pax_global_header00006660000000000000000000000064145701672130014520gustar00rootroot0000000000000052 comment=4471fba6a422f0f595da1bf211b4f72b12707b2f charliecloud-0.37/000077500000000000000000000000001457016721300141075ustar00rootroot00000000000000charliecloud-0.37/.github/000077500000000000000000000000001457016721300154475ustar00rootroot00000000000000charliecloud-0.37/.github/PERUSEME000066400000000000000000000110241457016721300166150ustar00rootroot00000000000000[This file is not called README because files named .github/README.* get picked up by GitHub and used as the main project README.] This directory defines our GitHub Actions test suite setup. The basic strategy is to start one “job” per builder; these run in parallel. Each job then cycles through several different configurations, which vary per builder. It is configured to “fail fast”, i.e., if one of the jobs fails, the others will be immediately cancelled. For example, we only run the quick test suite on one builder, but if it fails everything will stop and you still get notified quickly. The number of concurrent jobs is not clear to me, but I’ve seen 7 and the documentation [1] implies it’s at least 20 (though I assume there is some global limit for OSS projects too). Nominally, jobs are started from the left side of the list, so anything we think is likely to fail fast (e.g., the quick scope) should be leftward; in practice it seems to be random. We could add more matrix dimensions, but then we’d have to deal with ordering more carefully, and pass the Docker cache manually (or not use it for some things). [1]: https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration Conventions: * We install everything to start, then uninstall as needed for more bare-bones tests. * For the “extra things’ tests: * Docker is the fastest builder, so that’s where we put extra things. * We need to retain sudo for uninstalling stuff. * I could not figure out how to set a boolean variable for use in “if” conditions. (I *did* get an environment variable to work, but not using our set/unset convention, rather the strings “true” and “false”. This seemed error-prone.) Therefore the extra things tests all use the full expression. Miscellaneous notes and gotchas: * Runner specs (as of 2020-11-25), nominal: Azure Standard_DS2_v2 virtual machine: 2 vCPUs, 7 GiB RAM, 15 GiB SSD storage. The OS image is bare-bones but there is a lot of software installed in third-party locations [1]. Looking at the actual VM provisioned, the disk specs are a little different: it’s got an 84GiB root filesystem mounted, and another 9GiB mounted on /mnt. With a little deleting, maybe we can make room for a full-scope test. It does seem to boot faster than Travis; overall performance is worse; but total test time is lower (Travis took 50–55 minutes to complete a passing build). [1]: https://github.com/actions/virtual-environments/blob/ubuntu20/20201116.1/images/linux/Ubuntu2004-README.md * The default shell (Bash) does not read any init files [1], so you cannot configure it with e.g. .bashrc. [1]: https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#using-a-specific-shell * GitHub doesn’t seem to notice our setup if .github is a symlink. :( * GitHub seems to want us to encapsulate some of the steps that are now just shell scripts into “actions”. I haven’t looked into this. Issue #914. * Force-push does start a new build. * Commands in “run” blocks aren’t logged by default; you need “set -x” if you want to see them. However there seems to be a race condition, so the commands and their output aren’t always interleaved correctly. * “docker” does not require “sudo”. * There are several places where we configure, make, make install. These need to be kept in sync. Perhaps there is an opportunity for an “Action” here? But the configure output validation varies. * The .github directory doesn’t have a Makefile.am; the files are listed in the root Makefile.am. * Most variables are strings. It’s easy to get into a situation where you set a variable to “false” but it’s the string “false” so it’s true. * Viewing step output is glitchy: * While the job is in progress, sometimes the step headings are links and sometimes they aren’t. * If it does work, you can’t scroll back to the start. * The “in progress” throbber seems to often be on the wrong heading. * When it’s over, sometimes clicking on a heading opens it but the content is blank; in this case, clicking a different job and coming back seems to fix things. * Previously we listed $CH_TEST_TARDIR and $CH_TEST_IMGDIR between phases. I didn’t transfer that over. It must have been useful, so let’s pay attention to see if it needs to be re-added. charliecloud-0.37/.github/workflows/000077500000000000000000000000001457016721300175045ustar00rootroot00000000000000charliecloud-0.37/.github/workflows/main.yml000066400000000000000000000473351457016721300211670ustar00rootroot00000000000000name: test suite on: pull_request: # all pull requests push: branches: [ master ] # all commits on master schedule: - cron: '0 2 * * 0' # every Sunday at 2:00am UTC (Saturday 7:00pm MST) jobs: main: runs-on: ubuntu-22.04 timeout-minutes: 90 strategy: fail-fast: false # Actions seems kind of flaky lately matrix: builder: - ch-image pack_fmt: [squash-mount, tar-unpack, squash-unpack] keep_sudo: # if false, remove self from sudoers post-install/setup - false cache: - enabled include: - builder: none pack_fmt: squash-mount keep_sudo: false cache: disabled - builder: docker pack_fmt: tar-unpack keep_sudo: true cache: enabled - builder: ch-image pack_fmt: squash-mount keep_sudo: false cache: disabled env: CH_IMAGE_CACHE: ${{ matrix.cache }} CH_TEST_BUILDER: ${{ matrix.builder }} CH_TEST_TARDIR: /mnt/tarballs CH_TEST_IMGDIR: /mnt/images CH_TEST_PACK_FMT: ${{ matrix.pack_fmt }} CH_TEST_PERMDIRS: /mnt/perms_test /run/perms_test ch_prefix: /var/tmp steps: - uses: actions/checkout@v3 - name: early setup & validation run: | [[ -n $CH_TEST_BUILDER ]] sudo timedatectl set-timezone America/Denver sudo chmod 1777 /mnt /usr/local/src echo "ch_makej=-j$(getconf _NPROCESSORS_ONLN)" >> $GITHUB_ENV # Remove sbin directories from $PATH (see issue #43). Assume none of # these are the first entry in $PATH. echo "PATH=$PATH" path_new=$PATH for i in /sbin /usr/sbin /usr/local/sbin; do path_new=${path_new/:$i/} done echo "path_new=$path_new" echo "PATH=$path_new" >> $GITHUB_ENV # Set sudo umask to something quite restrictive. The default is # 0022, but we had a "make install" bug (issue #947) that was # tickled by 0027, which is a better setting. For reasons I don’t # understand, this only affects sudo, but since that’s what we want, # I left it. Note there are a few dependency installs below that # have similar permissions bugs; these relax the umask on a # case-by-case basis. sudo sed -i -E 's/^UMASK\s+022/UMASK 0077/' /etc/login.defs fgrep UMASK /etc/login.defs - name: print starting environment run: | echo builder: ${{ matrix.builder }} echo pack_fmt: ${{ matrix.pack_fmt }} echo keep_sudo: ${{ matrix.keep_sudo }} echo cache: ${{ matrix.cache }} uname -a lsb_release -a id pwd getconf _NPROCESSORS_ONLN free -m df -h locale -a timedatectl env | egrep '^(PATH|USER)=' env | egrep '^(ch|CH)_' [[ $PATH != */usr/local/sbin* ]] # verify sbin removal; see above printf 'umask for %s: ' $USER && umask printf 'umask under sudo: ' && sudo sh -c umask [[ $(umask) = 0022 ]] [[ $(sudo sh -c umask) = 0077 ]] - name: lines of code if: ${{ matrix.builder == 'none' }} run: | sudo apt-get install cloc misc/loc - name: install/configure dependencies, all run: | # configure doesn’t tell us about these. sudo apt-get install attr pigz pv # configure does tell us about these. sudo apt-get install bats squashfs-tools # Track newest Sphinx in case it breaks things. # FIXME: pip yells about docutils 0.19 so ask for latest 0.18 sudo su -c 'umask 0022 && pip3 install docutils==0.18.1 sphinx sphinx-rtd-theme sphinx-reredirects' # Use newest shellcheck. cd /usr/local/src wget -nv https://github.com/koalaman/shellcheck/releases/download/stable/shellcheck-stable.linux.x86_64.tar.xz tar xf shellcheck-stable.linux.x86_64.tar.xz sudo mv shellcheck-stable/shellcheck /usr/local/bin which shellcheck shellcheck --version - name: install/configure dependencies, ch-image if: ${{ matrix.builder == 'ch-image' }} run: | # Use most current packages b/c new versions sometimes break things. sudo sh -c 'umask 0022 && pip3 install lark requests wheel' # Configure Git and friends for build cache. if [[ $CH_IMAGE_CACHE = enabled ]]; then # Git command -v git git --version git config --global user.name 'Judit Polgár' git config --global user.email judit@example.com git config --global core.excludesfile ~/.gitignore echo __ch-test_ignore__ >> ~/.gitignore # Graphviz sudo apt-get install -y graphviz dot -V # git2dot sudo sh -c 'umask 0022 && pip install python-dateutil' git clone https://github.com/jlinoff/git2dot.git ~/git2dot cd ~/git2dot git checkout $(git tag | sort -Vr | head -1) sudo cp -a git2dot.py /usr/local/bin command -v git2dot.py git2dot.py --version fi - name: install libsquashfuse if: ${{ matrix.pack_fmt == 'squash-mount' }} run: | set -x sudo apt-get install -y libfuse3-dev cd /usr/local/src git clone https://github.com/vasi/squashfuse.git cd squashfuse # Check out the latest release. git checkout $(git tag -l '[0-9].*' | sort -V | tail -1) ./autogen.sh ./configure --prefix=/usr/local make sudo sh -c 'umask 0022 && make install' sudo ldconfig # Unset setuid on all fusermount3(1) on the system. sudo chmod -v u-s /usr/bin/fusermount* ls -lh $(command -v fusermount3) [[ ! -u $(command -v fusermount3) ]] - name: install musl and friends if: ${{ matrix.pack_fmt == 'tar-unpack' }} run: | sudo apt-get install -y musl-tools # At present we use argp to parse command line arguments. This is # not portable; it’s a glibc extension (see #1260). Thus, install a # standalone argp. cd /usr/local/src git clone --depth=1 https://github.com/ericonr/argp-standalone.git cd argp-standalone autoreconf --force --install CC=musl-gcc ./configure make $ch_makej sudo install -D -m644 argp.h /usr/include/x86_64-linux-musl/argp.h sudo install -D -m755 libargp.a /usr/lib/x86_64-linux-musl/libargp.a # musl-gcc does not install Linux headers. This is a bad kludge. # See: https://www.openwall.com/lists/musl/2017/11/23/1 sudo ln -s ../linux ../asm-generic ../x86_64-linux-gnu \ /usr/include/x86_64-linux-musl/ sudo ln -s x86_64-linux-gnu/asm /usr/include/x86_64-linux-musl/ - name: disable bundled lark, ch-image if: ${{ matrix.builder == 'ch-image' && matrix.pack_fmt == 'squash-mount' && matrix.cache == 'disabled' }} run: | set -x # no install, disable ./autogen.sh --no-lark ./configure --disable-bundled-lark --prefix=$ch_prefix/from-git fgrep '"lark" module ... external' config.log ! test -f ./lib/lark/lark.py make $ch_makej sudo make $ch_makej install [[ $(bin/ch-image -v --dependencies) = 'lark path: /usr/local/lib/python3.10/dist-packages/lark/__init__.py' ]] [[ $($ch_prefix/from-git/bin/ch-image -v --dependencies) = 'lark path: /usr/local/lib/python3.10/dist-packages/lark/__init__.py' ]] make clean # install, disable ./autogen.sh ./configure --disable-bundled-lark --prefix=$ch_prefix/from-git fgrep '"lark" module ... bundled' config.log test -f ./lib/lark/lark.py make $ch_makej sudo make $ch_makej install [[ $(bin/ch-image -v --dependencies) = 'lark path: /home/runner/work/charliecloud/charliecloud/lib/lark/__init__.py' ]] [[ $($ch_prefix/from-git/bin/ch-image -v --dependencies) = 'lark path: /usr/local/lib/python3.10/dist-packages/lark/__init__.py' ]] make clean # no install, enable: build fails, untested # install, enable: normal configuration, remainder of CI - name: build/install from Git run: | ./autogen.sh # Remove Autotools to make sure everything works without them. sudo apt-get remove autoconf automake # Configure and verify output. # # We want to support any standard libc, so build against musl for # some cases: # # squash-mount: glibc; libsquashfuse and libfuse3 are built with it # squash-unpack: glibc; to test ch-fromhost which assumes glibc # tar-unpack: musl # # See PR #1258, which is a FTBFS for musl. Other build steps in this # workflow also use glibc. if [[ $CH_TEST_PACK_FMT == tar-unpack ]]; then export CC=musl-gcc # GNU ldd(1) doesn’t work on musl binaries. LDD='/lib/ld-musl-x86_64.so.1 --list' else LDD=ldd fi # Go wild with install directories to make sure it works (see #683). # We do have a normal install below. ./configure --prefix=$ch_prefix/from-git \ --datarootdir=$ch_prefix/from-git_dataroot \ --libexecdir=$ch_prefix/from-git_libexec \ --libdir=$ch_prefix/from-git_lib \ --mandir=$ch_prefix/from-git_man \ --docdir=$ch_prefix/from-git_doc \ --htmldir=$ch_prefix/from-git_html set -x fgrep 'documentation: yes' config.log [[ $CH_TEST_BUILDER = docker ]] && command -v docker if [[ $CH_TEST_BUILDER = ch-image ]]; then fgrep 'with ch-image(1): yes' config.log fgrep '"lark" module ... bundled' config.log test -f ./lib/lark/lark.py fi fgrep 'recommended tests, tar-unpack mode: yes' config.log fgrep 'recommended tests, squash-unpack mode: yes' config.log if [[ $CH_TEST_PACK_FMT = squash-mount ]]; then fgrep 'recommended tests, squash-mount mode: yes' config.log fgrep 'internal SquashFS mounting ... yes' config.log else fgrep 'recommended tests, squash-mount mode: no' config.log fgrep 'internal SquashFS mounting ... no' config.log fi set +x # Build and install. make $ch_makej sudo make $ch_makej install $LDD bin/ch-run bin/ch-run --version $LDD $ch_prefix/from-git/bin/ch-run $ch_prefix/from-git/bin/ch-run --version # Ensure bundled Lark in tarball. make $ch_makej dist tar tf charliecloud-*.tar.gz | fgrep /lib/lark/lark.py - name: late setup & validation, ch-image if: ${{ matrix.builder == 'ch-image' }} run: | set -x [[ $(bin/ch-image gestalt python-path) = /usr/bin/python3 ]] [[ $(bin/ch-image -v --dependencies) = "lark path: /home/runner/work/charliecloud/charliecloud/lib/lark/__init__.py" ]] [[ $($ch_prefix/from-git/bin/ch-image gestalt python-path) = /usr/bin/python3 ]] [[ $($ch_prefix/from-git/bin/ch-image -v --dependencies) = "lark path: /var/tmp/from-git_lib/charliecloud/lark/__init__.py" ]] - name: late setup & validation, all run: | bin/ch-test --is-pedantic all bin/ch-test --is-sudo all - name: make filesystem permissions fixtures run: | bin/ch-test mk-perm-dirs - name: start local registry, ch-image if: ${{ matrix.builder == 'ch-image' }} # See HOWTO in Google Docs for details. run: | set -x mkdir ~/registry-etc cp test/registry-config.yml ~/registry-etc/config.yml cd ~/registry-etc openssl req -batch -subj '/C=US/ST=NM/L=LA/O=LANL/CN=localhost' \ -newkey rsa:4096 -nodes -sha256 -x509 -days 36500 \ -keyout localhost.key -out localhost.crt #openssl x509 -text -noout -in localhost.crt sudo apt-get install apache2-utils htpasswd -Bbn charlie test > ./htpasswd diff -u <(docker run --rm registry:2 \ cat /etc/docker/registry/config.yml) \ config.yml || true docker run -d --rm -p 127.0.0.1:5000:5000 \ -v ~/registry-etc:/etc/docker/registry registry:2 docker ps -a - name: build/install from tarball run: | # Create and unpack tarball. The wildcard saves us having to put the # version in a variable. This assumes there isn’t already a tarball # or unpacked directory in $ch_prefix, which is true on the clean # VMs GitHub gives us. Note that cd fails if it gets more than one # argument, which helps, but this is probably kind of brittle. make $ch_makej dist mv charliecloud-*.tar.gz $ch_prefix cd $ch_prefix tar xf charliecloud-*.tar.gz rm charliecloud-*.tar.gz # else cd fails with “too many arguments” cd charliecloud-* pwd # Configure and verify output. ./configure --prefix=$ch_prefix/from-tarball set -x fgrep 'documentation: yes' config.log [[ $CH_TEST_BUILDER = buildah* ]] && command -v buildah [[ $CH_TEST_BUILDER = docker ]] && command -v docker [[ $CH_TEST_BUILDER = ch-image ]] && fgrep 'with ch-image(1): yes' config.log fgrep 'recommended tests, tar-unpack mode: yes' config.log fgrep 'recommended tests, squash-unpack mode: yes' config.log if [[ ${{ matrix.pack_fmt }} = squash-mount ]]; then fgrep 'recommended tests, squash-mount mode: yes' config.log fgrep 'internal SquashFS mounting ... yes' config.log else fgrep 'recommended tests, squash-mount mode: no' config.log fgrep 'internal SquashFS mounting ... no' config.log fi set +x # Build and install. make $ch_makej sudo make $ch_makej install bin/ch-run --version $ch_prefix/from-tarball/bin/ch-run --version - name: configure sudo to user root, group non-root if: ${{ matrix.keep_sudo }} run: | sudo sed -Ei 's/=\(ALL\)/=(ALL:ALL)/g' /etc/sudoers.d/runner sudo cat /etc/sudoers.d/runner - name: remove sudo if: ${{ ! matrix.keep_sudo }} run: | sudo rm /etc/sudoers.d/runner ! sudo echo hello - name: run test suite (Git WD, standard) run: | bin/ch-test all # Validate that “rootemu” test phase didn’t run (skipped by default # on standard scope). [[ $(cat /tmp/ch-test.tmp.$USER/rootemu) = no ]] - name: run root emulation phase if: ${{ matrix.builder == 'ch-image' && matrix.pack_fmt == 'squash-mount' && matrix.cache == 'enabled' }} run: | # We only really need to run the root emulation tests once in CI. # ch-image with an enabled cache is required for these unit tests. # Since all CI tests that use ch-image with an enabled cache take # roughly the same amount of time, we arbitrarily chose squash-mount # as the pack format. bin/ch-test rootemu # Validate that “rootemu” test phase ran. [[ $(cat /tmp/ch-test.tmp.$USER/rootemu) = yes ]] - name: run test suite (installed from Git WD, standard) if: ${{ matrix.builder == 'ch-image' && matrix.pack_fmt == 'squash-mount' && matrix.cache == 'enabled' }} run: | $ch_prefix/from-git/bin/ch-test all [[ $(cat /tmp/ch-test.tmp.$USER/rootemu) = no ]] - name: run test suite (installed from tarball, standard) if: ${{ matrix.builder == 'ch-image' && matrix.pack_fmt == 'squash-mount' && matrix.cache == 'enabled' }} run: | $ch_prefix/from-tarball/bin/ch-test all [[ $(cat /tmp/ch-test.tmp.$USER/rootemu) = no ]] - name: uninstall from tarball if: ${{ matrix.keep_sudo }} run: | cd $ch_prefix/charliecloud-* sudo make uninstall diff -u - <(find $ch_prefix/from-tarball | sort) <<'EOF' /var/tmp/from-tarball /var/tmp/from-tarball/bin /var/tmp/from-tarball/lib /var/tmp/from-tarball/libexec /var/tmp/from-tarball/share /var/tmp/from-tarball/share/doc /var/tmp/from-tarball/share/man /var/tmp/from-tarball/share/man/man1 /var/tmp/from-tarball/share/man/man7 EOF - name: rebuild with most things --disable’d if: ${{ matrix.builder == 'docker' }} run: | make distclean ./configure --prefix=/doesnotexist \ --disable-html --disable-man --disable-ch-image set -x fgrep 'HTML documentation ... no' config.log fgrep 'man pages ... no' config.log fgrep 'ch-image(1) ... no' config.log fgrep 'with ch-image(1): no' config.log # This CI test requires Docker, but the “configure” script doesn’t # check for Docker, so we do it ourselves with “command -v”. command -v docker fgrep 'more complete tests: no' config.log set +x # Build. make $ch_makej bin/ch-run --version - name: run test suite (Git WD, standard) if: ${{ matrix.builder == 'docker' }} run: | bin/ch-test all [[ $(cat /tmp/ch-test.tmp.$USER/rootemu) = no ]] - name: remove non-essential dependencies if: ${{ matrix.builder == 'docker' }} run: | set -x # This breaks lots of dependencies unrelated to our build but YOLO. sudo dpkg --remove --force-depends \ pigz \ pv \ python3-requests \ squashfs-tools sudo pip3 uninstall -y sphinx sphinx-rtd-theme if [[ ${{ matrix.pack_fmt }} = squash-mount ]]; then ( cd /usr/local/src/squashfuse && sudo make uninstall ) fi ! python3 -c 'import requests' ! python3 -c 'import lark' - name: rebuild if: ${{ matrix.builder == 'docker' }} run: | make distclean ./configure --prefix=/doesnotexist set -x fgrep 'documentation: no' config.log command -v docker fgrep 'more complete tests: no' config.log fgrep 'recommended tests, squash-mount mode: no' config.log set +x # Build and install. make $ch_makej bin/ch-run --version - name: run test suite (Git WD, standard) if: ${{ matrix.builder == 'docker' }} run: | bin/ch-test all [[ $(cat /tmp/ch-test.tmp.$USER/rootemu) = no ]] - name: print ending environment if: ${{ always() }} run: | free -m df -h du -sch $CH_TEST_TARDIR/* || true du -sch $CH_TEST_IMGDIR/* || true charliecloud-0.37/LICENSE000066400000000000000000000261361457016721300151240ustar00rootroot00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. charliecloud-0.37/Makefile.am000066400000000000000000000022741457016721300161500ustar00rootroot00000000000000SUBDIRS = lib bin doc examples misc packaging test # The CI stuff isn't really relevant for the tarballs, but they should # have complete source code. EXTRA_DIST = .github/PERUSEME .github/workflows/main.yml EXTRA_DIST += LICENSE README.rst VERSION autogen.sh # Embedded paths are in the source code suitable for running from the source # directory (i.e., without install). When installing, those paths are often # wrong, so re-write them with the correct paths we got from configure. Note: # Some variables are in both Python and sh, so we use syntax valid for both; # others are just sh. install-exec-hook: @echo '### re-writing embedded paths ###' for i in $(DESTDIR)@bindir@/ch-convert \ $(DESTDIR)@bindir@/ch-fromhost \ $(DESTDIR)@bindir@/ch-image \ $(DESTDIR)@bindir@/ch-run-oci \ $(DESTDIR)@bindir@/ch-test \ $(DESTDIR)@libdir@/charliecloud/base.sh \ $(DESTDIR)@libexecdir@/charliecloud/doctest; \ do \ sed -Ei -e 's|^(ch_lib ?= ?).+/lib"?$$|\1"@libdir@/charliecloud"|' \ -e 's|^(CHTEST_DIR=).+$$|\1@libexecdir@/charliecloud|' \ -e 's|^(CHTEST_EXAMPLES_DIR=).+$$|\1@docdir@/examples|' \ $$i; \ done charliecloud-0.37/README.rst000066400000000000000000000147071457016721300156070ustar00rootroot00000000000000What is Charliecloud? --------------------- Charliecloud provides user-defined software stacks (UDSS) for high-performance computing (HPC) centers. This “bring your own software stack” functionality addresses needs such as: * software dependencies that are numerous, complex, unusual, differently configured, or simply newer/older than what the center provides; * build-time requirements unavailable within the center, such as relatively unfettered internet access; * validated software stacks and configuration to meet the standards of a particular field of inquiry; * portability of environments between resources, including workstations and other test and development system not managed by the center; * consistent environments, even archivally so, that can be easily, reliably, and verifiably reproduced in the future; and/or * usability and comprehensibility. How does it work? ----------------- Charliecloud uses Linux user namespaces to run containers with no privileged operations or daemons and minimal configuration changes on center resources. This simple approach avoids most security risks while maintaining access to the performance and functionality already on offer. Container images can be built using Docker or anything else that can generate a standard Linux filesystem tree. How do I learn more? -------------------- * Documentation: https://hpc.github.io/charliecloud * GitHub repository: https://github.com/hpc/charliecloud * Low-traffic mailing list for announcements: https://groups.io/g/charliecloud * We wrote an article for USENIX's magazine *;login:* that explains in more detail the motivation for Charliecloud and the technology upon which it is based: https://www.usenix.org/publications/login/fall2017/priedhorsky * For technical papers about Charliecloud, refer to the *Technical publications* section below. Who is responsible? ------------------- Contributors: * Richard Berger * Lucas Caudill * Rusty Davis * Hunter Easterday * Oliver Freyermuth * Shane Goff * Michael Jennings * Christoph Junghans * Dave Love * Jordan Ogas * Kevin Pelzel * Megan Phinney * Reid Priedhorsky , co-founder and project lead * Tim Randles , co-founder * Benjamin "The Storm" Stormer * Meisam Tabriz * Matthew Vernon * Peter Wienemann * Lowell Wofford How can I participate? ---------------------- Use our GitHub page: https://github.com/hpc/charliecloud Bug reports and feature requests should be filed as “Issues”. Questions, comments, support requests, and everything else should use our “Discussions”. Don't worry if you put something in the wrong place; we’ll be more than happy to help regardless. We also have a mailing list for announcements: https://groups.io/g/charliecloud Patches are much appreciated on the software itself as well as documentation. Optionally, please include in your first patch a credit for yourself in the list above. We are friendly and welcoming of diversity on all dimensions. Technical publications ---------------------- If Charliecloud helped your research, or it was useful to you in any other context where bibliographic citations are appropriate, please cite the following open-access paper: Reid Priedhorsky and Tim Randles. “Charliecloud: Unprivileged containers for user-defined software stacks in HPC”, 2017. In *Proc. Supercomputing*. DOI: `10.1145/3126908.3126925 `_. *Note:* This paper contains out-of-date number for the size of Charliecloud’s code. Please instead use the current number in the FAQ. Other publications: * We compare the performance of three HPC-specific container technologies against bare metal, finding no concerns about performance degradation. Alfred Torrez, Tim Randles, and Reid Priedhorsky. “HPC container runtimes have minimal or no performance impact”, 2019. In *Proc. CANOPIE HPC Workshop @ SC*. DOI: `10.1109/CANOPIE-HPC49598.2019.00010 `_. * A demonstration of how low-privilege containers solve increasing demand for software flexibility. Reid Priedhorsky, R. Shane Canon, Timothy Randles, and Andrew J. Younge. “Minimizing privilege for building HPC containers”, 2021. In *Proc. Supercomputing*. DOI: `10.6084/m9.figshare.14396099 `_. * Charliecloud’s build cache performs competitively with the standard many-layered union filesystem approach and has structural advantages including a better diff format, lower cache overhead, and better file de-duplication. Reid Priedhorsky, Jordan Ogas, Claude H. (Rusty) Davis IV, Z. Noah Hounshel, Ashlyn Lee, Benjamin Stormer, and R. Shane Goff. "Charliecloud’s layer-free, Git-based container build cache", 2023. In *Proc. Supercomputing*. DOI: `10.1145/3624062.3624585 `_. Copyright and license --------------------- Charliecloud is copyright © 2014–2023 Triad National Security, LLC and others. This software was produced under U.S. Government contract 89233218CNA000001 for Los Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC for the U.S. Department of Energy/National Nuclear Security Administration. This is open source software (LA-CC 14-096); you can redistribute it and/or modify it under the terms of the Apache License, Version 2.0. A copy is included in file LICENSE. You may not use this software except in compliance with the license. The Government is granted for itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide license in this material to reproduce, prepare derivative works, distribute copies to the public, perform publicly and display publicly, and to permit others to do so. Neither the government nor Triad National Security, LLC makes any warranty, express or implied, or assumes any liability for use of this software. If software is modified to produce derivative works, such derivative works should be clearly marked, so as not to confuse it with the version available from LANL. .. LocalWords: USENIX's CNA Meisam figshare charliecloud-0.37/VERSION000066400000000000000000000000051457016721300151520ustar00rootroot000000000000000.37 charliecloud-0.37/autogen.sh000077500000000000000000000066171457016721300161220ustar00rootroot00000000000000#!/bin/bash set -e lark_version=1.1.9 while [[ "$#" -gt 0 ]]; do case $1 in --clean) clean=yes ;; --no-lark) lark_no_install=yes ;; --rm-lark) lark_shovel=yes ;; *) help=yes ;; esac shift done if [[ $help ]]; then cat <&1 echo 'hint: Install "wheel" and then re-run with "--rm-lark"?' 2>&1 exit 1 fi set +x echo echo 'Done. Now you can "./configure".' fi charliecloud-0.37/bin/000077500000000000000000000000001457016721300146575ustar00rootroot00000000000000charliecloud-0.37/bin/Makefile.am000066400000000000000000000017321457016721300167160ustar00rootroot00000000000000# Bugs in this Makefile: # # 1. $(EXEEXT) not included for scripts. ## C programs bin_PROGRAMS = ch-checkns ch-run ch_checkns_SOURCES = ch-checkns.c ch_misc.h ch_misc.c ch_run_SOURCES = ch-run.c ch_core.h ch_core.c ch_misc.h ch_misc.c if HAVE_LIBSQUASHFUSE ch_run_SOURCES += ch_fuse.h ch_fuse.c endif ch_run_CFLAGS = $(CFLAGS) $(PTHREAD_CFLAGS) ch_run_LDADD = $(CH_RUN_LIBS) ## Shell scripts - distributed as-is dist_bin_SCRIPTS = ch-convert \ ch-fromhost \ ch-test ## Python scripts - need text processing bin_SCRIPTS = ch-run-oci # scripts to build EXTRA_SCRIPTS = ch-image # more scripts that *may* be built if ENABLE_CH_IMAGE bin_SCRIPTS += ch-image endif EXTRA_DIST = ch-image.py.in ch-run-oci.py.in CLEANFILES = $(bin_SCRIPTS) $(EXTRA_SCRIPTS) ch-image: ch-image.py.in ch-run-oci: ch-run-oci.py.in $(bin_SCRIPTS): %: %.py.in rm -f $@ sed -E 's|%PYTHON_SHEBANG%|@PYTHON_SHEBANG@|' < $< > $@ chmod +rx,-w $@ # respects umask charliecloud-0.37/bin/ch-checkns.c000066400000000000000000000137011457016721300170330ustar00rootroot00000000000000/* This example program walks through the complete namespace / pivot_root(2) dance to enter a Charliecloud container, with each step documented. If you can compile it and run it without error as a normal user, ch-run will work too (if not, that's a bug). If not, this will hopefully help you understand more clearly what went wrong. pivot_root(2) has a large number of error conditions resulting in EINVAL that are not documented in the man page [1]. The ones we ran into are: 1. The new root cannot be shared [2] outside the mount namespace. This makes sense, as we as an unprivileged user inside our namespace should not be able to change privileged things owned by other namespaces. This condition arises on systemd systems, which mount everything shared by default. 2. The new root must not have been mounted before unshare(2), and/or it must be a mount point. The man page says "new_root does not have to be a mount point", but the source code comment says "[i]t must be a mount point" [3]. (I haven't isolated which was our problem.) In either case, this is a very common situation. 3. The old root is a "rootfs" [4]. This is documented in a source code comment [3] but not the man page. This is an unusual situation for most contexts, because the rootfs is typically the initramfs overmounted during boot. However, some cluster provisioning systems, e.g. Perceus, use the original rootfs directly. Regarding overlayfs: It's very attractive to union-mount a tmpfs over the read-only image; then all programs can write to their hearts' desire, and the image does not change. This also simplifies the code. Unfortunately, overlayfs + userns is not allowed as of 4.4.23. See: https://lwn.net/Articles/671774/ [1]: http://man7.org/linux/man-pages/man2/pivot_root.2.html [2]: https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt [3]: http://lxr.free-electrons.com/source/fs/namespace.c?v=4.4#L2952 [4]: https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt */ #define _GNU_SOURCE #include #include #include #include #include #include #include #include #include #include "config.h" #include "ch_misc.h" const char usage[] = "\ \n\ Usage: ch-checkns\n\ \n\ Check \"ch-run\" prerequisites, e.g., namespaces and \"pivot_root(2)\".\n\ \n\ Example:\n\ \n\ $ ch-checkns\n\ ok\n"; #define TRY(x) if (x) fatal_(__FILE__, __LINE__, errno, #x) void fatal_(const char *file, int line, int errno_, const char *str) { char *url = "https://github.com/hpc/charliecloud/blob/master/bin/ch-checkns.c"; printf("error: %s: %d: %s\n", file, line, str); printf("errno: %d\nsee: %s\n", errno_, url); exit(1); } int main(int argc, char *argv[]) { unsigned long flags; if (argc >= 2 && strcmp(argv[1], "--help") == 0) { fprintf(stderr, usage); return 0; } if (argc >= 2 && strcmp(argv[1], "--version") == 0) { version(); return 0; } /* Ensure that our image directory exists. It doesn't really matter what's in it. */ if (mkdir("/tmp/newroot", 0755) && errno != EEXIST) TRY (errno); /* Enter the mount and user namespaces. Note that in some cases (e.g., RHEL 6.8), this will succeed even though the userns is not created. In that case, the following mount(2) will fail with EPERM. */ TRY (unshare(CLONE_NEWNS|CLONE_NEWUSER)); /* Claim the image for our namespace by recursively bind-mounting it over itself. This standard trick avoids conditions 1 and 2. */ TRY (mount("/tmp/newroot", "/tmp/newroot", NULL, MS_REC | MS_BIND | MS_PRIVATE, NULL)); /* The next few calls deal with condition 3. The solution is to overmount the root filesystem with literally anything else. We use the parent of the image, /tmp. This doesn't hurt if / is not a rootfs, so we always do it for simplicity. */ /* Claim /tmp for our namespace. You would think that because /tmp contains /tmp/newroot and it's a recursive bind mount, we could claim both in the same call. But, this causes pivot_root(2) to fail later with EBUSY. */ TRY (mount("/tmp", "/tmp", NULL, MS_REC | MS_BIND | MS_PRIVATE, NULL)); /* chdir to /tmp. This moves the process' special "." pointer to the soon-to-be root filesystem. Otherwise, it will keep pointing to the overmounted root. See the e-mail at the end of: https://git.busybox.net/busybox/tree/util-linux/switch_root.c?h=1_24_2 */ TRY (chdir("/tmp")); /* Move /tmp to /. (One could use this to directly enter the image, avoiding pivot_root(2) altogether. However, there are ways to remove all active references to the root filesystem. Then, the image could be unmounted, exposing the old root filesystem underneath. While Charliecloud does not claim a strong isolation boundary, we do want to make activating the UDSS irreversible.) */ TRY (mount("/tmp", "/", NULL, MS_MOVE, NULL)); /* Move the "/" special pointer to the new root filesystem, for the reasons above. (Similar reasoning applies for why we don't use chroot(2) to directly activate the UDSS.) */ TRY (chroot(".")); /* Make a place for the old (intermediate) root filesystem to land. */ if (mkdir("/newroot/oldroot", 0755) && errno != EEXIST) TRY (errno); /* Re-mount the image read-only. */ flags = path_mount_flags("/newroot") | MS_REMOUNT | MS_BIND | MS_RDONLY; TRY (mount(NULL, "/newroot", NULL, flags, NULL)); /* Finally, make our "real" newroot into the root filesystem. */ TRY (chdir("/newroot")); TRY (syscall(SYS_pivot_root, "/newroot", "/newroot/oldroot")); TRY (chroot(".")); /* Unmount the old filesystem and it's gone for good. */ TRY (umount2("/oldroot", MNT_DETACH)); /* Report success. */ printf("ok\n"); } charliecloud-0.37/bin/ch-completion.bash000066400000000000000000001047571457016721300202750ustar00rootroot00000000000000# Completion script for Charliecloud # SC2207 pops up whenever we do “COMPREPLY=( $(compgen [...]) )”. This seems to # be standard for implementations of bash completion, and we didn't like the # suggested alternatives, so we disable it here. # shellcheck disable=SC2207 # SC2034 complains about modifying variables by reference in # _ch_run_parse. Disable it. # shellcheck disable=SC2034 # Permissions for this file: # # This file needs to be sourced, not executed. Because of this, the execute bit # for the file shoud remain unset for all permission groups. # # (sourcing versus executing: https://superuser.com/a/176788) # Resources for understanding this script: # # * Everything bash: # https://www.gnu.org/software/bash/manual/html_node/index.html # # * Bash parameter expansion: # https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html # # * Bash completion builtins (compgen, comopt, etc.): # https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion-Builtins.html # # * Bash completion variables (e.g. COMPREPLY): # https://devmanual.gentoo.org/tasks-reference/completion/index.html # # * Call-by-reference for bash function args: # https://unix.stackexchange.com/a/224564 ## SYNTAX GLOSSARY ## # # Bash has some pretty unusual syntax, and this script has no shortage of # strange Bash-isms. I’m including this syntax glossary with the hope that it’ll # make this code more readable for Bash newbies and those who are rusty. For more # info, see the gnu.org “Bash parameter expansion” page linked above, which is also # the source for this glossary. # # ${array[i]} # Gives the ith element of “array”. Note that bash arrays are indexed at # zero, as all things should be. # # ${array[@]} # Expands “array” to its member elements as a sequence of words, one word # per element. # # ${#parameter} # Gives the length of “parameter”. If “parameter” is a string, this # expansion gives you the character length of the string. If “parameter” is # an array subscripted by “@” or “*” (e.g. “foo[@]”), then the expansion # gives you the number of elements in the array. # # ${parameter:offset:length} # A.k.a. substring expansion. If “parameter” is a string, expand up to # “length” characters, starting with the character at position “offset.” If # “offset” is unspecified, start at the first character. If “parameter” is # an array subscripted by “@” or “*,” (e.g. “foo[@]”) expand up to “length” # elements, starting at the element at position “offset” (e.g. # “${foo[offset]}”). # # Example 1 (string): # # $ foo="abcdef" # $ echo ${foo::3} # abc # $ echo ${foo:1:3} # bcd # # Example 2 (array): # # $ foo=("a" "b" "c" "d" "e" "f") # $ echo ${foo[@]::3} # a b c # $ echo ${foo[@]:1:3} # b c d # # ${parameter/pattern/string} # This is a form of pattern replacement in which “parameter” is expanded and # the first instance of “pattern” is replaced with “string”. # # ${parameter//pattern/string} # Similar to “${parameter/pattern/string}” above, except every instance of # “pattern” in the expanded parameter is replaced by “string” instead of only # the first. # ## Setup ## # According to this post (https://stackoverflow.com/a/50281697), Bash 4.3 alpha # added the feature that enables the use of out parameters for functions (or # passing variables by reference), which is an integral feature of this script. bash_vmin=4.3.0 # Check Bash version bash_v=$(bash --version | head -1 | grep -Eo "[0-9\.]{2,}[0-9]") if [[ $(printf "%s\n%s\n" "$bash_vmin" "$bash_v" | sort -V | head -1) != "$bash_vmin" ]]; then echo "ch-completion.bash: unsupported bash version ($bash_v < $bash_vmin)" return 1 fi # Check for bash completion, exit if not found. FIXME: #1640. if [[ -z "$(declare -f -F _get_comp_words_by_ref)" ]]; then if [[ -f /usr/share/bash-completion/bash_completion ]]; then . /usr/share/bash-completion/bash_completion elif [[ -f /etc/bash_completion ]]; then . /etc/bash_completion else echo "ch-completion.bash: dependency \"bash_completion\" not found, exiting" return 1 fi fi # https://stackoverflow.com/a/246128 _ch_completion_dir=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) _ch_completion_version="$("$_ch_completion_dir"/../misc/version)" _ch_completion_log="/tmp/ch-completion.log" # Record file being sourced. if [[ -n "$CH_COMPLETION_DEBUG" ]]; then printf "ch-completion.bash sourced\n\n" >> "$_ch_completion_log" fi _ch_completable_executables="ch-image ch-run ch-convert" ## ch-convert ## # Valid formats _convert_fmts="ch-image dir docker podman squash tar" # Options for ch-convert that accept args _convert_arg_opts="-i --in-fmt -o --out-fmt -s --storage --tmp" # All options for ch-convert _convert_opts="-h --help -n --dry-run --no-clobber --no-xattrs -v --verbose $_convert_arg_opts" # Completion function for ch-convert # _ch_convert_complete () { local prev local cur local fmt_in local fmt_out local words local opts_end=-1 local strg_dir local extras _get_comp_words_by_ref -n : cur prev words cword strg_dir=$(_ch_find_storage "${words[@]::${#words[@]}-1}") _ch_convert_parse "$strg_dir" "$cword" fmt_in fmt_out opts_end "${words[@]}" # Populate debug log _DEBUG "\$ ${words[*]}" _DEBUG " storage: dir: $strg_dir" _DEBUG " current: $cur" _DEBUG " previous: $prev" _DEBUG " input format: $fmt_in" _DEBUG " output format: $fmt_out" if [[ $opts_end != -1 ]]; then _DEBUG " input image: ${words[$opts_end]}" else _DEBUG " input image:" fi # Command line options if [[ ($opts_end == -1) || ($cword -lt $opts_end) ]]; then case "$prev" in -i|--in-fmt) COMPREPLY=( $(compgen -W "${_convert_fmts//$fmt_out/}" -- "$cur") ) return 0 ;; -o|--out-fmt) COMPREPLY=( $(compgen -W "${_convert_fmts//$fmt_in/}" -- "$cur") ) return 0 ;; -s|--storage|--tmp) # Avoid overzealous completion. E.g. if there’s only one subdir of the # current dir, this command completes to that dir even if $cur is empty # (i.e. the user hasn’t yet typed anything), which seems confusing for # the user. if [[ -n "$cur" ]]; then compopt -o nospace COMPREPLY=( $(compgen -d -S / -- "$cur") ) fi return 0 ;; *) # Not an option that requires an arg. COMPREPLY=( $(compgen -W "$_convert_opts" -- "$cur") ) ;; esac fi if [[ ($opts_end == -1) ]]; then # Input image not yet specified, complete potential input images. case "$fmt_in" in ch-image) COMPREPLY+=( $(compgen -W "$(_ch_list_images "$strg_dir")" -- "$cur") ) __ltrim_colon_completions "$cur" ;; dir) COMPREPLY+=( $(compgen -d -- "$cur") ) if [[ -n "$(compgen -d -- "$cur")" ]]; then compopt -o nospace fi return 0 ;; squash) COMPREPLY+=( $(_compgen_filepaths -X "!*.sqfs" "$cur") ) _space_filepath -X "!*.sqfs" "$cur" ;; tar) COMPREPLY+=( $(_compgen_filepaths -X "!*.tar.* !*tgz" "$cur") ) _space_filepath -X "!*.tar.* !*tgz" "$cur" ;; docker|podman) # We don’t attempt to complete in this case. return 0 ;; "") # No in fmt specified, could be anything COMPREPLY+=( $(_compgen_filepaths -X "!*.tar.* !*tgz !*.sqfs" "$cur") ) COMPREPLY+=( $(compgen -W "$(_ch_list_images "$strg_dir")" -- "$cur") ) _space_filepath -X "!*.tar.* !*tgz !*.sqfs" "$cur" __ltrim_colon_completions "$cur" return 0 ;; esac elif [[ ($cword -gt $opts_end) ]]; then # Input image has been specified and current word appears after it in # the command line. Assume we’re completing output image. If output # format COULD be dir, tar, or squash, complete valid directory paths. if ! _is_subword "$fmt_out" "ch-image docker podman"; then compopt -o nospace COMPREPLY+=( $(compgen -d -S / -- "$cur") ) fi return 0 fi return 0 } ## ch-image ## # Subcommands and options for ch-image # _image_build_opts="-b --bind --build-arg -f --file --force --force-cmd -n --dry-run --parse-only -t --tag" _image_common_opts="-a --arch --always-download --auth --break --cache --cache-large --dependencies -h --help --no-cache --no-lock --no-xattrs --profile --rebuild --password-many -q --quiet -s --storage --tls-no-verify -v --verbose --version --xattrs" _image_subcommands="build build-cache delete gestalt import list pull push reset undelete" # archs taken from ARCH_MAP in charliecloud.py _archs="amd64 arm/v5 arm/v6 arm/v7 arm64/v8 386 mips64le ppc64le s390x" # Completion function for ch-image # _ch_image_complete () { local prev local cur local cword local words local sub_cmd local strg_dir local extras= _get_comp_words_by_ref -n : cur prev words cword sub_cmd=$(_ch_image_subcmd_get "$cword" "${words[@]}") # To find the storage directory, we want to look at all the words in the # current command line except for the current word (“${words[$cword]}” # here). We do this to prevent unexpected behavior resulting from the # current word being incomplete. The bash syntax we use to accomplish this # is “"${array[@]::$i}" "${array[@]:$i+1:${#array[@]}-1}"” which is # analagous to “array[:i] + array[i+1:]” in Python, giving you all elements # of the array, except for the one at index “i”. The syntax glossary at the # top of this file gives a breakdown of the constituent elements of this # hideous expression. strg_dir=$(_ch_find_storage "${words[@]::$cword}" "${words[@]:$cword+1:${#array[@]}-1}") # Populate debug log _DEBUG "\$ ${words[*]}" _DEBUG " storage: dir: $strg_dir" _DEBUG " word index: $cword" _DEBUG " current: $cur" _DEBUG " previous: $prev" _DEBUG " sub command: $sub_cmd" # Common opts that take args # case "$prev" in -a|--arch) COMPREPLY=( $(compgen -W "host yolo $_archs" -- "$cur") ) return 0 ;; --break) # “--break” arguments take the form “MODULE:LINE”. Complete “MODULE:” # from python files in lib (we can’t complete line number). COMPREPLY=( $(compgen -S : -W "$(_compgen_py_libfiles)" -- "$cur") ) __ltrim_colon_completions compopt -o nospace return 0 ;; --cache-large) # This is just a user-specified number. Can’t autocomplete COMPREPLY=() return 0 ;; -s|--storage) # See comment about overzealous completion for the “--storage” option # under “_ch_convert_complete”. if [[ -n "$cur" ]]; then compopt -o nospace COMPREPLY=( $(compgen -d -S / -- "$cur") ) fi return 0 ;; esac case "$sub_cmd" in build) case "$prev" in # Go through a list of potential subcommand-specific opts to see if # $cur should be an argument. Otherwise, default to CONTEXT or any # valid option (common or subcommand-specific). -f|--file) COMPREPLY=( $(_compgen_filepaths "$cur") ) _space_filepath "$cur" return 0 ;; -t) # We can’t autocomplete a tag, so we're not even gonna try. COMPREPLY=() return 0 ;; *) # Autocomplete to context directory, common opt, or build-specific # opt --force can take “fakeroot” or “seccomp” as an argument, or # no argument at all. if [[ $prev == --force ]]; then extras+="$extras fakeroot seccomp" fi COMPREPLY=( $(compgen -W "$_image_build_opts $extras" -- "$cur") ) # By default, “complete” adds a space after each completed word. # This is incredibly inconvenient when completing directories and # filepaths, so we enable the “nospace” option. We want to make # sure that this option is only enabled if there are valid path # completions for $cur, otherwise spaces would never be added # after a completed word, which is also inconveninet. if [[ -n "$(compgen -d -S / -- "$cur")" ]]; then compopt -o nospace COMPREPLY+=( $(compgen -d -S / -- "$cur") ) fi ;; esac ;; build-cache) COMPREPLY=( $(compgen -W "--reset --gc --tree --dot" -- "$cur") ) ;; delete|list) if [[ "$sub_cmd" == "list" ]]; then if [[ "$prev" == "--undeletable" || "$prev" == "--undeleteable" || "$prev" == "-u" ]]; then COMPREPLY=( $(compgen -W "$(_ch_undelete_list "$strg_dir")" -- "$cur") ) return 0 fi extras+="$extras -l --long -u --undeletable" # If “cur” starts with “--undelete,” add “--undeleteable” (the less # correct version of “--undeletable”) to the list of possible # completions. if [[ ${cur::10} == "--undelete" ]]; then extras="$extras --undeleteable" fi fi COMPREPLY=( $(compgen -W "$(_ch_list_images "$strg_dir") $extras" -- "$cur") ) __ltrim_colon_completions "$cur" ;; gestalt) COMPREPLY=( $(compgen -W "bucache bucache-dot python-path storage-path" -- "$cur") ) ;; import) # Complete (1) directories and (2) files named like tarballs. COMPREPLY+=( $(_compgen_filepaths -X "!*.tar.* !*tgz" "$cur") ) if [[ ${#COMPREPLY} -gt 0 ]]; then compopt -o nospace fi ;; push) if [[ "$prev" == "--image" ]]; then compopt -o nospace COMPREPLY=( $(compgen -d -S / -- "$cur") ) return 0 fi COMPREPLY=( $(compgen -W "$(_ch_list_images "$strg_dir") --image" -- "$cur") ) __ltrim_colon_completions "$cur" ;; undelete) COMPREPLY=( $(compgen -W "$(_ch_undelete_list "$strg_dir")" -- "$cur") ) ;; '') # Only autocomplete subcommands if there's no subcommand present. COMPREPLY=( $(compgen -W "$_image_subcommands" -- "$cur") ) ;; esac # If we’ve made it this far, the last remaining option for completion is # common opts. COMPREPLY+=( $(compgen -W "$_image_common_opts" -- "$cur") ) return 0 } ## ch-run ## # Options for ch-run # _run_opts="-b --bind -c --cd --env-no-expand --feature -g --gid --home -j --join --join-pid --join-ct --join-tag -m --mount --no-passwd -q --quiet -s --storage --seccomp -t --private-tmp --set-env -u --uid --unsafe --unset-env -v --verbose -w --write -? --help --usage -V --version" _run_features="extglob seccomp squash" # args for the --feature option # Completion function for ch-run # _ch_run_complete () { local prev local cur local cword local words local strg_dir local extras= _get_comp_words_by_ref -n : cur prev words cword # See the comment above the first call to “_ch_find_storage” for an # explanation of the horrible syntax here. strg_dir=$(_ch_find_storage "${words[@]::$cword}" "${words[@]:$cword+1:${#array[@]}-1}") local cli_image local cmd_index=-1 _ch_run_parse "$strg_dir" "$cword" cli_image cmd_index "${words[@]}" # Populate debug log _DEBUG "\$ ${words[*]}" _DEBUG " storage: dir: $strg_dir" _DEBUG " word index: $cword" _DEBUG " current: $cur" _DEBUG " previous: $prev" _DEBUG " cli image: $cli_image" # Currently, we don’t try to suggest completions if you’re in the “command” # part of the ch-run CLI (i.e. entering commands to be run inside the # container). Implementing this *may* be possible, but doing so would likely # be absurdly complicated, so we don’t plan on it. if [[ $cmd_index != -1 && $cmd_index -lt $cword ]]; then COMPREPLY=() return 0 fi # Common opts that take args # case "$prev" in -b|--bind) COMPREPLY=() return 0 ;; -c|--cd) COMPREPLY=() return 0 ;; --feature) COMPREPLY=( $(compgen -W "$_run_features" -- "$cur") ) return 0 ;; -g|--gid) COMPREPLY=() return 0 ;; --join-pid) COMPREPLY=() return 0 ;; --join-ct) COMPREPLY=() return 0 ;; --join-tag) COMPREPLY=() return 0 ;; -m|--mount) compopt -o nospace COMPREPLY=( $(compgen -d -- "$cur") ) return 0 ;; -s|--storage) # See comment about overzealous completion for the “--storage” option # under “_ch_convert_complete”. if [[ -n "$cur" ]]; then compopt -o nospace COMPREPLY=( $(compgen -d -S / -- "$cur") ) fi return 0 ;; --set-env) extras+=$(compgen -f -- "$cur") ;; -u|--uid) COMPREPLY=() return 0 ;; --unset-env) COMPREPLY=() return 0 ;; esac if [[ -z $cli_image ]]; then # No image found in command line, complete dirs, tarfiles, and sqfs # archives COMPREPLY=( $(_compgen_filepaths -X "!*.sqfs" "$cur") ) # Complete images in storage. Note we don't use “ch-image list” here # because it can initialize an empty storage directory and we don't want # this script to have any such side effects. COMPREPLY+=( $(compgen -W "$(_ch_list_images "$strg_dir")" -- "$cur") ) __ltrim_colon_completions "$cur" fi _space_filepath -X "!*.sqfs" "$cur" COMPREPLY+=( $(compgen -W "$_run_opts $extras" -- "$cur") ) return 0 } ## Helper functions ## _ch_completion_help="Usage: ch-completion [ OPTION ] Utility function for Charliecloud tab completion. --disable disable tab completion for all Charliecloud executables --help show this help message --version check tab completion script version --version-ok check version compatibility between tab completion and Charliecloud executables " # Add debugging text to log file if CH_COMPLETION_DEBUG is specified. _DEBUG () { if [[ -n "$CH_COMPLETION_DEBUG" ]]; then #echo "$@" >> "$_ch_completion_log" printf "%s\n" "$@" >> "$_ch_completion_log" fi } # Utility function for Charliecloud tab completion that’s available to users. ch-completion () { while true; do case $1 in --disable) complete -r ch-convert complete -r ch-image complete -r ch-run ;; --help) printf "%s" "$_ch_completion_help" 1>&2 return 0 ;; --version) printf "%s\n" "$_ch_completion_version" 1>&2 ;; --version-ok) if _version_ok_ch_completion "ch-image"; then printf "version ok\n" 1>&2 return 0 else printf "ch-image: %s\n" "$(ch-image --version)" 1>&2 printf "ch-completion: %s\n" "$_ch_completion_version" 1>&2 printf "version incompatible!\n" 1>&2 return 1 fi ;; *) break ;; esac shift done } _completion_opts="--disable --help --version --version-ok" # Yes, the untility function needs completion too... # _ch_completion_complete () { local cur _get_comp_words_by_ref -n : cur COMPREPLY=( $(compgen -W "$_completion_opts" -- "$cur") ) return 0 } # Parser for ch-convert command line. Takes 6 arguments: # # 1.) A string representing the path to the storage directory. # # 2.) The current position (measured in words) of the cursor in the array # representing the command line (index starting at 0). # # 3.) An out parameter (explanation below). If “_ch_convert_parse” is able to # determine the format of the input image, it will pass that format back # to the caller as a string using this out parameter. There are two ways # that “_ch_convert_parse” can determine the input image format: # i.) If “-i” or “--in-fmt” is specified and is followed by a valid # image format, the out parameter will be set to a that format. # E.g. “ch-image”. # ii.) If the parser detects that an input image has been specified, # it will try to determine the format of that image. This does # not work for Docker or Podman images, and never will. # # 4.) Another out parameter. If the user has specified an output image format # using “-o” or “--out-fmt”, the parser will use this out parameter to # pass that format back to the caller. # # 5.) A string representing the expanded command line array (i.e. # "${array[@]}"). # # “Out parameter” here refers to a variable that is meant to pass information # from this function to its caller (here the “_ch_chonvert_complete” function). # Out parameters should be passed to a bash function as the unquoted names of # variables (e.g. “var” instead of “$var” or “"$var"”) within the caller’s # scope. Passing the variables to the function in this way allows it to change # their values, and for those changes to persist in the scope that called the # function (this is what makes them “out parameters”). # _ch_convert_parse () { local images images=$(_ch_list_images "$1") local cword="$2" local -n in_fmt=$3 local -n out_fmt=$4 local -n end_opts=$5 shift 5 local words=("$@") local ct=1 while ((ct < ${#words[@]})); do case ${words[$ct-1]} in -i|--in-fmt) if _is_subword "${words[$ct]}" "$_convert_fmts"; then in_fmt="${words[$ct]}" fi ;; -o|--out-fmt) if _is_subword "${words[$ct]}" "$_convert_fmts"; then out_fmt="${words[$ct]}" fi ;; esac if (! _is_subword "${words[$ct-1]}" "$_convert_arg_opts") \ && [[ ("${words[$ct]}" != "-"*) && ($ct -ne $cword) ]]; then # First non-opt arg found, assuming it’s the input image end_opts=$ct local word word="$(_sanitized_tilde_expand "${words[$ct]}")" if [[ -z "$in_fmt" ]]; then # If the parser hasn’t been told the input image format yet, try # to figure it out from available information. if _is_subword "${words[$ct]}" "$images"; then # Check for storage images first because this is what # ch-convert seems to default to in the case of a name # collision between different image formats (e.g. if “foo” # is an image in storage and “./foo/” is in the working # directory). in_fmt="ch-image" elif [[ -d "$word" ]]; then in_fmt="dir" elif [[ -f "$word" ]]; then if [[ ("${words[$ct]}" == *".tgz") || ("${words[$ct]}" == *".tar."*) ]]; then in_fmt="tar" elif [[ "${words[$ct]}" == *".sqfs" ]]; then in_fmt="squash" fi fi fi fi ((ct++)) done } # Figure out which storage directory to use (including cli-specified storage). # Remove trailing slash. Note that this isn't performed when the script is # sourced because the working storage directory can effectively change at any # time with “CH_IMAGE_STORAGE” or the “--storage” option. _ch_find_storage () { if echo "$@" | grep -Eq -- '\s(--storage|-\w*s)'; then # This if “--storage” or “-s” are in the command line. sed -Ee 's/(.*)(--storage=*|[^-]-s=*)\ *([^ ]*)(.*$)/\3/g' -Ee 's|/$||g' <<< "$@" elif [[ -n "$CH_IMAGE_STORAGE" ]]; then echo "$CH_IMAGE_STORAGE" | sed -Ee 's|/$||g' else echo "/var/tmp/$USER.ch" fi } # Print the subcommand in an array of words; if there is not one, print an empty # string. This feels a bit kludge-y, but it's the best I could come up with. # It's worth noting that the double for loop doesn't take that much time, since # the Charliecloud command line, even in the wost case, is relatively short. # # Usage: _ch_image_subcmd_get [words] # # Example: # >> _ch_image_subcmd_get "ch-image [...] build [...]" # build _ch_image_subcmd_get () { local cword="$1" shift 1 local subcmd local wrds=("$@") local ct=1 while ((ct < ${#wrds[@]})); do if [[ $ct -ne $cword ]]; then for subcmd_i in $_image_subcommands; do if [[ "${wrds[$ct]}" == "$subcmd_i" ]]; then subcmd="$subcmd_i" break 2 fi done fi ((ct++)) done echo "$subcmd" } # List images in storage directory. _ch_list_images () { # “find” throws an error if “img” subdir doesn't exist or is empty, so check # before proceeding. if [[ -d "$1/img" && -n "$(ls -A "$1/img")" ]]; then find "$1/img/"* -maxdepth 0 -printf "%f\n" | sed -e 's|+|:|g' -e 's|%|/|g' fi } # Horrible, disgusting function to find an image or image ref in the ch-run # command line. This function takes five arguments: # # 1.) A string representing the path to the storage directory. # # 2.) The current position (measured in words) of the cursor in the array # representing the command line (index starting at 0). # # 3.) An out parameter (see explanation above “_ch_convert_parse”). If # “_ch_run_parse” finds the name of an image in storage (e.g. # “alpine:latest”) or something that looks like an image path (i.e. a # directory, tarball or file named like a squash archive) in the command # line, the value of the variable will be updated to the image name or # path. If neither are found, the function will not modify the value of # this variable. # # 4.) Another out parameter. If this function finds “--” in the current # command line and it doesn't seem like the user is trying to complete # that “--” to an option, “_ch_run_parse” will assume that this is the # point beyond which the user specifies commands to be run inside the # container and will give the variable the index value of the “--”. Our # criterion for deciding that the user isn't trying to complete “--” to an # option is that the current index of the cursor in the word array # (argument 2, see above) is not equal to the position of the “--” in said # array. # # 5.) A string representing the expanded command line array (i.e. # "${array[@]}"). # _ch_run_parse () { # The essential purpose of this function is to try to find an image in the # current command line. If it finds one, it passes the “name” of the image # back to the caller in the form of an out parameter (see above). If it # doesn't find one, the out parameter remains unmodified. This function # assumes that the out parameter in question is the empty string before it # gets called. local images # these two lines are separate b/c SC2155 images=$(_ch_list_images "$1") # shift 1 local cword="$1" shift 1 local -n cli_img=$1 local -n cmd_pt=$2 shift 2 local wrds=("$@") local ct=1 # Check for tarballs and squashfs archives. while ((ct < ${#wrds[@]})); do # In bash, expansion of the “~” character to the value of $HOME doesn't # happen if a value is quoted (see # https://stackoverflow.com/a/52519780). To work around this, we add # “eval echo” (https://stackoverflow.com/a/6988394) to this test. if [[ $ct != "$cword" ]]; then if [[ ( -f "$(_sanitized_tilde_expand "${wrds[$ct]}")" \ && ( ${wrds[$ct]} == *.sqfs \ || ${wrds[$ct]} == *.tar.? \ || ${wrds[$ct]} == *.tar.?? \ || ${wrds[$ct]} == *.tgz ) ) \ || ( -d ${wrds[$ct]} \ && ${wrds[$ct-1]} != --mount \ && ${wrds[$ct-1]} != -m \ && ${wrds[$ct-1]} != --bind \ && ${wrds[$ct-1]} != -b \ && ${wrds[$ct-1]} != -c \ && ${wrds[$ct-1]} != --cd ) ]]; then cli_img="${wrds[$ct]}" fi if [[ ${wrds[$ct]} == "--" ]]; then cmd_pt=$ct fi # Check for refs to images in storage. if [[ -z $cli_img ]]; then if _is_subword "${wrds[$ct]}" "$images"; then cli_img="${wrds[$ct]}" fi fi fi ((ct++)) done } # List undeletable images in the build cache, if it exists. _ch_undelete_list () { if [[ -d "$1/bucache/" ]]; then git -C "$strg_dir/bucache/" tag -l | sed -e "s/&//g" \ -e "s/%/\//g" \ -e "s/+/:/g" fi } # Returns filenames and directories, appending a slash to directory names. # This function takes option “-X”, a string of space-separated glob patterns # to be excluded from file completion using the compgen option of the same # name (source: https://stackoverflow.com/a/40227233, see also: # https://devdocs.io/bash/programmable-completion-builtins#index-compgen) _compgen_filepaths () { local filterpats=("") if [[ "$1" == "-X" && 1 -lt ${#@} ]]; then # Read a string into an array: # https://stackoverflow.com/a/10586169 # Pitfalls: # https://stackoverflow.com/a/45201229 # FIXME: Need to modify $IFS before doing this? read -ra filterpats <<< "$2" shift 2 fi local cur="$1" # Files, excluding directories, with no trailing slashes. The grep # performs an inverted substring match on the list of directories and the # list of files respectively produced by compgen. The compgen statements # also prepend (-P) a “^” and append (-S) a “$” to the file/dir names to # avoid the case where a substring matching a dirname is erroniously # removed from a filename by the inverted match. These delimiters are then # removed by the “sed”. (See the StackOverflow post cited above for OP’s # explanation of this code). The for loop iterates through exclusion # patterns specified by the “-X” option. If “-X” isn't specified, the code # in the loop executes once, with no patterns excluded (“-X ""”). for pat in "${filterpats[@]}" do grep -v -F -f <(compgen -d -P ^ -S '$' -X "$pat" -- "$cur") \ <(compgen -f -P ^ -S '$' -X "$pat" -- "$cur") | sed -e 's/^\^//' -e 's/\$$/ /' \ -e 's/ $//g' # remove trailing space done # Directories with trailing slashes: compgen -d -S / -- "$cur" } # Wrapper for a horrible pipeline to complete python files in lib. _compgen_py_libfiles () { compgen -f "$_ch_completion_dir/../lib/" | grep -o -E ".*\.py" | sed "s|$_ch_completion_dir\/\.\.\/lib\/\(.*\)\.py|\1|" } # Return 0 if "$1" is a word in space-separated sequence of words "$2", e.g. # # >>> _is_subword "foo" "foo bar baz" # 0 # >>> _is_subword "foo" "foobar baz" # 1 # _is_subword () { local subword=$1 shift 1 #shellcheck disable=SC2068 for word in $@; do if [[ "$word" == "$subword" ]]; then return 0 fi done return 1 } # Expand tilde in quoted strings to the correct home path, if applicable, while # sanitizing to prevent code injection (see https://stackoverflow.com/a/38037679). # _sanitized_tilde_expand () { if [[ $1 == ~* ]]; then # Adding the “/” at the end here is important for ensuring that the “~” # always gets expanded, e.g. in the case where "$1" is “~” instead of # “~/”. user="$(echo "$1/" | sed -E 's|^~([^~/]*/).*|\1|g')" path="$(echo "$1" | sed -E 's|^~[^~/]*(.*)|\1|g')" eval "$(printf "home=~%q" "$user")" # Check if “home” is a vaild directory. # shellcheck disable=SC2154 if [[ -d "$home" ]]; then # The first character of “path” is “/”. Since we've added a “/” to # the end of “home” for proper “~” expansion, we now avoid the first # character of “path” in the concatenation of the two to avoid a # “//”. echo "$home${path:1:${#path}-1}" return 0 fi fi echo "$1" } # Wrapper for some tricky logic that determines whether or not to add a space at # the end of a path completion. For the sake of convenience we want to avoid # adding a space at the end if the completion is a directory path, because we # don’t know if the user is looking for the completed directory or one of its # subpaths (we may be able to figure this out in some cases, but I’m not gonna # worry about that now). We *do* want to add a space at the end if the # completion is the path to a file. _space_filepath () { local files files="$(_compgen_filepaths "$1" "$2" "$3")" if [[ (-n "$files") \ && (! -f "$(_sanitized_tilde_expand "$files")") ]]; then compopt -o nospace fi } _version_ok_ch_completion () { if [[ "$($1 --version 2>&1)" == "$_ch_completion_version" ]]; then return 0 else return 1 fi } complete -F _ch_completion_complete ch-completion complete -F _ch_convert_complete ch-convert complete -F _ch_image_complete ch-image complete -F _ch_run_complete ch-run charliecloud-0.37/bin/ch-convert000077500000000000000000000715771457016721300166760ustar00rootroot00000000000000#!/bin/sh ## preamble ################################################################## ch_lib=$(cd "$(dirname "$0")" && pwd)/../lib . "${ch_lib}/base.sh" PATH=${ch_bin}:$PATH usage=$(cat < "$2" } cv_dir_chimage () { dir_in_validate "$1" chimage_out_validate "$2" INFO 'importing ...' # “ch-image” recognizes “-q” as an argument, but we choose to quiet it with # “quiet” instead here because doing so is simpler from the perspective of # “ch-convert”. quiet ch-image import "$1" "$2" # FIXME: no progress meter } cv_dir_docker () { dir_in_validate "$1" docker_out_validate "$2" dir_to_dockman docker_ "$1" "$2" } cv_dir_podman () { dir_in_validate "$1" podman_out_validate "$2" dir_to_dockman podman_ "$1" "$2" } cv_dir_squash () { # FIXME: mksquashfs(1) is incredibly noisy. This can be mitigated with # -quiet, but that's not available until version 4.4 (2019). dir_in_validate "$1" squash_out_validate "$2" pflist=${tmpdir}/pseudofiles INFO 'packing ...' quiet touch "$pflist" mount_points_ensure "$1" "$pflist" # Exclude build cache metadata. 64kiB block size based on Shane’s # experiments. # shellcheck disable=SC2086 quiet mksquashfs "$1" "$2" $squash_xattr_arg -b 65536 -noappend -all-root \ -pf "$pflist" -e "$1"/ch/git -e "$1"/ch/git.pickle # Zero the archive’s internal modification time at bytes 8–11, 0-indexed # [1]. Newer SquashFS-Tools ≥4.3 have option “-fstime 0” to do this, but # CentOS 7 comes with 4.2. [1]: https://dr-emann.github.io/squashfs/ printf '\x00\x00\x00\x00' | quiet dd of="$2" bs=1 count=4 seek=8 conv=notrunc status=none quiet rm "$pflist" } cv_dir_tar () { dir_in_validate "$1" tar_out_validate "$2" # Don't add essential files & directories because that will happen later # when converted to dir or squash. INFO 'packing ...' tar_ "$1" | gzip_ > "$2" } cv_docker_chimage () { dockman_in_validate docker_ "$1" chimage_out_validate "$2" dockman_to_chimage docker_ "$1" "$2" } cv_docker_dir () { dockman_in_validate docker_ "$1" dir_out_validate "$2" dockman_to_dir docker_ "$1" "$2" } cv_docker_podman () { dockman_in_validate docker_ "$1" podman_out_validate "$2" docker_out=${tmpdir}/weirdal.tar.gz cv_docker_tar "$1" "$docker_out" # FIXME: needlessly compresses cv_tar_podman "$docker_out" "$2" quiet rm "$docker_out" } cv_docker_squash () { dockman_in_validate docker_ "$1" squash_out_validate "$2" dockman_to_squash docker_ "$1" "$2" } cv_docker_tar () { dockman_in_validate docker_ "$1" tar_out_validate "$2" dockman_to_tar docker_ "$1" "$2" } cv_podman_chimage () { dockman_in_validate podman_ "$1" chimage_out_validate "$2" dockman_to_chimage podman_ "$1" "$2" } cv_podman_dir () { dockman_in_validate podman_ "$1" dir_out_validate "$2" dockman_to_dir podman_ "$1" "$2" } cv_podman_docker () { dockman_in_validate podman_ "$1" docker_out_validate "$2" podman_out=${tmpdir}/weirdal.tar.gz cv_podman_tar "$1" "$podman_out" # FIXME: needlesly compresses cv_tar_docker "$podman_out" "$2" quiet rm "$podman_out" } cv_podman_squash () { dockman_in_validate podman_ "$1" squash_out_validate "$2" dockman_to_squash podman_ "$1" "$2" } cv_podman_tar () { dockman_in_validate podman_ "$1" tar_out_validate "$2" dockman_to_tar podman_ "$1" "$2" } cv_squash_chimage () { squash_in_validate "$1" chimage_out_validate "$2" unsquash_dir=${tmpdir}/weirdal cv_squash_dir "$1" "$unsquash_dir" cv_dir_chimage "$unsquash_dir" "$2" quiet rm -Rf --one-file-system "$unsquash_dir" } cv_squash_dir () { squash_in_validate "$1" dir_out_validate "$2" # Notes about unsquashfs(1): # # 1. It has no exclude filter, only include, so if the archive includes # bad files like devices, this will fail. I don't know to what degree # this will be a problem. # # 2. It has no option analagous to tar(1)’s -p, so we have to rely on the # umask to get correct permissions. (Weirdly, it seems to respect umask # for files but not directories.) umask_=$(umask) umask 0000 # To support conversion with an empty dir as output (#1612), we add the “-f” # option to our unsquashfs(1) call below. Without “-f”, if you try to use # unsquashfs(1) with an empty dir as the target, you get the error “failed # to make directory path/to/dir, because File exists”. The documentation for # “-f” says “if file already exists then overwrite”, but I’ve concluded that # the option actually doesn’t “overwrite” directories through the following # test: # # 1. Create an empty directory, let’s call it “parent”. # # 2. Create another empty directory (“child”) that is a subdirectory of # “parent”. # # 3. Change the owner and group of “parent” to root, ensure that the owner # and group of “child” are a user (rather than root). # # 4. Check that both “parent” and “child” have the following permissions: # “drwxr-xr-x”. # # 5. As the owner of “child”, try to remove “child” and confirm that linux # won’t allow it. # # 6. Use unsquashfs(1) with the “-f” option to unpack an archive to “child” # and confirm that it’s successful. # # (https://unix.stackexchange.com/a/583819) quiet unsquashfs -f -d "$2" -user-xattrs "$1" umask "$umask_" dir_fixup "$2" } cv_squash_docker () { squash_in_validate "$1" docker_out_validate "$2" squash_to_dockman docker_ "$1" "$2" } cv_squash_podman () { squash_in_validate "$1" podman_out_validate "$2" squash_to_dockman podman_ "$1" "$2" } cv_squash_tar () { squash_in_validate "$1" tar_out_validate "$2" unsquash_dir=${tmpdir}/weirdal cv_squash_dir "$1" "$unsquash_dir" cv_dir_tar "$unsquash_dir" "$2" quiet rm -Rf --one-file-system "$unsquash_dir" } cv_tar_chimage () { tar_in_validate "$1" chimage_out_validate "$2" INFO 'importing ...' quiet ch-image import "$1" "$2" # FIXME: no progress meter } cv_tar_dir () { tar_in_validate "$1" dir_out_validate "$2" INFO 'analyzing ...' root=$(tar_root "$1") INFO 'unpacking ...' dir_make "$2" # Unpack the tarball. There's a lot going on here. # # 1. Use a pipe b/c PV ignores arguments if it’s cat rather than PV. # # 2. Use --strip-components to turn non-tarbombs into tarbombs so we can # just provide our own root directory and not need to clean up later. # # 3. Use --xform to strip leading “./” so --strip-components gets the # right number. (Leading “/” is always stripped and does not count as # a component.) The trailing “x” gets extended regular expressions, # like “sed -E”. # # 4. Use --exclude to ignore the contents of “/dev”, because # unprivileged users can't make device files and we overmount that # directory anyway. The order appears to be: --exclude, --xform, # --strip-components, so we have to specify it thrice to account for # for leading “/” and “./”. if [ -z "$root" ]; then strip_ct=0 # tarbomb ex1='dev/*' else strip_ct=1 # not tarbomb # Escape tar glob wildcards; printf avoids echo’s trailing newline. root_escaped=$(printf "%s" "$root" | sed -E 's|([[*?])|\\\1|g') ex1="${root_escaped}/dev/*" fi ex2="/${ex1}" ex3="./${ex1}" VERBOSE "exclude patterns: ${ex1} ${ex2} ${ex3}" #shellcheck disable=SC2094,SC2086 pv_ -s "$(stat -c%s "$1")" < "$1" \ | quiet tar x"$(tar_decompress_arg "$1")" -pC "$2" -f - \ --xform 's|^\./||x' --strip-components=$strip_ct \ --anchored --no-wildcards-match-slash $tar_xattr_args \ --exclude="$ex1" --exclude="$ex2" --exclude="$ex3" dir_fixup "$2" } cv_tar_docker () { tar_in_validate "$1" docker_out_validate "$2" tar_to_dockman docker_ "$1" "$2" } cv_tar_podman () { tar_in_validate "$1" podman_out_validate "$2" tar_to_dockman podman_ "$1" "$2" } cv_tar_squash () { tar_in_validate "$1" squash_out_validate "$2" tar_dir=${tmpdir}/weirdal cv_tar_dir "$1" "$tar_dir" cv_dir_squash "$tar_dir" "$2" quiet rm -Rf --one-file-system "$tar_dir" } ## Dockman functions ## # Use for conversions involving docker or podman. Similarities between the # command line instructions for docker and podman allow to write generalized # functions that can be used for both (hence “dockman”). When calling a # “dockman” function, image format is specified by the first argument # (“docker_” or “podman_”). chimage_to_dockman () { chimage_tar=${tmpdir}/weirdal.tar.gz cv_chimage_tar "$2" "$chimage_tar" # FIXME: needlessly compresses? tar_to_dockman "$1" "$chimage_tar" "$3" quiet rm "$chimage_tar" } dir_to_dockman () { dirtar=${tmpdir}/weirdal.tar.gz # One could also use “docker build” with “FROM scratch” and “COPY”, # apparently saving a tar step. However, this will in fact tar the source # directory anyway to send it to the Docker daemon. cv_dir_tar "$2" "$dirtar" # FIXME: needlessly compresses tar_to_dockman "$1" "$dirtar" "$3" quiet rm "$dirtar" } dm_fmt_name () { # Format “docker_” and “podman_” as “Docker” and “Podman” respectively. case $1 in docker_) echo "$1" | tr "d" "D" | tr -d "_" ;; podman_) echo "$1" | tr "p" "P" | tr -d "_" ;; *) FATAL "unreachable code reached" ;; esac } dockman_in_validate () { digest=$("$1" image ls -q "$2") [ -n "$digest" ] || FATAL "source not found in $(dm_fmt_name "$1") storage: ${2}" } dockman_to_chimage () { dockman_out=${tmpdir}/weirdal.tar.gz "cv_${1}tar" "$2" "$dockman_out" # FIXME: needlessly compresses cv_tar_chimage "$dockman_out" "$3" quiet rm "$dockman_out" } dockman_to_dir () { dockman_out=${tmpdir}/weirdal.tar.gz "dockman_to_tar" "$1" "$2" "$dockman_out" # FIXME: needlessly compresses cv_tar_dir "$dockman_out" "$3" quiet rm "$dockman_out" } dockman_to_squash () { dockman_dir=${tmpdir}/weirdal dockman_to_dir "$1" "$2" "$dockman_dir" # FIXME: needlessly compresses cv_dir_squash "$dockman_dir" "$3" quiet rm -Rf --one-file-system "$dockman_dir" } dockman_to_tar () { tmptar=${tmpdir}/weirdal.tar tmpenv=${tmpdir}/weirdal.env INFO 'exporting ...' cid=$("$1" create --read-only "$2" /bin/true) # cmd needed but not run size=$("$1" image inspect "$2" --format='{{.Size}}') quiet "$1" export "$cid" | pv_ -s "$size" > "$tmptar" "$1" rm "$cid" > /dev/null INFO 'adding environment ...' "$1" inspect "$2" \ --format='{{range .Config.Env}}{{println .}}{{end}}' > "$tmpenv" # The tar flavor Docker gives us does not support UIDs or GIDs greater # than 2**21, so use 0/0 rather than what’s on the filesystem. See #1573. # shellcheck disable=SC2086 quiet tar rf "$tmptar" -b1 -P --owner=0 --group=0 $tar_xattr_args \ --xform="s|${tmpenv}|ch/environment|" "$tmpenv" INFO 'compressing ...' pv_ < "$tmptar" | gzip_ -6 > "$3" quiet rm "$tmptar" quiet rm "$tmpenv" } squash_to_dockman () { unsquash_tar=${tmpdir}/weirdal.tar.gz cv_squash_tar "$2" "$unsquash_tar" dockman=$(echo "$1" | tr -d "_") # remove trailing underscore "cv_tar_$dockman" "$unsquash_tar" "$3" quiet rm "$unsquash_tar" } tar_to_dockman () { INFO "importing ..." tmpimg=$(mktemp -u tmpimg.XXXXXX | tr '[:upper:]' '[:lower:]') quiet "$1" import "$2" "$tmpimg" # FIXME: no progress meter # Podman imports our tarballs with rw------- permissions on “/ch” (i.e., # no execute), which causes all kinds of breakage. Work around that. quiet "$1" build -t "$3" - <> "$2" else quiet mkdir "${1}/${i}" fi fi done # files for i in etc/hosts etc/resolv.conf; do if ! exist_p "${1}/${i}"; then if [ -n "$2" ]; then quiet echo "${i} f 644 root root true" >> "$2" else quiet touch "${1}/${i}" fi fi done } # Validate the parent or enclosing directory of $1 exists. parent_validate () { parent=$(dirname "$1") [ -d "$parent" ] || "not a directory: $parent" } # Exit with error if $1 exists and --no-clobber was given. path_noclobber () { if [ -e "$1" ] && [ -n "$no_clobber" ]; then FATAL "exists, not deleting per --no-clobber: ${1}" fi } # Tar $1 and emit the result on stdout, excluding build cache metadata. # Produce a tarbomb because Docker requires tarbombs. tar_ () { # shellcheck disable=SC2086 ( cd "$1" && tar cf - $tar_xattr_args \ --exclude=./ch/git \ --exclude=./ch/git.pickle . ) | pv_ } # Print the appropriate tar(1) decompression argument for file named $1, which # may be the empty string, because GNU tar is unable to infer it if input is a # pipe [1], and we want to keep pv(1). # # [1]: https://www.gnu.org/software/tar/manual/tar.html#gzip tar_decompress_arg () { case $1 in *.tar) echo ;; *.tar.gz) echo z ;; *.tar.xz) echo J ;; *.tgz) echo z ;; *) FATAL "unknown extension: ${1}" ;; esac } # Print the name of the root directory of tarball $1 on stdout, if there is # one. If not, i.e. $1 is a tarbomb, return the empty string. # # We don't use pv(1) for this and therefore let tar infer the compression. # # This is rather messy because: # # 1. Archive members can start with “/” (slash) or “./” (dot, slash), both # of which are ignored on unpacking. For example, the root directory as # listed in the tarball might be “foo”, “/foo”, or “./foo”; in all three # cases this function prints “foo”. # # 2. Tarballs have no index, so listing all members requires reading and # decompressing the entire archive (and unpacking is a second full read). # I have not found a way to detect a tarbomb without listing all members; # doing so is issue #1325. # # I have not tested decompressing once and then reading the decompressed # version twice. Some quick testing suggests that we spend almost all the # read time in gzip, but this approach adds time to write the # uncompressed tarball in addition to space to store it, so it's not an # appealing approach to me. # # Previously, we listed only the first N members, but this breaks on # Spack images if /spack is first in the archive because that directory # can contain tens of thousands of files (maybe more). # # 3. GNU tar lists members newline-separated. This still works for member # names containing newline, because it's escaped as “\n”. Some other # characters are escaped too, e.g. tab is “\t”. I am assuming this case # is unlikely for container image tarballs, so this function has not been # tested with such members. # # See also: https://unix.stackexchange.com/a/242712 tar_root () { # The three commands in this sed script are: (1) remove leading “/” or # “./” if present; (2) delete from the first slash to the end of the line # inclusive, i.e. everything except the first component; (3) delete blank # lines, because the first component often appears alone. We use sed # because --xform does not apply to listing. #shellcheck disable=SC2094 list=$( pv_ -s "$(stat -c%s "$1")" < "$1" \ | tar t"$(tar_decompress_arg "$1")" -f - \ | sed -E 's|^\.?/||; s|/.*$||; /^$/d') # Get the first path component of the first file in the tarball. root1=$(echo "$list" | head -n 1) # Find members whose first component does not match the first member’s. if echo "$list" | grep -Fxvq "$root1"; then VERBOSE 'tarbomb: yes' echo '' else VERBOSE 'tarbomb: no' echo "$root1" fi } # Set $tmpdir to be a new directory with a unique and unpredictable name, as a # subdirectory of --tmp, $TMPDIR, or /var/tmp, whichever is first set. tmpdir_setup () { if [ -z "$tmpdir" ]; then if [ -n "$TMPDIR" ]; then tmpdir=$TMPDIR else tmpdir=/var/tmp fi fi case $tmpdir in /*) ;; *) FATAL "temp dir must be absolute: ${tmpdir}" ;; esac tmpdir=$(mktemp -d --tmpdir="$tmpdir" ch-convert.XXXXXX) } # Error out if “--xattrs” and “--no-xattrs” are specified in the same command # line. xattr_opt_err () { if [ -n "$xattrs" ] && [ -n "$no_xattrs" ]; then FATAL "\"--xattrs\" incompatible with \"--no-xattrs\"" fi } ## main ###################################################################### while true; do if ! parse_basic_arg "$1"; then case $1 in -i|--in-fmt) shift in_fmt=$1 ;; -i=*|--in-fmt=*) in_fmt=${1#*=} ;; -n|--dry-run) dry_run=yes ;; --no-clobber) no_clobber=yes ;; --no-xattrs) no_xattrs=yes xattr_opt_err ;; -o|--out-fmt) shift out_fmt=$1 ;; -o=*|--out-fmt=*) out_fmt=${1#*=} ;; -s|--storage) shift cli_storage=$1 ;; --tmp) shift tmpdir=$1 ;; --xattrs) xattrs=yes xattr_opt_err ;; *) break ;; esac fi shift done if [ "$#" -ne 2 ]; then usage fi # This bizarre bit of syntax comes from https://unix.stackexchange.com/a/28782 if [ -n "$xattrs" ] || { [ -n "$CH_XATTRS" ] && [ -z "$no_xattrs" ]; }; then echo "preserving xattrs..." tar_xattr_args='--xattrs-include=user.* --xattrs-include=system.*' squash_xattr_arg=-xattrs else echo "discarding xattrs..." tar_xattr_args= squash_xattr_arg= fi in_desc=$1 out_desc=$2 VERBOSE "verbose level: ${log_level}" if command -v ch-image > /dev/null 2>&1; then have_ch_image=yes VERBOSE 'ch-image: found' else VERBOSE 'ch-image: not found' fi if command -v docker > /dev/null 2>&1; then have_docker=yes VERBOSE 'docker: found' else VERBOSE 'docker: not found' fi if command -v podman > /dev/null 2>&1; then have_podman=yes VERBOSE 'podman: found' else VERBOSE 'podman: not found' fi in_fmt=$(fmt_validate "$in_fmt" "$in_desc") out_fmt=$(fmt_validate "$out_fmt" "$out_desc") tmpdir_setup VERBOSE "temp dir: ${tmpdir}" VERBOSE "noclobber: ${no_clobber:-will clobber}" INFO 'input: %-8s %s' "$in_fmt" "$in_desc" INFO 'output: %-8s %s' "$out_fmt" "$out_desc" if [ "$in_fmt" = "$out_fmt" ]; then FATAL 'input and output formats must be different' fi if [ -z "$dry_run" ]; then # Dispatch to conversion function. POSIX sh does not support hyphen in # function names, so remove it. "cv_$(echo "$in_fmt" | tr -d '-')_$(echo "$out_fmt" | tr -d '-')" \ "$in_desc" "$out_desc" fi quiet rmdir "$tmpdir" INFO 'done' charliecloud-0.37/bin/ch-fromhost000077500000000000000000000436401457016721300170450ustar00rootroot00000000000000#!/bin/sh # The basic algorithm here is that we build up a list of file # source:destination pairs separated by newlines, then walk through them and # copy them into the image. # # The colon separator is to avoid the difficulty of iterating through a # sequence of pairs with no arrays or structures in POSIX sh. We could avoid # it by taking action immediately upon encountering each file in the argument # list, but that would (a) yield a half-injected image for basic errors like # misspellings on the command line and (b) would require the image to be first # on the command line, which seems awkward. # # The newline separator is for the same reason and also because it's # convenient for input from --cmd and --file. # # Note on looping through the newlines in a variable: The approach in this # script is to set IFS to newline, loop, then restore. This is awkward but # seemed the least bad. Alternatives include: # # 1. Piping echo into "while read -r": This executes the while in a # subshell, so variables don't stick. # # 2. Here document used as input, e.g.: # # while IFS= read -r FILE; do # ... # done < /dev/null 2>&1; then DEBUG "cray libfabric: ${f}" cray_fi_found=yes host_libfabric=$f else DEBUG "libfabric: ${f}" lib_found=yes fi ;; *-fi.so) DEBUG "libfabric shared provider: ${f}" fi_prov_found=yes # Providers, like Cray's libgnix-fi.so, link against paths that # need to be bind-mounted at run-time. Some of these paths need # to be added to ldconf; thus, add the linked paths to a ldconf # list to be added into the image. ldds=$(ldd "$f" 2>&1 | grep lib | awk '{print $3}' | sort -u) for l in $ldds; do ld=$(dirname "$(readlink -f "$l")") # Avoid duplicates and host libfabric.so. if [ "$(echo "$ld_conf" | grep -c "$ld")" -eq 0 ] \ && [ "$(echo "$ld" | grep -c "libfabric.so")" -eq 0 ]; \ then enqueue_ldconf "$ld" fi done ;; *) DEBUG "shared library: ${f}" lib_found=yes ;; esac fi # This adds a delimiter only for the second and subsequent files. # https://chris-lamb.co.uk/posts/joining-strings-in-posix-shell # # If destination empty, we'll infer it later. inject_files="${inject_files:+$inject_files$newline}$f:$d" done IFS="$old_ifs" } enqueue_ldconf () { [ "$1" ] ld_conf="${ld_conf:+$ld_conf$newline}$1" } queue_mkdir () { [ "$1" ] inject_mkdirs="${inject_mkdirs:+$inject_mkdirs$newline}$1" } queue_unlink () { [ "$1" ] inject_unlinks="${inject_unlinks:+$inject_unlinks$newline}$1" } warn () { printf '< warn >! %s\n' "$1" 1>&2 } warn_fi_var () { if [ -n "$FI_PROVIDER" ]; then warn "FI_PROVIDER=$FI_PROVIDER set; this will be preferred provider at runtime." fi if [ -n "$FI_PROVIDER_PATH" ]; then warn "FI_PROVIDER_PATH=$FI_PROVIDER_PATH set; --dest required" fi } if [ "$#" -eq 0 ]; then usage 1 fi while [ $# -gt 0 ]; do opt=$1; shift if ! parse_basic_arg "$opt"; then case $opt in -c|--cmd) ensure_nonempty --cmd "$1" out=$($1) || FATAL "command failed: $1" enqueue_file "$out" shift ;; --cray-cxi) warn_fi_var if [ -z "$CH_FROMHOST_OFI_CXI" ]; then FATAL "CH_FROMHOST_OFI_CXI is not set" fi enqueue_file "$CH_FROMHOST_OFI_CXI" ;; --cray-gni) warn_fi_var if [ -z "$CH_FROMHOST_OFI_GNI" ]; then FATAL "CH_FROMHOST_OFI_GNI is not set" fi enqueue_file "$CH_FROMHOST_OFI_GNI" ;; -d|--dest) ensure_nonempty --dest "$1" dest=$1 shift ;; -f|--file) ensure_nonempty --file "$1" out=$(cat "$1") || FATAL "cannot read file: ${1}" enqueue_file "$out" shift ;; # Note: Specifying any of the --print-* options along with one of # the file specification options will result in all the file # gathering and checking work being discarded. --print-cray-fi) cray_fi_found=yes print_cray_fi_dest=yes ;; --print-fi) fi_prov_found=yes print_fi_dest=yes ;; --print-lib) lib_found=yes print_lib_dest=yes ;; --no-ldconfig) no_ldconfig=yes ;; --nvidia) out=$(nvidia-container-cli list --binaries --libraries) \ || FATAL "nvidia-container-cli failed; does this host have GPUs?" enqueue_file "$out" ;; -p|--path) ensure_nonempty --path "$1" enqueue_file "$1" shift ;; -*) INFO "invalid option: ${opt}" usage ;; *) ensure_nonempty "image path" "${opt}" [ -z "$image" ] || FATAL "duplicate image: ${opt}" [ -d "$opt" ] || FATAL "image not a directory: ${opt}" image="$opt" ;; esac fi done if [ -n "$FI_PROVIDER_PATH" ] && [ -n "$fi_prov_found" ] && [ -z "$dest" ]; then FATAL "FI_PROVIDER_PATH set; missing --dest" fi [ "$image" ] || FATAL "no image specified" if [ -n "$cray_fi_found" ]; then # There is no Slingshot provider CXI; to leverage slingshot we need to # replace the image libfabric.so with Cray's. VERBOSE "searching image for inferred libfabric destiation" img_libfabric=$(find "$image" -name "libfabric.so") [ -n "$img_libfabric" ] || FATAL "libfabric.so not found in $image" DEBUG "found $img_libfabric" if [ "$(echo "$img_libfabric" | wc -l)" -ne 1 ]; then warn 'found more than one libfabric.so' fi img_libfabric_path=$(echo "$img_libfabric" | sed "s@$image@@") cray_fi_dest=$(dirname "/$img_libfabric_path") # Since cray's libfabric isn't a standard provider, to use slingshot we # must also add any missing linked libraries from the host. VERBOSE "adding cray libfabric libraries" ldds=$(ldd "$host_libfabric" 2>&1 | grep lib | awk '{print $3}' | sort -u) for l in $ldds; do # Do not replace any libraries found in the image, experimentation has # shown this to be problematic. Perhaps revisit in the future. For now, # both MPICH and OpenMPI examples work with this conservative approach. file_found=$(find "${image}" -name "$(basename "$l")") if [ -n "$file_found" ]; then DEBUG "skipping $l" continue fi enqueue_file "$l" file_dir=$(dirname "$l") # Avoid duplicates. if [ "$(echo "$ld_conf" | grep -c "$file_dir")" -eq 0 ]; then enqueue_ldconf "$file_dir" fi done fi if [ -n "$lib_found" ]; then # We want to put the libraries in the first directory that ldconfig # searches, so that we can override (or overwrite) any of the same library # that may already be in the image. VERBOSE "asking ldconfig for inferred shared library destination" # "ldconfig -Nv" gives pointless warnings on stderr even if successful; we # don't want to show those to users (unless -vv or higher). However, we # don't want to simply pipe stderr to /dev/null because this hides real # errors. Thus, use the following abomination to pipe stdout and stderr to # *separate grep commands*. See: https://stackoverflow.com/a/31151808 if [ "$log_level" -lt 2 ]; then # VERBOSE or lower stderr_filter='(^|dynamic linker, ignoring|given more than once|No such file or directory)$' else # DEBUG or higher stderr_filter=weird_al_yankovic_will_not_appear_in_ldconfig_output fi lib_dest=$( { "${ch_bin}/ch-run" "$image" -- /sbin/ldconfig -Nv \ 2>&1 1>&3 3>&- | grep -Ev "$stderr_filter" ; } \ 3>&1 1>&2 | grep -E '^/' | cut -d: -f1 | head -1 ) [ -n "$lib_dest" ] || FATAL 'empty path from ldconfig' [ -z "${lib_dest%%/*}" ] || FATAL "bad path from ldconfig: ${lib_dest}" VERBOSE "inferred shared library destination: ${image}/${lib_dest}" fi if [ -n "$fi_prov_found" ]; then # The libfabric provider can be specified with FI_PROVIDER. The path the # search for shared providers at can be specified with FI_PROVIDER_PATH # (undocumented). This complicates the inferred destination because these # variables can be inherited from the host or explicitly set in the # image's /ch/environment file. # # For simplicity, the inferred injection destination is the always the # 'libfabric' directory at the path where libfabric.so is found. If it # does not exist, create it. Warn if FI_PROVIDER_PATH or FI_PROVIDER is # found in the the image's /ch/environment file. VERBOSE "searching ${image} for libfabric shared provider destination" ch_env_p=$(grep -E '^FI_PROVIDER_PATH=' "${image}/ch/environment") \ || true # avoid -e exit ch_env_p=${ch_env_p##*=} if [ -n "$ch_env_p" ]; then warn "FI_PROVIDER_PATH in ${image}/ch/environment; consider --dest" fi img_libfabric=$(find "$image" -name 'libfabric.so') img_libfabric_path=$(echo "$img_libfabric" | sed "s@$image@@") DEBUG "found: ${image}${img_libfabric_path}" fi_prov_dest=$(dirname "/${img_libfabric_path}") fi_prov_dest="${fi_prov_dest}/libfabric" queue_mkdir "$fi_prov_dest" VERBOSE "inferred provider destination: $fi_prov_dest" fi if [ -n "$print_lib_dest" ]; then echo "$lib_dest" exit 0 fi if [ -n "$print_fi_dest" ]; then echo "$fi_prov_dest" fi if [ -n "$print_cray_fi_dest" ]; then echo "$cray_fi_dest" fi if [ -f /etc/opt/cray/release/cle-release ]; then # Cray needs a pile of hugetlbfs filesystems mounted at # /var/lib/hugetlbfs/global. Create mount point for ch-run. queue_mkdir /var/lib/hugetlbfs # UGNI if [ ! -L /etc/opt/cray/release/cle-release ]; then # ALPS libraries require the contents of this directory to be present # at the same path as the host. Create the mount point here, then # ch-run bind-mounts it later. queue_mkdir /var/opt/cray/alps/spool # The cray-ugni provider will link against cray’s libwlm_detect so. # Create the mount point for ch-run. queue_mkdir /opt/cray/wlm_detect # libwlm_detect.so requires file(s) to present at the same path as the # host. Create mount point for ch-run. queue_mkdir /etc/opt/cray/wlm_detect # OFI uGNI provider, libgnix-fi.so, links against the Cray host’s # libxpmem, libudreg, libalpsutil, libalpslli, and libugni; create # mount points for ch-run to use later. queue_mkdir /opt/cray/udreg queue_mkdir /opt/cray/xpmem queue_mkdir /opt/cray/ugni queue_mkdir /opt/cray/alps fi # CXI (slingshot) if [ -f /opt/cray/etc/release/cos ]; then # Newer Cray Shasta environments require the contents of this # directory to be present at the same path as the host. Create mount # points for ch-run to use later. queue_mkdir /var/spool/slurmd fi fi [ "$inject_files" ] || FATAL "empty file list" VERBOSE "injecting into image: ${image}" old_ifs="$IFS" IFS="$newline" # Process unlink list. for u in $inject_unlinks; do DEBUG "deleting: ${image}${u}" rm -f "${image}${u}" done # Process bind-mount destination targets. for d in $inject_mkdirs; do DEBUG "mkdir: ${image}${d}" mkdir -p "${image}${d}" done # Process ldconfig targets. if [ "$fi_prov_found" ] || [ "$cray_fi_found" ]; then if [ ! -f "${image}/etc/ld.so.conf" ]; then DEBUG "creating empty ld.so.conf" touch "${image}/etc/ld.so.conf" fi if ! grep -F 'include ld.so.conf.d/*.conf' "${image}/etc/ld.so.conf" \ > /dev/null 2>&1; then DEBUG "ld.so.conf: adding 'include ld.so.conf.d/*.conf'" echo 'include ld.so.conf.d/*.conf' >> "${image}/etc/ld.so.conf" fi # Prepare image ch-ofi.conf. printf '' > "${image}/etc/ld.so.conf.d/ch-ofi.conf" # add ofi dso provider ld library dirs. for c in $ld_conf; do DEBUG "ld.so.conf: adding ${c}" echo "$c" >> "${image}/etc/ld.so.conf.d/ch-ofi.conf" done fi for file in $inject_files; do f="${file%%:*}" d="${file#*:}" infer= if is_bin "$f" && [ -z "$d" ]; then d=/usr/bin infer=" (inferred)" elif is_so "$f" && [ -z "$d" ]; then case "$f" in *libfabric.so) d=$lib_dest if ldd "$f" | grep libcxi > /dev/null 2>&1; then d=$cray_fi_dest fi ;; *-fi.so) d=$fi_prov_dest ;; *) d=$lib_dest ;; esac infer=" (inferred)" fi VERBOSE "${f} -> ${d}${infer}" [ "$d" ] || FATAL "no destination for: ${f}" [ -z "${d%%/*}" ] || FATAL "not an absolute path: ${d}" [ -d "${image}${d}" ] || FATAL "not a directory: ${image}${d}" if [ ! -w "${image}${d}" ]; then # Some images unpack with unwriteable directories; fix. This seems # like a bit of a kludge to me, so I'd like to remove this special # case in the future if possible. (#323) INFO "${image}${d} not writeable; fixing" chmod u+w "${image}${d}" || FATAL "can't chmod u+w: ${image}${d}" fi cp --dereference --preserve=all "$f" "${image}${d}" \ || FATAL "cannot inject: ${f}" done IFS="$old_ifs" if [ -z "$no_ldconfig" ] \ && { [ "$lib_found" ] \ || [ "$fi_prov_found" ] \ || [ "$cray_fi_found" ] ;} then VERBOSE "running ldconfig" "${ch_bin}/ch-run" -w "$image" -- /sbin/ldconfig 2> /dev/null \ || FATAL 'ldconfig error' if [ -n "$fi_prov_found" ] || [ -n "$cray_fi_found" ]; then VERBOSE "validating ldconfig cache" for file in $inject_files; do f="$(basename "${file%%:*}")" f=$( "${ch_bin}/ch-run" "$image" \ -- find / \ -not \( -path /proc -prune \) \ -not \( -path /dev -prune \) \ -not \( -path /tmp -prune \) \ -not \( -path /sys -prune \) \ -not \( -path /var/opt/cray -prune \) \ -not \( -path /etc/opt/cray -prune \) \ -name "$f") if [ "$("${ch_bin}/ch-run" "$image" -- ldd "$f" | grep -c 'not found ')" -ne 0 ]; then FATAL "ldconfig: '${ch_bin}/ch-run $image -- ldd $f' failed" fi done fi else VERBOSE "not running ldconfig" fi echo 'done' charliecloud-0.37/bin/ch-image.py.in000066400000000000000000000450131457016721300173130ustar00rootroot00000000000000#!%PYTHON_SHEBANG% import argparse import ast import collections.abc import inspect import os.path import sys ch_lib = os.path.dirname(os.path.abspath(__file__)) + "/../lib" sys.path.insert(0, ch_lib) import charliecloud as ch import build import build_cache as bu import filesystem as fs import misc import pull import push ## Constants ## # FIXME: It’s currently easy to get the ch-run path from another script, but # hard from something in lib. So, we set it here for now. ch.CH_BIN = os.path.dirname(os.path.abspath( inspect.getframeinfo(inspect.currentframe()).filename)) ch.CH_RUN = ch.CH_BIN + "/ch-run" ## Main ## def main(): if (not os.path.exists(ch.CH_RUN)): ch.depfails.append(("missing", ch.CH_RUN)) ap = ch.ArgumentParser( description="Build and manage images; completely unprivileged.", epilog="""Storage directory is used for caching and temporary images. Location: first defined of --storage, $CH_IMAGE_STORAGE, and %s.""" % fs.Storage.root_default(), sub_title="subcommands", sub_metavar="CMD") # Common options. # # --dependencies (and --help and --version) are options rather than # subcommands for consistency with other commands. # # These are also accepted *after* the subcommand, as it makes wrapping # ch-image easier and possibly improve the UX. There are multiple ways to # do this, though no tidy ones unfortunately. Here, we build up a # dictionary of options we want, and pass it to both main and subcommand # parsers; this works because both go into the same Namespace object. There # are two quirks to be aware of: # # 1. We omit the common options from subcommand --help for clarity and # because before the subcommand is preferred. # # 2. We suppress defaults in the subcommand [1]. Without this, the # subcommand option value wins even it it’s the default. :P Currently, # if specified in both places, the subcommand value wins and the # before value is not considered at all, e.g. "ch-image -vv foo -v" # gives verbosity 1, not 3. This oddity seemed acceptable. # # Alternate approaches include: # # * Set the main parser as the “parent” of the subcommand parser [2]. # This may be the documented approach? However, it adds all the # subcommands to the subparser, which we don’t want. A workaround would # be to create a *third* parser that’s the parent of both the main and # subcommand parsers, but that seems like too much indirection to me. # # * A two-stage parse (parse_known_args(), then parse_args() to have the # main parser look again) works [3], but is complicated and has some # odd side effects e.g. multiple subcommands will be accepted. # # Each sub-list is a group of options. They key identifies the mutually # exclustive group, or non-mutually exclusive if None. # # [1]: https://bugs.python.org/issue9351#msg373665 # [2]: https://docs.python.org/3/library/argparse.html#parents # [3]: https://stackoverflow.com/a/54936198 common_opts = \ { ("bucache", "build cache common options"): [ [["--cache"], { "action": "store_const", "const": ch.Build_Mode.ENABLED, "dest": "bucache", "help": "enable build cache" }], [["--no-cache"], { "action": "store_const", "const": ch.Build_Mode.DISABLED, "dest": "bucache", "help": "disable build cache" }], [["--rebuild"], { "action": "store_const", "const": ch.Build_Mode.REBUILD, "dest": "bucache", "help": "force cache misses for non-FROM instructions" }] ], (None, "misc common options"): [ [["-a", "--arch"], { "metavar": "ARCH", "default": "host", "help": "architecture for image registries (default: host)"}], [["--always-download"], { "action": "store_true", "help": "redownload any image files when pulling"}], [["--auth"], { "action": "store_true", "help": "authenticated registry access; implied by push" }], [["--break"], { "metavar": "MODULE:LINE", "help": "break into PDB before LINE of MODULE" }], [["--cache-large"], { "metavar": "SIZE", "type": lambda s: ch.positive(s) * 2**20, # internal unit: bytes "default": ch.positive( os.environ.get("CH_IMAGE_CACHE_LARGE", 0)) * 2**20, "help": "large file threshold in MiB" }], [["--debug"], { "action": "store_true", "help": "add short traceback to fatal error hints" }], [["--dependencies"], { "action": misc.Dependencies, "help": "print any missing dependencies and exit" }], [["--no-lock"], { "action": "store_true", "help": "allow concurrent storage directory access (risky!)" }], [["--no-xattrs"], { "action": "store_true", "help": "disable xattrs and ACLs (overrides $CH_XATTRS)" }], [["--password-many"], { "action": "store_true", "help": "re-prompt each time a registry password is needed" }], [["--profile"], { "action": "store_true", "help": "dump profile to “./profile.{p,txt}”" }], [["-q", "--quiet"], { "action": "count", "default": 0, "help": "print less output (can be repeated)"}], [["-s", "--storage"], { "metavar": "DIR", "type": fs.Path, "help": "set builder internal storage directory to DIR" }], [["--tls-no-verify"], { "action": "store_true", "help": "don’t verify registry certificates (dangerous!)" }], [["-v", "--verbose"], { "action": "count", "default": 0, "help": "print extra chatter (can be repeated)" }], [["--version"], { "action": misc.Version, "help": "print version and exit" }], [["--xattrs"], { "action": "store_true", "help": "enable build cache support for xattrs and ACLs"}] ] } # Most, but not all, subcommands need to check dependencies before doing # anything (the exceptions being basic information commands like # storage-path). Similarly, only some need to initialize the storage # directory. These dictionaries map the dispatch function to a boolean # value saying whether to do those things. dependencies_check = dict() storage_init = dict() # Helper function to set up a subparser. The star forces the latter two # arguments to be called by keyword, for clarity. def add_opts(p, dispatch, *, deps_check, stog_init, help_=False): assert (not stog_init or deps_check) # can’t init storage w/o deps if (dispatch is not None): p.set_defaults(func=dispatch) dependencies_check[dispatch] = deps_check storage_init[dispatch] = stog_init for ((name, title), group) in common_opts.items(): if (name is None): p2 = p.add_argument_group(title=title) else: p2 = p.add_argument_group(title=title) p2 = p2.add_mutually_exclusive_group() for (args, kwargs) in group: if (help_): kwargs2 = kwargs else: kwargs2 = { **kwargs, "default": argparse.SUPPRESS } p2.add_argument(*args, **kwargs2) # main parser add_opts(ap, None, deps_check=False, stog_init=False, help_=True) # build sp = ap.add_parser("build", "build image from Dockerfile") add_opts(sp, build.main, deps_check=True, stog_init=True) sp.add_argument("-b", "--bind", metavar="SRC[:DST]", action="append", default=[], help="mount SRC at guest DST (default: same as SRC)") sp.add_argument("--build-arg", metavar="ARG[=VAL]", action="append", default=[], help="set build-time variable ARG to VAL, or $ARG if no VAL") sp.add_argument("-f", "--file", metavar="DOCKERFILE", help="Dockerfile to use (default: CONTEXT/Dockerfile)") sp.add_argument("--force", metavar="MODE", nargs="?", default="seccomp", type=ch.Force_Mode, const="seccomp", help="inject unprivileged build workarounds") sp.add_argument("--force-cmd", metavar="CMD,ARG1[,ARG2...]", action="append", default=[], help="command arg(s) to add under --force=seccomp") sp.add_argument("-n", "--dry-run", action="store_true", help="don’t execute instructions") sp.add_argument("--parse-only", action="store_true", help="stop after parsing the Dockerfile") sp.add_argument("-t", "--tag", metavar="TAG", help="name (tag) of image to create (default: inferred)") sp.add_argument("context", metavar="CONTEXT", help="context directory") # build-cache sp = ap.add_parser("build-cache", "print build cache information") add_opts(sp, misc.build_cache, deps_check=True, stog_init=True) sp.add_argument("--gc", action="store_true", help="run garbage collection first") sp.add_argument("--reset", action="store_true", help="clear and re-initialize first") sp.add_argument("--tree", action="store_true", help="print a text tree summary") sp.add_argument("--dot", nargs="?", metavar="PATH", const="build-cache", help="write DOT and PDF tree summaries") # delete sp = ap.add_parser("delete", "delete image from internal storage") add_opts(sp, misc.delete, deps_check=True, stog_init=True) sp.add_argument("image_ref", metavar="IMAGE_GLOB", help="image(s) to delete", nargs='+') # gestalt (has sub-subcommands) sp = ap.add_parser("gestalt", "query debugging configuration", sub_title="subsubcommands", sub_metavar="CMD") add_opts(sp, lambda x: False, deps_check=False, stog_init=False) # bucache tp = sp.add_parser("bucache", "exit successfully if build cache available") add_opts(tp, misc.gestalt_bucache, deps_check=True, stog_init=False) # bucache-dot tp = sp.add_parser("bucache-dot", "exit success if can produce DOT trees") add_opts(tp, misc.gestalt_bucache_dot, deps_check=True, stog_init=False) # storage-path tp = sp.add_parser("storage-path", "print storage directory path") add_opts(tp, misc.gestalt_storage_path, deps_check=False, stog_init=False) # python-path tp = sp.add_parser("python-path", "print path to python interpreter in use") add_opts(tp, misc.gestalt_python_path, deps_check=False, stog_init=False) # logging tp = sp.add_parser("logging", "print logging messages at all levels") add_opts(tp, misc.gestalt_logging, deps_check=False, stog_init=False) tp.add_argument("--fail", action="store_true", help="also generate a fatal error") # import sp = ap.add_parser("import", "copy external image into storage") add_opts(sp, misc.import_, deps_check=True, stog_init=True) sp.add_argument("path", metavar="PATH", help="directory or tarball to import") sp.add_argument("image_ref", metavar="IMAGE_REF", help="destination image name (tag)") # list sp = ap.add_parser("list", "print information about image(s)") add_opts(sp, misc.list_, deps_check=True, stog_init=True) sp.add_argument("-l", "--long", action="store_true", help="use long listing format") sp.add_argument("-u", "--undeletable", action="store_true", help="list images that can be restored with “undelete”") sp.add_argument("--undeleteable", action="store_true", dest="undeletable", help=argparse.SUPPRESS) sp.add_argument("image_ref", metavar="IMAGE_REF", nargs="?", help="print details of this image only") # pull sp = ap.add_parser("pull", "copy image from remote repository to local filesystem") add_opts(sp, pull.main, deps_check=True, stog_init=True) sp.add_argument("--last-layer", metavar="N", type=int, help="stop after unpacking N layers") sp.add_argument("--parse-only", action="store_true", help="stop after parsing the image reference(s)") sp.add_argument("source_ref", metavar="IMAGE_REF", help="image reference") sp.add_argument("dest_ref", metavar="DEST_REF", nargs="?", help="destination image reference (default: IMAGE_REF)") # push sp = ap.add_parser("push", "copy image from local filesystem to remote repository") add_opts(sp, push.main, deps_check=True, stog_init=True) sp.add_argument("--image", metavar="DIR", type=fs.Path, help="path to unpacked image (default: opaque path in storage dir)") sp.add_argument("source_ref", metavar="IMAGE_REF", help="image to push") sp.add_argument("dest_ref", metavar="DEST_REF", nargs="?", help="destination image reference (default: IMAGE_REF)") # reset sp = ap.add_parser("reset", "delete everything in ch-image builder storage") add_opts(sp, misc.reset, deps_check=True, stog_init=False) # undelete sp = ap.add_parser("undelete", "recover image from build cache") add_opts(sp, misc.undelete, deps_check=True, stog_init=True) sp.add_argument("image_ref", metavar="IMAGE_REF", help="image to recover") # Monkey patch problematic characters out of stdout and stderr. ch.monkey_write_streams() # Parse it up! if (len(sys.argv) < 2): ap.print_help(file=sys.stderr) ch.exit(1) cli = ap.parse_args() # Initialize. ch.init(cli) if (dependencies_check[cli.func]): ch.dependencies_check() if (storage_init[cli.func]): ch.storage.init() bu.init(cli) # Dispatch. ch.profile_start() cli.func(cli) ch.warnings_dump() ch.exit(0) ## Functions ## def breakpoint_inject(module_name, line_no): # Inject a PDB breakpoint into the module named module_name before the # statement on line line_no. See: https://stackoverflow.com/a/41858422 class PDB_Injector(ast.NodeTransformer): def __init__(self, *args, **kwargs): self.inject_ct = 0 return super().__init__(*args, **kwargs) def generic_visit(self, parent): # Operate on parent of target statement because we need to inject the # new code into the parent’s body (i.e., as siblings of the target # statement). if ( self.inject_ct == 0 and hasattr(parent, "body") and isinstance(parent.body, collections.abc.Sequence)): for (i, child) in enumerate(parent.body): if ( isinstance(child, ast.stmt) and hasattr(child, "lineno") and child.lineno == line_no): ch.WARNING( "--break: injecting PDB breakpoint: %s:%d (%s)" % (module_name, line_no, type(child).__name__)) parent.body[i:i] = inject_tree.body self.inject_ct += 1 break super().generic_visit(parent) # superclass actually visits children return parent if (module_name not in sys.modules): ch.FATAL("--break: no module named %s" % module_name) module = sys.modules[module_name] src_text = inspect.getsource(module) src_path = inspect.getsourcefile(module) module_tree = ast.parse(src_text, "%s " % src_path) inject_tree = ast.parse("import pdb; pdb.set_trace()", "Weird Al Yankovic") ijor = PDB_Injector() ijor.visit(module_tree) # calls generic_visit() on all nodes if (ijor.inject_ct < 1): ch.FATAL("--break: no statement found at %s:%d" % (module_name, line_no)) assert (ijor.inject_ct == 1) ast.fix_missing_locations(module_tree) exec(compile(module_tree, "%s " % src_path, "exec"), module.__dict__) # Set a global in the target module so it can test if it’s been # re-executed. This means re-execution is *complete*, so it will not be set # in module-level code run during re-execution, but if the original # execution continues *after* re-execution completes (this happens for # __main__), it *will* be set in that code. module.__dict__["breakpoint_reexecuted"] = "%s:%d" % (module_name, line_no) ## Bootstrap ## # This code is more complicated than the standard boilerplace (i.e., “if # (__name__ == "__main__"): main()”) for two reasons: # # 1. The mechanism for fatal errors is to raise ch.Fatal_Error. We catch # this to re-print warnings and print the error message before exiting. # (We used to priont an error message and then sys.exit(1), but this # approach lets us do things like rollback and fixes ordering problems # such as #1486.) # # 2. There is a big mess of hairy code to let us set PDB breakpoints in this # file (i.e., module __main__) with --break. See PR #1837. if (__name__ == "__main__"): try: # We can’t set these two module globals that support --break normally # (i.e., module-level code at the top of this file) because this module # might be executed twice, and thus any value we set would be # overwritten by the default when the module is re-executed. if ("breakpoint_considered" not in globals()): global breakpoint_considered breakpoint_considered = True # A few lines of bespoke CLI parsing so that we can inject # breakpoints into the CLI parsing code itself. for (opt, arg) in zip(sys.argv[1:], sys.argv[2:] + [None]): (opt, _, arg_eq) = opt.partition("=") if (opt == "--break"): if (arg_eq != ""): arg = arg_eq try: (module_name, line_no) = arg.split(":") line_no = int(line_no) except ValueError: ch.FATAL("--break: can’t parse MODULE:LIST: %s" % arg) breakpoint_inject(module_name, line_no) # If we injected into __main__, we already ran main() when re-executing # this module inside breakpoint_inject(). if ("breakpoint_reexecuted" not in globals()): main() except ch.Fatal_Error as x: ch.warnings_dump() ch.ERROR(*x.args, **x.kwargs) ch.exit(1) charliecloud-0.37/bin/ch-run-oci.py.in000066400000000000000000000256501457016721300176120ustar00rootroot00000000000000#!%PYTHON_SHEBANG% import argparse import inspect import json import os import re import signal import subprocess import sys import time import types ch_lib = os.path.dirname(os.path.abspath(__file__)) + "/../lib" sys.path.insert(0, ch_lib) import charliecloud as ch import misc import filesystem as fs BUNDLE_PREFIX = ["/tmp", "/var/tmp"] CH_BIN = os.path.dirname(os.path.abspath( inspect.getframeinfo(inspect.currentframe()).filename)) OCI_VERSION_MIN = "1.0.1" # inclusive OCI_VERSION_MAX = "1.0.999" # inclusive args = None # CLI Namespace state = None # state object def main(): global args, state ch.monkey_write_streams() args = args_parse() ch.VERBOSE("--- starting ------------------------------------") ch.VERBOSE("args: %s" % sys.argv) ch.VERBOSE("environment: %s" % { k: v for (k, v) in os.environ.items() if k.startswith("CH_RUN_OCI_") }) ch.VERBOSE("CLI: %s" % args) if (args.op.__name__ == "op_" + os.getenv("CH_RUN_OCI_HANG", default="")): ch.VERBOSE("hanging before %s per CH_RUN_OCI_HANG" % args.op.__name__) sleep_forever() assert False, "unreachable code reached" state = state_load() args.op() ch.VERBOSE("done") def args_parse(): ap = argparse.ArgumentParser(description='OCI wrapper for "ch-run".') ap.add_argument("-v", "--verbose", action="count", default=0, help="print extra chatter (can be repeated)") ap.add_argument("--debug", action="store_true", help="add short traceback to fatal error hints") ap.add_argument("--version", action=misc.Version, help="print version and exit") sps = ap.add_subparsers() sp = sps.add_parser("create") sp.set_defaults(op=op_create) sp.add_argument("--bundle", required=True, metavar="DIR") sp.add_argument("--console-socket", metavar="PATH") sp.add_argument("--pid-file", required=True, metavar="FILE") sp.add_argument("--no-new-keyring", action="store_true") sp.add_argument("cid", metavar="CONTAINER_ID") sp = sps.add_parser("delete") sp.set_defaults(op=op_delete) sp.add_argument("cid", metavar="CONTAINER_ID") sp = sps.add_parser("kill") sp.set_defaults(op=op_kill) sp.add_argument("cid", metavar="CONTAINER_ID") sp.add_argument("signal", metavar="SIGNAL") sp = sps.add_parser("start") sp.set_defaults(op=op_start) sp.add_argument("cid", metavar="CONTAINER_ID") sp = sps.add_parser("state") sp.set_defaults(op=op_state) sp.add_argument("cid", metavar="CONTAINER_ID") args_ = ap.parse_args() args_.arch = "yolo" # dummy args to make charliecloud.init() happy args_.always_download = None args_.auth = None args_.func = abs # needs to have __module__ attribute args_.no_cache = None args_.no_lock = False args_.no_xattrs = False args_.password_many = False args_.profile = False args_.quiet = False args_.storage = None args_.tls_no_verify = False args_.xattrs = False ch.init(args_) if len(sys.argv) < 2: ap.print_help(file=sys.stderr) ch.exit(1) bundle_ = bundle_from_cid(args_.cid) if ("bundle" in args_ and args_.bundle != bundle_): ch.FATAL("bundle argument “%s” differs from inferred bundle “%s”" % (args_.bundle, bundle_)) args_.bundle = bundle_ pid_file_ = pid_file_from_bundle(args_.bundle) if ("pid_file" in args_ and args_.pid_file != pid_file_): ch.FATAL("pid_file argument “%s” differs from inferred “%s”" % (args_.pid_file, pid_file_)) args_.pid_file = pid_file_ return args_ def bundle_from_cid(cid): m = re.search(r"^buildah-buildah(.+)$", cid) if (m is None): ch.FATAL("cannot parse container ID: %s" % cid) paths = [] for p in BUNDLE_PREFIX: paths.append("%s/buildah%s" % (p, m[1])) if (os.path.exists(paths[-1])): return paths[-1] ch.FATAL("can’t infer bundle path; none of these exist: %s" % " ".join(paths)) def debug_lines(s): for line in s.splitlines(): ch.VERBOSE(line) def image_fixup(path): ch.VERBOSE("fixing up image: %s" % path) # Metadata directory. fs.Path("%s/ch/bin" % path).mkdirs() # Mount points. fs.Path("%s/etc/hosts" % path).file_ensure_exists() fs.Path("%s/etc/resolv.conf" % path).file_ensure_exists() # /etc/{passwd,group} fs.Path("%s/etc/passwd" % path).file_write("""\ root:x:0:0:root:/root:/bin/sh nobody:x:65534:65534:nobody:/:/bin/false """) fs.Path("%s/etc/group" % path).file_write("""\ root:x:0: nogroup:x:65534: """) # Kludges to work around expectations of real root, not UID 0 in a # unprivileged user namespace. See also the default environment. # # Debian apt/dpkg/etc. want to chown(1), chgrp(1), etc. in various ways. fs.Path(path, "ch/bin/chgrp").symlink_to("/bin/true") fs.Path(path, "ch/bin/dpkg-statoverride").symlink_to("/bin/true") # Debian package management also wants to mess around with users. This is # causing problems with /etc/gshadow and other files. These links don’t # work if they are in /ch/bin, I think because dpkg is resetting the path? # For now we’ll do this, but I don’t like it. fakeroot(1) also solves the # problem (see issue #472). fs.Path(path, "bin/chown").symlink_to("/bin/true", clobber=True) fs.Path(path, "usr/sbin/groupadd").symlink_to("/bin/true", clobber=True) fs.Path(path, "usr/sbin/useradd").symlink_to("/bin/true", clobber=True) fs.Path(path, "usr/sbin/usermod").symlink_to("/bin/true", clobber=True) fs.Path(path, "usr/bin/chage").symlink_to("/bin/true", clobber=True) def op_create(): # Validate arguments. if (args.console_socket): ch.FATAL("--console-socket not supported") # Start dummy supervisor. if (state.pid is not None): ch.FATAL("container already created") pid = ch.ossafe("can’t fork", os.fork) if (pid == 0): # Child; the only reason to exist is so Buildah sees a process when it # looks for one. Sleep until told to exit. # # Note: I looked into changing the process title and this turns out to # be remarkably hairy unless you use a 3rd-party module. def exit_(sig, frame): ch.VERBOSE("dummy supervisor: done") ch.exit(0) signal.signal(signal.SIGTERM, exit_) ch.VERBOSE("dummy supervisor: starting") sleep_forever() else: state.pid = pid with args.pid_file.open("wt") as fp: print("%d" % pid, file=fp) ch.VERBOSE("dummy supervisor started with pid %d" % pid) def op_delete(): ch.VERBOSE("delete operation is a no-op") def op_kill(): ch.VERBOSE("kill operation is a no-op") def op_start(): # Note: Contrary to the implication of its name, the “start” operation # blocks until the user command is done. c = state.config # Unsupported features to barf about. if (state.pid is None): ch.FATAL("can’t start: not created yet") if (c["process"].get("terminal", False)): ch.FATAL("not supported: pseudoterminals") if ("annotations" in c): ch.FATAL("not supported: annotations") if ("hooks" in c): ch.FATAL("not supported: hooks") for d in c["linux"]["namespaces"]: if ("path" in d): ch.FATAL("not supported: joining existing namespaces") if ("intelRdt" in c["linux"]): ch.FATAL("not supported: Intel RDT") # Environment file. This is a list of lines, not a dict. # # GNU tar, when it thinks it’s running as root, tries to chown(2) and # chgrp(2) files to whatever’s in the tarball. --no-same-owner avoids this. with fs.Path(args.bundle + "/environment").open("wt") as fp: for line in ( c["process"]["env"] # from Dockerfile + [ "TAR_OPTIONS=--no-same-owner" ]): # ours line = re.sub(r"^(PATH=)", "\\1/ch/bin:", line) ch.VERBOSE("env: %s" % line) print(line, file=fp) # Build command line. cmd = CH_BIN + "/ch-run" ca = [cmd, "--cd", c["process"]["cwd"], "--no-passwd", "--gid", str(c["process"]["user"]["gid"]), "--uid", str(c["process"]["user"]["uid"]), "--unset-env=*", "--set-env=%s/environment" % args.bundle] if (not c["root"].get("readonly", False)): ca.append("--write") ca += [c["root"]["path"], "--"] ca += c["process"]["args"] # Fix up root filesystem. image_fixup(args.bundle + "/mnt/rootfs") # Execute user command. We can’t execv(2) because we have to do cleanup # after it exits. fs.Path(args.bundle + "/user_started").file_ensure_exists() ch.VERBOSE("user command: %s" % ca) # Standard output disappears, so send stdout to stderr. cp = subprocess.run(ca, stdout=2) fs.Path(args.bundle + "/user_done").file_ensure_exists() ch.VERBOSE("user command done") # Stop dummy supervisor. if (state.pid is None): ch.FATAL("no dummy supervisor PID found") try: os.kill(state.pid, signal.SIGTERM) state.pid = None os.unlink(args.pid_file) except OSError as x: ch.FATAL("can’t kill PID %d: %s (%d)" % (state.pid, x.strerror, x.errno)) # Puke if user command failed. if (cp.returncode != 0): ch.FATAL("user command failed: %d" % cp.returncode) def op_state(): def status(): if (state.user_command_started): if (state.user_command_done): return "stopped" else: return "running" if (state.pid is None): return "creating" else: return "created" st = { "ociVersion": OCI_VERSION_MAX, "id": args.cid, "status": status(), "bundle": args.bundle } if (state.pid is not None): st["pid"] = state.pid out = json.dumps(st, indent=2) debug_lines(out) print(out) def sleep_forever(): while True: time.sleep(60) # can’t provide infinity here def pid_file_from_bundle(bundle): return bundle + "/pid" def state_load(): st = types.SimpleNamespace() st.config = fs.Path(args.bundle, "config.json").json_from_file("state") #debug_lines(json.dumps(st.config, indent=2)) v_min = version_parse_oci(OCI_VERSION_MIN) v_actual = version_parse_oci(st.config["ociVersion"]) v_max = version_parse_oci(OCI_VERSION_MAX) if (not v_min <= v_actual <= v_max): ch.FATAL("unsupported OCI version: %s" % st.config["ociVersion"]) try: fp = open(args.pid_file, "rt") st.pid = int(ch.ossafe("can’t read: %s" % args.pid_file, fp.read)) ch.VERBOSE("found supervisor pid: %d" % st.pid) except FileNotFoundError: st.pid = None ch.VERBOSE("no supervisor pid found") st.user_command_started = os.path.isfile(args.bundle + "/user_started") st.user_command_done = os.path.isfile(args.bundle + "/user_done") return st def version_parse_oci(s): # Dead-simple version parsing for OCI; not intended for other uses. return tuple(re.split(r"[.-]", s)[:3]) if (__name__ == "__main__"): try: main() except ch.Fatal_Error as x: ch.ERROR(*x.args, **x.kwargs) ch.exit(1) charliecloud-0.37/bin/ch-run.c000066400000000000000000000473451457016721300162340ustar00rootroot00000000000000/* Copyright © Triad National Security, LLC, and others. */ /* Note: This program does not bother to free memory allocations, since they are modest and the program is short-lived. */ #define _GNU_SOURCE #include #include #include #include #include #include #include #include "config.h" #include "ch_core.h" #include "ch_misc.h" /** Constants and macros **/ /* Environment variables used by --join parameters. */ char *JOIN_CT_ENV[] = { "OMPI_COMM_WORLD_LOCAL_SIZE", "SLURM_STEP_TASKS_PER_NODE", "SLURM_CPUS_ON_NODE", NULL }; char *JOIN_TAG_ENV[] = { "SLURM_STEP_ID", NULL }; /* Default overlaid tmpfs size. */ char *WRITE_FAKE_DEFAULT = "12%"; /** Command line options **/ const char usage[] = "\ \n\ Run a command in a Charliecloud container.\n\ \v\ Example:\n\ \n\ $ ch-run /data/foo -- echo hello\n\ hello\n\ \n\ You cannot use this program to actually change your UID.\n"; const char args_doc[] = "IMAGE -- COMMAND [ARG...]"; /* Note: Long option numbers, once issued, are permanent; i.e., if you remove one, don’t re-number the others. */ const struct argp_option options[] = { { "bind", 'b', "SRC[:DST]", 0, "mount SRC at guest DST (default: same as SRC)"}, { "cd", 'c', "DIR", 0, "initial working directory in container"}, { "env-no-expand", -10, 0, 0, "don't expand $ in --set-env input"}, { "feature", -11, "FEAT", 0, "exit successfully if FEAT is enabled" }, { "gid", 'g', "GID", 0, "run as GID within container" }, { "home", -12, 0, 0, "mount host $HOME at guest /home/$USER" }, { "join", 'j', 0, 0, "use same container as peer ch-run" }, { "join-pid", -5, "PID", 0, "join a namespace using a PID" }, { "join-ct", -3, "N", 0, "number of join peers (implies --join)" }, { "join-tag", -4, "TAG", 0, "label for peer group (implies --join)" }, { "test", -17, "TEST", 0, "do test TEST" }, { "mount", 'm', "DIR", 0, "SquashFS mount point"}, { "no-passwd", -9, 0, 0, "don't bind-mount /etc/{passwd,group}"}, { "private-tmp", 't', 0, 0, "use container-private /tmp" }, { "quiet", 'q', 0, 0, "print less output (can be repeated)"}, #ifdef HAVE_SECCOMP { "seccomp", -14, 0, 0, "fake success for some syscalls with seccomp(2)"}, #endif { "set-env", -6, "ARG", OPTION_ARG_OPTIONAL, "set env. variables per ARG (newline-delimited)"}, { "set-env0", -15, "ARG", OPTION_ARG_OPTIONAL, "set env. variables per ARG (null-delimited)"}, { "storage", 's', "DIR", 0, "set DIR as storage directory"}, { "uid", 'u', "UID", 0, "run as UID within container" }, { "unsafe", -13, 0, 0, "do unsafe things (internal use only)" }, { "unset-env", -7, "GLOB", 0, "unset environment variable(s)" }, { "verbose", 'v', 0, 0, "be more verbose (can be repeated)" }, { "version", 'V', 0, 0, "print version and exit" }, { "warnings", -16, "NUM", 0, "log NUM warnings and exit" }, { "write", 'w', 0, 0, "mount image read-write (avoid)"}, { "write-fake", 'W', "SIZE", OPTION_ARG_OPTIONAL, "overlay read-write tmpfs on top of image" }, { 0 } }; /** Types **/ struct args { struct container c; struct env_delta *env_deltas; char *initial_dir; #ifdef HAVE_SECCOMP bool seccomp_p; #endif char *storage_dir; bool unsafe; }; /** Function prototypes **/ void fix_environment(struct args *args); bool get_first_env(char **array, char **name, char **value); void img_directory_verify(const char *img_path, const struct args *args); int join_ct(int cli_ct); char *join_tag(char *cli_tag); int parse_int(char *s, bool extra_ok, char *error_tag); static error_t parse_opt(int key, char *arg, struct argp_state *state); void parse_set_env(struct args *args, char *arg, int delim); void privs_verify_invoking(); char *storage_default(void); extern void warnings_reprint(void); /** Global variables **/ const struct argp argp = { options, parse_opt, args_doc, usage }; extern char **environ; // see environ(7) extern char *warnings; /** Main **/ int main(int argc, char *argv[]) { bool argp_help_fmt_set; struct args args; int arg_next; char ** c_argv; // initialize “warnings” buffer warnings = mmap(NULL, WARNINGS_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); T_ (warnings != MAP_FAILED); privs_verify_invoking(); Z_ (atexit(warnings_reprint)); #ifdef ENABLE_SYSLOG syslog(LOG_USER|LOG_INFO, "uid=%u args=%d: %s", getuid(), argc, argv_to_string(argv)); #endif username = getenv("USER"); Te (username != NULL, "$USER not set"); verbose = LL_INFO; // in ch_misc.c args = (struct args){ .c = (struct container){ .binds = list_new(sizeof(struct bind), 0), .container_gid = getegid(), .container_uid = geteuid(), .env_expand = true, .host_home = NULL, .img_ref = NULL, .newroot = NULL, .join = false, .join_ct = 0, .join_pid = 0, .join_tag = NULL, .overlay_size = NULL, .private_passwd = false, .private_tmp = false, .type = IMG_NONE, .writable = false }, .env_deltas = list_new(sizeof(struct env_delta), 0), .initial_dir = NULL, #ifdef HAVE_SECCOMP .seccomp_p = false, #endif .storage_dir = storage_default(), .unsafe = false }; /* I couldn't find a way to set argp help defaults other than this environment variable. Kludge sets/unsets only if not already set. */ if (getenv("ARGP_HELP_FMT")) argp_help_fmt_set = true; else { argp_help_fmt_set = false; Z_ (setenv("ARGP_HELP_FMT", "opt-doc-col=27,no-dup-args-note", 0)); } Z_ (argp_parse(&argp, argc, argv, 0, &arg_next, &args)); if (!argp_help_fmt_set) Z_ (unsetenv("ARGP_HELP_FMT")); if (arg_next >= argc - 1) { printf("usage: ch-run [OPTION...] IMAGE -- COMMAND [ARG...]\n"); FATAL("IMAGE and/or COMMAND not specified"); } args.c.img_ref = argv[arg_next++]; args.c.newroot = realpath_(args.c.newroot, true); args.storage_dir = realpath_(args.storage_dir, true); args.c.type = image_type(args.c.img_ref, args.storage_dir); switch (args.c.type) { case IMG_DIRECTORY: if (args.c.newroot != NULL) // --mount was set WARNING("--mount invalid with directory image, ignoring"); args.c.newroot = realpath_(args.c.img_ref, false); img_directory_verify(args.c.newroot, &args); break; case IMG_NAME: args.c.newroot = img_name2path(args.c.img_ref, args.storage_dir); Tf (!args.c.writable || args.unsafe, "--write invalid when running by name"); break; case IMG_SQUASH: #ifndef HAVE_LIBSQUASHFUSE FATAL("this ch-run does not support internal SquashFS mounts"); #endif break; case IMG_NONE: FATAL("unknown image type: %s", args.c.img_ref); break; } if (args.c.join) { args.c.join_ct = join_ct(args.c.join_ct); args.c.join_tag = join_tag(args.c.join_tag); } if (getenv("TMPDIR") != NULL) host_tmp = getenv("TMPDIR"); else host_tmp = "/tmp"; c_argv = list_new(sizeof(char *), argc - arg_next); for (int i = 0; i < argc - arg_next; i++) c_argv[i] = argv[i + arg_next]; VERBOSE("verbosity: %d", verbose); VERBOSE("image: %s", args.c.img_ref); VERBOSE("storage: %s", args.storage_dir); VERBOSE("newroot: %s", args.c.newroot); VERBOSE("container uid: %u", args.c.container_uid); VERBOSE("container gid: %u", args.c.container_gid); VERBOSE("join: %d %d %s %d", args.c.join, args.c.join_ct, args.c.join_tag, args.c.join_pid); VERBOSE("private /tmp: %d", args.c.private_tmp); #ifdef HAVE_SECCOMP VERBOSE("seccomp: %d", args.seccomp_p); #endif VERBOSE("unsafe: %d", args.unsafe); containerize(&args.c); fix_environment(&args); #ifdef HAVE_SECCOMP if (args.seccomp_p) seccomp_install(); #endif run_user_command(c_argv, args.initial_dir); // should never return exit(EXIT_FAILURE); } /** Supporting functions **/ /* Adjust environment variables. Call once containerized, i.e., already pivoted into new root. */ void fix_environment(struct args *args) { char *old_value, *new_value; // $HOME: If --home, set to “/home/$USER”. if (args->c.host_home) { Z_ (setenv("HOME", cat("/home/", username), 1)); } else if (path_exists("/root", NULL, true)) { Z_ (setenv("HOME", "/root", 1)); } else Z_ (setenv("HOME", "/", 1)); // $PATH: Append /bin if not already present. old_value = getenv("PATH"); if (old_value == NULL) { WARNING("$PATH not set"); } else if ( strstr(old_value, "/bin") != old_value && !strstr(old_value, ":/bin")) { T_ (1 <= asprintf(&new_value, "%s:/bin", old_value)); Z_ (setenv("PATH", new_value, 1)); VERBOSE("new $PATH: %s", new_value); } // $TMPDIR: Unset. Z_ (unsetenv("TMPDIR")); // --set-env and --unset-env. for (size_t i = 0; args->env_deltas[i].action != ENV_END; i++) { struct env_delta ed = args->env_deltas[i]; switch (ed.action) { case ENV_END: Te (false, "unreachable code reached"); break; case ENV_SET_DEFAULT: ed.arg.vars = env_file_read("/ch/environment", ed.arg.delim); // fall through case ENV_SET_VARS: for (size_t j = 0; ed.arg.vars[j].name != NULL; j++) env_set(ed.arg.vars[j].name, ed.arg.vars[j].value, args->c.env_expand); break; case ENV_UNSET_GLOB: env_unset(ed.arg.glob); break; } } // $CH_RUNNING is not affected by --unset-env or --set-env. Z_ (setenv("CH_RUNNING", "Weird Al Yankovic", 1)); } /* Find the first environment variable in array that is set; put its name in *name and its value in *value, and return true. If none are set, return false, and *name and *value are undefined. */ bool get_first_env(char **array, char **name, char **value) { for (int i = 0; array[i] != NULL; i++) { *name = array[i]; *value = getenv(*name); if (*value != NULL) return true; } return false; } /* Validate that it’s OK to run the IMG_DIRECTORY format image at path; if not, exit with error. */ void img_directory_verify(const char *newroot, const struct args *args) { Te (args->c.newroot != NULL, "can't find image: %s", args->c.newroot); Te (args->unsafe || !path_subdir_p(args->storage_dir, args->c.newroot), "can't run directory images from storage (hint: run by name)"); } /* Find an appropriate join count; assumes --join was specified or implied. Exit with error if no valid value is available. */ int join_ct(int cli_ct) { int j = 0; char *ev_name, *ev_value; if (cli_ct != 0) { VERBOSE("join: peer group size from command line"); j = cli_ct; goto end; } if (get_first_env(JOIN_CT_ENV, &ev_name, &ev_value)) { VERBOSE("join: peer group size from %s", ev_name); j = parse_int(ev_value, true, ev_name); goto end; } end: Te(j > 0, "join: no valid peer group size found"); return j; } /* Find an appropriate join tag; assumes --join was specified or implied. Exit with error if no valid value is found. */ char *join_tag(char *cli_tag) { char *tag; char *ev_name, *ev_value; if (cli_tag != NULL) { VERBOSE("join: peer group tag from command line"); tag = cli_tag; goto end; } if (get_first_env(JOIN_TAG_ENV, &ev_name, &ev_value)) { VERBOSE("join: peer group tag from %s", ev_name); tag = ev_value; goto end; } VERBOSE("join: peer group tag from getppid(2)"); T_ (1 <= asprintf(&tag, "%d", getppid())); end: Te(tag[0] != '\0', "join: peer group tag cannot be empty string"); return tag; } /* Parse an integer string arg and return the result. If an error occurs, print a message prefixed by error_tag and exit. If not extra_ok, additional characters remaining after the integer are an error. */ int parse_int(char *s, bool extra_ok, char *error_tag) { char *end; long l; errno = 0; l = strtol(s, &end, 10); Ze (end == s, "%s: no digits found", error_tag); Ze (errno == ERANGE || l < INT_MIN || l > INT_MAX, "%s: out of range", error_tag); Tf (errno == 0, error_tag); if (!extra_ok) Te (*end == 0, "%s: extra characters after digits", error_tag); return (int)l; } /* Parse one command line option. Called by argp_parse(). */ static error_t parse_opt(int key, char *arg, struct argp_state *state) { struct args *args = state->input; int i; switch (key) { case -3: // --join-ct args->c.join = true; args->c.join_ct = parse_int(arg, false, "--join-ct"); break; case -4: // --join-tag args->c.join = true; args->c.join_tag = arg; break; case -5: // --join-pid args->c.join_pid = parse_int(arg, false, "--join-pid"); break; case -6: // --set-env parse_set_env(args, arg, '\n'); break; case -7: { // --unset-env struct env_delta ed; Te (strlen(arg) > 0, "--unset-env: GLOB must have non-zero length"); ed.action = ENV_UNSET_GLOB; ed.arg.glob = arg; list_append((void **)&(args->env_deltas), &ed, sizeof(ed)); } break; case -9: // --no-passwd args->c.private_passwd = true; break; case -10: // --env-no-expand args->c.env_expand = false; break; case -11: // --feature if (!strcmp(arg, "extglob")) { #ifdef HAVE_FNM_EXTMATCH exit(0); #else exit(1); #endif } else if (!strcmp(arg, "seccomp")) { #ifdef HAVE_SECCOMP exit(0); #else exit(1); #endif } else if (!strcmp(arg, "squash")) { #ifdef HAVE_LIBSQUASHFUSE exit(0); #else exit(1); #endif } else FATAL("unknown feature: %s", arg); break; case -12: // --home Tf (args->c.host_home = getenv("HOME"), "--home failed: $HOME not set"); if (args->c.overlay_size == NULL) { VERBOSE("--home specified; also setting --write-fake"); args->c.overlay_size = WRITE_FAKE_DEFAULT; } break; case -13: // --unsafe args->unsafe = true; break; #ifdef HAVE_SECCOMP case -14: // --seccomp args->seccomp_p = true; break; #endif case -15: // --set-env0 parse_set_env(args, arg, '\0'); break; case -16: // --warnings for (int i = 1; i <= parse_int(arg, false, "--warnings"); i++) WARNING("this is warning %d!", i); exit(0); break; case -17: // --test if (!strcmp(arg, "log")) test_logging(false); else if (!strcmp(arg, "log-fail")) test_logging(true); else FATAL("invalid --test argument: %s; see source code", arg); break; case 'b': { // --bind char *src, *dst; for (i = 0; args->c.binds[i].src != NULL; i++) // count existing binds ; T_ (args->c.binds = realloc(args->c.binds, (i+2) * sizeof(struct bind))); args->c.binds[i+1].src = NULL; // terminating zero args->c.binds[i].dep = BD_MAKE_DST; // source src = strsep(&arg, ":"); T_ (src != NULL); Te (src[0] != 0, "--bind: no source provided"); args->c.binds[i].src = src; // destination dst = arg ? arg : src; Te (dst[0] != 0, "--bind: no destination provided"); Te (strcmp(dst, "/"), "--bind: destination can't be /"); Te (dst[0] == '/', "--bind: destination must be absolute"); args->c.binds[i].dst = dst; } break; case 'c': // --cd args->initial_dir = arg; break; case 'g': // --gid i = parse_int(arg, false, "--gid"); Te (i >= 0, "--gid: must be non-negative"); args->c.container_gid = (gid_t) i; break; case 'j': // --join args->c.join = true; break; case 'm': // --mount Ze ((arg[0] == '\0'), "mount point can't be empty string"); args->c.newroot = arg; break; case 's': // --storage args->storage_dir = arg; if (!path_exists(arg, NULL, false)) WARNING("storage directory not found: %s", arg); break; case 'q': // --quiet Te(verbose <= 0, "--quiet incompatible with --verbose"); verbose--; Te(verbose >= -3, "--quiet can be specified at most thrice"); break; case 't': // --private-tmp args->c.private_tmp = true; break; case 'u': // --uid i = parse_int(arg, false, "--uid"); Te (i >= 0, "--uid: must be non-negative"); args->c.container_uid = (uid_t) i; break; case 'V': // --version version(); exit(EXIT_SUCCESS); break; case 'v': // --verbose Te(verbose >= 0, "--verbose incompatible with --quiet"); verbose++; Te(verbose <= 3, "--verbose can be specified at most thrice"); break; case 'w': // --write args->c.writable = true; break; case 'W': // --write-fake args->c.overlay_size = arg != NULL ? arg : WRITE_FAKE_DEFAULT; break; case ARGP_KEY_NO_ARGS: argp_state_help(state, stderr, ( ARGP_HELP_SHORT_USAGE | ARGP_HELP_PRE_DOC | ARGP_HELP_LONG | ARGP_HELP_POST_DOC)); exit(EXIT_FAILURE); default: return ARGP_ERR_UNKNOWN; }; return 0; } void parse_set_env(struct args *args, char *arg, int delim) { struct env_delta ed; if (arg == NULL) { ed.action = ENV_SET_DEFAULT; ed.arg.delim = delim; } else { ed.action = ENV_SET_VARS; if (strchr(arg, '=') == NULL) ed.arg.vars = env_file_read(arg, delim); else { ed.arg.vars = list_new(sizeof(struct env_var), 1); ed.arg.vars[0] = env_var_parse(arg, NULL, 0); } } list_append((void **)&(args->env_deltas), &ed, sizeof(ed)); } /* Validate that the UIDs and GIDs are appropriate for program start, and abort if not. Note: If the binary is setuid, then the real UID will be the invoking user and the effective and saved UIDs will be the owner of the binary. Otherwise, all three IDs are that of the invoking user. */ void privs_verify_invoking() { uid_t ruid, euid, suid; gid_t rgid, egid, sgid; Z_ (getresuid(&ruid, &euid, &suid)); Z_ (getresgid(&rgid, &egid, &sgid)); // Calling the program if user is really root is OK. if ( ruid == 0 && euid == 0 && suid == 0 && rgid == 0 && egid == 0 && sgid == 0) return; // Now that we know user isn't root, no GID privilege is allowed. T_ (egid != 0); // no privilege T_ (egid == rgid && egid == sgid); // no setuid or funny business // No UID privilege allowed either. T_ (euid != 0); // no privilege T_ (euid == ruid && euid == suid); // no setuid or funny business } /* Return path to the storage directory, if -s is not specified. */ char *storage_default(void) { char *storage = getenv("CH_IMAGE_STORAGE"); if (storage == NULL) T_ (1 <= asprintf(&storage, "/var/tmp/%s.ch", username)); return storage; } charliecloud-0.37/bin/ch-test000077500000000000000000001034561457016721300161650ustar00rootroot00000000000000#!/bin/bash ### Setup we need right away set -e # Set the locale so we have predictable sorting. However, ch-image crashes on # Python 3.6 in locale “C” (see issues #970 and #1262), which is the only # locale guaranteed to be available, so use a UTF-8 locale if available. This # means tests will fail on Python 3.6 systems without this UTF-8 locale. if [[ $(locale -a) = *en_US.utf8* ]]; then export LC_ALL=en_US.utf8 else export LC_ALL=C fi # Set environment variable for ch-image --debug flag export CH_IMAGE_DEBUG=true # Set environment variable for ch-convert and ch-image --xattrs flag export CH_XATTRS=true ### Functions we need right away fatal () { printf 'error: %s\n' "$1" 1>&2 exit 1 } warning () { printf 'warning: %s\n' "$1" 1>&2 } ### Setup ch_lib=$(cd "$(dirname "$0")" && pwd)/../lib if [[ ! -f ${ch_lib}/base.sh ]]; then fatal "install or build problem: not found: ${ch_lib}/base.sh" fi . "${ch_lib}/base.sh" . "${ch_lib}/contributors.bash" export ch_bin export ch_lib if docker info > /dev/null 2>&1; then ch_docker_nosudo=yes else ch_docker_nosudo= fi export ch_docker_nosudo usage=$(cat < /dev/null 2>&1; then fatal 'builder: buildah: not installed.' fi bl=$(command -v buildah) bv=$(buildah --version | awk '{print $3}') min='1.11.2' ;; docker) if ! command -v docker > /dev/null 2>&1; then fatal 'builder: docker: not installed' fi bl=$(command -v docker) bv=$(docker_ --version | awk '{print $3}' | sed -e 's/,$//') ;; none) bl='none' bv= ;; *) fatal "builder: $CH_TEST_BUILDER: not supported" ;; esac printf 'found: %s %s\n\n' "$bl" "$bv" version_check 'builder' "$min" "$bv" } builder_set () { width=$1 if [[ -n $builder ]]; then CH_TEST_BUILDER=$builder method='command line' elif [[ -n $CH_TEST_BUILDER ]]; then method='environment' else CH_TEST_BUILDER=ch-image method='default' fi printf "%-*s %s (%s)\n" "$width" 'builder:' "$CH_TEST_BUILDER" "$method" if [[ $CH_TEST_BUILDER == ch-image ]]; then vset CH_IMAGE_STORAGE '' "$CH_IMAGE_STORAGE" "/var/tmp/${USER}.ch" \ "$width" 'ch-image storage' fi export CH_TEST_BUILDER } # Create CH_TEST_IMGDIR, avoiding #347. dir_img_mk () { dir_img_rm printf "creating %s\n\n" "$CH_TEST_IMGDIR" $ch_mpirun_node mkdir "$CH_TEST_IMGDIR" $ch_mpirun_node touch "$CH_TEST_IMGDIR/WEIRD_AL_YANKOVIC" } dir_img_rm () { dir_rm_safe "$CH_TEST_IMGDIR" } # Remove a filesystem permissions fixture directory. Ensure that the target # directory has exactly the two subdirectories expected first. dir_perm_rm () { if [[ $(find "${1}" -maxdepth 1 -mindepth 1 | wc -l) == 2 \ && -d "${1}/pass" && -d "${1}/nopass" ]]; then echo "removing ${1}" sudo rm -rf --one-file-system "$1" fi } # Remove directory $1 if it’s either 1) empty or 2) contains a sentinel file. dir_rm_safe () { if [[ $(find "$1" -maxdepth 1 -mindepth 1 | wc -l) == 0 \ || -e ${1}/WEIRD_AL_YANKOVIC ]]; then echo "removing $1" $ch_mpirun_node rm -rf --one-file-system "$1" else fatal "non-empty and missing sentinel file; manually delete: $1" fi } # The run phase requires artifacts from a successful build phase. Thus, we # check sanity based on the minimal set of artifacts (no builder). dir_tar_check () { printf 'checking %s: ' "$CH_TEST_TARDIR" dir_tar_check_file chtest{.tar.gz,.sqfs} printf 'ok\n\n' } dir_tar_check_file () { local missing for f in "$@"; do if [[ -f ${CH_TEST_TARDIR}/${f} ]]; then return 0 else missing+=("${CH_TEST_TARDIR}/${f}") fi done fatal "phase $phase: missing packed images: ${missing[*]}" } dir_tar_mk () { dir_tar_rm printf "creating %s\n\n" "$CH_TEST_TARDIR" $ch_mpirun_node mkdir "$CH_TEST_TARDIR" $ch_mpirun_node touch "${CH_TEST_TARDIR}/WEIRD_AL_YANKOVIC" } dir_tar_rm () { dir_rm_safe "$CH_TEST_TARDIR" } dir_tmp_rm () { if [[ $TMP_ == "/tmp/ch-test.tmp.${USER}" ]]; then echo "removing $TMP_" rm -rf --one-file-system "$TMP_" fi } dirs_unpriv_rm () { dir_tar_rm dir_img_rm dir_tmp_rm } pack_fmt_set () { width=$1 if command -v mksquashfs > /dev/null 2>&1; then have_mksquashfs=yes else have_mksquashfs= fi if [[ $(ldd "${ch_bin}/ch-run") = *"libsquashfuse"* ]]; then have_libsquashfuse=yes else have_libsquashfuse= fi if [[ -n $pack_fmt ]]; then CH_TEST_PACK_FMT=$pack_fmt method='command line' elif [[ -n $CH_TEST_PACK_FMT ]]; then method='environment' elif [[ -n $have_mksquashfs && -n $have_libsquashfuse ]]; then CH_TEST_PACK_FMT=squash-mount method='default' else CH_TEST_PACK_FMT=tar-unpack method='default' fi case $CH_TEST_PACK_FMT in '🐘') # elephant emoji U+1F418 CH_TEST_PACK_FMT=squash-mount ;; '📠') # fax machine emoji U+1F4E0 CH_TEST_PACK_FMT=tar-unpack ;; '🎃') # jack-o-lantern emoji U+1F383 CH_TEST_PACK_FMT=squash-unpack ;; esac export CH_TEST_PACK_FMT printf "%-*s %s (%s)\n" \ "$width" 'packed image format:' "$CH_TEST_PACK_FMT" "$method" case $CH_TEST_PACK_FMT in squash-mount) if [[ -z $have_mksquashfs ]]; then fatal "format invalid: ${CH_TEST_PACK_FMT}: no mksquashfs" fi if [[ -z $have_libsquashfuse ]]; then fatal "format invalid: ${CH_TEST_PACK_FMT}: ch-run not linked with libsquashfuse" fi ;; tar-unpack) ;; # nothing to check (assume we have tar) squash-unpack) if [[ -z $have_mksquashfs ]]; then fatal "format invalid: ${CH_TEST_PACK_FMT}: no mksquashfs" fi ;; *) fatal "format unknown: ${CH_TEST_PACK_FMT}" ;; esac } pedantry_set () { width=$1 default=no # Default to pedantic on CI or if user is a contributor. if [[ -n $ch_contributor || -n $CI ]]; then default=yes fi vset ch_pedantic "$pedantic" '' $default "$width" 'pedantic mode' if [[ $ch_pedantic == no ]]; then ch_pedantic= # proper boolean fi # The motivation here is that in pedantic mode, we want to run all the # tests we reasonably can. So, if the user *has* sudo, then default --sudo # to yes. What is a little awkward is that “sudo -v” can generate a # password prompt in the middle of the status output. An alternative is # “sudo -nv”, which doesn’t; drawbacks are that you have to analyze the # output (not exit code) and it generates a failed password log message if # there is not already a sudo session going. if [[ -n $ch_pedantic ]] \ && command -v sudo > /dev/null \ && sudo -v > /dev/null 2>&1; then use_sudo_default=yes else use_sudo_default= fi } pq_missing () { if [[ $phase == all || $phase == build ]]; then local img=$1 local out=$2 local tag tag=$(test_make_auto tag "$img") printf '%s\n' "$out" >> "${CH_TEST_TARDIR}/${tag}.pq_missing" fi } require_unset () { name=$1 value=${!1} if [[ -n $value ]]; then fatal "$name: no multiple assignment (already \"$value\")" fi } scope_check () { case $1 in quick|standard|full) return 0 ;; *) fatal "invalid scope: $1" ;; esac } # Assign scope a sortable opaque integer value. This value is used to help # filter images and tests that are out of scope. scope_to_digit () { case $1 in quick) echo 1 ;; standard) echo 2 ;; full) echo 3 ;; skip*) # skip has the highest value to ensure it is always filtered out echo 4 ;; *) fatal "scope '$scope' invalid" ;; esac } test_build () { echo 'executing build phase tests ...' bats build/*.bats } test_build_images () { echo 'building images ...' if [[ ! -f ${TMP_}/build_auto.bats ]]; then fatal "${TMP_}/build_auto.bats not found" fi bats "${TMP_}/build_auto.bats" } test_examples () { printf '\n' if [[ $CH_TEST_SCOPE == quick ]]; then echo "no examples for $CH_TEST_SCOPE scope" fi echo 'executing example phase tests ...' if find "$TMP_" -name '*_example.bats' | grep -q .; then bats "${TMP_}"/*_example.bats fi } test_make () { local bat_file local img_pack local pack_files local tag printf "finding tests compatible with %s phase settings ...\n" "$phase" case $phase in build|build-images) for i in $images $examples; do if test_make_check_image "$i"; then echo "found: $i" build_targets+=( "$i" ) fi done printf '\n' printf 'generate build_auto.bats ...\n' test_make_auto "$phase" "${build_targets[@]}" > "${TMP_}/build_auto.bats" printf 'ok\n\n' ;; run) # For each tarball or squashfs file in --pack-dir look for a # corresponding example or image that produces a matching tag. If # found, check the image for exclusion conidtions. pack_files=$(find "$CH_TEST_TARDIR" -name '*.tar.gz' \ -o -name '*.sqfs' | sort) for i in $pack_files; do img_pack=${i##*/} img_pack=${img_pack%%.*} for j in $images $examples; do if [[ $(test_make_tag_from_path "$j") == "$img_pack" ]]; then if test_make_check_image "$j"; then echo "found: $i" run_targets+=( "$j" ) fi fi done done printf '\n' printf 'generate run_auto.bats ...\n' test_make_auto "$phase" "${run_targets[@]}" > "${TMP_}/run_auto.bats" printf 'ok\n\n' ;; examples) if [[ $CH_TEST_SCOPE == quick ]]; then echo 'skipping examples phase in quick scope' return fi for i in $examples; do if test_make_check_image "$i"; then bat_file=$(dirname "$i")/test.bats tag=$(test_make_tag_from_path "$i") cp "$bat_file" "${TMP_}/${tag}_example.bats" # Substitute $ch_test_tag here with sed because we run all the # examples together later, but the value needs to vary between # the files. Watch the escaping here. sed -i "s/\\\$ch_test_tag/${tag}/g" \ "${TMP_}/${tag}_example.bats" echo "found: $(dirname "$i")" fi done printf '\n' printf 'generate example bat files ...\n' printf 'ok\n\n' ;; *) ;; esac } test_make_auto () { local mode mode=$1;shift if [[ $mode != tag ]]; then printf "# Do not edit this file; it's autogenerated\n\n" printf "load %s/common.bash\n\n" "$CHTEST_DIR" fi while [[ "$#" -gt 0 ]]; do path_=$1;shift basename_=$(basename "$path_") dirname_=$(dirname "$path_") tag=$(test_make_tag_from_path "$path_") if [[ $dir == "" ]];then dir='.' fi if [[ $mode == tag ]]; then echo "$tag" exit 0 fi if [[ $mode == build* ]]; then case $basename_ in Build|Build.*) test_make_template_print 'build_custom.bats.in' ;; Dockerfile|Dockerfile.*) test_make_template_print 'build.bats.in' test_make_template_print 'builder_to_archive.bats.in' ;; *) fatal "test_make_auto: unknown build type" ;; esac elif [[ $mode == run ]];then test_make_template_print 'unpack.bats.in' else fatal "test_make_auto: invalid mode '$mode'" fi done } test_make_check_image () { img_ok=yes img=$1 dir=$(basename "$(dirname "$img")") arch_exclude=$(cat "$img" | grep -F 'ch-test-arch-exclude: ' \ | sed 's/.*: //' | awk '{print $1}') builder_include=$(cat "$img" | grep -F 'ch-test-builder-include: ' \ | sed 's/.*: //' | awk '{print $1}') builder_exclude=$(cat "$img" | grep -F 'ch-test-builder-exclude: ' \ | sed 's/.*: //' | awk '{print $1}') img_scope_str=$(cat "$img" | grep -F 'ch-test-scope' \ | sed 's/.*: //' \ | awk '{print $1}') [[ -n $img_scope_str ]] || fatal "no scope: $img" img_scope_int=$(scope_to_digit "$img_scope_str") [[ -n $img_scope_int ]] || exit 1 # set -e not working, why? sudo_required=$(cat "$img" | grep -F 'ch-test-need-sudo') if [[ $phase == 'build' ]]; then # Exclude Dockerfiles if we have no builder. if [[ $CH_TEST_BUILDER == none && $img == *Dockerfile* ]]; then pq_missing "$img" 'builder required' img_ok= fi # Exclude if included builders are given and $CH_TEST_BUILDER isn’t one. if [[ -n $builder_include ]]; then builder_ok= for b in $builder_include; do if [[ $b == "$CH_TEST_BUILDER" ]]; then builder_ok=yes fi done if [[ -z $builder_ok ]]; then pq_missing "$img" "builder not included: ${CH_TEST_BUILDER}" img_ok= fi fi # Exclude images that are not compatible with CH_TEST_BUILDER. for b in $builder_exclude; do if [[ $b == "$CH_TEST_BUILDER" ]]; then pq_missing "$img" "builder excluded: ${CH_TEST_BUILDER}" img_ok= fi done fi # Exclude images with a scope that is not a subset of CH_TEST_SCOPE. if [[ $scope_int -lt "$img_scope_int" ]]; then pq_missing "$img" "not in scope: ${CH_TEST_SCOPE}" img_ok= fi # Exclude images that do not work with the host architecture. for a in $arch_exclude; do if [[ $a == "$(uname -m)" ]]; then pq_missing "$img" "incompatible architecture: ${a}" img_ok= fi done # Exclude images that require sudo if CH_TEST_SUDO is empty if [[ -n $sudo_required && -z $CH_TEST_SUDO ]]; then pq_missing "$img" 'generic sudo required' img_ok= fi # In examples phase, exclude chtest and any images not in a subdirectory of # examples. if [[ $phase == examples && ( $dir == chtest || $dir == examples ) ]]; then img_ok= fi if [[ -n $img_ok ]]; then return 0 # include image else return 1 # exclude image fi } test_make_tag_from_path () { # Generate a tag from given path. # # Consider the following path: $CHTEST/examples/Dockerfile.openmpi # First break the path into four components: # 1) dir: the parent directory of the file (examples) # 2) base: the full file name (Dockerfile.openmpi) # 3) basicname: the file name (Dockerfile) # 4) extension: the file's extension (openmpi) # # $basicname must be “Build” or “Dockerfile”; otherwise error. # if $dir is “.”, “test”, or “examples” then tag=$extension; otherwise # tag is $extension if set or $dir-$exenstion. local base local basicname local dir local extension local tag dir=$(basename "$(dirname "$1")") # last directory only base=$(basename "$1") basicname=${base%%.*} extension=${base##*.} if [[ $extension == "$basicname" ]]; then extension='' else extension=${extension/\./} # remove dot fi case $basicname in Build|Dockerfile) case $dir in .|examples|charliecloud|test) # dot is directory “test” if [[ -z $extension ]]; then fatal "can't compute tag: $1" else tag=$extension fi ;; *) if [[ -z $extension ]]; then tag=$(basename "$dir") else tag=$(basename "${dir}-${extension}") fi esac ;; *) fatal "test_make_auto: invalid basic name '$basicname'" ;; esac echo "$tag" } test_make_template_print () { local template template="./make-auto.d/$1" # Subsitute template variables and remove “source” command that is only # for ShellCheck. cat "$template" \ | sed -E -e 's/^(source common\.bash.*)$/#\1/' \ -e "s@%\(basename\)s@$basename_@g" \ -e "s@%\(dirname\)s@$dirname_@g" \ -e "s@%\(path\)s@$path_@g" \ -e "s@%\(scope\)s@$CH_TEST_SCOPE@g" \ -e "s@%\(tag\)s@$tag@g" printf '\n' } test_one_file () { if [[ $one_file != *.bats ]]; then # Assume it's a Dockerfile or Build file; derive tag and test.bats. ch_test_tag=$(test_make_tag_from_path "$one_file") export ch_test_tag one_file=$(dirname "$one_file")/test.bats printf 'tag: %s\n' "$ch_test_tag" fi printf 'file: %s\n' "$one_file" bats "$one_file" } test_rootemu () { if command -v ch-image > /dev/null 2>&1; then bats force-auto.bats echo yes > "$TMP_/rootemu" elif [[ -n "$1" ]]; then printf "error: ch-image required, not found\n" exit 1 fi } test_run () { echo 'executing run phase tests ...' if [[ ! -f ${TMP_}/run_auto.bats ]]; then fatal "${TMP_}/run_auto.bats not found" fi bats run_first.bats "${TMP_}/run_auto.bats" ./run/*.bats if [[ $CH_TEST_SCOPE != quick ]]; then for guest_user in $(id -un) root nobody; do for guest_group in $(id -gn) root $(id -gn nobody); do export GUEST_USER=$guest_user export GUEST_GROUP=$guest_group echo "testing as $GUEST_USER $GUEST_GROUP" bats run/ch-run_uidgid.bats done done fi } # Exit with failure if given version number is below a minimum. # # $1: human-readable descriptor # $2: minimum version # $3: actual version version_check () { desc=$1 min=$2 actual=$3 if [[ $( printf '%s\n%s\n' "$min" "$actual" \ | sort -V | head -n1) != "$min" ]]; then fatal "$desc: mininum version $min, found $actual" fi } win () { printf "\nAll tests passed.\n" } ### Body of script # Ensure ch-run has been compiled (issue #329). if ! "${ch_bin}/ch-run" --version > /dev/null 2>&1; then fatal "no working ch-run found in $ch_bin" fi # Some tests have specific libc requirements. case $(readelf -p .interp "${ch_bin}/ch-run") in *musl*) # e.g. /lib/ld-musl-x86_64.so.1 export ch_libc=musl ;; *) # e.g. /lib64/ld-linux-x86-64.so.2 export ch_libc=glibc ;; esac # Ensure we have Bash 4.1 or higher if /bin/bash -c 'set -e; [[ 1 = 0 ]]; exit 0'; then # Bash bug: [[ ... ]] expression doesn't exit with set -e # https://github.com/sstephenson/bats/issues/49 fatal 'Bash minimum version is 4.1' fi # Is the user a contributor? email= # First, ask Git for the configured e-mail address. if command -v git > /dev/null 2>&1; then email="$(git config --get user.email || true)" fi # If that doesn’t work, construct it from the environment. if [[ -z $email ]]; then email="$USER@$(hostname --domain)" fi ch_contributor= for i in "${ch_contributors[@]}"; do if [[ $i == "$email" ]]; then ch_contributor=yes fi done # Ensure Bats is installed. if command -v bats > /dev/null 2>&1; then bats=$(command -v bats) bats_version="$(bats --version | awk '{print $2}')" else fatal 'Bats not found' fi # Reject non-default registry on GitHub Actions. if [[ -n $GITHUB_ACTIONS && -n $CH_REGY_DEFAULT_HOST ]]; then fatal 'non-default registry on GitHub Actions invalid' fi # Create a directory to hold auto-generated test artifacts. TMP_=/tmp/ch-test.tmp.$USER if [[ ! -d $TMP_ ]]; then mkdir "$TMP_" chmod 700 "$TMP_" fi # Record that we haven’t (yet) run the “rootemu” tests echo no > "$TMP_/rootemu" # Find test directories. Note some of this gets rewritten at install time. CHTEST_DIR=${ch_base}/test CHTEST_EXAMPLES_DIR=${ch_base}/examples if [[ ! -f ${ch_base}/VERSION ]]; then # installed CHTEST_INSTALLED=yes CHTEST_GITWD= else # build dir CHTEST_INSTALLED= if [[ -e ${ch_base}/.git ]]; then CHTEST_GITWD=yes else CHTEST_GITWD= fi fi export ch_base export CHTEST_INSTALLED export CHTEST_GITWD export CHTEST_DIR export CHTEST_EXAMPLES_DIR export TMP_ # Check for test directory. if [[ ! -d $CHTEST_DIR ]]; then fatal "test directory not found: $CHTEST_DIR" fi if [[ ! -d $CHTEST_EXAMPLES_DIR ]]; then fatal "examples not found: $CHTEST_EXAMPLES_DIR" fi # Parse arguments. if [[ $# == 0 ]]; then usage 1 fi while [[ $# -gt 0 ]]; do opt=$1; shift case $opt in all|build|build-images|clean|examples|mk-perm-dirs|rm-perm-dirs|rootemu|run) require_unset phase phase=$opt ;; -b|--builder) require_unset builder builder=$1; shift ;; --builder=*) require_unset builder builder=${opt#*=} ;; -c|--bucache-mode) require_unset bucache_mode bucache_mode=$1; shift ;; --bucache-mode=*) require_unset bucache_mode bucache_mode=${opt#*=} ;; --dry-run) dry=true ;; -f|--file) require_unset phase phase=one-file one_file=$1; shift ;; --file=*) require_unset phase phase=one-file one_file=${opt#*=} ;; -h|--help) usage 0 ;; --img-dir) require_unset imgdir imgdir=$1; shift ;; --img-dir=*) require_unset imgdir imgdir=${opt#*=} ;; --is-pedantic) # undocumented; for CI is_pedantic=yes ;; --is-sudo) # undocumented; for CI is_sudo=yes ;; --fi-path) require_unset fi_path fi_path=$1; shift ;; --fi-path=*) require_unset fi_path fi_path=${opt#*=} ;; --pack-dir) require_unset tardir tardir=$1; shift ;; --pack-dir=*) require_unset tardir tardir=${opt#*=} ;; --pack-fmt) require_unset pack_fmt pack_fmt=$1; shift ;; --pack-fmt=*) require_unset pack_fmt pack_fmt=${opt#*=} ;; --pedantic) pedantic=$1; shift ;; --pedantic=*) pedantic=${opt#*=} ;; --perm-dir) use_sudo=yes permdirs+=("$1"); shift ;; --perm-dir=*) use_sudo=yes permdirs+=("${opt#*=}") ;; -s|--scope) require_unset scope scope_check "$1" scope=$1; shift ;; --scope=*) require_unset scope scope=${opt#*=} scope_check "$scope" ;; --sudo) use_sudo=yes ;; --srun-mpi=*) require_unset srun_mpi srun_mpi="${opt#*=}" ;; --srun-mpi) require_unset srun_mpi srun_mpi=$1; shift ;; --lustre) require_unset lustredir lustredir=$1; shift ;; --lustre=*) require_unset lustredir lustredir=${opt#*=} ;; --version) version ;; *) fatal "unrecognized argument: $opt" ;; esac done printf 'ch-run: %s (%s)\n' "${ch_bin}/ch-run" "$ch_libc" printf 'bats: %s (%s)\n' "$bats" "$bats_version" if [[ $(printf "%s\n%s" "$bats_version" "$bats_vmin" | sort -V | head -1) \ != "$bats_vmin" ]]; then warning "Bats version unsupported b/c < $bats_vmin" fi printf 'tests: %s\n' "$CHTEST_DIR" printf 'installed: %s\n' "${CHTEST_INSTALLED:-no}" printf 'locale: %s\n' "$LC_ALL" printf 'git workdir: %s\n' "${CHTEST_GITWD:-no}" if [[ -n $ch_contributor ]]; then ch_contributor_note="yes; $email in README.rst" else ch_contributor_note="no; $email not in README.rst" fi printf 'contributor: %s\n\n' "$ch_contributor_note" if [[ $phase = one-file ]]; then if [[ $one_file == *:* ]]; then x=$one_file one_file=${x%%:*} # before first colon export ch_one_test=${x#*:} # after first colon fi if [[ ! -f $one_file ]]; then fatal "not a file: $one_file" fi one_file=$(readlink -f "$one_file") # make absolute b/c we cd later if [[ $one_file = */test.bats ]]; then fatal '--file: must specify build recipe file, not test.bats' fi fi printf "%-21s %s" 'phase:' "$phase" if [[ $phase = one-file ]]; then printf ': %s (%s)' "$one_file" "$ch_one_test" fi if [[ -z $phase ]]; then fatal 'phase: no phase specified' fi printf '\n' # See issue #1580 # shellcheck disable=SC2016 if [[ -d /var/tmp/img ]] || [[ -d /var/tmp/tar ]]; then printf '\n' warning 'NOTE: default image and pack directories changed to:' warning ' CH_TEST_IMGDIR=/var/tmp/${USER}.img' warning ' CH_TEST_TARDIR=/var/tmp/${USER}.pack' fi printf '\n' # variable name CLI environment default # desc. width description vset CH_TEST_SCOPE "$scope" "$CH_TEST_SCOPE" standard \ 21 'scope' builder_set 21 pedantry_set 21 vset CH_TEST_SUDO "$use_sudo" "$CH_TEST_SUDO" "$use_sudo_default" \ 21 'use generic sudo' vset CH_TEST_IMGDIR "$imgdir" "$CH_TEST_IMGDIR" "/var/tmp/${USER}.img" \ 21 'unpacked images dir' vset CH_TEST_TARDIR "$tardir" "$CH_TEST_TARDIR" "/var/tmp/${USER}.pack" \ 21 'packed images dir' pack_fmt_set 21 vset CH_IMAGE_CACHE "$bucache_mode" "$CH_IMAGE_CACHE" enabled \ 21 'build cache mode' vset CH_TEST_PERMDIRS "${permdirs[*]}" "$CH_TEST_PERMDIRS" skip \ 21 'fs permissions dirs' vset CH_TEST_LUSTREDIR "$lustredir" "$CH_TEST_LUSTREDIR" skip \ 21 'Lustre test dir' vset CH_TEST_SLURM_MPI "$srun_mpi" "$CH_TEST_SLURM_MPI" "$SLURM_MPI_TYPE" \ 21 'srun mpi type' vset CH_TEST_OFI_PATH "$fi_path" "$CH_TEST_OFI_PATH" skip \ 21 'ofi provider(s) path' printf '\n' if [[ $phase == *'perm'* ]] && [[ $CH_TEST_PERMDIRS == skip ]]; then fatal "phase $phase: CH_TEST_PERMDIRS: can't be 'skip'" fi # Check that different sources of version number are consistent. printf 'ch-test version: %s\n' "$ch_version" ch_run_version=$("${ch_bin}/ch-run" --version 2>&1) if [[ $ch_version != "$ch_run_version" ]]; then warning "inconsistent ch-run version: ${ch_run_version}" fi if [[ -z $CHTEST_INSTALLED ]]; then cf_version=$("${ch_base}/configure" --version | head -1 | cut -d' ' -f3) if [[ $ch_version != "$cf_version" ]]; then warning "inconsistent configure version: ${cf_version}" fi # Charliecloud version. Prefer git; otherwise use simple version. src_version=$( "${ch_base}/misc/version" 2> /dev/null \ || cat "${ch_base}/VERSION") if [[ $ch_version != "$src_version" ]]; then warning "inconsistent source version: ${src_version}" fi fi printf '\n' # Ensure ofi path looks sane. if [[ -n "$fi_path" ]]; then case "$fi_path" in *-fi.so) true ;; *libfabric.so) true ;; *) fatal '--fi-path: must end in -fi.so or libfabric.so' esac fi # Don't allow FI_ variables. if [[ -n "$FI_PROVIDER" ]]; then fatal 'host FI_PROVIDER set' fi if [[ -n "$FI_PROVIDER_PATH" ]]; then fatal 'host FI_PROVIDER_PATH set' fi # If srun --mpi=pmix, set variable to avoid spurious error. See # https://github.com/open-mpi/ompi/issues/7516. if [[ "$CH_TEST_SLURM_MPI" == 'pmix'* ]]; then export PMIX_MCA_gds=hash fi # Ensure BATS_TMPDIR is set to /tmp (issue #278). if [[ -n $BATS_TMPDIR && $BATS_TMPDIR != '/tmp' ]]; then fatal "BATS_TMPDIR: must be /tmp; found '$BATS_TMPDIR' (issue #278)" fi # Ensure namespaces are configured properly. printf 'checking namespaces ...\n' if ! "${ch_bin}/ch-checkns"; then fatal 'namespace sanity check (ch-checkns) failed' fi printf '\n' if [[ $CH_TEST_SUDO ]]; then printf 'checking sudo ...\n' sudo echo ok printf '\n' fi if [[ -n $is_pedantic ]]; then printf 'exiting per --is-pedantic\n' if [[ -n $ch_pedantic ]]; then exit 0; else exit 1; fi fi if [[ -n $is_sudo ]]; then printf 'exiting per --is-sudo\n' if [[ -n $CH_TEST_SUDO ]]; then exit 0; else exit 1; fi fi if [[ -n $dry ]];then printf 'exiting per --dry-run\n' exit 0 fi cd "$CHTEST_DIR" export PATH=$ch_bin:$PATH # Now that CH_TEST_* variables, PATH, and BATS_TMPDIR has been set and checked, # we source CHTEST_DIR/common.bash. . "${CHTEST_DIR}/common.bash" # The distinction here is that “images” are purely for testing and have no # value as examples for the user, while “examples” are dual-purpose. We call # “find” twice for each to preserve desired sort order. images=$( find "$CHTEST_DIR" -name 'Dockerfile.*' | sort \ && find "$CHTEST_DIR" -name 'Build' \ -o -name 'Build.*' | sort) examples=$( find "$CHTEST_EXAMPLES_DIR" -name 'Dockerfile' \ -o -name 'Dockerfile.*' | sort \ && find "$CHTEST_EXAMPLES_DIR" -name 'Build' \ -o -name 'Build.*' | sort) scope_int=$(scope_to_digit "$CH_TEST_SCOPE") # Execute phase case $phase in all) phase=build dir_tar_mk builder_check test_make test_build_images test_build phase=rootemu if [[ "$scope" != "full" ]]; then echo "skipping root emulation tests (full scope only)" else echo 'executing root emulation tests...' test_rootemu fi phase=run dir_img_mk dir_tar_check test_make test_run phase=examples dir_tar_check test_make test_examples # Kobe. win ;; build) builder_check dir_tar_mk test_make test_build_images test_build win ;; clean) dirs_unpriv_rm if [[ -d $TMP_ ]] && [[ -e $TMP_/build_auto.bats ]]; then echo "removing $TMP_" rm -rf --one-file-system "$TMP_" fi ;; examples) test_make test_examples win ;; build-images) builder_check dir_tar_mk test_make test_build_images win ;; mk-perm-dirs) printf 'creating filesystem permissions fixtures ...\n' for d in $CH_TEST_PERMDIRS; do if [[ -d ${d} ]]; then printf '%s already exists\n' "$d" continue else sudo "${CHTEST_DIR}/make-perms-test" "$d" "$USER" nobody fi done echo ;; one-file) test_one_file win ;; rm-perm-dirs) for d in $CH_TEST_PERMDIRS; do dir_perm_rm "$d" done ;; rootemu) test_rootemu optional # ch-image optional ;; run) dir_img_mk test_make test_run win ;; esac charliecloud-0.37/bin/ch_core.c000066400000000000000000000761261457016721300164410ustar00rootroot00000000000000/* Copyright © Triad National Security, LLC, and others. */ #define _GNU_SOURCE #include "config.h" #include #include #include #ifdef HAVE_SECCOMP #include #include #include #endif #include #include #include #include #ifdef HAVE_SECCOMP #include #include #endif #include #include #include #include #include #include #include #include #include #include "ch_misc.h" #include "ch_core.h" #ifdef HAVE_LIBSQUASHFUSE #include "ch_fuse.h" #endif /** Macros **/ /* Timeout in seconds for waiting for join semaphore. */ #define JOIN_TIMEOUT 30 /* Maximum length of paths we're willing to deal with. (Note that system-defined PATH_MAX isn't reliable.) */ #define PATH_CHARS 4096 /** Constants **/ /* Default bind-mounts. */ struct bind BINDS_DEFAULT[] = { { "/dev", "/dev", BD_REQUIRED }, { "/proc", "/proc", BD_REQUIRED }, { "/sys", "/sys", BD_REQUIRED }, { "/etc/hosts", "/etc/hosts", BD_OPTIONAL }, { "/etc/machine-id", "/etc/machine-id", BD_OPTIONAL }, { "/etc/resolv.conf", "/etc/resolv.conf", BD_OPTIONAL }, /* Cray bind-mounts. See #1473. */ { "/var/lib/hugetlbfs", "/var/lib/hugetlbfs", BD_OPTIONAL }, /* Cray Gemini/Aries interconnect bind-mounts. */ { "/etc/opt/cray/wlm_detect", "/etc/opt/cray/wlm_detect", BD_OPTIONAL }, { "/opt/cray/wlm_detect", "/opt/cray/wlm_detect", BD_OPTIONAL }, { "/opt/cray/alps", "/opt/cray/alps", BD_OPTIONAL }, { "/opt/cray/udreg", "/opt/cray/udreg", BD_OPTIONAL }, { "/opt/cray/ugni", "/opt/cray/ugni", BD_OPTIONAL }, { "/opt/cray/xpmem", "/opt/cray/xpmem", BD_OPTIONAL }, { "/var/opt/cray/alps", "/var/opt/cray/alps", BD_OPTIONAL }, /* Cray Shasta/Slingshot bind-mounts. */ { "/var/spool/slurmd", "/var/spool/slurmd", BD_OPTIONAL }, { 0 } }; /* Special values for seccomp tables. These must be negative to avoid clashing with real syscall numbers (note zero is often a valid syscal number). */ #define NR_NON -1 // syscall does not exist on architecture #define NR_END -2 // end of table /* Architectures that we support for seccomp. Order matches the corresponding table below. Note: On some distros (e.g., CentOS 7), some of the architecture numbers are missing. The workaround is to use the numbers I have on Debian Bullseye. The reason I (Reid) feel moderately comfortable doing this is how militant Linux is about not changing the userspace API. */ #ifdef HAVE_SECCOMP #ifndef AUDIT_ARCH_AARCH64 #define AUDIT_ARCH_AARCH64 0xC00000B7u // undeclared on CentOS 7 #undef AUDIT_ARCH_ARM // uses undeclared EM_ARM on CentOS 7 #define AUDIT_ARCH_ARM 0x40000028u #endif int SECCOMP_ARCHS[] = { AUDIT_ARCH_AARCH64, // arm64 AUDIT_ARCH_ARM, // arm32 AUDIT_ARCH_I386, // x86 (32-bit) AUDIT_ARCH_PPC64LE, // PPC AUDIT_ARCH_S390X, // s390x AUDIT_ARCH_X86_64, // x86-64 NR_END }; #endif /* System call numbers that we fake with seccomp (by doing nothing and returning success). Some processors can execute multiple architectures (e.g., 64-bit Intel CPUs can run both x64-64 and x86 code), and a process’ architecture can even change (if you execve(2) binary of different architecture), so we can’t just use the build host’s architecture. I haven’t figured out how to gather these system call numbers automatically, so they are compiled from [1, 2, 3]. See also [4] for a more general reference. NOTE: The total number of faked syscalls (i.e., non-zero entries below) must be somewhat less than 256. I haven’t computed the exact limit. There will be an assertion failure at runtime if this is exceeded. WARNING: Keep this list consistent with the ch-image(1) man page! [1]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/constants/syscalls.md#Cross_arch-Numbers [2]: https://github.com/strace/strace/blob/v4.26/linux/powerpc64/syscallent.h [3]: https://github.com/strace/strace/blob/v6.6/src/linux/s390x/syscallent.h [4]: https://unix.stackexchange.com/questions/421750 */ #ifdef HAVE_SECCOMP int FAKE_SYSCALL_NRS[][6] = { // arm64 arm32 x86 PPC64 s390x x86-64 // ------ ------ ------ ------ ------ ------ { 91, 185, 185, 184, 185, 126 }, // capset { NR_NON, 182, 182, 181, 212, 92 }, // chown { NR_NON, 212, 212, NR_NON, NR_NON, NR_NON }, // chown32 { 55, 95, 95, 95, 207, 93 }, // fchown { NR_NON, 207, 207, NR_NON, NR_NON, NR_NON }, // fchown32 { 54, 325, 298, 289, 291, 260 }, // fchownat { NR_NON, 16, 16, 16, 198, 94 }, // lchown { NR_NON, 198, 198, NR_NON, NR_NON, NR_NON }, // lchown32 { 104, 347, 283, 268, 277, 246 }, // kexec_load { 152, 139, 139, 139, 216, 123 }, // setfsgid { NR_NON, 216, 216, NR_NON, NR_NON, NR_NON }, // setfsgid32 { 151, 138, 138, 138, 215, 122 }, // setfsuid { NR_NON, 215, 215, NR_NON, NR_NON, NR_NON }, // setfsuid32 { 144, 46, 46, 46, 214, 106 }, // setgid { NR_NON, 214, 214, NR_NON, NR_NON, NR_NON }, // setgid32 { 159, 81, 81, 81, 206, 116 }, // setgroups { NR_NON, 206, 206, NR_NON, NR_NON, NR_NON }, // setgroups32 { 143, 71, 71, 71, 204, 114 }, // setregid { NR_NON, 204, 204, NR_NON, NR_NON, NR_NON }, // setregid32 { 149, 170, 170, 169, 210, 119 }, // setresgid { NR_NON, 210, 210, NR_NON, NR_NON, NR_NON }, // setresgid32 { 147, 164, 164, 164, 208, 117 }, // setresuid { NR_NON, 208, 208, NR_NON, NR_NON, NR_NON }, // setresuid32 { 145, 70, 70, 70, 203, 113 }, // setreuid { NR_NON, 203, 203, NR_NON, NR_NON, NR_NON }, // setreuid32 { 146, 23, 23, 23, 213, 105 }, // setuid { NR_NON, 213, 213, NR_NON, NR_NON, NR_NON }, // setuid32 { NR_END }, // end }; int FAKE_MKNOD_NRS[] = { NR_NON, 14, 14, 14, 14, 133 }; int FAKE_MKNODAT_NRS[] = { 33, 324, 297, 288, 290, 259 }; #endif /** Global variables **/ /* Variables for coordinating --join. */ struct { bool winner_p; char *sem_name; sem_t *sem; char *shm_name; struct { pid_t winner_pid; // access anytime after initialization (write-once) int proc_left_ct; // access only while serial } *shared; } join; /* Bind mounts done so far; canonical host paths. If null, there are none. */ char **bind_mount_paths = NULL; /** Function prototypes (private) **/ void bind_mount(const char *src, const char *dst, enum bind_dep, const char *newroot, unsigned long flags, const char *scratch); void bind_mounts(const struct bind *binds, const char *newroot, unsigned long flags, const char * scratch); void enter_udss(struct container *c); #ifdef HAVE_SECCOMP void iw(struct sock_fprog *p, int i, uint16_t op, uint32_t k, uint8_t jt, uint8_t jf); #endif void join_begin(const char *join_tag); void join_namespace(pid_t pid, const char *ns); void join_namespaces(pid_t pid); void join_end(int join_ct); void sem_timedwait_relative(sem_t *sem, int timeout); void setup_namespaces(const struct container *c, uid_t uid_out, uid_t uid_in, gid_t gid_out, gid_t gid_in); void setup_passwd(const struct container *c); void tmpfs_mount(const char *dst, const char *newroot, const char *data); /** Functions **/ /* Bind-mount the given path into the container image. */ void bind_mount(const char *src, const char *dst, enum bind_dep dep, const char *newroot, unsigned long flags, const char *scratch) { char *dst_fullc, *newrootc; char *dst_full = cat(newroot, dst); Te (src[0] != 0 && dst[0] != 0 && newroot[0] != 0, "empty string"); Te (dst[0] == '/' && newroot[0] == '/', "relative path"); if (!path_exists(src, NULL, true)) { Te (dep == BD_OPTIONAL, "can't bind: source not found: %s", src); return; } if (!path_exists(dst_full, NULL, true)) switch (dep) { case BD_REQUIRED: FATAL("can't bind: destination not found: %s", dst_full); break; case BD_OPTIONAL: return; case BD_MAKE_DST: mkdirs(newroot, dst, bind_mount_paths, scratch); break; } newrootc = realpath_(newroot, false); dst_fullc = realpath_(dst_full, false); Tf (path_subdir_p(newrootc, dst_fullc), "can't bind: %s not subdirectory of %s", dst_fullc, newrootc); if (strcmp(newroot, "/")) // don't record if newroot is "/" list_append((void **)&bind_mount_paths, &dst_fullc, sizeof(char *)); Zf (mount(src, dst_full, NULL, MS_REC|MS_BIND|flags, NULL), "can't bind %s to %s", src, dst_full); } /* Bind-mount a null-terminated array of struct bind objects. */ void bind_mounts(const struct bind *binds, const char *newroot, unsigned long flags, const char * scratch) { for (int i = 0; binds[i].src != NULL; i++) bind_mount(binds[i].src, binds[i].dst, binds[i].dep, newroot, flags, scratch); } /* Set up new namespaces or join existing namespaces. */ void containerize(struct container *c) { if (c->join_pid) { join_namespaces(c->join_pid); return; } if (c->join) join_begin(c->join_tag); if (!c->join || join.winner_p) { // Set up two nested user+mount namespaces: the outer so we can run // fusermount3 non-setuid, and the inner so we get the desired UID // within the container. We do this even if the image is a directory, to // reduce the number of code paths. setup_namespaces(c, geteuid(), 0, getegid(), 0); #ifdef HAVE_LIBSQUASHFUSE if (c->type == IMG_SQUASH) sq_fork(c); #endif setup_namespaces(c, 0, c->container_uid, 0, c->container_gid); enter_udss(c); } else join_namespaces(join.shared->winner_pid); if (c->join) join_end(c->join_ct); } /* Enter the new root (UDSS). On entry, the namespaces are set up, and this does the mounting and filesystem setup. Note that pivot_root(2) requires a complex dance to work, i.e., to avoid multiple undocumented error conditions. This dance is explained in detail in bin/ch-checkns.c. */ void enter_udss(struct container *c) { char *nr_parent, *nr_base, *mkdir_scratch; LOG_IDS; mkdir_scratch = NULL; path_split(c->newroot, &nr_parent, &nr_base); // Claim new root for this namespace. Despite MS_REC in bind_mount(), we do // need both calls to avoid pivot_root(2) failing with EBUSY later. DEBUG("claiming new root for this namespace") bind_mount(c->newroot, c->newroot, BD_REQUIRED, "/", MS_PRIVATE, NULL); bind_mount(nr_parent, nr_parent, BD_REQUIRED, "/", MS_PRIVATE, NULL); // Re-mount new root read-only unless --write or already read-only. if (!c->writable && !(access(c->newroot, W_OK) == -1 && errno == EROFS)) { unsigned long flags = path_mount_flags(c->newroot) | MS_REMOUNT // Re-mount ... | MS_BIND // only this mount point ... | MS_RDONLY; // read-only. Z_ (mount(NULL, c->newroot, NULL, flags, NULL)); } // Overlay a tmpfs if --write-fake. See for useful details: // https://www.kernel.org/doc/html/v5.11/filesystems/tmpfs.html // https://www.kernel.org/doc/html/v5.11/filesystems/overlayfs.html if (c->overlay_size != NULL) { VERBOSE("overlaying tmpfs for --write-fake (%s)", c->overlay_size); char *options; T_ (1 <= asprintf(&options, "size=%s", c->overlay_size)); Zf (mount(NULL, "/mnt", "tmpfs", 0, options), // host should have /mnt "cannot mount tmpfs for overlay"); free(options); Z_ (mkdir("/mnt/upper", 0700)); Z_ (mkdir("/mnt/work", 0700)); Z_ (mkdir("/mnt/merged", 0700)); mkdir_scratch = "/mnt/mkdir_overmount"; Z_ (mkdir(mkdir_scratch, 0700)); T_ (1 <= asprintf(&options, "lowerdir=%s,upperdir=%s,workdir=%s," "index=on,userxattr,volatile", c->newroot, "/mnt/upper", "/mnt/work")); // update newroot c->newroot = "/mnt/merged"; free(nr_parent); free(nr_base); path_split(c->newroot, &nr_parent, &nr_base); Zf (mount(NULL, c->newroot, "overlay", 0, options), "can't overlay"); VERBOSE("newroot updated: %s", c->newroot); free(options); } DEBUG("starting bind-mounts"); // Bind-mount default files and directories. bind_mounts(BINDS_DEFAULT, c->newroot, MS_RDONLY, NULL); // /etc/passwd and /etc/group. if (!c->private_passwd) setup_passwd(c); // Container /tmp. if (c->private_tmp) { tmpfs_mount("/tmp", c->newroot, NULL); } else { bind_mount(host_tmp, "/tmp", BD_REQUIRED, c->newroot, 0, NULL); } // Bind-mount user’s home directory at /home/$USER if requested. if (c->host_home) { T_ (c->overlay_size != NULL); bind_mount(c->host_home, cat("/home/", username), BD_MAKE_DST, c->newroot, 0, mkdir_scratch); } // Bind-mount user-specified directories. bind_mounts(c->binds, c->newroot, 0, mkdir_scratch); // Overmount / to avoid EINVAL if it’s a rootfs. Z_ (chdir(nr_parent)); Z_ (mount(nr_parent, "/", NULL, MS_MOVE, NULL)); Z_ (chroot(".")); // Pivot into the new root. Use /dev because it’s available even in // extremely minimal images. c->newroot = cat("/", nr_base); Zf (chdir(c->newroot), "can't chdir into new root"); Zf (syscall(SYS_pivot_root, c->newroot, path_join(c->newroot, "dev")), "can't pivot_root(2)"); Zf (chroot("."), "can't chroot(2) into new root"); Zf (umount2("/dev", MNT_DETACH), "can't umount old root"); DEBUG("pivot_root(2) dance successful") } /* Return image type of path, or exit with error if not a valid type. */ enum img_type image_type(const char *ref, const char *storage_dir) { struct stat st; FILE *fp; char magic[4]; // four bytes, not a string // If there’s a directory in storage where we would expect there to be if // ref were an image name, assume it really is an image name. if (path_exists(img_name2path(ref, storage_dir), NULL, false)) return IMG_NAME; // Now we know ref is a path of some kind, so find it. Zf (stat(ref, &st), "can't stat: %s", ref); // If ref is the path to a directory, then it’s a directory. if (S_ISDIR(st.st_mode)) return IMG_DIRECTORY; // Now we know it’s file-like enough to read. See if it has the SquashFS // magic number. fp = fopen(ref, "rb"); Tf (fp != NULL, "can't open: %s", ref); Tf (fread(magic, sizeof(char), 4, fp) == 4, "can't read: %s", ref); Zf (fclose(fp), "can't close: %s", ref); VERBOSE("image file magic expected: 6873 7173; actual: %x%x %x%x", magic[0], magic[1], magic[2], magic[3]); // If magic number matches, it’s a squash. Note: Magic number is 6873 7173, // i.e. “hsqs”. I think “sqsh” was intended but the superblock designers // were confused about endianness. // See: https://dr-emann.github.io/squashfs/ if (memcmp(magic, "hsqs", 4) == 0) return IMG_SQUASH; // Well now we’re stumped. FATAL("unknown image type: %s", ref); } char *img_name2path(const char *name, const char *storage_dir) { char *path; char *name_fs = strdup(name); replace_char(name_fs, '/', '%'); replace_char(name_fs, ':', '+'); T_ (1 <= asprintf(&path, "%s/img/%s", storage_dir, name_fs)); free(name_fs); // make Tim happy return path; } /* Helper function to write seccomp-bpf programs. */ #ifdef HAVE_SECCOMP void iw(struct sock_fprog *p, int i, uint16_t op, uint32_t k, uint8_t jt, uint8_t jf) { p->filter[i] = (struct sock_filter){ op, jt, jf, k }; DEBUG("%4d: { op=%2x k=%8x jt=%3d jf=%3d }", i, op, k, jt, jf); } #endif /* Begin coordinated section of namespace joining. */ void join_begin(const char *join_tag) { int fd; join.sem_name = cat("/ch-run_sem-", join_tag); join.shm_name = cat("/ch-run_shm-", join_tag); // Serialize. join.sem = sem_open(join.sem_name, O_CREAT, 0600, 1); T_ (join.sem != SEM_FAILED); sem_timedwait_relative(join.sem, JOIN_TIMEOUT); // Am I the winner? fd = shm_open(join.shm_name, O_CREAT|O_EXCL|O_RDWR, 0600); if (fd > 0) { VERBOSE("join: I won"); join.winner_p = true; Z_ (ftruncate(fd, sizeof(*join.shared))); } else if (errno == EEXIST) { VERBOSE("join: I lost"); join.winner_p = false; fd = shm_open(join.shm_name, O_RDWR, 0); T_ (fd > 0); } else { T_ (0); } join.shared = mmap(NULL, sizeof(*join.shared), PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); T_ (join.shared != NULL); Z_ (close(fd)); // Winner keeps lock; losers parallelize (winner will be done by now). if (!join.winner_p) Z_ (sem_post(join.sem)); } /* End coordinated section of namespace joining. */ void join_end(int join_ct) { if (join.winner_p) { // winner still serial VERBOSE("join: winner initializing shared data"); join.shared->winner_pid = getpid(); join.shared->proc_left_ct = join_ct; } else // losers serialize sem_timedwait_relative(join.sem, JOIN_TIMEOUT); join.shared->proc_left_ct--; VERBOSE("join: %d peers left excluding myself", join.shared->proc_left_ct); if (join.shared->proc_left_ct <= 0) { VERBOSE("join: cleaning up IPC resources"); Te (join.shared->proc_left_ct == 0, "expected 0 peers left but found %d", join.shared->proc_left_ct); Zf (sem_unlink(join.sem_name), "can't unlink sem: %s", join.sem_name); Zf (shm_unlink(join.shm_name), "can't unlink shm: %s", join.shm_name); } Z_ (sem_post(join.sem)); // parallelize (all) Z_ (munmap(join.shared, sizeof(*join.shared))); Z_ (sem_close(join.sem)); VERBOSE("join: done"); } /* Join a specific namespace. */ void join_namespace(pid_t pid, const char *ns) { char *path; int fd; T_ (1 <= asprintf(&path, "/proc/%d/ns/%s", pid, ns)); fd = open(path, O_RDONLY); if (fd == -1) { if (errno == ENOENT) { Te (0, "join: no PID %d: %s not found", pid, path); } else { Tf (0, "join: can't open %s", path); } } /* setns(2) seems to be involved in some kind of race with syslog(3). Rarely, when configured with --enable-syslog, the call fails with EINVAL. We never figured out a proper fix, so just retry a few times in a loop. See issue #1270. */ for (int i = 1; setns(fd, 0) != 0; i++) if (i >= 5) { Tf (0, "can’t join %s namespace of pid %d", ns, pid); } else { WARNING("can’t join %s namespace; trying again", ns); sleep(1); } } /* Join the existing namespaces created by the join winner. */ void join_namespaces(pid_t pid) { VERBOSE("joining namespaces of pid %d", pid); join_namespace(pid, "user"); join_namespace(pid, "mnt"); } /* Replace the current process with user command and arguments. */ void run_user_command(char *argv[], const char *initial_dir) { LOG_IDS; if (initial_dir != NULL) Zf (chdir(initial_dir), "can't cd to %s", initial_dir); VERBOSE("executing: %s", argv_to_string(argv)); Zf (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0), "can't set no_new_privs"); if (verbose < LL_INFO) T_ (freopen("/dev/null", "w", stdout)); if (verbose < LL_STDERR) T_ (freopen("/dev/null", "w", stderr)); execvp(argv[0], argv); // only returns if error Tf (0, "can't execve(2): %s", argv[0]); } /* Set up the fake-syscall seccomp(2) filter. This computes and installs a long-ish but fairly simple BPF program to implement the filter. To understand this rather hairy language: 1. https://man7.org/training/download/secisol_seccomp_slides.pdf 2. https://www.kernel.org/doc/html/latest/userspace-api/seccomp_filter.html 3. https://elixir.bootlin.com/linux/latest/source/samples/seccomp */ #ifdef HAVE_SECCOMP void seccomp_install(void) { int arch_ct = sizeof(SECCOMP_ARCHS)/sizeof(SECCOMP_ARCHS[0]) - 1; int syscall_cts[arch_ct]; struct sock_fprog p = { 0 }; int ii, idx_allow, idx_fake, idx_mknod, idx_mknodat, idx_next_arch; // Lengths of certain instruction groups. These are all obtained manually // by counting below, violating DRY. We could automate these counts, but it // seemed like the cost of extra buffers and code to do that would exceed // that of maintaining the manual counts. int ct_jump_start = 4; // ld arch & syscall nr, arch test, end-of-arch jump int ct_mknod_jump = 2; // jump table handling for mknod(2) and mknodat(2) int ct_mknod = 2; // mknod(2) handling int ct_mknodat = 6; // mknodat(2) handling // Count how many syscalls we are going to fake in the standard way. We // need this to compute the right offsets for all the jumps. for (int ai = 0; SECCOMP_ARCHS[ai] != NR_END; ai++) { p.len += ct_jump_start + ct_mknod_jump; syscall_cts[ai] = 0; for (int si = 0; FAKE_SYSCALL_NRS[si][0] != NR_END; si++) { bool syscall_p = FAKE_SYSCALL_NRS[si][ai] != NR_NON; syscall_cts[ai] += syscall_p; p.len += syscall_p; // syscall jump table entry } DEBUG("seccomp: arch %x: found %d syscalls", SECCOMP_ARCHS[ai], syscall_cts[ai]); } // Initialize program buffer. p.len += ( 1 // return allow + 1 // return fake success + ct_mknod // mknod(2) handling + ct_mknodat); // mknodat(2) handling DEBUG("seccomp(2) program has %d instructions", p.len); T_ (p.filter = calloc(p.len, sizeof(struct sock_filter))); // Return call addresses. Allow needs to come first because we’ll jump to // it for unknown architectures. idx_allow = p.len - 2 - ct_mknod - ct_mknodat; idx_fake = p.len - 1 - ct_mknod - ct_mknodat; idx_mknod = p.len - ct_mknod - ct_mknodat; idx_mknodat = p.len - ct_mknodat; // Build a jump table for each architecture. The gist is: if architecture // matches, fall through into the jump table, otherwise jump to the next // architecture (or ALLOW for the last architecture). ii = 0; idx_next_arch = -1; // avoid warning on some compilers for (int ai = 0; SECCOMP_ARCHS[ai] != NR_END; ai++) { int jump; idx_next_arch = ii + syscall_cts[ai] + ct_jump_start + ct_mknod_jump; // load arch into accumulator iw(&p, ii++, BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, arch), 0, 0); // jump to next arch if arch doesn't match jump = idx_next_arch - ii - 1; T_ (jump <= 255); iw(&p, ii++, BPF_JMP|BPF_JEQ|BPF_K, SECCOMP_ARCHS[ai], 0, jump); // load syscall number into accumulator iw(&p, ii++, BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, nr), 0, 0); // jump table of syscalls for (int si = 0; FAKE_SYSCALL_NRS[si][0] != NR_END; si++) { int nr = FAKE_SYSCALL_NRS[si][ai]; if (nr != NR_NON) { jump = idx_fake - ii - 1; T_ (jump <= 255); iw(&p, ii++, BPF_JMP|BPF_JEQ|BPF_K, nr, jump, 0); } } // jump to mknod(2) handling (add even if syscall not implemented to // make the instruction counts simpler) jump = idx_mknod - ii - 1; T_ (jump <= 255); iw(&p, ii++, BPF_JMP|BPF_JEQ|BPF_K, FAKE_MKNOD_NRS[ai], jump, 0); // jump to mknodat(2) handling jump = idx_mknodat - ii - 1; T_ (jump <= 255); iw(&p, ii++, BPF_JMP|BPF_JEQ|BPF_K, FAKE_MKNODAT_NRS[ai], jump, 0); // unfiltered syscall, jump to allow (limit of 255 doesn’t apply to JA) jump = idx_allow - ii - 1; iw(&p, ii++, BPF_JMP|BPF_JA, jump, 0, 0); } T_ (idx_next_arch == idx_allow); // Returns. (Note that if we wanted a non-zero errno, we’d bitwise-or with // SECCOMP_RET_ERRNO. But because fake success is errno == 0, we don’t need // a no-op “| 0”.) T_ (ii == idx_allow); iw(&p, ii++, BPF_RET|BPF_K, SECCOMP_RET_ALLOW, 0, 0); T_ (ii == idx_fake); iw(&p, ii++, BPF_RET|BPF_K, SECCOMP_RET_ERRNO, 0, 0); // mknod(2) handling. This just loads the file mode and jumps to the right // place in the mknodat(2) handling. T_ (ii == idx_mknod); // load mode argument into accumulator iw(&p, ii++, BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, args[1]), 0, 0); // jump to mode test iw(&p, ii++, BPF_JMP|BPF_JA, 1, 0, 0); // mknodat(2) handling. T_ (ii == idx_mknodat); // load mode argument into accumulator iw(&p, ii++, BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, args[2]), 0, 0); // jump to fake return if trying to create a device. iw(&p, ii++, BPF_ALU|BPF_AND|BPF_K, S_IFMT, 0, 0); // file type only iw(&p, ii++, BPF_JMP|BPF_JEQ|BPF_K, S_IFCHR, 2, 0); iw(&p, ii++, BPF_JMP|BPF_JEQ|BPF_K, S_IFBLK, 1, 0); // returns iw(&p, ii++, BPF_RET|BPF_K, SECCOMP_RET_ALLOW, 0, 0); iw(&p, ii++, BPF_RET|BPF_K, SECCOMP_RET_ERRNO, 0, 0); // Install filter. Use prctl(2) rather than seccomp(2) for slightly greater // compatibility (Linux 3.5 rather than 3.17) and because there is a glibc // wrapper. T_ (ii == p.len); // next instruction now one past the end of the buffer Z_ (prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &p)); DEBUG("note: see FAQ to disassemble the above") // Test filter. This will fail if the kernel executes the call (because we // are not really privileged and the arguments are bogus) or succeed if // filter handles it. We selected it over something more naturally in the // filter, e.g. setuid(2), because (1) no container process should ever use // it and (2) it’s unlikely to be emulated by a smarter filter in the // future, i.e., it won’t silently start doing something. Zf (syscall(SYS_kexec_load, 0, 0, NULL, 0), "seccomp root emulation failed (is your architecture supported?)"); } #endif /* Wait for semaphore sem for up to timeout seconds. If timeout or an error, exit unsuccessfully. */ void sem_timedwait_relative(sem_t *sem, int timeout) { struct timespec deadline; // sem_timedwait() requires a deadline rather than a timeout. Z_ (clock_gettime(CLOCK_REALTIME, &deadline)); deadline.tv_sec += timeout; if (sem_timedwait(sem, &deadline)) { Ze (errno == ETIMEDOUT, "timeout waiting for join lock"); Tf (0, "failure waiting for join lock"); } } /* Activate the desired isolation namespaces. */ void setup_namespaces(const struct container *c, uid_t uid_out, uid_t uid_in, gid_t gid_out, gid_t gid_in) { int fd; LOG_IDS; Zf (unshare(CLONE_NEWNS|CLONE_NEWUSER), "can't init user+mount namespaces"); LOG_IDS; /* Write UID map. What we are allowed to put here is quite limited. Because we do not have CAP_SETUID in the *parent* user namespace, we can map exactly one UID: an arbitrary container UID to our EUID in the parent namespace. This is sufficient to change our UID within the container; no setuid(2) or similar required. This is because the EUID of the process in the parent namespace is unchanged, so the kernel uses our new 1-to-1 map to convert that EUID into the container UID for most (maybe all) purposes. */ T_ (-1 != (fd = open("/proc/self/uid_map", O_WRONLY))); T_ (1 <= dprintf(fd, "%d %d 1\n", uid_in, uid_out)); Z_ (close(fd)); LOG_IDS; T_ (-1 != (fd = open("/proc/self/setgroups", O_WRONLY))); T_ (1 <= dprintf(fd, "deny\n")); Z_ (close(fd)); T_ (-1 != (fd = open("/proc/self/gid_map", O_WRONLY))); T_ (1 <= dprintf(fd, "%d %d 1\n", gid_in, gid_out)); Z_ (close(fd)); LOG_IDS; } /* Build /etc/passwd and /etc/group files and bind-mount them into newroot. /etc/passwd contains root, nobody, and an entry for the container UID, i.e., three entries, or two if the container UID is 0 or 65534. We copy the host's user data for the container UID, if that exists, and use dummy data otherwise (see issue #649). /etc/group works similarly: root, nogroup, and an entry for the container GID. We build new files to capture the relevant host username and group name mappings regardless of where they come from. We used to simply bind-mount the host's /etc/passwd and /etc/group, but this fails for LDAP at least; see issue #212. After bind-mounting, we remove the files from the host; they persist inside the container and then disappear completely when the container exits. */ void setup_passwd(const struct container *c) { int fd; char *path; struct group *g; struct passwd *p; // /etc/passwd T_ (path = cat(host_tmp, "/ch-run_passwd.XXXXXX")); T_ (-1 != (fd = mkstemp(path))); // mkstemp(3) writes path if (c->container_uid != 0) T_ (1 <= dprintf(fd, "root:x:0:0:root:/root:/bin/sh\n")); if (c->container_uid != 65534) T_ (1 <= dprintf(fd, "nobody:x:65534:65534:nobody:/:/bin/false\n")); errno = 0; p = getpwuid(c->container_uid); if (p) { T_ (1 <= dprintf(fd, "%s:x:%u:%u:%s:/:/bin/sh\n", p->pw_name, c->container_uid, c->container_gid, p->pw_gecos)); } else { if (errno) { Tf (0, "getpwuid(3) failed"); } else { VERBOSE("UID %d not found; using dummy info", c->container_uid); T_ (1 <= dprintf(fd, "%s:x:%u:%u:%s:/:/bin/sh\n", "charlie", c->container_uid, c->container_gid, "Charlie")); } } Z_ (close(fd)); bind_mount(path, "/etc/passwd", BD_REQUIRED, c->newroot, 0, NULL); Z_ (unlink(path)); // /etc/group T_ (path = cat(host_tmp, "/ch-run_group.XXXXXX")); T_ (-1 != (fd = mkstemp(path))); if (c->container_gid != 0) T_ (1 <= dprintf(fd, "root:x:0:\n")); if (c->container_gid != 65534) T_ (1 <= dprintf(fd, "nogroup:x:65534:\n")); errno = 0; g = getgrgid(c->container_gid); if (g) { T_ (1 <= dprintf(fd, "%s:x:%u:\n", g->gr_name, c->container_gid)); } else { if (errno) { Tf (0, "getgrgid(3) failed"); } else { VERBOSE("GID %d not found; using dummy info", c->container_gid); T_ (1 <= dprintf(fd, "%s:x:%u:\n", "charliegroup", c->container_gid)); } } Z_ (close(fd)); bind_mount(path, "/etc/group", BD_REQUIRED, c->newroot, 0, NULL); Z_ (unlink(path)); } /* Mount a tmpfs at the given path. */ void tmpfs_mount(const char *dst, const char *newroot, const char *data) { char *dst_full = cat(newroot, dst); Zf (mount(NULL, dst_full, "tmpfs", 0, data), "can't mount tmpfs at %s", dst_full); } charliecloud-0.37/bin/ch_core.h000066400000000000000000000037431457016721300164410ustar00rootroot00000000000000/* Copyright © Triad National Security, LLC, and others. This interface contains Charliecloud's core containerization features. */ #define _GNU_SOURCE #include /** Types **/ enum bind_dep { BD_REQUIRED, // both source and destination must exist BD_OPTIONAL, // if either source or destination missing, do nothing BD_MAKE_DST, // source must exist, try to create destination if it doesn't }; struct bind { char *src; char *dst; enum bind_dep dep; }; enum img_type { IMG_DIRECTORY, // normal directory, perhaps an external mount of some kind IMG_SQUASH, // SquashFS archive file (not yet mounted) IMG_NAME, // name of image in storage IMG_NONE, // image type is not set yet }; struct container { struct bind *binds; gid_t container_gid; // GID to use in container uid_t container_uid; // UID to use in container bool env_expand; // expand variables in --set-env char *host_home; // if --home, host path to user homedir, else NULL char *img_ref; // image description from command line char *newroot; // path to new root directory bool join; // is this a synchronized join? int join_ct; // number of peers in a synchronized join pid_t join_pid; // process in existing namespace to join char *join_tag; // identifier for synchronized join char *overlay_size; // size of overlaid tmpfs (NULL for no overlay) bool private_passwd; // don't bind custom /etc/{passwd,group} bool private_tmp; // don't bind host's /tmp enum img_type type; // directory, SquashFS, etc. bool writable; // re-mount image read-write }; /** Function prototypes **/ void containerize(struct container *c); enum img_type image_type(const char *ref, const char *images_dir); char *img_name2path(const char *name, const char *storage_dir); void run_user_command(char *argv[], const char *initial_dir); #ifdef HAVE_SECCOMP void seccomp_install(void); #endif charliecloud-0.37/bin/ch_fuse.c000066400000000000000000000235711457016721300164470ustar00rootroot00000000000000/* Copyright © Triad National Security, LLC, and others. */ /* Function prefixes: fuse_ libfuse; docs: https://libfuse.github.io/doxygen/globals.html sqfs_ll_ SquashFUSE; no docs but: https://github.com/vasi/squashfuse sq_ Charliecloud */ #define _GNU_SOURCE #include #include #include #include #include #include // SquashFUSE has a bug [1] where ll.h includes SquashFUSE's own config.h. // This clashes with our own config.h, as well as the system headers because // it defines _POSIX_C_SOURCE. By defining SQFS_CONFIG_H, SquashFUSE's // config.h skips itself. // [1]: https://github.com/vasi/squashfuse/issues/65 #define SQFS_CONFIG_H // But then FUSE_USE_VERSION isn't defined, which makes other parts of ll.h // puke. Looking at their code, it seems the only values used are 32 (for // libfuse3) and 26 (for libfuse2), so we can just blindly define it. #define FUSE_USE_VERSION 32 // SquashFUSE redefines __le16 unless HAVE_LINUX_TYPES_LE16 is defined. We are // assuming it is defined in on your machine. #define HAVE_LINUX_TYPES_LE16 // Now we can include ll.h. #include #include "config.h" #include "ch_core.h" #include "ch_fuse.h" #include "ch_misc.h" /** Types **/ /* A SquashFUSE mount. SquashFUSE allocates ll for us but not chan; use pointers for both for consistency. */ struct squash { char *mountpt; // path to mount point sqfs_ll_chan *chan; // FUSE channel associated with SquashFUSE mount sqfs_ll *ll; // SquashFUSE low-level data structure }; /** Constants **/ /* This mapping tells libfuse what functions implement which FUSE operations. It is passed to sqfs_ll_mount(). Why it is not internal to SquashFUSE I have no idea. */ struct fuse_lowlevel_ops OPS = { .getattr = &sqfs_ll_op_getattr, .opendir = &sqfs_ll_op_opendir, .releasedir = &sqfs_ll_op_releasedir, .readdir = &sqfs_ll_op_readdir, .lookup = &sqfs_ll_op_lookup, .open = &sqfs_ll_op_open, .create = &sqfs_ll_op_create, .release = &sqfs_ll_op_release, .read = &sqfs_ll_op_read, .readlink = &sqfs_ll_op_readlink, .listxattr = &sqfs_ll_op_listxattr, .getxattr = &sqfs_ll_op_getxattr, .forget = &sqfs_ll_op_forget, .statfs = &stfs_ll_op_statfs }; /** Global variables **/ /* SquashFUSE mount. Initialized in sq_mount() and then used in most of the other functions in this file. It's a global because the signal handler needs access to it. */ struct squash sq; /* True if exit request signal handler received SIGCHLD. */ volatile bool sigchld_received; /* True if any exit request signal has been received. */ volatile bool loop_terminating = false; /** Function prototypes (private) **/ void sq_done_request(int signum); int sq_loop(); void sq_mount(const char *img_path, char *mountpt); /** Functions **/ /* Signal handler to end the FUSE loop. This simply requests FUSE to end its loop, causing fuse_session_loop() to exit. */ void sq_done_request(int signum) { if (!loop_terminating) { // only act on first signal loop_terminating = true; sigchld_received = (signum == SIGCHLD); fuse_session_exit(sq.chan->session); } } /* Mount SquashFS archive c->img_path on directory c->newroot. If the latter is NULL, then mkdir(2) the default mount point and assign its path to c->newroot. After mounting, fork; the child returns immediately while the parent runs the FUSE loop until the child exits and then exits itself, with the same exit code as the child (unless something else went wrong). */ void sq_fork(struct container *c) { pid_t pid_child; struct stat st; // Default mount point? if (c->newroot == NULL) { char *subdir; T_ (asprintf(&subdir, "/%s.ch/mnt", username) > 0); c->newroot = cat("/var/tmp", subdir); VERBOSE("using default mount point: %s", c->newroot); mkdirs("/var/tmp", subdir, NULL, NULL); } // Verify mount point exists and is a directory. (SquashFS file path // already checked in img_type_get().) Zf (stat(c->newroot, &st), "can't stat mount point: %s", c->newroot); Te (S_ISDIR(st.st_mode), "not a directory: %s", c->newroot); // Mount SquashFS. Use PR_SET_NO_NEW_PRIVS to actively reject running // fusermount3(1) setuid, even if it’s installed that way. Zf (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0), "can't set no_new_privs"); sq_mount(c->img_ref, c->newroot); // Now that the filesystem is mounted, we can fork without race condition. // The child returns to caller and runs the user command. When that exits, // the parent gets SIGCHLD. pid_child = fork(); Tf (pid_child >= 0, "can't fork"); if (pid_child > 0) // parent (child does nothing here) exit(sq_loop()); } /* Run the squash loop to completion and return the exit code of the user command. Warning: This sets up but does not restore signal handlers. */ int sq_loop(void) { struct sigaction fin, ign; int looped, exit_code, child_status; // Set up signal handlers. Avoid fuse_set_signal_handlers() because we need // to catch a different set of signals, letting some be handled by the user // command [1]. Use sigaction(2) instead of signal(2) because the latter's // man page [2] says “avoid its use” and there are reports of bad // interactions with libfuse [3]. // // [1]: https://unix.stackexchange.com/questions/176235 // [2]: https://man7.org/linux/man-pages/man2/signal.2.html // [3]: https://stackoverflow.com/a/8918597 fin.sa_handler = sq_done_request; Z_ (sigemptyset(&fin.sa_mask)); // block no other signals during handling fin.sa_flags = SA_NOCLDSTOP; // only SIGCHLD on child exit ign.sa_handler = SIG_IGN; Z_ (sigaction(SIGCHLD, &fin, NULL)); // user command exits Z_ (sigaction(SIGHUP, &ign, NULL)); // terminal/session terminated Z_ (sigaction(SIGINT, &ign, NULL)); // Control-C Z_ (sigaction(SIGPIPE, &ign, NULL)); // broken pipe; we don't use pipes Z_ (sigaction(SIGTERM, &fin, NULL)); // somebody asked us to exit // Run the FUSE loop, which services FUSE requests until sq_done_request() // is invoked by a signal and tells it to stop, or someone unmounts the // filesystem externally with e.g. fusermount(1). Because we don't use // fuse_set_signal_handlers(), the return value doesn't contain the signal // number that ended the loop, contrary to the documentation. // // FIXME: this is single-threaded; see issue #1157. looped = fuse_session_loop(sq.chan->session); if (looped < 0) { errno = -looped; // restore encoded errno so our logging finds it Tf (0, "FUSE session failed"); } VERBOSE("FUSE loop terminated successfully"); // Clean up zombie child if exit signal was SIGCHLD. if (!sigchld_received) exit_code = 0; else { Tf (wait(&child_status) >= 0, "can't wait for child"); if (WIFEXITED(child_status)) { exit_code = WEXITSTATUS(child_status); VERBOSE("child terminated normally with exit code %d", exit_code); } else { // We now know that the child did not exit normally; the two // remaining options are (a) killed by signal and (b) stopped [1]. // Because we didn't call waitpid(2) with WUNTRACED, we don't get // notified if the child is stopped [2], so it must have been // signaled, and we need not call WIFSIGNALED(). // // [1]: https://codereview.stackexchange.com/a/109349 // [2]: https://man7.org/linux/man-pages/man2/wait.2.html exit_code = 1; VERBOSE("child terminated by signal %d", WTERMSIG(child_status)) } } // Clean up SquashFS mount. These functions have no error reporting. VERBOSE("unmounting: %s", sq.mountpt); sqfs_ll_destroy(sq.ll); sqfs_ll_unmount(sq.chan, sq.mountpt); VERBOSE("FUSE loop done"); return exit_code; } /* Mount the SquashFS img_path at mountpt. Exit on any errors. */ void sq_mount(const char *img_path, char *mountpt) { // SquashFUSE mount takes basically a command line rather than having a // standard library API. It's unclear to me where this command line is // documented, but the libfuse docs [1] suggest mount(8). // [1]: https://libfuse.github.io/doxygen/fuse-3_810_83_2include_2fuse_8h.html#ad866b0fd4d81bdbf3e737f7273ba4520 char *mount_argv[] = {"WEIRDAL", "-d"}; int mount_argc = (verbose > 3) ? 2 : 1; // include -d if high verbosity struct fuse_args mount_args = FUSE_ARGS_INIT(mount_argc, mount_argv); sq.mountpt = mountpt; T_ (sq.chan = malloc(sizeof(sqfs_ll_chan))); sq.ll = sqfs_ll_open(img_path, 0); Te (sq.ll != NULL, "can't open SquashFS: %s; try ch-run -vv?", img_path); // sqfs_ll_mount() is squirrely for a couple reasons: // // 1. Error reporting. We get back only SQFS_OK or SQFS_ERR, with no // further detail. Looking at the source code [1], the latter says // either fuse_session_new() or fuse_session_mount() failed, but we // can't tell which, or get any further information about what went // wrong. Hopefully fusermount3 also printed an error message. // // 2. Race condition. We have been seeing intermittent errors in the test // suite about permission denied accessing the mount point (issue // #1364). I *think* this is because a previous mount on the same // location is not yet cleaned up. For this reason, we have a short // retry loop. // // [1]: https://github.com/vasi/squashfuse/blob/74f4fe8/ll.c#L399 for (int i = 5; true; i--) if (SQFS_OK == sqfs_ll_mount(sq.chan, sq.mountpt, &mount_args, &OPS, sizeof(OPS), sq.ll)) { break; // success } else if (i <= 0) { FATAL("too many FUSE errors; giving up"); } else { WARNING("FUSE error mounting SquashFS; will retry"); sleep(1); } } charliecloud-0.37/bin/ch_fuse.h000066400000000000000000000002231457016721300164410ustar00rootroot00000000000000/* Copyright © Triad National Security, LLC, and others. */ #define _GNU_SOURCE /** Function prototypes **/ void sq_fork(struct container *c); charliecloud-0.37/bin/ch_misc.c000066400000000000000000000704551457016721300164430ustar00rootroot00000000000000/* Copyright © Triad National Security, LLC, and others. */ #define _GNU_SOURCE #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "config.h" #include "ch_misc.h" /** Macros **/ /* FNM_EXTMATCH is a GNU extension to support extended globs in fnmatch(3). If not available, define as 0 to ignore this flag. */ #ifndef HAVE_FNM_EXTMATCH #define FNM_EXTMATCH 0 #endif /* Number of supplemental GIDs we can deal with. */ #define SUPP_GIDS_MAX 128 /** External variables **/ /* Level of chatter on stderr. */ enum log_level verbose; /* Path to host temporary directory. Set during command line processing. */ char *host_tmp = NULL; /* Username of invoking users. Set during command line processing. */ char *username = NULL; /* List of warnings to be re-printed on exit. This is a buffer of shared memory allocated by mmap(2), structured as a sequence of null-terminated character strings. Warnings that do not fit in this buffer will be lost, though we allocate enough memory that this is unlikely. See “string_append()” for more details. */ char *warnings; /* Current byte offset from start of “warnings” buffer. This gives the address where the next appended string will start. This means that the null terminator of the previous string is warnings_offset - 1. */ size_t warnings_offset = 0; /** Function prototypes (private) **/ void mkdir_overmount(const char *path, const char *scratch); void msgv(enum log_level level, const char *file, int line, int errno_, const char *fmt, va_list ap); /** Functions **/ /* Serialize the null-terminated vector of arguments argv and return the result as a newly allocated string. The purpose is to provide a human-readable reconstruction of a command line where each argument can also be recovered byte-for-byte; see ch-run(1) for details. */ char *argv_to_string(char **argv) { char *s = NULL; for (size_t i = 0; argv[i] != NULL; i++) { char *argv_, *x; bool quote_p = false; // Max length is escape every char plus two quotes and terminating zero. T_ (argv_ = calloc(2 * strlen(argv[i]) + 3, 1)); // Copy to new string, escaping as we go. Note lots of fall-through. I'm // not sure where this list of shell meta-characters came from; I just // had it on hand already from when we were deciding on the image // reference transformation for filesystem paths. for (size_t ji = 0, jo = 0; argv[i][ji] != 0; ji++) { char c = argv[i][ji]; if (isspace(c) || !isascii(c) || !isprint(c)) quote_p = true; switch (c) { case '!': // history expansion case '"': // string delimiter case '$': // variable expansion case '\\': // escape character case '`': // output expansion argv_[jo++] = '\\'; case '#': // comment case '%': // job ID case '&': // job control case '\'': // string delimiter case '(': // subshell grouping case ')': // subshell grouping case '*': // globbing case ';': // command separator case '<': // redirect case '=': // globbing case '>': // redirect case '?': // globbing case '[': // globbing case ']': // globbing case '^': // command “quick substitution” case '{': // command grouping case '|': // pipe case '}': // command grouping case '~': // home directory expansion quote_p = true; default: argv_[jo++] = c; break; } } if (quote_p) { x = argv_; T_ (1 <= asprintf(&argv_, "\"%s\"", argv_)); free(x); } if (i != 0) { x = s; s = cat(s, " "); free(x); } x = s; s = cat(s, argv_); free(x); free(argv_); } return s; } /* Iterate through buffer “buf” of size “s” consisting of null-terminated strings and return the number of strings in it. Key assumptions: 1. The buffer has been initialized to zero, i.e. all bytes that have not been explicitly set are null. 2. All strings have been appended to the buffer in full without truncation, including their null terminator. 3. The buffer contains no empty strings. These assumptions are consistent with the construction of the “warnings” shared memory buffer, which is the main justification for this function. Note that under these assumptions, the final byte in the buffer is guaranteed to be null. */ int buf_strings_count(char *buf, size_t size) { int count = 0; if (buf[0] != '\0') { for (size_t i = 0; i < size; i++) if (buf[i] == '\0') { // found string terminator count++; if (i < size - 1 && buf[i+1] == '\0') // two term. in a row; done break; } } return count; } /* Return true if buffer buf of length size is all zeros, false otherwise. */ bool buf_zero_p(void *buf, size_t size) { for (size_t i = 0; i < size; i++) if (((char *)buf)[i] != 0) return false; return true; } /* Concatenate strings a and b, then return the result. */ char *cat(const char *a, const char *b) { char *ret; if (a == NULL) a = ""; if (b == NULL) b = ""; T_ (asprintf(&ret, "%s%s", a, b) == strlen(a) + strlen(b)); return ret; } /* Like scandir(3), but (1) filter excludes “.” and “..”, (2) results are not sorted, and (3) cannot fail (exits with an error instead). */ int dir_ls(const char *path, struct dirent ***namelist) { int entry_ct; entry_ct = scandir(path, namelist, dir_ls_filter, NULL); Tf (entry_ct >= 0, "can't scan dir", path); return entry_ct; } /* Return the number of entries in directory path, not including “.” and “..”; i.e., the empty directory returns 0 despite them. */ int dir_ls_count(const char *path) { int ct; struct dirent **namelist; ct = dir_ls(path, &namelist); for (size_t i = 0; i < ct; i++) free(namelist[i]); free(namelist); return ct; } /* scandir(3) filter that excludes “.” and “..”: Return 0 if e->d_name is one of those strings, else 1. */ int dir_ls_filter(const struct dirent *e) { return !(!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")); } /* Read the file listing environment variables at path, with records separated by delim, and return a corresponding list of struct env_var. Reads the entire file one time without seeking. If there is a problem reading the file, or with any individual variable, exit with error. The purpose of delim is to allow both newline- and zero-delimited files. We did consider using a heuristic to choose the file’s delimiter, but there seemed to be two problems. First, every heuristic we considered had flaws. Second, use of a heuristic would require reading the file twice or seeking. We don’t want to demand non-seekable files (e.g., pipes), and if we read the file into a buffer before parsing, we’d need our own getdelim(3). See issue #1124 for further discussion. */ struct env_var *env_file_read(const char *path, int delim) { struct env_var *vars; FILE *fp; Tf (fp = fopen(path, "r"), "can't open: %s", path); vars = list_new(sizeof(struct env_var), 0); for (size_t line_no = 1; true; line_no++) { struct env_var var; char *line = NULL; size_t line_len = 0; // don't care but required by getline(3) errno = 0; if (-1 == getdelim(&line, &line_len, delim, fp)) { if (errno == 0) // EOF break; else Tf (0, "can't read: %s", path); } if (line[strlen(line) - 1] == '\n') // rm newline if present line[strlen(line) - 1] = 0; if (line[0] == 0) // skip blank lines continue; var = env_var_parse(line, path, line_no); list_append((void **)&vars, &var, sizeof(var)); } Zf (fclose(fp), "can't close: %s", path); return vars; } /* Set environment variable name to value. If expand, then further expand variables in value marked with "$" as described in the man page. */ void env_set(const char *name, const char *value, const bool expand) { char *value_, *value_expanded; bool first_written; // Walk through value fragments separated by colon and expand variables. T_ (value_ = strdup(value)); value_expanded = ""; first_written = false; while (true) { // loop executes ≥ once char *fgmt = strsep(&value_, ":"); // NULL -> no more items if (fgmt == NULL) break; if (expand && fgmt[0] == '$' && fgmt[1] != 0) { fgmt = getenv(fgmt + 1); // NULL if unset if (fgmt != NULL && fgmt[0] == 0) fgmt = NULL; // convert empty to unset } if (fgmt != NULL) { // NULL -> omit from output if (first_written) value_expanded = cat(value_expanded, ":"); value_expanded = cat(value_expanded, fgmt); first_written = true; } } // Save results. VERBOSE("environment: %s=%s", name, value_expanded); Z_ (setenv(name, value_expanded, 1)); } /* Remove variables matching glob from the environment. This is tricky, because there is no standard library function to iterate through the environment, and the environ global array can be re-ordered after unsetenv(3) [1]. Thus, the only safe way without additional storage is an O(n^2) search until no matches remain. Our approach is O(n): we build up a copy of environ, skipping variables that match the glob, and then assign environ to the copy. (This is a valid thing to do [2].) [1]: https://unix.stackexchange.com/a/302987 [2]: http://man7.org/linux/man-pages/man3/exec.3p.html */ void env_unset(const char *glob) { char **new_environ = list_new(sizeof(char *), 0); for (size_t i = 0; environ[i] != NULL; i++) { char *name, *value; int matchp; split(&name, &value, environ[i], '='); T_ (name != NULL); // environ entries must always have equals matchp = fnmatch(glob, name, FNM_EXTMATCH); // extglobs if available if (matchp == 0) { VERBOSE("environment: unset %s", name); } else { T_ (matchp == FNM_NOMATCH); *(value - 1) = '='; // rejoin line list_append((void **)&new_environ, &name, sizeof(name)); } } environ = new_environ; } /* Parse the environment variable in line and return it as a struct env_var. Exit with error on syntax error; if path is non-NULL, attribute the problem to that path at line_no. Note: Trailing whitespace such as newline is *included* in the value. */ struct env_var env_var_parse(const char *line, const char *path, size_t lineno) { char *name, *value, *where; if (path == NULL) { T_ (where = strdup(line)); } else { T_ (1 <= asprintf(&where, "%s:%zu", path, lineno)); } // Split line into variable name and value. split(&name, &value, line, '='); Te (name != NULL, "can't parse variable: no delimiter: %s", where); Te (name[0] != 0, "can't parse variable: empty name: %s", where); free(where); // for Tim // Strip leading and trailing single quotes from value, if both present. if ( strlen(value) >= 2 && value[0] == '\'' && value[strlen(value) - 1] == '\'') { value[strlen(value) - 1] = 0; value++; } return (struct env_var){ name, value }; } /* Copy the buffer of size size pointed to by new into the last position in the zero-terminated array of elements with the same size on the heap pointed to by *ar, reallocating it to hold one more element and setting list to the new location. *list can be NULL to initialize a new list. Return the new array size. Note: ar must be cast, e.g. "list_append((void **)&foo, ...)". Warning: This function relies on all pointers having the same representation, which is true on most modern machines but is not guaranteed by the standard [1]. We could instead return the new value of ar rather than using an out parameter, which would avoid the double pointer and associated non-portability but make it easy for callers to create dangling pointers, i.e., after "a = list_append(b, ...)", b will dangle. That problem could in turn be avoided by returning a *copy* of the array rather than a modified array, but then the caller has to deal with the original array itself. It seemed to me the present behavior was the best trade-off. [1]: http://www.c-faq.com/ptrs/genericpp.html */ void list_append(void **ar, void *new, size_t size) { int ct; T_ (new != NULL); // count existing elements if (*ar == NULL) ct = 0; else for (ct = 0; !buf_zero_p((char *)*ar + ct*size, size); ct++) ; T_ (*ar = realloc(*ar, (ct+2)*size)); // existing + new + terminator memcpy((char *)*ar + ct*size, new, size); // append new (no overlap) memset((char *)*ar + (ct+1)*size, 0, size); // set new terminator } /* Return a pointer to a new, empty zero-terminated array containing elements of size size, with room for ct elements without re-allocation. The latter allows to pre-allocate an arbitrary number of slots in the list, which can then be filled directly without testing the list's length for each one. (The list is completely filled with zeros, so every position has a terminator after it.) */ void *list_new(size_t size, size_t ct) { void *list; T_ (list = calloc(ct+1, size)); return list; } /* If verbose, print uids and gids on stderr prefixed with where. */ void log_ids(const char *func, int line) { uid_t ruid, euid, suid; gid_t rgid, egid, sgid; gid_t supp_gids[SUPP_GIDS_MAX]; int supp_gid_ct; if (verbose >= 3) { Z_ (getresuid(&ruid, &euid, &suid)); Z_ (getresgid(&rgid, &egid, &sgid)); fprintf(stderr, "%s %d: uids=%d,%d,%d, gids=%d,%d,%d + ", func, line, ruid, euid, suid, rgid, egid, sgid); supp_gid_ct = getgroups(SUPP_GIDS_MAX, supp_gids); if (supp_gid_ct == -1) { T_ (errno == EINVAL); Te (0, "more than %d groups", SUPP_GIDS_MAX); } for (int i = 0; i < supp_gid_ct; i++) { if (i > 0) fprintf(stderr, ","); fprintf(stderr, "%d", supp_gids[i]); } fprintf(stderr, "\n"); } } void test_logging(bool fail) { TRACE("trace"); DEBUG("debug"); VERBOSE("verbose"); INFO("info"); WARNING("warning"); if (fail) FATAL("the program failed inexplicably (\"log-fail\" specified)"); exit(0); } /* Create the directory at path, despite its parent not allowing write access, by overmounting a new, writeable directory atop it. We preserve the old contents by bind-mounting the old directory as a subdirectory, then setting up a symlink ranch. The new directory lives initially in scratch, which must not be used for any other purpose. No cleanup is done here, so a disposable tmpfs is best. If anything goes wrong, exit with an error message. */ void mkdir_overmount(const char *path, const char *scratch) { char *parent, *path2, *over, *path_dst; char *orig_dir = ".orig"; // resisted calling this .weirdal int entry_ct; struct dirent **entries; VERBOSE("making writeable via symlink ranch: %s", path); path2 = strdup(path); parent = dirname(path2); T_ (1 <= asprintf(&over, "%s/%d", scratch, dir_ls_count(scratch) + 1)); path_dst = path_join(over, orig_dir); // bind-mounts Z_ (mkdir(over, 0755)); Z_ (mkdir(path_dst, 0755)); Zf (mount(parent, path_dst, NULL, MS_REC|MS_BIND, NULL), "can't bind-mount: %s -> %s", parent, path_dst); Zf (mount(over, parent, NULL, MS_REC|MS_BIND, NULL), "can't bind-mount: %s- > %s", over, parent); // symlink ranch entry_ct = dir_ls(path_dst, &entries); DEBUG("existing entries: %d", entry_ct); for (int i = 0; i < entry_ct; i++) { char * src = path_join(parent, entries[i]->d_name); char * dst = path_join(orig_dir, entries[i]->d_name); Zf (symlink(dst, src), "can't symlink: %s -> %s", src, dst); free(src); free(dst); free(entries[i]); } free(entries); Zf (mkdir(path, 0755), "can't mkdir even after overmount: %s", path); free(path_dst); free(over); free(path2); } /* Create directories in path under base. Exit with an error if anything goes wrong. For example, mkdirs("/foo", "/bar/baz") will create directories /foo/bar and /foo/bar/baz if they don't already exist, but /foo must exist already. Symlinks are followed. path must remain under base, i.e. you can't use symlinks or ".." to climb out. denylist is a null-terminated array of paths under which no directories may be created, or NULL if none. Can defeat an un-writeable directory by overmounting a new writeable directory atop it. To enable this behavior, pass the path to an appropriate scratch directory in scratch. */ void mkdirs(const char *base, const char *path, char **denylist, const char *scratch) { char *basec, *component, *next, *nextc, *pathw, *saveptr; char *denylist_null[] = { NULL }; struct stat sb; T_ (base[0] != 0 && path[0] != 0); // no empty paths T_ (base[0] == '/' && path[0] == '/'); // absolute paths only if (denylist == NULL) denylist = denylist_null; // literal here causes intermittent segfaults basec = realpath_(base, false); TRACE("mkdirs: base: %s", basec); TRACE("mkdirs: path: %s", path); for (size_t i = 0; denylist[i] != NULL; i++) TRACE("mkdirs: deny: %s", denylist[i]); pathw = cat(path, ""); // writeable copy saveptr = NULL; // avoid warning (#1048; see also strtok_r(3)) component = strtok_r(pathw, "/", &saveptr); nextc = basec; next = NULL; while (component != NULL) { next = cat(nextc, "/"); next = cat(next, component); // canonical except for last component TRACE("mkdirs: next: %s", next) component = strtok_r(NULL, "/", &saveptr); // next NULL if current last if (path_exists(next, &sb, false)) { if (S_ISLNK(sb.st_mode)) { char buf; // we only care if absolute Tf (1 == readlink(next, &buf, 1), "can't read symlink: %s", next); Tf (buf != '/', "can't mkdir: symlink not relative: %s", next); Te (path_exists(next, &sb, true), // resolve symlink "can't mkdir: broken symlink: %s", next); } Tf (S_ISDIR(sb.st_mode) || !component, // last component not dir OK "can't mkdir: exists but not a directory: %s", next); nextc = realpath_(next, false); TRACE("mkdirs: exists, canonical: %s", nextc); } else { Te (path_subdir_p(basec, next), "can't mkdir: %s not subdirectory of %s", next, basec); for (size_t i = 0; denylist[i] != NULL; i++) Ze (path_subdir_p(denylist[i], next), "can't mkdir: %s under existing bind-mount %s", next, denylist[i]); if (mkdir(next, 0755)) { if (scratch && (errno == EACCES || errno == EPERM)) mkdir_overmount(next, scratch); else Tf (0, "can't mkdir: %s", next); } nextc = next; // canonical b/c we just created last component as dir TRACE("mkdirs: created: %s", nextc) } } TRACE("mkdirs: done"); } /* Print a formatted message on stderr if the level warrants it. */ void msg(enum log_level level, const char *file, int line, int errno_, const char *fmt, ...) { va_list ap; va_start(ap, fmt); msgv(level, file, line, errno_, fmt, ap); va_end(ap); } noreturn void msg_fatal(const char *file, int line, int errno_, const char *fmt, ...) { va_list ap; va_start(ap, fmt); msgv(LL_FATAL, file, line, errno_, fmt, ap); va_end(ap); exit(EXIT_FAILURE); } /* va_list form of msg(). */ void msgv(enum log_level level, const char *file, int line, int errno_, const char *fmt, va_list ap) { char *message, *ap_msg; if (level > verbose) return; T_ (1 <= asprintf(&message, "%s[%d]: ", program_invocation_short_name, getpid())); // Prefix for the more urgent levels. switch (level) { case LL_FATAL: message = cat(message, "error: "); // "fatal" too morbid for users break; case LL_WARNING: message = cat(message, "warning: "); break; default: break; } // Default message if not specified. Users should not see this. if (fmt == NULL) fmt = "please report this bug"; T_ (1 <= vasprintf(&ap_msg, fmt, ap)); if (errno_) { T_ (1 <= asprintf(&message, "%s%s: %s (%s:%d %d)", message, ap_msg, strerror(errno_), file, line, errno_)); } else { T_ (1 <= asprintf(&message, "%s%s (%s:%d)", message, ap_msg, file, line)); } if (level == LL_WARNING) { warnings_offset += string_append(warnings, message, WARNINGS_SIZE, warnings_offset); } fprintf(stderr, "%s\n", message); if (fflush(stderr)) abort(); // can't print an error b/c already trying to do that } /* Return true if the given path exists, false otherwise. On error, exit. If statbuf is non-null, store the result of stat(2) there. If follow_symlink is true and the last component of path is a symlink, stat(2) the target of the symlink; otherwise, lstat(2) the link itself. */ bool path_exists(const char *path, struct stat *statbuf, bool follow_symlink) { struct stat statbuf_; if (statbuf == NULL) statbuf = &statbuf_; if (follow_symlink) { if (stat(path, statbuf) == 0) return true; } else { if (lstat(path, statbuf) == 0) return true; } Tf (errno == ENOENT, "can't stat: %s", path); return false; } /* Concatenate paths a and b, then return the result. */ char *path_join(const char *a, const char *b) { char *ret; T_ (a != NULL); T_ (strlen(a) > 0); T_ (b != NULL); T_ (strlen(b) > 0); T_ (asprintf(&ret, "%s/%s", a, b) == strlen(a) + strlen(b) + 1); return ret; } /* Return the mount flags of the file system containing path, suitable for passing to mount(2). This is messy because, the flags we get from statvfs(3) are ST_* while the flags needed by mount(2) are MS_*. My glibc has a comment in bits/statvfs.h that the ST_* "should be kept in sync with" the MS_* flags, and the values do seem to match, but there are additional undocumented flags in there. Also, the kernel contains a test "unprivileged-remount-test.c" that manually translates the flags. Thus, I wasn't comfortable simply passing the output of statvfs(3) to mount(2). */ unsigned long path_mount_flags(const char *path) { struct statvfs sv; unsigned long known_flags = ST_MANDLOCK | ST_NOATIME | ST_NODEV | ST_NODIRATIME | ST_NOEXEC | ST_NOSUID | ST_RDONLY | ST_RELATIME | ST_SYNCHRONOUS; Z_ (statvfs(path, &sv)); // Flag 0x20 is ST_VALID according to the kernel [1], which clashes with // MS_REMOUNT, so inappropriate to pass through. Glibc unsets it from the // flag bits returned by statvfs(2) [2], but musl doesn’t [3], so unset it. // // [1]: https://github.com/torvalds/linux/blob/3644286f/include/linux/statfs.h#L27 // [2]: https://sourceware.org/git?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/internal_statvfs.c;h=b1b8dfefe6be909339520d120473bd67e4bece57 // [3]: https://git.musl-libc.org/cgit/musl/tree/src/stat/statvfs.c?h=v1.2.2 sv.f_flag &= ~0x20; Ze (sv.f_flag & ~known_flags, "unknown mount flags: 0x%lx %s", sv.f_flag & ~known_flags, path); return (sv.f_flag & ST_MANDLOCK ? MS_MANDLOCK : 0) | (sv.f_flag & ST_NOATIME ? MS_NOATIME : 0) | (sv.f_flag & ST_NODEV ? MS_NODEV : 0) | (sv.f_flag & ST_NODIRATIME ? MS_NODIRATIME : 0) | (sv.f_flag & ST_NOEXEC ? MS_NOEXEC : 0) | (sv.f_flag & ST_NOSUID ? MS_NOSUID : 0) | (sv.f_flag & ST_RDONLY ? MS_RDONLY : 0) | (sv.f_flag & ST_RELATIME ? MS_RELATIME : 0) | (sv.f_flag & ST_SYNCHRONOUS ? MS_SYNCHRONOUS : 0); } /* Split path into dirname and basename. */ void path_split(const char *path, char **dir, char **base) { char *path2; T_ (path2 = strdup(path)); T_ (*dir = strdup(dirname(path2))); free(path2); T_ (path2 = strdup(path)); T_ (*base = strdup(basename(path2))); free(path2); } /* Return true if path is a subdirectory of base, false otherwise. Acts on the paths as given, with no canonicalization or other reference to the filesystem. For example: path_subdir_p("/foo", "/foo/bar") => true path_subdir_p("/foo", "/bar") => false path_subdir_p("/foo/bar", "/foo/b") => false */ bool path_subdir_p(const char *base, const char *path) { int base_len = strlen(base); int path_len = strlen(base); // remove trailing slashes while (base[base_len-1] == '/' && base_len >= 1) base_len--; while (path[path_len-1] == '/' && path_len >= 1) path_len--; if (base_len > path_len) return false; if (!strcmp(base, "/")) // below logic breaks if base is root return true; return ( !strncmp(base, path, base_len) && (path[base_len] == '/' || path[base_len] == 0)); } /* Like realpath(3), but never returns an error. If the underlying realpath(3) fails or path is NULL, and fail_ok is true, then return a copy of the input; otherwise (i.e., fail_ok is false) exit with error. */ char *realpath_(const char *path, bool fail_ok) { char *pathc; if (path == NULL) return NULL; pathc = realpath(path, NULL); if (pathc == NULL) { if (fail_ok) { T_ (pathc = strdup(path)); } else { Tf (false, "can't canonicalize: %s", path); } } return pathc; } /* Replace all instances of character “old” in “s” with “new”. */ void replace_char(char *s, char old, char new) { for (int i = 0; s[i] != '\0'; i++) if(s[i] == old) s[i] = new; } /* Split string str at first instance of delimiter del. Set *a to the part before del, and *b to the part after. Both can be empty; if no token is present, set both to NULL. Unlike strsep(3), str is unchanged; *a and *b point into a new buffer allocated with malloc(3). This has two implications: (1) the caller must free(3) *a but not *b, and (2) the parts can be rejoined by setting *(*b-1) to del. The point here is to provide an easier wrapper for strsep(3). */ void split(char **a, char **b, const char *str, char del) { char *tmp; char delstr[2] = { del, 0 }; T_ (str != NULL); tmp = strdup(str); *b = tmp; *a = strsep(b, delstr); if (*b == NULL) *a = NULL; } /* Report the version number. */ void version(void) { fprintf(stderr, "%s\n", VERSION); } /* Append null-terminated string “str” to the memory buffer “offset” bytes after from the address pointed to by “addr”. Buffer length is “size” bytes. Return the number of bytes written. If there isn’t enough room for the string, do nothing and return zero. */ size_t string_append(char *addr, char *str, size_t size, size_t offset) { size_t written = strlen(str) + 1; if (size > (offset + written - 1)) // there is space memcpy(addr + offset, str, written); return written; } /* Reprint messages stored in “warnings” memory buffer. */ void warnings_reprint(void) { size_t offset = 0; int warn_ct = buf_strings_count(warnings, WARNINGS_SIZE); if (warn_ct > 0) fprintf(stderr, "%s[%d]: warning: reprinting first %d warning(s)\n", program_invocation_short_name, getpid(), warn_ct); while ( warnings[offset] != 0 || (offset < (WARNINGS_SIZE - 1) && warnings[offset+1] != 0)) { fputs(warnings + offset, stderr); fputc('\n', stderr); offset += strlen(warnings + offset) + 1; } if (fflush(stderr)) abort(); // can't print an error b/c already trying to do that } charliecloud-0.37/bin/ch_misc.h000066400000000000000000000131271457016721300164410ustar00rootroot00000000000000/* Copyright © Triad National Security, LLC, and others. This interface contains miscellaneous utility features. It is separate so that peripheral Charliecloud C programs don't have to link in the extra libraries that ch_core requires. */ #define _GNU_SOURCE #include #include #include #include /** Macros **/ /* Log the current UIDs. */ #define LOG_IDS log_ids(__func__, __LINE__) /* C99 does not have noreturn or _Noreturn (those are C11), but GCC, Clang, and hopefully others support the following extension. */ #define noreturn __attribute__ ((noreturn)) /* Size of “warnings” buffer, in bytes. We want this to be big enough that we don’t need to worry about running out of room. */ #define WARNINGS_SIZE (4*1024) /* Test some value, and if it's not what we expect, exit with a fatal error. These are macros so we have access to the file and line number. verify x is true (non-zero); otherwise print then exit: T_ (x) default error message including file, line, errno Tf (x, fmt, ...) printf-style message followed by file, line, errno Te (x, fmt, ...) same without errno verify x is zero (false); otherwise print as above & exit Z_ (x) Zf (x, fmt, ...) Ze (x, fmt, ...) errno is omitted if it's zero. Examples: Z_ (chdir("/does/not/exist")); -> ch-run: error: No such file or directory (ch-run.c:138 2) Zf (chdir("/does/not/exist"), "foo"); -> ch-run: foo: No such file or directory (ch-run.c:138 2) Ze (chdir("/does/not/exist"), "foo"); -> ch-run: foo (ch-run.c:138) errno = 0; Zf (0, "foo"); -> ch-run: foo (ch-run.c:138) Typically, Z_ and Zf are used to check system and standard library calls, while T_ and Tf are used to assert developer-specified conditions. errno is not altered by these macros unless they exit the program. FIXME: It would be nice if we could collapse these to fewer macros. However, when looking into that I ended up in preprocessor black magic (e.g. https://stackoverflow.com/a/2308651) that I didn't understand. */ #define T_(x) if (!(x)) msg_fatal(__FILE__, __LINE__, errno, NULL) #define Tf(x, ...) if (!(x)) msg_fatal(__FILE__, __LINE__, errno, __VA_ARGS__) #define Te(x, ...) if (!(x)) msg_fatal(__FILE__, __LINE__, 0, __VA_ARGS__) #define Z_(x) if (x) msg_fatal(__FILE__, __LINE__, errno, NULL) #define Zf(x, ...) if (x) msg_fatal(__FILE__, __LINE__, errno, __VA_ARGS__) #define Ze(x, ...) if (x) msg_fatal(__FILE__, __LINE__, 0, __VA_ARGS__) #define FATAL(...) msg_fatal( __FILE__, __LINE__, 0, __VA_ARGS__); #define WARNING(...) msg(LL_WARNING, __FILE__, __LINE__, 0, __VA_ARGS__); #define INFO(...) msg(LL_INFO, __FILE__, __LINE__, 0, __VA_ARGS__); #define VERBOSE(...) msg(LL_VERBOSE, __FILE__, __LINE__, 0, __VA_ARGS__); #define DEBUG(...) msg(LL_DEBUG, __FILE__, __LINE__, 0, __VA_ARGS__); #define TRACE(...) msg(LL_TRACE, __FILE__, __LINE__, 0, __VA_ARGS__); /** Types **/ enum env_action { ENV_END = 0, // terminate list of environment changes ENV_SET_DEFAULT, // set by /ch/environment within image ENV_SET_VARS, // set by list of variables ENV_UNSET_GLOB }; // unset glob matches struct env_var { char *name; char *value; }; struct env_delta { enum env_action action; union { int delim; // ENV_SET_DEFAULT struct env_var *vars; // ENV_SET_VARS char *glob; // ENV_UNSET_GLOB } arg; }; enum log_level { LL_FATAL = -3, LL_STDERR = -2, LL_WARNING = -1, LL_INFO = 0, // minimum number of -v to print the msg LL_VERBOSE = 1, LL_DEBUG = 2, LL_TRACE = 3 }; /** External variables **/ extern enum log_level verbose; extern char *host_tmp; extern char *username; extern char *warnings; extern size_t warnings_offset; /** Function prototypes **/ char *argv_to_string(char **argv); int buf_strings_count(char *str, size_t s); bool buf_zero_p(void *buf, size_t size); char *cat(const char *a, const char *b); int dir_ls(const char *path, struct dirent ***namelist); int dir_ls_count(const char *path); int dir_ls_filter(const struct dirent *e); struct env_var *env_file_read(const char *path, int delim); void env_set(const char *name, const char *value, const bool expand); void env_unset(const char *glob); struct env_var env_var_parse(const char *line, const char *path, size_t lineno); void list_append(void **ar, void *new, size_t size); void *list_new(size_t size, size_t ct); void log_ids(const char *func, int line); void test_logging(bool fail); void mkdirs(const char *base, const char *path, char **denylist, const char *scratch); void msg(enum log_level level, const char *file, int line, int errno_, const char *fmt, ...); noreturn void msg_fatal(const char *file, int line, int errno_, const char *fmt, ...); bool path_exists(const char *path, struct stat *statbuf, bool follow_symlink); char *path_join(const char *a, const char *b); unsigned long path_mount_flags(const char *path); void path_split(const char *path, char **dir, char **base); bool path_subdir_p(const char *base, const char *path); char *realpath_(const char *path, bool fail_ok); void replace_char(char *str, char old, char new); void split(char **a, char **b, const char *str, char del); void version(void); size_t string_append(char *addr, char *str, size_t size, size_t offset); void warnings_reprint(void); charliecloud-0.37/configure.ac000066400000000000000000001060001457016721300163720ustar00rootroot00000000000000# Gotchas: # # 1. Quadrigraphs. M4 consumes a number of important special characters, so # Autoconf uses 4-character sequences, e.g. "@%:@" is the octothorpe (#). # See: https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/Quadrigraphs.html # # 2. Booleans. The convention for Autoconf variables, which we follow, is # “yes” for true and “no” for false. This differs from the Charliecloud # convention of non-empty for true and empty for false. ### Prologue ################################################################# AC_INIT([Charliecloud], [m4_esyscmd_s([misc/version])], [https://github.com/hpc/charliecloud]) AC_PREREQ([2.69]) AC_CONFIG_SRCDIR([bin/ch-run.c]) AC_CONFIG_AUX_DIR([build-aux]) AC_CONFIG_MACRO_DIRS([misc/m4]) AC_CANONICAL_BUILD AC_CANONICAL_HOST AC_CANONICAL_TARGET AS_CASE([$host_os], [linux*], [], [*], [AC_MSG_ERROR([Linux is the only supported OS; see issue @%:@42.])] ) # Turn off maintainer mode by default. This appears to be controversial; see # issue #595 for links to some discussion. # # Bottom line for me: Maintainer mode has (1) never re-built the build system # in a situation where I felt it helped, but (2) fairly regularly re-builds or # re-configures at surprising times. # # In particular, it often rebuilds before “make clean” and friends, e.g. if # you change branches and then clean. This seems wrong. In my view, clean # should remove what is currently there, not what *would have been there* had # the build used a different, not-yet-existing build system. Disabling # maintainer mode also lets us put “make maintainer-clean” in autogen.sh # without triggering spurious rebuilds. AM_MAINTAINER_MODE([disable]) # By default, Autotools honors umask for directories but not files. Thus, if # you “sudo make install” with a umask more restrictive than 0022, the result # is an installation unavailable to most users (issue #947). This appears to # be a somewhat common complaint. # # Our workaround is to set the “mkdir -p” command [1]. (Note those # instructions also mention a different variable ac_cv_path_mkdir, but I # couldn’t figure out how to set it.) This needs to be before AM_INIT_AUTOMAKE # because that macro does something with the value. We use “install -d” rather # than “mkdir -m” because the latter still uses only umask for intermediate # directories [2]. # # This can still be overridden on the configure command line; for example, to # restore the previous behavior, use “./configure MKDIR_P='mkdir -p'” [3]. # # [1]: https://unix.stackexchange.com/a/436000 # [2]: http://gnu-automake.7480.n7.nabble.com/bug-12130-sudo-make-install-applies-umask-to-new-directories-tp18545p18548.html # [3]: https://lists.gnu.org/archive/html/automake/2004-01/msg00013.html MKDIR_P=${MKDIR_P:-install -d -m 0755} AM_INIT_AUTOMAKE([1.13 -Wall -Werror foreign subdir-objects]) AC_CONFIG_HEADERS([bin/config.h]) AC_CONFIG_FILES([Makefile bin/Makefile doc/Makefile examples/Makefile lib/Makefile misc/Makefile packaging/Makefile test/Makefile]) ### Options ################################################################## # Note: Variables must match option, e.g. --disable-foo-bar => enable_foo_bar. # Note: --with-sphinx-build provided by AX_WITH_PROG() below. AC_ARG_ENABLE([buggy-build], AS_HELP_STRING( [--enable-buggy-build], [omit -Werror; please see docs before use!]), [AS_CASE([$enableval], [yes], [use_werror=no], [no], [use_werror=yes], [*], [AC_MSG_ERROR([--enable-buggy-build: bad argument: $enableval])] )], [use_werror=yes]) AC_ARG_ENABLE([bundled-lark], AS_HELP_STRING([--disable-bundled-lark], [use system Lark (not recommended; see docs!)]), [], [enable_bundled_lark=yes]) AC_ARG_ENABLE([ch-image], AS_HELP_STRING([--disable-ch-image], [ch-image unprivileged builder & image manager]), [], [enable_ch_image=yes]) AC_ARG_ENABLE([html], AS_HELP_STRING([--disable-html], [HTML documentation]), [], [enable_html=yes]) AC_ARG_ENABLE([impolite-checks], AS_HELP_STRING([--disable-impolite-checks], [potentially troublesome informational checks]), [], [enable_impolite_checks=yes]) AC_ARG_ENABLE([man], AS_HELP_STRING([--disable-man], [man pages]), [], [enable_man=yes]) AC_ARG_ENABLE([syslog], AS_HELP_STRING([--disable-syslog], [logging to syslog]), [], [enable_syslog=yes]) AC_ARG_ENABLE([test], AS_HELP_STRING([--disable-test], [test suite]), [], [enable_test=yes]) AC_ARG_WITH([seccomp], AS_HELP_STRING([--with-seccomp=(yes|no)], [support for --seccomp])) AS_CASE([$with_seccomp], [yes], # explicit “yes” [want_seccomp=yes need_seccomp=yes msg_seccomp=yes], [no], # explicit “no” [want_seccomp=no need_seccomp=no msg_seccomp=no], [''], # option not specified [want_seccomp=yes need_seccomp=no msg_seccomp='if tested working'], [*], # anything else [AC_MSG_ERROR([invalid --with-seccomp arg: $with_seccomp])]) AC_ARG_WITH([libsquashfuse], AS_HELP_STRING([--with-libsquashfuse=@<:@yes|no|PATH@:>@], [whether to link with libsquashfuse])) AS_CASE([$with_libsquashfuse], [yes], # explicit “yes” [want_libsquashfuse=yes need_libsquashfuse=yes], [no], # explicit “no” [want_libsquashfuse=no need_libsquashfuse=no], [''], # option not specified [want_libsquashfuse=yes need_libsquashfuse=no], [*], # explicit path to libsquashfuse install [want_libsquashfuse=yes need_libsquashfuse=yes lib_libsquashfuse=$with_libsquashfuse/lib inc_libsquashfuse=$with_libsquashfuse/include]) AC_ARG_WITH([python], AS_HELP_STRING( [--with-python=SHEBANG], [Python shebang to use for scripts (default: "/usr/bin/env python3")]), [PYTHON_SHEBANG="$withval"], [PYTHON_SHEBANG='/usr/bin/env python3']) # Can’t deduce shebang from Gentoo “sphinx-python”; allow override. See #629. AC_ARG_WITH([sphinx-python], AS_HELP_STRING( [--with-sphinx-python=SHEBANG], [Python shebang used by Sphinx (default: deduced from sphinx-build executable]]), [sphinx_python="$withval"], [sphinx_python='']) ### Feature test macros ###################################################### # Macro to validate executable versions. Arguments: # # $1 name of variable containing executable name or absolute path # $2 minimum version # $3 append to $1 to make shell pipeline to get actual version only # (e.g., without program name) # # This macro is not able to determine if a program exists, only whether its # version is sufficient. ${!1} (i.e, the value of the variable whose name is # stored in $1) must be either empty, an absolute path to an executable, or # the name of a program in $PATH. A prior macro such as AX_WITH_PROG can be # used to ensure this condition. # # If ${!1} is an absolute path, and that file isn’t executable, error out. If # it’s something other than an absolute path, assume it’s the name of a # program in $PATH; if not, the behavior is undefined but not good (FIXME). # # Post-conditions: # # 1. If ${!1} is non-empty and the version reported by the program is # greater than or equal to the minimum, ${!1} is unchanged. If ${!1} is # empty or reported version is insufficient, ${!1} is the empty string. # This lets you test version sufficiency by whether ${!1} is empty. # # 2. $1_VERSION_NOTE contains a brief explanatory note. # AC_DEFUN([CH_CHECK_VERSION], [ AS_VAR_PUSHDEF([prog], [$1]) AS_IF([test -n "$prog"], [ # ${!1} is non-empty AS_CASE([$prog], # absolute path; check if executable [/*], [AC_MSG_CHECKING([if $prog is executable]) AS_IF([test -e "$prog"], [AC_MSG_RESULT([ok])], [AC_MSG_RESULT([no]) AC_MSG_ERROR([must be executable])])]) AC_MSG_CHECKING([if $prog version >= $2]) vact=$($prog $3) AX_COMPARE_VERSION([$2], [le], [$vact], [ AC_SUBST([$1_VERSION_NOTE], ["ok ($vact)"]) AC_MSG_RESULT([ok ($vact)]) ], [ AC_SUBST([$1_VERSION_NOTE], ["too old ($vact)"]) AC_MSG_RESULT([too old ($vact)]) AS_UNSET([$1]) ]) ], [ # ${!} is empty AC_SUBST([$1_VERSION_NOTE], ["not found"]) AS_UNSET([$1]) ]) AS_VAR_POPDEF([prog]) ]) ### C compiler ############################################################### # Need a C99 compiler. (See https://stackoverflow.com/a/28558338.) AC_PROG_CC # Set up CFLAGS. ch_cflags='-std=c99 -Wall' AS_IF([test -n "$lib_libsquashfuse"], [ch_cflags="$ch_cflags -I$inc_libsquashfuse -L$lib_libsquashfuse" # Without this, clang fails with “error: argument unused during # compilation” on the -L. GCC ignores it. ch_cflags="$ch_cflags -Wno-unused-command-line-argument"]) AS_IF([test $use_werror = yes], [ch_cflags="$ch_cflags -Werror"]) AX_CHECK_COMPILE_FLAG([$ch_cflags], [ CFLAGS="$CFLAGS $ch_cflags" ], [ AC_MSG_ERROR([no suitable C99 compiler found]) ]) AS_IF([test "$CC" = icc], [AC_MSG_ERROR([icc not supported (see PR @%:@481)])]) ### ch-run required ########################################################## # Only ch-run needs any kind of interesting library stuff; this variable holds # the library arguments we need. This also requires us to use AC_CHECK_LIB # instead of the (recommended by docs) AC_SEARCH_LIBS, because that adds # things to LIBS, which we don’t want because it’s applied to all executables. CH_RUN_LIBS= # asprintf(3) # # You can do this with AC_CHECK_FUNC or AC_CHECK_FUNCS, but those macros call # the function with no arguments. This causes a warning for asprintf() for # some compilers (and I have no clue why others accept it); see issue #798. # Instead, try to build a small test program that calls asprintf() correctly. AC_MSG_CHECKING([for asprintf in libc]) AC_COMPILE_IFELSE([AC_LANG_SOURCE([[ #define _GNU_SOURCE #include #include int main(void) { char *p; if (asprintf(&p, "WEIRD AL YANKOVIC\n") >= 0) free(p); return 0; } ]])], [AC_MSG_RESULT([yes])], [AC_MSG_RESULT([no]) AC_MSG_ERROR([asprintf(3) not found; please report this bug])]) # argp_parse(3), which is included with glibc but not other libc’s, e.g. musl. AC_MSG_CHECKING([for argp_parse in libc]) AC_LINK_IFELSE([AC_LANG_SOURCE([[ #include int main(void) { argp_parse(0, 1, NULL, 0, 0, 0); return 0; } ]])], [AC_MSG_RESULT([yes])], # built-in, no further action [AC_MSG_RESULT([no]) # try external libargp AC_CHECK_LIB( [argp], [argp_parse], [CH_RUN_LIBS="-largp $CH_RUN_LIBS"], [AC_MSG_ERROR([argp_parse(3) not found; please report this bug])])]) # pthreads; needed for “ch-run --join”. AX_PTHREAD # POSIX IPC lives in librt. AC_CHECK_LIB([rt], [shm_open], [CH_RUN_LIBS="-lrt $CH_RUN_LIBS"], [ AC_MSG_ERROR([shm_open(3) not found]) ]) # User namespaces AC_MSG_CHECKING([if in chroot]) # https://unix.stackexchange.com/a/14346 AS_IF([test "$(awk '$5=="/" {print $1}' int main(void) { if (unshare(CLONE_NEWNS|CLONE_NEWUSER)) return 1; // syscall failed else return 0; // syscall succeeded } ]])], [have_userns=yes], [have_userns=no], [AC_MSG_ERROR([cross-compilation not supported])]) AC_MSG_RESULT($have_userns) # overlayfs AC_DEFUN([CH_OVERLAY_C], [[ #define _GNU_SOURCE #include #include #include #include #include #include #include #include #define T_(x) if (!(x)) fatal_(__FILE__, __LINE__, errno, #x) #define Z_(x) if (x) fatal_(__FILE__, __LINE__, errno, #x) void fatal_(const char *file, int line, int errno_, const char *str) { fprintf(stderr, "error: %s: %d: %s\n", file, line, str); fprintf(stderr, "errno: %d: %s\n", errno_, strerror(errno_)); exit(1); } int main(void) { int fd; uid_t euid = geteuid(); gid_t egid = getegid(); // enter namespaces Z_ (unshare(CLONE_NEWNS|CLONE_NEWUSER)); // set up ID maps T_ (-1 != (fd = open("/proc/self/uid_map", O_WRONLY))); T_ (1 <= dprintf(fd, "%d %d 1\n", 0, euid)); Z_ (close(fd)); T_ (-1 != (fd = open("/proc/self/setgroups", O_WRONLY))); T_ (1 <= dprintf(fd, "deny\n")); Z_ (close(fd)); T_ (-1 != (fd = open("/proc/self/gid_map", O_WRONLY))); T_ (1 <= dprintf(fd, "%d %d 1\n", 0, egid)); Z_ (close(fd)); // set up overlayfs Z_ (mount("/", "/", NULL, MS_BIND | MS_REC | MS_PRIVATE, NULL)); Z_ (mount(NULL, "/mnt", "tmpfs", 0, NULL)); Z_ (mkdir("/mnt/upper", 0700)); Z_ (mkdir("/mnt/lower", 0700)); Z_ (mkdir("/mnt/lower/test", 0700)); Z_ (mkdir("/mnt/work", 0700)); Z_ (mkdir("/mnt/merged", 0700)); Z_ (mount(NULL, "/mnt/merged", "overlay", MS_NOATIME, "lowerdir=/mnt/lower," "upperdir=/mnt/upper," "workdir=/mnt/work," "index=on,userxattr,volatile")); // test if user xattrs are working #ifdef XATTRS Z_ (rmdir("/mnt/merged/test")); Z_ (mkdir("/mnt/merged/test", 0700)); #endif } ]]) AC_MSG_CHECKING([for unprivileged overlayfs]) have_overlayfs="check disabled" AS_IF([test $enable_impolite_checks = yes], [AC_RUN_IFELSE([AC_LANG_SOURCE(CH_OVERLAY_C)], [have_overlayfs=yes], [have_overlayfs=no], [AC_MSG_ERROR([cross-compilation not supported])])]) AC_MSG_RESULT($have_overlayfs) have_tmpfs_xattrs="check disabled" AC_MSG_CHECKING([for tmpfs user xattrs]) AS_IF([test $enable_impolite_checks = yes], [AC_RUN_IFELSE([#define XATTRS AC_LANG_SOURCE(CH_OVERLAY_C)], [have_tmpfs_xattrs=yes], [have_tmpfs_xattrs=no], [AC_MSG_ERROR([cross-compilation not supported])])]) AC_MSG_RESULT($have_tmpfs_xattrs) ### ch-run optional ########################################################## # FNM_EXTMATCH is a GNU extension to support extended globs in fnmatch(3). AC_CHECK_DECL(FNM_EXTMATCH, [have_fnm_extmatch=yes], [have_fnm_extmatch=no], [[#define _GNU_SOURCE #include ]]) # Should we build seccomp? AC_MSG_CHECKING([for seccomp filter support]) AC_RUN_IFELSE([AC_LANG_SOURCE([[ #define _GNU_SOURCE #include #include #include #include #include #include #define Z_(x) if (x) { perror(NULL); exit(1); } int main(void) { struct sock_filter f[] = { BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ALLOW) }; struct sock_fprog p = { 1, f }; Z_ (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0, 0)); Z_ (prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &p, 0, 0, 0)); return 0; } ]])], [AC_MSG_RESULT([yes]) test_seccomp=yes], [AC_MSG_RESULT([no]) test_seccomp=no], [AC_MSG_ERROR([cross-compilation not supported])]) AS_IF([test $test_seccomp = yes], [AS_IF([test $want_seccomp = yes], [have_seccomp=yes], # works and =yes or not given [have_seccomp=no])], # works and =no [AS_IF([test $need_seccomp = yes], [have_seccomp=yes], # doesn’t work and =yes (build anyway) [have_seccomp=no])]) # doesn’t work and =no or not given ### SquashFS ################################################################# # SquashFS Tools vmin_mksquashfs=4.2 # CentOS 7 AC_CHECK_PROG([MKSQUASHFS], [mksquashfs], [mksquashfs]) CH_CHECK_VERSION([MKSQUASHFS], [$vmin_mksquashfs], [-version | head -1 | cut -d' ' -f3]) # SquashFUSE executables vmin_squashfuse=0.1.100 # Ubuntu 16.04 (Xenial). CentOS 7 has 0.1.102. AC_CHECK_PROG([SQUASHFUSE], [squashfuse], [squashfuse]) CH_CHECK_VERSION([SQUASHFUSE], [$vmin_squashfuse], [--help 2>&1 | head -1 | cut -d' ' -f2]) # Check for libsquashfuse if it’s wanted. have_libfuse3=n/a have_libsquashfuse_ll=n/a have_ll_h=n/a AS_IF([test $want_libsquashfuse = yes], [ # libfuse3. As of version 0.5.0, SquashFUSE’s ll.h won’t build without an # appropriate -I [1]. Presently we use pkg-config to find it, but see #1844. # # We avoid PKG_CHECK_MODULES because it introduces a dependency on # pkg-config at autogen.sh time, with impressively incomprehensible error # messages if it’s not met [2]. The approach below also seems simpler [3]? # # [1]: https://github.com/vasi/squashfuse/commit/eca5764 # [2]: https://ae1020.github.io/undefined-macro-pkg-config/ # [3]: https://tirania.org/blog/archive/2012/Oct-20.html AC_CHECK_PROG(have_pkg_config, pkg-config, yes, no) AS_IF([test $have_pkg_config != yes], [AC_MSG_ERROR([need pkg-config to find libfuse3; try --with-libsquashfuse=no or see issue @%:@1844])]) AS_IF([pkg-config --exists fuse3], [ have_libfuse3=yes CFLAGS="$CFLAGS $(pkg-config --cflags fuse3)" # libsquashfuse? AC_CHECK_LIB([squashfuse_ll], [sqfs_ll_mount], [have_libsquashfuse_ll=yes], [have_libsquashfuse_ll=no]) # ll.h? AC_CHECK_HEADER([squashfuse/ll.h], [have_ll_h=yes], [have_ll_h=no], [#define SQFS_CONFIG_H #define FUSE_USE_VERSION 32 ]) # see comment in ch_fuse.c regarding these defines ], [have_libfuse3=no]) ]) # Should we link with libsquashfuse? AS_IF([ test $want_libsquashfuse = yes \ && test $have_libfuse3 = yes \ && test $have_libsquashfuse_ll = yes \ && test $have_ll_h = yes], [have_libsquashfuse=yes AS_IF([test -n "$lib_libsquashfuse"], [rpath_libsquashfuse=-Wl,-rpath=$lib_libsquashfuse], [rpath_libsquashfuse=]) CH_RUN_LIBS="-lsquashfuse_ll -lfuse3 $rpath_libsquashfuse $CH_RUN_LIBS"], [have_libsquashfuse=no]) AS_IF([ test $need_libsquashfuse = yes \ && test $have_libsquashfuse = no], [AC_MSG_ERROR([libsquashfuse requested but not found])]) # Any SquashFUSE support at all? AS_IF([ test -n "$SQUASHFUSE" \ || test $have_libsquashfuse = yes], [have_any_squashfuse=yes], [have_any_squashfuse=no]) ### ch-image ################################################################# # Python vmin_python=3.6 # NOTE: Keep in sync with lib/charliecloud.py AC_MSG_CHECKING([if "$PYTHON_SHEBANG" starts with slash]) AS_CASE([$PYTHON_SHEBANG], [/*], [AC_MSG_RESULT([ok])], [*], [AC_MSG_RESULT([no]) AC_MSG_ERROR([--with-python: must start with slash])]) python="${PYTHON_SHEBANG#/usr/bin/env }" # use shell to find it AS_CASE([$python], [/*], [PYTHON="$python"], # absolute [*], [AC_CHECK_PROG([PYTHON], [$python], [$python])] # verify it's in $PATH ) CH_CHECK_VERSION([PYTHON], [$vmin_python], [--version | head -1 | cut -d' ' -f2]) # Python module “requests” vmin_requests=2.6.0 # CentOS 7; FIXME: haven’t actually tested this AS_IF([test -n "$PYTHON"], [ AC_MSG_CHECKING([for requests module]) cat <&1 | cut -d' ' -f5]) # Git vmin_git=2.28.1 AC_CHECK_PROG([GIT], [git], [git]) CH_CHECK_VERSION([GIT], [$vmin_git], [--version | cut -d' ' -f3]) # git2dot vmin_git2dot=0.8.3 AC_CHECK_PROG([GIT2DOT], [git2dot.py], [git2dot.py]) CH_CHECK_VERSION([GIT2DOT], [$vmin_git2dot], [--version | cut -d' ' -f3]) # rsync vmin_rsync=3.1.0 # NOTE: keep in sync with lib/charliecloud.py AC_CHECK_PROG([RSYNC], [rsync], [rsync]) CH_CHECK_VERSION([RSYNC], [$vmin_rsync], [-V | sed -En 's/rsync +version (@<:@0-9@:>@+\.@<:@0-9@:>@+\.@<:@0-9@:>@+).*$/\1/p']) ### Docs ##################################################################### # Sphinx vmin_sphinx=1.2.3 AX_WITH_PROG([SPHINX], [sphinx-build]) CH_CHECK_VERSION([SPHINX], [$vmin_sphinx], [--version | sed -E 's/sphinx-build //']) # Get the Sphinx Python. We don’t care about version. AS_IF([test -n "$SPHINX"], [ AS_IF([test -z "$sphinx_python"], [ AC_MSG_CHECKING([for sphinx-build Python]) sphinx_python=$(head -1 "$SPHINX" | sed -E -e 's/^#!\s*//' -e 's/\s.*$//') AC_MSG_RESULT([$sphinx_python]) AC_MSG_CHECKING([if "$sphinx_python" starts with slash]) AS_CASE([$sphinx_python], [/*], [AC_MSG_RESULT([ok])], [*], [AC_MSG_RESULT([no]) AC_MSG_ERROR([--with-sphinx-python: must start with slash])]) ])]) # “docutils” module vmin_docutils=0.14 AS_IF([test -n "$SPHINX"], [ # Sphinx depends on docutils, so we don’t need to check if the module exists # before checking its version. (CH_CHECK_VERSION isn’t smart enough to deal # with Python being present but a module not.) DOCUTILS=$sphinx_python # FIXME: output is confusing CH_CHECK_VERSION([DOCUTILS], [$vmin_docutils], [-c 'import docutils; print(docutils.__version__)']) ], [DOCUTILS_VERSION_NOTE='moot b/c no sphinx-build']) # “sphinx-rtd-theme” module vmin_rtd=0.2.4 AS_IF([test -n "$SPHINX"], [ AC_MSG_CHECKING([for sphinx_rtd_theme module]) cat <&1) #AS_IF([ test -z "$sudo_out" \ # || echo "$sudo_out" | grep -Fq asswor], # [have_sudo=yes], # [have_sudo=no]) #AC_MSG_RESULT($have_sudo) # Wget vmin_wget=1.11 # 2008 AC_CHECK_PROG([WGET], [wget], [wget]) CH_CHECK_VERSION([WGET], [$vmin_wget], [--version | head -1 | cut -d' ' -f3]) ### Output variables ######################################################### # Autotools output variables are ... interesting. This is my best # understanding: # # 1. AC_SUBST(foo) does two things in Makefile.am: # # a. Replace the string "@foo@" with the value of foo anywhere it # appears. # # b. Set the Make variable foo to the same value, i.e., add “foo = @foo@” # which is then substituted as in item 1. # # So this is how you transfer a variable from configure to Make. # # 2. AC_SUBST_NOTMAKE(foo) does only 1a. # # 3. AM_CONDITIONAL(foo, test) creates a variable for use in Automake # conditionals. E.g. if you say in configure.ac: # # AM_CONDITIONAL([foo], [test $foo = yes]) # # and then in Makefile.am: # # if foo # ... bar ... # else # ... baz ... # endif # # then if the configure variable $foo is “yes”, lines “... bar ...” will # be placed in the Makefile; otherwise, “... baz ...” will be included. # # This is how you include and exclude portions of the Makefile.am from # the output Makefile. It *does not* create a Make variable. # # 4. AC_DEFINE(foo, value, comment) #define’s the preprocessor symbol foo to # value in config.h. (Supposedly value and comment are optional but I got # warnings doing that.) So this is how you make configure values # available in C code (as macros, not variables). Typically you would # define something or not (allowing #ifdef), rather than always define to # true or false (which would require #if). # # 5. AC_DEFINE_UNQUOTES adds some extra transformations to the above. I # didn’t quite follow. # # Below are all the variables we want available outside configure. AM_CONDITIONAL([ENABLE_CH_IMAGE], [test $enable_ch_image = yes]) AM_CONDITIONAL([ENABLE_HTML], [test $enable_html = yes]) AM_CONDITIONAL([ENABLE_LARK], [test $enable_bundled_lark = yes]) AM_CONDITIONAL([ENABLE_MAN], [test $enable_man = yes]) AS_IF([test $enable_syslog = yes], [AC_DEFINE([ENABLE_SYSLOG], [1], [log to syslog])]) AM_CONDITIONAL([ENABLE_TEST], [test $enable_test = yes]) AC_SUBST([CH_RUN_LIBS]) AC_SUBST([PYTHON_SHEBANG]) AC_SUBST([SPHINX]) AS_IF([test $have_fnm_extmatch = yes], [AC_DEFINE([HAVE_FNM_EXTMATCH], [1], [extended globs supported])]) AS_IF([test $have_seccomp = yes], [AC_DEFINE([HAVE_SECCOMP], [1], [seccomp supported])]) AM_CONDITIONAL([HAVE_LIBSQUASHFUSE], [test $have_libsquashfuse = yes]) AS_IF([test $have_libsquashfuse = yes], [AC_DEFINE([HAVE_LIBSQUASHFUSE], [1], [link with libsquashfuse])]) ### Prepare report ########################################################### # FIXME: Should replace all these with macros? # ch-run (needed below) AS_IF([ test $have_userns = yes], [have_ch_run=yes], [have_ch_run=no]) # image builders AS_IF([ test $enable_ch_image = yes \ && test -n "$PYTHON" \ && test -n "$PYTHON_SHEBANG" \ && test -n "$REQUESTS" \ && test $have_ch_run = yes], [have_ch_image=yes], [have_ch_image=no]) AS_IF([ test $have_ch_image = yes \ && test -n "$GIT"], [have_ch_image_bu=yes], [have_ch_image_bu=no]) AS_IF([ test $have_ch_image = yes \ && test -n "$RSYNC"], [have_ch_image_rsync=yes], [have_ch_image_rsync=no]) # managing container images AS_IF([ test $have_ch_image = yes], [have_any_builder=yes], [have_any_builder=no]) AS_IF([ test $have_ch_image = yes], [have_dockerfile_build=yes], [have_dockerfile_build=no]) AS_IF([ test $have_ch_image = yes], [have_builder_to_tar=yes], [have_builder_to_tar=no]) AS_IF([ test $have_ch_image = yes \ && test -n "$MKSQUASHFS"], [have_pack_squash=yes], [have_pack_squash=no]) # running containers AS_IF([ test -n "$NVIDIA_CLI" \ && test $have_nvidia_libs = yes], [have_nvidia=yes], [have_nvidia=no]) # test suite AS_IF([ test $enable_test = yes \ && test $have_ch_run = yes \ && test $have_ch_image = yes \ && test -n "$_BASH" \ && test -n "$BATS" \ && test -n "$WGET"], # assume access to Docker Hub or mirror [have_tests_basic=yes], [have_tests_basic=no]) AS_IF([ test $have_tests_basic = yes \ && test $have_docs = yes \ && test -n "$SHELLCHECK" ], # assume we do have generic sudo [have_tests_more=yes], [have_tests_more=no]) AS_IF([ test $have_tests_more = yes \ && test -n "$DOT" \ && test -n "$GIT2DOT" ], [have_tests_debug=yes], [have_tests_debug=no]) AS_IF([ test $have_tests_basic = yes \ && test $have_tests_more = yes], [have_tests_tar=yes], [have_tests_tar=no]) AS_IF([ test $have_tests_basic = yes \ && test $have_tests_more = yes \ && test $have_pack_squash = yes ], [have_tests_squashunpack=yes], [have_tests_squashunpack=no]) AS_IF([ test $have_tests_squashunpack = yes \ && test $have_libsquashfuse = yes], [have_tests_squashmount=yes], [have_tests_squashmount=no]) ### Write output files ####################################################### AC_OUTPUT ## Print report AS_IF([ test $have_userns = no \ && test $chrooted = yes], [ chroot_warning=$(cat <<'EOF' Warning: configure is running in a chroot, but user namespaces cannot be created in a chroot; see the man page unshare(2). Therefore, the above may be a false negative. However, note that like all the run-time configure tests, this is informational only and does not affect the build. EOF ) ]) AC_MSG_NOTICE([ Dependencies report =================== Below is a summary of configure's findings. Caveats ~~~~~~~ Charliecloud's run-time dependencies are lazy; features just try to use their dependencies and error if there's a problem. This report summarizes what configure found on *this system*, because that's often useful, but none of the run-time findings change what is built and installed. Listed versions are minimums. These are a bit fuzzy. Try it even if configure thinks a version is too old, and please report back to us. Building Charliecloud ~~~~~~~~~~~~~~~~~~~~~ will build and install: ch-image(1) ... ${enable_ch_image} HTML documentation ... ${enable_html} man pages ... ${enable_man} syslog ... ${enable_syslog} test suite ... ${enable_test} required: C99 compiler ... ${CC} ${CC_VERSION} optional: extended glob patterns in --unset-env ... ${have_fnm_extmatch} ch-run(1) internal SquashFS mounting: ${have_libsquashfuse} enabled ... ${want_libsquashfuse} libfuse3 ... ${have_libfuse3} ${fuse3_CFLAGS:-} libsquashfuse_ll ... ${have_libsquashfuse_ll} ll.h header ... ${have_ll_h} documentation: ${have_docs} sphinx-build(1) ≥ $vmin_sphinx ... ${SPHINX_VERSION_NOTE} sphinx-build(1) Python ... ${sphinx_python:-n/a} "docutils" module ≥ $vmin_docutils ... ${DOCUTILS_VERSION_NOTE} "sphinx-rtd-theme" module ≥ $vmin_rtd ... ${RTD_VERSION_NOTE} Building images ~~~~~~~~~~~~~~~ with ch-image(1): ${have_ch_image} enabled ... ${enable_ch_image} Python shebang line ... ${PYTHON_SHEBANG:-none} Python in shebang ≥ $vmin_python ... ${PYTHON_VERSION_NOTE} "lark" module ... ${lark_status} "requests" module ≥ $vmin_requests ... ${REQUESTS_VERSION_NOTE} ch-run(1) ... ${have_ch_run} with ch-image(1) using build cache: ${have_ch_image_bu} ch-image(1): ... ${have_ch_image} Git ≥ $vmin_git ... ${GIT_VERSION_NOTE} with ch-image(1) using RSYNC instruction: ${have_ch_image_rsync} ch-image(1): ... ${have_ch_image} rsync ≥ $vmin_rsync ... ${RSYNC_VERSION_NOTE} Managing container images ~~~~~~~~~~~~~~~~~~~~~~~~~ build from Dockerfile: ${have_dockerfile_build} ch-image(1) builder ... ${have_ch_image} access to an image repository ... assumed yes pack images from builder storage to tarball: ${have_builder_to_tar} ch-image(1) builder ... ${have_ch_image} pack images from builder storage to SquashFS: ${have_pack_squash} ch-image(1) builder ... ${have_ch_image} mksquashfs(1) ≥ $vmin_mksquashfs ... ${MKSQUASHFS_VERSION_NOTE} Note: Pulling/pushing images from/to a repository is currently done using the builder directly. Running containers ~~~~~~~~~~~~~~~~~~ ch-run(1): ${have_ch_run} user+mount namespaces ... ${have_userns}$chroot_warning run SquashFS images: ${have_any_squashfuse} manual mount with SquashFUSE ≥ $vmin_squashfuse ... ${SQUASHFUSE_VERSION_NOTE} internal mount with libsquashfuse ... ${have_libsquashfuse} fake system calls with seccomp(2): ${have_seccomp} enabled ... ${msg_seccomp} tested working ... ${test_seccomp} writeable overlay (--write-fake): ${have_overlayfs} fully functional ... ${have_tmpfs_xattrs} inject nVidia GPU libraries: ${have_nvidia} nvidia-container-cli(1) ≥ $vmin_nvidia_cli ... ${NVIDIA_CLI_VERSION_NOTE} nVidia libraries & executables present ... ${have_nvidia_libs} Test suite ~~~~~~~~~~ basic tests, all stages: ${have_tests_basic} test suite enabled ... ${enable_test} ch-run(1) ... ${have_ch_run} any builder above ... ${have_ch_image} access to Docker Hub or mirror ... assumed yes Bats ≥ $vmin_bats ... ${BATS_VERSION_NOTE} Bash ≥ $vmin_bash ... ${_BASH_VERSION_NOTE} wget(1) ≥ $vmin_wget ... ${WGET_VERSION_NOTE} more complete tests: ${have_tests_more} basic tests ... ${have_tests_basic} documentation built ... ${have_docs} ShellCheck ≥ $vmin_shellcheck ... ${SHELLCHECK_VERSION_NOTE} generic sudo ... assumed yes debugging tests: ${have_tests_debug} more tests ... ${have_tests_more} DOT ≥ $vmin_dot ... ${DOT_VERSION_NOTE} git2dot ≥ $vmin_git2dot ... ${GIT2DOT_VERSION_NOTE} recommended tests, tar-unpack mode: ${have_tests_tar} basic tests ... ${have_tests_basic} more tests ... ${have_tests_more} recommended tests, squash-unpack mode: ${have_tests_squashunpack} basic tests ... ${have_tests_basic} more tests ... ${have_tests_more} pack/unpack SquashFS images ... ${have_pack_squash} recommended tests, squash-mount mode: ${have_tests_squashmount} recommended, squash-unpack mode: ${have_tests_squashunpack} internal SquashFS mounting ... ${have_libsquashfuse} ]) charliecloud-0.37/doc/000077500000000000000000000000001457016721300146545ustar00rootroot00000000000000charliecloud-0.37/doc/Makefile.am000066400000000000000000000116701457016721300167150ustar00rootroot00000000000000# This Makefile started with the default Makefile produced by the Sphinx # initialization process, which we then modified over time. During the # Automake-ification, I stripped out most of the boilderplate and left only # the targets that we use. # We turn off parallel build in doc: # # 1. Sphinx handles building the whole documentation internally already, as # a unit, so we shouldn't call sphinx-build more than once for different # output files at all, let alone in parallel. # # 2. Serial build is plenty fast. # # 3. There is a race condition in Sphinx < 1.6.6 that's triggered when two # instances (e.g., for html and man targets) try to "mkdir doctrees" # simultaneously. See issue #115. # # This special target was introduced in GNU Make 3.79, in April 2000. .NOTPARALLEL: EXTRA_DIST = \ _loc.rst \ best_practices.rst \ bugs.rst \ charliecloud.rst \ ch-checkns.rst \ ch-completion.bash.rst \ ch-convert.rst \ ch-fromhost.rst \ ch-image.rst \ ch-run.rst \ ch-run-oci.rst \ ch-test.rst \ conf.py \ dev.rst \ faq.rst \ favicon.ico \ index.rst \ install.rst \ logo-sidebar.png \ make-deps-overview \ man/README \ py_env.rst \ rd100-winner.png \ see_also.rst \ tutorial.rst if ENABLE_MAN man_MANS = \ man/charliecloud.7 \ man/ch-checkns.1 \ man/ch-completion.bash.7 \ man/ch-convert.1 \ man/ch-fromhost.1 \ man/ch-image.1 \ man/ch-run.1 \ man/ch-run-oci.1 \ man/ch-test.1 endif if ENABLE_HTML nobase_html_DATA = \ html/searchindex.js \ html/_images/rd100-winner.png \ html/best_practices.html \ html/ch-checkns.html \ html/ch-completion.bash.html \ html/ch-convert.html \ html/ch-fromhost.html \ html/ch-image.html \ html/ch-run.html \ html/ch-run-oci.html \ html/ch-test.html \ html/command-usage.html \ html/dev.html \ html/faq.html \ html/index.html \ html/install.html \ html/search.html \ html/tutorial.html endif # NOTE: ./html might be a Git checkout to support "make web", so make sure not # to delete it. CLEANFILES = $(man_MANS) $(nobase_html_DATA) \ _deps.rst html/.buildinfo html/.nojekyll if ENABLE_HTML # Automake can't remove directories. clean-local: rm -Rf doctrees html/_sources html/_static html/_images endif # Automake can't install and uninstall directories. _static contains around # one hundred files in several directories, and I'm pretty sure the contents # change depending on Sphinx version and other details, so we can't just list # the files. These targets deal with it as an opaque directory. The _sources # directory is another generated directory that contains references to the # input .rst files which we need for searching to work so we give it a similar # treatment. if ENABLE_HTML install-data-hook: cp -r html/_sources $(DESTDIR)$(htmldir)/html cp -r html/_static $(DESTDIR)$(htmldir)/html find $(DESTDIR)$(htmldir)/html/_sources -xtype f -exec chmod 644 {} \; find $(DESTDIR)$(htmldir)/html/_static -xtype d -exec chmod 755 {} \; find $(DESTDIR)$(htmldir)/html/_static -xtype f -exec chmod 644 {} \; uninstall-hook: test -d $(DESTDIR)$(htmldir)/html/_sources \ && rm -Rf $(DESTDIR)$(htmldir)/html/_sources \; test -d $(DESTDIR)$(htmldir)/html/_static \ && rm -Rf $(DESTDIR)$(htmldir)/html/_static \; test -d $(DESTDIR)$(htmldir)/html/_images \ && rm -Rf $(DESTDIR)$(htmldir)/html/_images \; endif # You can set these variables from the command line. SPHINXOPTS = -W SPHINXBUILD = @SPHINX@ PAPER = BUILDDIR = . # Internal variables. ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(SPHINXOPTS) . _deps.rst: ../config.log make-deps-overview cat $< | ./make-deps-overview > $@ # Since we're not doing anything in parallel anyway, just put the HTML and the # man pages in the same target, with conditionals. Gotchas: # # 1. If we build both, the HTML needs to go first otherwise it doesn't get # curly quotes. ¯\_(ツ)_/¯ # # 2. This not a "grouped target" but rather an "independent target" [1], # because the former came in GNU Make 4.3 which is quite new. However it # does seem to get run only once. # # [1]: https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html $(nobase_html_DATA) $(man_MANS): ../lib/version.txt ../README.rst _deps.rst $(EXTRA_DIST) if ENABLE_HTML # Create dummy file in case the redirect is disabled for EPEL. mkdir -p html touch html/command-usage.html # Build. $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html # Avoid GitHub messing things up with Jekyll. touch html/.nojekyll # Some output files are copies with same timestamp as source; fix. Note # we need all the HTML output files, not just the one picked in $@. touch --no-create $(nobase_html_DATA) # remove unused files that Sphinx made rm -f $(BUILDDIR)/html/_deps.html \ $(BUILDDIR)/html/charliecloud.html \ $(BUILDDIR)/html/bugs.html \ $(BUILDDIR)/html/_loc.html \ $(BUILDDIR)/html/objects.inv \ $(BUILDDIR)/html/see_also.html endif if ENABLE_MAN $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man endif charliecloud-0.37/doc/_loc.rst000066400000000000000000000013431457016721300163230ustar00rootroot00000000000000.. Do not edit this file — it’s auto-generated. We pride ourselves on keeping Charliecloud lightweight and simple. The lines of code as of version 0.37 is: .. list-table:: * - Program itself - 9079 * - Test suite & examples - 12019 * - Documentation - 6416 * - Build system - 1294 * - Packaging - 629 * - Miscellaneous - 506 * - Total - 29943 These include code only, excluding blank lines and comments. They were counted using `cloc `_ version 1.96. We typically quote the "Program itself" number when describing the size of Charliecloud. (Please do not quote the size in Priedhorsky and Randles 2017, as that number is very out of date.) charliecloud-0.37/doc/best_practices.rst000066400000000000000000000317261457016721300204110ustar00rootroot00000000000000Best practices ************** Other best practices information ================================ This isn’t the last word. Also consider: * Many of Docker’s `Best practices for writing Dockerfiles `_ apply to Charliecloud images as well. * “`Recommendations for the packaging and containerizing of bioinformatics software `_”, Gruening et al. 2019, is a thoughtful editorial with eleven specific containerization recommendations for scientific software. * “`Application container security guide `_”, NIST Special Publication 800-190; Souppaya, Morello, and Scarfone 2017. Filesystems =========== There are two performance gotchas to be aware of for Charliecloud. Metadata traffic ---------------- Directory-format container images and the Charliecloud storage directory often contain, and thus Charliecloud must manipulate, a very large number of files. For example, after running the test suite, the storage directory contains almost 140,000 files. That is, metadata traffic can be quite high. Such images and the storage directory should be stored on a filesystem with reasonable metadata performance. Notably, this *excludes* Lustre, which is commonly used for scratch filesystems in HPC; i.e., don’t store these things on Lustre. NFS is usually fine, though in general it performs worse than a local filesystem. In contrast, SquashFS images, which encapsulate the image into a single file that is mounted using FUSE at runtime, insulate the filesystem from this metadata traffic. Images in this format are suitable for any filesystem, including Lustre. .. _best-practices_file-copy: File copy performance --------------------- :code:`ch-image` does a lot of file copying. The bulk of this is manipulating images in the storage directory. Importantly, this includes :ref:`large files ` stored by the build cache outside its Git repository, though this feature is disabled by default. Copies are costly both in time (to read, transfer, and write the duplicate bytes) and space (to store the bytes). However significant optimizations are sometimes available. Charliecloud’s internal file copies (unfortunately not sub-programs like Git) can take advantage of multiple optimized file-copy paths offered by Linux: in-kernel copy Copy data inside the kernel without passing through user-space. Saves time but not space. server-side copy Copy data on the server without sending it over the network, relevant only for network filesystems. Saves time but not space. reflink copy (best) Copy-on-write via “`reflink `_”. The destination file gets a new inode but shares the data extents of the source file — i.e., no data are copied! — with extents unshared later if/when are written. Saves both time and space (and potentially quite a lot). To use these optimizations, you need: 1. Python ≥3.8, for :code:`os.copy_file_range()` (`docs `_), which wraps :code:`copy_file_range(2)` (`man page `_), which selects the best method from the three above. 2. A new-ish Linux kernel (details vary). 3. The right filesystem. .. |yes| replace:: ✅ .. |no| replace:: ❌ The following table summarizes our (possibly incorrect) understanding of filesystem support as of October 2023. For current or historical information, see the `Linux source code `_ for in-kernel filesystems or specific filesystem release nodes, e.g. `ZFS `_. A checkmark |yes| indicates supported, |no| unsupported. We recommend using a filesystem that supports reflink and also (if applicable) server-side copy. +----------------------------+---------------+---------------+----------------+ | | in-kernel | server-side | reflink (best) | +============================+===============+===============+================+ | *local filesystems* | +----------------------------+---------------+---------------+----------------+ | BTRFS | |yes| | n/a | |yes| | +----------------------------+---------------+---------------+----------------+ | OCFS2 | |yes| | n/a | |yes| | +----------------------------+---------------+---------------+----------------+ | XFS | |yes| | n/a | |yes| | +----------------------------+---------------+---------------+----------------+ | ZFS | |yes| | n/a | |yes| [1] | +----------------------------+---------------+---------------+----------------+ | *network filesystems* | +----------------------------+---------------+---------------+----------------+ | CIFS/SMB | |yes| | |yes| | ? | +----------------------------+---------------+---------------+----------------+ | NFSv3 | |yes| | |no| | |no| | +----------------------------+---------------+---------------+----------------+ | NFSv4 | |yes| | |yes| | |yes| [2] | +----------------------------+---------------+---------------+----------------+ | *other situations* | +----------------------------+---------------+---------------+----------------+ | filesystems not listed | |yes| | |no| | |no| | +----------------------------+---------------+---------------+----------------+ | copies between filesystems | |no| [3] | |no| | |no| | +----------------------------+---------------+---------------+----------------+ Notes: 1. As of `ZFS 2.2.0 `_. 2. If the underlying exported filesystem also supports reflink. 3. Recent kernels (≥5.18 as well as stable kernels if backported) support in-kernel file copy between filesystems, but for many kernels it is `not stable `_, so Charliecloud does not currently attempt it. Installing your own software ============================ This section covers four situations for making software available inside a Charliecloud container: 1. Third-party software installed into the image using a package manager. 2. Third-party software compiled from source into the image. 3. Your software installed into the image. 4. Your software stored on the host but compiled in the container. .. note:: Maybe you don’t have to install the software at all. Is there already a trustworthy image on Docker Hub you can use as a base? Third-party software via package manager ---------------------------------------- This approach is the simplest and fastest way to install stuff in your image. The :code:`examples/hello` Dockerfile does this to install the package :code:`openssh-client`: .. literalinclude:: ../examples/hello/Dockerfile :language: docker :lines: 3-7 You can use distribution package managers such as :code:`dnf`, as demonstrated above, or others, such as :code:`pip` for Python packages. Be aware that the software will be downloaded anew each time you execute the instruction (unless you add an HTTP cache, which is out of scope of this documentation). .. note:: RPM and friends (:code:`yum`, :code:`dnf`, etc.) have traditionally been rather troublesome in containers, and we suspect there are bugs we haven’t ironed out yet. If you encounter problems, please do file a bug! Third-party software compiled from source ----------------------------------------- Under this method, one uses :code:`RUN` commands to fetch the desired software using :code:`curl` or :code:`wget`, compile it, and install. Our example (:code:`examples/Dockerfile.almalinux_8ch`) does this with ImageMagick: .. literalinclude:: ../examples/Dockerfile.almalinux_8ch :language: docker :lines: 2- So what is going on here? #. Use the latest AlmaLinux 8 as the base image. #. Install some packages using :code:`dnf`, the OS package manager, including a basic development environment. #. Install :code:`wheel` using :code:`pip` and adjust the shared library configuration. (These are not needed for ImageMagick but rather support derived images.) #. For ImageMagick itself: #. Download and untar. Note the use of the variable :code:`MAGICK_VERSION` and versions easier. #. Build and install. Note the :code:`getconf` trick to guess at an appropriate parallel build. #. Clean up, in order to reduce the size of the build cache as well as the resulting Charliecloud image (:code:`rm -Rf`). .. note:: Because it’s a container image, you can be less tidy than you might normally be. For example, we install ImageMagick directly into :code:`/usr/local` rather than using something like `GNU Stow `_ to organize this directory tree. Your software stored in the image --------------------------------- This method covers software provided by you that is included in the image. This is recommended when your software is relatively stable or is not easily available to users of your image, for example a library rather than simulation code under active development. The general approach is the same as installing third-party software from source, but you use the :code:`COPY` instruction to transfer files from the host filesystem (rather than the network via HTTP) to the image. For example, :code:`examples/mpihello/Dockerfile.openmpi` uses this approach: .. literalinclude:: ../examples/mpihello/Dockerfile.openmpi :language: docker These Dockerfile instructions: 1. Copy the host directory :code:`examples/mpihello` to the image at path :code:`/hello`. The host path is relative to the *context directory*, which is tarred up and sent to the Docker daemon. Docker builds have no access to the host filesystem outside the context directory. (Unlike HPC, Docker comes from a world without network filesystems. This tar-based approach lets the Docker daemon run on a different node from the client without needing any shared filesystems.) The usual convention, including for Charliecloud tests and examples, is that the context is the directory containing the Dockerfile in question. A common pattern, used here, is to copy in the entire context. 2. :code:`cd` to :code:`/hello`. 3. Compile our example. We include :code:`make clean` to remove any leftover build files, since they would be inappropriate inside the container. Once the image is built, we can see the results. (Install the image into :code:`/var/tmp` as outlined in the tutorial, if you haven’t already.) :: $ ch-run /var/tmp/mpihello-openmpi.sqfs -- ls -lh /hello total 32K -rw-rw---- 1 charlie charlie 908 Oct 4 15:52 Dockerfile -rw-rw---- 1 charlie charlie 157 Aug 5 22:37 Makefile -rw-rw---- 1 charlie charlie 1.2K Aug 5 22:37 README -rwxr-x--- 1 charlie charlie 9.5K Oct 4 15:58 hello -rw-rw---- 1 charlie charlie 1.4K Aug 5 22:37 hello.c -rwxrwx--- 1 charlie charlie 441 Aug 5 22:37 test.sh Your software stored on the host -------------------------------- This method leaves your software on the host but compiles it in the image. This is recommended when your software is volatile or each image user needs a different version, for example a simulation code under active development. The general approach is to bind-mount the appropriate directory and then run the build inside the container. We can re-use the :code:`mpihello` image to demonstrate this. :: $ cd examples/mpihello $ ls -l total 20 -rw-rw---- 1 charlie charlie 908 Oct 4 09:52 Dockerfile -rw-rw---- 1 charlie charlie 1431 Aug 5 16:37 hello.c -rw-rw---- 1 charlie charlie 157 Aug 5 16:37 Makefile -rw-rw---- 1 charlie charlie 1172 Aug 5 16:37 README $ ch-run -b .:/mnt/0 --cd /mnt/0 /var/tmp/mpihello.sqfs -- \ make mpicc -std=gnu11 -Wall hello.c -o hello $ ls -l total 32 -rw-rw---- 1 charlie charlie 908 Oct 4 09:52 Dockerfile -rwxrwx--- 1 charlie charlie 9632 Oct 4 10:43 hello -rw-rw---- 1 charlie charlie 1431 Aug 5 16:37 hello.c -rw-rw---- 1 charlie charlie 157 Aug 5 16:37 Makefile -rw-rw---- 1 charlie charlie 1172 Aug 5 16:37 README A common use case is to leave a container shell open in one terminal for building, and then run using a separate container invoked from a different terminal. .. LocalWords: userguide Gruening Souppaya Morello Scarfone openmpi nist .. LocalWords: ident OCFS MAGICK charliecloud-0.37/doc/bugs.rst000066400000000000000000000003631457016721300163500ustar00rootroot00000000000000.. only:: man Reporting bugs ============== If Charliecloud was obtained from your Linux distribution, use your distribution’s bug reporting procedures. Otherwise, report bugs to: https://github.com/hpc/charliecloud/issues charliecloud-0.37/doc/ch-checkns.rst000066400000000000000000000004251457016721300174150ustar00rootroot00000000000000:code:`ch-checkns` ++++++++++++++++++ .. only:: not man Check :code:`ch-run` prerequisites, e.g., namespaces and :code:`pivot_root(2)`. Synopsis ======== :: $ ch-checkns Example ======= :: $ ch-checkns ok .. include:: ./bugs.rst .. include:: ./see_also.rst charliecloud-0.37/doc/ch-completion.bash.rst000066400000000000000000000043131457016721300210640ustar00rootroot00000000000000.. _ch-completion.bash: :code:`ch-completion.bash` ++++++++++++++++++++++++++ .. only:: not man Tab completion for the Charliecloud command line. Synopsis ======== :: $ source ch-completion.bash Description =========== :code:`ch-completion.bash` provides tab completion for the charliecloud command line. Currently, tab completion is available for Bash users for the executables :code:`ch-image`, :code:`ch-run`, and :code:`ch-convert`. We do not currently install the file if Charliecloud is built from source (see `issue #1842 `_). In this case, source it from the Charliecloud source code:: $ source $CHARLIECLOUD_SOURCE_PATH/bin/ch-completion.bash If you installed with a distribution package, the procedure is probably nicer. See your distro’s docs if you installed a package.) Disable completion with the utility function :code:`ch-completion` added to your environment when the above is sourced:: $ ch-completion --disable Dependencies ============ Tab completion has these additional dependencies: * Bash ≥ 4.3.0 * :code:`bash-completion` library (`GitHub `_, or it probably comes with your distribution, `e.g. `_) .. _ch-completion_func: :code:`ch-completion` ===================== Utility function for :code:`ch-completion.bash`. Synopsis -------- :: $ ch-completion [ OPTIONS ] Description ----------- :code:`ch-completion` is a function to manage Charliecloud’s tab completion. It is added to the environment when completion is sourced. The option(s) given specify what to do: :code:`--disable` Disable tab completion for all Charliecloud executables. :code:`--help` Print help message. :code:`--version` Print version of tab completion that’s currently enabled. :code:`--version-ok` Verify that tab completion version is consistent with that of :code:`ch-image`. Debugging ========= Tab completion can write debugging logs to :code:`/tmp/ch-completion.log`. Enable this by setting the environment variable :code:`CH_COMPLETION_DEBUG`. (This is primarily intended for developers.) .. LocalWords: func charliecloud-0.37/doc/ch-convert.rst000066400000000000000000000152571457016721300174700ustar00rootroot00000000000000:code:`ch-convert` ++++++++++++++++++ .. only:: not man Convert an image from one format to another. Synopsis ======== :: $ ch-convert [-i FMT] [-o FMT] [OPTION ...] IN OUT Description =========== Copy image :code:`IN` to :code:`OUT` and convert its format. Replace :code:`OUT` if it already exists, unless :code:`--no-clobber` is specified. It is an error if :code:`IN` and :code:`OUT` have the same format; use the format’s own tools for that case. :code:`ch-run` can run container images that are plain directories or (optionally) SquashFS archives. However, images can take on a variety of other formats as well. The main purpose of this tool is to make images in those other formats available to :code:`ch-run`. For best performance, :code:`ch-convert` should be invoked only once, producing the final format actually needed. :code:`IN` Descriptor for the input image. For image builders, this is an image reference; otherwise, it’s a filesystem path. :code:`OUT` Descriptor for the output image. :code:`-h`, :code:`--help` Print help and exit. :code:`-i`, :code:`--in-fmt FMT` Input image format is :code:`FMT`. If omitted, inferred as described below. :code:`-n`, :code:`--dry-run` Don’t read the input or write the output. Useful for testing format inference. :code:`--no-clobber` Error if :code:`OUT` already exists, rather than replacing it. :code:`--no-xattrs` Ignore xattrs and ACLs when converting. Overrides :code:`$CH_XATTRS`. :code:`-o`, :code:`--out-fmt FMT` Output image format is :code:`FMT`; inferred if omitted. :code:`-q`, :code:`--quiet` Be quieter; can be repeated. Incompatible with :code:`-v`. See the :ref:`FAQ entry on verbosity ` for details. :code:`-s`, :code:`--storage DIR` Set the storage directory. Equivalent to the same option for :code:`ch-image(1)` and :code:`ch-run(1)`. :code:`--tmp DIR` A sub-directory for temporary storage is created in :code:`DIR` and removed at the end of a successful conversion. **If this script crashes or errors out, the temporary directory is left behind to assist in debugging.** Storage may be needed up to twice the uncompressed size of the image, depending on the input and output formats. Default: :code:`$TMPDIR` if specified; otherwise :code:`/var/tmp`. :code:`-v`, :code:`--verbose` Print extra chatter. Can be repeated. :code:`--xattrs` Preserve xattrs and ACLs when converting. .. Notes: 1. It’s a deliberate choice to use UNIXey options rather than the Skopeo syntax [1], e.g. "-i docker" rather than "docker:image-name". [1]: https://manpages.debian.org/unstable/golang-github-containers-image/containers-transports.5.en.html 2. There used to be an [OUT_ARG ...] that would be passed unchanged to the archiver, i.e. tar(1) or mksquashfs(1). However it wasn’t clear there were real use cases, and this has lots of opportunities to mess things up. Also, it’s not clear when it will be called. For example, if you convert a directory to a tarball, then passing e.g. -J to XZ-compress will work fine, but when converting from Docker, we just compress the tarball we got from Docker, so in that case -J wouldn’t work. 3. I also deliberately left out an option to change the output compression algorithm, under the assumption that the default is good enough. This can be revisited later IMO if needed. Image formats ============= :code:`ch-convert` knows about these values of :code:`FMT`: :code:`ch-image` Internal storage for Charliecloud’s unprivileged image builder (Dockerfile interpreter) :code:`ch-image`. :code:`dir` Ordinary filesystem directory (i.e., not a mount point) containing an unpacked image. Output directories that already exist are replaced if they look like an image. If the output directory is empty, the conversion should use the directory without overwriting it. If the directory doesn’t look like an image and isn’t empty, exit with an error. :code:`docker` Internal storage for Docker. :code:`podman` Internal storage for Podman. :code:`squash` SquashFS filesystem archive containing the flattened image. SquashFS archives are much like tar archives but are mountable, including by :code:`ch-run`'s internal SquashFUSE mounting. Most systems have at least the SquashFS-Tools installed which allows unpacking into a directory, just like tar. Due to this greater flexibility, SquashFS is preferred to tar. **Note:** Conversions to and from SquashFS are quite noisy due to the verbosity of the underlying :code:`mksquashfs(1)` and :code:`unsquashfs(1)` tools. :code:`tar` Tar archive containing the flattened image with no layer sub-archives; i.e., the output of :code:`docker export` works but the output of :code:`docker save` does not. Output tarballs are always gzipped and must end in :code:`.tar.gz`; input tarballs can be any compression acceptable to :code:`tar(1)`. All of these are local formats; :code:`ch-convert` does not know how to push or pull images. Format inference ================ :code:`ch-convert` tries to save typing by guessing formats when they are reasonably clear. This is done against filenames, rather than file contents, so the rules are the same for output descriptors that do not yet exist. Format inference is done for both :code:`IN` and :code:`OUT`. The first matching glob below yields the inferred format. Paths need not exist in the filesystem. 1. :code:`*.sqfs`, :code:`*.squash`, :code:`*.squashfs`: SquashFS. 2. :code:`*.tar`, :code:`*.t?z`, :code:`*.tar.?`, :code:`*.tar.??`: Tarball. 3. :code:`/*`, :code:`./*`, i.e. absolute path or relative path with explicit dot: Directory. 4. If `ch-image` is installed: :code:`ch-image` internal storage. 5. If Podman is installed: Podman internal storage. 6. If Docker is installed: Docker internal storage. 7. Otherwise: No format inference. Examples ======== Typical build-to-run sequence for image :code:`foo/bar` using :code:`ch-run`'s internal SquashFUSE code, inferring the output format:: $ sudo docker build -t foo/bar -f Dockerfile . [...] $ ch-convert foo/bar:latest /var/tmp/foobar.sqfs input: docker foo/bar:latest output: squashfs /var/tmp/foobar.sqfs copying ... done $ ch-run /var/tmp/foobar.sqfs -- echo hello hello Same conversion, but no format inference:: $ ch-convert -i ch-image -o squash foo/bar:latest /var/tmp/foobar.sqfs input: docker foo/bar:latest output: squashfs /var/tmp/foobar.sqfs copying ... done .. include:: ./bugs.rst .. include:: ./see_also.rst .. LocalWords: FMT fmt charliecloud-0.37/doc/ch-fromhost.rst000066400000000000000000000451421457016721300176450ustar00rootroot00000000000000:code:`ch-fromhost` +++++++++++++++++++ .. only:: not man Inject files from the host into an image directory, with various magic. Synopsis ======== :: $ ch-fromhost [OPTION ...] [FILE_OPTION ...] IMGDIR Description =========== .. note:: This command is experimental. Features may be incomplete and/or buggy. Please report any issues you find, so we can fix them! Inject files from the host into the Charliecloud image directory :code:`IMGDIR`. The purpose of this command is to inject arbitrary host files into a container necessary to access host specific resources; usually GPU or proprietary interconnects. **It is not a general copy-to-image tool**; see further discussion on use cases below. It should be run after:code:`ch-convert` and before :code:`ch-run`. After invocation, the image is no longer portable to other hosts. Injection is not atomic; if an error occurs partway through injection, the image is left in an undefined state and should be re-unpacked from storage. Injection is currently implemented using a simple file copy, but that may change in the future. Arbitrary file and libfabric injection are handled differently. Arbitrary files --------------- Arbitrary file paths that contain the strings :code:`/bin` or :code:`/sbin` are assumed to be executables and placed in :code:`/usr/bin` within the container. Paths that are not loadable libfabric providers and contain the strings :code:`/lib` or :code:`.so` are assumed to be shared libraries and are placed in the first-priority directory reported by :code:`ldconfig` (see :code:`--lib-path` below). Other files are placed in the directory specified by :code:`--dest`. If any shared libraries are injected, run :code:`ldconfig` inside the container (using :code:`ch-run -w`) after injection. Libfabric --------- MPI implementations have numerous ways of communicating messages over interconnects. We use libfabric (OFI), an OpenFabric framework that exports fabric communication services to applications, to manage these communications with built-in, or loadable, fabric providers. - https://ofiwg.github.io/libfabric - https://ofiwg.github.io/libfabric/v1.14.0/man/fi_provider.3.html Using OFI, we can (a) uniformly manage fabric communication services for both OpenMPI and MPICH, and (b) use simplified methods of accessing proprietary host hardware, e.g., Cray's Gemini/Aries and Slingshot (CXI). OFI providers implement the application facing software interfaces needed to access network specific protocols, drivers, and hardware. Loadable providers, i.e., compiled OFI libraries that end in :code:`-fi.so`, for example, Cray's :code:`libgnix-fi.so`, can be copied into, and used, by an image with a MPI configured against OFI. Alternatively, the image's :code:`libfabric.so` can be overwritten with the host's. See details and quirks below. Options ======= To specify which files to inject -------------------------------- :code:`-c`, :code:`--cmd CMD` Inject files listed in the standard output of command :code:`CMD`. :code:`-f`, :code:`--file FILE` Inject files listed in the file :code:`FILE`. :code:`-p`, :code:`--path PATH` Inject the file at :code:`PATH`. :code:`--cray-cxi` Inject cray-libfabric for slingshot. This is equivalent to :code:`--path $CH_FROMHOST_OFI_CXI`, where :code:`$CH_FROMHOST_OFI_CXI` is the path the Cray host libfabric :code:`libfabric.so`. :code:`--cray-gni` Inject cray gemini/aries GNI libfabric provider :code:`libgnix-fi.so`. This is equivalent to :code:`--fi-provider $CH_FROMHOST_OFI_GNI`, where :code:`CH_FROMHOST_OFI_GNI` is the path to the Cray host ugni provider :code:`libgnix-fi.so`. :code:`--nvidia` Use :code:`nvidia-container-cli list` (from :code:`libnvidia-container`) to find executables and libraries to inject. These can be repeated, and at least one must be specified. To specify the destination within the image ------------------------------------------- :code:`-d`, :code:`--dest DST` Place files specified later in directory :code:`IMGDIR/DST`, overriding the inferred destination, if any. If a file's destination cannot be inferred and :code:`--dest` has not been specified, exit with an error. This can be repeated to place files in varying destinations. Additional arguments -------------------- :code:`--print-cray-fi` Print inferred destination for libfabric replacement. :code:`--print-fi` Print the guest destination path for libfabric provider(s). :code:`--print-lib` Print the guest destination path for shared libraries inferred as described above. :code:`--no-ldconfig` Don't run :code:`ldconfig` even if we appear to have injected shared libraries. :code:`-h`, :code:`--help` Print help and exit. :code:`-v`, :code:`--verbose` Be more verbose about what is going on. Can be repeated. :code:`--version` Print version and exit. .. warning:: :code:`ldconfig` often prints scary-looking warnings on stderr even everything is going well. By default, we suppress these, but you can see them with sufficient verbosity. For example:: $ ch-fromhost --print-lib /var/tmp/bullseye /usr/local/lib $ ch-fromhost -v --print-lib /var/tmp/bullseye asking ldconfig for inferred shared library destination inferred shared library destination: /var/tmp/bullseye//usr/local/lib /usr/local/lib $ ch-fromhost -v -v --print-lib /var/tmp/bullseye asking ldconfig for inferred shared library destination /sbin/ldconfig: Can't stat /usr/local/lib/x86_64-linux-gnu: No such file or directory /sbin/ldconfig: Path `/lib/x86_64-linux-gnu' given more than once /sbin/ldconfig: Path `/usr/lib/x86_64-linux-gnu' given more than once /sbin/ldconfig: /lib/x86_64-linux-gnu/ld-2.31.so is the dynamic linker, ignoring inferred shared library destination: /var/tmp/bullseye//usr/local/lib /usr/local/lib See `issue #732 `_ for an example of how this was confusing for users. When to use :code:`ch-fromhost` =============================== This command does a lot of heuristic magic; while it *can* copy arbitrary files into an image, this usage is discouraged and prone to error. Here are some use cases and the recommended approach: 1. *I have some files on my build host that I want to include in the image.* Use the :code:`COPY` instruction within your Dockerfile. Note that it's OK to build an image that meets your specific needs but isn't generally portable, e.g., only runs on specific micro-architectures you're using. 2. *I have an already built image and want to install a program I compiled separately into the image.* Consider whether a building a new derived image with a Dockerfile is appropriate. Another good option is to bind-mount the directory containing your program at run time. A less good option is to :code:`cp(1)` the program into your image, because this permanently alters the image in a non-reproducible way. 3. *I have some shared libraries that I need in the image for functionality or performance, and they aren't available in a place where I can use* :code:`COPY`. This is the intended use case of :code:`ch-fromhost`. You can use :code:`--cmd`, :code:`--file`, :code:`--ofi`, and/or :code:`--path` to put together a custom solution. But, please consider filing an issue so we can package your functionality with a tidy option like :code:`--nvidia`. Libfabric usage and quirks ============================== The implementation of libfabric provider injection and replacement is experimental and has a couple quirks. 1. Containers must have the following software installed: a. libfabric (https://ofiwg.github.io/libfabric/). See :code:`charliecloud/examples/Dockerfile.libfabric`. b. Corresponding open source MPI implementation configured and built against the container libfabric, e.g., - `MPICH `_, or - `OpenMPI `_. See :code:`charliecloud/examples/Dockerfile.mpich` and :code:`charliecloud/examples/Dockerfile.openmpi`. 2. At run time, a libfabric provider can be specified with the variable :code:`FI_PROVIDER`. The path to search for shared providers can be specified with :code:`FI_PROVIDER_PATH`. These variables can be inherited from the host or explicitly set with the container's environment file :code:`/ch/environent` via :code:`--set-env`. To avoid issues and reduce complexity, the inferred injection destination for libfabric providers and replacement will always at the path in the image where :code:`libfabric.so` is found. 3. The Cray GNI loadable provider, :code:`libgnix-fi.so`, will link to compiler(s) in the programming environment by default. For example, if it is built under the :code:`PrgEnv-intel` programming environment, it will have links to files at paths :code:`/opt/gcc` and :code:`/opt/intel` that :code:`ch-run` will not bind automatically. Managing all possible bind mount paths is untenable. Thus, this experimental implementation injects libraries linked to a :code:`libgnix-fi.so` built with the minimal modules necessary to compile, i.e.: - modules - craype-network-aries - eproxy - slurm - cray-mpich - craype-haswell - craype-hugepages2M A Cray GNI provider linked against more complicated PE's will still work, assuming 1) the user explicitly bind-mounts missing libraries listed from its :code:`ldd` output, and 2) all such libraries do not conflict with container functionality, e.g., :code:`glibc.so`, etc. 4. At the time of this writing, a Cray Slingshot optimized provider is not available; however, recent libfabric source acitivity indicates there may be at some point, see: https://github.com/ofiwg/libfabric/pull/7839We. For now, on Cray systems with Slingshot, CXI, we need overwrite the container's :code:`libfabric.so` with the hosts using :code:`--path`. See examples for details. 5. Tested only for C programs compiled with GCC. Additional bind mount or kludging may be needed for untested use cases. If you'd like to use another compiler or programming environment, please get in touch so we can implement the necessary support. Please file a bug if we missed anything above or if you know how to make the code better. Notes ===== Symbolic links are dereferenced, i.e., the files pointed to are injected, not the links themselves. As a corollary, do not include symlinks to shared libraries. These will be re-created by :code:`ldconfig`. There are two alternate approaches for nVidia GPU libraries: 1. Link :code:`libnvidia-containers` into :code:`ch-run` and call the library functions directly. However, this would mean that Charliecloud would either (a) need to be compiled differently on machines with and without nVidia GPUs or (b) have :code:`libnvidia-containers` available even on machines without nVidia GPUs. Neither of these is consistent with Charliecloud's philosophies of simplicity and minimal dependencies. 2. Use :code:`nvidia-container-cli configure` to do the injecting. This would require that containers have a half-started state, where the namespaces are active and everything is mounted but :code:`pivot_root(2)` has not been performed. This is not feasible because Charliecloud has no notion of a half-started container. Further, while these alternate approaches would simplify or eliminate this script for nVidia GPUs, they would not solve the problem for other situations. Bugs ==== File paths may not contain colons or newlines. :code:`ldconfig` tends to print :code:`stat` errors; these are typically non-fatal and occur when trying to probe common library paths. See `issue #732 `_. Examples ======== libfabric --------- Cray Slingshot CXI injection. Replace image libabfric, i.e., :code:`libfabric.so`, with Cray host's libfabric at host path :code:`/opt/cray-libfabric/lib64/libfabric.so`. :: $ ch-fromhost -v --path /opt/cray-libfabric/lib64/libfabric.so /tmp/ompi [ debug ] queueing files [ debug ] cray libfabric: /opt/cray-libfabric/lib64/libfabric.so [ debug ] searching image for inferred libfabric destiation [ debug ] found /tmp/ompi/usr/local/lib/libfabric.so [ debug ] adding cray libfabric libraries [ debug ] skipping /lib64/libcom_err.so.2 [...] [ debug ] queueing files [ debug ] shared library: /usr/lib64/libcxi.so.1 [ debug ] queueing files [ debug ] shared library: /usr/lib64/libcxi.so.1.2.1 [ debug ] queueing files [ debug ] shared library: /usr/lib64/libjson-c.so.3 [ debug ] queueing files [ debug ] shared library: /usr/lib64/libjson-c.so.3.0.1 [...] [ debug ] queueing files [ debug ] shared library: /usr/lib64/libssh.so.4 [ debug ] queueing files [ debug ] shared library: /usr/lib64/libssh.so.4.7.4 [...] [ debug ] inferred shared library destination: /tmp/ompi//usr/local/lib [ debug ] injecting into image: /tmp/ompi/ [ debug ] mkdir -p /tmp/ompi//var/lib/hugetlbfs [ debug ] mkdir -p /tmp/ompi//var/spool/slurmd [ debug ] echo '/usr/lib64' >> /tmp/ompi//etc/ld.so.conf.d/ch-ofi.conf [ debug ] /opt/cray-libfabric/lib64/libfabric.so -> /usr/local/lib (inferred) [ debug ] /usr/lib64/libcxi.so.1 -> /usr/local/lib (inferred) [ debug ] /usr/lib64/libcxi.so.1.2.1 -> /usr/local/lib (inferred) [ debug ] /usr/lib64/libjson-c.so.3 -> /usr/local/lib (inferred) [ debug ] /usr/lib64/libjson-c.so.3.0.1 -> /usr/local/lib (inferred) [ debug ] /usr/lib64/libssh.so.4 -> /usr/local/lib (inferred) [ debug ] /usr/lib64/libssh.so.4.7.4 -> /usr/local/lib (inferred) [ debug ] running ldconfig [ debug ] ch-run -w /tmp/ompi/ -- /sbin/ldconfig [ debug ] validating ldconfig cache done Same as above, except also inject Cray's :code:`fi_info` to verify Slingshot provider access. :: $ ch-fromhost -v --path /opt/cray/libfabric/1.15.0.0/lib64/libfabric.so \ -d /usr/local/bin \ --path /opt/cray/libfabric/1.15.0.0/lib64/libfabric.so \ /tmp/ompi [...] $ ch-run /tmp/ompi/ -- fi_info -p cxi provider: cxi fabric: cxi [...] type: FI_EP_RDM protocol: FI_PROTO_CXI Cray GNI shared provider injection. Add Cray host built GNI provider :code:`libgnix-fi.so` to the image and verify with :code:`fi_info`. :: $ ch-fromhost -v --path /home/ofi/libgnix-fi.so /tmp/ompi [ debug ] queueing files [ debug ] libfabric shared provider: /home/ofi/libgnix-fi.so [ debug ] searching /tmp/ompi for libfabric shared provider destination [ debug ] found: /tmp/ompi/usr/local/lib/libfabric.so [ debug ] inferred provider destination: //usr/local/lib/libfabric [ debug ] injecting into image: /tmp/ompi [ debug ] mkdir -p /tmp/ompi//usr/local/lib/libfabric [ debug ] mkdir -p /tmp/ompi/var/lib/hugetlbfs [ debug ] mkdir -p /tmp/ompi/var/opt/cray/alps/spool [ debug ] mkdir -p /tmp/ompi/opt/cray/wlm_detect [ debug ] mkdir -p /tmp/ompi/etc/opt/cray/wlm_detect [ debug ] mkdir -p /tmp/ompi/opt/cray/udreg [ debug ] mkdir -p /tmp/ompi/opt/cray/xpmem [ debug ] mkdir -p /tmp/ompi/opt/cray/ugni [ debug ] mkdir -p /tmp/ompi/opt/cray/alps [ debug ] echo '/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf [ debug ] echo '/opt/cray/alps/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf [ debug ] echo '/opt/cray/udreg/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf [ debug ] echo '/opt/cray/ugni/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf [ debug ] echo '/opt/cray/wlm_detect/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf [ debug ] echo '/opt/cray/xpmem/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf [ debug ] echo '/usr/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf [ debug ] /home/ofi/libgnix-fi.so -> //usr/local/lib/libfabric (inferred) [ debug ] running ldconfig [ debug ] ch-run -w /tmp/ompi -- /sbin/ldconfig [ debug ] validating ldconfig cache done $ ch-run /tmp/ompi -- fi_info -p gni provider: gni fabric: gni [...] type: FI_EP_RDM protocol: FI_PROTO_GNI Arbitrary --------- Place shared library :code:`/usr/lib64/libfoo.so` at path :code:`/usr/lib/libfoo.so` (assuming :code:`/usr/lib` is the first directory searched by the dynamic loader in the image), within the image :code:`/var/tmp/baz` and executable :code:`/bin/bar` at path :code:`/usr/bin/bar`. Then, create appropriate symlinks to :code:`libfoo` and update the :code:`ld.so` cache. :: $ cat qux.txt /bin/bar /usr/lib64/libfoo.so $ ch-fromhost --file qux.txt /var/tmp/baz Same as above:: $ ch-fromhost --cmd 'cat qux.txt' /var/tmp/baz Same as above:: $ ch-fromhost --path /bin/bar --path /usr/lib64/libfoo.so /var/tmp/baz Same as above, but place the files into :code:`/corge` instead (and the shared library will not be found by :code:`ldconfig`):: $ ch-fromhost --dest /corge --file qux.txt /var/tmp/baz Same as above, and also place file :code:`/etc/quux` at :code:`/etc/quux` within the container:: $ ch-fromhost --file qux.txt --dest /etc --path /etc/quux /var/tmp/baz Inject the executables and libraries recommended by nVidia into the image, and then run :code:`ldconfig`:: $ ch-fromhost --nvidia /var/tmp/baz asking ldconfig for shared library destination /sbin/ldconfig: Can’t stat /libx32: No such file or directory /sbin/ldconfig: Can’t stat /usr/libx32: No such file or directory shared library destination: /usr/lib64//bind9-export injecting into image: /var/tmp/baz /usr/bin/nvidia-smi -> /usr/bin (inferred) /usr/bin/nvidia-debugdump -> /usr/bin (inferred) /usr/bin/nvidia-persistenced -> /usr/bin (inferred) /usr/bin/nvidia-cuda-mps-control -> /usr/bin (inferred) /usr/bin/nvidia-cuda-mps-server -> /usr/bin (inferred) /usr/lib64/libnvidia-ml.so.460.32.03 -> /usr/lib64//bind9-export (inferred) /usr/lib64/libnvidia-cfg.so.460.32.03 -> /usr/lib64//bind9-export (inferred) [...] /usr/lib64/libGLESv2_nvidia.so.460.32.03 -> /usr/lib64//bind9-export (inferred) /usr/lib64/libGLESv1_CM_nvidia.so.460.32.03 -> /usr/lib64//bind9-export (inferred) running ldconfig Acknowledgements ================ This command was inspired by the similar `Shifter `_ feature that allows Shifter containers to use the Cray Aries network. We particularly appreciate the help provided by Shane Canon and Doug Jacobsen during our implementation of :code:`--cray-mpi`. We appreciate the advice of Ryan Olson at nVidia on implementing :code:`--nvidia`. .. include:: ./bugs.rst .. include:: ./see_also.rst .. LocalWords: libmpi libmpich nvidia charliecloud-0.37/doc/ch-image.rst000066400000000000000000002405571457016721300170750ustar00rootroot00000000000000:code:`ch-image` ++++++++++++++++ .. only:: not man Build and manage images; completely unprivileged. Synopsis ======== .. Note: Keep these consistent with the synopses in each subcommand. :: $ ch-image [...] build [-t TAG] [-f DOCKERFILE] [...] CONTEXT $ ch-image [...] build-cache [...] $ ch-image [...] delete IMAGE_GLOB [IMAGE_GLOB ...] $ ch-image [...] gestalt [SELECTOR] $ ch-image [...] import PATH IMAGE_REF $ ch-image [...] list [-l] [IMAGE_REF] $ ch-image [...] pull [...] IMAGE_REF [DEST_REF] $ ch-image [...] push [--image DIR] IMAGE_REF [DEST_REF] $ ch-image [...] reset $ ch-image [...] undelete IMAGE_REF $ ch-image { --help | --version | --dependencies } Description =========== :code:`ch-image` is a tool for building and manipulating container images, but not running them (for that you want :code:`ch-run`). It is completely unprivileged, with no setuid/setgid/setcap helpers. Many operations can use caching for speed. The action to take is specified by a sub-command. Options that print brief information and then exit: :code:`-h`, :code:`--help` Print help and exit successfully. If specified before the sub-command, print general help and list of sub-commands; if after the sub-command, print help specific to that sub-command. :code:`--dependencies` Report dependency problems on standard output, if any, and exit. If all is well, there is no output and the exit is successful; in case of problems, the exit is unsuccessful. :code:`--version` Print version number and exit successfully. Common options placed before or after the sub-command: :code:`-a`, :code:`--arch ARCH` Use :code:`ARCH` for architecture-aware registry operations. (See section "Architecture" below for details.) :code:`--always-download` Download all files when pulling, even if they are already in builder storage. Note that :code:`ch-image pull` will always retrieve the most up-to-date image; this option is mostly for debugging. :code:`--auth` Authenticate with the remote repository, then (if successful) make all subsequent requests in authenticated mode. For most subcommands, the default is to never authenticate, i.e., make all requests anonymously. The exception is :code:`push`, which implies :code:`--auth`. :code:`--break MODULE:LINE` Set a `PDB `_ breakpoint at line number :code:`LINE` of module named :code:`MODULE` (typically the filename with :code:`.py` removed, or :code:`__main__` for :code:`ch-image` itself). That is, a PDB debugger shell will open before executing the specified line. This is accomplished by re-parsing the module, injecting :code:`import pdb; pdb.set_trace()` into the parse tree, re-compiling the tree, and replacing the module’s code with the result. This has various gotchas, including (1) module-level code in the target module is executed twice, (2) the option is parsed with bespoke early code so command line argument parsing itself can be debugged, (3) breakpoints on function definition will trigger while the module is being re-executed, not when the function is called (break on the first line of the function body instead), and (4) other weirdness we haven’t yet characterized. :code:`--cache` Enable build cache. Default if a sufficiently new Git is available. See section :ref:`Build cache ` for details. :code:`--cache-large SIZE` Set the cache’s large file threshold to :code:`SIZE` MiB, or :code:`0` for no large files, which is the default. Values greater than zero can speed up many builds but can also cause performance degradation. **Experimental.** See section :ref:`Large file threshold ` for details. :code:`--debug` Add a stack trace to fatal error hints. This can also be done by setting the environment variable :code:`CH_IMAGE_DEBUG`. :code:`--no-cache` Disable build cache. Default if a sufficiently new Git is not available. This option turns off the cache completely; if you want to re-execute a Dockerfile and store the new results in cache, use :code:`--rebuild` instead. :code:`--no-lock` Disable storage directory locking. This lets you run as many concurrent :code:`ch-image` instances as you want against the same storage directory, which risks corruption but may be OK for some workloads. :code:`--no-xattrs` Enforce default handling of xattrs, i.e. do not save them in the build cache or restore them on rebuild. This is the default, but the option is provided to override the :code:`$CH_XATTRS` environment variable. :code:`--password-many` Re-prompt the user every time a registry password is needed. :code:`--profile` Dump profile to files :code:`/tmp/chofile.p` (:code:`cProfile` dump format) and :code:`/tmp/chofile.txt` (text summary). You can convert the former to a PDF call graph with :code:`gprof2dot -f pstats /tmp/chofile.p | dot -Tpdf -o /tmp/chofile.pdf`. This excludes time spend in subprocesses. Profile data should still be written on fatal errors, but not if the program crashes. :code:`-q, --quiet` Be quieter; can be repeated. Incompatible with :code:`-v` and suppresses :code:`--debug` regardless of option order. See the :ref:`FAQ entry on verbosity ` for details. :code:`--rebuild` Execute all instructions, even if they are build cache hits, except for :code:`FROM` which is retrieved from cache on hit. :code:`-s`, :code:`--storage DIR` Set the storage directory (see below for important details). :code:`--tls-no-verify` Don’t verify TLS certificates of the repository. (Do not use this option unless you understand the risks.) :code:`-v`, :code:`--verbose` Print extra chatter; can be repeated. See the :ref:`FAQ entry on verbosity ` for details. :code:`--xattrs` Save xattrs and ACLs in the build cache, and restore them when rebuilding from the cache. Architecture ============ Charliecloud provides the option :code:`--arch ARCH` to specify the architecture for architecture-aware registry operations. The argument :code:`ARCH` can be: (1) :code:`yolo`, to bypass architecture-aware code and use the registry’s default architecture; (2) :code:`host`, to use the host’s architecture, obtained with the equivalent of :code:`uname -m` (default if :code:`--arch` not specified); or (3) an architecture name. If the specified architecture is not available, the error message will list which ones are. **Notes:** 1. :code:`ch-image` is limited to one image per image reference in builder storage at a time, regardless of architecture. For example, if you say :code:`ch-image pull --arch=foo baz` and then :code:`ch-image pull --arch=bar baz`, builder storage will contain one image called "baz", with architecture "bar". 2. Images’ default architecture is usually :code:`amd64`, so this is usually what you get with :code:`--arch=yolo`. Similarly, if a registry image is architecture-unaware, it will still be pulled with :code:`--arch=amd64` and :code:`--arch=host` on x86-64 hosts (other host architectures must specify :code:`--arch=yolo` to pull architecture-unaware images). 3. :code:`uname -m` and image registries often use different names for the same architecture. For example, what :code:`uname -m` reports as "x86_64" is known to registries as "amd64". :code:`--arch=host` should translate if needed, but it’s useful to know this is happening. Directly specified architecture names are passed to the registry without translation. 4. Registries treat architecture as a pair of items, architecture and sometimes variant (e.g., "arm" and "v7"). Charliecloud treats architecture as a simple string and converts to/from the registry view transparently. Authentication ============== Charliecloud does not have configuration files; thus, it has no separate :code:`login` subcommand to store secrets. Instead, Charliecloud will prompt for a username and password when authentication is needed. Note that some repositories refer to the secret as something other than a "password"; e.g., GitLab calls it a "personal access token (PAT)", Quay calls it an "application token", and nVidia NGC calls it an "API token". For non-interactive authentication, you can use environment variables :code:`CH_IMAGE_USERNAME` and :code:`CH_IMAGE_PASSWORD`. Only do this if you fully understand the implications for your specific use case, because it is difficult to securely store secrets in environment variables. By default for most subcommands, all registry access is anonymous. To instead use authenticated access for everything, specify :code:`--auth` or set the environment variable :code:`$CH_IMAGE_AUTH=yes`. The exception is :code:`push`, which always runs in authenticated mode. Even for pulling public images, it can be useful to authenticate for registries that have per-user rate limits, such as `Docker Hub `_. (Older versions of Charliecloud started with anonymous access, then tried to upgrade to authenticated if it seemed necessary. However, this turned out to be brittle; see issue `#1318 `_.) The username and password are remembered for the life of the process and silently re-offered to the registry if needed. One case when this happens is on push to a private registry: many registries will first offer a read-only token when :code:`ch-image` checks if something exists, then re-authenticate when upgrading the token to read-write for upload. If your site uses one-time passwords such as provided by a security device, you can specify :code:`--password-many` to provide a new secret each time. These values are not saved persistently, e.g. in a file. Note that we do use normal Python variables for this information, without pinning them into physical RAM with `mlock(2) `_ or any other special treatment, so we cannot guarantee they will never reach non-volatile storage. .. admonition:: Technical details Most registries use something called `Bearer authentication `_, where the client (e.g., Charliecloud) includes a *token* in the headers of every HTTP request. The authorization dance is different from the typical UNIX approach, where there is a separate login sequence before any content requests are made. The client starts by simply making the HTTP request it wants (e.g., to :code:`GET` an image manifest), and if the registry doesn’t like the client’s token (or if there is no token because the client doesn’t have one yet), it replies with HTTP 401 Unauthorized, but crucially it also provides instructions in the response header on how to get a token. The client then follows those instructions, obtains a token, re-tries the request, and (hopefully) all is well. This approach also allows a client to upgrade a token if needed, e.g. when transitioning from asking if a layer exists to uploading its content. The distinction between Charliecloud’s anonymous mode and authenticated modes is that it will only ask for anonymous tokens in anonymous mode and authenticated tokens in authenticated mode. That is, anonymous mode does involve an authentication procedure to obtain a token, but this "authentication" is done anonymously. (Yes, it’s confusing.) Registries also often reply HTTP 401 when an image does not exist, rather than the seemingly more correct HTTP 404 Not Found. This is to avoid information leakage about the existence of images the client is not allowed to pull, and it’s why Charliecloud never says an image simply does not exist. Storage directory ================= :code:`ch-image` maintains state using normal files and directories located in its *storage directory*; contents include various caches and temporary images used for building. In descending order of priority, this directory is located at: :code:`-s`, :code:`--storage DIR` Command line option. :code:`$CH_IMAGE_STORAGE` Environment variable. The path must be absolute, because the variable is likely set in a very different context than when it’s used, which seems error-prone on what a relative path is relative to. :code:`/var/tmp/$USER.ch` Default. (Previously, the default was :code:`/var/tmp/$USER/ch-image`. If a valid storage directory is found at the old default path, :code:`ch-image` tries to move it to the new default path.) Unlike many container implementations, there is no notion of storage drivers, graph drivers, etc., to select and/or configure. The storage directory can reside on any single filesystem (i.e., it cannot be split across multiple filesystems). However, it contains lots of small files and metadata traffic can be intense. For example, the Charliecloud test suite uses approximately 400,000 files and directories in the storage directory as of this writing. Place it on a filesystem appropriate for this; tmpfs’es such as :code:`/var/tmp` are a good choice if you have enough RAM (:code:`/tmp` is not recommended because :code:`ch-run` bind-mounts it into containers by default). While you can currently poke around in the storage directory and find unpacked images runnable with :code:`ch-run`, this is not a supported use case. The supported workflow uses :code:`ch-convert` to obtain a packed image; see the tutorial for details. The storage directory format changes on no particular schedule. :code:`ch-image` is normally able to upgrade directories produced by a given Charliecloud version up to one year after that version’s release. Upgrades outside this window and downgrades are not supported. In these cases, :code:`ch-image` will refuse to run until you delete and re-initialize the storage directory with :code:`ch-image reset`. .. warning:: Network filesystems, especially Lustre, are typically bad choices for the storage directory. This is a site-specific question and your local support will likely have strong opinions. .. _ch-image_build-cache: Build cache =========== Overview -------- Subcommands that create images, such as :code:`build` and :code:`pull`, can use a build cache to speed repeated operations. That is, an image is created by starting from the empty image and executing a sequence of instructions, largely Dockerfile instructions but also some others like "pull" and "import". Some instructions are expensive to execute (e.g., :code:`RUN wget http://slow.example.com/bigfile` or transferring data billed by the byte), so it’s often cheaper to retrieve their results from cache instead. The build cache uses a relatively new Git under the hood; see the installation instructions for version requirements. Charliecloud implements workarounds for Git’s various storage limitations, so things like file metadata and Git repositories within the image should work. **Important exception**: No files named :code:`.git*` or other Git metadata are permitted in the image’s root directory. `Extended attributes `_ (xattrs) are ignored by the build cache by default. Cache support for xattrs belonging to unprivileged xattr namespaces (e.g. :code:`user`) can be enabled by specifying the :code:`--xattrs` option or by setting the :code:`CH_XATTRS` environment variable. If :code:`CH_XATTRS` is set, you override it with :code:`--no-xattrs`. **Note that extended attributes in privileged xattr namespaces (e.g. :code:`trusted`) cannot be read by :code:`ch-image` and will always be lost without warning.** The cache has three modes: *enabled*, *disabled*, and a hybrid mode called *rebuild* where the cache is fully enabled for :code:`FROM` instructions, but all other operations re-execute and re-cache their results. The purpose of *rebuild* is to do a clean rebuild of a Dockerfile atop a known-good base image. Enabled mode is selected with :code:`--cache` or setting :code:`$CH_IMAGE_CACHE` to :code:`enabled`, disabled mode with :code:`--no-cache` or :code:`disabled`, and rebuild mode with :code:`--rebuild` or :code:`rebuild`. The default mode is *enabled* if an appropriate Git is installed, otherwise *disabled*. Compared to other implementations --------------------------------- .. note:: This section is a lightly edited excerpt from our paper “`Charliecloud’s layer-free, Git-based container build cache `_”. Existing tools such as Docker and Podman implement their build cache with a layered (union) filesystem such as `OverlayFS `_ or `FUSE-OverlayFS `_ and tar archives to represent the content of each layer; this approach is `standardized by OCI `_. The layered cache works, but it has drawbacks in three critical areas: 1. **Diff format.** The tar format is poorly standardized and `not designed for diffs `_. Notably, tar cannot represent file deletion. The workaround used for OCI layers is specially named *whiteout* files, which means the tar archives cannot be unpacked by standard UNIX tools and require special container-specific processing. 2. **Cache overhead.** Each time a Dockerfile instruction is started, a new overlay filesystem is mounted atop the existing layer stack. File metadata operations in the instruction then start at the top layer and descend the stack until the layer containing the desired file is reached. The cost of these operations is therefore proportional to the number of layers, i.e., the number of instructions between the empty root image and the instruction being executed. This results in a `best practice `_ of large, complex instructions to minimize their number, which can conflict with simpler, more numerous instructions the user might prefer. 3. **De-duplication.** Identical files on layers with an ancestry relationship (i.e., instruction *A* precedes *B* in a build) are stored only once. However, identical files on layers without this relationship are stored multiple times. For example, if instructions *B* and *B'* both follow *A* — perhaps because *B* was modified and the image rebuilt — then any files created by both *B* and *B'* will be stored twice. Also, similar files are never de-duplicated, regardless of ancestry. For example, if instruction *A* creates a file and subsequently instruction *B* modifies a single bit in that file, both versions are stored in their entirety. Our Git-based cache addresses the three drawbacks: (1) Git is purpose-built to store changing directory trees, (2) cache overhead is imposed only at instruction commit time, and (3) Git de-duplicates both identical and similar files. Also, it is based on an extremely widely used tool that enjoys development support from well-resourced actors, in particular on scaling (e.g., Microsoft’s large-repository accelerator `Scalar `_ was recently `merged into Git `_). In addition to these structural advantages, performance experiments reported in our paper above show that the Git-based approach is as good as (and sometimes better than) overlay-based caches. On build time, the two approaches are broadly similar, with one or the other being faster depending on context. Both had performance problems on NFS. Notably, however, the Git-based cache was much faster for a 129-instruction Dockerfile. On disk usage, the winner depended on the condition. For example, we saw the layered cache storing large sibling layers redundantly; on the other hand, the Git-based cache has some obvious redundancies as well, and one must compact it for full de-duplication benefit. However, Git’s de-duplication was quite effective in some conditions and we suspect will prove even better in more realistic scenarios. That is, we believe our results show that the Git-based build cache is highly competitive with the layered approach, with no obvious inferiority so far and hints that it may be superior on important dimensions. We have ongoing work to explore these questions in more detail. De-duplication and garbage collection ------------------------------------- Charliecloud’s build cache takes advantage of Git’s file de-duplication features. This operates across the entire build cache, i.e., files are de-duplicated no matter where in the cache they are found or the relationship between their container images. Files are de-duplicated at different times depending on whether they are identical or merely similar. *Identical* files are de-duplicated at :code:`git add` time; in :code:`ch-image build` terms, that’s upon committing a successful instruction. That is, it’s impossible to store two files with the same content in the build cache. If you try — say with :code:`RUN yum install -y foo` in one Dockerfile and :code:`RUN yum install -y foo bar` in another, which are different instructions but both install RPM :code:`foo`’s files — the content is stored once and each copy gets its own metadata and a pointer to the content, much like filesystem hard links. *Similar* files, however, are only de-duplicated during Git’s garbage collection process. When files are initially added to a Git repository (with :code:`git add`), they are stored inside the repository as (possibly compressed) individual files, called *objects* in Git jargon. Upon garbage collection, which happens both automatically when certain parameters are met and explicitly with :code:`git gc`, these files are archived and (re-)compressed together into a single file called a *packfile*. Also, existing packfiles may be re-written into the new one. During this process, similar files are identified, and each set of similar files is stored as one base file plus diffs to recover the others. (Similarity detection seems to be based primarily on file size.) This *delta* process is agnostic to alignment, which is an advantage over alignment-sensitive block-level de-duplicating filesystems. Exception: "Large" files are not compressed or de-duplicated. We use the Git default threshold of 512 MiB (as of this writing). Charliecloud runs Git garbage collection at two different times. First, a lighter-weight garbage pass runs automatically when the number of loose files (objects) grows beyond a limit. This limit is in flux as we learn more about build cache performance, but it’s quite a bit higher than the Git default. This garbage runs in the background and can continue after the build completes; you may see Git processes using a lot of CPU. An important limitation of the automatic garbage is that large packfiles (again, this is in flux, but it’s several GiB) will not be re-packed, limiting the scope of similar file detection. To address this, a heavier garbage collection can be run manually with :code:`ch-image build-cache --gc`. This will re-pack (and re-write) the entire build cache, de-duplicating all similar files. In both cases, garbage uses all available cores. :code:`git build-cache` prints the specific garbage collection parameters in use, and :code:`-v` can be added for more detail. .. _ch-image_bu-large: Large file threshold -------------------- Because Git uses content-addressed storage, upon commit, it must read in full all files modified by an instruction. This I/O cost can be a significant fraction of build time for some images. To mitigate this, regular files larger than the experimental *large file threshold* are stored outside the Git repository, somewhat like `Git Large File Storage `_. :code:`ch-image` copies large files in and out of images at each instruction commit. It tries to do this with a fast metadata-only copy-on-write operation called “reflink”, but that is only supported with the right Python version, Linux kernel version, and filesystem. If unsupported, Charliecloud falls back to an expensive standard copy, which is likely slower than letting Git deal with the files. See :ref:`File copy performance ` for details. Every version of a large file is stored verbatim and uncompressed (e.g., a large file with a one-byte change will be stored in full twice), so Git’s de-duplication does not apply. *However*, on filesystems with reflink support, files can share extents (e.g., each of the two files will have its own extent containing the changed byte, but the rest of the extents will remain shared). This provides de-duplication between large files images that share ancestry. Also, unused large files are deleted by :code:`ch-image build-cache --gc`. A final caveat: Large files in any image with the same path, mode, size, and mtime (to nanosecond precision if possible) are considered identical, even if their content is not actually identical (e.g., :code:`touch(1)` shenanigans can corrupt an image). Option :code:`--cache-large` sets the threshold in MiB; if not set, environment variable :code:`CH_IMAGE_CACHE_LARGE` is used; if that is not set either, the default value :code:`0` indicates that no files are considered large. (Note that Git has an unrelated setting called :code:`core.bigFileThreshold`.) Example ------- Suppose we have this Dockerfile:: $ cat a.df FROM alpine:3.17 RUN echo foo RUN echo bar On our first build, we get:: $ ch-image build -t foo -f a.df . 1. FROM alpine:3.17 [ ... pull chatter omitted ... ] 2. RUN echo foo copying image ... foo 3. RUN echo bar bar grown in 3 instructions: foo Note the dot after each instruction’s line number. This means that the instruction was executed. You can also see this by the output of the two :code:`echo` commands. But on our second build, we get:: $ ch-image build -t foo -f a.df . 1* FROM alpine:3.17 2* RUN echo foo 3* RUN echo bar copying image ... grown in 3 instructions: foo Here, instead of being executed, each instruction’s results were retrieved from cache. (Charliecloud uses lazy retrieval; nothing is actually retrieved until the end, as seen by the "copying image" message.) Cache hit for each instruction is indicated by an asterisk (:code:`*`) after the line number. Even for such a small and short Dockerfile, this build is noticeably faster than the first. We can also try a second, slightly different Dockerfile. Note that the first three instructions are the same, but the third is different:: $ cat c.df FROM alpine:3.17 RUN echo foo RUN echo qux $ ch-image build -t c -f c.df . 1* FROM alpine:3.17 2* RUN echo foo 3. RUN echo qux copying image ... qux grown in 3 instructions: c Here, the first two instructions are hits from the first Dockerfile, but the third is a miss, so Charliecloud retrieves that state and continues building. We can also inspect the cache:: $ ch-image build-cache --tree * (c) RUN echo qux | * (a) RUN echo bar |/ * RUN echo foo * (alpine+3.9) PULL alpine:3.17 * (root) ROOT named images: 4 state IDs: 5 commits: 5 files: 317 disk used: 3 MiB Here there are four named images: :code:`a` and :code:`c` that we built, the base image :code:`alpine:3.17` (written as :code:`alpine+3.9` because colon is not allowed in Git branch names), and the empty base of everything :code:`root`. Also note how :code:`a` and :code:`c` diverge after the last common instruction :code:`RUN echo foo`. :code:`build` ============= Build an image from a Dockerfile and put it in the storage directory. Synopsis -------- :: $ ch-image [...] build [-t TAG] [-f DOCKERFILE] [...] CONTEXT Description ----------- See below for differences with other Dockerfile interpreters. Charliecloud supports an extended instruction (:code:`RSYNC`), a few other instructions behave slightly differently, and a few are ignored. Note that :code:`FROM` implicitly pulls the base image if needed, so you may want to read about the :code:`pull` subcommand below as well. Required argument: :code:`CONTEXT` Path to context directory. This is the root of :code:`COPY` instructions in the Dockerfile. If a single hyphen (:code:`-`) is specified: (a) read the Dockerfile from standard input, (b) specifying :code:`--file` is an error, and (c) there is no context, so :code:`COPY` will fail. (See :code:`--file` for how to provide the Dockerfile on standard input while also having a context.) Options: :code:`-b`, :code:`--bind SRC[:DST]` For :code:`RUN` instructions only, bind-mount :code:`SRC` at guest :code:`DST`. The default destination if not specified is to use the same path as the host; i.e., the default is equivalent to :code:`--bind=SRC:SRC`. If :code:`DST` does not exist, try to create it as an empty directory, though images do have ten directories :code:`/mnt/[0-9]` already available as mount points. Can be repeated. **Note:** See documentation for :code:`ch-run --bind` for important caveats and gotchas. **Note:** Other instructions that modify the image filesystem, e.g. :code:`COPY`, can only access host files from the context directory, regardless of this option. :code:`--build-arg KEY[=VALUE]` Set build-time variable :code:`KEY` defined by :code:`ARG` instruction to :code:`VALUE`. If :code:`VALUE` not specified, use the value of environment variable :code:`KEY`. :code:`-f`, :code:`--file DOCKERFILE` Use :code:`DOCKERFILE` instead of :code:`CONTEXT/Dockerfile`. If a single hyphen (:code:`-`) is specified, read the Dockerfile from standard input; like :code:`docker build`, the context directory is still available in this case. :code:`--force[=MODE]` Use unprivileged build with root emulation mode :code:`MODE`, which can be :code:`fakeroot`, :code:`seccomp` (the default), or :code:`none`. See section “Privilege model” below for details on what this does and when you might need it. :code:`--force-cmd=CMD,ARG1[,ARG2...]` If command :code:`CMD` is found in a :code:`RUN` instruction, add the comma-separated :code:`ARGs` to it. For example, :code:`--force-cmd=foo,-a,--bar=baz` would transform :code:`RUN foo -c` into :code:`RUN foo -a --bar=baz -c`. This is intended to suppress validation that defeats :code:`--force=seccomp` and implies that option. Can be repeated. If specified, replaces (does not extend) the default suppression options. Literal commas can be escaped with backslash; importantly however, backslash will need to be protected from the shell also. Section “Privilege model” below explains why you might need this. :code:`-n`, :code:`--dry-run` Don’t actually execute any Dockerfile instructions. :code:`--parse-only` Stop after parsing the Dockerfile. :code:`-t`, :code:`--tag TAG` Name of image to create. If not specified, infer the name: 1. If Dockerfile named :code:`Dockerfile` with an extension: use the extension with invalid characters stripped, e.g. :code:`Dockerfile.@FOO.bar` → :code:`foo.bar`. 2. If Dockerfile has extension :code:`df` or :code:`dockerfile`: use the basename with the same transformation, e.g. :code:`baz.@QUX.dockerfile` -> :code:`baz.qux`. 3. If context directory is not :code:`/`: use its name, i.e. the last component of the absolute path to the context directory, with the same transformation, 4. Otherwise (context directory is :code:`/`): use :code:`root`. If no colon present in the name, append :code:`:latest`. Uses :code:`ch-run -w -u0 -g0 --no-passwd --unsafe` to execute :code:`RUN` instructions. Privilege model --------------- Overview ~~~~~~~~ :code:`ch-image` is a *fully* unprivileged image builder. It does not use any setuid or setcap helper programs, and it does not use configuration files :code:`/etc/subuid` or :code:`/etc/subgid`. This contrasts with the “rootless” or “`fakeroot `_” modes of some competing builders, which do require privileged supporting code or utilities. Without root emulation, this approach does confuse programs that expect to have real root privileges, most notably distribution package installers. This subsection describes why that happens and what you can do about it. :code:`ch-image` executes all instructions as the normal user who invokes it. For :code:`RUN`, this is accomplished with :code:`ch-run` arguments including :code:`-w --uid=0 --gid=0`. That is, your host EUID and EGID are both mapped to zero inside the container, and only one UID (zero) and GID (zero) are available inside the container. Under this arrangement, processes running in the container for each :code:`RUN` *appear* to be running as root, but many privileged system calls will fail without the root emulation methods described below. **This affects any fully unprivileged container build, not just Charliecloud.** The most common time to see this is installing packages. For example, here is RPM failing to :code:`chown(2)` a file, which makes the package update fail: .. code-block:: none Updating : 1:dbus-1.10.24-13.el7_6.x86_64 2/4 Error unpacking rpm package 1:dbus-1.10.24-13.el7_6.x86_64 error: unpacking of archive failed on file /usr/libexec/dbus-1/dbus-daemon-launch-helper;5cffd726: cpio: chown Cleanup : 1:dbus-libs-1.10.24-12.el7.x86_64 3/4 error: dbus-1:1.10.24-13.el7_6.x86_64: install failed This one is (ironically) :code:`apt-get` failing to drop privileges: .. code-block:: none E: setgroups 65534 failed - setgroups (1: Operation not permitted) E: setegid 65534 failed - setegid (22: Invalid argument) E: seteuid 100 failed - seteuid (22: Invalid argument) E: setgroups 0 failed - setgroups (1: Operation not permitted) Charliecloud provides two different mechanisms to avoid these problems. Both involve lying to the containerized process about privileged system calls, but at very different levels of complexity. Root emulation mode :code:`fakeroot` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This mode uses :code:`fakeroot(1)` to maintain an elaborate web of deceit that is internally consistent. This program intercepts both privileged system calls (e.g., :code:`setuid(2)`) as well as other system calls whose return values depend on those calls (e.g., :code:`getuid(2)`), faking success for privileged system calls (perhaps making no system call at all) and altering return values to be consistent with earlier fake success. Charliecloud automatically installs the :code:`fakeroot(1)` program inside the container and then wraps :code:`RUN` instructions having known privilege needs with it. Thus, this mode is only available for certain distributions. The advantage of this mode is its consistency; e.g., careful programs that check the new UID after attempting to change it will not notice anything amiss. Its disadvantage is complexity: detailed knowledge and procedures for multiple Linux distributions. This mode has three basic steps: 1. After :code:`FROM`, analyze the image to see what distribution it contains, which determines the specific workarounds. 2. Before the user command in the first :code:`RUN` instruction where the injection seems needed, install :code:`fakeroot(1)` in the image, if one is not already installed, as well as any other necessary initialization commands. For example, we turn off the :code:`apt` sandbox (for Debian Buster) and configure EPEL but leave it disabled (for CentOS/RHEL). 3. Prepend :code:`fakeroot` to :code:`RUN` instructions that seem to need it, e.g. ones that contain :code:`apt`, :code:`apt-get`, :code:`dpkg` for Debian derivatives and :code:`dnf`, :code:`rpm`, or :code:`yum` for RPM-based distributions. :code:`RUN` instructions that *do not* seem to need modification are unaffected by this mode. The details are specific to each distribution. :code:`ch-image` analyzes image content (e.g., grepping :code:`/etc/debian_version`) to select a configuration; see :code:`lib/force.py` for details. :code:`ch-image` prints exactly what it is doing. .. warning:: Because of :code:`fakeroot` mode’s complexity, we plan to remove it if :code:`seccomp` mode performs well enough. If you have a situation where :code:`fakeroot` mode works and :code:`seccomp` does not, please let us know. Root emulation mode :code:`seccomp` (default) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This mode uses the kernel’s :code:`seccomp(2)` system call filtering to intercept certain privileged system calls, do absolutely nothing, and return success to the program. Some system calls are quashed regardless of their arguments: :code:`capset(2)`; :code:`chown(2)` and friends; :code:`kexec_load(2)` (used to validate the filter itself); ; and :code:`setuid(2)`, :code:`setgid(2)`, and :code:`setgroups(2)` along with the other system calls that change user or group. :code:`mknod(2)` and :code:`mknodat(2)` are quashed if they try to create a device file (e.g., creating FIFOs works normally). The advantages of this approach is that it’s much simpler, it’s faster, it’s completely agnostic to libc, and it’s mostly agnostic to distribution. The disadvantage is that it’s a very lazy liar; even the most cursory consistency checks will fail, e.g., :code:`getuid(2)` after :code:`setuid(2)`. While this mode does not provide consistency, it does offer a hook to help prevent programs asking for consistency. For example, :code:`apt-get -o APT::Sandbox::User=root` will prevent :code:`apt-get` from attempting to drop privileges, which `it verifies `_, exiting with failure if the correct IDs are not found (which they won’t be under this approach). This can be expressed with :code:`--force-cmd=apt-get,-o,APT::Sandbox::User=root`, though this particular case is built-in and does not need to be specified. The full default configuration, which is applied regardless of the image distribution, can be examined in the source file :code:`force.py`. If any :code:`--force-cmd` are specified, this replaces (rather than extends) the default configuration. Note that because the substitutions are a simple regex with no knowledge of shell syntax, they can cause unwanted modifications. For example, :code:`RUN apt-get install -y apt-get` will be run as :code:`/bin/sh -c "apt-get -o APT::Sandbox::User=root install -y apt-get -o APT::Sandbox::User=root"`. One workaround is to add escape syntax transparent to the shell; e.g., :code:`RUN apt-get install -y a\pt-get`. This mode executes *all* :code:`RUN` instructions with the :code:`seccomp(2)` filter and has no knowledge of which instructions actually used the intercepted system calls. Therefore, the printed “instructions modified” number is only a count of instructions with a hook applied as described above. :code:`RUN` logging ~~~~~~~~~~~~~~~~~~~~ In terminal output, image metadata, and the build cache, the :code:`RUN` instruction is always logged as :code:`RUN.S`, :code:`RUN.F`, or :code:`RUN.N`. The letter appended to the instruction reflects the root emulation mode used during the build in which the instruction was executed. :code:`RUN.S` indicates :code:`seccomp`, :code:`RUN.F` indicates :code:`fakeroot`, and :code:`RUN.N` indicates that neither form of root emulation was used (:code:`--force=none`). Compatibility and behavior differences -------------------------------------- :code:`ch-image` is an independent implementation and shares no code with other Dockerfile interpreters. It uses a formal Dockerfile parsing grammar developed from the `Dockerfile reference documentation `_ and miscellaneous other sources, which you can examine in the source code. We believe this independence is valuable for several reasons. First, it helps the community examine Dockerfile syntax and semantics critically, think rigorously about what is really needed, and build a more robust standard. Second, it yields disjoint sets of bugs (note that Podman, Buildah, and Docker all share the same Dockerfile parser). Third, because it is a much smaller code base, it illustrates how Dockerfiles work more clearly. Finally, it allows straightforward extensions if needed to support scientific computing. :code:`ch-image` tries hard to be compatible with Docker and other interpreters, though as an independent implementation, it is not bug-compatible. The following subsections describe differences from the Dockerfile reference that we expect to be approximately permanent. For not-yet-implemented features and bugs in this area, see `related issues `_ on GitHub. None of these are set in stone. We are very interested in feedback on our assessments and open questions. This helps us prioritize new features and revise our thinking about what is needed for HPC containers. Context directory ~~~~~~~~~~~~~~~~~ The context directory is bind-mounted into the build, rather than copied like Docker. Thus, the size of the context is immaterial, and the build reads directly from storage like any other local process would (i.e., it is reasonable use :code:`/` for the context). However, you still can’t access anything outside the context directory. Variable substitution ~~~~~~~~~~~~~~~~~~~~~ Variable substitution happens for *all* instructions, not just the ones listed in the Dockerfile reference. :code:`ARG` and :code:`ENV` cause cache misses upon *definition*, in contrast with Docker where these variables miss upon *use*, except for certain cache-excluded variables that never cause misses, listed below. Note that :code:`ARG` and :code:`ENV` have different syntax despite very similar semantics. :code:`ch-image` passes the following proxy environment variables in to the build. Changes to these variables do not cause a cache miss. They do not require an :code:`ARG` instruction, as `documented `_ in the Dockerfile reference. Unlike Docker, they are available if the same-named environment variable is defined; :code:`--build-arg` is not required. .. code-block:: sh HTTP_PROXY http_proxy HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy NO_PROXY no_proxy In addition to those listed in the Dockerfile reference, these environment variables are passed through in the same way: .. code-block:: sh SSH_AUTH_SOCK USER Finally, these variables are also pre-defined but are unrelated to the host environment: .. code-block:: sh PATH=/ch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TAR_OPTIONS=--no-same-owner :code:`ARG` ~~~~~~~~~~~ Variables set with :code:`ARG` are available anywhere in the Dockerfile, unlike Docker, where they only work in :code:`FROM` instructions, and possibly in other :code:`ARG` before the first :code:`FROM`. :code:`FROM` ~~~~~~~~~~~~ The :code:`FROM` instruction accepts option :code:`--arg=NAME=VALUE`, which serves the same purpose as the :code:`ARG` instruction. It can be repeated. :code:`LABEL` ~~~~~~~~~~~~~ The :code:`LABEL` instruction accepts :code:`key=value` pairs to add metadata for an image. Unlike Docker, multiline values are not supported; see issue `#1512 `_. Can be repeated. :code:`COPY` ~~~~~~~~~~~~ .. note:: The behavior described here matches Docker’s `now-deprecated legacy builder `_. Docker’s new builder, BuildKit, has different behavior in some cases, which we have not characterized. Especially for people used to UNIX :code:`cp(1)`, the semantics of the Dockerfile :code:`COPY` instruction can be confusing. Most notably, when a source of the copy is a directory, the *contents* of that directory, not the directory itself, are copied. This is documented, but it’s a real gotcha because that’s not what :code:`cp(1)` does, and it means that many things you can do in one :code:`cp(1)` command require multiple :code:`COPY` instructions. Also, the reference documentation is incomplete. In our experience, Docker also behaves as follows; :code:`ch-image` does the same in an attempt to be bug-compatible. 1. You can use absolute paths in the source; the root is the context directory. 2. Destination directories are created if they don’t exist in the following situations: 1. If the destination path ends in slash. (Documented.) 2. If the number of sources is greater than 1, either by wildcard or explicitly, regardless of whether the destination ends in slash. (Not documented.) 3. If there is a single source and it is a directory. (Not documented.) 3. Symbolic links behave differently depending on how deep in the copied tree they are. (Not documented.) 1. Symlinks at the top level — i.e., named as the destination or the source, either explicitly or by wildcards — are dereferenced. They are followed, and whatever they point to is used as the destination or source, respectively. 2. Symlinks at deeper levels are not dereferenced, i.e., the symlink itself is copied. 4. If a directory appears at the same path in source and destination, and is at the 2nd level or deeper, the source directory’s metadata (e.g., permissions) are copied to the destination directory. (Not documented.) 5. If an object (a) appears in both the source and destination, (b) is at the 2nd level or deeper, and (c) is different file types in source and destination, the source object will overwrite the destination object. (Not documented.) We expect the following differences to be permanent: * Wildcards use Python glob semantics, not the Go semantics. * :code:`COPY --chown` is ignored, because it doesn’t make sense in an unprivileged build. Features we do not plan to support ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Parser directives are not supported. We have not identified a need for any of them. * :code:`EXPOSE`: Charliecloud does not use the network namespace, so containerized processes can simply listen on a host port like other unprivileged processes. * :code:`HEALTHCHECK`: This instruction’s main use case is monitoring server processes rather than applications. Also, it requires a container supervisor daemon, which we have no plans to add. * :code:`MAINTAINER` is deprecated. * :code:`STOPSIGNAL` requires a container supervisor daemon process, which we have no plans to add. * :code:`USER` does not make sense for unprivileged builds. * :code:`VOLUME`: Charliecloud has good support for bind mounts; we anticipate that it will continue to focus on that and will not introduce the volume management features that Docker has. .. _ch-image_rsync: :code:`RSYNC` (Dockerfile extension) ------------------------------------ .. warning:: This instruction is experimental and may change or be removed. Overview ~~~~~~~~ Copying files is often simple but has numerous difficult corner cases, e.g. when dealing with symbolic or hard links. The standard instruction :code:`COPY` deals with many of these corner cases differently from other UNIX utilities, lacks complete documentation, and behaves inconsistently between different Dockerfile interpreters (e.g., Docker’s legacy builder vs. BuildKit), as detailed above. On the other hand, :code:`rsync(1)` is an extremely capable, widely used file copy tool, with detailed options to specify behavior and 25 years of history dealing with weirdness. :code:`RSYNC` (also spelled :code:`NSYNC`) is a Charliecloud extension that gives copying behavior identical to :code:`rsync(1)`. In fact, Charliecloud’s current implementation literally calls the host’s :code:`rsync(1)` to do the copy, though this may change in the future. There is no list form of :code:`RSYNC`. The two key usage challenges are trailing slashes on paths and symlink handling. In particular, the default symlink handling seemed reasonable to us, but you may want something different. See the arguments and examples below. Importantly, :code:`COPY` is not any less fraught, and you have no choice about what to do with symlinks. Arguments ~~~~~~~~~ :code:`RSYNC` takes the same arguments as :code:`rsync(1)`, so refer to its `man page `_ for a detailed explanation of all the options (with possible emphasis on its `symlink options `_). Sources are relative to the context directory even if they look absolute with a leading slash. Any globbed sources are processed by :code:`ch-image(1)` using Python rules, i.e., :code:`rsync(1)` sees the expanded sources with no wildcards. Relative destinations are relative to the image’s current working directory, while absolute destinations refer to the image’s root. For arguments that read input from a file (e.g. :code:`--exclude-from` or :code:`--files-from`), relative paths are relative to the context directory, absolute paths refer to the image root, and :code:`-` (standard input) is an error. For example, .. code-block:: docker WORKDIR /foo RSYNC --foo src1 src2 dst is translated to (the equivalent of):: $ mkdir -p /foo $ rsync -@=-1 -AHSXpr --info=progress2 -l --safe-links \ --foo /context/src1 /context/src2 /storage/imgroot/foo/dst2 Note the extensive default arguments to :code:`rsync(1)`. :code:`RSYNC` takes a single instruction option beginning with :code:`+` (plus) that is shorthand for a group of :code:`rsync(1)` options. This single option is one of: :code:`+m` Preserves metadata and directory structure. Symlinks are skipped *with a warning*. Equivalent to all of: * :code:`-@=-1`: use nanosecond precision when comparing timestamps. * :code:`-A`: preserve ACLs. * :code:`-H`: preserve hard link groups. * :code:`-S`: preserve file sparseness when possible. * :code:`-X`: preserve xattrs in :code:`user.*` namespace. * :code:`-p`: preserve permissions. * :code:`-r`: recurse into directories. * :code:`--info=progress2` (only if stderr is a terminal): show progress meter (note `subtleties in interpretation `_). :code:`+l` (default) Like :code:`+u`, but *silently skips* “unsafe” symlinks whose target is outside the top-of-transfer directory. Preserves: * Metadata. * Directory structure. * Symlinks, if a link’s target is within the “top-of-transfer directory”. This is not the context directory and often not the source either. Also, this creates broken symlinks if the target is not within the source but is within the top-of-transfer. See examples below. Equivalent to the :code:`rsync(1)` options listed for :code:`+m` plus :code:`--links` (copy symlinks as symlinks unless otherwise specified) and :code:`--safe-links` (silently skip unsafe symlinks). :code:`+u` Like :code:`+l`, but *replaces* with their target “unsafe” symlinks whose target is outside the top-of-transfer directory, and thus *can copy data outside the context directory into the image*. Preserves: * Metadata. * Directory structure. * Symlinks, if a link’s target is within the “top-of-transfer directory”. This is not the context directory and often not the source either. Also, this creates broken symlinks if the target is not within the source but is within the top-of-transfer. See examples below. Equivalent to the :code:`rsync(1)` options listed for :code:`+m` plus :code:`--links` (copy symlinks as symlinks unless otherwise specified) and :code:`--copy-unsafe-links` (copy the target of unsafe symlinks). :code:`+z` No default arguments. Directories will not be descended, no metadata will be preserved, and both hard and symbolic links will be ignored, except as otherwise specified by :code:`rsync(1)` options starting with a hyphen. (Note that :code:`-a`/:code:`--archive` is discouraged because it omits some metadata and handles symlinks inappropriately for containers.) .. note:: :code:`rsync(1)` supports a configuration file :code:`~/.popt` that alters its command line processing. Currently, this configuration is respected for :code:`RSYNC` arguments, but that may change without notice. Disallowed :code:`rsync(1)` features ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A small number of :code:`rsync(1)` features are actively disallowed: 1. :code:`rsync:` and :code:`ssh:` transports are an error. Charliecloud needs access to the entire input to compute cache hit or miss, and these transports make that impossible. It is possible these will become available in the future (please let us know if that is your use case!). For now, the workaround is to install :code:`rsync(1)` in the image and use it in a :code:`RUN` instruction, though only the instruction text will be considered for the cache. 2. Option arguments must be delimited with :code:`=` (equals). For example, to set the block size to 4 MiB, you must say :code:`--block-size=4M` or :code:`-B=4M`. :code:`-B4M` will be interpreted as the three arguments :code:`-B`, :code:`-4`, and :code:`-M`; :code:`--block-size 4M` will be interpreted as :code:`--block-size` with no argument and a copy source named :code:`4M`. This is so Charliecloud can process :code:`rsync(1)` options without knowing which ones take an argument. 3. Invalid :code:`rsync(1)` options: :code:`--daemon` Running :code:`rsync(1)` in daemon mode does not make sense for container build. :code:`-n`, :code:`--dry-run` This makes the copy a no-op, and Charliecloud may want to use it internally in the future. :code:`--remove-source-files` This would let the instruction alter the context directory. Note that there are likely other flags that don’t make sense and/or cause undesirable behavior. We have not characterized this problem. Build cache ~~~~~~~~~~~ The instruction is a cache hit if the metadata of all source files is unchanged (specifically: filename, file type and permissions, xattrs, size, and last modified time). Unlike Docker, Charliecloud does not use file contents. This has two implications. First, it is possible to fool the cache by manually restoring the last-modified time. Second, :code:`RSYNC` is I/O-intensive even when it hits, because it must :code:`stat(2)` every source file before checking the cache. However, this is still less I/O than reading the file content too. Notably, Charliecloud’s cache ignores :code:`rsync(1)`’s own internal notion of whether anything would be transferred (e.g., :code:`rsync -ni`). This may change in the future. Examples and tutorial ~~~~~~~~~~~~~~~~~~~~~ All of these examples use the same input, whose content will be introduced gradually, using edited output of :code:`ls -oghR` (which is like :code:`ls -lhR` but omits user and group). Examples assume a umask of :code:`0007`. The Dockerfile instructions listed also assume a preceding: .. code-block:: docker FROM alpine:3.17 RUN mkdir /dst i.e., a simple base image containing a top-level directory :code:`dst`. Many additional examples are available in the source code in the file :code:`test/build/50_rsync.bats`. We begin by copying regular files. The context directory :code:`ctx` contains, in part, two directories containing one regular file each. Note that one of these files (:code:`file-basic1`) and one of the directories (:code:`basic1`) have strange permissions. :: ./ctx: drwx---r-x 2 60 Oct 11 13:20 basic1 drwxrwx--- 2 60 Oct 11 13:20 basic2 ./ctx/basic1: -rw----r-- 1 12 Oct 11 13:20 file-basic1 ./ctx/basic2: -rw-rw---- 1 12 Oct 11 13:20 file-basic2 The simplest form of :code:`RSYNC` is to copy a single file into a specified directory: .. code-block:: docker RSYNC /basic1/file-basic1 /dst resulting in:: $ ls -oghR dst dst: -rw----r-- 1 12 Oct 11 13:26 file-basic1 Note that :code:`file-basic1`’s metadata — here its odd permissions — are preserved. :code:`1` is the number of hard links to the file, and :code:`12` is the file size. One can also rename the destination by specifying a new file name, and with :code:`+z`, not copy metadata (from here on the :code:`ls` command is omitted for brevity): .. code-block:: docker RSYNC +z /basic1/file-basic1 /dst/file-basic1_nom :: dst: -rw------- 1 12 Sep 21 15:51 file-basic1_nom A trailing slash on the destination creates a new directory and places the source file within: .. code-block:: docker RSYNC /basic1/file-basic1 /dst/new/ :: dst: drwxrwx--- 1 22 Oct 11 13:26 new dst/new: -rw----r-- 1 12 Oct 11 13:26 file-basic1 With multiple source files, the destination trailing slash is optional: .. code-block:: docker RSYNC /basic1/file-basic1 /basic2/file-basic2 /dst/newB :: dst: drwxrwx--- 1 44 Oct 11 13:26 newB dst/newB: -rw----r-- 1 12 Oct 11 13:26 file-basic1 -rw-rw---- 1 12 Oct 11 13:26 file-basic2 For directory sources, the presence or absence of a trailing slash is highly significant. Without one, the directory itself is placed in the destination (recall that this would rename a source *file*): .. code-block:: docker RSYNC /basic1 /dst/basic1_new :: dst: drwxrwx--- 1 12 Oct 11 13:28 basic1_new dst/basic1_new: drwx---r-x 1 22 Oct 11 13:28 basic1 dst/basic1_new/basic1: -rw----r-- 1 12 Oct 11 13:28 file-basic1 A source trailing slash means copy the *contents of* a directory rather than the directory itself. Importantly, however, the directory’s metadata is copied to the destination directory. .. code-block:: docker RSYNC /basic1/ /dst/basic1_renamed :: dst: drwx---r-x 1 22 Oct 11 13:28 basic1_renamed dst/basic1_renamed: -rw----r-- 1 12 Oct 11 13:28 file-basic1 One gotcha is that :code:`RSYNC +z` is a no-op if the source is a directory: .. code-block:: docker RSYNC +z /basic1 /dst/basic1_newC :: dst: At least :code:`-r` is needed with :code:`+z` in this case: .. code-block:: docker RSYNC +z -r /basic1/ /dst/basic1_newD :: dst: drwx------ 1 22 Oct 11 13:28 basic1_newD dst/basic1_newD: -rw------- 1 12 Oct 11 13:28 file-basic1 Multiple source directories can be specified, including with wildcards. This example also illustrates that copies files are by default merged with content already existing in the image. .. code-block:: docker RUN mkdir /dst/dstC && echo file-dstC > /dst/dstC/file-dstC RSYNC /basic* /dst/dstC :: dst: drwxrwx--- 1 42 Oct 11 13:33 dstC dst/dstC: drwx---r-x 1 22 Oct 11 13:33 basic1 drwxrwx--- 1 22 Oct 11 13:33 basic2 -rw-rw---- 1 10 Oct 11 13:33 file-dstC dst/dstC/basic1: -rw----r-- 1 12 Oct 11 13:33 file-basic1 dst/dstC/basic2: -rw-rw---- 1 12 Oct 11 13:33 file-basic2 Trailing slashes can be specified independently for each source: .. code-block:: docker RUN mkdir /dst/dstF && echo file-dstF > /dst/dstF/file-dstF RSYNC /basic1 /basic2/ /dst/dstF :: dst: drwxrwx--- 1 52 Oct 11 13:33 dstF dst/dstF: drwx---r-x 1 22 Oct 11 13:33 basic1 -rw-rw---- 1 12 Oct 11 13:33 file-basic2 -rw-rw---- 1 10 Oct 11 13:33 file-dstF dst/dstF/basic1: -rw----r-- 1 12 Oct 11 13:33 file-basic1 Bare :code:`/` (i.e., the entire context directory) is considered to have a trailing slash: .. code-block:: docker RSYNC / /dst :: dst: drwx---r-x 1 22 Oct 11 13:33 basic1 drwxrwx--- 1 22 Oct 11 13:33 basic2 dst/basic1: -rw----r-- 1 12 Oct 11 13:33 file-basic1 dst/basic2: -rw-rw---- 1 12 Oct 11 13:33 file-basic2 To *replace* (rather than merge with) existing content, use :code:`--delete`. Note also that wildcards can be combined with trailing slashes and that the directory gets the metadata of the *first* slashed directory. .. code-block:: docker RUN mkdir /dst/dstG && echo file-dstG > /dst/dstG/file-dstG RSYNC --delete /basic*/ /dst/dstG :: dst: drwx---r-x 1 44 Oct 11 14:00 dstG dst/dstG: -rw----r-- 1 12 Oct 11 14:00 file-basic1 -rw-rw---- 1 12 Oct 11 14:00 file-basic2 Symbolic links in the source(s) add significant complexity. Like :code:`rsync(1)`, :code:`RSYNC` can do one of three things with a given symlink: 1. Ignore it, silently or with a warning. 2. Preserve it: copy as a symlink, with the same target. 3. Dereference it: copy the target instead. These actions are selected independently for *safe symlinks* and *unsafe symlinks*. Safe symlinks are those which point to a target within the *top of transfer*, which is the deepest directory in the source path with a trailing slash. For example, :code:`/foo/bar`’s top-of-transfer is :code:`/foo` (regardless of whether :code:`bar` is a directory or file), while :code:`/foo/bar/`’s top-of-transfer is :code:`/foo/bar`. For the symlink examples, the context contains two sub-directories with a variety of symlinks, as well as a sibling file and directory outside the context. All of these links are valid on the host. In this listing, the absolute path to the parent of the context directory is replaced with :code:`/...`. :: .: drwxrwx--- 9 200 Oct 11 14:00 ctx drwxrwx--- 2 60 Oct 11 14:00 dir-out -rw-rw---- 1 9 Oct 11 14:00 file-out ./ctx: drwxrwx--- 3 320 Oct 11 14:00 sym1 ./ctx/sym1: lrwxrwxrwx 1 13 Oct 11 14:00 dir-out_rel -> ../../dir-out drwxrwx--- 2 60 Oct 11 14:00 dir-sym1 lrwxrwxrwx 1 8 Oct 11 14:00 dir-sym1_direct -> dir-sym1 lrwxrwxrwx 1 10 Oct 11 14:00 dir-top_rel -> ../dir-top lrwxrwxrwx 1 47 Oct 11 14:00 file-out_abs -> /.../file-out lrwxrwxrwx 1 14 Oct 11 14:00 file-out_rel -> ../../file-out -rw-rw---- 1 10 Oct 11 14:00 file-sym1 lrwxrwxrwx 1 57 Oct 11 14:00 file-sym1_abs -> /.../ctx/sym1/file-sym1 lrwxrwxrwx 1 9 Oct 11 14:00 file-sym1_direct -> file-sym1 lrwxrwxrwx 1 17 Oct 11 14:00 file-sym1_upover -> ../sym1/file-sym1 lrwxrwxrwx 1 51 Oct 11 14:00 file-top_abs -> /.../ctx/file-top lrwxrwxrwx 1 11 Oct 11 14:00 file-top_rel -> ../file-top ./ctx/sym1/dir-sym1: -rw-rw---- 1 14 Oct 11 14:00 dir-sym1.file ./dir-out: -rw-rw---- 1 13 Oct 11 14:00 dir-out.file By default, safe symlinks are preserved while unsafe symlinks are silently ignored: .. code-block:: docker RSYNC /sym1 /dst :: dst: drwxrwx--- 1 206 Oct 11 17:10 sym1 dst/sym1: drwxrwx--- 1 26 Oct 11 17:10 dir-sym1 lrwxrwxrwx 1 8 Oct 11 17:10 dir-sym1_direct -> dir-sym1 lrwxrwxrwx 1 10 Oct 11 17:10 dir-top_rel -> ../dir-top -rw-rw---- 1 10 Oct 11 17:10 file-sym1 lrwxrwxrwx 1 9 Oct 11 17:10 file-sym1_direct -> file-sym1 lrwxrwxrwx 1 17 Oct 11 17:10 file-sym1_upover -> ../sym1/file-sym1 lrwxrwxrwx 1 17 Oct 11 17:10 file-sym2_upover -> ../sym2/file-sym2 lrwxrwxrwx 1 11 Oct 11 17:10 file-top_rel -> ../file-top dst/sym1/dir-sym1: -rw-rw---- 1 14 Oct 11 17:10 dir-sym1.file The source files have four rough fates: 1. Regular files and directories (:code:`file-sym1` and :code:`dir-sym1`). These are copied into the image unchanged, including metadata. 2. Safe symlinks, now broken. This is one of the gotchas of :code:`RSYNC`’s top-of-transfer directory (here host path :code:`./ctx`, image path :code:`/`) differing from the source directory (:code:`./ctx/sym1`, :code:`/sym1`), because the latter lacks a trailing slash. :code:`dir-top_rel`, :code:`file-sym2_upover`, and :code:`file-top_rel` all ascend only as high as :code:`./ctx` (host path, :code:`/` image) before re-descending. This is within the top-of-transfer, so the symlinks are safe and thus copied unchanged, but their targets were not included in the copy. 3. Safe symlinks, still valid. 1. :code:`dir-sym1_direct` and :code:`file-sym1_direct` point directly to files in the same directory. 2. :code:`dir-sym1_upover` and :code:`file-sym1_upover` point to files in the same directory, but by first ascending into their parent — within the top-of-transfer, so they are safe — and then re-descending. If :code:`sym1` were renamed during the copy, these links would break. 4. Unsafe symlinks, which are ignored by the copy and do not appear in the image. 1. Absolute symlinks are always unsafe (:code:`*_abs`). 2. :code:`dir-out_rel` and :code:`file-out_rel` are relative symlinks that ascend above the top-of-transfer, in this case to targets outside the context, and are thus unsafe. The top-of-transfer can be changed to :code:`sym1` with a trailing slash. This also adds :code:`sym1` to the destination so the resulting directory structure is the same. .. code-block:: docker RSYNC /sym1/ /dst/sym1 :: dst: drwxrwx--- 1 96 Oct 11 17:10 sym1 dst/sym1: drwxrwx--- 1 26 Oct 11 17:10 dir-sym1 lrwxrwxrwx 1 8 Oct 11 17:10 dir-sym1_direct -> dir-sym1 -rw-rw---- 1 10 Oct 11 17:10 file-sym1 lrwxrwxrwx 1 9 Oct 11 17:10 file-sym1_direct -> file-sym1 dst/sym1/dir-sym1: -rw-rw---- 1 14 Oct 11 17:10 dir-sym1.file :code:`*_upover` and :code:`*-out_rel` are now unsafe and replaced with their targets. Another common use case is to follow unsafe symlinks and copy their targets in place of the links. This is accomplished with :code:`+u`: .. code-block:: docker RSYNC +u /sym1/ /dst/sym1 :: dst: drwxrwx--- 1 352 Oct 11 17:10 sym1 dst/sym1: drwxrwx--- 1 24 Oct 11 17:10 dir-out_rel drwxrwx--- 1 26 Oct 11 17:10 dir-sym1 lrwxrwxrwx 1 8 Oct 11 17:10 dir-sym1_direct -> dir-sym1 drwxrwx--- 1 24 Oct 11 17:10 dir-top_rel -rw-rw---- 1 9 Oct 11 17:10 file-out_abs -rw-rw---- 1 9 Oct 11 17:10 file-out_rel -rw-rw---- 1 10 Oct 11 17:10 file-sym1 -rw-rw---- 1 10 Oct 11 17:10 file-sym1_abs lrwxrwxrwx 1 9 Oct 11 17:10 file-sym1_direct -> file-sym1 -rw-rw---- 1 10 Oct 11 17:10 file-sym1_upover -rw-rw---- 1 10 Oct 11 17:10 file-sym2_abs -rw-rw---- 1 10 Oct 11 17:10 file-sym2_upover -rw-rw---- 1 9 Oct 11 17:10 file-top_abs -rw-rw---- 1 9 Oct 11 17:10 file-top_rel dst/sym1/dir-out_rel: -rw-rw---- 1 13 Oct 11 17:10 dir-out.file dst/sym1/dir-sym1: -rw-rw---- 1 14 Oct 11 17:10 dir-sym1.file dst/sym1/dir-top_rel: -rw-rw---- 1 13 Oct 11 17:10 dir-top.file Now all the unsafe symlinks noted above are present in the image, but they have changed to the normal files and directories pointed to. .. warning:: This feature lets you copy files outside the context into the image, unlike other container builders where :code:`COPY` can never access anything outside the context. The sources themselves, if symlinks, do not get special treatment: .. code-block:: docker RSYNC /sym1/file-sym1_direct /sym1/file-sym1_upover /dst :: dst: lrwxrwxrwx 1 9 Oct 11 17:10 file-sym1_direct -> file-sym1 Note that :code:`file-sym1_upover` does not appear in the image, despite being named explicitly in the instruction, because it is an unsafe symlink. If the *destination* is a symlink to a file, and the source is a file, the link is replaced and the target is unchanged. (If the source is a directory, that is an error.) .. code-block:: docker RUN touch /dst/file-dst && ln -s file-dst /dst/file-dst_direct RSYNC /file-top /dst/file-dst_direct :: dst: -rw-rw---- 1 0 Oct 11 17:42 file-dst -rw-rw---- 1 9 Oct 11 17:42 file-dst_direct If the destination is a symlink to a directory, the link is followed: .. code-block:: docker RUN mkdir /dst/dir-dst && ln -s dir-dst /dst/dir-dst_direct RSYNC /file-top /dst/dir-dst_direct :: dst: drwxrwx--- 1 16 Oct 11 17:50 dir-dst lrwxrwxrwx 1 7 Oct 11 17:50 dir-dst_direct -> dir-dst dst/dir-dst: -rw-rw---- 1 9 Oct 11 17:50 file-top Examples -------- Build image :code:`bar` using :code:`./foo/bar/Dockerfile` and context directory :code:`./foo/bar`:: $ ch-image build -t bar -f ./foo/bar/Dockerfile ./foo/bar [...] grown in 4 instructions: bar Same, but infer the image name and Dockerfile from the context directory path:: $ ch-image build ./foo/bar [...] grown in 4 instructions: bar Build using humongous vendor compilers you want to bind-mount instead of installing into the image:: $ ch-image build --bind /opt/bigvendor:/opt . $ cat Dockerfile FROM centos:7 RUN /opt/bin/cc hello.c #COPY /opt/lib/*.so /usr/local/lib # fail: COPY doesn’t bind mount RUN cp /opt/lib/*.so /usr/local/lib # possible workaround RUN ldconfig :code:`build-cache` =================== :: $ ch-image [...] build-cache [...] Print basic information about the cache. If :code:`-v` is given, also print some Git statistics and the Git repository configuration. If any of the following options are given, do the corresponding operation before printing. Multiple options can be given, in which case they happen in this order. :code:`--dot` Create a DOT export of the tree named :code:`./build-cache.dot` and a PDF rendering :code:`./build-cache.pdf`. Requires :code:`graphviz` and :code:`git2dot`. :code:`--gc` Run Git garbage collection on the cache, including full de-duplication of similar files. This will immediately remove all cache entries not currently reachable from a named branch (which is likely to cause corruption if the build cache is being accessed concurrently by another process). The operation can take a long time on large caches. :code:`--reset` Clear and re-initialize the build cache. :code:`--tree` Print a text tree of the cache using Git’s :code:`git log --graph` feature. If :code:`-v` is also given, the tree has more detail. :code:`delete` ============== :: $ ch-image [...] delete IMAGE_GLOB [IMAGE_GLOB ... ] Delete the image(s) described by each :code:`IMAGE_GLOB` from the storage directory (including all build stages). :code:`IMAGE_GLOB` can be either a plain image reference or an image reference with glob characters to match multiple images. For example, :code:`ch-image delete 'foo*'` will delete all images whose names start with :code:`foo`. Multiple images and/or globs can also be given in a single command line. Importantly, this sub-command *does not* also remove the image from the build cache. Therefore, it can be used to reduce the size of the storage directory, trading off the time needed to retrieve an image from cache. .. warning:: Glob characters must be quoted or otherwise protected from the shell, which also desires to interpret them and will do so incorrectly. :code:`gestalt` =============== :: $ ch-image [...] gestalt [SELECTOR] Provide information about the `configuration and available features `_ of :code:`ch-image`. End users generally will not need this; it is intended for testing and debugging. :code:`SELECTOR` is one of: * :code:`bucache`. Exit successfully if the build cache is available, unsuccessfully with an error message otherwise. With :code:`-v`, also print version information about dependencies. * :code:`bucache-dot`. Exit successfully if build cache DOT trees can be written, unsuccessfully with an error message otherwise. With :code:`-v`, also print version information about dependencies. * :code:`python-path`. Print the path to the Python interpreter in use and exit successfully. * :code:`storage-path`. Print the storage directory path and exit successfully. :code:`list` ============ Print information about images. If no argument given, list the images in builder storage. Synopsis -------- :: $ ch-image [...] list [-l] [IMAGE_REF] Description ----------- Optional argument: :code:`-l`, :code:`--long` Use long format (name, last change timestamp) when listing images. :code:`-u`, :code:`--undeletable` List images that can be undeleted. Can also be spelled :code:`--undeleteable`. :code:`IMAGE_REF` Print details of what’s known about :code:`IMAGE_REF`, both locally and in the remote registry, if any. Examples -------- List images in builder storage:: $ ch-image list alpine:3.17 (amd64) alpine:latest (amd64) debian:buster (amd64) Print details about Debian Buster image:: $ ch-image list debian:buster details of image: debian:buster in local storage: no full remote ref: registry-1.docker.io:443/library/debian:buster available remotely: yes remote arch-aware: yes host architecture: amd64 archs available: 386 bae2738ed83 amd64 98285d32477 arm/v7 97247fd4822 arm64/v8 122a0342878 For remotely available images like Debian Buster, the associated digest is listed beside each available architecture. Importantly, this feature does *not* provide the hash of the local image, which is only calculated on push. :code:`import` ============== :: $ ch-image [...] import PATH IMAGE_REF Copy the image at :code:`PATH` into builder storage with name :code:`IMAGE_REF`. :code:`PATH` can be: * an image directory * a tarball with no top-level directory (a.k.a. a "`tarbomb `_") * a standard tarball with one top-level directory If the imported image contains Charliecloud metadata, that will be imported unchanged, i.e., images exported from :code:`ch-image` builder storage will be functionally identical when re-imported. .. warning:: Descendant images (i.e., :code:`FROM` the imported :code:`IMAGE_REF`) are linked using :code:`IMAGE_REF` only. If a new image is imported under a new :code:`IMAGE_REF`, all instructions descending from that :code:`IMAGE_REF` will still hit, even if the new image is different. :code:`pull` ============ Pull the image described by the image reference :code:`IMAGE_REF` from a repository to the local filesystem. Synopsis -------- :: $ ch-image [...] pull [...] IMAGE_REF [DEST_REF] See the FAQ for the gory details on specifying image references. Description ----------- Destination: :code:`DEST_REF` If specified, use this as the destination image reference, rather than :code:`IMAGE_REF`. This lets you pull an image with a complicated reference while storing it locally with a simpler one. Options: :code:`--last-layer N` Unpack only :code:`N` layers, leaving an incomplete image. This option is intended for debugging. :code:`--parse-only` Parse :code:`IMAGE_REF`, print a parse report, and exit successfully without talking to the internet or touching the storage directory. This script does a fair amount of validation and fixing of the layer tarballs before flattening in order to support unprivileged use despite image problems we frequently see in the wild. For example, device files are ignored, and file and directory permissions are increased to a minimum of :code:`rwx------` and :code:`rw-------` respectively. Note, however, that symlinks pointing outside the image are permitted, because they are not resolved until runtime within a container. The following metadata in the pulled image is retained; all other metadata is currently ignored. (If you have a need for additional metadata, please let us know!) * Current working directory set with :code:`WORKDIR` is effective in downstream Dockerfiles. * Environment variables set with :code:`ENV` are effective in downstream Dockerfiles and also written to :code:`/ch/environment` for use in :code:`ch-run --set-env`. * Mount point directories specified with :code:`VOLUME` are created in the image if they don’t exist, but no other action is taken. Note that some images (e.g., those with a "version 1 manifest") do not contain metadata. A warning is printed in this case. Examples -------- Download the Debian Buster image matching the host’s architecture and place it in the storage directory:: $ uname -m aarch32 pulling image: debian:buster requesting arch: arm64/v8 manifest list: downloading manifest: downloading config: downloading layer 1/1: c54d940: downloading flattening image layer 1/1: c54d940: listing validating tarball members resolving whiteouts layer 1/1: c54d940: extracting image arch: arm64 done Same, specifying the architecture explicitly:: $ ch-image --arch=arm/v7 pull debian:buster pulling image: debian:buster requesting arch: arm/v7 manifest list: downloading manifest: downloading config: downloading layer 1/1: 8947560: downloading flattening image layer 1/1: 8947560: listing validating tarball members resolving whiteouts layer 1/1: 8947560: extracting image arch: arm (may not match host arm64/v8) :code:`push` ============ Push the image described by the image reference :code:`IMAGE_REF` from the local filesystem to a repository. Synopsis -------- :: $ ch-image [...] push [--image DIR] IMAGE_REF [DEST_REF] See the FAQ for the gory details on specifying image references. Description ----------- Destination: :code:`DEST_REF` If specified, use this as the destination image reference, rather than :code:`IMAGE_REF`. This lets you push to a repository without permanently adding a tag to the image. Options: :code:`--image DIR` Use the unpacked image located at :code:`DIR` rather than an image in the storage directory named :code:`IMAGE_REF`. Because Charliecloud is fully unprivileged, the owner and group of files in its images are not meaningful in the broader ecosystem. Thus, when pushed, everything in the image is flattened to user:group :code:`root:root`. Also, setuid/setgid bits are removed, to avoid surprises if the image is pulled by a privileged container implementation. Examples -------- Push a local image to the registry :code:`example.com:5000` at path :code:`/foo/bar` with tag :code:`latest`. Note that in this form, the local image must be named to match that remote reference. :: $ ch-image push example.com:5000/foo/bar:latest pushing image: example.com:5000/foo/bar:latest layer 1/1: gathering layer 1/1: preparing preparing metadata starting upload layer 1/1: a1664c4: checking if already in repository layer 1/1: a1664c4: not present, uploading config: 89315a2: checking if already in repository config: 89315a2: not present, uploading manifest: uploading cleaning up done Same, except use local image :code:`alpine:3.17`. In this form, the local image name does not have to match the destination reference. :: $ ch-image push alpine:3.17 example.com:5000/foo/bar:latest pushing image: alpine:3.17 destination: example.com:5000/foo/bar:latest layer 1/1: gathering layer 1/1: preparing preparing metadata starting upload layer 1/1: a1664c4: checking if already in repository layer 1/1: a1664c4: not present, uploading config: 89315a2: checking if already in repository config: 89315a2: not present, uploading manifest: uploading cleaning up done Same, except use unpacked image located at :code:`/var/tmp/image` rather than an image in :code:`ch-image` storage. (Also, the sole layer is already present in the remote registry, so we don’t upload it again.) :: $ ch-image push --image /var/tmp/image example.com:5000/foo/bar:latest pushing image: example.com:5000/foo/bar:latest image path: /var/tmp/image layer 1/1: gathering layer 1/1: preparing preparing metadata starting upload layer 1/1: 892e38d: checking if already in repository layer 1/1: 892e38d: already present config: 546f447: checking if already in repository config: 546f447: not present, uploading manifest: uploading cleaning up done :code:`reset` ============= :: $ ch-image [...] reset Delete all images and cache from ch-image builder storage. :code:`undelete` ================ :: $ ch-image [...] undelete IMAGE_REF If :code:`IMAGE_REF` has been deleted but is in the build cache, recover it from the cache. Only available when the cache is enabled, and will not overwrite :code:`IMAGE_REF` if it exists. Environment variables ===================== :code:`CH_IMAGE_USERNAME`, :code:`CH_IMAGE_PASSWORD` Username and password for registry authentication. **See important caveats in section "Authentication" above.** .. include:: py_env.rst .. include:: ./bugs.rst .. include:: ./see_also.rst .. LocalWords: tmpfs'es bigvendor AUTH auth bucache buc bigfile df rfc bae .. LocalWords: dlcache graphviz packfile packfiles bigFileThreshold fd Tpdf .. LocalWords: pstats gprof chofile cffd cacdb ARGs NSYNC dst imgroot popt .. LocalWords: globbed ni AHSXpr drwxrwx ctx sym nom newB newC newD dstC .. LocalWords: dstB dstF dstG upover drwx kexec pdb charliecloud-0.37/doc/ch-run-oci.rst000066400000000000000000000132151457016721300173540ustar00rootroot00000000000000:code:`ch-run-oci` ++++++++++++++++++ .. only:: not man OCI wrapper for :code:`ch-run`. Synopsis ======== :: $ ch-run-oci OPERATION [ARG ...] Description =========== .. note:: This command is experimental. Features may be incomplete and/or buggy. The quality of code is not yet up to the usual Charliecloud standards, and error handling is poor. Please report any issues you find, so we can fix them! Open Containers Initiative (OCI) wrapper for :code:`ch-run(1)`. You probably don’t want to run this command directly; it is intended to interface with other software that expects an OCI runtime. The current goal is to support completely unprivileged image building (e.g. :code:`buildah --runtime=ch-run-oci`) rather than general OCI container running. *Support of the OCI runtime specification is only partial.* This is for two reasons. First, it’s an experimental and incomplete feature. More importantly, the philosophy and goals of OCI differ significantly from those of Charliecloud. Key differences include: * OCI is designed to run services, while Charliecloud is designed to run scientific applications. * OCI containers are persistent things with a complex lifecycle, while Charliecloud containers are simply UNIX processes. * OCI expects support for a variety of namespaces, while Charliecloud supports user and mount, no more and no less. * OCI expects runtimes to maintain a supervisor process in addition to user processes; Charliecloud has no need for this. * OCI expects runtimes to maintain state throughout the container lifecycle in a location independent from the caller. For these reasons, :code:`ch-run-oci` is a bit of a kludge, and much of what it does is provide scaffolding to satisfy OCI requirements. Which OCI features are and are not supported is provided in the rest of this man page, and technical analysis and discussion are in the Contributor’s Guide. This command supports OCI version 1.0.0 only and fails with an error if other versions are offered. Operations ========== All OCI operations are accepted, but some are no-ops or merely scaffolding to satisfy the caller. For comparison, see also: * `OCI runtime and lifecycle spec `_ * The `runc man pages `_ :code:`create` -------------- :: $ ch-run-oci create --bundle DIR --pid-file FILE [--no-new-keyring] CONTAINER_ID Create a container. Charliecloud does not have separate create and start phases, so this operation only sets up OCI-related scaffolding. Arguments: :code:`--bundle DIR` Directory containing the OCI bundle. This must be :code:`/tmp/buildahYYY`, where :code:`YYY` matches :code:`CONTAINER_ID` below. :code:`--pid-file FILE` Filename to write the "container" process PID to. Note that for Charliecloud, the process given is fake; see above. This must be :code:`DIR/pid`, where :code:`DIR` is given by :code:`--bundle`. :code:`--no-new-keyring` Ignored. (Charliecloud does not implement session keyrings.) :code:`CONTAINER_ID` String to use as the container ID. This must be :code:`buildah-buildahYYY`, where :code:`YYY` matches :code:`DIR` above. Unsupported arguments: :code:`--console-socket PATH` UNIX socket to pass pseudoterminal file descriptor. Charliecloud does not support pseudoterminals; fail with an error if this argument is given. For Buildah, redirect its input from :code:`/dev/null` to prevent it from requesting a pseudoterminal. :code:`delete` -------------- :: $ ch-run-oci delete CONTAINER_ID Clean up the OCI-related scaffolding for specified container. :code:`kill` ------------ :: $ ch-run-oci kill CONTAINER_ID No-op. :code:`start` ------------- :: $ ch-run-oci start CONTAINER_ID Eexecute the user command specified at create time in a Charliecloud container. :code:`state` ------------- :: $ ch-run-oci state CONTAINER_ID Print the state of the given container on standard output as an OCI compliant JSON document. Unsupported OCI features ======================== As noted above, various OCI features are not supported by Charliecloud. We have tried to guess which features would be essential to callers; :code:`ch-run-oci` fails with an error if these are requested. Otherwise, the request is simply ignored. We are interested in hearing about scientific-computing use cases for unsupported features, so we can add support for things that are needed. Our goal is for this man page to be comprehensive: every OCI runtime feature should either work or be listed as unsupported. Unsupported features that are an error: * Pseudoterminals * Hooks (prestart, poststart, and prestop) * Annotations * Joining existing namespaces * Intel Resource Director Technology (RDT) Unsupported features that are ignored: * Mounts other than the root filesystem * User/group mappings beyond one user mapped to EUID and one group mapped to EGID * Disabling :code:`prctl(PR_SET_NO_NEW_PRIVS)` * Root filesystem propagation mode * :code:`sysctl` directives * masked and read-only paths (remaining unprivileged protects you) * Capabilities * rlimits * Devices (all devices are inherited from the host) * cgroups * seccomp * SELinux * AppArmor * Container hostname setting Environment variables ===================== .. include:: py_env.rst :code:`CH_RUN_OCI_HANG` If set to the name of a command (e.g., :code:`create`), sleep indefinitely when that command is invoked. The purpose here is to halt a build so it can be examined and debugged. .. include:: ./bugs.rst .. include:: ./see_also.rst charliecloud-0.37/doc/ch-run.rst000066400000000000000000000730241457016721300166100ustar00rootroot00000000000000:code:`ch-run` ++++++++++++++ .. only:: not man Run a command in a Charliecloud container. Synopsis ======== :: $ ch-run [OPTION...] IMAGE -- COMMAND [ARG...] Description =========== Run command :code:`COMMAND` in a fully unprivileged Charliecloud container using the image specified by :code:`IMAGE`, which can be: (1) a path to a directory, (2) the name of an image in :code:`ch-image` storage (e.g. :code:`example.com:5050/foo`) or, if the proper support is enabled, a SquashFS archive. :code:`ch-run` does not use any setuid or setcap helpers, even for mounting SquashFS images with FUSE. :code:`-b`, :code:`--bind=SRC[:DST]` Bind-mount :code:`SRC` at guest :code:`DST`. The default destination if not specified is to use the same path as the host; i.e., the default is :code:`--bind=SRC:SRC`. Can be repeated. With a read-only image (the default), :code:`DST` must exist. However, if :code:`--write` or :code:`--write-fake` are given, :code:`DST` will be created as an empty directory (possibly with the tmpfs overmount trick described in :ref:`faq_mkdir-ro`). In this case, :code:`DST` must be entirely within the image itself, i.e., :code:`DST` cannot enter a previous bind mount. For example, :code:`--bind /foo:/tmp/foo` will fail because :code:`/tmp` is shared with the host via bind-mount (unless :code:`$TMPDIR` is set to something else or :code:`--private-tmp` is given). Most images have ten directories :code:`/mnt/[0-9]` already available as mount points. Symlinks in :code:`DST` are followed, and absolute links can have surprising behavior. Bind-mounting happens after namespace setup but before pivoting into the container image, so absolute links use the host root. For example, suppose the image has a symlink :code:`/foo -> /mnt`. Then, :code:`--bind=/bar:/foo` will bind-mount on the *host’s* :code:`/mnt`, which is inaccessible on the host because namespaces are already set up and *also* inaccessible in the container because of the subsequent pivot into the image. Currently, this problem is only detected when :code:`DST` needs to be created: :code:`ch-run` will refuse to follow absolute symlinks in this case, to avoid directory creation surprises. :code:`-c`, :code:`--cd=DIR` Initial working directory in container. :code:`--env-no-expand` Don’t expand variables when using :code:`--set-env`. :code:`--feature=FEAT` If feature :code:`FEAT` is enabled, exit with success. Valid values of :code:`FEAT` are :code:`extglob` for extended globs, :code:`seccomp` for :code:`seccomp(2)`, and :code:`squash` for squashfs archives. :code:`-g`, :code:`--gid=GID` Run as group :code:`GID` within container. :code:`--home` Bind-mount your host home directory (i.e., :code:`$HOME`) at guest :code:`/home/$USER`, hiding any existing image content at that path. Implies :code:`--write-fake` so the mount point can be created if needed. :code:`-j`, :code:`--join` Use the same container (namespaces) as peer :code:`ch-run` invocations. :code:`--join-pid=PID` Join the namespaces of an existing process. :code:`--join-ct=N` Number of :code:`ch-run` peers (implies :code:`--join`; default: see below). :code:`--join-tag=TAG` Label for :code:`ch-run` peer group (implies :code:`--join`; default: see below). :code:`-m`, :code:`--mount=DIR` Use :code:`DIR` for the SquashFS mount point, which must already exist. If not specified, the default is :code:`/var/tmp/$USER.ch/mnt`, which *will* be created if needed. :code:`--no-passwd` By default, temporary :code:`/etc/passwd` and :code:`/etc/group` files are created according to the UID and GID maps for the container and bind-mounted into it. If this is specified, no such temporary files are created and the image’s files are exposed. :code:`-q`, :code:`--quiet` Be quieter; can be repeated. Incompatible with :code:`-v`. See the :ref:`faq_verbosity` for details. :code:`-s`, :code:`--storage DIR` Set the storage directory. Equivalent to the same option for :code:`ch-image(1)`. :code:`--seccomp` Using seccomp, intercept some system calls that would fail due to lack of privilege, do nothing, and return fake success to the calling program. This is intended for use by :code:`ch-image(1)` when building images; see that man page for a detailed discussion. :code:`-t`, :code:`--private-tmp` By default, the host’s :code:`/tmp` (or :code:`$TMPDIR` if set) is bind-mounted at container :code:`/tmp`. If this is specified, a new :code:`tmpfs` is mounted on the container’s :code:`/tmp` instead. :code:`--set-env`, :code:`--set-env=FILE`, :code:`--set-env=VAR=VALUE` Set environment variables with newline-separated file (:code:`/ch/environment` within the image if not specified) or on the command line. See below for details. :code:`--set-env0`, :code:`--set-env0=FILE`, :code:`--set-env0=VAR=VALUE` Like :code:`--set-env`, but file is null-byte separated. :code:`-u`, :code:`--uid=UID` Run as user :code:`UID` within container. :code:`--unsafe` Enable various unsafe behavior. For internal use only. Seriously, stay away from this option. :code:`--unset-env=GLOB` Unset environment variables whose names match :code:`GLOB`. :code:`-v`, :code:`--verbose` Print extra chatter; can be repeated. See the :ref:`FAQ entry on verbosity ` for details. :code:`-w`, :code:`--write` Mount image read-write. By default, the image is mounted read-only. *This option should be avoided for most use cases,* because (1) changing images live (as opposed to prescriptively with a Dockerfile) destroys their provenance and (2) SquashFS images, which is the best-practice format on parallel filesystems, must be read-only. It is better to use :code:`--write-fake` (for disposable data) or bind-mount host directories (for retained data). :code:`-W`, :code:`--write-fake[=SIZE]` Overlay a writeable tmpfs on top of the image. This makes the image *appear* read-write, but it actually remains read-only and unchanged. All data “written” to the image are discarded when the container exits. The size of the writeable filesystem :code:`SIZE` is any size specification acceptable to :code:`tmpfs`, e.g. :code:`4m` for 4MiB or :code:`50%` for half of physical memory. If this option is specified without :code:`SIZE`, the default is :code:`12%`. Note (1) this limit is a maximum — only actually stored files consume virtual memory — and (2) :code:`SIZE` larger than memory can be requested without error (the failure happens later if the actual contents become too large). This requires kernel support and there are some caveats. See section “:ref:`ch-run_overlay`” below for details. :code:`-?`, :code:`--help` Print help and exit. :code:`--usage` Print a short usage message and exit. :code:`-V`, :code:`--version` Print version and exit. **Note:** Because :code:`ch-run` is fully unprivileged, it is not possible to change UIDs and GIDs within the container (the relevant system calls fail). In particular, setuid, setgid, and setcap executables do not work. As a precaution, :code:`ch-run` calls :code:`prctl(PR_SET_NO_NEW_PRIVS, 1)` to `disable these executables `_ within the container. This does not reduce functionality but is a "belt and suspenders" precaution to reduce the attack surface should bugs in these system calls or elsewhere arise. Image format ============ :code:`ch-run` supports two different image formats. The first is a simple directory that contains a Linux filesystem tree. This can be accomplished by: * :code:`ch-convert` directly from :code:`ch-image` or another builder to a directory. * Charliecloud’s tarball workflow: build or pull the image, :code:`ch-convert` it to a tarball, transfer the tarball to the target system, then :code:`ch-convert` the tarball to a directory. * Manually mount a SquashFS image, e.g. with :code:`squashfuse(1)` and then un-mount it after run with :code:`fusermount -u`. * Any other workflow that produces an appropriate directory tree. The second is a SquashFS image archive mounted internally by :code:`ch-run`, available if it’s linked with the optional :code:`libsquashfuse_ll` shared library. :code:`ch-run` mounts the image filesystem, services all FUSE requests, and unmounts it, all within :code:`ch-run`. See :code:`--mount` above to set the mount point location. Like other FUSE implementations, Charliecloud calls the :code:`fusermount3(1)` utility to mount the SquashFS filesystem. However, **this executable does not need to be installed setuid root**, and in fact :code:`ch-run` actively suppresses its setuid bit if set (using :code:`prctl(2)`). Prior versions of Charliecloud provided wrappers for the :code:`squashfuse` and :code:`squashfuse_ll` SquashFS mount commands and :code:`fusermount -u` unmount command. We removed these because we concluded they had minimal value-add over the standard, unwrapped commands. .. warning:: Currently, Charliecloud unmounts the SquashFS filesystem when user command :code:`COMMAND`’s process exits. It does not monitor any of its child processes. Therefore, if the user command spawns child processes and then exits before them (e.g., some daemons), those children will have the image unmounted from underneath them. In this case, the workaround is to mount/unmount using external tools. We expect to remove this limitation in a future version. Host files and directories available in container via bind mounts ================================================================= In addition to any directories specified by the user with :code:`--bind`, :code:`ch-run` has standard host files and directories that are bind-mounted in as well. The following host files and directories are bind-mounted at the same location in the container. These give access to the host’s devices and various kernel facilities. (Recall that Charliecloud provides minimal isolation and containerized processes are mostly normal unprivileged processes.) They cannot be disabled and are required; i.e., they must exist both on host and within the image. * :code:`/dev` * :code:`/proc` * :code:`/sys` Optional; bind-mounted only if path exists on both host and within the image, without error or warning if not. * :code:`/etc/hosts` and :code:`/etc/resolv.conf`. Because Charliecloud containers share the host network namespace, they need the same hostname resolution configuration. * :code:`/etc/machine-id`. Provides a unique ID for the OS installation; matching the host works for most situations. Needed to support D-Bus, some software licensing situations, and likely other use cases. See also `issue #1050 `_. * :code:`/var/lib/hugetlbfs` at guest :code:`/var/opt/cray/hugetlbfs`, and :code:`/var/opt/cray/alps/spool`. These support Cray MPI. Additional bind mounts done by default but can be disabled; see the options above. * :code:`$HOME` at :code:`/home/$USER` (and image :code:`/home` is hidden). Makes user data and init files available. * :code:`/tmp` (or :code:`$TMPDIR` if set) at guest :code:`/tmp`. Provides a temporary directory that persists between container runs and is shared with non-containerized application components. * temporary files at :code:`/etc/passwd` and :code:`/etc/group`. Usernames and group names need to be customized for each container run. Multiple processes in the same container with :code:`--join` ============================================================= By default, different :code:`ch-run` invocations use different user and mount namespaces (i.e., different containers). While this has no impact on sharing most resources between invocations, there are a few important exceptions. These include: 1. :code:`ptrace(2)`, used by debuggers and related tools. One can attach a debugger to processes in descendant namespaces, but not sibling namespaces. The practical effect of this is that (without :code:`--join`), you can’t run a command with :code:`ch-run` and then attach to it with a debugger also run with :code:`ch-run`. 2. *Cross-memory attach* (CMA) is used by cooperating processes to communicate by simply reading and writing one another’s memory. This is also not permitted between sibling namespaces. This affects various MPI implementations that use CMA to pass messages between ranks on the same node, because it’s faster than traditional shared memory. :code:`--join` is designed to address this by placing related :code:`ch-run` commands (the “peer group”) in the same container. This is done by one of the peers creating the namespaces with :code:`unshare(2)` and the others joining with :code:`setns(2)`. To do so, we need to know the number of peers and a name for the group. These are specified by additional arguments that can (hopefully) be left at default values in most cases: * :code:`--join-ct` sets the number of peers. The default is the value of the first of the following environment variables that is defined: :code:`OMPI_COMM_WORLD_LOCAL_SIZE`, :code:`SLURM_STEP_TASKS_PER_NODE`, :code:`SLURM_CPUS_ON_NODE`. * :code:`--join-tag` sets the tag that names the peer group. The default is environment variable :code:`SLURM_STEP_ID`, if defined; otherwise, the PID of :code:`ch-run`’s parent. Tags can be re-used for peer groups that start at different times, i.e., once all peer :code:`ch-run` have replaced themselves with the user command, the tag can be re-used. Caveats: * One cannot currently add peers after the fact, for example, if one decides to start a debugger after the fact. (This is only required for code with bugs and is thus an unusual use case.) * :code:`ch-run` instances race. The winner of this race sets up the namespaces, and the other peers use the winner to find the namespaces to join. Therefore, if the user command of the winner exits, any remaining peers will not be able to join the namespaces, even if they are still active. There is currently no general way to specify which :code:`ch-run` should be the winner. * If :code:`--join-ct` is too high, the winning :code:`ch-run`’s user command exits before all peers join, or :code:`ch-run` itself crashes, IPC resources such as semaphores and shared memory segments will be leaked. These appear as files in :code:`/dev/shm/` and can be removed with :code:`rm(1)`. * Many of the arguments given to the race losers, such as the image path and :code:`--bind`, will be ignored in favor of what was given to the winner. .. _ch-run_overlay: Writeable overlay with :code:`--write-fake` =========================================== If you need the image to stay read-only but appear writeable, you may be able to use :code:`--write-fake` to overlay a writeable tmpfs atop the image. This requires kernel support. Specifically: 1. To use the feature at all, you need unprivileged overlayfs support. This is available in `upstream 5.11 `_ (February 2021), but distributions vary considerably. If you don’t have this, the container will fail to start with error “operation not permitted”. 2. For a fully functional overlay, you need a tmpfs that supports xattrs in the :code:`user` namespace. This is available in `upstream 6.6 `_ (October 2023). If you don’t have this, most things will work fine, but some operations will fail with “I/O error”, for example creating a directory with the same path as a previously deleted directory. There will also be syslog noise about xattr problems. (overlayfs can also use xattrs in the :code:`trusted` namespace, but this requires :code:`CAP_SYS_ADMIN` `on the host `_ and thus is not helpful for unprivileged containers.) Environment variables ===================== :code:`ch-run` leaves environment variables unchanged, i.e. the host environment is passed through unaltered, except: * by default (:code:`--home` not specified), :code:`HOME` is set to :code:`/root`, if it exists, and :code:`/` otherwise. * limited tweaks to avoid significant guest breakage; * user-set variables via :code:`--set-env`; * user-unset variables via :code:`--unset-env`; and * set :code:`CH_RUNNING`. This section describes these features. The default tweaks happen first, then :code:`--set-env` and :code:`--unset-env` in the order specified on the command line, and then :code:`CH_RUNNING`. The two options can be repeated arbitrarily many times, e.g. to add/remove multiple variable sets or add only some variables in a file. Default behavior ---------------- By default, :code:`ch-run` makes the following environment variable changes: :code:`$CH_RUNNING` Set to :code:`Weird Al Yankovic`. While a process can figure out that it’s in an unprivileged container and what namespaces are active without this hint, that can be messy, and there is no way to tell that it’s a *Charliecloud* container specifically. This variable makes such a test simple and well-defined. (**Note:** This variable is unaffected by :code:`--unset-env`.) :code:`$HOME` If :code:`--home` is specified, then your home directory is bind-mounted into the guest at :code:`/home/$USER`. If you also have a different home directory path on the host, an inherited :code:`$HOME` will be incorrect inside the guest, which confuses lots of software, notably Spack. Thus, with :code:`--home`, :code:`$HOME` is set to :code:`/home/$USER` (by default, it is unchanged.) :code:`$PATH` Newer Linux distributions replace some root-level directories, such as :code:`/bin`, with symlinks to their counterparts in :code:`/usr`. Some of these distributions (e.g., Fedora 24) have also dropped :code:`/bin` from the default :code:`$PATH`. This is a problem when the guest OS does *not* have a merged :code:`/usr` (e.g., Debian 8 “Jessie”). Thus, we add :code:`/bin` to :code:`$PATH` if it’s not already present. Further reading: * `The case for the /usr Merge `_ * `Fedora `_ * `Debian `_ :code:`$TMPDIR` Unset, because this is almost certainly a host path, and that host path is made available in the guest at :code:`/tmp` unless :code:`--private-tmp` is given. Setting variables with :code:`--set-env` or :code:`--set-env0` -------------------------------------------------------------- The purpose of these two options is to set environment variables within the container. Values given replace any already in the environment (i.e., inherited from the host shell) or set by earlier uses of the options. These flags take an optional argument with two possible forms: 1. **If the argument contains an equals sign** (:code:`=`, ASCII 61), that sets an environment variable directly. For example, to set :code:`FOO` to the string value :code:`bar`:: $ ch-run --set-env=FOO=bar ... Single straight quotes around the value (:code:`'`, ASCII 39) are stripped, though be aware that both single and double quotes are also interpreted by the shell. For example, this example is similar to the prior one; the double quotes are removed by the shell and the single quotes are removed by :code:`ch-run`:: $ ch-run --set-env="'BAZ=qux'" ... 2. **If the argument does not contain an equals sign**, it is a host path to a file containing zero or more variables using the same syntax as above (except with no prior shell processing). With :code:`--set-env`, this file contains a sequence of assignments separated by newline (:code:`\n` or ASCII 10); with :code:`--set-env0`, the assignments are separated by the null byte (i.e., :code:`\0` or ASCII 0). Empty assignments are ignored, and no comments are interpreted. (This syntax is designed to accept the output of :code:`printenv` and be easily produced by other simple mechanisms.) The file need not be seekable. For example:: $ cat /tmp/env.txt FOO=bar BAZ='qux' $ ch-run --set-env=/tmp/env.txt ... For directory images only (because the file is read before containerizing), guest paths can be given by prepending the image path. 3. **If there is no argument**, the file :code:`/ch/environment` within the image is used. This file is commonly populated by :code:`ENV` instructions in the Dockerfile. For example, equivalently to form 2:: $ cat Dockerfile [...] ENV FOO=bar ENV BAZ=qux [...] $ ch-image build -t foo . $ ch-convert foo /var/tmp/foo.sqfs $ ch-run --set-env /var/tmp/foo.sqfs -- ... (Note the image path is interpreted correctly, not as the :code:`--set-env` argument.) At present, there is no way to use files other than :code:`/ch/environment` within SquashFS images. Environment variables are expanded for values that look like search paths, unless :code:`--env-no-expand` is given prior to :code:`--set-env`. In this case, the value is a sequence of zero or more possibly-empty items separated by colon (:code:`:`, ASCII 58). If an item begins with dollar sign (:code:`$`, ASCII 36), then the rest of the item is the name of an environment variable. If this variable is set to a non-empty value, that value is substituted for the item; otherwise (i.e., the variable is unset or the empty string), the item is deleted, including a delimiter colon. The purpose of omitting empty expansions is to avoid surprising behavior such as an empty element in :code:`$PATH` meaning `the current directory `_. For example, to set :code:`HOSTPATH` to the search path in the current shell (this is expanded by :code:`ch-run`, though letting the shell do it happens to be equivalent):: $ ch-run --set-env='HOSTPATH=$PATH' ... To prepend :code:`/opt/bin` to this current search path:: $ ch-run --set-env='PATH=/opt/bin:$PATH' ... To prepend :code:`/opt/bin` to the search path set by the Dockerfile, as retrieved from guest file :code:`/ch/environment` (here we really cannot let the shell expand :code:`$PATH`):: $ ch-run --set-env --set-env='PATH=/opt/bin:$PATH' ... Examples of valid assignment, assuming that environment variable :code:`BAR` is set to :code:`bar` and :code:`UNSET` is unset or set to the empty string: .. list-table:: :header-rows: 1 * - Assignment - Name - Value * - :code:`FOO=bar` - :code:`FOO` - :code:`bar` * - :code:`FOO=bar=baz` - :code:`FOO` - :code:`bar=baz` * - :code:`FLAGS=-march=foo -mtune=bar` - :code:`FLAGS` - :code:`-march=foo -mtune=bar` * - :code:`FLAGS='-march=foo -mtune=bar'` - :code:`FLAGS` - :code:`-march=foo -mtune=bar` * - :code:`FOO=$BAR` - :code:`FOO` - :code:`bar` * - :code:`FOO=$BAR:baz` - :code:`FOO` - :code:`bar:baz` * - :code:`FOO=` - :code:`FOO` - empty string * - :code:`FOO=$UNSET` - :code:`FOO` - empty string * - :code:`FOO=baz:$UNSET:qux` - :code:`FOO` - :code:`baz:qux` (not :code:`baz::qux`) * - :code:`FOO=:bar:baz::` - :code:`FOO` - :code:`:bar:baz::` * - :code:`FOO=''` - :code:`FOO` - empty string * - :code:`FOO=''''` - :code:`FOO` - :code:`''` (two single quotes) Example invalid assignments: .. list-table:: :header-rows: 1 * - Assignment - Problem * - :code:`FOO bar` - no equals separator * - :code:`=bar` - name cannot be empty Example valid assignments that are probably not what you want: .. Note: Plain leading space screws up ReST parser. We use ZERO WIDTH SPACE U+200B, then plain space. This will copy and paste incorrectly, but that seems unlikely. .. list-table:: :header-rows: 1 * - Assignment - Name - Value - Problem * - :code:`FOO="bar"` - :code:`FOO` - :code:`"bar"` - double quotes aren’t stripped * - :code:`FOO=bar # baz` - :code:`FOO` - :code:`bar # baz` - comments not supported * - :code:`FOO=bar\tbaz` - :code:`FOO` - :code:`bar\tbaz` - backslashes are not special * - :code:`​ FOO=bar` - :code:`​ FOO` - :code:`bar` - leading space in key * - :code:`FOO= bar` - :code:`FOO` - :code:`​ bar` - leading space in value * - :code:`$FOO=bar` - :code:`$FOO` - :code:`bar` - variables not expanded in key * - :code:`FOO=$BAR baz:qux` - :code:`FOO` - :code:`qux` - variable :code:`BAR baz` not set Removing variables with :code:`--unset-env` ------------------------------------------- The purpose of :code:`--unset-env=GLOB` is to remove unwanted environment variables. The argument :code:`GLOB` is a glob pattern (`dialect `_ :code:`fnmatch(3)` with the :code:`FNM_EXTMATCH` flag where supported); all variables with matching names are removed from the environment. .. warning:: Because the shell also interprets glob patterns, if any wildcard characters are in :code:`GLOB`, it is important to put it in single quotes to avoid surprises. :code:`GLOB` must be a non-empty string. Example 1: Remove the single environment variable :code:`FOO`:: $ export FOO=bar $ env | fgrep FOO FOO=bar $ ch-run --unset-env=FOO $CH_TEST_IMGDIR/chtest -- env | fgrep FOO $ Example 2: Hide from a container the fact that it’s running in a Slurm allocation, by removing all variables beginning with :code:`SLURM`. You might want to do this to test an MPI program with one rank and no launcher:: $ salloc -N1 $ env | egrep '^SLURM' | wc 44 44 1092 $ ch-run $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello [... long error message ...] $ ch-run --unset-env='SLURM*' $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello 0: MPI version: Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018 0: init ok cn001.localdomain, 1 ranks, userns 4026532530 0: send/receive ok 0: finalize ok Example 3: Clear the environment completely (remove all variables):: $ ch-run --unset-env='*' $CH_TEST_IMGDIR/chtest -- env $ Example 4: Remove all environment variables *except* for those prefixed with either :code:`WANTED_` or :code:`ALSO_WANTED_`:: $ export WANTED_1=yes $ export ALSO_WANTED_2=yes $ export NOT_WANTED_1=no $ ch-run --unset-env='!(WANTED_*|ALSO_WANTED_*)' $CH_TEST_IMGDIR/chtest -- env WANTED_1=yes ALSO_WANTED_2=yes $ Note that some programs, such as shells, set some environment variables even if started with no init files:: $ ch-run --unset-env='*' $CH_TEST_IMGDIR/debian_9ch -- bash --noprofile --norc -c env SHLVL=1 PWD=/ _=/usr/bin/env $ Examples ======== Run the command :code:`echo hello` inside a Charliecloud container using the unpacked image at :code:`/data/foo`:: $ ch-run /data/foo -- echo hello hello Run an MPI job that can use CMA to communicate:: $ srun ch-run --join /data/foo -- bar Syslog ====== By default, :code:`ch-run` logs its command line to `syslog `_. (This can be disabled by configuring with :code:`--disable-syslog`.) This includes: (1) the invoking real UID, (2) the number of command line arguments, and (3) the arguments, separated by spaces. For example:: Dec 10 18:19:08 mybox ch-run: uid=1000 args=7: ch-run -v /var/tmp/00_tiny -- echo hello "wor l}\$d" Logging is one of the first things done during program initialization, even before command line parsing. That is, almost all command lines are logged, even if erroneous, and there is no logging of program success or failure. Arguments are serialized with the following procedure. The purpose is to provide a human-readable reconstruction of the command line while also allowing each argument to be recovered byte-for-byte. .. Note: The next paragraph contains ​U+200B ZERO WIDTH SPACE after the backslash because backslash by itself won’t build and two backslashes renders as two backslashes. * If an argument contains only printable ASCII bytes that are not whitespace, shell metacharacters, double quote (:code:`"`, ASCII 34 decimal), or backslash (:code:`\​`, ASCII 92), then log it unchanged. * Otherwise, (a) enclose the argument in double quotes and (b) backslash-escape double quotes, backslashes, and characters interpreted by Bash (including POSIX shells) within double quotes. The verbatim command line typed in the shell cannot be recovered, because not enough information is provided to UNIX programs. For example, :code:`echo  'foo'` is given to programs as a sequence of two arguments, :code:`echo` and :code:`foo`; the two spaces and single quotes are removed by the shell. The zero byte, ASCII NUL, cannot appear in arguments because it would terminate the string. Exit status =========== If there is an error during containerization, :code:`ch-run` exits with status non-zero. If the user command is started successfully, the exit status is that of the user command, with one exception: if the image is an internally mounted SquashFS filesystem and the user command is killed by a signal, the exit status is 1 regardless of the signal value. .. include:: ./bugs.rst .. include:: ./see_also.rst .. LocalWords: mtune NEWROOT hugetlbfs UsrMerge fusermount mybox IMG HOSTPATH .. LocalWords: noprofile norc SHLVL PWD kernelnewbies extglob charliecloud-0.37/doc/ch-test.rst000066400000000000000000000233051457016721300167600ustar00rootroot00000000000000:code:`ch-test` +++++++++++++++ .. only:: not man Run some or all of the Charliecloud test suite. Synopsis ======== :: $ ch-test [PHASE] [--scope SCOPE] [--pack-fmt FMT] [ARGS] Description =========== Charliecloud comes with a comprehensive test suite that exercises the container workflow itself as well as a few example applications. :code:`ch-test` coordinates running the test suite. While the CLI has lots of options, the defaults are reasonable, and bare :code:`ch-test` will give useful results in a few minutes on single-node, internet-connected systems with a few GB available in :code:`/var/tmp`. The test suite requires a few GB (standard scope) or tens of GB (full scope) of storage for test fixtures: * *Builder storage* (e.g., layer cache). This goes wherever the builder puts it. * *Packed images directory*: image tarballs or SquashFS files. * *Unpacked images directory*. Images are unpacked into and then run from here. * *Filesystem permissions* directories. These are used to test that the kernel is enforcing permissions correctly. Note that this exercises the kernel, not Charliecloud, and can be omitted from routine Charliecloud testing. The first three are created when needed if they don’t exist, while the filesystem permissions fixtures must be created manually, in order to accommodate configurations where sudo is not available via the same login path used for running tests. The packed and unpacked image directories specified for testing are volatile. The contents of these directories are deleted before the build and run phases, respectively. In all four cases, when creating directories, only the final path component is created. Parent directories must already exist, i.e., :code:`ch-test` uses the behavior of :code:`mkdir` rather than :code:`mkdir -p`. Some of the tests exercise parallel functionality. If :code:`ch-test` is run on a single node, multiple cores will be used; if in a Slurm allocation, multiple nodes too. The subset of tests to run mostly splits along two key dimensions. The *phase* is which parts of the workflow to run. Different parts of the workflow can be tested on different systems by copying the necessary artifacts between them, e.g. by building images on one system and running them on another. The *scope* allows trading off thoroughness versus time. :code:`PHASE` must be one of the following: :code:`build` Image building and associated functionality, with the selected builder. :code:`run` Running containers and associated functionality. This requires a packed images directory produced by a successful :code:`build` phase, which can be copied from the build system if it’s not also the run system. :code:`rootemu` Test the root emulation modes (:code:`seccomp`, :code:`fakeroot`, and :code:`none`) on various linux distributions. :code:`examples` Example applications. Requires an unpacked images directory produced by a successful :code:`run` phase. :code:`all` Execute phases :code:`build`, :code:`rootemu`, :code:`run`, and :code:`examples`, in that order. :code:`mk-perm-dirs` Create the filesystem permissions directories. Requires :code:`--perm-dirs`. :code:`build-images` Build images from :code:`build` phase, without running the associated tests. :code:`clean` Delete automatically-generated test files, and packed and unpacked image directories. :code:`rm-perm-dirs` Remove the filesystem permissions directories. Requires :code:`--perm-dirs`. :code:`-f`, :code:`--file FILE[:TEST]` Run the tests in the given file only, which can be an arbitrary :code:`.bats` file, except for :code:`test.bats` under :code:`examples`, where you must specify the corresponding Dockerfile or :code:`Build` file instead. This is somewhat brittle and typically used for development or debugging. For example, it does not check whether the pre-requisites of whatever is in the file are satisfied. Often running :code:`build` and :code:`run` first is sufficient, but this varies. If :code:`TEST` is also given, then run only tests with name containing that string, skipping the others. The separator is a literal colon. If the string contains shell metacharacters such as space, you’ll need to quote the argument to protect it from the shell. Scope is specified with: :code:`-s`, :code:`--scope SCOPE` :code:`SCOPE` must be one of the following: * :code:`quick`: Most important subset of workflow. Handy for development. * :code:`standard`: All tested workflow functionality and a selection of more important examples. (Default.) * :code:`full`: All available tests, including all examples. Image format is specified with: :code:`--pack-fmt FMT` :code:`FMT` must be one of the following: * :code:`squash-mount` or 🐘: SquashFS archive, run directly from the archive using :code:`ch-run`’s internal SquashFUSE functionality. In this mode, tests that require writing to the image are skipped. * :code:`tar-unpack` or 📠: Tarball, and the images are unpacked before running. * :code:`squash-unpack` or 🎃: SquashFS, and the images are unpacked before running. Default: :code:`$CH_TEST_PACK_FMT` if set. Otherwise, if :code:`mksquashfs(1)` is available and :code:`ch-run` was built with :code:`libsquashfuse` support, then :code:`squash-mount`, else :code:`tar-unpack`. Additional arguments: :code:`-b`, :code:`--builder BUILDER` Image builder to use. Default: :code:`$CH_TEST_BUILDER` if set, otherwise :code:`ch-image`. :code:`--dry-run` Print summary of what would be tested and then exit. :code:`-h`, :code:`--help` Print usage and then exit. :code:`--img-dir DIR` Set unpacked images directory to :code:`DIR`. In a multi-node allocation, this directory may not be shared between nodes. Default: :code:`$CH_TEST_IMGDIR` if set; otherwise :code:`/var/tmp/${USER}.img`. :code:`--lustre DIR` Use :code:`DIR` for run-phase Lustre tests. Default: :code:`CH_TEST_LUSTREDIR` if set; otherwise skip them. The tests will create, populate, and delete a new subdirectory under :code:`DIR`, leaving everything else in :code:`DIR` untouched. :code:`--pack-dir DIR` Set packed images directory to :code:`DIR`. Default: :code:`$CH_TEST_TARDIR` if set; otherwise :code:`/var/tmp/${USER}.pack`. :code:`--pedantic (yes|no)` Some tests require configurations that are very specific (e.g., being a member of at least two groups) or unusual (e.g., sudo to a non-root group). If :code:`yes`, then fail if the requirement is not met; if :code:`no`, then skip. The default is :code:`yes` for CI environments or people listed in :code:`README.md`, :code:`no` otherwise. If :code:`yes` and sudo seems to be available, implies :code:`--sudo`. :code:`--perm-dir DIR` Add :code:`DIR` to filesystem permission fixture directories; can be specified multiple times. We recommend one such directory per mounted filesystem type whose kernel module you do not trust; e.g., you probably don’t need to test your :code:`tmpfs`\ es, but out-of-tree filesystems very likely need this. Implies :code:`--sudo`. Default: :code:`CH_TEST_PERMDIRS` if set; otherwise skip the filesystem permissions tests. :code:`--sudo` Enable things that require sudo, such as certain privilege escalation tests and creating/removing the filesystem permissions fixtures. Requires generic :code:`sudo` capabilities. Note that the Docker builder uses :code:`sudo docker` even without this option. Exit status =========== Zero if all tests passed; non-zero if any failed. For setup and teardown phases, zero if everything was created or deleted correctly, non-zero otherwise. Bugs ==== Bats will wait until all descendant processes finish before exiting, so if you get into a failure mode where a test sequence doesn’t clean up all its processes, :code:`ch-test` will hang. Examples ======== Many systems can simply use the defaults. To run the :code:`build`, :code:`run`, and :code:`examples` phases on a single system, without the filesystem permissions tests:: $ ch-test ch-test version 0.12 ch-run: 0.12 /usr/local/bin/ch-run bats: 0.4.0 /usr/bin/bats tests: /usr/local/libexec/charliecloud/test phase: build run examples scope: standard (default) builder: docker (default) use generic sudo: no (default) unpacked images dir: /var/tmp/img (default) packed images dir: /var/tmp/tar (default) fs permissions dirs: skip (default) checking namespaces ... ok checking builder ... found: /usr/bin/docker 19.03.2 bats build.bats build_auto.bats build_post.bats ✓ documentation seems sane ✓ version number seems sane [...] All tests passed. The next example is for a more complex setup like you might find in HPC centers: * Non-default fixture directories. * Non-default scope. * Different build and run systems. * Run the filesystem permissions tests. Output has been omitted. :: (mybox)$ ssh hpc-admin (hpc-admin)$ ch-test mk-perm-dirs --perm-dir /scratch/$USER/perms \ --perm-dir /home/$USER/perms (hpc-admin)$ exit (mybox)$ ch-test build --scope full (mybox)$ scp -r /var/tmp/pack hpc:/scratch/$USER/pack (mybox)$ ssh hpc (hpc)$ salloc -N2 (cn001)$ export CH_TEST_TARDIR=/scratch/$USER/pack (cn001)$ export CH_TEST_IMGDIR=/local/tmp (cn001)$ export CH_TEST_PERMDIRS="/scratch/$USER/perms /home/$USER/perms" (cn001)$ export CH_TEST_SCOPE=full (cn001)$ ch-test run (cn001)$ ch-test examples .. include:: ./bugs.rst .. include:: ./see_also.rst .. LocalWords: fmt img LUSTREDIR charliecloud-0.37/doc/charliecloud.rst000066400000000000000000000006171457016721300200500ustar00rootroot00000000000000:orphan: charliecloud man page +++++++++++++++++++++ .. include:: ../README.rst .. include:: ./bugs.rst See also -------- ch-checkns(1), ch-completion.bash(7), ch-convert(1), ch-fromhost(1), ch-image(1), ch-run(1), ch-run-oci(1), ch-test(1), Full documentation at: https://hpc.github.io/charliecloud Note ---- These man pages are for Charliecloud version |release| (Git commit |version|). charliecloud-0.37/doc/conf.py000066400000000000000000000254471457016721300161670ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # QUAC documentation build configuration file, created by # sphinx-quickstart on Wed Feb 20 12:04:35 2013. # # This file is execfile()d with the current directory set to its containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. import sys, os # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) # -- General configuration ----------------------------------------------------- # If your documentation needs a minimal Sphinx version, state it here. #needs_sphinx = '1.4.9' # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.mathjax', 'sphinx.ext.todo'] todo_include_todos = True # Workaround for EPEL, which prohibits pip and doesn’t have # sphinx_reredirects. The result will be no redirect, but a lot about the # EPEL-built docs is crappy. try: import sphinx_reredirects extensions.append('sphinx_reredirects') except ImportError: pass # Monkey-patch RSYNC keyword into Pygments. Ignore any problems. print() # statements here show up in the make(1) chatter. # # See: https://github.com/pygments/pygments/blob/e2cb7c9/pygments/lexers/configs.py#L667 try: import pygments.lexers.configs as plc import re for (i, tok) in enumerate(plc.DockerLexer.tokens["root"]): m = re.search(r"^(.*COPY.*)\)\)$", tok[0]) if (m is not None): re_new = m[1] + "|RSYNC))" plc.DockerLexer.tokens["root"][i] = (re_new, tok[1]) except Exception: pass # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = u'Charliecloud' copyright = u'2014–2023, Triad National Security, LLC and others' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. version = open("../lib/version.txt", "r").read().rstrip() # The full version, including alpha/beta/rc tags. release = version # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. #language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. today_fmt = '%Y-%m-%d %H:%M %Z' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = ["doctrees", "html", "man", "**/_*.rst"] # FIXME: Workaround for older Sphinx that barf with: # # WARNING: document isn't included in any toctree # # on files included via ".. include::'. I believe this was fixed in 1.4.3 and # the relevant issue is: https://github.com/sphinx-doc/sphinx/issues/2603 exclude_patterns += ["*_desc.rst", "_*.rst", "bugs.rst", "py_env.rst", "see_also.rst"] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. #pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'sphinx_rtd_theme' # FIXME: Workaround for older versions of Sphinx. This is not needed in 1.8.5, # but it is needed in 1.2.3. I don't know where the boundary is. We embed it # in try/except so that "docs-sane" can import the file too. try: import sphinx_rtd_theme html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] except ImportError: pass # error caught elsewhere highlight_language = 'console' # Don’t break links to the old command-usage.html. (#1461) redirects = { "command-usage": "index.html" } # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {'bodyfont': 'serif', # for agogo # 'pagewidth': '60em', # 'documentwidth': '43em', # 'sidebarwidth': '17em', # 'textalign':'left'} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". # #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. html_logo = "logo-sidebar.png" # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. html_favicon = "favicon.ico" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". #html_static_path = ['_static'] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. #html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. html_use_smartypants = True # deprecated in 1.6.6 smartquotes = True smartquotes_action = "qBD" # quotes, en and em dashes, but not ellipses # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. html_domain_indices = False # If false, no index is generated. html_use_index = False # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. html_show_sourcelink = False # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. html_show_sphinx = False # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. #html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = None # Output file base name for HTML help builder. htmlhelp_basename = 'charliedoc' # -- Options for LaTeX output -------------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). #'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). #'pointsize': '10pt', # Additional stuff for the LaTeX preamble. #'preamble': '', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ ('index', 'charlie.tex', u'Charliecloud Documentation', u'Reid Priedhorsky, Tim Randles, and others', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # If true, show page references after internal links. #latex_show_pagerefs = False # If true, show URL addresses after external links. #latex_show_urls = False # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_domain_indices = True # -- Options for manual page output -------------------------------------------- # Put all man pages in one directory regardless of section. Default changes to # True in Sphinx 4.0, which broke our builds (#1060). man_make_section_directory = False # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ("charliecloud", "charliecloud", "Lightweight user-defined software stacks for high-performance computing", [], 7), ("ch-checkns", "ch-checkns", 'Check "ch-run" prerequisites, e.g., namespaces and "pivot_root(2)"', [], 1), ("ch-completion.bash", "ch-completion.bash", 'Tab completion for the Charliecloud command line', [], 7), ("ch-convert", "ch-convert", 'Convert an image from one format to another', [], 1), ("ch-fromhost", "ch-fromhost", "Inject files from the host into an image directory, with various magic", [], 1), ("ch-image", "ch-image", "Build and manage images; completely unprivileged", [], 1), ("ch-run", "ch-run", "Run a command in a Charliecloud container", [], 1), ("ch-run-oci", "ch-run-oci", 'OCI wrapper for "ch-run"', [], 1), ("ch-test", "ch-test", "Run some or all of the Charliecloud test suite", [], 1), ] # If true, show URL addresses after external links. #man_show_urls = False # -- Options for Texinfo output ------------------------------------------------ # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ('index', 'Charliecloud', u'Charliecloud Documentation', u'Reid Priedhorsky, Tim Randles, and others', 'Charliecloud', 'One line description of project.', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. #texinfo_appendices = [] # If false, no module index is generated. #texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. #texinfo_show_urls = 'footnote' charliecloud-0.37/doc/dev.rst000066400000000000000000001601101457016721300161630ustar00rootroot00000000000000Contributor’s guide ******************* This section is notes on contributing to Charliecloud development. Currently, it is messy and incomplete. Patches welcome! It documents public stuff only. If you are on the core team at LANL, also consult the internal documentation and other resources. .. contents:: :depth: 2 :local: .. note:: We are interested in and will consider all good-faith contributions. While it does make things easier and faster if you follow the guidelines here, *they are not required*. We’ll either clean it up for you or walk you through any necessary changes. Workflow ======== We try to keep procedures and the Git branching model simple. Right now, we’re pretty similar to Scott Chacon’s “`GitHub Flow `_”: Master is stable; work on short-lived topic branches; use pull requests to ask for merging; keep issues organized with tags and milestones. The standard workflow is: 1. Propose a change in an issue. 2. Tag the issue with its kind (bug, enhancement, question). 3. Get consensus on what to do and how to do it, with key information recorded in the issue. 4. Submit a PR that refers to the issue. 5. Assign the issue to a milestone. 6. Review/iterate. 7. Project lead merges with “squash and merge”. Code review ----------- **Issues and pull requests.** The typical workflow is: #. Propose a change in an issue. #. Get consensus on what to do, whether in the issue or elsewhere. #. Create a `pull request `_ (PR) for the implementation. #. Iterate the PR until consensus is reached to either merge or abandon. #. Merge or close the PR accordingly. The issue, not the PR, should be tagged and milestoned so a given change shows up only once in the various views. GitHub PRs have two states, which are often poorly labeled. These states and our interpretations are: * *Ready for review* (the green *Create pull request* button). This means that the PR is ready to be merged once tests and code review pass. In-progress PRs headed in that direction should also be in this state (i.e., the trigger for review and possible merge is the review request, not a draft to ready-for-review transition). * *Draft*. This means not ready for merge even if tests and review pass. (GitLab would indicate this with a :code:`WIP:` prefix in the title.) **Stand-alone PRs.** If consensus is obtained through other means, e.g. out-of-band discussion, then a stand-alone PR is appropriate (i.e., don’t create an issue just for the sake of having an issue to link to a PR). A stand-alone PR should be tagged and milestoned, since there is no issue. Note that stand-alone PRs are generally not a good way to *propose* something. **Address a single concern.** When practical, issues and PRs should address completely one self-contained change. If there are multiple concerns, make separate issues and/or PRs. For example, PRs should not tidy unrelated code, and non-essential complications should be split into a follow-on issue. However, sometimes one PR addresses several related issues, which is fine. **Documentation and tests first.** The best practice for significant changes is to draft documentation and/or tests first, get feedback on that, and then implement the code. Reviews of the form "you need a completely different approach" are no fun. **CI must pass.** PRs will usually not be merged until they pass CI, with exceptions if the failures are clearly unconnected and we are confident they aren’t masking a real issue. If appropriate, tests should also pass on relevant supercomputers. **Use close keywords in PRs.** Use the issue-closing keywords (variations on `"closes", "fixes", and "resolves" `_) in PR descriptions to link it to the relevant issue(s). If this changes, edit the description to add/remove issues. **PR review procedure.** When your PR is ready for review — which may or may not be when you want it considered for merging! — do this: #. Request review from the person(s) you want to look at it. The purpose of requesting review is so the person is notified you need their help. #. If you think it’s ready to merge (even if you’re not sure), ensure the PR is (1) marked “ready for review” (green icon), and (2) the project lead is included in your review request. In both cases, the person from whom you requested review now owns the branch, and you should stop work on it unless and until you get it back (modulo other communication, of course). This is so they can make tidy commits if needed without collision. It is good practice to communicate with your reviewer directly to set expectations on review urgency. Review outcomes: * *Request changes*: The reviewer believes there are changes needed, *and* the PR needs re-review after these are done. * *Comment*: The reviewer has questions or comments, *and* the PR needs re-review after these are addressed. * *Approve*: The reviewer believes the branch is ready to proceed (further work if draft, merging if ready for review). Importantly, the review can include comments/questions/changes *but* the reviewer believes these don’t need re-review (i.e., the PR author can deal with them independently). *Use multi-comment reviews.* Review comments should all be packaged up into a single review; click *Start a review* rather than *Add single comment*. Then the PR author gets only a single notification instead of one for every comment you make, and it’s clear when the branch is theirs again. *Selecting a reviewer.* Generally, you want to find a reviewer with time to do the review and appropriate expertise. Feel free to ask if you’re not sure. Note that the project lead must approve any PRs before merge, so they are typically a reasonable choice if you don’t have someone else in mind. External contributions do not need to select a reviewer. The team will notice the PR and wrangle its review. *Special case 1:* Often, the review consists of code changes, and the reviewer will want you to assess those changes. GitHub doesn’t let you request review from the PR submitter, so this must be done with a comment, either online or offline. *Special case 2:* GitHub will not let you request review from external people, so this needs to be done with a comment too. Generally you should ask the original bug reporter to review, to make sure it solves their problem. Branching and merging --------------------- **Don’t commit directly to master.** Even the project lead doesn’t do this. While it may appear that some trivial fixes are being committed to the master directly, what really happened is that these were prototyped on a branch and then fast-forward merged after the tests pass. (Note we no longer do this.) **Merging to master.** Only the project lead should do this. **Branch naming convention.** Name the branch with a *brief* summary of the issue being fixed — just a couple words — with words separated by hyphens, then an underscore and the issue number being addressed. For example, issue `#1773 `_ is titled “:code:`ch-image build`: :code:`--force=fakeroot` outputs to stderr despite :code:`-q`”; the corresponding branch (for `PR #1812 `_) is called :code:`fakeroot-quiet-rhel_1773`. Something even shorter, such as :code:`fakeroot_1773`, would have been fine too. Stand-alone PRs do the same, just without an issue number. For example, `PR #1804 `_ is titled “add tab completion to :code:`ch-convert`” and the branch is :code:`convert-completion`. It’s okay if the branch name misses a little. For example, if you discover during work on a PR that you should close a second issue in the same PR, it’s not necessary to add the second issue number to the branch name. **Branch merge procedure.** Generally, branches are merged in the GitHub web interface with the *Squash and merge* button, which is :code:`git merge --squash` under the hood. This squashes the branch into a single commit on master. Commit message must be the PR number followed by the PR title, e.g.: PR #268: remove ch-docker-run The commit message should not mention issue numbers; let the PR itself do that. The reason to prefer merge via web interface is that GitHub often doesn’t notice merges done on the command line. After merge, delete the branch via the web interface. **Branch history tidiness.** Commit frequently at semantically relevant times, and keep in mind that this history will probably be squashed per above. It is not necessary to rebase or squash to keep branch history tidy. But, don’t go crazy. Commit messages like "try 2" and "fix CI again" are a bad sign; so are carefully proofread ones. Commit messages that are brief, technically relevant, and quick to write are what you want on feature branches. **Keep branches up to date.** Merge master into your branch, rather than rebasing. This lets you resolve conflicts once rather than multiple times as rebase works through a stack of commits. Note that PRs with merge conflicts will generally not be merged. Resolve conflicts before asking for review. **Remove obsolete branches.** Keep your repo free of old branches with the script :code:`misc/branches-tidy`. Miscellaneous issue and pull request notes ------------------------------------------ **Acknowledging issues.** Issues and PRs submitted from outside should be acknowledged promptly, including adding or correcting tags. **Closing issues.** We close issues when we’ve taken the requested action, decided not to take action, resolved the question, or actively determined an issue is obsolete. It is OK for “stale” issues to sit around indefinitely awaiting this. Unlike many projects, we do not automatically close issues just because they’re old. **Closing PRs.** Stale PRs, on the other hand, are to be avoided due to bit rot. We try to either merge or reject PRs in a timely manner. **Re-opening issues.** Closed issues can be re-opened if new information arises, for example a :code:`worksforme` issue with new reproduction steps. Continuous integration (CI) testing ----------------------------------- **Quality of testing.** Tagged versions currently get more testing for various reasons. We are working to improve testing for normal commits on master, but full parity is probably unlikely. **Cycles budget.** The resource is there for your use, so take advantage of it, but be mindful of the various costs of this compute time. Things you can do include focused local testing, cancelling jobs you know will fail or that won’t give you additional information, and not pushing every commit (CI tests only the most recent commit in a pushed group). Avoid making commits merely to trigger CI. **Purging Docker cache.** :code:`misc/docker-clean.sh` can be used to purge your Docker cache, either by removing all tags or deleting all containers and images. The former is generally preferred, as it lets you update only those base images that have actually changed (the ones that haven’t will be re-tagged). Issue labeling -------------- We use the following labels (a.k.a. tags) to organize issues. Each issue (or stand-alone PR) should have label(s) from each category, with the exception of disposition which only applies to closed issues. Labels are periodically validated using a script. Charliecloud team members should label their own issues. The general public are more than welcome to label their issues if they like, but in practice this is rare, which is fine. Whoever triages the incoming issue should add or adjust labels as needed. .. note:: This scheme is designed to organize open issues only. There have been previous schemes, and we have not re-labeled closed issues. What kind of change is it? ~~~~~~~~~~~~~~~~~~~~~~~~~~ Choose *one type* from: :code:`bug` Something doesn’t work; e.g., it doesn’t work as intended or it was mis-designed. This includes usability and documentation problems. Steps to reproduce with expected and actual behavior are almost always very helpful. :code:`enhancement` Things work, but it would be better if something was different. For example, a new feature proposal, an improvement in how a feature works, or clarifying an error message. Steps to reproduce with desired and current behavior are often helpful. :code:`refactor` Change that will improve Charliecloud but does not materially affect user-visible behavior. Note this doesn’t mean “invisible to the user”; even user-facing documentation or logging changes could feasibly be this, if they are more cleanup-oriented. How important/urgent is it? ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Choose *one priority* from: :code:`high` High priority. :code:`medium` Medium priority. :code:`low` Low priority. Note: Unfortunately, due to resource limitations, complex issues here are likely to wait a long time, perhaps forever. If that makes you particularly sad on a particular issue, please comment to say why. Maybe it’s mis-prioritized. :code:`deferred` No plans to do this, but not rejected. These issues stay open, because we do not consider the deferred state resolved. Submitting PRs on these issues is risky; you probably want to argue successfully that it should be done before starting work on it. Priority is indeed required, though it can be tricky because the levels are fuzzy. Do not hesitate to ask for advice. Considerations include: is customer or development work blocked by the issue; how valuable is the issue for customers; does the issue affect key customers; how many customers are affected; how much of Charliecloud is affected; what is the workaround like, if any. Difficulty of the issue is not a factor in priority, i.e., here we are trying to express benefit, not cost/benefit ratio. Perhaps the `Debian bug severity levels `_ provide inspiration. The number of :code:`high` priority issues should be relatively low. In part because priority is quite imprecise, issues are not a priority queue, i.e., we do work on lower-priority issues while higher-priority ones are still open. Related to this, issues do often move between priority levels. In particular, if you think we picked the wrong priority level, please say so. What part of Charliecloud is affected? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Choose *one or more components* from: :code:`runtime` The container runtime itself; largely :code:`ch-run`. :code:`image` Image building and interaction with image registries; largely :code:`ch-image`. (Not to be confused with image management tasks done by glue code.) :code:`glue` The “glue” that ties the runtime and image management (:code:`ch-image` or another builder) together. Largely shell scripts in :code:`bin`. :code:`install` Charliecloud build & install system, packaging, etc. (Not to be confused with image building.) :code:`doc` Documentation. :code:`test` Test suite and examples. :code:`misc` Everything else. Do not combine with another component. Special considerations ~~~~~~~~~~~~~~~~~~~~~~ Choose *one or more extras* from: :code:`blocked` We can’t do this yet because something else needs to happen first. If that something is another issue, mention it in a comment. :code:`hpc` Related specifically to HPC and HPC scaling considerations; e.g., interactions with job schedulers. :code:`uncertain` Course of action is unclear. For example: is the feature a good idea, what is a good approach to solve the bug, what additional information is needed. :code:`usability` Affects usability of any part of Charliecloud, including documentation and project organization. Why was it closed? ~~~~~~~~~~~~~~~~~~ If the issue was resolved (i.e., bug fixed or enhancement/refactoring implemented), there is no disposition tag. Otherwise, to explain why not, choose *one disposition* from: :code:`cantfix` The issue is not something we can resolve. Typically problems with other software, problems with containers in general that we can’t work around, or not actionable due to clarity or other reasons. *Use caution when blaming a problem on user error. Often (or usually) there is a documentation or usability bug that caused the "user error".* :code:`discussion` Converted to a discussion. The most common use is when someone asks a question rather than making a request for some change. :code:`duplicate` Same as some other issue. In addition to this tag, duplicates should refer to the other issue in a comment to record the link. Of the duplicates, the better one should stay open (e.g., clearer reproduction steps); if they are roughly equal in quality, the older one should stay open. :code:`moot` No longer relevant. Examples: withdrawn by reporter, fixed in current version (use :code:`duplicate` instead if it applies though), obsoleted by change in plans. :code:`wontfix` We are not going to do this, and we won’t merge PRs. Sometimes you’ll want to tag and then wait a few days before closing, to allow for further discussion to catch mistaken tags. :code:`worksforme` We cannot reproduce a bug, and it seems unlikely this will change given available information. Typically you’ll want to tag, then wait a few days for clarification before closing. Bugs closed with this tag that do gain a reproducer later should definitely be re-opened. For some bugs, it really feels like they should be reproducible but we’re missing it somehow; such bugs should be left open in hopes of new insight arising. .. note:: We do not use the GitHub “closed as not planned” feature, so everything is “closed as completed” even if the reason is one of the above. Deprecated labels ~~~~~~~~~~~~~~~~~ You might see these on old issues, but they are no longer in use. * :code:`help wanted`: This tended to get stale and wasn’t generating any leads. * :code:`key issue`: Replaced by priority labels. * :code:`question`: Replaced by Discussions. (If you report a bug that seems to be a discussion, we’ll be happy to convert it to you.) Test suite ========== Timing the tests ---------------- The :code:`ts` utility from :code:`moreutils` is quite handy. The following prepends each line with the elapsed time since the previous line:: $ ch-test -s quick | ts -i '%M:%.S' Note: a skipped test isn’t free; I see ~0.15 seconds to do a skip. :code:`ch-test` complains about inconsistent versions ----------------------------------------------------- There are multiple ways to ask Charliecloud for its version number. These should all give the same result. If they don’t, :code:`ch-test` will fail. Typically, something needs to be rebuilt. Recall that :code:`configure` contains the version number as a constant, so a common way to get into this situation is to change Git branches without rebuilding it. Charliecloud is small enough to just rebuild everything with:: $ ./autogen.sh && ./configure && make clean && make Special images -------------- For images not needed after completion of a test, tag them :code:`tmpimg`. This leaves only one extra image at the end of the test suite. Writing a test image using the standard workflow ------------------------------------------------ Summary ~~~~~~~ The Charliecloud test suite has a workflow that can build images by two methods: 1. From a Dockerfile, using :code:`ch-image` or another builder (see :code:`common.bash:build_()`). 2. By running a custom script. To create an image that will be built and unpacked and/or mounted, create a file in :code:`examples` (if the image recipe is useful as an example) or :code:`test` (if not) called :code:`{Dockerfile,Build}.foo`. This will create an image tagged :code:`foo`. Additional tests can be added to the test suite Bats files. To create an image with its own tests, documentation, etc., create a directory in :code:`examples`. In this directory, place :code:`{Dockerfile,Build}[.foo]` to build the image and :code:`test.bats` with your tests. For example, the file :code:`examples/foo/Dockerfile` will create an image tagged :code:`foo`, and :code:`examples/foo/Dockerfile.bar` tagged :code:`foo-bar`. These images also get the build and unpack/mount tests. Additional directories can be symlinked into :code:`examples` and will be integrated into the test suite. This allows you to create a site-specific test suite. :code:`ch-test` finds tests at any directory depth; e.g. :code:`examples/foo/bar/Dockerfile.baz` will create a test image tagged :code:`bar-baz`. Image tags in the test suite must be unique. Order of processing; within each item, alphabetical order: 1. Dockerfiles in :code:`test`. 2. :code:`Build` files in :code:`test`. 3. Dockerfiles in :code:`examples`. 4. :code:`Build` files in :code:`examples`. The purpose of doing :code:`Build` second is so they can leverage what has already been built by a Dockerfile, which is often more straightforward. How to specify when to include and exclude a test image ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each of these image build files must specify its scope for building and running, which must be greater than or equal than the scope of all tests in any corresponding :code:`.bats` files. Exactly one of the following strings must appear: .. code-block:: none ch-test-scope: skip ch-test-scope: quick ch-test-scope: standard ch-test-scope: full Other stuff on the line (e.g., comment syntax) is ignored. Optional test modification directives are: :code:`ch-test-arch-exclude: ARCH` If the output of :code:`uname -m` matches :code:`ARCH`, skip the file. :code:`ch-test-builder-exclude: BUILDER` If using :code:`BUILDER`, skip the file. :code:`ch-test-builder-include: BUILDER` If specified, run only if using :code:`BUILDER`. Can be repeated to include multiple builders. If specified zero times, all builders are included. :code:`ch-test-need-sudo` Run only if user has sudo. How to write a :code:`Dockerfile` recipe ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It’s a standard Dockerfile. How to write a :code:`Build` recipe ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is an arbitrary script or program that builds the image. It gets three command line arguments: * :code:`$1`: Absolute path to directory containing :code:`Build`. * :code:`$2`: Absolute path and name of output image, without extension. This can be either: * Tarball compressed with gzip or xz; append :code:`.tar.gz` or :code:`.tar.xz` to :code:`$2`. If :code:`ch-test --pack-fmt=squash`, then this tarball will be unpacked and repacked as a SquashFS. Therefore, only use tarball output if the image build process naturally produces it and you would have to unpack it to get a directory (e.g., :code:`docker export`). * Directory; use :code:`$2` unchanged. The contents of this directory will be packed without any enclosing directory, so if you want an enclosing directory, include one. Hidden (dot) files in :code:`$2` will be ignored. * :code:`$3`: Absolute path to temporary directory for use by the script. This can be used for whatever and need no be cleaned up; the test harness will delete it. Other requirements: * The script may write only in two directories: (a) the parent directory of :code:`$2` and (b) :code:`$3`. Specifically, it may not write to the current working directory. Everything written to the parent directory of :code:`$2` must have a name starting with :code:`$(basename $2)`. * The first entry in :code:`$PATH` will be the Charliecloud under test, i.e., bare :code:`ch-*` commands will be the right ones. * Any programming language is permitted. To be included in the Charliecloud source code, a language already in the test suite dependencies is required. * The script must test for its dependencies and fail with appropriate error message and exit code if something is missing. To be included in the Charliecloud source code, all dependencies must be something we are willing to install and test. * Exit codes: * 0: Image successfully created. * 65: One or more dependencies were not met. * 126 or 127: No interpreter available for script language (the shell takes care of this). * else: An error occurred. Building RPMs ============= We maintain :code:`.spec` files and infrastructure for building RPMs in the Charliecloud source code. This is for two purposes: 1. We maintain our own Fedora RPMs (see `packaging guidelines `_). 2. We want to be able to build an RPM of any commit. Item 2 is tested; i.e., if you break the RPM build, the test suite will fail. This section describes how to build the RPMs and the pain we’ve hopefully abstracted away. Dependencies ------------ * Charliecloud * Python 3.6+ * either: * the provided example :code:`centos_7ch` or :code:`almalinux_8ch` images, or * a RHEL/CentOS 7 or newer container image with (note there are different python version names for the listed packages in RHEL 8 and derivatives): * autoconf * automake * gcc * make * python36 * python36-sphinx * python36-sphinx_rtd_theme * rpm-build * rpmlint * rsync :code:`rpmbuild` wrapper script ------------------------------- While building the Charliecloud RPMs is not too weird, we provide a script to streamline it. The purpose is to (a) make it easy to build versions not matching the working directory, (b) use an arbitrary :code:`rpmbuild` directory, and (c) build in a Charliecloud container for non-RPM-based environments. The script must be run from the root of a Charliecloud Git working directory. Usage:: $ packaging/fedora/build [OPTIONS] IMAGE VERSION Options: * :code:`--install` : Install the RPMs after building into the build environment. * :code:`--rpmbuild=DIR` : Use RPM build directory root :code:`DIR` (default: :code:`~/rpmbuild`). For example, to build a version 0.9.7 RPM from the CentOS 7 image provided with the test suite, on any system, and leave the results in :code:`~/rpmbuild/RPMS` (note the test suite would also build the necessary image directory):: $ bin/ch-image build -f ./examples/Dockerfile.centos_7ch ./examples $ bin/ch-convert centos_7ch $CH_TEST_IMGDIR/centos_7ch $ packaging/fedora/build $CH_TEST_IMGDIR/centos_7ch 0.9.7-1 To build a pre-release RPM of Git HEAD using the CentOS 7 image:: $ bin/ch-image build -f ./examples/Dockerfile.centos_7ch ./examples $ bin/ch-convert centos_7ch $CH_TEST_IMGDIR/centos_7ch $ packaging/fedora/build ${CH_TEST_IMGDIR}/centos_7ch HEAD Gotchas and quirks ------------------ RPM versions and releases ~~~~~~~~~~~~~~~~~~~~~~~~~ If :code:`VERSION` is :code:`HEAD`, then the RPM version will be the content of :code:`VERSION.full` for that commit, including Git gobbledygook, and the RPM release will be :code:`0`. Note that such RPMs cannot be reliably upgraded because their version numbers are unordered. Otherwise, :code:`VERSION` should be a released Charliecloud version followed by a hyphen and the desired RPM release, e.g. :code:`0.9.7-3`. Other values of :code:`VERSION` (e.g., a branch name) may work but are not supported. Packaged source code and RPM build config come from different commits ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The spec file, :code:`build` script, :code:`.rpmlintrc`, etc. come from the working directory, but the package source is from the specified commit. This is what enables us to make additional RPM releases for a given Charliecloud release (e.g. 0.9.7-2). Corollaries of this policy are that RPM build configuration can be any or no commit, and it’s not possible to create an RPM of uncommitted source code. Changelog maintenance ~~~~~~~~~~~~~~~~~~~~~ The spec file contains a manually maintained changelog. Add a new entry for each new RPM release; do not include the Charliecloud release notes. For released versions, :code:`build` verifies that the most recent changelog entry matches the given :code:`VERSION` argument. The timestamp is not automatically verified. For other Charliecloud versions, :code:`build` adds a generic changelog entry with the appropriate version stating that it’s a pre-release RPM. .. _build-ova: Style hints =========== We haven’t written down a comprehensive style guide. Generally, follow the style of the surrounding code, think in rectangles rather than lines of code or text, and avoid CamelCase. Note that Reid is very picky about style, so don’t feel singled out if he complains (or even updates this section based on your patch!). He tries to be nice about it. Writing English --------------- * When describing what something does (e.g., your PR or a command), use the `imperative mood `_, i.e., write the orders you are giving rather than describe what the thing does. For example, do: | Inject files from the host into an image directory. | Add :code:`--join-pid` option to :code:`ch-run`. Do not (indicative mood): | Injects files from the host into an image directory. | Adds :code:`--join-pid` option to :code:`ch-run`. * Use sentence case for titles, not title case. * If it’s not a sentence, start with a lower-case character. * Use spell check. Keep your personal dictionary updated so your editor is not filled with false positives. Documentation ------------- Heading underline characters: 1. Asterisk, :code:`*`, e.g. "5. Contributor’s guide" 2. Equals, :code:`=`, e.g. "5.7 OCI technical notes" 3. Hyphen, :code:`-`, e.g. "5.7.1 Gotchas" 4. Tilde, :code:`~`, e.g. "5.7.1.1 Namespaces" (try to avoid) .. _dependency-policy: Dependency policy ----------------- Specific dependencies (prerequisites) are stated elsewhere in the documentation. This section describes our policy on which dependencies are acceptable. Generally ~~~~~~~~~ All dependencies must be stated and justified in the documentation. We want Charliecloud to run on as many systems as practical, so we work hard to keep dependencies minimal. However, because Charliecloud depends on new-ish kernel features, we do depend on standards of similar vintage. Core functionality should be available even on small systems with basic Linux distributions, so dependencies for run-time and build-time are only the bare essentials. Exceptions, to be used judiciously: * Features that add convenience rather than functionality may have additional dependencies that are reasonably expected on most systems where the convenience would be used. * Features that only work if some other software is present can add dependencies of that other software (e.g., :code:`ch-convert` depends on Docker to convert to/from Docker image storage). The test suite is tricky, because we need a test framework and to set up complex test fixtures. We have not yet figured out how to do this at reasonable expense with dependencies as tight as run- and build-time, so there are systems that do support Charliecloud but cannot run the test suite. Building the RPMs should work on RPM-based distributions with a kernel new enough to support Charliecloud. You might need to install additional packages (but not from third-party repositories). :code:`curl` vs. :code:`wget` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For URL downloading in shell code, including Dockerfiles, use :code:`wget -nv`. Both work fine for our purposes, and we need to use one or the other consistently. According to Debian’s popularity contest, 99.88% of reporting systems have :code:`wget` installed, vs. about 44% for :code:`curl`. On the other hand, :code:`curl` is in the minimal install of CentOS 7 while :code:`wget` is not. For now, Reid just picked :code:`wget` because he likes it better. Variable conventions in shell scripts and :code:`.bats` files ------------------------------------------------------------- * Separate words with underscores. * User-configured environment variables: all uppercase, :code:`CH_TEST_` prefix. Do not use in individual :code:`.bats` files; instead, provide an intermediate variable. * Variables local to a given file: lower case, no prefix. * Bats: set in :code:`common.bash` and then used in :code:`.bats` files: lower case, :code:`ch_` prefix. * Surround lower-case variables expanded in strings with curly braces, unless they’re the only thing in the string. E.g.: .. code-block:: none "${foo}/bar" # yes "$foo" # yes "$foo/bar" # no "${foo}" # no * Don’t quote variable assignments or other places where not needed (e.g., case statements). E.g.: .. code-block:: none foo=${bar}/baz # yes foo="${bar}/baz" # no Statement ordering within source files -------------------------------------- In general, we order things alphabetically. Python ~~~~~~ The module as a whole, and each class, comprise a sequence of ordering units separated by section header comments surrounded by two or more hashes, e.g. :code:`## Globals ##`. Sections with the following names must be in this order (omissions are fine). Other section names may appear in any order. There is also an unnamed zeroth section. #. Enums #. Constants #. Globals #. Exceptions #. Main #. Functions #. Supporting classes #. Core classes #. Classes Within each section, statements occur in the following order. #. imports #. standard library #. external imports not in the standard library #. :code:`import charliecloud` #. other Charliecloud imports #. assignments #. class definitions #. function definitions #. :code:`__init__` #. static methods #. class methods #. other double-underscore methods (e.g. :code:`__str__`) #. properties #. “normal” functions (instance methods) Within each group of statements above, identifiers must occur in alphabetical order. Exceptions: #. Classes must appear after their base class. #. Assignments may appear in any order. Statement types not listed above may appear in any order. A statement that must be out of order is exempted with a comment on its first line containing 👻, because a ghost says “OOO”, i.e. “out of order”. Python code ----------- Indentation width ~~~~~~~~~~~~~~~~~ `3 spaces `_ per level. No tab characters. C code ------ :code:`const` ~~~~~~~~~~~~~ The :code:`const` keyword is used to indicate that variables are read-only. It has a variety of uses; in Charliecloud, we use it for `function pointer arguments `_ to state whether or not the object pointed to will be altered by the function. For example: .. code-block:: c void foo(const char *in, char *out) is a function that will not alter the string pointed to by :code:`in` but may alter the string pointed to by :code:`out`. (Note that :code:`char const` is equivalent to :code:`const char`, but we use the latter order because that’s what appears in GCC error messages.) We do not use :code:`const` on local variables or function arguments passed by value. One could do this to be more clear about what is and isn’t mutable, but it adds quite a lot of noise to the source code, and in our evaluations didn’t catch any bugs. We also do not use it on double pointers (e.g., :code:`char **out` used when a function allocates a string and sets the caller’s pointer to point to it), because so far those are all out-arguments and C has `confusing rules `_ about double pointers and :code:`const`. Lists ~~~~~ The general convention is to use an array of elements terminated by an element containing all zeros (i.e., every byte is zero). While this precludes zero elements within the list, it makes it easy to iterate: .. code-block:: c struct foo { int a; float b; }; struct foo *bar = ...; for (int i = 0; bar[i].a != 0; i++) do_stuff(bar[i]); Note that the conditional checks that only one field of the struct (:code:`a`) is zero; this loop leverages knowledge of this specific data structure that checking only :code:`a` is sufficient. Lists can be set either as literals: .. code-block:: c struct foo bar[] = { {1, 2.0}, {3, 4.0}, {0, 0.0} }; or built up from scratch on the heap; the contents of this list are equivalent (note the C99 trick to avoid create a :code:`struct foo` variable): .. code-block:: c struct foo baz; struct foo *qux = list_new(sizeof(struct foo), 0); baz.a = 1; baz.b = 2.0; list_append((void **)&qux, &baz, sizeof(struct foo)); list_append((void **)&qux, &((struct foo){3, 4.0}), sizeof(struct foo)); This form of list should be used unless some API requires something else. .. warning:: Taking the address of an array in C yields the address of the first element, which is the same thing. For example, consider this list of strings, i.e. pointers to :code:`char`: .. code-block:: c char foo[] = "hello"; char **list = list_new(sizeof(char *), 0) list_append((void **)list, &foo, sizeof(char *)); // error! Because :code:`foo == &foo`, this will add to the list not a pointer to :code:`foo` but the *contents* of :code:`foo`, i.e. (on a machine with 64-bit pointers) :code:`'h'`, :code:`'e'`, :code:`'l'`, :code:`'l'`, :code:`'o'`, :code:`'\0'` followed by two bytes of whatever follows :code:`foo` in memory. This would work because :code:`bar != &bar`: .. code-block:: c char foo[] = "hello"; char bar = foo; char **list = list_new(sizeof(char *), 0) list_append((void **)list, &bar, sizeof(char *)); // OK Debugging ========= Python :code:`printf(3)`-style debugging ---------------------------------------- Consider :code:`ch.ILLERI()`. This uses the same mechanism as the standard logging functions (:code:`ch.INFO()`, :code:`ch.VERBOSE()`, etc.) but it (1) cannot be suppressed and (2) uses a color that stands out. All :code:`ch.ILLERI()` calls must be removed before a PR can be merged. :code:`seccomp(2)` BPF ---------------------- :code:`ch-run --seccomp -vv` will log the BPF instructions as they are computed, but it’s all in raw hex and hard to interpret, e.g.:: $ ch-run --seccomp -vv alpine:3.17 -- true [...] ch-run[62763]: seccomp: arch c00000b7: found 13 syscalls (ch_core.c:582) ch-run[62763]: seccomp: arch 40000028: found 27 syscalls (ch_core.c:582) [...] ch-run[62763]: seccomp(2) program has 156 instructions (ch_core.c:591) ch-run[62763]: 0: { op=20 k= 4 jt= 0 jf= 0 } (ch_core.c:423) ch-run[62763]: 1: { op=15 k=c00000b7 jt= 0 jf= 17 } (ch_core.c:423) ch-run[62763]: 2: { op=20 k= 0 jt= 0 jf= 0 } (ch_core.c:423) ch-run[62763]: 3: { op=15 k= 5b jt=145 jf= 0 } (ch_core.c:423) [...] ch-run[62763]: 154: { op= 6 k=7fff0000 jt= 0 jf= 0 } (ch_core.c:423) ch-run[62763]: 155: { op= 6 k= 50000 jt= 0 jf= 0 } (ch_core.c:423) ch-run[62763]: note: see FAQ to disassemble the above (ch_core.c:676) ch-run[62763]: executing: true (ch_core.c:538) You can instead use `seccomp-tools `_ to disassemble and pretty-print the BPF code in a far easier format, e.g.:: $ sudo apt install ruby-dev $ gem install --user-install seccomp-tools $ export PATH=~/.gem/ruby/3.1.0/bin:$PATH $ seccomp-tools dump -c 'ch-run --seccomp alpine:3.19 -- true' line CODE JT JF K ================================= 0000: 0x20 0x00 0x00 0x00000004 A = arch 0001: 0x15 0x00 0x11 0xc00000b7 if (A != ARCH_AARCH64) goto 0019 0002: 0x20 0x00 0x00 0x00000000 A = sys_number 0003: 0x15 0x91 0x00 0x0000005b if (A == aarch64.capset) goto 0149 [...] 0154: 0x06 0x00 0x00 0x7fff0000 return ALLOW 0155: 0x06 0x00 0x00 0x00050000 return ERRNO(0) Note that the disassembly is not perfect; e.g. if an architecture is not in your kernel headers, the system call name is wrong. OCI technical notes =================== This section describes our analysis of the Open Container Initiative (OCI) specification and implications for our implementations of :code:`ch-image`, and :code:`ch-run-oci`. Anything relevant for users goes in the respective man page; here is for technical details. The main goals are to guide Charliecloud development and provide and opportunity for peer-review of our work. ch-run-oci ---------- Currently, :code:`ch-run-oci` is only tested with Buildah. These notes describe what we are seeing from Buildah’s runtime expectations. Gotchas ~~~~~~~ Namespaces """""""""" Buildah sets up its own user and mount namespaces before invoking the runtime, though it does not change the root directory. We do not understand why. In particular, this means that you cannot see the container root filesystem it provides without joining those namespaces. To do so: #. Export :code:`CH_RUN_OCI_LOGFILE` with some logfile path. #. Export :code:`CH_RUN_OCI_DEBUG_HANG` with the step you want to examine (e.g., :code:`create`). #. Run :code:`ch-build -b buildah`. #. Make note of the PID in the logfile. #. :code:`$ nsenter -U -m -t $PID bash` Supervisor process and maintaining state """""""""""""""""""""""""""""""""""""""" OCI (and thus Buildah) expects a process that exists throughout the life of the container. This conflicts with Charliecloud’s lack of a supervisor process. Bundle directory ~~~~~~~~~~~~~~~~ * OCI documentation (very incomplete): https://github.com/opencontainers/runtime-spec/blob/master/bundle.md The bundle directory defines the container and is used to communicate between Buildah and the runtime. The root filesystem (:code:`mnt/rootfs`) is mounted within Buildah’s namespaces, so you’ll want to join them before examination. :code:`ch-run-oci` has restrictions on bundle directory path so it can be inferred from the container ID (see the man page). This lets us store state in the bundle directory instead of maintaining a second location for container state. Example:: # cd /tmp/buildah265508516 # ls -lR . | head -40 .: total 12 -rw------- 1 root root 3138 Apr 25 16:39 config.json d--------- 2 root root 40 Apr 25 16:39 empty -rw-r--r-- 1 root root 200 Mar 9 2015 hosts d--x------ 3 root root 60 Apr 25 16:39 mnt -rw-r--r-- 1 root root 79 Apr 19 20:23 resolv.conf ./empty: total 0 ./mnt: total 0 drwxr-x--- 19 root root 380 Apr 25 16:39 rootfs ./mnt/rootfs: total 0 drwxr-xr-x 2 root root 1680 Apr 8 14:30 bin drwxr-xr-x 2 root root 40 Apr 8 14:30 dev drwxr-xr-x 15 root root 720 Apr 8 14:30 etc drwxr-xr-x 2 root root 40 Apr 8 14:30 home [...] Observations: #. The weird permissions on :code:`empty` (000) and :code:`mnt` (100) persist within the namespaces, so you’ll want to be namespace root to look around. #. :code:`hosts` and :code:`resolv.conf` are identical to the host’s. #. :code:`empty` is still an empty directory with in the namespaces. What is this for? #. :code:`mnt/rootfs` contains the container root filesystem. It is a tmpfs. No other new filesystems are mounted within the namespaces. :code:`config.json` ~~~~~~~~~~~~~~~~~~~ * OCI documentation: * https://github.com/opencontainers/runtime-spec/blob/master/config.md * https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md This is the meat of the container configuration. Below is an example :code:`config.json` along with commentary and how it maps to :code:`ch-run` arguments. This was pretty-printed with :code:`jq . config.json`, and we re-ordered the keys to match the documentation. There are a number of additional keys that appear in the documentation but not in this example. These are all unsupported, either by ignoring them or throwing an error. The :code:`ch-run-oci` man page documents comprehensively what OCI features are and are not supported. .. code-block:: javascript { "ociVersion": "1.0.0", We validate that this is "1.0.0". .. code-block:: javascript "root": { "path": "/tmp/buildah115496812/mnt/rootfs" }, Path to root filesystem; maps to :code:`NEWROOT`. If key :code:`readonly` is :code:`false` or absent, add :code:`--write`. .. code-block:: javascript "mounts": [ { "destination": "/dev", "type": "tmpfs", "source": "/dev", "options": [ "private", "strictatime", "noexec", "nosuid", "mode=755", "size=65536k" ] }, { "destination": "/dev/mqueue", "type": "mqueue", "source": "mqueue", "options": [ "private", "nodev", "noexec", "nosuid" ] }, { "destination": "/dev/pts", "type": "devpts", "source": "pts", "options": [ "private", "noexec", "nosuid", "newinstance", "ptmxmode=0666", "mode=0620" ] }, { "destination": "/dev/shm", "type": "tmpfs", "source": "shm", "options": [ "private", "nodev", "noexec", "nosuid", "mode=1777", "size=65536k" ] }, { "destination": "/proc", "type": "proc", "source": "/proc", "options": [ "private", "nodev", "noexec", "nosuid" ] }, { "destination": "/sys", "type": "bind", "source": "/sys", "options": [ "rbind", "private", "nodev", "noexec", "nosuid", "ro" ] }, { "destination": "/etc/hosts", "type": "bind", "source": "/tmp/buildah115496812/hosts", "options": [ "rbind" ] }, { "destination": "/etc/resolv.conf", "type": "bind", "source": "/tmp/buildah115496812/resolv.conf", "options": [ "rbind" ] } ], This says what filesystems to mount in the container. It is a mix; it has tmpfses, bind-mounts of both files and directories, and other non-device-backed filesystems. The docs suggest a lot of flexibility, including stuff that won’t work in an unprivileged user namespace (e.g., filesystems backed by a block device). The things that matter seem to be the same as Charliecloud defaults. Therefore, for now we just ignore mounts. .. code-block:: javascript "process": { "terminal": true, This says that Buildah wants a pseudoterminal allocated. Charliecloud does not currently support that, so we error in this case. However, Buildah can be persuaded to set this :code:`false` if you redirect its standard input from :code:`/dev/null`, which is the current workaround. Things work fine. .. code-block:: javascript "cwd": "/", Maps to :code:`--cd`. .. code-block:: javascript "args": [ "/bin/sh", "-c", "apk add --no-cache bc" ], Maps to :code:`COMMAND [ARG ...]`. Note that we do not run :code:`ch-run` via the shell, so there aren’t worries about shell parsing. .. code-block:: javascript "env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "https_proxy=http://proxyout.lanl.gov:8080", "no_proxy=localhost,127.0.0.1,.lanl.gov", "HTTP_PROXY=http://proxyout.lanl.gov:8080", "HTTPS_PROXY=http://proxyout.lanl.gov:8080", "NO_PROXY=localhost,127.0.0.1,.lanl.gov", "http_proxy=http://proxyout.lanl.gov:8080" ], Environment for the container. The spec does not say whether this is the complete environment or whether it should be added to some default environment. We treat it as a complete environment, i.e., place the variables in a file and then :code:`--unset-env='*' --set-env=FILE`. .. code-block:: javascript "rlimits": [ { "type": "RLIMIT_NOFILE", "hard": 1048576, "soft": 1048576 } ] Process limits Buildah wants us to set with :code:`setrlimit(2)`. Ignored. .. code-block:: javascript "capabilities": { ... }, Long list of capabilities that Buildah wants. Ignored. (Charliecloud provides security by remaining an unprivileged process.) .. code-block:: javascript "user": { "uid": 0, "gid": 0 }, }, Maps to :code:`--uid=0 --gid=0`. .. code-block:: javascript "linux": { "namespaces": [ { "type": "pid" }, { "type": "ipc" }, { "type": "mount" }, { "type": "user" } ], Namespaces that Buildah wants. Ignored; Charliecloud just does user and mount. .. code-block:: javascript "uidMappings": [ { "hostID": 0, "containerID": 0, "size": 1 }, { "hostID": 1, "containerID": 1, "size": 65536 } ], "gidMappings": [ { "hostID": 0, "containerID": 0, "size": 1 }, { "hostID": 1, "containerID": 1, "size": 65536 } ], Describes the identity map between the namespace and host. Buildah wants it much larger than Charliecloud’s single entry and asks for container root to be host root, which we can’t do. Ignored. .. code-block:: javascript "maskedPaths": [ "/proc/acpi", "/proc/kcore", ... ], "readonlyPaths": [ "/proc/asound", "/proc/bus", ... ] Spec says to "mask over the provided paths ... so they cannot be read" and "sed the provided paths as readonly". Ignored. (Unprivileged user namespace protects us.) .. code-block:: javascript } } End of example. State ~~~~~ The OCI spec does not say how the JSON document describing state should be given to the caller. Buildah is happy to get it on the runtime’s standard output. :code:`ch-run-oci` provides an OCI compliant state document. Status :code:`creating` will never be returned, because the create operation is essentially a no-op, and annotations are not supported, so the :code:`annotations` key will never be given. Additional sources ~~~~~~~~~~~~~~~~~~ * :code:`buildah` man page: https://github.com/containers/buildah/blob/master/docs/buildah.md * :code:`buildah bud` man page: https://github.com/containers/buildah/blob/master/docs/buildah-bud.md * :code:`runc create` man page: https://raw.githubusercontent.com/opencontainers/runc/master/man/runc-create.8.md * https://github.com/opencontainers/runtime-spec/blob/master/runtime.md ch-image -------- pull ~~~~ Images pulled from registries come with OCI metadata, i.e. a "config blob". This is stored verbatim in :code:`/ch/config.pulled.json` for debugging. Charliecloud metadata, which includes a translated subset of the OCI config, is kept up to date in :code:`/ch/metadata.json`. push ~~~~ Image registries expect a config blob at push time. This blob consists of both OCI runtime and image specification information. * OCI run-time and image documentation: * https://github.com/opencontainers/runtime-spec/blob/master/config.md * https://github.com/opencontainers/image-spec/blob/master/config.md Since various OCI features are unsupported by Charliecloud we push only what is necessary to satisfy general image registry requirements. The pushed config is created on the fly, referencing the image’s metadata and layer tar hash. For example, including commentary: .. code-block:: javascript { "architecture": "amd64", "charliecloud_version": "0.26", "comment": "pushed with Charliecloud", "config": {}, "container_config": {}, "created": "2021-12-10T20:39:56Z", "os": "linux", "rootfs": { "diff_ids": [ "sha256:607c737779a53d3a04cbd6e59cae1259ce54081d9bafb4a7ab0bc863add22be8" ], "type": "layers" }, "weirdal": "yankovic" The fields above are expected by the registry at push time, with the exception of :code:`charliecloud_version` and :code:`weirdal`, which are Charliecloud extensions. .. code-block:: javascript "history": [ { "created": "2021-11-17T02:20:51.334553938Z", "created_by": "/bin/sh -c #(nop) ADD file:cb5ed7070880d4c0177fbe6dd278adb7926e38cd73e6abd582fd8d67e4bbf06c in / ", "empty_layer": true }, { "created": "2021-11-17T02:20:51.921052716Z", "created_by": "/bin/sh -c #(nop) CMD [\"bash\"]", "empty_layer": true }, { "created": "2021-11-30T20:14:08Z", "created_by": "FROM debian:buster", "empty_layer": true }, { "created": "2021-11-30T20:14:19Z", "created_by": "RUN ['/bin/sh', '-c', 'apt-get update && apt-get install -y bzip2 wget && rm -rf /var/lib/apt/lists/*']", "empty_layer": true }, { "created": "2021-11-30T20:14:19Z", "created_by": "WORKDIR /usr/local/src", "empty_layer": true }, { "created": "2021-11-30T20:14:19Z", "created_by": "ARG MC_VERSION='latest'", "empty_layer": true }, { "created": "2021-11-30T20:14:19Z", "created_by": "ARG MC_FILE='Miniconda3-latest-Linux-x86_64.sh'", "empty_layer": true }, { "created": "2021-11-30T20:14:21Z", "created_by": "RUN ['/bin/sh', '-c', 'wget -nv https://repo.anaconda.com/miniconda/$MC_FILE']", "empty_layer": true }, { "created": "2021-11-30T20:14:33Z", "created_by": "RUN ['/bin/sh', '-c', 'bash $MC_FILE -bf -p /usr/local']", "empty_layer": true }, { "created": "2021-11-30T20:14:33Z", "created_by": "RUN ['/bin/sh', '-c', 'rm -Rf $MC_FILE']", "empty_layer": true }, { "created": "2021-11-30T20:14:33Z", "created_by": "RUN ['/bin/sh', '-c', 'which conda && conda --version']", "empty_layer": true }, { "created": "2021-11-30T20:14:34Z", "created_by": "RUN ['/bin/sh', '-c', 'conda config --set auto_update_conda False']", "empty_layer": true }, { "created": "2021-11-30T20:14:34Z", "created_by": "RUN ['/bin/sh', '-c', 'conda config --add channels conda-forge']", "empty_layer": true }, { "created": "2021-11-30T20:15:07Z", "created_by": "RUN ['/bin/sh', '-c', 'conda install --yes obspy']", "empty_layer": true }, { "created": "2021-11-30T20:15:07Z", "created_by": "WORKDIR /", "empty_layer": true }, { "created": "2021-11-30T20:15:08Z", "created_by": "RUN ['/bin/sh', '-c', 'wget -nv http://examples.obspy.org/RJOB_061005_072159.ehz.new']", "empty_layer": true }, { "created": "2021-11-30T20:15:08Z", "created_by": "COPY ['hello.py'] -> '.'", "empty_layer": true }, { "created": "2021-11-30T20:15:08Z", "created_by": "RUN ['/bin/sh', '-c', 'chmod 755 ./hello.py']" } ], } The history section is collected from the image’s metadata and :code:`empty_layer` added to all entries except the last to represent a single-layer image. This is needed because Quay checks that the number of non-empty history entries match the number of pushed layers. Miscellaneous notes =================== Updating bundled Lark parser ---------------------------- In order to change the version of the bundled lark parser you must modify multiple files. To find them, e.g. for version 1.1.9 (the regex is hairy to catch both dot notation and tuples, but not the list of filenames in :code:`lib/Makefile.am`):: $ misc/grep -E '1(\.|, )1(\.|, )9($|\s|\))' What to do in each location should either be obvious or commented. .. LocalWords: milestoned gh nv cht Chacon’s scottchacon mis cantfix tmpimg .. LocalWords: rootfs cbd cae ce bafb bc weirdal yankovic nop cb fbe adb fd .. LocalWords: abd bbf LOGFILE logfile rtd Enums WIP rpmlintrc rhel ILLERI charliecloud-0.37/doc/faq.rst000066400000000000000000001524031457016721300161620ustar00rootroot00000000000000Frequently asked questions (FAQ) ******************************** .. contents:: :depth: 3 :local: About the project ================= Where did the name Charliecloud come from? ------------------------------------------ *Charlie* — Charles F. McMillan was director of Los Alamos National Laboratory from June 2011 until December 2017, i.e., at the time Charliecloud was started in early 2014. He is universally referred to as “Charlie” here. *cloud* — Charliecloud provides cloud-like flexibility for HPC systems. How do you spell Charliecloud? ------------------------------ We try to be consistent with *Charliecloud* — one word, no camel case. That is, *Charlie Cloud* and *CharlieCloud* are both incorrect. How large is Charliecloud? -------------------------- .. include:: _loc.rst Errors ====== How do I read the :code:`ch-run` error messages? ------------------------------------------------ :code:`ch-run` error messages look like this:: $ ch-run foo -- echo hello ch-run[25750]: can’t find image: foo: No such file or directory (ch-run.c:107 2) There is a lot of information here, and it comes in this order: 1. Name of the executable; always :code:`ch-run`. 2. Process ID in square brackets; here :code:`25750`. This is useful when debugging parallel :code:`ch-run` invocations. 3. Colon. 4. Main error message; here :code:`can’t find image: foo`. This should be informative as to what went wrong, and if it’s not, please file an issue, because you may have found a usability bug. Note that in some cases you may encounter the default message :code:`error`; if this happens and you’re not doing something very strange, that’s also a usability bug. 5. Colon (but note that the main error itself can contain colons too), if and only if the next item is present. 6. Operating system’s description of the the value of :code:`errno`; here :code:`No such file or directory`. Omitted if not applicable. 7. Open parenthesis. 8. Name of the source file where the error occurred; here :code:`ch-run.c`. This and the following item tell developers exactly where :code:`ch-run` became confused, which greatly improves our ability to provide help and/or debug. 9. Source line where the error occurred. 10. Value of :code:`errno` (see `C error codes in Linux `_ for the full list of possibilities). 11. Close parenthesis. *Note:* Despite the structured format, the error messages are not guaranteed to be machine-readable. :code:`ch-run` fails with “can’t re-mount image read-only” ---------------------------------------------------------- Normally, :code:`ch-run` re-mounts the image directory read-only within the container. This fails if the image resides on certain filesystems, such as NFS (see `issue #9 `_). There are two solutions: 1. Unpack the image into a different filesystem, such as :code:`tmpfs` or local disk. Consult your local admins for a recommendation. Note that Lustre is probably not a good idea because it can give poor performance for you and also everyone else on the system. 2. Use the :code:`-w` switch to leave the image mounted read-write. This may have an impact on reproducibility (because the application can change the image between runs) and/or stability (if there are multiple application processes and one writes a file in the image that another is reading or writing). :code:`ch-image` fails with "certificate verify failed" ------------------------------------------------------- When :code:`ch-image` interacts with a remote registry (e.g., via :code:`push` or :code:`pull` subcommands), it will verify the registry’s HTTPS certificate. If this fails, :code:`ch-image` will exit with the error "certificate verify failed". This situation tends to arise with self-signed or institutionally-signed certificates, even if the OS is configured to trust them. We use the Python HTTP library Requests, which on many platforms `includes its own CA certificates bundle `_, ignoring the bundle installed by the OS. Requests can be directed to use an alternate bundle of trusted CAs by setting environment variable :code:`REQUESTS_CA_BUNDLE` to the bundle path. (See `the Requests documentation `_ for details.) For example:: $ export REQUESTS_CA_BUNDLE=/usr/local/share/ca-certificates/registry.crt $ ch-image pull registry.example.com/image:tag Alternatively, certificate verification can be disabled entirely with the :code:`--tls-no-verify` flag. However, users should enable this option only if they have other means to be confident in the registry’s identity. "storage directory seems invalid" --------------------------------- Charliecloud uses its *storage directory* (:code:`/var/tmp/$USER.ch` by default) for various internal uses. As such, Charliecloud needs complete control over this directory’s contents. This error happens when the storage directory exists but its contents do not match what’s expected, including if it’s an empty directory, which is to protect against using common temporary directories like :code:`/tmp` or :code:`/var/tmp` as the storage directory. Let Charliecloud create the storage directory. For example, if you want to use :code:`/big/containers/$USER/charlie` for the storage directory (e.g., by setting :code:`CH_IMAGE_STORAGE`), ensure :code:`/big/containers/$USER` exists but do not create the final directory :code:`charlie`. "Transport endpoint is not connected" ------------------------------------- This error likely means that the SquashFS mount process has exited or been killed and you’re attempting to access the mount location. This is most often seen when a parallel launcher like :code:`srun` is used to run the mount command. :code:`srun` will see that the mount command has exited successfully and clean up all child processes, including that of the active mount. A workaround is to use a tool like :code:`pdsh`. For more details see Charliecloud issue `#230 `_. "fatal: :code:`$HOME` not set" from Git, or similar --------------------------------------------------- For example:: $ cat Dockerfile FROM alpine:3.17 RUN apk add git RUN git config --global http.sslVerify false $ ch-image build -t foo -f Dockerfile . 1 FROM alpine:3.17 2 RUN ['/bin/sh', '-c', 'apk add git'] [...] 3 RUN ['/bin/sh', '-c', 'git config --global http.sslVerify false'] fatal: $HOME not set error: build failed: RUN command exited with 128 The reason this happens is that :code:`ch-image build` executes :code:`RUN` instructions with :code:`ch-run` options including the absence of :code:`--home`, under which the environment variable :code:`$HOME` is unset. Thus, tools like Git that try to use it will fail. The reasoning for leaving the variable unset is that because Charliecloud runs unprivileged, it isn’t really meaningful for a container to have multiple users, and thus building images with things in the home directory is an antipattern. In fact, with :code:`--home` specified, :code:`ch-run` sets :code:`$HOME` to :code:`/home/$USER` and bind-mounts the user’s host home directory at that path. The concern with setting :code:`$HOME` to some default value during build is that it could simply hide the problem until runtime later, where it would be even more confusing. (That said, if this pattern is desired, it can be implemented with an :code:`ARG` or :code:`ENV` instruction.) The recommended workaround and best practice is to put configuration at the system level, not the user level. In the example above, this means changing :code:`git config --global` to :code:`git config --system`. See the man page for :code:`ch-run` for more on environment variable handling. :code:`ch-run` fails with “can’t execve(2): permission denied” -------------------------------------------------------------- For example:: $ ch-run /var/tmp/hello -- /bin/echo foo ch-run[154334]: error: can’t execve(2): /bin/echo: Permission denied (ch_core.c:387 13) But :code:`/bin/echo` *does* have execute permission:: $ ls -lh /var/tmp/hello/bin/echo -rwxr-xr-x 1 charlie charlie 51 Oct 8 2021 /var/tmp/hello/bin/echo In this case, the error indicates the container image is on a filesystem mounted with :code:`noexec`. To verify this, you can use e.g. :code:`findmnt(8)`:: $ findmnt TARGET SOURCE FSTYPE OPTIONS [...] └─/var/tmp tmpfs tmpfs rw,noexec,relatime,size=8675309k Note :code:`noexec` under :code:`OPTIONS`. To fix this, you can: 1. Use a different filesystem mounted :code:`exec` (i.e., the opposite of :code:`noexec` and typically the default). 2. Change the mount options for the filesystem (e.g., update :code:`/etc/fstab` or remount with :code:`exec`). 3. Use SquashFS format images (only for images exported from Charliecloud’s storage directory). Unexpected behavior =================== What do the version numbers mean? --------------------------------- Released versions of Charliecloud have a pretty standard version number, e.g. 0.9.7. Work leading up to a released version also has version numbers, to satisfy tools that require them and to give the executables something useful to report on :code:`--version`, but these can be quite messy. We refer to such versions informally as *pre-releases*, but Charliecloud does not have formal pre-releases such as alpha, beta, or release candidate. *Pre-release version numbers are not in order*, because this work is in a DAG rather than linear, except they precede the version we are working towards. If you’re dealing with these versions, use Git. Pre-release version numbers are the version we are working towards, followed by: :code:`~pre`, the branch name if not :code:`master` with non-alphanumerics removed, the commit hash, and finally :code:`dirty` if the working directory had uncommitted changes. Examples: * :code:`0.2.0` : Version 0.2.0. Released versions don’t include Git information, even if built in a Git working directory. * :code:`0.2.1~pre` : Some snapshot of work leading up to 0.2.1, built from source code where the Git information has been lost, e.g. the tarballs Github provides. This should make you wary because you don’t have any provenance. It might even be uncommitted work or an abandoned branch. * :code:`0.2.1~pre+1a99f42` : Master branch commit 1a99f42, built from a clean working directory (i.e., no changes since that commit). * :code:`0.2.1~pre+foo1.0729a78` : Commit 0729a78 on branch :code:`foo-1`, :code:`foo_1`, etc. built from clean working directory. * :code:`0.2.1~pre+foo1.0729a78.dirty` : Commit 0729a78 on one of those branches, plus un-committed changes. :code:`--uid 0` lets me read files I can’t otherwise! ----------------------------------------------------- Some permission bits can give a surprising result with a container UID of 0. For example:: $ whoami reidpr $ echo surprise > ~/cantreadme $ chmod 000 ~/cantreadme $ ls -l ~/cantreadme ---------- 1 reidpr reidpr 9 Oct 3 15:03 /home/reidpr/cantreadme $ cat ~/cantreadme cat: /home/reidpr/cantreadme: Permission denied $ ch-run /var/tmp/hello cat ~/cantreadme cat: /home/reidpr/cantreadme: Permission denied $ ch-run --uid 0 /var/tmp/hello cat ~/cantreadme surprise At first glance, it seems that we’ve found an escalation -- we were able to read a file inside a container that we could not read on the host! That seems bad. However, what is really going on here is more prosaic but complicated: 1. After :code:`unshare(CLONE_NEWUSER)`, :code:`ch-run` gains all capabilities inside the namespace. (Outside, capabilities are unchanged.) 2. This include :code:`CAP_DAC_OVERRIDE`, which enables a process to read/write/execute a file or directory mostly regardless of its permission bits. (This is why root isn’t limited by permissions.) 3. Within the container, :code:`exec(2)` capability rules are followed. Normally, this basically means that all capabilities are dropped when :code:`ch-run` replaces itself with the user command. However, if EUID is 0, which it is inside the namespace given :code:`--uid 0`, then the subprocess keeps all its capabilities. (This makes sense: if root creates a new process, it stays root.) 4. :code:`CAP_DAC_OVERRIDE` within a user namespace is honored for a file or directory only if its UID and GID are both mapped. In this case, :code:`ch-run` maps :code:`reidpr` to container :code:`root` and group :code:`reidpr` to itself. 5. Thus, files and directories owned by the host EUID and EGID (here :code:`reidpr:reidpr`) are available for all access with :code:`ch-run --uid 0`. This is not an escalation. The quirk applies only to files owned by the invoking user, because :code:`ch-run` is unprivileged outside the namespace, and thus he or she could simply :code:`chmod` the file to read it. Access inside and outside the container remains equivalent. References: * http://man7.org/linux/man-pages/man7/capabilities.7.html * http://lxr.free-electrons.com/source/kernel/capability.c?v=4.2#L442 * http://lxr.free-electrons.com/source/fs/namei.c?v=4.2#L328 .. _faq_mkdir-ro: :code:`--bind` creates mount points within un-writeable directories! -------------------------------------------------------------------- Consider this image:: $ ls /var/tmp/image bin dev home media opt root sbin sys usr ch etc lib mnt proc run srv tmp var $ ls -ld /var/tmp/image/mnt drwxr-xr-x 4 root root 80 Jan 5 09:52 /var/tmp/image/mnt $ ls /var/tmp/image/mnt bar foo That is, :code:`/mnt` is owned by root, un-writeable by us even considering the prior question, and contains two subdirectories. Indeed, we cannot create a new directory there:: $ mkdir /var/tmp/image/mnt/baz mkdir: cannot create directory ‘/var/tmp/image/mnt/baz’: Permission denied Recall that bind-mounting to a path that does not exist in a read-only image fails:: $ ch-run -b /tmp/baz:/mnt/baz /var/tmp/image -- ls /mnt ch-run[40498]: error: can't mkdir: /var/tmp/image/mnt/baz: Read-only file system (ch_misc.c:582 30) That’s fine; we’ll just use :code:`--write-fake` to create a writeable overlay on the container. Then we can make any mount points we need. Right? :: $ ch-run -W /var/tmp/image -- mkdir /qux # succeeds $ ch-run -W /var/tmp/image -- mkdir /mnt/baz # fails mkdir: can't create directory '/mnt/baz': Permission denied Wait — why could we create a subdirectory of (container path) :code:`/` but not a subdirectory of :code:`/mnt`? This is because the latter, which is at host path :code:`/var/tmp/image/mnt`, is not writeable by us: the overlayfs propagates the directory’s no-write permissions. Despite this, we can in fact use paths that do not yet exist for bind-mount destinations:: $ ch-run -W -b /tmp/baz:/mnt/baz /var/tmp/image -- ls /mnt bar baz foo What’s happening is bind-mount trickery and a symlink ranch. :code:`ch-run` creates a new directory on the overlaid tmpfs, bind-mounts the old (host path) :code:`/var/tmp/images/mnt` to a subdirectory of it, symlinks the old contents, and finally overmounts the old, un-writeable directory with the new one:: $ ch-run -W -b /tmp/baz:/mnt/baz /var/tmp/image -- ls -la /mnt drwxr-x--- 4 reidpr reidpr 120 Jan 5 17:11 . drwx------ 1 reidpr reidpr 40 Jan 5 17:11 .. drwxr-xr-x 4 nobody nogroup 80 Jan 5 16:52 .orig lrwxrwxrwx 1 reidpr reidpr 9 Jan 5 17:11 bar -> .orig/bar drwxr-x--- 2 reidpr reidpr 40 Jan 3 23:49 baz lrwxrwxrwx 1 reidpr reidpr 9 Jan 5 17:11 foo -> .orig/foo $ ch-run -W -b /tmp/baz:/mnt/baz /var/tmp/image -- cat /proc/mounts | fgrep ' /mnt' none /mnt tmpfs rw,relatime,size=3943804k,uid=1000,gid=1000,inode64 0 0 none /mnt/.orig overlay rw,relatime,lowerdir=/var/tmp/image,upperdir=/mnt/upper,workdir=/mnt/work,volatile,userxattr 0 0 tmpfs /mnt/baz tmpfs rw,relatime,size=8388608k,inode64 0 0 This new directory is writeable, and :code:`mkdir(2)` succeeds. (The overlaid tmpfs is mounted on *host* :code:`/mnt` during container assembly, which is why it appears in mount options.) There are differences from the original directory, of course. Most notably: * The ranched symlinks can be deleted by the user within the container, contrary to the old directory’s read-only permissions. * The contents of the “ranched” directory become symlinks rather than their original file type. Software that cares about these things may break. Why does :code:`ping` not work? ------------------------------- :code:`ping` fails with “permission denied” or similar under Charliecloud, even if you’re UID 0 inside the container:: $ ch-run $IMG -- ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes ping: permission denied (are you root?) $ ch-run --uid=0 $IMG -- ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes ping: permission denied (are you root?) This is because :code:`ping` needs a raw socket to construct the needed :code:`ICMP ECHO` packets, which requires capability :code:`CAP_NET_RAW` or root. Unprivileged users can normally use :code:`ping` because it’s a setuid or setcap binary: it raises privilege using the filesystem bits on the executable to obtain a raw socket. Under Charliecloud, there are multiple reasons :code:`ping` can’t get a raw socket. First, images are unpacked without privilege, meaning that setuid and setcap bits are lost. But even if you do get privilege in the container (e.g., with :code:`--uid=0`), this only applies in the container. Charliecloud uses the host’s network namespace, where your unprivileged host identity applies and :code:`ping` still can’t get a raw socket. The recommended alternative is to simply try the thing you want to do, without testing connectivity using :code:`ping` first. Why is MATLAB trying and failing to change the group of :code:`/dev/pts/0`? --------------------------------------------------------------------------- MATLAB and some other programs want pseudo-TTY (PTY) files to be group-owned by :code:`tty`. If it’s not, Matlab will attempt to :code:`chown(2)` the file, which fails inside a container. The scenario in more detail is this. Assume you’re user :code:`charlie` (UID=1000), your primary group is :code:`nerds` (GID=1001), :code:`/dev/pts/0` is the PTY file in question, and its ownership is :code:`charlie:tty` (:code:`1000:5`), as it should be. What happens in the container by default is: 1. MATLAB :code:`stat(2)`\ s :code:`/dev/pts/0` and checks the GID. 2. This GID is :code:`nogroup` (65534) because :code:`tty` (5) is not mapped on the host side (and cannot be, because only one’s EGID can be mapped in an unprivileged user namespace). 3. MATLAB concludes this is bad. 4. MATLAB executes :code:`chown("/dev/pts/0", 1000, 5)`. 5. This fails because GID 5 is not mapped on the guest side. 6. MATLAB pukes. The workaround is to map your EGID of 1001 to 5 inside the container (instead of the default 1001:1001), i.e. :code:`--gid=5`. Then, step 4 succeeds because the call is mapped to :code:`chown("/dev/pts/0", 1000, 1001)` and MATLAB is happy. :code:`ch-convert` from Docker incorrect image sizes ---------------------------------------------------- When converting from Docker, :code:`ch-convert` often finishes before the progress bar is complete. For example:: $ ch-convert -i docker foo /var/tmp/foo.tar.gz input: docker foo output: tar /var/tmp/foo.tar.gz exporting ... 373MiB 0:00:21 [============================> ] 65% [...] In this case, the :code:`.tar.gz` contains 392 MB uncompressed:: $ zcat /var/tmp/foo.tar.gz | wc 2740966 14631550 392145408 But Docker thinks the image is 597 MB:: $ sudo docker image inspect foo | fgrep -i size "Size": 596952928, "VirtualSize": 596952928, We’ve also seen cases where the Docker-reported size is an *under*\ estimate:: $ ch-convert -i docker bar /var/tmp/bar.tar.gz input: docker bar output: tar /var/tmp/bar.tar.gz exporting ... 423MiB 0:00:22 [============================================>] 102% [...] $ zcat /var/tmp/bar.tar.gz | wc 4181186 20317858 444212736 $ sudo docker image inspect bar | fgrep -i size "Size": 433812403, "VirtualSize": 433812403, We think that this is because Docker is computing size based on the size of the layers rather than the unpacked image. We do not currently have a fix; see `issue #165 `_. My password that contains digits doesn’t work in VirtualBox console ------------------------------------------------------------------- VirtualBox has confusing Num Lock behavior. Thus, you may be typing arrows, page up/down, etc. instead of digits, without noticing because console password fields give no feedback, not even whether a character has been typed. Try using the number row instead, toggling Num Lock key, or SSHing into the virtual machine. Mode bits (permission bits) are lost ------------------------------------ Charliecloud preserves only some mode bits, specifically user, group, and world permissions, and the `restricted deletion flag `_ on directories; i.e. 777 on files and 1777 on directories. The setuid (4000) and setgid (2000) bits are not preserved because ownership of files within Charliecloud images is that of the user who unpacks the image. Leaving these bits set could therefore surprise that user by unexpectedly creating files and directories setuid/gid to them. The sticky bit (1000) is not preserved for files because :code:`unsquashfs(1)` unsets it even with umask 000. However, this is bit is largely obsolete for files. Note the non-preserved bits may *sometimes* be retained, but this is undefined behavior. The specified behavior is that they may be zeroed at any time. Why is my wildcard in :code:`ch-run` not working? ------------------------------------------------- Be aware that wildcards in the :code:`ch-run` command are interpreted by the host, not the container, unless protected. One workaround is to use a sub-shell. For example:: $ ls /usr/bin/oldfind ls: cannot access '/usr/bin/oldfind': No such file or directory $ ch-run /var/tmp/hello.sqfs -- ls /usr/bin/oldfind /usr/bin/oldfind $ ls /usr/bin/oldf* ls: cannot access '/usr/bin/oldf*': No such file or directory $ ch-run /var/tmp/hello.sqfs -- ls /usr/bin/oldf* ls: cannot access /usr/bin/oldf*: No such file or directory $ ch-run /var/tmp/hello.sqfs -- sh -c 'ls /usr/bin/oldf*' /usr/bin/oldfind How do I ... ============ My app needs to write to :code:`/var/log`, :code:`/run`, etc. ------------------------------------------------------------- Because the image is mounted read-only by default, log files, caches, and other stuff cannot be written anywhere in the image. You have three options: 1. Configure the application to use a different directory. :code:`/tmp` is often a good choice, because it’s shared with the host and fast. 2. Use :code:`RUN` commands in your Dockerfile to create symlinks that point somewhere writeable, e.g. :code:`/tmp`, or :code:`/mnt/0` with :code:`ch-run --bind`. 3. Run the image read-write with :code:`ch-run -w`. Be careful that multiple containers do not try to write to the same files. OpenMPI Charliecloud jobs don’t work ------------------------------------ MPI can be finicky. This section documents some of the problems we’ve seen. :code:`mpirun` can’t launch jobs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For example, you might see:: $ mpirun -np 1 ch-run /var/tmp/mpihello-openmpi -- /hello/hello App launch reported: 2 (out of 2) daemons - 0 (out of 1) procs [cn001:27101] PMIX ERROR: BAD-PARAM in file src/dstore/pmix_esh.c at line 996 We’re not yet sure why this happens — it may be a mismatch between the OpenMPI builds inside and outside the container — but in our experience launching with :code:`srun` often works when :code:`mpirun` doesn’t, so try that. .. _faq_join: Communication between ranks on the same node fails ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ OpenMPI has many ways to transfer messages between ranks. If the ranks are on the same node, it is faster to do these transfers using shared memory rather than involving the network stack. There are two ways to use shared memory. The first and older method is to use POSIX or SysV shared memory segments. This approach uses two copies: one from Rank A to shared memory, and a second from shared memory to Rank B. For example, the :code:`sm` *byte transport layer* (BTL) does this. The second and newer method is to use the :code:`process_vm_readv(2)` and/or :code:`process_vm_writev(2)`) system calls to transfer messages directly from Rank A’s virtual memory to Rank B’s. This approach is known as *cross-memory attach* (CMA). It gives significant performance improvements in `benchmarks `_, though of course the real-world impact depends on the application. For example, the :code:`vader` BTL (enabled by default in OpenMPI 2.0) and :code:`psm2` *matching transport layer* (MTL) do this. The problem in Charliecloud is that the second approach does not work by default. We can demonstrate the problem with LAMMPS molecular dynamics application:: $ srun --cpus-per-task 1 ch-run /var/tmp/lammps_mpi -- \ lmp_mpi -log none -in /lammps/examples/melt/in.melt [cn002:21512] Read -1, expected 6144, errno = 1 [cn001:23947] Read -1, expected 6144, errno = 1 [cn002:21517] Read -1, expected 9792, errno = 1 [... repeat thousands of times ...] With :code:`strace(1)`, one can isolate the problem to the system call noted above:: process_vm_readv(...) = -1 EPERM (Operation not permitted) write(33, "[cn001:27673] Read -1, expected 6"..., 48) = 48 The `man page `_ reveals that these system calls require that the process have permission to :code:`ptrace(2)` one another, but sibling user namespaces `do not `_. (You *can* :code:`ptrace(2)` into a child namespace, which is why :code:`gdb` doesn’t require anything special in Charliecloud.) This problem is not specific to containers; for example, many settings of kernels with `YAMA `_ enabled will similarly disallow this access. So what can you do? There are a few options: * We recommend simply using the :code:`--join` family of arguments to :code:`ch-run`. This puts a group of :code:`ch-run` peers in the same namespaces; then, the system calls work. See the :doc:`ch-run` man page for details. * You can also sometimes turn off single-copy. For example, for :code:`vader`, set the MCA variable :code:`btl_vader_single_copy_mechanism` to :code:`none`, e.g. with an environment variable:: $ export OMPI_MCA_btl_vader_single_copy_mechanism=none :code:`psm2` does not let you turn off CMA, but it does fall back to two-copy if CMA doesn’t work. However, this fallback crashed when we tried it. * The kernel module `XPMEM `_ enables a different single-copy approach. We have not yet tried this, and the module needs to be evaluated for user namespace safety, but it’s quite a bit faster than CMA on benchmarks. .. Images by URL only works in Sphinx 1.6+. Debian Stretch has 1.4.9, so remove it for now. .. image:: https://media.giphy.com/media/1mNBTj3g4jRCg/giphy.gif :alt: Darth Vader bowling a strike with the help of the Force :align: center I get a bunch of independent rank-0 processes when launching with :code:`srun` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For example, you might be seeing this:: $ srun ch-run /var/tmp/mpihello-openmpi -- /hello/hello 0: init ok cn036.localdomain, 1 ranks, userns 4026554634 0: send/receive ok 0: finalize ok 0: init ok cn035.localdomain, 1 ranks, userns 4026554634 0: send/receive ok 0: finalize ok We were expecting a two-rank MPI job, but instead we got two independent one-rank jobs that did not coordinate. MPI ranks start as normal, independent processes that must find one another somehow in order to sync up and begin the coupled parallel program; this happens in :code:`MPI_Init()`. There are lots of ways to do this coordination. Because we are launching with the host’s Slurm, we need it to provide something for the containerized processes for such coordination. OpenMPI must be compiled to use what that Slurm has to offer, and Slurm must be told to offer it. What works for us is a something called "PMIx". You can see if your Slurm supports it with:: $ srun --mpi=list cray_shasta none pmi2 pmix If :code:`pmix` is not in the list, you must either (a) ask your admins to enable Slurm’s PMIx support, or (b) rebuild your container MPI against an PMI in the list. If it is in the list, but you’re seeing this problem, that means it is not the default, and you need to tell Slurm you want it. Try:: $ srun --mpi=pmix ch-run /var/tmp/mpihello-openmpi -- /hello/hello 0: init ok wc035.localdomain, 2 ranks, userns 4026554634 1: init ok wc036.localdomain, 2 ranks, userns 4026554634 0: send/receive ok 0: finalize ok How do I run X11 apps? ---------------------- X11 applications should “just work”. For example, try this Dockerfile: .. code-block:: docker FROM debian:stretch RUN apt-get update \ && apt-get install -y xterm Build it and unpack it to :code:`/var/tmp`. Then:: $ ch-run /scratch/ch/xterm -- xterm should pop an xterm. If your X11 application doesn’t work, please file an issue so we can figure out why. How do I specify an image reference? ------------------------------------ You must specify an image for many use cases, including :code:`FROM` instructions, the source of an image pull (e.g. :code:`ch-image pull` or :code:`docker pull`), the destination of an image push, and adding image tags. Charliecloud calls this an *image reference*, but there appears to be no established name for this concept. The syntax of an image reference is not well documented. This FAQ represents our understanding, which is cobbled together from the `Dockerfile reference `_, the :code:`docker tag` `documentation `_, and various forum posts. It is not a precise match for how Docker implements it, but it should be close enough. We’ll start with two complete examples with all the bells and whistles: 1. :code:`example.com:8080/foo/bar/hello-world:version1.0` 2. :code:`example.com:8080/foo/bar/hello-world@sha256:f6c68e2ad82a` These references parse into the following components, in this order: 1. A `valid hostname `_; we assume this matches the regular expression :code:`[A-Za-z0-9.-]+`, which is very approximate. Optional; here :code:`example.com`. 2. A colon followed by a decimal port number. If hostname is given, optional; otherwise disallowed; here :code:`8080`. 3. If hostname given, a slash. 4. A path, with one or more components separated by slash. Components match the regex :code:`[a-z0-9_.-]+`. Optional; here :code:`foo/bar`. Pedantic details: * Under the hood, the default path is :code:`library`, but this is generally not exposed to users. * Three or more underscores in a row is disallowed by Docker, but we don’t check this. 5. If path given, a slash. 6. The image name (tag), which matches :code:`[a-z0-9_.-]+`. Required; here :code:`hello-world`. 7. Zero or one of: * A tag matching the regular expression :code:`[A-Za-z0-9_.-]+` and preceded by a colon. Here :code:`version1.0` (example 1). * A hexadecimal hash preceded by the string :code:`@sha256:`. Here :code:`f6c68e2ad82a` (example 2). * Note: Digest algorithms other than SHA-256 are in principle allowed, but we have not yet seen any. Detail-oriented readers may have noticed the following gotchas: * A hostname without port number is ambiguous with the leading component of a path. For example, in the reference :code:`foo/bar/baz`, it is ambiguous whether :code:`foo` is a hostname or the first (and only) component of the path :code:`foo/bar`. The `resolution rule `_ is: if the ambiguous substring contains a dot, assume it’s a hostname; otherwise, assume it’s a path component. * The only character that cannot go in a POSIX filename is slash. Thus, Charliecloud uses image references in filenames, replacing slash with percent (:code:`%`). Because this character cannot appear in image references, the transformation is reversible. Git branch names do not allow a colon. Thus, to maintain the image reference as both the image filename and git branch in storage, we replace the colon with plus (:code:`+`). An alternate approach would be to replicate the reference path in the filesystem, i.e., path components in the reference would correspond directly to a filesystem path. This would yield a clearer filesystem structure. However, we elected not to do it because it complicates the code to save and clean up image reference-related data, and it does not address a few related questions, e.g. should the host and port also be a directory level. Usually, most of the components are omitted. For example, you’ll more commonly see image references like: * :code:`debian`, which refers to the tag :code:`latest` of image :code:`debian` from Docker Hub. * :code:`debian:stretch`, which is the same except for tag :code:`stretch`. * :code:`fedora/httpd`, which is tag :code:`latest` of :code:`fedora/httpd` from Docker Hub. See :code:`charliecloud.py` for a specific grammar that implements this. Can I build or pull images using a tool Charliecloud doesn’t know about? ------------------------------------------------------------------------ Yes. Charliecloud deals in well-known UNIX formats like directories, tarballs, and SquashFS images. So, once you get your image into some format Charliecloud likes, you can enter the workflow. For example, `skopeo `_ is a tool to pull images to OCI format, and `umoci `_ can flatten an OCI image to a directory. Thus, you can use the following commands to run an Alpine 3.9 image pulled from Docker hub:: $ skopeo copy docker://alpine:3.17 oci:/tmp/oci:img [...] $ ls /tmp/oci blobs index.json oci-layout $ umoci unpack --rootless --image /tmp/oci:img /tmp/alpine:3.17 [...] $ ls /tmp/alpine:3.17 config.json rootfs sha256_2ca27acab3a0f4057852d9a8b775791ad8ff62fbedfc99529754549d33965941.mtree umoci.json $ ls /tmp/alpine:3.17/rootfs bin etc lib mnt proc run srv tmp var dev home media opt root sbin sys usr $ ch-run /tmp/alpine:3.17/rootfs -- cat /etc/alpine-release 3.9.5 How do I authenticate with SSH during :code:`ch-image` build? ------------------------------------------------------------- The simplest approach is to run the `SSH agent `_ on the host. :code:`ch-image` then leverages this with two steps: 1. pass environment variable :code:`SSH_AUTH_SOCK` into the build, with no need to put :code:`ARG` in the Dockerfile or specify :code:`--build-arg` on the command line; and 2. bind-mount host :code:`/tmp` to guest :code:`/tmp`, which is where the SSH agent’s listening socket usually resides. Thus, SSH within the container will use this existing SSH agent on the host to authenticate without further intervention. For example, after making :code:`ssh-agent` available on the host, which is OS and site-specific:: $ echo $SSH_AUTH_SOCK /tmp/ssh-rHsFFqwwqh/agent.49041 $ ssh-add Enter passphrase for /home/charlie/.ssh/id_rsa: Identity added: /home/charlie/.ssh/id_rsa (/home/charlie/.ssh/id_rsa) $ ssh-add -l 4096 SHA256:aN4n2JeMah2ekwhyHnb0Ug9bYMASmY+5uGg6MrieaQ /home/charlie/.ssh/id_rsa (RSA) $ cat ./Dockerfile FROM alpine:latest RUN apk add openssh RUN echo $SSH_AUTH_SOCK RUN ssh git@github.com $ ch-image build -t foo -f ./Dockerfile . [...] 3 RUN ['/bin/sh', '-c', 'echo $SSH_AUTH_SOCK'] /tmp/ssh-rHsFFqwwqh/agent.49041 4 RUN ['/bin/sh', '-c', 'ssh git@github.com'] [...] Hi charlie! You’ve successfully authenticated, but GitHub does not provide shell access. Note this example is rather contrived — bare SSH sessions in a Dockerfile rarely make sense. In practice, SSH is used as a transport to fetch something, e.g. with :code:`scp(1)` or :code:`git(1)`. See the next entry for a more realistic example. SSH stops :code:`ch-image` build with interactive queries --------------------------------------------------------- This often occurs during an SSH-based Git clone. For example: .. code-block:: docker FROM alpine:latest RUN apk add git openssh RUN git clone git@github.com:hpc/charliecloud.git .. code-block:: console $ ch-image build -t foo -f ./Dockerfile . [...] 3 RUN ['/bin/sh', '-c', 'git clone git@github.com:hpc/charliecloud.git'] Cloning into 'charliecloud'... The authenticity of host 'github.com (140.82.113.3)' can’t be established. RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8. Are you sure you want to continue connecting (yes/no/[fingerprint])? At this point, the build stops while SSH waits for input. This happens even if you have :code:`github.com` in your :code:`~/.ssh/known_hosts`. This file is not available to the build because :code:`ch-image` runs :code:`ch-run` without :code:`--home`, so :code:`RUN` instructions can’t see anything in your home directory. Solutions include: 1. Change to anonymous HTTPS clone, if available. Most public repositories will support this. For example: .. code-block:: docker FROM alpine:latest RUN apk add git RUN git clone https://github.com/hpc/charliecloud.git 2. Approve the connection interactively by typing :code:`yes`. Note this will record details of the connection within the image, including IP address and the fingerprint. The build also remains interactive. 3. Edit the image’s system `SSH config `_ to turn off host key checking. Note this can be rather hairy, because the SSH config language is quite flexible and the first instance of a directive is the one used. However, often the changes can be simply appended: .. code-block:: docker FROM alpine:latest RUN apk add git openssh RUN printf 'StrictHostKeyChecking=no\nUserKnownHostsFile=/dev/null\n' \ >> /etc/ssh/ssh_config RUN git clone git@github.com:hpc/charliecloud.git Check your institutional policy on whether this is permissible, though it’s worth noting that users `almost never `_ verify the host fingerprints anyway. This will not record details of the connection in the image. 4. Turn off host key checking on the SSH command line. (See caveats in the previous item.) The wrapping tool should provide a way to configure this command line. For example, for Git: .. code-block:: docker FROM alpine:latest RUN apk add git openssh ARG GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" RUN git clone git@github.com:hpc/charliecloud.git 5. Add the remote host to the system known hosts file, e.g.: .. code-block:: docker FROM alpine:latest RUN apk add git openssh RUN echo 'github.com,140.82.112.4 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==' >> /etc/ssh/ssh_known_hosts RUN git clone git@github.com:hpc/charliecloud.git This records connection details in both the Dockerfile and the image. Other approaches could be found with web searches such as "automate unattended SSH" or "SSH in cron jobs". .. _faq_building-with-docker: How do I use Docker to build Charliecloud images? ------------------------------------------------- The short version is to run Docker commands like :code:`docker build` and :code:`docker pull` like usual, and then use :code:`ch-convert` to copy the image from Docker storage to a SquashFS archive, tarball, or directory. If you are behind an HTTP proxy, that requires some extra setup for Docker; see below. Security implications of Docker ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Because Docker (a) makes installing random crap from the internet simple and (b) is easy to deploy insecurely, you should take care. Some of the implications are below. This list should not be considered comprehensive nor a substitute for appropriate expertise; adhere to your ethical and institutional responsibilities. * **Docker equals root.** Anyone who can run the :code:`docker` command or interact with the Docker daemon can `trivially escalate to root `_. This is considered a feature. For this reason, don’t create the :code:`docker` group, as this will allow passwordless, unlogged escalation for anyone in the group. Run it with :code:`sudo docker`. Also, Docker runs container processes as root by default. In addition to being poor hygiene, this can be an escalation path, e.g. if you bind-mount host directories. * **Docker alters your network configuration.** To see what it did:: $ ifconfig # note docker0 interface $ brctl show # note docker0 bridge $ route -n * **Docker installs services.** If you don’t want the Docker service starting automatically at boot, e.g.:: $ systemctl is-enabled docker enabled $ systemctl disable docker $ systemctl is-enabled docker disabled Configuring for a proxy ~~~~~~~~~~~~~~~~~~~~~~~ By default, Docker does not work if you are behind a proxy, and it fails in two different ways. The first problem is that Docker itself must be told to use a proxy. This manifests as:: $ sudo docker run hello-world Unable to find image 'hello-world:latest' locally Pulling repository hello-world Get https://index.docker.io/v1/repositories/library/hello-world/images: dial tcp 54.152.161.54:443: connection refused If you have a systemd system, the `Docker documentation `_ explains how to configure this. If you don’t have a systemd system, then :code:`/etc/default/docker` might be the place to go? The second problem is that programs executed during build (:code:`RUN` instructions) need to know about the proxy as well. This manifests as images failing to build because they can’t download stuff from the internet. One fix is to configure your :code:`.bashrc` or equivalent to: 1. Set the proxy environment variables: .. code-block:: sh export HTTP_PROXY=http://proxy.example.com:8088 export http_proxy=$HTTP_PROXY export HTTPS_PROXY=$HTTP_PROXY export https_proxy=$HTTP_PROXY export NO_PROXY='localhost,127.0.0.1,.example.com' export no_proxy=$NO_PROXY 2. Configure a :code:`docker build` wrapper: .. code-block:: sh # Run "docker build" with specified arguments, adding proxy variables if # set. Assumes "sudo" is needed to run "docker". function docker-build () { if [[ -z $HTTP_PROXY ]]; then sudo docker build "$@" else sudo docker build --build-arg HTTP_PROXY="$HTTP_PROXY" \ --build-arg HTTPS_PROXY="$HTTPS_PROXY" \ --build-arg NO_PROXY="$NO_PROXY" \ --build-arg http_proxy="$http_proxy" \ --build-arg https_proxy="$https_proxy" \ --build-arg no_proxy="$no_proxy" \ "$@" fi } How can I build images for a foreign architecture? -------------------------------------------------- QEMU ~~~~ Suppose you want to build Charliecloud containers on a system which has a different architecture from the target system. It’s straightforward as long as you can install suitable packages on the build system (your personal computer?). You just need the magic of QEMU via a distribution package with a name like Debian’s :code:`qemu-user-static`. For use in an image root this needs to be the :code:`-static` version, not plain :code:`qemu-user`, and contain a :code:`qemu-*-static` executable for your target architecture. In case it doesn’t install “binfmt” hooks (telling Linux how to run foreign binaries), you’ll need to make that work — perhaps it’s in another package. That’s all you need to make building with :code:`ch-image` work with a base foreign architecture image and the :code:`--arch` option. It’s significantly slower than native, but quite usable — about half the speed of native for the ppc64le target with a build taking minutes on a laptop with a magnetic disc. There’s a catch that images in :code:`ch-image` storage aren’t distinguished by architecture except by any name you give them, e.g., a base image like :code:`debian:11` pulled with :code:`--arch ppc64le` will overwrite a native x86 one. For example, to build a ppc64le image on a Debian Buster amd64 host:: $ uname -m x86_64 $ sudo apt install qemu-user-static $ ch-image pull --arch ppc64le alpine:3.17 $ printf 'FROM alpine:3.17\nRUN apk add coreutils\n' | ch-image build -t foo - $ ch-convert alpine:3.17 /var/tmp/foo $ ch-run /var/tmp/foo -- uname -m ppc64le PRoot ~~~~~ Another way to build a foreign image, which works even without :code:`sudo` to install :code:`qemu-*-static`, is to populate a chroot for it with the `PRoot `_ tool, whose :code:`-q` option allows specifying a :code:`qemu-*-static` binary (perhaps obtained by unpacking a distribution package). How can I use tarball base images from e.g. linuxcontainers.org? ---------------------------------------------------------------- If you can’t find an image repository from which to pull for the distribution and architecture of interest, it is worth looking at the extensive collection of rootfs archives `maintained by linuxcontainers.org `_. They are meant for LXC, but are fine as a basis for Charliecloud. For example, this would leave a :code:`ppc64le/alpine:3.17` image du jour in the registry for use in a Dockerfile :code:`FROM` line. Note that linuxcontainers.org uses the opposite order for “le” in the architecture name. :: $ wget https://uk.lxd.images.canonical.com/images/alpine/3.15/ppc64el/default/20220304_13:00/rootfs.tar.xz $ ch-image import rootfs.tar.xz ppc64le/alpine:3.17 .. _faq_verbosity: How can I control Charliecloud’s quietness or verbosity? -------------------------------------------------------- Charliecloud logs various chatter about what is going on to standard error. This is distinct from *output*, e.g., :code:`ch-image list` prints the list of images to standard output. We use reasonably standard log levels: 1. **Error**. Some error condition that makes it impossible to proceed. The program exits soon after printing the error. Examples: unknown image type, Dockerfile parse error. (There is an internal distinction between “fatal” and “error” levels, but this isn’t really meaningful to users.) 2. **Warning**. Unexpected condition the user needs to know about but that should not stop the program. Examples: :code:`ch-run --mount` with a directory image (which does not use a mount point), unsupported Dockerfile instructions that are ignored. 3. **Info**. Chatter useful enough to be printed by default. Example: progress messages during image download and unpacking. (:code:`ch-run` is silent during normal operations and does not have any “info” logging.) 4. **Verbose**. Diagnostic information useful for debugging user containers, the Charliecloud installation, and Charliecloud itself. Examples: :code:`ch-run --join` coordination progress, :code:`ch-image` internal paths, Dockerfile parse tree. 5. **Debug**. More detailed diagnostic information useful for debugging Charliecloud. Examples: data structures unserialized from image registry metadata JSON, image reference parse tree. 6. **Trace**; printed if :code:`-vvv`. Grotesquely detailed diagnostic information for debugging Charliecloud, to the extent it interferes with normal use. A sensible person might use a `debugger `_ instead. Examples: component-by-component progress of bind-mount target directory analysis/creation, text of image registry JSON, every single file unpacked from image layers. Charliecloud also runs sub-programs at various times, notably commands in :code:`RUN` instructions and :code:`git(1)` to manage the build cache. These programs have their own standard error and standard output streams, which Charliecloud either suppresses or passes through depending on verbosity level. Most Charliecloud programs accept :code:`-v` to increase logging verbosity and :code:`-q` to decrease it. Generally: * Each :code:`-v` (up to three) makes Charliecloud noisier. * :code:`-q` suppresses normal logging. * :code:`-qq` also suppresses stdout for the program and its subprocesses, and warnings from the program. * :code:`-qqq` also suppresses subprocess stderr. (This means subprocesses are completely silenced no matter what goes wrong!) This table list which logging is printed at which verbosity levels (✅ indicates printed, ❌ suppressed). .. list-table:: :header-rows: 1 * - - :code:`-vvv` - :code:`-vv` - :code:`-v` - def. - :code:`-q` - :code:`-qq` - :code:`-qqq` * - trace - ✅ - ❌ - ❌ - ❌ - ❌ - ❌ - ❌ * - debug - ✅ - ✅ - ❌ - ❌ - ❌ - ❌ - ❌ * - verbose - ✅ - ✅ - ✅ - ❌ - ❌ - ❌ - ❌ * - info - ✅ - ✅ - ✅ - ✅ - ❌ - ❌ - ❌ * - program stdout - ✅ - ✅ - [1] - [1] - [1] - ❌ - ❌ * - subprocess stdout - ✅ - ✅ - [1] - [1] - [1] [2] - ❌ - ❌ * - warning - ✅ - ✅ - ✅ - ✅ - ✅ - ❌ - ❌ * - subprocess stderr - ✅ - ✅ - ✅ - ✅ - ✅ - ✅ - ❌ * - error - ✅ - ✅ - ✅ - ✅ - ✅ - ✅ - ✅ Notes: 1. Charliecloud handles subprocess stdout on case-by-case basis for these log levels. For example, sometimes it’s passed through by default (e.g., :code:`RUN`) and sometimes it’s captured for internal use (e.g., many :code:`git(1)` invocations). 2. In the case of :code:`ch-run`, the user command is considered a subprocess, e.g. :code:`ch-run -q example -- echo foo` will produce no output. .. _faq_xattrs: How do I handle extended attributes in Charliecloud? ---------------------------------------------------- As noted in section :ref:`ch-image_build-cache`, Charliecloud doesn’t support extended attributes (xattrs) by default. Support for xattrs can be enabled for :code:`ch-image` and :code:`ch-convert` by specifying :code:`--xattrs` or setting :code:`$CH_XATTRS`. This will make :code:`ch-image` save and restore xattrs via the build cache, and will make :code:`ch-convert` preserve xattrs on conversion. Important caveats include: 1. :code:`ch-image` and :code:`ch-convert` cannot read xattrs in privileged namespaces (e.g. :code:`trusted` and :code:`security`). Extended attributes in these namespaces will never be saved or restored via the cache, and will never be preserved when converting between image formats. 2. :code:`ch-image import` cannot handle xattrs. This is a limitation of the Python `tarfile `_ library, which as of version 3.12.1 doesn’t support xattrs (see CPython issue `#113293 `_). 3. :code:`ch-convert -o ch-image` uses :code:`ch-image import` under the hood. This in conjunction with (2) means that :code:`ch-convert` cannot preserve xattrs when converting to the :code:`ch-image` format. 4. :code:`ch-image pull` uses the tarfile library, so xattrs will be lost when pulling from a registry. 5. Support for xattrs varies among filesystems, e.g. tmpfs didn’t support xattrs in the :code:`user` namespace prior to Linux kernel `upstream 6.6 `_ (Oct 2023). .. LocalWords: CAs SY Gutmann AUTH rHsFFqwwqh MrieaQ Za loc mpihello mvo du .. LocalWords: VirtualSize linuxcontainers jour uk lxd rwxr xr qq qqq drwxr .. LocalWords: drwx charliecloud-0.37/doc/favicon.ico000066400000000000000000000104761457016721300170050ustar00rootroot00000000000000  (( @ #.#.[Op 6AGp[z3z [Y p(p3kA [3 O[3?(%[SG-"1= pfi'qfffQfY(q3LQ0  2=zfHf(([G \)charliecloud-0.37/doc/index.rst000066400000000000000000000010451457016721300165150ustar00rootroot00000000000000Overview ******** .. image:: rd100-winner.png :align: right :alt: R&D 100 2018 winner logo :width: 128px :target: https://www.lanl.gov/discover/news-release-archive/2018/November/1119-rd-100-awards.php .. include:: ../README.rst .. note:: This documentation is for Charliecloud version |version| and was built |today|. .. toctree:: :numbered: :hidden: install tutorial ch-checkns ch-completion.bash ch-convert ch-fromhost ch-image ch-run ch-run-oci ch-test faq best_practices dev charliecloud-0.37/doc/install.rst000066400000000000000000000512021457016721300170540ustar00rootroot00000000000000Installing ********** .. admonition:: Audience This section assumes a moderate level of experience installing UNIX software. This section describes what you need to install Charliecloud and how to do so. Note that installing and using Charliecloud can be done as a normal user with no elevated privileges, provided that user namespaces have been enabled. .. contents:: :depth: 2 :local: Build and install from source ============================= Using release tarball --------------------- We provide `tarballs `_ with a fairly standard :code:`configure` script. Thus, build and install can be as simple as:: $ ./configure $ make $ sudo make install If you don’t have sudo, you can: * Run Charliecloud directly from the build directory; add :code:`$BUILD_DIR/bin` to your :code:`$PATH` and you are good to go, without :code:`make install`. * Install in a prefix you have write access to, e.g. in your home directory with :code:`./configure --prefix=~`. :code:`configure` will provide a detailed report on what will be built and installed, along with what dependencies are present and missing. From Git checkout ----------------- If you obtain the source code with Git, you must build :code:`configure` and friends yourself. To do so, you will need the following. The versions in most common distributions should be sufficient. * Automake * Autoconf * Python’s :code:`pip3` package installer and its :code:`wheel` extension Create :code:`configure` with:: $ ./autogen.sh This script has a few options; see its :code:`--help`. Note that Charliecloud disables Automake’s "maintainer mode" by default, so the build system (Makefiles, :code:`configure`, etc.) will never automatically be rebuilt. You must run :code:`autogen.sh` manually if you need this. You can also re-enable maintainer mode with :code:`configure` if you like, though this is not a tested configuration. :code:`configure` options ------------------------- Charliecloud’s :code:`configure` has the following options in addition to the standard ones. Feature selection: :code:`--disable-FOO` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By default, all features that can be built will be built and installed. You can exclude some features with: ========================== ======================================================= option don’t build/install ========================== ======================================================= :code:`--disable-ch-image` :code:`ch-image` unprivileged builder & image manager :code:`--disable-html` HTML documentation :code:`--disable-man` man pages :code:`--disable-syslog` logging to syslog (see individual man pages) :code:`--disable-tests` test suite ========================== ======================================================= You can also say :code:`--enable-FOO` to fail the build if :code:`FOO` can’t be built. Dependency selection: :code:`--with-FOO` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Some dependencies can be specified as follows. Note only some of these support :code:`--with-FOO=no`, as listed. :code:`--with-libsquashfuse={yes,no,PATH}` Whether to link with :code:`libsquashfuse`. Options: * If not specified: Look for :code:`libsquashfuse` in standard install locations and link with it if found. Otherwise disable internal SquashFS mount, with no warning or error. * :code:`yes`: Look for :code:`libsquashfuse` in standard locations and link with it if found; otherwise, error. * :code:`no`: Disable :code:`libsquashfuse` linking and internal SquashFS mounting, even if it’s installed. * Path to :code:`libsquashfuse` install prefix: Link with :code:`libsquashfuse` found there, or error if not found, and add it to :code:`ch-run`'s RPATH. (Note this argument is *not* the directory containing the shared library or header file.) **Note:** A very specific version and configuration of SquashFUSE is required. See below for details. :code:`--with-python=SHEBANG` Shebang line to use for Python scripts. Default: :code:`/usr/bin/env python3`. :code:`--with-sphinx-build=PATH` Path to :code:`sphinx-build` executable. Default: the :code:`sphinx-build` found first in :code:`$PATH`. :code:`--with-sphinx-python=PATH` Path to Python used by :code:`sphinx-build`. Default: shebang of :code:`sphinx-build`. Less strict build: :code:`--enable-buggy-build` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Please do not use this option routinely, as that hides bugs that we cannot find otherwise.* By default, Charliecloud builds with :code:`CFLAGS` including :code:`-Wall -Werror`. The principle here is that we prefer diagnostics that are as noisy as practical, so that problems are identified early and we can fix them. We prefer :code:`-Werror` unless there is a specific reason to turn it off. For example, this approach identified a buggy :code:`configure` test (`issue #798 `_). Many others recommend the opposite. For example, Gentoo’s "`Common mistakes `_" guide advises against :code:`-Werror` because it causes breakage that is "random" and "without purpose". There is a well-known `blog post `_ from Flameeyes that recommends :code:`-Werror` be off by default and used by developers and testers only. In our opinion, for Charliecloud, these warnings are most likely the result of real bugs and shouldn’t be hidden (i.e., they are neither random nor without purpose). Our code should have no warnings, regardless of compiler, and any spurious warnings should be silenced individually. We do not have the resources to test with a wide variety of compilers, so enabling :code:`-Werror` only for development and testing, as recommended by others, means that we miss potentially important diagnostics — people typically do not pay attention to warnings, only errors. That said, we recognize that packagers and end users just want to build the code with a minimum of hassle. Thus, we provide the :code:`configure` flag: :code:`--enable-buggy-build` Remove :code:`-Werror` from :code:`CFLAGS` when building. Don’t hesitate to use it. But if you do, we would very much appreciate if you: 1. File a bug explaining why! We’ll fix it. 2. Remove it from your package or procedure once we fix that bug. Disable bundled Lark package: :code:`--disable-bundled-lark` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *This option is minimally supported and not recommended. Use only if you really know what you are doing.* Charliecloud uses the Python package `Lark `_ for parsing Dockerfiles and image references. Because this package is developed rapidly, and recent versions have important features and bug fixes not yet available in common distributions, we bundle the package with Charliecloud. If you prefer a separately-installed Lark, either via system packages or :code:`pip`, you can use :code:`./configure --disable-bundled-lark`. This excludes the bundled Lark from being installed or placed in :code:`make dist` tarballs. It *does not* remove the bundled Lark from the source directory; if you run from the source directory (i.e., without installing), the bundled Lark will be used if present regardless of this option. Bundled Lark is included in the tarballs we distribute. You can remove it and re-build :code:`configure` with :code:`./autogen.sh --rm-lark --no-lark`. If you are starting from a Git checkout, bundled Lark is installed by default by :code:`./autogen.sh`, but you can prevent this with :code:`./autogen.sh --no-lark`. The main use case for these options is to support package maintainers. If this is you and does not meet your needs, please get in touch with us and we will help. Avoid potentially troublesome informational tests: :code:`--disable-impolite-checks` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :code:`configure` performs a lot of checks that do not inform decisions but are simply informational, for the report at the end. These checks replicate run-time decisions; their purpose is to offer guidance on what to expect at run time. Some of these checks trigger alerts in some situations. for example, writing files in :code:`/proc` confuses the Gentoo package build `sandbox `_. Option :code:`--disable-impolite-checks` skips these checks. The only consequence is a somewhat less informative report. Install with package manager ============================ Charliecloud is also available using a variety of distribution and third-party package managers. Maintained by us: * `Spack `_; install with :code:`+builder` to get :code:`ch-image`. * `Fedora/EPEL `_; check for available versions with :code:`{yum,dnf} list charliecloud`. Maintained by others: * `Debian `_ * `Gentoo `_ * `NixOS `_ * `SUSE `_ and `openSUSE `_ Note that Charliecloud development moves quickly, so double-check that packages have the version and features you need. Pull requests and other collaboration to improve the packaging situation are particularly welcome! Dependencies ============ Charliecloud’s philosophy on dependencies is that they should be (1) minimal and (2) granular. For any given feature, we try to implement it with the minimum set of dependencies, and in any given environment, we try to make the maximum set of features available. This section documents Charliecloud’s dependencies in detail. Do you need to read it? If you are installing Charliecloud on the same system where it will be used, probably not. :code:`configure` will issue a report saying what will and won’t work. Otherwise, it may be useful to gain an understanding of what to expect when deploying Charliecloud. Note that we do not rigorously track dependency versions. We update the minimum versions stated below as we encounter problems, but they are not tight bounds and may be out of date. It is worth trying even if your version is documented to be too old. Please let us know any success or failure reports. Finally, the run-time dependencies are lazy; specific features just try to use their dependencies and fail if there’s a problem, hopefully with a useful error message. In other words, there’s no version checking or whatnot that will keep you from using a feature unless it truly doesn’t work in your environment. User namespaces --------------- Charliecloud’s fundamental principle of a workflow that is fully unprivileged end-to-end requires unprivileged `user namespaces `_. In order to enable them, you need a vaguely recent Linux kernel with the feature compiled in and active. Some distributions need configuration changes. For example: * Debian Stretch `needs sysctl `_ :code:`kernel.unprivileged_userns_clone=1`. * RHEL/CentOS 7.4 and 7.5 need both a `kernel command line option and a sysctl `_. RHEL/CentOS 7.6 and up need only the sysctl. Note that Docker does not work with user namespaces, so skip step 4 of the Red Hat instructions, i.e., don’t add :code:`--userns-remap` to the Docker configuration (see `issue #97 `_). Note: User namespaces `always fail in a chroot `_ with :code:`EPERM`. If :code:`configure` detects that it’s in a chroot, it will print a warning in its report. One common scenario where this comes up is packaging, where builds often happen in a chroot. However, like all the run-time :code:`configure` tests, this is informational only and does not affect the build. Supported architectures ----------------------- Charliecloud should work on any architecture supported by the Linux kernel, and we have run Charliecloud containers on x86-64, ARM, and Power. However, it is currently tested only on x86_64 and ARM. Most builders are also fairly portable; e.g., see `Docker’s supported platforms `_. libc ---- We want Charliecloud to work with any C99/POSIX libc, though it is only tested with `glibc `_ and `musl `_, and other libc’s are very likely to have problems. (Please report these bugs!) Non-glibc libc’s will currently need a `standalone libargp `_ (see issue `#1260 `_). Details by feature ------------------ This section is a comprehensive listing of the specific dependencies and versions by feature group. It is auto-generated from the definitive source, :code:`configure.ac`. Listed versions are minimums, with the caveats above. Everything needs a POSIX shell and utilities. The next section contains notes about some of the dependencies. .. include:: _deps.rst Notes on specific dependencies ------------------------------ This section describes additional details we have learned about some of the dependencies. Note that most of these are optional. It is in alphabetical order by dependency. Bash ~~~~ When Bash is needed, it’s because: * Shell scripting is a lot easier in Bash than POSIX shell, so we use it for scripts applicable in contexts where it’s very likely Bash is already available. * It is required by our testing framework, Bats. Buildah ~~~~~~~ Charliecloud uses Buildah’s "rootless" mode and :code:`ignore-chown-errors` storage configuration for a fully unprivileged workflow with no sudo and no setuid binaries. Note that in this mode, images in Buildah internal storage will have all user and group ownership flattened to UID/GID 0. If you prefer a privileged workflow, Charliecloud can also use Buildah with setuid helpers :code:`newuidmap` and :code:`newgidmap`. This will not remap ownership. To configure Buildah in rootless mode, make sure your config files are in :code:`~/.config/containers` and they are correct. Particularly if your system also has configuration in :code:`/etc/containers`, problems can be very hard to diagnose. .. For example, with different mistakes in :code:`~/.config/containers/storage.conf` and :code:`/etc/containers/storage.conf` present or absent, and all in rootless mode, we have seen various combinations of: * error messages about configuration * error messages about :code:`lchown` * using :code:`storage.conf` from :code:`/etc/containers` instead of :code:`~/.config/containers` * using default config documented for rootless * using default config documented for rootful * exiting zero * exiting non-zero * completing the build * not completing the build We assume this will be straightened out over time, but for the time being, if you encounter strange problems with Buildah, check that your config resides only in :code:`~/.config/containers` and is correct. C compiler ~~~~~~~~~~ We test with GCC. Core team members use whatever version comes with their distribution. In principle, any C99 compiler should work. Please let us know any success or failure reports. Intel :code:`icc` is not supported because it links extra shared libraries that our test suite can’t deal with. See `PR #481 `_. image repository access ~~~~~~~~~~~~~~~~~~~~~~~ :code:`FROM` instructions in Dockerfiles and image pushing/pulling require access to an image repository and configuring the builder for that repository. Options include: * `Docker Hub `_, or other public repository such as `gitlab.com `_ or NVIDIA’s `NCG container registry `_. * A private Docker-compatible registry, such as a private Docker Hub or GitLab instance. * Filesystem directory, for builders that support this (e.g., :code:`ch-image`). Python ~~~~~~ We use Python for scripts that would be really hard to do in Bash, when we think Python is likely to be available. ShellCheck ~~~~~~~~~~ `ShellCheck `_ is a very thorough and capable linter for shell scripts. In order to pass the full test suite, all the shell scripts need to pass ShellCheck. While it is widely available in distributions, the packaged version is usually too old. Building from source is tricky because it’s a Haskell program, which isn’t a widely available tool chain. Fortunately, the developers provide pre-compiled `static binaries `_ on their GitHub page. Sphinx ~~~~~~ We use Sphinx to build the documentation; the theme is `sphinx-rtd-theme `_. Minimum versions are listed above. Note that while anything greater than the minimum should yield readable documentation, we don’t test quality with anything other than what we use to build the website, which is usually but not always the most recent version available on PyPI. If you’re on Debian Stretch or some version of Ubuntu, installing with :code:`pip3` will silently install into :code:`~/.local`, leaving the :code:`sphinx-build` binary in :code:`~/.local/bin`, which is often not on your path. One workaround (untested) is to run :code:`pip3` as root, which violates principle of least privilege. A better workaround, assuming you can write to :code:`/usr/local`, is to add the undocumented and non-standard :code:`--system` argument to install in :code:`/usr/local` instead. (This matches previous :code:`pip` behavior.) See Debian bugs `725848 `_ and `820856 `_. SquashFS and SquashFUSE ~~~~~~~~~~~~~~~~~~~~~~~ The SquashFS workflow requires `SquashFS Tools `_ to create SquashFS archives. To mount these archives using :code:`ch-run`'s internal code, you need: 1. `libfuse3 `_, including: * development files, which are probably available in your distribution, e.g., :code:`libfuse3-dev`. (Build time only.) * The :code:`fusermount3` executable, which often comes in a distro package called something like :code:`fuse3`. **This is typically installed setuid, but Charliecloud does not need that**; you can :code:`chmod u-s` the file or build/install as a normal user. 2. `SquashFUSE `_ v0.1.105 or later, excluding v0.3.0 (we need the :code:`libsquashfuse_ll` shared library). Version 0.3.0 contains an incompatible function signature change that was reverted in 0.4.0. This must be installed, not linked from its build directory, though it can be installed in a non-standard location. Without these, you can still use a SquashFS workflow but must mount and unmount the filesystem archives manually. You can do this using the executables that come with SquashFUSE, and the version requirement is much less stringent. .. note:: If :code:`libfuse2` development files are available but those for :code:`libfuse3` are not, SquashFUSE will still build and install, but the proper components will not be available, so Charliecloud’s :code:`configure` will say it’s not found. sudo, generic ~~~~~~~~~~~~~ Privilege escalation via sudo is used in the test suite to: * Prepare fixture directories for testing filesystem permissions enforcement. * Test :code:`ch-run`'s behavior under different ownership scenarios. (Note that Charliecloud also uses :code:`sudo docker`; see above.) Wget ~~~~ Wget is used to demonstrate building an image without a builder (the main test image used to exercise Charliecloud itself). Command line tab completion =========================== Charliecloud offers experimental tab completion for Bash users. This feature is currently implemented for :code:`ch-image`, :code:`ch-run`, and :code:`ch-convert`. For details on setting up tab completion, as well as general documentation, see :ref:`ch-completion.bash`. .. LocalWords: Werror Flameeyes plougher deps libc’s ericonr charliecloud-0.37/doc/logo-sidebar.png000066400000000000000000000767111457016721300177450ustar00rootroot00000000000000PNG  IHDR Y pHYs.#.#x?vrPLTEUUU[[[jjjTTTTTTSSSQQQFFFOOOLLLSSSIII???DDD>>>888000&&& "tRNS --0@HPT`pw?|IDATxbJPC c"ilLamͪariTAyCyB@E*rWUUuiXUUUeYµMYOzWbŪ,tڗmsP.w2*Kfdq.!8Ur%΢(wad߳妘?dƇKNvct8kz1˕Sƛ ˵)fq *eXoK*N2QKr۵2јeuIq1 @q.;eıX]~?-EKj-oW`zx^ cqs3X`3kޝ$Db\I [#`<監28-˖{ Yl*$gT`jWp榝QtT~eޢt>^L 4ThZyc2! V`PKCKk0@d\hloilV魝92 עzDUtMWT'kk7W^e2@*ց^_ELoM*; GktY䜛J/'mY e232T2BPe2@Y(@ ;+r{g_8 @kOiF-v,Phr,m+ Jˬ-`zey0U]y0Vrwf5Y^묭 Kgdjz ȘQIJ "e^LoeYO|(IO 2!2` `2!@`zkCde2hŇ0Na8|>Py(-:|G`XQCMGۡ30b@*/XEU^9v|'ZPPFEkBh5@(^ ñ!ݔμhϙəm-R>Z B+ 0\x2 eFƄ2@L#3LndPC_bdL^{$C#>4E^ BX 1W8wXŸ,n!{ K"/$2^`-&-#3k 0mi Lf(lx $2AAvdd2DT6(+$2ph7 $2 $ !$2d8D A&Hdd2DO3ه aa(@"L  2@"L$2d̐LL*2mHf; \.n HdXӺ2HVd%^.{ @2+erbaD&֩NVd@"u:L>=D&uB|@R+L3O+uJ$2AO~RN(wb s 2 D>ȯtn >d@"`z|SP@l*v LPeCZ/ rKȡH1Lv&7v%d1dG[ɱt%Ɏ刐̗t%ɕ#BYU.zkg#CHنTk7Q% =3d B6$G`9Sl+D}t=mObcSj& Š#L ( d[}1F=-aur1R= O9# Fl$ge@,6@N?-b`rsrMeaBqqMR)L\) o&;-fir>| ]-ބ(qg5pYMufzPzPYyP5L^y09+vV:-Tܹu&; ҮW0N̙C@, =;zQԙ 8[gEWl:3B^|jĜN ²0lEWP`,bUeY~{-˲E좉PGpE$7LeY|.( !,E.]UNUU(:&,X+L^vyWe#{͋߭m+[лnEFɇm|e_p=Vȫfn" e9T&˥"6fj @NuL-7+~9E2y6[aXȩeder3_ayC"',tdc1{L<Υ('-(aE7c5$yl2SeTyL{R?ttLг\t׿dm }L%nW&dCAP&'!/` X&(T"erb>[TȥcjMb<T*ym<ɩ7B$| K29Ye ܪ"er.5*֤׳LV*֥y:XMSȞLLmL<0LiudH(@c˹zx4$ Njsd j̨lcZ dj$c5ced wVs9f9˚dž29wRCeH󭧌;"2ﯦp|~OJe2=a5W͞dW3K0w}I'9$'6N'Ĕ2}ژDFdL)3Bdة^Ӈĵcq,i\Qf"vmLF5kULF9?wkj5XYsxop,佖6Adzc4gY4 k,-|ƫڮu2ld#so e2ʄ3HmxW2eB+QW] ,dL@@-[/uwPUJ2eWz\b/_VƝ[lLŽ1P&%'^&#0/jMDVF&yQ£LfpJ:QʍLVx%eE$(2RXŷV]g_r\|Q(6.l6+tYxLR,`ʡE|Oef*?HS$Ur_R媸Z!D\Q_yJwY1;xXʿY&+^oG!Qfu{z)IgJޛkU2Y(߽Õ?ҘvP*`=LU:3nep-nD(!E2pWkaWV,sd}DO^'ȇmdɈdJ/O29D>tJ[EV25Ul3cѕUl2Y"^.A~|kqdᨀzRPoe|mddDQWL=",;V e \-D&K\|8n2Y"g;?H2:_ɦ72Y"[^ ɉ[LW{Lݕj"9[.eE2Y"_s`  jL섚 2Gkp|h+2MTHNkY!(aIeߤHNzUK5憐!^EIFn uǵCL'T=q1E22Y"bc:9K,ko\FR/D~')ybE2=;䀭G"9__/Vjj}dd/!fߢQ'/D@qM$'kثxߥLr7v--1-Lka-e]"!8:<w, kЀL5GDsDrCKӍ"6B0,cUN'="fXi}FWo#]s_e;ٞ1W=_Ft-~u1EB~S&cd'H?>T0m ˫-="9N1HfLa,{ze/QF?d*D>qwo#_Mg0 [0YkKDr}5<[hf%6iצ9? /=j:+!Yvb_~;)^I*Erxe9-O"szki1ԗHH&1]OhP֚!װ pd&^]GC"Y$H$34KZCx&ؗK%^y-iWy DH~BZ5bIӲkH~}L< 񅰒jeٵHؗY!^?*ƛH"fȆūK,NQ$9xɡ,l?I?I"MN U8mɁF[wo͏L'hr/ErS%Yux[DrF7~9 2D򏗱EyDr8Fb:y$=_lJR$O>B{GV}KHq?tB\9uuٵ"Y$Y!WO$jWDrǾO/$6\sKW $'~|9^~ usI Z$[htTˮmON@#upҲC xa&Er|B?lp^~y}_"Y$Y!Z^HΡ,þ[,E$6;FuLkL>4yN"7 T?Dr[ adD;oq~ƾ< $Qlgڮ$œH'jS{GZ:ױۓ%ErH^j(]+[\""9TO_Q&ryo{LɁDMҵԙlJ,Z'ܯFdX9ѯBP\\K2Y$Gc5 kR:p"Y$Y!eʀuТq,tx$o2kmBO1J%ww2yz]G~>6h g{cA0Q!EHZG1՝K>,|c~GM$geȳ40Z B!ҏd, ȧ1($E9%/@A;+JTIdHw9+i ]w%drV"yHƎ9.d qmWԷ_&{/B))3E^"9xi`D+UV&d<xֵg[^i-ErnGgy6s.L%$ȳl^v%qҫL= M>DV^%&eG8h/Gr`4r=gxL!DMg|6Dpµt6d,Er!aHJJu,5]_[t<%4܋o#T imdHɿ8+d_'T%.+}%?j㟨Kox`ljliiYHw\k6]$*m*D,ErH^f9DVP -k2Y$6+TcjrB㼐R ErDxIuIhR6٣3Y}*lsܳ/ynup ry3[䠥>4yonrmun@& |}ߗVc' j伐,Ergm@z;"iɋGD$O$67f˺3L6H>2Y$gV{Z[n<}ɓKؤXe`HN,>+8n }C5n;TO$h㻾EJ>4^䥷<>gȞpe |t"Y$O,wykJVk޻X&ujHN&c<+8ݺ\',]ӻBRɃF?0es=]y.]x<g|OZ? ,k|ӒC('7 o)M?2H!FtVq ʎAYM#ymƛ]Ĉь|s*c0< ,y-{欐%-y-cFo>I tS8<'xfNprc%Gᬐřc>Q!wr"7}a!#ulg DroǿrpBFfH>E?kAbvXlMs2[+^~|~mG[< up#Y'Hc9W, m'}j+ܿց=c4#V exf" ^ k oM׮&ErRA-]wMdw"/RAZ$wmNeb s8̷&Ӥۘ+^|pm|ɇrƐWխX";:U ⬐zƐl#Oa A"y6aUfqJzI|Er#oLn8 0 Ggy("MqTkfN?PvȸH,c6=<(C俍1Y$+dSA ge^X-X&6eg] ex7l%?3z`+ewA=?X\v&pʖy}7o$"c<(;Z$<29ڷ(;K$2Β뷧{*#GuNB<~"ًx8luǟnHfC/~{Wʳ}6){8gy VzMu(o>/ k<)ׯ_UErf ʶT k<ql/R9Ĩ >eE]GszИpX^JBe!4fp,o.O>^QGr^-8?ll9oϯ-xh1ߞ5 }2]CYٔ?l~@|M#;hJ,ƕ'i("YS^ǧ|=\(*?ٖ/4~<{SߞHF$`pt<Ը/.HMMFek{M*ڐw6.;_Do o&ɷ+tRHdzX[~-nO${7OϽo7.g$yVU "aլ^߻q&GO(s='eui$SCu~3d NzkrDrODn^ߊ"+g ЗnL6HSx 5>{={x|>Xcr>??>ލ3{:5,"-Ix%>[8~y~Sr8krDH6̏cG/&L6?"0u]2~}4/3MmNs|p K|'ڐ|$[`Vێ^nbLdkCD2AZ|Pd(kJL>{/7'MɘLN 囷 m"'{5Fxw;xe`7IOLh|>9mo+TH"Y$gm5j+u?c~;oWz$,67P|k0[0 /=no^G>A ےd6|tӐIR>F3c`/c4JL(FLo9oC oM@/5܉%ErFL~h߳8~oInɺQdw5RۀyP~.ۭc~?4Hə!B! G"Y$nV߷>7> {9[6ٽ?KXp-ErےwHE&[H$dq #Iƙ|:wuP,"9{M&~/_H䦙8wfH,N$af O\&vx%ԋRiLcaG:.m@4ozoU `8Z$dGj4ixkբp~/F8;K`tF"Y${sy췾vPu` VI5e_ EH/m=<u}mk(K$dd򗁣m&{Q}KÿZ7 |uu()E2=O&|06U}W+7\[gT ",=~<ܦz?euVw0 ZW$E2>'2a|<Y( t||" [=H\zL2R5N?fnWihuH,g4Ӭkp^n*P9\qQsݴS}uҍ7d7مmu|q:[LPiߦϽ^68Ӛ8($:HL&|M;Mwa?>أ~U۠oHN ^?$E2ȇԻWx3omJ+98(D$da2vںֻ)C/?-_ jz>%E2?uo/桎[ynyOj6 Dr쩋Z ~t;o_ngZC w-oJGh*Y$diӬ$韁ݷV_`\n}[/`*Y'[i~vuAB}C;j_V[xEH&knu_:#b*9gL`:M&߾Nz ^>Ҹ^w ɝL4ɤQyg~6k:~1= ȷ?K\֍}K ڹQ2<"ْZ4964{ed`=s@k:)7~2Ln s)՛f.۹sYD$O̷EHF9XIzN--k<z ,M䶃i/w-}^K坃?G-n3( ϡݍo*>{uH4Hn6uHB'NuVbk5kO`:v](>"wkZM&7}zQ@_۵#oy@d?L&4/nL~Zkv߶bnƈ\&V&٬]5M a&?w\mϿk%[#)-[7J䆙|O]&~ Zw|<Kw[#z銦ی}o]ԾyN(߶@:j$ەgk|X ~G/[}uu2$ەC|2ܗኮgՌN7⻬2 "Y$3Ɠ{o۴~uo]>u֑'ٰ|$~ӽ;;[R\+Cݗ;ɨn45*2ۮi}ַKf>x J5[O$Ii֞}:R"[9aacra|ݲû|U6-w6~fy0}$tkZwSsj#;KƩ̝$E2h8堌K㷭%}Jar*dlh۲J!ywߵn}N̖Xp=G,MԶU;^wkp^rH׮-hoω8 w%ўAYz^ot&qzpCiQ9H>׾/-&|h4'E29 Fv<άkӴ4<㦒bSrtu19M9Wy#Z_vyu[iZMdm*lHɄgҨ K~ߘnuZ[pP%۾OZUݑdaDTRgh_|l!ߘ"QUH놕&|$խ'cqWdۗ)X$ɣ| nuc˰qu8K$d>N&C^ljM<5z̬60#)ا5xi:~fȗfWal ۸ 8uO0\7RߛoMsBⳌaH;c4|(-^oL:&W%ݼZhG^T@J?I/_N/__Z;8&MA35k΃vzʸkm\Fvh3V~~|=<_[hPѕcMj9!ϸnvþ.|Xf$O1!(nd&G{mSFUs z5ko}kwU y*e­L)& Ԭ;4-?~ت^>.,VR]ʭ.4=S.uk]?~ⵡ_όt,Wt;sPLͺdt9nkǚpMίe{^}p 9wۙ y5~S_Z̪68w;|Fɹ+ԭ֞gTn#|7ӬKO:fu}RikAvLz~Ͽ.Ze2fu^ڹo{6FA2}G}'fYT ?kgr/8dҵ~\[rylr]}Riҵ5jkdR_pmu*.yq!7?y^G|iL g[G~U#+o3zyo6?(>75=zM$$sI̒T=S|nܘ탷ˈ7ر]\.*ژTݝ jMW]7]` f;So ԈK^A2g0jaS? MA}x`e{O*j6?["[5;ECR%NnrV-aD CKoϏw?@=>nb)Y2(۠^f2)Gd8%$]5f'5Obo7֏yJsuĠ,oq+79 uA5ϵj 5=.cR٘N鷸۠>?龥~_@3.#T9QwT㍺O$cL:OЗ!yZR K۠nN?ȅ拱>?ukk7~B܆2htJr+98n[3]ukC});W²H4nm߾=(zkS\gd= 1c}έ{ ow}}< zk')]wX<4l >v6۠ZRsne~̫v:lhw[7:ӄW];>Hw}Ϸќ)hxNUOEmPCai:c_vCdCȣ-:C/N'qHm?C'%8U&Ma!5؛ש~cEģut}uCkSfy{~?/֨e> 峝Oa2i{6ߥb]O|hz=%C(?D |'1Gܖnu"vo4GV7m#eG}}T-:dD4r%~n6GNvyh:T~fV#2w}-H2y-~Y+[U6drcmfYxsU~f|l4@YN|ǵ=bǧ7rXݐ{7ѷƝNn)̟|>]^jekz)$ pe(2j&Se-v@[^byFmpvlo^FYc;ɘJ;epی]o"s3BϬ%D4߼7p&걟!2W&VՄjYSTIE#nn R?EfMQ&;ܙ! 52p@CJCYM?XW%10bB.vAݎ/C|az> dL$ǨȮ9v 0|;;9G['8ytSỢ71x{x>;iLMK YMKgqŎmlR#_,lƴ}Zd*Or6Y(!Kvȴ]('ݴ;F#v[h'GsjOl/1[FxO$CT>8U: *$QהA,N"N嚔C=>oGqLֺ ukk/w_yz|5U!a*DV5T!5?[\r ]u>:\ ~[hq#6$[>Tu խ#e*`>k4e#B -TA"Tz 9;i3٨\D&;:D%Ȅ`:TS$28j3j\D&lMVL|=g65Hd`A5Y$2[il6[i fb@"3)~+m$rJ-Rc:$2,4k0`Bw/kHs #cq A$2wxY 阫b/oN"q-'K*YF/IjniBRHI[&s+EVii9 $F>cϛFaJDȺHE5*&Jv@)Arlu $4fk"PidRkm qH{"I>JBHA% bPYqͩrLarIvvYDžX+1rbWly&dT3mxu\+d@F6HNS5`xޣfG'ƒKާW8K0Y-)Z  &hAa2ApB6H6L& xJkdd|B|eB?eL,,"iδ6H6L&"x0 ^$ka]Aa2qTu$&l}2H6L&! _F6jȎB%70Lɵi)F"c`LZ2:H֠ 3ӐIHils5Ss\y4XH"NNLQ&g>(ҰV6H}F1]A mI&F cis>bԬ`JiTNm8/0&Y愚Ar+Ot (NYh: H'MNv&4ftr~u#dB%oH؁lUWlI0nqΪm4.:a2dzr;QA2LLO&8ΈO80x9}vug7SPvuԌ߀@3k 9b'~+@-v9[+M)zG͘0'̧'Λo PEeIa)z6{csrl"}k hKy)~6BPrnlIVwVDٱS?YE[/3OɭM)[jt2I(59<3([+]{ʱ[tHj:I QoV]NyQ`SfJϡv;=J׼c ּd2cꗭM C&lI ZsUhb j絝1F5 0ScUU۲,WEam4bUeY`F=75Gw+y{n6OƉ!tIKeQHZ={~-"DXER/w߳?d1[~*<K,"/]UEHg7ʭI]4AK˿j9t.PS\xpJZ˲k eb`]dYlm5A$2Xl]Hdd2V($22l V(L~2ؐl{2 w,"F q 12L OPHdd2 Hdd2DF& (2yHd/L@"#Rs' @ (5dL֚2DF&a!R:i'\c)UId @vB,1drdWDNV+mGA$r Id u`Yg:|"[ؕ _]yDiBBSyq@Nڄ2@ eBF6 @L#ۡ @FCTY[8 SJ"x 0={PVx 0=vB:v;s_9'vVZ;džD( |9]9l<ɋl6u{o?Vy^=ί-ӌVњkk߰7JY9&q9lF*d߰;$|-B{Cd([$[ lj `e a^gW@9&0PN I(|GWʆ(ǬIheB"V̵/7P6D@`?:]LM3f 1P&^y6-L):2P6D&-޳VSʩ)z{Bq"|=;!r*S&]üB";Vz8m+U/.Hqߜ9^ܼeW~őc|h\e^)jݝ/ز.,R:ߞ2XE=CYwιP7N"z@uhGTz}%^'cf! d7>;״5͑u}xk}լ陓C̪Ia)e}4˾M8<Wr)7,Ku^i<υUYVqu۹HY&2<e<+jXyQ;ުQyyyvoc:2BYܴwޘ\d>c.ٳ> eLZlHy:>f,NBY tuVoa`P&@f#SǴt<09rpN+aԇ p*Bvy<5z9nj s o>H-:MKBX YzQp-ZZ=&eR2IM*_ k_~M!~b=rC;H*z7YrvP krT0@PYrTvqȆ `dbM;}^) 3$e]eS*߹?X`\*:e`rTkz"_g>285539XQR9?G+IQR ?K\nLg*'3nqP򏳉?!ԗlJ˗idOCVY-^ XEVe#?E@RȕeqQWMi|ګ$c;Oud55)T6J]c|> I=ի8VƈyQ[Ʈk3o*Jd`u{plU>UJw~C!>ڈko*ծ,4{C::qol}]!0v*e_G%Ǣ(xrmisQ`rcdUrc`<,i)K7MKmDrVܻ"tq9|.HF$'2>)Hew\mv6(dDrҷæ=u缬NE2"9ve.HκsnV(H|s PY$948xdDr2Ӷ{/)s2&dDra5ErƝsѬ&dD@4ErƝsVdDreLBY$g97Y9"3sC(|;i;SɈȬ mW_|;iɈ,4~V$973TMHȓ}+9"9ιc嚋dDr4eRko W\$#c"o/S;6"Y$皋dD!e,EHF$'f{ CӁH"Y$#<\Y"Y$dDrJ֧K@MNt"Y$dDrBvV"Y<&k.Ɂ.H%dDr*K+Ersm|1Y$#ö2/CͳDriڸ"%TzA$9ϛk.]VC"9ι*KɈd<eqܰCP"yu \8"9ιaZZ$#M&;Fe[D2"Y"w6C;ET",md"D&;yO.HF$KydsHïəw5Ɉ@/T2Y$9z<]nHӪǸM]OH9J"dDr$U1[s~߇uE2"9'yUs*?,uγl}thHF$Mb{$EGC8\o.HF$keU^obx,wfu~xڭ)A۴C/e9EwE۲ `P,uθ䄯[YT=&H9#ch޷Qe"Y猻HO>|,O&uθZ|Z [~Y$qiuHa1ݓY'Ew?Yk%H9#fmװͷ.*"Y猻H1i߬N?qc$r7վ\OuhǪ*7E\ѺTHsr[U=_ږ"Ċ,6?Uվ,!Zw}QlOPUeY,gKe:>-^|?<_m-vC?x:Ueȍt򦵿rz[}څzETmvzq v9o:8uK>"Ʒ#;;ͪ6Aw~R$/ݦg1_T媷?]_ޱ_yx90_V?;Gt>-}.?9O%ɫ]y6luߡ\|חfl?RGrAico< "5zJN߯|=4οOUs5z_Zsi2+ծ%Dl}?n/M<-Ov^ui2eT:Hnٻj儕ty?z$P#L(bu 8g^nշ*g-vk/ GrN#]VoLPCCd"9^*RM$gmu$>M`|\.ɵnyH.~2r\,vKoNK$Dzkx"d/wenXع!fH޵ɽ\/=k?3^$]@{85B9HS:L3uh7>oɛkߌhUmٯtOqrP)W~uΑn/y7ъUżq$Ӹ}7a1#+"H^ɴbwN(=PdRQjk5ϜZ4j\>yo;E$8#yQ itqd$ogny$KEr%5Oػ0Xsxi}V 9P.5ýC^aR1Z4z띧6#F1r$_;81#42z<;T5-bheJt&\ܰ:.Mho=j$?u70#3:}fubǁ#yf*HQNe;K74t'ucFr0ףaN"yy_RZ6yF jbc7-WDfEvH}MwH%#"T0Gr1̳ڶ,:"ƶEHs-ג2!#{*z5x9ΒZDr.p[$cFZG.=t.G,or(z,FEɃBLp#4,H7Ϸđ<{~$Kg=pm旧K(ErQ&HU=-X&jK5gćD2#O>y$_'rHn`W$Oqdk=a$&vɱRa3a='jڊrH..Ӊz~߉n>_3!FrWe .[MlZhju OqJ$KEv8_Ͷul۽urGU4Hҏ7X"Eȼl>Aݰt!8VWQlmպs>xܰ.rc,.aM%"S/XqVV5qo3)HnfX$7૚}ż<896+MƊŁn}+ea/o)Y|\=>RGYaHnZk ,ˁ{(<=Oze|nV/}<~pm,y*HLݗ*H6o>@.wpC0xABX.>Sd\zy6[tuo?}o\l$&HL)#vl܆g٬ˆNF_wCI?WUlخz 1RF:H>|V7m˾_?ڝ@$89HnԐ:a7VޭϻԺu"S~n|.`$GKe?]$k4 ܆,Hzn>g}kG]ղ͓v짎W$7n d#9^*Ѹgֳ6keM#BbnH{;Zͺ"3u*iuqDrTqÿE>5oݴ\ŏUzt6/ۿӥw_]z8{=.hlgjfHnf?H.FJ&\x [̧֞c-CV6UGrT\!h#y;L9RlB?s.W1"{ 3D]Aq/+Ga-:f?չ:#DrH&k~+n>|gw$l#Dr1n&؏XUMξ٫H("9^*HEol:HnVc~Լ!\i#5Nnv}blA9J"7Eđk/f$G|RHHnXz˛^iEr9B$HtZhw}eIbcOW&;:uo28cDr4y1I"8s#kjS$>Yv R׳jv)HqGrݳ4uSDvιH;jmEq¢ݩB?a!cum/%voo?{ts<-~47N1=snWFrH:siPꣴ8?:^"nmcWrx7op{#7 X#9^J$GmF9Ϊsye =uνDr.uV=&SrznX͏Mo/%cVw_fO-o?v${I:CFo>Ggi$K#՝t}.Is@{ds/\uV#P"~ݺ@o^vř- 'H! "*}>J06#n" G^|s\ݥ"yhVE"9pb['5-Ճׯ2H[ɋc,{y>&hw֍YqrD`Urr<.@}$Rg:q#+17|/IDrWyb˿W,ݛFxկkeɆl7S|V$17CGG$՛'f^J#yiQɃɣH6\ldKV$weѥʵf$͊H$/S^d$z#P{fP}$R7GDroj٢~d(vf"9EB [ez##p"cTL}H\.:5K^$7+"GLW涺otܡ'|ɖM#BB/ƽYt9C|`!>d}^2mzlnH\._}qgF pH>gH#yȻݞV ^ɖM#|)͆2}HL$kOo258 mljUC_כdUʘl}X$X?{D˺Ou=6Cз䛁D*eL֕!0跻֏A}$=6fV/]M'klJj+\N"ZZD}SdV$A7dUyU>GrHG}3Q]l^ڕF*eMpanRCvwD,8Ra+%R' Nɦ5.&W˨!ŗcd"D$k$R?Y=B"Z<*O/Ov[dUʚC$7"{pjEЪ8D?x Kp:5!L$W%Er1ɋZ" 9A]Af"H&mEre@<|id"H&d"Udv5L$D2L$kyD2L$D2|ĺ.?=A$D2Hd3osL$Dr*x[cHvD{x7R[DrD2L$7>yRw$RD0lD2HJGr_n%qmd&Dr·,F#9 -% )CsJ$da-"fHg4ى ]n:A۽ɁWR_u==T_oJW__%UV\{H[;"9C5W+q$=>o㋿=uG*uw>("ٷDr؇{Em]{\?.M<ߙSIcL$7#ْE$}o3>ȶZGr!]]/ۈϽXl:2w{VɡQՍDr,mNm_74Gr O^mDϞPX2z'~68m<Ճ\yqUʨWU[d푼5) H]c){׀`{]U)Lj~?nξ,CwNN?J$.h雹^Z\Y"9tRWt3G eO*et%k-[{$-NJ$H`P:F]n_xɽ W er/tG$-u|ZMn&*J7nحR&W~>}39*ͬj59ՃWNi_/ג*arL3Y}$/ӈ fb6h^~e/v-_O~*ar#9|ԮރH161}[?J<=_T[z0ʮAuҝH9CŒG$K-j=M] zx#sfLZx]*a򵗜zP$Gz_DrlF+ܬiwSEL[Z}EsRL>-(+P$W c?$^jƘ_Bu=]׵tZ.K4tftU)ˆ+E ]#9j {1ƋH6ezQ뇬)4VH冘sGn:7E[RI|8)՛ DrqMuۘH:ktKbYOzT̿::'ZRugR&_@rH[7yתDr|R]?iF7jR6UQN`B#v|*>2*eۜ+ʝM"9f-oCJ>"9.ڄW"5v΍Z}ԣQØN`,Q4KbJ3Nug? w _ hnȉ/?ϔz_G%_D;T:ꡦJן|y2LB$GsDrDQu*{]@_"ڣ"Ǔ"qu90Z:u-C0}q['R~="9egrfueDr_GEVUj2b&&bVY;u-P?>ُ>GK$ǷPf&q.3އsqeDrlmOQы5R >?mJ=yp3_;ϫ&"90vqHlW"9 =%(R81 pzyO䐋do2mS|馜l#-HNi~t]铘?#";<~!"!dr?mL)g=|43O}I-o_LT0} йSIEr_n!~ō qglh^n/2n\$l$+ӑqfJy +rRCyJQ1+PRMCg1"9i~ ;Bwq|\O;U_vTg$''Oc7c"9؁W$>Ӑ(m _t M=5&πsf]%`W-~Go3=:^>jI6`J|ܭ>rS{C"9q_̏=꽤GN?~Sa6XalZr4ۿ=oDrF J{&e\ 0H~nR9\v(]Jߟ}C]+~)R7&D+k~-7ʜ:Dz~ޖ$]F?sK{-i?,U?oɃƓK$8AM跢+{C#9Nc$ ItS/+>*N$c"9uEORV#_;MԇHFk3FBW*E&+KdK.ErNF|VsqRմOmF>Wd{U>*|1uDI!"YlqRr;ja~W(wDL^ejX"s?Ջ7[{+E?[ULj~+_dUv)v2kx*FrGZk1L>cwRdV$[. 5nSjJGr{j1WSUL6!d+- 7>I eM7$mobH.Ɋ]؎H4"Yߍ3S:ʮڑܲػR&CUn> u[F72yL=fWHnwg|FU>;ULֶBc.u=BJ$+ү0j,ɭZ,i>f=R2SnHuHVuD$5tD$iSש vz?q$ډdeuOu])+UFeuH+2:<8ʹT{ҵzn5ψdUѶwٵP$>DwY`JuצGMZ W?:d,љ[Uҵf7?So댁Nt.1JH~;/gGrhdrL^^#?l۩RL^FyY_Q^Dr~?_vL\NmS~"Rw~ֽɯD ]urWƘ_0k ,0Sz_nmh;PN%rO,s:߉oDh08Ep&RwxګemsA#PgHj= Hw嚳~[KSL>\W[GeQ|QIE$7"8cʱ?vƽp$kZܥ (R79)B|*R"Y(?uuEP+/KW$YWl[xDuzug(H.b@vZ7(' G]h,8D.y3AozrލH-Xd2U+ؕe"ȕ"- T`R**%zQm>nA['%+||j3vKk$hHSO^ZQ[4R>--"Yh,,;Wd1UʥB9uN$^T˭hRO UT!%6yO$K b "YrjѦ\w%WAtVgKD="9wA$oy$'CbOd{2nH~ĽBڗV-*BCe/u%sf%/n/Bk*^O%75#YbncT\>UVVU}znԥ'HߵɓN#{rUnۨc+NY{jRLe7e \Ǫ͓:T{Rv26Ks:-L[]\{V vTyUJU=.Y7;'z +3vTߏ`:2/cGzznu~sa?U͓%GþկûF}wןK챹J׵wr~s#iZ5G߆b w*skwnZav={pQW75?Wx0T)=atn;_~q=mIENDB`charliecloud-0.37/doc/make-deps-overview000077500000000000000000000027001457016721300203130ustar00rootroot00000000000000#!/bin/sh # This script is a pipe that translates configure output to ReST markup # suitable for inclusion in the documentation. # # Note: Don't pipe configure into it, because that can alter the build in the # middle of it. E.g., I've had part of "make install" in --prefix and part in # the default /usr/local. Use config.log instead. # # I have very mixed feelings about this script. On one hand, we've defined a # new markup language. On the other hand, I feel that writing down the # dependencies really does need to be DRY, checking the run-time dependencies # too has a lot of value, translating from prose docs to a configure script # would be nearly impossible, and ReST is way too ugly to use as-is in # terminal output. # # Steps: # # 1. Remove everything before "Building Charliecloud" (inclusive) to the # next log section (exclusive). # 2. Remove "will build and install" paragraph. # 3. Remove any "Warning:" paragraphs. # 4. Remove results of tests: " ..." to EOL, ": yes", ": no". # 5. Convert indentation to bullet lists. # 6. Convert "foo(1)" to ":code:`foo`". # shellcheck disable=SC2016 awk '/^Building Charliecloud/,/^##/' | head -n-2 \ | awk -v RS='' '{gsub(/^ will build.*/, ""); print; print ""}' \ | awk -v RS='' '{gsub(/^ +Warning:.*/, ""); print; print ""}' \ | sed -r -e 's/ \.\.\..*$//' -e 's/ (yes|no)$//' \ -e 's/^ //' -e 's/(^( )+)/\1* /' -e 's/:$/:\n/' \ -e 's/([a-zA-Z0-9-]+)\(1\)/:code:`\1`/g' charliecloud-0.37/doc/man/000077500000000000000000000000001457016721300154275ustar00rootroot00000000000000charliecloud-0.37/doc/man/README000066400000000000000000000001361457016721300163070ustar00rootroot00000000000000This directory contains the compiled man pages. You can read them with: $ man -l man/foo.1 charliecloud-0.37/doc/publish000077500000000000000000000032041457016721300162470ustar00rootroot00000000000000#!/bin/bash # This script builds the documentation and then publishes it to the web. See # the internal documentation for usage and how to set it up. set -e doc_base=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) fatal () { echo "¯\_(ツ)_/¯ $1" 1>&2 exit 1 } # Parse command line. if [[ $1 == --force ]]; then clean_only= else clean_only=yes fi # Are there any uncommitted changes? echo 'checking for uncommitted changes' dirty= if ! git diff-index --quiet --cached HEAD; then dirty='+dirty' fi if ! git diff-files --quiet; then dirty='+dirty' fi if [[ $clean_only && $dirty ]]; then fatal 'uncommitted changes present' fi cd "$doc_base" # Clean up and prep. echo 'preparing to build' make clean > /dev/null # Did "make clean" work? The only files left should be .git and an empty # directory _images. leftovers=$(find html -mindepth 1 -name .git -prune \ -o -not \( -name _images \ -o -name '.git*' \) -print) if [[ -n "$leftovers" ]]; then echo "$leftovers" 1>&2 fatal 'mysterious files in doc/html after "make clean"' fi # Build. echo 'building docs' make cd html # Can we talk to GitHub? echo 'testing GitHub access' if ! git ls-remote > /dev/null; then fatal "can't talk to GitHub" fi # Publish it (note Unicode siren characters that don't appear in all editors). echo '🚨🚨🚨 publishing new docs 🚨🚨🚨' commit=$(cd .. && git rev-parse --short HEAD)${dirty} set -x git add --all git commit -a -m "docs for commit $commit" git push origin gh-pages set +x # Done. echo 'Done.' echo "Typos found: $((RANDOM%5+1))" charliecloud-0.37/doc/py_env.rst000066400000000000000000000007361457016721300167140ustar00rootroot00000000000000:code:`CH_LOG_FILE` If set, append log chatter to this file, rather than standard error. This is useful for debugging situations where standard error is consumed or lost. Also sets verbose mode if not already set (equivalent to :code:`--verbose`). :code:`CH_LOG_FESTOON` If set, prepend PID and timestamp to logged chatter. :code:`CH_XATTRS` If set, save xattrs in the build cache and restore them when rebuilding from the cache (equivalent to :code:`--xattrs`). charliecloud-0.37/doc/rd100-winner.png000066400000000000000000002520621457016721300175170ustar00rootroot00000000000000PNG  IHDR'@gAMA asRGB cHRMz&u0`:pQ< pHYsԍ9bKGDLIDATx]wE~zŀb)'΄L(*a"`@Q͇ @+h@i&e/]Q#F p}}Uo *E3#Ph^\j^nnaQ%}sy(2  "22==DFF Q WVV[7X@Cdv4PA .\88888pppppp .\88888pppppp .\88888pppppt>> L&b1L 8NI3!""bӐ$rvQϽ0Vh4b x<vi1a6V鬪r\}RRRNg]]2j>|1BRvkkkwrXxߟRq[[[ss3; 䈈QYYt:Ci4[ZZpp!*++1ƄgddDEE p8v޽k׮IbccӫK7++;'><..!RXX_@|||ZZZEEE&J]9Z,Ԗ*A$I6l 'A)mkk+//߶m[YYfaHII1͕uuu#FkN=ԼHɄb{p8>\XXw߽ }Jibbb)--Zϟ\ 5$''gee_cQ&HDl;ہm} 77`}?@WV |oqa[n`5 m;k֬29ҍuϺkG2)RLSѱbsfv̉t)lذtRb%&&޽[{uyc.5s f~v]nh)DЃ#{rɝ*nii}v7x# ",˲e˚t53/[KTUU}>lڵ_|Ŏ;jjjػZ|W@\\#< hPp9NylXpfRRR0ƾ|!W^y?s%''uy2111UUU]t]we2(Cׄ;w~'[l?RRRN|8[Y{-0dȐ̐˫֒; ??_QQQ6oެ8cl6[+Z~)O?tUW@IIIh 11nFVo "F#`Ν;.\8qڷ~E sssC&0]hvo>] 裏)?Zcǎў`]NOlJf!\Xzg/++c5A0_T_>}sժU BEEc/|ʕ%%%As@8p 77[o=p0yd=db]k&M$΂UUUs΍.** Ҡ-%%eݫWVEX,ӧO ~ngE :tѢEza F-9vlbX>T}}}jj?_ZZ DCIxAG${&''36ϲ=Y 6˦9 !O?tZZv>&>>JZ.'tʣZ'''$}GgqƼy233KKKv{bTUU;VaV[ne'̡!VAHHHqp8!bkk+nFXreKK|YdA@})Acc/]IOOojjڶmRSS^B#ZxصtMW^yessfdH1=XNn_py4f*5X,Fǁ9p(vgNgFFƶm۪=~6#**uxz?v/b9p6l 'y hjj뭷KKKSqRRnwAAn$k6""?\8B B#lfY>0ԩSTLyf]FqqqJaYL{{{N!IKJJz,U1f^nnn`\8B&]c I‰'C} QY+@S쩩SLe'SUUU }3h[ZZk2gM͔ ~> hnn.\8B 5:[@XGMLTcٺsݻ4)"""XUsTTT:6hd LSN9Uch'l7;,0l' G*'#Qѫ iӦ6kcV b+ơvR%;2#oM v{bb7|S__: ,K0`4K%۷o8ppt_RsŜx≌Tlu&.uֶ6휣ŗ$-.wcvq͛7 ֬_WbboHً:Rg-.gdMMMuuuY9S#<st7vݪ \L֯_/I?2H΄uoj АpB&{bE9996m2LCm럞u*c\SS&M Go[JY9TߤA7ZPP4t w'>""b'p=p߃c/޵ 9۷p 㜜tqILL˫ݱcرctr GI&H,t:+hiiQo9YShLX &L0n:tlf]|~o̝;777ʂ8냂1sO:UEgzpojrr23ULb=#,&!v4  (sojj:T HII=zlfny襠`Μ9ޕ GȨ]dI^^jwfڵ+;;[Knju*{F[j{ڟ!.*77뮻8džYpa~~͛GYTT8 } VkYYٸq㮼Jcy`-SFǺ\/Kb3,!t:dɒ?__ZeK.]v8`Xi.!#)JiZZ%?쟝]TTk=YYY>fXcBȾ},Xp-|lxy'x{{tx < /Amnn^`Abb/Oq/|'rrrX.]$m7y>voqQ6hӦM1>Jٳx_~9**nn;O8'#F|ᇗ_~Ν;###\^88t8477O>;va]xqqqLx&MMM~ Y}@Jivvvqq?m4_x]jNscǎ}w333_~SN9߹s7z 'Vj^^ޡC^|?O>SvGx8kB7wouP$IEDK KJJ|+[n%???""!666;;{ƍ{v)4`̘1V .f»{ٺofV~ih4߿_7aE>{SLX! n"""O={{c_6lXAA!`0y7|sCCV .KKKY>.(ʦO~}DІ nּ`H%\Tf"cg4Nsл:++kOWXK/ 6SWWwС7|nkkSi!bp L_ GHJJ:3}ٸ89O0Ν;/袬`wGm)zvn@G: mmmgURRr}͝;7''*BՒC 6lٲez? /I,.'f[|yzz6ꫯlllT-W['W__t 1aZS0?A`捙1cFttv]իW+]j*m6ܹs:$ :22K/.LT駟:TS__뭷|X.ƚ02ɔ^\\iӦO?](.X'`ua](uy*%%e͵w@,f0f477222AЎ3{୒$yزh#F& CFY_|1qD-B^z{wСEEE=` 04@YYY'tRYYY^ 1 5~嗌3-{&RvdٮqpaVPP~)S؟x͛1? J=466/DEEUUUM:533SEL@ S$3g\C b!\8[ /Բ?B7С^~裏f$ 2:ujT^7oo࡟Q0sh155^YXEwKII!;;`ڵ_|.f nY. 4vs=8dP38CkWTT,\0:: 8˃***R&bzpyZ>`eʗA:vK.DW\9c ǯ 4^9s0sC%''WVVs=;۷@bbbjǦ#k֬-ȸ[bUSNٳ_;<.Gsss VZ}ӧ3ah4w},Tuɓ'O8H0oJ3Tyc;@2vggg7772Bkab^lv;<ֲr| xo\877СC~ᥗ^jheŊ3f6lXqq1[_&I(qqq<3uv6lXQQ /pǪ/ggg3o{Y,>!@v N=ԇzhtȑ%%%{#VݻwÆ 6rGP}HRѱQ LGW:*\>!3Yr%[]Vu !o;vȐ!)=!C yyyժs`.] GfÇ̙3[ZZp8fΜ ~uW)3DE$QJ׭[Ǯ[4LvL*>|xn#Fy海k~]Ɵ [ WQO&w .wedz`EFJ`/x`Ȑ!999yyyٹyyyfΜY[[,@vvvHJ!6 jY__?w\Fܹ6-)))&&&:::666%%%''';;;)) x3ܲe dee^\p1\b -(**ZduO9v#, 2|gux=0955uZ.//EIFPzz:رCuoq\k֬9餓^l@LqFQuٿO @`&!G?*++_뮻NV9XzRf鯷?˗/}s9gÆ /((hmmlFJHH`OzgRRRtMw+p뭷~iii <|8uԉ'޽{ǎ{-++s8111999cƌ7nѣY* Zd??S? ~.G9XmL8S N.ZZZ׬Y>&I'xndUEyVZZoG{{{RRw.Z- ;a„ &Bn7?t> .dͷ98p谧&տv~GcbbnvѨ˃iot9Ik͛7/77ǭW__{ l6v}Dc,wɷƁ#g`4 *hE03Fa%%%999s?’}5 T Yv 㖖{lΜ9! ]®zȐ!_}Մ 6n.wi˗|X!O?M69pؿ(f⬬{l;wH/83~)۶mCl6}3h*1)S~{as17>%C͟?SN)**2dHyy9gVT^^n2nw2C!V5(VC~qӧO>|+ukk={yŋ2$---L~ߠ ߠ dMfyF #++7XhM7tW;699YmRа~Ga=[ZZT88 y\bqWyQ?h(w齑#8\888+x(.\88888pppppp .\88888pppppp .\88888&@)tqpq;5Puܱs @1($Q`j@r:1P"(Z?E(EI#\nq9]p.[=@p꟪p)c33<0(Gb~.ηl0ì=u ~(R6{8D!QEr{N;]v&!6pUе3rbb0@/:^FqZTgBͻXy}H(CASg @H$(yD7HnD0{aO}:ulzL@ vIк?*@- #]Uv6deW-!$zZ닋(mׁ_ݿo_!!F"A1<rQS%S ($a)NF:!HqJ;' iRSj_qZ]^V)B]5 "#BV€ ,R"Ini7nn,)/(CU%%I ŀ s`$` w{ Tf#amF(q/< |uޞ)W % 2<1_(UUu{;)RHL 0Bc0B,M} "@Fv XKR'ejb@Qh;}5un^ҸmPyu,:HHT]]@ @(* CO)ѣ20 ;KJQ۞ҝKz`P ޟP/_G{ٿs{_]Mѐ]t?z"Q*u uSf;>5kljje?De8U4LT;R~|c_CiDT[H' SGi\C#T>iSPt\EA@ 9g :ꚦ~YO~QtJsJ0hd]1 8$ " 85-˧]qť--+>ZKBH:;jN?]A芍r>S]WTdM9S@<<|#\ !J)%8+H3sl{#6Wl|`ZRC#A |(PLLԩ6O+V|h}#TXQ\-??(ٟbв?hbץ}1Gǟ""8cu1;恻\ j/vc)!+Ah~.Aڿ`ٿ c, Bl6SP+KJ:aDKT?t:Nz )h__DO5MR(!* `[$v׾3痯|?M8)xf+Mp!$Ƙ$GFFؼpU)%uazwCT>=?99ʤ}PYڏ[bM*mTT1ϐOTak*aP !`vpxmOW7ֳ2"pM'Gz:&)ˉc$v|(8s^1?uwa+XT].g#0j9Qu؟ju1i=K4/}3VA@Slw:=Gy}~/ܳ}7Q&> UW;m[iT쏔 `ؖ䥧lY9t YGsP /]W TmhNƧOAZlu Ʉ;,/ /kh]y-ꙐǑ$G ˆ:f(\z1-yYWk1 $쏺{\0`c.A0D~晧g7xUgz9_E|ɗ^N)?d+L*b S*W$Rm%Zg֩A@VGuܘ/̛}GI (cG}3?/iS a.}n O^)f{@7_#'0+4HĀ3H0F!ʈ{ ~rέg-. ԟRC?x@\`VϽƵkbB>wUҔ IU\D'7TQvT'S*h)&cH"i)QO>qY+04։/%(^|Ѡ _2RO K>%>)ٟ(_t˜"͎xg.Z輄Xx@rv9>CGWB/C^ 竦>= +Mb|iu^|Ѯ / հ?hؿcO |e#68. Opbo#|ˤ^*M)~8N‚}뮝ٚܯ!||iz.} /9PnQ?հ?9Iw_O0@1/wӔ߽o@!@@ą 3bDN :_f Ycw+)/p|)>/M @&D3!|SOXĸA8`|)'$Ok dcT2)//z|ieF'y|9U"z#1mhujM;il,!2: n?{_8mI<<%)*A|_.WD&}c6`0gZJ\~a|K.$/ //%Z?*q'~8G]Wx雂)9)UNd UT975ϻ257/: ~("$Ygw /{]Eb/e@*5/m";N~P1?  Tvt# i zvW?$l^'O0 (7 a]_| .9&̚u嫺?l]J`R{VL?*|)J;bħ%EgV|Sٰ}7Aw$Wm~UAMد<.3x_G$Jc1 WWW];׊r_1aIw_B5))"H/ @ Bmlv|H@_QG3#C!<iѢՍMzg(@!P[ڏ  xggH) KlltrRBZzrFƐ!C*, wn)=c"Nywqz{˅MoR%J OOà2>u:!+5^ _2g {?e.Bc=?B~S}"G8`$䙅몪[GAp~~ƞ}99??`%QD]zY/0ʝ1Iy_-ZawhTN2( fVWdUɾF\B:kӍwRJRB$"ID(!X5䳑^%@w|nD'Y`@Eu ZZ]ϽZ=3 @fГo8R 0Bt`F( IK~wi;kW\yAbJ=$bAЋK7u μk~捠~~sU|K_>9l2z+mhCal2LfCd"QHST%"d#G$J(z x|؟vώԱ!"?oD}cG  @Hg0 t<D)]?λo7+)9U(P|2豿SogsB*/J0߮%tQl991*mH\Vf-31/;9';9#-6%1&"ڂ[`0A|Ao @K>Ad[BRn'O7M?؏b_ԑJGQ͈jks<_Xs톿$yAO.ٟRB:#f̸ui,f0 B$I"g >*b0N!? ÍÍm7c2k;ayɉ Q(:I0F(- (%*TKJdTHtc/5CQ׏N yB-7-{.--] 苂/-SJD(PA F#` $5/~u][~,SrR|||Ld cj J0EB(!(Hs潏v@|SN>kԩsMpBu Ӆc.)GbE}S. k^le_ I@J'[H?/pD5aA`6*^.'txBi„?lŔ)%!@|B(!`r-[ΒD6m2O8ɓN<1"#q:@)p@6_*(Fթ3ێ]U;vUg^={DbBv("шiӦV~'۝JA:97(6_nmp1yW_uޕM6,:NBq0_4p}ЏE-mQZJܿqe'dn[bN!myW__Jxٿi?o?7+ZyF0;+Kf:tH̄c!/J(`0̖xz.(Mv]{ c|d6EẌ́H|iTfOKZO~q9<6g\JBZ :?r(V<"DGϱG@H>kn cϸ=.RF%ID6-^p~ޣ\]8$B(|Δ-"b4m',+|voƟ`aE|@_/gcֈ S%{(GlP?g[wб,BK4ǎ4MhP %n74`t{:)J`0kK?%V=RgcNz_DFYHDv_/^̓EV3e? ,xD"v.:>(co\,{+|qGDy%mzODQ27ƛ]l{0[Zӯ{۞A& K5> ~p\YL͠ |6I cƓ[3OMU\1n/ /5e>سg/LާSD՟}Ư~Dޥٌ]ʪ b5J}bu 񢍄.cv;O}X r*+qWN$upꣂ/+|QIX@ e?}aesY;֒fo/b&D _}p7zbMAxd|u쏽r2 opA7ȷ{_]JYɋskZ[  :PBϹ| X Ӏ{/vf#0`ݝ}f5 -={!*n";]e#5` -\Ԩ_ Bn?K+]w+KG!صhƬ1:%d{r\0.}{;`DD$ISEB\c_BXiLGCYtQ7}Q.bp{"k;{eKATg:]>Z_yk(%FЙc΂/l#pkIpf:^-[~pLU&?(zͷ \/~QQT_sW Me7sT$T_ g?{IMfTp_?~;PUYH\)/Q&W~e]/T6  * րO*-&*k c_&F]<3sa?#} G,,*R.1-~1a $oTT^pdIƮX芳\[g2Jb` tk![P0u$u!IT?pw Kl -701Y[-&BI594k{߮(__@.3!c1Q\/ mNPVV!pgI`|{)RٲRF ϲZM}< Eڲuä5)Tu{yLQp|a?*go EFnf40:ڤU>n?!FyϿ3zTΛo<|hڽ}XʧƟ8of삖b6Q>] ǡŜ{65FҰ|u@Hv):$CSե@pb466u_ㅵya:otOK,f ĸ/?o|?sff/lrUtx̒ COo6cJV_GX!Дȿ\> `|_$}=<Hڥp"I?peYmnG"P".z3[DK1ʏ`tէbb(B 4*goa]nzy :Ô]@~xhG5ت .i 'h lq mM3:m{fX6ml2zA- >l„)|iZ;a SL. :ˀ`do/ &{ֆg7Se;NA˳P1p{K?7Laq8= L$]2L6\RC]Z5 vY-ꂯx` /{7ϼ1+.* ʽ@:̋ 22@O@o_|ct;7c=n77ۖر Fv?e Ko62")bCOR[ ǀ)yjFtpCz, MȡĄ8o\w7 _aHAG%+-Αy0˝ģ(H?.nW`*j` oi؟vӌ &epO;3AZZ2%P0:琔')VoAvM<'qV\"2MR? :vTGE HJ'.?5쯠PICMg.!a/e(e$Ly7_omko y&=« ea$'a`^&hx( oAx4뮫_#lYafG{SH"BF8ańp|>uuHig <3ʂ#~T? v}a:6Ӊ0VWZ W]y.3/nwzz~U[uW%F#a,%}j:QaY02ET7оI >N9"u:5=)!PPXva:-jA* P\ts[T?pTO*+M&JoKXpya@upu}P/%/C03I"_%Ղ2,֐RB0BQSNH3Г6Gw={Հ hJjЏvF.ߗ9jdN*)1:*"J_ğ!ύf3"v wૃ;D=Q)h*7pQ6,ɍ"@h[=gTI"$+#=!23"$t}`ZIQ( @%wcHٙqC|Z;󈳿* ȽﾹQ+юg;n?JHBB aF7BHT|^  P|!_ w)#%Y ls);pҳݵR%XTζ/Nz:uG;_ Bhd%:9\ r'PJ/h B :G^?cBd4;r 80,IVCh?(ӻݩ je4PDD_$ɭ仏VoHTC)5 pcB`NU~;v \x d;_v:IEV@ Lt^Z'\cRc6@ q}cqs55 &PRuV>XpFRbYn#$bo@ݕXew(R̹_A|+ߗT(<[@ v}T5-)͛yӜkECtcv I n˓{xh:ո8)FtZ|InjdN aV&Y,bGP&20#Ҵi`l_]@ݵS* %`0 ٙ0P%0iG}ԟ7OthP/s$ICFKvw0F>%DZM;cڭAͼf$:Ji>Xeޞ|A/Tٿe@hH? #y}{?<,ˎ[%#7ٛ@%r'qD) r"",*O3:yvK$M2<%vHIR|A9-҆Ds8pgQ̙۳DGn _vC')AP@z%|!z=1*&⥥֬!3N# PYǁ{M1F)'}c@BS#4 RFM mT?B%݀TUU fveΘG)i쯞P&`6] r|!CVPl\ԪzS?0o'7.TtRuOAlQ&I^/Px и4 _jװy[zq'sY'O g&/O!(\t\uu(9Lj:H-ν|>_ݺuի[]}%@i~uJ'JC헒$0<ɭbÎ|^{~#GNڹ_L, +RMB@m 8@4z4CeKDEq],E!tFW\6 LC|Ƕ/J $"3EbV[UЭ8Cn];lظ/x/e@g:)cktOFz,5AwΤK3_iQVѢȶ(=^2Q(B2-5$ςE?:"" Cd 7-o\^ߐ傀h}'t/er[ٳQ"BXZڿ'$R1t2d*[~$r:-E] i?h49O4vlEsdyoGj5dy#_{M O?[n{B6C];>n#|.YrX+H&ݶ0BMi_ y `9o3 0 Ӧ>mcv_wNuA9ogr*DRXt`݆P@8B,!z_x_exg!p P,Co 'y;$9g"`z]k!|"[rbB X_D29 E i1hːc A]0s@&h_7M3XKvXk0"BGxoV_ D0Q4]576F*ѿ!$7Ze-b2޴J ynOҖ5R4_q4wj#_v)i诅Q` h_i~k}4ħ_ _‘H]o[h.wjk\al"[vh,y7^|YXp 'p<ѳ- z1 ͇&2^ECSN:l옝 e]qԫ(!TWGKhpnW=7ru-0H7QCi49M3|Yd tqۅwnң#h4c~_),w(&RR$"H)uEC l鯕c~9^qh:ݧ=TUQ ;w,i*DȲ'_/CSe V@A"3> 2s .yf8nd~_f'?H  @xN_oe_?D >5ȇk~.eh9=1Y)| !Ӕi pDcٜCR)-diZFEiBY|ؤ[ɠ/`9]N!D;9_~cyGUU݁$37uߡc~/߿Fѡ=zhd_0- c`L^Dt';@D*<^͹v8ɕ'%v앿xk~0syO]3u;LN1flOHA H f_-eat#@_wyE ¹*xTU]NEƥ=M/Y92`(RS8y:܍JNNZLeK)缲:Tǭp YŢ˚o| W]uO^ݪg!%oxyϿIyWUĒ,6G!D.8Vcm|馧) 2* .!{п_W5}K?,z 3|/~,+06|7?u-|خKG |iƠ@B`C?TWYA/fg! % )$)IJ)I!㏓?4hBķ d?!B'K HU9zڊW#P~lz]BȄ # Hr΢iPRL\ OL1 W\S.j cWI,D@ "4VE?ۂ/"RJ}='^k6V96qb*cp?qb-a4ш!8g47[B@"r(lPX,."Bd,}7nq: m$MZX`8s1W^yP Yn_I)% l|腹*J"D2tsn(qζV" a;uVUbIOSR$dáזB9Jty!BBO7lؚwGc ۈrFϭ>q5yOA7/r4mkJ"q媍Edg}FE|Οxt))zA<+)V:s JTO3|I!%޻z_oF]pDEZXe8~P@`;h{6[p:%Qv_tr]#/Bfuga;?P]]o-ھ DXb?trA$x?~[yu&n_s -,2Il`p"= r9O+.üj O+NN[\E~觝9{-ٽ :=.#UNGmm`Ԯ^{VwD ‹ y;e%nU:Gž} "*~ػo8")p gMeqj8gueBDz #F\[ 0'G˩Y҂/Rp{wܻ]w߷O@09#"kG=Ѕp 7̣DIrlyEP f{Lܳ;*L{dKevr.k*韢 (V+#'D7x_d_`q#_B/n#W]]0Q@QX?pqx~#?$%t:_.YpU6߿xiD$Kue HC Zw᧊%TՔȤqCݑ;#`[8gu/>&jnV8̇kAK!\.e8ϻށ8z_,K/A|W((Ga,9p-[>|ea CkWdɊg};\%&XCC=^vZU<@;[IWUUIuDBpƞ~wi/ء5>Aq гT2|A/L_f!J"6l ,0=^Qٺyӡ¹`M$qϹP8kmUtf#N_zjKip o ZZg}矶kN%ј@~̷6ZXh s?C((!Xk?3AǺuPR+) Wqê}-]mkoM z7rjBCw3ù8oaquc\[Ux:7YUO h (:~(]X:||#+Yt;7W$}Q;c )%&Y诽P2i7h͡hYW_0̥ O_X+I)q}߫,RVPUq-C%@&) \n[9|^2jK:@a̳=Z=)8>Qg%{/+ |{(RW r/J7/dPm<>]tY|@RJ}(v9ɐqb]!B(S'XXqΐJ;|nvŻ"OGs ad"Kٰѿo W 7?r^{ݽRЮ ݒ =9R"suDPUr*O>raN%t7wݥ+G;s@@C 6ׂ/0߈I>-B<h!ma,`|җ_~v*5KeHҔϿvoksRr9c`O/̽sN_ҟ;19g|A&/DDF -Z@-,x ƬkW(#J?~ ~p8FE`^SG:xО5A(o7_g(9 |a3_FE*^k>쏢h@˧X\ 4Dm՚ǝ!Db,ZM?/r Gb`!4't~%Ntڌ5c<%š^>%>7k/h˔ * +K`;} _`&Ƈc唒 ҽYt8W ޑZL3F_tQFP(r '?tG:xuTrKq$1/>|:1P3_b|&y=l=(gEݰa<~>4:ѿ/|ԿD.vCՃs?B>i  rp@Z-by ZO-pqSrП,qF4h?w3 !а+x9lj-'/hKnJJ/.X2RbÂKm/ȳ;|c׭5ϼNjiuz$E=XoF`iyubHXH 𜃏8x׼ {NÆ 6=rONA 7F 2b^c \w=X (dF3Ǭk\ xI.N6Sݶ/+7FS!iNڌ*%b~ʀ:M{c qkz?^ާ*4J ,I%>ϓykkP`:Dߞxu۶U@/+[~s ,Ea(IZRAʶ?6H4ֱ}ɓ6|H@1 " `1;X*J_xWI"ǹM-4z`łܸn0 h,sp5k7A]WXR1Q#?pЪ+kRJ@um}KG>W] pF9|uXoʿ~vh0omyK]<˜U@&)oj}/[S!2 t_22I1mSW喻?*K4 &l!/83Fj,&,ؕsWʢ I奞OXs_SP@o9Jǥ P/ j^G8u s; mB8cK[W+w1u[nj̻/.;uZbȐ[P6 _k3'dE6Зn1`I,#J! DPT ϼA4 )Mʿ  "bz]ֿNOܭ' T "OH{|ؓJ m@e27hW+X댻)Q.yǞsr>,Ύ9}Ͽm.-qj%IN~ܠY{uZ`1G//+|ݟW~ڐ\ƵIm@׵?8cА< ooo٨SfKZ}cT?xc=uhmuXm@u %%t};g]0l£n~+*#)|a_IBu/ɶ%+%={|a?|x TU*-"|oRJ[W F1#}zMp9VWfÎgC.zKӭ& 3Dfn υeޛ|gkXQ.|> Bue>UU4i?okm/}}"FeI0@ Bi7íj4 ީ];*Lnj޽k ʪL+m)d @Co~}@T^g_U{Ӄ|էE"QE '}~|tT_/yfGLXӼ&!aJ_s%qu?zωǝ8Gc&k1ØvuPvy05("2Ƹț)D_7'}K!J=m'T) /1UO xh[}t LO@$v*s|b ;)0L/h-YODj?y#d8lA p[:h@}Ee݊c*#`xQd4=^.2R>- &4'mBU+0-iG6ADF΢p}H2GKyūC 4 !HM*cg]vȒWoW L8/+]+:tuݥSIrTFEc6#Rc9qz49û/ޡ 2b| Jg?㯛@) d*mWܧޚq 867@ٵKR6d|nٳ p`| xynbWޱon%5@FHݺGö$$ ]Z $W7OUš_\쫛_[v9`+9C&JB`] & Ci~|as_x1ҟuqs6(,B^п-g?["9󜐹 ;g_[zv}t98gi#FN Gc 4#wOé8b|Ϻ@3¸qO<$ i~K"n@` sЕJ_w;qܮ]<'"Qh^#4Cdqf̾?3)/|i;Y^x-m.)MH=XeKJ oCGJزBHe^<_TTT!g,'f' N=dsN#Oϫ,x] b5@0D@4 D@ Rj0 `$2 em VS+15#'/2F/ Z >DS8S8 1 _?6?w/(^9{]B6V?Ms©y*ǵ?9Vy!RGѣvj/?{v>)eB`Ov3v-)( _sɔCىtEK(sb(cq 0cگ$L 18 \+\+~{Wg^S]L6?_+0=y-{Mfv/ J) J=]-P18kf&ǟxvǓ&X PE'пyĭ(LUEz 21"D2s˫W{kZ_^pQ)Lޞ'̕$bB•2\cͩ~_Em:UAI6іJgШLb½90OdmOҿ^ U~M)%_1h4wٮmؕ7<{kB)IIHHebslλ5WߜGog||%QZ J)B?*+Jn@g㾱|c}HdL(ҥ#, Ac~P(dxS, h=/}Džpqԧ:4$lT!JK<0vw=%O8F% dD9N^sTxYؤ.݈!ڕqD9(tI,#YIMumxf={8zDu$?uҭݛx*ڕ֮:ǴJێ- // rW;y.C03LaA!䉣oֽg~c'ݿߞ;^zƍ(hTFcA\x':tڵ[_~}=27$Xnމ>ðV;\NBU@02f}pk\Jιi}0p"r#a:c$a/^u)so lk26ܤ}f/o}b[uKt71 G?r_/=OOӟws=:8 $>B-[j7׿ i 7祷?~THT"Ǵ||> rO.U2)FH' 3; !KK=+t~tKWs9|wҹ@0yKk_K∨0@#aG4tm|nPv0icJ&BD@"R%U{+\2a[S SaƵ{?"` baOu@0_ eo_ PПɻsb 1d J:*+*<觳.zuur Ec}m.(xBﳈHO.'p v[tD41+oɱ~{ Ti'?#@ $IHjWk^r#~t3A`+TߋԦm߮=w-.Ü1?C8eAa~잨}q);(B.璁 5_ht=te7~?f[y⹆dYߌ6ظ^q:R#c0 "C;9mPrׇ1UuZXe}x~|RasPSyp"}=(5,V1 /lF%E p֡Oܗc'>7nrRSaX)X*yDLUY<>KIBp$ZVιƝ׎c, /2@zPH\NC@'>33/_Ew[0B_ZvV=v.>{DJhR*)PUSQ ">y∶-=isIOjLQX ,ʨ hUJK*|kW_;#uoGa˹ ?G D)6nG{n!A$IJNppD C?7?.YOmIM14jXwO}>='\$Z;KFV\~p@7 I"v;ng] =?˯(:۞HʞKsP?lذu8<2+.9jճ.D6m]>OemhϹ 7S;jxWz c;-’sȅ+񰹂/$IZWq;UUZ彏~y%+~ޢ?Mц "Th6@c_[|+V!}}sGw]3aO}[_,WuԖy`0 GN^Ugv=p,P̽{URwDeʸrDc՟/}[~dB(FYȺ◩ӶKP<q|e/x"yqF^GLg"qD8XUMI;<|3.}vmc&ZYIѿ_5w Pvk#?x>dÆ6 gfUdR2.x| 9~V9'n{ۡ:yrC@RXU:!Pk?-{5l꿶2ҿe_\;C`9Pں﫷E_翯:O.haI\Zbb1&@|C\5k:b1ѿ9Glm;8z-WOv:XeMPH$ ݆)FDcً !;xkes&)}0,> RA5dɧH53`ćpE 3DTU b6֮vź׬Zfm SdK] c6zUn*^Y[l,X^./yK|qө( cR2Ej8 H$&}܃G>eH QyD21H0$@0zI#N9mdb|Qϵ[f:LL1ӫۥ_ݻwڥ}2or9NEL˿ESjJtպBbj8 "~:׬ߛW?+aDӦ5ku;$ILA/\Q=tM[7uޞ~^U"{.^Dk./I"!IRF2ѨbP,UUա歁M6o l d&"Q@Dc1{k`$"J)x5Op{ؐЫON:x9II$!Lz$' ra#cXsw`,)")e( XM+~^/~mm@mzdݏ. Cyy>fYӲ{G%-7ۉרiMԦVB!g*ÆgϡFypڻ]N@Hك(%;cJ^ wHWPo4JI$E_"hΡSǒn]vat,۲7.[g_wiꑨgTZ]+!xd*Sħ{xmyҥtu8Ȯ<8zS?ΐqDkP6mں-euwؔɣ'i]2jLbuA"oAS"0:nA|Y|+ih<DX,21(>>6U/~;2E(#{K\8=.>eç}ln|OT(\{=k9g"RV6)%SxYDRh{#wa{vb UUDHm42gZ/q&3NSLb~&+=c ,ŽDSE4#"dбC!:y?[/7ZC!-7BE}vchLwSS_[>^jk21EPr DXbnf%Ld(f_ҳADIH4綰C{'5uϝw9b`9K$m dQ?Ɏ["cDB-(S#|)4eo}s_uoKvy?p.v)g X?3@`0{p)t| ,~š(h;vX]TU圧qGv3AIAjvl鲩2Lv].8йcpTdL_ӟɰ62Lލ;RV#AMBkC@ԣkgyұ-{]ݏ[W{>^s:}F;|h}:\BP8EHAr41خ5s&?ykUx h{RFy,`̱XdX=Py]/vK}p$ZS`!'VON=/7X0iM>3H4 G}>w( rJ OZ7-,4o;SIO@B#I&WW;>gkxgBa2s'_zl׻g~}:վ[vZްhLTU5e"SfRm"c SVʼW]tqGM=qT4 mD4ZOv m`:Wx^L=_W\tLpmmspf ۂӟ]qg 27}7BDsDTr'5yhO,(+aݻ2F z":g 2* S(v9\.}Wi,+uV{<+RxEADI2Pe?]Ls(n9*BfcT/zqRYס}]7wܑ/>[Y4ő|$0.D+6OfJDaC?ܱw #~qXkC'Y=`M\_`GП:$㊊Lu{RΨ$I"qTU**^=Ƀ.TI0Wݾ#aC33/LעY1CD w)H$$HA@ k#qbd-bny咧^+^Ŷ.; s!0c+/>r10H?ϾD^-\6TȄ`A[fzx%j&K@EAUU5]w+o-iڕ{gqS)Ua%L^HP2mOⴊx.2MP610lOG?7`O;q& >'?l[lŠOs \~ҒXL5":mNg_[6Vt\[̞u CsDǏnq0Z_d|Mʿ~HUV?[!±.!I7^nWơvD@}wL_UƒɳqRi.3$"gj?M|@H~ v%߭ E#ިqíuzv|g]~6(tsa?~qH*a=S>-X\kEǰpƏO6"ޜ lR롿a J3$+7=Cqh/>cW;SOUusRD=jKjh(B)Âԁ0y[|Vx 3s^LUWxZ=o˔o+sg$DmPZH6eAjT8QYK2yvч*1M BПW=N f-nZ4ls?խo8'\7al^ @OD'8 OEBN_m@#wمG1z# #А ƊtuA ~JPJY[q7\k;wP zl( GӟYc}s_۞"6;.vD)6auFr^[.eNy[E`I|Sͧ>;/{G.>uƔ3mR3|=M/k>O&/=4@joM:Z < nա N~땻%ݵSvڡCUMġp#X tOL8ևd 6_&"#kOáyP4Oo˯t*R sW&)Z06-zLKL_KoKFCK]H vA$Π&|։Cf_2 }AO4pv51EƩqom!'LПG꧿ sxlh\@E᱘ Μm Ec9ͷjLMCӊ/S6$w{ 0N::EIk9.M|Q_?td\k$ƨ&|.9c0ks'^GlaXmioҘ=h#5KۂaMc9eT{޸Q۵PVDv<矿Ka(X:nSW('g1#@OҦOnNA ?PSj3g]T?Xo7m`l+4o FecM[=}v}: @C}tgT6_)sؿ|Ɂ@tl> *X! aV7WkE\;ꀽVVk~= )?`ykI ?s Fums;hجKN۷{l4dLߏ+ fan)B; }BA/×, W)Il.`CґDž pOseuD1,2ϤHGǂ/-+OT @q#&[~牅Üs+)2L*8>ەFѸ))P /П9֔4Oe2}py!e_26(^kKl tWU)/h-z+/:ڜ@m;(9~!˖.xe; $KC/hHei) ]>juuxeЂ/"$% !*UUBT !DƢC) Ҧ܀'9jhf?Z;)Hp].EFN_2f(BDH;񟒄RH)IM I/;f?CӀ:o X@Xi'~:r2zE$c0035 Kos;'&z5|d񢷮R ) W ‘JI)c1j! kP˘iG~pG>aো7}u.xv٩*cHo |ac_q!r9µ2ZQ!D$ T!՘PURj%o0'2:bw.9gpI$* O':5}®&~v񨝪jJ3_Y|I"8r:x4&k57nR Ѩ*8cnӡ}INeݺoWRsP8 `b&uH5 Wx Soq @KKѿp(:d]w:r;#w=v׾F߯¡h8*8"2C/;S/;'e%% 9`0.~o2 E_Žuݎeje~ B_>eΣF?Q!$缑/lhu;' I~hZ&EVswخKݻwݫ[=ѷO=vԞ+ TIAV xт/|?g cv<Ɂ@H,/_EBUN^ro=_.z~A6-y}G87[ײP(c6_0IL8Gmdowo-]w$Plַ/kťg6.]~<ͧLiڑcCY d trcɢ:kD)hr;Nko׮vJK}ҒR_t;xbj4B$rƌ/hˆp(<qI|A_p-z6S}s{U5Q=(c, 8`sg[.Ib (J \. )=v%?yN6_r/҂/!TJ|[{SiY&PX;v2Ta! r? gwUq@63g.6uIKPgCnK4 q}ܐ,MOc_8*۩0[t/o g5~SoӇ6u8BEa| 'Xq? bvmw^PbU3GpWD%h|||2֭ko}'M >6,k[n}x G1vHȂ+ٰ4g,T:)ϼ_W @(  jL !%!gLL\snˀP.JG<+]aPE[#_&~_*J}MkNZb5%*zL5e*!Jf>ej;$׶(#8[Ԟc:5Q'<ǽѿdyB@RRvޏ?mw?wI6F/g~f)w^u|\n&47 -FE8 pdLQf1SX#qO'YkÀ^P!6K_WT!e>̵x^vӣ.e?74)(8gnrΥ faNџ<ˈh=& e^U;7Ii?b4)BHo粋 @ف4Dn@{9A "wqp>]0 dAE|Bij6ߜqe}~YyKj/Ӭ~~Q{qs8ġns_V5,""Ƙӡq'^XbPο!imyW[ӿq/DCQ`mhʁwѯhrJ3-6nYϠNj K'p( ;I"cf k2ϫo-;uӭu̼W\QQV c2h*mq/!?g<R>וE:ۭg 2#1]Â/(9/hdomL9?󺄔|5)CBH}9 @NSɎ&9/ ͉Yt"--BZjis?J]ƤOep6NsiO9 Cp*>5S u:xi#_S{>׭ 6q_ q:-Y; /IHC朇BMm]EB6Fi 4^Y|-l ͜8)%o͜ $oq#.!S.xs&I6L E}X4Jr6ʷhYM=lOWZ r}}JƘ/g1 5xyI/̨ }eٕe2vY R20 [{?~~TK8poQ$Y5tfnP~/DkqΧzfg&$ Q_DPJ07ebY!˿W^*׍|fukvyP,/ꯙ4|^3~píIJhW>R߾lDKJ`L e `l/C"H&;~\oC[:20 "4o(k_v)+:)Go ޒmM\]|*kF=^7NOyUr3f3VK)kր/:1Ţu6E}|SӪhE^3P,i' 4}V.M\W(JD5.$>W?-?(_ȦK/=<rTIa$>!Q)\@';j&k߀{>K/ť-_$ {7XmOnܵgb岮BA"F|)ꏠXU &&.g'=>gYa&=^4_~L},B૶R\R lG?{8dCX˽^*h*z-,wtr%?0s)Y2^<0QJ!?sԩ'581"l E  qFMtF;Z ?vI0xC@Ӗ]xDRQ' ~t <%8 kw}#g0qxx{xr7;<7$>8Q`a$]ׄ.@ƲB $Lc˶/y{r9StW`q!尟vq@5=_:W}W3ٲ% "@3k'55DD عh1'w:O;vCC8 _Wby J`~vdٓS0{λ9llvǧ g8* |ꯢȤ%NX".3NIy l']ӝldQ/F>WHJ-{_:9J ?/uWQۛoI&>;~p/ XB>q/ij] -"EZ|~*evfm1 XAMdž3$Q3+/_W֦>uH"zT?r~TUl|>W XB"b-_"/, *_s7׏ouY9~I'Ks;*!v8{2?^wO4S ظy_dž>ePvj"\}̜qɢR7jK;i#S[/AΉM_AooRR"ZfY'A~J' cW-&^˿_"W>cSݺ81Y\7FrކvuŒ܁.˒8K iGK%߳ {%r2@׮Yi'4!˿WOL-Z8` :˿;N_b5I28L3{$)Dfhl2р0+k a-:W#B۵{$زu(:F/LX[q'E|i՟t]e!_&2LvXъؐ> _uʻ4/lQ*|㘕s +,_>w3vxZmێ93R_j Q?"cL bN ]"@4ՊP XbA6B{%YI&""I9R=d#s@,+_ޕd,/fJ}ǑX߽{ O6<_ށKz2$.HNhKL4C_<ʧ]b>gl_F`e v/ 32KM"+DhlҾ0G>KIi7/J!b(vxz@My38ȫ/G%,Y2Q'ԍ_ /e,jK 7"; {R5B૒yY.ز% QoIgI|\)<{`]t4h ʬX,ig|WMMK%"ts84S2u}BJ{p|dž^=/Y_ȤBTBz/ԨnҀAS5nS1NC;\[`|֟%w@Me<KԿ2p ̑RK76.S <bჇKqIQOybv鞮"TBX=Q s6:EZ7ewzU1Xir53cKapGJB=^5a.W8tx"gc_ބIБxxw`DZuȔ0.lw'X(ZKyueSR[f|)kR#1c|AK c%8kt+AJdsavr.X/}D<|d ۡã  |xoT0J) Z) ИfK=a@*R_ |Xr9\{cik,eSup2IJfBy݈@|Md\j_N3rΜZ=~v@@IPm*6!α*J 1ѱ8ɣcq6 v ty{/5|AD044xxdjz8RKFO9"lOOŏO'g?vX*hp`/ET˿wV#SqO>2VbG|U>Kgl@v>Cs2:K_w1!ę\ޚ̡F)e6;ŏO'ԟ4 |_אdw܁rWu(RP ht<) /ݜ2uD|tDĹ0כ_U%!GF!GSy,"4ZA܁V__{ԉB KK|>P[w|Q♴?al]&Jr Y@e 9}])Iʌbw K%96&>g(/tTLs! 4iU |+쉥vdQY>M ]Dt^b$e|" f|ArJ8UZА+'y@دNKe5o!],4 `|D2_auRRoO OR,_Z'BD^(Z`b|I/ Rx8!iZ)૮ZBvw{f399Yhv[jӖf ՙKd+=]f2a[c$6IM彳/%3傾?8R+ :˿qKs#[6$IX $_?˗ނhrsw钐T] R@d(1T"aU*0ws&?evbjB@N#j5 |Ax= #ͩ?Rdn@T)@I+*QXBNMO礰KG|ꯪIH̺l6a|!x_BND`eYgyj25i3 ]?8$էKoE]@C7TB W_ȲtM,rnW|% I4DLsngTx x5 2筶'@?Y5_>S])ZU!%2 տa/P_,yq#i,@u/|| D %rX8g!dEt 2a3 | 0 !jVPn5XعM h ?#KQg-P[) iw)ž :̆/ | ?Կ|((8lD |GQMzqBԪ~%!|c; ~P RZA*@`7VH/r{0HF B@Z@ZW?Ձ [^=NRʲqjZiV2?: C_j_TZ)Bxi-kkz)"R R@p_]7v"zcu/kK|0"oQ-~t@.Q_J'hXVi/1`{-a.>髺jf ҏXinX{_މrZ@v@_z'O(L|RT>ZXI)y4rWJ0 p3) d2h_ѥP_~kč@Qh!d5fD|Ae׳E;3Ҡ3{Ps)os=^uQP8zL/&wR@m0"uPͷ=aӚRMN@|A8 |~Do>tp/B,|i=.QdvO@˿k* J B([4 -_n0a˒DR[J/~K- vJqƪȁ/w½XԯcYRS42K[^‡.38G'9S!Q' I-NMv@؟H-7`I76=^@|G3l!CRY hbn!KPDf_>:y F4 LCx5S\ˋKYZneYS!K Nҍ7 h&@3",_ h ZPHQ QCzJGBɄ>(TXm' AdHXhٝP;CPUKp[P,>3OD~ 2˖Lap*:fr~Rg , s%R6t_y {:hEt:Y5˓⫑qI|b=i=^HY>p[W U*Yӧ/[$EX$ܙ F|s.!,/G%@23K.iX!ݍ]@*Y|; -ZDBA|Q/8$gLeJrTR~?(YJbALXD Rͼi }MNfV=R@YN'Ux<'nSS9hP/x4:PAU K_,!Re|=%jY$"c6hI' p~ @j!cUTAHruf|n&,R)y˿V~ 819ݙ9 ($ -^ ,rREu`wV)(5E )/z@R=9/YP&l(49o޻;vD |UߨqMLL2ES_9)emj1˿N\ c%A6&+LsVUgQL165]21/$COAh/g2E"H&mߕM+.rK33pMtf5|Cmyof:w-Dl Бc c|^NLdPCOz':g!@"J$lMy]4 GG;Կ1+ox/SM__=fkkFޞcvZ_Ac SV>_b?@T2ݝ L==2%+Ja'nIN3cϒo :2:]/ҘAA'縯'Nc3W0~4JrrxmS@ۓiENg PD@GBkTڣ%D|i_'dxhjiJ{U~ [ooBe"*Al"B%&q"|yf/92 חm՝!Rs'ʐ!@t-G|yj?Ȁ!IL"*KQgd"dkknszTw8Հ O\%TSzJ}; Z@mî4IXNjw9B8tdb=,t"@Te8ء#dgr%ӗۇN$!,_E2&$:2Y;9`HU M\FUz RF/lҪ? SGZ}`t>8µ3W^I m(:|d5I_՗)c).)ތ_@;aYP#CSc<)fҞD`NS@1<2QmL -D{8ovBȞvYNJigzdBUbcB`pИ39}Ccvs{I_Ϋk1j:PgG|y՟ቩcH@>f|ԿIHedRr&$؁ h]y glrxT`1x@sxW$ B-?&)TO tC[t@9*Qs3V56Eԥƀ/Wտ\S)Օ5cOtw% w+ | 9p&Ja}J0Rx쯵h\ƕrO3LL}(; NkڥS ,%䜩i/$$I)I6o03f/%Soصk_Nh)p/GT <|,4GT.[WʔHclQbw{^`"1 _SV0 y&)sL j_>eBd/ t<"BlK#rܫĎQ:VH1F/R՟|5=UW}˿lL-Ylq3[w+k;v5Q09 qO5 |Q拪1aCӹ"gk3ҒdQ% 3ܢAkSS;m;l? Z رXN:Q_z0ۻP$: G|j!S@{GǦ9G2Z˵VG~,KΛJǸJ@* ,1"g? Nj؎[,Z$1@Qc]Ȧ:3#Z cy_B@gQmjZ<4nXqF|yտ$,Y/_c]sҖ|im84NJ%~Wn;8+q?IJҒ˖yy? }/Y. }ӂNe8RKG ǀ(@Kr[ bf1lR8)/" k([w L:pE.qvJC6ء~㦭ѝJоQ ( [E7/-ۆ# _QO+ Ea?n뎟gE|JCy롆9k߁Q4]Sę_SD5VD- _ȵ3?`IQڴik,gvNjR,wʓ ].񅃒j__rl9vgj%:7l߲ir"oL'OX iJeŪ+_O\@D4޼y{,R'Ԑ E3|P^<4+2Q_. PL,X*< V3PML6m>|e潪3irfBWxªys+u^>tعoz m |U|&优|ECn/n9td48I=|TA@JbE, hArɢREfr=DIh&Ƿkts{%Kci3QDiɓNXyN?})]h.e6n-_?i=^j_l,@ OXf!U]7&w t'eS.K[@/eԵQ)r~P"P WJm(%{ V;%Yv> Mܳo8e3;D|EJb3N=&[85$ 8ֆǟüj_HAPoUW~-D?>ܶ?0VIvb7Ī9}H8e(U/_n%a,K>Ԯ2*Ͽ/Q5E|ݕJIf*yk!nKwRPB쩭|z3 HCx, _ඁOSy^) m$U?D|:vEv/fncͱ@B_ Uc'ܬv""yWQ}! y+Bܖ?>QY篞f/r$N{.P<&O$QKw& HFچ%ba/P*@HvhJ$5/q4?KUc SO^/;ۮ1{k6Y p'ԽB_I泛& &UN9rO8_tN?d%/`irQ૆S[Ey$J&̧t.eSM$ "X NWsObT_tgCd6@L: ,@Uպ{G7.*˽xhsb<|+ EA/%Աߑ"xM| ݴ74%, -C r&g׮>nU(V =_9SH?"#aN?1ҸNSW]˿[|5.y豝E Z૒񯪁G"Ft*K~Ⱦ[_5LR=˛C@ dܺۆ `rEy 4.aP:E^..W_qVM"= nMtPhS=0ʩȀ/+/Xg^s񅢝-K"JD|[gEy];vN&MZuj?!H&c:N=eu)sj͖PO0=>Y!S' yE`ۥ$DL|]}EsZ^~5 Y< Ѳoo)?lI%.8 ૦g+\ӳw5<WzD}ׯ,^2 @$)b) =&#_o_<s7,mN-0{DH{3 wg|yTU@KW,x[/iI^WW W^,JEX(?lcU[#rxi\'ȫP/ /PC`3&ܟJ2!g|K)1Ʀ]pD+Kүةc8ԑtۧ{~̞}# \d<&ϯ\|q)_@_|հտ_~@'4|Qi%mfս[\+M_Sr+hNw\ ǿ]%U`wW<w,=ҙ!e} M^q\qZISPpw@FB:-63ІCc'tPS9R@b<`N wSR |ճ/p̪;6JnTV7?t=Ţrm0Ex_䉔N'|bt$ *7; |a Krq6=W1G)z=^&5/Ѯ n.aN9 p s=^tK3Rtio ]-ewlbº N]nIxwL] [k!t/6Dryq}O3I))B˩S@%$>L}לp/CS;TK0 R;fNQuv6 |S2ª7C$ KH&*'BJ?a@B )HJ&U?~wC@!1[L~^ݕ,°6|yԟRIsCLW`>WtY}0 !X~2wڙk>hVX]#ywY<)#!$y]ƧsPcUjELyN7; _zkw̤lfX6óe,flʲ,*òiͰLeF6fͮLpg@udsZ `yʫ/[=6^6'JIt_?mYx-ܖ$݌/+,nWsKR%3{rypn5*gtp+jtO`) 7g/IJ@'ۣd~ eyI_`{e-ngw"&KS|г/^+KQ<||k]OJFrÆG:&`}x3i{Ytw% /nK3LL nuۛO=l|O-/f_ |_.a+g߱RPcEoDN @h _A.kV=ɫ_l<p&SN{WiwӖ{/oQH%΋T&i_oc0F͟4>"Xt𦛾L&p舀jo{~@6E5/݋2@ԇ\k6PQPԿXWgzɒŸo8id4gz_#g_Sf Og/7u`H"aV7|)#@QE+&,?|/=1[sd~yח,[PlOSqȍ I~< !U5Ep!ZӨ?W0Ho]ϼ1|z9#9C|& ﭟUw`'nDdK_;/;j1v^N<)+ Է14͑ᡯ~;U^o_er.jX7Z`ãm~ó*s?ƛ>sGr3}>;[%Q6|]7t C|CGIC-|-A_1$wO|-GAsI{'z|~rsCIi]X _I@ܺo;v=F/_XڃS•ZuW:;}W]yz_2o3j{}e2BJ/ c$ea*dR?{~Xr?"K4_(>rߌ ` ZЕҬRy?XnpsU?߹ 77#9ؙ -K _n+?[v8N%HB4j\ 8 G'yzw_saYa`O~|:,U ˿FJ)>}E~azܩ!_@ zoLHd>˿ש?<4 >9m1>O_wbo{u&J|5`33 qNx_=IJ)!˟54ſvӧOXT;Ʉ񏟸g~v_"!; ٯ;w?8rɦ\=\9G&zSzl?:w'a0 IknyͿZ(yrQ|_7_=o;㼳WHqj@|Xʷ|ͦy|~ۛ/<ՓyX$W@q!] 0RoyC~{S9ڑK׼~BE{P{A?Iƿצ /r 雮dOWJHUߒKU3v A:,15]`)j__,? doOY%NE:q;]YT# Bo e~}]Wð_7/( S@usZɿѽwݿ?;xDhuc眹՗w9+W-& u}VqdhSѿS>ckRk8 no~۫K7={׏MrY"]KϾs)}}TO397*; d=^PHa {o %&GH2%wu~߽mNjagGOhFpA2/XUyN)*u==뷂' J.b >}Y-p2HBMMؼww\uh~;Fw`Pn|˕02zSstrg 9ktt†/уWHKo߶s( ၇6纻??7  U_=ClcaCcO:cG?^xq[vuCONaLNς+/Zz/_(@)_ML0t3J s#Gm2ޭd Cd;X:&dۂY~Bu=\@HUC֭I}.U{UkHiu{8F8;l~SDwkn$s@K $@˜cU"Qe`)DLӖ0DI $ a 7BQLL19Z_"d{`ӗ~cx%SΙ?sOyA}3_z]U8 Rcs.\2_a`RDB}`9(rvt78uz`/~iI\{xbSL ?,TfetOk_vB}5pWu$We]O~vJ%xdݽ [  |^/ Ҫ?Կ| EE)"KչG6*mtv10/܇2J<GQ[3Qe&URTWUk [6|EPJLZNvuS$R dHPYtJjLi`on|$娿SO5m7M/֯-&5SDkA4w R~RyE 7v@U_ Z5 ! CJxf1;'}-I!gBJgfFz|x4X?e_Аo|n嗿}r2e ߣ4 W8˿VY AZ-J纬Dܥ_~Go0(<ӄ aFBSRI^ޯM\ Ǝu_N$4hY/ׁ%1sy~‹"E'~/:Z*;|qtMj٩iui} +߲xʷ}nnB`v ߾/HD*!,_z R>6(/y)E@VIzslxx}c_EO:Q*_gԇ]c/ |ZG{s_XHJ޲W~vĥ nG^w?Odz%f h%FW_u-4%IJAf[~~WOOet@E_zW-U?w5CdNE] ?X.(%I z_[ Ve ϟ{>}BHr׊/|廞2}B8 a oofԾ_}//h=K>RչH |YG˲D2ivwwm_x]\nfQvEDBX6c}` *5[/WguHlv/ЩG B@)IH?7kmm B}+MMs-! F| /!f׃>|ʩڏ{UAPoUW~w =P./忕/",J=3/_ݳ z?U?|׿z4;He /xI/)_x[w<"ʳڿ /h *??sy<H`w"ǟi!,R_BHswz{5XF/kkۇ %tE-KJi i[i~e /u/6oJAc]$HdҰ/ | /"B"|o;g9}K?!$P_Re Lڱ};.{_v*4Tf%X 7l?|T"$)5./"B2to_KŒ|9ŏO_;YvNr,Q9_R_P >fСßg[}ُnRVMU.B\>-!%BH'/OJe*C!&%$]Lw\>#} HWRz=.*j-e;pȁ5'(9 ++~8VYʃZSy׮7zU'~ҼT|1BVY0 %i4"n~7||k/އW.%t*ZXnÌ/OH/D!"f&xɧ[?wZ* ӈ"q It"Mԛ",߼S]~K97yPRCFb_@JitCSy'>vO,ʶk'>_럮{UmSz!)KU1ϯz+2'$ID1n&7w޹n^+%LMot:!$)ت7WTZ|ܠn lAz^Z]v{+_@,;nE5ܗ٫qRA 6p~2< Ǟyɇïd_ug,^70K,Dr8Stz8 vL`H@pxhǶy?=!X<dɛ~rM?9W+_wg׏D% }\쫿}o$D4L3Cwn実vo.WH\@./9I-&C[hZp9sqgMJ%Q,R$f1mR-.U9C4 09O7o9S;{pooj4;;;h{k-[o^E)td䥤جV rFipNpx adZpMQC(EQ(Ȑiy<5^aA_S'^ӫ`iEdfo:wZEyQNnVbJRpىs4"9ǡу݁""2`?t~k67 )O  `vLtto}և QqUUUY%yiYI)ͦ( cB!AP A hj==Vw_l?^*FH@|hz䱷n̜1|RGiA~AvVVzJJ5!N*v6wt759_!A)$&0V!݆9 9)9ٙIIiɶ$[Bfl%pj~>U=~w5swMq}ŢS532 s2󳳳ҳSSRRl6j*"9'RU|~7;ra޲$Hıf$^7y<mc{Q'˻~̯+@\B2IB,R1)@bD#T?:m^2O5$qroƤ؂G !!!!dHHHHHH !!!!!dHHHHHH !!!!!dHHHHHH !!!!!dHHHHHH !!!!!dHHHHHH !!!!!dHHHHH !!!!!dHHHHHL1X(Z9! [/))%%%yc8u%$m ? h"!d)dL5ܓocߣ"C|aʠ3566ʡ$!!!a`Z@s @00`Omރh^:Y (Mi_<#=k4h Ms]ZUaZq9đy>Fc1>hL=2WFD׍QN#_"h9; 36) Q0r8E{x4c4"0DOjDpȏlQ]9~.e܁x6UhPHBdX)- AT@Ad "( @Ad   0@ԫ_2bNoW1K#B}eD  !ؾz \ЯKۊ(CO] ʡZd9D~E ?eHfy0PGE\t-T"KEnpC'rj^MU~ h8 !h, 91{qK *P֌%8L0ODuX^DR!4*:#4L' @`4aFa"an I?$}BgʁZ 5A" /.p?-Ecęb CP- ZJ($Tė  ø"}X((%|Ckh J n#t8p zpHq~I.-Z'X?# 20I/2P,w!@9 QTY9`G!r K@4&zbaQ Pd` ((@@4h?: k" Pr578![pаF=&0eF"sNɨR^JTơiHD/DV>-5B3^'Qm%p(#tT, E(B U #1JhD =T]:c@C_)qBdPP{Ģ\>FImᑔ"L$CGe h8GYi ::?pvy?G`0rxM1(l;cM(}9(h;D~?ZfUi2r`"XͦPx;IY(r^߸^VU6RVobaCZu>L%h+Ikj KN$'%0@inC0Wk?=VVT֎\ᶛ/]D֑_ y_KΨBtW(po{ve$PMӶnvrgZ6U՘b Xva?sڴi҈¡jg^xwmIL!kovݰ/fH+jcϿIII1Ǣ={.;͎}D4d6mT]0;;{D Ξ=kɑ/x]8o0@FFfee 51==F!ԲFܹUuD1GKjjJCC0S^Dv.lbx4$^ӆ3bFs<'';%%ƺCDjww_\=,35I̘M2a+[Ӓ:~jkhhi,++{{^p\#h6xݏl<ȹ :hYcd /*̻u՜lDz)]]]f485rCWWׄkʛ<|  "c,999%%%))i``^|.K,EQ"N.0n¼>,W%&}?rDއđ?bē|0Uλ_,e pyf [z5k[l }fB𬫫+++}!''HLiZKKKmmmkkkJJJyy?\{=f|2Un/9991OpYYYWŪ(1Q֮]yՍ󔮮G}4>˿=+V\mmm1Onnn>Ӧmooo/\lwh*oȬiZGGGZZ!GlޙA#gAXbkVx)6ݪrI9 PAӖiZJr2|UӘgkꚛ  /9X f_~)/ DLNN]Q$ ---ee?Mw~W"'*Ats_ó>WVV󔬬,8%iVS}^fXQQq-w߳DP#LQs47OW\|߀/ mY|i ^~sfWÓ^o@wwϨNommMMM= k{Kmo1"bvd>kcՕ;=~DΧ0"VUUFĤ!h rUTT/?)ϫ8o|:"JKKKMM &HNN?e,yjE4G4jkkeey6[>3}[dh;o[,L+= OQYeyr7{/)Zt]=/;;hq 333- Bns|s2} 555?u댲90T* OOOha'@(bcw\:`///lnn64<xO.Ykl,ecwݗvcR^DxUx~QcT|3nC188p8~Ͼm}7]`{䑧*tzzzCCÑG.5(`=s{{{qqou ^k]%R욀c'ܳW q HNN?C6߿)dREqwee D7t[$4#Ay_@}C"uwwi_rKqFU%X}|hH,e`E[q$'',|>%FF y'X,1=̂]礤(UO?Ծλ>)]]]?!'f֮][^^~8 Z6-{򀮮O?lu^HIj%J< ??fa }P)Qx2f5g0%DΙ7nn{mdHMD9x}wlm}o 5M<)ا_1bF*z,^ph]Ng/::;;gx2:ͭs.fCoox׿ 4`ƀs|4Cmq̮:缕Ͻځ%)$GCz5-$6iTE|4E0MCCT U4 Uq>..qjjjj 4 nn^-'4M35K8k=LQ("FzDP%5#1&-++{饗!FdE\F'ӯ40"൵7psnnOnn.,_~ZnN1d'ՕGkMhYK.WzEv)Ǩn+ҭ"_-N[/-*xQpy O='%%x>h!ͦ®S G!|+z{vygggii?_*|5$$$Agj766Wp : bs-|JEaG4_H[,l( !c@^fsSWWWaaAV9"qlSKO2aʭٽW\~q__uG^uM͌15D~]_)/w |^z郀TNH{}$:댍6`Ȧvpֶܲ&MyNl 6sD\7p*ͭ]C2a90(fR%%%i0)5-#<^^sI̠-??磌ƍ,?|7|S^^xb>qPaħ*iwww~~W_}u/'Llqvttv--lH1Ml˾vͬV ;0DǙ։^U=#d>k3gs8gsnX୷ט*"dӹpnoV>)rRVVcOt~!j0ճ%$$,ލvK6TT8 !!3 ,H0 N\qmmm%%%/ʪ -.) n~xFuR)m 9sfloo~Ma1FTrEEEs/((>1k|/`XDow2d;qٹo~"x6x?H!p)(pݽAۈ$Ac@fL@z]?Pu`xA.%''3sV";^IH]w[KKK$W%-- :ñ'}DOJ8---2f;K_\]exc0̸0@nvz~By T`KL6i]|YZZ/bM}{k~vAB֬r g֑%DHC4s? iL,YwݙrnmcÁɨ~a~]Pr(~/++Da~z<2Ĺ4slkkgdnEQ.fbag@zz;묛6[QDtuygV1Ԟ~eGFFFFCCQG9kV1뿌1">}nՕeƲ߯(kJf\qEQw/yFFF A,Urio`+'b"s$9ZSHΜb=`nb(?q;.'л¹s|E;s;AVkg=ۈyVOOxFD_|R]x?'P4MңO8؎!'4[Cl3*au:SSc3oxlqGDv8f9e^<}}233L%L<\dCo jliӦm@䄢(gAzPlqvN96)݇vH}}}~~I%O>]y< NA"J)ޕ\Jw%ջꛒ]Iߦc7?Vcs6Y\-dP` 0 #%#lHSZƻ^v9EEE.R]{.jULmZ +I%/zdu8^y_X{b-7Ǹ 9A @˼)&TޥxXh LW^}M$6Cv]./߸*M,#N=?$'''& "MSW?ɑab |ُBÈƘ``IƐC{|UUUuu57awN+x_;0ċbHf瑻;43rA=[rkK;'Alca<kߪ 缣~C4##HP3ܥebGxsCUe\{/,7'gmv\w,ޮkdE RUs/~f}GĄ;t I;mqII1vZ[rrWoK"R~c=|##;;e37`SeȾNZvNeeeMM ƌӳ}nNI ׇ9{YUUY]]*=¼;v1KSU 6n@kkP-)))77sjkCe'mTrcv}©w LՑie g/--]Z9)q8窪jcƖ%$t:bJ~Autt̚5kJq(Ħ̘?c(K)-[t}f{ڬ\f$\`.QHE%+jicH> HoYKDhsOd f*i}fW;<} &!f.ٻ+.q8fCUULNN>[ͳ-]f ; ̸Q\\8=xYQw3eE(w}:TN0~-U999f?SSSN~ʛ؎86CĦ}ֹW>;;dg8ũVS+"q9sfwwwi=)8i6S$" ?hYPP`FKJJ?ι2LAqʊkVUU'oAq'9۶~;)L"#CxY|~0w0 蛑ܡIjAgmI!=rͯjG<_}%%@SM?F7ƿȚwOd)0n]"D3YFn4)(`Z٧=2ׯjBFl`N"n/Zp8F" ]82# 0>pgfxg~M"nsvsLfE8%쾣o6@*}obir0I>3?ۛDG% $wܱn9 &~} YsW^rAWW[}ߝW^d~Z---G!2{f'`fQ?>` Ŵ?sԬ5hrPe`Y|?RM< AV5%%%y 8Qn9 GQ&IѿhYb$ "Ц+^^k mo4Yi@w)577\OdE4|H(S6_(5d8xEvd( L0=k[rsNADr5϶Z:lquc~ q00xs]DhQLy;9\m,>0_ PE؈mS 9W[lVrAKf"9믿@eOu ojK_t3nwqq+vu 7Isnz?u{C.+扃B(^';(;7i19|Eֱ; q ddd``^$"nM&'imvu_xXҴIrg<ԁ%<੧HO7TU|fL|W%s"j{ŗ2*LD#'0$3ON2!N09O(s7rqJJgֻ.[~M'I1˖?L3:;; ?㋯)} ބRy34;`hy uORMGӴ򢢢ꃃo|ŕצ]uґ3'69@@BD$| &-?`Xtm4>b-Pݪr_|ERR]'l9Ф95%¸@z͚~'O2{ύϘf33r9G}byW͙L3}R5c ۥ!$(O1–xᗯX"%myws涵VO6VT߿h _a&9V摝xiN( FED8d[NB`ÞdsYUUS,4QI·],t l·{K 67.[ɻwBt_q͝]vo4l44 1MCMC@ʙ1M@!G~q V*rX*,3\}{}[ly[[[aa!kkk >줚FYLn;+ C;wy"1a,X43#"wt֛0LOE9<'D -> k՗TWW̚ FEkk Ӷā-s*~ӯ}J0 ݭ{7{B 6 s{ H>i=U0i^|^x w)Nt\lSUU އTq VGzY* ӝNŇϝh5#t#qKK\z83HIIq:-l80N{&0:ę't,,45:;;餺a 2YXs VWW{)U_j!2 "w`eg•pqǸ\.3>VQQq}m,M* *̷- ))iLhdFF:}~ vjfck;cغv Ec]j%B LA)+.Z;\b+y3JIxdl/+**2L.tI^ESiϘ1v'$$a(}Zې=6uF=\t:ͤuu]vy EO9V]sygg+VUį 30Fhۧ4Q9%$+(sMMMB X( XRwOp~Ղŭ0 KNdߠȼ>:ݲh}<=$rrr抩g<fT}aU\\4oA(3';7!!16YSŐ׋˸sMI{BGz-%NYclY477Dt:KJJvضb"r;oySSIp:Ţ@ q0w;kjjdMjTPUovɟ@`ы_5cp8}W?xU&!R5Kao+^ mns}"pC ֙0edd444.]z̬%Ȥh~>I7^I ݿ.3>,Sڎ3guuuO9( LJv ZbXz{{`g`̙]tכIzlvƹL "zl?~{Cց jꊊ K8${vzzzʞ߿T{SpNJ \Dm\vn^^^WW׵.+-smf8lpsB$5k[I Q,vvZp@"ý,?HZ'X zכocYP sY2tk-BDe-﷟LKKm Šud$I?<ׇܬ޻n>^(pf&jvkE(9`}׮]>LcFzvetwwpy7xk+L-٤htfʼyꜣuq\wFI1 % 9Y9#`cLĬ.XHG(0 9ie}pkkkaaT֖4Ҳjnr"UUE$Qa(++ ) > ΢B @XZȮ[uYooPιb>#u0 UcޓO\涊4 v8h5myt:GǍrrrZ[[=w_6(%qHY m2k$ys+~xb7x%D²AvGAC f,ѵP!a%hnsnjVk[\qՍmj / wغO_:kTeGp̴y2M:D2F=LIa-Rn媿)cFf} 9y @ b~@ĉDD:32GGGǚZ4'+$"k_t5n:DjjjGGǒY5+c22WQ&j&=H^{ni N:[nxsa3㜲٩;:5foHl+VjWXW/hϼlͯNdA!{ߝEؿw&EN#ED̿ͮYإ=HMMu:srs((HN;myIIIccc̳z{{.]bեXajYl?j_{l3{mNfX;dUZZZW].o[oXEz+[4r"U̴)))119w}_n"K{IN:J9䐃^xE3 {''iu%VxEy#?R0? 9[[ŗk<~m6kyY}s[Ro;j( \.5A0#U[z^e&:zOQC Liqh.>l /12HLLt:m6[+"gYW]u577iժ+30,g͐sJ,vMtlyyy}}}o~ MKHm]~k׮5#{n`I+SX>*63&zTƃ7Ҍ Zs:#"g:ㄬ I|vv`d:D[m & j*rCggef&H/xq}rN[F[ܚRiE`iܜnsފvfQ&>(l-@rrP*\&V^raEseV\8}ԁ8eRtD`L+++SRM'[~RΑ;a6JgN\~_d wcSMut:"rV^[rC\x ռޫ.bXwf&p(-5ȏ98 [}leoooVVfM\!xRzr˪o1N^Q#>$DP5LKю=@2w>E͟ "KM/=? ;+@(ňx?N^@l9XN:7x3;;ؙ;*C l(..Fv6mT 7mW;;;!qJbf/߳Ξ={```HZZhɢ0̌>==9~Kg轧l:Ԝܜm Р04ŀeiSz&VYkjjLtdfu8wv=hgilB an2XHx'AS 742}2rjk6ަjD{I_TT425U䀦q쌙|B$~gϝ˻ݹKKK@ =iwwO]s"U%yvGz߯L>N:n<+n枙ڄ)8b>2+ d@2 |E "C@q(@Hۂ0BtDY4:F-.$r 2}ҫ3?33snJ.632]#5v#2"2Ǟxet@ZZZOO^{s%'ۉ'@,௡ܸ1 j26y"ZDv2c Buk~GLxdgL'4"D$AC`!ѐ;FBBO&9ʃ2ʷ@~J yY."+" "BV7NcnPCkXUґR "$  RaPZgj ~@/u%^/0CPYָ k-Вj%ŸhÙE3' #:u`z5)tnTeDPa~(xj?)1F mShΑaPN~u5NbAAA ͯrGQBz`ZψMDƫFD&`cا@EwjYxia+)pΐH2@뭫s[, HO.`(2ɥj(1jʐQp ~b =0-:? z! 55l!? ];H8p hG Á$(и0@}  ].MjPPL(M kd/ ۯZx 4Eeجd,vDjZ7@D IQxذtBd9HE&"" ' r T!p 3,3#!" jFLԄMFh|34)B"fF9©yDD(qH!9!qDN@VPqch|Cx[ <=HQdLqM`z. ^CyH$8F9G 4# \qGƕ cc N^S.s*Kh D@6!֐" $cCmm1G6O lBGos7HCRaG: lKŇl=CitQx ŹL<نr$w>B8K h *̃B }P7)HC ֙ &#p0$$$$$ @BBBBB2 J؝ ^:I1NԘLD@X"PBZ4*@bDEH @B@` &IĂS0W(_q^/P ;Arп3"IwN" Y`^#W DERKG!~ӣנ{ pMQV F-he538q$cp@YPhН8gaa%z^"Bj' clLwda(jU0cDʣ\ C'$#{`hO@p60z ޸^8;/, s_G4pAB 6} ES  \%p:AcH/`)n{."mCD1Ãt+b" #+ (C0("rQ;r`=W E OT ƘP~ C' Pn 8t!8E`CczO*09iláٸFF O2=D+1_BV(<Ɩyx07\0O4\ghvu)X7W2IcSNð-t2}QD1VF 3)jnњ(Fq t!:G!FA( Qē9Z1["E^"p=boSg8'E0fS(+X'"ThЈHߠ [J[ ib_| *+**Cee%ܵ.*a9@-.P?EM*W8%'b*i^'ݽ˭'y|#׈4tlj9$K >EY(da{o -&Q7Ltp `Pa0&?qUv"dHHHHHH !!!!!dHHHHHH !!!!!dHHHHHH !!!!!dHHHHHH !!!!!dHHHHHH !!!!!dHHHHHH !!!!d XdŢ$&&ʇ(!!!a(J0M<|%M4@@$ζXHHHHl00ꭖ?"r F$$$$66> UX aAj@*\x֟oߎ`]@u%~b^~ CY%tEXtdate:create2019-03-13T12:57:06-06:00'%tEXtdate:modify2019-03-13T12:57:06-06:00VM6V tEXtdc:formatapplication/postscriptTr' tEXtillustrator:StartupProfilePrintMlj$tEXtpdf:ProducerAdobe PDF library 10.01?|$tEXtps:HiResBoundingBox115x148+0+0b_tEXtps:LevelAdobe-3.0 EPSF-3.0 .@7tEXtps:SpotColor-0procset Adobe_CoolType_Utility_T42 1.0 0mtEXtstRef:originalDocumentIDuuid:5D20892493BFDB11914A8590D31508C8htEXtstRef:renditionClassproof:pdf."&(tEXtxmp:CreateDate2018-10-30T12:07:03-04:00K/tEXtxmp:CreatorToolAdobe Illustrator CS6 (Windows)y*tEXtxmp:MetadataDate2018-10-30T12:07:04-04:00_(tEXtxmp:ModifyDate2018-10-30T12:07:04-04:00:|9tEXtxmpMM:DocumentIDxmp.did:1DDA8BBB5DDCE811B208EDCE842B44C1￀`9tEXtxmpMM:InstanceIDxmp.iid:1DDA8BBB5DDCE811B208EDCE842B44C1R@>tEXtxmpMM:OriginalDocumentIDuuid:5D20892493BFDB11914A8590D31508C8&tEXtxmpMM:RenditionClassproof:pdf߉ tEXtxmpTPg:HasVisibleOverprintFalsey#tEXtxmpTPg:HasVisibleTransparencyFalse:\=tEXtxmpTPg:NPages1Ɂ۲IENDB`charliecloud-0.37/doc/see_also.rst000066400000000000000000000001731457016721300172010ustar00rootroot00000000000000.. only:: man See also ======== charliecloud(7) Full documentation at: charliecloud-0.37/doc/tutorial.rst000066400000000000000000001304231457016721300172540ustar00rootroot00000000000000Tutorial ******** This tutorial will teach you how to create and run Charliecloud images, using both examples included with the source code as well as new ones you create from scratch. This tutorial assumes that: (a) Charliecloud is in your path, including Charliecloud’s fully unprivileged image builder :code:`ch-image` and (b) Charliecloud is installed under :code:`/usr/local`. (If the second assumption isn’t true, you will just need to modify some paths.) If you want to use Docker to build images, see the :ref:`FAQ `. .. contents:: :depth: 2 :local: .. note:: Shell sessions throughout this documentation will use the prompt :code:`$` to indicate commands executed natively on the host and :code:`>` for commands executed in a container. 90 seconds to Charliecloud ========================== This section is for the impatient. It shows you how to quickly build and run a “hello world” Charliecloud container. If you like what you see, then proceed with the rest of the tutorial to understand what is happening and how to use Charliecloud for your own applications. Using a SquashFS image ---------------------- The preferred workflow uses our internal SquashFS mounting code. Your sysadmin should be able to tell you if this is linked in. :: $ cd /usr/local/share/doc/charliecloud/examples/hello $ ch-image build . inferred image name: hello [...] grown in 3 instructions: hello $ ch-convert hello /var/tmp/hello.sqfs input: ch-image hello output: squash /var/tmp/hello.sqfs packing ... Parallel mksquashfs: Using 8 processors Creating 4.0 filesystem on /var/tmp/hello.sqfs, block size 65536. [=============================================|] 10411/10411 100% [...] done $ ch-run /var/tmp/hello.sqfs -- echo "I’m in a container" I’m in a container Using a directory image ----------------------- If not, you can create image in plain directory format instead. Most of this tutorial uses SquashFS images, but you can adapt it analogously to this section. :: $ cd /usr/local/share/doc/charliecloud/examples/hello $ ch-image build . inferred image name: hello [...] grown in 4 instructions: hello $ ch-convert hello /var/tmp/hello input: ch-image hello output: dir /var/tmp/hello exporting ... done $ ch-run /var/tmp/hello -- echo "I’m in a container" I’m in a container .. note:: You can run perfectly well out of :code:`/tmp`, but because it is bind-mounted automatically, the image root will then appear in multiple locations in the container’s filesystem tree. This can cause confusion for both users and programs. Getting help ============ All the executables have decent help and can tell you what version of Charliecloud you have (if not, please report a bug). For example:: $ ch-run --help Usage: ch-run [OPTION...] IMAGE -- COMMAND [ARG...] Run a command in a Charliecloud container. [...] $ ch-run --version 0.26 Man pages for all commands are provided in this documentation (see table of contents at left) as well as via :code:`man(1)`. Pull an image ============= To start, let’s obtain a container image that someone else has already built. The containery way to do this is the pull operation, which means to move an image from a remote repository into local storage of some kind. First, browse the Docker Hub repository of `official AlmaLinux images `_. Note the list of tags; this is a partial list of image versions that are available. We’ll use the tag “:code:`8`”. Use the Charliecloud program :code:`ch-image` to pull this image to Charliecloud’s internal storage directory:: $ ch-image pull almalinux:8 pulling image: almalinux:8 requesting arch: amd64 manifest list: downloading: 100% manifest: downloading: 100% config: downloading: 100% layer 1/1: 3239c63: downloading: 68.2/68.2 MiB (100%) pulled image: adding to build cache flattening image layer 1/1: 3239c63: listing validating tarball members layer 1/1: 3239c63: changed 42 absolute symbolic and/or hard links to relative resolving whiteouts layer 1/1: 3239c63: extracting image arch: amd64 done $ ch-image list almalinux:8 Images come in lots of different formats; :code:`ch-run` can use directories and SquashFS archives. For this example, we’ll use SquashFS. We use the command :code:`ch-convert` to create a SquashFS image from the image in internal storage, then run it:: $ ch-convert almalinux:8 almalinux.sqfs $ ch-run almalinux.sqfs -- /bin/bash > pwd / > ls bin ch dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var > cat /etc/redhat-release AlmaLinux release 8.7 (Stone Smilodon) > exit What do these commands do? 1. Create a SquashFS-format image (:code:`ch-convert ...`). 2. Create a running container using that image (:code:`ch-run almalinux.sqfs`). 3. Stop processing :code:`ch-run` options (:code:`--`). (This is standard notation for UNIX command line programs.) 4. Run the program :code:`/bin/bash` inside the container, which starts an interactive shell, where we enter a few commands and then exit, returning to the host. Containers are not special ========================== Many folks would like you to believe that containers are magic and special (especially if they want to sell you their container product). This is not the case. To demonstrate, we’ll create a working container image using standard UNIX tools. Many Linux distributions provide tarballs containing installed base images, including Alpine. We can use these in Charliecloud directly:: $ wget -O alpine.tar.gz 'https://github.com/alpinelinux/docker-alpine/blob/v3.16/x86_64/alpine-minirootfs-3.16.3-x86_64.tar.gz?raw=true' $ tar tf alpine.tar.gz | head -10 ./ ./root/ ./var/ ./var/log/ ./var/lock/ ./var/lock/subsys/ ./var/spool/ ./var/spool/cron/ ./var/spool/cron/crontabs ./var/spool/mail This tarball is what’s called a “tarbomb”, so we need to provide an enclosing directory to avoid making a mess:: $ mkdir alpine $ cd alpine $ tar xf ../alpine.tar.gz $ ls bin etc lib mnt proc run srv tmp var dev home media opt root sbin sys usr $ du -sh 5.6M . $ cd .. Now, run a shell in the container! (Note that base Alpine does not have Bash, so we run :code:`/bin/sh` instead.) :: $ ch-run ./alpine -- /bin/sh > pwd / > ls bin etc lib mnt proc run srv tmp var dev home media opt root sbin sys usr > cat /etc/alpine-release 3.16.3 > exit .. warning:: Generally, you should avoid directory-format images on shared filesystems such as NFS and Lustre, in favor of local storage such as :code:`tmpfs` and local hard disks. This will yield better performance for you and anyone else on the shared filesystem. In contrast, SquashFS images should work fine on shared filesystems. Build from Dockerfile ===================== The other containery way to get an image is the build operation. This interprets a recipe, usually a Dockerfile, to create an image and place it into builder storage. We can then extract the image from builder storage to a directory and run it. Charliecloud supports arbitrary image builders. In this tutorial, we use :code:`ch-image`, which comes with Charliecloud, but you can also use others, e.g. Docker or Podman. :code:`ch-image` is a big deal because it is completely unprivileged. Other builders typically run as root or require setuid root helper programs; this raises a number of security questions. We’ll write a “Hello World” Python program and put it into an image we specify with a Dockerfile. Set up a directory to work in:: $ mkdir hello.src $ cd hello.src Type in the following program as :code:`hello.py` using your least favorite editor: .. code-block:: python #!/usr/bin/python3 print("Hello World!") Next, create a file called :code:`Dockerfile` and type in the following recipe: .. code-block:: docker FROM almalinux:8 RUN yum -y install python36 COPY ./hello.py / RUN chmod 755 /hello.py These four instructions say: 1. :code:`FROM`: We are extending the :code:`almalinux:8` *base image*. 2. :code:`RUN`: Install the :code:`python36` RPM package, which we need for our Hello World program. 3. :code:`COPY`: Copy the file :code:`hello.py` we just made to the root directory of the image. In the source argument, the path is relative to the *context directory*, which we’ll see more of below. 4. :code:`RUN`: Make that file executable. .. note:: :code:`COPY` is a standard instruction but has a number of disadvantages in its corner cases. Charliecloud also has :code:`RSYNC`, which addresses these; see :ref:`its documentation ` for details. Let’s build this image:: $ ch-image build -t hello -f Dockerfile . 1. FROM almalinux:8 [...] 4. RUN chmod 755 /hello.py grown in 4 instructions: hello This command says: 1. Build (:code:`ch-image build`) an image named (a.k.a. tagged) “hello” (:code:`-t hello`). 2. Use the Dockerfile called “Dockerfile” (:code:`-f Dockerfile`). 3. Use the current directory as the context directory (:code:`.`). Now, list the images :code:`ch-image` knows about:: $ ch-image list almalinux:8 hello And run the image we just made:: $ cd .. $ ch-convert hello hello.sqfs $ ch-run hello.sqfs -- /hello.py Hello World! This time, we’ve run our application directly rather than starting an interactive shell. Push an image ============= The containery way to share your images is by pushing them to a container registry. In this section, we will set up a registry on GitLab and push the hello image to that registry, then pull it back to compare. Destination setup ----------------- Create a private container registry: 1. Browse to https://gitlab.com (or any other GitLab instance). 2. Log in. You should end up on your *Projects* page. 3. Click *New project* then *Create blank project*. 4. Name your project “:code:`test-registry`”. Leave *Visibility Level* at *Private*. Click *Create project*. You should end up at your project’s main page. 5. At left, choose *Settings* (the gear icon) → *General*, then *Visibility, project features, permissions*. Enable *Container registry*, then click *Save changes*. 6. At left, choose Packages & Registries (the box icon) → Container registry. You should see the message “There are no container images stored for this project”. At this point, we have a container registry set up, and we need to teach :code:`ch-image` how to log into it. On :code:`gitlab.com` and some other instances, you can use your GitLab password. However, GitLab has a thing called a *personal access token* (PAT) that can be used no matter how you log into the GitLab web app. To create one: 1. Click on your avatar at the top right. Choose *Edit Profile*. 2. At left, choose *Access Tokens* (the three-pin plug icon). 3. Type in the name “:code:`registry`”. Tick the boxes *read_registry* and *write_registry*. Click *Create personal access token*. 4. Your PAT will be displayed at the top of the result page under *Your new personal access token*. Copy this string and store it somewhere safe & policy-compliant for your organization. (Also, you can revoke it at the end of the tutorial if you like.) Push ---- We can now use :code:`ch-image push` to push the image to GitLab. (Note that the tagging step you would need for Docker is unnecessary here, because we can just specify a destination reference at push time.) You will need to substitute your GitLab username for :code:`$USER` below. When you are prompted for credentials, enter your GitLab username and copy-paste the PAT you created earlier (or enter your password). .. note:: The specific GitLab path may vary depending on how your GitLab is set up. Check the Docker examples on the empty container registry page for the value you need. For example, if you put your container registry in a group called “containers”, the image reference would be :code:`gitlab.com/$USER/containers/myproj/hello:latest`. :: $ ch-image push hello gitlab.com:5050/$USER/myproj/hello:latest pushing image: hello destination: gitlab.com:5050/$USER/myproj/hello:latest layer 1/1: gathering layer 1/1: preparing preparing metadata starting upload layer 1/1: bca515d: checking if already in repository Username: $USER Password: layer 1/1: bca515d: not present, uploading: 139.8/139.8 MiB(100% config: f969909: checking if already in repository config: f969909: not present, uploading manifest: uploading cleaning up done Go back to your container registry page. You should see your image listed now! Pull and compare ---------------- Let’s pull that image and see how it looks:: $ ch-image pull --auth registry.gitlab.com/$USER/myproj/hello:latest hello.2 pulling image: gitlab.com:5050/$USER/myproj/hello:latest destination: hello.2 [...] $ ch-image list almalinux:8 hello hello.2 $ ch-convert hello.2 ./hello.2 $ ls ./hello.2 bin etc lib mnt proc run srv tmp var dev home media opt root sbin sys usr MPI Hello World =============== In this section, we’ll build and run a simple MPI parallel program. Image builds can be chained. Here, we’ll build a chain of four images: the official :code:`almalinux:8` image, a customized AlmaLinux 8 image, an OpenMPI image, and finally the application image. Important: Many of the specifics in this section will vary from site to site. In that case, follow your site’s instructions instead. Build base images ----------------- First, build two images using the Dockerfiles provided with Charliecloud. These two build should take about 15 minutes total, depending on the speed of your system. Note that Charliecloud infers their names from the Dockerfile name, so we don’t need to specify :code:`-t`. :: $ ch-image build \ -f /usr/local/share/doc/charliecloud/examples/Dockerfile.almalinux_8ch \ /usr/local/share/doc/charliecloud/examples $ ch-image build \ -f /usr/local/share/doc/charliecloud/examples/Dockerfile.openmpi \ /usr/local/share/doc/charliecloud/examples Build image ----------- Next, create a new directory for this project, and within it the following simple C program called :code:`mpihello.c`. (Note the program contains a bug; consider fixing it.) :: #include #include int main (int argc, char **argv) { int msg, rank, rank_ct; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &rank_ct); MPI_Comm_rank(MPI_COMM_WORLD, &rank); printf("hello from rank %d of %d\n", rank, rank_ct); if (rank == 0) { for (int i = 1; i < rank_ct; i++) { MPI_Send(&msg, 1, MPI_INT, i, 0, MPI_COMM_WORLD); printf("rank %d sent %d to rank %d\n", rank, msg, i); } } else { MPI_Recv(&msg, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("rank %d received %d from rank 0\n", rank, msg); } MPI_Finalize(); } Add this :code:`Dockerfile`:: FROM openmpi RUN mkdir /hello WORKDIR /hello COPY mpihello.c . RUN mpicc -o mpihello mpihello.c . (The instruction :code:`WORKDIR` changes directories; the default working directory within a Dockerfile is :code:`/`). Now build. The default Dockerfile is :code:`./Dockerfile`, so we can omit :code:`-f`. :: $ ls Dockerfile mpihello.c $ ch-image build -t mpihello $ ch-image list almalinux:8 almalinux_8ch mpihello openmpi Finally, create a squashball image and copy it to the supercomputer:: $ ch-convert mpihello mpihello.sqfs $ scp mpihello.sqfs super-fe: Run the container ----------------- We’ll run this application interactively. One could also put similar steps in a Slurm batch script. First, obtain a two-node allocation and load Charliecloud:: $ salloc -N2 -t 1:00:00 salloc: Granted job allocation 599518 [...] $ module load charliecloud Then, run the application on all cores in your allocation:: $ srun -c1 ch-run ~/mpihello.sqfs -- /hello/mpihello hello from rank 1 of 72 rank 1 received 0 from rank 0 [...] hello from rank 63 of 72 rank 63 received 0 from rank 0 Win! Build cache =========== :code:`ch-image` subcommands that create images, such as build and pull, can use a build cache to speed repeated operations. That is, an image is created by starting from the empty image and executing a sequence of instructions, largely Dockerfile instructions but also some others like “pull” and “import”. Some instructions are expensive to execute so it’s often cheaper to retrieve their results from cache instead. Let’s set up this example by first resetting the build cache:: $ ch-image build-cache --reset $ mkdir cache-test $ cd cache-test Suppose we have a Dockerfile :code:`a.df`: .. code-block:: docker FROM almalinux:8 RUN sleep 2 && echo foo RUN sleep 2 && echo bar On our first build, we get:: $ ch-image build -t a -f a.df . 1. FROM almalinux:8 [ ... pull chatter omitted ... ] 2. RUN echo foo copying image ... foo 3. RUN echo bar bar grown in 3 instructions: a Note the dot after each instruction’s line number. This means that the instruction was executed. You can also see this in the output of the two :code:`echo` commands. But on our second build, we get:: $ ch-image build -t a -f a.df . 1* FROM almalinux:8 2* RUN sleep 2 && echo foo 3* RUN sleep 2 && echo bar copying image ... grown in 3 instructions: a Here, instead of being executed, each instruction’s results were retrieved from cache. Cache hit for each instruction is indicted by an asterisk (“:code:`*`”) after the line number. Even for such a small and short Dockerfile, this build is noticeably faster than the first. Let’s also try a second, slightly different Dockerfile, :code:`b.df`. The first two instructions are the same, but the third is different. .. code-block:: docker FROM almalinux:8 RUN sleep 2 && echo foo RUN sleep 2 && echo qux Build it:: $ ch-image build -t b -f b.df . 1* FROM almalinux:8 2* RUN sleep 2 && echo foo 3. RUN sleep 2 && echo qux copying image qux grown in 3 instructions: b Here, the first two instructions are hits from the first Dockerfile, but the third is a miss, so Charliecloud retrieves that state and continues building. Finally, inspect the cache:: $ ch-image build-cache --tree * (b) RUN sleep 2 && echo qux | * (a) RUN sleep 2 && echo bar |/ * RUN sleep 2 && echo foo * (almalinux:8) PULL almalinux:8 * (root) ROOT named images: 4 state IDs: 5 commits: 5 files: 317 disk used: 3 MiB Here there are four named images: :code:`a` and :code:`b` that we built, the base image :code:`almalinux:8`, and the empty base of everything :code:`ROOT`. Also note that :code:`a` and :code:`b` diverge after the last common instruction :code:`RUN sleep 2 && echo foo`. Appendices ========== These appendices contain further tutorials that may be enlightening but are less essential to understanding Charliecloud. Namespaces with :code:`unshare(1)` ---------------------------------- :code:`unshare(1)` is a shell command that comes with most new-ish Linux distributions in the :code:`util-linux` package. We will use it to explore a little about how namespaces, which are the basis of containers, work. Identifying the current namespaces ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are several kinds of namespaces, and every process is always in one namespace of each kind. Namespaces within each kind form a tree. Every namespace has an ID number, which you can see in :code:`/proc` with some magic symlinks:: $ ls -l /proc/self/ns total 0 lrwxrwxrwx 1 charlie charlie 0 Mar 31 16:44 cgroup -> 'cgroup:[4026531835]' lrwxrwxrwx 1 charlie charlie 0 Mar 31 16:44 ipc -> 'ipc:[4026531839]' lrwxrwxrwx 1 charlie charlie 0 Mar 31 16:44 mnt -> 'mnt:[4026531840]' lrwxrwxrwx 1 charlie charlie 0 Mar 31 16:44 net -> 'net:[4026531992]' lrwxrwxrwx 1 charlie charlie 0 Mar 31 16:44 pid -> 'pid:[4026531836]' lrwxrwxrwx 1 charlie charlie 0 Mar 31 16:44 pid_for_children -> 'pid:[4026531836]' lrwxrwxrwx 1 charlie charlie 0 Mar 31 16:44 user -> 'user:[4026531837]' lrwxrwxrwx 1 charlie charlie 0 Mar 31 16:44 uts -> 'uts:[4026531838]' Let’s start a new shell with different user and mount namespaces. Note how the ID numbers change for these two, but not the others. :: $ unshare --user --mount > ls -l /proc/self/ns | tee inside.txt total 0 lrwxrwxrwx 1 nobody nogroup 0 Mar 31 16:46 cgroup -> 'cgroup:[4026531835]' lrwxrwxrwx 1 nobody nogroup 0 Mar 31 16:46 ipc -> 'ipc:[4026531839]' lrwxrwxrwx 1 nobody nogroup 0 Mar 31 16:46 mnt -> 'mnt:[4026532733]' lrwxrwxrwx 1 nobody nogroup 0 Mar 31 16:46 net -> 'net:[4026531992]' lrwxrwxrwx 1 nobody nogroup 0 Mar 31 16:46 pid -> 'pid:[4026531836]' lrwxrwxrwx 1 nobody nogroup 0 Mar 31 16:46 pid_for_children -> 'pid:[4026531836]' lrwxrwxrwx 1 nobody nogroup 0 Mar 31 16:46 user -> 'user:[4026532732]' lrwxrwxrwx 1 nobody nogroup 0 Mar 31 16:46 uts -> 'uts:[4026531838]' > exit These IDs are available both in the name and inode number of the magic symlink target:: $ stat -L /proc/self/ns/user File: /proc/self/ns/user Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 4h/4d Inode: 4026531837 Links: 1 Access: (0444/-r--r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2022-12-16 10:56:54.916459868 -0700 Modify: 2022-12-16 10:56:54.916459868 -0700 Change: 2022-12-16 10:56:54.916459868 -0700 Birth: - $ unshare --user --mount -- stat -L /proc/self/ns/user File: /proc/self/ns/user Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 4h/4d Inode: 4026532565 Links: 1 Access: (0444/-r--r--r--) Uid: (65534/ nobody) Gid: (65534/ nogroup) Access: 2022-12-16 10:57:07.136561077 -0700 Modify: 2022-12-16 10:57:07.136561077 -0700 Change: 2022-12-16 10:57:07.136561077 -0700 Birth: - The user namespace ~~~~~~~~~~~~~~~~~~ Unprivileged user namespaces let you map your effective user id (UID) to any UID inside the namespace, and your effective group ID (GID) to any GID. Let’s try it. First, who are we? :: $ id uid=1000(charlie) gid=1000(charlie) groups=1000(charlie),24(cdrom),25(floppy),27(sudo),29(audio) This shows our user (1000 :code:`charlie`), our primary group (1000 :code:`charlie`), and a bunch of supplementary groups. Let’s start a user namespace, mapping our UID to 0 (:code:`root`) and our GID to 0 (:code:`root`):: $ unshare --user --map-root-user > id uid=0(root) gid=0(root) groups=0(root),65534(nogroup) This shows that our UID inside the container is 0, our GID is 0, and all supplementary groups have collapsed into 65534:code:`nogroup`, because they are unmapped inside the namespace. (If :code:`id` complains about not finding names for IDs, just ignore it.) We are root!! Let’s try something sneaky!!! :: > cat /etc/shadow cat: /etc/shadow: Permission denied Drat! The kernel followed the UID map outside the namespace and used that for access control; i.e., we are still acting as us, a normal unprivileged user who cannot read :code:`/etc/shadow`. Something else interesting:: > ls -l /etc/shadow -rw-r----- 1 nobody nogroup 2151 Feb 10 11:51 /etc/shadow > exit This shows up as :code:`nobody:nogroup` because UID 0 and GID 0 outside the container are not mapped to anything inside (i.e., they are *unmapped*). The mount namespace ~~~~~~~~~~~~~~~~~~~ This namespace lets us set up an independent filesystem tree. For this exercise, you will need two terminals. In Terminal 1, set up namespaces and mount a new tmpfs over your home directory:: $ unshare --mount --user > mount -t tmpfs none /home/charlie mount: only root can use "--types" option Wait! What!? The problem now is that you still need to be root inside the container to use the :code:`mount(2)` system call. Try again:: $ unshare --mount --user --map-root-user > mount -t tmpfs none /home/charlie > mount | fgrep /home/charlie none on /home/charlie type tmpfs (rw,relatime,uid=1000,gid=1000) > touch /home/charlie/foo > ls /home/charlie foo In Terminal 2, which is not in the container, note how the mount doesn’t show up in :code:`mount` output and the files you created are not present:: $ ls /home/charlie articles.txt flu-index.tsv perms_test [...] $ mount | fgrep /home/charlie $ Exit the container in Terminal 1:: > exit Namespaces in Charliecloud -------------------------- Let’s revisit the symlinks in :code:`/proc`, but this time with Charliecloud:: $ ls -l /proc/self/ns total 0 lrwxrwxrwx 1 charlie charlie 0 Sep 28 11:24 ipc -> ipc:[4026531839] lrwxrwxrwx 1 charlie charlie 0 Sep 28 11:24 mnt -> mnt:[4026531840] lrwxrwxrwx 1 charlie charlie 0 Sep 28 11:24 net -> net:[4026531969] lrwxrwxrwx 1 charlie charlie 0 Sep 28 11:24 pid -> pid:[4026531836] lrwxrwxrwx 1 charlie charlie 0 Sep 28 11:24 user -> user:[4026531837] lrwxrwxrwx 1 charlie charlie 0 Sep 28 11:24 uts -> uts:[4026531838] $ ch-run /var/tmp/hello -- ls -l /proc/self/ns total 0 lrwxrwxrwx 1 charlie charlie 0 Sep 28 17:34 ipc -> ipc:[4026531839] lrwxrwxrwx 1 charlie charlie 0 Sep 28 17:34 mnt -> mnt:[4026532257] lrwxrwxrwx 1 charlie charlie 0 Sep 28 17:34 net -> net:[4026531969] lrwxrwxrwx 1 charlie charlie 0 Sep 28 17:34 pid -> pid:[4026531836] lrwxrwxrwx 1 charlie charlie 0 Sep 28 17:34 user -> user:[4026532256] lrwxrwxrwx 1 charlie charlie 0 Sep 28 17:34 uts -> uts:[4026531838] The container has different mount (:code:`mnt`) and user (:code:`user`) namespaces, but the rest of the namespaces are shared with the host. This highlights Charliecloud’s focus on functionality (make your container run), rather than isolation (protect the host from your container). Normally, each invocation of :code:`ch-run` creates a new container, so if you have multiple simultaneous invocations, they will not share containers. In some cases this can cause problems with MPI programs. However, there is an option :code:`--join` that can solve them; see the :ref:`FAQ ` for details. All you need is Bash -------------------- In this exercise, we’ll use shell commands to create minimal container image with a working copy of Bash, and that’s all. To do so, we need to set up a directory with the Bash binary, the shared libraries it uses, and a few other hooks needed by Charliecloud. **Important:** Your Bash is almost certainly linked differently than described below. Use the paths from your terminal, not this tutorial. Adjust the steps below as needed. It will not work otherwise. :: $ ldd /bin/bash linux-vdso.so.1 (0x00007ffdafff2000) libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f6935cb6000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6935cb1000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6935af0000) /lib64/ld-linux-x86-64.so.2 (0x00007f6935e21000) $ ls -l /lib/x86_64-linux-gnu/libc.so.6 lrwxrwxrwx 1 root root 12 May 1 2019 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.28.so The shared libraries pointed to are symlinks, so we’ll use :code:`cp -L` to dereference them and copy the target files. :code:`linux-vdso.so.1` is a kernel thing, not a shared library file, so we don’t copy that. Set up the container:: $ mkdir alluneed $ cd alluneed $ mkdir bin $ mkdir dev $ mkdir lib $ mkdir lib64 $ mkdir lib/x86_64-linux-gnu $ mkdir proc $ mkdir sys $ mkdir tmp $ cp -pL /bin/bash ./bin $ cp -pL /lib/x86_64-linux-gnu/libtinfo.so.6 ./lib/x86_64-linux-gnu $ cp -pL /lib/x86_64-linux-gnu/libdl.so.2 ./lib/x86_64-linux-gnu $ cp -pL /lib/x86_64-linux-gnu/libc.so.6 ./lib/x86_64-linux-gnu $ cp -pL /lib64/ld-linux-x86-64.so.2 ./lib64/ld-linux-x86-64.so.2 $ cd .. $ ls -lR alluneed ./alluneed: total 0 drwxr-x--- 2 charlie charlie 60 Mar 31 17:15 bin drwxr-x--- 2 charlie charlie 40 Mar 31 17:26 dev drwxr-x--- 2 charlie charlie 80 Mar 31 17:27 etc drwxr-x--- 3 charlie charlie 60 Mar 31 17:17 lib drwxr-x--- 2 charlie charlie 60 Mar 31 17:19 lib64 drwxr-x--- 2 charlie charlie 40 Mar 31 17:26 proc drwxr-x--- 2 charlie charlie 40 Mar 31 17:26 sys drwxr-x--- 2 charlie charlie 40 Mar 31 17:27 tmp ./alluneed/bin: total 1144 -rwxr-xr-x 1 charlie charlie 1168776 Apr 17 2019 bash ./alluneed/dev: total 0 ./alluneed/lib: total 0 drwxr-x--- 2 charlie charlie 100 Mar 31 17:19 x86_64-linux-gnu ./alluneed/lib/x86_64-linux-gnu: total 1980 -rwxr-xr-x 1 charlie charlie 1824496 May 1 2019 libc.so.6 -rw-r--r-- 1 charlie charlie 14592 May 1 2019 libdl.so.2 -rw-r--r-- 1 charlie charlie 183528 Nov 2 12:16 libtinfo.so.6 ./alluneed/lib64: total 164 -rwxr-xr-x 1 charlie charlie 165632 May 1 2019 ld-linux-x86-64.so.2 ./alluneed/proc: total 0 ./alluneed/sys: total 0 ./alluneed/tmp: total 0 Next, start a container and run :code:`/bin/bash` within it. Option :code:`--no-passwd` turns off some convenience features that this image isn’t prepared for. :: $ ch-run --no-passwd ./alluneed -- /bin/bash > pwd / > echo "hello world" hello world > ls / bash: ls: command not found > echo * bin dev home lib lib64 proc sys tmp > exit It’s not very useful since the only commands we have are Bash built-ins, but it’s a container! Interacting with the host ------------------------- Charliecloud is not an isolation layer, so containers have full access to host resources, with a few quirks. This section demonstrates how that works. Filesystems ~~~~~~~~~~~ Charliecloud makes host directories available inside the container using bind mounts, which is somewhat like a hard link in that it causes a file or directory to appear in multiple places in the filesystem tree, but it is a property of the running kernel rather than the filesystem. Several host directories are always bind-mounted into the container. These include system directories such as :code:`/dev`, :code:`/proc`, :code:`/sys`, and :code:`/tmp`. Others can be requested with a command line option, e.g. :code:`--home` bind-mounts the invoking user’s home directory. Charliecloud uses recursive bind mounts, so for example if the host has a variety of sub-filesystems under :code:`/sys`, as Ubuntu does, these will be available in the container as well. In addition to these, arbitrary user-specified directories can be added using the :code:`--bind` or :code:`-b` switch. By default, mounts use the same path as provided from the host. In the case of directory images, which are writeable, the target mount directory will be automatically created before the container is started:: $ mkdir /var/tmp/foo0 $ echo hello > /var/tmp/foo0/bar $ mkdir /var/tmp/foo1 $ echo world > /var/tmp/foo1/bar $ ch-run -b /var/tmp/foo0 -b /var/tmp/foo1 /var/tmp/hello -- bash > cat /var/tmp/foo0/bar hello > cat /var/tmp/foo1/bar world However, as SquashFS filesystems are read-only, in this case you must provide a destination that already exists, like those created under :code:`/mnt`:: $ mkdir /var/tmp/foo0 $ echo hello > /var/tmp/foo0/bar $ mkdir /var/tmp/foo1 $ echo world > /var/tmp/foo1/bar $ ch-run -b /var/tmp/foo0 -b /var/tmp/foo1 /var/tmp/hello -- bash ch-run[1184427]: error: can’t mkdir: /var/tmp/hello/var/tmp/foo0: Read-only file system (ch_misc.c:142 30) $ ch-run -b /var/tmp/foo0:/mnt/0 -b /var/tmp/foo1:/mnt/1 /var/tmp/hello -- bash > ls /mnt 0 1 2 3 4 5 6 7 8 9 > cat /mnt/0/bar hello > cat /mnt/1/bar world Network ~~~~~~~ Charliecloud containers share the host’s network namespace, so most network things should be the same. However, SSH is not aware of Charliecloud containers. If you SSH to a node where Charliecloud is installed, you will get a shell on the host, not in a container, even if :code:`ssh` was initiated from a container:: $ stat -L --format='%i' /proc/self/ns/user 4026531837 $ ssh localhost stat -L --format='%i' /proc/self/ns/user 4026531837 $ ch-run /var/tmp/hello.sqfs -- /bin/bash > stat -L --format='%i' /proc/self/ns/user 4026532256 > ssh localhost stat -L --format='%i' /proc/self/ns/user 4026531837 There are a couple ways to SSH to a remote node and run commands inside a container. The simplest is to manually invoke :code:`ch-run` in the :code:`ssh` command:: $ ssh localhost ch-run /var/tmp/hello.sqfs -- stat -L --format='%i' /proc/self/ns/user 4026532256 .. note:: Recall that by default, each :code:`ch-run` invocation creates a new container. That is, the :code:`ssh` command above has not entered an existing user namespace :code:`’2256`; rather, it has re-used the namespace ID :code:`’2256`. Another may be to edit one's shell initialization scripts to check the command line and :code:`exec(1)` :code:`ch-run` if appropriate. This is brittle but avoids wrapping :code:`ssh` or altering its command line. User and group IDs ~~~~~~~~~~~~~~~~~~ Unlike Docker and some other container systems, Charliecloud tries to make the container’s users and groups look the same as the host’s. This is accomplished by bind-mounting a custom :code:`/etc/passwd` and :code:`/etc/group` into the container. For example:: $ id -u 901 $ whoami charlie $ ch-run /var/tmp/hello.sqfs -- bash > id -u 901 > whoami charlie More specifically, the user namespace, when created without privileges as Charliecloud does, lets you map any container UID to your host UID. :code:`ch-run` implements this with the :code:`--uid` switch. So, for example, you can tell Charliecloud you want to be root, and it will tell you that you’re root:: $ ch-run --uid 0 /var/tmp/hello.sqfs -- bash > id -u 0 > whoami root But, as shown above, this doesn’t get you anything useful, because the container UID is mapped back to your UID on the host before permission checks are applied:: > dd if=/dev/mem of=/tmp/pwned dd: failed to open '/dev/mem': Permission denied This mapping also affects how users are displayed. For example, if a file is owned by you, your host UID will be mapped to your container UID, which is then looked up in :code:`/etc/passwd` to determine the display name. In typical usage without :code:`--uid`, this mapping is a no-op, so everything looks normal:: $ ls -nd ~ drwxr-xr-x 87 901 901 4096 Sep 28 12:12 /home/charlie $ ls -ld ~ drwxr-xr-x 87 charlie charlie 4096 Sep 28 12:12 /home/charlie $ ch-run /var/tmp/hello.sqfs -- bash > ls -nd ~ drwxr-xr-x 87 901 901 4096 Sep 28 18:12 /home/charlie > ls -ld ~ drwxr-xr-x 87 charlie charlie 4096 Sep 28 18:12 /home/charlie But if :code:`--uid` is provided, things can seem odd. For example:: $ ch-run --uid 0 /var/tmp/hello.sqfs -- bash > ls -nd /home/charlie drwxr-xr-x 87 0 901 4096 Sep 28 18:12 /home/charlie > ls -ld /home/charlie drwxr-xr-x 87 root charlie 4096 Sep 28 18:12 /home/charlie This UID mapping can contain only one pair: an arbitrary container UID to your effective UID on the host. Thus, all other users are unmapped, and they show up as :code:`nobody`:: $ ls -n /tmp/foo -rw-rw---- 1 902 902 0 Sep 28 15:40 /tmp/foo $ ls -l /tmp/foo -rw-rw---- 1 sig sig 0 Sep 28 15:40 /tmp/foo $ ch-run /var/tmp/hello.sqfs -- bash > ls -n /tmp/foo -rw-rw---- 1 65534 65534 843 Sep 28 21:40 /tmp/foo > ls -l /tmp/foo -rw-rw---- 1 nobody nogroup 843 Sep 28 21:40 /tmp/foo User namespaces have a similar mapping for GIDs, with the same limitation --- exactly one arbitrary container GID maps to your effective *primary* GID. This can lead to some strange-looking results, because only one of your GIDs can be mapped in any given container. All the rest become :code:`nogroup`:: $ id uid=901(charlie) gid=901(charlie) groups=901(charlie),903(nerds),904(losers) $ ch-run /var/tmp/hello.sqfs -- id uid=901(charlie) gid=901(charlie) groups=901(charlie),65534(nogroup) $ ch-run --gid 903 /var/tmp/hello.sqfs -- id uid=901(charlie) gid=903(nerds) groups=903(nerds),65534(nogroup) However, this doesn’t affect access. The container process retains the same GIDs from the host perspective, and as always, the host IDs are what control access:: $ ls -l /tmp/primary /tmp/supplemental -rw-rw---- 1 sig charlie 0 Sep 28 15:47 /tmp/primary -rw-rw---- 1 sig nerds 0 Sep 28 15:48 /tmp/supplemental $ ch-run /var/tmp/hello.sqfs -- bash > cat /tmp/primary > /dev/null > cat /tmp/supplemental > /dev/null One area where functionality *is* reduced is that :code:`chgrp(1)` becomes useless. Using an unmapped group or :code:`nogroup` fails, and using a mapped group is a no-op because it’s mapped back to the host GID:: $ ls -l /tmp/bar rw-rw---- 1 charlie charlie 0 Sep 28 16:12 /tmp/bar $ ch-run /var/tmp/hello.sqfs -- chgrp nerds /tmp/bar chgrp: changing group of '/tmp/bar': Invalid argument $ ch-run /var/tmp/hello.sqfs -- chgrp nogroup /tmp/bar chgrp: changing group of '/tmp/bar': Invalid argument $ ch-run --gid 903 /var/tmp/hello.sqfs -- chgrp nerds /tmp/bar $ ls -l /tmp/bar -rw-rw---- 1 charlie charlie 0 Sep 28 16:12 /tmp/bar Workarounds include :code:`chgrp(1)` on the host or fastidious use of setgid directories:: $ mkdir /tmp/baz $ chgrp nerds /tmp/baz $ chmod 2770 /tmp/baz $ ls -ld /tmp/baz drwxrws--- 2 charlie nerds 40 Sep 28 16:19 /tmp/baz $ ch-run /var/tmp/hello.sqfs -- touch /tmp/baz/foo $ ls -l /tmp/baz/foo -rw-rw---- 1 charlie nerds 0 Sep 28 16:21 /tmp/baz/foo Apache Spark ------------ This example is in :code:`examples/spark`. Build a SquashFS image of it and upload it to your supercomputer. Interactive ~~~~~~~~~~~ We need to first create a basic configuration for Spark, as the defaults in the Dockerfile are insufficient. For real jobs, you’ll want to also configure performance parameters such as memory use; see `the documentation `_. First:: $ mkdir -p ~/sparkconf $ chmod 700 ~/sparkconf We’ll want to use the supercomputer’s high-speed network. For this example, we’ll find the Spark master’s IP manually:: $ ip -o -f inet addr show | cut -d/ -f1 1: lo inet 127.0.0.1 2: eth0 inet 192.168.8.3 8: eth1 inet 10.8.8.3 Your site support can tell you which to use. In this case, we’ll use 10.8.8.3. Create some configuration files. Replace :code:`[MYSECRET]` with a string only you know. Edit to match your system; in particular, use local disks instead of :code:`/tmp` if you have them:: $ cat > ~/sparkconf/spark-env.sh SPARK_LOCAL_DIRS=/tmp/spark SPARK_LOG_DIR=/tmp/spark/log SPARK_WORKER_DIR=/tmp/spark SPARK_LOCAL_IP=127.0.0.1 SPARK_MASTER_HOST=10.8.8.3 $ cat > ~/sparkconf/spark-defaults.conf spark.authenticate true spark.authenticate.secret [MYSECRET] We can now start the Spark master:: $ ch-run -b ~/sparkconf /var/tmp/spark.sqfs -- /spark/sbin/start-master.sh Look at the log in :code:`/tmp/spark/log` to see that the master started correctly:: $ tail -7 /tmp/spark/log/*master*.out 17/02/24 22:37:21 INFO Master: Starting Spark master at spark://10.8.8.3:7077 17/02/24 22:37:21 INFO Master: Running Spark version 2.0.2 17/02/24 22:37:22 INFO Utils: Successfully started service 'MasterUI' on port 8080. 17/02/24 22:37:22 INFO MasterWebUI: Bound MasterWebUI to 127.0.0.1, and started at http://127.0.0.1:8080 17/02/24 22:37:22 INFO Utils: Successfully started service on port 6066. 17/02/24 22:37:22 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066 17/02/24 22:37:22 INFO Master: I have been elected leader! New state: ALIVE If you can run a web browser on the node, browse to :code:`http://localhost:8080` for the Spark master web interface. Because this capability varies, the tutorial does not depend on it, but it can be informative. Refresh after each key step below. The Spark workers need to know how to reach the master. This is via a URL; you can get it from the log excerpt above, or consult the web interface. For example:: $ MASTER_URL=spark://10.8.8.3:7077 Next, start one worker on each compute node. In this tutorial, we start the workers using :code:`srun` in a way that prevents any subsequent :code:`srun` invocations from running until the Spark workers exit. For our purposes here, that’s OK, but it’s a significant limitation for some jobs. (See `issue #230 `_.) Alternatives include :code:`pdsh`, which is the approach we use for the Spark tests (:code:`examples/other/spark/test.bats`), or a simple for loop of :code:`ssh` calls. Both of these are also quite clunky and do not scale well. :: $ srun sh -c " ch-run -b ~/sparkconf /var/tmp/spark.sqfs -- \ spark/sbin/start-slave.sh $MASTER_URL \ && sleep infinity" & One of the advantages of Spark is that it’s resilient: if a worker becomes unavailable, the computation simply proceeds without it. However, this can mask issues as well. For example, this example will run perfectly fine with just one worker, or all four workers on the same node, which aren’t what we want. Check the master log to see that the right number of workers registered:: $ fgrep worker /tmp/spark/log/*master*.out 17/02/24 22:52:24 INFO Master: Registering worker 127.0.0.1:39890 with 16 cores, 187.8 GB RAM 17/02/24 22:52:24 INFO Master: Registering worker 127.0.0.1:44735 with 16 cores, 187.8 GB RAM 17/02/24 22:52:24 INFO Master: Registering worker 127.0.0.1:22445 with 16 cores, 187.8 GB RAM 17/02/24 22:52:24 INFO Master: Registering worker 127.0.0.1:29473 with 16 cores, 187.8 GB RAM Despite the workers calling themselves 127.0.0.1, they really are running across the allocation. (The confusion happens because of our :code:`$SPARK_LOCAL_IP` setting above.) This can be verified by examining logs on each compute node. For example (note single quotes):: $ ssh 10.8.8.4 -- tail -3 '/tmp/spark/log/*worker*.out' 17/02/24 22:52:24 INFO Worker: Connecting to master 10.8.8.3:7077... 17/02/24 22:52:24 INFO TransportClientFactory: Successfully created connection to /10.8.8.3:7077 after 263 ms (216 ms spent in bootstraps) 17/02/24 22:52:24 INFO Worker: Successfully registered with master spark://10.8.8.3:7077 We can now start an interactive shell to do some Spark computing:: $ ch-run -b ~/sparkconf /var/tmp/spark.sqfs -- /spark/bin/pyspark --master $MASTER_URL Let’s use this shell to estimate 𝜋 (this is adapted from one of the Spark `examples `_): .. code-block:: pycon >>> import operator >>> import random >>> >>> def sample(p): ... (x, y) = (random.random(), random.random()) ... return 1 if x*x + y*y < 1 else 0 ... >>> SAMPLE_CT = int(2e8) >>> ct = sc.parallelize(xrange(0, SAMPLE_CT)) \ ... .map(sample) \ ... .reduce(operator.add) >>> 4.0*ct/SAMPLE_CT 3.14109824 (Type Control-D to exit.) We can also submit jobs to the Spark cluster. This one runs the same example as included with the Spark source code. (The voluminous logging output is omitted.) :: $ ch-run -b ~/sparkconf /var/tmp/spark.sqfs -- \ /spark/bin/spark-submit --master $MASTER_URL \ /spark/examples/src/main/python/pi.py 1024 [...] Pi is roughly 3.141211 [...] Exit your allocation. Slurm will clean up the Spark daemons. Success! Next, we’ll run a similar job non-interactively. Non-interactive ~~~~~~~~~~~~~~~ We’ll re-use much of the above to run the same computation non-interactively. For brevity, the Slurm script at :code:`examples/other/spark/slurm.sh` is not reproduced here. Submit it as follows. It requires three arguments: the squashball, the image directory to unpack into, and the high-speed network interface. Again, consult your site administrators for the latter. :: $ sbatch -N4 slurm.sh spark.sqfs /var/tmp ib0 Submitted batch job 86754 Output:: $ fgrep 'Pi is' slurm-86754.out Pi is roughly 3.141393 Success! (to four significant digits) .. LocalWords: NEWROOT rhel oldfind oldf mem drwxr xr sig drwxrws mpihello .. LocalWords: openmpi rwxr rwxrwx cn cpus sparkconf MasterWebUI MasterUI .. LocalWords: StandaloneRestServer MYSECRET TransportClientFactory sc tf .. LocalWords: containery lockdev subsys cryptsetup utmp xf bca Recv df af .. LocalWords: minirootfs alpinelinux cdrom ffdafff cb alluneed myproj fe .. LocalWords: pL ib charliecloud-0.37/examples/000077500000000000000000000000001457016721300157255ustar00rootroot00000000000000charliecloud-0.37/examples/.dockerignore000066400000000000000000000002321457016721300203760ustar00rootroot00000000000000# Exclude everything by default * # Needed for Dockerfile.openmpi !dont-init-ucx-on-intel-cray.patch # Needed for Dockerfile.exhaustive !Dockerfile.* charliecloud-0.37/examples/Dockerfile.almalinux_8ch000066400000000000000000000053411457016721300224550ustar00rootroot00000000000000# ch-test-scope: standard FROM almalinux:8 # This image has three purposes: (1) demonstrate we can build a AlmaLinux 8 # image, (2) provide a build environment for Charliecloud EPEL 8 RPMs, and (3) # provide image packages necessary for Obspy and Paraview. # # Quirks: # # 1. Install the dnf ovl plugin to work around RPMDB corruption when # building images with Docker and the OverlayFS storage driver. # # 2. Enable PowerTools repo, because some packages in EPEL depend on it. # # 3. Install packages needed to build el8 rpms. # # 4. Issue #1103: Install libarchive to resolve cmake bug # # 5. AlmaLinux lost their GPG key, so manual intervention is required to # install current packages [1]. # # [1]: https://almalinux.org/blog/2023-12-20-almalinux-8-key-update/ RUN rpm --import https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux RUN dnf install -y --setopt=install_weak_deps=false \ epel-release \ 'dnf-command(config-manager)' RUN dnf config-manager --enable powertools RUN dnf install -y --setopt=install_weak_deps=false \ dnf-plugin-ovl \ autoconf \ automake \ gcc \ git \ libarchive \ libpng-devel \ make \ python3 \ python3-devel \ python3-lark-parser \ python3-requests \ python3-sphinx \ python3-sphinx_rtd_theme \ rpm-build \ rpmlint \ rsync \ squashfs-tools \ squashfuse \ wget \ which \ && dnf clean all # Need wheel to install bundled Lark, and the RPM version doesn’t work. RUN pip3 install wheel # AlmaLinux's linker doesn’t search these paths by default; add them because we # will install stuff later into /usr/local. RUN echo "/usr/local/lib" > /etc/ld.so.conf.d/usrlocal.conf \ && echo "/usr/local/lib64" >> /etc/ld.so.conf.d/usrlocal.conf \ && ldconfig # Install ImageMagick # The latest, 7.1.0, fails to install with a cryptic libtool error. ¯\_(ツ)_/¯ ARG MAGICK_VERSION=7.0.11-14 RUN wget -nv -O ImageMagick-${MAGICK_VERSION}.tar.gz \ "https://github.com/ImageMagick/ImageMagick/archive/refs/tags/${MAGICK_VERSION}.tar.gz" \ && tar xf ImageMagick-${MAGICK_VERSION}.tar.gz \ && cd ImageMagick-${MAGICK_VERSION} \ && ./configure --prefix=/usr/local \ && make -j $(getconf _NPROCESSORS_ONLN) install \ && rm -Rf ../ImageMagick-${MAGICK_VERSION} # Add mount points for files and directories for paraview and obspy comparison # tests. RUN mkdir /diff \ && echo "example bind mount file" > /a.png \ && echo "example bind mount file" > /b.png charliecloud-0.37/examples/Dockerfile.centos_7ch000066400000000000000000000024131457016721300217520ustar00rootroot00000000000000# ch-test-scope: full FROM centos:7 # This image has two purposes: (1) demonstrate we can build a CentOS 7 image # and (2) provide a build environment for Charliecloud EPEL 7 RPMs. # Install our dependencies, ensuring we fail out if any are missing. RUN yum install -y epel-release \ && yum install -y --setopt=skip_missing_names_on_install=0 \ autoconf \ automake \ bats \ fakeroot \ gcc \ git \ make \ python3-devel \ python3 \ python36-lark-parser \ python36-requests \ python36-sphinx \ python36-sphinx_rtd_theme \ rpm-build \ rpmlint \ rsync \ squashfs-tools \ squashfuse \ wget \ && yum clean all # We need to install epel rpm-macros after python3-devel to get the correct # python package version for our spec file macros. # https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/K4EH7V3OUFJFVL6A72IILJUA6JFX2HZW/ RUN yum install -y epel-rpm-macros # Need wheel to install bundled Lark, and the RPM version doesn’t work. RUN pip3 install wheel charliecloud-0.37/examples/Dockerfile.debian_11ch000066400000000000000000000003031457016721300217500ustar00rootroot00000000000000# ch-test-scope: standard FROM debian:bullseye ARG DEBIAN_FRONTEND=noninteractive RUN apt-get update \ && apt-get install -y --no-install-recommends apt-utils \ && rm -rf /var/lib/apt/lists/* charliecloud-0.37/examples/Dockerfile.libfabric000066400000000000000000000153341457016721300216410ustar00rootroot00000000000000# ch-test-scope: full FROM almalinux_8ch # A key goal of this Dockerfile is to demonstrate best practices for building # OpenMPI and MPICH for use inside a container. # # This Dockerfile aspires to work close to optimally on clusters with any of the # following interconnects: # # - Ethernet (TCP/IP) # - InfiniBand (IB) # - Omni-Path (OPA) # - RDMA over Converged Ethernet (RoCE) interconnects # - Gemini/Aries (UGNI) ** # - Slingshot (CXI) ** # # with no environment variables, command line arguments, additional # configuration files, and minimal runtime manipulation. # # MPI implementations have numerous ways of communicating messages over # interconnects. We use Libfabric (OFI), an OpenFabric framework that # exports fabric communication services to applications, to manage these # communications with built-in, or loadable, fabric providers. # # - https://ofiwg.github.io/libfabric # - https://ofiwg.github.io/libfabric/v1.14.0/man/fi_provider.3.html # # Using OFI, we can: 1) uniformly manage fabric communications services for both # OpenMPI and MPICH; 2) use host-built OFI shared object providers to use # proprietary host hardware, e.g., Cray Gemini/Aries; and 3) replace the # container’s OFI with that of the hosts to leverage special fabric interfaces, # e.g., Cray’s Slingshot CXI. # # Providers implement the application facing software interfaces needed to # access network specific protocols, drivers, and hardware. The built-in # providers relevant here are: # # Provider included reason Eth IB OPA RoCE Slingshot Gemini/Aries # -------- -------- ------ --- -- --- ---- --------- ------------ # # opx No a N N Y N # psm2 No b N N Y N # psm3 Yes c Y N Y Y X # shm Yes d # tcp Yes Y* X X X X X # verbs Yes N Y N Y # # cxi No f X Y* # ugni No f Y* # # Y : supported # Y*: best choice for that interconnect # X : supported but sub-optimal # : unclear # # a : OPA is covered by psm3. # b : psm3 is preffered over psm2. # c : psm3 provides optimized performance for most verbs and socket devices # Additionally, PSM3.x: 1) fully integrates the OFI provider and # underlying PSM3 protocols/implementation, and 2) exports only OFI APIs. # c : requires cray interconnect and libraries # d : shm enables applications using OFI to be run over shared memory. # f : Requires access to hardware specific libraries at build time; these # providers need to be injected at run-time. See ch-fromhost man page. # # The full list of OFI providers can be seen here: # # - https://github.com/ofiwg/libfabric/blob/main/README.md # # PMI: # # We build OpenPMIx, PMI2, and FLUX-PMI. # OS packages needed to build libfabric providers. # # Note that libpsm2 is x86-64 only so we skip if missing RUN dnf install -y --setopt=install_weak_deps=false \ automake \ brotli \ file \ flex \ gcc \ gcc-c++ \ gcc-gfortran \ git \ hwloc \ hwloc-devel \ hwloc-libs \ hwloc-plugins \ ibacm \ libatomic \ libevent-devel \ libtool \ libibumad \ libibumad-devel \ librdmacm \ librdmacm-devel \ libssh \ rdma-core \ make \ numactl-devel \ wget \ && dnf install -y --setopt=install_weak_deps=false --skip-broken \ libpsm2 \ libpsm2-devel \ && dnf clean all WORKDIR /usr/local/src # Libfabric (OFI) # # PSM3 is our preferred provider for OPA, however, it requires libpsm2, which # is x86_64 only. Thus, the 'PSM_CONFIG' variable is used to avoid building PSM3 # on aarch64 machines. ARG LIBFABRIC_VERSION=1.15.1 RUN git clone --branch v${LIBFABRIC_VERSION} --depth 1 \ https://github.com/ofiwg/libfabric/ \ && cd libfabric \ && ./autogen.sh \ && if [[ $(uname -m) == x86_64 ]]; then PSM_CONFIG=enable; \ else PSM_CONFIG=disable; fi \ && ./configure --prefix=/usr/local \ --disable-opx \ --disable-psm2 \ --disable-efa \ --disable-rxm \ --disable-sockets \ "--${PSM_CONFIG}-psm3" \ --enable-rxm \ --enable-shm \ --enable-tcp \ --enable-verbs \ && make -j$(getconf _NPROCESSORS_ONLN) install \ && rm -Rf ../libfabric* # PMIX. # # There isn’t a package available with the PMIX libraries we need, so # build them. # # Note: PMIX_VERSION is a variable used by OpenMPI at configure time; we use # PMIX_VER to avoid issues. ARG PMIX_VER=3.2.4 RUN git clone https://github.com/openpmix/openpmix.git \ && cd openpmix \ && git checkout v$PMIX_VER \ && ./autogen.pl \ && ./configure --prefix=/usr/local \ --with-libevent \ --with-hwloc \ && make -j$(getconf _NPROCESSORS_ONLN) install \ && rm -Rf ../openpmix* # FLUX-PMI # # Flux requires a number of additional packages. We install them here to # distinquish between libfabric provider and flux-pmi dependencies. ARG FLUX_VERSION=0.45.0 RUN dnf install -y \ czmq \ czmq-devel \ cppzmq-devel \ jansson \ jansson-devel \ libarchive-devel \ libsqlite3x-devel \ lua-devel \ lz4-devel \ ncurses-devel \ python3-cffi \ python3-jsonschema \ python3-yaml \ tree \ && dnf clean all \ && git clone https://github.com/flux-framework/flux-core \ && cd flux-core \ && git checkout v${FLUX_VERSION} \ && ./autogen.sh \ && ./configure --prefix=/usr/local \ && make -j$(getconf _NPROCESSORS_ONLN) install \ && rm -Rf ../flux-core \ && echo '/usr/local/lib/flux' >> /etc/ld.so.conf.d/usrlocal.conf \ && ldconfig # PMI2 # # We prefer PMIx, it scales better than PMI2. PMI2 will no longer be supported # by OpenMPI starting with version 5. # ARG SLURM_VERSION=21-08-6-1 RUN wget https://github.com/SchedMD/slurm/archive/slurm-${SLURM_VERSION}.tar.gz \ && tar -xf slurm-${SLURM_VERSION}.tar.gz \ && cd slurm-slurm-${SLURM_VERSION} \ && ./configure --prefix=/usr/local \ && cd contribs/pmi2 \ && make -j$(getconf _NPROCESSORS_ONLN) install \ && rm -Rf ../../../slurm* RUN ls -lh /usr/local/lib/flux charliecloud-0.37/examples/Dockerfile.mpich000066400000000000000000000026341457016721300210230ustar00rootroot00000000000000# ch-test-scope: full # See Dockerfile.libfabric for MPI goals and details. FROM libfabric WORKDIR /usr/local/src # Configure MPICH with OpenPMIx. Note we did attempt to configure MPICH # against both PMI2 and PMIx, as we do with OpenMPI, but the examples only # pass testing when pmix is specified. # # Note --with-pm=no disables the hydra and gforker process manager; this # allows us to launch parallel jobs with slurm using PMIx or PMI2. As a # consequence, the mpiexec exectuable is no longer compiled or installed; # thus, single-node guest launch using mpiexec inside container is not # poassible. # # Slingshot CXI requires MPICH version 4.1 or greater. ARG MPI_VERSION=4.1.1 ARG MPI_URL=http://www.mpich.org/static/downloads/${MPI_VERSION} RUN wget -nv ${MPI_URL}/mpich-${MPI_VERSION}.tar.gz \ && tar xf mpich-${MPI_VERSION}.tar.gz \ && cd mpich-${MPI_VERSION} \ && CFLAGS=-O3 \ CXXFLAGS=-O3 \ ./configure --prefix=/usr/local \ --enable-fast=O3 \ --enable-g=none \ --enable-ofi-domain \ --enable-threads=multiple \ --with-ch4-shmmods=posix \ --with-device=ch4:ofi \ --with-libfabric=/usr/local \ --with-pm=no \ --with-pmix=/usr/local/lib RUN cd mpich-${MPI_VERSION} \ && make -j$(getconf _NPROCESSORS_ONLN) install \ && rm -Rf ../mpich-${MPI_VERSION}* \ && ldconfig charliecloud-0.37/examples/Dockerfile.nvidia000066400000000000000000000043741457016721300212000ustar00rootroot00000000000000# ch-test-scope: full # ch-test-arch-exclude: aarch64 # only x86-64, ppc64le supported by nVidia # This Dockerfile demonstrates a multi-stage build. With a single-stage build # that brings along the nVidia build environment, the resulting unpacked image # is 2.9 GiB; with the multi-stage build, it’s 146 MiB. # # See: https://docs.docker.com/develop/develop-images/multistage-build ## Stage 1: Install the nVidia build environment and build a sample app. FROM ubuntu:20.04 # OS packages needed ARG DEBIAN_FRONTEND=noninteractive RUN apt-get update \ && apt-get install -y --no-install-recommends \ ca-certificates \ gnupg \ make \ wget \ && rm -rf /var/lib/apt/lists/* # Install CUDA from nVidia. # See: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=2004&target_type=debnetwork WORKDIR /usr/local/src ARG nvidia_pub=3bf863cc.pub RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin \ && mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 \ && wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/$nvidia_pub \ && apt-key add $nvidia_pub \ && echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" >> /etc/apt/sources.list \ && apt-get update \ && apt-get install -y --no-install-recommends cuda-toolkit-11-2 \ && rm -rf /var/lib/apt/lists/* $nvidia_pub # Build the sample app we’ll use to test. WORKDIR /usr/local/cuda-11.2/samples/0_Simple/matrixMulCUBLAS RUN make ## Stage 2: Copy the built sample app into a clean Ubuntu image. FROM ubuntu:20.04 COPY --from=0 /usr/local/cuda-11.2/samples/0_Simple/matrixMulCUBLAS / # These are the two nVidia shared libraries that the sample app needs. We could # be smarter about finding this path. However, one thing to avoid is copying in # all of /usr/local/cuda-11.2/targets/x86_64-linux/lib, because that directory # is quite large. COPY --from=0 /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcublas.so.11.4.1.1043 /usr/local/lib COPY --from=0 /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcublasLt.so.11.4.1.1043 /usr/local/lib RUN ldconfig charliecloud-0.37/examples/Dockerfile.openmpi000066400000000000000000000032531457016721300213700ustar00rootroot00000000000000# ch-test-scope: full FROM libfabric # See Dockerfile.libfabric for MPI goals and details. # OpenMPI. # # Build with PMIx, PMI2, and FLUX-PMI support. # # 1. --disable-pty-support to avoid “pipe function call failed when # setting up I/O forwarding subsystem”. # # 2. --enable-mca-no-build=plm-slurm to support launching processes using the # host’s srun (i.e., the container OpenMPI needs to talk to the host Slurm’s # PMIx) but prevent OpenMPI from invoking srun itself from within the # container, where srun is not installed (the error messages from this are # inscrutable). ARG MPI_URL=https://www.open-mpi.org/software/ompi/v4.1/downloads ARG MPI_VERSION=4.1.4 RUN wget -nv ${MPI_URL}/openmpi-${MPI_VERSION}.tar.gz \ && tar xf openmpi-${MPI_VERSION}.tar.gz RUN cd openmpi-${MPI_VERSION} \ && CFLAGS=-O3 \ CXXFLAGS=-O3 \ FLUX_PMI_CFLAGS=-I/usr/local/include/flux/core,-L/usr/local/lib/flux \ FLUX_PMI_LIBS=-lpmi \ ./configure --prefix=/usr/local \ --sysconfdir=/mnt/0 \ --with-pmix=/usr/local \ --with-pmi=/usr/local \ --with-flux-pmi-library \ --with-libfabric=/usr/local \ --disable-pty-support \ --enable-mca-no-build=btl-openib,plm-slurm \ && make -j$(getconf _NPROCESSORS_ONLN) install \ && rm -Rf ../openmpi-${MPI_VERSION}* RUN ldconfig # OpenMPI expects this program to exist, even if it’s not used. Default is # “ssh : rsh”, but that’s not installed. RUN echo 'plm_rsh_agent = false' >> /mnt/0/openmpi-mca-params.conf # Silence spurious pmix error. https://github.com/open-mpi/ompi/issues/7516. ENV PMIX_MCA_gds=hash charliecloud-0.37/examples/Makefile.am000066400000000000000000000110151457016721300177570ustar00rootroot00000000000000examplesdir = $(docdir)/examples execs = \ chtest/Build \ chtest/bind_priv.py \ chtest/dev_proc_sys.py \ chtest/fs_perms.py \ chtest/printns \ chtest/signal_out.py \ distroless/hello.py \ hello/hello.sh \ obspy/hello.py noexecs = \ Dockerfile.centos_7ch \ Dockerfile.almalinux_8ch \ Dockerfile.debian_11ch \ Dockerfile.libfabric \ Dockerfile.mpich \ Dockerfile.nvidia \ Dockerfile.openmpi \ chtest/Makefile \ chtest/chroot-escape.c \ chtest/mknods.c \ chtest/setgroups.c \ chtest/setuid.c \ copy/Dockerfile \ copy/dirA/fileAa \ copy/dirB/fileBa \ copy/dirB/fileBb \ copy/dirCa/dirCb/fileCba \ copy/dirCa/dirCb/fileCbb \ copy/dirD/fileDa \ copy/dirEa/dirEb/fileEba \ copy/dirEa/dirEb/fileEbb \ copy/dirF/dir19a3/file19b1 \ copy/dirF/file19a3 \ copy/dirF/file19a2 \ copy/dirF/dir19a2/file19b2 \ copy/dirF/dir19a2/dir19b2/file19c1 \ copy/dirF/dir19a2/dir19b3/file19c1 \ copy/dirF/dir19a2/file19b3 \ copy/dirG/diry/file_ \ copy/dirG/filey \ copy/dirG/s_dir1 \ copy/dirG/s_dir4/file_ \ copy/dirG/s_file1 \ copy/dirG/s_file4/file_ \ copy/fileA \ copy/fileB \ copy/test.bats \ distroless/Dockerfile \ exhaustive/Dockerfile \ hello/Dockerfile \ hello/README \ lammps/Dockerfile \ lammps/melt.patch \ lammps/simple.patch \ lustre/Dockerfile \ mpibench/Dockerfile.mpich \ mpibench/Dockerfile.openmpi \ mpihello/Dockerfile.mpich \ mpihello/Dockerfile.openmpi \ mpihello/Makefile \ mpihello/hello.c \ mpihello/slurm.sh \ multistage/Dockerfile \ obspy/Dockerfile \ obspy/README \ obspy/obspy.png \ paraview/Dockerfile \ paraview/cone.2ranks.vtk \ paraview/cone.nranks.vtk \ paraview/cone.png \ paraview/cone.py \ paraview/cone.serial.vtk \ seccomp/Dockerfile \ seccomp/mknods.c \ seccomp/test.bats \ spack/Dockerfile \ spark/Dockerfile \ spark/slurm.sh batsfiles = \ distroless/test.bats \ exhaustive/test.bats \ hello/test.bats \ lammps/test.bats \ lustre/test.bats \ mpibench/test.bats \ mpihello/test.bats \ multistage/test.bats \ obspy/test.bats \ paraview/test.bats \ spack/test.bats \ spark/test.bats nobase_examples_SCRIPTS = $(execs) nobase_examples_DATA = $(noexecs) if ENABLE_TEST nobase_examples_DATA += $(batsfiles) endif EXTRA_DIST = $(execs) $(noexecs) $(batsfiles) # Automake is completely unable to deal with symlinks; we cannot include them # in the source code or "make dist" won't work, and we can't include them in # the files to install or "make install" won't work. These targets take care # of everything manually. # # Note: -T prevents ln(1) from dereferencing and descending into symlinks to # directories. Without this, new symlinks are created within such directories, # instead of replacing the existing symlink as we wanted. See PR #722. all-local: ln -fTs dirCb copy/dirCa/symlink-to-dirCb ln -fTs fileDa copy/dirD/symlink-to-fileDa ln -fTs dirEb copy/dirEa/symlink-to-dirEb ln -fTs filey copy/dirG/s_dir2 ln -fTs diry copy/dirG/s_dir3 ln -fTs filey copy/dirG/s_file2 ln -fTs diry copy/dirG/s_file3 ln -fTs fileA copy/symlink-to-fileA ln -fTs fileB copy/symlink-to-fileB-A ln -fTs fileB copy/symlink-to-fileB-B clean-local: rm -f copy/dirCa/symlink-to-dirCb rm -f copy/dirD/symlink-to-fileDa rm -f copy/dirEa/symlink-to-dirEb rm -f copy/dirG/s_dir2 rm -f copy/dirG/s_dir3 rm -f copy/dirG/s_file2 rm -f copy/dirG/s_file3 rm -f copy/symlink-to-fileA rm -f copy/symlink-to-fileB-A rm -f copy/symlink-to-fileB-B install-data-hook: ln -fTs dirCb $(DESTDIR)$(examplesdir)/copy/dirCa/symlink-to-dirCb ln -fTs fileDa $(DESTDIR)$(examplesdir)/copy/dirD/symlink-to-fileDa ln -fTs dirEb $(DESTDIR)$(examplesdir)/copy/dirEa/symlink-to-dirEb ln -fTs filey $(DESTDIR)$(examplesdir)/copy/dirG/s_dir2 ln -fTs diry $(DESTDIR)$(examplesdir)/copy/dirG/s_dir3 ln -fTs filey $(DESTDIR)$(examplesdir)/copy/dirG/s_file2 ln -fTs diry $(DESTDIR)$(examplesdir)/copy/dirG/s_file3 ln -fTs fileA $(DESTDIR)$(examplesdir)/copy/symlink-to-fileA ln -fTs fileB $(DESTDIR)$(examplesdir)/copy/symlink-to-fileB-A ln -fTs fileB $(DESTDIR)$(examplesdir)/copy/symlink-to-fileB-B uninstall-local: rm -f $(DESTDIR)$(examplesdir)/copy/dirCa/symlink-to-dirCb rm -f $(DESTDIR)$(examplesdir)/copy/dirD/symlink-to-fileDa rm -f $(DESTDIR)$(examplesdir)/copy/dirEa/symlink-to-dirEb rm -f $(DESTDIR)$(examplesdir)/copy/dirG/s_dir2 rm -f $(DESTDIR)$(examplesdir)/copy/dirG/s_dir3 rm -f $(DESTDIR)$(examplesdir)/copy/dirG/s_file2 rm -f $(DESTDIR)$(examplesdir)/copy/dirG/s_file3 rm -f $(DESTDIR)$(examplesdir)/copy/symlink-to-fileA rm -f $(DESTDIR)$(examplesdir)/copy/symlink-to-fileB-A rm -f $(DESTDIR)$(examplesdir)/copy/symlink-to-fileB-B uninstall-hook: rmdir $$(find $(docdir) -type d | sort -r) charliecloud-0.37/examples/chtest/000077500000000000000000000000001457016721300172175ustar00rootroot00000000000000charliecloud-0.37/examples/chtest/Build000077500000000000000000000116401457016721300202060ustar00rootroot00000000000000#!/bin/bash # Build an Alpine Linux image roughly following the chroot(2) instructions: # https://wiki.alpinelinux.org/wiki/Installing_Alpine_Linux_in_a_chroot # # We deliberately do not sudo. It’s a little rough around the edges, because # apk expects root, but it better follows the principle of least privilege. We # could tidy by using the fakeroot utility, but AFAICT that’s not particularly # common and we’d prefer not to introduce another dependency. For example, # it's a standard tool on Debian but only in EPEL for CentOS. # # FIXME: Despite the guidance in the Build script API docs, this produces a # tarball even though the process does not naturally produce one. This is # because we are also creating some rather bizarre tar edge cases. These # should be moved to a separate script. # # ch-test-scope: quick set -ex srcdir=$1 tarball_uncompressed=${2}.tar tarball=${tarball_uncompressed}.gz workdir=$3 arch=$(uname -m) mirror=http://dl-cdn.alpinelinux.org/alpine/v3.9 # Dynamically select apk-tools-static version. We would prefer to hard-code a # version (and upgrade on our schedule), but we can’t because Alpine does not # keep old package versions. If we try, the build breaks every few months (for # example, see issue #242). apk_tools=$( wget -qO - "${mirror}/main/${arch}" \ | grep -F apk-tools-static \ | sed -E 's/^.*(apk-tools-static-[0-9.r-]+\.apk).*$/\1/') img=${workdir}/img cd "$workdir" # “apk add” wants to install a bunch of files root:root. Thus, if we don’t map # ourselves to root:root, we get thousands of errors about “Failed to set # ownership”. # # For most Build scripts, we’d simply error out with missing prerequisites, # but this is a core image that much of the test suite depends on. ch_run="ch-run -u0 -g0 -w ${img}" ## Bootstrap base Alpine Linux. # Download statically linked apk. wget "${mirror}/main/${arch}/${apk_tools}" # Bootstrap directories. mkdir img mkdir img/{dev,etc,proc,sys,tmp} touch img/etc/{group,hosts,passwd,resolv.conf} # Bootstrap static apk. (cd img && tar xf "../${apk_tools}") mkdir img/etc/apk echo ${mirror}/main > img/etc/apk/repositories # Install the base system and a dynamically linked apk. # # This will give a few errors about chown failures. However, the install does # seem to work, so we ignore the failed exit code. $ch_run -- /sbin/apk.static \ --allow-untrusted --initdb --update-cache \ add alpine-base apk-tools \ || true # Now that we’ve bootstrapped, we don’t need apk.static any more. It wasn’t # installed using apk, so it’s not in the database and can just be rm’ed. rm img/sbin/apk.static.* # Install packages we need for our tests. $ch_run -- /sbin/apk add gcc make musl-dev python3 || true # Validate the install. $ch_run -- /sbin/apk audit --system $ch_run -- /sbin/apk stats # Fix permissions. # # Note that this removes setuid/setgid bits from a few files (and # directories). There is not a race condition, i.e., a window where setuid # executables could become the invoking users, which would be a security hole, # because the setuid/setgid binaries are not group- or world-readable until # after this chmod. chmod -R u+rw,ug-s img ## Install our test stuff. # Fixtures for --bind tests mkdir img/home/directory-in-home touch img/home/file-in-home # Test programs. cp -r "$srcdir" img/test $ch_run --cd /test -- sh -c 'make clean && make' # Fixtures for /dev cleaning. touch img/dev/deleteme mkdir -p img/mnt/dev touch img/mnt/dev/dontdeleteme # Fixture to make sure we raise hidden files in non-tarbombs. touch img/.hiddenfile1 img/..hiddenfile2 img/...hiddenfile3 # Fixtures for bind-mounting ln -s ../bind4 img/mnt/bind4 ln -s ./doesnotexist img/mnt/link-b0rken-rel ln -s /doesnotexist img/mnt/link-b0rken-abs ln -s /tmp img/mnt/link-bad-abs ln -s ../.. img/mnt/link-bad-rel # Fixture to test resolv.conf as symlink (issue #1015). mv img/etc/resolv.conf img/etc/resolv.conf.real ln -s /etc/resolv.conf.real img/etc/resolv.conf # Fixtures to validate permissions are retained on export (issue #1241). See # FAQ for why this isn’t 7777. touch img/maxperms_file chmod 0777 img/maxperms_file mkdir img/maxperms_dir chmod 1777 img/maxperms_dir # Get rid of “/root” directory, used for “HOME” test in “ch-run_misc.bats”. rmdir "$img"/root ## Tar it up. # Using pigz saves about 8 seconds. Normally we wouldn’t care about that, but # this script is part of the quick scope, which we’d like developers to use # frequently, so every second matters. if command -v pigz > /dev/null 2>&1; then gzip_cmd=pigz else gzip_cmd=gzip fi # Charliecloud supports images both with a single top level directory and # without (tarbomb). The Docker images in the test suite are all tarbombs # (because that’s what “docker export” gives us), so use a containing # directory for this one. tar cf "$tarball_uncompressed" -- img # Finalize the tarball. $gzip_cmd -f "$tarball_uncompressed" [[ -f $tarball ]] charliecloud-0.37/examples/chtest/Makefile000066400000000000000000000002331457016721300206550ustar00rootroot00000000000000BINS := chroot-escape mknods setgroups setuid ALL := $(BINS) CFLAGS := -std=c11 -Wall -Werror .PHONY: all all: $(ALL) .PHONY: clean clean: rm -f $(ALL) charliecloud-0.37/examples/chtest/bind_priv.py000077500000000000000000000022461457016721300215540ustar00rootroot00000000000000#!/usr/bin/env python3 # This script tries to bind to a privileged port on each of the IP addresses # specified on the command line. import errno import socket import sys PORT = 7 # echo results = dict() try: for ip in sys.argv[1:]: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) try: s.bind((ip, PORT)) except OSError as x: if (x.errno in (errno.EACCES, errno.EADDRNOTAVAIL)): results[ip] = x.errno else: raise else: results[ip] = 0 except Exception as x: print('ERROR\texception: %s' % x) rc = 1 else: if (len(results) < 1): print('ERROR\tnothing to test', end='') rc = 1 elif (len(set(results.values())) != 1): print('ERROR\tmixed results: ', end='') rc = 1 else: result = next(iter(results.values())) if (result != 0): print('SAFE\t%d (%s) ' % (result, errno.errorcode[result]), end='') rc = 0 else: print('RISK\tsuccessful bind ', end='') rc = 1 explanation = ' '.join('%s=%d' % (ip, e) for (ip, e) in sorted(results.items())) print(explanation) sys.exit(rc) charliecloud-0.37/examples/chtest/chroot-escape.c000066400000000000000000000042131457016721300221170ustar00rootroot00000000000000/* This program tries to escape a chroot using well-established methods, which are not an exploit but rather take advantage of chroot(2)'s well-defined behavior. We use device and inode numbers to test whether the root directory is the same before and after the escape. References: https://filippo.io/escaping-a-chroot-jail-slash-1/ http://www.bpfh.net/simes/computing/chroot-break.html */ #define _DEFAULT_SOURCE #include #include #include #include #include #include #include #include void fatal(char * msg) { printf("ERROR\t%s: %s\n", msg, strerror(errno)); exit(EXIT_FAILURE); } int main() { struct stat before, after; int fd; int status = EXIT_FAILURE; char tmpdir_template[] = "/tmp/chtest.tmp.chroot.XXXXXX"; char * tmpdir_name; if (stat("/", &before)) fatal("stat before"); tmpdir_name = mkdtemp(tmpdir_template); if (tmpdir_name == NULL) fatal("mkdtemp"); if ((fd = open(".", O_RDONLY)) < 0) fatal("open"); if (chroot(tmpdir_name)) { if (errno == EPERM) { printf("SAFE\tchroot(2) failed with EPERM\n"); status = EXIT_SUCCESS; } else { fatal("chroot"); } } else { if (fchdir(fd)) fatal("fchdir"); if (close(fd)) fatal("close"); for (int i = 0; i < 1024; i++) if (chdir("..")) fatal("chdir"); /* If we got this far, we should be able to call chroot(2), so failure is an error. */ if (chroot(".")) fatal("chroot"); /* If root directory is the same before and after the attempted escape, then the escape failed, and we should be happy. */ if (stat("/", &after)) fatal("stat after"); if (before.st_dev == after.st_dev && before.st_ino == after.st_ino) { printf("SAFE\t"); status = EXIT_SUCCESS; } else { printf("RISK\t"); status = EXIT_FAILURE; } printf("dev/inode before %lu/%lu, after %lu/%lu\n", before.st_dev, before.st_ino, after.st_dev, after.st_ino); } if (rmdir(tmpdir_name)) fatal("rmdir"); return status; } charliecloud-0.37/examples/chtest/dev_proc_sys.py000077500000000000000000000026421457016721300222770ustar00rootroot00000000000000#!/usr/bin/env python3 import os.path import sys # Files in /dev and /sys seem to vary between Linux systems. Thus, try a few # candidates and use the first one that exists. What we want is a file with # permissions root:root -rw------- that’s in a directory readable and # executable by unprivileged users, so we know we’re testing permissions on # the file rather than any of its containing directories. This may help for # finding such a file in /sys: # # $ find /sys -type f -a -perm 600 -ls # sys_file = None for f in ("/sys/devices/cpu/rdpmc", "/sys/kernel/mm/page_idle/bitmap", "/sys/module/nf_conntrack_ipv4/parameters/hashsize", "/sys/kernel/slab/request_sock_TCP/red_zone"): if (os.path.exists(f)): sys_file = f break if (sys_file is None): print("ERROR\tno test candidates in /sys exist") sys.exit(1) dev_file = None for f in ("/dev/cpu_dma_latency", "/dev/mem"): if (os.path.exists(f)): dev_file = f break if (dev_file is None): print("ERROR\tno test candidates in /dev exist") sys.exit(1) problem_ct = 0 for f in (dev_file, "/proc/kcore", sys_file): try: open(f, "rb").read(1) print("RISK\t%s: read allowed" % f) problem_ct += 1 except PermissionError: print("SAFE\t%s: read not allowed" % f) except OSError as x: print("ERROR\t%s: exception: %s" % (f, x)) problem_ct += 1 sys.exit(problem_ct != 0) charliecloud-0.37/examples/chtest/fs_perms.py000077500000000000000000000067241457016721300214230ustar00rootroot00000000000000#!/usr/bin/env python3 # This script walks the directories specified in sys.argv[1:] prepared by # make-perms-test.sh and attempts to read, write, and traverse (cd) each of # the entries within. It compares the result to the expectation encoded in the # filename. # # A summary line is printed on stdout. Running chatter describing each # evaluation is printed on stderr. # # Note: This works more or less the same as an older version embodied by # `examples/sandbox.py --filesystem` but is implemented in pure Python without # shell commands. Thus, the whole script must be run as root if you want to # see what root can do. import os.path import random import re import sys EXPECTED_RE = re.compile(r'~(...)$') class Makes_No_Sense(TypeError): pass VERBOSE = False def main(): if (sys.argv[1] == '--verbose'): global VERBOSE VERBOSE = True sys.argv.pop(1) d = sys.argv[1] mismatch_ct = 0 test_ct = 0 for path in sorted(os.listdir(d)): test_ct += 1 mismatch_ct += not test('%s/%s' % (d, path)) if (test_ct <= 0 or test_ct % 2887 != 0): error("unexpected number of tests: %d" % test_ct) if (mismatch_ct == 0): print('SAFE\t', end='') else: print('RISK\t', end='') print('%d mismatches in %d tests' % (mismatch_ct, test_ct)) sys.exit(mismatch_ct != 0) # Table of test function name fragments. testvec = { (False, False, False): ('X', 'bad'), (False, False, True ): ('l', 'broken_symlink'), (False, True, False): ('f', 'file'), (False, True, True ): ('f', 'file'), (True, False, False): ('d', 'dir'), (True, False, True ): ('d', 'dir') } def error(msg): print('ERROR\t%s' % msg) sys.exit(1) def expected(path): m = EXPECTED_RE.search(path) if (m is None): return '*' else: return m[1] def test(path): filetype = (os.path.isdir(path), os.path.isfile(path), os.path.islink(path)) report = '%s %-24s ' % (testvec[filetype][0], path) expect = expected(path) result = '' for op in 'r', 'w', 't': # read, write, traverse f = globals()['try_%s_%s' % (op, testvec[filetype][1])] try: f(path) except (PermissionError, Makes_No_Sense): result += '-' except Exception as x: error('exception on %s: %s' % (path, x)) else: result += op report += result if (expect != '*' and result != expect): print('%s mismatch' % report) return False else: if (VERBOSE): print('%s ok' % report) return True def try_r_bad(path): error('bad file type: %s' % path) try_t_bad = try_r_bad try_w_bad = try_r_bad def try_r_broken_symlink(path): raise Makes_No_Sense() try_t_broken_symlink = try_r_broken_symlink try_w_broken_symlink = try_r_broken_symlink def try_r_dir(path): os.listdir(path) def try_t_dir(path): try_r_file(path + '/file') def try_w_dir(path): fpath = '%s/a%d' % (path, random.getrandbits(64)) try_w_file(fpath) os.unlink(fpath) def try_r_file(path): with open(path, 'rb', buffering=0) as fp: fp.read(1) def try_t_file(path): raise Makes_No_Sense() def try_w_file(path): # The file should exist, but this will create it if it doesn’t. We don't # check for that error condition because we *only* want to touch the OS for # open(2) and write(2). with open(path, 'wb', buffering=0) as fp: fp.write(b'written by fs_test.py\n') if (__name__ == '__main__'): main() charliecloud-0.37/examples/chtest/mknods.c000066400000000000000000000061261457016721300206630ustar00rootroot00000000000000/* Try to make some device files, and print a message to stdout describing what happened. See: https://www.kernel.org/doc/Documentation/devices.txt */ #define _GNU_SOURCE #include #include #include #include #include #include #include #include const unsigned char_devs[] = { 1, 3, /* /dev/null -- most innocuous */ 1, 1, /* /dev/mem -- most juicy */ 0 }; int main(int argc, char ** argv) { dev_t dev; char * dir; int i, j; unsigned maj, min; bool open_ok; char * path; for (i = 1; i < argc; i++) { dir = argv[i]; for (j = 0; char_devs[j] != 0; j += 2) { maj = char_devs[j]; min = char_devs[j + 1]; if (0 > asprintf(&path, "%s/c%d.%d", dir, maj, min)) { printf("ERROR\tasprintf() failed with errno=%d\n", errno); return 1; } fprintf(stderr, "trying to mknod %s: ", path); dev = makedev(maj, min); if (mknod(path, S_IFCHR | 0500, dev)) { // Could not create device; make sure it's an error we expected. switch (errno) { case EACCES: case EINVAL: // e.g. /sys/firmware/efi/efivars case ENOENT: // e.g. /proc case ENOTDIR: // for bind-mounted files e.g. /etc/passwd case EPERM: case EROFS: fprintf(stderr, "failed as expected with errno=%d\n", errno); break; default: fprintf(stderr, "failed with unexpected errno\n"); printf("ERROR\tmknod(2) failed on %s with errno=%d\n", path, errno); return 1; } } else { // Device created; safe if we can't open it (see issue #381). fprintf(stderr, "succeeded\n"); fprintf(stderr, "trying to open %s: ", path); if (open(path, O_RDONLY) != -1) { fprintf(stderr, "succeeded\n"); open_ok = true; } else { open_ok = false; switch (errno) { case EACCES: fprintf(stderr, "failed as expected with errno=%d\n", errno); break; default: fprintf(stderr, "failed with unexpected errno\n"); printf("ERROR\topen(2) failed on %s with errno=%d\n", path, errno); return 1; } } // Remove the device, whether or not we were able to open it. if (unlink(path)) { printf("ERROR\tunlink(2) failed on %s with errno=%d", path, errno); return 1; } if (open_ok) { printf("RISK\tmknod(2), open(2) succeeded on %s (now removed)\n", path); return 1; } } } } printf("SAFE\t%d devices in %d dirs failed\n", (i - 1) * (j / 2), i - 1); return 0; } charliecloud-0.37/examples/chtest/printns000077500000000000000000000012261457016721300206430ustar00rootroot00000000000000#!/usr/bin/env python3 # Print out my namespace IDs, to stdout or (if specified) the path in $2. # Then, if $1 is specified, wait that number of seconds before exiting. import glob import os import socket import sys import time if (len(sys.argv) > 1): pause = float(sys.argv[1]) else: pause = 0 if (len(sys.argv) > 2): out = open(sys.argv[2], "wt") else: out = sys.stdout hostname = socket.gethostname() for ns in glob.glob("/proc/self/ns/*"): stat = os.stat(ns) print("%s:%s:%d" % (ns, hostname, stat.st_ino), file=out, flush=True) out.close() # close the file ASAP to not collide with a later printns if (pause): time.sleep(pause) charliecloud-0.37/examples/chtest/setgroups.c000066400000000000000000000016461457016721300214250ustar00rootroot00000000000000/* Try to drop the last supplemental group, and print a message to stdout describing what happened. */ #define _DEFAULT_SOURCE #include #include #include #include #include #define NGROUPS_MAX 128 int main() { int group_ct; gid_t groups[NGROUPS_MAX]; group_ct = getgroups(NGROUPS_MAX, groups); if (group_ct == -1) { printf("ERROR\tgetgroups(2) failed with errno=%d\n", errno); return 1; } fprintf(stderr, "found %d groups; trying to drop last group %d\n", group_ct, groups[group_ct - 1]); if (setgroups(group_ct - 1, groups)) { if (errno == EPERM) { printf("SAFE\tsetgroups(2) failed with EPERM\n"); return 0; } else { printf("ERROR\tsetgroups(2) failed with errno=%d\n", errno); return 1; } } else { printf("RISK\tsetgroups(2) succeeded\n"); return 1; } } charliecloud-0.37/examples/chtest/setuid.c000066400000000000000000000015441457016721300206640ustar00rootroot00000000000000/* Try to change effective UID. */ #define _GNU_SOURCE #include #include #include #include #define NOBODY 65534 #define NOBODY2 65533 int main(int argc, char ** argv) { // target UID is nobody, unless we're already nobody uid_t start = geteuid(); uid_t target = start != NOBODY ? NOBODY : NOBODY2; int result; fprintf(stderr, "current EUID=%u, attempting EUID=%u\n", start, target); result = seteuid(target); // setuid(2) fails with EINVAL in user namespaces and EPERM if not root. if (result == 0) { printf("RISK\tsetuid(2) succeeded for EUID=%u\n", target); return 1; } else if (errno == EINVAL) { printf("SAFE\tsetuid(2) failed as expected with EINVAL\n"); return 0; } printf("ERROR\tsetuid(2) failed unexpectedly with errno=%d\n", errno); return 1; } charliecloud-0.37/examples/chtest/signal_out.py000077500000000000000000000020671457016721300217450ustar00rootroot00000000000000#!/usr/bin/env python3 # Send a signal to a process outside the container. # # This is a little tricky. We want a process that: # # 1. is certain to exist, to avoid false negatives # 2. we shouldn’t be able to signal (specifically, we can’t create a process # to serve as the target) # 3. is outside the container # 4. won’t crash the host too badly if killed by the signal # # We want a signal that: # # 5. will be harmless if received # 6. is not blocked # # Accordingly, this test sends SIGCONT to the youngest getty process. The # thinking is that the virtual terminals are unlikely to be in use, so losing # one will be straightforward to clean up. import os import signal import subprocess import sys try: pdata = subprocess.check_output(["pgrep", "-nl", "getty"]) except subprocess.CalledProcessError: print("ERROR\tpgrep failed") sys.exit(1) pid = int(pdata.split()[0]) try: os.kill(pid, signal.SIGCONT) except PermissionError as x: print("SAFE\tfailed as expected: %s" % x) sys.exit(0) print("RISK\tsucceeded") sys.exit(1) charliecloud-0.37/examples/copy/000077500000000000000000000000001457016721300166775ustar00rootroot00000000000000charliecloud-0.37/examples/copy/Dockerfile000066400000000000000000000215021457016721300206710ustar00rootroot00000000000000# Exercise the COPY instruction, which has rather strange semantics compared # to what we are used to in cp(1). See FAQ. # # ch-test-scope: standard # ch-test-builder-exclude: buildah # ch-test-builder-exclude: buildah-runc # ch-test-builder-exclude: buildah-setuid FROM alpine:3.17 # Test directory RUN mkdir /test WORKDIR /test ## Source: Regular file(s) # Source: one file # Dest: new file, relative to workdir COPY fileA file1a # Source: one file # Dest: new file, absolute path COPY fileA /test/file1b # Source: one file, absolute path (root is context directory) # Dest: new file COPY /fileB file2 # Source: one file # Dest: existing file RUN echo 'this should be overwritten' > file3 COPY fileA file3 # Source: one file # Dest: symlink to existing file, relative path RUN echo 'this should be overwritten' > file4 \ && ln -s file4 symlink-to-file4 COPY fileA symlink-to-file4 # Source: one file # Dest: symlink to existing file, absolute path RUN echo 'this should be overwritten' > file5 \ && ln -s /test/file5 symlink-to-file5 COPY fileA symlink-to-file5 # Source: one file # Dest: existing directory, no trailing slash # # Note: This behavior is inconsistent with the Dockerfile reference, which # implies that dir1a must be a file because it does not end in slash. RUN mkdir dir01a COPY fileA dir01a # Source: one file # Dest: existing directory, trailing slash RUN mkdir dir01b COPY fileA dir01b/ # Source: one file # Dest: symlink to existing directory, relative, no trailing slash RUN mkdir dir01c \ && ln -s dir01c symlink-to-dir01c COPY fileA symlink-to-dir01c # Source: one file # Dest: symlink to existing directory, absolute, no trailing slash RUN mkdir dir01d \ && ln -s /test/dir01d symlink-to-dir01d COPY fileA symlink-to-dir01d # Source: one file # Dest: symlink to existing directory, relative, trailing slash RUN mkdir dir01e \ && ln -s dir01e symlink-to-dir01e COPY fileA symlink-to-dir01e/ # Source: one file # Dest: symlink to existing directory, absolute, trailing slash RUN mkdir dir01f \ && ln -s /test/dir01f symlink-to-dir01f COPY fileA symlink-to-dir01f/ # Source: one file # Dest: symlink to existing directory, multi-level, relative, no slash RUN mkdir -p dir01g/dir \ && ln -s dir01g symlink-to-dir01g COPY fileA symlink-to-dir01g/dir # Source: one file # Dest: symlink to existing directory, multi-level, absolute, no slash RUN mkdir -p dir01h/dir \ && ln -s /test/dir01h symlink-to-dir01h COPY fileA symlink-to-dir01h/dir # Source: one file # Dest: new directory, one level of creation COPY fileA dir02/ # Source: one file # Dest: new directory, two levels of creation COPY fileA dir03a/dir03b/ # Source: two files, explicit # Dest: existing directory RUN mkdir dir04 COPY fileA fileB dir04/ # Source: two files, explicit # Dest: new directory, one level COPY fileA fileB dir05/ # Source: two files, wildcard # Dest: existing directory RUN mkdir dir06 COPY file* dir06/ ## Source: Director(y|ies) # Source: one directory # Dest: existing directory, no trailing slash # # Note: Again, the reference seems to imply this shouldn’t work. RUN mkdir dir07a COPY dirA dir07a # Source: one directory # Dest: existing directory, trailing slash RUN mkdir dir07b COPY dirA dir07b/ # Source: one directory # Dest: symlink to existing directory, relative, no trailing slash RUN mkdir dir07c \ && ln -s dir07c symlink-to-dir07c COPY dirA symlink-to-dir07c # Source: one directory # Dest: symlink to existing directory, absolute, no trailing slash RUN mkdir dir07d \ && ln -s /test/dir07d symlink-to-dir07d COPY dirA symlink-to-dir07d # Source: one directory # Dest: symlink to existing directory, relative, trailing slash RUN mkdir dir07e \ && ln -s dir07e symlink-to-dir07e COPY dirA symlink-to-dir07e/ # Source: one directory # Dest: symlink to existing directory, absolute, trailing slash RUN mkdir dir07f \ && ln -s /test/dir07f symlink-to-dir07f COPY dirA symlink-to-dir07f/ # Source: one directory # Dest: new directory, one level, no trailing slash # # Note: Again, the reference seems to imply this shouldn’t work. COPY dirA dir08a # Source: one directory # Dest: new directory, one level, trailing slash COPY dirA dir08b/ # NOTE: Not currently tested (see #1707). We keep it around but commented out # to illustrate what we really would like to test. # # # Source: one directory # # Dest: existing file, 2nd level # # # # Note: While this fails if the existing file is at the top level (which we # # verify in test/build/50_dockerfile.bats), if the existing file is at the 2nd # # level, it's overwritten by the directory. # RUN touch dir08a/dirCb # COPY dirCa dir08a # Source: two directories, explicit # Dest: existing directory RUN mkdir dir09 COPY dirA dirB dir09/ # Source: two directories, explicit # Dest: new directory, one level COPY dirA dirB dir10/ # Source: two directories, wildcard # Dest: existing directory RUN mkdir dir11 COPY dir[AB] dir11/ # Source: two directories, wildcard # Dest: new directory, one level COPY dir[AB] dir12/ ## Source: Symlink(s) # Note: Behavior for symlinks is not documented. See FAQ. # Source: one symbolic link, to file, named explicitly # Dest: existing directory COPY symlink-to-fileA ./ # Source: one symbolic link, to directory, named explicitly # Dest: existing directory RUN mkdir dir13 COPY dirCa/symlink-to-dirCb dir13/ # Source: one symbolic link, to file, in a directory # Dest: existing directory RUN mkdir dir14 COPY dirD dir14/ # Source: one symbolic link, to file, in a directory # Dest: new directory, one level COPY dirD dir15/ # Source: one symbolic link, to directory, in a directory # Dest: existing directory RUN mkdir dir16 COPY dirEa dir16/ # Source: two symbolic links, to files, named explicitly # Dest: existing directory RUN mkdir dir17 COPY fileB symlink-to-fileB-A symlink-to-fileB-B dir17/ # Source: two symbolic links, to files, wildcard # Dest: existing directory RUN mkdir dir18 COPY fileB symlink-to-fileB-* dir18/ ## Merge directory trees # Set up destination directory tree. RUN mkdir dir19 \ && mkdir dir19/dir19a1 \ && mkdir dir19/dir19a2 \ && mkdir dir19/dir19a2/dir19b1 \ && mkdir dir19/dir19a2/dir19b2 \ && echo old > dir19/file19a1 \ && echo old > dir19/file19a2 \ && echo old > dir19/dir19a1/file19b1 \ && echo old > dir19/dir19a2/file19b1 \ && echo old > dir19/dir19a2/file19b2 \ && echo old > dir19/dir19a2/dir19b2/file19c1 \ && chmod 777 dir19/dir19a2 # Copy in the new directory tree. This is supposed to merge the two trees. # Important considerations, from perspective of destination tree: # # 1. File at top level, new. # 2. File at top level, existing (should overwrite). # 3. File at 2nd level, new. # 4. File at 2nd level, existing (should overwrite). # 5. Directory at top level, new. # 6. Directory at top level, existing (permissions should overwrite). # 7. Directory at 2nd level, new. # 8. Directory at 2nd level, existing (permissions should overwrite). # # The directories should be o-rwx so we can see if the permissions were from # the old or new version. RUN test $(stat -c '%A' dir19/dir19a2 | cut -c8-) = 'rwx' \ && stat -c '%n: %A' dir19/dir19a2 COPY dirF dir19/ RUN test $(stat -c '%A' dir19/dir19a2 | cut -c8-) != 'rwx' \ && stat -c '%n: %A' dir19/dir19a2 ## Destination: Symlink, 2nd level. # NOTE: Not currently tested (see #1707). # # # Note: This behavior is DIFFERENT from the symlink at 1st level tests above # # (recall we are trying to be bug-compatible with Docker). # # # Set up destination. # RUN mkdir dir20 \ # && echo new > dir20/filex \ # && mkdir dir20/dirx \ # && for i in $(seq 4); do \ # echo file$i > dir20/file$i \ # && ln -s file$i dir20/s_file$i \ # && mkdir dir20/dir$i \ # && echo dir$i/file_ > dir20/dir$i/file_ \ # && ln -s dir$i dir20/s_dir$i; \ # done \ # && ls -lR dir20 # # # Copy in the new directory tree. In all of these cases, the source simply # # overwrites the destination; symlinks are not followed. # # # # name source destination # # ------- ------------ ------------ # # 1. s_file1 file link to file # # 2. s_dir1 file link to dir # # 3. s_file2 link to file link to file # # 4. s_dir2 link to file link to dir # # 5. s_file3 link to dir link to file # # 6. s_dir3 link to dir link to dir # # 7. s_file4 directory link to file # # 8. s_dir4 directory link to dir # # # COPY dirG dir20/ ## Wrap up; this output helps to build the expectations in test.bats. # Need GNU find, not BusyBox find RUN apk add --no-cache findutils # File tree with type annotation characters. RUN ls -1FR . # Regular file contents. RUN find . -type f -printf '%y: %p: ' -a -exec cat {} \; | sort # Symlink targets. RUN find . -type l -printf '%y: %p -> %l\n' | sort charliecloud-0.37/examples/copy/dirA/000077500000000000000000000000001457016721300175565ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirA/fileAa000066400000000000000000000000141457016721300206550ustar00rootroot00000000000000dirA/fileAa charliecloud-0.37/examples/copy/dirB/000077500000000000000000000000001457016721300175575ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirB/fileBa000066400000000000000000000000141457016721300206570ustar00rootroot00000000000000dirB/fileBa charliecloud-0.37/examples/copy/dirB/fileBb000066400000000000000000000000141457016721300206600ustar00rootroot00000000000000dirB/fileBb charliecloud-0.37/examples/copy/dirCa/000077500000000000000000000000001457016721300177215ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirCa/dirCb/000077500000000000000000000000001457016721300207445ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirCa/dirCb/fileCba000066400000000000000000000000241457016721300222100ustar00rootroot00000000000000dirCa/dirCb/fileCba charliecloud-0.37/examples/copy/dirCa/dirCb/fileCbb000066400000000000000000000000241457016721300222110ustar00rootroot00000000000000dirCa/dirCb/fileCbb charliecloud-0.37/examples/copy/dirD/000077500000000000000000000000001457016721300175615ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirD/fileDa000066400000000000000000000000141457016721300206630ustar00rootroot00000000000000dirD/fileDa charliecloud-0.37/examples/copy/dirEa/000077500000000000000000000000001457016721300177235ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirEa/dirEb/000077500000000000000000000000001457016721300207505ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirEa/dirEb/fileEba000066400000000000000000000000241457016721300222160ustar00rootroot00000000000000dirEa/dirEb/fileEba charliecloud-0.37/examples/copy/dirEa/dirEb/fileEbb000066400000000000000000000000241457016721300222170ustar00rootroot00000000000000dirEa/dirEb/fileEbb charliecloud-0.37/examples/copy/dirF/000077500000000000000000000000001457016721300175635ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirF/dir19a2/000077500000000000000000000000001457016721300207365ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirF/dir19a2/dir19b2/000077500000000000000000000000001457016721300221125ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirF/dir19a2/dir19b2/file19c1000066400000000000000000000000041457016721300233440ustar00rootroot00000000000000new charliecloud-0.37/examples/copy/dirF/dir19a2/dir19b3/000077500000000000000000000000001457016721300221135ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirF/dir19a2/dir19b3/file19c1000066400000000000000000000000041457016721300233450ustar00rootroot00000000000000new charliecloud-0.37/examples/copy/dirF/dir19a2/file19b2000066400000000000000000000000041457016721300221700ustar00rootroot00000000000000new charliecloud-0.37/examples/copy/dirF/dir19a2/file19b3000066400000000000000000000000041457016721300221710ustar00rootroot00000000000000new charliecloud-0.37/examples/copy/dirF/dir19a3/000077500000000000000000000000001457016721300207375ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirF/dir19a3/file19b1000066400000000000000000000000041457016721300221700ustar00rootroot00000000000000new charliecloud-0.37/examples/copy/dirF/file19a2000066400000000000000000000000041457016721300210140ustar00rootroot00000000000000new charliecloud-0.37/examples/copy/dirF/file19a3000066400000000000000000000000041457016721300210150ustar00rootroot00000000000000new charliecloud-0.37/examples/copy/dirG/000077500000000000000000000000001457016721300175645ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirG/diry/000077500000000000000000000000001457016721300205335ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirG/diry/file_000066400000000000000000000000131457016721300215260ustar00rootroot00000000000000diry/file_ charliecloud-0.37/examples/copy/dirG/filey000066400000000000000000000000041457016721300206110ustar00rootroot00000000000000new charliecloud-0.37/examples/copy/dirG/s_dir1000066400000000000000000000000041457016721300206620ustar00rootroot00000000000000new charliecloud-0.37/examples/copy/dirG/s_dir4/000077500000000000000000000000001457016721300207505ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirG/s_dir4/file_000066400000000000000000000000151457016721300217450ustar00rootroot00000000000000s_dir4/file_ charliecloud-0.37/examples/copy/dirG/s_file1000066400000000000000000000000041457016721300210230ustar00rootroot00000000000000new charliecloud-0.37/examples/copy/dirG/s_file4/000077500000000000000000000000001457016721300211115ustar00rootroot00000000000000charliecloud-0.37/examples/copy/dirG/s_file4/file_000066400000000000000000000000161457016721300221070ustar00rootroot00000000000000s_file4/file_ charliecloud-0.37/examples/copy/fileA000066400000000000000000000000061457016721300176360ustar00rootroot00000000000000fileA charliecloud-0.37/examples/copy/fileB000066400000000000000000000000061457016721300176370ustar00rootroot00000000000000fileB charliecloud-0.37/examples/copy/test.bats000066400000000000000000000113751457016721300205400ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" @test "${ch_tag}/ls" { scope standard prerequisites_ok copy # “ls -F” trailing symbol list: https://unix.stackexchange.com/a/82358 diff -u - <(ch-run --cd /test "$ch_img" -- ls -1FR .) < %l\n' | sort) < fileDa l: ./dir15/symlink-to-fileDa -> fileDa l: ./dir16/symlink-to-dirEb -> dirEb l: ./symlink-to-dir01c -> dir01c l: ./symlink-to-dir01d -> /test/dir01d l: ./symlink-to-dir01e -> dir01e l: ./symlink-to-dir01f -> /test/dir01f l: ./symlink-to-dir01g -> dir01g l: ./symlink-to-dir01h -> /test/dir01h l: ./symlink-to-dir07c -> dir07c l: ./symlink-to-dir07d -> /test/dir07d l: ./symlink-to-dir07e -> dir07e l: ./symlink-to-dir07f -> /test/dir07f l: ./symlink-to-file4 -> file4 l: ./symlink-to-file5 -> /test/file5 EOF } charliecloud-0.37/examples/distroless/000077500000000000000000000000001457016721300201205ustar00rootroot00000000000000charliecloud-0.37/examples/distroless/Dockerfile000066400000000000000000000006021457016721300221100ustar00rootroot00000000000000# Skip this test because of issues with gcr.io (see #896). # ch-test-scope: skip # ch-test-arch-exclude: ppc64le # base image unavailable # Distroless is a Google project providing slim images that contain runtime # dependencies only. https://github.com/GoogleContainerTools/distroless # The python3 image was chosen for ease of testing. FROM gcr.io/distroless/python3 COPY hello.py / charliecloud-0.37/examples/distroless/hello.py000077500000000000000000000000521457016721300215750ustar00rootroot00000000000000#!/usr/bin/python3 print("Hello, World!") charliecloud-0.37/examples/distroless/test.bats000066400000000000000000000004151457016721300217520ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" setup () { scope standard prerequisites_ok distroless } @test "${ch_tag}/hello" { run ch-run "$ch_img" -- /hello.py echo "$output" [[ $status -eq 0 ]] [[ $output = 'Hello, World!' ]] } charliecloud-0.37/examples/exhaustive/000077500000000000000000000000001457016721300201125ustar00rootroot00000000000000charliecloud-0.37/examples/exhaustive/Dockerfile000066400000000000000000000027621457016721300221130ustar00rootroot00000000000000# This Dockerfile aims to have at least one of everything, to exercise the # comprehensiveness of Dockerfile feature support. # # FIXME: That focus is a bit out of date. I think really what is here is the # ways we want to exercise ch-image in ways we care about the resulting image. # Exercises where we don’t care are in test/build/50_dockerfile.bats. But, I # don't want to do the refactoring right now. # # See: https://docs.docker.com/engine/reference/builder # # ch-test-scope: full # ch-test-builder-include: ch-image # Use a moderately complex image reference. FROM registry-1.docker.io:443/library/alpine:3.17 AS stage1 RUN pwd WORKDIR /usr/local/src RUN pwd RUN ls --color=no -lh RUN apk add --no-cache bc RUN ["echo", "hello \n${chse_2} \${chse_2} ${NOTSET}"] # should print: # a -${chse_2}- b -value2- c -c- d -d- RUN echo 'a -${chse_2}-' "b -${chse_2}-" "c -${NOTSET:-c}-" "d -${chse_2:+d}-" RUN env # WORKDIR. See test/build/50_ch-image.bats where we validate this all worked OK. # FIXME: test with variable # # filesystem root WORKDIR / RUN mkdir workdir # absolute path, no mkdir WORKDIR /workdir RUN touch file # absolute path, mkdir RUN mkdir /workdir/abs2 WORKDIR /workdir/abs2 RUN touch file # relative path, no mkdir WORKDIR rel1 RUN touch file1 # relative path, 2nd level, no mkdir WORKDIR rel2 RUN touch file # relative path, parent dir, no mkdir WORKDIR .. RUN touch file2 # results RUN ls -R /workdir # TODO: # comment with trailing backslash (line continuation does not work in comments) charliecloud-0.37/examples/exhaustive/test.bats000066400000000000000000000007041457016721300217450ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" setup () { scope standard prerequisites_ok exhaustive } @test "${ch_tag}/WORKDIR" { output_expected=$(cat <<'EOF' /workdir: abs2 file /workdir/abs2: file rel1 /workdir/abs2/rel1: file1 file2 rel2 /workdir/abs2/rel1/rel2: file EOF ) run ch-run "$ch_img" -- ls -R /workdir echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$output_expected") <(echo "$output") } charliecloud-0.37/examples/hello/000077500000000000000000000000001457016721300170305ustar00rootroot00000000000000charliecloud-0.37/examples/hello/Dockerfile000066400000000000000000000002221457016721300210160ustar00rootroot00000000000000# ch-test-scope: standard FROM almalinux:8 RUN dnf install -y --setopt=install_weak_deps=false openssh-clients \ && dnf clean all COPY . hello charliecloud-0.37/examples/hello/README000066400000000000000000000001751457016721300177130ustar00rootroot00000000000000This example is a hello world Charliecloud container. It demonstrates running a command on the host from inside a container. charliecloud-0.37/examples/hello/hello.sh000077500000000000000000000000461457016721300204720ustar00rootroot00000000000000#!/bin/sh set -e echo 'hello world' charliecloud-0.37/examples/hello/test.bats000066400000000000000000000013461457016721300206660ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" setup () { scope standard prerequisites_ok hello pmix_or_skip LC_ALL=C # no other locales installed in container } @test "${ch_tag}/hello" { run ch-run "$ch_img" -- /hello/hello.sh echo "$output" [[ $status -eq 0 ]] [[ $output = 'hello world' ]] } @test "${ch_tag}/distribution sanity" { # Try various simple things that should work in a basic Debian # distribution. (This does not test anything Charliecloud manipulates.) ch-run "$ch_img" -- /bin/bash -c true ch-run "$ch_img" -- /bin/true ch-run "$ch_img" -- find /etc -name 'a*' ch-run "$ch_img" -- sh -c 'echo foo | /bin/grep -E foo' ch-run "$ch_img" -- nice true } charliecloud-0.37/examples/lammps/000077500000000000000000000000001457016721300172165ustar00rootroot00000000000000charliecloud-0.37/examples/lammps/Dockerfile000066400000000000000000000034111457016721300212070ustar00rootroot00000000000000# ch-test-scope: full FROM openmpi WORKDIR /usr/local/src # Packages for building. RUN dnf install -y --setopt=install_weak_deps=false \ cmake \ patch \ python3-devel \ python3-pip \ python3-setuptools \ && dnf clean all # Building mpi4py from source to ensure it is built against our MPI build # Building numpy from source to work around issues seen on Aarch64 systems RUN pip3 install --no-binary :all: cython==0.29.24 mpi4py==3.1.1 numpy==1.19.5 #RUN ln -s /usr/bin/python3 /usr/bin/python # Build LAMMPS. ARG LAMMPS_VERSION=29Sep2021 RUN wget -nv https://github.com/lammps/lammps/archive/patch_${LAMMPS_VERSION}.tar.gz \ && tar xf patch_$LAMMPS_VERSION.tar.gz \ && mkdir lammps-${LAMMPS_VERSION}.build \ && cd lammps-${LAMMPS_VERSION}.build \ && cmake -DCMAKE_INSTALL_PREFIX=/usr/local \ -DCMAKE_BUILD_TYPE=Release \ -DBUILD_MPI=yes \ -DBUILD_LIB=on \ -DBUILD_SHARED_LIBS=on \ -DPKG_DIPOLE=yes \ -DPKG_KSPACE=yes \ -DPKG_POEMS=yes \ -DPKG_PYTHON=yes \ -DPKG_USER-REAXC=yes \ -DPKG_USER-MEAMC=yes \ -DLAMMPS_MACHINE=mpi \ ../lammps-patch_${LAMMPS_VERSION}/cmake \ && make -j $(getconf _NPROCESSORS_ONLN) install \ && ln -s /usr/local/src/lammps-patch_${LAMMPS_VERSION}/ /lammps \ && rm -f ../patch_$LAMMPS_VERSION.tar.gz RUN ldconfig # Patch in.melt to increase problem dimensions. COPY melt.patch /lammps/examples/melt RUN patch -p1 -d / < /lammps/examples/melt/melt.patch # Patch simple.py to uncomment mpi4py calls and disable file output. # Patch in.simple to increase problem dimensions. COPY simple.patch /lammps/python/examples RUN patch -p1 -d / < /lammps/python/examples/simple.patch charliecloud-0.37/examples/lammps/melt.patch000066400000000000000000000004741457016721300212050ustar00rootroot00000000000000--- a/lammps/examples/melt/in.melt 2014-01-07 14:43:31.000000000 -0700 +++ b/lammps/examples/melt/in.melt 2018-03-16 14:37:02.000000000 -0600 @@ -6,3 +6,3 @@ lattice fcc 0.8442 -region box block 0 10 0 10 0 10 +region box block 0 120 0 120 0 120 create_box 1 box @@ -32,2 +32,2 @@ thermo 50 -run 250 +run 3 charliecloud-0.37/examples/lammps/simple.patch000066400000000000000000000040141457016721300215270ustar00rootroot00000000000000--- /lammps/python/examples/simple.py 2019-09-20 09:51:15.000000000 -0600 +++ /lammps/python/examples/simple.py 2019-09-23 16:58:28.950720810 -0600 @@ -1,4 +1,4 @@ -#!/usr/bin/env python -i +#!/usr/bin/python3 # preceding line should have path for Python on your machine # simple.py @@ -28,12 +28,12 @@ me = 0 # uncomment this if running in parallel via mpi4py -#from mpi4py import MPI -#me = MPI.COMM_WORLD.Get_rank() -#nprocs = MPI.COMM_WORLD.Get_size() +from mpi4py import MPI +me = MPI.COMM_WORLD.Get_rank() +nprocs = MPI.COMM_WORLD.Get_size() from lammps import lammps -lmp = lammps() +lmp = lammps("mpi") # run infile one line at a time @@ -85,7 +85,7 @@ # test of new gather/scatter and box extract/reset methods # can try this in parallel and with/without atom_modify sort enabled -lmp.command("write_dump all custom tmp.simple id type x y z fx fy fz"); +#lmp.command("write_dump all custom tmp.simple id type x y z fx fy fz"); x = lmp.gather_atoms("x",1,3) f = lmp.gather_atoms("f",1,3) @@ -123,10 +123,10 @@ boxlo,boxhi,xy,yz,xz,periodicity,box_change = lmp.extract_box() if me == 0: print("Box info",boxlo,boxhi,xy,yz,xz,periodicity,box_change) -lmp.reset_box([0,0,0],[10,10,8],0,0,0) +#lmp.reset_box([0,0,0],[10,10,8],0,0,0) -boxlo,boxhi,xy,yz,xz,periodicity,box_change = lmp.extract_box() -if me == 0: print("Box info",boxlo,boxhi,xy,yz,xz,periodicity,box_change) +#boxlo,boxhi,xy,yz,xz,periodicity,box_change = lmp.extract_box() +#if me == 0: print("Box info",boxlo,boxhi,xy,yz,xz,periodicity,box_change) # uncomment if running in parallel via mpi4py -#print("Proc %d out of %d procs has" % (me,nprocs), lmp) +print("Proc %d out of %d procs has" % (me,nprocs), lmp) --- /lammps/python/examples/in.simple 2019-10-02 16:09:55.198770328 -0600 +++ /lammps/python/examples/in.simple 2019-10-02 16:10:21.263332834 -0600 @@ -5,7 +5,7 @@ atom_style atomic atom_modify map array lattice fcc 0.8442 -region box block 0 4 0 4 0 4 +region box block 0 120 0 120 0 120 create_box 1 box create_atoms 1 box mass 1 1.0 charliecloud-0.37/examples/lammps/test.bats000066400000000000000000000065621457016721300210610ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" # LAMMPS does have a test suite, but we do not use it, because it seems too # fiddly to get it running properly. # # 1. Running the command listed in LAMMPS’ Jenkins tests [2] fails with a # strange error: # # $ python run_tests.py tests/test_commands.py tests/test_examples.py # Loading tests from tests/test_commands.py... # Traceback (most recent call last): # File "run_tests.py", line 81, in # tests += load_tests(f) # File "run_tests.py", line 22, in load_tests # for testname in list(tc): # TypeError: 'Test' object is not iterable # # Looking in run_tests.py, this sure looks like a bug (it’s expecting a # list of Tests, I think, but getting a single Test). But it works in # Jenkins. Who knows. # # 2. The files test/test_*.py say that the tests can be run with # “nosetests”, which they can, after setting several environment # variables. But some of the tests fail for me. I didn’t diagnose. # # Instead, we simply run some of the example problems in a loop and see if # they exit with return code zero. We don’t check output. # # Note that a lot of the other examples crash. I haven’t diagnosed or figured # out if we care. # # We are open to patches if anyone knows how to fix this situation reliably. # # [1]: https://github.com/lammps/lammps-testing # [2]: https://ci.lammps.org/job/lammps/job/master/job/testing/lastSuccessfulBuild/console setup () { scope full prerequisites_ok "$ch_tag" multiprocess_ok pmix_or_skip [[ -n "$ch_cray" ]] && export FI_PROVIDER=$cray_prov } lammps_try () { # These examples cd because some (not all) of the LAMMPS tests expect to # find things based on $CWD. infiles=$(ch-run --cd "/lammps/examples/${1}" "$ch_img" -- \ bash -c "ls in.*") for i in $infiles; do printf '\n\n%s\n' "$i" # shellcheck disable=SC2086 $ch_mpirun_core ch-run --join --cd /lammps/examples/$1 "$ch_img" -- \ lmp_mpi -log none -in "$i" done } @test "${ch_tag}/inject host cray mpi ($cray_prov)" { cray_ofi_or_skip "$ch_img" run ch-run "$ch_img" -- fi_info echo "$output" [[ $output == *"provider: $cray_prov"* ]] [[ $output == *"fabric: $cray_prov"* ]] [[ $status -eq 0 ]] } @test "${ch_tag}/using all cores" { # shellcheck disable=SC2086 run $ch_mpirun_core ch-run --join "$ch_img" -- \ lmp_mpi -log none -in /lammps/examples/melt/in.melt echo "$output" [[ $status -eq 0 ]] ranks_found=$( echo "$output" \ | grep -F 'MPI tasks' \ | tail -1 \ | sed -r 's/^.+with ([0-9]+) MPI tasks.+$/\1/') echo "ranks expected: ${ch_cores_total}" echo "ranks found: ${ranks_found}" [[ $ranks_found -eq "$ch_cores_total" ]] } @test "${ch_tag}/crack" { lammps_try crack; } @test "${ch_tag}/dipole" { lammps_try dipole; } @test "${ch_tag}/flow" { lammps_try flow; } @test "${ch_tag}/friction" { lammps_try friction; } @test "${ch_tag}/melt" { lammps_try melt; } @test "${ch_tag}/mpi4py simple" { $ch_mpirun_core ch-run --join --cd /lammps/python/examples "$ch_img" -- \ ./simple.py in.simple } @test "${ch_tag}/revert image" { unpack_img_all_nodes "$ch_cray" } charliecloud-0.37/examples/lustre/000077500000000000000000000000001457016721300172435ustar00rootroot00000000000000charliecloud-0.37/examples/lustre/Dockerfile000066400000000000000000000016711457016721300212420ustar00rootroot00000000000000# ch-test-scope: full # ch-test-arch-exclude: aarch64 # No lustre RPMS for aarch64 FROM almalinux:8 # Install lustre-client dependencies RUN dnf install -y --setopt=install_weak_deps=false \ e2fsprogs-libs \ wget \ perl \ && dnf clean all ARG LUSTRE_VERSION=2.12.6 ARG LUSTRE_URL=https://downloads.whamcloud.com/public/lustre/lustre-${LUSTRE_VERSION}/el8/client/RPMS/x86_64/ # The lustre-client rpm has a dependency on the kmod-lustre-client rpm, this is # not required for our tests and frequently is incompatible with the kernel # headers in the container, using the --nodeps flag to work around this. # NOTE: The --nodeps flag ignores all dependencies not just kmod-lustre-client, # this could surpress a legitimate failure at build time and lead to odd # behavior at runtime. RUN wget ${LUSTRE_URL}/lustre-client-${LUSTRE_VERSION}-1.el8.x86_64.rpm \ && rpm -i --nodeps *.rpm \ && rm -f *.rpm charliecloud-0.37/examples/lustre/test.bats000066400000000000000000000045711457016721300211040ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" setup () { scope full prerequisites_ok lustre if [[ $CH_TEST_LUSTREDIR = skip ]]; then # Assume that in a Slurm allocation, even if one node, Lustre should # be available for testing. msg='no Lustre test directory to bind mount' if [[ $SLURM_JOB_ID ]]; then pedantic_fail "$msg" else skip "$msg" fi elif [[ ! -d $CH_TEST_LUSTREDIR ]]; then echo "'${CH_TEST_LUSTREDIR}' is not a directory" 1>&2 exit 1 fi } clean_dir () { rmdir "${1}/set_stripes" rmdir "${1}/test_create_dir" rm "${1}/test_write.txt" rmdir "$1" } tidy_run () { ch-run -b "$binds" "$ch_img" -- "$@" } binds=${CH_TEST_LUSTREDIR}:/mnt/0 work_dir=/mnt/0/charliecloud_test @test "${ch_tag}/start clean" { clean_dir "${CH_TEST_LUSTREDIR}/charliecloud_test" || true mkdir "${CH_TEST_LUSTREDIR}/charliecloud_test" # fail if not cleaned up } @test "${ch_tag}/create directory" { tidy_run mkdir "${work_dir}/test_create_dir" } @test "${ch_tag}/create file" { tidy_run touch "${work_dir}/test_create_file" } @test "${ch_tag}/delete file" { tidy_run rm "${work_dir}/test_create_file" } @test "${ch_tag}/write file" { # sh wrapper to get echo output to the right place. Without it, the output # from echo goes outside the container. tidy_run sh -c "echo hello > ${work_dir}/test_write.txt" } @test "${ch_tag}/read file" { output_expected=$(cat <<'EOF' hello 0+1 records in 0+1 records out EOF ) # Using dd allows us to skip the write cache and hit the disk. run tidy_run dd if="${work_dir}/test_write.txt" iflag=nocache status=noxfer diff -u <(echo "$output_expected") <(echo "$output") } @test "${ch_tag}/striping" { tidy_run mkdir "${work_dir}/set_stripes" stripe_ct_old=$(tidy_run lfs getstripe --stripe-count "${work_dir}/set_stripes/") echo "old stripe count: $stripe_ct_old" expected_new=$((stripe_ct_old * 2)) echo "expected new stripe count: $expected_new" tidy_run lfs setstripe -c "$expected_new" "${work_dir}/set_stripes" stripe_ct_new=$(tidy_run lfs getstripe --stripe-count "${work_dir}/set_stripes") echo "actual new stripe count: $stripe_ct_new" [[ $expected_new -eq $stripe_ct_new ]] } @test "${ch_tag}/clean up" { clean_dir "${CH_TEST_LUSTREDIR}/charliecloud_test" } charliecloud-0.37/examples/mpibench/000077500000000000000000000000001457016721300175125ustar00rootroot00000000000000charliecloud-0.37/examples/mpibench/Dockerfile.mpich000066400000000000000000000005641457016721300226100ustar00rootroot00000000000000# ch-test-scope: full FROM mpich RUN dnf install -y which \ && dnf clean all # Compile the Intel MPI benchmark WORKDIR /usr/local/src ARG IMB_VERSION=IMB-v2021.3 RUN git clone --branch $IMB_VERSION --depth 1 \ https://github.com/intel/mpi-benchmarks \ && cd mpi-benchmarks/src_c \ && make CC=mpicc -j$(getconf _NPROCESSORS_ONLN) -f Makefile TARGET=MPI1 charliecloud-0.37/examples/mpibench/Dockerfile.openmpi000066400000000000000000000005661457016721300231610ustar00rootroot00000000000000# ch-test-scope: full FROM openmpi RUN dnf install -y which \ && dnf clean all # Compile the Intel MPI benchmark WORKDIR /usr/local/src ARG IMB_VERSION=IMB-v2021.3 RUN git clone --branch $IMB_VERSION --depth 1 \ https://github.com/intel/mpi-benchmarks \ && cd mpi-benchmarks/src_c \ && make CC=mpicc -j$(getconf _NPROCESSORS_ONLN) -f Makefile TARGET=MPI1 charliecloud-0.37/examples/mpibench/test.bats000066400000000000000000000126001457016721300213430ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" setup () { scope full prerequisites_ok "$ch_tag" pmix_or_skip # One iteration on most of these tests because we just care about # correctness, not performance. (If we let the benchmark choose, there is # an overwhelming number of errors when MPI calls start failing, e.g. if # CMA isn’t working, and this makes the test take really long.) # # Large -npmin because we only want to test all cores. imb_mpi1=/usr/local/src/mpi-benchmarks/src_c/IMB-MPI1 imb_args="-iter 1 -npmin 1000000000" # On the HSN performance test, we do want to run multiple iterations to # reduce variability. The benchmark will automatically scale down the # number of iterations based on the expected run time, disabling that so # we get more consistent results. Npmin is ommitted as we are only running # with two processes, one per node. imb_perf_args="-iter 100 -iter_policy off" } check_errors () { [[ ! "$1" =~ 'errno =' ]] } check_finalized () { [[ "$1" =~ 'All processes entering MPI_Finalize' ]] } check_process_ct () { ranks_expected="$1" echo "ranks expected: ${ranks_expected}" ranks_found=$( echo "$output" \ | grep -F '#processes =' \ | sed -r 's/^.+#processes = ([0-9]+)\s+$/\1/') echo "ranks found: ${ranks_found}" [[ $ranks_found -eq "$ranks_expected" ]] } # one from "Single Transfer Benchmarks" @test "${ch_tag}/pingpong (guest launch)" { openmpi_or_skip # shellcheck disable=SC2086 run ch-run $ch_unslurm "$ch_img" -- \ "$ch_mpi_exe" $ch_mpirun_np "$imb_mpi1" $imb_args PingPong echo "$output" [[ $status -eq 0 ]] check_errors "$output" check_process_ct 2 "$output" check_finalized "$output" } # one from "Parallel Transfer Benchmarks" @test "${ch_tag}/sendrecv (guest launch)" { openmpi_or_skip # shellcheck disable=SC2086 run ch-run $ch_unslurm "$ch_img" -- \ "$ch_mpi_exe" $ch_mpirun_np "$imb_mpi1" $imb_args Sendrecv echo "$output" [[ $status -eq 0 ]] check_errors "$output" check_process_ct "$ch_cores_node" "$output" check_finalized "$output" } # one from "Collective Benchmarks" @test "${ch_tag}/allreduce (guest launch)" { openmpi_or_skip # shellcheck disable=SC2086 run ch-run $ch_unslurm "$ch_img" -- \ "$ch_mpi_exe" $ch_mpirun_np "$imb_mpi1" $imb_args Allreduce echo "$output" [[ $status -eq 0 ]] check_errors "$output" check_process_ct "$ch_cores_node" "$output" check_finalized "$output" } @test "${ch_tag}/inject cray mpi ($cray_prov)" { cray_ofi_or_skip "$ch_img" run ch-run "$ch_img" -- fi_info echo "$output" [[ $output == *"provider: $cray_prov"* ]] [[ $output == *"fabric: $cray_prov"* ]] [[ $status -eq 0 ]] } # This test compares OpenMPI’s point to point bandwidth with all high speed # plugins enabled against the performance just using TCP. Pass if HSN # performance is at least double TCP. @test "${ch_tag}/using the high-speed network (host launch)" { multiprocess_ok [[ $ch_multinode ]] || skip "multinode only" if [[ $ch_cray ]]; then [[ $cray_prov == 'gni' ]] && skip "gni doesn't support tcp" fi openmpi_or_skip # Verify we have known HSN devices present. (Note that -d tests for # directory, not device.) if [[ ! -d /dev/infiniband ]] && [[ ! -e /dev/cxi0 ]]; then pedantic_fail "no high speed network detected" fi # shellcheck disable=SC2086 hsn_enabled_bw=$($ch_mpirun_2_2node ch-run \ "$ch_img" -- "$imb_mpi1" $imb_perf_args Sendrecv \ | tail -n +35 | sort -nrk6 | head -1 | awk '{print $6}') # Configure network transport plugins to TCP only. # shellcheck disable=SC2086 hsn_disabled_bw=$(OMPI_MCA_pml=ob1 OMPI_MCA_btl=tcp,self \ $ch_mpirun_2_2node ch-run "$ch_img" -- \ "$imb_mpi1" $imb_perf_args Sendrecv | tail -n +35 \ | sort -nrk6 | head -1 | awk '{print $6}') echo "Max bandwidth with high speed network: $hsn_enabled_bw MB/s" echo "Max bandwidth without high speed network: $hsn_disabled_bw MB/s" [[ ${hsn_disabled_bw%\.*} -lt $((${hsn_enabled_bw%\.*} / 2)) ]] } @test "${ch_tag}/pingpong (host launch)" { multiprocess_ok # shellcheck disable=SC2086 run $ch_mpirun_core ch-run --join "$ch_img" -- \ "$imb_mpi1" $imb_args PingPong echo "$output" [[ $status -eq 0 ]] check_errors "$output" check_process_ct 2 "$output" check_finalized "$output" } @test "${ch_tag}/sendrecv (host launch)" { multiprocess_ok # shellcheck disable=SC2086 run $ch_mpirun_core ch-run --join "$ch_img" -- \ "$imb_mpi1" $imb_args Sendrecv echo "$output" [[ $status -eq 0 ]] check_errors "$output" check_process_ct "$ch_cores_total" "$output" check_finalized "$output" } @test "${ch_tag}/allreduce (host launch)" { multiprocess_ok # shellcheck disable=SC2086 run $ch_mpirun_core ch-run --join "$ch_img" -- \ "$imb_mpi1" $imb_args Allreduce echo "$output" [[ $status -eq 0 ]] check_errors "$output" check_process_ct "$ch_cores_total" "$output" check_finalized "$output" } @test "${ch_tag}/revert image" { unpack_img_all_nodes "$ch_cray" } charliecloud-0.37/examples/mpihello/000077500000000000000000000000001457016721300175365ustar00rootroot00000000000000charliecloud-0.37/examples/mpihello/Dockerfile.mpich000066400000000000000000000001261457016721300226260ustar00rootroot00000000000000# ch-test-scope: full FROM mpich COPY . /hello WORKDIR /hello RUN make clean && make charliecloud-0.37/examples/mpihello/Dockerfile.openmpi000066400000000000000000000001471457016721300232000ustar00rootroot00000000000000# ch-test-scope: full FROM openmpi # This example COPY . /hello WORKDIR /hello RUN make clean && make charliecloud-0.37/examples/mpihello/Makefile000066400000000000000000000002351457016721300211760ustar00rootroot00000000000000BINS := hello CFLAGS := -std=gnu11 -Wall .PHONY: all all: $(BINS) .PHONY: clean clean: rm -f $(BINS) $(BINS): Makefile %: %.c mpicc $(CFLAGS) $< -o $@ charliecloud-0.37/examples/mpihello/hello.c000066400000000000000000000042251457016721300210100ustar00rootroot00000000000000/* MPI test program. Reports user namespace and rank, then sends and receives some simple messages. Patterned after: http://en.wikipedia.org/wiki/Message_Passing_Interface#Example_program */ #define _GNU_SOURCE #include #include #include #include #include #include #include #include #define TAG 0 #define MSG_OUT 8675309 void fatal(char * fmt, ...); int op(int rank, int i); int rank, rank_ct; int main(int argc, char ** argv) { char hostname[HOST_NAME_MAX+1]; char mpi_version[MPI_MAX_LIBRARY_VERSION_STRING]; int mpi_version_len; int msg; MPI_Status mstat; struct stat st; stat("/proc/self/ns/user", &st); MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &rank_ct); MPI_Comm_rank(MPI_COMM_WORLD, &rank); if (rank == 0) { MPI_Get_library_version(mpi_version, &mpi_version_len); printf("%d: MPI version:\n%s\n", rank, mpi_version); } gethostname(hostname, HOST_NAME_MAX+1); printf("%d: init ok %s, %d ranks, userns %lu\n", rank, hostname, rank_ct, st.st_ino); fflush(stdout); if (rank == 0) { for (int i = 1; i < rank_ct; i++) { msg = MSG_OUT; MPI_Send(&msg, 1, MPI_INT, i, TAG, MPI_COMM_WORLD); msg = 0; MPI_Recv(&msg, 1, MPI_INT, i, TAG, MPI_COMM_WORLD, &mstat); if (msg != op(i, MSG_OUT)) fatal("0: expected %d back but got %d", op(i, MSG_OUT), msg); } } else { msg = 0; MPI_Recv(&msg, 1, MPI_INT, 0, TAG, MPI_COMM_WORLD, &mstat); if (msg != MSG_OUT) fatal("%d: expected %d but got %d", rank, MSG_OUT, msg); msg = op(rank, msg); MPI_Send(&msg, 1, MPI_INT, 0, TAG, MPI_COMM_WORLD); } if (rank == 0) printf("%d: send/receive ok\n", rank); MPI_Finalize(); if (rank == 0) printf("%d: finalize ok\n", rank); return 0; } void fatal(char * fmt, ...) { va_list ap; fprintf(stderr, "rank %d:", rank); va_start(ap, fmt); vfprintf(stderr, fmt, ap); va_end(ap); fprintf(stderr, "\n"); exit(EXIT_FAILURE); } int op(int rank, int i) { return i * rank; } charliecloud-0.37/examples/mpihello/slurm.sh000077500000000000000000000013351457016721300212410ustar00rootroot00000000000000#!/bin/bash #SBATCH --time=0:10:00 # Arguments: Path to tarball, path to image parent directory. set -e tar=$1 imgdir=$2 img=${2}/$(basename "${tar%.tar.gz}") if [[ -z $tar ]]; then echo 'no tarball specified' 1>&2 exit 1 fi printf 'tarball: %s\n' "$tar" if [[ -z $imgdir ]]; then echo 'no image directory specified' 1>&2 exit 1 fi printf 'image: %s\n' "$img" # Make Charliecloud available (varies by site). module purge module load friendly-testing module load charliecloud # Unpack image. srun ch-convert -o dir "$tar" "$imgdir" # MPI version in container. printf 'container: ' ch-run "$img" -- mpirun --version | grep -E '^mpirun' # Run the app. srun --cpus-per-task=1 ch-run "$img" -- /hello/hello charliecloud-0.37/examples/mpihello/test.bats000066400000000000000000000073741457016721300214030ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" setup () { scope full prerequisites_ok "$ch_tag" pmix_or_skip if [[ $srun_mpi != pmix* ]]; then skip 'pmix required' fi } count_ranks () { echo "$1" \ | grep -E '^0: init ok' \ | tail -1 \ | sed -r 's/^.+ ([0-9]+) ranks.+$/\1/' } @test "${ch_tag}/guest starts ranks" { openmpi_or_skip # shellcheck disable=SC2086 run ch-run $ch_unslurm "$ch_img" -- mpirun $ch_mpirun_np /hello/hello echo "$output" [[ $status -eq 0 ]] rank_ct=$(count_ranks "$output") echo "found ${rank_ct} ranks, expected ${ch_cores_node}" [[ $rank_ct -eq "$ch_cores_node" ]] [[ $output = *'0: send/receive ok'* ]] [[ $output = *'0: finalize ok'* ]] } @test "${ch_tag}/inject cray mpi ($cray_prov)" { cray_ofi_or_skip "$ch_img" run ch-run "$ch_img" -- fi_info echo "$output" [[ $output == *"provider: $cray_prov"* ]] [[ $output == *"fabric: $cray_prov"* ]] [[ $status -eq 0 ]] } @test "${ch_tag}/validate $cray_prov injection" { [[ -n "$ch_cray" ]] || skip "host is not cray" [[ -n "$CH_TEST_OFI_PATH" ]] || skip "--fi-provider not set" run $ch_mpirun_node ch-run --join "$ch_img" -- sh -c \ "FI_PROVIDER=$cray_prov FI_LOG_LEVEL=info /hello/hello 2>&1" echo "$output" [[ $status -eq 0 ]] if [[ "$cray_prov" == gni ]]; then [[ "$output" == *' registering provider: gni'* ]] [[ "$output" == *'gni:'*'gnix_ep_nic_init()'*'Allocated new NIC for EP'* ]] fi if [[ "$cray_prov" == cxi ]]; then [[ "$output" == *'cxi:mr:ofi_'*'stats:'*'searches'*'deletes'*'hits'* ]] fi } @test "${ch_tag}/MPI version" { [[ -z $ch_cray ]] || skip 'serial launches unsupported on Cray' # shellcheck disable=SC2086 run ch-run $ch_unslurm "$ch_img" -- /hello/hello echo "$output" [[ $status -eq 0 ]] if [[ $ch_mpi = openmpi ]]; then [[ $output = *'Open MPI'* ]] else [[ $ch_mpi = mpich ]] if [[ $ch_cray ]]; then [[ $output = *'CRAY MPICH'* ]] else [[ $output = *'MPICH Version:'* ]] fi fi } @test "${ch_tag}/empty stderr" { multiprocess_ok output=$($ch_mpirun_core ch-run --join "$ch_img" -- \ /hello/hello 2>&1 1>/dev/null) echo "$output" [[ -z "$output" ]] } @test "${ch_tag}/serial" { [[ -z $ch_cray ]] || skip 'serial launches unsupported on Cray' # This seems to start up the MPI infrastructure (daemons, etc.) within the # guest even though there's no mpirun. # shellcheck disable=SC2086 run ch-run $ch_unslurm "$ch_img" -- /hello/hello echo "$output" [[ $status -eq 0 ]] [[ $output = *' 1 ranks'* ]] [[ $output = *'0: send/receive ok'* ]] [[ $output = *'0: finalize ok'* ]] } @test "${ch_tag}/host starts ranks" { multiprocess_ok echo "starting ranks with: ${ch_mpirun_core}" guest_mpi=$(ch-run "$ch_img" -- mpirun --version | head -1) echo "guest MPI: ${guest_mpi}" # shellcheck disable=SC2086 run $ch_mpirun_core ch-run --join "$ch_img" -- /hello/hello 2>&1 echo "$output" [[ $status -eq 0 ]] rank_ct=$(count_ranks "$output") echo "found ${rank_ct} ranks, expected ${ch_cores_total}" [[ $rank_ct -eq "$ch_cores_total" ]] [[ $output = *'0: send/receive ok'* ]] [[ $output = *'0: finalize ok'* ]] } @test "${ch_tag}/Cray bind mounts" { [[ $ch_cray ]] || skip 'host is not a Cray' ch-run "$ch_img" -- mount | grep -F /dev/hugepages if [[ $cray_prov == 'gni' ]]; then ch-run "$ch_img" -- mount | grep -F /var/opt/cray/alps/spool else ch-run "$ch_img" -- mount | grep -F /var/spool/slurmd fi } @test "${ch_tag}/revert image" { unpack_img_all_nodes "$ch_cray" } charliecloud-0.37/examples/multistage/000077500000000000000000000000001457016721300201035ustar00rootroot00000000000000charliecloud-0.37/examples/multistage/Dockerfile000066400000000000000000000030561457016721300221010ustar00rootroot00000000000000# This image tests multi-stage build using GNU Hello. In the first stage, we # install a build environment and build Hello. In the second stage, we start # fresh again with a base image and copy the Hello executables. Tests # demonstrate that Hello runs and none of the build environment is present. # # ch-test-scope: standard FROM almalinux:8 AS buildstage # Build environment RUN dnf install -y \ gcc \ make \ wget WORKDIR /usr/local/src # GNU Hello. Install using DESTDIR to make copying below easier. # # This downloads from a specific mirror [1] that smelled reliable because both # ftp.gnu.org itself and the mirror alias ftpmirror.gnu.org are unreliable. # Specifically, ftpmirror.gnu.org frequently ends up a tripadvisor.com, which # is frequently HTTP 500. # # [1]: https://www.gnu.org/prep/ftp.html ARG gnu_mirror=mirrors.kernel.org/gnu ARG version=2.12.1 RUN wget -nv https://${gnu_mirror}/hello/hello-${version}.tar.gz RUN tar xf hello-${version}.tar.gz \ && cd hello-${version} \ && ./configure \ && make -j $(getconf _NPROCESSORS_ONLN) \ && make install DESTDIR=/hello RUN ls -ld /hello/usr/local/*/* FROM almalinux:8 RUN dnf install -y man # COPY the hello install over, by both name and index, making sure not to # overwrite existing contents. Recall that COPY works different than cp(1). COPY --from=0 /hello/usr/local/bin /usr/local/bin COPY --from=buildstage /hello/usr/local/share /usr/local/share COPY --from=buildstage /hello/usr/local/share/locale /usr/local/share/locale RUN ls -ld /usr/local/*/* charliecloud-0.37/examples/multistage/test.bats000066400000000000000000000027051457016721300217410ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" setup () { prerequisites_ok multistage } @test "${ch_tag}/hello" { run ch-run "$ch_img" -- hello -g 'Hello, Charliecloud!' echo "$output" [[ $status -eq 0 ]] [[ $output = 'Hello, Charliecloud!' ]] } @test "${ch_tag}/man hello" { ch-run "$ch_img" -- man hello > /dev/null } @test "${ch_tag}/files seem OK" { [[ $CH_TEST_PACK_FMT = squash-mount ]] && skip 'need directory image' # hello executable itself. test -x "${ch_img}/usr/local/bin/hello" # Present by default. test -d "${ch_img}/usr/local/share/applications" test -d "${ch_img}/usr/local/share/info" test -d "${ch_img}/usr/local/share/man" # Copied from first stage. test -d "${ch_img}/usr/local/share/locale" # Correct file count in directories. ls -lh "${ch_img}/usr/local/bin" [[ $(find "${ch_img}/usr/local/bin" -mindepth 1 -maxdepth 1 | wc -l) -eq 1 ]] ls -lh "${ch_img}/usr/local/share" [[ $(find "${ch_img}/usr/local/share" -mindepth 1 -maxdepth 1 | wc -l) -eq 4 ]] } @test "${ch_tag}/no first-stage stuff present" { # Can’t run GCC. run ch-run "$ch_img" -- gcc --version echo "$output" [[ $status -eq 1 ]] [[ $output = *'gcc: No such file or directory'* ]] # No GCC or Make. ls -lh "${ch_img}/usr/bin/gcc" || true [[ ! -f "${ch_img}/usr/bin/gcc" ]] ls -lh "${ch_img}/usr/bin/make" || true [[ ! -f "${ch_img}/usr/bin/make" ]] } charliecloud-0.37/examples/obspy/000077500000000000000000000000001457016721300170615ustar00rootroot00000000000000charliecloud-0.37/examples/obspy/Dockerfile000066400000000000000000000030611457016721300210530ustar00rootroot00000000000000# ch-test-scope: full # ch-test-arch-exclude: aarch64 # no obspy Conda package # ch-test-arch-exclude: ppc64le # no obspy Conda package? FROM almalinux_8ch RUN dnf install -y --setopt=install_weak_deps=false \ zlib-devel \ && dnf clean all WORKDIR /usr/local/src # Install Miniconda. Notes/gotchas: # # 1. Install into /usr/local. Some of the instructions [e.g., 1] warn # against putting conda in $PATH; others don’t. However it seems to work # and then we don’t need to muck with the path. # # 2. Use latest version so we catch sooner if things explode. # # [1]: https://docs.anaconda.com/anaconda/user-guide/faq/ ARG MC_VERSION=latest ARG MC_FILE=Miniconda3-$MC_VERSION-Linux-x86_64.sh RUN wget -nv https://repo.anaconda.com/miniconda/$MC_FILE # Miniconda will fail if the HOME variable is not set. RUN HOME=/home bash $MC_FILE -bf -p /usr/local RUN rm -Rf $MC_FILE RUN which conda && conda --version # Disable automatic conda upgrades for predictable versioning. RUN conda config --set auto_update_conda False # Install obspy, also latest. This is a container, so don’t bother creating a # new environment for obspy. # See: https://github.com/obspy/obspy/wiki/Installation-via-Anaconda RUN conda config --add channels conda-forge # Use numpy 1.21 to avoid isse: https://github.com/obspy/obspy/issues/2940 RUN conda install --yes obspy=1.4.0 RUN conda update obspy # Hello world program and input from docs. WORKDIR / RUN wget -nv http://examples.obspy.org/RJOB_061005_072159.ehz.new COPY hello.py . RUN chmod 755 ./hello.py RUN ldconfig charliecloud-0.37/examples/obspy/README000066400000000000000000000022701457016721300177420ustar00rootroot00000000000000We’d prefer to run the ObsPy test suite, but it seems quite finicky and we weren’t able to get it to work. Problems with the test suite include: 1. Requires network access even for the non-network modules. We filed an issue about this [1] that did result in likely-actionable exclusions, though we haven’t followed up. ObsPy also has a PR [2] unmerged as of 2021-08-04 that could replay the network traffic offline. 2. Expects to write within the install directory (e.g., site-packages/obspy/clients/filesystem/tests/data/tsindex_data), which is an antipattern even when not containerized. 3. LOTS of warnings, e.g. hundreds of deprecation gripes from NumPy as well as ObsPy itself. 4. Various errors, e.g. “AttributeError: 'bool' object has no attribute 'lower'” from within Matplotlib. (I was able to solve this one by choosing an older version of Matplotlib than the one depended on by the ObsPy Conda package, but we don't have time to maintain that.) 5. Can’t get it to pass. ;) See also issue #64. Bottom line, I would love for an ObsPy person to maintain this example with passing ObsPy tests, but we don't have time to do so. charliecloud-0.37/examples/obspy/hello.py000066400000000000000000000011531457016721300205360ustar00rootroot00000000000000#!/usr/bin/env python3 # “Reading Seismograms” example from §3 of the ObsPy tutorial, with some of # the prints commented out and taking the plot file from the command line. # # See: https://docs.obspy.org/tutorial/code_snippets/reading_seismograms.html import sys # §3.0 from obspy import read st = read('RJOB_061005_072159.ehz.new') #print(st) #print(len(st)) tr = st[0] # assign first and only trace to new variable print(tr) # §3.1 print(tr.stats) #print(tr.stats.station) #print(tr.stats.datatype) # §3.2 #print(tr.data) print(tr.data[0:3]) print(len(tr)) # §3.3 tr.plot(outfile=sys.argv[1]) charliecloud-0.37/examples/obspy/obspy.png000066400000000000000000000517531457016721300207360ustar00rootroot00000000000000PNG  IHDR Y=#9tEXtSoftwareMatplotlib version3.4.2, https://matplotlib.org/+X pHYsaa?iSXIDATxyXTe?6 "Bf方/e{iOi=V2yR4MsO3T@qEP\a`e>?qT߯ss}3DD@DDDDDdv=<0!"""""aBDDDDD6l  """""& DDDDDd3L@f0!"""""aBDDDDD6l  """""& DU`ȑ#t&,X5kք7LQꢢWWWt .]RF'''jjGnn.{9ԭ[* &YYY;fe-.Xb6l7774nΝÇ~GGGezܸq[GyܹsGi3J?l14k  ҥKK/ F#OuoIxb+ٳgΝ;YYO^u&M@e˖طoI}DDh0rHdgg+uh߾=\]]ѼysDGG+uf2SZm5c"((* 63 OOOԮ]g-kq͛ 777?ի8aooo!!!&jggO>b|7oF۶m???̞=|k׮Jڵk~xxx`Ĉ0 Ef@~˖-/r5iܸ,_\DDz%++KL":tP>|̟?DqŋСC@_nR?ydիˁCΞ=kڴiɓ'h4ʅ &m̟?_nR"oNC/l1hqqqVk~ҥr!ɑ'OJZ?(Ѿ(~^l4mTҶm[YtyDɓ'%77WO.O>RߩS'YfXJìY󒗗'?xyy)1&&F$**JҤs2c eVZɬY$++K>siР䈈̙3eر%cɒ%w^C}KJJ9sF|||dEe-cǎI@@8p@ʕ+r׬Y#:u2)+8/֭[RZ59Hٵkztux{{+ӶJ@ GԩSeܸqJݻA""rh4b0z%$_@@YOڵk3fȑ#-._\\ e˖YRRq^W_}%O84jժUS6mjk׮7q_4pM^z׿^1X[zxexzz>xx9lݺǏ7722M65) ׮]CZZY͚5Ck.!""O<٘c?FBB._nݺwEAi1ydxxx|<?/Μ97n࣏>ȑ#q碸y U_lj's Eő#G^PvÇ+e'NG}GGG bZݻwGdd$Ν;Nܓb阴Nj/ 3fW_}/Z{Vӂ{18qظq#~G|gfQ 0`QF)jZVjM ?Bj0tPm;w`zlQ!//KJnZ\...wyo~}T[z5^x_ [t)֯_uAR;%1cЫW/nZB޽Qn]|7ذaoo߾/֭[V)SVqȑbի=,"1bjժYf)喎'88>>>ppp@K/aӦMEᶊ׿PNT^'O.122O?4jժUl7oƜ9syf峲qF888gϞ.X=zIЫW/*.&M}cQѣi& {}ԩS ڴi6ly%h4ڵkpwwJ* $""""O"wvv\CyPp,G\\BCCz777!..&aaa8rH  eի.-!""""III+0,R {ｇuGNFc2FNN=\]]2|̞=۬<))l=DDDDDjQ^=w(E2 ҥK~z8p@JVC՚̧jVVL% ɯ+ʴi0i$իF*|{@H@~̛7'jԨcٲetFF /// 66V$::!!!E NNNgll,~wh4w;ڵ+7n\Q%Rih4"''zժUî]0aڵ 7YstK/se˖>5kp)lڴ WĿ|r,YիWG`` d=MDDz-KUf#FIٞ={0{l߿Dxx<(=ϟGVrJ%˜1cqFxyyᣏ>!CJVӭ^Gرc+"//~)"##eԮ]C""""ziy4 HET7`ݨV #,[:O?4~m<UaӧO۷_w(DDZeH@*O͛7Sq֭GDD]%&&BRAV>(VXa2JBrr2￣]vPըQF7o*f͂f-*#\\\Rpey+ DDDDd[L@*'''t:ܽ{~)^y={{E1zhܼy111SO=^7zht:hZkxJӕg俺u놗^z DDDDpbR ވX?m4=z4\\\닕+W֭[WT2d_{gѢE8}4&z=OA0`D[l[hV#Gg1)wvvFϞ=gezj4jOTTf͚B?QƤI59 U`'z= ,X-ZwFծ]2b ]"{)==/">C wwaq>///ٙܐ/%%I#GDZZ233g5 Ǐ/qL 5k=QFa׮]2& #Νxlذ [ƍMz=mۆ.]-Rжm[4lw.QK,#G|""""" T@x7Zl2|w/F6leƍ;Xv-h[ bRzUd1|p$&&\~R]tƍ7ߠFhҤ \]]{n8;;+/_jnnn4h}zj(7ƍ,^x͞~ +QiQwUIupBDEE~t:h4 4~W^x;"*<Q*1yODDE+Q*~ !!!. KD";;C!"* H׳gO\t |MyBDW_-0 bb* Fݶm[)GS`g[^_!QUe9;v ڵFA`` VX-X5kք7Lbr qTTN:ҥK[͚5\.etHMM5yQyرcT*^ZޡQ1L2l0iiiXnܹs߰d_8u6oެ$'ĉ'x/rֱcGdeea}&òeː;w8DTnݺeu\#^x!"RRe.t ;;;4o7ٳg_~V¨Qw^j5 f͚tJ-^Ò%K/ 00־… HKK[oZjwHDT Y&v܉~ڦ1FYr,"$ Ǐʕ+1}t;v IIIhӦ {=aaa6m ..Jg11 0 ʴV-q|GF֭g$''W3!vvv0`vZQk"""7?[U&ٳ' 9s-[Zj)ȧhp}aٳ;PJG~WΑQq= oF~xb 8q>/j3Zj 6m4+(" OEDD_H@.^<쳰G&Mйsg۷U捎FHHedd >>Fc""q?!"D裏ݻ_!"8s ~w"<<K,ABBh":w N ̝;-[,ЉK3DDGH@<<<?`ƌh4޽;Ə={O>;v,Zjƍwʈ*NNNX~=-ZOOO8p+W,!""k233_lQ%UenBѣzanڴiWj 111e݇j̟?sEVкuk:&%DD_8BDD"\TT YYY`2,zL@*>& DDT"!)<ȵ3?*JlDDT<& DDT".\v ;v(TDDSVSNpB^T\BDDπ=rJi4e%s@&& DDT"e~߾}gϞ{^7UL@DN8Q߼yAFFF""gey~ex_Q*I۷ ^EDTq,""2Sn-ֽ[Xhdg@*!""3s)uY|Q$$$X,EϞ=ѣݫLM^EDT,""2S%M)))ؾ}{cǎHJJz^1!"QQQ[nzF#F}/_+^EDTUdٲehӦ 7n\:QNXj`޽P5jѣ< BD0K.ARR}-KDDSI@RSSxb̚5ˤ<..tXXN:e AAAEi0jM^DDUQJJJmԨ|{j??ap=-WX^EDTyTd ///rNFLk4t:u ?><<L.#"W%"""LMiӦYUVATDDU_Ϟ=OСCV+ā!"|%XDDT?|2Ν;W.1L@~'_J!""k޼9ڵkhԨۇ:X΀ /HHH֭[e˖R3 DDT&.]EYZWg;o1qD?Tggghxa""?L@̜=z#[o?$''?ŋ+V+W#!CkZT*h4֭[HHH@N0b =\TŸpM1{AzW_k׮.jth4DJDTTaP)7o}]_|Ǜ׫WOn݊x{{ͬ=Jgggt:իW?*XlT*[n_UVaĈf#ݼyz3ik޽T>QΈ*JTH $cƌ ٰaxyyIjjj˥ IOOADD!z_@ڷo/^Z~mi޼4h:t?Sycǎ*?RwiݺhZ;wI}s,!!A$77W^y4h,YDf͚%7nP ~6mH˖-wwwJ%g϶s玤h۷oh8`/xQ9x?^rrrz`\z\~j;w ",&eYYYwAS> ݻwZjr5Ce+CWfFm㋯K/h>OOOeˤK.ʴgggm۶_l5j:u""B”nݺիE˷~+ ,7oʶm۔y7n,2|8pKTT#y 'Z0QEP/z/rtfh,>vP^L>C BZ VkRjV6m&Md2_M6mŋԩGGG 8AAA0a>}HKKCLL2ԩn OOO}46ml<䓰Czz:Μ9MBR?@֭a4qիÇnݺ8u֬Ym۶x憄|ץ ǎÆ 憰0;gΜ1 _~dgg[6?&OիWcΜ98{, ݻwGǣM6ׯ6l؀ 6`ҥ&UHwWŐHx=̞{r&ʿWÆ ݭ.ӼysYpźX,S aÆɅ $%%Ez)_G_|!"")))˅ dܸqr-1 ;HFFrqdՒ+{%yyy!6msΉȧ~*_~$$$_|!gϞ۷o[dddԩSerU垄RFr3|S̙3j/R+Ye~ 7/^;wbĈ*0gDDh; *gpttDff&Pn]YYY%n+x,((H޶mz衬N:^''',^]tAϞ=aÆ᭷;wСC١ZjKP^=?«T*ѣx饗{q4kLĉ_1cƌmo||<AT*΀*7nH^E6l(;w,r!$"W/bbDȺڵkS[7nƍYo0ٳuVDK۶mK;m4 j@+8?~FҴqFΖ#GJBB\zUDD[ N2Y˗effNN :lHX~q/j[Ie~T&"W/OdGWf\]]M@N8!&udbcceٲeңG ׯ_c qqq/-=z7o:uRb),##dڰ=z[Z=֌FDGG[Me~K@8EDt 5k֔wW^~[<-ZPnzg}uW_}[.\b4nݺի.\ɓ'6Y6''4j۷qq<%^*LL% m,qpp';w`ڵ7FRR௿RF)2eI/"{%I&ҥKf8::իWgADeL11s.v &w DNNMfV[n pׯo2ڵk͖I71!""L@ʕ+.Qprr2|с򠠠{%XDDdn \lYY\+aÆYLĬYL+bVrhxG/iRTZ 0ӦMÔ)S:e̙&U59LM= Zj8}4MʧO'*1^u 0ÇWлwx ={amߴn^ &.\P?쳰CJJ cB+ݻwt٤lܸu߿_~%!"LTVW" )x_}r[Ė /""R裏"&&O~vJ;)| /!"֣Gl۶T*K5j*!"oZ[QR/X* Vʲwww ???h49ٶҫHI ;v_:;;c̘1Tqc>>&3gDjj*^k^ùs4i~$%%!11s-*Һ\i׮])rYw7ߔFXDDT*[={8+W̙3hЮ];k׮DFF_D˖-3f`ժU>QԵkW:}QiADTUd޽u}Y;w 99_XXN:3KHH@VV hZ ŽbBDTU$77o&/^lV`ooWWWL@)Ƥ.ܒCyիW`ѢE<"۽{w۷b{www}l' ;vZ?`۷?͛ӧ.\#GhӦ ::!!!`*mEGGApqq.''2HDTL:׮]C޽;ټysy@DD Hq"""VZ믿FΝf:u 6m_2d:wW^yAAA7oc3*:`ɒ%ʏ9ŋ |XDȺ͛pDDTv*}i2mooooo咫9s`̘1SW_QFP|'ׯZ-{9" }|@ hؿQQ*V NDT999`֧z w~uHo$Vp~[#Ye~ZG"""*Q}glDDTQp}ѱcR""2c2)B$DDTkIu""'L@ATaxȶ&SO!33 "`BDDfhn̙ JcX`"" G'""3X~qmQU3 DDdnݺ6[פIbQbBDD%V|'&U\*C "*wDDtQSЉlJ$ k֬> F-[J]VVYdوA`ȑζuDDO?!0jԨSO@_#GbٲeHOOǫA)3gDjj*^k^ùs4i~$%%!11s-M!"\\\_a(F˗wDDt*}r5Ԯ];wJBxx8\wV\3gBѠ]v߿?֮] ċ/-[3fUss*J={wDDTETYf_>vڅ>۷+7ohl͛qJQO@cǎ-yxx`ǎHOO;＃ݻXL4 s*˾K޽;RSS1j(<͵6Q#wDDTT""DIT*\~>>>ESn]lڴ -ZiӐ%K~w3/^ٳgѺukܼyժU`ʕرcj@zz:4̓o )U崙 HI%&&"55U,..J}XX84jHI>iӦ8uT hZ\I@rrr0|pL<Ngtfu:u̟?ʫ^ze%DDdI޽矗wDD=޽;- ޯa`ĈUf͚jjY]~Z.r=ӦMCzzJJJ-%"e? "Pرۘ0a]m۶9Upp0bcch4h...ٳgGGG@LL &O\z= HI z0sL8p7n4K ǎCzz:͛p@FШQ#,X_}9<Q HF_̙ӧOWjP|'ׯP^=ʲضm<==7`pp(BDDDDDUV Üã2|? ?wpDDDDT/<'s8^""""H޽<%Xh4ٳFRRR=EiZԫW}Xc>؇ݻ5"`ggu!Vnʏ}X5+?a>*ꙏ|3-"""""*  䄙3g cV ʏ}Xx: π0!"""""aBDDDDD6l#Gܹ3bcc f͚Ɣ)SP(:u¥K#F jj!!!EƐ{uօJBrrI}VVY6Y VX  7ƹs*:99Q7n`֭xGgywQ̟7R?[/""͚5;tRٳ۷/jԨ5k"<}Faqƙ/GGGbI&!00hٲ%Wd<FӦMZjatJ}||<ڷoWWW4oѥ'*/[̙3Q^=h44l+V}h!?/11...xo*ZHLJN9sHRR'|"""exv4nX/_.""z^d咕%SL:(>\ϟ_rrrdr! ׯ_7siР<.lчΝNS؇Y(3 z @>}0j(XjyZ-jź?8PZ5 :m۶Ν;!!!ʍVpKj5򐙙iq=۲Wwyo~S[z5^x8::ҥK~z[*ʤ.!!ˋ}cm$sz]l{`ĈUf͚Ujժ^zaذa%ac>J6mX|yb*x\U>,ÒE?^z}}E*b"D& >>EQ-kqyyyפ\~ei{/^\*eXnWE0o$4l>>>Xp!{b;,ޛ؇H*JF!ݻwl͛7K@@\xQ_.!!!f`XBzL:du։NYvKBBB1z(YYYJo-}V+9sLXkJ[W^&MȊ+Lڰ6 ֶm۔" a_"SDdRfM6KKKMy}aMLLx{{-iiiSO|^^}U̔7#5jԐ={Xm}Kf$==xl"gΜ(׮]ݻ /ԷjJ̙#z^K!oFܹ#yyyw^h4E\q>4WS뭷ޒaÆɭ[,⽫h}"ݫRK@8;;ڷo|Rzuɓ'|>r䈄tAiJD}hCZn-vG@@0y̔!Cիe-. cƌF#% >"3:矗ӧ-o>qssS;w,&;`RWpy晌UܾYbZÇ[,ݸqCz%...ҰaCٹsRyf󓼼<傃eժU"R> NNN&u^tIҥK""wI Uԩ#F2Ү];qvvf͚bT*g}VEVKpp|J]>~\U>آ *<?!?tT"%v0DDDDDD1!"""""aBDDDDD6l  """""& DDDDDd3L@f0!"""""aBDDDDD6l  """""& DDDDDd3L@f0!"""""aBDDDDD6PfxKIENDB`charliecloud-0.37/examples/obspy/test.bats000066400000000000000000000010451457016721300207130ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" setup () { scope standard prerequisites_ok obspy indir=$CHTEST_EXAMPLES_DIR/obspy outdir=$BATS_TMPDIR/obspy } @test "${ch_tag}/hello" { # Remove prior test’s plot to avoid using it if something else breaks. mkdir -p "$outdir" rm -f "$outdir"/obspy.png ch-run -b "${outdir}:/mnt" "$ch_img" -- /hello.py /mnt/obspy.png } @test "${ch_tag}/hello PNG" { pict_ok pict_assert_equal "${indir}/obspy.png" \ "${outdir}/obspy.png" 1 } charliecloud-0.37/examples/paraview/000077500000000000000000000000001457016721300175435ustar00rootroot00000000000000charliecloud-0.37/examples/paraview/Dockerfile000066400000000000000000000036321457016721300215410ustar00rootroot00000000000000# ch-test-scope: skip #1810 FROM openmpi WORKDIR /usr/local/src # The mesa rpms introduce explicit dependencies python3.11-libs; ParaView will # error at configure time unless we provide the python3.11-devel package. RUN dnf install -y --setopt=install_weak_deps=false \ cmake \ expat-devel \ llvm \ llvm-devel \ mesa-libGL \ mesa-libGL-devel \ mesa-libOSMesa \ mesa-libOSMesa-devel \ python3-mako \ python3-pip \ python3.11-devel \ zlib-devel \ && dnf clean all RUN pip3 install --no-binary=mpi4py \ cython \ mpi4py WORKDIR /usr/local/src # ParaView. Use system libpng to work around issues linking with NEON specific # symbols on ARM. ARG PARAVIEW_MAJORMINOR=5.11 ARG PARAVIEW_VERSION=5.11.2 RUN wget -nv -O ParaView-v${PARAVIEW_VERSION}.tar.xz "https://www.paraview.org/paraview-downloads/download.php?submit=Download&version=v${PARAVIEW_MAJORMINOR}&type=binary&os=Sources&downloadFile=ParaView-v${PARAVIEW_VERSION}.tar.xz" \ && tar xf ParaView-v${PARAVIEW_VERSION}.tar.xz \ && mkdir ParaView-v${PARAVIEW_VERSION}.build \ && cd ParaView-v${PARAVIEW_VERSION}.build \ && cmake -DCMAKE_INSTALL_PREFIX=/usr/local \ -DCMAKE_BUILD_TYPE=Release \ -DBUILD_TESTING=OFF \ -DBUILD_SHARED_LIBS=ON \ -DPARAVIEW_ENABLE_PYTHON=ON \ -DPARAVIEW_BUILD_QT_GUI=OFF \ -DVTK_USE_X=OFF \ -DOPENGL_INCLUDE_DIR=IGNORE \ -DOPENGL_gl_LIBRARY=IGNORE \ -DVTK_OPENGL_HAS_OSMESA=ON \ -DVTK_USE_OFFSCREEN=OFF \ -DPARAVIEW_USE_MPI=ON \ -DPYTHON_EXECUTABLE=/usr/bin/python3 \ -DVTK_USE_SYSTEM_PNG=ON \ ../ParaView-v${PARAVIEW_VERSION} \ && make -j $(getconf _NPROCESSORS_ONLN) install \ && rm -Rf ../ParaView-v${PARAVIEW_VERSION}* charliecloud-0.37/examples/paraview/cone.2ranks.vtk000066400000000000000000000006611457016721300224170ustar00rootroot00000000000000# vtk DataFile Version 5.1 vtk output ASCII DATASET POLYDATA POINTS 12 float 0.5 0 0 -0.5 0.5 0 -0.5 0.25 0.433013 -0.5 -0.25 0.433013 -0.5 -0.5 6.12323e-17 -0.5 -0.25 -0.433013 -0.5 0.25 -0.433013 0.5 0 0 -0.5 -0.5 6.12323e-17 -0.5 -0.25 -0.433013 -0.5 0.25 -0.433013 -0.5 0.5 -1.22465e-16 POLYGONS 8 24 OFFSETS vtktypeint64 0 6 9 12 15 18 21 24 CONNECTIVITY vtktypeint64 6 5 4 3 2 1 0 1 2 0 2 3 0 3 4 7 8 9 7 9 10 7 10 11 charliecloud-0.37/examples/paraview/cone.nranks.vtk000066400000000000000000000011271457016721300225110ustar00rootroot00000000000000# vtk DataFile Version 5.1 vtk output ASCII DATASET POLYDATA POINTS 22 float 0.5 0 0 -0.5 0.5 0 -0.5 0.25 0.433013 -0.5 -0.25 0.433013 -0.5 -0.5 6.12323e-17 -0.5 -0.25 -0.433013 -0.5 0.25 -0.433013 0.5 0 0 -0.5 0.25 0.433013 -0.5 -0.25 0.433013 0.5 0 0 -0.5 -0.25 0.433013 -0.5 -0.5 6.12323e-17 0.5 0 0 -0.5 -0.5 6.12323e-17 -0.5 -0.25 -0.433013 0.5 0 0 -0.5 -0.25 -0.433013 -0.5 0.25 -0.433013 0.5 0 0 -0.5 0.25 -0.433013 -0.5 0.5 -1.22465e-16 POLYGONS 8 24 OFFSETS vtktypeint64 0 6 9 12 15 18 21 24 CONNECTIVITY vtktypeint64 6 5 4 3 2 1 0 1 2 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 charliecloud-0.37/examples/paraview/cone.png000066400000000000000000000075021457016721300212010ustar00rootroot00000000000000PNG  IHDRݡ IDATx^ݏ]y`;m-F/bk' T-mJiBrDeڪRՋREBI#EJ>6xl=3{19|zz+3^ }MO= , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ˜k:'g{'Oo: $`}d}UUؿ!cؿK Bu`>F?%[أ? [u>XK 5>XK ~pp)(+Xw߁NmH` _L`Z ր탐-^A}]*(X_L}lI샵d &`d &`d UDZyjdm)"XnlVg탐-GW#-MJa%[0̃>XK`p+}l 2V`-قrV`-ق]w6Xvv.`Mq%[&`źnlA z>٢dy+W#-ʔgk0XdrdЯ_L(Any-X9~52"WA*X]BIVy*Xd }p@E\ׯF&[DI jdE,W-$X'[/`nolw0"Mm:YX.~wYW?v}{>/]z0q9LX owֿlpᘶHG."Yk&XӕC暂ux)di).X]c-Cfgg4]^5٢Տ Ef65kpIMXk$ɭ!X)0v![E'^d dWb^d)/bp%[ BU慗lџ`K؈`ERTd; VTK` /b`e%Kl/*`!Q$Xel쒭VlC,%[%,z𒭼 ;vVDlGZxVND<1vVtEl%XLV;lE$Xt$ͱKb, xVŔ/J`.d+eE>vVM1^"K!XDl@˱KKJ7i,5Q'XocluI(˄%[,zdk /ٚhk쒭I,v  dxV[ 6څlO`t#]5v /`Al K`/dkp?v  `xVSv([,HEu`{}_3yJw_'ݳޛ~J1{Ѧ3_8-l>~׶VU鷛JJXUՃ6Jb6o^m:(`]o:(`ϙ)knFDIkJH2XH1X.݁:Ik^) VZY;P#`- P#`^,FZ,FZIWL]iXY+s?̧a$'VUKzF'mvC7Bq ii֯{/^?|87RJ_~G?q^G6^ء?va' @r֏W_|}7W=W_[[:7un-+[VWonuue~}إ >֭G=O5Ámz/5Գ{u媪v-UR妇z=ܼO++ydU'6}զ瀤%7aֻ~7mv'gZYY[]ė\Buv7g{gg_RSv֍#~=p[ 6, , , , , , , , , , , , , , , , , , , , , , , ?rLBkIENDB`charliecloud-0.37/examples/paraview/cone.py000066400000000000000000000024611457016721300210440ustar00rootroot00000000000000# Draw a cone and write it out to sys.argv[1] in a few different ways. All # output files should be bit-for-bit reproducible, i.e., no embedded # timestamps, hostnames, floating point error, etc. from __future__ import print_function import os import platform import sys import mpi4py.MPI import paraview.simple as pv # Version information. print("ParaView %d.%d.%d on Python %s" % (pv.paraview.servermanager.vtkSMProxyManager.GetVersionMajor(), pv.paraview.servermanager.vtkSMProxyManager.GetVersionMinor(), pv.paraview.servermanager.vtkSMProxyManager.GetVersionPatch(), platform.python_version())) # Even if you start multiple pvbatch using MPI, this script is only # executed by rank 0. Check this assumption. assert mpi4py.MPI.COMM_WORLD.rank == 0 # Output directory provided on command line. outdir = sys.argv[1] # Render a cone. pv.Cone() pv.Show() pv.Render() print("rendered") # PNG image (serial). filename = "%s/cone.png" % outdir pv.SaveScreenshot(filename) print(filename) # Legacy VTK file (ASCII, serial). filename = "%s/cone.vtk" % outdir pv.SaveData(filename, FileType="Ascii") print(filename) # XML VTK files (parallel). filename=("%s/cone.pvtp" % outdir) writer = pv.XMLPPolyDataWriter(FileName=filename) writer.UpdatePipeline() print(filename) # Done. print("done") charliecloud-0.37/examples/paraview/cone.serial.vtk000066400000000000000000000007511457016721300224760ustar00rootroot00000000000000# vtk DataFile Version 5.1 vtk output ASCII DATASET POLYDATA POINTS 7 float 0.5 0 0 -0.5 0.5 0 -0.5 0.25 0.433013 -0.5 -0.25 0.433013 -0.5 -0.5 6.12323e-17 -0.5 -0.25 -0.433013 -0.5 0.25 -0.433013 METADATA INFORMATION 2 NAME L2_NORM_RANGE LOCATION vtkDataArray DATA 2 0.5 0.707107 NAME L2_NORM_FINITE_RANGE LOCATION vtkDataArray DATA 2 0.5 0.707107 POLYGONS 8 24 OFFSETS vtktypeint64 0 6 9 12 15 18 21 24 CONNECTIVITY vtktypeint64 6 5 4 3 2 1 0 1 2 0 2 3 0 3 4 0 4 5 0 5 6 0 6 1 charliecloud-0.37/examples/paraview/test.bats000066400000000000000000000062331457016721300214010ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" setup () { scope full prerequisites_ok paraview pmix_or_skip indir=${CHTEST_EXAMPLES_DIR}/paraview outdir=$BATS_TMPDIR/paraview inbind=${indir}:/mnt/0 outbind=${outdir}:/mnt/1 if [[ $ch_multinode ]]; then # Bats only creates $BATS_TMPDIR on the first node. # shellcheck disable=SC2086 $ch_mpirun_node mkdir -p "$outdir" else mkdir -p "$outdir" fi } # The first two tests demonstrate ParaView as an “executable” to process a # non-containerized input deck (cone.py) and produce non-containerized output. # # .png: In previous versions, PNG output is antialiased with a single rank # and not with multiple ranks depending on the execution environment. # This is no longer the case as of version 5.5.4 but may change with # a new version of Paraview. # # .vtk: The number of extra and/or duplicate points and indexing of these # points into polygons varied by rank count on my VM, but not on the # cluster. The resulting VTK file is dependent on whether an image was # rendered serially or using 2 or n processes. # # We do not check .pvtp (and its companion .vtp) output because it’s a # collection of XML files containing binary data and it seems too hairy to me. @test "${ch_tag}/inject cray mpi ($cray_prov)" { cray_ofi_or_skip "$ch_img" run ch-run "$ch_img" -- fi_info echo "$output" [[ $output == *"provider: $cray_prov"* ]] [[ $output == *"fabric: $cray_prov"* ]] [[ $status -eq 0 ]] } @test "${ch_tag}/cone serial" { [[ -z $ch_cray ]] || skip 'serial launches unsupported on Cray' # shellcheck disable=SC2086 ch-run $ch_unslurm -b "$inbind" -b "$outbind" "$ch_img" -- \ pvbatch /mnt/0/cone.py /mnt/1 mv "$outdir"/cone.png "$outdir"/cone.serial.png ls -l "$outdir"/cone* diff -u "${indir}/cone.serial.vtk" "${outdir}/cone.vtk" } @test "${ch_tag}/cone serial PNG" { [[ -z $ch_cray ]] || skip 'serial launches unsupported on Cray' pict_ok pict_assert_equal "${indir}/cone.png" "${outdir}/cone.serial.png" 1000 } @test "${ch_tag}/cone ranks=2" { multiprocess_ok # shellcheck disable=SC2086 $ch_mpirun_2 ch-run --join -b "$inbind" -b "$outbind" "$ch_img" -- \ pvbatch /mnt/0/cone.py /mnt/1 mv "$outdir"/cone.png "$outdir"/cone.2ranks.png ls -l "$outdir"/cone* diff -u "${indir}/cone.2ranks.vtk" "${outdir}/cone.vtk" } @test "${ch_tag}/cone ranks=2 PNG" { multiprocess_ok pict_ok pict_assert_equal "${indir}/cone.png" "${outdir}/cone.2ranks.png" 1000 } @test "${ch_tag}/cone ranks=N" { multiprocess_ok # shellcheck disable=SC2086 $ch_mpirun_core ch-run --join -b "$inbind" -b "$outbind" "$ch_img" -- \ pvbatch /mnt/0/cone.py /mnt/1 mv "$outdir"/cone.png "$outdir"/cone.nranks.png ls -l "$outdir"/cone* diff -u "${indir}/cone.nranks.vtk" "${outdir}/cone.vtk" } @test "${ch_tag}/cone ranks=N PNG" { multiprocess_ok pict_ok pict_assert_equal "${indir}/cone.png" "${outdir}/cone.nranks.png" 1000 } @test "${ch_tag}/revert image" { unpack_img_all_nodes "$ch_cray" } charliecloud-0.37/examples/seccomp/000077500000000000000000000000001457016721300173565ustar00rootroot00000000000000charliecloud-0.37/examples/seccomp/Dockerfile000066400000000000000000000004611457016721300213510ustar00rootroot00000000000000# ch-test-scope: standard # ch-test-builder-include: ch-image FROM alpine:3.17 RUN apk add gcc musl-dev strace RSYNC / / RUN gcc -std=c11 -Wall -Werror -fmax-errors=1 -o mknods mknods.c RUN strace ./mknods RUN ls -lh /_* RUN test $(ls /_* | wc -l) == 2 RUN test -p /_mknod_fifo RUN test -p /_mknodat_fifo charliecloud-0.37/examples/seccomp/mknods.c000066400000000000000000000016761457016721300210270ustar00rootroot00000000000000/* Use mknod(2) and mknodat(2) to create character and block devices (which should be blocked by the seccomp filters) and FIFOs (which should not.) */ #define _GNU_SOURCE #include #include #include #include #include #include #include #define DEVNULL makedev(1,3) // character device /dev/null #define DEVRAM0 makedev(1,0) // block device /dev/ram0 #define Z_(x) if (x) (fprintf(stderr, "failed: %d: %s (%d)\n", \ __LINE__, strerror(errno), errno), \ exit(1)) int main(void) { Z_ (mknod("/_mknod_chr", S_IFCHR, DEVNULL)); Z_ (mknod("/_mknod_blk", S_IFBLK, DEVRAM0)); Z_ (mknod("/_mknod_fifo", S_IFIFO, 0)); Z_ (mknodat(AT_FDCWD, "./_mknodat_chr", S_IFCHR, DEVNULL)); Z_ (mknodat(AT_FDCWD, "./_mknodat_blk", S_IFBLK, DEVRAM0)); Z_ (mknodat(AT_FDCWD, "./_mknodat_fifo", S_IFIFO, 0)); } charliecloud-0.37/examples/seccomp/test.bats000066400000000000000000000005501457016721300212100ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "$CHTEST_DIR"/common.bash setup () { prerequisites_ok seccomp } @test "${ch_tag}/fifos only" { ch-run "$ch_img" -- sh -c 'ls -lh /_*' # shellcheck disable=SC2016 ch-run "$ch_img" -- sh -c 'test $(ls /_* | wc -l) == 2' ch-run "$ch_img" -- test -p /_mknod_fifo ch-run "$ch_img" -- test -p /_mknodat_fifo } charliecloud-0.37/examples/spack/000077500000000000000000000000001457016721300170265ustar00rootroot00000000000000charliecloud-0.37/examples/spack/Dockerfile000066400000000000000000000062031457016721300210210ustar00rootroot00000000000000# ch-test-scope: full FROM almalinux:8 # Note: Spack is a bit of an odd duck testing wise. Because it’s a package # manager, the key tests we want are to install stuff (this includes the Spack # test suite), and those don’t make sense at run time. Thus, most of what we # care about is here in the Dockerfile, and test.bats just has a few # trivialities. # # bzip, file, patch, unzip, and which are packages needed to install # Charliecloud with Spack. These are in Spack’s Docker example [2] but are not # documented as prerequisites [1]. texinfo is an undocumented dependency of # Spack’s m4, and that package is in PowerTools, which we enable using sed(1) # to avoid installing the config-manager DNF plugin. # # [1]: https://spack.readthedocs.io/en/latest/getting_started.html # [2]: https://spack.readthedocs.io/en/latest/workflows.html#using-spack-to-create-docker-images RUN sed -Ei 's/enabled=0/enabled=1/' \ /etc/yum.repos.d/almalinux-powertools.repo RUN dnf install -y --setopt=install_weak_deps=false \ bzip2 \ gcc \ gcc-c++ \ git \ gnupg2-smime \ file \ make \ patch \ python3 \ texinfo \ unzip \ which \ && dnf clean all # Certain Spack packages (e.g., tar) puke if they detect themselves being # configured as UID 0. This is the override. See issue #540 and [2]. ARG FORCE_UNSAFE_CONFIGURE=1 # Install Spack. This follows the documented procedure to run it out of the # source directory. There apparently is no “make install” type operation to # place it at a standard path (“spack clone” simply clones another working # directory to a new path). # # Depending on what’s commented below, we get either Spack’s “develop” branch # or the latest released version. Using develop catches problems earlier, but # that branch has a LOT more churn and some of the problems might not occur in # a released version. I expect the right choice will change over time. ARG SPACK_REPO=https://github.com/spack/spack #RUN git clone --depth 1 $SPACK_REPO # tip of develop; faster clone RUN git clone $SPACK_REPO && cd spack && git checkout releases/latest # slow RUN cd spack && git status && git rev-parse --short HEAD # Set up environment to use Spack. (We can’t use setup-env.sh because the # Dockerfile shell is sh, not Bash.) ENV PATH /spack/bin:$PATH RUN spack compiler find --scope system # Test: Some basic commands. RUN which spack RUN spack --version RUN spack compiler find RUN spack compiler list RUN spack compiler list --scope=system RUN spack compiler list --scope=user RUN spack compilers RUN spack spec charliecloud # Test: Install Charliecloud. # Kludge: here we specify an older python sphinx rtd_theme version because # newer default version, 0.5.0, introduces a dependency on node-js which doesn’t # appear to build on gcc 4.8 or gcc 8.3 # (see: https://github.com/spack/spack/issues/19310). RUN spack spec charliecloud+docs^py-sphinx-rtd-theme@0.4.3 RUN spack install charliecloud+docs^py-sphinx-rtd-theme@0.4.3 # Clean up. RUN spack clean --all charliecloud-0.37/examples/spack/test.bats000066400000000000000000000015371457016721300206660ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" setup() { scope full prerequisites_ok spack export PATH=/spack/bin:$PATH } @test "${ch_tag}/version" { # Spack likes to write to $HOME/.spack; thus, we bind it. ch-run --home "$ch_img" -- spack --version } @test "${ch_tag}/compilers" { echo "spack compiler list" ch-run --home "$ch_img" -- spack compiler list echo "spack compiler list --scope=system" ch-run --home "$ch_img" -- spack compiler list --scope=system echo "spack compiler list --scope=user" ch-run --home "$ch_img" -- spack compiler list --scope=user echo "spack compilers" ch-run --home "$ch_img" -- spack compilers } @test "${ch_tag}/find" { run ch-run --home "$ch_img" -- spack find charliecloud echo "$output" [[ $status -eq 0 ]] [[ $output = *'charliecloud@'* ]] } charliecloud-0.37/examples/spark/000077500000000000000000000000001457016721300170455ustar00rootroot00000000000000charliecloud-0.37/examples/spark/Dockerfile000066400000000000000000000026761457016721300210520ustar00rootroot00000000000000# ch-test-scope: full # # Use Buster because Stretch JRE install fails with: # # tempnam() is so ludicrously insecure as to defy implementation. # tempnam: Cannot allocate memory # dpkg: error processing package openjdk-8-jre-headless:amd64 (--configure): # subprocess installed post-installation script returned error exit status 1 FROM debian:buster ARG DEBIAN_FRONTEND=noninteractive # Install needed OS packages. RUN apt-get update \ && apt-get install -y --no-install-recommends \ default-jre-headless \ less \ procps \ python3 \ wget \ && rm -rf /var/lib/apt/lists/* # Download and install Spark. Notes: # # 1. We aren’t using SPARK_NO_DAEMONIZE to make sure can deal with daemonized # applications. # # 2. Spark is installed to /opt/spark, which is Spark’s new default location. ARG URLPATH=https://archive.apache.org/dist/spark/spark-3.2.0/ ARG DIR=spark-3.2.0-bin-hadoop3.2 ARG TAR=$DIR.tgz RUN wget -nv $URLPATH/$TAR \ && tar xf $TAR \ && mv $DIR /opt/spark \ && rm $TAR # Very basic default configuration, to make it run and not do anything stupid. RUN printf '\ SPARK_LOCAL_IP=127.0.0.1\n\ SPARK_LOCAL_DIRS=/tmp\n\ SPARK_LOG_DIR=/tmp\n\ SPARK_WORKER_DIR=/tmp\n\ ' > /opt/spark/conf/spark-env.sh # Move config to /mnt/0 so we can provide a different config if we want RUN mv /opt/spark/conf /mnt/0 \ && ln -s /mnt/0 /opt/spark/conf charliecloud-0.37/examples/spark/slurm.sh000077500000000000000000000043401457016721300205470ustar00rootroot00000000000000#!/bin/bash #SBATCH --time=0:10:00 # Run an example non-interactive Spark computation. Requires three arguments: # # 1. Image tarball # 2. Directory in which to unpack tarball # 3. High-speed network interface name # # Example: # # $ sbatch slurm.sh /scratch/spark.tar.gz /var/tmp ib0 # # Spark configuration will be generated in ~/slurm-$SLURM_JOB_ID.spark; any # configuration already there will be clobbered. set -e if [[ -z $SLURM_JOB_ID ]]; then echo "not running under Slurm" 1>&2 exit 1 fi tar=$1 img=$2 img=${img}/spark dev=$3 conf=${HOME}/slurm-${SLURM_JOB_ID}.spark # Make Charliecloud available (varies by site) module purge module load friendly-testing module load charliecloud # What IP address to use for master? if [[ -z $dev ]]; then echo "no high-speed network device specified" exit 1 fi master_ip=$( ip -o -f inet addr show dev "$dev" \ | sed -r 's/^.+inet ([0-9.]+).+/\1/') master_url=spark://${master_ip}:7077 if [[ -n $master_ip ]]; then echo "Spark master IP: ${master_ip}" else echo "no IP address for ${dev} found" exit 1 fi # Unpack image srun ch-convert -o dir "$tar" "$img" # Make Spark configuration mkdir "$conf" chmod 700 "$conf" cat < "${conf}/spark-env.sh" SPARK_LOCAL_DIRS=/tmp/spark SPARK_LOG_DIR=/tmp/spark/log SPARK_WORKER_DIR=/tmp/spark SPARK_LOCAL_IP=127.0.0.1 SPARK_MASTER_HOST=${master_ip} JAVA_HOME=/usr/lib/jvm/default-java/ EOF mysecret=$(cat /dev/urandom | tr -dc '0-9a-f' | head -c 48) cat < "${conf}/spark-defaults.sh" spark.authenticate true spark.authenticate.secret $mysecret EOF chmod 600 "${conf}/spark-defaults.sh" # Start the Spark master ch-run -b "$conf" "$img" -- /spark/sbin/start-master.sh sleep 10 tail -7 /tmp/spark/log/*master*.out grep -Fq 'New state: ALIVE' /tmp/spark/log/*master*.out # Start the Spark workers srun sh -c " ch-run -b '${conf}' '${img}' -- \ /spark/sbin/start-slave.sh ${master_url} \ && sleep infinity" & sleep 10 grep -F worker /tmp/spark/log/*master*.out tail -3 /tmp/spark/log/*worker*.out # Compute pi ch-run -b "$conf" "$img" -- \ /spark/bin/spark-submit --master "$master_url" \ /spark/examples/src/main/python/pi.py 1024 # Let Slurm kill the workers and master charliecloud-0.37/examples/spark/test.bats000066400000000000000000000121241457016721300206770ustar00rootroot00000000000000CH_TEST_TAG=$ch_test_tag load "${CHTEST_DIR}/common.bash" # Note: If you get output like the following (piping through cat turns off # BATS terminal magic): # # $ ./bats ../examples/spark/test.bats | cat # 1..5 # ok 1 spark/configure # ok 2 spark/start # [...]/test/bats.src/libexec/bats-exec-test: line 329: /tmp/bats.92406.src: No such file or directory # [...]/test/bats.src/libexec/bats-exec-test: line 329: /tmp/bats.92406.src: No such file or directory # [...]/test/bats.src/libexec/bats-exec-test: line 329: /tmp/bats.92406.src: No such file or directory # # that means that mpirun is starting too many processes per node (you want 1). # One solution is to export OMPI_MCA_rmaps_base_mapping_policy= (i.e., set but # empty). setup () { scope standard prerequisites_ok spark pmix_or_skip [[ $CH_TEST_PACK_FMT = *-unpack ]] || skip 'issue #1161' umask 0077 # Unset these Java variables so the container doesn’t use host paths. unset JAVA_BINDIR JAVA_HOME JAVA_ROOT spark_dir=${TMP_}/spark # runs before each test, so no mktemp spark_config=$spark_dir spark_log=/tmp/sparklog confbind=${spark_config}:/mnt/0 if [[ $ch_multinode ]]; then # We use hostname to determine the interface to use for this test, # avoiding complicated logic determining which interface is the HSN. # In many environments this likely results in the tests running over # the slower management interface, which is fine for testing, but # should be avoided for large scale runs. master_host="$(hostname)" # Start Spark workers using pdsh. We would really prefer to do this # using srun, but that doesn’t work; see issue #230. command -v pdsh >/dev/null 2>&1 || pedantic_fail "pdsh not in path" pernode="pdsh -R ssh -w ${SLURM_NODELIST} -- PATH='${PATH}'" else master_host=localhost pernode= fi master_url=spark://${master_host}:7077 master_log="${spark_log}/*master.Master*.out" # expand globs later } @test "${ch_tag}/configure" { # check for restrictive umask run umask -S echo "$output" [[ $status -eq 0 ]] [[ $output = 'u=rwx,g=,o=' ]] # create config $ch_mpirun_node mkdir -p "$spark_config" # We set JAVA_HOME in the spark environment file as this appears to be the # idiomatic method for ensuring spark finds the java install. tee < "${spark_config}/spark-env.sh" SPARK_LOCAL_DIRS=/tmp/spark SPARK_LOG_DIR=$spark_log SPARK_WORKER_DIR=/tmp/spark SPARK_LOCAL_IP=127.0.0.1 SPARK_MASTER_HOST=${master_host} JAVA_HOME=/usr/lib/jvm/default-java/ EOF my_secret=$(cat /dev/urandom | tr -dc '0-9a-f' | head -c 48) tee < "${spark_config}/spark-defaults.conf" spark.authenticate.true spark.authenticate.secret ${my_secret} EOF if [[ $ch_multinode ]]; then sbcast -f "${spark_config}/spark-env.sh" "${spark_config}/spark-env.sh" sbcast -f "${spark_config}/spark-defaults.conf" "${spark_config}/spark-defaults.conf" fi } @test "${ch_tag}/start" { # remove old master logs so new one has predictable name rm -Rf --one-file-system "$spark_log" # start the master ch-run -b "$confbind" "$ch_img" -- /opt/spark/sbin/start-master.sh sleep 15 # shellcheck disable=SC2086 cat $master_log # shellcheck disable=SC2086 grep -Fq 'New state: ALIVE' $master_log # start the workers # shellcheck disable=SC2086 $pernode ch-run -b "$confbind" "$ch_img" -- \ /opt/spark/sbin/start-worker.sh "$master_url" sleep 15 } @test "${ch_tag}/worker count" { # Note that in the log, each worker shows up as 127.0.0.1, which might # lead you to believe that all the workers started on the same (master) # node. However, I believe this string is self-reported by the workers and # is an artifact of SPARK_LOCAL_IP=127.0.0.1 above, which AFAICT just # tells the workers to put their web interfaces on localhost. They still # connect to the master and get work OK. [[ -z $ch_multinode ]] && SLURM_NNODES=1 # shellcheck disable=SC2086 worker_ct=$(grep -Fc 'Registering worker' $master_log || true) echo "node count: $SLURM_NNODES; worker count: ${worker_ct}" [[ $worker_ct -eq "$SLURM_NNODES" ]] } @test "${ch_tag}/pi" { run ch-run -b "$confbind" "$ch_img" -- \ /opt/spark/bin/spark-submit --master "$master_url" \ /opt/spark/examples/src/main/python/pi.py 64 echo "$output" [[ $status -eq 0 ]] # This computation converges quite slowly, so we only ask for two correct # digits of pi. [[ $output = *'Pi is roughly 3.1'* ]] } @test "${ch_tag}/stop" { $pernode ch-run -b "$confbind" "$ch_img" -- /opt/spark/sbin/stop-worker.sh ch-run -b "$confbind" "$ch_img" -- /opt/spark/sbin/stop-master.sh sleep 2 # Any Spark processes left? # (Use egrep instead of fgrep so we don’t match the grep process.) # shellcheck disable=SC2086 $pernode ps aux | ( ! grep -E '[o]rg\.apache\.spark\.deploy' ) } @test "${ch_tag}/hang" { # If there are any test processes remaining, this test will hang. true } charliecloud-0.37/lib/000077500000000000000000000000001457016721300146555ustar00rootroot00000000000000charliecloud-0.37/lib/Makefile.am000066400000000000000000000056201457016721300167140ustar00rootroot00000000000000# Define an alias for pkglibdir to override Automake helpfulness: # # error: 'pkglibdir' is not a legitimate directory for 'DATA' # # See: https://www.gnu.org/software/automake/manual/html_node/Uniform.html mylibdir = $(pkglibdir) dist_mylib_DATA = base.sh \ build.py \ build_cache.py \ charliecloud.py \ filesystem.py \ force.py \ image.py \ misc.py \ pull.py \ push.py \ registry.py mylib_DATA = contributors.bash \ version.py \ version.sh \ version.txt # Bundled Lark (currently version 1.1.9); Automake does not support wildcards # [1], so list the files. Note it's version-specific. Hopefully if a new # version of Lark adds a file and we omit it here by mistake, the tests will # catch it. To get this list: # # $ (cd lib && find lark lark-*.dist-info -xtype f) | LC_ALL=C sort | sed -E 's/$/ \\/' # # Then, copy-n-paste & remove the last backslash. PROOFREAD YOUR DIFF!!! LARK = \ lark-1.1.9.dist-info/INSTALLER \ lark-1.1.9.dist-info/LICENSE \ lark-1.1.9.dist-info/METADATA \ lark-1.1.9.dist-info/RECORD \ lark-1.1.9.dist-info/WHEEL \ lark-1.1.9.dist-info/entry_points.txt \ lark-1.1.9.dist-info/top_level.txt \ lark/__init__.py \ lark/ast_utils.py \ lark/common.py \ lark/exceptions.py \ lark/grammar.py \ lark/grammars/__init__.py \ lark/grammars/common.lark \ lark/grammars/lark.lark \ lark/grammars/python.lark \ lark/grammars/unicode.lark \ lark/indenter.py \ lark/lark.py \ lark/lexer.py \ lark/load_grammar.py \ lark/parse_tree_builder.py \ lark/parser_frontends.py \ lark/parsers/__init__.py \ lark/parsers/cyk.py \ lark/parsers/earley.py \ lark/parsers/earley_common.py \ lark/parsers/earley_forest.py \ lark/parsers/grammar_analysis.py \ lark/parsers/lalr_analysis.py \ lark/parsers/lalr_interactive_parser.py \ lark/parsers/lalr_parser.py \ lark/parsers/lalr_parser_state.py \ lark/parsers/xearley.py \ lark/py.typed \ lark/reconstruct.py \ lark/tools/__init__.py \ lark/tools/nearley.py \ lark/tools/serialize.py \ lark/tools/standalone.py \ lark/tree.py \ lark/tree_matcher.py \ lark/tree_templates.py \ lark/utils.py \ lark/visitors.py if ENABLE_LARK nobase_dist_mylib_DATA = $(LARK) endif CLEANFILES = $(mylib_DATA) contributors.bash: ../README.rst rm -f $@ printf '# shellcheck shell=bash\n' >> $@ printf 'declare -a ch_contributors\n' >> $@ sed -En 's/^\*.+<(.+@.+)>.*$$/ch_contributors+=('"'"'\1'"'"')/p' < $< >> $@ # Remove empty charliecloud directories after uninstallation. uninstall-hook: rmdir $$(find $(pkglibdir) -type d | sort -r) version.txt: ../configure printf '@PACKAGE_VERSION@\n' > $@ version.py: ../configure printf "VERSION='@PACKAGE_VERSION@'\n" > $@ version.sh: ../configure printf "# shellcheck shell=sh disable=SC2034\n" > $@ printf "ch_version='@PACKAGE_VERSION@'\n" >> $@ charliecloud-0.37/lib/base.sh000066400000000000000000000134701457016721300161300ustar00rootroot00000000000000# shellcheck shell=sh set -e ch_bin="$(cd "$(dirname "$0")" && pwd)" # shellcheck disable=SC2034 ch_base=${ch_bin%/*} ch_lib=${ch_bin}/../lib . "${ch_lib}/version.sh" # Log level. Incremented by “--verbose” and decremented by “--quiet”, as in the # Python code. log_level=0 # Logging functions. Note that we disable SC2059 because we want these functions # to behave exactly like printf(1), e.g. we want # # >>> VERBOSE "foo %s" "bar" # foo bar # # Implementing the suggestion in SC2059 would instead result in something like # # >>> VERBOSE "foo %s" "bar" # foo %sbar DEBUG () { if [ "$log_level" -ge 2 ]; then # shellcheck disable=SC2059 printf "$@" 1>&2 printf '\n' 1>&2 fi } FATAL () { printf 'error: ' 1>&2 # shellcheck disable=SC2059 printf "$@" 1>&2 printf '\n' 1>&2 exit 1 } INFO () { if [ "$log_level" -ge 0 ]; then # shellcheck disable=SC2059 printf "$@" 1>&2 printf '\n' 1>&2 fi } VERBOSE () { if [ "$log_level" -ge 1 ]; then # shellcheck disable=SC2059 printf "$@" 1>&2 printf '\n' 1>&2 fi } WARNING () { if [ "$log_level" -ge -1 ]; then printf 'warning: ' 1>&2 # shellcheck disable=SC2059 printf "$@" 1>&2 printf '\n' 1>&2 fi } # Return success if path $1 exists, without dereferencing links, failure # otherwise. (“test -e” dereferences.) exist_p () { stat "$1" > /dev/null 2>&1 } # Try to parse $1 as a common argument. If accepted, either exit (for things # like --help) or return success; otherwise, return failure (i.e., not a # common argument). parse_basic_arg () { case $1 in --_lib-path) # undocumented echo "$ch_lib" exit 0 ;; --help) usage 0 # exits ;; -q|--quiet) if [ $log_level -gt 0 ]; then FATAL "incompatible options: --quiet, --verbose" fi log_level=$((log_level-1)) return 0 ;; -v|--verbose) if [ $log_level -lt 0 ]; then FATAL "incompatible options: --quiet, --verbose" fi log_level=$((log_level+1)) return 0 ;; --version) version # exits ;; esac return 1 # not a basic arg } # Redirect standard streams (or not) depending on “quiet” level. See table in # FAQ. quiet () { if [ $log_level -lt -2 ]; then "$@" 1>/dev/null 2>/dev/null elif [ $log_level -lt -1 ]; then "$@" 1>/dev/null else "$@" fi } # Convert container registry path to filesystem compatible path. # # NOTE: This is used both to name user-visible stuff like tarballs as well as # dig around in the ch-image storage directory. tag_to_path () { echo "$1" | tr '/:' '%+' } usage () { echo "${usage:?}" 1>&2 exit "${1:-1}" } version () { echo 1>&2 "$ch_version" exit 0 } # Set a variable and print its value, human readable description, and origin. # Parameters: # # $1: string: variable name # $2: string: command line argument value (1st priority) # $3: string: environment variable value (2nd priority) # $4: string: default value (3rd priority) # $5: int: width of description (use -1 for natural width) # $6: string: human readable description for stdout # $7: boolean: if true, suppress chatter # # FIXME: Shouldn't export the variable, and no Bash indirection available. # There are safe eval solution out there, but I was too lazy to deal with it. vset () { var_name=$1 cli_value=$2 env_value=$3 def_value=$4 desc_width=$5 var_desc=$6 quiet=$7 if [ "$cli_value" ]; then export "$var_name"="$cli_value" value=$cli_value method='command line' elif [ "$env_value" ]; then export "$var_name"="$env_value" value=$env_value method='environment' else export "$var_name"="$def_value" value=$def_value method='default' fi # FIXME: Kludge: Assume it's a boolean variable and the empty string means # false. Print "no" instead of the empty string. if [ -z "$value" ]; then value=no fi if [ -z "$quiet" ]; then var_desc="$var_desc:" printf "%-*s %s (%s)\n" "$desc_width" "$var_desc" "$value" "$method" fi } # Is Docker present, and if so, do we need sudo? If docker is a wrapper for # podman, “docker info” hangs (#1656), so treat that as not found. if ( ! command -v docker > /dev/null 2>&1 ) \ || ( docker --help 2>&1 | grep -Fqi podman ); then docker_ () { echo 'docker not found; unreachable code reached' 1>&1 exit 1 } elif docker info > /dev/null 2>&1; then docker_ () { docker "$@" } else docker_ () { sudo docker "$@" } fi # Wrapper for rootless podman (for consistency w/ docker). # The only thing we're really concerned with here is the trailing underscore, # since we use it to construct function calls. podman_ () { podman "$@" } # Use parallel gzip if it's available. if command -v pigz > /dev/null 2>&1; then gzip_ () { pigz "$@" } else gzip_ () { gzip "$@" } fi # Use pv(1) to show a progress bar, if it’s available and the quiet level is # less than one, otherwise cat(1). WARNING: You must pipe in the file because # arguments are ignored if this is cat(1). (We also don’t want a progress bar if # stdin is not a terminal, but pv takes care of that). Note that we put the if # statement in the scope of the function because doing so ensures that it gets # evaulated after “quiet” is assigned an appropriate value by “parse_basic_arg”. pv_ () { if command -v pv > /dev/null 2>&1 && [ "$log_level" -gt -1 ]; then pv -pteb "$@" else cat fi } charliecloud-0.37/lib/build.py000066400000000000000000001460561457016721300163420ustar00rootroot00000000000000# Implementation of "ch-image build". import abc import ast import enum import glob import json import os import os.path import re import shutil import sys import charliecloud as ch import build_cache as bu import filesystem as fs import force import image as im ## Globals ## # ARG values that are set before FROM. argfrom = {} # Namespace from command line arguments. FIXME: be more tidy about this ... cli = None # --force injector object (initialized to something meaningful during FROM). forcer = None # Images that we are building. Each stage gets its own image. In this # dictionary, an image appears exactly once or twice. All images appear with # an int key counting stages up from zero. Images with a name (e.g., “FROM ... # AS foo”) have a second string key of the name. images = dict() # Number of stages. This is obtained by counting FROM instructions in the # parse tree, so we can use it for error checking. image_ct = None ## Imports not in standard library ## # See image.py for the messy import of this. lark = im.lark ## Exceptions ## class Instruction_Ignored(Exception): pass ## Main loop ## class Environment: """The state we are in: environment variables, working directory, etc. Most of this is just passed through from the image metadata.""" class Main_Loop(lark.Visitor): __slots__ = ("instruction_total_ct", "miss_ct", # number of misses during this stage "inst_prev") # last instruction executed def __init__(self, *args, **kwargs): self.miss_ct = 0 self.inst_prev = None self.instruction_total_ct = 0 super().__init__(*args, **kwargs) def __default__(self, tree): class_ = tree.data.title() + "_G" if (class_ in globals()): inst = globals()[class_](tree) if (self.instruction_total_ct == 0): if (not (isinstance(inst, Directive_G) or isinstance(inst, From__G) or isinstance(inst, Instruction_No_Image))): ch.FATAL("first instruction must be ARG or FROM") inst.init(self.inst_prev) # The three announce_maybe() calls are clunky but I couldn’t figure # out how to avoid the repeats. try: self.miss_ct = inst.prepare(self.miss_ct) inst.announce_maybe() except Instruction_Ignored: inst.announce_maybe() return except ch.Fatal_Error: inst.announce_maybe() inst.prepare_rollback() raise if (inst.miss): if (self.miss_ct == 1): inst.checkout_for_build() try: inst.execute() except ch.Fatal_Error: inst.rollback() raise if (inst.image_i >= 0): inst.metadata_update() inst.commit() self.inst_prev = inst self.instruction_total_ct += 1 def main(cli_): # CLI namespace. :P global cli cli = cli_ # Infer input file if needed. if (cli.file is None): cli.file = cli.context + "/Dockerfile" # Infer image name if needed. if (cli.tag is None): path = os.path.basename(cli.file) if ("." in path): (base, ext_all) = str(path).split(".", maxsplit=1) (base_all, ext_last) = str(path).rsplit(".", maxsplit=1) else: base = None ext_last = None if (base == "Dockerfile"): cli.tag = ext_all ch.VERBOSE("inferring name from Dockerfile extension: %s" % cli.tag) elif (ext_last in ("df", "dockerfile")): cli.tag = base_all ch.VERBOSE("inferring name from Dockerfile basename: %s" % cli.tag) elif (os.path.abspath(cli.context) != "/"): cli.tag = os.path.basename(os.path.abspath(cli.context)) ch.VERBOSE("inferring name from context directory: %s" % cli.tag) else: assert (os.path.abspath(cli.context) == "/") cli.tag = "root" ch.VERBOSE("inferring name with root context directory: %s" % cli.tag) cli.tag = re.sub(r"[^a-z0-9_.-]", "", cli.tag.lower()) ch.INFO("inferred image name: %s" % cli.tag) # --force and friends. if (cli.force_cmd and cli.force == ch.Force_Mode.FAKEROOT): ch.FATAL("--force-cmd and --force=fakeroot are incompatible") if (not cli.force_cmd): cli.force_cmd = force.FORCE_CMD_DEFAULT else: cli.force = ch.Force_Mode.SECCOMP # convert cli.force_cmd to parsed dict force_cmd = dict() for line in cli.force_cmd: (cmd, args) = force.force_cmd_parse(line) force_cmd[cmd] = args cli.force_cmd = force_cmd ch.VERBOSE("force mode: %s" % cli.force) if (cli.force == ch.Force_Mode.SECCOMP): for (cmd, args) in cli.force_cmd.items(): ch.VERBOSE("force command: %s" % ch.argv_to_string([cmd] + args)) if ( cli.force == ch.Force_Mode.SECCOMP and ch.cmd([ch.CH_BIN + "/ch-run", "--feature=seccomp"], fail_ok=True) != 0): ch.FATAL("ch-run was not built with seccomp(2) support") # Deal with build arguments. def build_arg_get(arg): kv = arg.split("=") if (len(kv) == 2): return kv else: v = os.getenv(kv[0]) if (v is None): ch.FATAL("--build-arg: %s: no value and not in environment" % kv[0]) return (kv[0], v) cli.build_arg = dict( build_arg_get(i) for i in cli.build_arg ) ch.DEBUG(cli) # Guess whether the context is a URL, and error out if so. This can be a # typical looking URL e.g. “https://...” or also something like # “git@github.com:...”. The line noise in the second line of the regex is # to match this second form. Username and host characters from # https://tools.ietf.org/html/rfc3986. if (re.search(r""" ^((git|git+ssh|http|https|ssh):// | ^[\w.~%!$&'\(\)\*\+,;=-]+@[\w.~%!$&'\(\)\*\+,;=-]+:)""", cli.context, re.VERBOSE) is not None): ch.FATAL("not yet supported: issue #773: URL context: %s" % cli.context) if (os.path.exists(cli.context + "/.dockerignore")): ch.WARNING("not yet supported, ignored: issue #777: .dockerignore file") # Read input file. if (cli.file == "-" or cli.context == "-"): text = ch.ossafe("can’t read stdin", sys.stdin.read) elif (not os.path.isdir(cli.context)): ch.FATAL("context must be a directory: %s" % cli.context) else: fp = fs.Path(cli.file).open("rt") text = ch.ossafe("can’t read: %s" % cli.file, fp.read) ch.close_(fp) # Parse it. parser = lark.Lark(im.GRAMMAR_DOCKERFILE, parser="earley", propagate_positions=True, tree_class=im.Tree) # Avoid Lark issue #237: lark.exceptions.UnexpectedEOF if the file does not # end in newline. text += "\n" try: tree = parser.parse(text) except lark.exceptions.UnexpectedInput as x: ch.VERBOSE(x) # noise about what was expected in the grammar ch.FATAL("can’t parse: %s:%d,%d\n\n%s" % (cli.file, x.line, x.column, x.get_context(text, 39))) ch.VERBOSE(tree.pretty()[:-1]) # rm trailing newline # Sometimes we exit after parsing. if (cli.parse_only): ch.exit(0) # Count the number of stages (i.e., FROM instructions) global image_ct image_ct = sum(1 for i in tree.children_("from_")) # If we use RSYNC, error out quickly if appropriate rsync(1) not present. if (tree.child("rsync") is not None): try: ch.version_check(["rsync", "--version"], ch.RSYNC_MIN) except ch.Fatal_Error: ch.ERROR("Dockerfile uses RSYNC, so rsync(1) is required") raise # Traverse the tree and do what it says. # # We don’t actually care whether the tree is traversed breadth-first or # depth-first, but we *do* care that instruction nodes are visited in # order. Neither visit() nor visit_topdown() are documented as of # 2020-06-11 [1], but examining source code [2] shows that visit_topdown() # uses Tree.iter_trees_topdown(), which *is* documented to be in-order [3]. # # This change seems to have been made in 0.8.6 (see PR #761); before then, # visit() was in order. Therefore, we call that instead, if visit_topdown() # is not present, to improve compatibility (see issue #792). # # [1]: https://lark-parser.readthedocs.io/en/latest/visitors/#visitors # [2]: https://github.com/lark-parser/lark/blob/445c8d4/lark/visitors.py#L211 # [3]: https://lark-parser.readthedocs.io/en/latest/classes/#tree ml = Main_Loop() if (hasattr(ml, 'visit_topdown')): ml.visit_topdown(tree) else: ml.visit(tree) if (ml.instruction_total_ct > 0): if (ml.miss_ct == 0): ml.inst_prev.checkout() ml.inst_prev.ready() # Check that all build arguments were consumed. if (len(cli.build_arg) != 0): ch.FATAL("--build-arg: not consumed: " + " ".join(cli.build_arg.keys())) # Print summary & we’re done. if (ml.instruction_total_ct == 0): ch.FATAL("no instructions found: %s" % cli.file) assert (ml.inst_prev.image_i + 1 == image_ct) # should’ve errored already if ((cli.force != ch.Force_Mode.NONE) and ml.miss_ct != 0): ch.INFO("--force=%s: modified %d RUN instructions" % (cli.force.value, forcer.run_modified_ct)) ch.INFO("grown in %d instructions: %s" % (ml.instruction_total_ct, ml.inst_prev.image)) # FIXME: remove when we’re done encouraging people to use the build cache. if (isinstance(bu.cache, bu.Disabled_Cache)): ch.INFO("build slow? consider enabling the build cache", "https://hpc.github.io/charliecloud/command-usage.html#build-cache") ## Functions ## def unescape(sl): # FIXME: This is also ugly and should go in the grammar. # # The Dockerfile spec does not precisely define string escaping, but I’m # guessing it’s the Go rules. You will note that we are using Python rules. # This is wrong but close enough for now (see also gripe in previous # paragraph). if ( not sl.startswith('"') # no start quote and (not sl.endswith('"') or sl.endswith('\\"'))): # no end quote sl = '"%s"' % sl assert (len(sl) >= 2 and sl[0] == '"' and sl[-1] == '"' and sl[-2:] != '\\"') return ast.literal_eval(sl) ## Supporting classes ## class Instruction(abc.ABC): __slots__ = ("announced_p", "commit_files", # modified files; default: anything "git_hash", # Git commit where sid was found "image", "image_alias", "image_i", "lineno", "options", # consumed "options_str", # saved at instantiation "parent", "sid", "tree") def __init__(self, tree): """Note: When this is called, all we know about the instruction is what’s in the parse tree. In particular, you must not call ch.variables_sub() here.""" self.announced_p = False self.commit_files = set() self.git_hash = bu.GIT_HASH_UNKNOWN self.lineno = tree.meta.line self.options = dict() # saving options with only 1 saved value for st in tree.children_("option"): k = st.terminal("OPTION_KEY") v = st.terminal("OPTION_VALUE") if (k in self.options): ch.FATAL("%3d %s: repeated option --%s" % (self.lineno, self.str_name, k)) self.options[k] = v # saving keypair options in a dictionary for st in tree.children_("option_keypair"): k = st.terminal("OPTION_KEY") s = st.terminal("OPTION_VAR") v = st.terminal("OPTION_VALUE") # assuming all key pair options allow multiple options self.options.setdefault(k, {}).update({s: v}) ol = list() for (k, v) in self.options.items(): if (isinstance(v, dict)): for (k2, v) in v.items(): ol.append("--%s=%s=%s" % (k, k2, v)) else: ol.append("--%s=%s" % (k, v)) self.options_str = " ".join(ol) self.tree = tree # These are set in init(). self.image = None self.parent = None self.image_alias = None self.image_i = None def __str__(self): options = self.options_str if (options != ""): options = " " + options return "%s%s %s" % (self.str_name, options, self.str_) @property def env_arg(self): if (self.image is None): assert False, "unimplemented" # return dict() else: return self.image.metadata["arg"] @property def env_build(self): return { **self.env_arg, **self.env_env } @property def env_env(self): if (self.image is None): assert False, "unimplemented" # return dict() else: return self.image.metadata["env"] @property def miss(self): """This is actually a three-valued property: 1. True => miss 2. False => hit 3. None => unknown or n/a""" if (self.git_hash == bu.GIT_HASH_UNKNOWN): return None else: return (self.git_hash is None) @property def shell(self): if (self.image is None): assert False, "unimplemented" # return ["/bin/false"] else: return self.image.metadata["shell"] @shell.setter def shell(self, x): self.image.metadata["shell"] = x @property def sid_input(self): return str(self).encode("UTF-8") @property def status_char(self): return bu.cache.status_char(self.miss) @property @abc.abstractmethod def str_(self): ... @property def str_name(self): return self.__class__.__name__.split("_")[0].upper() @property def workdir(self): return fs.Path(self.image.metadata["cwd"]) @workdir.setter def workdir(self, x): self.image.metadata["cwd"] = str(x) def announce_maybe(self): "Announce myself if I haven’t already been announced." if (not self.announced_p): self_ = str(self) if (ch.user() == "qwofford" and sys.stderr.isatty()): self_ = re.sub(r"^RSYNC", "NSYNC", self_) ch.INFO("%3s%s %s" % (self.lineno, self.status_char, self_)) self.announced_p = True def chdir(self, path): if (path.is_absolute()): self.workdir = path else: self.workdir //= path def checkout(self, base_image=None): bu.cache.checkout(self.image, self.git_hash, base_image) def checkout_for_build(self, base_image=None): self.parent.checkout(base_image) global forcer forcer = force.new(self.image.unpack_path, cli.force, cli.force_cmd) def commit(self): path = self.image.unpack_path self.git_hash = bu.cache.commit(path, self.sid, str(self), self.commit_files) def execute(self): """Do what the instruction says. At this point, the unpack directory is all ready to go. Thus, the method is cache-ignorant.""" pass def init(self, parent): """Initialize attributes defining this instruction’s context, much of which is not available until the previous instruction is processed. After this is called, the instruction has a valid image and parent instruction, unless it’s the first instruction, in which case prepare() does the initialization.""" # Separate from prepare() because subclasses shouldn’t need to override # it. If a subclass doesn’t like the result, it can just change things # in prepare(). self.parent = parent if (self.parent is None): self.image_i = -1 else: self.image = self.parent.image self.image_alias = self.parent.image_alias self.image_i = self.parent.image_i def metadata_update(self): self.image.metadata["history"].append( { "created": ch.now_utc_iso8601(), "created_by": "%s %s" % (self.str_name, self.str_)}) self.image.metadata_save() def options_assert_empty(self): try: k = next(iter(self.options.keys())) ch.FATAL("%s: invalid option --%s" % (self.str_name, k)) except StopIteration: pass def prepare(self, miss_ct): """Set up for execution; parent is the parent instruction and miss_ct is the number of misses in this stage so far. Returns the new number of misses; usually miss_ct if this instruction hit or miss_ct + 1 if it missed. Some instructions (e.g., FROM) resets the miss count. Announce self as soon as hit/miss status is known, hopefully before doing anything complicated or time-consuming. Typically, subclasses will set up enough state for self.sid_input to be valid, then call super().prepare(). Gotchas: 1. Announcing the instruction: Subclasses that are fast can let the caller announce. However, subclasses that consume non-trivial time in prepare() should call announce_maybe() as soon as they know hit/miss status. 2. Errors: The caller catches Fatal_Error, announces, calls prepare_rollback(), and then re-raises. This to ensure the instruction is announced (see #1486) and any possibly-inconsistent state is fixed before existing. 3. Modifying image metadata: Instructions like ARG, ENV, FROM, LABEL, SHELL, and WORKDIR must modify metadata here, not in execute(), so it’s available to later instructions even on cache hit.""" self.sid = bu.cache.sid_from_parent(self.parent.sid, self.sid_input) self.git_hash = bu.cache.find_sid(self.sid, self.image.ref.for_path) return miss_ct + int(self.miss) def prepare_rollback(self): pass # typically a no-op def ready(self): bu.cache.ready(self.image) def rollback(self): """Discard everything done by execute(), which may have completed partially, fully, or not at all.""" bu.cache.rollback(self.image.unpack_path) def unsupported_forever_warn(self, msg): ch.WARNING("not supported, ignored: %s %s" % (self.str_name, msg)) def unsupported_yet_fatal(self, msg, issue_no): ch.FATAL("not yet supported: issue #%d: %s %s" % (issue_no, self.str_name, msg)) def unsupported_yet_warn(self, msg, issue_no): ch.WARNING("not yet supported, ignored: issue #%d: %s %s" % (issue_no, self.str_name, msg)) class Copy(Instruction): # Superclass for instructions that do some flavor of file copying (ADD, # COPY, RSYNC). __slots__ = ("dst", # string b/c trailing slash is significant "dst_raw", "from_", "src_metadata", "srcs", # strings b/c trailing slashes are significant "srcs_base", "srcs_raw") @property def sid_input(self): return super().sid_input + self.src_metadata def expand_dest(self): """Set self.dst from self.dst_raw with environment variables expanded and image root prepended.""" dst_raw = ch.variables_sub(self.dst_raw, self.env_build) if (len(dst_raw) < 1): ch.FATAL("destination is empty after expansion: %s" % self.dst_raw) base = self.image.unpack_path if (dst_raw[0] != "/"): base //= self.workdir self.dst = base // ch.variables_sub(self.dst_raw, self.env_build) def expand_sources(self): """Set self.srcs from self.srcs_raw with environment variables and globs expanded, absolute paths with appropriate base, and validate that they are within the sources base.""" if (cli.context == "-" and self.from_ is None): ch.FATAL("no context because “-” given") if (len(self.srcs_raw) < 1): ch.FATAL("source or destination missing") self.srcs_base_set() self.srcs = list() for src in (ch.variables_sub(i, self.env_build) for i in self.srcs_raw): # glob can’t take Path matches = sorted(fs.Path(i) for i in glob.glob("%s/%s" % (self.srcs_base, src))) if (len(matches) == 0): ch.FATAL("source not found: %s" % src) for m in matches: self.srcs.append(m) ch.VERBOSE("source: %s" % m) # Validate source is within context directory. (We need the source # as given later, so don’t canonicalize persistently.) There is no # clear subsitute for commonpath() in pathlib. mc = m.resolve() if (not os.path.commonpath([mc, self.srcs_base]) .startswith(self.srcs_base)): ch.FATAL("can’t copy from outside context: %s" % src) def srcs_base_set(self): "Set self.srcs_base according to context and --from." if (self.from_ is None): self.srcs_base = cli.context else: if (self.from_ == self.image_i or self.from_ == self.image_alias): ch.FATAL("--from: stage %s is the current stage" % self.from_) if (not self.from_ in images): # FIXME: Would be nice to also report if a named stage is below. if (isinstance(self.from_, int) and self.from_ < image_ct): if (self.from_ < 0): ch.FATAL("--from: invalid negative stage index %d" % self.from_) else: ch.FATAL("--from: stage %d does not exist yet" % self.from_) else: ch.FATAL("--from: stage %s does not exist" % self.from_) self.srcs_base = images[self.from_].unpack_path self.srcs_base = os.path.realpath(self.srcs_base) ch.VERBOSE("context: %s" % self.srcs_base) class Instruction_No_Image(Instruction): # This is a class for instructions that do not affect the image, i.e., # no-op from the image’s perspective, but executed for their side effects, # e.g., changing some configuration. These instructions do not interact # with the build cache and can be executed when no image exists (i.e., # before FROM). # FIXME: Only tested with instructions before the first FROM. I doubt it # works for instructions elsewhere. @property def miss(self): return True @property def status_char(self): return bu.cache.status_char(None) def checkout_for_build(self): pass def commit(self): pass def prepare(self, miss_ct): return miss_ct + int(self.miss) class Instruction_Unsupported(Instruction): __slots__ = () @property def miss(self): return None @property def str_(self): return "(unsupported)" class Instruction_Supported_Never(Instruction_Unsupported): __slots__ = () def prepare(self, *args): self.unsupported_forever_warn("instruction") raise Instruction_Ignored() ## Core classes ## class Arg(Instruction): __slots__ = ("key", "value") def __init__(self, *args): super().__init__(*args) self.commit_files.add(fs.Path("ch/metadata.json")) self.key = self.tree.terminal("WORD", 0) if (self.key in cli.build_arg): self.value = cli.build_arg[self.key] del cli.build_arg[self.key] else: self.value = self.value_default() @property def sid_input(self): if (self.key in im.ARGS_MAGIC): return (self.str_name + self.key).encode("UTF-8") else: return super().sid_input @property def str_(self): s = "%s=" % self.key if (self.value is not None): s += "'%s'" % self.value if (self.key in im.ARGS_MAGIC): s += " [special]" return s def prepare(self, *args): if (self.value is not None): self.value = ch.variables_sub(self.value, self.env_build) self.env_arg[self.key] = self.value return super().prepare(*args) class Arg_Bare_G(Arg): __slots__ = () def value_default(self): return None class Arg_Equals_G(Arg): __slots__ = () def value_default(self): v = self.tree.terminal("WORD", 1) if (v is None): v = unescape(self.tree.terminal("STRING_QUOTED")) return v class Arg_First(Instruction_No_Image): __slots__ = ("key", "value") def __init__(self, *args): super().__init__(*args) self.key = self.tree.terminal("WORD", 0) if (self.key in cli.build_arg): self.value = cli.build_arg[self.key] del cli.build_arg[self.key] else: self.value = self.value_default() @property def str_(self): s = "%s=" % self.key if (self.value is not None): s += "'%s'" % self.value if (self.key in im.ARGS_MAGIC): s += " [special]" return s def prepare(self, *args): if (self.value is not None): argfrom.update({self.key: self.value}) return super().prepare(*args) class Arg_First_Bare_G(Arg_First): __slots__ = () def value_default(self): return None class Arg_First_Equals_G(Arg_First): __slots__ = () def value_default(self): v = self.tree.terminal("WORD", 1) if (v is None): v = unescape(self.tree.terminal("STRING_QUOTED")) return v class Copy_G(Copy): # ABANDON ALL HOPE YE WHO ENTER HERE # # Note: The Dockerfile specification for COPY is complex, messy, # inexplicably different from cp(1), and incomplete. We try to be # bug-compatible with Docker (legacy builder, not BuildKit -- yes, they are # different) but probably are not 100%. See the FAQ. # # Because of these weird semantics, none of this abstracted into a general # copy function. I don’t want people calling it except from here. __slots__ = () def __init__(self, *args): super().__init__(*args) self.from_ = self.options.pop("from", None) if (self.from_ is not None): try: self.from_ = int(self.from_) except ValueError: pass # No subclasses, so check what parse tree matched. if (self.tree.child("copy_shell") is not None): args = list(self.tree.child_terminals("copy_shell", "WORD")) elif (self.tree.child("copy_list") is not None): args = list(self.tree.child_terminals("copy_list", "STRING_QUOTED")) for i in range(len(args)): args[i] = args[i][1:-1] # strip quotes else: assert False, "unreachable code reached" self.srcs_raw = args[:-1] self.dst_raw = args[-1] @property def str_(self): dst = repr(self.dst) if hasattr(self, "dst") else self.dst_raw return "%s -> %s" % (self.srcs_raw, dst) def copy_src_dir(self, src, dst): """Copy the contents of directory src, named by COPY, either explicitly or with wildcards, to dst. src might be a symlink, but dst is a canonical path. Both must be at the top level of the COPY instruction; i.e., this function must not be called recursively. dst must exist already and be a directory. Unlike subdirectories, the metadata of dst will not be altered to match src.""" def onerror(x): ch.FATAL("can’t scan directory: %s: %s" % (x.filename, x.strerror)) # Use Path objects in this method because the path arithmetic was # getting too hard with strings. src = src.resolve() # alternative to os.path.realpath() dst = fs.Path(dst) assert (src.is_dir() and not src.is_symlink()) assert (dst.is_dir() and not dst.is_symlink()) ch.DEBUG("copying named directory: %s -> %s" % (src, dst)) for (dirpath, dirnames, filenames) in ch.walk(src, onerror=onerror): subdir = dirpath.relative_to(src) dst_dir = dst // subdir # dirnames can contain symlinks, which we handle as files, so we’ll # rebuild it; the walk will not descend into those “directories”. dirnames2 = dirnames.copy() # shallow copy dirnames[:] = list() # clear in place for d in dirnames2: src_path = dirpath // d dst_path = dst_dir // d ch.TRACE("dir: %s -> %s" % (src_path, dst_path)) if (os.path.islink(src_path)): filenames.append(d) # symlink, handle as file ch.TRACE("symlink to dir, will handle as file") continue else: dirnames.append(d) # directory, descend into later # If destination exists, but isn’t a directory, remove it. if (os.path.exists(dst_path)): if (os.path.isdir(dst_path) and not os.path.islink(dst_path)): ch.TRACE("dst_path exists and is a directory") else: ch.TRACE("dst_path exists, not a directory, removing") dst_path.unlink() # If destination directory doesn’t exist, create it. if (not os.path.exists(dst_path)): ch.TRACE("mkdir dst_path") ch.ossafe("can’t mkdir: %s" % dst_path, os.mkdir, dst_path) # Copy metadata, now that we know the destination exists and is a # directory. ch.ossafe("can’t copy metadata: %s -> %s" % (src_path, dst_path), shutil.copystat, src_path, dst_path, follow_symlinks=False) for f in filenames: src_path = dirpath // f dst_path = dst_dir // f ch.TRACE("file or symlink via copy2: %s -> %s" % (src_path, dst_path)) if (not (os.path.isfile(src_path) or os.path.islink(src_path))): ch.FATAL("can’t COPY: unknown file type: %s" % src_path) if (os.path.exists(dst_path)): ch.TRACE("destination exists, removing") if (os.path.isdir(dst_path) and not os.path.islink(dst_path)): dst_path.rmtree() else: dst_path.unlink() src_path.copy(dst_path) def copy_src_file(self, src, dst): """Copy file src to dst. src might be a symlink, but dst is a canonical path. Both must be at the top level of the COPY instruction; i.e., this function must not be called recursively. dst has additional constraints: 1. If dst is a directory that exists, src will be copied into that directory like cp(1); e.g. “COPY file_ /dir_” will produce a file in the imaged called. “/dir_/file_”. 2. If dst is a regular file that exists, src will overwrite it. 3. If dst is another type of file that exists, that’s an error. 4. If dst does not exist, the parent of dst must be a directory that exists.""" assert (src.is_file()) assert (not dst.is_symlink()) assert ( (dst.exists() and (dst.is_dir() or dst.is_file())) or (not dst.exists() and dst.parent.is_dir())) if (dst.is_dir()): dst //= src.name src = src.resolve() ch.DEBUG("copying named file: %s -> %s" % (src, dst)) src.copy(dst) def dest_realpath(self, unpack_path, dst): """Return the canonicalized version of path dst within (canonical) image path unpack_path. We can’t use os.path.realpath() because if dst is an absolute symlink, we need to use the *image’s* root directory, not the host. Thus, we have to resolve symlinks manually.""" dst_canon = unpack_path dst_parts = list(reversed(dst.parts)) # easier to operate on end of list iter_ct = 0 while (len(dst_parts) > 0): iter_ct += 1 if (iter_ct > 100): # arbitrary ch.FATAL("can’t COPY: too many path components") ch.TRACE("current destination: %d %s" % (iter_ct, dst_canon)) #ch.TRACE("parts remaining: %s" % dst_parts) part = dst_parts.pop() if (part == "/" or part == "//"): # 3 or more slashes yields "/" ch.TRACE("skipping root") continue cand = dst_canon // part ch.TRACE("checking: %s" % cand) if (not cand.is_symlink()): ch.TRACE("not symlink") dst_canon = cand else: target = fs.Path(os.readlink(cand)) ch.TRACE("symlink to: %s" % target) assert (len(target.parts) > 0) # POSIX says no empty symlinks if (target.is_absolute()): ch.TRACE("absolute") dst_canon = fs.Path(unpack_path) else: ch.TRACE("relative") dst_parts.extend(reversed(target.parts)) return dst_canon def execute(self): # Locate the destination. unpack_canon = fs.Path(self.image.unpack_path).resolve() if (self.dst.startswith("/")): dst = fs.Path(self.dst) else: dst = self.workdir // self.dst ch.VERBOSE("destination, as given: %s" % dst) dst_canon = self.dest_realpath(unpack_canon, dst) # strips trailing slash ch.VERBOSE("destination, canonical: %s" % dst_canon) if (not os.path.commonpath([dst_canon, unpack_canon]) .startswith(str(unpack_canon))): ch.FATAL("can’t COPY: destination not in image: %s" % dst_canon) # Create the destination directory if needed. if ( self.dst.endswith("/") or len(self.srcs) > 1 or self.srcs[0].is_dir()): if (not dst_canon.exists()): dst_canon.mkdirs() elif (not dst_canon.is_dir()): # not symlink b/c realpath() ch.FATAL("can’t COPY: not a directory: %s" % dst_canon) if (dst_canon.parent.exists()): if (not dst_canon.parent.is_dir()): ch.FATAL("can’t COPY: not a directory: %s" % dst_canon.parent) else: dst_canon.parent.mkdirs() # Copy each source. for src in self.srcs: if (src.is_file()): self.copy_src_file(src, dst_canon) elif (src.is_dir()): self.copy_src_dir(src, dst_canon) else: ch.FATAL("can’t COPY: unknown file type: %s" % src) def prepare(self, miss_ct): # Complain about unsupported stuff. if (self.options.pop("chown", False)): self.unsupported_forever_warn("--chown") # Any remaining options are invalid. self.options_assert_empty() # Expand operands. self.expand_sources() self.dst = ch.variables_sub(self.dst_raw, self.env_build) # Gather metadata for hashing. self.src_metadata = fs.Path.stat_bytes_all(self.srcs) # Pass on to superclass. return super().prepare(miss_ct) class Directive_G(Instruction_Supported_Never): __slots__ = () @property def str_name(self): return "#%s" % self.tree.terminal("DIRECTIVE_NAME") def prepare(self, *args): ch.WARNING("not supported, ignored: parser directives") raise Instruction_Ignored() class Env(Instruction): __slots__ = ("key", "value") def __init__(self, *args): super().__init__(*args) self.commit_files |= {fs.Path("ch/environment"), fs.Path("ch/metadata.json")} @property def str_(self): return "%s='%s'" % (self.key, self.value) def execute(self): with (self.image.unpack_path // "/ch/environment").open("wt") as fp: for (k, v) in self.env_env.items(): print("%s=%s" % (k, v), file=fp) def prepare(self, *args): self.value = ch.variables_sub(unescape(self.value), self.env_build) self.env_env[self.key] = self.value return super().prepare(*args) class Env_Equals_G(Env): __slots__ = () def __init__(self, *args): super().__init__(*args) self.key = self.tree.terminal("WORD", 0) self.value = self.tree.terminal("WORD", 1) if (self.value is None): self.value = self.tree.terminal("STRING_QUOTED") class Env_Space_G(Env): __slots__ = () def __init__(self, *args): super().__init__(*args) self.key = self.tree.terminal("WORD") self.value = self.tree.terminals_cat("LINE_CHUNK") class From__G(Instruction): __slots__ = ("alias", "base_alias", "base_image", "base_text") # Not meaningful for FROM. sid_input = None def __init__(self, *args): super().__init__(*args) argfrom.update(self.options.pop("arg", {})) @property def str_(self): if (hasattr(self, "base_alias")): base_text = str(self.base_alias) elif (hasattr(self, "base_image")): base_text = str(self.base_image.ref) else: # Initialization failed, but we want to print *something*. base_text = self.base_text return base_text + ((" AS " + self.alias) if self.alias else "") def checkout_for_build(self): assert (isinstance(bu.cache, bu.Disabled_Cache)) super().checkout_for_build(self.base_image) def execute(self): # Everything happens in prepare(). pass def metadata_update(self, *args): # FROM doesn’t update metadata because it never misses when the cache is # enabled, so this would never be called, and we want disabled results # to be the same. In particular, FROM does not generate history entries. pass def prepare(self, miss_ct): # FROM is special because its preparation involves opening a new stage # and closing the previous if there was one. Because of this, the actual # parent is the last instruction of the base image. self.base_text = self.tree.child_terminals_cat("image_ref", "IMAGE_REF") self.alias = self.tree.child_terminal("from_alias", "IR_PATH_COMPONENT") # Validate instruction. if (self.options.pop("platform", False)): self.unsupported_yet_fatal("--platform", 778) self.options_assert_empty() # Update context. self.image_i += 1 self.image_alias = self.alias if (self.image_i == image_ct - 1): # Last image; use tag unchanged. tag = cli.tag elif (self.image_i > image_ct - 1): # Too many images! ch.FATAL("expected %d stages but found at least %d" % (image_ct, self.image_i + 1)) else: # Not last image; append stage index to tag. tag = "%s_stage%d" % (cli.tag, self.image_i) if self.base_text in images: # Is alias; store base_text as the “alias used” to target a previous # stage as the base. self.base_alias = self.base_text self.base_text = str(images[self.base_text].ref) self.base_image = im.Image(im.Reference(self.base_text, argfrom)) self.image = im.Image(im.Reference(tag)) images[self.image_i] = self.image if (self.image_alias is not None): images[self.image_alias] = self.image ch.VERBOSE("image path: %s" % self.image.unpack_path) # More error checking. if (str(self.image.ref) == str(self.base_image.ref)): ch.FATAL("output image ref same as FROM: %s" % self.base_image.ref) # Close previous stage if needed. In particular, we need the previous # stage’s image directory to exist because (a) we need to read its # metadata and (b) in case there’s a COPY later. Cache disabled will # already have the image directory and there is no notion of branch # “ready”, so do nothing in that case. if (self.image_i > 0 and not isinstance(bu.cache, bu.Disabled_Cache)): if (miss_ct == 0): # No previous miss already checked out the image. This will still # be fast most of the time since the correct branch is likely # checked out already. self.parent.checkout() self.parent.ready() # At this point any meaningful parent of FROM, e.g., previous stage, has # been closed; thus, act as own parent. self.parent = self # Pull base image if needed. This tells us hit/miss. (self.sid, self.git_hash) = bu.cache.find_image(self.base_image) unpack_no_git = ( self.base_image.unpack_exist_p and not self.base_image.unpack_cache_linked) # Announce (before we start pulling). self.announce_maybe() # FIXME: shouldn’t know or care whether build cache is enabled here. if (self.miss): if (unpack_no_git): # Use case is mostly images built by old ch-image still in storage. if (not isinstance(bu.cache, bu.Disabled_Cache)): ch.WARNING("base image only exists non-cached; adding to cache") (self.sid, self.git_hash) = bu.cache.adopt(self.base_image) else: (self.sid, self.git_hash) = bu.cache.pull_lazy(self.base_image, self.base_image.ref) elif (unpack_no_git): ch.WARNING("base image also exists non-cached; using cache") # Load metadata self.image.metadata_load(self.base_image) self.env_arg.update(argfrom) # from pre-FROM ARG # Done. return int(self.miss) # will still miss in disabled mode def prepare_rollback(self): # AFAICT the only thing that might be busted is the unpack directories # for either the base image or the image. We could probably be smarter # about this, but for now just delete them. try: base_image = self.base_image except AttributeError: base_image = None try: image = self.image except AttributeError: image = None if (base_image is not None or image is not None): ch.INFO("something went wrong, rolling back ...") if (base_image is not None): bu.cache.unpack_delete(base_image, missing_ok=True) if (image is not None): bu.cache.unpack_delete(image, missing_ok=True) class Label(Instruction): __slots__ = ("key", "value") def __init__(self, *args): super().__init__(*args) self.commit_files |= {ch.Path("ch/metadata.json")} @property def str_(self): return "%s='%s'" % (self.key, self.value) def prepare(self, *args): self.value = ch.variables_sub(unescape(self.value), self.env_build) self.image.metadata["labels"][self.key] = self.value return super().prepare(*args) class Label_Equals_G(Label): __slots__ = () def __init__(self, *args): super().__init__(*args) self.key = self.tree.terminal("WORD", 0) self.value = self.tree.terminal("WORD", 1) if (self.value is None): self.value = self.tree.terminal("STRING_QUOTED") class Label_Space_G(Label): __slots__ = () def __init__(self, *args): super().__init__(*args) self.key = self.tree.terminal("WORD") self.value = self.tree.terminals_cat("LINE_CHUNK") class Rsync_G(Copy): __slots__ = ("plus_option", "rsync_options") def __init__(self, *args): super().__init__(*args) self.from_ = None # not supported yet line_no = self.tree.meta.line st = self.tree.child("option_plus") self.plus_option = "l" if st is None else st.terminal("OPTION_LETTER") options_done = False self.rsync_options = list() self.srcs_raw = list() for word in self.tree.terminals("WORDE"): if (not options_done and word.startswith("-")): # Option. See assumption in docs that makes parsing a lot easier. if (word == "--"): # end of options options_done = True elif (word.startswith("--")): # long option self.rsync_options.append(word) else: # short option(s) if (len(word) == 1): ch.FATAL("RSYNC: %d: invalid argument: %s" % (line_no, word)) # Append options individually so we can process them more later. for m in re.finditer(r"[^=]=.*$|[^=]", word[1:]): self.rsync_options.append("-" + m[0]) continue # Not an option, so it must be a source or destination path. self.srcs_raw.append(word) if (len(self.srcs_raw) == 0): ch.FATAL("RSYNC: %d: source and destination missing" % line_no) self.dst_raw = self.srcs_raw.pop() @property def rsync_options_concise(self): "Return self.rsync_options with short options coalesced." # We don’t group short options with an argument even though we could # because it seems confusing, e.g. “-ab=c” vs. “-a -b=c”. def ship_out(): nonlocal group if (group != ""): ret.append(group) group = "" ret = list() group = "" for o in self.rsync_options: if (o.startswith("--")): # long option, not grouped ship_out() ret.append(o) elif (len(o) > 2): # short option with argument, not grouped ship_out() ret.append(o) else: # short option without argument, grouped if (group == ""): group = "-" group += o[1:] # add to group ship_out() return ret @property def str_(self): ret = list() if (self.plus_option is not None): ret.append("+" + self.plus_option) if (len(self.rsync_options_concise) > 0): ret += self.rsync_options_concise ret += self.srcs_raw ret.append(self.dst_raw) return " ".join(ret) def execute(self): plus_options = list() if (self.plus_option in "lmu"): # no action needed for +z # see man page for explanations plus_options = ["-@=-1", "-AHSXpr"] if (sys.stderr.isatty()): plus_options += ["--info=progress2"] if (self.plus_option == "l"): plus_options += ["-l", "--safe-links"] elif (self.plus_option == "u"): plus_options += ["-l", "--copy-unsafe-links"] ch.cmd(["rsync"] + plus_options + self.rsync_options_concise + self.srcs + [self.dst]) def expand_rsync_froms(self): for i in range(len(self.rsync_options)): o = self.rsync_options[i] m = re.search("^--([a-z]+)-from=(.+)$", o) if (m is not None): key = m[1] if (m[2] == "-"): ch.FATAL("--*-from: can’t use standard input") elif (":" in m[2]): ch.FATAL("--*-from: can’t use remote hosts (colon in path)") path = ch.Path(m[2]) if (path.is_absolute()): path = self.image.unpack_path // path else: path = self.srcs_base // path self.rsync_options[i] = "--%s-from=%s" % (key, path) def prepare(self, miss_ct): self.rsync_validate() # Expand operands. self.expand_sources() self.expand_dest() self.expand_rsync_froms() # Gather metadata for hashing. self.src_metadata = fs.Path.stat_bytes_all(self.srcs) # Pass on to superclass. return super().prepare(miss_ct) def rsync_validate(self): # Reject bad + options. if (self.plus_option not in ("mluz")): ch.FATAL("invalid plus option: %s" % self.plus_option) # Reject SSH and rsync transports. I *believe* simply the presence of # “:” (colon) in the filename triggers this behavior. for src in self.srcs_raw: if (":" in src): ch.FATAL("SSH and rsync transports not supported: %s" % src) # Reject bad flags. bad = { "--daemon", "-n", "--dry-run", "--remove-source-files" } for o in self.rsync_options: if (o in bad): ch.FATAL("disallowed option: %s" % o) class Run(Instruction): __slots__ = ("cmd") @property def str_name(self): # Can’t get this from the forcer object because it might not have been # initialized yet. if (cli.force == ch.Force_Mode.NONE): tag = ".N" elif (cli.force == ch.Force_Mode.FAKEROOT): # FIXME: This causes spurious misses because it adds the force tag to # *all* RUN instructions, not just those that actually were modified # (i.e, any RUN instruction will miss the equivalent RUN without # --force=fakeroot). But we don’t know know if an instruction needs # modifications until the result is checked out, which happens after # we check the cache. See issue #1339. tag = ".F" elif (cli.force == ch.Force_Mode.SECCOMP): tag = ".S" else: assert False, "unreachable code reached (force mode = %s)" % cli.force return super().str_name + tag def execute(self): rootfs = self.image.unpack_path cmd = forcer.run_modified(self.cmd, self.env_build) exit_code = ch.ch_run_modify(rootfs, cmd, self.env_build, self.workdir, cli.bind, forcer.ch_run_args, fail_ok=True) if (exit_code != 0): ch.FATAL("build failed: RUN command exited with %d" % exit_code) class Run_Exec_G(Run): __slots__ = () @property def str_(self): return json.dumps(self.cmd) # double quotes, shlex.quote is less verbose def prepare(self, *args): self.cmd = [ ch.variables_sub(unescape(i), self.env_build) for i in self.tree.terminals("STRING_QUOTED")] return super().prepare(*args) class Run_Shell_G(Run): # Note re. line continuations and whitespace: Whitespace before the # backslash is passed verbatim to the shell, while the newline and any # whitespace between the newline and baskslash are deleted. __slots__ = ("_str_") @property def str_(self): return self._str_ # can’t replace abstract property with attribute def prepare(self, *args): cmd = self.tree.terminals_cat("LINE_CHUNK") self.cmd = self.shell + [cmd] self._str_ = cmd return super().prepare(*args) class Shell_G(Instruction): def __init__(self, *args): super().__init__(*args) self.commit_files.add(fs.Path("ch/metadata.json")) @property def str_(self): return str(self.shell) def prepare(self, *args): self.shell = [ ch.variables_sub(unescape(i), self.env_build) for i in self.tree.terminals("STRING_QUOTED")] return super().prepare(*args) class Uns_Forever_G(Instruction_Supported_Never): __slots__ = ("name") def __init__(self, *args): super().__init__(*args) self.name = self.tree.terminal("UNS_FOREVER") @property def str_name(self): return self.name class Uns_Yet_G(Instruction_Unsupported): __slots__ = ("issue_no", "name") def __init__(self, *args): super().__init__(*args) self.name = self.tree.terminal("UNS_YET") self.issue_no = { "ADD": 782, "CMD": 780, "ENTRYPOINT": 780, "ONBUILD": 788 }[self.name] @property def str_name(self): return self.name def prepare(self, *args): self.unsupported_yet_warn("instruction", self.issue_no) raise Instruction_Ignored() class Workdir_G(Instruction): __slots__ = ("path") @property def str_(self): return str(self.path) def execute(self): (self.image.unpack_path // self.workdir).mkdirs() def prepare(self, *args): self.path = fs.Path(ch.variables_sub( self.tree.terminals_cat("LINE_CHUNK"), self.env_build)) self.chdir(self.path) return super().prepare(*args) charliecloud-0.37/lib/build_cache.py000066400000000000000000001724511457016721300174630ustar00rootroot00000000000000# Note on how we use Git: # # Git is extremely flexible and can be configured in many ways, including # various configuration files [1] as well as environment variables [2]. We do # our best to use a fully-isolated Git that brings along no external # configuration the user or system may have, by excluding configuration files # other than Charliecloud’s and clearing the environment. # # Another gotcha that is not (yet?) documented is $PATH. git(1) re-executes # itself in the same way that it was invoked; e.g., if you invoke it with # plain “git”, which looks up the binary in the path, then sub-invocations # will do the same and look up “git” again [3]. If somehow you use different # paths to find the outer and inner Git — which is easy to do accidentally # with subprocess — you can run a mixed-version Git, which is bad (see #1606). # We work around this by looking up git(1) once and then calling it by its # absolute path, with an empty environment including unset $PATH. # # Alternately, we could have sanitized the environment more carefully, passing # through $PATH and perhaps other variables. This seemed difficult to do # correctly (e.g., should we keep $LD_LIBRARY_PATH?), and it seemed unlikely # that Git would be executing programs other than Git that weren’t readily # available in the standard paths or using any non-standard features. # # [1]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration # [2]: https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables # [3]: https://lore.kernel.org/git/E7D87B07-C416-4A58-8726-CCDA0907AC66@lanl.gov/t/#u import configparser import datetime import glob import hashlib import itertools import os import pickle import re import shutil import stat import tempfile import textwrap import charliecloud as ch import filesystem as fs import image as im import pull ## Constants ## # Required versions. DOT_MIN = (2, 30, 1) GIT_MIN = (2, 28, 1) GIT2DOT_MIN = (0, 8, 3) # Git configuration. Note some of these are overridden in specific commands. # Documentation for these variables: https://git-scm.com/docs/git-config GIT_CONFIG = { # Do parallel checkouts with all cores. "checkout.workers": "-1", # Prioritize write speed over data safety; i.e., increase the risk of cache # corruption on system crash while (hopefully) decreasing write speed. "core.fsync": "none", # We want to keep access to commits on deleted branches so they are still # available for cache hits. This setting is necessary but not sufficient; # see branch_delete() below. "core.logAllRefUpdates": "true", # Try to maximize “git add” speed. "core.looseCompression": "0", # Enable incremental indexes [1]; it should speed things like “git add”. # [1]: https://git-scm.com/docs/git-update-index "core.splitIndex": "true", # “ctime” marks when the file *or its inode* was last changed. Twiddling # the various metadata will alter this, so Git shouldn’t use it when # deciding if a file may have changed. "core.trustctime": "false", # Enable the “untracked cache” [1], which saves directory mtimes to # eliminate the need to re-stat(2) in some cases. # [1]: https://git-scm.com/docs/git-update-index "core.untrackedCache": "true", # Quick-and-dirty results suggest that commit is not faster after garbage # collection, and checkout is actually a little faster if *not* garbage # collected. Therefore, it’s not a high priority to run garbage collection. # Further, I would assume garbaging a lot of files rather than a few gives # better opportunities for delta compression. Our most file-ful example # image is obspy at about 50K files. "gc.auto": "100000", # Leave packs larger than this alone during automatic GC. This is to avoid # excessive resource consumption during GC the user didn’t ask for. "gc.bigPackThreshold": "12G", # Anything unreachable from a named branch or the reflog is unavailable to # the build cache, so we may as well delete it immediately. However, there # might be a concurrent Git operation in progress, so don’t use “now”. "gc.pruneExpire": "12.hours.ago", # Use the newest index version, which does “simple pathname compression # that reduces index size by 30%-50% on large repositories” [1]. # [1]: https://git-scm.com/docs/git-update-index "index.version": "4", # Print logs in short format by default. This helps ensure consistency # across different systems and git versions. "log.decorate": "short", # States on the reflog are available to the build cache, but the default # prune time is 90 and 30 days respectively, which seems too long. #"gc.reflogExpire": "14.days.ago", # changed my mind # In some quick-and-dirty tests (see issue #1412), pack.compression=1 is # 50% faster than the default 6 at the cost of 6% more size, while # Compression 0 is twice as fast but also over twice the size; 9 doubles # the time with no space savings. 1 seems like the right balance. "pack.compression": "1", # These two are guesses based on the fact that HPC machines tend to have a # lot of memory and more caching is faster. "pack.deltaCacheLimit": "4096", "pack.deltaCacheSize": "1G", # These two are guesses based on [1] and its links, particularly [2]. # [1]: https://stackoverflow.com/questions/28720151 # [2]: https://web.archive.org/web/20170526024841/https://vcscompare.blogspot.com/2008/06/git-repack-parameters.html "pack.depth": "36", "pack.window": "24", # Our Git repo is purely local, so it doesn’t really matter who owns the # commits. Set these in case the user hasn’t configured them and they can’t # be guessed. (Issue #1535.) "user.email": "charlie@localhost", "user.name": "Charlie", # Always fail if Git doesn’t know who the user is, rather than guessing if # possible. Makes #1535 happen for everyone. "user.useConfigOnly": "true", } # Placeholder for Git hash values that are unknown. This deliberately does not # support str operations (e.g., indexing), so trying those will fail loudly. GIT_HASH_UNKNOWN = -1 ## Globals ## # The active build cache. cache = None # Path to DOT output (.gv and .pdf will be appended) dot_base = None # Absolute path of Git binary we’re using. git = None # Default path within image to metadata pickle. PICKLE_PATH = fs.Path("ch/git.pickle") ## Functions ## def have_deps(required=True): """Return True if dependencies for the build cache are present, False otherwise. Note this does not include the --dot debugging option; that checks its own dependencies when invoked. This function also figures out which Git to use and sets the appropriate variables.""" global git git = shutil.which("git") if (git is None): (ch.FATAL if required else ch.VERBOSE)("no git(1) found") return False # As of 2.34.1, we get: "git version 2.34.1\n". return ch.version_check([git, "--version"], GIT_MIN, required=required) def have_dot(): ch.version_check(["git2dot.py", "--version"], GIT2DOT_MIN) ch.version_check(["dot", "-V"], DOT_MIN) def init(cli): # At this point --bucache is what the user wanted, either directly or via # --no-cache. If it’s None, chose the right default; otherwise, try what # the user asked for and fail if we can’t do it. if (cli.bucache != ch.Build_Mode.DISABLED): ok = have_deps(False) if (cli.bucache is None): cli.bucache = ch.Build_Mode.ENABLED if ok else ch.Build_Mode.DISABLED ch.VERBOSE("using default build cache mode") if (cli.bucache != ch.Build_Mode.DISABLED and not ok): ch.FATAL("insufficient Git for build cache mode: %s" % cli.bucache.value) # Set cache appropriately. We could also do this with a factory method, but # that seems overkill. global cache if (cli.bucache == ch.Build_Mode.ENABLED): cache = Enabled_Cache(cli.cache_large) elif (cli.bucache == ch.Build_Mode.REBUILD): cache = Rebuild_Cache(cli.cache_large) elif (cli.bucache == ch.Build_Mode.DISABLED): cache = Disabled_Cache(cli.cache_large) else: assert False, "unreachable" ch.VERBOSE("build cache mode: %s" % cache) # DOT output path try: global dot_base dot_base = cli.dot except AttributeError: pass ## Supporting classes ## class File_Metadata: # This class holds metadata we care about for a file (in the general sense, # not necessarily a regular file), along with methods for dealing with that # metadata for the build cache. This includes re-creating some files from # their metadata, for files that Git can’t store or breaks. # # Importantly, this class must support un-pickling of old versions of # itself, to support existing build caches on upgrade. At present, we do # this attribute-by-attribute, without explicit versioning. We omit # __slots__ so old versions with since-deleted attributes can be unpickled. # # Note that ctime can’t be restored: https://unix.stackexchange.com/a/36105 # # Attributes corresponding directly to inode(7) fields (and pickled): # # atime_ns # mtime_ns # mode # # size .......... Size of the file in bytes. Note large file thresholds # vary between builds, so this attribute should not be # used to determine if a file is in large file storage. # WARNING: May be -1 if read from an old pickle. # # Other attributes stored in pickle: # # children ...... Insertion-ordered mapping from child names to their # File_Metadata objects. Empty if non-directory, or upon # object creation (i.e., caller must assemble the tree). # # dont_restore .. If True, file should not be restored (e.g., RPM # database cache files). # # large_name .... If non-None, file is stored out-of-band as a large file # with this name. This is a function of all the # attributes that make file large-unique, i.e., two files # with the same large_name are considered the same file. # # hardlink_to ... If non-None, file is a hard link to this other file, # stored as a Path object relative to the image root. # # xattrs ........ Extended attributes of the file, stored as a dictionary. # The keys take the form “namespace.name”, where # “namespace” is the namespace the xattr belongs to (e.g. # “user” or “system”) and “name” is the actual name of the # xattr. The value in the dictionary is the value assigned # to the xattr. # # Attributes not stored (recomputed on unpickle): # # image_root .... Absolute path to the image directory to which path is # relative. That is, image_root // path is the absolute # version of path. # # path .......... Path to the file within the image, relative to the # image root. Empty for the image root itself. For # example, an image’s “/bin/true” has path “bin/true”. # # path_abs ...... Absolute path to the file (under the host root). # # st ............ Stat object for the file. Absent after un-pickling. def __init__(self, image_root, path): # Note: Constructor not called during unpickle. self.image_root = image_root self.path = path self.path_abs = image_root // path self.st = self.path_abs.stat(False) self.stat_cache_update() self.children = dict() self.dont_restore = False self.hardlink_to = None self.large_name = None self.xattrs = dict() if ch.xattrs_save: for xattr in ch.ossafe("can’t list xattrs: %s" % self.path_abs, os.listxattr, self.path_abs, follow_symlinks=False): self.xattrs[xattr] = \ ch.ossafe(("can’t get xattr: %s: %s" % (self.path_abs, xattr)), os.getxattr, self.path_abs, xattr, follow_symlinks=False) @classmethod def git_prepare(class_, image_root, large_file_thresh, path=None, hardlinks=None): """Recursively walk the given image root, prepare it for a Git commit, and return the resulting File_Metadata tree describing it. This is mostly reversed by git_restore_walk(); anything not is noted. path is the path relative to image_root currently being examined; hardlinks is a dictionary used to track what hard link groups have been seen already. External callers should pass None. For each file, in this order: 1. Record file metadata, specifically mode and timestamps, because Git does not save metadata beyond a limited executable bit. (More may be saved in the future; see issue #1287.) This captures FIFOs (named pipes), which are ignored by Git. Exception: Sockets are ignored. Like FIFOs, sockets are ignored by Git, but there isn’t a meaningful way to re-create them. Their presence in a container image that is not in use, which this image shouldn’t be, most likely reflects a bug in something. We do print a warning in this case. 2. For directories, record the number of children. Git does not save empty directories, so this is used to re-create them. 3. For devices, exit with error. Such files should not appear in unprivileged container images, so their presence means something is wrong. 4. For non-directories with link count greater than 1 (i.e., hard links), do nothing when the first link is encountered, but second and subsequent links are deleted, to be restored on checkout. (Git splits multiply-linked files into separate files, and directories cannot be hard-linked [1].) 5. Files special due to their name: a. Names starting in “.git” are special to Git. Therefore, except at the root where “.git” files support the cache’s Git worktree, rename them to begin with “.weirdal_” instead. b. Files matching the pattern /var/lib/rpm/__db.* are Berkeley DB database support files used by RPM. Sometimes, something mishandles the last-modified dates on these files, fooling Git into thinking they have not been modified, and so they don’t get committed or restored, which confuses BDB/RPM. Fortunately, they can be safely deleted, and that’s a simple workaround, so we do it. See issue #1351. Return the File_Metadata tree, and if write is True, also save it in “ch/git.pickle”. [1]: https://en.wikipedia.org/wiki/Hard_link#Limitations""" # Setup. if (path is None): assert (hardlinks is None) path = fs.Path() hardlinks = dict() fm = class_(image_root, path) if (fm.path == im.GIT_DIR): # skip Git stuff at image root fm.dont_restore = True return fm # Ensure minimum permissions. Some tools like to make files without # necessary owner permissions, because root ignores the permissions bits # (CAP_DAC_OVERRIDE). See e.g. #1765. fm.st = fm.path_abs.chmod_min(fm.st) fm.stat_cache_update() # Validate file type and recurse if needed. (Don’t use os.walk() because # it’s iterative, and our algorithm is better expressed recursively.) if ( stat.S_ISREG(fm.mode) or stat.S_ISLNK(fm.mode) or stat.S_ISFIFO(fm.mode)): # RPM databases get corrupted. Easy fix is delete them. See #1351. if (path.match("var/lib/rpm/__db.*")): ch.VERBOSE("deleting, see issue #1351: %s" % path) fm.path_abs.unlink() fm.dont_restore = True return fm elif ( stat.S_ISSOCK(fm.mode)): ch.WARNING("socket in image, deleting: %s" % path) fm.path_abs.unlink() fm.dont_restore = True return fm elif ( stat.S_ISCHR(fm.mode) or stat.S_ISBLK(fm.mode)): ch.FATAL("device files invalid in image: %s" % path) elif ( stat.S_ISDIR(fm.mode)): entries = sorted(fm.path_abs.listdir()) for i in entries: # Recurse fm.children[i] = class_.git_prepare(image_root, large_file_thresh, path // i, hardlinks) else: ch.FATAL("unexpected file type in image: %x: %s" % (stat.IFMT(fm.mode), path)) # Deal with hard links (directories can’t be hard-linked). if (fm.st.st_nlink > 1 and not stat.S_ISDIR(fm.mode)): if ((fm.st.st_dev, fm.st.st_ino) in hardlinks): ch.DEBUG("hard link: deleting subsequent: %d %d %s" % (fm.st.st_dev, fm.st.st_ino, path)) fm.hardlink_to = hardlinks[(fm.st.st_dev, fm.st.st_ino)] fm.path_abs.unlink() return fm else: ch.DEBUG("hard link: recording first: %d %d %s" % (fm.st.st_dev, fm.st.st_ino, path)) hardlinks[(fm.st.st_dev, fm.st.st_ino)] = path # Deal with large files. This comparison is a little sloppy (no files # named “git.pickle” are large, not just the one in /ch), but it works # for now. if ( fm.size >= large_file_thresh and stat.S_ISREG(fm.mode) and fm.path.name != PICKLE_PATH.name and fm.hardlink_to is None): fm.large_name = fm.large_prepare() else: fm.large_name = None # Remove empty directories. Git will ignore them, including leaving them # there when switching the worktree to a different branch, which is bad. if (fm.empty_dir_p): fm.path_abs.rmdir() return fm # Remove FIFOs for the same reason. if (stat.S_ISFIFO(fm.mode)): fm.path_abs.unlink() return fm # Rename if necessary. if (not path.git_compatible_p): ch.DEBUG("renaming: %s -> %s" % (path, path.git_escaped)) fm.path_abs.rename(fm.path_abs.git_escaped) # Done. return fm @classmethod def unpickle(self, image_root, data=None): if (data is None): data = (image_root // PICKLE_PATH).file_read_all(text=False) fm_tree = pickle.loads(data) fm_tree.unpickle_fix(image_root, path=fs.Path(".")) return fm_tree def __getstate__(self): return { a:v for (a,v) in self.__dict__.items() if (a not in { "image_root", "path", "path_abs", "st" }) } @property def empty_dir_p(self): """True if I represent either an empty directory, or a directory that contains only children where empty_dir_p is true. E.g., the root of a directory tree containing only empty directories returns true.""" # In principle this could do a lot of recursion, but in practice I’m # guessing it’s not too much. if (not stat.S_ISDIR(self.mode)): return False # not a directory # True if no children (truly empty directory), or each child is unstored # or empty_dir_p (recursively empty directory tree). return all((c.unstored or c.empty_dir_p) for c in self.children.values()) @property def unstored(self): """True if I represent something not stored, either ignored by Git or deleted by us before committing.""" return ( stat.S_ISFIFO(self.mode) or stat.S_ISSOCK(self.mode) or self.large_name is not None or self.hardlink_to is not None) def get(self, path): "Return the File_Metadata object at path." fm = self for name in path.parts: fm = fm.children[name] return fm def git_restore(self, quick): #ch.TRACE(self.str_for_log()) # output is extreme even for TRACE? # Do-nothing case. Exclude RPM databases explicitly because old caches # can have them left over without being tagged don’t restore. if (self.dont_restore or self.path.match("var/lib/rpm/__db.*")): if (not quick and self.path != im.GIT_DIR): ch.INFO("ignoring un-restorable file: /%s" % self.path) return # Make sure I exist, and with the correct name. if (self.hardlink_to is not None): # This relies on prepare and restore having the same traversal order, # so the first (stored) link is always available by the time we get # to subsequent (unstored) links. target = self.image_root // self.hardlink_to ch.DEBUG("hard link: restoring: %s -> %s" % (self.path_abs, target)) ch.ossafe("can’t hardlink: %s -> %s" % (self.path_abs, target), os.link, target, self.path_abs, follow_symlinks=False) elif (self.large_name is not None): self.large_restore() elif (self.empty_dir_p): ch.ossafe("can’t mkdir: %s" % self.path, os.mkdir, self.path_abs) elif (stat.S_ISFIFO(self.mode)): ch.ossafe("can’t make FIFO: %s" % self.path, os.mkfifo, self.path_abs) elif (not self.path.git_compatible_p): self.path_abs.git_escaped.rename(self.path_abs) for (xattr, val) in self.xattrs.items(): self.path_abs.setxattr(xattr, val) # Recurse children. if (len(self.children) > 0): for child in self.children.values(): child.git_restore(quick) # Restore my metadata. if (( not quick # Git broke metadata or self.hardlink_to is not None # we just made the hardlink or stat.S_ISDIR(self.mode) # maybe just created or modified or stat.S_ISFIFO(self.mode)) # we just made the FIFO and not stat.S_ISLNK(self.mode)): # can’t not follow symlinks ch.ossafe("can’t restore times: %s" % self.path_abs, os.utime, self.path_abs, ns=(self.atime_ns, self.mtime_ns)) ch.ossafe("can’t restore mode: %s" % self.path_abs, os.chmod, self.path_abs, stat.S_IMODE(self.mode)) def large_name_get(self): "Return my name for use in large file storage." assert (self.size >= 0) h = hashlib.md5() for attr in ("mtime_ns", "mode", "size", "path"): h.update(bytes(repr(getattr(self, attr)).encode("UTF-8"))) # The digest is unique, but add an encoded path to aid debugging. return ( h.hexdigest() + "%" + str(self.path).replace("/", "%"))[:ch.FILENAME_MAX_CHARS] def large_names(self): "Return a set containing the large names of myself and all descendants." if (self.large_name is None): names = set() else: names = { self.large_name } for c in self.children.values(): names |= c.large_names() return names def large_prepare(self): """Move my file to large file storage, or delete it if it already exists, then return the appropriate large name.""" large_name = self.large_name_get() target = ch.storage.build_large_path(large_name) if (target.exists()): op = "found" self.path_abs.unlink() else: op = "moving to" self.path_abs.rename(target) ch.DEBUG("large file: %s: %s: %s" % (self.path, op, large_name)) return large_name def large_restore(self): "Restore large file from OOB storage." target = ch.storage.build_large_path(self.large_name) ch.DEBUG("large file: %s: copying: %s" % (self.path_abs, self.large_name)) fs.copy(target, self.path_abs) def pickle(self): (self.image_root // PICKLE_PATH) \ .file_write(pickle.dumps(self, protocol=4)) def stat_cache_update(self): for attr in ("atime_ns", "mtime_ns", "mode", "size"): setattr(self, attr, getattr(self.st, "st_" + attr)) def str_for_log(self): # Truncate reported time to seconds. fmt = "%Y-%m-%d.%H:%M:%S" mstr = datetime.datetime.fromtimestamp(self.mtime_ns // 1e9).strftime(fmt) astr = datetime.datetime.fromtimestamp(self.mtime_ns // 1e9).strftime(fmt) return ("%s%s %s [%d %d %s %s %s %s %s] %s %s" % (" " * len(self.path), stat.filemode(self.mode), self.path.name, self.size, len(self.children), "dont_restore" if self.dont_restore else "-", "empty_dir" if self.empty_dir_p else "-", "unstored" if self.unstored else "-", mstr, astr, self.hardlink_to if self.hardlink_to else "-", self.large_name if self.large_name else "-")) def unpickle_fix(self, image_root, path): "Does no I/O." # old: large_name, size, xattrs: no such attribute if (not (hasattr(self, "large_name"))): self.large_name = None if (not (hasattr(self, "size"))): self.size = -1 if (not (hasattr(self, "xattrs"))): self.xattrs = dict() # old: hardlink_to: stored as string if (isinstance(self.hardlink_to, str)): self.hardlink_to = fs.Path(self.hardlink_to) # old: children, name: just a list, and instances know their names if (isinstance(self.children, list)): children_new = dict() for c in self.children: children_new[c.name] = c delattr(c, "name") self.children = children_new # all: set non-stored attributes self.image_root = image_root self.path = path self.path_abs = image_root // path # recurse for (name, child) in self.children.items(): child.unpickle_fix(image_root, path // name) def update(self, path): "Recompute the File_Metadata object at path in the tree rooted by me." # FIXME: Can’t handle anything other than regular, non-large files that # don’t need renaming. assert (stat.S_ISDIR(self.mode) and len(path) >= 1) fm = self for name in path.parts[:-1]: fm = fm.children[name] fm.children[path.name] = self.__class__(self.image_root, path) assert (stat.S_ISREG(fm.children[path.name].mode)) class State_ID: __slots__ = ('id_') def __init__(self, id_): # Constructor should only be called internally, so verify id_ type. assert (isinstance(id_, bytes)) self.id_ = id_ @classmethod def from_parent(class_, psid, input_): """Return the State_ID corresponding to parent State_ID psid and data input_ describing the transition, which can be either bytes or str.""" h = hashlib.md5(psid.id_) if (isinstance(input_, str)): input_ = input_.encode("UTF-8") h.update(input_) return class_(h.digest()) @classmethod def from_text(class_, text): """The argument is stringified; then this string must end with 32 hex digits, possibly interspersed with other characters. Any preceding characters are ignored.""" text = re.sub(r"[^0-9A-Fa-f]", "", str(text))[-32:] if (len(text) < 32): ch.FATAL("state ID too short: %s" % text) try: b = bytes.fromhex(text) except ValueError: ch.FATAL("state ID: malformed hex: %s" % text); return class_(b) def __eq__(self, other): return self.id_ == other.id_ def __hash__(self): return hash(self.id_) def __str__(self): s = self.id_.hex().upper() return ":".join((s[0:4], s[4:8], s[8:16], s[16:24], s[24:32])) @property def short(self): return str(self)[:4] ## Core classes ## class Enabled_Cache: root_id = State_ID.from_text("4A6F:73C3:A9204361:7061626C:616E6361") __slots__ = ("bootstrap_ct", "file_metadata", "large_threshold") def __init__(self, large_threshold): self.bootstrap_ct = 0 self.large_threshold = large_threshold if (not os.path.isdir(self.root)): self.root.mkdir() ls = self.root.listdir() if (len(ls) == 0): self.bootstrap() # empty; initialize a new cache elif (not {"HEAD", "objects", "refs"} <= ls): # Non-empty but not an existing cache. # See: https://git-scm.com/docs/gitrepository-layout ch.FATAL("storage broken: not a build cache: %s" % self.root) else: self.configure() # updates config if needed self.worktrees_fix() @staticmethod def branch_name_ready(ref): return ref.for_path @staticmethod def branch_name_unready(ref): return ref.for_path + "#" @staticmethod def commit_hash_p(commit_ish): """Return True if commit_ish looks like a commit hash, False otherwise. Note this is a text-based heuristic only. It will return True for hashes that don’t exist in the repo, and false positives for branch/tag names that look like hashes.""" return (re.search(r"^[0-9a-f]{7,}$", commit_ish) is not None) def __str__(self): return ("enabled (large=%g)" % self.large_threshold) @property def root(self): return ch.storage.build_cache def adopt(self, img): self.worktree_adopt(img, "root") img.metadata_load() img.metadata_save() log = "IMPORT %s" % img.ref sid = self.sid_from_parent(self.root_id, log) gh = self.commit(img.unpack_path, sid, log, []) self.ready(img) return (sid, gh) def bootstrap(self): ch.INFO("initializing empty build cache") self.bootstrap_ct += 1 # Initialize bare repo. Don’t use wrapper because the build cache # doesn’t exist yet. ch.cmd_quiet([git, "init", "--bare", "-b", "root", self.root], env={}) self.configure() # Create empty root commit. This is done in a strange way with no real # working directory at all, because (1) cloning the bucache doesn’t # clone the config, which we care about, and (2) worktrees cannot be # used on empty repositories. # See: https://stackoverflow.com/a/29396902/396038 try: with tempfile.TemporaryDirectory(prefix="weirdal.") as td: env = { "GIT_DIR": self.root, "GIT_WORK_TREE": td, "GIT_INDEX_FILE": "%s/bootstrap-index" % td } self.git(["read-tree", "--empty"], env=env) # Note: complaints about empty commits go to stdout, not stderr. self.git(["commit", "--allow-empty", "-m", "ROOT\n\n%s" % self.root_id], env=env) except OSError as x: ch.FATAL("can’t create or delete temporary directory: %s: %s" % (x.filename, x.strerror)) def branch_delete(self, branch): """Delete branch branch if it exists; otherwise, do nothing. This removes only the branch ref; its commits remain until garbage collected.""" # Note: in a typical Git working directory, HEAD has followed the branch # around, so when we delete a branch ref *and necessarily its reflog # too*, that branch’s commits remain accessible via HEAD’s reflog until # the reflog entries expire. However, in our case, it’s the worktree # HEAD that did the following, and that reflog goes away when the # worktree is deleted, so the branch’s commits become inaccessible # immediately upon branch deletion. Here, the first “update-ref” # shenanigan logs the branch tip in the bare repo’s HEAD reflog, keeping # the commits accessible. The second puts HEAD back where it was. branches = [branch] if (self.ready_p(branch) and (self.cached_p(branch))): branches.append(self.unready_of(branch)) # Tag deleted branch. This is allows images to be recovered with # “undelete.” Note that the “-f” flag overwrites existing tags with the # same name, meaning we only track the most recently deleted branch. self.git(["tag", "-a", "-f", "&%s" % branch, branch, "-m", "''"]) for brnch in branches: if (self.git(["show-ref", "--quiet", "--heads", brnch], fail_ok=True).returncode == 0): # branch found head_old = self.git(["rev-parse", "HEAD"]).stdout.strip() self.git(["update-ref", "HEAD", brnch]) self.git(["update-ref", "HEAD", head_old]) self.git(["branch", "-D", brnch]) def branch_nocheckout(self, src_ref, dest): """Create ready branch for Ref src_ref pointing to dest, which can be either an Ref or a Git commit reference (as a string).""" if (isinstance(dest, im.Reference)): dest = self.branch_name_ready(dest) # Some versions of Git won’t let us update a branch that’s already # checked out, so detach that worktree if it exists. src_img = im.Image(src_ref) if (src_img.unpack_exist_p): self.git(["checkout", "--detach"], cwd=src_img.unpack_path) self.git(["branch", "-f", self.branch_name_ready(src_ref), dest]) def cached_p(self, git_id): """True iff image corresponding to “git_id” is in the cache.""" return self.find_commit(git_id)[1] != None def checkout(self, image, git_hash, base_image): # base_image used in other subclasses self.worktree_add(image, git_hash) self.git_restore(image.unpack_path, [], False) def checkout_ready(self, image, git_hash, base_image=None): """“checkout()” followed by “ready()” is an operation that appears several times throughout the code, so we wrap it here.""" self.checkout(image, git_hash, base_image) self.ready(image) def commit(self, path, sid, msg, files): # Commit image at path into the build cache. If set files is empty, any # and all image content may have been changed. Otherwise, assume files # lists the only files in the image that have changed. These must be # relative paths relative to the image root (i.e., path), may only be # regular files, and get none of the special handling we have for # general image content. # # WARNING: files must be empty for the first image commit. self.git_prepare(path, files) t = ch.Timer() if (len(files) == 0): git_files = ["-A"] else: git_files = list(files) + ["ch/git.pickle"] self.git(["add"] + git_files, cwd=path) t.log("prepared index") t = ch.Timer() self.git(["commit", "-q", "--allow-empty", "-m", "%s\n\n%s" % (msg, sid)], cwd=path) t.log("committed") # “git commit” does print the new commit’s hash without “-q”, but it # also prints every file commited, which is rather enormous for us. # Therefore, retrieve the hash separately. cp = self.git(["rev-parse", "--short", "HEAD"], cwd=path) git_hash = cp.stdout.strip() self.git_restore(path, files, True) return git_hash def commit_find_deleted(self, git_id): deleted = self.git(["log", "--format=%h%n%B", "-n", "1", "&%s" % git_id],fail_ok=True) if (deleted.returncode == 0): # Commit was previously deleted but is still cached. Get info. sid = State_ID.from_text(deleted.stdout) commit = deleted.stdout.split("\n", maxsplit=1)[0] else: sid = None commit = None return (sid, commit) def configure(self): # Configuration. path = self.root // "config" fp = path.open("r+") config = configparser.ConfigParser() config.read_file(fp, source=path) changed = False for (k, v) in GIT_CONFIG.items(): (s, k) = k.lower().split(".", maxsplit=1) if (config.get(s, k, fallback=None) != v): changed = True try: config.add_section(s) except configparser.DuplicateSectionError: pass config.set(s, k, v) if (changed): ch.VERBOSE("writing updated Git config") fp.seek(0) fp.truncate() ch.ossafe("can’t write Git config: %s" % path, config.write, fp) ch.close_(fp) # Ignore list entries: # # 1. Git has no default gitignore, but cancel any global gitignore rules # the user might have. https://stackoverflow.com/a/26681066 # # 2. The oddly-named GIT_DIR. # # It is easier to just write the list we want every time, rather than # trying to figure out if an update is needed. (self.root // "info/exclude").file_write("!*\n%s\n" % im.GIT_DIR) # Remove old .gitignore files from all commits. While there are nice # tools to do this (e.g. “git filter-repo”), we don’t want to depend on # an external tool. Thus, the options seem to be “filter-branch” or # “export” followed by “import”. Around half of the “filter-branch” man # page is devoted to explaining why not to use it, so we use the latter. # # NOTE: Without --reflog, “fast-export” will omit commits on unnamed # branches (i.e., accessible only via reflog), but with it, we get # “commit” instructions in the stream with no branch name, which # “fast-import” won’t accept. Therefore, we just delete those commits, # to prevent deleted .gitignore files from creeping back in. (I couldn’t # figure out how to fix this for “filter-branch” either, which I didn’t # want to use anyway for the reasons above.) if (ch.storage.bucache_needs_ignore_upgrade.exists()): ch.INFO("upgrading build cache to v6+, some cached data may be lost", "see release notes for v0.32") text = self.git(["fast-export", "--no-data", "--", "--all"], encoding="UTF-8").stdout #fs.Path("/tmp/old").file_write(text) # There is a bug in Git that loses files that become directories [1]. # We work around this by moving delete commands within each commit to # be first. This makes a number of assumptions about the output of # “fast-export” that are true only for us, e.g. that it’s all # line-based, including data like commit messages. # # [1]: https://lore.kernel.org/git/6486D136-23D8-4C90-AEDA-DD037A5CD2B5@lanl.gov/T/#t lines = text.split("\n") data_p = re.compile(r"^[DM] ") i = 0 while i < len(lines): if (data_p.search(lines[i])): j = i + 1 while (data_p.search(lines[j])): j += 1 lines[i:j] = sorted(lines[i:j], key=lambda x: x[0]) i = j - 1 i += 1 text = "\n".join(lines) text = re.sub(r"^(D|M [0-7]+ [0-9a-f]+) \.(git|weirdal_)ignore$", r"#\g<0>", text, flags=re.MULTILINE) #fs.Path("/tmp/new").file_write(text) self.git(["fast-import", "--force"], input=text) self.git(["reflog", "expire", "--all", "--expire=now"]) ch.storage.bucache_needs_ignore_upgrade.unlink() def find_commit(self, git_id): """Return (state ID, commit) of commit-ish git_id, or (None, None) if it doesn’t exist.""" # Note abbreviated commit hash %h is automatically long enough to avoid # collisions. cp = self.git(["log", "--format=%h%n%B", "-n", "1", git_id], fail_ok=True) if (cp.returncode == 0): # branch exists sid = State_ID.from_text(cp.stdout) commit = cp.stdout.split("\n", maxsplit=1)[0] else: sid = None commit = None ch.VERBOSE("commit-ish %s: %s %s" % (git_id, commit, sid)) return (sid, commit) def find_deleted_image(self, image): return self.commit_find_deleted(image.ref.for_path) def find_image(self, image): """Return (state ID, commit) of branch tip for image, or (None, None) if no such branch.""" return self.find_commit(image.ref.for_path) def find_sid(self, sid, branch): """Return the hash of the commit matching State_ID, or None if no such commit exists. First search branch, then if not found, the entire repo including commits not reachable from any branch.""" commit = self.find_sid_(sid, branch) if (commit is None): commit = self.find_sid_(sid) ch.VERBOSE("commit for %s: %s" % (sid, commit)) return commit def find_sid_(self, sid, branch=None): """Return the hash of the most recent commit matching State_ID sid, or None if no such commit exists. If branch is given, search only that branch; otherwise, search the entire repo, including commits not reachable from any branch.""" argv = ["log", "--grep", sid, "-F", "--format=%h", "-n", "1"] if (branch is not None): fail_ok = True argv += [branch] else: fail_ok = False argv += ["--all", "--reflog"] cp = self.git(argv, fail_ok=fail_ok) if (cp.returncode != 0 or len(cp.stdout) == 0): return None else: return cp.stdout.split(maxsplit=1)[0] def garbageinate(self): ch.INFO("collecting cache garbage") t = ch.Timer() # Expire the reflog with a recent time instead of now in case there is a # parallel Git operation in progress. self.git(["-c", "gc.bigPackthreshold=0", "-c", "gc.pruneExpire=now", "-c", "gc.reflogExpire=now", "gc"], quiet=False) t.log("collected garbage") t = ch.Timer() digests = self.git(["rev-list", "--all", "--reflog", "--date-order"]).stdout.split("\n") assert (digests[-1] == "") # trailing newline digests[-2:] = [] # discard root commit and trailing newline p = ch.Progress("enumerating large files", "commits", 1, len(digests)) larges_used = set() for d in digests: data = self.git(["show", "%s:%s" % (d, PICKLE_PATH)], encoding=None).stdout fm = File_Metadata.unpickle(fs.Path("/DUMMY"), data) larges_used |= fm.large_names() p.update(1) p.done() t.log("enumerated large files") t = ch.Timer() ch.INFO("found %d large files used; deleting others" % len(larges_used)) for l in ch.storage.build_large.listdir(): if (l not in larges_used): (ch.storage.build_large // l).unlink() t.log("deleted unused large files") def git(self, argv, cwd=None, quiet=True, *args, **kwargs): """Run the given git(1) command with appropriate environment and return the resulting CompletedProcess object. If cwd is None, run with CWD set to the build cache bare repo; otherwise, it must be the path to an unpacked image. If quiet is true, read Git’s stdout and return it in cp.stdout; otherwise, leave Git’s stdout unchanged. Any additional arguments are passed through to ch.cmd_stdout().""" if (cwd is None): cwd = self.root else: if ("env" not in kwargs): kwargs["env"] = dict() kwargs["env"].update({ "GIT_DIR": str(cwd // im.GIT_DIR), "GIT_WORK_TREE": str(cwd) }) return (ch.cmd_stdout if quiet else ch.cmd)([git] + argv, cwd=cwd, *args, **kwargs) def git_prepare(self, unpack_path, files, write=True): """Prepare unpack_path for Git operations (see File_Metadata.git_prepare() for lots of details). If files is None, regenerate self.file_metadata by walking the directory tree. Otherwise, update metadata only for files in files.""" t = ch.Timer() if (len(files) == 0): self.file_metadata = File_Metadata.git_prepare(unpack_path, self.large_threshold) else: for path in files: self.file_metadata.update(path) t.log("gathered file metadata") if (write): self.file_metadata.pickle() def git_restore(self, unpack_path, files, quick): """Opposite of git_prepare. If files is non-empty, only restore those files. If quick, assuming that unpack_path is unchanged since file_metadata was collected earlier in this process, rather than the directory being checked out from Git, i.e., only restore things that we broke in git_prepare() (e.g., renaming .git files), not things that Git breaks (e.g., file permissions). Otherwise (i.e., not quick), read the File_Metadata tree the pickled file and do a full restore. This method will dirty the Git working directory.""" t = ch.Timer() if (not quick): self.file_metadata = File_Metadata.unpickle(unpack_path) if (len(files) == 0): self.file_metadata.git_restore(quick) else: for path in files: self.file_metadata.get(path).git_restore(quick) t.log("restored file metadata (%s)" % ("quick" if quick else "full")) def pull_eager(self, img, src_ref, last_layer=None): """Pull image, always checking if the repository version is newer. This is the pull operation invoked from the command line.""" pullet = pull.Image_Puller(img, src_ref) pullet.download() # will use dlcache if appropriate dl_sid = self.sid_from_parent(self.root_id, pullet.sid_input) dl_git_hash = self.find_sid(dl_sid, img.ref.for_path) if (dl_git_hash is not None): # Downloaded image is in cache, check it out. ch.INFO("pulled image: found in build cache") # Remove tag for previously deleted branch, if it exists. self.tag_delete(img.ref.for_path, fail_ok=True) self.checkout_ready(img, dl_git_hash) else: # Unpack and commit downloaded image. This also creates the worktree. ch.INFO("pulled image: adding to build cache") self.pull_lazy(img, src_ref, last_layer, pullet) def pull_lazy(self, img, src_ref, last_layer=None, pullet=None): """Pull img from src_ref if it does not exist in the build cache, i.e., do not ask the registry if there is a newer version. This is the pull operation invoked by FROM. If pullet is not None, use that Image_Puller and do not download anything (i.e., assume Image_Puller.download() has already been called).""" if (pullet is None): # a young hen, especially one less than one year old pullet = pull.Image_Puller(img, src_ref) pullet.download() pullet.unpack(last_layer) sid = self.sid_from_parent(self.root_id, pullet.sid_input) pullet.done() self.worktree_adopt(img, "root") commit = self.commit(img.unpack_path, sid, "PULL %s" % src_ref, []) self.ready(img) if (img.ref != src_ref): self.branch_nocheckout(src_ref, img.ref) return (sid, commit) def ready(self, image): (_, git_hash) = self.find_deleted_image(image) if (not (git_hash is None)): self.tag_delete(image.ref.for_path) # Branch was deleted. self.git(["checkout", "-B", self.branch_name_ready(image.ref)], cwd=image.unpack_path) self.branch_delete(self.branch_name_unready(image.ref)) def ready_p(self, branch): return (not branch.endswith("#")) def reset(self): if (self.bootstrap_ct >= 1): ch.WARNING("not resetting brand-new cache") else: # Kill any Git garbage collection that may be running, to avoid race # conditions while deleting the cache (see issue #1406). Open # directly to avoid a TOCTOU race. pid_path = ch.storage.build_cache // "gc.pid" try: fp = open(pid_path, "rt", encoding="UTF-8") text = ch.ossafe("can’t read: %s" % pid_path, fp.read) pid = int(text.split()[0]) ch.INFO("stopping build cache garbage collection, PID %d" % pid) ch.kill_blocking(pid) ch.close_(fp) except FileNotFoundError: # no PID file, therefore no GC running pass except OSError as x: ch.FATAL("can’t open GC PID file: %s: %s" % (pid_path, x.strerror)) # Delete images that are worktrees referring back to the build cache. ch.INFO("deleting build cache") for d in ch.storage.unpack_base.listdir(): dotgit = ch.storage.unpack_base // d // im.GIT_DIR if (os.path.exists(dotgit)): ch.VERBOSE("deleting cached image: %s" % d) (ch.storage.unpack_base // d).rmtree() # Delete build cache. self.root.rmtree() ch.storage.build_large.rmtree() # Create new. self.root.mkdir() ch.storage.build_large.mkdir() self.bootstrap() def rollback(self, path): """Restore path to the last committed state, including both tracked and untracked files.""" ch.INFO("something went wrong, rolling back ...") self.git_prepare(path, [], write=False) t = ch.Timer() self.git(["reset", "--hard", "HEAD"], cwd=path, quiet=False) self.git(["clean", "-fdq"], cwd=path, quiet=False) t.log("reverted worktree") self.git_restore(path, [], False) def sid_from_parent(self, *args): # This lets us intercept the call and return None in disabled mode. return State_ID.from_parent(*args) def status_char(self, miss): "Return single character to indicate whether miss is true or not." if (miss is None): return " " elif (miss): return "." else: return "*" def summary_print(self): # state IDs msgs = self.git(["log", "--all", "--reflog", "--format=format:%b"]).stdout states = set() for msg in msgs.splitlines(): if (msg != ""): states.add(State_ID.from_text(msg)) # branches (FIXME: how to count unnamed branch tips?) image_ct = self.git(["branch", "--list"]).stdout.count("\n") # file count and size on disk (file_ct, byte_ct) = fs.Path(self.root).du() commit_ct = int(self.git(["rev-list", "--all", "--reflog", "--count"]).stdout) (file_ct, file_suffix) = ch.si_decimal(file_ct) (byte_ct, byte_suffix) = ch.si_binary_bytes(byte_ct) # print it print("named images: %5d" % image_ct) print("state IDs: %5d" % len(states)) print("large files: %5d" % len(ch.storage.build_large.listdir())) print("commits: %5d" % commit_ct) print("internal files: %5d %s" % (file_ct, file_suffix)) print("disk used: %5d %s" % (byte_ct, byte_suffix)) # some information directly from Git if (ch.log_level >= ch.Log_Level.VERBOSE): out = self.git(["count-objects", "-vH"]).stdout print("Git statistics:") print(textwrap.indent(out, " "), end="") out = (self.root // "config").file_read_all() print("Git config:") print(textwrap.indent(out, " "), end="") def tag_delete(self, tag, *args, **kwargs): """Delete specified git tag. Used for recovering deleted branches.""" return self.git(["tag", "-d", "&%s" % tag], *args, **kwargs) def tree_dot(self): have_dot() path_gv = fs.Path(dot_base + ".gv") path_pdf = fs.Path(dot_base + ".pdf") if (not path_gv.is_absolute()): path_gv = os.getcwd() // path_gv path_pdf = os.getcwd() // path_pdf ch.VERBOSE("writing %s" % path_gv) ch.cmd_quiet( ["git2dot.py", "--range", "--all --reflog --topo-order", "--font-name", "Nimbus Mono", "-d", 'graph[rankdir="TB", bgcolor="white"]', "-d", 'edge[dir=forward, arrowsize=0.5]', "--bedge", '[color=gray80, dir=none]', "--bnode", '[label="{label}", shape=box, height=0.20, color=gray80]', "--cnode", '[label="{label}", shape=box, color=black, fillcolor=white]', "--mnode", '[label="{label}", shape=box, color=black, fillcolor=white]', "-D", "@SID@", "([0-9A-F]{4}):", "-l", "@SID@|%s", str(path_gv)], cwd=self.root) ch.VERBOSE("writing %s" % path_pdf) ch.cmd_quiet(["dot", "-Tpdf", "-o%s" % path_pdf, str(path_gv)]) def tree_print(self): # Note the percent codes are interpreted by Git. # See: https://git-scm.com/docs/git-log#_pretty_formats args = ["log", "--graph", "--all", "--reflog", "--topo-order"] if (ch.log_level == ch.Log_Level.INFO): # ref names, subject (instruction), branch heads. fmt = "%C(auto)%d %Creset%<|(77,trunc)%s" args.append("--decorate-refs=refs/heads") else: # ref names, short commit hash, subject (instruction), body (state ID) # FIXME: The body contains a trailing newline I can’t figure out how # to remove. fmt = "%C(auto)%d%C(yellow) %h %Creset%s %b" self.git(args + ["--format=%s" % fmt], quiet=False) print() # blank line to separate from summary def unpack_delete(self, image, missing_ok=False): """Wrapper for Image.unpack_delete() that first detaches the work tree's head. If we delete an image's unpack path without first detaching HEAD, the corresponding work tree must also be deleted before the bucache branch. This involves multiple calls to worktrees_fix(), which is clunky, so we use this method instead.""" if (not image.unpack_exist_p and missing_ok): return (_, commit) = self.find_commit(image.ref.for_path) if (commit is not None): # Off with her head! self.git(["checkout", "%s" % commit], cwd=image.unpack_path) image.unpack_delete() def unready_of(self, branch): if (self.ready_p(branch)): return branch + "#" else: return branch def worktree_add(self, image, base): if (image.unpack_cache_linked): self.git_prepare(image.unpack_path, [], write=False) # clean worktree if (self.commit_hash_p(base) and base == self.worktree_head(image)): ch.VERBOSE("already checked out: %s %s" % (image.unpack_path, base)) else: ch.INFO("updating existing image ...") t = ch.Timer() self.git(["checkout", "-B", self.branch_name_unready(image.ref), base], cwd=image.unpack_path) t.log("adjusted worktree") else: ch.INFO("copying image from cache ...") image.unpack_clear() t = ch.Timer() self.git(["worktree", "add", "-f", "-B", self.branch_name_unready(image.ref), image.unpack_path, base]) # Move GIT_DIR from default location to where we want it. git_dir_default = image.unpack_path // ".git" git_dir_new = image.unpack_path // im.GIT_DIR git_dir_new.parent.mkdir() git_dir_default.rename(git_dir_new) t.log("created worktree") def worktree_adopt(self, image, base): """Create a new worktree with the contents of existing directory image.unpack_path. Note shenanigans because “git worktree add” *cannot* use an existing directory but shutil.copytree *must* create its own directory (until Python 3.8, and we have to support 3.6). So we use some renaming.""" if (os.path.isdir(ch.storage.image_tmp)): ch.WARNING("temporary image still exists, deleting", "maybe a previous command crashed?") ch.storage.image_tmp.rmtree() image.unpack_path.rename(ch.storage.image_tmp) self.worktree_add(image, base) (image.unpack_path // im.GIT_DIR).rename( ch.storage.image_tmp // im.GIT_DIR) image.unpack_path.rmtree() ch.storage.image_tmp.rename(image.unpack_path) def worktree_head(self, image): cp = self.git(["rev-parse", "--short", "HEAD"], fail_ok=True, cwd=image.unpack_path) if (cp.returncode != 0): return None else: return cp.stdout.strip() def worktrees_fix(self): """Git stores pointers (paths) both from the main repository to each worktree, and in the other direction from each worktree back to the main repo. These are absolute paths, so if the storage directory gets moved, they need updating. Also, worktrees can disappear without telling Git. This method cleans all that up. This method does roughly the same thing as “git worktree repair” and “git worktree prune”, but we do it manually [1,2] because we know more about what is going on than Git does: (1) which images are worktrees vs. plain directories; (2) which images changed from worktree to plain directory w/o telling Git; (3) where the worktrees and main repo are relative to one another. In particular, I don’t see a simple way to trust the exit code of “git worktree repair” without doing most of this work first anyway. [1]: https://git-scm.com/docs/git-worktree [2]: https://git-scm.com/docs/gitrepository-layout""" t = ch.Timer() wt_actuals = { fs.Path(i).parts[-(len(im.GIT_DIR)+1)] for i in glob.iglob(str( ch.storage.unpack_base // "*" // im.GIT_DIR)) } wt_gits = { fs.Path(i).name for i in glob.iglob("%s/worktrees/*" % self.root) } # Unlink images that think they are in Git but are not. This should not # happen, but it does, and I wasn’t able to figure out how it happened. wt_gits_orphaned = wt_actuals - wt_gits for img_dir in wt_gits_orphaned: link = ch.storage.unpack_base // img_dir // im.GIT_DIR ch.WARNING("image erroneously marked cached, fixing: %s" % link, ch.BUG_REPORT_PLZ) link.unlink() wt_actuals -= wt_gits_orphaned # Delete worktree data for images that no longer exist or aren’t # Git-enabled any more. wt_gits_deleted = wt_gits - wt_actuals for wt in wt_gits_deleted: (ch.storage.build_cache // "worktrees" // wt).rmtree() ch.VERBOSE("deleted %d stale worktree metadatas" % len(wt_gits_deleted)) wt_gits -= wt_gits_deleted # Validate that the pointers are in sync now. if (wt_gits != wt_actuals): ch.ERROR("found images -> cache links: %s" % " ".join(wt_actuals)) ch.ERROR("found cache -> images links: %s" % " ".join(wt_gits)) ch.FATAL("build cache is desynchronized, cannot proceed", ch.BUG_REPORT_PLZ) # If storage directory moved, repair all the paths. if (len(wt_gits) > 0): wt_dir_stored = fs.Path(( ch.storage.build_cache // "worktrees" // next(iter(wt_gits)) // "gitdir").file_read_all()) if (not wt_dir_stored.is_relative_to(ch.storage.root)): for wt in wt_actuals: wt_repo_dir = ch.storage.build_cache // "worktrees" // wt wt_img_git = ch.storage.unpack_base // wt // im.GIT_DIR wt_img_git.file_write("gitdir: %s\n" % str(wt_repo_dir)) (wt_repo_dir // "gitdir").file_write(str(wt_img_git) + "\n") ch.VERBOSE("fixed %d worktrees" % len(wt_actuals)) t.log("re-linked worktrees") class Rebuild_Cache(Enabled_Cache): def __str__(self): return ("rebuild (large=%g)" % self.large_threshold) def find_sid(self, sid, branch): return None class Disabled_Cache(Rebuild_Cache): def __init__(self, *args): pass def __str__(self): return "disabled" def checkout(self, image, git_hash, base_image): ch.INFO("copying image ...") image.unpack_clear() image.copy_unpacked(base_image) def commit(self, path, *args): self.permissions_fix(path) return None def find_image(self, *args): return (None, None) def permissions_fix(self, path): # Some distributions create unreadable files; e.g., CentOS 7 after # installing “openssh”: # # $ ls -lh /scratch/reidpr.ch/img/centos_7ch/usr/bin/ssh-agent # ---x--s--x 1 reidpr reidpr 374K Nov 24 2021 [...]/ssh-agent # # This makes the image un-copyable, so it can’t be used as a base image. # # Enabled_Cache takes care of this in git_prepare(), and # --force=fakeroot bypasses it in some other way I haven’t looked into. for (dir_, subdirs, files) in ch.walk(path): for i in itertools.chain(subdirs, files): (dir_ // i).chmod_min() def pull_lazy(self, img, src_ref, last_layer=None, pullet=None): if (pullet is None and os.path.exists(img.unpack_path)): ch.VERBOSE("base image already exists, skipping pull") else: if (pullet is None): pullet = pull.Image_Puller(img, src_ref) pullet.download() img.unpack_clear() pullet.unpack(last_layer) pullet.done() return (None, None) def ready(self, *args): pass def rollback(self, path): self.permissions_fix(path) def sid_from_parent(self, *args): return None def worktree_add(self, *args): pass def worktree_adopt(self, *args): pass def worktrees_prune(self, *args): pass charliecloud-0.37/lib/charliecloud.py000066400000000000000000001004751457016721300176740ustar00rootroot00000000000000import argparse import atexit import cProfile import collections import collections.abc import datetime import enum import functools import hashlib import io import locale import os import platform import pstats import re import shlex import shutil import signal import subprocess import sys import time import traceback import warnings # List of dependency problems. This variable needs to be created before we # import any other Charliecloud stuff to avoid #806. depfails = [] # 👻 import filesystem as fs import registry as rg import version ## Enums ## # Build cache mode. class Build_Mode(enum.Enum): ENABLED = "enabled" DISABLED = "disabled" REBUILD = "rebuild" # Download cache mode. class Download_Mode(enum.Enum): ENABLED = "enabled" WRITE_ONLY = "write-only" # Root emulation mode class Force_Mode(enum.Enum): FAKEROOT = "fakeroot" SECCOMP = "seccomp" NONE = "none" # Log level @functools.total_ordering class Log_Level(enum.Enum): TRACE = 3 DEBUG = 2 VERBOSE = 1 INFO = 0 WARNING = -1 STDERR = -2 QUIET_STDERR = -3 # To support comparisons, we need to define at least one “ordering” # operator. See: https://stackoverflow.com/a/39269589 def __lt__(self, other): if self.__class__ is other.__class__: return self.value < other.value return NotImplemented ## Constants ## # Architectures. This maps the “machine” field returned by uname(2), also # available as "uname -m" and platform.machine(), into architecture names that # image registries use. It is incomplete (see e.g. [1], which is itself # incomplete) but hopefully includes most architectures encountered in # practice [e.g. 2]. Registry architecture and variant are separated by a # slash. Note it is *not* 1-to-1: multiple uname(2) architectures map to the # same registry architecture. # # [1]: https://stackoverflow.com/a/45125525 # [2]: https://github.com/docker-library/bashbrew/blob/v0.1.0/vendor/github.com/docker-library/go-dockerlibrary/architecture/oci-platform.go ARCH_MAP = { "armv5l": "arm/v5", "armv6l": "arm/v6", "aarch32": "arm/v7", "armv7l": "arm/v7", "aarch64": "arm64/v8", "armv8l": "arm64/v8", "i386": "386", "i686": "386", "mips64le": "mips64le", "ppc64le": "ppc64le", "s390x": "s390x", # a.k.a. IBM Z "x86_64": "amd64" } # Some images have oddly specified architecture. For example, as of # 2022-06-08, on Docker Hub, opensuse/leap:15.1 offers architectures amd64, # arm/v7, arm64/v8, and ppc64le, while opensuse/leap:15.2 offers amd64, arm, # arm64, and ppc64le, i.e., with no variants. This maps architectures to a # sequence of fallback architectures that we hope are equivalent. See class # Arch_Dict below. ARCH_MAP_FALLBACK = { "arm/v7": ("arm",), "arm64/v8": ("arm64",) } # Incompatible option pairs for the ch-image command line CLI_INCOMPATIBLE_OPTS = [("quiet", "verbose"), ("xattrs", "no_xattrs")] # String to use as hint when we throw an error that suggests a bug. BUG_REPORT_PLZ = "please report this bug: https://github.com/hpc/charliecloud/issues" # Maximum filename (path component) length, in *characters*. All Linux # filesystems of note that I could identify support at least 255 *bytes*. The # problem is filenames with multi-byte characters: you cannot simply truncate # byte-wise because you might do so in the middle of a character. So this is a # somewhat random guess with hopefully enough headroom not to cause problems. FILENAME_MAX_CHARS = 192 # Chunk size in bytes when streaming HTTP. Progress meter is updated once per # chunk, which means the display is updated roughly every 20s at 100 Kbit/s # and every 2s at 1Mbit/s; beyond that, the once-per-second display throttling # takes over. HTTP_CHUNK_SIZE = 256 * 1024 # Minimum versions. NOTE: Keep in sync with configure.ac. PYTHON_MIN = (3,6) RSYNC_MIN = (3,1,0) ## Globals ## # Compatibility link. Sometimes we load pickled data from when Path was # defined in this file. This alias lets us still load such pickles. Path = fs.Path # Active architecture (both using registry vocabulary) arch = None # requested by user arch_host = None # of host # FIXME: currently set in ch-image :P CH_BIN = None CH_RUN = None # Logging; set using init() below. log_level = Log_Level(0) # Verbosity level. log_festoon = False # If true, prepend pid and timestamp to chatter. log_fp = sys.stderr # File object to print logs to. trace_fatal = False # Add abbreviated traceback to fatal error hint. # Warnings to be re-printed when program exits warns = list() # True if the download cache is enabled. dlcache_p = None # Profiling. profiling = False profile = None # Width of terminal. term_width = shutil.get_terminal_size(fallback=(sys.maxsize, -1))[0] ## Exceptions ## class Fatal_Error(Exception): def __init__(self, *args, **kwargs): self.args = args self.kwargs = kwargs class Image_Unavailable_Error(Exception): pass class No_Fatman_Error(Exception): pass ## Classes ## class Arch_Dict(collections.UserDict): """Dictionary that overloads subscript and “in” to consider ARCH_MAP_FALLBACK.""" def __contains__(self, k): # “in” operator if (k in self.data): return True try: return self._fallback_key(k) in self.data except KeyError: return False def __getitem__(self, k): try: return self.data.__getitem__(k) except KeyError: return self.data.__getitem__(self._fallback_key(k)) def _fallback_key(self, k): """Return fallback key corresponding to key k, or raise KeyError if there is no fallback.""" assert (k not in self.data) if (k not in ARCH_MAP_FALLBACK): raise KeyError("no fallbacks: %s" % k) for f in ARCH_MAP_FALLBACK[k]: if (f in self.data): return f raise KeyError("fallbacks also missing: %s" % k) def in_warn(self, k): """Return True if k in self, False otherwise, just like the “in“ operator, but also log a warning if fallback is used.""" result = k in self if (result and k not in self.data): WARNING("arch %s requested but falling back to %s" % (k, self._fallback_key(k))) return result class ArgumentParser(argparse.ArgumentParser): class HelpFormatter(argparse.HelpFormatter): # Suppress duplicate metavar printing when option has both short and # long flavors. E.g., instead of: # # -s DIR, --storage DIR set builder internal storage directory to DIR # # print: # # -s, --storage DIR set builder internal storage directory to DIR # # From https://stackoverflow.com/a/31124505. def _format_action_invocation(self, action): if (not action.option_strings or action.nargs == 0): return super()._format_action_invocation(action) default = self._get_default_metavar_for_optional(action) args_string = self._format_args(action, default) return ', '.join(action.option_strings) + ' ' + args_string def __init__(self, sub_title=None, sub_metavar=None, *args, **kwargs): super().__init__(formatter_class=self.HelpFormatter, *args, **kwargs) self._optionals.title = "options" # https://stackoverflow.com/a/16981688 if (sub_title is not None): self.subs = self.add_subparsers(title=sub_title, metavar=sub_metavar) def add_parser(self, title, desc, *args, **kwargs): return self.subs.add_parser(title, help=desc, description=desc, *args, **kwargs) def parse_args(self, *args, **kwargs): cli = super().parse_args(*args, **kwargs) if (not hasattr(cli, "func")): self.error("CMD not specified") # Bring in environment variables that set options. if (cli.bucache is None and "CH_IMAGE_CACHE" in os.environ): try: cli.bucache = Build_Mode(os.environ["CH_IMAGE_CACHE"]) except ValueError: FATAL("$CH_IMAGE_CACHE: invalid build cache mode: %s" % os.environ["CH_IMAGE_CACHE"]) return cli class OrderedSet(collections.abc.MutableSet): # Note: The superclass provides basic implementations of all the other # methods. I didn’t evaluate any of these. __slots__ = ("data",) def __init__(self, others=None): self.data = collections.OrderedDict() if (others is not None): self.data.update((i, None) for i in others) def __contains__(self, item): return (item in self.data) def __iter__(self): return iter(self.data.keys()) def __len__(self): return len(self.data) def __repr__(self): return "%s(%s)" % (self.__class__.__name__, list(iter(self))) def add(self, x): self.data[x] = None def clear(self): # Superclass provides an implementation but warns it’s slow (and it is). self.data.clear() def discard(self, x): self.data.pop(x, None) class Progress: """Simple progress meter for countable things that updates at most once per second. Writes first update upon creation. If length is None, then just count up (this is for registries like Red Hat that sometimes don’t provide a Content-Length header for blobs). The purpose of the divisor is to allow counting things that are much more numerous than what we want to display; for example, to count bytes but report MiB, use a divisor of 1048576. By default, moves to a new line at first update, then assumes exclusive control of this line in the terminal, rewriting the line as needed. If output is not a TTY or global log_festoon is set, each update is one log entry with no overwriting.""" __slots__ = ("display_last", "divisor", "msg", "length", "unit", "overwrite_p", "precision", "progress") def __init__(self, msg, unit, divisor, length): self.msg = msg self.unit = unit self.divisor = divisor self.length = length if (not os.isatty(log_fp.fileno()) or log_festoon): self.overwrite_p = False # updates all use same line else: self.overwrite_p = True # each update on new line self.precision = 1 if self.divisor >= 1000 else 0 self.progress = 0 self.display_last = float("-inf") self.update(0) def done(self): self.update(0, True) if (self.overwrite_p): INFO("") # newline to release display line def update(self, increment, last=False): now = time.monotonic() self.progress += increment if (last or now - self.display_last > 1): if (self.length is None): line = ("%s: %.*f %s" % (self.msg, self.precision, self.progress / self.divisor, self.unit)) else: ct = "%.*f/%.*f" % (self.precision, self.progress / self.divisor, self.precision, self.length / self.divisor) pct = "%d%%" % (100 * self.progress / self.length) if (ct == "0.0/0.0"): # too small, don’t print count line = "%s: %s" % (self.msg, pct) else: line = ("%s: %s %s (%s)" % (self.msg, ct, self.unit, pct)) INFO(line, end=("\r" if self.overwrite_p else "\n")) self.display_last = now class Progress_Reader: """Wrapper around a binary file object to maintain a progress meter while reading.""" __slots__ = ("fp", "msg", "progress") def __init__(self, fp, msg): self.fp = fp self.msg = msg self.progress = None def __iter__(self): return self def __next__(self): data = self.read(HTTP_CHUNK_SIZE) if (len(data) == 0): raise StopIteration return data def close(self): if (self.progress is not None): self.progress.done() close_(self.fp) def read(self, size=-1): data = ossafe("can’t read: %s" % self.fp.name, self.fp.read, size) self.progress.update(len(data)) return data def seek(self, *args): raise io.UnsupportedOperation def start(self): # Get file size. This seems awkward, but I wasn’t able to find anything # better. See: https://stackoverflow.com/questions/283707 old_pos = self.fp.tell() assert (old_pos == 0) # math will be wrong if this isn’t true length = self.fp.seek(0, os.SEEK_END) self.fp.seek(old_pos) self.progress = Progress(self.msg, "MiB", 2**20, length) class Progress_Writer: """Wrapper around a binary file object to maintain a progress meter while data are written. Overwrite the file if it already exists. This downloads to a temporary file to ease recovery if the download is interrupted. This uses a predictable name to support restarts in the future, which would probably require verification after download. For now, we just delete any leftover temporary files in Storage.init(). An interesting alternative is to download to an anonymous temporary file that vanishes if not linked into the filesystem. Recent Linux provides a very cool procedure to do this -- open(2) with O_TMPFILE followed by linkat(2) [1] -- but it’s not always supported and the workaround (create, then immediately unlink(2)) does not support re-linking [2]. This would also not support restarting the download. [1]: https://man7.org/linux/man-pages/man2/open.2.html [2]: https://stackoverflow.com/questions/4171713""" __slots__ = ("fp", "msg", "path", "path_tmp", "progress") def __init__(self, path, msg): self.msg = msg self.path = path self.path_tmp = path.with_name("part_" + path.name) self.progress = None def close(self): if (self.progress is not None): self.progress.done() close_(self.fp) self.path.unlink(missing_ok=True) self.path_tmp.rename(self.path) def start(self, length): self.progress = Progress(self.msg, "MiB", 2**20, length) self.fp = self.path_tmp.open("wb") def write(self, data): self.progress.update(len(data)) ossafe("can’t write: %s" % self.path, self.fp.write, data) class Timer: __slots__ = ("start") def __init__(self): self.start = time.time() def log(self, msg): VERBOSE("%s in %.3fs" % (msg, time.time() - self.start)) ## Supporting functions ## def DEBUG(msg, hint=None, **kwargs): if (log_level >= Log_Level.DEBUG): log(msg, hint, None, "38;5;6m", "", **kwargs) # dark cyan (same as 36m) def ERROR(msg, hint=None, trace=None, **kwargs): log(msg, hint, trace, "1;31m", "error: ", **kwargs) # bold red def FATAL(msg, hint=None, **kwargs): if (trace_fatal): # One-line traceback, skipping top entry (which is always bootstrap code # calling ch-image.main()) and last entry (this function). tr = ", ".join("%s:%d:%s" % (os.path.basename(f.filename), f.lineno, f.name) for f in reversed(traceback.extract_stack()[1:-1])) else: tr = None raise Fatal_Error(msg, hint, tr, **kwargs) def ILLERI(msg, hint=None, **kwargs): # For temporary debugging only. See contributors’ guide. log(msg, hint, None, "38;5;207m", "", **kwargs) # hot pink def INFO(msg, hint=None, **kwargs): "Note: Use print() for output; this function is for logging." if (log_level >= Log_Level.INFO): log(msg, hint, None, "33m", "", **kwargs) # yellow def TRACE(msg, hint=None, **kwargs): if (log_level >= Log_Level.TRACE): log(msg, hint, None, "38;5;6m", "", **kwargs) # dark cyan (same as 36m) def VERBOSE(msg, hint=None, **kwargs): if (log_level >= Log_Level.VERBOSE): log(msg, hint, None, "38;5;14m", "", **kwargs) # light cyan (1;36m, not bold) def WARNING(msg, hint=None, msg_save=True, **kwargs): if (log_level > Log_Level.STDERR): if (msg_save): warns.append(msg) log(msg, hint, None, "31m", "warning: ", **kwargs) # red def arch_host_get(): "Return the registry architecture of the host." arch_uname = platform.machine() VERBOSE("host architecture from uname: %s" % arch_uname) try: arch_registry = ARCH_MAP[arch_uname] except KeyError: FATAL("unknown host architecture: %s" % arch_uname, BUG_REPORT_PLZ) VERBOSE("host architecture for registry: %s" % arch_registry) return arch_registry def argv_to_string(argv): return " ".join(shlex.quote(i).replace("\n", "\\n") for i in argv) def bytes_hash(data): "Return the hash of data, as a hex string with no leading algorithm tag." h = hashlib.sha256() h.update(data) return h.hexdigest() def ch_run_modify(img, args, env, workdir="/", binds=[], ch_run_args=[], fail_ok=False): # Note: If you update these arguments, update the ch-image(1) man page too. args = ( [CH_BIN + "/ch-run"] + ch_run_args + ["-w", "-u0", "-g0", "--no-passwd", "--cd", workdir, "--unsafe"] + sum([["-b", i] for i in binds], []) + [img, "--"] + args) return cmd(args, env=env, stderr=None, fail_ok=fail_ok) def close_(fp): try: path = fp.name except AttributeError: path = "(no path)" ossafe("can’t close: %s" % path, fp.close) def cmd(argv, fail_ok=False, **kwargs): """Run command using cmd_base(). If fail_ok, return the exit code whether or not the process succeeded; otherwise, return (zero) only if the process succeeded and exit with fatal error if it failed.""" if (log_level < Log_Level.WARNING): kwargs["stdout"] = subprocess.DEVNULL if (log_level <= Log_Level.QUIET_STDERR): kwargs["stderr"] = subprocess.DEVNULL cp = cmd_base(argv, fail_ok=fail_ok, **kwargs) return cp.returncode def cmd_base(argv, fail_ok=False, **kwargs): """Run a command to completion. If not fail_ok, exit with a fatal error if the command fails (i.e., doesn’t exit with code zero). Return the CompletedProcess object. The command’s stderr is suppressed unless (1) logging is DEBUG or higher or (2) fail_ok is False and the command fails.""" argv = [str(i) for i in argv] VERBOSE("executing: %s" % argv_to_string(argv)) if ("env" in kwargs): for (k,v) in sorted(kwargs["env"].items()): VERBOSE("env: %s=%s" % (k,v)) if ("stderr" not in kwargs): if (log_level <= Log_Level.INFO): # VERBOSE or lower: capture for printing on fail only kwargs["stderr"] = subprocess.PIPE if ("input" not in kwargs): kwargs["stdin"] = subprocess.DEVNULL try: profile_stop() cp = subprocess.run(argv, **kwargs) profile_start() except OSError as x: VERBOSE("can’t execute %s: %s" % (argv[0], x.strerror)) # Most common reason we are here is that the command isn’t found, which # generates a FileNotFoundError. Use fake return value 127; this is # consistent with the shell [1]. This is a kludge, but we assume the # caller doesn’t care about the distinction between some problem within # the subprocess and inability to start the subprocess. # # [1]: https://devdocs.io/bash/exit-status#Exit-Status cp = subprocess.CompletedProcess(argv, 127) if (not fail_ok and cp.returncode != 0): if (cp.stderr is not None): if (isinstance(cp.stderr, bytes)): cp.stderr = cp.stderr.decode("UTF-8") sys.stderr.write(cp.stderr) sys.stderr.flush() FATAL("command failed with code %d: %s" % (cp.returncode, argv_to_string(argv))) return cp def cmd_quiet(argv, **kwargs): """Run command using cmd() and return the exit code. If logging is verbose or lower, discard stdout.""" if (log_level >= Log_Level.DEBUG): # debug or higher stdout=None else: stdout=subprocess.DEVNULL return cmd(argv, stdout=stdout, **kwargs) def cmd_stdout(argv, encoding="UTF-8", **kwargs): """Run command using cmd_base(), capturing its standard output. Return the CompletedProcess object (its stdout is available in the “stdout” attribute). If logging is debug or higher, print stdout.""" cp = cmd_base(argv, encoding=encoding, stdout=subprocess.PIPE, **kwargs) if (log_level >= Log_Level.DEBUG): # debug or higher # just dump to stdout rather than using DEBUG() to match cmd_quiet sys.stdout.write(cp.stdout) sys.stdout.flush() return cp def color_reset(*fps): for fp in fps: color_set("0m", fp) def color_set(color, fp): if (fp.isatty()): print("\033[" + color, end="", flush=True, file=fp) def dependencies_check(): """Check more dependencies. If any dependency problems found, here or above (e.g., lark module checked at import time), then complain and exit.""" # enforce Python minimum version vsys_py = sys.version_info[:3] # 4th element is a string if (vsys_py < PYTHON_MIN): vmin_py_str = ".".join(("%d" % i) for i in PYTHON_MIN) vsys_py_str = ".".join(("%d" % i) for i in vsys_py) depfails.append(("bad", ("need Python %s but running under %s: %s" % (vmin_py_str, vsys_py_str, sys.executable)))) # report problems & exit for (p, v) in depfails: ERROR("%s dependency: %s" % (p, v)) if (len(depfails) > 0): exit(1) def digest_trim(d): """Remove the algorithm tag from digest d and return the rest. >>> digest_trim("sha256:foobar") 'foobar' Note: Does not validate the form of the rest.""" try: return d.split(":", maxsplit=1)[1] except AttributeError: FATAL("not a string: %s" % repr(d)) except IndexError: FATAL("no algorithm tag: %s" % d) def done_notify(): if (user() == "jogas"): INFO("!!! KOBE !!!") else: INFO("done") def exit(code): profile_stop() profile_dump() sys.exit(code) def init(cli): # logging global log_festoon, log_fp, log_level, trace_fatal, xattrs_save incomp_opts = 0 for (x,y) in CLI_INCOMPATIBLE_OPTS: if (getattr(cli, x) and getattr(cli, y)): ERROR("“--%s” incompatible with “--%s”" % ((x.replace("_","-"), y.replace("_","-")))) incomp_opts += 1 if (incomp_opts > 0): FATAL("%d incompatible option pair(s)" % incomp_opts) xattrs_save = ((cli.xattrs) or (("CH_XATTRS" in os.environ) and (not cli.no_xattrs))) trace_fatal = (cli.debug or bool(os.environ.get("CH_IMAGE_DEBUG", False))) log_level = Log_Level(cli.verbose - cli.quiet) assert (-3 <= log_level.value <= 3) if (log_level <= Log_Level.STDERR): # suppress writing to stdout (particularly “print”). sys.stdout = open(os.devnull, 'w') if ("CH_LOG_FESTOON" in os.environ): log_festoon = True file_ = os.getenv("CH_LOG_FILE") if (file_ is not None): log_fp = file_.open("at") atexit.register(color_reset, log_fp) VERBOSE("version: %s" % version.VERSION) VERBOSE("verbose level: %d (%s))" % (log_level.value, log_level.name)) VERBOSE("save xattrs: %s" % str(xattrs_save)) # signal handling signal.signal(signal.SIGINT, sigterm) signal.signal(signal.SIGTERM, sigterm) # storage directory global storage storage = fs.Storage(cli.storage) fs.storage_lock = not cli.no_lock # architecture global arch, arch_host assert (cli.arch is not None) arch_host = arch_host_get() if (cli.arch == "host"): arch = arch_host else: arch = cli.arch # download cache if (cli.always_download): dlcache = Download_Mode.WRITE_ONLY else: dlcache = Download_Mode.ENABLED global dlcache_p dlcache_p = (dlcache == Download_Mode.ENABLED) # registry authentication if (cli.func.__module__ == "push"): rg.auth_p = True elif (cli.auth): rg.auth_p = True elif ("CH_IMAGE_AUTH" in os.environ): rg.auth_p = (os.environ["CH_IMAGE_AUTH"] == "yes") else: rg.auth_p = False VERBOSE("registry authentication: %s" % rg.auth_p) # Red Hat Python warns about tar bugs, citing CVE-2007-4559. # We mitigate this already, so suppress the noise. (#1818) warnings.filterwarnings("ignore", module=r"^tarfile$", message=( "^The default behavior of tarfile" + " extraction has been changed to disallow" + " common exploits")) # misc global password_many, profiling password_many = cli.password_many profiling = cli.profile if (cli.tls_no_verify): rg.tls_verify = False rpu = rg.requests.packages.urllib3 rpu.disable_warnings(rpu.exceptions.InsecureRequestWarning) def kill_blocking(pid, timeout=10): """Kill process pid with SIGTERM (the friendly one) and wait for it to exit. If timeout (in seconds) is exceeded and it’s still running, exit with a fatal error. It is *not* an error if pid does not exist, to avoid race conditions where we decide to kill a process and it exits before we can send the signal.""" sig = signal.SIGTERM try: os.kill(pid, sig) except ProcessLookupError: # ESRCH, no such process return except OSError as x: FATAL("can’t signal PID %d with %d: %s" % (pid, sig, x.strerror)) for i in range(timeout*2): try: os.kill(pid, 0) # no effect on process except ProcessLookupError: # done return except OSError as x: FATAL("can’t signal PID %s with 0: %s" % (pid, x.strerror)) time.sleep(0.5) FATAL("timeout of %ds exceeded trying to kill PID %d" % (timeout, pid), BUG_REPORT_PLZ) def log(msg, hint, trace, color, prefix, end="\n"): if (color is not None): color_set(color, log_fp) if (log_festoon): ts = datetime.datetime.now().isoformat(timespec="milliseconds") festoon = ("%5d %s " % (os.getpid(), ts)) else: festoon = "" print(festoon, prefix, msg, sep="", file=log_fp, end=end, flush=True) if (hint is not None): print(festoon, "hint: ", hint, sep="", file=log_fp, flush=True) if (trace is not None): print(festoon, "trace: ", trace, sep="", file=log_fp, flush=True) if (color is not None): color_reset(log_fp) def monkey_write_streams(): """Monkey patch to replace problematic characters in stdout and stderr streams when running Python 3.6. (see #1629).""" def monkey_write_insert(f): write_orig = f.write def write_monkey(text): text = text.replace("“", "\"").replace("”", "\"").replace("’", "'") write_orig(text) f.write = write_monkey # Try to encode test string of problematic characters. If unsuccessful, # monkey patch them out. for stream in sys.stdout, sys.stderr: for encoding in stream.encoding, locale.getpreferredencoding(), "ASCII": if (encoding is not None): try: "“”’".encode(encoding=encoding) except UnicodeEncodeError: monkey_write_insert(stream) break def now_utc_iso8601(): return datetime.datetime.utcnow().isoformat(timespec="seconds") + "Z" def ossafe(msg, f, *args, **kwargs): """Call f with args and kwargs. Catch OSError and other problems and fail with a nice error message.""" try: return f(*args, **kwargs) except OSError as x: FATAL("%s: %s" % (msg, x.strerror)) def positive(x): """Convert x to float, then if ≤ 0, change to positive infinity. This is monstly a convenience function to let 0 express “unlimited”.""" x = float(x) if (x <= 0): x = float("inf") return x def prefix_path(prefix, path): """"Return True if prefix is a parent directory of path. Assume that prefix and path are strings.""" return prefix == path or (prefix + '/' == path[:len(prefix) + 1]) def profile_dump(): "If profiling, dump the profile data." if (profiling): INFO("writing profile files ...") fp = fs.Path("/tmp/chofile.txt").open("wt") ps = pstats.Stats(profile, stream=fp) ps.sort_stats(pstats.SortKey.CUMULATIVE) ps.dump_stats("/tmp/chofile.p") ps.print_stats() close_(fp) def profile_start(): "If profiling, start the profiler." global profile if (profiling): if (profile is None): INFO("initializing profiler") profile = cProfile.Profile() profile.enable() def profile_stop(): "If profiling, stop the profiler." if (profiling and profile is not None): profile.disable() def si_binary_bytes(ct): # FIXME: varies between 1 and 3 significant figures ct = float(ct) for suffix in ("B", "KiB", "MiB", "GiB", "TiB", "PiB", "EiB", "ZiB"): if (ct < 1024): return (ct, suffix) ct /= 1024 assert False, "unreachable" def si_decimal(ct): ct = float(ct) for suffix in ("", "K", "M", "G", "T", "P", "E", "Z"): if (ct < 1000): return (ct, suffix) ct /= 1000 assert False, "unreachable" def sigterm(signum, frame): "Handler for SIGTERM and friends." # Ignore further signals because we are already cleaning up. signal.signal(signal.SIGINT, signal.SIG_IGN) signal.signal(signal.SIGTERM, signal.SIG_IGN) # Don’t stomp on progress meter if one is being printed. print() signame = signal.Signals(signum).name ERROR("received %s, exiting" % signame) FATAL("received %s" % signame) def user(): "Return the current username; exit with error if it can’t be obtained." try: return os.environ["USER"] except KeyError: FATAL("can’t get username: $USER not set") def variables_sub(s, variables): if (s is None): return s # FIXME: This should go in the grammar rather than being a regex kludge. # # Dockerfile spec does not say what to do if substituting a value that’s # not set. We ignore those subsitutions. This is probably wrong (the shell # substitutes the empty string). for (k, v) in variables.items(): # FIXME: remove when issue #774 is fixed m = re.search(r"(? v): too_old("%s is too old: %d.%d.%d < %d.%d.%d" % ((prog,) + v + min_)) return False VERBOSE("%s version OK: %d.%d.%d ≥ %d.%d.%d" % ((prog,) + v + min_)) return True def walk(*args, **kwargs): """Wrapper for os.walk(). Return a generator of the files in a directory tree (root specified in *args). For each directory in said tree, yield a 3-tuple (dirpath, dirnames, filenames), where dirpath is a Path object, and dirnames and filenames are lists of Path objects. For insight into these being lists rather than generators, see use of ch.walk() in Copy_G.copy_src_dir().""" for (dirpath, dirnames, filenames) in os.walk(*args, **kwargs): yield (fs.Path(dirpath), [fs.Path(dirname) for dirname in dirnames], [fs.Path(filename) for filename in filenames]) def warnings_dump(): if (len(warns) > 0): WARNING("reprinting %d warning(s)" % len(warns), msg_save=False) for msg in warns: WARNING(msg, msg_save=False) charliecloud-0.37/lib/filesystem.py000066400000000000000000001425721457016721300174260ustar00rootroot00000000000000import errno import fcntl import fnmatch import glob import hashlib import json import os import pprint import re import shutil import stat import struct import tarfile import charliecloud as ch ## Constants ## # Storage directory format version. We refuse to operate on storage # directories with non-matching versions. Increment this number when the # format changes non-trivially. # # To see the directory formats in released versions: # # $ git grep -E '^STORAGE_VERSION =' $(git tag | sort -V) STORAGE_VERSION = 7 ## Globals ## # True if we lock storage directory to prevent concurrent access; false for no # locking (which is very YOLO and may break the storage directory). storage_lock = True ### Functions ### def copy(src, dst, follow_symlinks=False): """Copy file src to dst. Wrapper function providing same signature as shutil.copy2(). See Path.copy() for lots of gory details. Accepts follow_symlinks, but the only valid value is False.""" assert (not follow_symlinks) if (isinstance(src, str)): src = Path(src) if (isinstance(dst, str)): dst = Path(dst) src.copy(dst) ## Classes ## class Path(os.PathLike): """Path class roughly corresponding to pathlib.PosixPath. While it does subclass os.PathLike, it does not subclass anything in pathlib because: 1. Only in 3.12 does pathlib.Path actually support subclasses [1]. Before then it can be done, but it’s messy and brittle. 2. pathlib.Path seems overcomplicated for our use case and is often slow. This class implements (incompletely) the pathlib.PosixPath API, with many extensions and two important differences: 1. Trailing slash. Objects remember whether a trailing slash is present, and append it when str() or repr(). “/” is considered to *not* have a trailing slash. Subprograms might interpret this differently. Notably, rsync(1) *does* interpret “/” as trailing-slashed. 2. Path join operator. This class uses the “//” operator, not “/”, for joining paths, with different semantics. When appending an absolute path to a pathlib.PosixPath object, the left operand is ignored, leaving only the absolute right operand: >>> import pathlib >>> a = pathlib.Path("/foo/bar") >>> a / "baz" PosixPath('/foo/bar/baz') >>> a / "/baz" PosixPath('/baz') This is contrary to long-standing UNIX/POSIX, where extra slashes in a path are ignored, e.g. “/foo//bar” is equivalent to “/foo/bar”. os.path.join() behaves the same way. This behavior caused quite a few Charliecloud bugs. IMO it’s too error-prone to manually manage whether paths are absolute or relative. Thus, the operator to join instances of this class is “//”, which does the POSIX thing, i.e., if the right operand is absolute, that fact is just ignored. E.g.: >>> a = Path("/foo/bar") >>> a // "/baz" Path('/foo/bar/baz') >>> "/baz" // a Path('/baz/foo/bar') We used a different operator because it seemed a source of confusion to change the behavior of “/” (which is not provided by this class). An alternative was “+” like strings, but that led to silently wrong results when the paths *were* strings (simple string concatenation with no slash). [1]: https://docs.python.org/3/whatsnew/3.12.html#pathlib""" # Store the path as a string. Assume: # # 1. No multiple slashes. # 2. Length at least one character. # 3. Does not begin with redundant “./” (but can be just “.”). # # Call self._tidy() if these can’t be assumed. __slots__ = ("path",) # Name of the gzip(1) to use for file_gzip(); set on first call. gzip = None def __init__(self, *segments): """e.g.: >>> Path("/a/b") Path('/a/b') >>> Path("/", "a", "b") Path('/a/b') >>> Path("/", "a", "b", "/") Path('/a/b/') >>> Path("a/b") Path('a/b') >>> Path("/a/b/") Path('/a/b/') >>> Path("//") Path('/') >>> Path("") Path('.') >>> Path("./a") Path('a')""" segments = [ (i.__fspath__() if isinstance(i, os.PathLike) else i) for i in segments] self.path = "/".join(segments) self._tidy() ## Internal ## @classmethod def _gzip_set(cls): """Set gzip class attribute on first call to file_gzip(). Note: We originally thought this could be accomplished WITHOUT calling a class method (by setting the attribute, e.g. “self.gzip = 'foo'”), but it turned out that this would only set the attribute for the single instance. To set self.gzip for all instances, we need the class method.""" if (cls.gzip is None): if (shutil.which("pigz") is not None): cls.gzip = "pigz" elif (shutil.which("gzip") is not None): cls.gzip = "gzip" else: ch.FATAL("can’t find path to gzip or pigz") def _tidy(self): "Repair self.path assumptions (see attribute docs above)." if (self.path == ""): self.path = "." else: self.path = re.sub(r"/{2,}", "/", self.path) self.path = re.sub(r"^\./", "", self.path) ## pathlib.PosixPath API ## def __eq__(self, other): """e.g.: >>> a1 = Path("a") >>> a2 = Path("a") >>> b = Path("b") >>> a1 == a1 True >>> a1 is a1 True >>> a1 == a2 True >>> a1 is a2 False >>> a1 == b False >>> a1 != b True >>> Path("a") == Path("a/") False >>> Path("a") == Path("a/").untrailed True >>> Path("") == Path(".") True >>> Path("/") == Path("//") True >>> Path("a/b") == Path("a//b") == Path("a///b") True""" return (self.path == Path(other).path) def __fspath__(self): return self.path def __ge__(self, other): return not self.__lt__(other) def __gt__(self, other): return not self.__le__(other) def __hash__(self): return hash(self.path) def __le__(self, other): return (self.path <= Path(other).path) def __lt__(self, other): return (self.path < Path(other).path) def __ne__(self, other): return not self.__eq__(other) def __repr__(self): """e.g.: >>> repr(Path("a")) "Path('a')" >>> repr(Path("a'b")) 'Path("a\\'b")' """ return 'Path(%s)' % repr(self.path) def __str__(self): """e.g.: >>> str(Path("a")) 'a'""" return self.path @property def name(self): """e.g.: >>> Path("a").name 'a' >>> Path("/a/b").name 'b' >>> Path("a/b").name 'b' >>> Path("a/b/").name 'b' Note: Unlike pathlib.Path, dot and slash return themselves: >>> Path("/").name '/' >>> Path(".").name '.' """ if (self.root_p): return "/" return self.untrailed.path.rpartition("/")[-1] @property def parent(self): """e.g.: >>> Path("/a/b").parent Path('/a') >>> Path("a/b").parent Path('a') >>> Path("/a").parent Path('/') >>> Path("a/b/").parent Path('a') >>> Path("a").parent Path('.') >>> Path(".").parent Path('.') Note that the parent of “/” is “/”, per POSIX: >>> Path("/").parent Path('/')""" if (self.root_p): return self.deepcopy() (parent, slash, _) = self.untrailed.path.rpartition("/") if (parent != ""): return self.__class__(parent) elif (slash == "/"): # absolute path with single non-root component return self.__class__("/") else: # relative path with single component return self.__class__(".") @property def parts(self): """e.g.: >>> Path("/a/b").parts ['/', 'a', 'b'] >>> Path("a/b/").parts ['a', 'b'] >>> Path("/").parts ['/'] >>> Path(".").parts []""" if (self.path == "."): return [] ret = self.path.split("/") if (ret[0] == ""): ret[0] = "/" if (ret[-1] == ""): del ret[-1] return ret def exists(self, links=False): """Return True if I exist, False otherwise. Iff links, follow symlinks. >>> Path("/").exists() True >>> Path("/doesnotexist").exists() False >>> Path("/proc/self/cmdline").exists(False) True""" try: os.stat(self, follow_symlinks=links) except FileNotFoundError: return False except OSError as x: ch.FATAL("can’t stat: %s: %s" % (self, x.strerror)) return True def glob(self, pattern): oldcwd = self.chdir() # No root_dir in glob.glob() until 3.10. ret = glob.glob(pattern, recursive=True) oldcwd.chdir() return ret def hardlink_to(self, target): ch.ossafe("can’t hard link: %s -> %s", os.link, target, self) def is_absolute(self): return (self.path[0] == "/") def is_dir(self): """e.g.: >>> Path("/proc").is_dir() True >>> Path("/proc/self").is_dir() True >>> Path("/proc/cmdline").is_dir() False >>> Path("/doesnotexist").is_dir() False""" return os.path.isdir(self) def is_file(self): return os.path.isfile(self) def is_relative_to(self, other): """e.g.: >>> Path("/a/b").is_relative_to("/a") True >>> Path("/a/b/").is_relative_to("/a") True >>> Path("/a/b").is_relative_to("/c") False >>> Path("/a/b").is_relative_to("c") False""" try: self.relative_to(other) return True except ValueError: return False def is_symlink(self): return os.path.islink(self) def match(self, pattern): """e.g.: >>> a = Path("/foo/bar.txt") >>> a.match("*.txt") True >>> a.match("*.TXT") False""" return fnmatch.fnmatchcase(self.__fspath__(), pattern) def mkdir(self): ch.TRACE("ensuring directory: %s" % self) if (self.is_dir()): return # target exists and is a directory, do nothing try: os.mkdir(self) except FileExistsError as x: ch.FATAL("can’t mkdir: exists and not a directory: %s" % x.filename) except OSError as x: ch.FATAL("can’t mkdir: %s: %s" % (x.filename, x.strerror)) def open(self, mode, *args, **kwargs): return ch.ossafe("can’t open for %s: %s" % (mode, self), open, self, mode, *args, **kwargs) def relative_to(self, other): """e.g. absolute paths: >>> a = Path("/a/b") >>> a.relative_to(Path("/")) Path('a/b') >>> a.relative_to("/a") Path('b') >>> Path("/a/b/").relative_to("/a") Path('b/') e.g. relative paths: >>> a = Path("a/b") >>> a.relative_to("a") Path('b') e.g. problems: >>> Path("/a/b").relative_to("a") Traceback (most recent call last): ... ValueError: Can't mix absolute and relative paths >>> Path("/a/b").relative_to("/c") Traceback (most recent call last): ... ValueError: /a/b not a subpath of /c """ if (isinstance(other, Path)): other = other.untrailed.__fspath__() common = os.path.commonpath([self, other]) if (common != other): raise ValueError("%s not a subpath of %s" % (self, other)) return self.__class__(self.path[ len(other) + (0 if other == "/" else 1):]) def rename(self, path_new): path_new = self.__class__(path_new) ch.ossafe("can’t rename: %s -> %s" % (self, path_new), os.rename, self, path_new) return path_new def resolve(self): """e.g.: >>> import os >>> real = Path("/proc/%d" % os.getpid()) >>> link = Path("/proc/self") >>> link.resolve() == real True""" return self.__class__(os.path.realpath(self)) def rmdir(self): ch.ossafe("can’t rmdir: %s" % self, os.rmdir, self) def stat(self, links): """e.g.: >>> import stat >>> st = Path("/proc/self").stat(False) >>> stat.S_ISDIR(st.st_mode) False >>> stat.S_ISLNK(st.st_mode) True >>> st = Path("/proc/self").stat(True) >>> stat.S_ISDIR(st.st_mode) True >>> stat.S_ISLNK(st.st_mode) False""" return ch.ossafe("can’t stat: %s" % self, os.stat, self, follow_symlinks=links) def symlink_to(self, target, clobber=False): if (clobber and self.is_file()): self.unlink() try: os.symlink(target, self) except FileExistsError: if (not self.is_symlink()): ch.FATAL("can’t symlink: source exists and isn’t a symlink: %s" % self) if (self.readlink() != target): ch.FATAL("can’t symlink: %s exists; want target %s but existing is %s" % (self, target, self.readlink())) except OSError as x: ch.FATAL("can’t symlink: %s -> %s: %s" % (self, target, x.strerror)) def unlink(self, missing_ok=False): if (missing_ok and not self.exists()): return ch.ossafe("can’t unlink: %s" % self, os.unlink, self) def with_name(self, name_new): """e.g.: >>> Path("a").with_name("b") Path('b') >>> Path("a/b").with_name("c") Path('a/c') >>> Path(".").with_name("a") Path('a') Not available for “/” because this would change an absolute path to relative, and that seems too surprising: >>> Path("/").with_name("a") Traceback (most recent call last): ... ValueError: with_name() invalid for /""" if (self.root_p): raise ValueError("with_name() invalid for /") return self.parent // name_new ## Extensions ## @staticmethod def stat_bytes_all(paths): "Return concatenation of metadata_bytes() on each given Path object." md = bytearray() for path in paths: md += path.stat_bytes_recursive() return md def __floordiv__(self, right): left = self.path try: right = right.__fspath__() except AttributeError: pass # assume right is a string return self.__class__(left + "/" + right) def __len__(self): """The length of a Path is the number of components, including the root directory. “.” has zero components. >>> len(Path("a")) 1 >>> len(Path("/")) 1 >>> len(Path("/a")) 2 >>> len(Path("a/b")) 2 >>> len(Path("/a/b")) 3 >>> len(Path("/a/")) 2 >>> len(Path(".")) 0""" return len(self.parts) def __rfloordiv__(self, left): return self.__class__(left).__floordiv__(self) @property def empty_p(self): return (self.path == ".") @property def first(self): """Return my first component as a new Path object, e.g.: >>> a = Path("/") >>> b = a.first >>> b Path('/') >>> a == b True >>> a is b False >>> Path("").first Path('.') >>> Path("./a").first Path('a') >>> Path("a/b").first Path('a')""" if (self.root_p): return self.deepcopy() return self.__class__(self.path.partition("/")[0]) @property def git_compatible_p(self): """Return True if my filename can be stored in Git, false otherwise. >>> Path("/gitignore").git_compatible_p True >>> Path("/.gitignore").git_compatible_p False""" return (not self.name.startswith(".git")) @property def git_escaped(self): """Return a copy of me escaped for Git storage, possibly unchanged. >>> Path("/gitignore").git_escaped Path('/gitignore') >>> Path("/.gitignore").git_escaped Path('/.weirdal_ignore') >>> Path("/.gitignore/").git_escaped Path('/.weirdal_ignore/') """ ret = self.with_name(self.name.replace(".git", ".weirdal_")) if (self.trailed_p): ret.path += "/" return ret @property def root_p(self): return (self.path == "/") @property def trailed_p(self): """e.g.: >>> Path("a").trailed_p False >>> Path("a/").trailed_p True >>> Path("/").trailed_p False >>> (Path("a") // "b").trailed_p False >>> (Path("a/") // "b").trailed_p False >>> (Path("a") // "b/").trailed_p True >>> (Path("a") // "/").trailed_p True""" return (self.path != "/" and self.path[-1] == "/") @property def untrailed(self): """Return self with trailing slash removed (if any). E.g.: >>> Path("a").untrailed Path('a') >>> Path("a/").untrailed Path('a') >>> Path("/").untrailed Path('/') >>> Path(".").untrailed Path('.')""" if (self.root_p): return self.deepcopy() else: return self.__class__(self.path.rstrip("/")) def chdir(self): "Change CWD to path and return previous CWD. Exit on error." old = ch.ossafe("can’t getcwd(2)", os.getcwd) ch.ossafe("can’t chdir(2): %s" % self, os.chdir, self) return self.__class__(old) def chmod_min(self, st_old=None): """Set my permissions to at least 0o700 for directories and 0o400 otherwise. For symlinks, do nothing, because we don’t want to follow symlinks and follow_symlinks=False (or os.lchmod) is not supported on some (all?) Linux. (Also, symlink permissions are ignored on Linux, so it doesn’t matter anyway.) If given, st_old is a stat_result object for self, to avoid another stat(2) call. In this case, also return the resulting stat_result object, which is st itself if nothing was modified, or a new stat_result object if the mode was changed.""" st = self.stat(False) if not st_old else st_old if (stat.S_ISLNK(st.st_mode)): return st_old perms_old = stat.S_IMODE(st.st_mode) perms_new = perms_old | (0o700 if stat.S_ISDIR(st.st_mode) else 0o400) if (perms_new != perms_old): ch.VERBOSE("fixing permissions: %s: %03o -> %03o" % (self, perms_old, perms_new)) ch.ossafe("can’t chmod: %s" % self, os.chmod, self, perms_new) if (st_old): # stat_result is a deeply weird object (a “structsec” rather than a # named tuple), including multiple values for the same field when # accessed by index vs. name. I did figure out how to create a # modified copy, which is the commented code below, but it seems too # brittle and scary, so just re-stat(2) the modified metadata. # # st_list = list(st_old) # st_dict = { k:getattr(a, k) for k in dir(a) if k[:3] == "st_" } # st_list[0] |= perms_new # st_dict["st_mode"] |= perms_new # st_new = os.stat_result(st_list, st_dict) # assert (st_new[0] == st_new.st_mode) # return st_new if (perms_new == perms_old): return st_old else: return self.stat(False) def copy(self, dst): """Copy file myself to dst, including metadata, overwriting dst if it exists. dst must be the actual destination path, i.e., it may not be a directory. Does not follow symlinks. If (a) src is a regular file, (b) src and dst are on the same filesystem, and (c) Python is version ≥3.8, then use os.copy_file_range() [1,2], which at a minimum does an in-kernel data transfer. If that filesystem also (d) supports copy-on-write [3], then this is a very fast lazy reflink copy. [1]: https://docs.python.org/3/library/os.html#os.copy_file_range [2]: https://man7.org/linux/man-pages/man2/copy_file_range.2.html [3]: https://elixir.bootlin.com/linux/latest/A/ident/remap_file_range """ src_st = self.stat(False) # dst is not a directory, so parent must be on the same filesystem. We # *do* want to follow symlinks on the parent. dst_dev = dst.parent.stat(True).st_dev if ( stat.S_ISREG(src_st.st_mode) and src_st.st_dev == dst_dev and hasattr(os, "copy_file_range")): # Fast path. The same-filesystem restriction is because reliable # copy_file_range(2) between filesystems seems quite new (maybe # kernel 5.18?). try: if (dst.exists()): # If dst is a symlink, we get OLOOP from os.open(). Delete it # unconditionally though, for simplicity. dst.unlink() src_fd = os.open(self, os.O_RDONLY|os.O_NOFOLLOW) dst_fd = os.open(dst, os.O_WRONLY|os.O_NOFOLLOW|os.O_CREAT) # I’m not sure why we need to loop this -- there’s no explanation # of *when* fewer bytes than requested would be copied -- but the # man page example does. remaining = src_st.st_size while (remaining > 0): copied = os.copy_file_range(src_fd, dst_fd, remaining) if (copied == 0): ch.FATAL("zero bytes copied: %s -> %s" % (self, dst)) remaining -= copied os.close(src_fd) os.close(dst_fd) except OSError as x: ch.FATAL("can’t copy data (fast): %s -> %s: %s" % (self, dst, x.strerror)) else: # Slow path. try: shutil.copyfile(self, dst, follow_symlinks=False) except OSError as x: ch.FATAL("can’t copy data (slow): %s -> %s: %s" % (self, dst, x.strerror)) try: # Metadata. shutil.copystat(self, dst, follow_symlinks=False) except OSError as x: ch.FATAL("can’t copy metadata: %s -> %s" % (self, dst, x.strerror)) def copytree(self, *args, **kwargs): "Wrapper for shutil.copytree() that exits on the first error." shutil.copytree(self, copy_function=copy, *args, **kwargs) def deepcopy(self): """Return a copy of myself. E.g.: >>> a = Path("a") >>> b = a.deepcopy() >>> b Path('a') >>> a == b True >>> a is b False""" return self.__class__(self.path) def disk_bytes(self): """Return the number of disk bytes consumed by path. Note this is probably different from the file size.""" return self.stat(False).st_blocks * 512 def du(self): """Return a tuple (number of files, total bytes on disk) for everything under path. Warning: double-counts files with multiple hard links and any shared data extents.""" file_ct = 1 byte_ct = self.disk_bytes() for (dir_, subdirs, files) in ch.walk(self): file_ct += len(subdirs) + len(files) byte_ct += sum((self.__class__(dir_) // i).disk_bytes() for i in subdirs + files) return (file_ct, byte_ct) def file_ensure_exists(self): """If the final element of path exists (without dereferencing if it’s a symlink), do nothing; otherwise, create it as an empty regular file.""" if (not os.path.lexists(self)): fp = self.open("w") ch.close_(fp) def file_gzip(self, args=[]): """Run pigz(1) if it’s available, otherwise gzip(1), on file at path and return the file’s new name. Pass args to the gzip executable. This lets us gzip files (a) in parallel if pigz(1) is installed and (b) without reading them into memory.""" path_c = self.suffix_add(".gz") # On first call, remember first available of pigz and gzip using class # attribute 'gzip'. self.__class__._gzip_set() # Remove destination if it already exists, because “gzip --force” does # several other things too. Also, pigz(1) sometimes confusingly reports # “Inappropriate ioctl for device” if destination already exists. if (path_c.exists()): path_c.unlink() # Compress. ch.cmd([self.gzip] + args + [str(self)]) # Zero out GZIP header timestamp, bytes 4–7 zero-indexed inclusive [1], # to ensure layer hash is consistent. See issue #1080. # [1]: https://datatracker.ietf.org/doc/html/rfc1952 §2.3.1 fp = path_c.open("r+b") ch.ossafe("can’t seek: %s" % fp, fp.seek, 4) ch.ossafe("can’t write: %s" % fp, fp.write, b'\x00\x00\x00\x00') ch.close_(fp) return path_c def file_hash(self): """Return the hash of data in file at path, as a hex string with no algorithm tag. File is read in chunks and can be larger than memory. >>> Path("/dev/null").file_hash() 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' """ fp = self.open("rb") h = hashlib.sha256() while True: data = ch.ossafe("can’t read: %s" % self, fp.read, 2**18) if (len(data) == 0): # EOF break h.update(data) ch.close_(fp) return h.hexdigest() def file_read_all(self, text=True): """Return the contents of file at path, or exit with error. If text, read in “rt” mode with UTF-8 encoding; otherwise, read in mode “rb”. >>> Path("/dev/null").file_read_all() '' >>> Path("/dev/null").file_read_all(False) b''""" if (text): mode = "rt" encoding = "UTF-8" else: mode = "rb" encoding = None fp = self.open(mode, encoding=encoding) data = ch.ossafe("can’t read: %s" % self, fp.read) ch.close_(fp) return data def file_size(self, follow_symlinks=False): """Return the size of file at path in bytes. >>> Path("/dev/null").file_size() 0""" return self.stat(follow_symlinks).st_size def file_write(self, content): """e.g.: >>> Path("/dev/null").file_write("Weird Al Yankovic") """ if (isinstance(content, str)): content = content.encode("UTF-8") fp = self.open("wb") ch.ossafe("can’t write: %s" % self, fp.write, content) ch.close_(fp) def grep_p(self, rx): """Return True if file at path contains a line matching regular expression rx, False if it does not. >>> Path("/dev/null").grep_p(r"foo") False""" try: with open(self, "rt") as fp: for line in fp: if (re.search(rx, line) is not None): return True return False except OSError as x: ch.FATAL("can’t read %s: %s" % (self, x.strerror)) def iterdir(self): """e.g.: >>> import os >>> dir = Path("/proc/self/task") >>> set(dir.iterdir()) == { dir // str(os.getpid()) } True""" for entry in ch.ossafe("can’t scan: %s" % self, os.scandir, self): yield self.__class__(entry.path) def json_from_file(self, msg): ch.DEBUG("loading JSON: %s: %s" % (msg, self)) text = self.file_read_all() ch.TRACE("text:\n%s" % text) try: data = json.loads(text) ch.DEBUG("result:\n%s" % pprint.pformat(data, indent=2)) except json.JSONDecodeError as x: ch.FATAL("can’t parse JSON: %s:%d: %s" % (self, x.lineno, x.msg)) return data def listdir(self): """Return set of entries in directory path, as strings, without self (.) and parent (..). We considered changing this to use os.scandir() for #992, but decided that the advantages it offered didn’t warrant the effort required to make the change. >>> import os >>> Path("/proc/self/task").listdir() == { str(os.getpid()) } True""" return set(ch.ossafe("can’t list: %s" % self, os.listdir, self)) def mkdirs(self, exist_ok=True): "Like “mkdir -p”." ch.TRACE("ensuring directory and parents: %s" % self) try: os.makedirs(self, exist_ok=exist_ok) except OSError as x: # x.filename might be an intermediate directory ch.FATAL("can’t mkdir: %s: %s: %s" % (self, x.filename, x.strerror)) def mountpoint(self): """Return the mount point of the filesystem containing, or, if symlink, the file pointed to. E.g.: >>> Path("/proc").mountpoint() Path('/proc') >>> Path("/proc/self").mountpoint() Path('/proc') >>> Path("/").mountpoint() Path('/')""" # https://stackoverflow.com/a/4453715 try: pc = self.resolve() except RuntimeError: ch.FATAL("not found, can’t resolve: %s" % self) dev_child = pc.stat(False).st_dev while (not pc.root_p): dev_parent = pc.parent.stat(False).st_dev if (dev_child != dev_parent): return pc pc = pc.parent # Got all the way up to root without finding a transition, so we’re on # the root filesystem. return self.__class__("/") def rmtree(self): ch.TRACE("deleting directory: %s" % self) try: shutil.rmtree(self) except OSError as x: ch.FATAL("can’t recursively delete directory %s: %s: %s" % (self, x.filename, x.strerror)) def setxattr(self, name, value): if (ch.xattrs_save): try: os.setxattr(self, name, value, follow_symlinks=False) except OSError as x: if (x.errno == errno.ENOTSUP): # no OSError subclass ch.WARNING("xattrs not supported on %s, setting --no-xattr" % self.mountpoint()) ch.xattrs_save = False else: ch.FATAL("can’t set xattr: %s: %s: %s" % (self, name, x.strerror)) if (not ch.xattrs_save): # not “else” because maybe changed in “if” ch.DEBUG("xattrs disabled, ignoring: %s: %s" % (self, name)) return def stat_bytes(self, links): "Return self.stat() encoded as an opaque bytearray." st = self.stat(links) return ( self.path.encode("UTF-8") + struct.pack("=HQQ", st.st_mode, st.st_size, st.st_mtime_ns)) def stat_bytes_recursive(self): """Return concatenation of self.stat() and all my children as an opaque bytearray, in unspecified but consistent order. Follow symlinks in self but not its descendants.""" # FIXME: Locale issues related to sorting? md = self.stat_bytes(True) if (self.is_dir()): for (dir_, dirs, files) in ch.walk(self): md += dir_.stat_bytes(False) for f in sorted(files): md += (dir_ // f).stat_bytes(False) dirs.sort() return md def strip(self, left=0, right=0): """Return a copy of self with n leading components removed. E.g.: >>> a = Path("/a/b/c") >>> a.strip(left=1) Path('a/b/c') >>> a.strip(right=1) Path('/a/b') >>> a.strip(left=1, right=1) Path('a/b') >>> Path("/a/b/").strip(right=1) Path('/a/') It is an error if self doesn’t have at least left + right components, i.e., you can strip a path down to nothing but not further. >>> Path("/").strip(left=1, right=1) Traceback (most recent call last): ... ValueError: can't strip 2 components from a path with only 1""" parts = self.parts if (len(parts) < left + right): raise ValueError("can't strip %d components from a path with only %d" % (left + right, len(parts))) ret = self.__class__(*self.parts[left:len(self.parts)-right]) if (self.trailed_p): ret.path += "/" return ret def suffix_add(self, suffix): """Append the given suffix and return the result. Dot (“.”) is not special and must be specified explicitly if needed. E.g.: >>> Path("a").suffix_add(".txt") Path('a.txt') >>> Path("a").suffix_add("_txt") Path('a_txt') >>> Path("a/").suffix_add(".txt") Path('a.txt/')""" return self.__class__( self.untrailed.path + suffix + ("/" if self.trailed_p else "")) class Storage: """Source of truth for all paths within the storage directory. Do not compute any such paths elsewhere!""" __slots__ = ("lockfile_fp", "root") def __init__(self, storage_cli): self.root = storage_cli if (self.root is None): self.root = self.root_env() if (self.root is None): self.root = self.root_default() if (not self.root.is_absolute()): self.root = os.getcwd() // self.root @staticmethod def root_default(): # FIXME: Perhaps we should use getpass.getch.user() instead of the $USER # environment variable? It seems a lot more robust. But, (1) we’d have # to match it in some scripts and (2) it makes the documentation less # clear becase we have to explain the fallback behavior. return Path("/var/tmp/%s.ch" % ch.user()) @staticmethod def root_env(): if (not "CH_IMAGE_STORAGE" in os.environ): return None path = Path(os.environ["CH_IMAGE_STORAGE"]) if (not path.is_absolute()): ch.FATAL("$CH_IMAGE_STORAGE: not absolute path: %s" % path) return path @property def bucache_needs_ignore_upgrade(self): return self.build_cache // "ch_upgrade-ignore" @property def build_cache(self): return self.root // "bucache" @property def build_large(self): return self.root // "bularge" @property def download_cache(self): return self.root // "dlcache" @property def image_tmp(self): return self.root // "imgtmp" @property def lockfile(self): return self.root // "lock" @property def mount_point(self): return self.root // "mnt" @property def unpack_base(self): return self.root // "img" @property def upload_cache(self): return self.root // "ulcache" @property def valid_p(self): """Return True if storage present and seems valid, even if old, False otherwise. This answers “is the storage directory real”, not “can this storage directory be used”; it should return True for more or less any Charliecloud storage directory we might feasibly come across, even if it can’t be upgraded. See also #1147.""" return (os.path.isdir(self.unpack_base) and os.path.isdir(self.download_cache)) @property def version_file(self): return self.root // "version" def build_large_path(self, name): return self.build_large // name def cleanup(self): "Called during initialization after we know the storage dir is valid." # Delete partial downloads. part_ct = 0 for path in self.download_cache.glob("part_*"): path = Path(path) ch.VERBOSE("deleting: %s" % path) path.unlink() part_ct += 1 if (part_ct > 0): ch.WARNING("deleted %d partially downloaded files" % part_ct) def fatman_for_download(self, image_ref): return self.download_cache // ("%s.fat.json" % image_ref.for_path) def init(self): """Ensure the storage directory exists, contains all the appropriate top-level directories & metadata, and is the appropriate version.""" # WARNING: This function contains multiple calls to self.lock(). The # point is to lock as soon as we know the storage directory exists, and # definitely before writing anything, to reduce the race conditions that # surely exist. Ensure new code paths also call self.lock(). if (not os.path.isdir(self.root)): op = "initializing" v_found = None else: op = "upgrading" # not used unless upgrading if (not self.valid_p): if (os.path.exists(self.root) and not self.root.listdir()): hint = "let Charliecloud create %s; see FAQ" % self.root.name else: hint = None ch.FATAL("storage directory seems invalid: %s" % self.root, hint) v_found = self.version_read() if (v_found == STORAGE_VERSION): ch.VERBOSE("found storage dir v%d: %s" % (STORAGE_VERSION, self.root)) self.lock() elif (v_found in {None, 4, 5, 6}): # initialize/upgrade ch.INFO("%s storage directory: v%d %s" % (op, STORAGE_VERSION, self.root)) self.root.mkdir() self.lock() # These directories appeared in various storage versions, but since # the thing to do on upgrade is the same as initialize, we don’t # track the details. self.download_cache.mkdir() self.build_cache.mkdir() self.build_large.mkdir() self.unpack_base.mkdir() self.upload_cache.mkdir() if (v_found is not None): # upgrade if (v_found < 6): # Git metadata moved from /.git to /ch/.git, and /.gitignore # went out-of-band (to info/exclude in the repository). for img in self.unpack_base.iterdir(): old = img // ".git" new = img // "ch/git" if (old.exists()): new.parent.mkdir() old.rename(new) gi = img // ".gitignore" if (gi.exists()): gi.unlink() # Must also remove .gitignore from all commits. This requires # Git operations, which we can’t do here because the build # cache may be disabled. Do it in Enabled_Cache.configure(). if (len(self.build_cache.listdir()) > 0): self.bucache_needs_ignore_upgrade.file_ensure_exists() if (v_found == 6): # Charliecloud 0.32 had a bug where symlinks to fat manifests # that were really skinny were erroneously absolute, making the # storage directory immovable (PR #1657). Remove all symlinks # in dlcache; they’ll be re-created later. for entry in self.download_cache.iterdir(): if (entry.is_symlink()): ch.DEBUG("deleting bad v6 symlink: %s" % entry) entry.unlink() self.version_file.file_write("%d\n" % STORAGE_VERSION) else: # can’t upgrade ch.FATAL("incompatible storage directory v%d: %s" % (v_found, self.root), "you can delete and re-initialize with “ch-image reset”") self.validate_strict() self.cleanup() def lock(self): """Lock the storage directory. Charliecloud does not at present support concurrent use of ch-image(1) against the same storage directory.""" # File locking on Linux is a disaster [1, 2]. Currently, we use POSIX # fcntl(2) locking, which has major pitfalls but should be fine for our # use case. It apparently works on NFS [3] and does not require # cleanup/stealing like a lock file would. # # [1]: https://apenwarr.ca/log/20101213 # [2]: http://0pointer.de/blog/projects/locking.html # [3]: https://stackoverflow.com/a/22411531 if (not storage_lock): return self.lockfile_fp = self.lockfile.open("w") try: fcntl.lockf(self.lockfile_fp, fcntl.LOCK_EX | fcntl.LOCK_NB) except OSError as x: if (x.errno in { errno.EACCES, errno.EAGAIN }): ch.FATAL("storage directory is already in use", "concurrent instances of ch-image cannot share the same storage directory") else: ch.FATAL("can’t lock storage directory: %s" % x.strerror) def manifest_for_download(self, image_ref, digest): if (digest is None): digest = "skinny" return ( self.download_cache // ("%s%%%s.manifest.json" % (image_ref.for_path, digest))) def reset(self): if (self.valid_p): self.root.rmtree() self.init() # largely for debugging else: ch.FATAL("%s not a builder storage" % (self.root)); def unpack(self, image_ref): return self.unpack_base // image_ref.for_path def validate_strict(self): """Validate storage directory structure; if something is wrong, exit with an error message. This is a strict validation; the version must be current, the structure of the directory must be current, and nothing unexpected may be present. However, it is not comprehensive. The main purpose is to check for bad upgrades and other programming errors, not meddling.""" ch.DEBUG("validating storage directory: %s" % self.root) msg_prefix = "invalid storage directory" # Check that all expected files exist, and no others. Note that we don’t # verify file *type*, assuming that kind of error is rare. entries = self.root.listdir() for entry in { i.name for i in (self.build_cache, self.build_large, self.download_cache, self.unpack_base, self.upload_cache, self.version_file) }: try: entries.remove(entry) except KeyError: ch.FATAL("%s: missing file or directory: %s" % (msg_prefix, entry)) # Ignore some files that may or may not exist. entries -= { i.name for i in (self.lockfile, self.mount_point) } # Delete some files that exist only if we crashed. for i in (self.image_tmp, ): if (i.name in entries): ch.WARNING("deleting leftover temporary file/dir: %s" % i.name) i.rmtree() entries.remove(i.name) # If anything is left, yell about it. if (len(entries) > 0): ch.FATAL("%s: extraneous file(s): %s" % (msg_prefix, " ".join(sorted(entries)))) # check version v_found = self.version_read() if (v_found != STORAGE_VERSION): ch.FATAL("%s: version mismatch: %d expected, %d found" % (msg_prefix, STORAGE_VERSION, v_found)) # check that no image directories have “:” in filename assert isinstance(self.unpack_base, Path) # remove if test suite passes imgs = self.unpack_base.listdir() imgs_bad = set() for img in imgs: if (":" in img): # bad char check b/c problem here is bad upgrade ch.FATAL("%s: storage directory broken: bad image dir name: %s" % (msg_prefix, img), ch.BUG_REPORT_PLZ) def version_read(self): # While support for storage v1 was dropped some time ago, let’s at least # retain the ability to recognize it. if (not os.path.isfile(self.version_file)): return 1 text = self.version_file.file_read_all() try: return int(text) except ValueError: ch.FATAL('malformed storage version: "%s"' % text) class TarFile(tarfile.TarFile): # This subclass augments tarfile.TarFile to add safety code. While the # tarfile module docs [1] say “do not use this class [TarFile] directly”, # they also say “[t]he tarfile.open() function is actually a shortcut” to # class method TarFile.open(), and the source code recommends subclassing # TarFile [2]. # # It’s here because the standard library class has problems with symlinks # and replacing one file type with another; see issues #819 and #825 as # well as multiple unfixed Python bugs [e.g. 3,4,5]. We work around this # with manual deletions. # # [1]: https://docs.python.org/3/library/tarfile.html # [2]: https://github.com/python/cpython/blob/2bcd0fe7a5d1a3c3dd99e7e067239a514a780402/Lib/tarfile.py#L2159 # [3]: https://bugs.python.org/issue35483 # [4]: https://bugs.python.org/issue19974 # [5]: https://bugs.python.org/issue23228 @staticmethod def fix_link_target(ti, tb): """Deal with link (symbolic or hard) weirdness or breakage. If it can be fixed, fix it; if not, abort the program.""" src = Path(ti.name) tgt = Path(ti.linkname) fix_ct = 0 # Empty target not allowed; have to check string b/c "" -> Path("."). if (len(ti.linkname) == 0): ch.FATAL("rejecting link with empty target: %s: %s" % (tb, ti.name)) # Fix absolute link targets. if (tgt.is_absolute()): if (ti.issym()): # Change symlinks to relative for correct interpretation inside or # outside the container. kind = "symlink" new = ( Path(*(("..",) * (len(src.parts) - 1))) // Path(*(tgt.parts[1:]))) elif (ti.islnk()): # Hard links refer to tar member paths; just strip leading slash. kind = "hard link" new = tgt.relative_to("/") else: assert False, "not a link" ch.DEBUG("absolute %s: %s -> %s: changing target to: %s" % (kind, src, tgt, new)) tgt = new fix_ct = 1 # Reject links that climb out of image (FIXME: repair instead). if (".." in os.path.normpath(src // tgt).split("/")): ch.FATAL("rejecting too many up-levels: %s: %s -> %s" % (tb, src, tgt)) # Done. ti.linkname = str(tgt) return fix_ct @staticmethod def fix_member_uidgid(ti): assert (ti.name[0] != "/") # absolute paths unsafe but shouldn’t happen if (not (ti.isfile() or ti.isdir() or ti.issym() or ti.islnk())): ch.FATAL("invalid file type: %s" % ti.name) ti.uid = 0 ti.uname = "root" ti.gid = 0 ti.gname = "root" if (ti.mode & stat.S_ISUID): ch.VERBOSE("stripping unsafe setuid bit: %s" % ti.name) ti.mode &= ~stat.S_ISUID if (ti.mode & stat.S_ISGID): ch.VERBOSE("stripping unsafe setgid bit: %s" % ti.name) ti.mode &= ~stat.S_ISGID # Need new method name because add() is called recursively and we don’t # want those internal calls to get our special sauce. def add_(self, name, **kwargs): def filter_(ti): assert (ti.name == "." or ti.name[:2] == "./") if (ti.name in ("./ch/git", "./ch/git.pickle")): ch.DEBUG("omitting from push: %s" % ti.name) return None self.fix_member_uidgid(ti) return ti kwargs["filter"] = filter_ super().add(name, **kwargs) def clobber(self, targetpath, regulars=False, symlinks=False, dirs=False): assert (regulars or symlinks or dirs) try: st = os.lstat(targetpath) except FileNotFoundError: # We could move this except clause after all the stat.S_IS* calls, # but that risks catching FileNotFoundError that came from somewhere # other than lstat(). st = None except OSError as x: ch.FATAL("can’t lstat: %s" % targetpath, targetpath) if (st is not None): if (stat.S_ISREG(st.st_mode)): if (regulars): Path(targetpath).unlink() elif (stat.S_ISLNK(st.st_mode)): if (symlinks): Path(targetpath).unlink() elif (stat.S_ISDIR(st.st_mode)): if (dirs): Path(targetpath).rmtree() else: ch.FATAL("invalid file type 0%o in previous layer; see inode(7): %s" % (stat.S_IFMT(st.st_mode), targetpath)) def makedir(self, tarinfo, targetpath): # Note: This gets called a lot, e.g. once for each component in the path # of the member being extracted. ch.TRACE("makedir: %s" % targetpath) self.clobber(targetpath, regulars=True, symlinks=True) super().makedir(tarinfo, targetpath) def makefile(self, tarinfo, targetpath): ch.TRACE("makefile: %s" % targetpath) self.clobber(targetpath, symlinks=True, dirs=True) super().makefile(tarinfo, targetpath) def makelink(self, tarinfo, targetpath): ch.TRACE("makelink: %s -> %s" % (targetpath, tarinfo.linkname)) self.clobber(targetpath, regulars=True, symlinks=True, dirs=True) super().makelink(tarinfo, targetpath) charliecloud-0.37/lib/force.py000066400000000000000000000433211457016721300163300ustar00rootroot00000000000000import re import charliecloud as ch import filesystem as fs ## Globals ## FAKEROOT_DEFAULT_CONFIGS = { # General notes: # # 1. Semantics of these configurations. (Character limits are to support # tidy code and message formatting.) # # a. This is a dictionary of configurations, which themselves are # dictionaries. # # b. Key is an arbitrary tag; user-visible. There’s no enforced # character set but let’s stick with [a-z0-9_] for now and limit to # at most 10 characters. # # c. A configuration has the following keys. # # name ... Human-readable name for the configuration. Max 46 chars. # # match .. Tuple; first item is the name of a file and the second is # a regular expression. If the regex matches any line in the # file, that configuration is used for the image. # # init ... List of tuples containing POSIX shell commands to perform # fakeroot installation and any other initialization steps. # # Item 1: Command to detect if the step is necessary. If the # command exits successfully, the step is already # complete; if unsuccessful, it is still needed. The sense # of the test is so something like "is command FOO # available?", which seems the most common command, does # not require negation. # # The test should be fairly permissive; e.g., if the image # already has a fakeroot implementation installed, but # it’s a different one than we would have chosen, the # command should succeed. # # IMPORTANT: This command must have no side effects, # because it is normally run in all matching images, even # if --force is not specified. Note that talking to the # internet is a side effect! # # Item 2: Command to do the init step. # # I.e., to perform each fakeroot initialization step, # ch-image does roughly: # # if ( ! $CMD_1 ); then # $CMD_2 # fi # # For both commands, the output is visible to the user but # is not analyzed. # # cmds ... List of RUN command words that need fakeroot injection. # Each item in the list is matched against each # whitespace-separated word in the RUN instructions. For # example, suppose that each is the list “dnf”, “rpm”, and # “yum”; consider the following RUN instructions: # # RUN ['dnf', 'install', 'foo'] # RUN dnf install foo # # These are fairly standard forms. “dnf” matches both, the # first on the first element in the list and the second # after breaking the shell command on whitespace. # # RUN true&&dnf install foo # # This third example does *not* match (false negative) # because breaking on whitespace yields “true&&dnf”, # “install”, and “foo”; none of these words are “dnf”. # # RUN echo dnf install foo # # This final example *does* match (false positive) becaus # the second word *is* “dnf”; the algorithm isn’t smart # enough to realize that it’s an argument to “echo”. # # The last two illustrate that the algorithm uses simple # whitespace delimiters, not even a partial shell parser. # # each ... List of words to prepend to RUN instructions that match # cmd_each. For example, if each is ["fr", "-z"], then these # instructions: # # RUN ['dnf', 'install', 'foo'] # RUN dnf install foo # # become: # # RUN ['fr', '-z', 'dnf', 'install', 'foo'] # RUN ['fr', '-z', '/bin/sh', '-c', 'dnf install foo'] # # (Note that “/bin/sh -c” is how shell-form RUN instructions # are executed regardless of --force.) # # 2. The first match wins. However, because dictionary ordering can’t be # relied on yet, since it was introduced in Python 3.6 [1], matches # should be disjoint. # # [1]: https://docs.python.org/3/library/stdtypes.html#dict # # 3. A matching configuration is considered applicable if any of the # fakeroot-able commands are present. We do nothing if the config isn’t # applicable. We do not look for other matches. # # 4. There are three implementations of fakeroot that I could find: # fakeroot, fakeroot-ng, and pseudo. As of 2020-09-02: # # * fakeroot-ng and pseudo use a daemon process, while fakeroot does # not. pseudo also uses a persistent database. # # * fakeroot-ng does not support ARM; pseudo supports many architectures # including ARM. # # * “Old” fakeroot seems to have had version 1.24 on 2019-09-07 with # the most recent commit 2020-08-12. # # * fakeroot-ng is quite old: last upstream release was 0.18 in 2013, # and its source code is on Sourceforge. # # * pseudo is aslo a bit old: last upstream version was 1.9.0 on # 2018-01-20, and the last Git commit was 2019-08-02. # # Generally, we select the first one that seems to work in the order # fakeroot, pseudo, fakeroot-ng. # # 5. Why grep a specified file vs. simpler alternatives? # # * Look at image name: Misses derived images, large number of names # seems a maintenance headache, :latest changes. # # * grep the same file for each distro: No standardized file for this. # [FIXME: This may be wrong; see issue #1292.] # # * Ask lsb_release(1): Not always installed, requires executing ch-run. # Fedora notes: # # 1. The minimum supported version was chosen somewhat arbitrarily based on # versions available for testing, i.e., what was on Docker Hub. # # 2. The fakeroot package is in the base repository set so enabling EPEL is # not required. # # 3. Must be before “rhel8” because that matches Fedora too. "fedora": { "name": "Fedora 24+", "match": ("/etc/fedora-release", r"release (?!1?[0-9] |2[0-3] )"), "init": [ ("command -v fakeroot > /dev/null", "dnf install -y fakeroot") ], "cmds": ["dnf", "rpm", "yum"], "each": ["fakeroot"] }, # RHEL (and rebuilds like CentOS, Alma, Springdale, Rocky) notes: # # 1. These seem to have only fakeroot, which is in EPEL, not the standard # repos. # # 2. Unlike some derivatives, RHEL itself doesn’t have the epel-release rpm # in the standard repos; install via rpm for both to be consistent. # # 3. Enabling EPEL can have undesirable side effects, e.g. different # version of things in the base repo that breaks other things. Thus, # when we are done with EPEL, we uninstall it. Existing EPEL # installations are left alone. (Such breakage is an EPEL bug, but we do # commonly encounter it.) # # 4. “yum repolist” has a lot of side effects, e.g. locking the RPM # database and asking configured repos for something or other. "rhel7": { "name": "RHEL 7 and derivatives", "match": ("/etc/redhat-release", r"release 7\."), "init": [ ("command -v fakeroot > /dev/null", "set -e; " r"if ! grep -Eq '\[epel\]' /etc/yum.conf /etc/yum.repos.d/*; then " "yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm; " "yum install -y fakeroot; " "yum remove -y epel-release; " "else " "yum install -y fakeroot; " "fi; ") ], "cmds": ["dnf", "rpm", "yum"], "each": ["fakeroot"] }, "rhel8": { "name": "RHEL 8+ and derivatives", "match": ("/etc/redhat-release", r"release (?![0-7]\.)"), "init": [ ("command -v fakeroot > /dev/null", "set -e; " r"if ! grep -Eq '\[epel\]' /etc/yum.conf /etc/yum.repos.d/*; then " # Macro %rhel from *-release* RPM, e.g. redhat-release-server # or centos-linux-release; thus reliable. "dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(rpm -E %rhel).noarch.rpm; " "dnf install -y fakeroot; " "dnf remove -y epel-release; " "else " "dnf install -y fakeroot; " "fi; ") ], "cmds": ["dnf", "rpm", "yum"], "each": ["fakeroot"] }, # Debian/Ubuntu notes: # # 1. In recent Debian-based distributions apt(8) runs as an unprivileged # user by default. This makes *all* apt operations fail in an # unprivileged container because it can’t drop privileges. There are # multiple ways to turn the “sandbox” off. AFAICT, none are documented, # but this one at least appears in Google searches a lot. # # apt also doesn’t drop privileges if there is no user _apt; in my # testing, sometimes this user is present and sometimes not, for reasons # I don’t understand. If not present, you get this warning: # # W: No sandbox user '_apt' on the system, can not drop privileges # # Configuring apt not to use the sandbox seemed cleaner than deleting # this user and eliminates the warning. # # 2. If we wanted to test if a fakeroot package was installed, we could say: # # dpkg-query -Wf '${Package}\n' \ # | egrep '^(fakeroot|fakeroot-ng|pseudo)$' "debderiv": { "name": "Debian 9+, Ubuntu 14+, or other derivative", "match": ("/etc/os-release", r"Debian GNU/Linux (?![0-8] )|Ubuntu (?![0-9]\.|1[0-3]\.)|ID_LIKE=debian"), "init": [ ("apt-config dump | fgrep -q 'APT::Sandbox::User \"root\"'" " || ! fgrep -q _apt /etc/passwd", "echo 'APT::Sandbox::User \"root\";'" " > /etc/apt/apt.conf.d/no-sandbox"), ("command -v fakeroot > /dev/null", # update b/c base image ships with no package indexes "apt-get update && apt-get install -y fakeroot") ], "cmds": ["apt", "apt-get", "dpkg"], "each": ["fakeroot"] }, "suse": { "name": "(Open)SUSE 42.2+", # no fakeroot before this # I don’t know if there are OpenSUSE derivatives "match": ("/etc/os-release", r"ID_LIKE=.*suse"), "init": [ ("command -v fakeroot > /dev/null", # fakeroot seems to have a missing dependency, otherwise # failing with missing getopt in the fakeroot script. "zypper refresh; zypper install -y fakeroot /usr/bin/getopt") ], "cmds": ["zypper", "rpm"], "each": ["fakeroot"] }, # pacman doesn’t seem to have proper dependencies like dpkg and rpm. It # happens that fakeroot can fail because the downloaded version is linked # against a newer glibc (at least) than is in the base image. We could also # update glibc (and its dependency util-linux), but that causes the tests # to fail with fchownat() errors. Another possiblity is to add “-u” to # update all installed packages, but that may not be what a user wants. "arch": { "name": "Arch Linux", "match": ("/etc/os-release", r"ID=arch"), # /etc/arch-release empty "init": [ ("command -v fakeroot > /dev/null", "pacman -Syq --noconfirm fakeroot") ], "cmds": ["pacman"], "each": ["fakeroot"] }, "alpine": { "name": "Alpine, any version", "match": ("/etc/alpine-release", r"[0-9]\.[0-9]+\.[0-9]+"), "init": [ ("command -v fakeroot > /dev/null", "apk update; apk add fakeroot") ], "cmds": ["apk"], "each": ["fakeroot"] }, } # Default value of --force-cmd. # # NOTE: apt(8) tells people not to use it in scripts, but they do it anyway. FORCE_CMD_DEFAULT = { "apt": ["-o", "APT::Sandbox::User=root"], "apt-get": ["-o", "APT::Sandbox::User=root"] } ## Functions ### def force_cmd_parse(text): # 1. Split on “,” preceded by even number of backslashes. # # FIXME: Said backslashes are removed in the split, so you can’t have a # component with trailing backslashes. That seems rare so I’m not fixing # for now. args = re.split(r"(? STRING_QUOTED """ # Where the .git “directory” in the image is located. (Normally it’s a # directory, and that’s what the Git docs call it, but it’s a file for # worktrees.) We deliberately do not call it “.git” because that makes it # hidden, but also more importantly it confuses Git into thinking /ch is a # different Git repo. GIT_DIR = ch.Path("ch/git") # Dockerfile grammar. Note image references are not parsed during Dockerfile # parsing. GRAMMAR_DOCKERFILE = r""" start: dockerfile // First instruction must be ARG or FROM, but that is not a syntax error. dockerfile: _NEWLINES? ( arg_first | directive | comment )* ( instruction | comment )* ?instruction: _WS? ( arg | copy | env | from_ | label | rsync | run | shell | workdir | uns_forever | uns_yet ) directive.2: _WS? "#" _WS? DIRECTIVE_NAME "=" _line _NEWLINES DIRECTIVE_NAME: ( "escape" | "syntax" ) comment: _WS? _COMMENT_BODY _NEWLINES _COMMENT_BODY: /#[^\n]*/ arg: "ARG"i _WS ( arg_bare | arg_equals ) _NEWLINES arg_bare: WORD arg_equals: WORD "=" ( WORD | STRING_QUOTED ) arg_first.2: "ARG"i _WS ( arg_first_bare | arg_first_equals ) _NEWLINES arg_first_bare: WORD arg_first_equals: WORD "=" ( WORD | STRING_QUOTED ) copy: "COPY"i ( _WS option )* _WS ( copy_list | copy_shell ) _NEWLINES copy_list.2: _string_list copy_shell: WORD ( _WS WORD )+ env: "ENV"i _WS ( env_space | env_equalses ) _NEWLINES env_space: WORD _WS _line env_equalses: env_equals ( _WS env_equals )* env_equals: WORD "=" ( WORD | STRING_QUOTED ) from_: "FROM"i ( _WS ( option | option_keypair ) )* _WS image_ref ( _WS from_alias )? _NEWLINES from_alias: "AS"i _WS IR_PATH_COMPONENT // FIXME: undocumented; this is guess label: "LABEL"i _WS ( label_space | label_equalses ) _NEWLINES label_space: WORD _WS _line label_equalses: label_equals ( _WS label_equals )* label_equals: WORD "=" ( WORD | STRING_QUOTED ) rsync: ( "RSYNC"i | "NSYNC"i ) ( _WS option_plus )? _WS WORDE ( _WS WORDE )+ _NEWLINES run: "RUN"i _WS ( run_exec | run_shell ) _NEWLINES run_exec.2: _string_list run_shell: _line shell: "SHELL"i _WS _string_list _NEWLINES workdir: "WORKDIR"i _WS _line _NEWLINES uns_forever: UNS_FOREVER _WS _line _NEWLINES UNS_FOREVER: ( "EXPOSE"i | "HEALTHCHECK"i | "MAINTAINER"i | "STOPSIGNAL"i | "USER"i | "VOLUME"i ) uns_yet: UNS_YET _WS _line _NEWLINES UNS_YET: ( "ADD"i | "CMD"i | "ENTRYPOINT"i | "ONBUILD"i ) /// Common /// option: "--" OPTION_KEY "=" OPTION_VALUE option_keypair: "--" OPTION_KEY "=" OPTION_VAR "=" OPTION_VALUE option_plus: "+" OPTION_LETTER OPTION_KEY: /[a-z]+/ OPTION_LETTER: /[a-z]/ OPTION_VALUE: /[^= \t\n]+/ OPTION_VAR: /[a-z]+/ image_ref: IMAGE_REF IMAGE_REF: /[${}A-Za-z0-9:._\/-]+/ // variable substitution chars ${} added """ + GRAMMAR_COMMON # Grammar for image references. GRAMMAR_IMAGE_REF = r""" // Note: Hostnames with no dot and no port get parsed as a hostname, which // is wrong; it should be the first path component. We patch this error later. // FIXME: Supposedly this can be fixed with priorities, but I couldn’t get it // to work with brief trying. start: image_ref image_ref: ir_hostport? ir_path? ir_name ( ir_tag | ir_digest )? ir_hostport: IR_HOST ( ":" IR_PORT )? "/" ir_path: ( IR_PATH_COMPONENT "/" )+ ir_name: IR_PATH_COMPONENT ir_tag: ":" IR_TAG ir_digest: "@sha256:" HEX_STRING IR_HOST: /[A-Za-z0-9_.-]+/ IR_PORT: /[0-9]+/ IR_TAG: /[A-Za-z0-9_.-]+/ """ + GRAMMAR_COMMON # Top-level directories we create if not present. STANDARD_DIRS = { "bin", "dev", "etc", "mnt", "proc", "sys", "tmp", "usr" } # Width of token name when truncating text to fit on screen. WIDTH_TOKEN_MAX = 10 ## Classes ## class Image: """Container image object. Constructor arguments: ref........... Reference object to identify the image. unpack_path .. Directory to unpack the image in; if None, infer path in storage dir from ref.""" __slots__ = ("metadata", "ref", "unpack_path") def __init__(self, ref, unpack_path=None): if (isinstance(ref, str)): ref = Reference(ref) assert isinstance(ref, Reference) self.ref = ref if (unpack_path is not None): assert isinstance(unpack_path, fs.Path) self.unpack_path = unpack_path else: self.unpack_path = ch.storage.unpack(self.ref) self.metadata_init() @classmethod def glob(class_, image_glob): """Return a possibly-empty iterator of images in the storage directory matching the given glob.""" for ref in Reference.glob(image_glob): yield class_(ref) def __str__(self): return str(self.ref) @property def deleteable(self): """True if it’s OK to delete me, either my unpack directory (a) is at the expected location within the storage directory xor (b) is not not but it looks like an image; False otherwise.""" if (self.unpack_path == ch.storage.unpack_base // self.unpack_path.name): return True else: if (all(os.path.isdir(self.unpack_path // i) for i in ("bin", "dev", "usr"))): return True return False @property def last_modified(self): # Return the last modified time of self as a datetime.datetime object in # the local time zone. return datetime.datetime.fromtimestamp( (self.metadata_path // "metadata.json").stat(False).st_mtime, datetime.timezone.utc).astimezone() @property def metadata_path(self): return self.unpack_path // "ch" @property def unpack_cache_linked(self): return (self.unpack_path // GIT_DIR).exists() @property def unpack_exist_p(self): return os.path.exists(self.unpack_path) def commit(self): "Commit the current unpack directory into the layer cache." assert False, "unimplemented" def copy_unpacked(self, other): """Copy image other to my unpack directory, which may not exist. other can be either a path (string or fs.Path object) or an Image object; in the latter case other.unpack_path is used. other need not be a valid image; the essentials will be created if needed.""" if (isinstance(other, str) or isinstance(other, fs.Path)): src_path = other else: src_path = other.unpack_path ch.VERBOSE("copying image: %s -> %s" % (src_path, self.unpack_path)) fs.Path(src_path).copytree(self.unpack_path, symlinks=True) # Simpler to copy this file then delete it, rather than filter it out. (self.unpack_path // GIT_DIR).unlink(missing_ok=True) self.unpack_init() def layers_open(self, layer_tars): """Open the layer tarballs and read some metadata (which unfortunately means reading the entirety of every file). Return an OrderedDict: keys: layer hash (full) values: namedtuple with two fields: fp: open TarFile object members: sequence of members (OrderedSet) Empty layers are skipped. Important note: TarFile.extractall() extracts the given members in the order they are specified, so we need to preserve their order from the file, as returned by getmembers(). We also need to quickly remove members we don’t want from this sequence. Thus, we use the OrderedSet class defined in this module.""" TT = collections.namedtuple("TT", ["fp", "members"]) layers = collections.OrderedDict() # Schema version one (v1) allows one or more empty layers for Dockerfile # entries like CMD (https://github.com/containers/skopeo/issues/393). # Unpacking an empty layer doesn’t accomplish anything, so ignore them. empty_cnt = 0 for (i, path) in enumerate(layer_tars, start=1): lh = os.path.basename(path).split(".", 1)[0] lh_short = lh[:7] ch.INFO("layer %d/%d: %s: listing" % (i, len(layer_tars), lh_short)) try: fp = fs.TarFile.open(path) members = ch.OrderedSet(fp.getmembers()) # reads whole file :( except tarfile.TarError as x: ch.FATAL("cannot open: %s: %s" % (path, x)) if (lh in layers and len(members) > 0): ch.WARNING("ignoring duplicate non-empty layer: %s" % lh_short) if (len(members) > 0): layers[lh] = TT(fp, members) else: ch.WARNING("ignoring empty layer: %s" % lh_short) empty_cnt += 1 ch.VERBOSE("skipped %d empty layers" % empty_cnt) return layers def metadata_init(self): "Initialize empty metadata structure." # Elsewhere can assume the existence and types of everything here. self.metadata = { "arch": ch.arch_host.split("/")[0], # no variant "arg": { **ARG_DEFAULTS_MAGIC, **ARG_DEFAULTS }, "cwd": "/", "env": dict(), "history": list(), "labels": dict(), "shell": ["/bin/sh", "-c"], "volumes": list() } # set isn’t JSON-serializable def metadata_load(self, target_img=None): """Load metadata file, replacing the existing metadata object. If metadata doesn’t exist, warn and use defaults. If target_img is non-None, use that image’s metadata instead of self’s.""" if (target_img is not None): path = target_img.metadata_path else: path = self.metadata_path path //= "metadata.json" if (path.exists()): ch.VERBOSE("loading metadata") else: ch.WARNING("no metadata to load; using defaults") self.metadata_init() return self.metadata = path.json_from_file("metadata") # upgrade old metadata self.metadata.setdefault("arg", dict()) self.metadata.setdefault("history", list()) # add default ARG variables self.metadata["arg"].update({ **ARG_DEFAULTS_MAGIC, **ARG_DEFAULTS }) def metadata_merge_from_config(self, config): """Interpret all the crap in the config data structure that is meaningful to us, and add it to self.metadata. Ignore anything we expect in config that’s missing.""" def get(*keys): d = config keys = list(keys) while (len(keys) > 1): try: d = d[keys.pop(0)] except KeyError: return None assert (len(keys) == 1) return d.get(keys[0]) def set_(dst_key, *src_keys): v = get(*src_keys) if (v is not None and v != ""): self.metadata[dst_key] = v if ("config" not in config): ch.FATAL("config missing key 'config'") # architecture set_("arch", "architecture") # $CWD set_("cwd", "config", "WorkingDir") # environment env = get("config", "Env") if (env is not None): for line in env: try: (k,v) = line.split("=", maxsplit=1) except AttributeError: ch.FATAL("can’t parse config: bad Env line: %s" % line) self.metadata["env"][k] = v # History. if ("history" not in config): ch.FATAL("invalid config: missing history") self.metadata["history"] = config["history"] # labels set_("labels", "config", "Labels") # copy reference # shell set_("shell", "config", "Shell") # Volumes. FIXME: Why is this a dict with empty dicts as values? vols = get("config", "Volumes") if (vols is not None): for k in config["config"]["Volumes"].keys(): self.metadata["volumes"].append(k) def metadata_replace(self, config_json): self.metadata_init() if (config_json is None): ch.INFO("no config found; initializing empty metadata") else: # Copy pulled config file into the image so we still have it. path = self.metadata_path // "config.pulled.json" config_json.copy(path) ch.VERBOSE("pulled config path: %s" % path) self.metadata_merge_from_config(path.json_from_file("config")) self.metadata_save() def metadata_save(self): """Dump image’s metadata to disk, including the main data structure but also all auxiliary files, e.g. ch/environment.""" # Adjust since we don’t save everything. metadata = copy.deepcopy(self.metadata) for k in ARGS_MAGIC: metadata["arg"].pop(k, None) # Serialize. We take care to pretty-print this so it can (sometimes) be # parsed by simple things like grep and sed. out = json.dumps(metadata, indent=2, sort_keys=True) ch.DEBUG("metadata:\n%s" % out) # Main metadata file. path = self.metadata_path // "metadata.json" ch.VERBOSE("writing metadata file: %s" % path) path.file_write(out + "\n") # /ch/environment path = self.metadata_path // "environment" ch.VERBOSE("writing environment file: %s" % path) path.file_write( ( "\n".join("%s=%s" % (k,v) for (k,v) in sorted(metadata["env"].items())) + "\n")) # mkdir volumes ch.VERBOSE("ensuring volume directories exist") for path in metadata["volumes"]: (self.unpack_path // path).mkdirs() def tarballs_write(self, tarball_dir): """Write one uncompressed tarball per layer to tarball_dir. Return a sequence of tarball basenames, with the lowest layer first.""" # FIXME: Yes, there is only one layer for now and we’ll need to update # it when (if) we have multiple layers. But, I wanted the interface to # support multiple layers. base = "%s.tar" % self.ref.for_path path = tarball_dir // base try: ch.INFO("layer 1/1: gathering") ch.VERBOSE("writing tarball: %s" % path) fp = fs.TarFile.open(path, "w", format=tarfile.PAX_FORMAT) unpack_path = self.unpack_path.resolve() # aliases use symlinks ch.VERBOSE("canonicalized unpack path: %s" % unpack_path) fp.add_(unpack_path, arcname=".") fp.close() except OSError as x: ch.FATAL("can’t write tarball: %s" % x.strerror) return [base] def unpack(self, layer_tars, last_layer=None): """Unpack config_json (path to JSON config file) and layer_tars (sequence of paths to tarballs, with lowest layer first) into the unpack directory, validating layer contents and dealing with whiteouts. Empty layers are ignored. The unpack directory must not exist.""" if (last_layer is None): last_layer = sys.maxsize ch.INFO("flattening image") self.unpack_layers(layer_tars, last_layer) self.unpack_init() def unpack_cache_unlink(self): (self.unpack_path // ".git").unlink() def unpack_clear(self): """If the unpack directory does not exist, do nothing. If the unpack directory is already an image, remove it. Otherwise, error.""" if (not os.path.exists(self.unpack_path)): ch.VERBOSE("no image found: %s" % self.unpack_path) else: if (not os.path.isdir(self.unpack_path)): ch.FATAL("can’t flatten: %s exists but is not a directory" % self.unpack_path) if (not self.deleteable): ch.FATAL("can’t flatten: %s exists but does not appear to be an image" % self.unpack_path) ch.VERBOSE("removing image: %s" % self.unpack_path) t = ch.Timer() self.unpack_path.rmtree() t.log("removed image") def unpack_delete(self): ch.VERBOSE("unpack path: %s" % self.unpack_path) if (not self.unpack_exist_p): ch.FATAL("image not found, can’t delete: %s" % self.ref) if (self.deleteable): ch.INFO("deleting image: %s" % self.ref) self.unpack_path.chmod_min() for (dir_, subdirs, _) in os.walk(self.unpack_path): # must fix as subdirs so we can traverse into them for subdir in subdirs: (fs.Path(dir_) // subdir).chmod_min() self.unpack_path.rmtree() else: ch.FATAL("storage directory seems broken: not an image: %s" % self.ref) def unpack_init(self): """Initialize the unpack directory, which must exist. Any setup already present will be left unchanged. After this, self.unpack_path is a valid Charliecloud image directory.""" # Metadata directory. (self.unpack_path // "ch").mkdir() (self.unpack_path // "ch/environment").file_ensure_exists() # Essential directories & mount points. Do nothing if something already # exists, without dereferencing, in case it’s a symlink, which will work # for bind-mount later but won’t resolve correctly now outside the # container (e.g. linuxcontainers.org images; issue #1015). # # WARNING: Keep in sync with shell scripts. for d in list(STANDARD_DIRS) + ["mnt/%d" % i for i in range(10)]: d = self.unpack_path // d if (not os.path.lexists(d)): d.mkdirs() (self.unpack_path // "etc/hosts").file_ensure_exists() (self.unpack_path // "etc/resolv.conf").file_ensure_exists() def unpack_layers(self, layer_tars, last_layer): layers = self.layers_open(layer_tars) self.validate_members(layers) self.whiteouts_resolve(layers) self.unpack_path.mkdir() # create directory in case no layers for (i, (lh, (fp, members))) in enumerate(layers.items(), start=1): lh_short = lh[:7] if (i > last_layer): ch.INFO("layer %d/%d: %s: skipping per --last-layer" % (i, len(layers), lh_short)) else: ch.INFO("layer %d/%d: %s: extracting" % (i, len(layers), lh_short)) try: fp.extractall(path=self.unpack_path, members=members) except OSError as x: ch.FATAL("can’t extract layer %d: %s" % (i, x.strerror)) def validate_members(self, layers): ch.INFO("validating tarball members") top_dirs = set() ch.VERBOSE("pass 1: canonicalizing member paths") for (i, (lh, (fp, members))) in enumerate(layers.items(), start=1): abs_ct = 0 for m in list(members): # copy b/c we remove items from the set # Remove members with empty paths. if (len(m.name) == 0): ch.WARNING("layer %d/%d: %s: skipping member with empty path" % (i, len(layers), lh[:7])) members.remove(m) # Convert member paths to fs.Path objects for easier processing. # Note: In my testing, parsing a string into a fs.Path object took # about 2.5µs, so this should be plenty fast. m.name = fs.Path(m.name) # Reject members with up-levels. if (".." in m.name.parts): ch.FATAL("rejecting up-level member: %s: %s" % (fp.name, m.name)) # Correct absolute paths. if (m.name.is_absolute()): m.name = m.name.relative_to("/") abs_ct += 1 # Record top-level directory. if (len(m.name.parts) > 1 or m.isdir()): top_dirs.add(m.name.first) if (abs_ct > 0): ch.WARNING("layer %d/%d: %s: fixed %d absolute member paths" % (i, len(layers), lh[:7], abs_ct)) top_dirs.discard(None) # ignore “.” # Convert to tarbomb if (1) there is a single enclosing directory and # (2) that directory is not one of the standard directories, e.g. to # allow images containing just “/bin/fooprog”. if (len(top_dirs) != 1 or not top_dirs.isdisjoint(STANDARD_DIRS)): ch.VERBOSE("pass 2: conversion to tarbomb not needed") else: ch.VERBOSE("pass 2: converting to tarbomb") for (i, (lh, (fp, members))) in enumerate(layers.items(), start=1): for m in members: if (len(m.name.parts) > 0): # ignore “.” m.name = fs.Path(*m.name.parts[1:]) # strip first component ch.VERBOSE("pass 3: analyzing members") for (i, (lh, (fp, members))) in enumerate(layers.items(), start=1): dev_ct = 0 link_fix_ct = 0 for m in list(members): # copy again m.name = str(m.name) # other code assumes strings if (m.isdev()): # Device or FIFO: Ignore. dev_ct += 1 ch.VERBOSE("ignoring device file: %s" % m.name) members.remove(m) continue elif (m.issym() or m.islnk()): link_fix_ct += fs.TarFile.fix_link_target(m, fp.name) elif (m.isdir()): # Directory: Fix bad permissions (hello, Red Hat). m.mode |= 0o700 elif (m.isfile()): # Regular file: Fix bad permissions (HELLO RED HAT!!). m.mode |= 0o600 else: ch.FATAL("unknown member type: %s" % m.name) # Discard Git metadata (files that begin with “.git”). if (re.search(r"^(\./)?\.git", m.name)): ch.WARNING("ignoring member: %s" % m.name) members.remove(m) continue # Discard anything under /dev. Docker puts regular files and # directories in here on “docker export”. Note leading slashes # already taken care of in TarFile.fix_member_path() above. if (re.search(r"^(\./)?dev/.", m.name)): ch.VERBOSE("ignoring member under /dev: %s" % m.name) members.remove(m) continue fs.TarFile.fix_member_uidgid(m) if (dev_ct > 0): ch.WARNING("layer %d/%d: %s: ignored %d devices and/or FIFOs" % (i, len(layers), lh[:7], dev_ct)) if (link_fix_ct > 0): ch.INFO("layer %d/%d: %s: changed %d absolute symbolic and/or hard links to relative" % (i, len(layers), lh[:7], link_fix_ct)) def whiteout_rm_prefix(self, layers, max_i, prefix): """Ignore members of all layers from 1 to max_i inclusive that have path prefix of prefix. For example, if prefix is foo/bar, then ignore foo/bar and foo/bar/baz but not foo/barbaz. Return count of members ignored.""" ch.TRACE("finding members with prefix: %s" % prefix) prefix = os.path.normpath(prefix) # "./foo" == "foo" ignore_ct = 0 for (i, (lh, (fp, members))) in enumerate(layers.items(), start=1): if (i > max_i): break members2 = list(members) # copy b/c we’ll alter members for m in members2: if (ch.prefix_path(prefix, m.name)): ignore_ct += 1 members.remove(m) ch.TRACE("layer %d/%d: %s: ignoring %s" % (i, len(layers), lh[:7], m.name)) return ignore_ct def whiteouts_resolve(self, layers): """Resolve whiteouts. See: https://github.com/opencontainers/image-spec/blob/master/layer.md""" ch.INFO("resolving whiteouts") for (i, (lh, (fp, members))) in enumerate(layers.items(), start=1): wo_ct = 0 ig_ct = 0 members2 = list(members) # copy b/c we’ll alter members for m in members2: dir_ = os.path.dirname(m.name) filename = os.path.basename(m.name) if (filename.startswith(".wh.")): wo_ct += 1 members.remove(m) if (filename == ".wh..wh..opq"): # “Opaque whiteout”: remove contents of dir_. ch.DEBUG("found opaque whiteout: %s" % m.name) ig_ct += self.whiteout_rm_prefix(layers, i - 1, dir_) else: # “Explicit whiteout”: remove same-name file without ".wh.". ch.DEBUG("found explicit whiteout: %s" % m.name) ig_ct += self.whiteout_rm_prefix(layers, i - 1, dir_ + "/" + filename[4:]) if (wo_ct > 0): ch.VERBOSE("layer %d/%d: %s: %d whiteouts; %d members ignored" % (i, len(layers), lh[:7], wo_ct, ig_ct)) class Reference: """Reference to an image in a remote repository. The constructor takes one argument, which is interpreted differently depending on type: None or omitted... Build an empty Reference (all fields None). string ........... Parse it; see FAQ for syntax. Can be either the standard form (e.g., as in a FROM instruction) or our filename form with percents replacing slashes. Lark parse tree .. Must be same result as parsing a string. This allows the parse step to be embedded in a larger parse (e.g., a Dockerfile). Warning: References containing a hostname without a dot and no port cannot be round-tripped through a string, because the hostname will be assumed to be a path component.""" __slots__ = ("host", "port", "path", "name", "tag", "digest", "variables") # Reference parser object. Instantiating a parser took 100ms when we tested # it, which means we can’t really put it in a loop. But, at parse time, # “lark” may refer to a dummy module (see above), so we can’t populate the # parser here either. We use a class varible and populate it at the time of # first use. parser = None def __init__(self, src=None, variables=None): self.host = None self.port = None self.path = [] self.name = None self.tag = None self.digest = None self.variables = dict() if variables is None else variables if (isinstance(src, str)): src = self.parse(src, self.variables) if (isinstance(src, lark.tree.Tree)): self.from_tree(src) elif (src is not None): assert False, "unsupported initialization type" @staticmethod def path_to_ref(path): if (isinstance(path, fs.Path)): path = path.name return path.replace("+", ":").replace("%", "/") @staticmethod def ref_to_pathstr(ref_str): return ref_str.replace("/", "%").replace(":", "+") @classmethod def glob(class_, image_glob): """Return a possibly-empty iterator of references in the storage directory matching the given glob.""" for path in ch.storage.unpack_base.glob(class_.ref_to_pathstr(image_glob)): yield class_(class_.path_to_ref(path)) @classmethod def parse(class_, s, variables): if (class_.parser is None): class_.parser = lark.Lark(GRAMMAR_IMAGE_REF, parser="earley", propagate_positions=True, tree_class=Tree) s = s.translate(str.maketrans("%+", "/:", "&")) hint="https://hpc.github.io/charliecloud/faq.html#how-do-i-specify-an-image-reference" s = ch.variables_sub(s, variables) if "$" in s: ch.FATAL("image reference contains an undefined variable: %s" % s) try: tree = class_.parser.parse(s) except lark.exceptions.UnexpectedInput as x: if (x.column == -1): ch.FATAL("image ref syntax, at end: %s" % s, hint); else: ch.FATAL("image ref syntax, char %d: %s" % (x.column, s), hint) except lark.exceptions.UnexpectedEOF as x: # We get UnexpectedEOF because of Lark issue #237. This exception # doesn’t have a column location. ch.FATAL("image ref syntax, at end: %s" % s, hint) ch.DEBUG(tree.pretty()) return tree def __str__(self): out = "" if (self.host is not None): out += self.host if (self.port is not None): out += ":" + str(self.port) if (self.host is not None): out += "/" out += self.path_full if (self.tag is not None): out += ":" + self.tag if (self.digest is not None): out += "@sha256:" + self.digest return out @property def as_verbose_str(self): def fmt(x): if (x is None): return None else: return repr(x) return """\ as string: %s for filename: %s fields: host %s port %s path %s name %s tag %s digest %s\ """ % tuple( [str(self), self.for_path] + [fmt(i) for i in (self.host, self.port, self.path, self.name, self.tag, self.digest)]) @property def canonical(self): "Copy of self with all the defaults filled in." ref = self.copy() ref.defaults_add() return ref @property def for_path(self): return self.ref_to_pathstr(str(self)) @property def path_full(self): out = "" if (len(self.path) > 0): out += "/".join(self.path) + "/" out += self.name return out @property def version(self): if (self.tag is not None): return self.tag if (self.digest is not None): return "sha256:" + self.digest assert False, "version invalid with no tag or digest" def copy(self): "Return an independent copy of myself." return copy.deepcopy(self) def defaults_add(self): "Set defaults for all empty fields." if (self.host is None): if ("CH_REGY_DEFAULT_HOST" not in os.environ): self.host = "registry-1.docker.io" else: self.host = os.getenv("CH_REGY_DEFAULT_HOST") self.port = int(os.getenv("CH_REGY_DEFAULT_PORT", 443)) prefix = os.getenv("CH_REGY_PATH_PREFIX") if (prefix is not None): self.path = prefix.split("/") + self.path if (self.port is None): self.port = 443 if (self.host == "registry-1.docker.io" and len(self.path) == 0): # FIXME: For Docker Hub only, images with no path need a path of # “library” substituted. Need to understand/document the rules here. self.path = ["library"] if (self.tag is None and self.digest is None): self.tag = "latest" def from_tree(self, t): self.host = t.child_terminal("ir_hostport", "IR_HOST") self.port = t.child_terminal("ir_hostport", "IR_PORT") if (self.port is not None): self.port = int(self.port) self.path = [ ch.variables_sub(s, self.variables) for s in t.child_terminals("ir_path", "IR_PATH_COMPONENT")] self.name = t.child_terminal("ir_name", "IR_PATH_COMPONENT") self.tag = t.child_terminal("ir_tag", "IR_TAG") self.digest = t.child_terminal("ir_digest", "HEX_STRING") for a in ("host", "port", "name", "tag", "digest"): setattr(self, a, ch.variables_sub(getattr(self, a), self.variables)) # Resolve grammar ambiguity for hostnames w/o dot or port. if ( self.host is not None and "." not in self.host and self.port is None): self.path.insert(0, self.host) self.host = None class Tree(lark.tree.Tree): def _pretty(self, level, istr): # Re-implement with less space optimization and more debugging info. # See: https://github.com/lark-parser/lark/blob/262ab71/lark/tree.py#L78 pfx = "%4d %3d%s" % (self.meta.line, self.meta.column, istr*(level+1)) yield (pfx + self._pretty_label() + "\n") for c in self.children: if (isinstance(c, Tree)): yield from c._pretty(level + 1, istr) else: text = c type_ = c.type width = len(pfx) + len(istr) + len(text) + len(type_) + 2 over = width - ch.term_width if (len(type_) > WIDTH_TOKEN_MAX): # trim token (unconditionally for consistent alignment) token_rm = len(type_) - WIDTH_TOKEN_MAX type_ = type_[:-token_rm] over -= token_rm if (over > 0): # trim text (if needed) text = text[:-(over + 3)] + "..." yield "%s%s %s %s\n" % (pfx, istr, type_, text) def child(self, cname): """Locate a descendant subtree named cname using breadth-first search and return it. If no such subtree exists, return None.""" return next(self.children_(cname), None) def child_terminal(self, cname, tname, i=0): """Locate a descendant subtree named cname using breadth-first search and return its first child terminal named tname. If no such subtree exists, or it doesn’t have such a terminal, return None.""" st = self.child(cname) if (st is not None): return st.terminal(tname, i) else: return None def child_terminals(self, cname, tname): """Locate a descendant substree named cname using breadth-first search and yield the values of its child terminals named tname. If no such subtree exists, or it has no such terminals, yield empty sequence.""" for d in self.iter_subtrees_topdown(): if (d.data == cname): return d.terminals(tname) return [] def child_terminals_cat(self, cname, tname): """Return the concatenated values of all child terminals named tname as a string, with no delimiters. If none, return the empty string.""" return "".join(self.child_terminals(cname, tname)) def children_(self, cname): "Yield children of tree named cname using breadth-first search." for st in self.iter_subtrees_topdown(): if (st.data == cname): yield st def iter_subtrees_topdown(self, *args, **kwargs): return super().iter_subtrees_topdown(*args, **kwargs) def terminal(self, tname, i=0): """Return the value of the ith child terminal named tname (zero-based), or None if not found.""" for (j, t) in enumerate(self.terminals(tname)): if (j == i): return t return None def terminals(self, tname): """Yield values of all child terminals named tname, or empty list if none found.""" for j in self.children: if (isinstance(j, lark.lexer.Token) and j.type == tname): yield j.value def terminals_cat(self, tname): """Return the concatenated values of all child terminals named tname as a string, with no delimiters. If none, return the empty string.""" return "".join(self.terminals(tname)) charliecloud-0.37/lib/misc.py000066400000000000000000000153141457016721300161660ustar00rootroot00000000000000# Subcommands not exciting enough for their own module. import argparse import inspect import itertools import os import os.path import sys import charliecloud as ch import build_cache as bu import filesystem as fs import image as im import pull import version ## argparse “actions” ## class Action_Exit(argparse.Action): def __init__(self, *args, **kwargs): super().__init__(nargs=0, *args, **kwargs) class Dependencies(Action_Exit): def __call__(self, ap, cli, *args, **kwargs): # ch.init() not yet called, so must get verbosity from arguments. ch.dependencies_check() if (cli.verbose >= 1): print("lark path: %s" % os.path.normpath(inspect.getfile(im.lark))) sys.exit(0) class Version(Action_Exit): def __call__(self, *args, **kwargs): print(version.VERSION) sys.exit(0) ## Plain functions ## # Argument: command line arguments Namespace. Do not need to call sys.exit() # because caller manages that. def build_cache(cli): if (cli.bucache == ch.Build_Mode.DISABLED): ch.FATAL("build-cache subcommand invalid with build cache disabled") if (cli.reset): bu.cache.reset() if (cli.gc): bu.cache.garbageinate() if (cli.tree): bu.cache.tree_print() if (cli.dot): bu.cache.tree_dot() bu.cache.summary_print() def delete(cli): fail_ct = 0 for ref in cli.image_ref: delete_ct = 0 for img in itertools.chain(im.Image.glob(ref), im.Image.glob(ref + "_stage[0-9]*")): bu.cache.unpack_delete(img) to_delete = im.Reference.ref_to_pathstr(str(img)) bu.cache.branch_delete(to_delete) delete_ct += 1 if (delete_ct == 0): fail_ct += 1 ch.ERROR("no matching image, can’t delete: %s" % ref) bu.cache.worktrees_fix() if (fail_ct > 0): ch.FATAL("unable to delete %d invalid image(s)" % fail_ct) def gestalt_bucache(cli): bu.have_deps() def gestalt_bucache_dot(cli): bu.have_deps() bu.have_dot() def gestalt_logging(cli): ch.TRACE("trace") ch.DEBUG("debug") ch.VERBOSE("verbose") ch.INFO("info") ch.WARNING("warning") ch.ERROR("error") if (cli.fail): ch.FATAL("the program failed inexplicably") def gestalt_python_path(cli): print(sys.executable) def gestalt_storage_path(cli): print(ch.storage.root) def import_(cli): if (not os.path.exists(cli.path)): ch.FATAL("can’t copy: not found: %s" % cli.path) if (ch.xattrs_save): ch.WARNING("--xattrs unsupported by “ch-image import” (see FAQ)") pathstr = im.Reference.ref_to_pathstr(cli.image_ref) if (cli.bucache == ch.Build_Mode.ENABLED): # Un-tag previously deleted branch, if it exists. bu.cache.tag_delete(pathstr, fail_ok=True) dst = im.Image(im.Reference(cli.image_ref)) ch.INFO("importing: %s" % cli.path) ch.INFO("destination: %s" % dst) dst.unpack_clear() if (os.path.isdir(cli.path)): dst.copy_unpacked(cli.path) else: # tarball, hopefully dst.unpack([cli.path]) bu.cache.adopt(dst) if (dst.metadata["history"] == []): dst.metadata["history"].append({ "empty_layer": False, "command": "ch-image import"}) dst.metadata_save() ch.done_notify() def list_(cli): if (cli.undeletable): # list undeletable images imgdir = ch.storage.build_cache // "refs/tags" else: # list images imgdir = ch.storage.unpack_base if (cli.image_ref is None): # list all images if (not os.path.isdir(ch.storage.root)): ch.FATAL("does not exist: %s" % ch.storage.root) if (not ch.storage.valid_p): ch.FATAL("not a storage directory: %s" % ch.storage.root) images = sorted(imgdir.listdir()) if (len(images) >= 1): img_width = max(len(ref) for ref in images) for ref in images: img = im.Image(im.Reference(fs.Path(ref).parts[-1])) if cli.long: print("%-*s | %s" % (img_width, img, img.last_modified.ctime())) else: print(img) else: # list specified image img = im.Image(im.Reference(cli.image_ref)) print("details of image: %s" % img.ref) # present locally? if (not img.unpack_exist_p): stored = "no" else: img.metadata_load() stored = "yes (%s), modified: %s" % (img.metadata["arch"], img.last_modified.ctime()) print("in local storage: %s" % stored) # in cache? (sid, commit) = bu.cache.find_image(img) if (sid is None): cached = "no" else: cached = "yes (state ID %s, commit %s)" % (sid.short, commit[:7]) if (os.path.exists(img.unpack_path)): wdc = bu.cache.worktree_head(img) if (wdc is None): ch.WARNING("stored image not connected to build cache") elif (wdc != commit): ch.WARNING("stored image doesn’t match build cache: %s" % wdc) print("in build cache: %s" % cached) # present remotely? print("full remote ref: %s" % img.ref.canonical) pullet = pull.Image_Puller(img, img.ref) try: pullet.fatman_load() remote = "yes" arch_aware = "yes" arch_keys = sorted(pullet.architectures.keys()) try: fmt_space = len(max(arch_keys,key=len)) arch_avail = [] for key in arch_keys: arch_avail.append("%-*s %s" % (fmt_space, key, pullet.digests[key][:11])) except ValueError: # handles case where arch_keys is empty, e.g. # mcr.microsoft.com/windows:20H2. arch_avail = [None] except ch.Image_Unavailable_Error: remote = "no (or you are not authorized)" arch_aware = "n/a" arch_avail = ["n/a"] except ch.No_Fatman_Error: remote = "yes" arch_aware = "no" arch_avail = ["unknown"] pullet.done() print("available remotely: %s" % remote) print("remote arch-aware: %s" % arch_aware) print("host architecture: %s" % ch.arch_host) print("archs available: %s" % arch_avail[0]) for arch in arch_avail[1:]: print((" " * 21) + arch) def reset(cli): ch.storage.reset() def undelete(cli): if (cli.bucache != ch.Build_Mode.ENABLED): ch.FATAL("only available when cache is enabled") img = im.Image(im.Reference(cli.image_ref)) if (img.unpack_exist_p): ch.FATAL("image exists; will not overwrite") (_, git_hash) = bu.cache.find_deleted_image(img) if (git_hash is None): ch.FATAL("image not in cache") bu.cache.checkout_ready(img, git_hash) charliecloud-0.37/lib/pull.py000066400000000000000000000311351457016721300162060ustar00rootroot00000000000000import json import os import os.path import charliecloud as ch import build_cache as bu import image as im import registry as rg ## Constants ## # Internal library of manifests, e.g. for “FROM scratch” (issue #1013). manifests_internal = { "scratch": { # magic empty image "schemaVersion": 2, "config": { "digest": None }, "layers": [] } } ## Main ## def main(cli): # Set things up. src_ref = im.Reference(cli.source_ref) dst_ref = src_ref if cli.dest_ref is None else im.Reference(cli.dest_ref) if (cli.parse_only): print(src_ref.as_verbose_str) ch.exit(0) if (ch.xattrs_save): ch.WARNING("--xattrs unsupported for “ch-image pull” (see FAQ)") dst_img = im.Image(dst_ref) ch.INFO("pulling image: %s" % src_ref) if (src_ref != dst_ref): ch.INFO("destination: %s" % dst_ref) ch.INFO("requesting arch: %s" % ch.arch) bu.cache.pull_eager(dst_img, src_ref, cli.last_layer) ch.done_notify() ## Classes ## class Image_Puller: __slots__ = ("architectures", # key: architecture, value: manifest digest "config_hash", "digests", "image", "layer_hashes", "registry", "sid_input", "src_ref") def __init__(self, image, src_ref): self.architectures = None self.config_hash = None self.digests = dict() self.image = image self.layer_hashes = None self.registry = rg.HTTP(src_ref) self.sid_input = None self.src_ref = src_ref @property def config_path(self): if (self.config_hash is None): return None else: return ch.storage.download_cache // (self.config_hash + ".json") @property def fatman_path(self): return ch.storage.fatman_for_download(self.image.ref) @property def manifest_path(self): if (str(self.image.ref) in manifests_internal): return "[internal library]" else: if (ch.arch == "yolo" or self.architectures is None): digest = None else: digest = self.architectures[ch.arch] return ch.storage.manifest_for_download(self.image.ref, digest) def done(self): self.registry.close() def download(self): "Download image metadata and layers and put them in the download cache." # Spec: https://docs.docker.com/registry/spec/manifest-v2-2/ ch.VERBOSE("downloading image: %s" % self.image) have_skinny = False try: # fat manifest if (ch.arch != "yolo"): try: self.fatman_load() if (not self.architectures.in_warn(ch.arch)): ch.FATAL("requested arch unavailable: %s" % ch.arch, ("available: %s" % " ".join(sorted(self.architectures.keys())))) except ch.No_Fatman_Error: # currently, this error is only raised if we’ve downloaded the # skinny manifest. have_skinny = True if (ch.arch == "amd64"): # We’re guessing that enough arch-unaware images are amd64 to # barge ahead if requested architecture is amd64. ch.arch = "yolo" ch.WARNING("image is architecture-unaware") ch.WARNING("requested arch is amd64; using --arch=yolo") else: ch.FATAL("image is architecture-unaware", "consider --arch=yolo") # manifest self.manifest_load(have_skinny) except ch.Image_Unavailable_Error: if (ch.user() == "qwofford"): h = "Quincy, use --auth!!" else: h = "if your registry needs authentication, use --auth" ch.FATAL("unauthorized or not in registry: %s" % self.registry.ref, h) # config ch.VERBOSE("config path: %s" % self.config_path) if (self.config_path is not None): if (os.path.exists(self.config_path) and ch.dlcache_p): ch.INFO("config: using existing file") else: self.registry.blob_to_file(self.config_hash, self.config_path, "config: downloading") # layers for (i, lh) in enumerate(self.layer_hashes, start=1): path = self.layer_path(lh) ch.VERBOSE("layer path: %s" % path) msg = "layer %d/%d: %s" % (i, len(self.layer_hashes), lh[:7]) if (os.path.exists(path) and ch.dlcache_p): ch.INFO("%s: using existing file" % msg) else: self.registry.blob_to_file(lh, path, "%s: downloading" % msg) # done self.registry.close() def error_decode(self, data): """Decode first error message in registry error blob and return a tuple (code, message).""" try: code = data["errors"][0]["code"] msg = data["errors"][0]["message"] except (IndexError, KeyError): ch.FATAL("malformed error data", "yes, this is ironic") return (code, msg) def fatman_load(self): """Download the fat manifest and load it. If the image has a fat manifest populate self.architectures; this may be an empty dictionary if no valid architectures were found. Raises: * Image_Unavailable_Error if the image does not exist or we are not authorized to have it. * No_Fatman_Error if the image exists but has no fat manifest, i.e., is architecture-unaware. In this case self.architectures is set to None.""" self.architectures = None if (str(self.src_ref) in manifests_internal): # cheat; internal manifest library matches every architecture self.architectures = ch.Arch_Dict({ ch.arch_host: None }) # Assume that image has no digest. This is a kludge, but it makes my # solution to issue #1365 work so ¯\_(ツ)_/¯ self.digests[ch.arch_host] = "no digest" return # raises Image_Unavailable_Error if needed self.registry.fatman_to_file(self.fatman_path, "manifest list: downloading") fm = self.fatman_path.json_from_file("fat manifest") if ("layers" in fm or "fsLayers" in fm): # Check for skinny manifest. If not present, create a symlink to the # “fat manifest” with the conventional name for a skinny manifest. # This works because the file we just saved as the “fat manifest” is # actually a misleadingly named skinny manifest. Link is relative to # avoid embedding the storage directory path within the storage # directory (see PR #1657). if (not self.manifest_path.exists()): self.manifest_path.symlink_to(self.fatman_path.name) raise ch.No_Fatman_Error() if ("errors" in fm): # fm is an error blob. (code, msg) = self.error_decode(fm) if (code == "MANIFEST_UNKNOWN"): ch.INFO("manifest list: no such image") return else: ch.FATAL("manifest list: error: %s" % msg) self.architectures = ch.Arch_Dict() if ("manifests" not in fm): ch.FATAL("manifest list has no key 'manifests'") for m in fm["manifests"]: try: if (m["platform"]["os"] != "linux"): continue arch = m["platform"]["architecture"] if ("variant" in m["platform"]): arch = "%s/%s" % (arch, m["platform"]["variant"]) digest = m["digest"] except KeyError: ch.FATAL("manifest lists missing a required key") if (arch in self.architectures): ch.FATAL("manifest list: duplicate architecture: %s" % arch) self.architectures[arch] = ch.digest_trim(digest) self.digests[arch] = digest.split(":")[1] if (len(self.architectures) == 0): ch.WARNING("no valid architectures found") def layer_path(self, layer_hash): "Return the path to tarball for layer layer_hash." return ch.storage.download_cache // (layer_hash + ".tar.gz") def manifest_digest_by_arch(self): "Return skinny manifest digest for target architecture." fatman = self.fat_manifest_path.json_from_file() arch = None digest = None variant = None try: arch, variant = ch.arch.split("/", maxsplit=1) except ValueError: arch = ch.arch if ("manifests" not in fatman): ch.FATAL("manifest list has no manifests") for k in fatman["manifests"]: if (k.get('platform').get('os') != 'linux'): continue elif ( k.get('platform').get('architecture') == arch and ( variant is None or k.get('platform').get('variant') == variant)): digest = k.get('digest') if (digest is None): ch.FATAL("arch not found for image: %s" % arch, 'try "ch-image list IMAGE_REF"') return digest def manifest_load(self, have_skinny=False): """Download the manifest file, parse it, and set self.config_hash and self.layer_hashes. If the image does not exist, exit with error.""" def bad_key(key): ch.FATAL("manifest: %s: no key: %s" % (self.manifest_path, key)) self.config_hash = None self.layer_hashes = None # obtain the manifest try: # internal manifest library, e.g. for “FROM scratch” manifest = manifests_internal[str(self.src_ref)] ch.INFO("manifest: using internal library") except KeyError: # download the file and parse it if (ch.arch == "yolo" or self.architectures is None): digest = None else: digest = self.architectures[ch.arch] ch.DEBUG("manifest digest: %s" % digest) if (not have_skinny): self.registry.manifest_to_file(self.manifest_path, "manifest: downloading", digest=digest) manifest = self.manifest_path.json_from_file("manifest") # validate schema version try: version = manifest['schemaVersion'] except KeyError: bad_key("schemaVersion") if (version not in {1,2}): ch.FATAL("unsupported manifest schema version: %s" % repr(version)) # load config hash # # FIXME: Manifest version 1 does not list a config blob. It does have # things (plural) that look like a config at history/v1Compatibility as # an embedded JSON string :P but I haven’t dug into it. if (version == 1): ch.VERBOSE("no config; manifest schema version 1") self.config_hash = None else: # version == 2 try: self.config_hash = manifest["config"]["digest"] if (self.config_hash is not None): self.config_hash = ch.digest_trim(self.config_hash) except KeyError: bad_key("config/digest") # load layer hashes if (version == 1): key1 = "fsLayers" key2 = "blobSum" else: # version == 2 key1 = "layers" key2 = "digest" if (key1 not in manifest): bad_key(key1) self.layer_hashes = list() for i in manifest[key1]: if (key2 not in i): bad_key("%s/%s" % (key1, key2)) self.layer_hashes.append(ch.digest_trim(i[key2])) if (version == 1): self.layer_hashes.reverse() # Remember State_ID input. We can’t rely on the manifest existing in # serialized form (e.g. for internal manifests), so re-serialize. self.sid_input = json.dumps(manifest, sort_keys=True) def unpack(self, last_layer=None): layer_paths = [self.layer_path(h) for h in self.layer_hashes] bu.cache.unpack_delete(self.image, missing_ok=True) self.image.unpack(layer_paths, last_layer) self.image.metadata_replace(self.config_path) # Check architecture we got. This is limited because image metadata does # not store the variant. Move fast and break things, I guess. arch_image = self.image.metadata["arch"] or "unknown" arch_short = ch.arch.split("/")[0] arch_host_short = ch.arch_host.split("/")[0] if (arch_image != "unknown" and arch_image != arch_host_short): host_mismatch = " (may not match host %s)" % ch.arch_host else: host_mismatch = "" ch.INFO("image arch: %s%s" % (arch_image, host_mismatch)) if (ch.arch != "yolo" and arch_short != arch_image): ch.WARNING("image architecture does not match requested: %s ≠ %s" % (ch.arch, arch_image)) charliecloud-0.37/lib/push.py000066400000000000000000000163731457016721300162200ustar00rootroot00000000000000import json import os.path import charliecloud as ch import image as im import registry as rg import version ## Main ## def main(cli): src_ref = im.Reference(cli.source_ref) ch.INFO("pushing image: %s" % src_ref) image = im.Image(src_ref, cli.image) # FIXME: validate it’s an image using Megan’s new function (PR #908) if (not os.path.isdir(image.unpack_path)): if (cli.image is not None): ch.FATAL("can’t push: %s does not appear to be an image" % cli.image) else: ch.FATAL("can’t push: no image %s" % src_ref) if (cli.image is not None): ch.INFO("image path: %s" % image.unpack_path) else: ch.VERBOSE("image path: %s" % image.unpack_path) if (cli.dest_ref is not None): dst_ref = im.Reference(cli.dest_ref) ch.INFO("destination: %s" % dst_ref) else: dst_ref = im.Reference(cli.source_ref) up = Image_Pusher(image, dst_ref) up.push() ch.done_notify() ## Classes ## class Image_Pusher: # Note; We use functions to create the blank config and manifest to to # avoid copy/deepcopy complexity from just copying a default dict. __slots__ = ("config", # sequence of bytes "dst_ref", # destination of upload "image", # Image object we are uploading "layers", # list of (digest, .tar.gz path), lowest first "manifest", # sequence of bytes "registry") # destination registry def __init__(self, image, dst_ref): self.config = None self.dst_ref = dst_ref self.image = image self.layers = None self.manifest = None self.registry = None @classmethod def config_new(class_): "Return an empty config, ready to be filled in." # FIXME: URL of relevant docs? # FIXME: tidy blank/empty fields? return { "architecture": ch.arch_host_get(), "charliecloud_version": version.VERSION, "comment": "pushed with Charliecloud", "config": {}, "container_config": {}, "created": ch.now_utc_iso8601(), "history": [], "os": "linux", "rootfs": { "diff_ids": [], "type": "layers" }, "weirdal": "yankovic" } @classmethod def manifest_new(class_): "Return an empty manifest, ready to be filled in." return { "schemaVersion": 2, "mediaType": rg.TYPES_MANIFEST["docker2"], "config": { "mediaType": rg.TYPE_CONFIG, "size": None, "digest": None }, "layers": [], "weirdal": "yankovic" } def cleanup(self): ch.INFO("cleaning up") # Delete the tarballs since we can’t yet cache them. for (_, tar_c) in self.layers: ch.VERBOSE("deleting tarball: %s" % tar_c) tar_c.unlink() def prepare(self): """Prepare self.image for pushing to self.dst_ref. Return tuple: (list of gzipped layer tarball paths, config as a sequence of bytes, manifest as a sequence of bytes). There is not currently any support for re-using any previously prepared files already in the upload cache, because we don’t yet have a way to know if these have changed until they are already build.""" # Initializing an HTTP instance for the registry and doing a 'GET' # request right out the gate ensures the user needs to authenticate # before we prepare the image for upload (#1426). self.registry = rg.HTTP(self.dst_ref) self.registry.request("GET", self.registry._url_base) tars_uc = self.image.tarballs_write(ch.storage.upload_cache) tars_c = list() config = self.config_new() manifest = self.manifest_new() # Prepare layers. for (i, tar_uc) in enumerate(tars_uc, start=1): ch.INFO("layer %d/%d: preparing" % (i, len(tars_uc))) path_uc = ch.storage.upload_cache // tar_uc hash_uc = path_uc.file_hash() config["rootfs"]["diff_ids"].append("sha256:" + hash_uc) size_uc = path_uc.file_size() path_c = path_uc.file_gzip(["-9", "--no-name"]) tar_c = path_c.name hash_c = path_c.file_hash() size_c = path_c.file_size() tars_c.append((hash_c, path_c)) manifest["layers"].append({ "mediaType": rg.TYPE_LAYER, "size": size_c, "digest": "sha256:" + hash_c }) # Prepare metadata. ch.INFO("preparing metadata") self.image.metadata_load() # Environment. Note that this is *not* a dictionary for some reason but # a list of name/value pairs separated by equals [1], with no quoting. # # [1]: https://github.com/opencontainers/image-spec/blob/main/config.md config['config']['Env'] = ["%s=%s" % (k, v) for k, v in self.image.metadata.get("env", {}).items()] # History. Some registries, e.g., Quay, use history metadata for simple # sanity checks. For example, when an image’s number of "empty_layer" # history entries doesn’t match the number of layers being uploaded, # Quay will reject the image upload. # # This type of error checking is odd as the empty_layer key is optional # (https://github.com/opencontainers/image-spec/blob/main/config.md). # # Thus, to push images built (or pulled) with Charliecloud we ensure the # the total number of non-empty layers always totals one (1). To do this # we iterate over the history entires backward searching for the first # non-empty entry and preserve it; all others are set to empty. hist = self.image.metadata["history"] non_empty_winner = None for i in range(len(hist) - 1, -1, -1): if ( "empty_layer" not in hist[i].keys() or ( "empty_layer" in hist[i].keys() and not hist[i]["empty_layer"])): non_empty_winner = i break assert(non_empty_winner is not None) for i in range(len(hist) - 1): if (i != non_empty_winner): hist[i]["empty_layer"] = True config["history"] = hist # Pack it up to go. config_bytes = json.dumps(config, indent=2).encode("UTF-8") config_hash = ch.bytes_hash(config_bytes) manifest["config"]["size"] = len(config_bytes) manifest["config"]["digest"] = "sha256:" + config_hash ch.DEBUG("config: %s\n%s" % (config_hash, config_bytes.decode("UTF-8"))) manifest_bytes = json.dumps(manifest, indent=2).encode("UTF-8") ch.DEBUG("manifest:\n%s" % manifest_bytes.decode("UTF-8")) # Store for the next steps. self.layers = tars_c self.config = config_bytes self.manifest = manifest_bytes def push(self): self.prepare() self.upload() self.cleanup() def upload(self): ch.INFO("starting upload") for (i, (digest, tarball)) in enumerate(self.layers, start=1): self.registry.layer_from_file(digest, tarball, "layer %d/%d: " % (i, len(self.layers))) self.registry.config_upload(self.config) self.registry.manifest_upload(self.manifest) self.registry.close() charliecloud-0.37/lib/registry.py000066400000000000000000000536051457016721300171100ustar00rootroot00000000000000import getpass import hashlib import io import os import re import types import urllib import charliecloud as ch ## Hairy imports ## # Requests is not bundled, so this noise makes the file parse and # --version/--help work even if it’s not installed. try: import requests import requests.auth import requests.exceptions except ImportError: ch.depfails.append(("missing", 'Python module "requests"')) # Mock up a requests.auth module so the rest of the file parses. requests = types.ModuleType("requests") requests.auth = types.ModuleType("requests.auth") requests.auth.AuthBase = object ## Constants ## # Content types for some stuff we care about. # See: https://github.com/opencontainers/image-spec/blob/main/media-types.md TYPES_MANIFEST = \ {"docker2": "application/vnd.docker.distribution.manifest.v2+json", "oci1": "application/vnd.oci.image.manifest.v1+json"} TYPES_INDEX = \ {"docker2": "application/vnd.docker.distribution.manifest.list.v2+json", "oci1": "application/vnd.oci.image.index.v1+json"} TYPE_CONFIG = "application/vnd.docker.container.image.v1+json" TYPE_LAYER = "application/vnd.docker.image.rootfs.diff.tar.gzip" ## Globals ## # Verify TLS certificates? Passed to requests. tls_verify = True # True if we talk to registries authenticated; false if anonymously. auth_p = False ## Classes ## class Auth(requests.auth.AuthBase): # Every registry request has an “authorization object”. This starts as no # authentication at all. If we get HTTP 401 Unauthorized, we try to # “escalate” to a higher level of authorization; some classes have multiple # escalators that we try in order. Escalation can fail either if # authentication fails or there is nothing to escalate to. # # Class attributes: # # anon_p ...... True if the authorization object is anonymous; False if # it needed authentication. # # escalators .. Sequence of classes we can escalate to. Empty if no # escalation possible. This is actually a property rather # than a class attribute because it needs to refer to # classes that may not have been defined when the module is # created, e.g. classes later in the file, or some can # escalate to themselves. # # auth_p ...... True if appropriate for authenticated mode, False if # anonymous (i.e., --auth or not, respectively). Everything # must be one or the other. # # scheme ...... Auth scheme string (from WWW-Authenticate header) that # this class matches. __slots__ = ("auth_h_next",) # WWW-Authenticate header for next escalator @classmethod def authenticate(class_, creds, auth_d): """Authenticate using the given credentials and parsed WWW-Authenticate dictionary. Return a new Auth object if successful, None if not. The caller is responsible for dealing with the failure.""" ... def __eq__(self, other): return (type(self) == type(other)) @property def escalators(self): ... def escalate(self, reg, res): """Escalate to a higher level of authorization. Use the WWW-Authenticate header in failed response res if there is one.""" ch.VERBOSE("escalating from %s" % self) assert (res.status_code == 401) # Get authentication instructions. if ("WWW-Authenticate" in res.headers): auth_h = res.headers["WWW-Authenticate"] elif (self.auth_h_next is not None): auth_h = self.auth_h_next else: ch.FATAL("don’t know how to authenticate: WWW-Authenticate not found") # We use two “undocumented (although very stable and frequently cited)” # methods to parse the authentication response header (thanks Andy, # i.e., @adrecord on GitHub). (auth_scheme, auth_d) = auth_h.split(maxsplit=1) auth_d = urllib.request.parse_keqv_list( urllib.request.parse_http_list(auth_d)) ch.VERBOSE("WWW-Authenticate parsed: %s %s" % (auth_scheme, auth_d)) # Is escalation possible in principle? if (len(self.escalators) == 0): ch.FATAL("no further authentication possible, giving up") # Try to escalate. for class_ in self.escalators: if (class_.scheme == auth_scheme): if (class_.auth_p != auth_p): ch.VERBOSE("skipping %s: auth mode mismatch" % class_.__name__) else: ch.VERBOSE("authenticating using %s" % class_.__name__) auth = class_.authenticate(reg, auth_d) if (auth is None): ch.VERBOSE("authentication failed; trying next") elif (auth == self): ch.VERBOSE("authentication did not escalate; trying next") else: return auth # success! ch.VERBOSE("no authentication left to try") return None class Auth_Basic(Auth): anon_p = False scheme = "Basic" auth_p = True __slots__ = ("basic") @classmethod def authenticate(class_, reg, auth_d): # Note: Basic does not validate the credentials until we try to use it. if ("realm" not in auth_d): ch.FATAL("WWW-Authenticate missing realm") (username, password) = reg.creds.get() i = class_() i.basic = requests.auth.HTTPBasicAuth(username, password) return i def __call__(self, *args, **kwargs): return self.basic(*args, **kwargs) def __eq__(self, other): return super().__eq__(other) and (self.basic == other.basic) def __str__(self): return self.basic.__str__() @property def escalators(self): return () class Auth_Bearer_IDed(Auth): # https://stackoverflow.com/a/58055668 anon_p = False scheme = "Bearer" auth_p = True variant = "IDed" __slots__ = ("auth_d", "token") def __init__(self, token, auth_d): self.token = token self.auth_d = auth_d @classmethod def authenticate(class_, reg, auth_d): # Registries and endpoints vary in what they put in WWW-Authenticate. We # need realm because it’s the URL to use for a token. Otherwise, just # give back all the keys we got. for k in ("realm",): if (k not in auth_d): ch.FATAL("WWW-Authenticate missing key: %s" % k) params = { (k,v) for (k,v) in auth_d.items() if k != "realm" } # Request a Bearer token. res = reg.request_raw("GET", auth_d["realm"], {200,401,403}, auth=class_.token_auth(reg.creds), params=params) if (res.status_code != 200): ch.VERBOSE("bearer token request rejected") return None # Create new instance. i = class_(res.json()["token"], auth_d) ch.VERBOSE("received bearer token: %s" % (i.token_short)) return i @classmethod def token_auth(class_, creds): """Return a requests.auth.AuthBase object used to authenticate the token request.""" (username, password) = creds.get() return requests.auth.HTTPBasicAuth(username, password) def __call__(self, req): req.headers["Authorization"] = "Bearer %s" % self.token return req def __eq__(self, other): return super().__eq__(other) and (self.auth_d == other.auth_d) def __str__(self): return ("Bearer (%s) %s" % (self.__class__.__name__.split("_")[-1], self.token_short)) @property def escalators(self): # One can escalate to an authenticated Bearer with a greater scope. I’m # pretty sure this doesn’t create an infinite loop because eventually # the token request will fail. return (Auth_Bearer_IDed,) @property def token_short(self): return ("%s..%s" % (self.token[:8], self.token[-8:])) class Auth_Bearer_Anon(Auth_Bearer_IDed): anon_p = True scheme = "Bearer" auth_p = False __slots__ = () @classmethod def token_auth(class_, creds): # The way to get an anonymous Bearer token is to give no Basic auth # header in the token request. return None @property def escalators(self): return (Auth_Bearer_IDed,) class Auth_None(Auth): anon_p = True scheme = None #auth_p = # not meaningful b/c we start here in both modes def __call__(self, req): return req def __str__(self): return "no authorization" @property def escalators(self): return (Auth_Basic, Auth_Bearer_Anon, Auth_Bearer_IDed) class Credentials: __slots__ = ("password", "username") def __init__(self): self.username = None self.password = None def get(self): # If stored, return those. if (self.username is not None): username = self.username password = self.password else: try: # Otherwise, use environment variables. username = os.environ["CH_IMAGE_USERNAME"] password = os.environ["CH_IMAGE_PASSWORD"] except KeyError: # Finally, prompt the user. # FIXME: This hangs in Bats despite sys.stdin.isatty() == True. try: username = input("\nUsername: ") except KeyboardInterrupt: ch.FATAL("authentication cancelled") password = getpass.getpass("Password: ") if (not ch.password_many): # Remember the credentials. self.username = username self.password = password return (username, password) class HTTP: """Transfers image data to and from a remote image repository via HTTPS. Note that ref refers to the *remote* image. Objects of this class have no information about the local image.""" __slots__ = ("auth", "creds", "ref", "session") def __init__(self, ref): # Need an image ref with all the defaults filled in. self.ref = ref.canonical self.auth = Auth_None() self.creds = Credentials() self.session = None # This is commented out because it prints full request and response # bodies to standard output (not stderr), which overwhelms the terminal. # Normally, a better debugging approach if you need this is to sniff the # connection using e.g. mitmproxy. #if (verbose >= 2): # http.client.HTTPConnection.debuglevel = 1 @staticmethod def headers_log(hs): """Log the headers.""" # All headers first. for h in hs: h = h.lower() if (h == "www-authenticate"): f = ch.VERBOSE else: f = ch.DEBUG f("%s: %s" % (h, hs[h])) # Friendly message for Docker Hub rate limit. pull_ct = period = left_ct = reason = "???" # keep as strings if ("ratelimit-limit" in hs): h = hs["ratelimit-limit"] m = re.search(r"^(\d+);w=(\d+)$", h) if (m is None): ch.WARNING("can’t parse RateLimit-Limit: %s" % h) else: pull_ct = m[1] period = str(int(m[2]) / 3600) # seconds to hours if ("ratelimit-remaining" in hs): h = hs["ratelimit-remaining"] m = re.search(r"^(\d+);", h) if (m is None): ch.WARNING("can’t parse RateLimit-Remaining: %s" % h) else: left_ct = m[1] if ("docker-ratelimit-source" in hs): h = hs["docker-ratelimit-source"] m = re.search(r"^[0-9.a-f:]+$", h) # IPv4 or IPv6 if (m is not None): reason = m[0] else: m = re.search(r"^[0-9A-Fa-f-]+$", h) # user UUID if (m is not None): reason = "auth" else: # Overall limits yield HTTP 429 so warning seems legitimate? ch.WARNING("can’t parse Docker-RateLimit-Source: %s" % h) if (any(i != "???" for i in (pull_ct, period, left_ct))): ch.INFO("Docker Hub rate limit: %s pulls left of %s per %s hours (%s)" % (left_ct, pull_ct, period, reason)) @property def _url_base(self): return "https://%s:%d/v2/" % (self.ref.host, self.ref.port) def _url_of(self, type_, address): "Return an appropriate repository URL." return self._url_base + "/".join((self.ref.path_full, type_, address)) def blob_exists_p(self, digest): """Return true if a blob with digest (hex string) exists in the remote repository, false otherwise.""" # Gotchas: # # 1. HTTP 401 means both unauthorized *or* not found, I assume to avoid # information leakage about the presence of stuff one isn’t allowed # to see. By the time it gets here, we should be authenticated, so # interpret it as not found. # # 2. Sometimes we get 301 Moved Permanently. It doesn’t bubble up to # here because requests.request() follows redirects. However, # requests.head() does not follow redirects, and it seems like a # weird status, so I worry there is a gotcha I haven’t figured out. url = self._url_of("blobs", "sha256:%s" % digest) res = self.request("HEAD", url, {200,401,404}) return (res.status_code == 200) def blob_to_file(self, digest, path, msg): "GET the blob with hash digest and save it at path." # /v2/library/hello-world/blobs/ url = self._url_of("blobs", "sha256:" + digest) sw = ch.Progress_Writer(path, msg) self.request("GET", url, out=sw, hd=digest) sw.close() def blob_upload(self, digest, data, note=""): """Upload blob with hash digest to url. data is the data to upload, and can be anything requests can handle; if it’s an open file, then it’s wrapped in a Progress_Reader object. note is a string to prepend to the log messages; default empty string.""" ch.INFO("%s%s: checking if already in repository" % (note, digest[:7])) # 1. Check if blob already exists. If so, stop. if (self.blob_exists_p(digest)): ch.INFO("%s%s: already present" % (note, digest[:7])) return msg = "%s%s: not present, uploading" % (note, digest[:7]) if (isinstance(data, io.IOBase)): data = ch.Progress_Reader(data, msg) data.start() else: ch.INFO(msg) # 2. Get upload URL for blob. url = self._url_of("blobs", "uploads/") res = self.request("POST", url, {202}) # 3. Upload blob. We do a “monolithic” upload (i.e., send all the # content in a single PUT request) as opposed to a “chunked” upload # (i.e., send data in multiple PATCH requests followed by a PUT request # with no body). url = res.headers["Location"] res = self.request("PUT", url, {201}, data=data, params={ "digest": "sha256:%s" % digest }) if (isinstance(data, ch.Progress_Reader)): data.close() # 4. Verify blob now exists. if (not self.blob_exists_p(digest)): ch.FATAL("blob just uploaded does not exist: %s" % digest[:7]) def close(self): if (self.session is not None): self.session.close() def config_upload(self, config): "Upload config (sequence of bytes)." self.blob_upload(ch.bytes_hash(config), config, "config: ") def escalate(self, res): "Try to escalate authorization; return True if successful, else False." auth = self.auth.escalate(self, res) if (auth is None): return False else: self.auth = auth return True def fatman_to_file(self, path, msg): """GET the manifest for self.image and save it at path. This seems to have four possible results: 1. HTTP 200, and body is a fat manifest: image exists and is architecture-aware. 2. HTTP 200, but body is a skinny manifest: image exists but is not architecture-aware. 3. HTTP 401/404: image does not exist or is unauthorized. 4. HTTP 429: rate limite exceeded. This method raises Image_Unavailable_Error in case 3. The caller is responsible for distinguishing cases 1 and 2.""" url = self._url_of("manifests", self.ref.version) pw = ch.Progress_Writer(path, msg) # Including TYPES_MANIFEST avoids the server trying to convert its v2 # manifest to a v1 manifest, which currently fails for images # Charliecloud pushes. The error in the test registry is “empty history # when trying to create schema1 manifest”. accept = "%s;q=0.5" % ",".join( list(TYPES_INDEX.values()) + list(TYPES_MANIFEST.values())) res = self.request("GET", url, out=pw, statuses={200, 401, 404, 429}, headers={ "Accept" : accept }) pw.close() if (res.status_code == 429): if (self.auth.anon_p): hint = "consider --auth" else: hint = None ch.FATAL("registry rate limit exceeded (HTTP 429)", hint) elif (res.status_code != 200): ch.DEBUG(res.content) raise ch.Image_Unavailable_Error() def layer_from_file(self, digest, path, note=""): "Upload gzipped tarball layer at path, which must have hash digest." # NOTE: We don’t verify the digest b/c that means reading the whole file. ch.VERBOSE("layer tarball: %s" % path) fp = path.open("rb") # open file avoids reading it all into memory self.blob_upload(digest, fp, note) ch.close_(fp) def manifest_to_file(self, path, msg, digest=None): """GET manifest for the image and save it at path. If digest is given, use that to fetch the appropriate architecture; otherwise, fetch the default manifest using the exising image reference.""" if (digest is None): digest = self.ref.version else: digest = "sha256:" + digest url = self._url_of("manifests", digest) pw = ch.Progress_Writer(path, msg) accept = "%s;q=0.5" % ",".join(TYPES_MANIFEST.values()) res = self.request("GET", url, out=pw, statuses={200, 401, 404}, headers={ "Accept" : accept }) pw.close() if (res.status_code != 200): ch.DEBUG(res.content) raise ch.Image_Unavailable_Error() def manifest_upload(self, manifest): "Upload manifest (sequence of bytes)." # Note: The manifest is *not* uploaded as a blob. We just do one PUT. ch.INFO("manifest: uploading") url = self._url_of("manifests", self.ref.tag) self.request("PUT", url, {201}, data=manifest, headers={ "Content-Type": TYPES_MANIFEST["docker2"] }) def request(self, method, url, statuses={200}, out=None, hd=None, **kwargs): """Request url using method and return the response object. If statuses is given, it is set of acceptable response status codes, defaulting to {200}; any other response is a fatal error. If out is given, response content will be streamed to this Progress_Writer object and must be non-zero length. If hd is given, validate integrity of downloaded data using expected hash digest. Use current session if there is one, or start a new one if not. If authentication fails (or isn’t initialized), then authenticate harder and re-try the request.""" # Set up. assert (out or hd is None), "digest only checked if streaming" self.session_init_maybe() ch.VERBOSE("auth: %s" % self.auth) if (out is not None): kwargs["stream"] = True # Make the request. while True: res = self.request_raw(method, url, statuses | {401}, **kwargs) if (res.status_code != 401): break else: ch.VERBOSE("HTTP 401 unauthorized") if (self.escalate(res)): # success ch.VERBOSE("retrying with auth: %s" % self.auth) elif (401 in statuses): # caller can deal with it break else: ch.FATAL("unhandled authentication failure") # Stream response if needed. m = hashlib.sha256() if (out is not None and res.status_code == 200): try: length = int(res.headers["Content-Length"]) except KeyError: length = None except ValueError: ch.FATAL("invalid Content-Length in response") out.start(length) for chunk in res.iter_content(ch.HTTP_CHUNK_SIZE): out.write(chunk) m.update(chunk) # store downloaded hash digest # Validate integrity of downloaded data if (hd is not None and hd != m.hexdigest()): ch.FATAL("registry streamed response content is invalid") # Done. return res def request_raw(self, method, url, statuses, auth=None, **kwargs): """Request url using method. statuses is an iterable of acceptable response status codes; any other response is a fatal error. Return the requests.Response object. Session must already exist. If auth arg given, use it; otherwise, use object’s stored authentication if initialized; otherwise, use no authentication.""" ch.VERBOSE("%s: %s" % (method, url)) if (auth is None): auth = self.auth try: res = self.session.request(method, url, auth=auth, **kwargs) ch.VERBOSE("response status: %d" % res.status_code) self.headers_log(res.headers) if (res.status_code not in statuses): ch.FATAL("%s failed; expected status %s but got %d: %s" % (method, statuses, res.status_code, res.reason)) except requests.exceptions.RequestException as x: ch.FATAL("%s failed: %s" % (method, x)) return res def session_init_maybe(self): "Initialize session if it’s not initialized; otherwise do nothing." if (self.session is None): ch.VERBOSE("initializing session") self.session = requests.Session() self.session.verify = tls_verify charliecloud-0.37/misc/000077500000000000000000000000001457016721300150425ustar00rootroot00000000000000charliecloud-0.37/misc/Makefile.am000066400000000000000000000000521457016721300170730ustar00rootroot00000000000000EXTRA_DIST = docker-clean.sh grep version charliecloud-0.37/misc/branches-tidy000077500000000000000000000051641457016721300175320ustar00rootroot00000000000000#!/usr/bin/env python3 # FIXME: pretty colors? import argparse import collections import re import subprocess class Branch: __slots__ = ("local", "repo", "remote", "status") def __str__(self): s = self.local if (self.remote is not None): s += " → " if (self.repo == repo_max and self.remote == self.local): s += "•" else: s += "%s/%s" % (self.repo, self.remote) if (self.status is not None): s += " [%s]" % self.status return s def delete(name): subprocess.run(["git", "branch", "-qD", name], check=True) # globals remote_dangling = set() remote_matched = set() other = set() repos = collections.Counter() repo_max = None delete_ct = 0 p = argparse.ArgumentParser( description = "List summary of Git branches.", epilog = "Dot (•) indicates branch is at most common remote with same name.") p.add_argument("-d", "--delete", action="store_true", help="delete dangling branches (remote branch missing)") p.add_argument("-r", "--delete-remote", metavar="REMOTE", action="append", default=list(), help="delete branches pointing to REMOTE (can be repeated)") args = p.parse_args() cp = subprocess.run(["git", "branch", "--format", "%(refname:short) %(upstream:short) %(upstream:track)"], stdout=subprocess.PIPE, encoding="UTF-8", check=True) for m in re.finditer(r"^(\S+)\s((\S+)/(\S+))?\s(\[(.+)\])?$", cp.stdout, re.MULTILINE): b = Branch() b.local = m[1] b.repo = m[3] b.remote = m[4] b.status = m[6] if (b.remote is None): other.add(b) else: repos[b.repo] += 1 if (b.status == "gone"): remote_dangling.add(b) else: remote_matched.add(b) assert( len(cp.stdout.splitlines()) == len(other) + len(remote_matched) + len(remote_dangling)) (repo_max, repo_max_ct) = repos.most_common(1)[0] print("found %d repos; most common: %s (%d)" % (len(repos), repo_max, repo_max_ct)) print("remote dangling (%d):" % len(remote_dangling)) for b in remote_dangling: print(" %s" % b, end="") if (args.delete): delete(b.local) delete_ct += 1 print(" ☠️", end="") print() print("remote (%d):" % len(remote_matched)) for b in remote_matched: print(" %s" % b, end="") if (b.repo in args.delete_remote): delete(b.local) delete_ct += 1 print(" ☠️", end="") print() print("other (%d):" % len(other)) for b in other: print(" %s" % b) if (delete_ct > 0): print("deleted %d branches" % delete_ct) charliecloud-0.37/misc/docker-clean.sh000077500000000000000000000020771457016721300177360ustar00rootroot00000000000000#!/bin/bash # FIXME: Give up after a certain number of iterations. set -e # Remove all containers. while true; do cmd='sudo docker ps -aq' cs_ct=$($cmd | wc -l) echo "found $cs_ct containers" [[ 0 -eq $cs_ct ]] && break # shellcheck disable=SC2046 sudo docker rm $($cmd) done # Untag all images. This fails with: # # Error response from daemon: invalid reference format # # sometimes. I don’t know why. if [[ $1 != --all ]]; then while true; do cmd='sudo docker images --filter dangling=false --format {{.Repository}}:{{.Tag}}' tag_ct=$($cmd | wc -l) echo "found $tag_ct tagged images" [[ 0 -eq $tag_ct ]] && break # shellcheck disable=SC2046 sudo docker rmi -f --no-prune $($cmd) done fi # If --all specified, remove all images. if [[ $1 = --all ]]; then while true; do cmd='sudo docker images -q' img_ct=$($cmd | wc -l) echo "found $img_ct images" [[ 0 -eq $img_ct ]] && break # shellcheck disable=SC2046 sudo docker rmi -f $($cmd) done fi charliecloud-0.37/misc/grep000077500000000000000000000026571457016721300157370ustar00rootroot00000000000000#!/bin/bash # The purpose of this script is to make it more convenient to search the # project source code. It’s a wrapper for grep(1) to exclude directories that # yield a ton of false positives, in particular the documentation’s JavaScript # index that is one line thousands of characters long. It's tedious to derive # the exclusions manually each time. # # grep(1) has an --exclude-dir option, but I can’t get it to work with rooted # directories, e.g., doc-src/_build vs _build anywhere in the tree, so we use # find(1) instead. You may hear complaints about how find is too slow for # this, but the project is small enough we don’t care. set -e cd "$(dirname "$0")"/.. find . \( -path ./.git \ -o -path ./autom4te.cache \ -o -path ./doc/doctrees \ -o -path ./bin/ch-checkns \ -o -path ./bin/ch-image \ -o -path ./bin/ch-run \ -o -path ./bin/ch-run-oci \ -o -path ./doc/html \ -o -path ./doc/man \ -o -path ./test/approved-trailing-whitespace \ -o -path './lib/lark*' \ -o -path '*.patch' \ -o -name '*.pyc' \ -o -name '*.vtk' \ -o -name ._.DS_Store \ -o -name .DS_Store \ -o -name configure \ -o -name 'config.*' \ -o -name Makefile \ -o -name Makefile.in \ \) -prune \ -o -type f \ -exec grep --color=auto -HIn "$@" {} \; charliecloud-0.37/misc/loc000077500000000000000000000243321457016721300155510ustar00rootroot00000000000000#!/bin/bash # Count lines of code in the project in an intelligent way. # # Notes/gotchas: # # 1. Use temporary files instead of variables to allow easier examination # after the script is run, and also because newline handling in shell # variables is rather hairy & error prone (e.g., command substitution # strips trailing newlines but echo adds a trailing newline, hiding it). # # 2. To add/update language definitions, cloc only has “merge, but mine are # lower priority” (--read-lang-def) and “use mine only, with no default # definitions” (--force-lang-def). We need “merge my definitions with # default, but with mine higher priority”, so we emulate it with # --force-lang-def, providing all the language definitions we need, some # of which are altered. # # 3. cloc also does not have a way to define file prefixes (say, # Dockerfile.*). So we list all the Dockerfiles we have manually in the # language definitions. # # 4. To debug “cloc ignoring wrong number of files”: just look in # /tmp/loc.ignored.out. However, you need cloc 1.84 or higher because of # cloc issue #401 [1]; otherwise the file is empty. # # [1]: https://github.com/AlDanial/cloc/issues/401 set -e -o pipefail export LC_ALL=C cd "$(dirname "$0")"/.. countem () { msg=$1 infile=$2 outfile=${2}.out catfile=${2}.cat ignore_ct=${3:-0} nolist=$4 echo echo "*** $msg ***" #cat "$infile" cloc --force-lang-def=/dev/stdin \ --categorized="$catfile" \ --list-file="$infile" \ --ignored=/tmp/loc.ignored.out \ <<'EOF' > "$outfile" Bash filter remove_matches ^\s*# filter remove_inline #.*$ extension bash extension bats extension bats.in script_exe bash 3rd_gen_scale 3.81 end_of_line_continuation \\$ C filter rm_comments_in_strings " /* */ filter rm_comments_in_strings " // filter call_regexp_common C++ extension c extension h 3rd_gen_scale 0.77 end_of_line_continuation \\$ conf-like filter remove_matches ^\s*# filter remove_inline #.*$ extension rpmlintrc extension spec filename .dockerignore filename .gitattributes filename .gitignore 3rd_gen_scale 1.00 end_of_line_continuation \\$ Dockerfile filter remove_matches ^\s*# filter remove_inline #.*$ extension df filename Dockerfile filename Dockerfile.argenv filename Dockerfile.centos_7ch filename Dockerfile.almalinux_8ch filename Dockerfile.debian_11ch filename Dockerfile.file-quirks filename Dockerfile.libfabric filename Dockerfile.metadata filename Dockerfile.mpich filename Dockerfile.nvidia filename Dockerfile.ocimanifest filename Dockerfile.openmpi filename Dockerfile.quick 3rd_gen_scale 2.00 end_of_line_continuation \\$ m4 filter remove_matches ^dnl\s filter remove_matches ^\s*# filter remove_inline #.*$ extension ac extension m4 3rd_gen_scale 1.00 Make filter remove_matches ^\s*# filter remove_inline #.*$ extension Gnumakefile extension Makefile extension am extension gnumakefile extension makefile extension mk filename Gnumakefile filename Makefile filename gnumakefile filename makefile script_exe make 3rd_gen_scale 2.50 end_of_line_continuation \\$ patch filter remove_matches ^# filter remove_matches ^\-\-\- filter remove_matches ^\+\+\+ filter remove_matches ^\s filter remove_matches ^@@ extension diff extension patch 3rd_gen_scale 1.00 plain text filter remove_matches TEXT_HAS_NO_COMMENTS_BUT_A_FILTER_IS_REQUIRED extension txt filename PERUSEME filename README filename VERSION filename approved-trailing-whitespace 3rd_gen_scale 1.00 POSIX sh filter remove_matches ^\s*# filter remove_inline #.*$ extension sh script_exe sh 3rd_gen_scale 3.81 end_of_line_continuation \\$ Python filter remove_matches ^\s*# filter docstring_to_C filter call_regexp_common C filter remove_inline #.*$ extension py extension py.in script_exe python script_exe python2.6 script_exe python2.7 script_exe python3 script_exe python3.3 script_exe python3.4 script_exe python3.5 3rd_gen_scale 4.20 end_of_line_continuation \\$ ReST filter remove_between_regex ^\.\. ^[^\s\.] extension rst 3rd_gen_scale 1.50 Ruby filter remove_matches ^\s*# filter remove_below_above ^=begin ^=end filter remove_inline #.*$ extension rake extension rb filename Rakefile filename rakefile filename Vagrantfile script_exe ruby 3rd_gen_scale 4.20 end_of_line_continuation \\$ VTK filter remove_matches ^\s*# extension vtk 3rd_gen_scale 1.00 end_of_line_continuation \\$ YAML filter remove_matches ^\s*# filter remove_inline #.*$ extension yaml extension yml 3rd_gen_scale 0.90 EOF if [[ -z $nolist ]]; then cat "$catfile" fi cat "$outfile" if ! grep -Eq "\b${ignore_ct} files ignored" "$outfile"; then grep -F '(unknown)' "$catfile" || true echo echo "🚨🚨🚨 cloc ignoring wrong number of files; expected ${ignore_ct} 🚨🚨🚨" cat /tmp/loc.ignored.out exit 1 fi } if [[ -e ./Makefile.in ]]; then echo '🚨🚨🚨 Makefile.in seems to exist 🚨🚨🚨' echo 'did you "./autogen.sh --clean --rm-lark"?' exit 1 fi min_cloc=1.81 if ! command -v cloc > /dev/null \ || [[ $( printf '%s\n%s\n' "$min_cloc" "$(cloc --version)" \ | sort -V | head -1 ) != "$min_cloc" ]]; then echo "🚨🚨🚨 no cloc version ≥ $min_cloc 🚨🚨🚨" echo 'did you try a better operating system?' exit 1 fi # program (Charliecloud itself) find ./bin ./lib -type f -a \( \ -name '*.c' \ -o -name '*.h' \ -o -name '*.py' \ -o -name '*.py.in' \ -o -name '*.sh' \ -o -path './bin/ch-*' \) | grep -Fv ./bin/ch-test | sort > /tmp/loc.program countem "PROGRAM" /tmp/loc.program # test suite find ./.github ./examples ./test -type f -a \( \ -name '*.bats' \ -o -name '*.bats.in' \ -o -name '*.c' \ -o -name '*.df' \ -o -name '*.patch' \ -o -name '*.py' \ -o -name '*.py.in' \ -o -name '*.sh' \ -o -name '*.vtk' \ -o -name '*.yml' \ -o -name 'Build.*' \ -o -name 'Dockerfile.*' \ -o -name .dockerignore \ -o -name Build \ -o -name Dockerfile \ -o -name Makefile \ -o -name README \ -o -path ./test/fixtures/README \ -o -path ./.github/PERUSEME \ -o -path ./examples/chtest/printns \ -o -path ./test/approved-trailing-whitespace \ -o -path ./test/common.bash \ -o -path ./test/doctest-auto \ -o -path ./test/old-storage \ -o -path ./test/sotest/files_inferrable.txt \ -o -path ./test/whiteout \) | sort > /tmp/loc.test echo ./bin/ch-test >> /tmp/loc.test countem "TEST SUITE & EXAMPLES" /tmp/loc.test # documentation find . -type f -a \( \ -path './doc/*.rst' \ -o -path ./README.rst \ -o -path ./doc/conf.py \ -o -path ./doc/make-deps-overview \ -o -path ./doc/man/README \ -o -path ./doc/publish \) | sort > /tmp/loc.doc countem "DOCUMENTATION" /tmp/loc.doc # build system find . -type f -a \( \ -name Makefile.am \ -o -path ./autogen.sh \ -o -path ./configure.ac \ -o -path ./misc/version \ -o -path ./misc/m4/README \) | sort > /tmp/loc.build countem "BUILD SYSTEM" /tmp/loc.build # packaging code find ./packaging \( -name Makefile.am \) -prune -o -type f \ -print | sort > /tmp/loc.packaging countem "PACKAGING" /tmp/loc.packaging # misc find . -type f -a \( \ -path ./.gitattributes \ -o -path ./.gitignore \ -o -path ./VERSION \ -o -path ./misc/docker-clean.sh \ -o -path ./misc/branches-tidy \ -o -path ./misc/grep \ -o -path ./misc/loc \) | sort > /tmp/loc.misc countem "MISCELLANEOUS" /tmp/loc.misc # ignored - this includes binaries and files we distribute but didn’t write find . -type f -a \( \ -name '*.ico' \ -o -name '.gitmodules' \ -o -name '*.png' \ -o -name '*.tar.gz' \ -o -name 'file[A-Z0-9y]*' \ -o -name 's_dir?' \ -o -name 's_file?' \ -o -path './misc/m4/*.m4' \ -o -path './packaging/debian/*' \ -o -name 'file_' \ -o -path ./LICENSE \ -o -path ./test/fixtures/empty-file \) | sort > /tmp/loc.ignored echo echo "*** IGNORED ***" cat /tmp/loc.ignored # everything find . \( -path ./.git \ -o -path ./doc/html \ -o -name .DS_Store \ -o -name ._.DS_Store \ -o -name __pycache__ \) \ -prune -o -type f -print \ | sort > /tmp/loc.all cat /tmp/loc.{all,ignored} | sort | uniq -u > /tmp/loc.total countem "TOTAL" /tmp/loc.total 0 nolist # test for files we forgot cat /tmp/loc.{program,test,doc,build,packaging,misc,ignored,all} \ | sort | uniq -c | (grep -Ev '^\s*2' || true) > /tmp/loc.extra if [[ -s /tmp/loc.extra ]]; then echo echo 'UNKNOWN FILES' cat /tmp/loc.extra echo echo '🚨🚨🚨 unknown files found 🚨🚨🚨' exit 1 fi # generate loc.rst cmd='s/^SUM:.* ([0-9]+)$/\1/p' cat < doc/_loc.rst .. Do not edit this file — it’s auto-generated. We pride ourselves on keeping Charliecloud lightweight and simple. The lines of code as of version $(cat VERSION) is: .. list-table:: * - Program itself - $(sed -En "${cmd}" /tmp/loc.program.out) * - Test suite & examples - $(sed -En "${cmd}" /tmp/loc.test.out) * - Documentation - $(sed -En "${cmd}" /tmp/loc.doc.out) * - Build system - $(sed -En "${cmd}" /tmp/loc.build.out) * - Packaging - $(sed -En "${cmd}" /tmp/loc.packaging.out) * - Miscellaneous - $(sed -En "${cmd}" /tmp/loc.misc.out) * - Total - $(sed -En "${cmd}" /tmp/loc.total.out) These include code only, excluding blank lines and comments. They were counted using \`cloc \`_ version $(cloc --version). We typically quote the "Program itself" number when describing the size of Charliecloud. (Please do not quote the size in Priedhorsky and Randles 2017, as that number is very out of date.) EOF echo cat doc/_loc.rst charliecloud-0.37/misc/m4/000077500000000000000000000000001457016721300153625ustar00rootroot00000000000000charliecloud-0.37/misc/m4/README000066400000000000000000000030271457016721300162440ustar00rootroot00000000000000This directory contains additional M4 macros for the build system. Currently, these are all from Autoconf Archive. While many distributions have an Autoconf Archive package, which autogen.sh can use if present, it’s a little uncommon to have installed, and we keep running into boxen where we want to run autogen.sh, but Autoconf Archive is not installed and we can’t install it promptly. There is a licensing exception for these macros that lets us redistribute them: “Every single one of those macros can be re-used without imposing any restrictions whatsoever on the licensing of the generated configure script. In particular, it is possible to use all those macros in configure scripts that are meant for non-free software.” [1] To add a macro: 1. Browse the Autoconf Archive documentation [1] and select the macro you want to use. 2. Download the macro file from the "m4" directory of the GitHub source code mirror [2] and put it in this directory. Use a release tag rather than a random Git commit. You can "wget" the URL you get with the "raw" button. (You could also use the master Git repo on Savannah [3], but GitHub is a lot easier to use.) 3. Record the macro and its last updated version in the list below. Macros in use: v2021.02.19 AX_CHECK_COMPILE_FLAG v2021.02.19 AX_COMPARE_VERSION v2021.02.19 AX_PTHREAD v2021.02.19 AX_WITH_PROG [1]: https://www.gnu.org/software/autoconf-archive/ [2]: https://github.com/autoconf-archive/autoconf-archive [3]: http://savannah.gnu.org/projects/autoconf-archive/ charliecloud-0.37/misc/m4/ax_check_compile_flag.m4000066400000000000000000000040701457016721300220730ustar00rootroot00000000000000# =========================================================================== # https://www.gnu.org/software/autoconf-archive/ax_check_compile_flag.html # =========================================================================== # # SYNOPSIS # # AX_CHECK_COMPILE_FLAG(FLAG, [ACTION-SUCCESS], [ACTION-FAILURE], [EXTRA-FLAGS], [INPUT]) # # DESCRIPTION # # Check whether the given FLAG works with the current language's compiler # or gives an error. (Warnings, however, are ignored) # # ACTION-SUCCESS/ACTION-FAILURE are shell commands to execute on # success/failure. # # If EXTRA-FLAGS is defined, it is added to the current language's default # flags (e.g. CFLAGS) when the check is done. The check is thus made with # the flags: "CFLAGS EXTRA-FLAGS FLAG". This can for example be used to # force the compiler to issue an error when a bad flag is given. # # INPUT gives an alternative input source to AC_COMPILE_IFELSE. # # NOTE: Implementation based on AX_CFLAGS_GCC_OPTION. Please keep this # macro in sync with AX_CHECK_{PREPROC,LINK}_FLAG. # # LICENSE # # Copyright (c) 2008 Guido U. Draheim # Copyright (c) 2011 Maarten Bosmans # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice # and this notice are preserved. This file is offered as-is, without any # warranty. #serial 6 AC_DEFUN([AX_CHECK_COMPILE_FLAG], [AC_PREREQ(2.64)dnl for _AC_LANG_PREFIX and AS_VAR_IF AS_VAR_PUSHDEF([CACHEVAR],[ax_cv_check_[]_AC_LANG_ABBREV[]flags_$4_$1])dnl AC_CACHE_CHECK([whether _AC_LANG compiler accepts $1], CACHEVAR, [ ax_check_save_flags=$[]_AC_LANG_PREFIX[]FLAGS _AC_LANG_PREFIX[]FLAGS="$[]_AC_LANG_PREFIX[]FLAGS $4 $1" AC_COMPILE_IFELSE([m4_default([$5],[AC_LANG_PROGRAM()])], [AS_VAR_SET(CACHEVAR,[yes])], [AS_VAR_SET(CACHEVAR,[no])]) _AC_LANG_PREFIX[]FLAGS=$ax_check_save_flags]) AS_VAR_IF(CACHEVAR,yes, [m4_default([$2], :)], [m4_default([$3], :)]) AS_VAR_POPDEF([CACHEVAR])dnl ])dnl AX_CHECK_COMPILE_FLAGS charliecloud-0.37/misc/m4/ax_compare_version.m4000066400000000000000000000146531457016721300215200ustar00rootroot00000000000000# =========================================================================== # https://www.gnu.org/software/autoconf-archive/ax_compare_version.html # =========================================================================== # # SYNOPSIS # # AX_COMPARE_VERSION(VERSION_A, OP, VERSION_B, [ACTION-IF-TRUE], [ACTION-IF-FALSE]) # # DESCRIPTION # # This macro compares two version strings. Due to the various number of # minor-version numbers that can exist, and the fact that string # comparisons are not compatible with numeric comparisons, this is not # necessarily trivial to do in a autoconf script. This macro makes doing # these comparisons easy. # # The six basic comparisons are available, as well as checking equality # limited to a certain number of minor-version levels. # # The operator OP determines what type of comparison to do, and can be one # of: # # eq - equal (test A == B) # ne - not equal (test A != B) # le - less than or equal (test A <= B) # ge - greater than or equal (test A >= B) # lt - less than (test A < B) # gt - greater than (test A > B) # # Additionally, the eq and ne operator can have a number after it to limit # the test to that number of minor versions. # # eq0 - equal up to the length of the shorter version # ne0 - not equal up to the length of the shorter version # eqN - equal up to N sub-version levels # neN - not equal up to N sub-version levels # # When the condition is true, shell commands ACTION-IF-TRUE are run, # otherwise shell commands ACTION-IF-FALSE are run. The environment # variable 'ax_compare_version' is always set to either 'true' or 'false' # as well. # # Examples: # # AX_COMPARE_VERSION([3.15.7],[lt],[3.15.8]) # AX_COMPARE_VERSION([3.15],[lt],[3.15.8]) # # would both be true. # # AX_COMPARE_VERSION([3.15.7],[eq],[3.15.8]) # AX_COMPARE_VERSION([3.15],[gt],[3.15.8]) # # would both be false. # # AX_COMPARE_VERSION([3.15.7],[eq2],[3.15.8]) # # would be true because it is only comparing two minor versions. # # AX_COMPARE_VERSION([3.15.7],[eq0],[3.15]) # # would be true because it is only comparing the lesser number of minor # versions of the two values. # # Note: The characters that separate the version numbers do not matter. An # empty string is the same as version 0. OP is evaluated by autoconf, not # configure, so must be a string, not a variable. # # The author would like to acknowledge Guido Draheim whose advice about # the m4_case and m4_ifvaln functions make this macro only include the # portions necessary to perform the specific comparison specified by the # OP argument in the final configure script. # # LICENSE # # Copyright (c) 2008 Tim Toolan # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice # and this notice are preserved. This file is offered as-is, without any # warranty. #serial 13 dnl ######################################################################### AC_DEFUN([AX_COMPARE_VERSION], [ AC_REQUIRE([AC_PROG_AWK]) # Used to indicate true or false condition ax_compare_version=false # Convert the two version strings to be compared into a format that # allows a simple string comparison. The end result is that a version # string of the form 1.12.5-r617 will be converted to the form # 0001001200050617. In other words, each number is zero padded to four # digits, and non digits are removed. AS_VAR_PUSHDEF([A],[ax_compare_version_A]) A=`echo "$1" | sed -e 's/\([[0-9]]*\)/Z\1Z/g' \ -e 's/Z\([[0-9]]\)Z/Z0\1Z/g' \ -e 's/Z\([[0-9]][[0-9]]\)Z/Z0\1Z/g' \ -e 's/Z\([[0-9]][[0-9]][[0-9]]\)Z/Z0\1Z/g' \ -e 's/[[^0-9]]//g'` AS_VAR_PUSHDEF([B],[ax_compare_version_B]) B=`echo "$3" | sed -e 's/\([[0-9]]*\)/Z\1Z/g' \ -e 's/Z\([[0-9]]\)Z/Z0\1Z/g' \ -e 's/Z\([[0-9]][[0-9]]\)Z/Z0\1Z/g' \ -e 's/Z\([[0-9]][[0-9]][[0-9]]\)Z/Z0\1Z/g' \ -e 's/[[^0-9]]//g'` dnl # In the case of le, ge, lt, and gt, the strings are sorted as necessary dnl # then the first line is used to determine if the condition is true. dnl # The sed right after the echo is to remove any indented white space. m4_case(m4_tolower($2), [lt],[ ax_compare_version=`echo "x$A x$B" | sed 's/^ *//' | sort -r | sed "s/x${A}/false/;s/x${B}/true/;1q"` ], [gt],[ ax_compare_version=`echo "x$A x$B" | sed 's/^ *//' | sort | sed "s/x${A}/false/;s/x${B}/true/;1q"` ], [le],[ ax_compare_version=`echo "x$A x$B" | sed 's/^ *//' | sort | sed "s/x${A}/true/;s/x${B}/false/;1q"` ], [ge],[ ax_compare_version=`echo "x$A x$B" | sed 's/^ *//' | sort -r | sed "s/x${A}/true/;s/x${B}/false/;1q"` ],[ dnl Split the operator from the subversion count if present. m4_bmatch(m4_substr($2,2), [0],[ # A count of zero means use the length of the shorter version. # Determine the number of characters in A and B. ax_compare_version_len_A=`echo "$A" | $AWK '{print(length)}'` ax_compare_version_len_B=`echo "$B" | $AWK '{print(length)}'` # Set A to no more than B's length and B to no more than A's length. A=`echo "$A" | sed "s/\(.\{$ax_compare_version_len_B\}\).*/\1/"` B=`echo "$B" | sed "s/\(.\{$ax_compare_version_len_A\}\).*/\1/"` ], [[0-9]+],[ # A count greater than zero means use only that many subversions A=`echo "$A" | sed "s/\(\([[0-9]]\{4\}\)\{m4_substr($2,2)\}\).*/\1/"` B=`echo "$B" | sed "s/\(\([[0-9]]\{4\}\)\{m4_substr($2,2)\}\).*/\1/"` ], [.+],[ AC_WARNING( [invalid OP numeric parameter: $2]) ],[]) # Pad zeros at end of numbers to make same length. ax_compare_version_tmp_A="$A`echo $B | sed 's/./0/g'`" B="$B`echo $A | sed 's/./0/g'`" A="$ax_compare_version_tmp_A" # Check for equality or inequality as necessary. m4_case(m4_tolower(m4_substr($2,0,2)), [eq],[ test "x$A" = "x$B" && ax_compare_version=true ], [ne],[ test "x$A" != "x$B" && ax_compare_version=true ],[ AC_WARNING([invalid OP parameter: $2]) ]) ]) AS_VAR_POPDEF([A])dnl AS_VAR_POPDEF([B])dnl dnl # Execute ACTION-IF-TRUE / ACTION-IF-FALSE. if test "$ax_compare_version" = "true" ; then m4_ifvaln([$4],[$4],[:])dnl m4_ifvaln([$5],[else $5])dnl fi ]) dnl AX_COMPARE_VERSION charliecloud-0.37/misc/m4/ax_pthread.m4000066400000000000000000000540461457016721300177540ustar00rootroot00000000000000# =========================================================================== # https://www.gnu.org/software/autoconf-archive/ax_pthread.html # =========================================================================== # # SYNOPSIS # # AX_PTHREAD([ACTION-IF-FOUND[, ACTION-IF-NOT-FOUND]]) # # DESCRIPTION # # This macro figures out how to build C programs using POSIX threads. It # sets the PTHREAD_LIBS output variable to the threads library and linker # flags, and the PTHREAD_CFLAGS output variable to any special C compiler # flags that are needed. (The user can also force certain compiler # flags/libs to be tested by setting these environment variables.) # # Also sets PTHREAD_CC and PTHREAD_CXX to any special C compiler that is # needed for multi-threaded programs (defaults to the value of CC # respectively CXX otherwise). (This is necessary on e.g. AIX to use the # special cc_r/CC_r compiler alias.) # # NOTE: You are assumed to not only compile your program with these flags, # but also to link with them as well. For example, you might link with # $PTHREAD_CC $CFLAGS $PTHREAD_CFLAGS $LDFLAGS ... $PTHREAD_LIBS $LIBS # $PTHREAD_CXX $CXXFLAGS $PTHREAD_CFLAGS $LDFLAGS ... $PTHREAD_LIBS $LIBS # # If you are only building threaded programs, you may wish to use these # variables in your default LIBS, CFLAGS, and CC: # # LIBS="$PTHREAD_LIBS $LIBS" # CFLAGS="$CFLAGS $PTHREAD_CFLAGS" # CXXFLAGS="$CXXFLAGS $PTHREAD_CFLAGS" # CC="$PTHREAD_CC" # CXX="$PTHREAD_CXX" # # In addition, if the PTHREAD_CREATE_JOINABLE thread-attribute constant # has a nonstandard name, this macro defines PTHREAD_CREATE_JOINABLE to # that name (e.g. PTHREAD_CREATE_UNDETACHED on AIX). # # Also HAVE_PTHREAD_PRIO_INHERIT is defined if pthread is found and the # PTHREAD_PRIO_INHERIT symbol is defined when compiling with # PTHREAD_CFLAGS. # # ACTION-IF-FOUND is a list of shell commands to run if a threads library # is found, and ACTION-IF-NOT-FOUND is a list of commands to run it if it # is not found. If ACTION-IF-FOUND is not specified, the default action # will define HAVE_PTHREAD. # # Please let the authors know if this macro fails on any platform, or if # you have any other suggestions or comments. This macro was based on work # by SGJ on autoconf scripts for FFTW (http://www.fftw.org/) (with help # from M. Frigo), as well as ac_pthread and hb_pthread macros posted by # Alejandro Forero Cuervo to the autoconf macro repository. We are also # grateful for the helpful feedback of numerous users. # # Updated for Autoconf 2.68 by Daniel Richard G. # # LICENSE # # Copyright (c) 2008 Steven G. Johnson # Copyright (c) 2011 Daniel Richard G. # Copyright (c) 2019 Marc Stevens # # This program is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation, either version 3 of the License, or (at your # option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General # Public License for more details. # # You should have received a copy of the GNU General Public License along # with this program. If not, see . # # As a special exception, the respective Autoconf Macro's copyright owner # gives unlimited permission to copy, distribute and modify the configure # scripts that are the output of Autoconf when processing the Macro. You # need not follow the terms of the GNU General Public License when using # or distributing such scripts, even though portions of the text of the # Macro appear in them. The GNU General Public License (GPL) does govern # all other use of the material that constitutes the Autoconf Macro. # # This special exception to the GPL applies to versions of the Autoconf # Macro released by the Autoconf Archive. When you make and distribute a # modified version of the Autoconf Macro, you may extend this special # exception to the GPL to apply to your modified version as well. #serial 30 AU_ALIAS([ACX_PTHREAD], [AX_PTHREAD]) AC_DEFUN([AX_PTHREAD], [ AC_REQUIRE([AC_CANONICAL_TARGET]) AC_REQUIRE([AC_PROG_CC]) AC_REQUIRE([AC_PROG_SED]) AC_LANG_PUSH([C]) ax_pthread_ok=no # We used to check for pthread.h first, but this fails if pthread.h # requires special compiler flags (e.g. on Tru64 or Sequent). # It gets checked for in the link test anyway. # First of all, check if the user has set any of the PTHREAD_LIBS, # etcetera environment variables, and if threads linking works using # them: if test "x$PTHREAD_CFLAGS$PTHREAD_LIBS" != "x"; then ax_pthread_save_CC="$CC" ax_pthread_save_CFLAGS="$CFLAGS" ax_pthread_save_LIBS="$LIBS" AS_IF([test "x$PTHREAD_CC" != "x"], [CC="$PTHREAD_CC"]) AS_IF([test "x$PTHREAD_CXX" != "x"], [CXX="$PTHREAD_CXX"]) CFLAGS="$CFLAGS $PTHREAD_CFLAGS" LIBS="$PTHREAD_LIBS $LIBS" AC_MSG_CHECKING([for pthread_join using $CC $PTHREAD_CFLAGS $PTHREAD_LIBS]) AC_LINK_IFELSE([AC_LANG_CALL([], [pthread_join])], [ax_pthread_ok=yes]) AC_MSG_RESULT([$ax_pthread_ok]) if test "x$ax_pthread_ok" = "xno"; then PTHREAD_LIBS="" PTHREAD_CFLAGS="" fi CC="$ax_pthread_save_CC" CFLAGS="$ax_pthread_save_CFLAGS" LIBS="$ax_pthread_save_LIBS" fi # We must check for the threads library under a number of different # names; the ordering is very important because some systems # (e.g. DEC) have both -lpthread and -lpthreads, where one of the # libraries is broken (non-POSIX). # Create a list of thread flags to try. Items with a "," contain both # C compiler flags (before ",") and linker flags (after ","). Other items # starting with a "-" are C compiler flags, and remaining items are # library names, except for "none" which indicates that we try without # any flags at all, and "pthread-config" which is a program returning # the flags for the Pth emulation library. ax_pthread_flags="pthreads none -Kthread -pthread -pthreads -mthreads pthread --thread-safe -mt pthread-config" # The ordering *is* (sometimes) important. Some notes on the # individual items follow: # pthreads: AIX (must check this before -lpthread) # none: in case threads are in libc; should be tried before -Kthread and # other compiler flags to prevent continual compiler warnings # -Kthread: Sequent (threads in libc, but -Kthread needed for pthread.h) # -pthread: Linux/gcc (kernel threads), BSD/gcc (userland threads), Tru64 # (Note: HP C rejects this with "bad form for `-t' option") # -pthreads: Solaris/gcc (Note: HP C also rejects) # -mt: Sun Workshop C (may only link SunOS threads [-lthread], but it # doesn't hurt to check since this sometimes defines pthreads and # -D_REENTRANT too), HP C (must be checked before -lpthread, which # is present but should not be used directly; and before -mthreads, # because the compiler interprets this as "-mt" + "-hreads") # -mthreads: Mingw32/gcc, Lynx/gcc # pthread: Linux, etcetera # --thread-safe: KAI C++ # pthread-config: use pthread-config program (for GNU Pth library) case $target_os in freebsd*) # -kthread: FreeBSD kernel threads (preferred to -pthread since SMP-able) # lthread: LinuxThreads port on FreeBSD (also preferred to -pthread) ax_pthread_flags="-kthread lthread $ax_pthread_flags" ;; hpux*) # From the cc(1) man page: "[-mt] Sets various -D flags to enable # multi-threading and also sets -lpthread." ax_pthread_flags="-mt -pthread pthread $ax_pthread_flags" ;; openedition*) # IBM z/OS requires a feature-test macro to be defined in order to # enable POSIX threads at all, so give the user a hint if this is # not set. (We don't define these ourselves, as they can affect # other portions of the system API in unpredictable ways.) AC_EGREP_CPP([AX_PTHREAD_ZOS_MISSING], [ # if !defined(_OPEN_THREADS) && !defined(_UNIX03_THREADS) AX_PTHREAD_ZOS_MISSING # endif ], [AC_MSG_WARN([IBM z/OS requires -D_OPEN_THREADS or -D_UNIX03_THREADS to enable pthreads support.])]) ;; solaris*) # On Solaris (at least, for some versions), libc contains stubbed # (non-functional) versions of the pthreads routines, so link-based # tests will erroneously succeed. (N.B.: The stubs are missing # pthread_cleanup_push, or rather a function called by this macro, # so we could check for that, but who knows whether they'll stub # that too in a future libc.) So we'll check first for the # standard Solaris way of linking pthreads (-mt -lpthread). ax_pthread_flags="-mt,-lpthread pthread $ax_pthread_flags" ;; esac # Are we compiling with Clang? AC_CACHE_CHECK([whether $CC is Clang], [ax_cv_PTHREAD_CLANG], [ax_cv_PTHREAD_CLANG=no # Note that Autoconf sets GCC=yes for Clang as well as GCC if test "x$GCC" = "xyes"; then AC_EGREP_CPP([AX_PTHREAD_CC_IS_CLANG], [/* Note: Clang 2.7 lacks __clang_[a-z]+__ */ # if defined(__clang__) && defined(__llvm__) AX_PTHREAD_CC_IS_CLANG # endif ], [ax_cv_PTHREAD_CLANG=yes]) fi ]) ax_pthread_clang="$ax_cv_PTHREAD_CLANG" # GCC generally uses -pthread, or -pthreads on some platforms (e.g. SPARC) # Note that for GCC and Clang -pthread generally implies -lpthread, # except when -nostdlib is passed. # This is problematic using libtool to build C++ shared libraries with pthread: # [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=25460 # [2] https://bugzilla.redhat.com/show_bug.cgi?id=661333 # [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=468555 # To solve this, first try -pthread together with -lpthread for GCC AS_IF([test "x$GCC" = "xyes"], [ax_pthread_flags="-pthread,-lpthread -pthread -pthreads $ax_pthread_flags"]) # Clang takes -pthread (never supported any other flag), but we'll try with -lpthread first AS_IF([test "x$ax_pthread_clang" = "xyes"], [ax_pthread_flags="-pthread,-lpthread -pthread"]) # The presence of a feature test macro requesting re-entrant function # definitions is, on some systems, a strong hint that pthreads support is # correctly enabled case $target_os in darwin* | hpux* | linux* | osf* | solaris*) ax_pthread_check_macro="_REENTRANT" ;; aix*) ax_pthread_check_macro="_THREAD_SAFE" ;; *) ax_pthread_check_macro="--" ;; esac AS_IF([test "x$ax_pthread_check_macro" = "x--"], [ax_pthread_check_cond=0], [ax_pthread_check_cond="!defined($ax_pthread_check_macro)"]) if test "x$ax_pthread_ok" = "xno"; then for ax_pthread_try_flag in $ax_pthread_flags; do case $ax_pthread_try_flag in none) AC_MSG_CHECKING([whether pthreads work without any flags]) ;; *,*) PTHREAD_CFLAGS=`echo $ax_pthread_try_flag | sed "s/^\(.*\),\(.*\)$/\1/"` PTHREAD_LIBS=`echo $ax_pthread_try_flag | sed "s/^\(.*\),\(.*\)$/\2/"` AC_MSG_CHECKING([whether pthreads work with "$PTHREAD_CFLAGS" and "$PTHREAD_LIBS"]) ;; -*) AC_MSG_CHECKING([whether pthreads work with $ax_pthread_try_flag]) PTHREAD_CFLAGS="$ax_pthread_try_flag" ;; pthread-config) AC_CHECK_PROG([ax_pthread_config], [pthread-config], [yes], [no]) AS_IF([test "x$ax_pthread_config" = "xno"], [continue]) PTHREAD_CFLAGS="`pthread-config --cflags`" PTHREAD_LIBS="`pthread-config --ldflags` `pthread-config --libs`" ;; *) AC_MSG_CHECKING([for the pthreads library -l$ax_pthread_try_flag]) PTHREAD_LIBS="-l$ax_pthread_try_flag" ;; esac ax_pthread_save_CFLAGS="$CFLAGS" ax_pthread_save_LIBS="$LIBS" CFLAGS="$CFLAGS $PTHREAD_CFLAGS" LIBS="$PTHREAD_LIBS $LIBS" # Check for various functions. We must include pthread.h, # since some functions may be macros. (On the Sequent, we # need a special flag -Kthread to make this header compile.) # We check for pthread_join because it is in -lpthread on IRIX # while pthread_create is in libc. We check for pthread_attr_init # due to DEC craziness with -lpthreads. We check for # pthread_cleanup_push because it is one of the few pthread # functions on Solaris that doesn't have a non-functional libc stub. # We try pthread_create on general principles. AC_LINK_IFELSE([AC_LANG_PROGRAM([#include # if $ax_pthread_check_cond # error "$ax_pthread_check_macro must be defined" # endif static void *some_global = NULL; static void routine(void *a) { /* To avoid any unused-parameter or unused-but-set-parameter warning. */ some_global = a; } static void *start_routine(void *a) { return a; }], [pthread_t th; pthread_attr_t attr; pthread_create(&th, 0, start_routine, 0); pthread_join(th, 0); pthread_attr_init(&attr); pthread_cleanup_push(routine, 0); pthread_cleanup_pop(0) /* ; */])], [ax_pthread_ok=yes], []) CFLAGS="$ax_pthread_save_CFLAGS" LIBS="$ax_pthread_save_LIBS" AC_MSG_RESULT([$ax_pthread_ok]) AS_IF([test "x$ax_pthread_ok" = "xyes"], [break]) PTHREAD_LIBS="" PTHREAD_CFLAGS="" done fi # Clang needs special handling, because older versions handle the -pthread # option in a rather... idiosyncratic way if test "x$ax_pthread_clang" = "xyes"; then # Clang takes -pthread; it has never supported any other flag # (Note 1: This will need to be revisited if a system that Clang # supports has POSIX threads in a separate library. This tends not # to be the way of modern systems, but it's conceivable.) # (Note 2: On some systems, notably Darwin, -pthread is not needed # to get POSIX threads support; the API is always present and # active. We could reasonably leave PTHREAD_CFLAGS empty. But # -pthread does define _REENTRANT, and while the Darwin headers # ignore this macro, third-party headers might not.) # However, older versions of Clang make a point of warning the user # that, in an invocation where only linking and no compilation is # taking place, the -pthread option has no effect ("argument unused # during compilation"). They expect -pthread to be passed in only # when source code is being compiled. # # Problem is, this is at odds with the way Automake and most other # C build frameworks function, which is that the same flags used in # compilation (CFLAGS) are also used in linking. Many systems # supported by AX_PTHREAD require exactly this for POSIX threads # support, and in fact it is often not straightforward to specify a # flag that is used only in the compilation phase and not in # linking. Such a scenario is extremely rare in practice. # # Even though use of the -pthread flag in linking would only print # a warning, this can be a nuisance for well-run software projects # that build with -Werror. So if the active version of Clang has # this misfeature, we search for an option to squash it. AC_CACHE_CHECK([whether Clang needs flag to prevent "argument unused" warning when linking with -pthread], [ax_cv_PTHREAD_CLANG_NO_WARN_FLAG], [ax_cv_PTHREAD_CLANG_NO_WARN_FLAG=unknown # Create an alternate version of $ac_link that compiles and # links in two steps (.c -> .o, .o -> exe) instead of one # (.c -> exe), because the warning occurs only in the second # step ax_pthread_save_ac_link="$ac_link" ax_pthread_sed='s/conftest\.\$ac_ext/conftest.$ac_objext/g' ax_pthread_link_step=`AS_ECHO(["$ac_link"]) | sed "$ax_pthread_sed"` ax_pthread_2step_ac_link="($ac_compile) && (echo ==== >&5) && ($ax_pthread_link_step)" ax_pthread_save_CFLAGS="$CFLAGS" for ax_pthread_try in '' -Qunused-arguments -Wno-unused-command-line-argument unknown; do AS_IF([test "x$ax_pthread_try" = "xunknown"], [break]) CFLAGS="-Werror -Wunknown-warning-option $ax_pthread_try -pthread $ax_pthread_save_CFLAGS" ac_link="$ax_pthread_save_ac_link" AC_LINK_IFELSE([AC_LANG_SOURCE([[int main(void){return 0;}]])], [ac_link="$ax_pthread_2step_ac_link" AC_LINK_IFELSE([AC_LANG_SOURCE([[int main(void){return 0;}]])], [break]) ]) done ac_link="$ax_pthread_save_ac_link" CFLAGS="$ax_pthread_save_CFLAGS" AS_IF([test "x$ax_pthread_try" = "x"], [ax_pthread_try=no]) ax_cv_PTHREAD_CLANG_NO_WARN_FLAG="$ax_pthread_try" ]) case "$ax_cv_PTHREAD_CLANG_NO_WARN_FLAG" in no | unknown) ;; *) PTHREAD_CFLAGS="$ax_cv_PTHREAD_CLANG_NO_WARN_FLAG $PTHREAD_CFLAGS" ;; esac fi # $ax_pthread_clang = yes # Various other checks: if test "x$ax_pthread_ok" = "xyes"; then ax_pthread_save_CFLAGS="$CFLAGS" ax_pthread_save_LIBS="$LIBS" CFLAGS="$CFLAGS $PTHREAD_CFLAGS" LIBS="$PTHREAD_LIBS $LIBS" # Detect AIX lossage: JOINABLE attribute is called UNDETACHED. AC_CACHE_CHECK([for joinable pthread attribute], [ax_cv_PTHREAD_JOINABLE_ATTR], [ax_cv_PTHREAD_JOINABLE_ATTR=unknown for ax_pthread_attr in PTHREAD_CREATE_JOINABLE PTHREAD_CREATE_UNDETACHED; do AC_LINK_IFELSE([AC_LANG_PROGRAM([#include ], [int attr = $ax_pthread_attr; return attr /* ; */])], [ax_cv_PTHREAD_JOINABLE_ATTR=$ax_pthread_attr; break], []) done ]) AS_IF([test "x$ax_cv_PTHREAD_JOINABLE_ATTR" != "xunknown" && \ test "x$ax_cv_PTHREAD_JOINABLE_ATTR" != "xPTHREAD_CREATE_JOINABLE" && \ test "x$ax_pthread_joinable_attr_defined" != "xyes"], [AC_DEFINE_UNQUOTED([PTHREAD_CREATE_JOINABLE], [$ax_cv_PTHREAD_JOINABLE_ATTR], [Define to necessary symbol if this constant uses a non-standard name on your system.]) ax_pthread_joinable_attr_defined=yes ]) AC_CACHE_CHECK([whether more special flags are required for pthreads], [ax_cv_PTHREAD_SPECIAL_FLAGS], [ax_cv_PTHREAD_SPECIAL_FLAGS=no case $target_os in solaris*) ax_cv_PTHREAD_SPECIAL_FLAGS="-D_POSIX_PTHREAD_SEMANTICS" ;; esac ]) AS_IF([test "x$ax_cv_PTHREAD_SPECIAL_FLAGS" != "xno" && \ test "x$ax_pthread_special_flags_added" != "xyes"], [PTHREAD_CFLAGS="$ax_cv_PTHREAD_SPECIAL_FLAGS $PTHREAD_CFLAGS" ax_pthread_special_flags_added=yes]) AC_CACHE_CHECK([for PTHREAD_PRIO_INHERIT], [ax_cv_PTHREAD_PRIO_INHERIT], [AC_LINK_IFELSE([AC_LANG_PROGRAM([[#include ]], [[int i = PTHREAD_PRIO_INHERIT; return i;]])], [ax_cv_PTHREAD_PRIO_INHERIT=yes], [ax_cv_PTHREAD_PRIO_INHERIT=no]) ]) AS_IF([test "x$ax_cv_PTHREAD_PRIO_INHERIT" = "xyes" && \ test "x$ax_pthread_prio_inherit_defined" != "xyes"], [AC_DEFINE([HAVE_PTHREAD_PRIO_INHERIT], [1], [Have PTHREAD_PRIO_INHERIT.]) ax_pthread_prio_inherit_defined=yes ]) CFLAGS="$ax_pthread_save_CFLAGS" LIBS="$ax_pthread_save_LIBS" # More AIX lossage: compile with *_r variant if test "x$GCC" != "xyes"; then case $target_os in aix*) AS_CASE(["x/$CC"], [x*/c89|x*/c89_128|x*/c99|x*/c99_128|x*/cc|x*/cc128|x*/xlc|x*/xlc_v6|x*/xlc128|x*/xlc128_v6], [#handle absolute path differently from PATH based program lookup AS_CASE(["x$CC"], [x/*], [ AS_IF([AS_EXECUTABLE_P([${CC}_r])],[PTHREAD_CC="${CC}_r"]) AS_IF([test "x${CXX}" != "x"], [AS_IF([AS_EXECUTABLE_P([${CXX}_r])],[PTHREAD_CXX="${CXX}_r"])]) ], [ AC_CHECK_PROGS([PTHREAD_CC],[${CC}_r],[$CC]) AS_IF([test "x${CXX}" != "x"], [AC_CHECK_PROGS([PTHREAD_CXX],[${CXX}_r],[$CXX])]) ] ) ]) ;; esac fi fi test -n "$PTHREAD_CC" || PTHREAD_CC="$CC" test -n "$PTHREAD_CXX" || PTHREAD_CXX="$CXX" AC_SUBST([PTHREAD_LIBS]) AC_SUBST([PTHREAD_CFLAGS]) AC_SUBST([PTHREAD_CC]) AC_SUBST([PTHREAD_CXX]) # Finally, execute ACTION-IF-FOUND/ACTION-IF-NOT-FOUND: if test "x$ax_pthread_ok" = "xyes"; then ifelse([$1],,[AC_DEFINE([HAVE_PTHREAD],[1],[Define if you have POSIX threads libraries and header files.])],[$1]) : else ax_pthread_ok=no $2 fi AC_LANG_POP ])dnl AX_PTHREAD charliecloud-0.37/misc/m4/ax_with_prog.m4000066400000000000000000000047121457016721300203220ustar00rootroot00000000000000# =========================================================================== # https://www.gnu.org/software/autoconf-archive/ax_with_prog.html # =========================================================================== # # SYNOPSIS # # AX_WITH_PROG([VARIABLE],[program],[VALUE-IF-NOT-FOUND],[PATH]) # # DESCRIPTION # # Locates an installed program binary, placing the result in the precious # variable VARIABLE. Accepts a present VARIABLE, then --with-program, and # failing that searches for program in the given path (which defaults to # the system path). If program is found, VARIABLE is set to the full path # of the binary; if it is not found VARIABLE is set to VALUE-IF-NOT-FOUND # if provided, unchanged otherwise. # # A typical example could be the following one: # # AX_WITH_PROG(PERL,perl) # # NOTE: This macro is based upon the original AX_WITH_PYTHON macro from # Dustin J. Mitchell . # # LICENSE # # Copyright (c) 2008 Francesco Salvestrini # Copyright (c) 2008 Dustin J. Mitchell # # Copying and distribution of this file, with or without modification, are # permitted in any medium without royalty provided the copyright notice # and this notice are preserved. This file is offered as-is, without any # warranty. #serial 17 AC_DEFUN([AX_WITH_PROG],[ AC_PREREQ([2.61]) pushdef([VARIABLE],$1) pushdef([EXECUTABLE],$2) pushdef([VALUE_IF_NOT_FOUND],$3) pushdef([PATH_PROG],$4) AC_ARG_VAR(VARIABLE,Absolute path to EXECUTABLE executable) AS_IF(test -z "$VARIABLE",[ AC_MSG_CHECKING(whether EXECUTABLE executable path has been provided) AC_ARG_WITH(EXECUTABLE,AS_HELP_STRING([--with-EXECUTABLE=[[[PATH]]]],absolute path to EXECUTABLE executable), [ AS_IF([test "$withval" != yes && test "$withval" != no],[ VARIABLE="$withval" AC_MSG_RESULT($VARIABLE) ],[ VARIABLE="" AC_MSG_RESULT([no]) AS_IF([test "$withval" != no], [ AC_PATH_PROG([]VARIABLE[],[]EXECUTABLE[],[]VALUE_IF_NOT_FOUND[],[]PATH_PROG[]) ]) ]) ],[ AC_MSG_RESULT([no]) AC_PATH_PROG([]VARIABLE[],[]EXECUTABLE[],[]VALUE_IF_NOT_FOUND[],[]PATH_PROG[]) ]) ]) popdef([PATH_PROG]) popdef([VALUE_IF_NOT_FOUND]) popdef([EXECUTABLE]) popdef([VARIABLE]) ]) charliecloud-0.37/misc/version000077500000000000000000000021571457016721300164620ustar00rootroot00000000000000#!/bin/sh # Compute and print out the full version number. See FAQ for details. # # This script should usually be run once, by Autotools, and the result # propagated using Autotools. This propagates the Git information into # tarballs, and otherwise, you can get a mismatch between different parts of # the software. set -e ch_base=$(cd "$(dirname "$0")" && pwd)/.. version_file=${ch_base}/VERSION version_simple=$(cat "$version_file") case $version_simple in *~*) prerelease=yes ;; *) prerelease= ;; esac if [ ! -e "${ch_base}/.git" ] || [ -z "$prerelease" ]; then # no Git or release version; use simple version printf "%s\n" "$version_simple" else # add Git stuff git_branch=$( git rev-parse --abbrev-ref HEAD \ | sed 's/[^A-Za-z0-9]//g' \ | sed 's/$/./g' \ | sed 's/master.//g') git_hash=$(git rev-parse --short HEAD) dirty=$(git diff-index --quiet HEAD || echo .dirty) printf '%s+%s%s%s\n' "$version_simple" \ "$git_branch" \ "$git_hash" \ "$dirty" fi charliecloud-0.37/packaging/000077500000000000000000000000001457016721300160335ustar00rootroot00000000000000charliecloud-0.37/packaging/Makefile.am000066400000000000000000000002441457016721300200670ustar00rootroot00000000000000EXTRA_DIST = \ README \ requirements.txt \ fedora/build \ fedora/charliecloud.rpmlintrc \ fedora/charliecloud.spec \ fedora/el7-pkgdir.patch \ fedora/upstream.spec charliecloud-0.37/packaging/README000066400000000000000000000014101457016721300167070ustar00rootroot00000000000000openSUSE -------- openSUSE packages are maintained and built using the openSUSE Build Service (OBS) at https://build.opensuse.org To use OBS to create your own charliecloud packages, you’ll need to create a user account and then copy the packaging from the devel package at: https://build.opensuse.org/package/show/network:cluster/charliecloud This can be done from the web interface clicking on “Branch package”. Then you will be able to fetch locally all the packaging files from your copy (or branch), make your changes and send them to OBS that will build packages and create package repositories for your branch. The beginner’s guide on how to use OBS with the osc command-line tool can be found at https://openbuildservice.org/help/manuals/obs-user-guide/ charliecloud-0.37/packaging/fedora/000077500000000000000000000000001457016721300172735ustar00rootroot00000000000000charliecloud-0.37/packaging/fedora/build000077500000000000000000000223211457016721300203200ustar00rootroot00000000000000#!/usr/bin/env python3 # See contributors’ guide for documentation of this script. from __future__ import print_function import argparse import errno import os import platform import pwd import re import shlex import shutil import socket import subprocess import sys import time CH_BASE = os.path.abspath(os.path.dirname(__file__) + "/../..") CH_RUN = [CH_BASE + "/bin/ch-run"] PACKAGES = ["charliecloud", "charliecloud-builder", "charliecloud-debuginfo", "charliecloud-doc", "charliecloud-test"] ARCH = platform.machine() def main(): # Parse arguments. ap = argparse.ArgumentParser() ap.add_argument("image", metavar="DIR") ap.add_argument("version"), ap.add_argument("--install", action="store_true") ap.add_argument("--rpmbuild", metavar="DIR", default="%s/rpmbuild" % os.getenv("HOME")) args = ap.parse_args() print("# Charliecloud root: %s" % CH_BASE) print("architecture: %s" % ARCH) print("""\ version: %(version)s image: %(image)s install: %(install)s rpmbuild root: %(rpmbuild)s """ % args.__dict__) # Use the generic base locale, rather than whatever the user has set, as # that’s the only one guaranteed to be installed in the containers. os.environ["LC_ALL"] = "C" # What's the real Git version? if (args.version == "HEAD"): try: # If we’re on a branch, we want to build on that branch so the branch # name shows up in the version name. commit = subprocess.check_output(["git", "symbolic-ref", "-q", "--short", "HEAD"], encoding="UTF-8")[:-1] except subprocess.CalledProcessError as x: if (x.returncode != 1): raise # Detached HEAD (e.g. CI) is also fine; use commit hash. commit = subprocess.check_output(["git", "rev-parse", "--verify", "HEAD"], encoding="UTF-8")[:-1] rpm_release = "0" else: m = re.search(r"([0-9.]+)-([0-9]+)", args.version) commit = "v" + m[1] rpm_release = m[2] # Create rpmbuild root rpm_sources = args.rpmbuild + '/SOURCES' rpm_specs = args.rpmbuild + '/SPECS' rpm_pips = args.rpmbuild + '/pip' for d in (rpm_sources, rpm_specs, rpm_pips): print("# mkdir -p %s" % d) try: os.makedirs(d) except OSError as x: if (x.errno != errno.EEXIST): raise # Determine distro. rpms_dist = cmd_out(CH_RUN, args.image, '--', 'rpmbuild', '--eval', '%{?dist}') rpms_dist = rpms_dist.split('.')[-1] # FIXME: el8 doesn’t have bats; skip for now. if (rpms_dist == 'el8'): PACKAGES.remove('charliecloud-test') # Get a clean Git checkout of the desired version. We do this by making a # temporary clone so as not to mess up the WD. git_tmp = rpm_sources + '/charliecloud' print("# cloning into %s and checking out commit %s" % (git_tmp, commit)) cmd("git", "clone", '.', git_tmp) cmd("git", "checkout", commit, cwd=git_tmp) # Build tarball. print("# building source tarball") # pip3 expects $HOME, here /root, to be writeable for its cache. If not, # the warning is that the directory “is not owned by the current user”, # which is not true. The warning further claims that “*caching* wheels has # been disabled” (emphasis added), but in fact wheels are disabled # entirely, so autogen.sh fails with “embedded Lark is broken”. Thanks pip! cmd(CH_RUN, "-b", "%s:/mnt/0" % git_tmp, "-b", "%s:/root" % rpm_pips, "-c", "/mnt/0", args.image, "--", "./autogen.sh") cmd(CH_RUN, "-b", "%s:/mnt/0" % git_tmp, "-c", "/mnt/0", args.image, "--", "./configure") cmd(CH_RUN, "-b", "%s:/mnt/0" % git_tmp, "-c", "/mnt/0", args.image, "--", "make") cmd(CH_RUN, "-b", "%s:/mnt/0" % git_tmp, "-c", "/mnt/0", args.image, "--", "make", "dist") ch_version = open(git_tmp + "/lib/version.txt").read()[:-1] ch_tarball = "charliecloud-%s.tar.gz" % ch_version print("# Charliecloud version: %s" % ch_version) print("# source tarball: %s" % ch_tarball) os.rename("%s/%s" % (git_tmp, ch_tarball), "%s/%s" % (rpm_sources, ch_tarball)) # Copy lint configuration and patchs. # FIXME: Put version into destination sometime? shutil.copy("%s/packaging/fedora/charliecloud.rpmlintrc" % CH_BASE, "%s/charliecloud.rpmlintrc" % rpm_specs) shutil.copy("%s/packaging/fedora/el7-pkgdir.patch" % CH_BASE, "%s/el7-pkgdir.patch" % rpm_sources) # Remove temporary Git directory. print("# rm -rf %s" % git_tmp) shutil.rmtree(git_tmp) # Copy and patch spec file. rpm_vr = "%s-%s" % (ch_version, rpm_release) # Fedora requires no version in spec file. Add a version for pre-release # specs to make it hard to upload one to Fedora by mistake. if ("~pre" not in ch_version): spec = "charliecloud.spec" else: spec = "charliecloud-%s.spec" % rpm_vr with open("%s/packaging/fedora/charliecloud.spec" % CH_BASE, "rt") as in_, \ open("%s/%s" % (rpm_specs, spec), "wt") as out: print("# writing %s" % out.name) t = in_.read() t = t.replace("@VERSION@", ch_version) t = t.replace("@RELEASE@", rpm_release) if ("~pre" in ch_version): # Add dummy changelog entry. timestamp = time.strftime("%a %b %d %Y") # required format name = pwd.getpwuid(os.geteuid()).pw_gecos.split(",")[0] moniker = pwd.getpwuid(os.geteuid()).pw_name domain = re.sub(r"^[^.]+.", "", socket.getfqdn()) t = t.replace("%changelog\n", """\ %%changelog * %s %s <%s@%s> %s - Pre-release package. See Git history for what is going on. """ % (timestamp, name, moniker, domain, rpm_vr)) else: # Verify requested version matches changelog. m = re.search(r"%changelog\n.+?([0-9.-]+)\n", t) if (m[1] != rpm_vr): print("requested version %s != changelog %s" % (rpm_vr, m[1])) sys.exit(1) out.write(t) # Prepare build and rpmlint arguments. container = [] rpmbuild_args = [] rpmlint_args = [] # Use /usr/local/src because rpmbuild fails with “%{_topdir}/BUILD” # shorter than “/usr/src/debug” (yes, really!) [1,2]. # # [1]: https://access.redhat.com/solutions/1426113 # [2]: https://gbenson.net/?p=367 rpms_src = "/usr/local/src/RPMS" rpms_arch = "%s/%s" % (rpms_src, ARCH) rpms_noarch = "%s/noarch" % rpms_src rpms = {'charliecloud': [rpms_arch, '%s.%s.rpm' % (rpms_dist, ARCH)], 'charliecloud-builder': [rpms_noarch, '%s.noarch.rpm' % rpms_dist], 'charliecloud-debuginfo': [rpms_arch, '%s.%s.rpm' % (rpms_dist, ARCH)], 'charliecloud-doc': [rpms_noarch, '%s.noarch.rpm' % rpms_dist], 'charliecloud-test': [rpms_arch, '%s.%s.rpm' % (rpms_dist, ARCH)]} rpm_specs = "/usr/local/src/SPECS" rpm_sources = "/usr/local/src/SOURCES" rpmbuild_args += ["--define", "_topdir /usr/local/src"] rpmlint_args += ["--file", "%s/charliecloud.rpmlintrc" % rpm_specs] container += [CH_BASE + "/bin/ch-run", "-w", "-b", "%s:/usr/local/src" % args.rpmbuild, args.image, "--"] # Build RPMs. cmd(container, "rpmbuild", rpmbuild_args, "--version") cmd(container, "rpmbuild", rpmbuild_args, "-ba", "%s/%s" % (rpm_specs, spec)) cmd(container, "ls", "-lh", rpms_arch) cmd(container, "ls", "-lh", rpms_noarch) # Install RPMs. if (args.install): print("# uninstalling (most errors can be ignored)") cmd_ok(container, "rpm", "--erase", PACKAGES) print("# installing") for p in PACKAGES: rpm_path = rpms[p][0] rpm_ext = rpms[p][-1] file = '%s/%s-%s.%s' % (rpm_path, p, rpm_vr, rpm_ext) cmd(container, "rpm", "--install", file) cmd(container, "rpm", "-qa", "charliecloud*") # Lint RPMs and spec file. Last so problems that don’t result in program # returning error are more obvious. print("# linting") cmd(container, "rpmlint", rpmlint_args, "%s/%s" % (rpm_specs, spec)) for p in PACKAGES: rpm_path = rpms[p][0] rpm_ext = rpms[p][-1] file = '%s/%s-%s.%s' % (rpm_path, p, rpm_vr, rpm_ext) cmd(container, "test", "-e", file) cmd(container, "rpmlint", rpmlint_args, file) # Success! print("# done") def cmd(*args, **kwargs): cmd_real(subprocess.check_call, *args, **kwargs) def cmd_ok(*args, **kwargs): rc = cmd_real(subprocess.call, *args, **kwargs) return (rc == 0) def cmd_out(*args, **kwargs): out = cmd_real(subprocess.check_output, *args, encoding="UTF-8", **kwargs) return out.rstrip() # remove trailing newline def cmd_real(runf, *args, **kwargs): # flatten any sub-lists (kludge) args2 = [] for arg in args: if (isinstance(arg, list)): args2 += arg else: args2.append(arg) # print and run print("$", end="") for arg in args2: arg = shlex.quote(arg) print(" " + arg, end="") print() return runf(args2, **kwargs) if (__name__ == "__main__"): main() charliecloud-0.37/packaging/fedora/charliecloud.rpmlintrc000066400000000000000000000055771457016721300237030ustar00rootroot00000000000000# This file is used to supress false positive errors and warnings generated by # rpmlint when used with our charliecloud packages. ### charliecloud.spec ### # The chroot tests are very fragile and have been removed upstream. See: # https://github.com/rpm-software-management/rpmlint/commit/83f915a54d23f7a912ed42b84ccb4e373bec8ad9 addFilter(r'missing-call-to-chdir-with-chroot') # The RPM build script will generate invalid source URLs for non-release # versions, e.g., '0.9.8~pre+epelpackage.41fe9fd'. addFilter(r'invalid-url') # charliecloud # We don't have architecture specific libraries, thus we can put files in /lib. # The rpm macro, _lib, expands to lib64, which is not what we want. Rather than # patch our install to an incorrect library path we ignore the lint error. addFilter(r'hardcoded-library-path') # Charliecloud uses pivot_root(2), not chroot(2), for containerization. The # calls to chroot(2) are part of the pivot_root(2) dance and not relevant to # Charliecloud's security posture. addFilter(r'missing-call-to-chdir-with-chroot') # The charliecloud example, chtest, has python scripts. addFilter(r'doc-file-dependency') # charliecloud-debuginfo # The only files under /usr/lib are those placed there by rpmbuild. addFilter(r'only-non-binary-in-usr-lib') # Ignore a false positive warning concerning pycache files and byte code. # https://bugzilla.redhat.com/show_bug.cgi?id=1286382 addFilter(r'python-bytecode-without-source') # We don't specify a version because the offending package was not out long and # we intend to remove Obsolete lines in the near future. addFilter(r'unversioned-explicit-obsoletes') ### charliecloud-test ### # Charliecloud is a container runtime. These shared objects are never used in # the host environment; rather, they are compiled by the test suite (both # running and examination of which serve as end-user documentation) and injected # into the container (guest) via utility script 'ch-fromhost'. The ldconfig # links are generated inside the container runtime environment. For more # information, see the test file: test/run/ch-fromhost.bats (line 108). addFilter(r'no-ldconfig-symlink') addFilter(r'library-without-ldconfig-postin') addFilter(r'library-without-ldconfig-postun') # The test suite has a few C files, e.g. userns.c, pivot_root.c, # chroot-escape.c, sotest.c, setgroups.c, mknods.c, setuid.c, etc., that # document -- line-by-line in some cases -- various components of the open source # runtime. These C files serve to show end users how containers work; some of # them are used explicitly during test suite runtime. addFilter(r'devel-file-in-non-devel-package') # The symlink to /usr/bin is created and does exist. addFilter(r'dangling-relative-symlink') # Funky files used as test fixtures: addFilter(r'dangling-symlink') # to /tmp addFilter(r'hidden-file-or-dir') # .dockerignore addFilter(r'zero-length') # for file copy test charliecloud-0.37/packaging/fedora/charliecloud.spec000066400000000000000000000125421457016721300226110ustar00rootroot00000000000000# Charliecloud fedora package spec file # # Contributors: # Dave Love @loveshack # Michael Jennings @mej # Jordan Ogas @jogas # Reid Priedhorksy @reidpr # Don't try to compile python3 files with /usr/bin/python. %{?el7:%global __python %__python3} Name: charliecloud Version: @VERSION@ Release: @RELEASE@%{?dist} Summary: Lightweight user-defined software stacks for high-performance computing License: ASL 2.0 URL: https://hpc.github.io/%{name}/ Source0: https://github.com/hpc/%{name}/releases/downloads/v%{version}/%{name}-%{version}.tar.gz BuildRequires: gcc rsync bash Requires: squashfuse squashfs-tools Patch1: el7-pkgdir.patch %description Charliecloud uses Linux user namespaces to run containers with no privileged operations or daemons and minimal configuration changes on center resources. This simple approach avoids most security risks while maintaining access to the performance and functionality already on offer. Container images can be built using Docker or anything else that can generate a standard Linux filesystem tree. For more information: https://hpc.github.io/charliecloud %package builder Summary: Charliecloud container image building tools License: ASL 2.0 and MIT BuildArch: noarch BuildRequires: python3-devel BuildRequires: python%{python3_pkgversion}-lark-parser BuildRequires: python%{python3_pkgversion}-requests Requires: %{name} Requires: python3 Requires: python%{python3_pkgversion}-lark-parser Requires: python%{python3_pkgversion}-requests Provides: bundled(python%{python3_pkgversion}-lark-parser) = 1.1.9 %description builder This package provides ch-image, Charliecloud's completely unprivileged container image manipulation tool. %package doc Summary: Charliecloud html documentation License: BSD and ASL 2.0 BuildArch: noarch Obsoletes: %{name}-doc < %{version}-%{release} BuildRequires: python%{python3_pkgversion}-sphinx BuildRequires: python%{python3_pkgversion}-sphinx_rtd_theme Requires: python%{python3_pkgversion}-sphinx_rtd_theme %description doc Html and man page documentation for %{name}. %package test Summary: Charliecloud test suite License: ASL 2.0 Requires: %{name} %{name}-builder /usr/bin/bats Obsoletes: %{name}-test < %{version}-%{release} %description test Test fixtures for %{name}. %prep %setup -q %if 0%{?el7} %patch1 -p1 %endif %build # Use old inlining behavior, see: # https://github.com/hpc/charliecloud/issues/735 CFLAGS=${CFLAGS:-%optflags -fgnu89-inline}; export CFLAGS %configure --docdir=%{_pkgdocdir} \ --libdir=%{_prefix}/lib \ --with-python=/usr/bin/python3 \ %if 0%{?el7} --with-sphinx-build=%{_bindir}/sphinx-build-3.6 %else --with-sphinx-build=%{_bindir}/sphinx-build %endif %install %make_install cat > README.EL7 </etc/sysctl.d/51-userns.conf sysctl -p /etc/sysctl.d/51-userns.conf Note for versions below RHEL7.6, you will also need to enable user namespaces: grubby --args=namespace.unpriv_enable=1 --update-kernel=ALL reboot Please visit https://hpc.github.io/charliecloud/ for more information. EOF # Remove bundled sphinx bits. %{__rm} -rf %{buildroot}%{_pkgdocdir}/html/_static/css %{__rm} -rf %{buildroot}%{_pkgdocdir}/html/_static/fonts %{__rm} -rf %{buildroot}%{_pkgdocdir}/html/_static/js # Use Fedora package sphinx bits. sphinxdir=%{python3_sitelib}/sphinx_rtd_theme/static ln -s "${sphinxdir}/css" %{buildroot}%{_pkgdocdir}/html/_static/css ln -s "${sphinxdir}/fonts" %{buildroot}%{_pkgdocdir}/html/_static/fonts ln -s "${sphinxdir}/js" %{buildroot}%{_pkgdocdir}/html/_static/js # Remove bundled license and readme (prefer license and doc macros). %{__rm} -f %{buildroot}%{_pkgdocdir}/LICENSE %{__rm} -f %{buildroot}%{_pkgdocdir}/README.rst %files %license LICENSE %doc README.rst %{?el7:README.EL7} %{_bindir}/ch-checkns %{_bindir}/ch-convert %{_bindir}/ch-fromhost %{_bindir}/ch-run %{_bindir}/ch-run-oci %{_mandir}/man1/ch-checkns.1* %{_mandir}/man1/ch-convert.1* %{_mandir}/man1/ch-fromhost.1* %{_mandir}/man1/ch-run.1* %{_mandir}/man1/ch-run-oci.1* %{_mandir}/man7/charliecloud.7* %{_mandir}/man7/ch-completion.bash.7* %{_prefix}/lib/%{name}/base.sh %{_prefix}/lib/%{name}/contributors.bash %{_prefix}/lib/%{name}/version.sh %{_prefix}/lib/%{name}/version.txt %files builder %{_bindir}/ch-image %{_mandir}/man1/ch-image.1* %{_prefix}/lib/%{name}/build.py %{_prefix}/lib/%{name}/build_cache.py %{_prefix}/lib/%{name}/charliecloud.py %{_prefix}/lib/%{name}/filesystem.py %{_prefix}/lib/%{name}/force.py %{_prefix}/lib/%{name}/image.py %{_prefix}/lib/%{name}/lark %{_prefix}/lib/%{name}/lark-1.1.9.dist-info %{_prefix}/lib/%{name}/misc.py %{_prefix}/lib/%{name}/pull.py %{_prefix}/lib/%{name}/push.py %{_prefix}/lib/%{name}/registry.py %{_prefix}/lib/%{name}/version.py %{?el7:%{_prefix}/lib/%{name}/__pycache__} %files doc %license LICENSE %{_pkgdocdir}/examples %{_pkgdocdir}/html %{?el7:%exclude %{_pkgdocdir}/examples/*/__pycache__} %files test %{_bindir}/ch-test %{_libexecdir}/%{name} %{_mandir}/man1/ch-test.1* %changelog * Thu Apr 16 2020 - @VERSION@-@RELEASE@ - Add new charliecloud package. charliecloud-0.37/packaging/fedora/el7-pkgdir.patch000066400000000000000000000007771457016721300222740ustar00rootroot00000000000000diff -ru charliecloud/bin/ch-test charliecloud-lib/bin/ch-test --- charliecloud/bin/ch-test 2020-04-07 12:19:37.054609706 -0600 +++ charliecloud-lib/bin/ch-test 2020-04-15 16:36:55.128831767 -0600 @@ -662,7 +662,7 @@ CHTEST_INSTALLED=yes CHTEST_GITWD= CHTEST_DIR=${ch_base}/libexec/charliecloud/test - CHTEST_EXAMPLES_DIR=${ch_base}/share/doc/charliecloud/examples + CHTEST_EXAMPLES_DIR=${ch_base}/share/doc/charliecloud-${ch_version}/examples else # build dir CHTEST_INSTALLED= charliecloud-0.37/packaging/fedora/printf.patch000066400000000000000000000007321457016721300216200ustar00rootroot00000000000000diff -ur charliecloud/bin/ch_misc.c charliecloud-patch/bin/ch_misc.c --- charliecloud/bin/ch_misc.c 2022-01-24 13:12:23.980046774 -0500 +++ charliecloud-patch/bin/ch_misc.c 2022-01-24 13:25:34.854133321 -0500 @@ -252,7 +252,7 @@ if (path == NULL) { T_ (where = strdup(line)); } else { - T_ (1 <= asprintf(&where, "%s:%lu", path, lineno)); + T_ (1 <= asprintf(&where, "%s:%zu", path, lineno)); } // Split line into variable name and value. charliecloud-0.37/packaging/fedora/upstream.spec000066400000000000000000000252211457016721300220110ustar00rootroot00000000000000# Charliecloud fedora package spec file # # Contributors: # Dave Love @loveshack # Michael Jennings @mej # Jordan Ogas @jogas # Reid Priedhorksy @reidpr # Don't try to compile python3 files with /usr/bin/python. %{?el7:%global __python %__python3} Name: charliecloud Version: 0.26 Release: 1%{?dist} Summary: Lightweight user-defined software stacks for high-performance computing License: ASL 2.0 URL: https://hpc.github.io/%{name}/ Source0: https://github.com/hpc/%{name}/releases/downloads/v%{version}/%{name}-%{version}.tar.gz BuildRequires: gcc rsync bash Requires: squashfuse squashfs-tools Patch1: el7-pkgdir.patch Patch2: printf.patch %description Charliecloud uses Linux user namespaces to run containers with no privileged operations or daemons and minimal configuration changes on center resources. This simple approach avoids most security risks while maintaining access to the performance and functionality already on offer. Container images can be built using Docker or anything else that can generate a standard Linux filesystem tree. For more information: https://hpc.github.io/charliecloud %package builder Summary: Charliecloud container image building tools License: ASL 2.0 and MIT BuildArch: noarch BuildRequires: python3-devel BuildRequires: python%{python3_pkgversion}-lark-parser BuildRequires: python%{python3_pkgversion}-requests Requires: %{name} Requires: python3 Requires: python%{python3_pkgversion}-lark-parser Requires: python%{python3_pkgversion}-requests Provides: bundled(python%{python3_pkgversion}-lark-parser) = 0.11.3 %description builder This package provides ch-image, Charliecloud's completely unprivileged container image manipulation tool. %package doc Summary: Charliecloud html documentation License: BSD and ASL 2.0 BuildArch: noarch Obsoletes: %{name}-doc < %{version}-%{release} BuildRequires: python%{python3_pkgversion}-sphinx BuildRequires: python%{python3_pkgversion}-sphinx_rtd_theme Requires: python%{python3_pkgversion}-sphinx_rtd_theme %description doc Html and man page documentation for %{name}. %package test Summary: Charliecloud test suite License: ASL 2.0 Requires: %{name} %{name}-builder /usr/bin/bats Obsoletes: %{name}-test < %{version}-%{release} %description test Test fixtures for %{name}. %prep %setup -q %if 0%{?el7} %patch1 -p1 %endif %patch2 -p1 %build # Use old inlining behavior, see: # https://github.com/hpc/charliecloud/issues/735 CFLAGS=${CFLAGS:-%optflags -fgnu89-inline}; export CFLAGS %configure --docdir=%{_pkgdocdir} \ --libdir=%{_prefix}/lib \ --with-python=/usr/bin/python3 \ %if 0%{?el7} --with-sphinx-build=%{_bindir}/sphinx-build-3.6 %else --with-sphinx-build=%{_bindir}/sphinx-build %endif %install %make_install cat > README.EL7 </etc/sysctl.d/51-userns.conf sysctl -p /etc/sysctl.d/51-userns.conf Note for versions below RHEL7.6, you will also need to enable user namespaces: grubby --args=namespace.unpriv_enable=1 --update-kernel=ALL reboot Please visit https://hpc.github.io/charliecloud/ for more information. EOF # Remove bundled sphinx bits. %{__rm} -rf %{buildroot}%{_pkgdocdir}/html/_static/css %{__rm} -rf %{buildroot}%{_pkgdocdir}/html/_static/fonts %{__rm} -rf %{buildroot}%{_pkgdocdir}/html/_static/js # Use Fedora package sphinx bits. sphinxdir=%{python3_sitelib}/sphinx_rtd_theme/static ln -s "${sphinxdir}/css" %{buildroot}%{_pkgdocdir}/html/_static/css ln -s "${sphinxdir}/fonts" %{buildroot}%{_pkgdocdir}/html/_static/fonts ln -s "${sphinxdir}/js" %{buildroot}%{_pkgdocdir}/html/_static/js # Remove bundled license and readme (prefer license and doc macros). %{__rm} -f %{buildroot}%{_pkgdocdir}/LICENSE %{__rm} -f %{buildroot}%{_pkgdocdir}/README.rst %files %license LICENSE %doc README.rst %{?el7:README.EL7} %{_bindir}/ch-checkns %{_bindir}/ch-convert %{_bindir}/ch-fromhost %{_bindir}/ch-run %{_bindir}/ch-run-oci %{_mandir}/man1/ch-checkns.1* %{_mandir}/man1/ch-convert.1* %{_mandir}/man1/ch-fromhost.1* %{_mandir}/man1/ch-run.1* %{_mandir}/man1/ch-run-oci.1* %{_mandir}/man7/charliecloud.7* %{_prefix}/lib/%{name}/base.sh %{_prefix}/lib/%{name}/contributors.bash %{_prefix}/lib/%{name}/version.sh %{_prefix}/lib/%{name}/version.txt %files builder %{_bindir}/ch-image %{_mandir}/man1/ch-image.1* %{_prefix}/lib/%{name}/build.py %{_prefix}/lib/%{name}/charliecloud.py %{_prefix}/lib/%{name}/fakeroot.py %{_prefix}/lib/%{name}/lark %{_prefix}/lib/%{name}/lark-1.1.8.dist-info %{_prefix}/lib/%{name}/lark-stubs %{_prefix}/lib/%{name}/misc.py %{_prefix}/lib/%{name}/pull.py %{_prefix}/lib/%{name}/push.py %{_prefix}/lib/%{name}/version.py %{?el7:%{_prefix}/lib/%{name}/__pycache__} %files doc %license LICENSE %{_pkgdocdir}/examples %{_pkgdocdir}/html %{?el7:%exclude %{_pkgdocdir}/examples/*/__pycache__} %files test %{_bindir}/ch-test %{_libexecdir}/%{name}/test %{_mandir}/man1/ch-test.1* %changelog * Mon Jan 24 2022 Jordan Ogas - 0.25-2 - Rebuilt for https://fedoraproject.org/wiki/Fedora_36_Mass_Rebuild * Mon Sep 20 2021 Jordan Ogas 0.24-12 - remove version numbers from Obsolete - remove Provides tag - replace package name with macro - tidy * Thu Jul 29 2021 Jordan Ogas 0.24-11 - move -builder to noarch - move examples back to -doc - add versions to obsoletes - use name macro * Wed Jul 28 2021 Jordan Ogas 0.24-10 - fix yet another typo; BuildRequires * Wed Jul 28 2021 Jordan Ogas 0.24-9 - add version to obsoletes * Wed Jul 28 2021 Jordan Ogas 0.24-8 - fix provides typo * Wed Jul 28 2021 Jordan Ogas 0.24-7 - add -common to obsoletes and provides * Wed Jul 28 2021 Jordan Ogas - 0.24-6 * revert to meta-package; separate builder to -builder * Wed Jul 21 2021 Fedora Release Engineering - 0.24-5 - Rebuilt for https://fedoraproject.org/wiki/Fedora_35_Mass_Rebuild * Mon Jul 19 2021 Jordan Ogas - 0.24-4 - fix epel7 python cache files * Mon Jul 19 2021 Jordan Ogas - 0.24-3 - Tidy, alphabatize files - Move builder exlusive python files out from -common - Move generic helper scripts to -common - Add requires runtime to -builders * Tue Jul 13 2021 Dave Love - 0.24-2 - Obsolete previous packge by -runtime, not -common * Wed Jun 30 2021 Dave Love - 0.24-1 - New version * Sun Apr 18 2021 Dave Love - 0.23-1 - New version - Split main package into runtime, builder, and common sub-packages - Require buildah and squashfs at run time - Use /lib, not /lib64 for noarch; drop lib64 patch - Don't BR squashfs-tools, squashfuse, buildah - Require squashfs-tools in -builders * Mon Mar 8 2021 Dave Love - 0.22-2 - Fix source0 path - Put man7 in base package * Tue Feb 9 2021 Dave Love - 0.22-1 - New version - update lib64.patch - add pull.py and push.py - (Build)Require python3-lark-parser, python3-requests * Wed Feb 3 2021 - 0.21-2 - Fix lib64.patch path for ch-image * Tue Jan 05 2021 - 0.21-1 - New version - Ship charlicloud.7 - Require fakeroot - Install fakeroot.py - Always ship patch1 - Get python3_sitelib defined - Move examples to -test and require sphinx_rtd_theme - Include __pycache__ on el7 - Use %%python3_pkgversion - BR python3, not /usr/bin/python3 - Fix comment capitalization and spacing * Tue Sep 22 2020 - 0.19-1 - Package build.py and misc.py - Remove unnecessary patch - New release * Mon Jul 27 2020 Fedora Release Engineering - 0.15-2 - Rebuilt for https://fedoraproject.org/wiki/Fedora_33_Mass_Rebuild * Thu Apr 16 2020 - 0.15-1 - Add test suite package - Update spec for autoconf - New release * Tue Jan 28 2020 Fedora Release Engineering - 0.10-2 - Rebuilt for https://fedoraproject.org/wiki/Fedora_32_Mass_Rebuild * Wed Sep 04 2019 - 0.10-1 - Patch doc-src/conf.py for epel - Fix doc-src/dev.rst - Fix libexec and doc install path for 0.10 changes - Tidy comments - New release * Thu Aug 22 2019 - 0.9.10-12 - Upate doc subpackage obsoletes * Mon Aug 19 2019 Dave love - 0.9.10-11 - Use canonical form for Source0 - Remove main package dependency from doc, and make it noarch * Fri Aug 02 2019 0.9.10-10 - Tidy comments; fix typ * Thu Jul 25 2019 0.9.10-9 - Use python site variable; fix doc file reference * Tue Jul 23 2019 0.9.10-8 - Remove bundled js, css, and font bits * Mon Jul 22 2019 0.9.10-6 - Temporarily remove test suite * Wed Jul 10 2019 0.9.10-5 - Revert test and example install path change - Update test readme * Wed Jul 3 2019 0.9.10-4 - Add doc package * Tue Jul 2 2019 0.9.10-3 - Tidy comments - Update source URL - Build html documentation; add rsync dependency - Add el7 conditionals for documentation - Remove libexecdir definition - Add test suite README.TEST * Wed May 15 2019 0.9.10-2 - Fix comment typo - Move test suite install path * Tue May 14 2019 0.9.10-1 - New version - Fix README.EL7 sysctl command instruction - Add pre-built html documentation - Fix python dependency - Remove temporary test-package readme - Fixed capitalization of change log messages * Tue Apr 30 2019 0.9.9-4 - Move global python declaration * Mon Apr 29 2019 0.9.9-3 - Match bin files with wildcard * Mon Apr 29 2019 0.9.9-2 - Update macro comment - Fix release tag history * Tue Apr 16 2019 0.9.9-1 - New version - Move temp readme creation to install segment - Fix spec file macro * Tue Apr 02 2019 0.9.8-2 - Remove python2 build option * Thu Mar 14 2019 0.9.8-1 - Add initial Fedora/EPEL package charliecloud-0.37/packaging/requirements.txt000066400000000000000000000007361457016721300213250ustar00rootroot00000000000000# This file contains the list of required python packages for the project in # requirements format. NOTE: It is provided on a “best effort” basis and is # not supported or maintained by the Charliecloud team. The source of truth # for all version dependencies is configure.ac. However, patches to keep it up # date are always welcome. # Users can install all our python dependencies simply by running: # > pip install -r requirements.txt lark-parser wheel requests>=2.6.0 charliecloud-0.37/test/000077500000000000000000000000001457016721300150665ustar00rootroot00000000000000charliecloud-0.37/test/.dockerignore000066400000000000000000000000631457016721300175410ustar00rootroot00000000000000# Nothing yet; used for testing ch-image warnings. charliecloud-0.37/test/Build.centos7xz000077500000000000000000000013741457016721300200230ustar00rootroot00000000000000#!/bin/bash # Download an xz-compressed CentOS 7 tarball. These are the base images for # the official CentOS Docker images. # # https://github.com/CentOS/sig-cloud-instance-images # # This GitHub repository is arranged with CentOS version and architecture in # different branches. We download the latest for a given architecture. # # To check what version is in a tarball (on any architecture): # # $ tar xf centos-7-${arch}-docker.tar.xz --to-stdout ./etc/centos-release # # ch-test-scope: standard # ch-test-builder-exclude: none set -ex #srcdir=$1 # unused tarball=${2}.tar.xz #workdir=$3 # unused wget -nv -O "$tarball" "https://github.com/CentOS/sig-cloud-instance-images/blob/CentOS-7-$(uname -m)/docker/centos-7-$(uname -m)-docker.tar.xz?raw=true" charliecloud-0.37/test/Build.docker_pull000077500000000000000000000011341457016721300203540ustar00rootroot00000000000000#!/bin/bash # ch-test-scope: quick # ch-test-builder-include: docker # ch-test-need-sudo # # Pull a docker image directly from Dockerhub and pack it into an image tarball. set -e #srcdir=$1 # unused tarball_gz=${2}.tar.gz workdir=$3 tag=docker_pull addr=alpine:3.17 img=$tag:latest cd "$workdir" sudo docker pull "$addr" sudo docker tag "$addr" "$tag" # FIXME: do we need a ch_version_docker equivalent? sudo docker tag "$tag" "$img" hash_=$(sudo docker images -q "$img" | sort -u) if [[ -z $hash_ ]]; then echo "no such image '$img'" exit 1 fi ch-convert -i docker "$tag" "$tarball_gz" charliecloud-0.37/test/Build.missing000077500000000000000000000001431457016721300175210ustar00rootroot00000000000000#!/bin/bash # ch-test-scope: quick # This image’s prerequisites can never be satisfied. exit 65 charliecloud-0.37/test/Dockerfile.argenv000066400000000000000000000011301457016721300203340ustar00rootroot00000000000000# Test how ARG and ENV variables flow around. This does not address syntax # quirks; for that see test “Dockerfile: syntax quirks” in # build/50_dockerfile.bats. Results are checked in both test “Dockerfile: ARG # and ENV values” in build/50_dockerfile.bats and multiple tests in # run/ch-run_misc.bats. The latter is why this is a separate Dockerfile # instead of embedded in a .bats file. # ch-test-scope: standard FROM alpine:3.17 ARG chse_arg1_df ARG chse_arg2_df=arg2 ARG chse_arg3_df="arg3 ${chse_arg2_df}" ENV chse_env1_df env1 ENV chse_env2_df="env2 ${chse_env1_df}" RUN env | sort charliecloud-0.37/test/Dockerfile.file-quirks000066400000000000000000000075261457016721300213240ustar00rootroot00000000000000# This Dockerfile is used to test that pull deals with quirky files, e.g. # replacement by different types (issues #819 and #825)`. Scope is “skip” # because we pull the image to test it; see test/build/50_pull.bats. # # To build and push: # # $ VERSION=$(date +%Y-%m-%d) # or other date as appropriate # $ sudo docker login # if needed # $ sudo docker build -t file-quirks -f Dockerfile.file-quirks . # $ sudo docker tag file-quirks:latest charliecloud/file-quirks:$VERSION # $ sudo docker images | fgrep file-quirks # $ sudo docker push charliecloud/file-quirks:$VERSION # # ch-test-scope: skip FROM alpine:3.17 WORKDIR /test ## Replace symlink with symlink. # Set up a symlink & targets. RUN echo target1 > ss_target1 \ && echo target2 > ss_target2 \ && ln -s ss_target1 ss_link # link and target should both contain “target1” RUN ls -l \ && for i in ss_*; do printf '%s : ' $i; cat $i; done # Overwrite it with a new symlink. RUN rm ss_link \ && ln -s ss_target2 ss_link # Now link should still be a symlink but contain “target2”. RUN ls -l \ && for i in ss_*; do printf '%s : ' $i; cat $i; done ## Replace symlink with regular file (issue #819). # Set up a symlink. RUN echo target > sf_target \ && ln -s sf_target sf_link # Link and target should both contain “target”. RUN ls -l \ && for i in sf_*; do printf '%s : ' $i; cat $i; done # Overwrite it with a regular file. RUN rm sf_link \ && echo regular > sf_link # Now link should be a regular file and contain “regular”. RUN ls -l \ && for i in sf_*; do printf '%s : ' $i; cat $i; done ## Replace regular file with symlink. # Set up two regular files. RUN echo regular > fs_link \ && echo target > fs_target # Link should be a regular file and contain “regular”. RUN ls -l \ && for i in fs_*; do printf '%s : ' $i; cat $i; done # Overwrite it with a symlink. RUN rm fs_link \ && ln -s fs_target fs_link # Now link should be a symlink; both should contain “target”. RUN ls -l \ && for i in fs_*; do printf '%s : ' $i; cat $i; done ## Replace symlink with directory. # Set up a symlink. RUN echo target > sd_target \ && ln -s sd_target sd_link # link and target should both contain “target”. RUN ls -l \ && for i in sd_*; do printf '%s : ' $i; cat $i; done # Overwrite it with a directory. RUN rm sd_link \ && mkdir sd_link # Now link should be a directory. RUN ls -l ## Replace directory with symlink. # I think this is what’s in image ppc64le.neo4j/2.3.5, as reported in issue # #825, but it doesn’t cause the same infinite recursion. # Set up a directory and a target. RUN mkdir ds_link \ && echo target > ds_target # It should be a directory. RUN ls -l # Overwrite it with a symlink. RUN rmdir ds_link \ && ln -s ds_target ds_link # Now link should be a symlink; both should contain “target”. RUN ls -l \ && for i in ds_*; do printf '%s : ' $i; cat $i; done ## Replace regular file with directory. # Set up a file. RUN echo regular > fd_member # It should be a file. RUN ls -l \ && for i in fd_*; do printf '%s : ' $i; cat $i; done # Overwrite it with a directory. RUN rm fd_member \ && mkdir fd_member # Now it should be a directory. RUN ls -l ## Replace directory with regular file. # Set up a directory. RUN mkdir df_member # It should be a directory. RUN ls -l # Overwrite it with a file. RUN rmdir df_member \ && echo regular > df_member # Now it should be a file. RUN ls -l \ && for i in df_*; do printf '%s : ' $i; cat $i; done ## Symlink with cycle (https://bugs.python.org/file37774). # Set up a symlink pointing to itself. RUN ln -s link_self link_self # List. RUN ls -l ## Broken symlinks (https://bugs.python.org/file37774). # Set up a symlink pointing to (1) a nonexistent file and (2) a directory that # only exists in the image. RUN ln -s doesnotexist link_b0rken \ && ln -s /test link_imageonly # List. RUN ls -l charliecloud-0.37/test/Dockerfile.metadata000066400000000000000000000021011457016721300206310ustar00rootroot00000000000000# This Dockerfile is used to test metadata pulling (issue #651). It includes # all the instructions that seemed like they ought to create metadata, even if # unsupported by ch-image. # # Scope is “skip” because we pull the image to test it; see # test/build/50_pull.bats. # # To build and push: # # $ VERSION=$(date +%Y-%m-%d) # or other date as appropriate # $ sudo docker login # if needed # $ sudo docker build -t charliecloud/metadata:$VERSION \ # -f Dockerfile.metadata . # $ sudo docker images | fgrep metadata # $ sudo docker push charliecloud/metadata:$VERSION # # ch-test-scope: skip FROM alpine:3.17 CMD ["bar", "baz"] ENTRYPOINT ["/bin/echo","foo"] ENV ch_foo=foo-ev ch_bar=bar-ev EXPOSE 867 5309/udp HEALTHCHECK --interval=60s --timeout=5s CMD ["/bin/true"] LABEL ch-foo=foo-label ch-bar=bar-label MAINTAINER charlie@example.com ONBUILD RUN echo hello RUN echo hello RUN ["/bin/echo", "world"] SHELL ["/bin/ash", "-c"] STOPSIGNAL SIGWINCH USER charlie:chargrp WORKDIR /mnt VOLUME /mnt/foo /mnt/bar /mnt/foo charliecloud-0.37/test/Dockerfile.ocimanifest000066400000000000000000000017551457016721300213700ustar00rootroot00000000000000# This Dockerfile is used to test image with an OCI manifest (issue #1184). # # WARNING: The manifest is produced by the build tool and is rather opaque. # Specifically, re-building the image might silently produce a different # manifest that also works, negating the value of this test. Building this # image with Podman 3.0.1 did trigger the above issue; Podman 3.4.0 very # likely has the same behavior. Bottom line, be very cautious about # re-building this image. One approach would be to comment out the content # types added by #1184 and see if the updated image still triggers the bug. # # Scope is “skip” because we pull it to test; see test/build/50_pull.bats. # ch-test-scope: skip # # To build and push: # # $ VERSION=$(date +%Y-%m-%d) # $ podman build -t charliecloud/ocimanifest:$VERSION \ # -f Dockerfile.ocimanifest . # $ podman images | fgrep ocimanifest # $ podman login # $ podman push charliecloud/ocimanifest:$VERSION # FROM alpine:3.17 RUN echo hello charliecloud-0.37/test/Dockerfile.quick000066400000000000000000000001741457016721300201750ustar00rootroot00000000000000# Minimal test image to exercise a Dockerfile build in quick scope. # ch-test-scope: quick FROM alpine:3.17 RUN apk add bc charliecloud-0.37/test/Makefile.am000066400000000000000000000074251457016721300171320ustar00rootroot00000000000000testdir = $(pkglibexecdir) # These test files require no special handling. testfiles = \ .dockerignore \ Dockerfile.argenv \ Dockerfile.quick \ approved-trailing-whitespace \ bucache/a.df \ bucache/a-fail.df \ bucache/argenv.df \ bucache/argenv-special.df \ bucache/argenv2.df \ bucache/b.df \ bucache/c.df \ bucache/copy.df \ bucache/difficult.df \ bucache/force.df \ bucache/from.df \ bucache/rsync.df \ build/10_sanity.bats \ build/40_pull.bats \ build/50_ch-image.bats \ build/50_dockerfile.bats \ build/50_localregistry.bats \ build/50_misc.bats \ build/99_cleanup.bats \ common.bash \ fixtures/empty-file \ fixtures/README \ make-auto.d/build.bats.in \ make-auto.d/build_custom.bats.in \ make-auto.d/builder_to_archive.bats.in \ make-auto.d/unpack.bats.in \ registry-config.yml \ run/build-rpms.bats \ run/ch-fromhost.bats \ run/ch-run_escalated.bats \ run/ch-run_isolation.bats \ run/ch-run_join.bats \ run/ch-run_misc.bats \ run/ch-run_uidgid.bats \ run_first.bats \ sotest/files_inferrable.txt \ sotest/libsotest.c \ sotest/sotest.c # Test files that should be executable. testfiles_exec = \ Build.centos7xz \ Build.docker_pull \ Build.missing \ docs-sane \ doctest \ doctest-auto \ force-auto \ make-perms-test \ old-storage \ order-py # Program and shared library used for testing shared library injection. It's # built according to the rules below. In principle, we could use libtool for # that, but I'm disinclined to add that in since it's one test program and # does not require any libtool portability. sobuilts = \ sotest/bin/sotest \ sotest/lib/libsotest.so.1.0 \ sotest/lib/libfabric/libsotest-fi.so \ sotest/libsotest.so \ sotest/libsotest.so.1 \ sotest/libsotest.so.1.0 \ sotest/sotest CLEANFILES = $(sobuilts) \ docs-sane \ doctest build/30_doctest-auto.bats \ force-auto force-auto.bats \ make-perms-test order-py if ENABLE_TEST nobase_test_DATA = $(testfiles) nobase_test_SCRIPTS = $(testfiles_exec) nobase_nodist_test_SCRIPTS = $(sobuilts) if ENABLE_CH_IMAGE # this means we have Python nobase_test_DATA += force-auto.bats force-auto.bats: force-auto ./$< > $@ nobase_test_DATA += build/30_doctest-auto.bats build/30_doctest-auto.bats: doctest-auto ./$< > $@ endif # See comment about symlinks in examples/Makefile.am. all-local: ln -fTs /tmp fixtures/symlink-to-tmp clean-local: rm -f fixtures/symlink-to-tmp install-data-hook: $(MKDIR_P) $(DESTDIR)$(testdir)/fixtures ln -fTs /tmp $(DESTDIR)$(testdir)/fixtures/symlink-to-tmp uninstall-hook: rm -f $(DESTDIR)$(testdir)/fixtures/symlink-to-tmp rmdir $(DESTDIR)$(testdir)/fixtures || true rmdir $$(find $(pkglibexecdir) -type d | sort -r) endif EXTRA_DIST = $(testfiles) \ $(testfiles_exec) \ docs-sane.py.in \ doctest.py.in \ force-auto.py.in \ make-perms-test.py.in \ order-py.py.in EXTRA_SCRIPTS = $(sobuilts) ## Python scripts - need text processing docs-sane doctest force-auto make-perms-test order-py: %: %.py.in rm -f $@ sed -E 's|%PYTHON_SHEBANG%|@PYTHON_SHEBANG@|' < $< > $@ chmod +rx,-w $@ # respects umask sotest/sotest: sotest/sotest.c sotest/libsotest.so.1.0 sotest/libsotest.so sotest/libsotest.so.1 $(CC) -o $@ $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) -L./sotest -lsotest $^ sotest/libsotest.so.1.0: sotest/libsotest.c $(CC) -o $@ $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) -shared -fPIC -Wl,-soname,libsotest.so.1 -lc $^ sotest/libsotest.so: sotest/libsotest.so.1.0 ln -fTs ./libsotest.so.1.0 $@ sotest/libsotest.so.1: sotest/libsotest.so.1.0 ln -fTs ./libsotest.so.1.0 $@ sotest/bin/sotest: sotest/sotest mkdir -p sotest/bin cp -a $^ $@ sotest/lib/libsotest.so.1.0: sotest/libsotest.so.1.0 mkdir -p sotest/lib cp -a $^ $@ sotest/lib/libfabric/libsotest-fi.so: sotest/libsotest.so.1.0 mkdir -p sotest/lib/libfabric cp -a $^ $@ charliecloud-0.37/test/approved-trailing-whitespace000066400000000000000000000012541457016721300225740ustar00rootroot00000000000000./misc/loc:60: filter rm_comments_in_strings " // ./test/build/50_dockerfile.bats:60:RUN true ./test/build/50_dockerfile.bats:63: ./test/build/50_dockerfile.bats:65: ./test/build/50_dockerfile.bats:66: ./test/build/50_dockerfile.bats:85:RUN echo test3\ ./test/build/50_dockerfile.bats:88:RUN echo test3\ ./test/build/50_dockerfile.bats:89:b\ ./test/build/50_dockerfile.bats:93:RUN echo test4 \ ./test/build/50_dockerfile.bats:96:RUN echo test4 \ ./test/build/50_dockerfile.bats:97:b \ ./test/build/50_dockerfile.bats:131: 4. RUN.S true ./test/build/50_dockerfile.bats:434:#ENV chse_1a value 1a ./test/build/50_dockerfile.bats:437:#ENV chse_1c=value\ 1c\ charliecloud-0.37/test/bucache/000077500000000000000000000000001457016721300164605ustar00rootroot00000000000000charliecloud-0.37/test/bucache/a-fail.df000066400000000000000000000000651457016721300201250ustar00rootroot00000000000000FROM alpine:3.17 RUN echo foo RUN false RUN echo bar charliecloud-0.37/test/bucache/a.df000066400000000000000000000000531457016721300172110ustar00rootroot00000000000000FROM alpine:3.17 RUN echo foo RUN echo bar charliecloud-0.37/test/bucache/argenv-special.df000066400000000000000000000001261457016721300216720ustar00rootroot00000000000000FROM alpine:3.17 ARG argA=vargA ARG SSH_AUTH_SOCK=sockA RUN echo $argA $SSH_AUTH_SOCK charliecloud-0.37/test/bucache/argenv.df000066400000000000000000000001661457016721300202600ustar00rootroot00000000000000FROM alpine:3.17 ARG argA=vargA ARG argB=vargB$argA ENV envA=venvA envB=venvB$argA RUN echo 1 $argA $argB $envA $envB charliecloud-0.37/test/bucache/argenv2.df000066400000000000000000000001661457016721300203420ustar00rootroot00000000000000FROM alpine:3.17 ARG argA=vargA ARG argB=vargB$argA ENV envA=venvA envB=venvB$argA RUN echo 2 $argA $argB $envA $envB charliecloud-0.37/test/bucache/b.df000066400000000000000000000000241457016721300172100ustar00rootroot00000000000000FROM a RUN echo baz charliecloud-0.37/test/bucache/c.df000066400000000000000000000000531457016721300172130ustar00rootroot00000000000000FROM alpine:3.17 RUN echo foo RUN echo qux charliecloud-0.37/test/bucache/copy.df000066400000000000000000000001371457016721300177460ustar00rootroot00000000000000# Context directory must be fixtures prepared in 55_cache.bats:COPY. FROM alpine:3.17 COPY * / charliecloud-0.37/test/bucache/difficult.df000066400000000000000000000012551457016721300207470ustar00rootroot00000000000000FROM alpine:3.17 WORKDIR /test # Directory and file with full permissions. RUN mkdir dir_all && chmod 4777 dir_all RUN touch dir_all/file_all && chmod 4777 dir_all/file_all # Directory and file with minimal permissions. RUN mkdir dir_min && chmod 700 dir_min RUN touch dir_min/file_min && chmod 400 dir_min/file_min # FIFO RUN mkfifo fifo_ # Empty directories RUN mkdir dir_empty RUN mkdir -p dir_empty_empty/dir_empty # Hard link RUN touch hard_target RUN ln hard_target hard_src # Symlink RUN touch soft_target RUN ln -s soft_target soft_src # Git repository RUN apk add git RUN git init gitrepo # Well-known last instruction so we can check if it’s cached. RUN echo last charliecloud-0.37/test/bucache/force.df000066400000000000000000000002461457016721300200730ustar00rootroot00000000000000# Use an almalinux:8 image because it can install some RPMs without --force. FROM almalinux:8 WORKDIR / RUN dnf install -y ed # doesn’t need --force WORKDIR /usr charliecloud-0.37/test/bucache/from.df000066400000000000000000000000211457016721300177270ustar00rootroot00000000000000FROM alpine:3.17 charliecloud-0.37/test/bucache/rsync.df000066400000000000000000000001421457016721300201260ustar00rootroot00000000000000# Context directory must be fixtures prepared in 55_cache.bats:RSYNC. FROM alpine:3.17 RSYNC /* / charliecloud-0.37/test/build/000077500000000000000000000000001457016721300161655ustar00rootroot00000000000000charliecloud-0.37/test/build/10_sanity.bats000066400000000000000000000164631457016721300206610ustar00rootroot00000000000000load ../common @test 'documentation seems sane' { scope standard if ( ! command -v sphinx-build > /dev/null 2>&1 ); then skip 'Sphinx is not installed' fi if [[ ! -d ../doc ]]; then skip 'documentation source code absent' fi if [[ ! -f ../doc/html/index.html || ! -f ../doc/man/ch-run.1 ]]; then skip 'documentation not built' fi (cd ../doc && make -j "$(getconf _NPROCESSORS_ONLN)") ./docs-sane } @test 'version number seems sane' { # This checks the form of the version number but not whether it’s # consistent with anything, because so far that level of strictness has # yielded hundreds of false positives but zero actual bugs. scope quick echo "version: ${ch_version}" re='^0\.[0-9]+(\.[0-9]+)?(~pre\+([A-Za-z0-9]+\.)?([0-9a-f]+(\.dirty)?)?)?$' [[ $ch_version =~ $re ]] } @test 'executables seem sane' { scope quick # Assume that everything in $ch_bin is ours if it starts with “ch-” and # either (1) is executable or (2) ends in “.c”. Demand satisfaction from # each. The latter is to catch cases when we haven't compiled everything; # if we have, the test makes duplicate demands, but that’s low cost. while IFS= read -r -d '' path; do path=${path%.c} filename=$(basename "$path") echo echo "$path" # --version run "$path" --version echo "$output" [[ $status -eq 0 ]] # --help: returns 0, says “Usage:” somewhere. run "$path" --help echo "$output" [[ $status -eq 0 ]] [[ $output = *'sage:'* ]] # Most, but not all, executables should print usage and exit # unsuccessfully when run without arguments. case $filename in ch-checkns) ;; *) run "$path" echo "$output" [[ $status -eq 1 ]] [[ $output = *'sage:'* ]] ;; esac # not setuid or setgid ls -l "$path" [[ ! -u $path ]] [[ ! -g $path ]] done < <( find "$ch_bin" -name 'ch-*' -a \( -executable -o -name '*.c' \) \ -print0 ) } @test 'lint shell scripts' { # ShellCheck excludes used below: # # SC1112 curly quotes in strings # SC2002 useless use of cat # SC2103 cd exit code unchecked (Bats checks for failure) # SC2164 same as SC2103 # # Excludes that work around issue #1625: # # SC2030 lost variable modification in subshell # SC2031 same as SC2030 scope standard arch_exclude ppc64le # no ShellCheck pre-built # Only do this test in build directory; the reasoning is that we don’t # alter the shell scripts during install enough to re-test, and it means # we only have to find everything in one path. if [[ $CHTEST_INSTALLED ]]; then skip 'only in build directory' fi # ShellCheck present? if ( ! command -v shellcheck >/dev/null 2>&1 ); then pedantic_fail 'no ShellCheck found' fi # ShellCheck minimum version? version=$(shellcheck --version | grep -E '^version:' | cut -d' ' -f2) needed=0.9.0 lesser=$(printf "%s\n%s\n" "$version" "$needed" | sort -V | head -1) echo "shellcheck: have ${version}, need ${needed}, lesser ${lesser}" if [[ $lesser != "$needed" ]]; then pedantic_fail 'shellcheck too old' fi # Shell scripts and libraries: appropriate extension or shebang. # For awk program, see: https://unix.stackexchange.com/a/66099 while IFS= read -r i; do echo "shellcheck: ${i}" shellcheck -x -P "$ch_lib" -e SC1112,SC2002 "$i" done < <( find "$ch_base" \ \( -name .git \ -o -name build-aux \) -prune \ -o \( -name '*.sh' -print \) \ -o \( -name '*.bash' -print \) \ -o \( -type f -exec awk '/^#!\/bin\/(ba)?sh/ {print FILENAME} {nextfile}' {} + \) ) # Bats scripts. Use sed to do several things: # # 1. Remove ch-test substitutions “%(foo)”, which confuse Bats. # # 2. Add the name of each command to a “true” argument to avoid warnings # about variables whos only reference is in that name. # # 3. Add extension “.bash” to “common” when needed. # # 4. Change “load” to “source”, which is close enough for this purpose. # # WARNING: If you change these expressions, ensure none of them changes # the number of lines, so line numbers (used in reporting) stay the same. while IFS= read -r i; do echo "shellcheck: ${i}" sed -E "$i" -e 's/%\(([a-zA-Z0-9_]+)\)/SUBST_\1/g' \ -e 's/^(@test (.+) \{)/\1 true \2;/g' \ -e 's/^load (.*)common$/load common.bash/g' \ -e 's/^load /source /g' \ | shellcheck -s bash -e SC1112,SC2002,SC2030,SC2031,SC2103,SC2164 \ - "$CHTEST_DIR"/common.bash done < <( find "$ch_base" -name '*.bats' -o -name '*.bats.in' ) } @test 'proxy variables' { scope standard # Proxy variables are a mess on UNIX. There are a lot them, and different # programs use them inconsistently. This test is based on the assumption # that if one of the proxy variables are set, then they all should be, in # order to prepare for diverse internet access at build time. # # Coordinate this test with common.bash:build_(). # # Note: ALL_PROXY and all_proxy aren’t currently included, because they # cause image builds to fail until Docker 1.13 # (https://github.com/docker/docker/pull/27412). v=' no_proxy http_proxy https_proxy' v+=$(echo "$v" | tr '[:lower:]' '[:upper:]') empty_ct=0 for i in $v; do if [[ -n ${!i} ]]; then echo "${i} is non-empty" for j in $v; do echo " $j=${!j}" if [[ -z ${!j} ]]; then (( ++empty_ct )) fi done break fi done [[ $empty_ct -eq 0 ]] } @test 'trailing whitespace' { scope standard [[ -z $CHTEST_INSTALLED ]] || skip 'build directory only' # Can’t use a here document to store the approved trailing-whitespace # lines because we’re grepping *this* file, so we’d have to add the here # document, which would expand the here document, etc. # # When updating CI to Ubuntu 22.04 (#1561), this test started failing because # the output of the “grep” started printing in a different order than what # was expected. Piping it into the “sort” ensures ordering consistency. The # command sorts first alphabetically by file path, then numerically by line # number. # # Note you can update the file by piping this “grep” and "sort" into it, # assuming there is no bogus trailing whitespace present. I have had trouble # with copy-and-paste removing the trailing whitespace. ../misc/grep -E '\s+$' \ | LC_ALL=C sort -t: -k1,1 -k2n,2 \ | diff -u approved-trailing-whitespace - } @test 'python object order' { scope standard status_all=0 for f in "$ch_lib"/*.py; do run ./order-py "$f" echo "$output" status_all=$((status_all+status)) done [[ $status_all -eq 0 ]] } charliecloud-0.37/test/build/40_pull.bats000066400000000000000000000413161457016721300203240ustar00rootroot00000000000000load ../common setup () { scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' } image_ref_parse () { # Try to parse image ref $1; expected output is provided on stdin and # expected exit code in $2. ref=$1 retcode_expected=$2 echo "--- parsing: ${ref}" set +e out=$(ch-image pull --parse-only "$ref" 2>&1) retcode=$? set -e echo "--- return code: ${retcode}" echo '--- output:' echo "$out" if [[ $retcode -ne "$retcode_expected" ]]; then echo "fail: return code differs from expected ${retcode_expected}" exit 1 fi diff -u -I '^hint: https://' -I '^trace:' - <(echo "$out") } @test 'image ref parsing' { # simplest cat <<'EOF' | image_ref_parse name 0 as string: name for filename: name fields: host None port None path [] name 'name' tag None digest None EOF # one-component path cat <<'EOF' | image_ref_parse path1/name 0 as string: path1/name for filename: path1%name fields: host None port None path ['path1'] name 'name' tag None digest None EOF # two-component path cat <<'EOF' | image_ref_parse path1/path2/name 0 as string: path1/path2/name for filename: path1%path2%name fields: host None port None path ['path1', 'path2'] name 'name' tag None digest None EOF # host with dot cat <<'EOF' | image_ref_parse example.com/name 0 as string: example.com/name for filename: example.com%name fields: host 'example.com' port None path [] name 'name' tag None digest None EOF # host with dot, with port cat <<'EOF' | image_ref_parse example.com:8080/name 0 as string: example.com:8080/name for filename: example.com+8080%name fields: host 'example.com' port 8080 path [] name 'name' tag None digest None EOF # host without dot, with port cat <<'EOF' | image_ref_parse examplecom:8080/name 0 as string: examplecom:8080/name for filename: examplecom+8080%name fields: host 'examplecom' port 8080 path [] name 'name' tag None digest None EOF # no path, tag cat <<'EOF' | image_ref_parse name:tag 0 as string: name:tag for filename: name+tag fields: host None port None path [] name 'name' tag 'tag' digest None EOF # no path, digest cat <<'EOF' | image_ref_parse name@sha256:feeddad 0 as string: name@sha256:feeddad for filename: name@sha256+feeddad fields: host None port None path [] name 'name' tag None digest 'feeddad' EOF # everything, tagged cat <<'EOF' | image_ref_parse example.com:8080/path1/path2/name:tag 0 as string: example.com:8080/path1/path2/name:tag for filename: example.com+8080%path1%path2%name+tag fields: host 'example.com' port 8080 path ['path1', 'path2'] name 'name' tag 'tag' digest None EOF # everything, tagged, filename component cat <<'EOF' | image_ref_parse example.com:8080%path1%path2%name:tag 0 as string: example.com:8080/path1/path2/name:tag for filename: example.com+8080%path1%path2%name+tag fields: host 'example.com' port 8080 path ['path1', 'path2'] name 'name' tag 'tag' digest None EOF # everything, digest cat <<'EOF' | image_ref_parse example.com:8080/path1/path2/name@sha256:feeddad 0 as string: example.com:8080/path1/path2/name@sha256:feeddad for filename: example.com+8080%path1%path2%name@sha256+feeddad fields: host 'example.com' port 8080 path ['path1', 'path2'] name 'name' tag None digest 'feeddad' EOF # errors # invalid character in image name cat <<'EOF' | image_ref_parse 'name*' 1 error: image ref syntax, char 5: name* hint: https://hpc.github.io/charliecloud/faq.html#how-do-i-specify-an-image-reference EOF # missing port number cat <<'EOF' | image_ref_parse 'example.com:/path1/name' 1 error: image ref syntax, char 13: example.com:/path1/name hint: https://hpc.github.io/charliecloud/faq.html#how-do-i-specify-an-image-reference EOF # path with leading slash cat <<'EOF' | image_ref_parse '/path1/name' 1 error: image ref syntax, char 1: /path1/name hint: https://hpc.github.io/charliecloud/faq.html#how-do-i-specify-an-image-reference EOF # path but no name cat <<'EOF' | image_ref_parse 'path1/' 1 error: image ref syntax, at end: path1/ hint: https://hpc.github.io/charliecloud/faq.html#how-do-i-specify-an-image-reference EOF # bad digest algorithm cat <<'EOF' | image_ref_parse 'name@sha512:feeddad' 1 error: image ref syntax, char 5: name@sha512:feeddad hint: https://hpc.github.io/charliecloud/faq.html#how-do-i-specify-an-image-reference EOF # both tag and digest cat <<'EOF' | image_ref_parse 'name:tag@sha512:feeddad' 1 error: image ref syntax, char 9: name:tag@sha512:feeddad hint: https://hpc.github.io/charliecloud/faq.html#how-do-i-specify-an-image-reference EOF } @test 'pull image with quirky files' { arch_exclude aarch64 # test image not available arch_exclude ppc64le # test image not available # Validate that layers replace symlinks correctly. See # test/Dockerfile.symlink and issues #819 & #825. CH_IMAGE_STORAGE=$BATS_TMPDIR/pull-quirks img="${CH_IMAGE_STORAGE}/img/charliecloud%file-quirks+2020-10-21" ch-image pull charliecloud/file-quirks:2020-10-21 ls -lh "${img}/test" output_expected=$(cat <<'EOF' regular file 'df_member' symbolic link 'ds_link' -> 'ds_target' regular file 'ds_target' directory 'fd_member' symbolic link 'fs_link' -> 'fs_target' regular file 'fs_target' symbolic link 'link_b0rken' -> 'doesnotexist' symbolic link 'link_imageonly' -> '../test' symbolic link 'link_self' -> 'link_self' directory 'sd_link' regular file 'sd_target' regular file 'sf_link' regular file 'sf_target' symbolic link 'ss_link' -> 'ss_target2' regular file 'ss_target1' regular file 'ss_target2' EOF ) cd "${img}/test" run stat -c '%-14F %N' -- * echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$output_expected") <(echo "$output") cd - } @test 'pull images with uncommon manifests' { arch_exclude aarch64 # test image not available arch_exclude ppc64le # test image not available if [[ -n $CH_REGY_DEFAULT_HOST ]]; then # Manifests seem to vary by registry; we need Docker Hub. skip 'default registry host set' fi storage="${BATS_TMPDIR}/tmp" cache=$storage/dlcache CH_IMAGE_STORAGE=$storage # OCI manifest; see issue #1184. img=charliecloud/ocimanifest:2021-10-12 ch-image pull "$img" # Manifest schema version one (v1); see issue #814. Use debian:squeeze # because 1) it always returns a v1 manifest schema (regardless of media # type specified), and 2) it isn't very large, thus keeps test time down. img=debian+squeeze ch-image pull "$img" grep -F '"schemaVersion": 1' "${cache}/${img}%skinny.manifest.json" rm -Rf --one-file-system "$storage" } @test 'pull from public repos' { if [[ -n $CH_REGY_DEFAULT_HOST ]]; then skip 'default registry host set' # avoid Docker Hub fi if [[ -z $CI ]]; then # Verify we can reach the public internet, except on CI, where we # insist this should work. ping -c3 8.8.8.8 || skip "can't ping 8.8.8.8" fi # These images are selected to be official-ish and small. My rough goal is # to keep them under 10MiB uncompressed, but this isn’t working great. It # may be worth our while to upload some small test images to these places. # Docker Hub: https://hub.docker.com/_/alpine ch-image pull registry-1.docker.io/library/alpine:3.17 # quay.io: https://quay.io/repository/quay/busybox ch-image pull quay.io/quay/busybox:latest # gitlab.com: https://gitlab.com/pages/hugo # FIXME: 50 MiB, try to do better; seems to be the slowest repo. ch-image pull registry.gitlab.com/pages/hugo:latest # Google Container Registry: # https://console.cloud.google.com/gcr/images/google-containers/GLOBAL # FIXME: “latest” tags do not work, but they do in Docker (issue #896) # FIXME: arch-aware pull does not work either (issue #1100) ch-image pull --arch=yolo gcr.io/google-containers/busybox:1.27 # nVidia NGC: https://ngc.nvidia.com # FIXME: 96 MiB unpacked; also kind of slow # Note: Can’t pull this image with LC_ALL=C under Python 3.6 (issue #970). ch-image pull nvcr.io/hpc/foldingathome/fah-gpu:7.6.21 # Red Hat registry: https://catalog.redhat.com/software/containers/explore # FIXME: 77 MiB unpacked, should find a smaller public image ch-image pull registry.access.redhat.com/ubi7/ubi-minimal:latest # Microsoft Container Registry: # https://github.com/microsoft/containerregistry ch-image pull mcr.microsoft.com/mcr/hello-world:latest # Things not here (yet?): # # 1. Harbor (issue #899): Has a demo repo (https://demo.goharbor.io) that # you can make an account on, but I couldn’t find a public repo, and # the demo repo gets reset every two days. # # 2. Docker registry container (https://hub.docker.com/_/registry): Would # need to set up an instance. # # 3. Amazon public repo (issue #901, # https://aws.amazon.com/blogs/containers/advice-for-customers-dealing-with-docker-hub-rate-limits-and-a-coming-soon-announcement/): # Does not exist yet; coming “within weeks” of 2020-11-02. # # 4. Microsoft Azure registry [1] (issue #902): I could not find any # public images. It seems that public pull is “currently a preview # feature” as of 2020-11-06 [2]. # # [1]: https://azure.microsoft.com/en-us/services/container-registry # [2]: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-faq#how-do-i-enable-anonymous-pull-access # # 5. JFrog / Artifactory (https://jfrog.com/container-registry/): Could # not find any public registry. } @test 'pull image with metadata' { arch_exclude aarch64 # test image not available arch_exclude ppc64le # test image not available tag=2021-01-15 name=charliecloud/metadata:$tag img=$CH_IMAGE_STORAGE/img/charliecloud%metadata+$tag ch-image pull "$name" # Volume mount points exist? ls -lh "${img}/mnt" test -d "${img}/mnt/foo" test -d "${img}/mnt/bar" # /ch/environment contents diff -u - "${img}/ch/environment" <<'EOF' PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ch_bar=bar-ev ch_foo=foo-ev EOF # /ch/metadata.json contents diff -u -I '^.*"created":.*,$' - "${img}/ch/metadata.json" <<'EOF' { "arch": "amd64", "arg": { "FAKEROOTDONTTRYCHOWN": "1", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "TAR_OPTIONS": "--no-same-owner" }, "cwd": "/mnt", "env": { "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "ch_bar": "bar-ev", "ch_foo": "foo-ev" }, "history": [ { "created": "2020-04-24T01:05:35.458457398Z", "created_by": "/bin/sh -c #(nop) ADD file:a0afd0b0db7f9ee9496186ead087ec00edd1386ea8c018557d15720053f7308e in / " }, { "created": "2020-04-24T01:05:35.807646609Z", "created_by": "/bin/sh -c #(nop) CMD [\"/bin/sh\"]", "empty_layer": true }, { "created": "2020-12-10T18:26:16.62246537Z", "created_by": "/bin/sh -c #(nop) CMD [\"true\"]", "empty_layer": true }, { "created": "2021-01-08T00:57:33.450706788Z", "created_by": "/bin/sh -c #(nop) CMD [\"bar\" \"baz\"]", "empty_layer": true }, { "created": "2021-01-08T00:57:33.675120552Z", "created_by": "/bin/sh -c #(nop) ENTRYPOINT [\"/bin/echo\" \"foo\"]", "empty_layer": true }, { "created": "2021-01-16T00:12:10.147564398Z", "created_by": "/bin/sh -c #(nop) ENV ch_foo=foo-ev ch_bar=bar-ev", "empty_layer": true }, { "created": "2021-01-16T00:12:10.340268945Z", "created_by": "/bin/sh -c #(nop) EXPOSE 5309/udp 867", "empty_layer": true }, { "created": "2021-01-16T00:12:10.590808975Z", "created_by": "/bin/sh -c #(nop) HEALTHCHECK &{[\"CMD\" \"/bin/true\"] \"1m0s\" \"5s\" \"0s\" '\\x00'}", "empty_layer": true }, { "created": "2021-01-16T00:12:10.749205247Z", "created_by": "/bin/sh -c #(nop) LABEL ch_foo=foo-label ch_bar=bar-label", "empty_layer": true }, { "author": "charlie@example.com", "created": "2021-01-16T00:12:10.919558634Z", "created_by": "/bin/sh -c #(nop) MAINTAINER charlie@example.com", "empty_layer": true }, { "author": "charlie@example.com", "created": "2021-01-16T00:12:11.080200702Z", "created_by": "/bin/sh -c #(nop) ONBUILD RUN echo hello", "empty_layer": true }, { "author": "charlie@example.com", "created": "2021-01-16T00:12:11.900757214Z", "created_by": "/bin/sh -c echo hello", "empty_layer": true }, { "author": "charlie@example.com", "created": "2021-01-16T00:12:12.868439691Z", "created_by": "/bin/echo world", "empty_layer": true }, { "author": "charlie@example.com", "created": "2021-01-16T00:12:13.055783024Z", "created_by": "/bin/ash -c #(nop) SHELL [/bin/ash -c]", "empty_layer": true }, { "author": "charlie@example.com", "created": "2021-01-16T00:12:13.473299627Z", "created_by": "/bin/ash -c #(nop) STOPSIGNAL SIGWINCH", "empty_layer": true }, { "author": "charlie@example.com", "created": "2021-01-16T00:12:13.644005108Z", "created_by": "/bin/ash -c #(nop) USER charlie:chargrp", "empty_layer": true }, { "author": "charlie@example.com", "created": "2021-01-16T00:12:13.83546594Z", "created_by": "/bin/ash -c #(nop) WORKDIR /mnt", "empty_layer": true }, { "author": "charlie@example.com", "created": "2021-01-16T00:12:14.042791834Z", "created_by": "/bin/ash -c #(nop) VOLUME [/mnt/foo /mnt/bar /mnt/foo]", "empty_layer": true } ], "labels": { "ch_bar": "bar-label", "ch_foo": "foo-label" }, "shell": [ "/bin/ash", "-c" ], "volumes": [ "/mnt/bar", "/mnt/foo" ] } EOF } @test 'pull by arch' { # Has fat manifest; requested arch exists. There’s not much simple to look # for in the output, so just see if it works. NOTE: As a temporary fix for # some test suite problems, I’m changing all instances of alpine:latest here # to alpine:3.15. We really need a more permanent solution for this (see #1485) ch-image --arch=yolo pull alpine:3.15 ch-image --arch=host pull alpine:3.15 ch-image --arch=amd64 pull alpine:3.15 ch-image --arch=arm64/v8 pull alpine:3.15 # Has fat manifest, but requested arch does not exist. run ch-image --arch=doesnotexist pull alpine:3.15 echo "$output" [[ $status -eq 1 ]] [[ $output = *'requested arch unavailable:'*'available:'* ]] # Delete it so we don’t try to use a non-matching arch for other testing. # FIXME: After #1485 is closed, revert to alpine:latest or alpine:3.17 and # delete cache along with image. ch-image delete alpine:3.15 || true # No fat manifest. ch-image --arch=yolo pull charliecloud/metadata:2021-01-15 ch-image --arch=amd64 pull charliecloud/metadata:2021-01-15 if [[ $(uname -m) == 'x86_64' ]]; then ch-image --arch=host pull charliecloud/metadata:2021-01-15 run ch-image --arch=arm64/v8 pull charliecloud/metadata:2021-01-15 echo "$output" [[ $status -eq 1 ]] [[ $output = *'image is architecture-unaware'*'consider --arch=yolo'* ]] fi } @test 'pull images that do not exist' { if [[ -n $CH_REGY_DEFAULT_HOST ]]; then skip 'default registry host set' # errors are Docker Hub specific fi # name does not exist remotely, in library run ch-image pull doesnotexist:latest echo "$output" [[ $status -eq 1 ]] [[ $output = *'registry-1.docker.io:443/library/doesnotexist:latest'* ]] # tag does not exist remotely, in library run ch-image pull alpine:doesnotexist echo "$output" [[ $status -eq 1 ]] [[ $output = *'registry-1.docker.io:443/library/alpine:doesnotexist'* ]] # name does not exist remotely, not in library run ch-image pull charliecloud/doesnotexist:latest echo "$output" [[ $status -eq 1 ]] [[ $output = *'registry-1.docker.io:443/charliecloud/doesnotexist:latest'* ]] # tag does not exist remotely, not in library run ch-image pull charliecloud/metadata:doesnotexist echo "$output" [[ $status -eq 1 ]] [[ $output = *'registry-1.docker.io:443/charliecloud/metadata:doesnotexist'* ]] } charliecloud-0.37/test/build/50_ch-image.bats000066400000000000000000000673661457016721300210400ustar00rootroot00000000000000load ../common setup () { scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' } tmpimg_build () { for img in "$@"; do ch-image build -t "$img" -f - . << 'EOF' FROM alpine:3.17 EOF run ch-image list [[ $status -eq 0 ]] [[ $output == *"$img"* ]] done } @test 'ch-image common options' { # no common options run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output != *'verbose level'* ]] # before only run ch-image -vv list echo "$output" [[ $status -eq 0 ]] [[ $output = *'verbose level: 2'* ]] # after only run ch-image list -vv echo "$output" [[ $status -eq 0 ]] [[ $output = *'verbose level: 2'* ]] # before and after; after wins run ch-image -vv list -v echo "$output" [[ $status -eq 0 ]] [[ $output = *'verbose level: 1'* ]] # unset debug in preparation for “--quiet” tests unset CH_IMAGE_DEBUG # test gestalt logging run ch-image gestalt logging echo "$output" [[ $status -eq 0 ]] [[ $output = *"info"* ]] [[ $output = *'warning: warning'* ]] [[ $output = *'error: error'* ]] # quiet level 1 run ch-image gestalt -q logging echo "$output" [[ $status -eq 0 ]] [[ $output != *"info"* ]] [[ $output = *'warning: warning'* ]] [[ $output = *'error: error'* ]] # quiet level 2 run ch-image build --rebuild -t tmpimg -qq -f - . << 'EOF' FROM alpine:3.17 RUN echo 'this is stdout' RUN echo 'this is stderr' 1>&2 EOF echo "$output" [[ $status -eq 0 ]] [[ $output != *'Dependencies resolved.'* ]] [[ $output != *'this is stdout'* ]] [[ $output = *'this is stderr'* ]] [[ $output != *'grown in 4 instructions: tmpimg'* ]] # quiet level 3 run ch-image gestalt logging -qqq echo "$output" [[ $status -eq 0 ]] [[ $output != *'info'* ]] [[ $output != *'warning: warning'* ]] [[ $output = *'error: error'* ]] # failure at quiet level 3 run ch-image gestalt logging -qqq --fail echo "$output" [[ $status -eq 1 ]] [[ $output != *'info'* ]] [[ $output != *'warning: warning'* ]] [[ $output = *'error: the program failed inexplicably'* ]] } @test 'ch-image delete' { # Verify image doesn’t exist. run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output != *"delete/test"* ]] # Build image. It’s called called delete/test to check ref parsing with # slash present. ch-image build -t delete/test -f - . << 'EOF' FROM alpine:3.17 FROM alpine:3.17 FROM alpine:3.17 FROM alpine:3.17 EOF run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output = *"delete/test"* ]] [[ $output = *"delete/test_stage0"* ]] [[ $output = *"delete/test_stage1"* ]] [[ $output = *"delete/test_stage2"* ]] # Delete image. ch-image delete delete/test run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output != *"delete/test"* ]] [[ $output != *"delete/test_stage0"* ]] [[ $output != *"delete/test_stage1"* ]] [[ $output != *"delete/test_stage2"* ]] tmpimg_build tmpimg1 tmpimg2 ch-image delete tmpimg1 tmpimg2 run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output != *"tmpimg1"* ]] [[ $output != *"tmpimg2"* ]] # Delete list of images with invalid image tmpimg_build tmpimg1 tmpimg2 run ch-image delete tmpimg1 doesnotexist tmpimg2 echo "$output" [[ $status -eq 1 ]] [[ $output == *"deleting image: tmpimg1"* ]] [[ $output == *"error: no matching image, can"?"t delete: doesnotexist"* ]] [[ $output == *"deleting image: tmpimg2"* ]] [[ $output == *"error: unable to delete 1 invalid image(s)"* ]] # Delete list of images with multiple invalid images tmpimg_build tmpimg1 tmpimg2 run ch-image delete tmpimg1 doesnotexist tmpimg2 doesnotexist2 echo "$output" [[ $status -eq 1 ]] [[ $output == *"deleting image: tmpimg1"* ]] [[ $output == *"error: no matching image, can"?"t delete: doesnotexist"* ]] [[ $output == *"deleting image: tmpimg2"* ]] [[ $output == *"error: no matching image, can"?"t delete: doesnotexist2"* ]] [[ $output == *"error: unable to delete 2 invalid image(s)"* ]] } @test 'broken image delete' { # Verify image doesn’t exist. run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output != *"deletetest"* ]] # Build image. ch-image build -t deletetest -f - . << 'EOF' FROM alpine:3.17 EOF run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output = *"deletetest"* ]] # Break image. rmdir "$CH_IMAGE_STORAGE"/img/deletetest/dev # Delete image. ch-image delete deletetest run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output != *"deletetest"* ]] } @test 'broken image overwrite' { # Build image. ch-image build -t tmpimg -f - . << 'EOF' FROM alpine:3.17 EOF # Break image. rmdir "$CH_IMAGE_STORAGE"/img/tmpimg/dev # Rebuild image. ch-image build -t tmpimg -f - . << 'EOF' FROM alpine:3.17 EOF run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output = *"tmpimg"* ]] } @test 'ch-image import' { # Note: We don’t test importing a real image because (1) when this is run # during the build phase there aren’t any unpacked images and (2) I can’t # think of a way import could fail that would be real image-specific. ## Test image (not runnable) fixtures=${BATS_TMPDIR}/import rm -Rfv --one-file-system "$fixtures" mkdir "$fixtures" \ "${fixtures}/empty" \ "${fixtures}/nonempty" \ "${fixtures}/nonempty/ch" \ "${fixtures}/nonempty/bin" (cd "$fixtures" && ln -s nonempty nelink) touch "${fixtures}/nonempty/bin/foo" cat <<'EOF' > "${fixtures}/nonempty/ch/metadata.json" { "arch": "corn", "cwd": "/", "env": {}, "labels": {}, "shell": [ "/bin/sh", "-c" ], "volumes": [] } EOF ls -lhR "$fixtures" ## Tarballs # tarbomb (cd "${fixtures}/nonempty" && tar czvf ../bomb.tar.gz .) run ch-image import -v "${fixtures}/bomb.tar.gz" imptest echo "$output" [[ $status -eq 0 ]] [[ $output = *"importing: ${fixtures}/bomb.tar.gz"* ]] [[ $output = *'conversion to tarbomb not needed'* ]] [[ -f "${CH_IMAGE_STORAGE}/img/imptest/bin/foo" ]] grep -F '"arch": "corn"' "${CH_IMAGE_STORAGE}/img/imptest/ch/metadata.json" ch-image delete imptest # non-tarbomb (cd "$fixtures" && tar czvf standard.tar.gz nonempty) run ch-image import -v "${fixtures}/standard.tar.gz" imptest echo "$output" [[ $status -eq 0 ]] [[ $output = *"importing: ${fixtures}/standard.tar.gz"* ]] [[ $output = *'converting to tarbomb'* ]] [[ -f "${CH_IMAGE_STORAGE}/img/imptest/bin/foo" ]] grep -F '"arch": "corn"' "${CH_IMAGE_STORAGE}/img/imptest/ch/metadata.json" ch-image delete imptest # non-tarbomb, but enclosing directory is a standard dir (cd "${fixtures}/nonempty" && tar czvf ../tricky.tar.gz bin) run ch-image import -v "${fixtures}/tricky.tar.gz" imptest echo "$output" [[ $status -eq 0 ]] [[ $output = *"importing: ${fixtures}/tricky.tar.gz"* ]] [[ $output = *'conversion to tarbomb not needed'* ]] [[ -f "${CH_IMAGE_STORAGE}/img/imptest/bin/foo" ]] ch-image delete imptest # empty, uncompressed tarfile (cd "${fixtures}" && tar cvf empty.tar empty) run ch-image import -v "${fixtures}/empty.tar" imptest echo "$output" [[ $status -eq 0 ]] [[ $output = *"importing: ${fixtures}/empty.tar"* ]] [[ $output = *'converting to tarbomb'* ]] [[ $output = *'warning: no metadata to load; using defaults'* ]] ch-image delete imptest ## Directories # non-empty directory run ch-image import -v "${fixtures}/nonempty" imptest echo "$output" [[ $status -eq 0 ]] [[ $output = *"importing: ${fixtures}/nonempty"* ]] [[ $output = *"copying image: ${fixtures}/nonempty -> ${CH_IMAGE_STORAGE}/img/imptest"* ]] [[ -f "${CH_IMAGE_STORAGE}/img/imptest/bin/foo" ]] grep -F '"arch": "corn"' "${CH_IMAGE_STORAGE}/img/imptest/ch/metadata.json" ch-image delete imptest # empty directory run ch-image import -v "${fixtures}/empty" imptest echo "$output" [[ $status -eq 0 ]] [[ $output = *"importing: ${fixtures}/empty"* ]] [[ $output = *"copying image: ${fixtures}/empty -> ${CH_IMAGE_STORAGE}/img/imptest"* ]] [[ $output = *'warning: no metadata to load; using defaults'* ]] ch-image delete imptest # symlink to directory run ch-image import -v "${fixtures}/nelink" imptest echo "$output" [[ $status -eq 0 ]] [[ $output = *"importing: ${fixtures}/nelink"* ]] [[ $output = *"copying image: ${fixtures}/nelink -> ${CH_IMAGE_STORAGE}/img/imptest"* ]] [[ -f "${CH_IMAGE_STORAGE}/img/imptest/bin/foo" ]] grep -F '"arch": "corn"' "${CH_IMAGE_STORAGE}/img/imptest/ch/metadata.json" ch-image delete imptest ## Errors # input does not exist run ch-image import -v /doesnotexist imptest echo "$output" [[ $status -eq 1 ]] [[ $output = *"error: can"?"t copy: not found: /doesnotexist"* ]] # invalid destination reference run ch-image import -v "${fixtures}/empty" 'badchar*' echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: image ref syntax, char 8: badchar*'* ]] # non-empty file that’s not a tarball run ch-image import -v "${fixtures}/nonempty/ch/metadata.json" imptest echo "$output" [[ $status -eq 1 ]] [[ $output = *"error: cannot open: ${fixtures}/nonempty/ch/metadata.json"* ]] ## Clean up [[ ! -e "${CH_IMAGE_STORAGE}/img/imptest" ]] rm -Rfv --one-file-system "$fixtures" } @test 'ch-image list' { # list all images run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output = *"alpine:3.17"* ]] # name does not exist remotely, in library run ch-image list doesnotexist:latest echo "$output" [[ $status -eq 0 ]] [[ $output = *'in local storage: no'* ]] [[ $output = *'available remotely: no'* ]] [[ $output = *'remote arch-aware: n/a'* ]] [[ $output = *'archs available: n/a'* ]] # tag does not exist remotely, in library run ch-image list alpine:doesnotexist echo "$output" [[ $status -eq 0 ]] [[ $output = *'in local storage: no'* ]] [[ $output = *'available remotely: no'* ]] [[ $output = *'remote arch-aware: n/a'* ]] [[ $output = *'archs available: n/a'* ]] # name does not exist remotely, not in library run ch-image list charliecloud/doesnotexist:latest echo "$output" [[ $status -eq 0 ]] [[ $output = *'in local storage: no'* ]] [[ $output = *'available remotely: no'* ]] [[ $output = *'remote arch-aware: n/a'* ]] [[ $output = *'archs available: n/a'* ]] # tag does not exist remotely, not in library run ch-image list charliecloud/metadata:doesnotexist echo "$output" [[ $status -eq 0 ]] [[ $output = *'in local storage: no'* ]] [[ $output = *'available remotely: no'* ]] [[ $output = *'remote arch-aware: n/a'* ]] [[ $output = *'archs available: n/a'* ]] # in storage, does not exist remotely run ch-image list argenv echo "$output" [[ $status -eq 0 ]] [[ $output = *'in local storage: yes'* ]] [[ $output = *'available remotely: no'* ]] [[ $output = *'remote arch-aware: n/a'* ]] [[ $output = *'archs available: n/a'* ]] # not in storage, exists remotely, fat manifest exists run ch-image list debian:buster-slim echo "$output" [[ $status -eq 0 ]] [[ $output = *'in local storage: no'* ]] [[ $output = *'available remotely: yes'* ]] [[ $output = *'remote arch-aware: yes'* ]] [[ $output = *'archs available:'*'386'*'amd64'*'arm/v7'*'arm64/v8'* ]] # in storage, exists remotely, no fat manifest run ch-image list charliecloud/metadata:2021-01-15 echo "$output" [[ $status -eq 0 ]] [[ $output = *'in local storage: yes'* ]] [[ $output = *'available remotely: yes'* ]] [[ $output = *'remote arch-aware: no'* ]] [[ $output = *'archs available: unknown'* ]] # exists remotely, fat manifest exists, no Linux architectures run ch-image list mcr.microsoft.com/windows:20H2 echo "$output" [[ $status -eq 0 ]] [[ $output = *'in local storage: no'* ]] [[ $output = *'available remotely: yes'* ]] [[ $output = *'remote arch-aware: yes'* ]] [[ $output = *'warning: no valid architectures found'* ]] # scratch is weird and tells lies run ch-image list scratch echo "$output" [[ $status -eq 0 ]] [[ $output = *'available remotely: yes'* ]] [[ $output = *'remote arch-aware: yes'* ]] } @test 'ch-image reset' { CH_IMAGE_STORAGE="$BATS_TMPDIR"/sd-reset # Ensure our test storage dir doesn’t exist yet. [[ -e $CH_IMAGE_STORAGE ]] && rm -Rf --one-file-system "$CH_IMAGE_STORAGE" # Put an image innit. ch-image pull alpine:3.17 ls "$CH_IMAGE_STORAGE" # List images; should be only the one we just pulled. run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output = "alpine:3.17" ]] # Reset. ch-image reset # Image storage directory should be empty now. expected=$(cat <<'EOF' .: bucache bularge dlcache img lock ulcache version ./bucache: ./bularge: ./dlcache: ./img: ./ulcache: EOF ) actual=$(cd "$CH_IMAGE_STORAGE" && ls -1R) diff -u <(echo "$expected") <(echo "$actual") # Remove storage directory. rm -Rf --one-file-system "$CH_IMAGE_STORAGE" # Reset again; should error. run ch-image reset echo "$output" [[ $status -eq 1 ]] [[ $output = *"$CH_IMAGE_STORAGE not a builder storage"* ]] } @test 'ch-image storage-path' { run ch-image gestalt storage-path echo "$output" [[ $status -eq 0 ]] [[ $output = /* ]] # absolute path [[ $CH_IMAGE_STORAGE && $output = "$CH_IMAGE_STORAGE" ]] # what we set } @test 'ch-image build --bind' { ch-image --no-cache build -t tmpimg -f - \ -b "${PWD}/fixtures" -b ./fixtures:/mnt/0 . < file_ % && mkdir dir_empty % && mkdir dir_nonempty % && mkfifo fifo_ % EOF ) ch-image build -t tmpimg - < "$CH_IMAGE_STORAGE"/version cat "$CH_IMAGE_STORAGE"/version # Version mismatch; fail. run ch-image -v list echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: incompatible storage directory v-1'* ]] # Reset. run ch-image reset echo "$output" [[ $status -eq 0 ]] [[ $output = *"initializing storage directory: v${v_current} ${CH_IMAGE_STORAGE}"* ]] # Version matches again; success. run ch-image -v list echo "$output" [[ $status -eq 0 ]] [[ $output = *"found storage dir v${v_current}: ${CH_IMAGE_STORAGE}"* ]] } @test 'ch-run --unsafe' { my_storage=${BATS_TMPDIR}/unsafe # Default storage location. if [[ $CH_IMAGE_STORAGE = /var/tmp/$USER.ch ]]; then sold=$CH_IMAGE_STORAGE unset CH_IMAGE_STORAGE [[ ! -e .%3.17 ]] ch-run --unsafe alpine:3.17 -- /bin/true CH_IMAGE_STORAGE=$sold fi # Rest of test uses custom storage path. rm -rf "$my_storage" mkdir -p "$my_storage"/img ch-convert -i ch-image -o dir alpine:3.17 "${my_storage}/img/alpine+3.17" unset CH_IMAGE_STORAGE # Specified on command line. ch-run --unsafe -s "$my_storage" alpine:3.17 -- /bin/true # Specified with environment variable. export CH_IMAGE_STORAGE=$my_storage # Basic environment-variable specified. ch-run --unsafe alpine:3.17 -- /bin/true } @test 'ch-run storage errors' { run ch-run -v -w alpine:3.17 -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: --write invalid when running by name'* ]] run ch-run -v "$CH_IMAGE_STORAGE"/img/alpine+3.17 -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"error: can't run directory images from storage (hint: run by name)"* ]] run ch-run -v -s /doesnotexist alpine:3.17 -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *'warning: storage directory not found: /doesnotexist'* ]] [[ $output = *"error: can't stat: alpine:3.17: No such file or directory"* ]] } @test 'ch-run image in both storage and cwd' { cd "$BATS_TMPDIR" # Set up a fixure image in $CWD that causes a collision with the named # image, and that’s missing /bin/true so it pukes if we try to run it. # That is, in both cases, we want run-by-name to win. rm -rf ./alpine+3.17 ch-convert -i ch-image -o dir alpine:3.17 ./alpine+3.17 rm ./alpine+3.17/bin/true # Default. ch-run alpine:3.17 -- /bin/true # With --unsafe. ch-run --unsafe alpine:3.17 -- /bin/true } @test "IMPORT cache miss" { # issue #1638 [[ $CH_IMAGE_CACHE = enabled ]] || skip 'build cache enabled only' ch-convert alpine:3.17 "$BATS_TMPDIR"/alpine317.tar.gz ch-convert alpine:3.16 "$BATS_TMPDIR"/alpine316.tar.gz export CH_IMAGE_STORAGE=$BATS_TMPDIR/import_1638 rm -Rf --one-file-system "$CH_IMAGE_STORAGE" ch-image import "$BATS_TMPDIR"/alpine317.tar.gz alpine:3.17 ch-image import "$BATS_TMPDIR"/alpine316.tar.gz alpine:3.16 df1=$BATS_TMPDIR/import_1638.1.df cat > "$df1" <<'EOF' FROM alpine:3.17 RUN true EOF df2=$BATS_TMPDIR/import_1638.2.df cat > "$df2" <<'EOF' FROM alpine:3.16 RUN true EOF echo echo '*** Build once: miss' run ch-image build -t tmpimg -f "$df1" "$BATS_TMPDIR" echo "$output" [[ $status -eq 0 ]] [[ $output = *'1* FROM alpine:3.17'* ]] [[ $output = *'2. RUN.S true'* ]] echo echo '*** Build again: hit' run ch-image build -t tmpimg -f "$df1" "$BATS_TMPDIR" echo "$output" [[ $status -eq 0 ]] [[ $output = *'1* FROM alpine:3.17'* ]] [[ $output = *'2* RUN.S true'* ]] echo echo '*** Build a 3rd time with the second base image: should now miss' run ch-image build -t tmpimg -f "$df2" "$BATS_TMPDIR" echo "$output" [[ $status -eq 0 ]] [[ $output = *'1* FROM alpine:3.16'* ]] [[ $output = *'2. RUN.S true'* ]] } @test "dnf --installroot" { # issue #1765 export CH_IMAGE_STORAGE=$BATS_TMPDIR/dnf_installroot df=$BATS_TMPDIR/dnf_installroot.df cat > "$df" < "$df" <<'EOF' ARG i=qux ARG img=alpine # “FROM alpine:3.17” for search when updating (see next line). ARG ver=3.17 FROM ${img}:${ver} ENV A=foo ENV A_B=/bar WORKDIR ${A_B}/baz RUN env | egrep '^PWD=' EOF run ch-image build --rebuild -f "$df" "$BATS_TMPDIR" echo "$output" [[ $status -eq 0 ]] [[ $output = *'PWD=/bar/baz'* ]] } charliecloud-0.37/test/build/50_dockerfile.bats000066400000000000000000001047071457016721300214640ustar00rootroot00000000000000load ../common setup () { [[ $CH_TEST_BUILDER != none ]] || skip 'no builder' } @test 'Dockerfile: syntax quirks' { # These should all yield an output image, but we don’t actually care about # it, so re-use the same one. export CH_IMAGE_CACHE=disabled scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' # FIXME: other builders? # No newline at end of file. printf 'FROM alpine:3.17\nRUN echo hello' \ | ch-image build -t tmpimg -f - . # Newline before FROM. ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 RUN echo hello EOF # Comment before FROM. ch-image build -t tmpimg -f - . <<'EOF' # foo FROM alpine:3.17 RUN echo hello EOF # Single instruction. ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 EOF # Whitespace around comment hash. run ch-image -v build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 #no whitespace #before only # after only # both before and after # multiple before # tab before EOF echo "$output" [[ $status -eq 0 ]] [[ $(echo "$output" | grep -Fc 'comment') -eq 6 ]] # Whitespace and newlines (turn on whitespace highlighting in your editor): run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 # trailing whitespace: shell sees it verbatim RUN true # whitespace-only line: ignored # two in a row # line continuation, no whitespace: shell sees one word RUN echo test1\ a # two in a row RUN echo test1\ b\ c # whitespace before line continuation: shell sees whitespace verbatim RUN echo test2 \ a # two in a row RUN echo test2 \ b \ c # whitespace after line continuation: shell sees one word RUN echo test3\ a # two in a row RUN echo test3\ b\ c # whitespace before & after line continuation: shell sees before only RUN echo test4 \ a # two in a row RUN echo test4 \ b \ c # whitespace on continued line: shell sees continued line's whitespace RUN echo test5\ a # two in a row RUN echo test5\ b\ c # whitespace-only continued line: shell sees whitespace verbatim RUN echo test6\ \ a # two in a row RUN echo test6\ \ \ b # backslash that is not a continuation: shell sees it verbatim RUN echo test\ 7\ a # two in a row RUN echo test\ 7\ \ b EOF echo "$output" [[ $status -eq 0 ]] output_expected=$(cat <<'EOF' warning: not yet supported, ignored: issue #777: .dockerignore file 1. FROM alpine:3.17 copying image ... 4. RUN.S true 13. RUN.S echo test1a test1a 16. RUN.S echo test1bc test1bc 21. RUN.S echo test2 a test2 a 24. RUN.S echo test2 b c test2 b c 29. RUN.S echo test3a test3a 32. RUN.S echo test3bc test3bc 37. RUN.S echo test4 a test4 a 40. RUN.S echo test4 b c test4 b c 45. RUN.S echo test5 a test5 a 48. RUN.S echo test5 b c test5 b c 53. RUN.S echo test6 a test6 a 57. RUN.S echo test6 b test6 b 63. RUN.S echo test\ 7a test 7a 66. RUN.S echo test\ 7\ b test 7 b --force=seccomp: modified 0 RUN instructions grown in 16 instructions: tmpimg build slow? consider enabling the build cache hint: https://hpc.github.io/charliecloud/command-usage.html#build-cache warning: reprinting 1 warning(s) warning: not yet supported, ignored: issue #777: .dockerignore file EOF ) diff -u <(echo "$output_expected") <(echo "$output") } @test 'Dockerfile: syntax errors' { scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' # Bad instruction. Also, -v should give interal blabber about the grammar. run ch-image -v build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 WEIRDAL EOF echo "$output" [[ $status -eq 1 ]] # error message [[ $output = *"can"?"t parse: -:2,1"* ]] # internal blabber (varies by version) [[ $output = *'No terminal'*"'W'"*'at line 2 col 1'* ]] # Bad long option. run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY --chown= foo bar EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *"can"?"t parse: -:2,14"* ]] # Empty input. run ch-image build -t tmpimg -f /dev/null . echo "$output" [[ $status -eq 1 ]] [[ $output = *'no instructions found: /dev/null'* ]] # Newline only. run ch-image build -t tmpimg -f - . <<'EOF' EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *'no instructions found: -'* ]] # Comment only. run ch-image build -t tmpimg -f - . <<'EOF' # foo EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *'no instructions found: -'* ]] # Only newline, then comment. run ch-image build -t tmpimg -f - . <<'EOF' # foo EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *'no instructions found: -'* ]] # Non-ARG instruction before FROM run ch-image build -t tmpimg -f - . <<'EOF' RUN echo uh oh FROM alpine:3.17 EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *'first instruction must be ARG or FROM'* ]] } @test 'Dockerfile: semantic errors' { scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' # Repeated instruction option. run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY --chown=foo --chown=bar fixtures/empty-file . EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *' 2 COPY: repeated option --chown'* ]] # COPY invalid option. run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY --foo=foo fixtures/empty-file . EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *'COPY: invalid option --foo'* ]] # FROM invalid option. run ch-image build -t tmpimg -f - . <<'EOF' FROM --foo=bar alpine:3.17 EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *'FROM: invalid option --foo'* ]] } @test 'Dockerfile: not-yet-supported features' { # This test also creates images we don’t care about. scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' # FROM --platform run ch-image build -t tmpimg -f - . <<'EOF' FROM --platform=foo alpine:3.17 EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: not yet supported: issue #778: FROM --platform'* ]] # other instructions run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 ADD foo CMD foo ENTRYPOINT foo ONBUILD foo EOF echo "$output" [[ $status -eq 0 ]] [[ $(echo "$output" | grep -Ec 'not yet supported.+instruction') -eq 8 ]] [[ $output = *'warning: not yet supported, ignored: issue #782: ADD instruction'* ]] [[ $output = *'warning: not yet supported, ignored: issue #780: CMD instruction'* ]] [[ $output = *'warning: not yet supported, ignored: issue #780: ENTRYPOINT instruction'* ]] [[ $output = *'warning: not yet supported, ignored: issue #788: ONBUILD instruction'* ]] # .dockerignore files run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *'warning: not yet supported, ignored: issue #777: .dockerignore file'* ]] # URL (Git repo) contexts run ch-image build -t not-yet-supported -f - \ git@github.com:hpc/charliecloud.git <<'EOF' FROM alpine:3.17 EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: not yet supported: issue #773: URL context'* ]] run ch-image build -t tmpimg -f - \ https://github.com/hpc/charliecloud.git <<'EOF' FROM alpine:3.17 EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: not yet supported: issue #773: URL context'* ]] # variable expansion modifiers run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 ARG foo=README COPY fixtures/${foo:+bar} . EOF echo "$output" [[ $status -eq 1 ]] # shellcheck disable=SC2016 [[ $output = *'error: modifiers ${foo:+bar} and ${foo:-bar} not yet supported (issue #774)'* ]] run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 ARG foo=README COPY fixtures/${foo:-bar} . EOF echo "$output" [[ $status -eq 1 ]] # shellcheck disable=SC2016 [[ $output = *'error: modifiers ${foo:+bar} and ${foo:-bar} not yet supported (issue #774)'* ]] } @test 'Dockerfile: unsupported features' { # This test also creates images we don’t care about. scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' # parser directives run ch-image build -t tmpimg -f - . <<'EOF' # escape=foo # syntax=foo #syntax=foo # syntax=foo #syntax=foo # foo=bar # comment FROM alpine:3.17 EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *'warning: not supported, ignored: parser directives'* ]] [[ $(echo "$output" | grep -Fc 'parser directives') -eq 10 ]] # COPY --from run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY --chown=foo fixtures/empty-file . EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *'warning: not supported, ignored: COPY --chown'* ]] # Unsupported instructions run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 EXPOSE foo HEALTHCHECK foo MAINTAINER foo STOPSIGNAL foo USER foo VOLUME foo EOF echo "$output" [[ $status -eq 0 ]] [[ $(echo "$output" | grep -Fc 'not supported') -eq 12 ]] [[ $output = *'warning: not supported, ignored: EXPOSE instruction'* ]] [[ $output = *'warning: not supported, ignored: HEALTHCHECK instruction'* ]] [[ $output = *'warning: not supported, ignored: MAINTAINER instruction'* ]] [[ $output = *'warning: not supported, ignored: STOPSIGNAL instruction'* ]] [[ $output = *'warning: not supported, ignored: USER instruction'* ]] [[ $output = *'warning: not supported, ignored: VOLUME instruction'* ]] } @test 'Dockerfile: ENV parsing' { scope standard env_expected=$(cat <<'EOF' ('chse_0a', 'value 0a') ('chse_0b', 'value 0b') ('chse_1b', 'value 1b ') ('chse_2a', 'value2a') ('chse_2b', 'value2b') ('chse_2c', 'chse2: value2a') ('chse_2d', 'chse2: value2a') ('chse_3a', '"value3a"') ('chse_4a', 'value4a') ('chse_4b', 'value4b') ('chse_5a', 'value5a') ('chse_5b', 'value5b') ('chse_6a', 'value6a') ('chse_6b', 'value6b') EOF ) run build_ --no-cache -t tmpimg -f - . <<'EOF' FROM almalinux_8ch # FIXME: make this more comprehensive, e.g. space-separate vs. # equals-separated for everything. # Value has internal space. ENV chse_0a value 0a ENV chse_0b="value 0b" # Value has internal space and trailing space. NOTE: Beware your editor # "helpfully" removing the trailing space. # # FIXME: Docker removes the trailing space! #ENV chse_1a value 1a ENV chse_1b="value 1b " # FIXME: currently a parse error. #ENV chse_1c=value\ 1c\ # Value surrounded by double quotes, which are not part of the value. ENV chse_2a "value2a" ENV chse_2b="value2b" # Substitute previous value, space-separated, without quotes. ENV chse_2c chse2: ${chse_2a} # Substitute a previous value, equals-separated, with quotes. ENV chse_2d="chse2: ${chse_2a}" # Backslashed quotes are included in value. ENV chse_3a \"value3a\" # FIXME: backslashes end up literal #ENV chse_3b=\"value3b\" # Multiple variables in the same instruction. ENV chse_4a=value4a chse_5a=value5a ENV chse_4b=value4b \ chse_5b=value5b # Value contains line continuation. FIXME: I think something isn't quite right # here. The backslash, newline sequence appears in the parse tree but not in # the output. That doesn't seem right. ENV chse_6a value\ 6a ENV chse_6b "value\ 6b" # FIXME: currently a parse error. #ENV chse_4=value4 chse_5="value5 foo" chse_6=value6\ foo chse_7=\"value7\" # Print output with Python to avoid ambiguity. RUN python3 -c 'import os; [print((k,v)) for (k,v) in sorted(os.environ.items()) if "chse_" in k]' EOF echo "$output" env_actual=$( echo "$output" \ | sed -En "s/^(#[0-9]+ [0-9.]+ )?(\('chse_.+\))$/\2/p") echo "$env_actual" [[ $status -eq 0 ]] diff -u <(echo "$env_expected") <(echo "$env_actual") } @test 'Dockerfile: LABEL parsing' { scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' label_expected=$(cat <<'EOF' ('chsl_0a', 'value 0a') ('chsl_0b', 'value 0b') ('chsl_2a', 'value2a') ('chsl_2b', 'value2b') ('chsl_3a', 'value3a') ('chsl_3b', 'value3b') EOF ) run build_ --no-cache -t tmpimg -f - . <<'EOF' FROM almalinux_8ch # Value has internal space. LABEL chsl_0a value 0a LABEL chsl_0b="value 0b" # FIXME: See issue #1533. Quotes around keys are not removed in metadata. #LABEL "chsl_1"="value 1" # Multiple variables in the same instruction. LABEL chsl_2a=value2a chsl_3a=value3a LABEL chsl_2b=value2b \ chsl_3b=value3b # FIXME: currently a parse error. #LABEL chsl_4=value4 chsl_5="value5 foo" chsl_6=value6\ foo chsl_7=\"value7\" # FIXME: See issue #1512. Multiline values currently not supported. #LABEL chsl_5 = "value\ #5" # Print output with Python to avoid ambiguity. RUN python3 -c 'import os; import json; labels = json.loads(open("/ch/metadata.json", "r").read())["labels"]; \ [print((k,v)) for (k,v) in sorted(labels.items()) if "chsl_" in k]' EOF echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$label_expected") <(echo "$output" | grep -E "^\('chsl_") } @test 'Dockerfile: SHELL' { scope standard [[ $CH_TEST_BUILDER = buildah* ]] && skip "Buildah doesn't support SHELL" # test that SHELL command can change executables and parameters run build_ -t tmpimg --no-cache -f - . <<'EOF' FROM alpine:3.17 RUN echo default: $0 SHELL ["/bin/ash", "-c"] RUN echo ash: $0 SHELL ["/bin/sh", "-v", "-c"] RUN echo sh-v: $0 EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *"default: /bin/sh"* ]] [[ $output = *"ash: /bin/ash"* ]] [[ $output = *"sh-v: /bin/sh"* ]] # test that it fails if shell doesn’t exist run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 SHELL ["/doesnotexist", "-c"] RUN print("hello") EOF echo "$output" [[ $status -eq 1 ]] if [[ $CH_TEST_BUILDER = ch-image ]]; then [[ $output = *"/doesnotexist: No such file or directory"* ]] else [[ $output = *"/doesnotexist: no such file or directory"* ]] fi # test that it fails if no paramaters run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 SHELL ["/bin/sh"] RUN true EOF echo "$output" [[ $status -ne 0 ]] # different builders use different error exit codes [[ $output = *"/bin/sh: can't open 'true': No such file or directory"* ]] # test that it works with python3 run build_ -t tmpimg -f - . <<'EOF' FROM almalinux_8ch SHELL ["/usr/bin/python3", "-c"] RUN print ("hello") EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *"grown in 3 instructions: tmpimg"* \ || $output = *"Successfully built"* \ || $output = *"naming to"*"tmpimg done"* ]] } @test 'Dockerfile: ARG and ENV values' { # We use full scope for builders other than ch-image because (1) with # ch-image, we are responsible for --build-arg being implemented correctly # and (2) Docker and Buildah take a full minute for this test, vs. three # seconds for ch-image. if [[ $CH_TEST_BUILDER = ch-image ]]; then scope standard else scope full fi prerequisites_ok argenv sed_ () { # Print only lines listing a test variable, with instruction number # prefixes added by BuildKit stripped if present. sed -En "s/^(#[0-9]+ [0-9.]+ )?(chse_.+)$/\2/p" } # Note that this test illustrates a number of behavior differences between # the builders. For most of these, but not all, Docker and Buildah have # the same behavior and ch-image differs. echo '*** default (no --build-arg)' env_expected=$(cat <<'EOF' chse_arg2_df=arg2 chse_arg3_df=arg3 arg2 chse_env1_df=env1 chse_env2_df=env2 env1 EOF ) run build_ --no-cache -t tmpimg -f ./Dockerfile.argenv . echo "$output" [[ $status -eq 0 ]] env_actual=$(echo "$output" | sed_) echo "$env_actual" diff -u <(echo "$env_expected") <(echo "$env_actual") echo '*** one --build-arg, has no default' env_expected=$(cat <<'EOF' chse_arg1_df=foo1 chse_arg2_df=arg2 chse_arg3_df=arg3 arg2 chse_env1_df=env1 chse_env2_df=env2 env1 EOF ) run build_ --build-arg chse_arg1_df=foo1 \ --no-cache -t tmpimg -f ./Dockerfile.argenv . echo "$output" [[ $status -eq 0 ]] env_actual=$(echo "$output" | sed_) echo "$env_actual" diff -u <(echo "$env_expected") <(echo "$env_actual") echo '*** one --build-arg, has default' env_expected=$(cat <<'EOF' chse_arg2_df=foo2 chse_arg3_df=arg3 foo2 chse_env1_df=env1 chse_env2_df=env2 env1 EOF ) run build_ --build-arg chse_arg2_df=foo2 \ --no-cache -t tmpimg -f ./Dockerfile.argenv . echo "$output" [[ $status -eq 0 ]] env_actual=$(echo "$output" | sed_) echo "$env_actual" diff -u <(echo "$env_expected") <(echo "$env_actual") echo '*** one --build-arg from environment' if [[ $CH_TEST_BUILDER == ch-image ]]; then env_expected=$(cat <<'EOF' chse_arg1_df=foo1 chse_arg2_df=arg2 chse_arg3_df=arg3 arg2 chse_env1_df=env1 chse_env2_df=env2 env1 EOF ) else # Docker and Buildah do not appear to take --build-arg values from the # environment. This is contrary to the “docker build” documentation; # “buildah bud” does not mention it either way. Tested on 18.09.7 and # 1.9.1-dev, respectively. env_expected=$(cat <<'EOF' chse_arg2_df=arg2 chse_arg3_df=arg3 arg2 chse_env1_df=env1 chse_env2_df=env2 env1 EOF ) fi chse_arg1_df=foo1 \ run build_ --build-arg chse_arg1_df \ --no-cache -t tmpimg -f ./Dockerfile.argenv . echo "$output" [[ $status -eq 0 ]] env_actual=$(echo "$output" | sed_) echo "$env_actual" diff -u <(echo "$env_expected") <(echo "$env_actual") echo '*** one --build-arg set to empty string' env_expected=$(cat <<'EOF' chse_arg1_df= chse_arg2_df=arg2 chse_arg3_df=arg3 arg2 chse_env1_df=env1 chse_env2_df=env2 env1 EOF ) chse_arg1_df=foo1 \ run build_ --build-arg chse_arg1_df= \ --no-cache -t tmpimg -f ./Dockerfile.argenv . echo "$output" [[ $status -eq 0 ]] env_actual=$(echo "$output" | sed_) echo "$env_actual" diff -u <(echo "$env_expected") <(echo "$env_actual") echo '*** two --build-arg' env_expected=$(cat <<'EOF' chse_arg2_df=bar2 chse_arg3_df=bar3 chse_env1_df=env1 chse_env2_df=env2 env1 EOF ) run build_ --build-arg chse_arg2_df=bar2 \ --build-arg chse_arg3_df=bar3 \ --no-cache -t tmpimg -f ./Dockerfile.argenv . echo "$output" [[ $status -eq 0 ]] env_actual=$(echo "$output" | sed_) echo "$env_actual" diff -u <(echo "$env_expected") <(echo "$env_actual") echo '*** repeated --build-arg' env_expected=$(cat <<'EOF' chse_arg2_df=bar2 chse_arg3_df=arg3 bar2 chse_env1_df=env1 chse_env2_df=env2 env1 EOF ) run build_ --build-arg chse_arg2_df=FOO \ --build-arg chse_arg2_df=bar2 \ --no-cache -t tmpimg -f ./Dockerfile.argenv . echo "$output" [[ $status -eq 0 ]] env_actual=$(echo "$output" | sed_) echo "$env_actual" diff -u <(echo "$env_expected") <(echo "$env_actual") echo '*** two --build-arg with substitution' if [[ $CH_TEST_BUILDER == ch-image ]]; then env_expected=$(cat <<'EOF' chse_arg2_df=bar2 chse_arg3_df=bar3 bar2 chse_env1_df=env1 chse_env2_df=env2 env1 EOF ) else # Docker and Buildah don’t substitute provided values. env_expected=$(cat <<'EOF' chse_arg2_df=bar2 chse_arg3_df=bar3 ${chse_arg2_df} chse_env1_df=env1 chse_env2_df=env2 env1 EOF ) fi # shellcheck disable=SC2016 run build_ --build-arg chse_arg2_df=bar2 \ --build-arg chse_arg3_df='bar3 ${chse_arg2_df}' \ --no-cache -t tmpimg -f ./Dockerfile.argenv . echo "$output" [[ $status -eq 0 ]] env_actual=$(echo "$output" | sed_) echo "$env_actual" diff -u <(echo "$env_expected") <(echo "$env_actual") echo '*** ARG not in Dockerfile' # Note: We don’t test it, but for Buildah, the variable does show up in # the build environment. run build_ --build-arg chse_doesnotexist=foo \ --no-cache -t tmpimg -f ./Dockerfile.argenv . echo "$output" if [[ $CH_TEST_BUILDER = ch-image ]]; then [[ $status -eq 1 ]] [[ $output = *'not consumed'* ]] [[ $output = *'chse_doesnotexist'* ]] else # Docker now (with BuildKit) just ignores the missing variable. [[ $status -eq 0 ]] fi echo '*** ARG not in environment' run build_ --build-arg chse_arg1_df \ --no-cache -t tmpimg -f ./Dockerfile.argenv . echo "$output" if [[ $CH_TEST_BUILDER = ch-image ]]; then [[ $status -eq 1 ]] [[ $output = *'--build-arg: chse_arg1_df: no value and not in environment'* ]] else [[ $status -eq 0 ]] fi } @test 'Dockerfile: ARG before FROM' { scope standard # single-stage run build_ --no-cache -t tmpimg - <<'EOF' ARG os=alpine:3.17 ARG foo=bar FROM $os ARG baz=qux RUN echo "os=$os foo=$foo baz=$baz" RUN echo alpine=$(cat /etc/alpine-release | cut -d. -f1-2) EOF echo "$output" [[ $status -eq 0 ]] if [[ $CH_TEST_BUILDER != docker ]]; then # Docker weird and inconsistent [[ $output = *'FROM alpine:3.17'* ]] [[ $output = *'os=alpine:3.17 foo=bar baz=qux'* ]] fi [[ $output = *'alpine=3.17'* ]] # multi-stage run build_ --no-cache -t tmpimg - <<'EOF' ARG os1=alpine:3.16 ARG os2=alpine:3.17 FROM $os1 RUN echo "1: os1=$os1 os2=$os2" RUN echo alpine1=$(cat /etc/alpine-release | cut -d. -f1-2) FROM $os2 RUN echo "2: os1=$os1 os2=$os2" RUN echo alpine2=$(cat /etc/alpine-release | cut -d. -f1-2) EOF echo "$output" [[ $status -eq 0 ]] if [[ $CH_TEST_BUILDER != docker ]]; then [[ $output = *'FROM alpine:3.16'* ]] [[ $output = *'FROM alpine:3.17'* ]] [[ $output = *'1: os1=alpine:3.16 os2=alpine:3.17'* ]] [[ $output = *'2: os1=alpine:3.16 os2=alpine:3.17'* ]] [[ $output = *'alpine1=3.16'* ]] [[ $output = *'alpine2=3.17'* ]] fi # no default value run build_ --no-cache -t tmpimg - <<'EOF' ARG os FROM $os EOF echo "$output" [[ $status -eq 1 ]] if [[ $CH_TEST_BUILDER = docker ]]; then # shellcheck disable=SC2016 [[ $output = *'base name ($os) should not be blank'* ]] else # shellcheck disable=SC2016 [[ ${lines[-2]} = 'error: image reference contains an undefined variable: $os' ]] fi # set with --build-arg run build_ --no-cache --build-arg=os=alpine:3.17 -t tmpimg - <<'EOF' ARG os=alpine:3.17 FROM $os RUN echo "os=$os" RUN echo alpine=$(cat /etc/alpine-release | cut -d. -f1-2) EOF echo "$output" [[ $status -eq 0 ]] if [[ $CH_TEST_BUILDER != docker ]]; then [[ $output = *'FROM alpine:3.17'* ]] [[ $output = *'os=alpine:3.17'* ]] fi [[ $output = *'alpine=3.17'* ]] # both before and after FROM run build_ --no-cache -t tmpimg - <<'EOF' ARG foo=bar FROM alpine:3.17 ARG foo=baz RUN echo "foo=$foo" EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *'foo=baz'* ]] # second wins } @test 'Dockerfile: FROM multistage alias' { scope standard # Ensure multisage alias works and reports correct base with ARG. run build_ --no-cache -t tmpimg -f - . <<'EOF' ARG BASEIMG=alpine:3.17 FROM $BASEIMG as a RUN true FROM a as b RUN true FROM b RUN true EOF # We only care that other builders return 0; we only check ch-image output. echo "$output" [[ $status -eq 0 ]] # There is a distinction between the image tag, displayed base/alias text, # and internal storage tag (e.g., _stage%s suffix). Exercise the following. # # 1. checkout base image ARG, as stage_0 with alias 'a', and display # correct base text; # # 2. checkout stage_0 as stage_1, with alias 'b', and display correct base # text (alias 'a', not ARG); # # 3. checkout stage_1 as image tag and display correct base text, (alias # 'b', not 'a' or ARG). if [[ $CH_TEST_BUILDER = ch-image ]]; then [[ $output = *"ARG BASEIMG='alpine:3.17'"* ]] [[ $output = *'FROM alpine:3.17 AS a'* ]] [[ $output = *'RUN.S true'* ]] [[ $output = *'FROM a AS b'* ]] [[ $output = *'RUN.S true'* ]] [[ $output = *'FROM b'* ]] [[ $output = *'RUN.S true'* ]] run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output = *'alpine:3.17'* ]] [[ $output = *'tmpimg'* ]] [[ $output = *'tmpimg_stage0'* ]] [[ $output = *'tmpimg_stage1'* ]] fi } @test 'Dockerfile: FROM --arg' { scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' # --arg present but not used in image name run ch-image build --no-cache -t tmpimg -f - . <<'EOF' FROM --arg=foo=bar alpine:3.17 RUN echo "1: foo=$foo" EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *'FROM --arg=foo=bar alpine:3.17'* ]] [[ $output = *'1: foo=bar'* ]] # --arg used in image name run ch-image build --no-cache -t tmpimg -f - . <<'EOF' FROM --arg=os=alpine:3.17 $os RUN echo "1: os=$os" EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *'FROM --arg=os=alpine:3.17 alpine:3.17'* ]] [[ $output = *'1: os=alpine:3.17'* ]] # multiple --arg run ch-image build --no-cache -t tmpimg -f - . <<'EOF' FROM --arg=foo=bar --arg=os=alpine:3.17 $os RUN echo "1: foo=$foo os=$os" EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *'FROM --arg=foo=bar --arg=os=alpine:3.17 alpine:3.17'* ]] [[ $output = *'1: foo=bar os=alpine:3.17'* ]] } @test 'Dockerfile: COPY list form' { scope standard [[ $CH_TEST_BUILDER == ch-image ]] || skip 'ch-image only' # single source run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY ["fixtures/empty-file", "."] EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *"COPY ['fixtures/empty-file'] -> '.'"* ]] test -f "$CH_IMAGE_STORAGE"/img/tmpimg/empty-file # multiple source run ch-image build -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY ["fixtures/empty-file", "fixtures/README", "."] EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *"COPY ['fixtures/empty-file', 'fixtures/README'] -> '.'"* ]] test -f "$CH_IMAGE_STORAGE"/img/tmpimg/empty-file test -f "$CH_IMAGE_STORAGE"/img/tmpimg/README } @test 'Dockerfile: COPY to nonexistent directory' { scope standard # file to one directory that doesn’t exist build_ --no-cache -t tmpimg -f - . <<'EOF' FROM alpine:3.17 RUN ! test -e /foo COPY fixtures/empty-file /foo/file_ RUN test -f /foo/file_ EOF # file to multiple directories that don’t exist build_ --no-cache -t tmpimg -f - . <<'EOF' FROM alpine:3.17 RUN ! test -e /foo COPY fixtures/empty-file /foo/bar/file_ RUN test -f /foo/bar/file_ EOF # directory to one directory that doesn’t exist build_ --no-cache -t tmpimg -f - . <<'EOF' FROM alpine:3.17 RUN ! test -e /foo COPY fixtures /foo/dir_ RUN test -d /foo/dir_ && test -f /foo/dir_/empty-file EOF # directory: multiple parents DNE build_ --no-cache -t tmpimg -f - . <<'EOF' FROM alpine:3.17 RUN ! test -e /foo COPY fixtures /foo/bar/dir_ RUN test -d /foo/bar/dir_ && test -f /foo/bar/dir_/empty-file EOF } @test 'Dockerfile: COPY errors' { scope standard [[ $CH_TEST_BUILDER = buildah* ]] && skip 'Buildah untested' # Dockerfile on stdin, so no context directory. run build_ -t tmpimg - <<'EOF' FROM alpine:3.17 COPY doesnotexist . EOF echo "$output" [[ $status -ne 0 ]] if [[ $CH_TEST_BUILDER = docker ]]; then # This error message seems wrong. I was expecting something about # no context, so COPY not allowed. [[ $output = *'file does not exist'* \ || $output = *'not found'* ]] else [[ $output = *'no context'* ]] fi # SRC not inside context directory. # # Case 1: leading “..”. run build_ -t tmpimg -f - sotest <<'EOF' FROM alpine:3.17 COPY ../common.bash . EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'outside'*'context'* \ || $output = *'not found'* ]] # Case 2: “..” inside path. run build_ -t tmpimg -f - sotest <<'EOF' FROM alpine:3.17 COPY lib/../../common.bash . EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'outside'*'context'* \ || $output = *'not found'* ]] # Case 3: symlink leading outside context directory. run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY fixtures/symlink-to-tmp . EOF echo "$output" [[ $status -ne 0 ]] if [[ $CH_TEST_BUILDER = docker ]]; then [[ $output = *'file does not exist'* \ || $output = *'not found'* ]] else [[ $output = *'outside'*'context'* ]] fi # Multiple sources and non-directory destination. run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY Build.missing common.bash /etc/fstab/ EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'not a directory'* ]] # OK so with Docker now that BuildKit is the default (v24.0.5), this build # *succeeds* and /etc/fstab is overwritten with the contents of # common.bash (and Build.missing is ignored AFAICT). 👎 if [[ $CH_TEST_BUILDER != docker ]]; then run build_ -t foo -f - . <<'EOF' FROM alpine:3.17 COPY Build.missing common.bash /etc/fstab EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'not a directory'* ]] fi run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY run /etc/fstab/ EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'not a directory'* ]] run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY run /etc/fstab EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'not a directory'* \ || $output = *'cannot copy to non-directory'* ]] # No sources given. run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY . EOF echo "$output" [[ $status -ne 0 ]] if [[ $CH_TEST_BUILDER = ch-image ]]; then [[ $output = *"error: can"?"t parse: -:2,7"* ]] else [[ $output = *'COPY requires at least two arguments'* ]] fi run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY ["."] EOF echo "$output" [[ $status -ne 0 ]] if [[ $CH_TEST_BUILDER = ch-image ]]; then [[ $output = *'error: source or destination missing'* ]] else [[ $output = *'COPY requires at least two arguments'* ]] fi # No sources found. run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY doesnotexist . EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'not found'* ]] # Some sources found. run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY fixtures/README doesnotexist . EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'not found'* ]] # No context with Dockerfile on stdin by context “-” run build_ -t tmpimg - <<'EOF' FROM alpine:3.17 COPY fixtures/README . EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'error: no context because '?'-'?' given'* \ || $output = *'COPY failed: file not found in build context or'* \ || $output = *'no such file or directory'* ]] } @test 'Dockerfile: COPY --from errors' { scope standard [[ $CH_TEST_BUILDER = buildah* ]] && skip 'Buildah untested' # Note: Docker treats several types of erroneous --from names as another # image and tries to pull it. To avoid clashes with real, pullable images, # we use the random name “uhigtsbjmfps” (https://www.random.org/strings/). # current index run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY --from=0 /etc/fstab / EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'current'*'stage'* \ || $output = *'circular dependency'* ]] # current name run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 AS uhigtsbjmfps COPY --from=uhigtsbjmfps /etc/fstab / EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'current stage'* \ || $output = *'access denied'*'repository does not exist'* \ || $output = *'circular dependency'* ]] # index does not exist run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY --from=1 /etc/fstab / EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'does not exist'* \ || $output = *'index out of bounds'* \ || $output = *'invalid stage index'* ]] # name does not exist run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY --from=uhigtsbjmfps /etc/fstab / EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'does not exist'* \ || $output = *'pull access denied'*'repository does not exist'* ]] # index exists, but is later if [[ $CH_TEST_BUILDER != docker ]]; then # BuildKit can work out of order run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY --from=1 /etc/fstab / FROM alpine:3.17 EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'does not exist yet'* ]] fi # name is later if [[ $CH_TEST_BUILDER != docker ]]; then # BuildKit can work out of order run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY --from=uhigtsbjmfps /etc/fstab / FROM alpine:3.17 AS uhigtsbjmfps EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'does not exist'* ]] [[ $output != *'does not exist yet'* ]] # so we review test fi # negative index run build_ -t tmpimg -f - . <<'EOF' FROM alpine:3.17 COPY --from=-1 /etc/fstab / FROM alpine:3.17 EOF echo "$output" [[ $status -ne 0 ]] [[ $output = *'invalid negative stage index'* \ || $output = *'index out of bounds'* \ || $output = *'invalid stage index'* ]] } @test 'Dockerfile: COPY from previous stage, no context' { # Normally, COPY is disallowed if there’s no context directory, but if # it’s from a previous stage, it should work. See issue #1381. scope standard [[ $CH_TEST_BUILDER == ch-image ]] || skip 'ch-image only' run ch-image build --no-cache -t foo - <<'EOF' FROM alpine:3.16 FROM alpine:3.17 COPY --from=0 /etc/os-release / EOF echo "$output" [[ "$status" -eq 0 ]] } @test 'Dockerfile: FROM scratch' { scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' # remove if it exists ch-image delete scratch || true # pull and validate special handling run ch-image pull -v scratch echo "$output" [[ $status -eq 0 ]] [[ $output = *'manifest: using internal library'* ]] [[ $output != *'layer 1'* ]] # no layers } @test 'Dockerfile: bad image reference' { scope standard [[ $CH_TEST_BUILDER == ch-image ]] || skip 'ch-image only' run ch-image build -t tmpimg - <<'EOF' FROM /alpine:3.17 EOF echo "$output" [[ $status -eq 1 ]] [[ ${lines[-3]} = 'error: image ref syntax, char 1: /alpine:3.17' ]] } charliecloud-0.37/test/build/50_localregistry.bats000066400000000000000000000114371457016721300222350ustar00rootroot00000000000000load ../common tag='ch-image push' # Note: These tests use a local registry listening on localhost:5000 but do # not start it. Therefore, they do not depend on whether the pushed images are # already present. setup () { scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' localregistry_init } @test "${tag}: without destination reference" { # FIXME: This test copies an image manually so we can use it to push. # Remove when we have real aliasing support for images. ch-image build -t localhost:5000/alpine:3.17 - <<'EOF' FROM alpine:3.17 EOF run ch-image -v --tls-no-verify push localhost:5000/alpine:3.17 echo "$output" [[ $status -eq 0 ]] [[ $output = *'pushing image: localhost:5000/alpine:3.17'* ]] [[ $output = *"image path: ${CH_IMAGE_STORAGE}/img/localhost+5000%alpine+3.17"* ]] ch-image delete localhost:5000/alpine:3.17 } @test "${tag}: without metadata history" { ch-image build -t tmpimg - <<'EOF' FROM alpine:3.17 EOF ch-convert tmpimg "$BATS_TMPDIR/tmpimg" rm -rf "$BATS_TMPDIR/tmpimg/ch" ch-image delete tmpimg ch-image import "$BATS_TMPDIR/tmpimg" tmpimg run ch-image -v --tls-no-verify push tmpimg localhost:5000/tmpimg echo "$output" [[ $status -eq 0 ]] [[ $output = *'pushing image: tmpimg'* ]] [[ $output = *'destination: localhost:5000/tmpimg'* ]] [[ $output = *"image path: ${CH_IMAGE_STORAGE}/img/tmpimg"* ]] } @test "${tag}: with destination reference" { run ch-image -v --tls-no-verify push alpine:3.17 localhost:5000/alpine:3.17 echo "$output" [[ $status -eq 0 ]] [[ $output = *'pushing image: alpine:3.17'* ]] [[ $output = *'destination: localhost:5000/alpine:3.17'* ]] [[ $output = *"image path: ${CH_IMAGE_STORAGE}/img/alpine+3.17"* ]] # FIXME: Can’t re-use layer from previous test because it’s a copy. #re='layer 1/1: [0-9a-f]{7}: already present' #[[ $output =~ $re ]] } @test "${tag}: with --image" { # NOTE: This also tests round-tripping and a more complex destination ref. img="$BATS_TMPDIR"/pushtest-up img2="$BATS_TMPDIR"/pushtest-down mkdir -p "$img" "$img"/{bin,dev,usr} # Set up setuid/setgid files and directories. touch "$img"/{setuid_file,setgid_file} chmod 4640 "$img"/setuid_file chmod 2640 "$img"/setgid_file mkdir -p "$img"/{setuid_dir,setgid_dir} chmod 4750 "$img"/setuid_dir chmod 2750 "$img"/setgid_dir ls -l "$img" [[ $(stat -c '%A' "$img"/setuid_file) = -rwSr----- ]] [[ $(stat -c '%A' "$img"/setgid_file) = -rw-r-S--- ]] [[ $(stat -c '%A' "$img"/setuid_dir) = drwsr-x--- ]] [[ $(stat -c '%A' "$img"/setgid_dir) = drwxr-s--- ]] # Create fake history. mkdir -p "$img"/ch cat <<'EOF' > "$img"/ch/metadata.json { "history": [ {"created_by": "ch-test" } ] } EOF # Push the image run ch-image -v --tls-no-verify push --image "$img" \ localhost:5000/foo/bar:weirdal echo "$output" [[ $status -eq 0 ]] [[ $output = *'pushing image: localhost:5000/foo/bar:weirdal'* ]] [[ $output = *"image path: ${img}"* ]] [[ $output = *'stripping unsafe setgid bit: ./setgid_dir'* ]] [[ $output = *'stripping unsafe setgid bit: ./setgid_file'* ]] [[ $output = *'stripping unsafe setuid bit: ./setuid_dir'* ]] [[ $output = *'stripping unsafe setuid bit: ./setuid_file'* ]] # Pull it back ch-image -v --tls-no-verify pull localhost:5000/foo/bar:weirdal ch-convert localhost:5000/foo/bar:weirdal "$img2" ls -l "$img2" [[ $(stat -c '%A' "$img2"/setuid_file) = -rw-r----- ]] [[ $(stat -c '%A' "$img2"/setgid_file) = -rw-r----- ]] [[ $(stat -c '%A' "$img2"/setuid_dir) = drwxr-x--- ]] [[ $(stat -c '%A' "$img2"/setgid_dir) = drwxr-x--- ]] } @test "${tag}: consistent layer hash" { run ch-image push --tls-no-verify alpine:3.17 localhost:5000/alpine:3.17 echo "$output" [[ $status -eq 0 ]] push1=$(echo "$output" | grep -E 'layer 1/1: .+: checking') run ch-image push --tls-no-verify alpine:3.17 localhost:5000/alpine:3.17 echo "$output" [[ $status -eq 0 ]] push2=$(echo "$output" | grep -E 'layer 1/1: .+: checking') diff -u <(echo "$push1") <(echo "$push2") } @test "${tag}: environment variables round-trip" { # Delete “tmpimg” from previous test to avoid issues ch-image delete tmpimg cat <<'EOF' | ch-image build -t tmpimg - FROM alpine:3.17 ENV weird="al yankovic" EOF ch-image push --tls-no-verify tmpimg localhost:5000/tmpimg ch-image pull --tls-no-verify localhost:5000/tmpimg ch-convert localhost:5000/tmpimg "$BATS_TMPDIR"/tmpimg run ch-run "$BATS_TMPDIR"/tmpimg --unset-env='*' --set-env -- env echo "$output" [[ $status -eq 0 ]] [[ $output = *'weird=al yankovic'* ]] } charliecloud-0.37/test/build/50_misc.bats000066400000000000000000000003001457016721300202700ustar00rootroot00000000000000load ../common @test 'sotest executable works' { scope quick [[ $ch_libc = glibc ]] || skip 'glibc only' export LD_LIBRARY_PATH=./sotest ldd sotest/sotest sotest/sotest } charliecloud-0.37/test/build/50_rsync.bats000066400000000000000000000501561457016721300205110ustar00rootroot00000000000000load ../common # NOTE: ls(1) output is not checked; this is for copy-paste into docs # shellcheck disable=SC2034 tag=RSYNC setup () { scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' umask 0007 fixtures=$BATS_TMPDIR/rsync context=$fixtures/ctx dst=$CH_IMAGE_STORAGE/img/tmpimg/dst } ls_ () { # ls(1)-alike but more predictable output and only the fields we want. See # also “compare-ls” in ch-convert.bats. ( cd "$1" find . -mindepth 1 -printf '%M %n %3s%y %P -> %l\n' \ | LC_ALL=C sort -k4 \ | sed -E -e 's# ([[:alnum:]._-]+/){4}# #' \ -e 's# ([[:alnum:]._-]+/){3}# #' \ -e 's# ([[:alnum:]._-]+/){2}# #' \ -e 's# ([[:alnum:]._-]+/){1}# #' \ -e 's# -> $##' \ -e 's#([0-9]+)[f]#\1#' \ -e 's#([0-9]+ )[0-9 ]+[a-z] #\1 #' \ -e 's#^(d[rwx-]{9}) [0-9]#\1 .#' \ -e "s#$1#/...#" ) } ls_dump () { target=$1 target_basename=$(basename "$target") target_parent=$(dirname "$target") out_basename=$2 ( cd "$target_parent" \ && ls -oghR "$target_basename" > "${BATS_TMPDIR}/rsync_${out_basename}" ) } @test "${tag}: set up fixtures" { rm -Rf --one-file-system "$fixtures" mkdir "$fixtures" cd "$fixtures" mkdir "$context" # outside context echo file-out > file-out mkdir dir-out echo dir-out.file > dir-out/dir-out.file # top level of context cd "$context" printf 'basic1/file-basic1\nbasic2\n' > file-top # also list of files mkdir dir-top echo dir-top.file > dir-top/dir-top.file # plain files and directories mkdir basic1 chmod 705 basic1 # weird permissions echo file-basic1 > basic1/file-basic1 chmod 604 basic1/file-basic1 # weird permissions mkdir basic2 echo file-basic2 > basic2/file-basic2 # symlinks cd "$context" mkdir sym2 echo file-sym2 > sym2/file-sym2 mkdir sym1 cd sym1 echo file-sym1 > file-sym1 mkdir dir-sym1 echo dir-sym1.file > dir-sym1/dir-sym1.file # target outside context ln -s "$fixtures"/file-out file-out_abs ln -s ../../file-out file-out_rel ln -s ../../dir-out dir-out_rel # target outside source (but inside context) ln -s "$context"/file-top file-top_abs ln -s ../file-top file-top_rel ln -s ../dir-top dir-top_rel # target inside source ln -s "$context"/sym1/file-sym1 file-sym1_abs ln -s file-sym1 file-sym1_direct ln -s ../sym1/file-sym1 file-sym1_upover ln -s dir-sym1 dir-sym1_direct # target inside other source ln -s "$context"/sym2/file-sym2 file-sym2_abs ln -s ../sym2/file-sym2 file-sym2_upover # broken symlink cd "$context" mkdir sym-broken cd sym-broken ln -s doesnotexist doesnotexist_broken_direct # hard links cd "$context" mkdir hard cd hard echo hard-file > hard-file1 ln hard-file1 hard-file2 echo "## created fixtures ##" ls_ "$fixtures" ls_ "$fixtures" > "$BATS_TMPDIR"/rsync_fixtures-ls_ ls_dump "$fixtures" fixtures } @test "${tag}: source: file(s)" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst # single file RSYNC /basic1/file-basic1 /dst # ... renamed RSYNC /basic1/file-basic1 /dst/file-basic1_renamed # ... without metadata RSYNC +z /basic1/file-basic1 /dst/file-basic1_nom # ... with trailing slash on *destination* RSYNC /basic1/file-basic1 /dst/new/ # multiple files RSYNC /basic1/file-basic1 /basic2/file-basic2 /dst/newB EOF ch-image build --rebuild -f "$ch_tmpimg_df" "$context" ls_dump "$dst" files run ls_ "$dst" echo "$output" [[ $status -eq -0 ]] cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst # directory RSYNC /basic1 /dst # ... renamed? RSYNC /basic1 /dst/basic1_new # ... renamed (trailing slash on source) RSYNC /basic1/ /dst/basic1_renamed # ... destination trailing slash has no effect for directory sources RSYNC /basic1 /dst/basic1_newB RSYNC /basic1/ /dst/basic1_renamedB/ # ... with +z (no-op!!) RSYNC +z /basic1 /dst/basic1_newC # ... need -r at least RSYNC +z -r /basic1/ /dst/basic1_newD EOF ch-image build --rebuild -f "$ch_tmpimg_df" "$context" ls_dump "$dst" dir1 run ls_ "$dst" echo "$output" [[ $status -eq -0 ]] cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst # two directories explicitly RUN mkdir /dst/dstB && echo file-dstB > /dst/dstB/file-dstB RSYNC /basic1 /basic2 /dst/dstB # ... with wildcards RUN mkdir /dst/dstC && echo file-dstC > /dst/dstC/file-dstC RSYNC /basic* /dst/dstC # ... with trailing slashes RUN mkdir /dst/dstD && echo file-dstD > /dst/dstD/file-dstD RSYNC /basic1/ /basic2/ /dst/dstD # ... with trailing slashes and wildcards RUN mkdir /dst/dstE && echo file-dstE > /dst/dstE/file-dstE RSYNC /basic*/ /dst/dstE # ... with one trailing slash and one not RUN mkdir /dst/dstF && echo file-dstF > /dst/dstF/file-dstF RSYNC /basic1 /basic2/ /dst/dstF # ... replace (do not merge with) existing contents RUN mkdir /dst/dstG && echo file-dstG > /dst/dstG/file-dstG RSYNC --delete /basic*/ /dst/dstG EOF ch-image build --rebuild -f "$ch_tmpimg_df" "$context" ls_dump "$dst" dir2 run ls_ "$dst" echo "$output" [[ $status -eq -0 ]] cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC / /dst EOF ch-image build --rebuild -f "$ch_tmpimg_df" "$context" ls_dump "$dst" dir-root run ls_ "$dst" echo "$output" [[ $status -eq -0 ]] cat < doesnotexist drwxrwx--- . sym1 drwxrwx--- . dir-sym1 -rw-rw---- 1 14 dir-sym1.file lrwxrwxrwx 1 dir-sym1_direct -> dir-sym1 lrwxrwxrwx 1 dir-top_rel -> ../dir-top -rw-rw---- 1 10 file-sym1 lrwxrwxrwx 1 file-sym1_direct -> file-sym1 lrwxrwxrwx 1 file-sym1_upover -> ../sym1/file-sym1 lrwxrwxrwx 1 file-sym2_upover -> ../sym2/file-sym2 lrwxrwxrwx 1 file-top_rel -> ../file-top drwxrwx--- . sym2 -rw-rw---- 1 10 file-sym2 EOF } @test "${tag}: symlinks: default" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC /sym1 /dst EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] [[ 0 -eq $(echo "$output" | grep -Fc 'skipping non-regular file') ]] ls_dump "$dst" sym-default run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < dir-sym1 lrwxrwxrwx 1 dir-top_rel -> ../dir-top -rw-rw---- 1 10 file-sym1 lrwxrwxrwx 1 file-sym1_direct -> file-sym1 lrwxrwxrwx 1 file-sym1_upover -> ../sym1/file-sym1 lrwxrwxrwx 1 file-sym2_upover -> ../sym2/file-sym2 lrwxrwxrwx 1 file-top_rel -> ../file-top EOF } @test "${tag}: symlinks: default, source trailing slash" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC /sym1/ /dst/sym1 EOF ch-image build --rebuild -f "$ch_tmpimg_df" "$context" ls_dump "$dst" sym-slashed run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < dir-sym1 -rw-rw---- 1 10 file-sym1 lrwxrwxrwx 1 file-sym1_direct -> file-sym1 EOF } @test "${tag}: symlinks: +m" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC +m /sym1/ /dst/sym1 EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] [[ 12 -eq $(echo "$output" | grep -Fc 'skipping non-regular file') ]] ls_dump "$dst" sym-m run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC +u /sym1/ /dst/sym1 EOF ch-image build --rebuild -f "$ch_tmpimg_df" "$context" ls_dump "$dst" sym-u run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < dir-sym1 drwxrwx--- . dir-top_rel -rw-rw---- 1 13 dir-top.file -rw-rw---- 1 9 file-out_abs -rw-rw---- 1 9 file-out_rel -rw-rw---- 1 10 file-sym1 -rw-rw---- 1 10 file-sym1_abs lrwxrwxrwx 1 file-sym1_direct -> file-sym1 -rw-rw---- 1 10 file-sym1_upover -rw-rw---- 1 10 file-sym2_abs -rw-rw---- 1 10 file-sym2_upover -rw-rw---- 1 26 file-top_abs -rw-rw---- 1 26 file-top_rel EOF } @test "${tag}: symlinks: between sources" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC /sym1 /sym2 /dst EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] ls_dump "$dst" sym-between run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < dir-sym1 lrwxrwxrwx 1 dir-top_rel -> ../dir-top -rw-rw---- 1 10 file-sym1 lrwxrwxrwx 1 file-sym1_direct -> file-sym1 lrwxrwxrwx 1 file-sym1_upover -> ../sym1/file-sym1 lrwxrwxrwx 1 file-sym2_upover -> ../sym2/file-sym2 lrwxrwxrwx 1 file-top_rel -> ../file-top drwxrwx--- . sym2 -rw-rw---- 1 10 file-sym2 EOF } @test "${tag}: symlinks: sources are symlinks to file" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC /sym1/file-sym1_direct /sym1/file-sym1_upover /dst EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] ls_dump "$dst" sym-to-file run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < file-sym1 EOF } @test "${tag}: symlinks: source is symlink to directory" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC /sym1/dir-sym1_direct /dst EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] ls_dump "$dst" sym-to-dir run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < dir-sym1 EOF } @test "${tag}: symlinks: source is symlink to directory (trailing slash)" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC /sym1/dir-sym1_direct/ /dst/dir-sym1_direct EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] ls_dump "$dst" sym-to-dir-slashed run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC /sym-broken /dst EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] ls_dump "$dst" sym-broken run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < doesnotexist EOF } @test "${tag}: symlinks: broken (--copy-links)" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC +m --copy-links /sym-broken /dst EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 1 ]] [[ $output = *"symlink has no referent: \"${context}/sym-broken/doesnotexist_broken_direct\""* ]] } @test "${tag}: symlinks: src file, dst symlink to file" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst \ && touch /dst/file-dst \ && ln -s file-dst /dst/file-dst_direct \ && ls -lh /dst RSYNC /file-top /dst/file-dst_direct EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] ls_dump "$dst" sym-dst-symlink-file run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst \ && mkdir /dst/dir-dst \ && ln -s dir-dst /dst/dir-dst_direct \ && ls -lh /dst RSYNC /file-top /dst/dir-dst_direct EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] ls_dump "$dst" sym-dst-symlink-dir-src-file run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < dir-dst EOF } @test "${tag}: symlinks: src dir, dst symlink to dir" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst \ && mkdir /dst/dir-dst \ && ln -s dir-dst /dst/dir-dst_direct \ && ls -lh /dst RSYNC /dir-top /dst/dir-dst_direct EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] ls_dump "$dst" sym-dst-symlink-dir run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < dir-dst EOF } @test "${tag}: symlinks: src dir (slashed), dst symlink to dir" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst \ && mkdir /dst/dir-dst \ && ln -s dir-dst /dst/dir-dst_direct \ && ls -lh /dst RSYNC /dir-top/ /dst/dir-dst_direct EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] ls_dump "$dst" sym-dst-symlink-dir-slashed run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < dir-dst EOF } @test "${tag}: hard links" { inode_src=$(stat -c %i "$context"/hard/hard-file1) [[ $inode_src -eq $(stat -c %i "$context"/hard/hard-file2) ]] cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC /hard /dst EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 0 ]] ls_dump "$dst" hard run ls_ "$dst" echo "$output" [[ $status -eq 0 ]] cat < "$ch_tmpimg_df" FROM alpine:3.17 WORKDIR /dst RSYNC file-basic1 . EOF ch-image build --rebuild -f "$ch_tmpimg_df" "$context"/basic1 ls_dump "$dst" files run ls_ "$dst" echo "$output" [[ $status -eq -0 ]] cat < "$ch_tmpimg_df" FROM alpine:3.17 RSYNC foo bar EOF run ch-image build -t tmpimg - < "$ch_tmpimg_df" echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: no context'* ]] } @test "${tag}: bad + option" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RSYNC +y foo bar EOF run ch-image build -t tmpimg - < "$ch_tmpimg_df" echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: invalid plus option: y'* ]] } @test "${tag}: remote transports" { cat < "$ch_tmpimg_df" FROM alpine:3.17 RSYNC foo://bar baz EOF run ch-image build -t tmpimg - < "$ch_tmpimg_df" echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: SSH and rsync transports not supported'* ]] } @test "${tag}: excluded options" { # We only test one of them, for DRY, though I did pick the one that seemed # most dangerous. cat < "$ch_tmpimg_df" FROM alpine:3.17 RSYNC --remove-source-files foo bar EOF run ch-image build -t tmpimg - < "$ch_tmpimg_df" echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: disallowed option: --remove-source-files'* ]] } @test "${tag}: --*-from translation" { # relative (context) cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RSYNC --files-from=./file-top / /dst EOF ch-image build --rebuild -f "$ch_tmpimg_df" "$context" ls_dump "$dst" files run ls_ "$dst" echo "$output" [[ $status -eq -0 ]] cat < "$ch_tmpimg_df" FROM alpine:3.17 RUN mkdir /dst RUN printf 'file-top\n' > /fls RSYNC --files-from=/fls / /dst EOF ch-image build --rebuild -f "$ch_tmpimg_df" "$context" ls_dump "$dst" files run ls_ "$dst" echo "$output" [[ $status -eq -0 ]] cat < "$ch_tmpimg_df" FROM alpine:3.17 RSYNC --files-from=- / /dst EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: --*-from: can'?'t use standard input'* ]] # colon disallowed cat < "$ch_tmpimg_df" FROM alpine:3.17 RSYNC --files-from=foo:bar / /dst EOF run ch-image build --rebuild -f "$ch_tmpimg_df" "$context" echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: --*-from: can'?'t use remote hosts'* ]] } charliecloud-0.37/test/build/55_cache.bats000066400000000000000000001354131457016721300204230ustar00rootroot00000000000000load ../common # shellcheck disable=SC2034 tag=cache # WARNING: Git timestamp precision is only one second [1]. This can cause # unstable sorting within --tree output because the tests commit very fast. If # it matters, add a “sleep 1”. # # [1]: https://stackoverflow.com/questions/28237043 treeonly () { # Remove (1) everything including and after first blank line and (2) # trailing whitespace on each line. sed -E -e '/^$/Q' -e 's/\s+$//' } setup () { scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' [[ $CH_IMAGE_CACHE = enabled ]] || skip 'build cache enabled only' export CH_IMAGE_STORAGE=$BATS_TMPDIR/butest # don’t mess up main storage dot_base=$BATS_TMPDIR/bu_ ch-image gestalt bucache-dot } ### Test cases for build cache paper figures (DOI: 10.1145/3624062.3624585) ### # Not all of these ended up as figures in the published paper, but I’m leaving # them here because they were targeted to the paper and were used in some # versions. If they are in the published paper, the figure number is noted. @test "${tag}: Fig. 2: empty cache" { rm -Rf --one-file-system "$CH_IMAGE_STORAGE" blessed_tree=$(cat << EOF initializing storage directory: v7 ${CH_IMAGE_STORAGE} initializing empty build cache * (root) ROOT EOF ) run ch-image build-cache --tree --dot="${dot_base}empty" echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_tree") <(echo "$output" | treeonly) } @test "${tag}: Fig. 3: initial pull" { ch-image pull alpine:3.17 blessed_tree=$(cat << 'EOF' * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree --dot="${dot_base}initial-pull" echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_tree") <(echo "$output" | treeonly) } @test "${tag}: FROM" { # FROM pulls ch-image build-cache --reset run ch-image build -v -t d -f bucache/from.df . echo "$output" [[ $status -eq 0 ]] [[ $output = *'1. FROM alpine:3.17'* ]] blessed_tree=$(cat << 'EOF' * (d, alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree --dot="${dot_base}from" echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_tree") <(echo "$output" | treeonly) # FROM doesn’t pull (same target name) run ch-image build -v -t d -f bucache/from.df . echo "$output" [[ $status -eq 0 ]] [[ $output = *'1* FROM alpine:3.17'* ]] blessed_tree=$(cat << 'EOF' * (d, alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_tree") <(echo "$output" | treeonly) # FROM doesn’t pull (different target name) run ch-image build -v -t d2 -f bucache/from.df . echo "$output" [[ $status -eq 0 ]] [[ $output = *'1* FROM alpine:3.17'* ]] blessed_tree=$(cat << 'EOF' * (d2, d, alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_tree") <(echo "$output" | treeonly) } @test "${tag}: Fig. 4: a.df" { ch-image build-cache --reset ch-image build -t a -f bucache/a.df . blessed_out=$(cat << 'EOF' * (a) RUN.S echo bar * RUN.S echo foo * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree --dot="${dot_base}a" echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: Fig. 5: b.df" { ch-image build-cache --reset ch-image build -t a -f bucache/a.df . ch-image build -t b -f bucache/b.df . blessed_out=$(cat << 'EOF' * (b) RUN.S echo baz * (a) RUN.S echo bar * RUN.S echo foo * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree --dot="${dot_base}b" echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: Fig. 6: c.df" { ch-image build-cache --reset ch-image build -t a -f bucache/a.df . ch-image build -t b -f bucache/b.df . sleep 1 ch-image build -t c -f bucache/c.df . blessed_out=$(cat << 'EOF' * (c) RUN.S echo qux | * (b) RUN.S echo baz | * (a) RUN.S echo bar |/ * RUN.S echo foo * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree --dot="${dot_base}c" echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: rebuild a.df" { # Forcing a rebuild show produce a new pair of FOO and BAR commits from # from the alpine branch. blessed_out=$(cat << 'EOF' * (a) RUN.S echo bar * RUN.S echo foo | * (c) RUN.S echo qux | | * (b) RUN.S echo baz | | * RUN.S echo bar | |/ | * RUN.S echo foo |/ * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) ch-image --rebuild build -t a -f bucache/a.df . run ch-image build-cache --tree [[ $status -eq 0 ]] echo "$output" diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: rebuild b.df" { # Rebuild of B. Since A was rebuilt in the last test, and because # the rebuild behavior only forces misses on non-FROM instructions, it # should now be based on A's new commits. blessed_out=$(cat << 'EOF' * (b) RUN.S echo baz * (a) RUN.S echo bar * RUN.S echo foo | * (c) RUN.S echo qux | | * RUN.S echo baz | | * RUN.S echo bar | |/ | * RUN.S echo foo |/ * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) ch-image --rebuild build -t b -f bucache/b.df . run ch-image build-cache --tree [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: c.df" { # Rebuild C. Since C doesn’t reference img_a (like img_b does) rebuilding # causes a miss on FOO. Thus C makes new FOO and QUX commits. # # Shouldn’t FOO hit? --reidpr 2/16 # - No! Rebuild forces misses; since c.df has it’s own FOO it should miss. # --jogas 2/24 blessed_out=$(cat << 'EOF' * (c) RUN.S echo qux * RUN.S echo foo | * (b) RUN.S echo baz | * (a) RUN.S echo bar | * RUN.S echo foo |/ | * RUN.S echo qux | | * RUN.S echo baz | | * RUN.S echo bar | |/ | * RUN.S echo foo |/ * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) # avoid race condition sleep 1 ch-image --rebuild build -t c -f bucache/c.df . run ch-image build-cache --tree [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: Fig. 7: change then revert" { ch-image build-cache --reset ch-image build -t e -f bucache/a.df . # “change” by using a different Dockerfile sleep 1 ch-image build -t e -f bucache/c.df . blessed_out=$(cat << 'EOF' * (e) RUN.S echo qux | * RUN.S echo bar |/ * RUN.S echo foo * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree --dot="${dot_base}revert1" echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) # “revert change”; no need to check for miss b/c it will show up in graph ch-image build -t e -f bucache/a.df . blessed_out=$(cat << 'EOF' * RUN.S echo qux | * (e) RUN.S echo bar |/ * RUN.S echo foo * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree --dot="${dot_base}revert2" echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: two pulls, same" { ch-image build-cache --reset ch-image pull alpine:3.17 ch-image pull alpine:3.17 blessed_out=$(cat << 'EOF' * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: two pulls, different" { localregistry_init unset CH_IMAGE_AUTH # don’t give local creds to Docker Hub # Simulate a change in an image from a remote repo; ensure that “ch-image # pull” downloads the next image. Note ch-image pull behavior is the same # with or without the build cache. This test is here for two reasons: # # 1. The build cache interactions with pull is more complex, i.e., we # assume that if pull works here with the cache enabled, it also # works without it. # # 2. We emit debugging PDFs for use in the paper, and doing that # anywhere else would be too surprising. df_ours=$(cat <<'EOF' FROM localhost:5000/champ RUN cat /worldchampion EOF ) tree_ours_before=$(cat <<'EOF' * (wc) RUN.S cat /worldchampion * (localhost+5000%champ) PULL localhost:5000/champ * (root) ROOT EOF ) echo echo '*** Prepare “ours” and “theirs” storages.' so=$BATS_TMPDIR/pull-local st=$BATS_TMPDIR/pull-remote rm -Rf --one-file-system "$so" "$st" echo echo '*** Them: Create the initial image state.' ch-image -s "$st" build -v -t capablanca -f - . < /worldchampion EOF ch-image -s "$st" --auth --tls-no-verify \ push capablanca localhost:5000/champ echo '*** Us: Build image using theirs as base.' # Both download and build caches are cold; FROM will do a (lazy) pull. # Files should be downloaded and all instructions should miss. Then do it # again; nothing should download and it should be all hits. run ch-image -s "$so" --auth --tls-no-verify \ build -t wc -f <(echo "$df_ours") /tmp echo "$output" [[ $status -eq 0 ]] [[ $output = *'. FROM'* ]] [[ $output = *'manifest list: downloading'* ]] [[ $output != *'manifest: downloading'* ]] [[ $output = *'config: downloading'* ]] [[ $output = *'. RUN.S'* ]] run ch-image -s "$so" --tls-no-verify build -t wc -f <(echo "$df_ours") /tmp echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output != *'manifest list: downloading'* ]] [[ $output != *'manifest: downloading'* ]] [[ $output != *'config: downloading'* ]] [[ $output = *'* RUN.S'* ]] run ch-image -s "$so" build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$tree_ours_before") <(echo "$output" | treeonly) echo echo '*** Us: explicit (eager) pull.' # This should download the manifest list and manifest, see that there are # no changes, and not download the config or layers. run ch-image -s "$so" --auth --tls-no-verify pull localhost:5000/champ echo "$output" [[ $status -eq 0 ]] [[ $output = *'manifest list: downloading'* ]] [[ $output != *'manifest: downloading'* ]] [[ $output = *'config: using existing file'* ]] [[ $output = *'layer'*'using existing file'* ]] run ch-image -s "$so" build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$tree_ours_before") <(echo "$output" | treeonly) echo echo '*** Them: Change and push the image.' ch-image -s "$st" build -t fischer -f - . < /worldchampion EOF ch-image -s "$st" --auth --tls-no-verify push fischer localhost:5000/champ echo echo '*** Us: Rebuild our image (lazy pull does not update).' # FROM should not notice the updated remote image. run ch-image -s "$so" --tls-no-verify build -t wc -f <(echo "$df_ours") /tmp echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output != *'manifest list: downloading'* ]] [[ $output != *'manifest: downloading'* ]] [[ $output != *'config: downloading'* ]] [[ $output = *'* RUN.S'* ]] run ch-image -s "$so" build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$tree_ours_before") <(echo "$output" | treeonly) echo echo '*** Us: Explicitly pull updated image.' # Returned config hash should differ from what is in storage; thus, the # new layer(s) should be pulled and the image branch in the cache updated. run ch-image -s "$so" --auth --tls-no-verify pull localhost:5000/champ echo "$output" [[ $status -eq 0 ]] [[ $output = *'manifest list: downloading'* ]] [[ $output != *'manifest: downloading'* ]] [[ $output = *'config: downloading'* ]] [[ $output = *'layer'*'downloading:'*'100%'* ]] run ch-image -s "$so" build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u - <(echo "$output" | treeonly) <<'EOF' * (localhost+5000%champ) PULL localhost:5000/champ | * (wc) RUN.S cat /worldchampion | * PULL localhost:5000/champ |/ * (root) ROOT EOF echo echo '*** Us: Rebuild our image (uses updated image).' # After the eager pull above, the base image exists in storage. Thus, the # FROM instruction hits; however, the resulting SID differs from the # original. Thus, intructions after FROM should miss. run ch-image -s "$so" --tls-no-verify build -t wc -f <(echo "$df_ours") /tmp echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output != *'manifest list: downloading'* ]] [[ $output != *'manifest: downloading'* ]] [[ $output != *'config: using existing file'* ]] [[ $output = *'. RUN.S'* ]] run ch-image -s "$so" build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u - <(echo "$output" | treeonly) <<'EOF' * (wc) RUN.S cat /worldchampion * (localhost+5000%champ) PULL localhost:5000/champ | * RUN.S cat /worldchampion | * PULL localhost:5000/champ |/ * (root) ROOT EOF } # FIXME: for issue #1359, add test here where they revert the image in the # remote registry to a previous state; our next pull will hit, and so too # should any subsequent previously cached instructions based on the FROM SID. @test "${tag}: branch ready" { ch-image build-cache --reset # Build A as “foo”. ch-image build -t foo -f bucache/a.df ./bucache # Rebuild A, except this is a broken version; the second instruction fails # leaving the new branch in a not-ready state pointing to “echo foo”. # The old branch remains. run ch-image build -t foo -f ./bucache/a-fail.df ./bucache sleep 1 echo "$output" [[ $status -eq 1 ]] run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] blessed_out=$(cat << 'EOF' * (foo) RUN.S echo bar * (foo#) RUN.S echo foo * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) # Build C as “foo”. Now branch “foo” points to the completed build of the # new Dockerfile, and the not-ready branch is gone. ch-image build -t foo -f ./bucache/c.df ./bucache run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] blessed_out=$(cat << 'EOF' * (foo) RUN.S echo qux | * RUN.S echo bar |/ * RUN.S echo foo * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: --force" { ch-image build-cache --reset # First build, without --force. ch-image build --force=none -t force -f ./bucache/force.df ./bucache # Second build, with --force. This should diverge after the first WORKDIR. sleep 1 ch-image build --force=seccomp -t force -f ./bucache/force.df ./bucache run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u - <(echo "$output" | treeonly) <<'EOF' * (force) WORKDIR /usr * RUN.S dnf install -y ed # doesn’t need --force | * WORKDIR /usr | * RUN.N dnf install -y ed # doesn’t need --force |/ * WORKDIR / * (almalinux+8) PULL almalinux:8 * (root) ROOT EOF # Third build, without --force. This should re-use the first build. sleep 1 ch-image build --force=none -t force -f ./bucache/force.df ./bucache run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u - <(echo "$output" | treeonly) <<'EOF' * WORKDIR /usr * RUN.S dnf install -y ed # doesn’t need --force | * (force) WORKDIR /usr | * RUN.N dnf install -y ed # doesn’t need --force |/ * WORKDIR / * (almalinux+8) PULL almalinux:8 * (root) ROOT EOF } @test "${tag}: Fig. 8: rebuild" { ch-image build-cache --reset # Build. Mode should not matter here, but we use enabled because that’s # more lifelike. ch-image build -t a -f ./bucache/a.df ./bucache # Re-build in “rebuild” mode. FROM should hit, others miss, and we should # have two branches. sleep 1 run ch-image build --rebuild -t a -f ./bucache/a.df ./bucache echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'. RUN.S echo foo'* ]] [[ $output = *'. RUN.S echo bar'* ]] blessed_out=$(cat << 'EOF' * (a) RUN.S echo bar * RUN.S echo foo | * RUN.S echo bar | * RUN.S echo foo |/ * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree --dot="${dot_base}rebuild" echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) # Re-build again in “rebuild” mode. The branch pointer should move to the # newer execution. run ch-image build-cache -v --tree echo "$output" [[ $status -eq 0 ]] commit_before=$(echo "$output" | sed -En 's/^.+\(a\) ([0-9a-f]+).+$/\1/p') echo "before: ${commit_before}" sleep 1 ch-image build --rebuild -t a -f ./bucache/a.df ./bucache run ch-image build-cache -v --tree echo "$output" [[ $status -eq 0 ]] commit_after=$(echo "$output" | sed -En 's/^.+\(a\) ([0-9a-f]+).+$/\1/p') echo "after: ${commit_after}" [[ $commit_before != "$commit_after" ]] } ### Additional test cases for correctness ### @test "${tag}: reset" { # re-init run ch-image build-cache --reset echo "$output" [[ $status -eq 0 ]] [[ $output = *'deleting build cache'* ]] [[ $output = *'initializing empty build cache'* ]] # fail if build cache disabled run ch-image build-cache --no-cache --reset [[ $status -eq 1 ]] echo "$output" [[ $output = *'build-cache subcommand invalid with build cache disabled'* ]] } @test "${tag}: gc" { ch-image build-cache --reset # Initial number of commits. diff -u <( ch-image build-cache \ | grep "commits" | awk '{print $2}') <(echo 1) # Number of commits after A. ch-image build -t a -f ./bucache/a.df . diff -u <( ch-image build-cache \ | grep "commits" | awk '{print $2}') <(echo 4) # Number of commits after 2x forced rebuilds of A (4 dangling) ch-image build --rebuild -t a -f ./bucache/a.df . ch-image build --rebuild -t a -f ./bucache/a.df . diff -u <( ch-image build-cache \ | grep "commits" | awk '{print $2}') <(echo 8) # Number of commits after garbage collecting. ch-image build-cache --tree ch-image build-cache --gc diff -u <( ch-image build-cache \ | grep "commits" | awk '{print $2}') <(echo 4) ch-image build-cache --tree } @test "${tag}: ARG and ENV" { ch-image build-cache --reset # Build. run ch-image build -t ae1 -f ./bucache/argenv.df ./bucache echo "$output" [[ $status -eq 0 ]] [[ $output = *'1 vargA vargBvargA venvA venvBvargA'* ]] # Rebuild; this should hit and print the correct values. run ch-image build -t ae1 -f ./bucache/argenv.df ./bucache echo "$output" [[ $status -eq 0 ]] [[ $output = *'5* RUN.S'* ]] # Re-build, with partial hits. ARG and ENV from first build should pass # through with correct values. sleep 1 run ch-image build -t ae2 -f ./bucache/argenv2.df ./bucache echo "$output" [[ $status -eq 0 ]] [[ $output = *'2 vargA vargBvargA venvA venvBvargA'* ]] # Re-build, setting ARG from the command line. This should miss. sleep 1 run ch-image build --build-arg=argB=foo \ -t ae3 -f ./bucache/argenv.df ./bucache echo "$output" [[ $status -eq 0 ]] [[ $output = *"3. ARG argB='foo'"* ]] # Re-build, setting ARG from the command line to the same. Should hit. sleep 1 run ch-image build --build-arg=argB=foo \ -t ae4 -f ./bucache/argenv.df ./bucache echo "$output" [[ $status -eq 0 ]] [[ $output = *"3* ARG argB='foo'"* ]] # Re-build, setting ARG from the command line to thing different. Miss. sleep 1 run ch-image build --build-arg=argB=bar \ -t ae5 -f ./bucache/argenv.df ./bucache echo "$output" [[ $status -eq 0 ]] [[ $output = *"3. ARG argB='bar'"* ]] [[ $output = *'1 vargA bar venvA venvB'* ]] # Check for expected tree. run ch-image build-cache --tree --dot="${dot_base}argenv" echo "$output" [[ $status -eq 0 ]] blessed_out=$(cat << 'EOF' * (ae5) RUN.S echo 1 $argA $argB $envA $envB * ENV envB='venvBvargA' * ENV envA='venvA' * ARG argB='bar' | * (ae4, ae3) RUN.S echo 1 $argA $argB $envA $envB | * ENV envB='venvBvargA' | * ENV envA='venvA' | * ARG argB='foo' |/ | * (ae2) RUN.S echo 2 $argA $argB $envA $envB | | * (ae1) RUN.S echo 1 $argA $argB $envA $envB | |/ | * ENV envB='venvBvargA' | * ENV envA='venvA' | * ARG argB='vargBvargA' |/ * ARG argA='vargA' * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: ARG special variables" { ch-image build-cache --reset unset SSH_AUTH_SOCK # Build. Should miss. run ch-image build -t foo -f ./bucache/argenv-special.df ./bucache echo "$output" [[ $status -eq 0 ]] [[ $output = *'1. FROM'* ]] [[ $output = *'2. ARG'* ]] [[ $output = *'3. ARG'* ]] [[ $output = *'4. RUN.S'* ]] [[ $output = *'vargA sockA'* ]] # Re-build. All hits. run ch-image build -t foo -f ./bucache/argenv-special.df ./bucache echo "$output" [[ $status -eq 0 ]] [[ $output = *'1* FROM'* ]] [[ $output = *'2* ARG'* ]] [[ $output = *'3* ARG'* ]] [[ $output = *'4* RUN.S'* ]] [[ $output != *'vargA sockA'* ]] # Re-build with new value from command line. All hits again. run ch-image build --build-arg=SSH_AUTH_SOCK=sockB \ -t foo -f ./bucache/argenv-special.df ./bucache echo "$output" [[ $status -eq 0 ]] [[ $output = *'1* FROM'* ]] [[ $output = *'2* ARG'* ]] [[ $output = *'3* ARG'* ]] [[ $output = *'4* RUN.S'* ]] [[ $output != *'vargA sockA'* ]] [[ $output != *'vargA sockB'* ]] } @test "${tag}: COPY" { ch-image build-cache --reset # Prepare fixtures. These need various manipulating during the test, which # is why they're built here on the fly. fixtures=${BATS_TMPDIR}/copy-cache mkdir -p "$fixtures" echo hello > "$fixtures"/file1 rm -f "$fixtures"/file1a mkdir -p "$fixtures"/dir1 touch "$fixtures"/dir1/file1 "$fixtures"/dir1/file2 printf '\n*** Build; all misses.\n\n' run ch-image build -t foo -f ./bucache/copy.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'. FROM'* ]] [[ $output = *'. COPY'* ]] printf '\n*** Re-build; all hits.\n\n' run ch-image build -t foo -f ./bucache/copy.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* COPY'* ]] printf '\n*** Add remove file in directory; should miss b/c dir mtime.\n\n' touch "$fixtures"/dir1/file2 rm "$fixtures"/dir1/file2 run ch-image build -t foo -f ./bucache/copy.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'. COPY'* ]] printf '\n*** Re-build; all hits.\n\n' run ch-image build -t foo -f ./bucache/copy.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* COPY'* ]] printf '\n*** Touch file; should miss because file mtime.\n\n' touch "$fixtures"/file1 run ch-image build -t foo -f ./bucache/copy.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'. COPY'* ]] printf '\n*** Re-build; all hits.\n\n' run ch-image build -t foo -f ./bucache/copy.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* COPY'* ]] printf '\n*** Rename file; should miss because filename.\n\n' mv "$fixtures"/file1 "$fixtures"/file1a run ch-image build -t foo -f ./bucache/copy.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'. COPY'* ]] printf '\n*** Re-build; all hits.\n\n' run ch-image build -t foo -f ./bucache/copy.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* COPY'* ]] printf '\n*** Rename file back; all hits.\n\n' stat "$fixtures"/file1a mv "$fixtures"/file1a "$fixtures"/file1 stat "$fixtures"/file1 run ch-image build -t foo -f ./bucache/copy.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* COPY'* ]] printf '\n*** Update content, same length, reset mtime; all hits.\n\n' cat "$fixtures"/file1 stat "$fixtures"/file1 mtime=$(stat -c %y "$fixtures"/file1) echo world > "$fixtures"/file1 touch -d "$mtime" "$fixtures"/file1 cat "$fixtures"/file1 stat "$fixtures"/file1 run ch-image build -t foo -f ./bucache/copy.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* COPY'* ]] } @test "${tag}: FROM non-cached base image" { ch-image build-cache --reset # Pull base image w/o cache. ch-image pull --no-cache alpine:3.17 [[ ! -e $CH_IMAGE_STORAGE/img/alpine+3.17/.git ]] # Build child image. run ch-image build -t foo - <<'EOF' FROM alpine:3.17 RUN echo foo EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *'. FROM'* ]] [[ $output = *'base image only exists non-cached; adding to cache'* ]] [[ $output = *'. RUN.S'* ]] # Check tree. blessed_out=$(cat << 'EOF' * (foo) RUN.S echo foo * (alpine+3.17) IMPORT alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: RSYNC" { ch-image build-cache --reset # Prepare fixtures. These need various manipulating during the test, which # is why they’re built here on the fly. fixtures=${BATS_TMPDIR}/rsync-cache rm -Rf --one-file-system "$fixtures" mkdir "$fixtures" echo hello > "$fixtures"/file1 mkdir "$fixtures"/dir1 touch "$fixtures"/dir1/file1 "$fixtures"/dir1/file2 printf '\n*** Build; all misses.\n\n' run ch-image build -f ./bucache/rsync.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'. FROM'* ]] [[ $output = *'. RSYNC'* ]] printf '\n*** Re-build; all hits.\n\n' run ch-image build -f ./bucache/rsync.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* RSYNC'* ]] printf '\n*** Add remove file in directory; should miss b/c dir mtime.\n\n' touch "$fixtures"/dir1/file2 rm "$fixtures"/dir1/file2 run ch-image build -f ./bucache/rsync.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'. RSYNC'* ]] printf '\n*** Re-build; all hits.\n\n' run ch-image build -f ./bucache/rsync.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* RSYNC'* ]] printf '\n*** Touch file; should miss because file mtime.\n\n' touch "$fixtures"/file1 run ch-image build -f ./bucache/rsync.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'. RSYNC'* ]] printf '\n*** Re-build; all hits.\n\n' run ch-image build -f ./bucache/rsync.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* RSYNC'* ]] printf '\n*** Rename file; should miss because filename.\n\n' mv "$fixtures"/file1 "$fixtures"/file1a run ch-image build -f ./bucache/rsync.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'. RSYNC'* ]] printf '\n*** Re-build; all hits.\n\n' run ch-image build -f ./bucache/rsync.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* RSYNC'* ]] printf '\n*** Rename file back; all hits.\n\n' stat "$fixtures"/file1a mv "$fixtures"/file1a "$fixtures"/file1 stat "$fixtures"/file1 run ch-image build -f ./bucache/rsync.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* RSYNC'* ]] printf '\n*** Update content, same length, reset mtime; all hits.\n\n' cat "$fixtures"/file1 stat "$fixtures"/file1 mtime=$(stat -c %y "$fixtures"/file1) echo world > "$fixtures"/file1 touch -d "$mtime" "$fixtures"/file1 cat "$fixtures"/file1 stat "$fixtures"/file1 run ch-image build -f ./bucache/rsync.df "$fixtures" echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* RSYNC'* ]] } @test "${tag}: all hits, new name" { ch-image build-cache --reset blessed_out=$(cat << 'EOF' * (a2, a) RUN.S echo bar * RUN.S echo foo * (alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) ch-image build -t a -f ./bucache/a.df . ch-image build -t a2 -f ./bucache/a.df . run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: pull to default destination" { ch-image build-cache --reset printf '\n*** Case 1: Not in build cache\n\n' run ch-image pull alpine:3.17 echo "$output" [[ $status -eq 0 ]] [[ $output = *'pulling image: alpine:3.17'* ]] [[ $output = *'pulled image: adding to build cache'* ]] # C1, C4 [[ $output != *'pulled image: found in build cache'* ]] # C2, C3 printf '\n*** Case 2: In build cache, up to date\n\n' run ch-image pull alpine:3.17 echo "$output" [[ $status -eq 0 ]] [[ $output = *'pulling image: alpine:3.17'* ]] [[ $output != *'pulled image: adding to build cache'* ]] # C1, C4 [[ $output = *'pulled image: found in build cache'* ]] # C2, C3 printf '\n*** Case 3: In build cache, not UTD, UTD commit present\n\n' printf 'FROM alpine:3.17\n' | ch-image build -t foo - printf 'FROM foo\nRUN echo foo\n' | ch-image build -t alpine:3.17 - blessed_out=$(cat << 'EOF' * (alpine+3.17) RUN.S echo foo * (foo) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) sleep 1 run ch-image pull alpine:3.17 echo "$output" [[ $status -eq 0 ]] [[ $output = *'pulling image: alpine:3.17'* ]] [[ $output != *'pulled image: adding to build cache'* ]] # C1, C4 [[ $output = *'pulled image: found in build cache'* ]] # C2, C3 blessed_out=$(cat << 'EOF' * RUN.S echo foo * (foo, alpine+3.17) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) printf '\n*** Case 4: In build cache, not UTD, UTD commit absent\n\n' sleep 1 printf 'FROM alpine:3.17\n' | ch-image build -t alpine:3.16 - blessed_out=$(cat << 'EOF' * RUN.S echo foo * (foo, alpine+3.17, alpine+3.16) PULL alpine:3.17 * (root) ROOT EOF ) run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) run ch-image pull alpine:3.16 echo "$output" [[ $status -eq 0 ]] [[ $output = *'pulling image: alpine:3.16'* ]] [[ $output = *'pulled image: adding to build cache'* ]] # C1, C4 [[ $output != *'pulled image: found in build cache'* ]] # C2, C3 blessed_out=$(cat << 'EOF' * (alpine+3.16) PULL alpine:3.16 | * RUN.S echo foo | * (foo, alpine+3.17) PULL alpine:3.17 |/ * (root) ROOT EOF ) run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_out") <(echo "$output" | treeonly) } @test "${tag}: multistage COPY" { # Multi-stage build with no instructions in the first stage. df_no=$(cat <<'EOF' FROM alpine:3.17 FROM alpine:3.16 COPY --from=0 /etc/os-release / EOF ) # Multi-stage build with instruction in the first stage. df_yes=$(cat <<'EOF' FROM alpine:3.17 RUN echo foo FROM alpine:3.16 COPY --from=0 /etc/os-release / EOF ) ch-image build-cache --reset run ch-image build -t tmpimg -f <(echo "$df_no") . # cold echo "$output" [[ $status -eq 0 ]] [[ $output = *'. COPY'* ]] run ch-image build -t tmpimg -f <(echo "$df_no") . # hot echo "$output" [[ $status -eq 0 ]] [[ $output = *'* COPY'* ]] ch-image build-cache --reset run ch-image build -t tmpimg -f <(echo "$df_yes") . # cold echo "$output" [[ $status -eq 0 ]] [[ $output = *'. COPY'* ]] run ch-image build -t tmpimg -f <(echo "$df_yes") . # hot echo "$output" [[ $status -eq 0 ]] [[ $output = *'* COPY'* ]] } @test "${tag}: pull to specified destination" { ch-image reset # pull special image to weird destination ch-image pull scratch foo # pull normal image to weird destination sleep 1 ch-image pull alpine:3.17 bar # everything in order? blessed_tree=$(cat << 'EOF' * (bar, alpine+3.17) PULL alpine:3.17 | * (scratch, foo) PULL scratch |/ * (root) ROOT EOF ) run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_tree") <(echo "$output" | treeonly) ls -x "$CH_IMAGE_STORAGE"/img [[ $(ls -x "$CH_IMAGE_STORAGE"/img) == "bar foo" ]] # pull same normal image normally sleep 1 ch-image pull alpine:3.17 # everything still in order? run ch-image build-cache --tree echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$blessed_tree") <(echo "$output" | treeonly) ls -x "$CH_IMAGE_STORAGE"/img [[ $(ls -x "$CH_IMAGE_STORAGE"/img) == "alpine+3.17 bar foo" ]] } @test "${tag}: empty dir persistence" { ch-image build-cache --reset ch-image delete tmpimg || true ch-image build -t tmpimg - <<'EOF' FROM alpine:3.17 RUN mkdir /foo && mkdir /foo/bar EOF sleep 1 ch-image build -t tmpimg - <<'EOF' FROM alpine:3.17 RUN true # miss RUN mkdir /foo # should not collide with leftover /foo from above EOF } @test "${tag}: garbage vs. reset" { scope full rm -Rf --one-file-system "$CH_IMAGE_STORAGE" # Init build cache. ch-image list cd "$CH_IMAGE_STORAGE"/bucache # Turn off auto-gc so it’s not triggered during the build itself. git config gc.auto 0 # Build an image that’s going to be annoying to garbage collect, but not # too annoying, so the test isn’t too long. Keep in mind this is probably # happening on a tmpfs. ch-image build -t tmpimg - <<'EOF' FROM alpine:3.17 RUN for i in $(seq 0 1024); do \ dd if=/dev/urandom of=/$i bs=4096K count=1 status=none; \ done EOF # Turn auto-gc back on, and configure it to run basically always. git config gc.auto 1 #git config gc.autoDetach false # for testing cat config # Garbage collect. Use raw Git commands so we can control exactly what is # going on. git gc --auto # Reset the cache while garbage collection is still running. cd .. run ch-image build-cache --reset echo "$output" [[ $status -eq 0 ]] [[ $output = *'stopping build cache garbage collection'* ]] } @test "${tag}: all hits, no image" { df=$(cat <<'EOF' FROM alpine:3.17 RUN echo foo EOF ) ch-image build-cache --reset ch-image build -t tmpimg -f <(echo "$df") . ch-image delete tmpimg [[ ! -e $CH_IMAGE_STORAGE/img/tmpimg ]] run ch-image build -v -t tmpimg -f <(echo "$df") . echo "$output" [[ $status -eq 0 ]] [[ $output = *'* FROM'* ]] [[ $output = *'* RUN.S'* ]] [[ $output = *"no image found: $CH_IMAGE_STORAGE/img/tmpimg"* ]] [[ $output = *'created worktree'* ]] } @test "${tag}: difficult files" { ch-image build-cache --reset statwalk () { # Remove (1) mtime and atime for symlinks, where it cannot be set, and # (2) directory sizes, which vary by filesystem and maybe other stuff. ( cd "$CH_IMAGE_STORAGE"/img/tmpimg/test find . -printf '%n %M %4s m=%TFT%TT a=%AFT%AT %p (%l)\n' \ | sed -E 's/^(1 l.+)([am]=[0-9T:.-]+ ){2}(.+)$/\1m=::: a=::: \3/' \ | sed -E 's/^([1-9] d[rwxs-]{9}) [0-9 ]{4}/\1 0000/' \ | LC_ALL=C sort -k6 ) } # Set umask so permissions match our reference data. umask 0027 # Build it. Every instruction does a quick restore, so this validates that # works, aside from mtime and atime which are expected to vary. Note that # “--force=none” is necessary because the dockerfile includes a call to # mkfifo(1), which uses the system call mknod(2), which is intercepted by # our seccomp(2) filter (see also: #1646). ch-image build --force=none -t tmpimg -f ./bucache/difficult.df . stat "$CH_IMAGE_STORAGE"/img/tmpimg/test/fifo_ stat1=$(statwalk) diff -u - <(echo "$stat1" | sed -E 's/([am])=[0-9T:.-]+/\1=:::/g') <<'EOF' 7 drwxr-x--- 0000 m=::: a=::: . () 2 drwsrwxrwx 0000 m=::: a=::: ./dir_all () 1 -rwsrwxrwx 0 m=::: a=::: ./dir_all/file_all () 2 drwxr-x--- 0000 m=::: a=::: ./dir_empty () 3 drwxr-x--- 0000 m=::: a=::: ./dir_empty_empty () 2 drwxr-x--- 0000 m=::: a=::: ./dir_empty_empty/dir_empty () 2 drwx------ 0000 m=::: a=::: ./dir_min () 1 -r-------- 0 m=::: a=::: ./dir_min/file_min () 1 prw-r----- 0 m=::: a=::: ./fifo_ () 3 drwxr-x--- 0000 m=::: a=::: ./gitrepo () 7 drwxr-x--- 0000 m=::: a=::: ./gitrepo/.git () 1 -rw-r----- 23 m=::: a=::: ./gitrepo/.git/HEAD () 2 drwxr-x--- 0000 m=::: a=::: ./gitrepo/.git/branches () 1 -rw-r----- 92 m=::: a=::: ./gitrepo/.git/config () 1 -rw-r----- 73 m=::: a=::: ./gitrepo/.git/description () 2 drwxr-x--- 0000 m=::: a=::: ./gitrepo/.git/hooks () 1 -rwxr-x--- 478 m=::: a=::: ./gitrepo/.git/hooks/applypatch-msg.sample () 1 -rwxr-x--- 896 m=::: a=::: ./gitrepo/.git/hooks/commit-msg.sample () 1 -rwxr-x--- 189 m=::: a=::: ./gitrepo/.git/hooks/post-update.sample () 1 -rwxr-x--- 424 m=::: a=::: ./gitrepo/.git/hooks/pre-applypatch.sample () 1 -rwxr-x--- 1643 m=::: a=::: ./gitrepo/.git/hooks/pre-commit.sample () 1 -rwxr-x--- 416 m=::: a=::: ./gitrepo/.git/hooks/pre-merge-commit.sample () 1 -rwxr-x--- 1374 m=::: a=::: ./gitrepo/.git/hooks/pre-push.sample () 1 -rwxr-x--- 4898 m=::: a=::: ./gitrepo/.git/hooks/pre-rebase.sample () 1 -rwxr-x--- 544 m=::: a=::: ./gitrepo/.git/hooks/pre-receive.sample () 1 -rwxr-x--- 1492 m=::: a=::: ./gitrepo/.git/hooks/prepare-commit-msg.sample () 1 -rwxr-x--- 2783 m=::: a=::: ./gitrepo/.git/hooks/push-to-checkout.sample () 1 -rwxr-x--- 3650 m=::: a=::: ./gitrepo/.git/hooks/update.sample () 2 drwxr-x--- 0000 m=::: a=::: ./gitrepo/.git/info () 1 -rw-r----- 240 m=::: a=::: ./gitrepo/.git/info/exclude () 4 drwxr-x--- 0000 m=::: a=::: ./gitrepo/.git/objects () 2 drwxr-x--- 0000 m=::: a=::: ./gitrepo/.git/objects/info () 2 drwxr-x--- 0000 m=::: a=::: ./gitrepo/.git/objects/pack () 4 drwxr-x--- 0000 m=::: a=::: ./gitrepo/.git/refs () 2 drwxr-x--- 0000 m=::: a=::: ./gitrepo/.git/refs/heads () 2 drwxr-x--- 0000 m=::: a=::: ./gitrepo/.git/refs/tags () 2 -rw-r----- 0 m=::: a=::: ./hard_src () 2 -rw-r----- 0 m=::: a=::: ./hard_target () 1 lrwxrwxrwx 11 m=::: a=::: ./soft_src (soft_target) 1 -rw-r----- 0 m=::: a=::: ./soft_target () EOF # Build again; tests full restore because we delete the image. Compare # against the (already validated) results of the first build, this time # including timestamps. ch-image delete tmpimg [[ ! -e $CH_IMAGE_STORAGE/img/tmpimg ]] run ch-image build --force=none -t tmpimg -f ./bucache/difficult.df . echo "$output" [[ $status -eq 0 ]] [[ $output = *'* RUN.N echo last'* ]] statwalk | diff -u <(echo "$stat1") - } @test "${tag}: ignore patterns" { # fails unless “__ch-test_ignore__” is included in the global gitignore file. git check-ignore -q __ch-test_ignore__ \ || pedantic_fail 'global ignore not configured' ch-image build-cache --reset df=$(cat <<'EOF' FROM alpine:3.17 RUN touch __ch-test_ignore__ EOF ) echo "$df" | ch-image build -t tmpimg - ch-image delete tmpimg echo "$df" | ch-image build -t tmpimg - ls -lh "$CH_IMAGE_STORAGE"/img/tmpimg/__ch-test_ignore__ } @test "${tag}: delete" { ch-image build-cache --reset printf 'FROM alpine:3.17\nRUN echo 1a\n' | ch-image build -t 1a - printf 'FROM alpine:3.17\nRUN echo 1b\n' | ch-image build -t 1b - printf 'FROM alpine:3.17\nRUN echo 2a\n' | ch-image build -t 2a - # Blessèd tree, with substitutions corresponding to images that will be # deleted. blessed_tree=$( ch-image build-cache --tree \ | treeonly \ | sed -E 's/\((..|alpine\+3\.17)\) //') echo "$blessed_tree" # starting point diff -u <(printf "1a\n1b\n2a\nalpine:3.17\n") <(ch-image list) # no glob ch-image delete 2a # the blessed tree needs to be updated, since 2a is now untagged diff -u <(printf "1a\n1b\nalpine:3.17\n") <(ch-image list) # matches none (non-empty) run ch-image delete 'foo*' echo "$output" [[ $status -eq 1 ]] [[ $output = *'no matching image, can'?'t delete: foo*'* ]] # matches some ch-image delete '1*' diff -u <(printf "alpine:3.17\n") <(ch-image list) # matches all ch-image delete '*' diff -u <(printf "") <(ch-image list) # matches none (empty) run ch-image delete '*' echo "$output" [[ $status -eq 1 ]] [[ $output = *'no matching image, can'?'t delete: *'* ]] # build cache unchanged diff -u <(echo "$blessed_tree") <(ch-image build-cache --tree | treeonly) } @test "${tag}: large files" { # We use files of size 3, 4, 5 MiB to avoid /lib/libcrypto.so.1.1, which # is about 2.5 MIB and which we don’t have control over. df=$(cat <<'EOF' FROM alpine:3.17 RUN dd if=/dev/urandom of=/bigfile3 bs=1M count=3 \ && dd if=/dev/urandom of=/bigfile4 bs=1M count=4 \ && dd if=/dev/urandom of=/bigfile5 bs=1M count=5 \ && touch -t 198005120000.00 /bigfile? \ && chmod 644 /bigfile? RUN ls -l /bigfile? /lib/libcrypto* EOF ) echo echo '*** no large files' ch-image build-cache --reset echo "$df" | ch-image build --cache-large=0 -t tmpimg - run ls "$CH_IMAGE_STORAGE"/bularge echo "$output" [[ $status -eq 0 ]] [[ -z $output ]] echo echo '*** threshold = 5' ch-image build-cache --reset echo "$df" | ch-image build --cache-large=5 -t tmpimg - run ls "$CH_IMAGE_STORAGE"/bularge echo "$output" [[ $status -eq 0 ]] diff -u - <(echo "$output") <<'EOF' b2dbc2a2bb35d6d0d5590aedc122cab6%bigfile5 EOF echo echo '*** threshold = 4, rebuild' echo "$df" | ch-image build --rebuild --cache-large=4 -t tmpimg - run ls "$CH_IMAGE_STORAGE"/bularge echo "$output" [[ $status -eq 0 ]] # should re-use existing bigfile5 diff -u - <(echo "$output") <<'EOF' 6f7a3513121d79c42283f6f758439c3a%bigfile4 b2dbc2a2bb35d6d0d5590aedc122cab6%bigfile5 EOF echo echo '*** threshold = 4, reset' ch-image build-cache --reset echo "$df" | ch-image build --rebuild --cache-large=4 -t tmpimg - run ls "$CH_IMAGE_STORAGE"/bularge echo "$output" [[ $status -eq 0 ]] diff -u - <(echo "$output") <<'EOF' 6f7a3513121d79c42283f6f758439c3a%bigfile4 b2dbc2a2bb35d6d0d5590aedc122cab6%bigfile5 EOF } @test "${tag}: hard links with Git-incompatible name" { # issue #1569 ch-image build-cache --reset ch-image build -t tmpimg - <<'EOF' FROM alpine:3.17 RUN mkdir -p a/b RUN mkdir -p a/c RUN touch a/b/.gitignore RUN ln a/b/.gitignore a/c/.gitignore RUN stat -c'%n %h %d/%i' a/?/.gitignore EOF } @test "${tag}: Git commands at image root" { # issue 1285 ch-image build-cache --reset # Use mount(8) to create a private /tmp; otherwise the bucache repo under # $BATS_TMPDIR *does* exist because /tmp is shared with the host. ch-image build -t tmpimg - <<'EOF' FROM alpine:3.17 RUN apk add git RUN cat /proc/mounts RUN mount -t tmpfs -o size=4m none /tmp \ && git config --system http.sslVerify false EOF } @test "${tag}: delete RPM databases" { # issue #1351 ch-image build-cache --reset run ch-image build -v -t tmpimg - <<'EOF' FROM alpine:3.17 RUN mkdir -p /var/lib/rpm RUN touch /var/lib/rpm/__db.001 EOF echo "$output" [[ $status -eq 0 ]] [[ $output = *'deleting, see issue #1351: var/lib/rpm/__db.001'* ]] [[ ! -e $CH_IMAGE_STORAGE/img/tmpimg/var/lib/rpm/__db.001 ]] } @test "${tag}: restore ACLs, xattrs" { # issue #1287 # Check if test needs to be skipped touch "$BATS_TMPDIR/tmpfs_test" if ! setfattr -n user.foo -v bar "$BATS_TMPDIR/tmpfs_test" \ && [[ -z $GITHUB_ACTIONS ]]; then skip "xattrs unsupported in ${BATS_TMPDIR}" fi # Build an image, then re-build from cache to test xattr/ACL cache # functionality. TMP_CX=$BATS_TMPDIR TMP_DF=$BATS_TMPDIR/weirdal.df cat <<'EOF' > "$TMP_DF" FROM alpine:3.17 RUN apk add attr RUN apk add acl RUN touch /home/foo RUN setfattr -n user.foo -v bar /home/foo RUN setfacl -m u:root:r /home/foo EOF ch-image build-cache --reset ch-image build -t tmpimg -f "$TMP_DF" "$TMP_CX" ch-image delete tmpimg ch-image build -t tmpimg -f "$TMP_DF" "$TMP_CX" run ch-run tmpimg -- getfattr home/foo # don’t check for ACL xattr bc it’s more straightforward to use getfacl(1). echo "$output" [[ $status -eq 0 ]] [[ $output = *'# file: home/foo'* ]] [[ $output = *'user.foo'* ]] run ch-run tmpimg -- getfacl home/foo echo "$output" [[ $status -eq 0 ]] [[ $output = *"user:$USER:r--"* ]] } @test "${tag}: orphaned worktrees" { # PR #1824 img_metadata=$CH_IMAGE_STORAGE/img/tmpimg/ch img_to_git=$img_metadata/git git_worktrees=$CH_IMAGE_STORAGE/bucache/worktrees git_to_img=$git_worktrees/tmpimg # pull image, should be unlinked ch-image pull --no-cache scratch tmpimg ch-image build-cache # rm leftover $git_to_img if it exists ls -lh "$img_metadata" "$git_worktrees" [[ ! -e "$img_to_git" ]] [[ ! -e "$git_to_img" ]] # add fake link touch "$img_to_git" ls -lh "$img_metadata" "$git_worktrees" [[ -e "$img_to_git" ]] [[ ! -e "$git_to_img" ]] # ch-image should warn and fix instead of crashing run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output = *'image erroneously marked cached, fixing'* ]] # warning should now be gone and the state be good ls -lh "$img_metadata" "$git_worktrees" [[ ! -e "$img_to_git" ]] [[ ! -e "$git_to_img" ]] run ch-image list echo "$output" [[ $status -eq 0 ]] [[ $output != *'image erroneously marked cached, fixing'* ]] } charliecloud-0.37/test/build/60_force.bats000066400000000000000000000044711457016721300204510ustar00rootroot00000000000000load ../common tag='ch-image --force' setup () { [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' export CH_IMAGE_CACHE=disabled } @test "${tag}: no matching distro" { scope standard # with --force run ch-image -v build --force=fakeroot -t tmpimg -f - . <<'EOF' FROM hello-world:latest EOF echo "$output" [[ $status -eq 1 ]] [[ $output = *'--force=fakeroot not available (no suitable config found)'* ]] } @test "${tag}: misc errors" { scope standard run ch-image build --force=fakeroot --force-cmd=foo,bar . echo "$output" [[ $status -eq 1 ]] [[ $output = *'are incompatible'* ]] } @test "${tag}: multiple RUN" { scope standard # 1. List form of RUN. # 2. apt-get not at beginning. run ch-image -v build --force -t tmpimg -f - . <<'EOF' FROM debian:buster RUN true RUN true && apt-get update RUN ["apt-get", "install", "-y", "hello"] EOF echo "$output" [[ $status -eq 0 ]] [[ $(echo "$output" | grep -Fc 'RUN: new command:') -eq 2 ]] [[ $output = *'--force=seccomp: modified 2 RUN instructions'* ]] [[ $output = *'grown in 4 instructions: tmpimg'* ]] } @test "${tag}: dpkg(8)" { # Typically folks will use apt-get(8), but bare dpkg(8) also happens. scope standard [[ $(uname -m) = x86_64 ]] || skip 'amd64 only' # NOTE: This produces a broken system because we ignore openssh-client’s # dependencies, but it’s good enough to test --force. ch-image -v build --force -t tmpimg -f - . <<'EOF' FROM debian:buster RUN apt-get update && apt install -y wget RUN wget -nv https://snapshot.debian.org/archive/debian/20230213T151507Z/pool/main/o/openssh/openssh-client_8.4p1-5%2Bdeb11u1_amd64.deb RUN dpkg --install --force-depends *.deb EOF } @test "${tag}: rpm(8)" { # Typically folks will use yum(8) or dnf(8), but bare rpm(8) also happens. scope standard [[ $(uname -m) = x86_64 ]] || skip 'amd64 only' ch-image -v build --force -t tmpimg -f - . <<'EOF' FROM almalinux:8 RUN curl -sO https://repo.almalinux.org/vault/8.6/BaseOS/x86_64/os/Packages/openssh-8.0p1-13.el8.x86_64.rpm RUN rpm --install *.rpm EOF } @test "${tag}: list form" { scope standard ch-image -v build --force -t tmpimg -f - . <<'EOF' FROM debian:buster RUN ["apt-get", "update"] RUN ["apt-get", "install", "-y", "openssh-client"] EOF } charliecloud-0.37/test/build/99_cleanup.bats000066400000000000000000000006671457016721300210210ustar00rootroot00000000000000load ../common @test 'nothing unexpected in tarball directory' { scope quick run find "$ch_tardir" -mindepth 1 -maxdepth 1 \ -not \( -name 'WEIRD_AL_YANKOVIC' \ -o -name '*.sqfs' \ -o -name '*.tar.gz' \ -o -name '*.tar.xz' \ -o -name '*.pq_missing' \) echo "$output" [[ $output = '' ]] } charliecloud-0.37/test/common.bash000066400000000000000000000336101457016721300172200ustar00rootroot00000000000000# shellcheck shell=bash # These variables are set, but in ways ShellCheck can’t figure out. Some may # be movable into this script but I haven’t looked in detail. We list them in # this unreachable code block to convince ShellCheck that they are assigned # (SC2154), and amusingly ShellCheck doesn’t know this code is unreachable. 😂 # # shellcheck disable=SC2034 if false; then # from ch-test ch_base= ch_bin= ch_lib= ch_libc= ch_test_tag= # from Bats lines= output= status= fi # Some defaults ch_tmpimg_df="$BATS_TMPDIR"/tmpimg.df arch_exclude () { # Skip the test if architecture (from “uname -m”) matches $1. [[ $(uname -m) != "$1" ]] || skip "arch ${1}" } archive_grep () { image="$1" case $image in *.sqfs) unsquashfs -l "$image" | grep 'squashfs-root/ch/environment' ;; *) tar -tf "$image" | grep -E '^([^/]*/)?ch/environment$' ;; esac } archive_ok () { ls -ld "$1" || true test -f "$1" test -s "$1" } build_ () { case $CH_TEST_BUILDER in ch-image) "$ch_bin"/ch-image build "$@" ;; docker) # Coordinate this list with test “build.bats/proxy variables”. # shellcheck disable=SC2154 docker_ build --build-arg HTTP_PROXY="$HTTP_PROXY" \ --build-arg HTTPS_PROXY="$HTTPS_PROXY" \ --build-arg NO_PROXY="$NO_PROXY" \ --build-arg http_proxy="$http_proxy" \ --build-arg https_proxy="$https_proxy" \ --build-arg no_proxy="$no_proxy" \ "$@" ;; *) printf 'invalid builder: %s\n' "$CH_TEST_BUILDER" >&2 exit 1 ;; esac } builder_ok () { # FIXME: Currently we make fairly limited tagging for some builders. # Uncomment below when they can be supported by all the builders. builder_tag_p "$1" #builder_tag_p "${1}:latest" #docker_tag_p "${1}:$(ch-run --version |& tr '~+' '--')" } builder_tag_p () { printf 'image tag %s ... ' "$1" case $CH_TEST_BUILDER in buildah*) hash_=$(buildah images -q "$1" | sort -u) if [[ $hash_ ]]; then echo "$hash_" return 0 fi ;; ch-image) if [[ -d ${CH_IMAGE_STORAGE}/img/${1} ]]; then echo "ok" return 0 fi ;; docker) hash_=$(docker_ images -q "$1" | sort -u) if [[ $hash_ ]]; then echo "$hash_" return 0 fi ;; esac echo 'not found' return 1 } chtest_fixtures_ok () { echo "checking chtest fixtures in: ${1}" # Did we raise hidden files correctly? [[ -e ${1}/.hiddenfile1 ]] [[ -e ${1}/..hiddenfile2 ]] [[ -e ${1}/...hiddenfile3 ]] # Did we remove the right /dev stuff? [[ -e ${1}/mnt/dev/dontdeleteme ]] ls -Aq "${1}/dev" [[ $(ls -Aq "${1}/dev") = '' ]] ch-run "$1" -- test -e /mnt/dev/dontdeleteme # Are permissions still good? ls -ld "$1"/maxperms_* [[ $(stat -c %a "${1}/maxperms_dir") = 1777 ]] [[ $(stat -c %a "${1}/maxperms_file") = 777 ]] } cray_ofi_or_skip () { if [[ $ch_cray ]]; then [[ -n "$CH_TEST_OFI_PATH" ]] || skip 'CH_TEST_OFI_PATH not set' [[ -z "$FI_PROVIDER_PATH" ]] || skip 'host FI_PROVIDER_PATH set' if [[ $cray_prov == 'gni' ]]; then export CH_FROMHOST_OFI_GNI=$CH_TEST_OFI_PATH $ch_mpirun_node ch-fromhost -v --cray-gni "$1" fi if [[ $cray_prov == 'cxi' ]]; then export CH_FROMHOST_OFI_CXI=$CH_TEST_OFI_PATH $ch_mpirun_node ch-fromhost --cray-cxi "$1" # Examples use libfabric's fi_info to ensure injection works; when # replacing libfabric we also need to replace this binary. fi_info="$(dirname "$(dirname "$CH_TEST_OFI_PATH")")/bin/fi_info" [[ -x "$fi_info" ]] $ch_mpirun_node ch-fromhost -v -d /usr/local/bin \ -p "$fi_info" \ "$1" fi else skip 'host is not a Cray' fi } env_require () { if [[ -z ${!1} ]]; then printf '$%s is empty or not set\n\n' "$1" >&2 exit 1 fi } image_ok () { test -d "$1" ls -ld "$1" || true byte_ct=$(du -s -B1 "$1" | cut -f1) echo "$byte_ct" [[ $byte_ct -ge 3145728 ]] # image is at least 3MiB [[ -d $1/bin && -d $1/dev && -d $1/usr ]] } localregistry_init () { # Skip unless GitHub Actions or there is a listener on localhost:5000. if [[ -z $GITHUB_ACTIONS ]] && ! ( command -v ss > /dev/null 2>&1 \ && ss -lnt | grep -F :5000); then skip 'no local registry' fi # Note: These will only stick if function is called *not* in a subshell. export CH_IMAGE_AUTH=yes export CH_IMAGE_USERNAME=charlie export CH_IMAGE_PASSWORD=test } multiprocess_ok () { [[ $ch_multiprocess ]] || skip 'no multiprocess launch tool found' true } openmpi_or_skip () { [[ $ch_mpi == 'openmpi' ]] || skip "openmpi only" } pedantic_fail () { msg=$1 if [[ -n $ch_pedantic ]]; then echo "$msg" 1>&2 return 1 else skip "$msg" fi } # If the two images (graphics, not container) are not “almost equal”, fail. # The first argument is the reference image; the second is the test image. The # third argument, if given, is the maximum number of differing pixels (default # zero). Also produce a diff image, which highlights the differing pixels in # red, based on the sample, e.g. foo.png -> foo.diff.png. pict_assert_equal () { ref=$1 sample=$2 pixel_max_ct=${3:-0} sample_base=$(basename "${sample%.*}") sample_ext=${sample##*.} diff_dir=${BATS_TMPDIR}/"$(basename "$(dirname "$sample")")" ref_bind="${ref}:/a.png" sample_bind="${sample}:/b.png" diff_bind="${diff_dir}:/diff" diff_="/diff/${sample_base}.diff.${sample_ext}" echo "reference: $ref" echo " bind: $ref_bind" echo "sample: $sample" echo " bind: $sample_bind" echo "diff: $diff_" echo " bind: $diff_bind" # See: https://imagemagick.org/script/command-line-options.php#metric pixel_ct=$(ch-run "$ch_img" -b "$ref_bind" \ -b "$sample_bind" \ -b "$diff_bind" -- \ compare -metric AE /a.png /b.png "$diff_" 2>&1 || true) echo "diff count: ${pixel_ct} pixels, max ${pixel_max_ct}" [[ $pixel_ct -le $pixel_max_ct ]] } # Check if the pict_ functions are usable; if not, pedantic-fail. pict_ok () { if "$ch_mpirun_node" ch-run "$ch_img" -- compare > /dev/null 2>&1; then pedantic_fail 'need ImageMagick' fi } pmix_or_skip () { if [[ $srun_mpi != pmix* ]]; then skip 'pmix required' fi } prerequisites_ok () { if [[ -f $CH_TEST_TARDIR/${1}.pq_missing ]]; then skip 'build prerequisites not met' fi } scope () { if [[ -n $ch_one_test ]]; then # Ignore scope if a single test is given. if [[ $BATS_TEST_DESCRIPTION != *"$ch_one_test"* ]]; then skip 'per --file' else return 0 fi fi case $1 in # $1 is the test’s scope quick) ;; # always run quick-scope tests standard) if [[ $CH_TEST_SCOPE = quick ]]; then skip "${1} scope" fi ;; full) if [[ $CH_TEST_SCOPE = quick || $CH_TEST_SCOPE = standard ]]; then skip "${1} scope" fi ;; skip) skip "developer-skipped; see comments and/or issues" ;; *) exit 1 esac } unpack_img_all_nodes () { if [[ $1 ]]; then case $CH_TEST_PACK_FMT in squash-mount) # Lots of things expect no extension here, so go with that # even though it’s a file, not a directory. $ch_mpirun_node ln -s "${ch_tardir}/${ch_tag}.sqfs" "${ch_imgdir}/${ch_tag}" ;; squash-unpack) $ch_mpirun_node ch-convert -o dir "${ch_tardir}/${ch_tag}.sqfs" "${ch_imgdir}/${ch_tag}" ;; tar-unpack) $ch_mpirun_node ch-convert -o dir "${ch_tardir}/${ch_tag}.tar.gz" "${ch_imgdir}/${ch_tag}" ;; *) false # unknown format ;; esac else skip 'not needed' fi } # Do we need sudo to run docker? if [[ -n $ch_docker_nosudo ]]; then docker_ () { docker "$@" } else docker_ () { sudo docker "$@" } fi # Podman wrapper (for consistency w docker) podman_ () { podman "$@" } # Do we have what we need? env_require CH_TEST_TARDIR env_require CH_TEST_IMGDIR env_require CH_TEST_PERMDIRS env_require CH_TEST_BUILDER if [[ $CH_TEST_BUILDER == ch-image ]]; then env_require CH_IMAGE_STORAGE fi # User-private temporary directory in case multiple users are running the # tests simultaneously. btnew=$TMP_/bats.tmp mkdir -p "$btnew" chmod 700 "$btnew" export BATS_TMPDIR=$btnew [[ $(stat -c %a "$BATS_TMPDIR") = '700' ]] ch_runfile=$(command -v ch-run) # Charliecloud version. ch_version=$(ch-run --version 2>&1) ch_version_base=$(echo "$ch_version" | sed -E 's/~.+//') ch_version_docker=$(echo "$ch_version" | tr '~+' '--') # Separate directories for tarballs and images. # # Canonicalize both so the have consistent paths and we can reliably use them # in tests (see issue #143). We use readlink(1) rather than realpath(2), # despite the admonition in the man page, because it's more portable [1]. # # We use “readlink -m” rather than “-e” or “-f” to account for the possibility # of some directory anywhere the path not existing [2], which has bitten us # multiple times; see issues #347 and #733. With this switch, if something is # missing, readlink(1) returns the path unchanged, and checks later convert # that to a proper error. # # [1]: https://unix.stackexchange.com/a/136527 # [2]: http://man7.org/linux/man-pages/man1/readlink.1.html ch_imgdir=$(readlink -m "$CH_TEST_IMGDIR") ch_tardir=$(readlink -m "$CH_TEST_TARDIR") # Image information. ch_tag=${CH_TEST_TAG:-NO_TAG_SET} # set by Makefile; many tests don’t need it ch_img=${ch_imgdir}/${ch_tag} ch_tar=${ch_tardir}/${ch_tag}.tar.gz ch_ttar=${ch_tardir}/chtest.tar.gz ch_timg=${ch_imgdir}/chtest if [[ $ch_tag = *'-mpich' ]]; then ch_mpi=mpich # As of MPICH 4.0.2, using SLURM as the MPICH process manager requires two # configure options that disable the compilation of mpiexec. This may not # always be the case. ch_mpi_exe=mpiexec else ch_mpi=openmpi ch_mpi_exe=mpirun fi # Crays are special. if [[ -f /etc/opt/cray/release/cle-release ]]; then ch_cray=yes # Prefer gni provider on Cray ugni machines if [[ -d /opt/cray/ugni ]]; then cray_prov=gni elif [[ -f /opt/cray/etc/release/cos ]]; then cray_prov=cxi fi else ch_cray= fi # Multi-node and multi-process stuff. Do not use Slurm variables in tests; use # these instead: # # ch_multiprocess can run multiple processes # ch_multinode can run on multiple nodes # ch_nodes number of nodes in job # ch_cores_node number of cores per node # ch_cores_total total cores in job ($ch_nodes × $ch_cores_node) # # ch_mpirun_node command to run one rank per node # ch_mpirun_core command to run one rank per physical core # ch_mpirun_2 command to run two ranks per job launcher default # ch_mpirun_2_1node command to run two ranks on one node # ch_mpirun_2_2node command to run two ranks on two nodes (one rank/node) # if [[ $SLURM_JOB_ID ]]; then ch_nodes=$SLURM_JOB_NUM_NODES else ch_nodes=1 fi # One rank per hyperthread can exhaust hardware contexts, resulting in # communication failure. Use one rank per core to avoid this. There are ways # to do this with Slurm, but they need Slurm configuration that seems # unreliably present. This seems to be the most portable way to do this. ch_cores_node=$(lscpu -p | tail -n +5 | sort -u -t, -k 2 | wc -l) ch_cores_total=$((ch_nodes * ch_cores_node)) ch_mpirun_node= ch_mpirun_np="-np ${ch_cores_node}" ch_unslurm= if [[ $SLURM_JOB_ID ]]; then [[ -z "$CH_TEST_SLURM_MPI" ]] || srun_mpi="--mpi=$CH_TEST_SLURM_MPI" ch_multiprocess=yes ch_mpirun_node="srun $srun_mpi --ntasks-per-node 1" ch_mpirun_core="srun $srun_mpi --ntasks-per-node $ch_cores_node" ch_mpirun_2="srun $srun_mpi -n2" ch_mpirun_2_1node="srun $srun_mpi -N1 -n2" # OpenMPI 3.1 pukes when guest-launched and Slurm environment variables # are present. Work around this by fooling OpenMPI into believing it’s not # in a Slurm allocation. if [[ $ch_mpi = openmpi ]]; then ch_unslurm='--unset-env=SLURM*' fi if [[ $ch_nodes -eq 1 ]]; then ch_multinode= ch_mpirun_2_2node=false else ch_multinode=yes ch_mpirun_2_2node="srun $srun_mpi -N2 -n2" fi else ch_multinode= ch_mpirun_2_2node=false if command -v mpirun > /dev/null 2>&1; then ch_multiprocess=yes ch_mpirun_node='mpirun --map-by ppr:1:node' ch_mpirun_core="mpirun ${ch_mpirun_np}" ch_mpirun_2='mpirun -np 2' ch_mpirun_2_1node='mpirun -np 2 --host localhost:2' else ch_multiprocess= ch_mpirun_node='' ch_mpirun_core=false ch_mpirun_2=false ch_mpirun_2_1node=false fi fi # Do we have and want sudo? if [[ $CH_TEST_SUDO ]] \ && command -v sudo >/dev/null 2>&1 \ && sudo -v > /dev/null 2>&1; then # This isn’t super reliable; it returns true if we have *any* sudo # privileges, not specifically to run the commands we want to run. ch_have_sudo=yes fi charliecloud-0.37/test/docs-sane.py.in000066400000000000000000000146741457016721300177350ustar00rootroot00000000000000#!%PYTHON_SHEBANG% # coding: utf-8 # This script performs sanity checking on the documentation: # # 1. Man page consistency. # # a. man/charliecloud.7 exists. # # b. The correct files FOO in bin have: # # - doc/FOO.rst # - doc/man/FOO.N # - an entry under “See also” in charliecloud.7 # # Where “N” is the appropriate man section number (e.g. 1 for # executables). Currently, the “correct” files in bin are: # # - All executables # - ch-completion.bash # # c. There aren’t any unexpcected .rst files, man files, or charliecloud.7 # “See also” entries. # # d. Synopsis in “FOO --help” (if applicable) matches FOO.rst and conf.py. from __future__ import print_function import glob import re import os import subprocess import sys # Dict of documentation files. Executables are added in “main()”. Files that are # not executables should be manually added here. man_targets = {"charliecloud": {"synopsis": "", "sec": 7}, "ch-completion.bash": {"synopsis": "Tab completion for the Charliecloud command line.", "sec": 7}} CH_BASE = os.path.abspath(os.path.dirname(__file__) + "/..") if (not os.path.isfile("%s/bin/ch-run" % CH_BASE)): print("not found: %s/bin/ch-run" % CH_BASE, file=sys.stderr) sys.exit(1) win = True def main(): check_man() if (win): print("ok") sys.exit(0) else: sys.exit(1) # This is the function that actually performs the sanity check for the docs (see # the comment at the top of this file). def check_man(): global man_targets # Add entries for executables to “man_targets”. “sec” is set to 1, “synopsis” # is set using the executable’s “--help” option (see “help_get”). Note that # this code assumes that a file is an executable if the execute bit for any # permission group. os.chdir(CH_BASE + "/bin") for f in os.listdir("."): if (os.path.isfile(f) and os.stat(f).st_mode & 0o111): man_targets[f] = {"synopsis": help_get(f), "sec": 1} # Check that all the expected .rst files are in doc/ and that no unexpected # .rst files are present. os.chdir(CH_BASE + "/doc") man_rsts = set(glob.glob("ch*.rst")) man_rsts_expected = { i + ".rst" for i in man_targets } lose_lots("unexpected .rst", man_rsts - man_rsts_expected) lose_lots("missing .rst", man_rsts_expected - man_rsts) # Construct a dictionary of synopses from the .rst files in doc. We’ll # compare these against the synopses in “man_targets”, which have either been # entered manually (for non-executables), or obtained from the help message # (for executables). man_synopses = dict() for man in man_targets: m = re.search(r"^\s+(.+)$\n\n\n^Synopsis", open(man + ".rst").read(), re.MULTILINE) if (m is not None): man_synopses[man] = m[1] elif (man_targets[man]["synopsis"] == ""): # No synopsis expected. man_synopses[man] = "" # Check for missing or unexpected synopses. lose_lots("missing synopsis", set(man_targets) - set(man_synopses)) lose_lots("unexpected synopsis", set(man_synopses) - set(man_targets)) # Check for synopses that don’t match the expectation provided in # “man_targets”. lose_lots("bad synopsis in man page", { "%s: %s (expected: %s)" % (p, man_targets[p]["synopsis"]) for (p, s) in man_synopses.items() if ( p in man_targets and summary_unrest(s) != man_targets[p]["synopsis"]) and "deprecated" not in s.lower() }) # Check for “see also” entries in charliecloud.rst. sees = { m[0] for m in re.finditer(r"ch-[a-z0-9-.]+\([1-8]\)", open("charliecloud.rst").read()) } sees_expected = { i + "(%d)" % (man_targets[i]["sec"]) for i in man_targets } - {"charliecloud(7)"} lose_lots("unexpected see-also in charliecloud.rst", sees - sees_expected) lose_lots("missing see-also in charliecloud.rst", sees_expected - sees) # Check for consistency with “conf.py” conf = {} execfile("./conf.py", conf) for (docname, name, desc, authors, section) in conf["man_pages"]: if (docname != name): lose("conf.py: startdocname != name: %s != %s" % (docname, name)) if (len(authors) != 0): lose("conf.py: bad authors: %s: %s" % (name, authors)) if (name != "charliecloud"): if (section != man_targets[name]["sec"]): lose("conf.py: bad section: %s: %s != %d" % (name, section, man_targets[name]["sec"])) if (name not in man_targets): lose("conf.py: unexpected man page: %s" % name) elif (desc + "." != man_targets[name]["synopsis"] and "deprecated" not in desc.lower()): lose("conf.py: bad summary: %s: %s" % (name, desc)) # Check that all expected man pages are present, and *only* the expected man # pages. os.chdir(CH_BASE + "/doc/man") mans = set(glob.glob("*.[1-8]")) mans_expected = { i + ".%d" % (man_targets[i]["sec"]) for i in man_targets} lose_lots("unexpected man", mans - mans_expected) lose_lots("missing man", mans_expected - mans) try: execfile # Python 2 except NameError: # Python 3; provide our own. See: https://stackoverflow.com/questions/436198 def execfile(path, globals_): with open(path, "rb") as fp: code = compile(fp.read(), path, "exec") exec(code, globals_) # Get an executable’s synopsis from its help message. def help_get(prog): if (not os.path.isfile(prog)): lose("not a file: %s" % prog) try: out = subprocess.check_output(["./" + prog, "--help"], universal_newlines=True, stderr=subprocess.STDOUT) except Exception as x: lose("%s --help failed: %s" % (prog, str(x))) return None m = re.search(r"^(?:[Uu]sage:[^\n]+\n| +[^\n]+\n|\n)*([^\n]+)\n", out) if (m is None): lose("%s --help: no summary found" % prog) return None else: return m[1] def lose(msg): print(msg) global win win = False def lose_lots(prefix, losers): for loser in losers: lose("%s: %s" % (prefix, loser)) def summary_unrest(rest): t = rest t = t.replace(r":code:`", '"') t = t.replace(r"`", '"') return t if (__name__ == "__main__"): main() charliecloud-0.37/test/doctest-auto000077500000000000000000000006751457016721300174370ustar00rootroot00000000000000#!/bin/bash # Print (on stdout) BATS tests to run doctests on each file in lib/. set -e -o pipefail cat < 0] tests = [i for i in tests_nonempty if re.search(object_re, i.name_short)] print("will run %d/%d tests" % (len(tests), len(tests_nonempty))) # Run tests. out = "" def out_save(text): global out out += text runner = doctest.DocTestRunner(optionflags=( doctest.DONT_ACCEPT_TRUE_FOR_1 | doctest.ELLIPSIS)) for test in tests: print("%s ... " % test.name_short, end="") out = "" results = runner.run(test, out=out_save) assert (results.attempted == len(test.examples)) if (results.failed == 0): print("ok (%d examples)" % results.attempted) else: print("%d/%d failed" % (results.failed, results.attempted)) print(out) print("big L, stopping tests") sys.exit(1) # Summarize. print("all tests passed") charliecloud-0.37/test/fixtures/000077500000000000000000000000001457016721300167375ustar00rootroot00000000000000charliecloud-0.37/test/fixtures/README000066400000000000000000000001061457016721300176140ustar00rootroot00000000000000You can see what tests use the fixtures with "misc/grep 'fixtures/'". charliecloud-0.37/test/fixtures/empty-file000066400000000000000000000000001457016721300207230ustar00rootroot00000000000000charliecloud-0.37/test/force-auto.py.in000066400000000000000000000255241457016721300201210ustar00rootroot00000000000000#!%PYTHON_SHEBANG% # This script generates a BATS file to exercise “ch-image build --force” # across a variety of distributions. It’s used by Makefile.am. # # About each distribution, we remember: # # - base image name # - config name it should select # - scope # standard: all tests in standard scope # full: one test in standard scope, the rest in full # - any tests invalid for that distro # # For each distribution, we test these factors: # # - the value of --force (absent, fakeroot, seccomp) (3) # - whether or not preparation for --force is already done (2) # - commands that (4) # - don’t need --force, and fail # - don’t need --force, and succeed # - apparently need --force but in fact do not # - really do need --force # # This would appear to yield 3×2×4 = 24 tests per distribution. However: # # 1. We only try pre-prepared images for “really need” commands with --force # given, to save time, so it’s at most 9 potential tests. # # 2. The pre-preparation step doesn’t make sense for some distros or for # --force=seccomp. # # 3. We’ve not yet determined an “apparently need” command for some distros. # # Bottom line, the number of tests per distro varies. See the code below for # specific details. import abc import enum import inspect import sys @enum.unique class Scope(enum.Enum): STANDARD = "standard" FULL = "full" @enum.unique class Run(enum.Enum): UNNEEDED_FAIL = "unneeded fail" UNNEEDED_WIN = "unneeded win" FAKE_NEEDED = "fake needed" NEEDED = "needed" class Test(abc.ABC): arch_excludes = [] force_excludes = [] base = None config = None scope = Scope.FULL prep_run = None runs = { Run.UNNEEDED_FAIL: "false", Run.UNNEEDED_WIN: "true" } def __init__(self, run, force, preprep): self.run = run self.force = force self.preprep = preprep def __str__(self): preprep = "preprep" if self.preprep else "no preprep" return f"{self.base}, {self.run.value}, {self.force}, {preprep}" @property def build1_post_hook(self): return "" @property def build2_post_hook(self): return "" def as_grep_files(self, grep_files, image, invert=False): cmds = [] for (re, path) in grep_files: path = f"\"$CH_IMAGE_STORAGE\"/img/{image}/{path}" cmd = f"ls -lh {path}" if (invert): cmd = f"! ( {cmd} )" cmds.append(cmd) if (not invert): cmds.append(f"grep -Eq -- '{re}' {path}") return "\n".join(cmds) def as_outputs(self, outputs, invert=False): cmds = [] for out in outputs: out = f"echo \"$output\" | grep -Eq -- \"{out}\"" if (invert): out = f"! ( {out} )" cmds.append(out) return "\n".join(cmds) def as_runs(self, runs): return "\n".join("RUN %s" % run for run in runs) def test(self): # skip? if (self.preprep and not (self.force and self.run == Run.NEEDED)): print(f"\n# skip: {self}: not needed") return if (self.preprep and self.prep_run is None): print(f"\n# skip: {self}: no preprep command") return if (self.preprep and self.force == "seccomp"): print(f"\n# skip: {self}: no preprep for --force=seccomp") return if (self.force in self.force_excludes): print(f"\n# skip: {self}: --force=%s excluded" % self.force) return # scope if ( (self.scope == Scope.STANDARD or self.run == Run.NEEDED) and self.force != "fakeroot"): scope = "standard" else: scope = "full" # architecture excludes arch_excludes = "\n".join("arch_exclude %s" % i for i in self.arch_excludes) # build 1 to make prep-prepped image (e.g. install EPEL) if needed if (not self.preprep): build1 = "# skipped: no separate prep" build2_base = self.base else: build2_base = "tmpimg" build1 = f"""\ run ch-image -v build -t tmpimg -f - . << 'EOF' FROM {self.base} RUN {self.prep_run} EOF echo "$output" [[ $status -eq 0 ]] {self.build1_post_hook}""" # force force = "--force=%s" % (self.force) if self.force else "--force=none" # run command we’re testing try: run = self.runs[self.run] except KeyError: print(f"\n# skip: {self}: no run command") return # status if ( self.run == Run.UNNEEDED_FAIL or ( self.run == Run.NEEDED and not self.force )): status = 1 else: status = 0 # output outs = [] if (self.force == "fakeroot"): outs += [f"--force=fakeroot: will use: {self.config}"] if (self.run in { Run.NEEDED, Run.FAKE_NEEDED }): outs += ["--force=fakeroot: modified 1 RUN instructions"] out = self.as_outputs(outs) # emit the test print(f""" @test "ch-image --force: {self}" {{ scope {scope} {arch_excludes} # build 1: intermediate image for preparatory commands {build1} # build 2: image we're testing run ch-image -v build {force} -t tmpimg2 -f - . << 'EOF' FROM {build2_base} RUN {run} EOF echo "$output" [[ $status -eq {status} ]] {out} {self.build2_post_hook} }} """, end="") class EPEL_Mixin: # Mixin class for RPM distros where we want to pre-install EPEL. I think # this should maybe go away and just go into a _Red_Hat base class, i.e. # test all RPM distros with EPEL pre-installed, but this matches what # existed in 50_fakeroot.bats. Note the install-EPEL command is elsewhere. epel_outputs = ["(Updating|Installing).+: epel-release"] epel_greps = [("enabled=1", "/etc/yum.repos.d/epel*.repo")] @property def build1_post_hook(self): return "\n".join(["# validate EPEL installed", self.as_outputs(self.epel_outputs), self.as_grep_files(self.epel_greps, "tmpimg")]) @property def build2_post_hook(self): return "\n".join([ "# validate EPEL present if installed by build 1, gone if by --force", self.as_grep_files(self.epel_greps, "tmpimg2", not self.preprep)]) class RHEL7(Test): config = "rhel7" runs = { **Test.runs, **{ Run.FAKE_NEEDED: "yum install -y ed", Run.NEEDED: "yum install -y openssh" } } class T_CentOS_7(RHEL7, EPEL_Mixin): scope = Scope.STANDARD base = "centos:7" prep_run = "yum install -y epel-release" class RHEL8(Test): config = "rhel8" runs = { **Test.runs, **{ Run.FAKE_NEEDED: "dnf install -y" " --setopt=install_weak_deps=false ed", Run.NEEDED: "dnf install -y" " --setopt=install_weak_deps=false openssh" } } class T_RHEL_UBI_8(RHEL8): base = "registry.access.redhat.com/ubi8/ubi" class CentOS_8(RHEL8, EPEL_Mixin): prep_run = "dnf install -y epel-release" class T_CentOS_8_Stream(CentOS_8): # CentOS Stream pulls from quay.io per the CentOS wiki: # https://wiki.centos.org/FAQ/CentOSStream#What_artifacts_are_built.3F base = "quay.io/centos/centos:stream8" #class T_CentOS_9_Stream(CentOS_8): # FIXME: fails importing GPG key # base = "quay.io/centos/centos:stream9" class T_Alma_8(CentOS_8): scope = Scope.STANDARD base = "almalinux:8" # :latest same as :8 as of 2022-03-01 class T_Rocky_8(CentOS_8): base = "rockylinux:8" # :latest same as :8 as of 2022-03-01 # With many of the following images, we test two versions of each image, the # latest version and the minimum supported version. “Minimum supported” here # meaning the minimum of (a) what’s available on Docker hub and we think won’t # vanish too quickly and (b) what we support with fakeroot. class Fedora(RHEL8): config = "fedora" class T_Fedora_26(Fedora): # We would prefer to test the lowest supported --force version, 24, # but the ancient version of dnf it has doesn't fail the transaction when # a package fails so we test with 26 instead. base = "fedora:26" class T_Fedora_34(Fedora): base = "fedora:34" # No worky as of Fedora 35; see issue #1163. class Fedora_Latest(Fedora): base = "fedora:latest" class Debian(Test): config = "debderiv" runs = { **Test.runs, **{ Run.NEEDED: " apt-get update" " && apt-get install -y openssh-client" } } class T_Debian_10(Debian): base = "debian:10" arch_excludes = ["ppc64le"] # base image unavailable class T_Debian_Latest(Debian): scope = Scope.STANDARD base = "debian:latest" class T_Ubuntu_16(Debian): base = "ubuntu:16.04" class T_Ubuntu_Latest(Debian): base = "ubuntu:latest" class SUSE(Test): config = "suse" runs = { **Test.runs, # No openssh packages seem to need --force. **{ Run.FAKE_NEEDED: "zypper install -y ed", Run.NEEDED: "zypper install -y dbus-1" } } # As of 2022-06-03, fails with “Signature verification failed for file # 'repomd.xml' from repository 'OSS Update'.”. Neither opensuse/archive:42.3 # nor opensuse/leap:42.3 work, though the latter has more architectures. #class T_OpenSUSE_42_3(SUSE): # base = "opensuse/archive:42.3" # arch_excludes = ["aarch64", "ppc64le"] # base only amd64 class T_OpenSUSE_Leap_15_0(SUSE): base = "opensuse/leap:15.0" class T_OpenSUSE_Leap_Latest(SUSE): base = "opensuse/leap:latest" # Arch has tons of old tags, versioned by date, on Docker Hub. However, only # :latest (or equivalently :base) is documented. class T_Arch_Latest(Test): config = "arch" base = "archlinux:latest" arch_excludes = ["aarch64", "ppc64le"] # base only amd64 force_excludes = ["seccomp"] # issue #1567 # pacman does not exit non-zero when installing openssh fails; work # around this bug by grepping the pacman log. runs = { **Test.runs, **{ Run.FAKE_NEEDED: "pacman -Syq --noconfirm ed; ! fgrep failed: /var/log/pacman.log", Run.NEEDED: "pacman -Syq --noconfirm openssh; ! fgrep failed: /var/log/pacman.log" } } class Alpine(Test): config = "alpine" # openssh does not need --force on Alpine runs = { **Test.runs, **{ Run.FAKE_NEEDED: "apk add ed", Run.NEEDED: "apk add dbus" } } class T_Alpine_39(Alpine): base = "alpine:3.9" class T_Alpine_Latest(Alpine): base = "alpine:latest" # main loop print("""\ # NOTE: This file is auto-generated. Do not modify. load common setup () { [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' [[ $CH_IMAGE_CACHE = enabled ]] || skip 'bucache enabled only' } """) # All classes starting with T_ get turned into a test. for (name, test) in (i for i in inspect.getmembers(sys.modules[__name__]) if i[0].startswith("T_")): for run in Run: for force in (None, "fakeroot", "seccomp"): for preprep in (False, True): test(run, force, preprep).test() charliecloud-0.37/test/make-auto.d/000077500000000000000000000000001457016721300171735ustar00rootroot00000000000000charliecloud-0.37/test/make-auto.d/build.bats.in000066400000000000000000000004171457016721300215540ustar00rootroot00000000000000source common.bash # for ShellCheck; removed by ch-test @test 'build %(tag)s' { scope %(scope)s # shellcheck disable=SC2086 build_ -t %(tag)s --file="%(path)s" "%(dirname)s" #sudo docker tag %(tag)s "%(tag)s:$ch_version_docker" builder_ok %(tag)s } charliecloud-0.37/test/make-auto.d/build_custom.bats.in000066400000000000000000000045441457016721300231530ustar00rootroot00000000000000source common.bash # for ShellCheck; removed by ch-test @test 'custom build %(tag)s' { scope %(scope)s out="${ch_tardir}/%(tag)s" pq="${ch_tardir}/%(tag)s.pq_missing" workdir="${ch_tardir}/%(tag)s.tmp" rm -f "$pq" mkdir "$workdir" cd "%(dirname)s" run ./%(basename)s "$PWD" "$out" "$workdir" echo "$output" if [[ $status -eq 0 ]]; then if [[ -f ${out}.tar.gz || -f ${out}.tar.xz ]]; then # tarball # Validate exactly one tarball came out. tarballs=( "$out".tar.* ) [[ ${#tarballs[@]} -eq 1 ]] tarball=${tarballs[0]} # Convert to SquashFS if needed. if [[ $CH_TEST_PACK_FMT = squash* ]]; then # With the centos7xz image, we run into permission errors if we # try to use the tar “--xattrs-include” option. Using strace(1), # we determined that with the xattrs option specified, tar first # calls mknodat(2) to create a file with permissions 000, then # openat(2) on the same file, which fails with EACCESS. Without # the xattrs option, the file is created by a call to openat(2) # with the O_CREAT flag (rather than mknodat(2)), so the # permission error is avoided. (See # https://savannah.gnu.org/support/index.php?110903). if [[ $tarball = *centos7xz* ]]; then xattrs_arg=--no-xattrs else xattrs_arg= fi ch-convert $xattrs_arg "$tarball" "${tarball/tar.?z/sqfs}" rm "$tarball" fi elif [[ -d $out ]]; then # directory case $CH_TEST_PACK_FMT in squash-*) ext=sqsh ;; tar-unpack) ext=tar.gz ;; *) false # unknown format ;; esac ch-convert "$out" "${out}.${ext}" else false # unknown format fi fi rm -Rf --one-file-system "$out" "$workdir" if [[ $status -eq 65 ]]; then touch "$pq" rm -Rf --one-file-system "$out".tar.{gz,xz} skip 'prerequisites not met' else return "$status" fi } charliecloud-0.37/test/make-auto.d/builder_to_archive.bats.in000066400000000000000000000007411457016721300243060ustar00rootroot00000000000000source common.bash # for ShellCheck; removed by ch-test @test 'builder to archive %(tag)s' { scope %(scope)s case $CH_TEST_PACK_FMT in squash*) ext=sqfs ;; tar-unpack) ext=tar.gz ;; *) false # unknown format ;; esac archive=${ch_tardir}/%(tag)s.${ext} ch-convert -i "$CH_TEST_BUILDER" '%(tag)s' "$archive" archive_grep "$archive" archive_ok "$archive" } charliecloud-0.37/test/make-auto.d/unpack.bats.in000066400000000000000000000002711457016721300217340ustar00rootroot00000000000000source common.bash # for ShellCheck; removed by ch-test @test 'unpack %(tag)s' { scope %(scope)s prerequisites_ok %(tag)s ch_tag=%(tag)s unpack_img_all_nodes "true" } charliecloud-0.37/test/make-perms-test.py.in000066400000000000000000000144311457016721300210660ustar00rootroot00000000000000#!%PYTHON_SHEBANG% # This script sets up a test directory for testing filesystem permissions # enforcement in UDSS such as virtual machines and containers. It must be run # as root. For example: # # $ sudo ./make-perms-test /data/perms_test $USER nobody # $ ./fs_perms.py /data/perms_test/pass 2>&1 | egrep -v 'ok$' # d /data/perms_test/pass/ld.out-a~--- --- rwt mismatch # d /data/perms_test/pass/ld.out-r~--- --- rwt mismatch # f /data/perms_test/pass/lf.out-a~--- --- rw- mismatch # f /data/perms_test/pass/lf.out-r~--- --- rw- mismatch # RISK 4 mismatches in 1 directories # # In this case, there will be four mismatches because the symlinks are # expected to be invalid after the pass directory is attached to the UDSS. # # Roughly 3,000 permission settings are evaluated in order to check files and # directories against user, primary group, and supplemental group access. # # For files, we test read and write. For directories, read, write, and # traverse. Files are not tested for execute because it’s a more complicated # test (new process needed) and if readable, someone could simply make their # own executable copy. # # Compatibility: As of February 2016, this needs to be compatible with Python # 2.6 because that’s the highest version that comes with RHEL 6. We’re also # aiming to be source-compatible with Python 3.4+, but that’s untested. # # Help: http://python-future.org/compatible_idioms.html from __future__ import division, print_function, unicode_literals import grp import os import os.path import pwd import sys if (len(sys.argv) != 4): print('usage error (PEBKAC)', file=sys.stderr) sys.exit(1) FILE_PERMS = set([0, 2, 4, 6]) DIR_PERMS = set([0, 1, 2, 3, 4, 5, 6, 7]) ALL_PERMS = FILE_PERMS | DIR_PERMS FILE_CONTENT = 'gary' * 19 + '\n' testdir = os.path.abspath(sys.argv[1]) my_user = sys.argv[2] yr_user = sys.argv[3] me = pwd.getpwnam(my_user) you = pwd.getpwnam(yr_user) my_uid = me.pw_uid my_gid = me.pw_gid my_group = grp.getgrgid(my_gid).gr_name yr_uid = you.pw_uid yr_gid = you.pw_gid yr_group = grp.getgrgid(yr_gid).gr_name # find an arbitrary supplemental group for my_user my_group2 = None my_gid2 = None for g in grp.getgrall(): if (my_user in g.gr_mem and g.gr_name != my_group): my_group2 = g.gr_name my_gid2 = g.gr_gid break if (my_group2 is None): print("couldn't find supplementary group for %s" % my_user, file=sys.stderr) sys.exit(1) if (my_gid == yr_gid or my_gid == my_gid2): print('%s and %s share a group' % (my_user, yr_user), file=sys.stderr) sys.exit(1) print('''\ test directory: %(testdir)s me: %(my_user)s %(my_uid)d you: %(yr_user)s %(yr_uid)d my primary group: %(my_group)s %(my_gid)d my supp. group: %(my_group2)s %(my_gid2)d your primary group: %(yr_group)s %(yr_gid)d ''' % locals()) def set_perms(name, uid, gid, mode): os.chown(name, uid, gid) os.chmod(name, mode) def symlink(src, link_name): if (not os.path.exists(src)): print('link target does not exist: %s' % src) sys.exit(1) os.symlink(src, link_name) class Test(object): def __init__(self, uid, gid, up, gp, op, name=None): self.uid = uid self.group = grp.getgrgid(gid).gr_name self.gid = gid self.user = pwd.getpwuid(uid).pw_name self.up = up self.gp = gp self.op = op self.name_override = name self.mode = up << 6 | gp << 3 | op # Which permission bits govern? if (self.uid == my_uid): self.p = self.up elif (self.gid in (my_gid, my_gid2)): self.p = self.gp else: self.p = self.op @property def name(self): if (self.name_override is not None): return self.name_override else: return ('%s.%s-%s.%03o~%s' % (self.type_, self.user, self.group, self.mode, self.expect)) @property def valid(self): return (all(x in self.valid_perms for x in (self.up, self.gp, self.op))) def write(self): if (not self.valid): return 0 self.write_real() set_perms(self.name, self.uid, self.gid, self.mode) return 1 class Test_Directory(Test): type_ = 'd' valid_perms = DIR_PERMS @property def expect(self): return ( ('r' if (self.p & 4) else '-') + ('w' if (self.p & 3 == 3) else '-') + ('t' if (self.p & 1) else '-')) def write_real(self): os.mkdir(self.name) # Create a file R/W by me, for testing traversal. file_ = self.name + '/file' with open(file_, 'w') as fp: fp.write(FILE_CONTENT) set_perms(file_, my_uid, my_uid, 0o660) class Test_File(Test): type_ = 'f' valid_perms = FILE_PERMS @property def expect(self): return ( ('r' if (self.p & 4) else '-') + ('w' if (self.p & 2) else '-') + '-') def write_real(self): with open(self.name, 'w') as fp: fp.write(FILE_CONTENT) try: os.mkdir(testdir) except OSError as x: print("can't mkdir %s: %s" % (testdir, str(x))) sys.exit(1) set_perms(testdir, my_uid, my_gid, 0o770) os.chdir(testdir) Test_Directory(my_uid, my_gid, 7, 7, 0, 'nopass').write() os.chdir('nopass') Test_Directory(my_uid, my_gid, 7, 7, 0, 'dir').write() Test_File(my_uid, my_gid, 6, 6, 0, 'file').write() os.chdir('..') Test_Directory(my_uid, my_gid, 7, 7, 0, 'pass').write() os.chdir('pass') ct = 0 for uid in (my_uid, yr_uid): for gid in (my_gid, my_gid2, yr_gid): if (uid == my_uid and gid == my_gid): # Files owned by my_uid:my_gid are not a meaningful access control # test; check the documentation for why. continue for up in ALL_PERMS: for gp in ALL_PERMS: for op in ALL_PERMS: f = Test_File(uid, gid, up, gp, op) #print(f.name) ct += f.write() d = Test_Directory(uid, gid, up, gp, op) #print(d.name) ct += d.write() #print(ct) symlink('f.%s-%s.600~rw-' % (my_user, yr_group), 'lf.in~rw-') symlink('d.%s-%s.700~rwt' % (my_user, yr_group), 'ld.in~rwt') symlink('%s/nopass/file' % testdir, 'lf.out-a~---') symlink('%s/nopass/dir' % testdir, 'ld.out-a~---') symlink('../nopass/file', 'lf.out-r~---') symlink('../nopass/dir', 'ld.out-r~---') print("created %d files and directories" % ct) charliecloud-0.37/test/old-storage000077500000000000000000000067171457016721300172470ustar00rootroot00000000000000#!/bin/bash set -e -o pipefail ### Functions fatal_msg () { printf '💀💀💀 error: %s 💀💀💀\n' "$1" 1>&2 } usage () { cat <<'EOF' 1>&2 Test that ch-image can upgrade storage directories generated by old versions, i.e., unpack each old storage directory from a tarball, then try to upgrade it and run the test suite. Usage: $ old-storage.sh SCOPE WORKDIR (DIR|TAR1) [TAR2 ...] WORKDIR is where ch-image storage are unpacked. It must be empty and have enough space for one storage directory. TARn are old storage directories archived as tarballs. These must have certain properties: 1. Named storage-$VERSION.$ARCH.tar.gz, e.g. “storage-0.27.x86_64.tar.gz”. (Note: $ARCH is not currently validated but may be in the future.) 2. Is a tarbomb, e.g.: $ tar tf storage-0.26.x86_64.tar.gz | head -3 ./ ./dlcache/ ./dlcache/alpine:3.9.fat.json 3. The result of: $ rm -Rf $(ch-image storage-path) && ch-test -b ch-image build or equivalent, though mostly rather than fully successful tests are fine. Note: Best practice is to generate the tarball at the time of release, because old test suites often don’t pass due to changing source images. If a directory DIR is given instead, use all tarballs in that directory that have last-modified dates less than one year in the past. (See #1507.) EOF } INFO () { printf '📣 %s\n' "$1" } ### Parse arguments & setup if [[ $1 = --help || $1 = -? ]]; then usage exit 0 fi if [[ $# -lt 3 ]]; then usage exit 1 fi scope=$1; shift workdir=$1; shift trap 'fatal_msg "command failed on line $LINENO"' ERR PATH=$(cd "$(dirname "$0")" && pwd)/../bin:$PATH export PATH if [[ -d $1 ]]; then oldtars=$(find "$1" -mindepth 1 -mtime -365 -print | sort) else oldtars=$(printf '%s ' "$@") # https://www.shellcheck.net/wiki/SC2124 fi summary='' pass_ct=0 fail_ct=0 ### Main loop INFO "workdir: $workdir" for oldtar in $oldtars; do base=$(basename "$oldtar") base=${base%.*} # rm .gz base=${base%.*} # rm .tar base=${base%.*} # rm architecture storage=${workdir}/${base} INFO "old tar: $oldtar ($(stat -c %y "$oldtar"))" INFO "unpacking: $storage" [[ -d $workdir ]] [[ ! -d $storage ]] mkdir "$storage" tar xf "$oldtar" -C "$storage" [[ -d $storage ]] export CH_IMAGE_STORAGE=$storage INFO "unpacked: $(du -sh "$storage" | cut -f1) bytes in $(du --inodes -sh "$storage" | cut -f1) inodes" case ${storage#*-} in 0.29|0.30|0.31) INFO "working around bug fixed by PR #1662" (cd "$storage"/bucache && git branch -D alpine+latest) ;; esac INFO "upgrading" ch-image list # These are images that contain references to things on the internet that # go out of date, so builds based on them fail. Re-pull them to get a # current base image. ch-image pull archlinux:latest INFO "testing" if (ch-test -b ch-image --pedantic=no -s "$scope" all); then pass_ct=$((pass_ct + 1)) summary+="😁 ${oldtar}: PASS"$'\n' else fail_ct=$((fail_ct + 1)) summary+="🤦 ${oldtar}: FAIL"$'\n' fi INFO "deleting: $storage" rm -Rf --one-file-system "$storage" [[ ! -d $storage ]] done cat <" % self.name @classmethod def parse(cls, lineno, end_lineno): lineno_found = lineno - 1 for line in lines[lineno:end_lineno+1]: lineno_found += 1 m = re.search(r"^ *##+ (.+) ##+", line) if (m is not None): return cls(lineno_found, lineno_found, m[1]) return None def FAIL(lineno, msg): print("😭 %d: %s" % (lineno, msg)) global error_ct error_ct += 1 def FAILO(before, after): FAIL(before.lineno, ( "%s precedes %s (line %d) but should follow" % (before.name, after.name, after.lineno))) def inherits(child, parent): "Return True if child inherits from parent (perhaps indirectly), else False." # FIXME: Considers only first parent. if (not ( isinstance(child, ast.ClassDef) and isinstance(parent, ast.ClassDef))): return False # not both classes while True: if (len(child.bases) < 1): return False # no base clase elif (not isinstance(child.bases[0], ast.Name)): return False # dot in name so can’t be in this file elif (child.bases[0].id == parent.name): return True # found a match else: # Move child up one generation. All we have here is the name, so # search all the module statements for a matching class. 🤪 child_new = None for stmt in tree.body: if ( isinstance(stmt, ast.ClassDef) and stmt.name == child.bases[0].id): child_new = stmt break if (child_new is not None): child = child_new else: return False # not found, so parent must be in another file def parse(statements_raw): statements = [Section(0, 0, "UNNAMED")] # Add in section comments. for i in range(len(statements_raw) - 1): s_cur = statements_raw[i] s_next = statements_raw[i+1] statements.append(s_cur) if (s_cur.end_lineno + 1 != s_next.lineno): # Gap between statements parsed by ast.parse(). Maybe it’s a section # comment? section = Section.parse(s_cur.end_lineno + 1, s_next.lineno - 1) if (section is not None): statements.append(section) try: statements.append(s_next) except UnboundLocalError: pass # empty body # Remove statement types exempt from ordering. Iterate backwards so we can # modify in-place. for i in reversed(range(len(statements))): if ( statements[i].__class__ not in CLASS_ORDER and statements[i].__class__ != Section): del statements[i] # Remove statements special-case exempt from ordering. for i in reversed(range(len(statements))): if (re.search(r"# +👻", lines[statements[i].lineno]) is not None): if (isinstance(statements[i], ast.ClassDef)): FAIL("%s: no exemptions for classes" % statements[i].name) else: del statements[i] # Set statement names. for stmt in statements: if (isinstance(stmt, ast.Import)): if (len(stmt.names) != 1): FAIL(stmt.lineno, "too many imports on same line") stmt.name = stmt.names[0].name elif (isinstance(stmt, ast.Assign)): if (isinstance(stmt.targets[0], ast.Name)): stmt.name = stmt.targets[0].id elif (isinstance(stmt.targets[0], (ast.Tuple, ast.List))): stmt.name = ",".join(i.id for i in stmt.targets[0].elts) elif (isinstance(stmt, (Section, ast.ClassDef, ast.FunctionDef))): pass # already has name attribute else: assert False, "invalid statement type: %s" % type(stmt) # Done. return statements def sort_key(stmt): """Return a tuple for sort order: (int: statement type, int: statement subtype, str: type-specific, str: object name)""" ret = list() ret.append(CLASS_ORDER[stmt.__class__]) if (isinstance(stmt, ast.Import)): name = stmt.name.split(".")[0] if (name == "charliecloud"): ret.append(3) elif (name in STDLIB_MODULES): ret.append(1) elif (name in CH_MODULES): ret.append(4) else: # neither standard library nor Charliecloud ret.append(2) ret.append(stmt.name) elif (isinstance(stmt, ast.Assign)): pass # all assignments are equal elif (isinstance(stmt, ast.FunctionDef)): dl = stmt.decorator_list try: #print(ast.dump(stmt, indent=2)) try: decorator = dl[0].id except AttributeError: if (isinstance(dl[0], ast.Attribute)): # dotted decorator if (dl[0].attr == "setter"): decorator = "property" # setter is close enough else: decorator = dl[0].value.id + "." + dl[0].attr except IndexError: decorator = None if (stmt.name == "__init__"): ret.append(1) elif (re.search(r"^__.+__$", stmt.name)): ret.append(4) elif (decorator == "staticmethod"): ret.append(2) elif (decorator == "classmethod"): ret.append(3) elif (decorator == "property"): ret.append(5) else: ret.append(6) ret.append(stmt.name) elif (isinstance(stmt, ast.ClassDef)): # NOTE: This does *not* consider inheritance relationships. That is # special cased in the main loop. ret.append(stmt.name) else: assert False, "unreachable code reached" return tuple(ret) def validate(statements): statements = parse(statements) # validate section order sections = [stmt for stmt in statements if isinstance(stmt, Section) and stmt.name in SECTION_ORDER] for i in range(len(sections) - 1): before = sections[i] for j in range(i + 1, len(sections)): after = sections[j] if (SECTION_ORDER[before.name] >= SECTION_ORDER[after.name]): FAILO(before, after) if ( (before.name == "Classes" and "classes" in after.name.lower()) or (after.name == "Classes" and "classes" in before.name.lower())): FAIL(before.lineno, ("§%s not allowed if §%s (line %d) present" % (before.name, after.name, after.lineno))) # build statements within sections sections = dict() # retains insertion order ≥3.6 for stmt in statements: if (isinstance(stmt, Section)): sections[stmt.name] = list() section_cur = stmt.name else: sections[section_cur].append(stmt) # Validate order within each section. This uses an O(n²) all-to-all # comparison so we can consider class inheritance. for (section_name, section) in sections.items(): #print("analyzing section %s" % section_name) for i in range(len(section) - 1): before = section[i] for j in range(i + 1, len(section)): after = section[j] if (not ( inherits(after, before) or sort_key(before) <= sort_key(after))): #print(before.lineno, sort_key(before), sort_key(after)) FAILO(before, after) validate(tree.body) for stmt in tree.body: if (isinstance(stmt, ast.ClassDef)): validate(stmt.body) print("total errors: %d" % error_ct) sys.exit(int(error_ct != 0)) charliecloud-0.37/test/registry-config.yml000066400000000000000000000012131457016721300207210ustar00rootroot00000000000000# Sample registry configuration file. Used for CI testing. # # WARNING: Ports will be incorrect for mitmproxy. See HOWTO in Google Docs. version: 0.1 log: fields: service: registry storage: cache: blobdescriptor: inmemory filesystem: rootdirectory: /var/lib/registry auth: htpasswd: realm: i-lost-on-jeopardy path: /etc/docker/registry/htpasswd http: addr: :5000 headers: X-Content-Type-Options: [nosniff] X-Weird-Al: [Yankovic] tls: certificate: /etc/docker/registry/localhost.crt key: /etc/docker/registry/localhost.key health: storagedriver: enabled: true interval: 10s threshold: 3 charliecloud-0.37/test/run/000077500000000000000000000000001457016721300156725ustar00rootroot00000000000000charliecloud-0.37/test/run/build-rpms.bats000066400000000000000000000127221457016721300206270ustar00rootroot00000000000000load ../common setup () { [[ $CH_TEST_PACK_FMT = *-unpack ]] || skip 'need writeable image' [[ $CHTEST_GITWD ]] || skip "not in Git working directory" if ! command -v sphinx-build > /dev/null 2>&1 \ && ! command -v sphinx-build-3.6 > /dev/null 2>&1; then skip 'Sphinx is not installed' fi } @test 'build/install el7 RPMs' { scope full prerequisites_ok centos_7ch img=${ch_imgdir}/centos_7ch image_ok "$img" rm -rf --one-file-system "${BATS_TMPDIR}/rpmbuild" # Build and install RPMs into CentOS 7 image. (cd .. && packaging/fedora/build --install "$img" \ --rpmbuild="$BATS_TMPDIR/rpmbuild" HEAD) } @test 'check el7 RPM files' { scope full prerequisites_ok centos_7ch img=${ch_imgdir}/centos_7ch # Do installed RPMs look sane? run ch-run "$img" -- rpm -qa "charliecloud*" echo "$output" [[ $status -eq 0 ]] [[ $output = *'charliecloud-'* ]] [[ $output = *'charliecloud-builder'* ]] [[ $output = *'charliecloud-debuginfo-'* ]] [[ $output = *'charliecloud-doc'* ]] [[ $output = *'charliecloud-test-'* ]] run ch-run "$img" -- rpm -ql "charliecloud" echo "$output" [[ $status -eq 0 ]] [[ $output = *'/usr/bin/ch-run'* ]] [[ $output = *'/usr/lib/charliecloud/base.sh'* ]] [[ $output = *'/usr/share/man/man7/charliecloud.7.gz'* ]] run ch-run "$img" -- rpm -ql "charliecloud-builder" echo "$output" [[ $status -eq 0 ]] [[ $output = *'/usr/bin/ch-image'* ]] [[ $output = *'/usr/lib/charliecloud/charliecloud.py'* ]] run ch-run "$img" -- rpm -ql "charliecloud-debuginfo" echo "$output" [[ $status -eq 0 ]] [[ $output = *'/usr/lib/debug/usr/bin/ch-run.debug'* ]] [[ $output = *'/usr/lib/debug/usr/libexec/charliecloud/test/sotest/lib/libsotest.so.1.0.debug'* ]] run ch-run "$img" -- rpm -ql "charliecloud-test" echo "$output" [[ $status -eq 0 ]] [[ $output = *'/usr/bin/ch-test'* ]] [[ $output = *'/usr/libexec/charliecloud/test/Build.centos7xz'* ]] [[ $output = *'/usr/libexec/charliecloud/test/sotest/lib/libsotest.so.1.0'* ]] run ch-run "$img" -- rpm -ql "charliecloud-doc" echo "$output" [[ $output = *'/usr/share/doc/charliecloud-'*'/html'* ]] [[ $output = *'/usr/share/doc/charliecloud-'*'/examples/lammps/Dockerfile'* ]] } @test 'remove el7 RPMs' { scope full prerequisites_ok centos_7ch img=${ch_imgdir}/centos_7ch # Uninstall to avoid interfering with the rest of the test suite. run ch-run -w "$img" -- rpm -v --erase charliecloud-test \ charliecloud-debuginfo \ charliecloud-doc \ charliecloud-builder \ charliecloud echo "$output" [[ $status -eq 0 ]] [[ $output = *'charliecloud-'* ]] [[ $output = *'charliecloud-debuginfo-'* ]] [[ $output = *'charliecloud-doc'* ]] [[ $output = *'charliecloud-test-'* ]] # All gone? run ch-run "$img" -- rpm -qa "charliecloud*" echo "$output" [[ $status -eq 0 ]] [[ $output = '' ]] } @test 'build/install el8 RPMS' { scope standard prerequisites_ok almalinux_8ch img=${ch_imgdir}/almalinux_8ch image_ok "$img" rm -Rf --one-file-system "${BATS_TMPDIR}/rpmbuild" # Build and install RPMs into AlmaLinux 8 image. (cd .. && packaging/fedora/build --install "$img" \ --rpmbuild="$BATS_TMPDIR/rpmbuild" HEAD) } @test 'check el8 RPM files' { scope standard prerequisites_ok almalinux_8ch img=${ch_imgdir}/almalinux_8ch # Do installed RPMs look sane? run ch-run "$img" -- rpm -qa "charliecloud*" echo "$output" [[ $status -eq 0 ]] [[ $output = *'charliecloud-'* ]] [[ $output = *'charliecloud-builder'* ]] [[ $output = *'charliecloud-debuginfo-'* ]] [[ $output = *'charliecloud-doc'* ]] run ch-run "$img" -- rpm -ql "charliecloud" echo "$output" [[ $status -eq 0 ]] [[ $output = *'/usr/bin/ch-run'* ]] [[ $output = *'/usr/lib/charliecloud/base.sh'* ]] [[ $output = *'/usr/share/man/man7/charliecloud.7.gz'* ]] run ch-run "$img" -- rpm -ql "charliecloud-builder" echo "$output" [[ $status -eq 0 ]] [[ $output = *'/usr/bin/ch-image'* ]] [[ $output = *'/usr/lib/charliecloud/charliecloud.py'* ]] run ch-run "$img" -- rpm -ql "charliecloud-debuginfo" echo "$output" [[ $status -eq 0 ]] [[ $output = *'/usr/lib/debug/usr/bin/ch-run'*'debug'* ]] run ch-run "$img" -- rpm -ql "charliecloud-doc" echo "$output" [[ $output = *'/usr/share/doc/charliecloud/html'* ]] [[ $output = *'/usr/share/doc/charliecloud/examples/lammps/Dockerfile'* ]] } @test 'remove el8 RPMs' { scope standard prerequisites_ok almalinux_8ch img=${ch_imgdir}/almalinux_8ch # Uninstall to avoid interfering with the rest of the test suite. run ch-run -w "$img" -- rpm -v --erase charliecloud-debuginfo \ charliecloud-doc \ charliecloud-builder \ charliecloud echo "$output" [[ $status -eq 0 ]] [[ $output = *'charliecloud-'* ]] [[ $output = *'charliecloud-debuginfo-'* ]] [[ $output = *'charliecloud-doc'* ]] # All gone? run ch-run "$img" -- rpm -qa "charliecloud*" echo "$output" [[ $status -eq 0 ]] [[ $output = '' ]] } charliecloud-0.37/test/run/ch-convert.bats000066400000000000000000000441551457016721300206260ustar00rootroot00000000000000load ../common # Testing strategy overview: # # The most efficient way to test conversion through all formats would be to # start with a directory, cycle through all the formats one at a time, with # directory being last, then compare the starting and ending directories. That # corresponds to visiting all the cells in the matrix below, starting from one # labeled “a”, ending in one labeled “b”, and skipping those labeled with a # dash. Also, if visit n is in column i, then the next visit n+1 must be in # row i. This approach does each conversion exactly once. # # output -> # | dir | ch-image | docker | squash | tar | # input +----------+----------+----------+----------+---------+ # | dir | — | a | a | a | a | # v ch-image | b | — | | | | # docker | b | | — | | | # squash | b | | | — | | # tar | b | | | | — | # +----------+----------+----------+----------+---------+ # # Because we start with a directory already available, this yields 5*5 - 5 - 1 # = 19 conversions. However, I was not able to figure out a traversal order # that would meet the constraints. # # Thus, we use the following algorithm. # # for every format i except dir: (4 iterations) # convert start_dir -> i # for every format j except dir: (4) # if i≠j: convert i -> j # convert j -> finish_dir # compare start_dir with finish_dir # # This yields 4 * (3*2 + 1*1) = 28 conversions, due to excess conversions to # dir. However, it can better isolate where the conversion went wrong, because # the chain is 3 conversions long rather than 19. # # The outer loop is unrolled into four separate tests to avoid having one test # that runs for two minutes. # This is a little goofy, because several of the tests need *all* the # builders. Thus, we (a) run only for builder ch-image but (b) # pedantic-require Docker to also be installed. setup () { scope standard [[ $CH_TEST_BUILDER = ch-image ]] || skip 'ch-image only' [[ $CH_TEST_PACK_FMT = *-unpack ]] || skip 'needs directory images' if ! command -v docker > /dev/null 2>&1; then pedantic_fail 'docker not found' fi } # Return success if directories $1 and $2 are recursively the same, failure # otherwise. This compares only metadata. False positives are possible if a # file’s content changes but the size and all other metadata stays the same; # this seems unlikely. # # We use a text diff of the two directory listings. Alternatives include: # # 1. “diff -qr --no-dereference”: compares file content, which we probably # don’t need, and I’m not sure about metadata. # # 2. “rsync -nv -aAX --delete "${1}/" "$2"”: does compare only metadata, but # hard to filter for symlink destination changes. # # The listings are retained for examination later if the test fails. compare () { echo "COMPARING ${1} to ${2}" compare-ls "$1" > "$BATS_TMPDIR"/compare-ls.1 compare-ls "$2" > "$BATS_TMPDIR"/compare-ls.2 diff -u "$BATS_TMPDIR"/compare-ls.1 "$BATS_TMPDIR"/compare-ls.2 # Ensure build cache metadata is not in $2. [[ ! -e ./.git ]] [[ ! -e ./.gitignore ]] [[ ! -e ./ch/git.pickle ]] } # This prints a not very nicely formatted recursive directory listing, with # metadata including xattrs. ACLs are included in the xattrs but are encoded # somehow, so you can see if they change but what exactly changed is an # exercise for the reader. We don’t use simple “ls -lR” because it only lists # the presence of ACLs and xattrs (+ or @ after the mode respectively); we # don’t use getfacl(1) because I couldn’t make it not follow symlinks and # getfattr(1) does the job, just more messily. # # Notes/Gotchas: # # 1. Seconds are omitted from timestamp because I couldn’t figure out how to # not include fractional seconds, which is often not preserved. # # 2. The image root directory tends to be volatile (e.g., number of links, # size), and it doesn’t matter much, so exclude it with “-mindepth 1”. # # 3. Also exclude several paths which are expected not to round-trip. # # 4. %n (number of links) is omitted from -printf format because ch-convert # does not round-trip hard links correctly. (They are split into multiple # independent files.) See issue #1310. # # sed(1) modifications (-e in order below): # # 1. Because ch-image changes absolute symlinks to relative using a sequence # of up-dirs (“..”), remove these sequences. # # 2. For the same reason, remove symlink file sizes (symlinks contain the # destination path). # # 3. Symlink timestamps seem not to be stable, so remove them. # # 4. Directory sizes also seem not to be stable. # # See also “ls_” in 50_rsync.bats. compare-ls () { cd "$1" || exit # to make -path reasonable find . -mindepth 1 \ \( -path ./.dockerenv \ -o -path ./ch \ -o -path ./run \) -prune \ -o -not \( -path ./.git \ -o -path ./ch/git.pickle \ -o -path ./dev \ -o -path ./etc \ -o -path ./etc/hostname \ -o -path ./etc/hosts \ -o -path ./etc/resolv.conf \ -o -path ./etc/resolv.conf.real \) \ -printf '/%P %y%s %g:%u %M %y%TF_%TH:%TM %l\n' \ -exec getfattr -dhm - {} \; \ | sed -E -e 's|(\.\./)+|/|' \ -e 's/ l[0-9]{1,3}/ lXX/' \ -e 's/ l[0-9_:-]{16}/ lXXXX-XX-XX_XX:XX/' \ -e 's/ d[0-9]{2,5}/ dXXXXX/' \ | LC_ALL=C sort cd - } # Kludge to cook up the right input and output descriptors for ch-convert. convert-img () { ct=$1 in_fmt=$2 out_fmt=$3; case $in_fmt in ch-image) in_desc=tmpimg ;; dir) in_desc=$ch_timg ;; docker) in_desc=tmpimg ;; podman) in_desc=tmpimg ;; tar) in_desc=${BATS_TMPDIR}/convert.tar.gz ;; squash) in_desc=${BATS_TMPDIR}/convert.sqfs ;; *) echo "unknown input format: $in_fmt" false ;; esac case $out_fmt in ch-image) out_desc=tmpimg ;; dir) out_desc=${BATS_TMPDIR}/convert.dir ;; docker) out_desc=tmpimg ;; podman) out_desc=tmpimg ;; tar) out_desc=${BATS_TMPDIR}/convert.tar.gz ;; squash) out_desc=${BATS_TMPDIR}/convert.sqfs ;; *) echo "unknown output format: $out_fmt" false ;; esac echo echo "CONVERT ${ct}: ${in_desc} ($in_fmt) -> ${out_desc} (${out_fmt})" delete "$out_fmt" "$out_desc" if [[ $in_fmt = ch-image && $CH_IMAGE_CACHE = enabled ]]; then # round-trip the input image through Git ch-image delete "$in_desc" ch-image undelete "$in_desc" fi ch-convert --no-clobber -v -i "$in_fmt" -o "$out_fmt" "$in_desc" "$out_desc" # Doing it twice doubles the time but also tests that both new conversions # and overwrite work. Hence, full scope only. if [[ $CH_TEST_SCOPE = full ]]; then ch-convert -v -i "$in_fmt" -o "$out_fmt" "$in_desc" "$out_desc" fi } delete () { fmt=$1 desc=$2 case $fmt in ch-image) ch-image delete "$desc" || true ;; dir) rm -Rf --one-file-system "$desc" ;; docker) docker_ rmi -f "$desc" ;; podman) podman_ rmi -f "$desc" || true ;; tar) rm -f "$desc" ;; squash) rm -f "$desc" ;; *) echo "unknown format: $fmt" false ;; esac } empty_dir_init () { rm -rf --one-file-system "$1" mkdir "$1" } # Test conversions dir -> $1 -> (all) -> dir. test_from () { end=${BATS_TMPDIR}/convert.dir ct=1 convert-img "$ct" dir "$1" for j in ch-image docker podman squash tar; do if [[ $1 != "$j" ]]; then ct=$((ct+1)) convert-img "$ct" "$1" "$j" fi ct=$((ct+1)) convert-img "$ct" "$j" dir image_ok "$end" compare "$ch_timg" "$end" chtest_fixtures_ok "$end" done } @test 'ch-convert: format inference' { # Test input only; output uses same code. Test cases match all the # criteria to validate the priority. We don’t exercise every possible # descriptor pattern, only those I thought had potential for error. # SquashFS run ch-convert -n ./foo:bar.sqfs out.tar echo "$output" [[ $status -eq 0 ]] [[ $output = *'input: squash'* ]] # tar run ch-convert -n ./foo:bar.tar out.sqfs echo "$output" [[ $status -eq 0 ]] [[ $output = *'input: tar'* ]] run ch-convert -n ./foo:bar.tgz out.sqfs echo "$output" [[ $status -eq 0 ]] [[ $output = *'input: tar'* ]] run ch-convert -n ./foo:bar.tar.Z out.sqfs echo "$output" [[ $status -eq 0 ]] [[ $output = *'input: tar'* ]] run ch-convert -n ./foo:bar.tar.gz out.sqfs echo "$output" [[ $status -eq 0 ]] [[ $output = *'input: tar'* ]] # directory run ch-convert -n ./foo:bar out.tar echo "$output" [[ $status -eq 0 ]] [[ $output = *'input: dir'* ]] # builders run ch-convert -n foo out.tar echo "$output" if command -v ch-image > /dev/null 2>&1; then [[ $status -eq 0 ]] [[ $output = *'input: ch-image'* ]] elif command -v docker > /dev/null 2>&1; then [[ $status -eq 0 ]] [[ $output = *'input: docker'* ]] elif command -v podman > /dev/null 2>&1; then [[ $status -eq 0 ]] [[ $output = *'input: podman'* ]] else [[ $status -eq 1 ]] [[ $output = *'no builder found' ]] fi } @test 'ch-convert: errors' { # same format run ch-convert -n foo.tar foo.tar.gz echo "$output" [[ $status -eq 1 ]] [[ $output = *'error: input and output formats must be different'* ]] # output directory not an image touch "${BATS_TMPDIR}/foo.tar" run ch-convert "${BATS_TMPDIR}/foo.tar" "$BATS_TMPDIR" echo "$output" [[ $status -eq 1 ]] [[ $output = *"error: exists but does not appear to be an image and is not empty: ${BATS_TMPDIR}"* ]] rm "${BATS_TMPDIR}/foo.tar" } @test 'ch-convert: --no-clobber' { # ch-image printf 'FROM alpine:3.17\n' | ch-image build -t tmpimg -f - "$BATS_TMPDIR" run ch-convert --no-clobber -o ch-image "$BATS_TMPDIR" tmpimg echo "$output" [[ $status -eq 1 ]] [[ $output = *"error: exists in ch-image storage, not deleting per --no-clobber: tmpimg" ]] # convert ch_timg into ch-image format ch-image delete timg || true if [[ $(stat -c %F "$ch_timg") = 'symbolic link' ]]; then # symlink to squash archive fmt="squash" else # directory fmt="dir" fi ch-convert -i "$fmt" -o ch-image "$ch_timg" timg # dir ch-convert -i ch-image -o dir timg "$BATS_TMPDIR/timg" run ch-convert --no-clobber -i ch-image -o dir timg "$BATS_TMPDIR/timg" echo "$output" [[ $status -eq 1 ]] [[ $output = *"error: exists, not deleting per --no-clobber: ${BATS_TMPDIR}/timg" ]] rm -Rf --one-file-system "${BATS_TMPDIR:?}/timg" # docker printf 'FROM alpine:3.17\n' | docker_ build -t tmpimg - run ch-convert --no-clobber -o docker "$BATS_TMPDIR" tmpimg echo "$output" [[ $status -eq 1 ]] [[ $output = *"error: exists in Docker storage, not deleting per --no-clobber: tmpimg" ]] # podman printf 'FROM alpine:3.17\n' | podman_ build -t tmpimg - run ch-convert --no-clobber -o podman "$BATS_TMPDIR" tmpimg echo "$output" [[ $status -eq 1 ]] [[ $output = *"error: exists in Podman storage, not deleting per --no-clobber: tmpimg" ]] # squash touch "${BATS_TMPDIR}/timg.sqfs" run ch-convert --no-clobber -i ch-image -o squash timg "$BATS_TMPDIR/timg.sqfs" echo "$output" [[ $status -eq 1 ]] [[ $output = *"error: exists, not deleting per --no-clobber: ${BATS_TMPDIR}/timg.sqfs" ]] rm "${BATS_TMPDIR}/timg.sqfs" # tar touch "${BATS_TMPDIR}/timg.tar.gz" run ch-convert --no-clobber -i ch-image -o tar timg "$BATS_TMPDIR/timg.tar.gz" echo "$output" [[ $status -eq 1 ]] [[ $output = *"error: exists, not deleting per --no-clobber: ${BATS_TMPDIR}/timg.tar.gz" ]] rm "${BATS_TMPDIR}/timg.tar.gz" } @test 'ch-convert: empty target dir' { empty=${BATS_TMPDIR}/test-empty ## setup source images ## # ch-image printf 'FROM alpine:3.17\n' | ch-image build -t tmpimg -f - "$BATS_TMPDIR" # docker printf 'FROM alpine:3.17\n' | docker_ build -t tmpimg - # podman printf 'FROM alpine:3.17\n' | podman_ build -t tmpimg - # squash touch "${BATS_TMPDIR}/tmpimg.sqfs" ch-convert -i ch-image -o squash tmpimg "$BATS_TMPDIR/tmpimg.sqfs" # tar ch-convert -i ch-image -o tar tmpimg "$BATS_TMPDIR/tmpimg.tar.gz" ## run test ## # ch-image empty_dir_init "$empty" run ch-convert -i ch-image -o dir tmpimg "$empty" echo "$output" [[ $status -eq 0 ]] [[ $output = *"using empty directory: $empty"* ]] # docker empty_dir_init "$empty" run ch-convert -i docker -o dir tmpimg "$empty" echo "$output" [[ $status -eq 0 ]] [[ $output = *"using empty directory: $empty"* ]] # podman empty_dir_init "$empty" run ch-convert -i podman -o dir tmpimg "$empty" echo "$output" [[ $status -eq 0 ]] [[ $output = *"using empty directory: $empty"* ]] # squash empty_dir_init "$empty" run ch-convert -i squash -o dir "$BATS_TMPDIR/tmpimg.sqfs" "$empty" echo "$output" [[ $status -eq 0 ]] [[ $output = *"using empty directory: $empty"* ]] # tar empty_dir_init "$empty" run ch-convert -i tar -o dir "$BATS_TMPDIR/tmpimg.tar.gz" "$empty" echo "$output" [[ $status -eq 0 ]] [[ $output = *"using empty directory: $empty"* ]] } @test 'ch-convert: pathological tarballs' { [[ $CH_TEST_PACK_FMT = tar-unpack ]] || skip 'tar mode only' out=${BATS_TMPDIR}/convert.dir # Are /dev fixtures present in tarball? (issue #157) present=$(tar tf "$ch_ttar" | grep -F deleteme) echo "$present" [[ $(echo "$present" | wc -l) -eq 2 ]] echo "$present" | grep -E '^img/dev/deleteme$' echo "$present" | grep -E '^img/mnt/dev/dontdeleteme$' # Convert to dir. ch-convert "$ch_ttar" "$out" image_ok "$out" chtest_fixtures_ok "$out" } # The next three tests are for issue #1241. @test 'ch-convert: permissions retained (dir)' { out=${BATS_TMPDIR}/convert.dir ch-convert timg "$out" ls -ld "$out"/maxperms_* [[ $(stat -c %a "${out}/maxperms_dir") = 1777 ]] [[ $(stat -c %a "${out}/maxperms_file") = 777 ]] } @test 'ch-convert: permissions retained (squash)' { squishy=${BATS_TMPDIR}/convert.sqfs out=${BATS_TMPDIR}/convert.dir ch-convert timg "$squishy" ch-convert "$squishy" "$out" ls -ld "$out"/maxperms_* [[ $(stat -c %a "${out}/maxperms_dir") = 1777 ]] [[ $(stat -c %a "${out}/maxperms_file") = 777 ]] } @test 'ch-convert: permissions retained (tar)' { tarball=${BATS_TMPDIR}/convert.tar.gz out=${BATS_TMPDIR}/convert.dir ch-convert timg "$tarball" ch-convert "$tarball" "$out" ls -ld "$out"/maxperms_* [[ $(stat -c %a "${out}/maxperms_dir") = 1777 ]] [[ $(stat -c %a "${out}/maxperms_file") = 777 ]] } @test 'ch-convert: b0rked xattrs' { # Check if test needs to be skipped touch "$BATS_TMPDIR/tmpfs_test" if ! setfattr -n user.foo -v bar "$BATS_TMPDIR/tmpfs_test" \ && [[ -z $GITHUB_ACTIONS ]]; then skip "xattrs unsupported in ${BATS_TMPDIR}" fi # b0rked: (adj) broken, messed up # # In this test, we create a tarball with “unusual” xattrs that we don’t want # to restore (i.e. a borked tarball), and try to convert it into a ch-image. [[ -n $CH_TEST_SUDO ]] || skip 'sudo required' cd "$BATS_TMPDIR" borked_img="borked_image" borked_file="${borked_img}/home/foo" borked_tar="borked.tgz" borked_out="borked_dir" rm -rf "$borked_img" "$borked_tar" "$borked_out" ch-image build -t tmpimg - <<'EOF' FROM alpine:3.17 RUN touch /home/foo EOF # convert image to dir and actually bork it ch-convert -i ch-image -o dir tmpimg "$borked_img" setfattr -n user.foo -v bar "$borked_file" sudo setfattr -n security.foo -v bar "$borked_file" sudo setfattr -n trusted.foo -v bar "$borked_file" setfacl -m "u:$USER:r" "$borked_file" # confirm it worked run sudo getfattr -dm - -- "$borked_file" echo "$output" [[ $status -eq 0 ]] [[ $output = *"# file: $borked_file"* ]] [[ $output = *'security.foo="bar"'* ]] [[ $output = *'trusted.foo="bar"'* ]] [[ $output = *'user.foo="bar"'* ]] run getfacl "$borked_file" echo "$output" [[ $status -eq 0 ]] [[ $output = *"user:$USER:r--"* ]] # tar it up sudo tar --xattrs-include='user.*' \ --xattrs-include='system.*' \ --xattrs-include='security.*' \ --xattrs-include='trusted.*' \ -czvf "$borked_tar" "$borked_img" ch-convert -i tar -o dir "$borked_tar" "$borked_out" run sudo getfattr -dm - -- "$borked_out/home/foo" echo "$output" [[ $status -eq 0 ]] [[ $output != *'security.foo="bar"'* ]] [[ $output != *'trusted.foo="bar"'* ]] [[ $output = *'user.foo="bar"'* ]] run getfacl "$borked_out/home/foo" echo "$output" [[ $status -eq 0 ]] [[ $output = *"user:$USER:r--"* ]] } @test 'ch-convert: dir -> ch-image -> X' { test_from ch-image } @test 'ch-convert: dir -> docker -> X' { test_from docker } @test 'ch-convert: dir -> podman -> X' { test_from podman } @test 'ch-convert: dir -> squash -> X' { test_from squash } @test 'ch-convert: dir -> tar -> X' { test_from tar } charliecloud-0.37/test/run/ch-fromhost.bats000066400000000000000000000333651457016721300210100ustar00rootroot00000000000000load ../common setup () { [[ $CH_TEST_PACK_FMT = *-unpack ]] || skip 'need writeable image' [[ $ch_libc = glibc ]] || skip 'glibc only' fi_provider_path=$FI_PROVIDER_PATH } fromhost_clean () { [[ $1 ]] # We used to delete only specific paths, but this turned into an unwieldy # mess of wildcards that obscured the original specificity purpose. rm -f "${1}/ld.so.cache" find "$1" -xdev \( \ -name 'libcuda*' \ -o -name 'libnvidia*' \ -o -name 'libfabric' \ -o -name libsotest-fi.so \ -o -name libsotest.so.1 \ -o -name libsotest.so.1.0 \ -o -name sotest \ -o -name sotest.c \ \) -print -delete ch-run -w "$1" -- /sbin/ldconfig # restore default cache fromhost_clean_p "$1" } fromhost_clean_p () { ch-run "$1" -- /sbin/ldconfig -p | grep -F libsotest && return 1 run fromhost_ls "$1" echo "$output" [[ $status -eq 0 ]] [[ -z $output ]] } fromhost_ls () { find "$1" -xdev -name '*sotest*' -ls } glibc_version_ok () { # glibc 2.34 and later is incompatible with older versions. If you try # to run a container with glibc < 2.34 on a host with glibc ≥ 2.34, you # get an error. This function ensures that host glibc and guest glibc are # compatible (see #1430). host=$(ldd --version | grep -oE '[0-9.]+[^.]$') guest=$(ch-run "$1" -- ldd --version | grep -oE '[0-9.]+[^.]$') # Check version compatibility. If host glibc ≥ 2.34 and guest glibc < 2.34, # skip the test. if [[ $(printf "%s\n2.34" "$host" | sort -V | head -1) = 2.34 \ && $(printf "%s\n2.34" "$guest" | sort -V | head -1) != 2.34 ]]; then skip "host glibc $host ≥ 2.34 > $guest" fi } @test 'ch-fromhost (CentOS)' { scope standard prerequisites_ok almalinux_8ch img=${ch_imgdir}/almalinux_8ch # check glibc version compatibility. glibc_version_ok "$img" libpath=$(ch-fromhost --print-lib "$img") echo "libpath: ${libpath}" # --file fromhost_clean "$img" ch-fromhost -v --file sotest/files_inferrable.txt "$img" fromhost_ls "$img" test -f "${img}/usr/bin/sotest" test -f "${img}${libpath}/libsotest.so.1.0" test -L "${img}${libpath}/libsotest.so.1" ch-run "$img" -- /sbin/ldconfig -p | grep -F libsotest ch-run "$img" -- sotest rm "${img}/usr/bin/sotest" rm "${img}${libpath}/libsotest.so.1.0" rm "${img}${libpath}/libsotest.so.1" ch-run -w "$img" -- /sbin/ldconfig fromhost_clean_p "$img" # --cmd ch-fromhost -v --cmd 'cat sotest/files_inferrable.txt' "$img" ch-run "$img" -- sotest # --path ch-fromhost -v --path sotest/bin/sotest \ --path sotest/lib/libsotest.so.1.0 \ "$img" ch-run "$img" -- sotest fromhost_clean "$img" # --cmd and --file ch-fromhost -v --cmd 'cat sotest/files_inferrable.txt' \ --file sotest/files_inferrable.txt "$img" ch-run "$img" -- sotest fromhost_clean "$img" # --dest ch-fromhost -v --file sotest/files_inferrable.txt \ --dest /mnt "$img" \ --path sotest/sotest.c ch-run "$img" -- sotest ch-run "$img" -- test -f /mnt/sotest.c fromhost_clean "$img" # --dest overrides inference, but ldconfig still run ch-fromhost -v --dest /lib \ --file sotest/files_inferrable.txt \ "$img" ch-run "$img" -- /lib/sotest fromhost_clean "$img" # --no-ldconfig ch-fromhost -v --no-ldconfig --file sotest/files_inferrable.txt "$img" [[ -f "${img}/usr/bin/sotest" ]] [[ -f "${img}${libpath}/libsotest.so.1.0" ]] [[ ! -L "${img}${libpath}/libsotest.so.1" ]] ch-run "$img" -- /sbin/ldconfig -p | grep -FL libsotest run ch-run "$img" -- sotest echo "$output" [[ $status -eq 127 ]] [[ $output = *'libsotest.so.1: cannot open shared object file'* ]] fromhost_clean "$img" # no --verbose ch-fromhost --file sotest/files_inferrable.txt "$img" ch-run "$img" -- sotest fromhost_clean "$img" # destination directory not writeable (#323) chmod -v u-w "${img}/mnt" ch-fromhost --dest /mnt --path sotest/sotest.c "$img" test -w "${img}/mnt" test -f "${img}/mnt/sotest.c" fromhost_clean "$img" } @test 'ch-fromhost (Debian)' { scope full prerequisites_ok debian_9ch img=${ch_imgdir}/debian_9ch # check glibc version compatibility. glibc_version_ok "$img" libpath=$(ch-fromhost --print-lib "$img") echo "libpath: ${libpath}" fromhost_clean "$img" ch-fromhost -v --file sotest/files_inferrable.txt "$img" fromhost_ls "$img" test -f "${img}/usr/bin/sotest" test -f "${img}/${libpath}/libsotest.so.1.0" test -L "${img}/${libpath}/libsotest.so.1" ch-run "$img" -- /sbin/ldconfig -p | grep -F libsotest ch-run "$img" -- sotest rm "${img}/usr/bin/sotest" rm "${img}/${libpath}/libsotest.so.1.0" rm "${img}/${libpath}/libsotest.so.1" rm "${img}/etc/ld.so.cache" fromhost_clean_p "$img" } @test 'ch-fromhost errors' { scope standard prerequisites_ok almalinux_8ch img=${ch_imgdir}/almalinux_8ch # no image run ch-fromhost --path sotest/sotest.c echo "$output" [[ $status -eq 1 ]] [[ $output = *'no image specified'* ]] fromhost_clean_p "$img" # image is not a directory run ch-fromhost --path sotest/sotest.c /etc/motd echo "$output" [[ $status -eq 1 ]] [[ $output = *'image not a directory: /etc/motd'* ]] fromhost_clean_p "$img" # two image arguments run ch-fromhost --path sotest/sotest.c "$img" foo echo "$output" [[ $status -eq 1 ]] [[ $output = *'duplicate image: foo'* ]] fromhost_clean_p "$img" # no files argument run ch-fromhost "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'empty file list'* ]] fromhost_clean_p "$img" # file that needs --dest but not specified run ch-fromhost -v --path sotest/sotest.c "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'no destination for: sotest/sotest.c'* ]] fromhost_clean_p "$img" # file with colon in name run ch-fromhost -v --path 'foo:bar' "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *"paths can't contain colon: foo:bar"* ]] fromhost_clean_p "$img" # file with newlines in name run ch-fromhost -v --path $'foo\nbar' "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *"no destination for: foo"* ]] fromhost_clean_p "$img" # --cmd no argument run ch-fromhost "$img" --cmd echo "$output" [[ $status -eq 1 ]] [[ $output = *'--cmd must not be empty'* ]] fromhost_clean_p "$img" # --cmd empty run ch-fromhost --cmd true "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'empty file list'* ]] fromhost_clean_p "$img" # --cmd fails run ch-fromhost --cmd false "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'command failed: false'* ]] fromhost_clean_p "$img" # --file no argument run ch-fromhost "$img" --file echo "$output" [[ $status -eq 1 ]] [[ $output = *'--file must not be empty'* ]] fromhost_clean_p "$img" # --file empty run ch-fromhost --file /dev/null "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'empty file list'* ]] fromhost_clean_p "$img" # --file does not exist run ch-fromhost --file /doesnotexist "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'/doesnotexist: No such file or directory'* ]] [[ $output = *'cannot read file: /doesnotexist'* ]] fromhost_clean_p "$img" # --path no argument run ch-fromhost "$img" --path echo "$output" [[ $status -eq 1 ]] [[ $output = *'--path must not be empty'* ]] fromhost_clean_p "$img" # --path does not exist run ch-fromhost --dest /mnt --path /doesnotexist "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'No such file or directory'* ]] [[ $output = *'cannot inject: /doesnotexist'* ]] fromhost_clean_p "$img" # --dest no argument run ch-fromhost "$img" --dest echo "$output" [[ $status -eq 1 ]] [[ $output = *'--dest must not be empty'* ]] fromhost_clean_p "$img" # --dest not an absolute path run ch-fromhost --dest relative --path sotest/sotest.c "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'not an absolute path: relative'* ]] fromhost_clean_p "$img" # --dest does not exist run ch-fromhost --dest /doesnotexist --path sotest/sotest.c "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'not a directory:'* ]] fromhost_clean_p "$img" # --dest is not a directory run ch-fromhost --dest /bin/sh --file sotest/sotest.c "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'not a directory:'* ]] fromhost_clean_p "$img" # image does not exist run ch-fromhost --file sotest/files_inferrable.txt /doesnotexist echo "$output" [[ $status -eq 1 ]] [[ $output = *'image not a directory: /doesnotexist'* ]] fromhost_clean_p "$img" # image specified twice run ch-fromhost --file sotest/files_inferrable.txt "$img" "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'duplicate image'* ]] fromhost_clean_p "$img" # ldconfig gives no shared library path (#324) # # (I don’t think this is the best way to get ldconfig to fail, but I # couldn’t come up with anything better. E.g., bad ld.so.conf or broken # .so’s seem to produce only warnings.) mv "${img}/sbin/ldconfig" "${img}/sbin/ldconfig.foo" run ch-fromhost --print-lib "$img" mv "${img}/sbin/ldconfig.foo" "${img}/sbin/ldconfig" echo "$output" [[ $status -eq 1 ]] [[ $output = *'empty path from ldconfig'* ]] fromhost_clean_p "$img" } @test 'ch-fromhost --path with libfabric' { scope full prerequisites_ok openmpi img=${ch_imgdir}/openmpi unset FI_PROVIDER_PATH ofidest=$(ch-fromhost --print-fi "$img") echo "provider dest: ${ofidest}" # The libsotest-fi.so is a dummy provider intended to exercise ch-fromhost # script logic. Succeed if ch-fromhost finds the container libfabric.so and # injects the libfabric-fi.so dummy executable in the directory /libfabric # where libfabric.so is found. # # Inferred dest from image libfabric.so. img=${ch_imgdir}/openmpi ofi=${CHTEST_DIR}/sotest/lib/libfabric/libsotest-fi.so run ch-fromhost -p "${ofi}" "$img" echo "$output" [[ $status -eq 0 ]] test -f "${img}/${ofidest}/libsotest-fi.so" fromhost_clean "$img" # host FI_PROVIDER_PATH with --dest export FI_PROVIDER_PATH=/usr/lib run ch-fromhost "$img" -d /usr/lib -p "$ofi" -v echo "$output" [[ $status -eq 0 ]] [[ $output = *'warn'*'FI_PROVIDER_PATH'* ]] test -f "${img}/usr/lib/libsotest-fi.so" fromhost_clean "$img" export FI_PROVIDER_PATH=$fi_provider_path } @test 'ch-fromhost --nvidia with GPU' { scope full prerequisites_ok nvidia command -v nvidia-container-cli >/dev/null 2>&1 \ || skip 'nvidia-container-cli not in PATH' img=${ch_imgdir}/nvidia # nvidia-container-cli --version (to make sure it’s linked correctly) nvidia-container-cli --version # Skip if nvidia-container-cli can’t find CUDA. run nvidia-container-cli list --binaries --libraries echo "$output" if [[ $status -eq 1 ]]; then if [[ $output = *'cuda error'* ]]; then skip "nvidia-container-cli can't find CUDA" fi false fi # --nvidia ch-fromhost -v --nvidia "$img" # nvidia-smi runs in guest ch-run "$img" -- nvidia-smi -L # nvidia-smi -L matches host host=$(nvidia-smi -L) echo "host GPUs:" echo "$host" guest=$(ch-run "$img" -- nvidia-smi -L) echo "guest GPUs:" echo "$guest" cmp <(echo "$host") <(echo "$guest") # --nvidia and --cmd fromhost_clean "$img" ch-fromhost --nvidia --file sotest/files_inferrable.txt "$img" ch-run "$img" -- nvidia-smi -L ch-run "$img" -- sotest # --nvidia and --file fromhost_clean "$img" ch-fromhost --nvidia --cmd 'cat sotest/files_inferrable.txt' "$img" ch-run "$img" -- nvidia-smi -L ch-run "$img" -- sotest # CUDA sample sample=/matrixMulCUBLAS # should fail without ch-fromhost --nvidia fromhost_clean "$img" run ch-run "$img" -- "$sample" echo "$output" [[ $status -eq 1 ]] [[ $output = *'CUDA error at'* ]] # should succeed with it fromhost_clean_p "$img" ch-fromhost --nvidia "$img" run ch-run "$img" -- "$sample" echo "$output" [[ $status -eq 0 ]] [[ $output =~ 'Comparing CUBLAS Matrix Multiply with CPU results: PASS' ]] } @test 'ch-fromhost --nvidia without GPU' { scope full prerequisites_ok nvidia img=${ch_imgdir}/nvidia # --nvidia should give a proper error whether or not nvidia-container-cli # is available. if ( command -v nvidia-container-cli >/dev/null 2>&1 ); then # nvidia-container-cli in $PATH run nvidia-container-cli list --binaries --libraries echo "$output" if [[ $status -eq 0 ]]; then # found CUDA; skip skip 'nvidia-container-cli found CUDA' else [[ $status -eq 1 ]] [[ $output = *'cuda error'* ]] run ch-fromhost -v --nvidia "$img" echo "$output" [[ $status -eq 1 ]] [[ $output = *'does this host have GPUs'* ]] fi else # nvidia-container-cli not in $PATH run ch-fromhost -v --nvidia "$img" echo "$output" [[ $status -eq 1 ]] r="nvidia-container-cli: (command )?not found" [[ $output =~ $r ]] [[ $output =~ 'nvidia-container-cli failed' ]] fi } charliecloud-0.37/test/run/ch-run_escalated.bats000066400000000000000000000052751457016721300217570ustar00rootroot00000000000000load ../common @test 'ch-run refuses to run if setgid' { scope standard ch_run_tmp=$BATS_TMPDIR/ch-run.setgid gid=$(id -g) gid2=$(id -G | cut -d' ' -f2) echo "gids: ${gid} ${gid2}" [[ $gid != "$gid2" ]] cp -a "$ch_runfile" "$ch_run_tmp" ls -l "$ch_run_tmp" chgrp "$gid2" "$ch_run_tmp" chmod g+s "$ch_run_tmp" ls -l "$ch_run_tmp" [[ -g $ch_run_tmp ]] run "$ch_run_tmp" --version echo "$output" [[ $status -eq 1 ]] [[ $output = *': please report this bug ('* ]] rm "$ch_run_tmp" } @test 'ch-run refuses to run if setuid' { scope standard [[ -n $ch_have_sudo ]] || skip 'sudo not available' ch_run_tmp=$BATS_TMPDIR/ch-run.setuid cp -a "$ch_runfile" "$ch_run_tmp" ls -l "$ch_run_tmp" sudo chown root "$ch_run_tmp" sudo chmod u+s "$ch_run_tmp" ls -l "$ch_run_tmp" [[ -u $ch_run_tmp ]] run "$ch_run_tmp" --version echo "$output" [[ $status -eq 1 ]] [[ $output = *': please report this bug ('* ]] sudo rm "$ch_run_tmp" } @test 'ch-run as root: --version and --test' { scope standard [[ -n $ch_have_sudo ]] || skip 'sudo not available' sudo "$ch_runfile" --version sudo "$ch_runfile" --help } @test 'ch-run as root: run image' { scope standard # Running an image should work as root, but it doesn’t, and I'm not sure # why, so skip this test. This fails in the test suite with: # # ch-run: couldn’t resolve image path: No such file or directory (ch-run.c:139:2) # # but when run manually (with same arguments?) it fails differently with: # # $ sudo bin/ch-run $ch_imgdir/chtest -- true # ch-run: [...]/chtest: Permission denied (ch-run.c:195:13) # skip 'issue #76' sudo "$ch_runfile" "$ch_timg" -- true } @test 'ch-run as root: root with non-zero gid refused' { scope standard [[ -n $ch_have_sudo ]] || skip 'sudo not available' if ! (sudo -u root -g "$(id -gn)" true); then # Allowing sudo to user root but group non-root is an unusual # configuration. You need e.g. “%foo ALL=(ALL:ALL)” instead of the # more common “%foo ALL=(ALL)”. See issue #485. pedantic_fail 'sudo not configured for user root and group non-root' fi run sudo -u root -g "$(id -gn)" "$ch_runfile" -v --version echo "$output" [[ $status -eq 1 ]] [[ $output = *'please report this bug ('* ]] } @test 'non-setuid fusermount3' { [[ $CH_TEST_PACK_FMT == squash-mount ]] || skip 'squash-mount format only' if [[ -u $(command -v fusermount3) ]]; then ls -lh "$(command -v fusermount3)" pedantic_fail 'fusermount3(1) is setuid' fi true # other tests validate it actually works } charliecloud-0.37/test/run/ch-run_isolation.bats000066400000000000000000000037031457016721300220250ustar00rootroot00000000000000load ../common @test 'mountns id differs' { scope full host_ns=$(stat -Lc '%i' /proc/self/ns/mnt) echo "host: ${host_ns}" guest_ns=$(ch-run "$ch_timg" -- stat -Lc %i /proc/self/ns/mnt) echo "guest: ${guest_ns}" [[ -n $host_ns && -n $guest_ns && $host_ns -ne $guest_ns ]] } @test 'userns id differs' { scope full host_ns=$(stat -Lc '%i' /proc/self/ns/user) echo "host: ${host_ns}" guest_ns=$(ch-run "$ch_timg" -- stat -Lc %i /proc/self/ns/user) echo "guest: ${guest_ns}" [[ -n $host_ns && -n $guest_ns && $host_ns -ne $guest_ns ]] } @test 'distro differs' { scope full # This is a catch-all and a bit of a guess. Even if it fails, however, we # get an empty string, which is fine for the purposes of the test. host_distro=$( cat /etc/os-release /etc/*-release /etc/*_version \ | grep -Em1 '[A-Za-z] [0-9]' \ | sed -r 's/^(.*")?(.+)(")$/\2/') echo "host: ${host_distro}" guest_expected='Alpine Linux v3.9' echo "guest expected: ${guest_expected}" if [[ $host_distro = "$guest_expected" ]]; then pedantic_fail 'host matches expected guest distro' fi guest_distro=$(ch-run "$ch_timg" -- \ cat /etc/os-release \ | grep -F PRETTY_NAME \ | sed -r 's/^(.*")?(.+)(")$/\2/') echo "guest: ${guest_distro}" [[ $guest_distro = "$guest_expected" ]] [[ $guest_distro != "$host_distro" ]] } @test 'user and group match host' { scope full host_uid=$(id -u) guest_uid=$(ch-run "$ch_timg" -- id -u) [[ $host_uid = "$guest_uid" ]] host_pgid=$(id -g) guest_pgid=$(ch-run "$ch_timg" -- id -g) [[ $host_pgid = "$guest_pgid" ]] host_username=$(id -un) guest_username=$(ch-run "$ch_timg" -- id -un) [[ $host_username = "$guest_username" ]] host_pgroup=$(id -gn) guest_pgroup=$(ch-run "$ch_timg" -- id -gn) [[ $host_pgroup = "$guest_pgroup" ]] } charliecloud-0.37/test/run/ch-run_join.bats000066400000000000000000000367471457016721300210010ustar00rootroot00000000000000load ../common setup () { scope standard } ipc_clean () { rm -v /dev/shm/*ch-run* } ipc_clean_p () { sem="$(find /dev/shm -maxdepth 1 -name '*ch-run*')" echo "$sem" 1>&2 [[ -z $sem ]] } joined_ok () { # parameters proc_ct_total=$1 # total number of processes peer_ct_node=$2 # size of each peer group (peers per node) namespace_ct=$3 # number of different namespace IDs status=$4 # exit status output="$5" # output echo "$output" # exit success printf ' exit status: ' 1>&2 if [[ $status -eq 0 ]]; then printf 'ok\n' 1>&2 else printf 'fail (%d)\n' "$status" 1>&2 return 1 fi # number of processes printf ' process count; expected %d: ' "$proc_ct_total" 1>&2 proc_ct_found=$(echo "$output" | grep -Ec 'join: 1 [0-9]+ [0-9a-z]+') if [[ $proc_ct_total -eq "$proc_ct_found" ]]; then printf 'ok\n' else printf 'fail (%d)\n' "$proc_ct_found" 1>&2 return 1 fi # number of peers printf ' peer group size; expected %d: ' "$peer_ct_node" 1>&2 peer_cts=$( echo "$output" \ | sed -rn 's/^ch-run\[[0-9]+\]: join: 1 ([0-9]+) .+$/\1/p') peer_ct_found=$(echo "$peer_cts" | sort -u) peer_cts_found=$(echo "$peer_ct_found" | wc -l) if [[ $peer_cts_found -ne 1 ]]; then printf 'fail (%d different counts reported)\n' "$peer_cts_found" 1>&2 return 1 fi if [[ $peer_ct_found -eq "$peer_ct_node" ]]; then printf 'ok\n' 1>&2 else printf 'fail (%d)\n' "$peer_ct_found" 1>&2 return 1 fi # correct number of namespace IDs for i in /proc/self/ns/*; do printf ' namespace count; expected %d: %s: ' "$namespace_ct" "$i" 1>&2 namespace_ct_found=$( echo "$output" \ | grep -E "^${i}:" \ | sort -u \ | wc -l) if [[ $namespace_ct -eq "$namespace_ct_found" ]]; then printf 'ok\n' 1>&2 else printf 'fail (%d)\n' "$namespace_ct_found" 1>&2 return 1 fi done } sleepcat () { # Wait up to $1 seconds for file $2 to appear, then wait one more second, # then cat the file. Return successfully whether or not the file appears # and was catted. timeout=$1 path=$2 for (( i=0; i < timeout; i++ )); do if [[ -f $path ]]; then sleep 1 # wait for write to complete cat "$path" return 0 fi sleep 1 done echo "timed out waiting for $path to appear" 1>&2 return 0 } # Unset environment variables that might be used. unset_vars () { unset OMPI_COMM_WORLD_LOCAL_SIZE unset SLURM_CPUS_ON_NODE unset SLURM_STEP_ID unset SLURM_STEP_TASKS_PER_NODE } @test 'ch-run --join: /dev/shm starts clean' { if ( ! ipc_clean_p ); then echo 'warning: /dev/shm contains leftover ch-run IPC' ipc_clean false fi } @test 'ch-run --join: one peer, direct launch' { unset_vars ipc_clean_p # --join-ct run ch-run -v --join-ct=1 "$ch_timg" -- /test/printns joined_ok 1 1 1 "$status" "$output" r='join: 1 1 [0-9]+ 0' # status from getppid(2) is all digits [[ $output =~ $r ]] [[ $output = *'join: peer group size from command line'* ]] ipc_clean_p # join count from an environment variable SLURM_CPUS_ON_NODE=1 run ch-run -v --join "$ch_timg" -- /test/printns joined_ok 1 1 1 "$status" "$output" [[ $output = *'join: peer group size from SLURM_CPUS_ON_NODE'* ]] ipc_clean_p # join count from an environment variable with extra goop SLURM_CPUS_ON_NODE=1foo ch-run --join "$ch_timg" -- /test/printns joined_ok 1 1 1 "$status" "$output" [[ $output = *'join: peer group size from SLURM_CPUS_ON_NODE'* ]] ipc_clean_p # join tag run ch-run -v --join-ct=1 --join-tag=foo "$ch_timg" -- /test/printns joined_ok 1 1 1 "$status" "$output" [[ $output = *'join: 1 1 foo 0'* ]] [[ $output = *'join: peer group tag from command line'* ]] ipc_clean_p SLURM_STEP_ID=bar run ch-run -v --join-ct=1 "$ch_timg" -- /test/printns joined_ok 1 1 1 "$status" "$output" [[ $output = *'join: 1 1 bar 0'* ]] [[ $output = *'join: peer group tag from SLURM_STEP_ID'* ]] ipc_clean_p } @test 'ch-run --join: two peers, direct launch' { unset_vars ipc_clean_p rm -f "$BATS_TMPDIR"/join.?.* # first peer (winner) ch-run -v --join-ct=2 --join-tag=foo "$ch_timg" -- \ /test/printns 10 "${BATS_TMPDIR}/join.1.ns" \ >& "${BATS_TMPDIR}/join.1.err" & sleepcat 2 "${BATS_TMPDIR}/join.1.ns" cat "${BATS_TMPDIR}/join.1.err" grep -Fq 'join: 1 2' "${BATS_TMPDIR}/join.1.err" grep -Fq 'join: I won' "${BATS_TMPDIR}/join.1.err" grep -Fq 'join: cleaning up IPC' "${BATS_TMPDIR}/join.1.err" && exit 1 # IPC resources present? (glibc and musl naming patterns are different) ls -lh /dev/shm [[ -e /dev/shm/ch-run_shm-foo ]] [[ -e /dev/shm/ch-run_sem-foo || -e /dev/shm/sem.ch-run_sem-foo ]] # second peer (loser) run ch-run -v --join-ct=2 --join-tag=foo "$ch_timg" -- \ /test/printns 0 "${BATS_TMPDIR}/join.2.ns" echo "$output" [[ $status -eq 0 ]] cat "${BATS_TMPDIR}/join.2.ns" echo "$output" | grep -Fq 'join: 1 2' echo "$output" | grep -Fq 'join: I lost' echo "$output" | grep -Fq 'joining namespaces of pid' echo "$output" | grep -Fq 'join: cleaning up IPC' # same namespaces? for i in /proc/self/ns/*; do [[ 1 = $( cat "$BATS_TMPDIR"/join.?.ns \ | grep -E "^${i}:" | uniq | wc -l) ]] done ipc_clean_p } @test 'ch-run --join: three peers, direct launch' { unset_vars ipc_clean_p rm -f "$BATS_TMPDIR"/join.?.* # first peer (winner) ch-run -v --join-ct=3 --join-tag=foo "$ch_timg" -- \ /test/printns 15 "${BATS_TMPDIR}/join.1.ns" \ >& "${BATS_TMPDIR}/join.1.err" & sleepcat 2 "${BATS_TMPDIR}/join.1.ns" cat "${BATS_TMPDIR}/join.1.err" grep -Fq 'join: 1 3' "${BATS_TMPDIR}/join.1.err" grep -Fq 'join: I won' "${BATS_TMPDIR}/join.1.err" grep -Fq 'join: 2 peers left' "${BATS_TMPDIR}/join.1.err" grep -Fq 'join: cleaning up IPC' "${BATS_TMPDIR}/join.1.err" && exit 1 # second peer (loser, no cleanup) ch-run -v --join-ct=3 --join-tag=foo "${ch_timg}" -- \ /test/printns 0 "${BATS_TMPDIR}/join.2.ns" \ >& "${BATS_TMPDIR}/join.2.err" & sleepcat 6 "${BATS_TMPDIR}/join.2.ns" cat "${BATS_TMPDIR}/join.2.err" grep -Fq 'join: 1 3' "${BATS_TMPDIR}/join.2.err" grep -Fq 'join: I lost' "${BATS_TMPDIR}/join.2.err" grep -Fq 'joining namespaces of pid' "${BATS_TMPDIR}/join.2.err" grep -Fq 'join: 1 peers left' "${BATS_TMPDIR}/join.2.err" grep -Fq 'join: cleaning up IPC' "${BATS_TMPDIR}/join.2.err" && exit 1 # IPC resources present? ls -lh /dev/shm [[ -e /dev/shm/ch-run_shm-foo ]] [[ -e /dev/shm/ch-run_sem-foo || -e /dev/shm/sem.ch-run_sem-foo ]] # third peer (loser, cleanup) ch-run -v --join-ct=3 --join-tag=foo "$ch_timg" -- \ /test/printns 0 "${BATS_TMPDIR}/join.3.ns" \ >& "${BATS_TMPDIR}/join.3.err" & sleepcat 6 "${BATS_TMPDIR}/join.3.ns" cat "${BATS_TMPDIR}/join.3.err" grep -Fq 'join: 1 3' "${BATS_TMPDIR}/join.3.err" grep -Fq 'join: I lost' "${BATS_TMPDIR}/join.2.err" grep -Fq 'joining namespaces of pid' "${BATS_TMPDIR}/join.2.err" grep -Fq 'join: 0 peers left' "${BATS_TMPDIR}/join.3.err" grep -Fq 'join: cleaning up IPC' "${BATS_TMPDIR}/join.3.err" # same namespaces? for i in /proc/self/ns/*; do [[ 1 = $( cat "$BATS_TMPDIR"/join.?.ns \ | grep -E "^$i:" | uniq | wc -l) ]] done ipc_clean_p } @test 'ch-run --join: multiple peers, framework launch' { multiprocess_ok ipc_clean_p # Two peers, one node. Should be one of each of the namespaces. Make sure # everyone chdir(2)s properly. # shellcheck disable=SC2086 run $ch_mpirun_2_1node ch-run -v --join --cd /test "$ch_timg" -- ./printns 2 echo "$output" [[ $status -eq 0 ]] ipc_clean_p joined_ok 2 2 1 "$status" "$output" # One peer per core across the allocation. Should be $ch_nodes of each # of the namespaces. # shellcheck disable=SC2086 run $ch_mpirun_core ch-run -v --join "$ch_timg" -- /test/printns 4 echo "$output" [[ $status -eq 0 ]] joined_ok "$ch_cores_total" "$ch_cores_node" "$ch_nodes" \ "$status" "$output" ipc_clean_p } @test 'ch-run --join: peer group size errors' { unset_vars # --join but no join count run ch-run --join "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ 'join: no valid peer group size found' ]] ipc_clean_p # join count no digits run ch-run --join-ct=a "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ 'join-ct: no digits found' ]] SLURM_CPUS_ON_NODE=a run ch-run --join "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ 'SLURM_CPUS_ON_NODE: no digits found' ]] ipc_clean_p # join count empty string run ch-run --join-ct='' "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ '--join-ct: no digits found' ]] SLURM_CPUS_ON_NODE=-1 run ch-run --join "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ 'join: no valid peer group size found' ]] ipc_clean_p # --join-ct digits followed by extra goo (OK from environment variable) run ch-run --join-ct=1a "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ '--join-ct: extra characters after digits' ]] ipc_clean_p # Regex for out-of-range error. range_re='.*: .*out of range' # join count above INT_MAX run ch-run --join-ct=2147483648 "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ $range_re ]] SLURM_CPUS_ON_NODE=2147483648 \ run ch-run --join "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ $range_re ]] ipc_clean_p # join count below INT_MIN run ch-run --join-ct=-2147483649 "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ $range_re ]] SLURM_CPUS_ON_NODE=-2147483649 \ run ch-run --join "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ $range_re ]] ipc_clean_p # join count above LONG_MAX run ch-run --join-ct=9223372036854775808 "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ $range_re ]] SLURM_CPUS_ON_NODE=9223372036854775808 \ run ch-run --join "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ $range_re ]] ipc_clean_p # join count below LONG_MIN run ch-run --join-ct=-9223372036854775809 "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ $range_re ]] SLURM_CPUS_ON_NODE=-9223372036854775809 \ run ch-run --join "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ $range_re ]] ipc_clean_p } @test 'ch-run --join: peer group tag errors' { unset_vars # Use a join count of 1 throughout. export SLURM_CPUS_ON_NODE=1 # join tag empty string run ch-run --join-tag='' "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ 'join: peer group tag cannot be empty string' ]] SLURM_STEP_ID='' run ch-run --join "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output =~ 'join: peer group tag cannot be empty string' ]] ipc_clean_p } @test 'ch-run --join-pid: without prior --join' { unset_vars ipc_clean_p rm -f "$BATS_TMPDIR"/join.?.* # First ch-run creates the namespaces with no joining at all. ch-run -v "$ch_timg" -- \ /test/printns 5 "${BATS_TMPDIR}/join.1.ns" \ >& "${BATS_TMPDIR}/join.1.err" & sleepcat 3 "${BATS_TMPDIR}/join.1.ns" cat "${BATS_TMPDIR}/join.1.err" grep -Fq "join: 0 0 (null) 0" "${BATS_TMPDIR}/join.1.err" # PID of ch-run/printns above. pid=$(sed -En 's/^ch-run\[([0-9]+)\]: executing:.+$/\1/p' \ "${BATS_TMPDIR}/join.1.err") echo "found pid: ${pid}" [[ -n $pid ]] # Second ch-run joins the first’s namespaces. run ch-run -v --join-pid="$pid" "$ch_timg" -- \ /test/printns 0 "${BATS_TMPDIR}/join.2.ns" echo "$output" [[ $status -eq 0 ]] cat "${BATS_TMPDIR}/join.2.ns" echo "$output" | grep -Fq "join: 0 0 (null) ${pid}" # Same namespaces? for i in /proc/self/ns/*; do [[ 1 = $( cat "$BATS_TMPDIR"/join.?.ns \ | grep -E "^${i}:" | uniq | wc -l) ]] done ipc_clean_p } @test 'ch-run --join-pid: with prior --join' { unset_vars ipc_clean_p rm -f "$BATS_TMPDIR"/join.?.* # First of two peers (winner). ch-run -v --join-ct=2 --join-tag=bar "$ch_timg" -- \ /test/printns 5 "${BATS_TMPDIR}/join.1.ns" \ >& "${BATS_TMPDIR}/join.1.err" & sleepcat 3 "${BATS_TMPDIR}/join.1.ns" cat "${BATS_TMPDIR}/join.1.err" grep -Fq 'join: 1 2' "${BATS_TMPDIR}/join.1.err" grep -Fq 'join: I won' "${BATS_TMPDIR}/join.1.err" grep -Fq 'join: cleaning up IPC' "${BATS_TMPDIR}/join.1.err" && exit 1 # PID of first peer. pid=$(sed -En 's/^ch-run\[([0-9]+)\]: join: winner initializing.+$/\1/p' \ "${BATS_TMPDIR}/join.1.err") echo "found pid: ${pid}" [[ -n $pid ]] # Second of two peers (loser). ch-run -v --join-ct=2 --join-tag=bar "${ch_timg}" -- \ /test/printns 10 "${BATS_TMPDIR}/join.2.ns" \ >& "${BATS_TMPDIR}/join.2.err" & sleepcat 6 "${BATS_TMPDIR}/join.2.ns" cat "${BATS_TMPDIR}/join.2.err" grep -Fq 'join: 1 2' "${BATS_TMPDIR}/join.2.err" grep -Fq 'join: I lost' "${BATS_TMPDIR}/join.2.err" grep -Fq "joining namespaces of pid ${pid}" "${BATS_TMPDIR}/join.2.err" grep -Fq 'join: 0 peers left' "${BATS_TMPDIR}/join.2.err" grep -Fq 'join: cleaning up IPC' "${BATS_TMPDIR}/join.2.err" # Third ch-run simulates unplanned, joins existing namespaces. run ch-run -v --join-pid="$pid" "$ch_timg" -- \ /test/printns 0 "${BATS_TMPDIR}/join.3.ns" echo "$output" [[ $status -eq 0 ]] cat "${BATS_TMPDIR}/join.3.ns" echo "$output" | grep -Fq "join: 0 0 (null) ${pid}" echo "$output" | grep -Fq "joining namespaces of pid ${pid}" echo "$output" | grep -Fq 'join: I won' && exit 1 echo "$output" | grep -Fq 'join: I lost' && exit 1 echo "$output" | grep -q 'join: .+ peers left' && exit 1 echo "$output" | grep -Fq 'join: cleaning up IPC' && exit 1 # Same namespaces? for i in /proc/self/ns/*; do [[ 1 = $( cat "$BATS_TMPDIR"/join.?.ns \ | grep -E "^${i}:" | uniq | wc -l) ]] done ipc_clean_p } @test 'ch-run --join-pid: errors' { # Can’t join namespaces of processes we don’t own. run ch-run -v --join-pid=1 "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output = *"join: can't open /proc/1/ns/user: Permission denied"* ]] # Can’t join namespaces of processes that don’t exist. pid=2147483647 run ch-run -v --join-pid="$pid" "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output = *"join: no PID ${pid}: /proc/${pid}/ns/user not found"* ]] } @test 'ch-run --join: /dev/shm ends clean' { if ( ! ipc_clean_p ); then echo 'warning: /dev/shm contains leftover ch-run IPC' ipc_clean false fi } charliecloud-0.37/test/run/ch-run_misc.bats000066400000000000000000001021771457016721300207640ustar00rootroot00000000000000load ../common bind1_dir=$BATS_TMPDIR/bind1 bind2_dir=$BATS_TMPDIR/bind2 setup () { mkdir -p "$bind1_dir" echo bind1_dir.file1 > "${bind1_dir}/file1" mkdir -p "$bind2_dir" echo bind2_dir.file2 > "${bind2_dir}/file2" } demand-overlayfs () { ch-run -W "$ch_timg" -- true || skip 'no unpriv overlayfs' } @test 'relative path to image' { # issue #6 scope full cd "$(dirname "$ch_timg")" && ch-run "$(basename "$ch_timg")" -- /bin/true } @test 'symlink to image' { # issue #50 scope full ln -sf "$ch_timg" "${BATS_TMPDIR}/symlink-test" ch-run "${BATS_TMPDIR}/symlink-test" -- /bin/true } @test 'mount image read-only' { scope standard run ch-run "$ch_timg" sh < write' ch-run -w "$ch_timg" rm write } @test 'optional default bind mounts silently skipped' { scope standard [[ ! -e "${ch_timg}/var/opt/cray/alps/spool" ]] [[ ! -e "${ch_timg}/var/opt/cray/hugetlbfs" ]] ch-run "$ch_timg" -- mount | ( ! grep -F /var/opt/cray/alps/spool ) ch-run "$ch_timg" -- mount | ( ! grep -F /var/opt/cray/hugetlbfs ) } @test "\$CH_RUNNING" { scope standard if [[ -v CH_RUNNING ]]; then echo "\$CH_RUNNING already set: $CH_RUNNING" false fi run ch-run "$ch_timg" -- /bin/sh -c 'env | grep -E ^CH_RUNNING' echo "$output" [[ $status -eq 0 ]] [[ $output = 'CH_RUNNING=Weird Al Yankovic' ]] } @test "\$HOME" { [[ $CH_TEST_BUILDER != 'none' ]] || skip 'image builder required' demand-overlayfs LC_ALL=C scope quick echo "host: $HOME" [[ $HOME ]] [[ $USER ]] # default: no change # shellcheck disable=SC2016 run ch-run "${ch_imgdir}"/quick -- /bin/sh -c 'echo $HOME' echo "$output" [[ $status -eq 0 ]] [[ $output = "/root" ]] # default: no “/root” # shellcheck disable=SC2016 run ch-run "$ch_timg" -- /bin/sh -c 'echo $HOME' echo "$output" [[ $status -eq 0 ]] [[ $output = "/" ]] # set $HOME if --home # shellcheck disable=SC2016 run ch-run --home "$ch_timg" -- /bin/sh -c 'echo $HOME' echo "$output" [[ $status -eq 0 ]] [[ $output = /home/$USER ]] # /home is merged if --home run ch-run --home "$ch_timg" -- ls -1 /home echo "$output" [[ $status -eq 0 ]] [[ $output = *directory-in-home* ]] [[ $output = *file-in-home* ]] [[ $output = *"$USER"* ]] # puke if $HOME not set home_tmp=$HOME unset HOME # shellcheck disable=SC2016 run ch-run --home "$ch_timg" -- /bin/sh -c 'echo $HOME' export HOME="$home_tmp" echo "$output" [[ $status -eq 1 ]] # shellcheck disable=SC2016 [[ $output = *'--home failed: $HOME not set'* ]] # puke if $USER not set user_tmp=$USER unset USER # shellcheck disable=SC2016 run ch-run --home "$ch_timg" -- /bin/sh -c 'echo $HOME' export USER=$user_tmp echo "$output" [[ $status -eq 1 ]] # shellcheck disable=SC2016 [[ $output = *'$USER not set'* ]] } @test "\$PATH: add /bin" { scope quick echo "$PATH" # if /bin is in $PATH, latter passes through unchanged PATH2="$ch_bin:/bin:/usr/bin" echo "$PATH2" # shellcheck disable=SC2016 PATH=$PATH2 run ch-run "$ch_timg" -- /bin/sh -c 'echo $PATH' echo "$output" [[ $status -eq 0 ]] [[ $output = "$PATH2" ]] PATH2="/bin:$ch_bin:/usr/bin" echo "$PATH2" # shellcheck disable=SC2016 PATH=$PATH2 run ch-run "$ch_timg" -- /bin/sh -c 'echo $PATH' echo "$output" [[ $status -eq 0 ]] [[ $output = "$PATH2" ]] # if /bin isn’t in $PATH, former is added to end PATH2="$ch_bin:/usr/bin" echo "$PATH2" # shellcheck disable=SC2016 PATH=$PATH2 run ch-run "$ch_timg" -- /bin/sh -c 'echo $PATH' echo "$output" [[ $status -eq 0 ]] [[ $output = $PATH2:/bin ]] } @test "\$PATH: unset" { scope standard old_path=$PATH unset PATH run "$ch_runfile" "$ch_timg" -- \ /usr/bin/python3 -c 'import os; print(os.getenv("PATH") is None)' PATH=$old_path echo "$output" [[ $status -eq 0 ]] # shellcheck disable=SC2016 [[ $output = *': $PATH not set'* ]] [[ $output = *'True'* ]] } @test "\$TMPDIR" { scope standard mkdir -p "${BATS_TMPDIR}/tmpdir" touch "${BATS_TMPDIR}/tmpdir/file-in-tmpdir" TMPDIR=${BATS_TMPDIR}/tmpdir run ch-run "$ch_timg" -- ls -1 /tmp echo "$output" [[ $status -eq 0 ]] [[ $output = file-in-tmpdir ]] } @test 'ch-run --cd' { scope quick # Default initial working directory is /. run ch-run "$ch_timg" -- pwd echo "$output" [[ $status -eq 0 ]] [[ $output = '/' ]] # Specify initial working directory. run ch-run --cd /dev "$ch_timg" -- pwd echo "$output" [[ $status -eq 0 ]] [[ $output = '/dev' ]] # Error if directory does not exist. run ch-run --cd /goops "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output =~ "can't cd to /goops: No such file or directory" ]] } @test 'ch-run --bind' { scope quick demand-overlayfs # one bind, default destination ch-run -b /mnt "$ch_timg" -- ls -lh /mnt # one bind, explicit destination ch-run -b "${bind1_dir}:/mnt/9" "$ch_timg" -- cat /mnt/9/file1 # one bind, create destination, one level ch-run -W -b "${bind1_dir}:/bind3" "$ch_timg" -- cat /bind3/file1 # one bind, create destination, two levels ch-run -W -b "${bind1_dir}:/bind4/a" "$ch_timg" -- cat /bind4/a/file1 # two binds, default destination ch-run -b /mnt -b /var "$ch_timg" -- ls -lh /mnt /var # two binds, explicit destinations ch-run -b "${bind1_dir}:/mnt/8" -b "${bind2_dir}:/mnt/9" "$ch_timg" \ -- cat /mnt/8/file1 /mnt/9/file2 # two binds, default/explicit ch-run -b /var -b "${bind2_dir}:/mnt/9" "$ch_timg" \ -- ls -lh /var /mnt/9/file2 # two binds, explicit/default ch-run -b "${bind1_dir}:/mnt/8" -b /var "$ch_timg" \ -- ls -lh /mnt/8/file1 /var # bind one source at two destinations ch-run -b "${bind1_dir}:/mnt/8" -b "${bind1_dir}:/mnt/9" "$ch_timg" \ -- diff -u /mnt/8/file1 /mnt/9/file1 # bind two sources at one destination ch-run -b "${bind1_dir}:/mnt/9" -b "${bind2_dir}:/mnt/9" "$ch_timg" \ -- sh -c '[ ! -e /mnt/9/file1 ] && cat /mnt/9/file2' } @test 'ch-run --bind with tmpfs overmount' { [[ -n $CH_TEST_SUDO ]] || skip 'sudo required' demand-overlayfs img=$BATS_TMPDIR/bind-overmount src=$BATS_TMPDIR/bind-overmount-src rm-img () { # Remove existing fixture, avoiding “sudo rm -Rf” b/c it’s too scary. sudo rm -f "$img"/foo/file-in-foo sudo rmdir "$img"/foo/directory-in-foo || true sudo rmdir "$img"/foo || true sudo rm -f "$img"/home/file-in-home sudo rmdir "$img"/home/directory-in-home || true sudo rmdir "$img"/home || true rm -Rf --one-file-system "$img" } rm-img ch-convert "$ch_tardir"/chtest.* "$img" ls -l "$img" mkdir "$img"/foo touch "$img"/foo/file-in-foo mkdir "$img"/foo/directory-in-foo sudo chown root:root "$img"/foo "$img"/home sudo chmod 755 "$img"/foo "$img"/home ls -ld "$img"/foo "$img"/home ls -l "$img"/foo "$img"/home mkdir -p "$src" touch "$src"/file-in-src mkdir -p "$src"/directory-in-src ls -ld "$src" ls -l "$src" # --bind run ch-run -W -b "$src":/foo/bar "$img" -- ls -lahR /foo echo "$output" [[ $status -eq 0 ]] # --home run ch-run --home "$img" -- ls -lah /home echo "$output" [[ $status -eq 0 ]] [[ $(echo "$output" | wc -l) -eq 3 ]] [[ $output = *directory-in-home* ]] [[ $output = *file-in-home* ]] [[ $output = *"$USER"* ]] rm-img } @test 'ch-run --bind errors' { scope quick [[ $CH_TEST_PACK_FMT == squash-mount ]] || skip 'squash-mount format only' demand-overlayfs # no argument to --bind run ch-run "$ch_timg" -b echo "$output" [[ $status -eq 64 ]] [[ $output = *'option requires an argument'* ]] # empty argument to --bind run ch-run -b '' "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *'--bind: no source provided'* ]] # source not provided run ch-run -b :/mnt/9 "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *'--bind: no source provided'* ]] # destination not provided run ch-run -b "${bind1_dir}:" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *'--bind: no destination provided'* ]] # destination is / run ch-run -b "${bind1_dir}:/" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"--bind: destination can't be /"* ]] # destination is relative run ch-run -b "${bind1_dir}:foo" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"--bind: destination must be absolute"* ]] # destination climbs out of image, exists run ch-run -b "${bind1_dir}:/.." "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't bind: "*"/${USER}.ch not subdirectory of "*"/${USER}.ch/mnt"* ]] # destination climbs out of image, does not exist run ch-run -b "${bind1_dir}:/../doesnotexist/a" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't mkdir: "*"/${USER}.ch/doesnotexist not subdirectory of "*"/${USER}.ch/mnt"* ]] [[ ! -e ${ch_imgdir}/doesnotexist ]] # source does not exist run ch-run -b "/doesnotexist" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't bind: source not found: /doesnotexist"* ]] # destination does not exist and image is not writeable run ch-run -b "${bind1_dir}:/doesnotexist" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't mkdir: "*"/${USER}.ch/mnt/doesnotexist: Read-only file system"* ]] # neither source nor destination exist run ch-run -b /doesnotexist-out:/doesnotexist-in "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't bind: source not found: /doesnotexist-out"* ]] # correct bind followed by source does not exist run ch-run -b "${bind1_dir}:/mnt/0" -b /doesnotexist "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't bind: source not found: /doesnotexist"* ]] # correct bind followed by destination does not exist run ch-run -b "${bind1_dir}:/mnt/0" -b "${bind2_dir}:/doesnotexist" \ "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't mkdir: "*"/${USER}.ch/mnt/doesnotexist: Read-only file system"* ]] # destination is broken symlink run ch-run -b "${bind1_dir}:/mnt/link-b0rken-abs" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't mkdir: symlink not relative: "*"/${USER}.ch/mnt/mnt/link-b0rken-abs"* ]] # destination is absolute symlink outside image run ch-run -b "${bind1_dir}:/mnt/link-bad-abs" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't bind: "*" not subdirectory of"* ]] # destination relative symlink outside image run ch-run -b "${bind1_dir}:/mnt/link-bad-rel" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't bind: "*" not subdirectory of"* ]] # mkdir(2) under existing bind-mount, default, first level run ch-run -b "${bind1_dir}:/proc/doesnotexist" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't mkdir: "*"/${USER}.ch/mnt/proc/doesnotexist under existing bind-mount "*"/${USER}.ch/mnt/proc "* ]] # mkdir(2) under existing bind-mount, user-supplied, first level run ch-run -b "${bind1_dir}:/mnt/0" \ -b "${bind2_dir}:/mnt/0/foo" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't mkdir: "*"/${USER}.ch/mnt/mnt/0/foo under existing bind-mount "*"/${USER}.ch/mnt/mnt/0 "* ]] # mkdir(2) under existing bind-mount, default, 2nd level run ch-run -b "${bind1_dir}:/proc/sys/doesnotexist" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't mkdir: "*"/${USER}.ch/mnt/proc/sys/doesnotexist under existing bind-mount "*"/${USER}.ch/mnt/proc "* ]] } @test 'ch-run --set-env' { scope standard # Quirk that is probably too obscure to put in the documentation: The # string containing only two straight quotes does not round-trip through # “printenv” or “env”, though it does round-trip through Bash “set”: # # $ export foo="''" # $ echo [$foo] # [''] # $ set | fgrep foo # foo=''\'''\''' # $ eval $(set | fgrep foo) # $ echo [$foo] # [''] # $ printenv | fgrep foo # foo='' # $ eval $(printenv | fgrep foo) # $ echo $foo # [] # Valid inputs. Use Python to print the results to avoid ambiguity. export SET=foo export SET2=boo f_in=${BATS_TMPDIR}/env.txt cat <<'EOF' > "$f_in" chse_a1=bar chse_a2=bar=baz chse_a3=bar baz chse_a4='bar' chse_a5= chse_a6='' chse_a7='''' chse_b1="bar" chse_b2=bar # baz chse_b3=bar chse_b4= bar chse_c1=foo chse_c1=bar chse_d1=foo: chse_d2=:foo chse_d3=: chse_d4=:: chse_d5=$SET chse_d6=$SET:$SET2 chse_d7=bar:$SET chse_d8=bar:baz:$SET chse_d9=$SET:bar chse_dA=$SET:bar:baz chse_dB=bar:$SET:baz chse_dC=bar:baz:$SET:bar:baz chse_e1=:$SET chse_e2=::$SET chse_e3=$SET: chse_e4=$SET:: chse_e5=bar:$ chse_e6=bar:* chse_e7=bar$SET chse_e8=bar::$SET chse_f1=$UNSET chse_f2=foo:$UNSET chse_f3=foo:$UNSET: chse_f4=$UNSET:foo chse_f5=:$UNSET:foo chse_f6=foo:$UNSET:$UNSET2 chse_f7=foo:$UNSET:$UNSET2: chse_f8=$UNSET:$UNSET2:foo chse_f9=:$UNSET:$UNSET2:foo chse_fA=foo:$UNSET:bar chse_fB=foo:$UNSET:$UNSET2:bar chse_fC=:$UNSET chse_fD=::$UNSET chse_fE=$UNSET: chse_fF=$UNSET:: EOF cat "$f_in" output_expected=$(cat <<'EOF' (' chse_b3', 'bar') ('chse_a1', 'bar') ('chse_a2', 'bar=baz') ('chse_a3', 'bar baz') ('chse_a4', 'bar') ('chse_a5', '') ('chse_a6', '') ('chse_a7', "''") ('chse_b1', '"bar"') ('chse_b2', 'bar # baz') ('chse_b4', ' bar') ('chse_c1', 'bar') ('chse_d1', 'foo:') ('chse_d2', ':foo') ('chse_d3', ':') ('chse_d4', '::') ('chse_d5', 'foo') ('chse_d6', 'foo:boo') ('chse_d7', 'bar:foo') ('chse_d8', 'bar:baz:foo') ('chse_d9', 'foo:bar') ('chse_dA', 'foo:bar:baz') ('chse_dB', 'bar:foo:baz') ('chse_dC', 'bar:baz:foo:bar:baz') ('chse_e1', ':foo') ('chse_e2', '::foo') ('chse_e3', 'foo:') ('chse_e4', 'foo::') ('chse_e5', 'bar:$') ('chse_e6', 'bar:*') ('chse_e7', 'bar$SET') ('chse_e8', 'bar::foo') ('chse_f1', '') ('chse_f2', 'foo') ('chse_f3', 'foo:') ('chse_f4', 'foo') ('chse_f5', ':foo') ('chse_f6', 'foo') ('chse_f7', 'foo:') ('chse_f8', 'foo') ('chse_f9', ':foo') ('chse_fA', 'foo:bar') ('chse_fB', 'foo:bar') ('chse_fC', '') ('chse_fD', ':') ('chse_fE', '') ('chse_fF', ':') EOF ) run ch-run --set-env="$f_in" "$ch_timg" -- python3 -c 'import os; [print((k,v)) for (k,v) in sorted(os.environ.items()) if "chse_" in k]' echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$output_expected") <(echo "$output") } @test 'ch-run --set-env0' { scope standard export SET=foo f_in=${BATS_TMPDIR}/env.bin { printf 'chse_a1=bar\0' printf "chse_a4='bar'\0" #shellcheck disable=SC2016 printf 'chse_d7=bar:$SET\0' printf 'chse_g1=foo\nbar\0' } > "$f_in" hd "$f_in" | sed -E 's/^0000//' # trim a few zeros to make it fit output_expected=$(cat <<'EOF' ('chse_a1', 'bar') ('chse_a4', 'bar') ('chse_d7', 'bar:foo') ('chse_g1', 'foo\nbar') EOF ) run ch-run --set-env0="$f_in" "$ch_timg" -- python3 -c 'import os; [print((k,v)) for (k,v) in sorted(os.environ.items()) if "chse_" in k]' echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$output_expected") <(echo "$output") } @test 'ch-run --set-env from Dockerfile' { scope standard prerequisites_ok argenv img=${ch_imgdir}/argenv output_expected=$(cat <<'EOF' chse_env1_df=env1 chse_env2_df=env2 env1 EOF ) run ch-run --set-env "$img" -- sh -c 'env | grep -E "^chse_"' echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$output_expected") <(echo "$output") } @test 'ch-run --set-env errors' { scope standard f_in=${BATS_TMPDIR}/env.txt # file does not exist run ch-run --set-env=doesnotexist.txt "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't open: doesnotexist.txt: No such file or directory"* ]] # /ch/environment missing run ch-run --set-env "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't open: /ch/environment: No such file or directory"* ]] # Note: I’m not sure how to test an error during reading, i.e., getline(3) # rather than fopen(3). Hence no test for “error reading”. # invalid line: missing “=” echo 'FOO bar' > "$f_in" run ch-run --set-env="$f_in" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't parse variable: no delimiter: ${f_in}:1"* ]] # invalid line: no name echo '=bar' > "$f_in" run ch-run --set-env="$f_in" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't parse variable: empty name: ${f_in}:1"* ]] } # shellcheck disable=SC2016 @test 'ch-run --set-env command line' { scope standard # missing “'” # shellcheck disable=SC2086 run ch-run --set-env=foo='$test:app' --env-no-expand -v "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 0 ]] [[ $output = *'environment: foo=$test:app'* ]] # missing environment variable run ch-run --set-env='$PATH:foo' "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *'$PATH:foo: No such file or directory'* ]] } @test 'ch-run --unset-env' { scope standard export chue_1=foo export chue_2=bar printf '\n# Nothing\n\n' run ch-run --unset-env=doesnotmatch "$ch_timg" -- env echo "$output" | sort [[ $status -eq 0 ]] ex='^(_|CH_RUNNING|HOME|PATH|SHLVL|TMPDIR)=' # expected to change diff -u <(env | grep -Ev "$ex" | sort) \ <(echo "$output" | grep -Ev "$ex" | sort) printf '\n# Everything\n\n' run ch-run --unset-env='*' "$ch_timg" -- env echo "$output" [[ $status -eq 0 ]] [[ $output = 'CH_RUNNING=Weird Al Yankovic' ]] printf '\n# Everything, plus shell re-adds\n\n' run ch-run --unset-env='*' "$ch_timg" -- /bin/sh -c 'env | sort' echo "$output" [[ $status -eq 0 ]] diff -u <(printf 'CH_RUNNING=Weird Al Yankovic\nPWD=/\nSHLVL=1\n') \ <(echo "$output") printf '\n# Without wildcards\n\n' run ch-run --unset-env=chue_1 "$ch_timg" -- env echo "$output" [[ $status -eq 0 ]] diff -u <(printf 'chue_2=bar\n') <(echo "$output" | grep -E '^chue_') printf '\n# With wildcards\n\n' run ch-run --unset-env='chue_*' "$ch_timg" -- env echo "$output" [[ $status -eq 0 ]] [[ $(echo "$output" | grep -E '^chue_') = '' ]] printf '\n# Empty string\n\n' run ch-run --unset-env= "$ch_timg" -- env echo "$output" [[ $status -eq 1 ]] [[ $output = *'--unset-env: GLOB must have non-zero length'* ]] } @test 'ch-run --unset-env extglobs' { scope standard ch-run --feature extglob || skip 'extended globs not available' export chue_1=foo export chue_2=bar printf '\n# With extended globs to select\n\n' run ch-run --unset-env='chue_@(1|2)' "$ch_timg" -- env echo "$output" [[ $status -eq 0 ]] [[ $(echo "$output" | grep -E '^chue_') = '' ]] printf '\n# With extended globs to deselect\n\n' run ch-run --unset-env='!(chue_*)' "$ch_timg" -- env echo "$output" [[ $status -eq 0 ]] output_expected=$(cat <<'EOF' CH_RUNNING=Weird Al Yankovic chue_1=foo chue_2=bar EOF ) diff -u <(echo "$output_expected") <(echo "$output" | LC_ALL=C sort) } @test 'ch-run mixed --set-env and --unset-env' { scope standard # Input. export chmix_a1=z export chmix_a2=y export chmix_a3=x f1_in=${BATS_TMPDIR}/env1.txt cat <<'EOF' > "$f1_in" chmix_b1=w chmix_b2=v EOF f2_in=${BATS_TMPDIR}/env2.txt cat <<'EOF' > "$f2_in" chmix_c1=u chmix_c2=t EOF # unset, unset output_expected=$(cat <<'EOF' chmix_a3=x EOF ) run ch-run --unset-env=chmix_a1 --unset-env=chmix_a2 "$ch_timg" -- \ sh -c 'env | grep -E ^chmix_ | sort' echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$output_expected") <(echo "$output") echo '# set, set' output_expected=$(cat <<'EOF' chmix_a1=z chmix_a2=y chmix_a3=x chmix_b1=w chmix_b2=v chmix_c1=u chmix_c2=t EOF ) run ch-run --set-env="$f1_in" --set-env="$f2_in" "$ch_timg" -- \ sh -c 'env | grep -E ^chmix_ | sort' echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$output_expected") <(echo "$output") echo '# unset, set' output_expected=$(cat <<'EOF' chmix_a2=y chmix_a3=x chmix_b1=w chmix_b2=v EOF ) run ch-run --unset-env=chmix_a1 --set-env="$f1_in" "$ch_timg" -- \ sh -c 'env | grep -E ^chmix_ | sort' echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$output_expected") <(echo "$output") echo '# set, unset' output_expected=$(cat <<'EOF' chmix_a1=z chmix_a2=y chmix_a3=x chmix_b1=w EOF ) run ch-run --set-env="$f1_in" --unset-env=chmix_b2 "$ch_timg" -- \ sh -c 'env | grep -E ^chmix_ | sort' echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$output_expected") <(echo "$output") echo '# unset, set, unset' output_expected=$(cat <<'EOF' chmix_a2=y chmix_a3=x chmix_b1=w EOF ) run ch-run --unset-env=chmix_a1 \ --set-env="$f1_in" \ --unset-env=chmix_b2 \ "$ch_timg" -- sh -c 'env | grep -E ^chmix_ | sort' echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$output_expected") <(echo "$output") echo '# set, unset, set' output_expected=$(cat <<'EOF' chmix_a1=z chmix_a2=y chmix_a3=x chmix_b1=w chmix_c1=u chmix_c2=t EOF ) run ch-run --set-env="$f1_in" \ --unset-env=chmix_b2 \ --set-env="$f2_in" \ "$ch_timg" -- sh -c 'env | grep -E ^chmix_ | sort' echo "$output" [[ $status -eq 0 ]] diff -u <(echo "$output_expected") <(echo "$output") } @test 'ch-run: internal SquashFUSE mounting' { scope standard [[ $CH_TEST_PACK_FMT == squash-mount ]] || skip 'squash-mount format only' ch_mnt="/var/tmp/${USER}.ch/mnt" # default mount point run ch-run -v "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 0 ]] [[ $output = *"newroot: (null)"* ]] [[ $output = *"using default mount point: ${ch_mnt}"* ]] [[ -d ${ch_mnt} ]] rmdir "${ch_mnt}" # -m option mountpt="${BATS_TMPDIR}/sqfs_tmpdir" mountpt_real=$(realpath "$mountpt") [[ -e $mountpt ]] || mkdir "$mountpt" run ch-run -m "$mountpt" -v "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 0 ]] [[ $output = *"newroot: ${mountpt_real}"* ]] rmdir "$mountpt" # -m with non-sqfs img img=$(realpath "${BATS_TMPDIR}/dirimg") ch-convert -i squash "$ch_timg" "$img" run ch-run -m /doesnotexist -v "$img" -- /bin/true echo "$output" [[ $status -eq 0 ]] [[ $output = *"warning: --mount invalid with directory image, ignoring"* ]] [[ $output = *"newroot: ${img}"* ]] rm -Rf --one-file-system "$img" } @test 'ch-run: internal SquashFUSE errors' { scope standard [[ $CH_TEST_PACK_FMT == squash-mount ]] || skip 'squash-mount format only' # mount point is empty string run ch-run --mount= "$ch_timg" -- /bin/true echo "$output" [[ $status -ne 0 ]] # exits with status of 139 [[ $output = *"mount point can't be empty string"* ]] # mount point doesn’t exist run ch-run -m /doesnotexist "$ch_timg" -- /bin/true echo "$output" [[ $status -ne 0 ]] # exits with status of 139 [[ $output = *"can't stat mount point: /doesnotexist: No such file or directory"* ]] # mount point is a file run ch-run -m ./fixtures/README "$ch_timg" -- /bin/true echo "$output" [[ $status -ne 0 ]] [[ $output = *'not a directory: '*'/fixtures/README'* ]] # image is file but not sqfs run ch-run -vv ./fixtures/README -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *'magic expected: 6873 7173; actual: 596f 7520'* ]] [[ $output = *'unknown image type: '*'/fixtures/README'* ]] # image is a broken sqfs sq_tmp="$BATS_TMPDIR"/b0rken.sqfs cp "$ch_timg" "$sq_tmp" # corrupt inode count (bytes 4–7, 0-indexed) printf '\xED\x5F\x84\x00' | dd of="$sq_tmp" bs=1 count=4 seek=4 conv=notrunc ls -l "$ch_timg" "$sq_tmp" run ch-run -vv "$sq_tmp" -- ls -l / echo "$output" [[ $status -ne 0 ]] [[ $output = *'magic expected: 6873 7173; actual: 6873 7173'* ]] [[ $output = *"can't open SquashFS: ${sq_tmp}"* ]] rm "$sq_tmp" } @test 'broken image errors' { scope standard img=${BATS_TMPDIR}/broken-image tmpdir=${TMPDIR:-/tmp} # Create an image skeleton. dirs=$(echo {dev,proc,sys}) files=$(echo etc/{group,passwd}) files_optional=$(echo etc/{hosts,resolv.conf}) mkdir -p "$img" for d in $dirs; do mkdir -p "${img}/$d"; done mkdir -p "${img}/etc" "${img}/home" "${img}/usr/bin" "${img}/tmp" for f in $files $files_optional; do touch "${img}/${f}"; done # This should start up the container OK but fail to find the user command. run ch-run "$img" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't execve(2): /bin/true: No such file or directory"* ]] # For each required file, we want a correct error if it’s missing. for f in $files; do echo "required: ${f}" rm "${img}/${f}" ls -l "${img}/${f}" || true run ch-run "$img" -- /bin/true touch "${img}/${f}" # restore before test fails for idempotency echo "$output" [[ $status -eq 1 ]] r="can't bind: destination not found: .+/${f}" echo "expected: ${r}" [[ $output =~ $r ]] done # For each optional file, we want no error if it’s missing. for f in $files_optional; do echo "optional: ${f}" rm "${img}/${f}" run ch-run "$img" -- /bin/true touch "${img}/${f}" # restore before test fails for idempotency echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't execve(2): /bin/true: No such file or directory"* ]] done # For all files, we want a correct error if it’s not a regular file. for f in $files $files_optional; do echo "not a regular file: ${f}" rm "${img}/${f}" mkdir "${img}/${f}" run ch-run "$img" -- /bin/true rmdir "${img}/${f}" # restore before test fails for idempotency touch "${img}/${f}" echo "$output" [[ $status -eq 1 ]] r="can't bind .+ to /.+/${f}: Not a directory" echo "expected: ${r}" [[ $output =~ $r ]] done # For each directory, we want a correct error if it’s missing. for d in $dirs tmp; do echo "required: ${d}" rmdir "${img}/${d}" run ch-run "$img" -- /bin/true mkdir "${img}/${d}" # restore before test fails for idempotency echo "$output" [[ $status -eq 1 ]] r="can't bind: destination not found: .+/${d}" echo "expected: ${r}" [[ $output =~ $r ]] done # For each directory, we want a correct error if it’s not a directory. for d in $dirs tmp; do echo "not a directory: ${d}" rmdir "${img}/${d}" touch "${img}/${d}" run ch-run "$img" -- /bin/true rm "${img}/${d}" # restore before test fails for idempotency mkdir "${img}/${d}" echo "$output" [[ $status -eq 1 ]] r="can't bind .+ to /.+/${d}: Not a directory" echo "expected: ${r}" [[ $output =~ $r ]] done # --private-tmp rmdir "${img}/tmp" run ch-run --private-tmp "$img" -- /bin/true mkdir "${img}/tmp" # restore before test fails for idempotency echo "$output" [[ $status -eq 1 ]] r="can't mount tmpfs at /.+/tmp: No such file or directory" echo "expected: ${r}" [[ $output =~ $r ]] # default shouldn’t care if /home is missing rmdir "${img}/home" run ch-run "$img" -- /bin/true mkdir "${img}/home" # restore before test fails for idempotency echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't execve(2): /bin/true: No such file or directory"* ]] # Everything should be restored and back to the original error. run ch-run "$img" -- /bin/true echo "$output" [[ $status -eq 1 ]] [[ $output = *"can't execve(2): /bin/true: No such file or directory"* ]] # At this point, there should be exactly two each of passwd and group # temporary files. Remove them. [[ $(find -H "$tmpdir" -maxdepth 1 -name 'ch-run_passwd*' | wc -l) -eq 2 ]] [[ $(find -H "$tmpdir" -maxdepth 1 -name 'ch-run_group*' | wc -l) -eq 2 ]] rm -v "$tmpdir"/ch-run_{passwd,group}* [[ $(find -H "$tmpdir" -maxdepth 1 -name 'ch-run_passwd*' | wc -l) -eq 0 ]] [[ $(find -H "$tmpdir" -maxdepth 1 -name 'ch-run_group*' | wc -l) -eq 0 ]] } @test 'UID and/or GID invalid on host' { scope standard uid_bad=8675309 gid_bad=8675310 # UID run ch-run -v --uid="$uid_bad" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 0 ]] [[ $output = *"UID ${uid_bad} not found; using dummy info"* ]] # GID run ch-run -v --gid="$gid_bad" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 0 ]] [[ $output = *"GID ${gid_bad} not found; using dummy info"* ]] # both run ch-run -v --uid="$uid_bad" --gid="$gid_bad" "$ch_timg" -- /bin/true echo "$output" [[ $status -eq 0 ]] [[ $output = *"UID ${uid_bad} not found; using dummy info"* ]] [[ $output = *"GID ${gid_bad} not found; using dummy info"* ]] } @test 'syslog' { # This test depends on a fairly specific syslog configuration, so just do # it on GitHub Actions. [[ -n $GITHUB_ACTIONS ]] || skip 'GitHub Actions only' [[ -n $CH_TEST_SUDO ]] || skip 'sudo required' expected="uid=$(id -u) args=6: ch-run ${ch_timg} -- echo foo \"b a}\\\$r\"" echo "$expected" #shellcheck disable=SC2016 ch-run "$ch_timg" -- echo foo 'b a}$r' text=$(sudo tail -n 10 /var/log/syslog) echo "$text" echo "$text" | grep -F "$expected" } @test 'reprint warnings' { run ch-run --warnings=0 [[ $status -eq 0 ]] [[ $(echo "$output" | grep -Fc 'this is warning 1!') -eq 0 ]] [[ $(echo "$output" | grep -Fc 'this is warning 2!') -eq 0 ]] [[ "$output" != *'reprinting first'* ]] run ch-run --warnings=1 echo "$output" [[ $status -eq 0 ]] [[ $output == *'ch-run['*']: warning: reprinting first 1 warning(s)'* ]] [[ $(echo "$output" | grep -Fc 'this is warning 1!') -eq 2 ]] [[ $(echo "$output" | grep -Fc 'this is warning 2!') -eq 0 ]] # Warnings list is a statically sized memory buffer. Ensure it works as # intended by printing more warnings than can be saved to this buffer. run ch-run --warnings=100 echo "$output" [[ $status -eq 0 ]] [[ $output == *'ch-run['*']: warning: reprinting first '*' warning(s)'* ]] [[ $(echo "$output" | grep -Fc 'this is warning 1!') -eq 2 ]] [[ $(echo "$output" | grep -Fc 'this is warning 100!') -eq 1 ]] } @test 'ch-run --quiet' { # test --logging-test run ch-run --test=log echo "$output" [[ $status -eq 0 ]] [[ $output = *'info'* ]] [[ $output = *'warning: warning'* ]] # quiet level 1 run ch-run -q --test=log echo "$output" [[ $status -eq 0 ]] [[ $output != *'info'* ]] [[ $output = *'warning: warning'* ]] # quiet level 2 run ch-run -qq --test=log echo "$output" [[ $status -eq 0 ]] [[ $output != *'info'* ]] [[ $output != *'warning: warning'* ]] # subprocess failure at quiet level 2 run ch-run -qq "$ch_timg" -- doesnotexist echo "$output" [[ $status -eq 1 ]] [[ $output = *"error: can't execve(2): doesnotexist: No such file or directory"* ]] # quiet level 3 run ch-run -qqq --test=log echo "$output" [[ $status -eq 0 ]] [[ $output != *'info'* ]] [[ $output != *"warning: warning"* ]] # subprocess failure at quiet level 3 run ch-run -qqq "$ch_timg" -- doesnotexist echo "$output" [[ $status -eq 1 ]] [[ $output != *"error: can't execve(2): doesnotexist: No such file or directory"* ]] # failure at quiet level 3 run ch-run -qqq --test=log-fail echo "$output" [[ $status -eq 1 ]] [[ $output != *'info'* ]] [[ $output != *'warning: warning'* ]] [[ $output = *'error: the program failed inexplicably'* ]] } @test 'ch-run --write-fake errors' { demand-overlayfs # bad tmpfs size run ch-run --write-fake=foo "$ch_timg" -- true echo "$output" [[ $status -eq 1 ]] [[ $output == *'cannot mount tmpfs for overlay: Invalid argument'* ]] } charliecloud-0.37/test/run/ch-run_uidgid.bats000066400000000000000000000132101457016721300212630ustar00rootroot00000000000000load ../common setup () { scope full if [[ -n $GUEST_USER ]]; then # Specific user requested for testing. [[ -n $GUEST_GROUP ]] guest_uid=$(id -u "$GUEST_USER") guest_gid=$(getent group "$GUEST_GROUP" | cut -d: -f3) uid_args="-u ${guest_uid}" gid_args="-g ${guest_gid}" echo "ID args: ${GUEST_USER}/${guest_uid} ${GUEST_GROUP}/${guest_gid}" echo else # No specific user requested. [[ -z $GUEST_GROUP ]] GUEST_USER=$(id -un) guest_uid=$(id -u) [[ $GUEST_USER = "$USER" ]] [[ $guest_uid -ne 0 ]] GUEST_GROUP=$(id -gn) guest_gid=$(id -g) [[ $guest_gid -ne 0 ]] uid_args= gid_args= echo "no ID arguments" echo fi } @test 'user and group as specified' { # shellcheck disable=SC2086 g=$(ch-run $uid_args $gid_args "$ch_timg" -- id -un) [[ $GUEST_USER = "$g" ]] # shellcheck disable=SC2086 g=$(ch-run $uid_args $gid_args "$ch_timg" -- id -u) [[ $guest_uid = "$g" ]] # shellcheck disable=SC2086 g=$(ch-run $uid_args $gid_args "$ch_timg" -- id -gn) [[ $GUEST_GROUP = "$g" ]] # shellcheck disable=SC2086 g=$(ch-run $uid_args $gid_args "$ch_timg" -- id -g) [[ $guest_gid = "$g" ]] } @test 'chroot escape' { # Try to escape a chroot(2) using the standard approach. # shellcheck disable=SC2086 ch-run $uid_args $gid_args "$ch_timg" -- /test/chroot-escape } @test '/dev /proc /sys' { # Read some files in /dev, /proc, and /sys that I shouldn’t have access to. # shellcheck disable=SC2086 ch-run $uid_args $gid_args "$ch_timg" -- /test/dev_proc_sys.py } @test 'filesystem permission enforcement' { [[ $CH_TEST_PERMDIRS = skip ]] && skip 'user request' for d in $CH_TEST_PERMDIRS; do d="${d}/pass" echo "verifying: ${d}" # shellcheck disable=SC2086 ch-run --private-tmp \ $uid_args $gid_args -b "$d:/mnt/0" "$ch_timg" -- \ /test/fs_perms.py /mnt/0 done } @test 'mknod(2)' { # Make some device files. If this works, we might be able to later read or # write them to do things we shouldn’t. Try on all mount points. # shellcheck disable=SC2016,SC2086 ch-run $uid_args $gid_args "$ch_timg" -- \ sh -c '/test/mknods $(cat /proc/mounts | cut -d" " -f2)' } @test 'privileged IPv4 bind(2)' { # Bind to privileged ports on all host IPv4 addresses. # # Some supported distributions don’t have “hostname --all-ip-addresses”. # Hence the awk voodoo. addrs=$(ip -o addr | awk '/inet / {gsub(/\/.*/, " ",$4); print $4}') # shellcheck disable=SC2086 ch-run $uid_args $gid_args "$ch_timg" -- /test/bind_priv.py $addrs } @test 'remount host root' { # Re-mount the root filesystem. Notes: # # - Because we have /dev from the host, we don’t need to create a new # device node. This makes the test simpler. In particular, we can # treat network and local root the same. # # - We leave the filesystem mounted even if successful, again to make # the test simpler. The rest of the tests will ignore it or maybe # over-mount something else. # # shellcheck disable=SC2086 ch-run $uid_args $gid_args "$ch_timg" -- \ sh -c '[ -f /bin/mount -a -x /bin/mount ]' dev=$(findmnt -n -o SOURCE -T /) type=$(findmnt -n -o FSTYPE -T /) opts=$(findmnt -n -o OPTIONS -T /) # shellcheck disable=SC2086 run ch-run $uid_args $gid_args "$ch_timg" -- \ /bin/mount -n -o "$opts" -t "$type" "$dev" /mnt/0 echo "$output" # return codes from http://man7.org/linux/man-pages/man8/mount.8.html # busybox seems to use the same list case $status in 0) # “success” printf 'RISK\tsuccessful mount\n' false ;; 1) ;& # “incorrect invocation or permissions” (we care which) 111) ;& # undocumented 255) # undocumented if [[ $output = *'ermission denied'* ]]; then printf 'SAFE\tmount exit %d, permission denied\n' "$status" return 0 elif [[ $dev = 'rootfs' && $output =~ 'No such device' ]]; then printf 'SAFE\tmount exit %d, no such device' "$status" return 0 else printf 'RISK\tmount exit %d w/o known explanation\n' "$status" false fi ;; 32) # “mount failed” printf 'SAFE\tmount exited with code 32\n' return 0 ;; esac printf 'ERROR\tunknown exit code: %s\n' "$status" false } @test 'setgroups(2)' { # Can we change our supplemental groups? # shellcheck disable=SC2086 ch-run $uid_args $gid_args "$ch_timg" -- /test/setgroups } @test 'seteuid(2)' { # Try to seteuid(2) to another UID we shouldn’t have access to # shellcheck disable=SC2086 ch-run $uid_args $gid_args "$ch_timg" -- /test/setuid } @test 'signal process outside container' { # Send a signal to a process we shouldn’t be able to signal, in this case # getty. This requires at least one getty running, i.e., at least one # virtual console waiting for login. In the past, distributions ran gettys # on several VCs by default, but in recent years they are often started # dynamically, so there may be none running. See your distro’s # documentation on how to configure this. See also e.g. issue #840. [[ $(pgrep -c getty) -eq 0 ]] && pedantic_fail 'no getty process found' # shellcheck disable=SC2086 ch-run $uid_args $gid_args "$ch_timg" -- /test/signal_out.py } charliecloud-0.37/test/run_first.bats000066400000000000000000000007011457016721300177520ustar00rootroot00000000000000load common @test 'permissions test directories exist' { scope standard [[ $CH_TEST_PERMDIRS = skip ]] && skip 'user request' for d in $CH_TEST_PERMDIRS; do echo "$d" test -d "${d}" test -d "${d}/pass" test -f "${d}/pass/file" test -d "${d}/nopass" test -d "${d}/nopass/dir" test -f "${d}/nopass/file" done } @test 'ch-checkns' { scope quick "${ch_bin}/ch-checkns" } charliecloud-0.37/test/sotest/000077500000000000000000000000001457016721300164075ustar00rootroot00000000000000charliecloud-0.37/test/sotest/files_inferrable.txt000066400000000000000000000000561457016721300224440ustar00rootroot00000000000000sotest/bin/sotest sotest/lib/libsotest.so.1.0 charliecloud-0.37/test/sotest/libsotest.c000066400000000000000000000001011457016721300205530ustar00rootroot00000000000000int increment(int a); int increment(int a) { return a + 1; } charliecloud-0.37/test/sotest/sotest.c000066400000000000000000000002631457016721300200750ustar00rootroot00000000000000#include #include int increment(int a); int main() { int b = 8675308; printf("libsotest says %d incremented is %d\n", b, increment(b)); exit(0); } charliecloud-0.37/test/unused/000077500000000000000000000000001457016721300163715ustar00rootroot00000000000000charliecloud-0.37/test/unused/echo-euid.c000066400000000000000000000004051457016721300203760ustar00rootroot00000000000000/* This program prints the effective user ID on stdout and exits. It is useful for testing whether the setuid bit was effective. */ #include #include #include int main(void) { printf("%u\n", geteuid()); return 0; } charliecloud-0.37/test/unused/su_wrap.py000077500000000000000000000036721457016721300204360ustar00rootroot00000000000000#!/usr/bin/env python3 # This script tries to use su to gain root privileges, assuming that # /etc/shadow has been changed such that no password is required. It uses # pexpect to emulate the terminal that su requires. # # WARNING: This does not work. For example: # # $ whoami ; echo $UID EUID # reidpr # 1001 1001 # $ /bin/su -c whoami # root # $ ./su_wrap.py 2>> /dev/null # SAFE escalation failed: empty password rejected # # That is, manual su can escalate without a password (and doesn't without the # /etc/shadow hack), but when this program tries to do apparently the same # thing, su wants a password. # # I have not been able to track down why this happens. I suspect that PAM has # some extra smarts about TTY that causes it to ask for a password under # pexpect. I'm leaving the code in the repository in case some future person # can figure it out. import sys import pexpect # Invoke su. This will do one of three things: # # 1. Print 'root'; the escalation was successful. # 2. Ask for a password; the escalation was unsuccessful. # 3. Something else; this is an error. # p = pexpect.spawn('/bin/su', ['-c', 'whoami'], timeout=5, encoding='UTF-8', logfile=sys.stderr) i = p.expect_exact(['root', 'Password:']) try: if (i == 0): # printed "root" print('RISK\tescalation successful: no password requested') elif (i == 1): # asked for password p.sendline() # try empty password i = p.expect_exact(['root', 'Authentication failure']) if (i == 0): # printed "root" print('RISK\tescalation successful: empty password accepted') elif (i == 1): # explicit failure print('SAFE\tescalation failed: empty password rejected') else: assert False else: assert False except p.EOF: print('ERROR\tsu exited unexpectedly') except p.TIMEOUT: print('ERROR\ttimed out waiting for su') except AssertionError: print('ERROR\tassertion failed') charliecloud-0.37/test/whiteout000077500000000000000000000052751457016721300166750ustar00rootroot00000000000000#!/usr/bin/env python3 # This Python script produces (on stdout) a Dockerfile that produces a large # number of whiteouts. At the end, the Dockerfile prints some output that can # be compared with the flattened image. The purpose is to test whiteout # interpretation during flattening. # # See: https://github.com/opencontainers/image-spec/blob/master/layer.md # # There are a few factors to consider: # # * files vs. directories # * white-out explicit files vs. everything in a directory # * restore the files vs. not (in the same layer as deletion) # # Currently, we don't do recursion, operating only on the specified directory. # We do this at two different levels in the directory tree. # # It's easy to bump into the 127-layer limit with this script. # # To build and push: # # $ version=2020-01-09 # use today's date # $ sudo docker login # if needed # $ ./whiteout | sudo docker build -t whiteout -f - . # $ sudo docker tag whiteout:latest charliecloud/whiteout:$version # $ sudo docker images | fgrep whiteout # $ sudo docker push charliecloud/whiteout:$version # # Then your new image will be at: # # https://hub.docker.com/repository/docker/charliecloud/whiteout import sys INF = 99 def discotheque(prefix, et): if (et == "file"): mk_cmd = "echo orig > %s" rm_cmd = "rm %s" rt_cmd = "echo rest > %s" elif (et == "dir"): mk_cmd = "mkdir -p %s/orig" rm_cmd = "rm -Rf %s" rt_cmd = "mkdir -p %s/rest" for mk_ct in [0, 1, 2]: for rm_ct in [0, 1, INF]: if ( (rm_ct == INF and mk_ct == 0) or (rm_ct != INF and rm_ct > mk_ct)): continue for rt_ct in [0, 1, 2]: if (rt_ct > rm_ct or rt_ct > mk_ct): continue base = "%s/%s_mk-%d_rm-%d_rt-%d" % (prefix, et, mk_ct, rm_ct, rt_ct) mks = ["mkdir %s" % base] rms = [] print("") for mk in range(mk_ct): mks.append(mk_cmd % ("%s/%d" % (base, mk))) if (rm_ct == INF): rms.append(rm_cmd % ("%s/*" % base)) else: for rm in range(rm_ct): rms.append(rm_cmd % ("%s/%d" % (base, rm))) for rt in range(rt_ct): rms.append(rt_cmd % ("%s/%d" % (base, rt))) if (len(mks) > 0): print("RUN " + " && ".join(mks)) if (len(rms) > 0): print("RUN " + " && ".join(rms)) print("FROM alpine:3.17") print("RUN mkdir /w /w/v") discotheque("/w", "file") discotheque("/w", "dir") discotheque("/w/v", "file") discotheque("/w/v", "dir") print("") print("RUN ls -aR /w") print("RUN find /w -type f -exec sh -c 'printf \"{} \" && cat {}' \; | sort")