././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.049557
spython-0.3.13/ 0000755 0001751 0000177 00000000000 14536152245 012716 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/LICENSE 0000644 0001751 0000177 00000036726 14536152223 013735 0 ustar 00runner docker Mozilla Public License, version 2.0
1. Definitions
1.1. “Contributor”
means each individual or legal entity that creates, contributes to the
creation of, or owns Covered Software.
1.2. “Contributor Version”
means the combination of the Contributions of others (if any) used by a
Contributor and that particular Contributor’s Contribution.
1.3. “Contribution”
means Covered Software of a particular Contributor.
1.4. “Covered Software”
means Source Code Form to which the initial Contributor has attached the
notice in Exhibit A, the Executable Form of such Source Code Form,
and Modifications of such Source Code Form, in each case
including portions thereof.
1.5. “Incompatible With Secondary Licenses”
means
a. that the initial Contributor has attached the notice described
in Exhibit B to the Covered Software; or
b. that the Covered Software was made available under the terms of
version 1.1 or earlier of the License, but not also under the terms
of a Secondary License.
1.6. “Executable Form”
means any form of the work other than Source Code Form.
1.7. “Larger Work”
means a work that combines Covered Software with other material,
in a separate file or files, that is not Covered Software.
1.8. “License”
means this document.
1.9. “Licensable”
means having the right to grant, to the maximum extent possible,
whether at the time of the initial grant or subsequently,
any and all of the rights conveyed by this License.
1.10. “Modifications”
means any of the following:
a. any file in Source Code Form that results from an addition to,
deletion from, or modification of the contents of Covered Software; or
b. any new file in Source Code Form that contains any Covered Software.
1.11. “Patent Claims” of a Contributor
means any patent claim(s), including without limitation, method, process,
and apparatus claims, in any patent Licensable by such Contributor that
would be infringed, but for the grant of the License, by the making,
using, selling, offering for sale, having made, import, or transfer of
either its Contributions or its Contributor Version.
1.12. “Secondary License”
means either the GNU General Public License, Version 2.0, the
GNU Lesser General Public License, Version 2.1, the GNU Affero General
Public License, Version 3.0, or any later versions of those licenses.
1.13. “Source Code Form”
means the form of the work preferred for making modifications.
1.14. “You” (or “Your”)
means an individual or a legal entity exercising rights under this License.
For legal entities, “You” includes any entity that controls,
is controlled by, or is under common control with You. For purposes of
this definition, “control” means (a) the power, direct or indirect,
to cause the direction or management of such entity, whether by contract
or otherwise, or (b) ownership of more than fifty percent (50%) of the
outstanding shares or beneficial ownership of such entity.
2. License Grants and Conditions
2.1. Grants
Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:
a. under intellectual property rights (other than patent or trademark)
Licensable by such Contributor to use, reproduce, make available,
modify, display, perform, distribute, and otherwise exploit its
Contributions, either on an unmodified basis, with Modifications,
or as part of a Larger Work; and
b. under Patent Claims of such Contributor to make, use, sell,
offer for sale, have made, import, and otherwise transfer either
its Contributions or its Contributor Version.
2.2. Effective Date
The licenses granted in Section 2.1 with respect to any Contribution
become effective for each Contribution on the date the Contributor
first distributes such Contribution.
2.3. Limitations on Grant Scope
The licenses granted in this Section 2 are the only rights granted
under this License. No additional rights or licenses will be implied
from the distribution or licensing of Covered Software under this License.
Notwithstanding Section 2.1(b) above, no patent license is granted
by a Contributor:
a. for any code that a Contributor has removed from
Covered Software; or
b. for infringements caused by: (i) Your and any other third party’s
modifications of Covered Software, or (ii) the combination of its
Contributions with other software (except as part of its
Contributor Version); or
c. under Patent Claims infringed by Covered Software in the
absence of its Contributions.
This License does not grant any rights in the trademarks, service marks,
or logos of any Contributor (except as may be necessary to comply with
the notice requirements in Section 3.4).
2.4. Subsequent Licenses
No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this
License (see Section 10.2) or under the terms of a Secondary License
(if permitted under the terms of Section 3.3).
2.5. Representation
Each Contributor represents that the Contributor believes its
Contributions are its original creation(s) or it has sufficient rights
to grant the rights to its Contributions conveyed by this License.
2.6. Fair Use
This License is not intended to limit any rights You have under
applicable copyright doctrines of fair use, fair dealing,
or other equivalents.
2.7. Conditions
Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the
licenses granted in Section 2.1.
3. Responsibilities
3.1. Distribution of Source Form
All distribution of Covered Software in Source Code Form, including
any Modifications that You create or to which You contribute, must be
under the terms of this License. You must inform recipients that the
Source Code Form of the Covered Software is governed by the terms
of this License, and how they can obtain a copy of this License.
You may not attempt to alter or restrict the recipients’ rights
in the Source Code Form.
3.2. Distribution of Executable Form
If You distribute Covered Software in Executable Form then:
a. such Covered Software must also be made available in Source Code
Form, as described in Section 3.1, and You must inform recipients of
the Executable Form how they can obtain a copy of such Source Code
Form by reasonable means in a timely manner, at a charge no more than
the cost of distribution to the recipient; and
b. You may distribute such Executable Form under the terms of this
License, or sublicense it under different terms, provided that the
license for the Executable Form does not attempt to limit or alter
the recipients’ rights in the Source Code Form under this License.
3.3. Distribution of a Larger Work
You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for
the Covered Software. If the Larger Work is a combination of
Covered Software with a work governed by one or more Secondary Licenses,
and the Covered Software is not Incompatible With Secondary Licenses,
this License permits You to additionally distribute such Covered Software
under the terms of such Secondary License(s), so that the recipient of
the Larger Work may, at their option, further distribute the
Covered Software under the terms of either this License or such
Secondary License(s).
3.4. Notices
You may not remove or alter the substance of any license notices
(including copyright notices, patent notices, disclaimers of warranty,
or limitations of liability) contained within the Source Code Form of
the Covered Software, except that You may alter any license notices to
the extent required to remedy known factual inaccuracies.
3.5. Application of Additional Terms
You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of
Covered Software. However, You may do so only on Your own behalf,
and not on behalf of any Contributor. You must make it absolutely clear
that any such warranty, support, indemnity, or liability obligation is
offered by You alone, and You hereby agree to indemnify every Contributor
for any liability incurred by such Contributor as a result of warranty,
support, indemnity or liability terms You offer. You may include
additional disclaimers of warranty and limitations of liability
specific to any jurisdiction.
4. Inability to Comply Due to Statute or Regulation
If it is impossible for You to comply with any of the terms of this License
with respect to some or all of the Covered Software due to statute,
judicial order, or regulation then You must: (a) comply with the terms of
this License to the maximum extent possible; and (b) describe the limitations
and the code they affect. Such description must be placed in a text file
included with all distributions of the Covered Software under this License.
Except to the extent prohibited by statute or regulation, such description
must be sufficiently detailed for a recipient of ordinary skill
to be able to understand it.
5. Termination
5.1. The rights granted under this License will terminate automatically
if You fail to comply with any of its terms. However, if You become
compliant, then the rights granted under this License from a particular
Contributor are reinstated (a) provisionally, unless and until such
Contributor explicitly and finally terminates Your grants, and (b) on an
ongoing basis, if such Contributor fails to notify You of the
non-compliance by some reasonable means prior to 60 days after You have
come back into compliance. Moreover, Your grants from a particular
Contributor are reinstated on an ongoing basis if such Contributor
notifies You of the non-compliance by some reasonable means,
this is the first time You have received notice of non-compliance with
this License from such Contributor, and You become compliant prior to
30 days after Your receipt of the notice.
5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions,
counter-claims, and cross-claims) alleging that a Contributor Version
directly or indirectly infringes any patent, then the rights granted
to You by any and all Contributors for the Covered Software under
Section 2.1 of this License shall terminate.
5.3. In the event of termination under Sections 5.1 or 5.2 above, all
end user license agreements (excluding distributors and resellers) which
have been validly granted by You or Your distributors under this License
prior to termination shall survive termination.
6. Disclaimer of Warranty
Covered Software is provided under this License on an “as is” basis, without
warranty of any kind, either expressed, implied, or statutory, including,
without limitation, warranties that the Covered Software is free of defects,
merchantable, fit for a particular purpose or non-infringing. The entire risk
as to the quality and performance of the Covered Software is with You.
Should any Covered Software prove defective in any respect, You
(not any Contributor) assume the cost of any necessary servicing, repair,
or correction. This disclaimer of warranty constitutes an essential part of
this License. No use of any Covered Software is authorized under this
License except under this disclaimer.
7. Limitation of Liability
Under no circumstances and under no legal theory, whether tort
(including negligence), contract, or otherwise, shall any Contributor, or
anyone who distributes Covered Software as permitted above, be liable to
You for any direct, indirect, special, incidental, or consequential damages
of any character including, without limitation, damages for lost profits,
loss of goodwill, work stoppage, computer failure or malfunction, or any and
all other commercial damages or losses, even if such party shall have been
informed of the possibility of such damages. This limitation of liability
shall not apply to liability for death or personal injury resulting from
such party’s negligence to the extent applicable law prohibits such
limitation. Some jurisdictions do not allow the exclusion or limitation of
incidental or consequential damages, so this exclusion and limitation may
not apply to You.
8. Litigation
Any litigation relating to this License may be brought only in the courts of
a jurisdiction where the defendant maintains its principal place of business
and such litigation shall be governed by laws of that jurisdiction, without
reference to its conflict-of-law provisions. Nothing in this Section shall
prevent a party’s ability to bring cross-claims or counter-claims.
9. Miscellaneous
This License represents the complete agreement concerning the subject matter
hereof. If any provision of this License is held to be unenforceable,
such provision shall be reformed only to the extent necessary to make it
enforceable. Any law or regulation which provides that the language of a
contract shall be construed against the drafter shall not be used to construe
this License against a Contributor.
10. Versions of the License
10.1. New Versions
Mozilla Foundation is the license steward. Except as provided in
Section 10.3, no one other than the license steward has the right to
modify or publish new versions of this License. Each version will be
given a distinguishing version number.
10.2. Effect of New Versions
You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published
by the license steward.
10.3. Modified Versions
If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a modified
version of this License if you rename the license and remove any
references to the name of the license steward (except to note that such
modified license differs from this License).
10.4. Distributing Source Code Form that is
Incompatible With Secondary Licenses
If You choose to distribute Source Code Form that is
Incompatible With Secondary Licenses under the terms of this version of
the License, the notice described in Exhibit B of this
License must be attached.
Exhibit A - Source Code Form License Notice
This Source Code Form is subject to the terms of the
Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular file,
then You may include the notice in a location (such as a LICENSE file in a
relevant directory) where a recipient would be likely to
look for such a notice.
You may add additional accurate notices of copyright ownership.
Exhibit B - “Incompatible With Secondary Licenses” Notice
This Source Code Form is “Incompatible With Secondary Licenses”,
as defined by the Mozilla Public License, v. 2.0.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/MANIFEST.in 0000644 0001751 0000177 00000000326 14536152223 014451 0 ustar 00runner docker include README.md LICENSE
recursive-include spython *
recursive-exclude * __pycache__
recursive-exclude * *.pyc
recursive-exclude * *.pyo
recursive-exclude * *.simg
recursive-exclude * *.img
prune docs
prune .docs
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.049557
spython-0.3.13/PKG-INFO 0000644 0001751 0000177 00000020625 14536152245 014020 0 ustar 00runner docker Metadata-Version: 2.1
Name: spython
Version: 0.3.13
Summary: Command line python tool for working with singularity.
Home-page: https://github.com/singularityhub/singularity-cli
Author: Vanessa Sochat
Author-email: vsoch@users.noreply.github.com
Maintainer: Vanessa Sochat
Maintainer-email: vsoch@users.noreply.github.com
License: LICENSE
Keywords: singularity python client (spython)
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development
Classifier: Topic :: Scientific/Engineering
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python :: 3
Description-Content-Type: text/markdown
License-File: LICENSE
# Singularity Python
[](https://travis-ci.org/singularityhub/singularity-cli)
[](https://github.com/singularityhub/singularity-cli/actions?query=branch%3Amaster+workflow%3Aspython-ci)
Singularity Python (spython) is the Python API for working with Singularity containers. See
the [documentation](https://singularityhub.github.io/singularity-cli) for installation and usage, and
the [install instructions](https://singularityhub.github.io/singularity-cli/install) for a quick start.
**This library does not support Singularity 2.x! It won't work and we no longer support it.**
We provide a [Singularity](Singularity) recipe for you to use if more convenient, along with the [full modules docstring](https://singularityhub.github.io/singularity-cli/api/source/spython.main.base.html#module-spython.main.base).
As of version 0.1.0, we only support Singularity > 3.5.2. This is done to encourage using
newer versions of Singularity with security fixes. If you want to use an older version of Singularity,
you will need to use version 0.0.85 or earlier.
## 😁️ Contributors 😁️
We use the [all-contributors](https://github.com/all-contributors/all-contributors)
tool to generate a contributors graphic below.
## License
This code is licensed under the MPL 2.0 [LICENSE](LICENSE).
## Help and Contribution
Please contribute to the package, or post feedback and questions as issues. For points that require discussion of the larger group, please use the Singularity List
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/README.md 0000644 0001751 0000177 00000017242 14536152223 014177 0 ustar 00runner docker # Singularity Python
[](https://travis-ci.org/singularityhub/singularity-cli)
[](https://github.com/singularityhub/singularity-cli/actions?query=branch%3Amaster+workflow%3Aspython-ci)
Singularity Python (spython) is the Python API for working with Singularity containers. See
the [documentation](https://singularityhub.github.io/singularity-cli) for installation and usage, and
the [install instructions](https://singularityhub.github.io/singularity-cli/install) for a quick start.
**This library does not support Singularity 2.x! It won't work and we no longer support it.**
We provide a [Singularity](Singularity) recipe for you to use if more convenient, along with the [full modules docstring](https://singularityhub.github.io/singularity-cli/api/source/spython.main.base.html#module-spython.main.base).
As of version 0.1.0, we only support Singularity > 3.5.2. This is done to encourage using
newer versions of Singularity with security fixes. If you want to use an older version of Singularity,
you will need to use version 0.0.85 or earlier.
## 😁️ Contributors 😁️
We use the [all-contributors](https://github.com/all-contributors/all-contributors)
tool to generate a contributors graphic below.
## License
This code is licensed under the MPL 2.0 [LICENSE](LICENSE).
## Help and Contribution
Please contribute to the package, or post feedback and questions as issues. For points that require discussion of the larger group, please use the Singularity List
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/pyproject.toml 0000644 0001751 0000177 00000000204 14536152223 015622 0 ustar 00runner docker [tool.black]
profile = "black"
exclude = ["^env/"]
[tool.isort]
profile = "black" # needed for black/isort compatibility
skip = []
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.049557
spython-0.3.13/setup.cfg 0000644 0001751 0000177 00000000404 14536152245 014535 0 ustar 00runner docker [flake8]
exclude = benchmarks docs
max-line-length = 100
ignore = E1 E2 E5 W5
per-file-ignores =
spython/__init__.py:F401
spython/utils/__init__.py:F401
spython/clint/__init__.py:F401
spython/logger/__init__.py:F401
[egg_info]
tag_build =
tag_date = 0
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/setup.py 0000644 0001751 0000177 00000006450 14536152223 014431 0 ustar 00runner docker import os
from setuptools import find_packages, setup
################################################################################
# HELPER FUNCTIONS #############################################################
################################################################################
def get_lookup():
"""get version by way of singularity.version, returns a
lookup dictionary with several global variables without
needing to import spython
"""
lookup = dict()
version_file = os.path.join("spython", "version.py")
with open(version_file) as filey:
exec(filey.read(), lookup)
return lookup
def get_requirements(lookup=None):
"""get_requirements reads in requirements and versions from
the lookup obtained with get_lookup"""
if lookup is None:
lookup = get_lookup()
install_requires = []
for module in lookup["INSTALL_REQUIRES"]:
module_name = module[0]
module_meta = module[1]
if "exact_version" in module_meta:
dependency = "%s==%s" % (module_name, module_meta["exact_version"])
elif "min_version" in module_meta:
if module_meta["min_version"] is None:
dependency = module_name
else:
dependency = "%s>=%s" % (module_name, module_meta["min_version"])
install_requires.append(dependency)
return install_requires
# Make sure everything is relative to setup.py
install_path = os.path.dirname(os.path.abspath(__file__))
os.chdir(install_path)
# Get version information from the lookup
lookup = get_lookup()
VERSION = lookup["__version__"]
NAME = lookup["NAME"]
AUTHOR = lookup["AUTHOR"]
AUTHOR_EMAIL = lookup["AUTHOR_EMAIL"]
PACKAGE_URL = lookup["PACKAGE_URL"]
KEYWORDS = lookup["KEYWORDS"]
DESCRIPTION = lookup["DESCRIPTION"]
LICENSE = lookup["LICENSE"]
with open("README.md") as readme:
LONG_DESCRIPTION = readme.read()
##########################################################################################
# MAIN ###################################################################################
##########################################################################################
if __name__ == "__main__":
INSTALL_REQUIRES = get_requirements(lookup)
TESTS_REQUIRES = get_requirements(lookup)
setup(
name=NAME,
version=VERSION,
author=AUTHOR,
author_email=AUTHOR_EMAIL,
maintainer=AUTHOR,
maintainer_email=AUTHOR_EMAIL,
packages=find_packages(),
include_package_data=True,
zip_safe=False,
url=PACKAGE_URL,
license=LICENSE,
description=DESCRIPTION,
long_description=LONG_DESCRIPTION,
long_description_content_type="text/markdown",
keywords=KEYWORDS,
setup_requires=["pytest-runner"],
tests_require=TESTS_REQUIRES,
install_requires=INSTALL_REQUIRES,
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"Programming Language :: Python",
"Topic :: Software Development",
"Topic :: Scientific/Engineering",
"Operating System :: Unix",
"Programming Language :: Python :: 3",
],
entry_points={"console_scripts": ["spython=spython.client:main"]},
)
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.037557
spython-0.3.13/spython/ 0000755 0001751 0000177 00000000000 14536152245 014422 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/README.md 0000644 0001751 0000177 00000002405 14536152223 015676 0 ustar 00runner docker # Singularity Python Organization
This will briefly outline the content of the folders here.
## Main Functions
- [main](main) holds the primary client functions to interact with Singularity (e.g., exec, run, pull), and the subfolders within represent command groups (e.g., instance, image) along with the base of the client ([base](main/base)). This folder is where **commands** go!
- [image](image.py) is a class that represents and holds an image object, in the case that the user wants to initialize a client with an image. This holds the ImageClass, which is only needed to instantiate an image.
- [instance](instance) is a class that represents and holds an instance object. The user can instantiate the class, and then run it's sub functions to interact with it. The higher level "list" command is provided on the level of the client.
- [cli](cli): is the actual entry point that connects the user to the python API client from the command line. The user inputs are parsed, and then passed into functions from main.
## Supporting
- [utils](utils) are various file and other utilities shared across submodules.
- [logger](logger) includes functions for progress bars, and logging levels.
- [tests](tests) are important for continuous integration, but likely overlooked.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/__init__.py 0000644 0001751 0000177 00000000050 14536152223 016522 0 ustar 00runner docker from spython.version import __version__
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.037557
spython-0.3.13/spython/client/ 0000755 0001751 0000177 00000000000 14536152245 015700 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/client/__init__.py 0000644 0001751 0000177 00000011143 14536152223 020005 0 ustar 00runner docker #!/usr/bin/env python
# Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import argparse
import os
import sys
def get_parser():
parser = argparse.ArgumentParser(
description="Singularity Client",
formatter_class=argparse.RawTextHelpFormatter,
add_help=False,
)
# Global Options
parser.add_argument(
"--debug",
"-d",
dest="debug",
help="use verbose logging to debug.",
default=False,
action="store_true",
)
parser.add_argument(
"--quiet",
"-q",
dest="quiet",
help="suppress all normal output",
default=False,
action="store_true",
)
parser.add_argument(
"--version",
dest="version",
help="show singularity and spython version",
default=False,
action="store_true",
)
subparsers = parser.add_subparsers(
help="description",
title="actions",
description="actions for Singularity",
dest="command",
metavar="general usage",
)
# Recipes
recipe = subparsers.add_parser("recipe", help="Recipe conversion and parsing")
recipe.add_argument(
"--entrypoint",
dest="entrypoint",
help="define custom entry point and prevent discovery",
default=None,
type=str,
)
recipe.add_argument(
"--json",
dest="json",
help="dump the (base) recipe content as json to the terminal",
default=False,
action="store_true",
)
recipe.add_argument(
"--force",
dest="force",
help="if the output file exists, overwrite.",
default=False,
action="store_true",
)
recipe.add_argument(
"files",
nargs="*",
help="the recipe input file and [optional] output file",
type=str,
)
recipe.add_argument(
"--parser",
type=str,
default="auto",
dest="parser",
choices=["auto", "docker", "singularity"],
help="Is the input a Dockerfile or Singularity recipe?",
)
recipe.add_argument(
"--writer",
type=str,
default="auto",
dest="writer",
choices=["auto", "docker", "singularity"],
help="Should we write to Dockerfile or Singularity recipe?",
)
# General Commands
subparsers.add_parser("shell", help="Interact with singularity python")
subparsers.add_parser("test", help="""Container testing (TBD)""")
return parser
def set_verbosity(args):
"""determine the message level in the environment to set based on args."""
level = "INFO"
if args.debug:
level = "DEBUG"
elif args.quiet:
level = "QUIET"
os.environ["MESSAGELEVEL"] = level
os.putenv("MESSAGELEVEL", level)
os.environ["SINGULARITY_MESSAGELEVEL"] = level
os.putenv("SINGULARITY_MESSAGELEVEL", level)
# Import logger to set
from spython.logger import bot
bot.debug("Logging level %s" % level)
import spython
bot.debug("Singularity Python Version: %s" % spython.__version__)
def version():
"""version prints the version, both for the user and help output"""
import spython
return spython.__version__
def main():
parser = get_parser()
def print_help(return_code=0):
"""print help, including the software version and active client
and exit with return code.
"""
v = version()
print("\nSingularity Python [v%s]\n" % (v))
parser.print_help()
sys.exit(return_code)
if len(sys.argv) == 1:
print_help()
try:
# We capture all primary arguments, and take secondary to pass on
args, options = parser.parse_known_args()
except Exception:
sys.exit(0)
# The main function
func = None
# If the user wants the version
if args.version:
print(version())
sys.exit(0)
# if environment logging variable not set, make silent
set_verbosity(args)
# Does the user want help for a subcommand?
if args.command == "recipe":
from .recipe import main as func
elif args.command == "shell":
from .shell import main as func
elif args.command == "test":
from .test import main as func
else:
print_help()
# Pass on to the correct parser
if args.command is not None:
func(args=args, options=options, parser=parser)
if __name__ == "__main__":
main()
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/client/recipe.py 0000644 0001751 0000177 00000006042 14536152223 017517 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import json
import os
import sys
from spython.logger import bot
from spython.main.parse.parsers import get_parser
from spython.main.parse.writers import get_writer
from spython.utils import write_file, write_json
def main(args, options, parser):
"""This function serves as a wrapper around the DockerParser,
SingularityParser, DockerWriter, and SingularityParser converters.
We can either save to file if args.outfile is defined, or print
to the console if not.
"""
# We need something to work with
if not args.files:
parser.print_help()
sys.exit(1)
# Get the user specified input and output files
outfile = None
if len(args.files) > 1:
outfile = args.files[1]
# First try to get writer and parser, if not defined will return None
writer = get_writer(args.writer)
parser = get_parser(args.parser)
# If the user wants to auto-detect the type
if args.parser == "auto":
if "dockerfile" in args.files[0].lower():
parser = get_parser("docker")
elif "singularity" in args.files[0].lower():
parser = get_parser("singularity")
# If the parser still isn't defined, no go.
if parser is None:
bot.exit(
"Please provide a Dockerfile or Singularity recipe, or define the --parser type."
)
# If the writer needs auto-detect
if args.writer == "auto":
if parser.name == "docker":
writer = get_writer("singularity")
else:
writer = get_writer("docker")
# If the writer still isn't defined, no go
if writer is None:
bot.exit("Please define the --writer type.")
# Initialize the chosen parser
recipeParser = parser(args.files[0])
# By default, discover entrypoint / cmd from Dockerfile
entrypoint = "/bin/bash"
force = False
if args.entrypoint is not None:
entrypoint = args.entrypoint
# This is only done if the user intended to print json here
recipeParser.entrypoint = args.entrypoint
recipeParser.cmd = None
force = True
if args.json:
if outfile is not None:
if not os.path.exists(outfile):
if force:
write_json(outfile, recipeParser.recipe.json())
else:
bot.exit("%s exists, set --force to overwrite." % outfile)
else:
print(json.dumps(recipeParser.recipe.json(), indent=4))
else:
# Do the conversion
recipeWriter = writer(recipeParser.recipe)
result = recipeWriter.convert(runscript=entrypoint, force=force)
# If the user specifies an output file, save to it
if outfile is not None:
write_file(outfile, result)
# Otherwise, convert and print to screen
else:
print(result)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/client/shell.py 0000644 0001751 0000177 00000003316 14536152223 017360 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
def main(args, options, parser):
# If we have options, first is image
image = None
if options:
image = options.pop(0)
lookup = {"ipython": ipython, "python": python, "bpython": run_bpython}
shells = ["ipython", "python", "bpython"]
# Otherwise present order of liklihood to have on system
for shell in shells:
try:
return lookup[shell](image)
except ImportError:
pass
def prepare_client(image):
"""prepare a client to embed in a shell with recipe parsers and writers."""
# The client will announce itself (backend/database) unless it's get
from spython.main import get_client
from spython.main.parse import parsers, writers
client = get_client()
if image:
client.load(image)
# Add recipe parsers
client.parsers = parsers
client.writers = writers
return client
def ipython(image):
"""give the user an ipython shell"""
client = prepare_client(image) # noqa
try:
from IPython import embed
except ImportError:
return python(image)
embed()
def run_bpython(image):
"""give the user a bpython shell"""
client = prepare_client(image)
try:
import bpython
except ImportError:
return python(image)
bpython.embed(locals_={"client": client})
def python(image):
"""give the user a python shell"""
import code
client = prepare_client(image)
code.interact(local={"client": client})
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/client/test.py 0000644 0001751 0000177 00000000724 14536152223 017230 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
def main(args, options, parser):
print("TBA, additional tests for Singularity containers.")
print("What would you like to see? Let us know!")
print("https://www.github.com/singularityhub/singularity-cli/issues")
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/image.py 0000644 0001751 0000177 00000004257 14536152223 016062 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import hashlib
import os
from spython.logger import bot
from spython.utils import split_uri
class ImageBase:
def __str__(self):
protocol = getattr(self, "protocol", None)
if protocol:
return "%s://%s" % (protocol, self.image)
return self.image
def __repr__(self):
return self.__str__()
def parse_image_name(self, image):
"""
simply split the uri from the image. Singularity handles
parsing of registry, namespace, image.
Parameters
==========
image: the complete image uri to load (e.g., docker://ubuntu)
"""
self._image = image
self.protocol, self.image = split_uri(image)
class Image(ImageBase):
def __init__(self, image=None):
"""An image here is an image file or a record.
The user can choose to load the image when starting the client, or
update the main client with an image. The image object is kept
with the main client to make running additional commands easier.
Parameters
==========
image: the image uri to parse (required)
"""
super(Image, self).__init__()
self.parse_image_name(image)
def get_hash(self, image=None):
"""return an md5 hash of the file based on a criteria level. This
is intended to give the file a reasonable version. This only is
useful for actual image files.
Parameters
==========
image: the image path to get hash for (first priority). Second
priority is image path saved with image object, if exists.
"""
hasher = hashlib.md5()
image = image or self.image
if os.path.exists(image):
with open(image, "rb") as f:
for chunk in iter(lambda: f.read(4096), b""):
hasher.update(chunk)
return hasher.hexdigest()
bot.warning("%s does not exist." % image)
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.037557
spython-0.3.13/spython/instance/ 0000755 0001751 0000177 00000000000 14536152245 016226 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/instance/__init__.py 0000644 0001751 0000177 00000007265 14536152223 020345 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
from spython.image import ImageBase
from spython.utils.fileio import read_file
class Instance(ImageBase):
def __init__(self, image, start=True, name=None, quiet=True, **kwargs):
"""An instance is an image running as an instance with services.
This class has functions appended under cmd/__init__ and is
instantiated when the user calls Client.
Parameters
==========
image: the Singularity image uri to parse (required)
start: boolean to start the instance (default is True)
name: a name for the instance (will generate RobotName
if not provided)
"""
super(Instance, self).__init__()
self.parse_image_name(image)
self.generate_name(name)
# Update metadats from arguments
self._update_metadata(kwargs)
self.options = kwargs.get("options", [])
self.args = kwargs.get("args", [])
self.cmd = []
self.quiet = quiet
# Start the instance
if start:
self.start(quiet=quiet, **kwargs)
# Unique resource identifier
def generate_name(self, name=None):
"""generate a Robot Name for the instance to use, if the user doesn't
supply one.
"""
# If no name provided, use robot name
if name is None:
name = self.RobotNamer.generate()
# dash allowed in instance name.
# authorizedChars = `^[a-zA-Z0-9._-]+$` from instance_linux.go
# self.name = name.replace("-", "_")
self.name = name
def parse_image_name(self, image):
"""
simply split the uri from the image. Singularity handles
parsing of registry, namespace, image.
Parameters
==========
image: the complete image uri to load (e.g., docker://ubuntu)
"""
self._image = image
self.protocol = "instance"
def get_uri(self):
"""return the image uri (instance://) along with it's name"""
return self.__str__()
# Logs
def read_logs_out(self):
"""Read output log file, if it exists"""
if hasattr(self, "log_out_path"):
if os.path.exists(self.log_out_path):
return read_file(self.log_out_path)
def read_logs_err(self):
"""Read error log file, if it exists"""
if hasattr(self, "log_err_path"):
if os.path.exists(self.log_err_path):
return read_file(self.log_err_path)
# Metadata
def _update_metadata(self, kwargs=None):
"""Extract any additional attributes to hold with the instance
from kwargs
"""
# If not given metadata, use instance.list to get it for container
if kwargs is None and hasattr(self, "name"):
kwargs = self._list(self.name, quiet=True, return_json=True)
# Add acceptable arguments
for arg in ["pid", "name", "ip_address", "log_err_path", "log_out_path", "img"]:
# Skip over non-iterables:
if arg in kwargs:
setattr(self, arg, kwargs[arg])
if "image" in kwargs:
self._image = kwargs["image"]
elif "container_image" in kwargs:
self._image = kwargs["container_image"]
def __str__(self):
if hasattr(self, "name"):
if self.protocol:
return "%s://%s" % (self.protocol, self.name)
return os.path.basename(self._image)
def __repr__(self):
return self.__str__()
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.037557
spython-0.3.13/spython/instance/cmd/ 0000755 0001751 0000177 00000000000 14536152245 016771 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/instance/cmd/__init__.py 0000644 0001751 0000177 00000002653 14536152223 021104 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
def generate_instance_commands():
"""The Instance client holds the Singularity Instance command group
The levels of verbosity (debug and quiet) are passed from the main
client via the environment variable MESSAGELEVEL.
"""
from spython.instance import Instance
# run_command uses run_cmd, but wraps to catch error
from spython.main.base.command import init_command, run_command
from spython.main.base.generate import RobotNamer
from spython.main.base.logger import println
from spython.main.instances import list_instances
from .logs import _logs, error_logs, output_logs
from .start import start
from .stop import stop
Instance.RobotNamer = RobotNamer()
Instance._init_command = init_command
Instance.run_command = run_command
Instance._list = list_instances # list command is used to get metadata
Instance._println = println
Instance.start = start # intended to be called on init, not by user
Instance.stop = stop
Instance.error_logs = error_logs
Instance.output_logs = output_logs
Instance._logs = _logs
# Give an instance the ability to breed :)
Instance.instance = Instance
return Instance
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/instance/cmd/logs.py 0000644 0001751 0000177 00000003610 14536152223 020303 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import platform
from spython.logger import bot
from spython.utils import get_userhome, get_username
def error_logs(self, print_logs=False):
"""For Singularity 3.5 and later, we are able to programatically
derive the name of the log. In this case, return the content
to the user. See
https://github.com/sylabs/singularity/issues/1115#issuecomment-560457918
for when this was added.
Parameters
==========
print_logs: boolean to indicate to print to the screen along with
return (defaults to False to just return log string)
"""
return self._logs(print_logs, "err")
def output_logs(self, print_logs=False):
"""Get output logs for the user, if they exist.
Parameters
==========
print_logs: boolean to indicate to print to the screen along with
return (defaults to False to just return log string)
"""
return self._logs(print_logs, "out")
def _logs(self, print_logs=False, ext="out"):
"""A shared function to print log files. The only differing element is
the extension (err or out)
"""
from spython.utils import check_install
check_install()
# Formulate the path of the logs
hostname = platform.node()
logpath = os.path.join(
get_userhome(),
".singularity",
"instances",
"logs",
hostname,
get_username(),
"%s.%s" % (self.name, ext),
)
if os.path.exists(logpath):
with open(logpath, "r") as filey:
logs = filey.read()
if print_logs is True:
print(logs)
else:
bot.warning("No log files have been produced.")
return logs
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/instance/cmd/start.py 0000644 0001751 0000177 00000005473 14536152223 020505 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
from spython.logger import bot
def start(
self,
image=None,
name=None,
args=None,
sudo=False,
sudo_options=None,
options=None,
capture=False,
singularity_options=None,
environ=None,
quiet=True,
):
"""start an instance. This is done by default when an instance is created.
Parameters
==========
image: optionally, an image uri (if called as a command from Client)
name: a name for the instance
sudo: if the user wants to run the command with sudo
capture: capture output, default is False. With True likely to hang.
args: arguments to provide to the instance (supported Singularity 3.1+)
singularity_options: a list of options to provide to the singularity client
quiet: Do not print verbose output.
options: a list of tuples, each an option to give to the start command
[("--bind", "/tmp"),...]
USAGE:
singularity [...] instance.start [...]
"""
from spython.utils import check_install, run_command
check_install()
# If name provided, over write robot (default)
if name is not None:
self.name = name
# If an image isn't provided, we have an initialized instance
if image is None:
# Not having this means it was called as a command, without an image
if not hasattr(self, "_image"):
bot.exit("Please provide an image, or create an Instance first.")
image = self._image
cmd = self._init_command(["instance", "start"], singularity_options)
# Set options and args
args = args or self.args
options = options or self.options
# Add options, if they are provided
if not isinstance(options, list):
options = [] if options is None else options.split(" ")
# Assemble the command!
cmd = cmd + options + [image, self.name]
# If arguments are provided
if args is not None:
if not isinstance(args, list):
args = [args]
cmd = cmd + args
# Print verbose output
if not (quiet or self.quiet):
bot.info(" ".join(cmd))
# Save the options and cmd, if the user wants to see them later
self.options = options
self.args = args
self.cmd = cmd
output = run_command(
cmd,
sudo=sudo,
sudo_options=sudo_options,
quiet=True,
capture=capture,
environ=environ,
)
if output["return_code"] == 0:
self._update_metadata()
else:
message = "%s : return code %s" % (output["message"], output["return_code"])
bot.error(message)
return self
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/instance/cmd/stop.py 0000644 0001751 0000177 00000003266 14536152223 020333 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
from spython.logger import bot
def stop(
self,
name=None,
sudo=False,
sudo_options=None,
timeout=None,
singularity_options=None,
quiet=True,
):
"""stop an instance. This is done by default when an instance is created.
Parameters
==========
name: a name for the instance
sudo: if the user wants to run the command with sudo
singularity_options: a list of options to provide to the singularity client
quiet: Do not print verbose output.
timeout: forcebly kill non-stopped instance after the
timeout specified in seconds
USAGE:
singularity [...] instance.stop [...]
"""
from spython.utils import check_install, run_command
check_install()
subgroup = ["instance", "stop"]
if timeout:
subgroup += ["-t", str(timeout)]
cmd = self._init_command(subgroup, singularity_options)
# If name is provided assume referencing an instance
instance_name = self.name
if name is not None:
instance_name = name
cmd = cmd + [instance_name]
# Print verbose output
if not (quiet or self.quiet):
bot.info(" ".join(cmd))
output = run_command(cmd, sudo=sudo, sudo_options=sudo_options, quiet=True)
if output["return_code"] != 0:
message = "%s : return code %s" % (output["message"], output["return_code"])
bot.error(message)
return output["return_code"]
return output["return_code"]
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.037557
spython-0.3.13/spython/logger/ 0000755 0001751 0000177 00000000000 14536152245 015701 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/logger/__init__.py 0000644 0001751 0000177 00000000147 14536152223 020010 0 ustar 00runner docker from .compatibility import decodeUtf8String
from .message import bot
from .progress import ProgressBar
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/logger/compatibility.py 0000644 0001751 0000177 00000001134 14536152223 021117 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
def decodeUtf8String(inputStr):
"""Convert an UTF8 sequence into a string
Required for compatibility with Python 2 where str==bytes
inputStr -- Either a str or bytes instance with UTF8 encoding
"""
return (
inputStr
if isinstance(inputStr, str) or not isinstance(inputStr, bytes)
else inputStr.decode("utf8")
)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/logger/message.py 0000644 0001751 0000177 00000022354 14536152223 017701 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import sys
from spython.logger import decodeUtf8String
from .spinner import Spinner
ABORT = -5
CRITICAL = -4
ERROR = -3
WARNING = -2
LOG = -1
INFO = 1
CUSTOM = 1
QUIET = 0
VERBOSE = VERBOSE1 = 2
VERBOSE2 = 3
VERBOSE3 = 4
DEBUG = 5
PURPLE = "\033[95m"
YELLOW = "\033[93m"
RED = "\033[91m"
DARKRED = "\033[31m"
CYAN = "\033[36m"
class SingularityMessage:
def __init__(self, MESSAGELEVEL=None):
self.level = get_logging_level()
self.history = []
self.errorStream = sys.stderr
self.outputStream = sys.stdout
self.colorize = self.useColor()
self.colors = {
ABORT: DARKRED,
CRITICAL: RED,
ERROR: RED,
WARNING: YELLOW,
LOG: PURPLE,
CUSTOM: PURPLE,
DEBUG: CYAN,
"OFF": "\033[0m", # end sequence
"CYAN": CYAN,
"PURPLE": PURPLE,
"RED": RED,
"DARKRED": DARKRED,
"YELLOW": YELLOW,
}
# Colors --------------------------------------------
def useColor(self):
"""useColor will determine if color should be added
to a print. Will check if being run in a terminal, and
if has support for ascii
"""
COLORIZE = get_user_color_preference()
if COLORIZE is not None:
return COLORIZE
streams = [self.errorStream, self.outputStream]
for stream in streams:
if not hasattr(stream, "isatty"):
return False
if not stream.isatty():
return False
return True
def addColor(self, level, text):
"""addColor to the prompt (usually prefix) if terminal
supports, and specified to do so
"""
if self.colorize:
if level in self.colors:
text = "%s%s%s" % (self.colors[level], text, self.colors["OFF"])
return text
def emitError(self, level):
"""determine if a level should print to
stderr, includes all levels but INFO and QUIET
"""
if level in [
ABORT,
ERROR,
WARNING,
VERBOSE,
VERBOSE1,
VERBOSE2,
VERBOSE3,
DEBUG,
]:
return True
return False
def emitOutput(self, level):
"""determine if a level should print to stdout
only includes INFO"""
if level in [LOG, INFO]:
return True
return False
def isEnabledFor(self, messageLevel):
"""check if a messageLevel is enabled to emit a level"""
if messageLevel <= self.level:
return True
return False
def emit(self, level, message, prefix=None, color=None):
"""emit is the main function to print the message
optionally with a prefix
:param level: the level of the message
:param message: the message to print
:param prefix: a prefix for the message
"""
if color is None:
color = level
if prefix is not None:
prefix = self.addColor(color, "%s " % (prefix))
else:
prefix = ""
message = self.addColor(color, message)
# Add the prefix
message = "%s%s" % (prefix, message)
if not message.endswith("\n"):
message = "%s\n" % message
# If the level is quiet, only print to error
if self.level == QUIET:
pass
# Otherwise if in range print to stdout and stderr
elif self.isEnabledFor(level):
if self.emitError(level):
self.write(self.errorStream, message)
else:
self.write(self.outputStream, message)
# Add all log messages to history
self.history.append(message)
def write(self, stream, message):
"""write will write a message to a stream,
first checking the encoding
"""
stream.write(decodeUtf8String(message))
def get_logs(self, join_newline=True):
"""'get_logs will return the complete history, joined by newline
(default) or as is.
"""
if join_newline:
return "\n".join(self.history)
return self.history
def show_progress(
self,
iteration,
total,
length=40,
min_level=0,
prefix=None,
carriage_return=True,
suffix=None,
symbol=None,
):
"""create a terminal progress bar, default bar shows for verbose+
:param iteration: current iteration (Int)
:param total: total iterations (Int)
:param length: character length of bar (Int)
"""
percent = 100 * (iteration / float(total))
progress = int(length * iteration // total)
if suffix is None:
suffix = ""
if prefix is None:
prefix = "Progress"
# Download sizes can be imperfect, setting carriage_return to False
# and writing newline with caller cleans up the UI
if percent >= 100:
percent = 100
progress = length
if symbol is None:
symbol = "="
if progress < length:
bar = symbol * progress + "|" + "-" * (length - progress - 1)
else:
bar = symbol * progress + "-" * (length - progress)
# Only show progress bar for level > min_level
if self.level > min_level:
percent = "%5s" % ("{0:.1f}").format(percent)
output = "\r" + prefix + " |%s| %s%s %s" % (bar, percent, "%", suffix)
sys.stdout.write(output)
if iteration == total and carriage_return:
sys.stdout.write("\n")
sys.stdout.flush()
# Logging ------------------------------------------
def abort(self, message):
self.emit(ABORT, message, "ABORT")
def critical(self, message):
self.emit(CRITICAL, message, "CRITICAL")
def error(self, message):
self.emit(ERROR, message, "ERROR")
def exit(self, message, return_code=1):
self.emit(ERROR, message, "ERROR")
sys.exit(return_code)
def warning(self, message):
self.emit(WARNING, message, "WARNING")
def log(self, message):
self.emit(LOG, message, "LOG")
def custom(self, prefix, message, color=PURPLE):
self.emit(CUSTOM, message, prefix, color)
def info(self, message):
self.emit(INFO, message)
def newline(self):
return self.info("")
def verbose(self, message):
self.emit(VERBOSE, message, "VERBOSE")
def println(self, message):
print(message)
def verbose1(self, message):
self.emit(VERBOSE, message, "VERBOSE1")
def verbose2(self, message):
self.emit(VERBOSE2, message, "VERBOSE2")
def verbose3(self, message):
self.emit(VERBOSE3, message, "VERBOSE3")
def debug(self, message):
self.emit(DEBUG, message, "DEBUG")
def is_quiet(self):
"""is_quiet returns true if the level is under 1"""
if self.level < 1:
return False
return True
# Terminal ------------------------------------------
def table(self, rows, col_width=2):
"""table will print a table of entries. If the rows is
a dictionary, the keys are interpreted as column names. if
not, a numbered list is used.
"""
labels = [str(x) for x in range(1, len(rows) + 1)]
if isinstance(rows, dict):
labels = list(rows.keys())
rows = list(rows.values())
for row in rows:
label = labels.pop(0)
label = label.ljust(col_width)
message = "\t".join(row)
self.custom(prefix=label, message=message)
def get_logging_level():
"""configure a logging to standard out based on the user's
selected level, which should be in an environment variable called
MESSAGELEVEL. if MESSAGELEVEL is not set, the info level
(1) is assumed (all informational messages).
"""
level = os.environ.get("MESSAGELEVEL", "1") # INFO
if level == "CRITICAL":
return CRITICAL
elif level == "ABORT":
return ABORT
elif level == "ERROR":
return ERROR
elif level == "WARNING":
return WARNING
elif level == "LOG":
return LOG
elif level == "INFO":
return INFO
elif level == "QUIET":
return QUIET
elif level.startswith("VERBOSE"):
return VERBOSE3
elif level == "LOG":
return LOG
elif level == "DEBUG":
return DEBUG
else:
level = int(level)
return level
def get_user_color_preference():
COLORIZE = os.environ.get("SINGULARITY_COLORIZE", None)
if COLORIZE is not None:
COLORIZE = convert2boolean(COLORIZE)
return COLORIZE
def convert2boolean(arg):
"""convert2boolean is used for environmental variables that must be
returned as boolean"""
if not isinstance(arg, bool):
return arg.lower() in ("yes", "true", "t", "1", "y")
return arg
SingularityMessage.spinner = Spinner()
bot = SingularityMessage()
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/logger/progress.py 0000644 0001751 0000177 00000010546 14536152223 020121 0 ustar 00runner docker # -*- coding: utf-8 -*-
# clint.textui.progress
# ~~~~~~~~~~~~~~~~~~~~~
# A derivation of clint version, to not introduce a dependency and add custom functionality.
# Credit to base code goes to https://github.com/kennethreitz/clint/blob/master/clint/textui/progress.py
from __future__ import absolute_import
import sys
import time
STREAM = sys.stderr
BAR_TEMPLATE = "%s[%s%s] %i/%i MB - %s\r"
BAR_FILLED_CHAR = "-"
BAR_EMPTY_CHAR = " "
# How long to wait before recalculating the ETA
ETA_INTERVAL = 1
# How many intervals (excluding the current one) to calculate the simple moving
# average
ETA_SMA_WINDOW = 9
class ProgressBar:
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.done()
return False # we're not suppressing exceptions
def __init__(
self,
label="",
width=32,
hide=None,
empty_char=BAR_EMPTY_CHAR,
filled_char=BAR_FILLED_CHAR,
expected_size=None,
every=1,
):
self.label = label
self.width = width
self.hide = hide
# Only show bar in terminals by default (better for piping, logging etc.)
if hide is None:
try:
self.hide = not STREAM.isatty()
except AttributeError: # output does not support isatty()
self.hide = True
self.empty_char = empty_char
self.filled_char = filled_char
self.expected_size = expected_size
self.every = every
self.start = time.time()
self.ittimes = []
self.eta = 0
self.etadelta = time.time()
self.etadisp = self.format_time(self.eta)
self.last_progress = 0
if self.expected_size:
self.show(0)
def show(self, progress, count=None):
if count is not None:
self.expected_size = count
if self.expected_size is None:
raise Exception("expected_size not initialized")
self.last_progress = progress
if (time.time() - self.etadelta) > ETA_INTERVAL:
self.etadelta = time.time()
self.ittimes = self.ittimes[-ETA_SMA_WINDOW:] + [
-(self.start - time.time()) / (progress + 1)
]
self.eta = (
sum(self.ittimes)
/ float(len(self.ittimes))
* (self.expected_size - progress)
)
self.etadisp = self.format_time(self.eta)
x = int(self.width * progress / self.expected_size)
if not self.hide:
if (progress % self.every) == 0 or ( # True every "every" updates
progress == self.expected_size
): # And when we're done
STREAM.write(
BAR_TEMPLATE
% (
self.label,
self.filled_char * x,
self.empty_char * (self.width - x),
progress,
self.expected_size,
self.etadisp,
)
)
STREAM.flush()
def done(self):
self.elapsed = time.time() - self.start
elapsed_disp = self.format_time(self.elapsed)
if not self.hide:
# Print completed bar with elapsed time
STREAM.write(
BAR_TEMPLATE
% (
self.label,
self.filled_char * self.width,
self.empty_char * 0,
self.last_progress,
self.expected_size,
elapsed_disp,
)
)
STREAM.write("\n")
STREAM.flush()
def format_time(self, seconds):
return time.strftime("%H:%M:%S", time.gmtime(seconds))
def bar(
it,
label="",
width=32,
hide=None,
empty_char=BAR_EMPTY_CHAR,
filled_char=BAR_FILLED_CHAR,
expected_size=None,
every=1,
):
"""Progress iterator. Wrap your iterables with it."""
count = len(it) if expected_size is None else expected_size
with ProgressBar(
label=label,
width=width,
hide=hide,
empty_char=BAR_EMPTY_CHAR,
filled_char=BAR_FILLED_CHAR,
expected_size=count,
every=every,
) as pbar:
for i, item in enumerate(it):
yield item
pbar.show(i + 1)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/logger/spinner.py 0000644 0001751 0000177 00000003607 14536152223 017733 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import sys
import threading
import time
from random import choice
class Spinner:
spinning = False
delay = 0.1
@staticmethod
def spinning_cursor():
while 1:
for cursor in "|/-\\":
yield cursor
@staticmethod
def balloons_cursor():
while 1:
for cursor in ". o O @ *":
yield cursor
@staticmethod
def changing_arrows():
while 1:
for cursor in "<^>v":
yield cursor
def select_generator(self, generator):
if generator is None:
generator = choice(["cursor", "arrow", "balloons"])
return generator
def __init__(self, delay=None, generator=None):
generator = self.select_generator(generator)
if generator == "cursor":
self.spinner_generator = self.spinning_cursor()
elif generator == "arrow":
self.spinner_generator = self.changing_arrows()
elif generator == "balloons":
self.spinner_generator = self.balloons_cursor()
if delay is None:
delay = 0.2
else:
self.spinner_generator = self.spinning_cursor()
if delay and float(delay):
self.delay = delay
def run(self):
while self.spinning:
sys.stdout.write(next(self.spinner_generator))
sys.stdout.flush()
time.sleep(self.delay)
sys.stdout.write("\b")
sys.stdout.flush()
def start(self):
self.spinning = True
threading.Thread(target=self.run).start()
def stop(self):
self.spinning = False
time.sleep(self.delay)
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.037557
spython-0.3.13/spython/main/ 0000755 0001751 0000177 00000000000 14536152245 015346 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/__init__.py 0000644 0001751 0000177 00000004141 14536152223 017453 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
def get_client(quiet=False, debug=False):
"""
get the client and perform imports not on init, in case there are any
initialization or import errors.
Parameters
==========
quiet: if True, suppress most output about the client
debug: turn on debugging mode
"""
from .base import Client as client
client.quiet = quiet
client.debug = debug
# Do imports here, can be customized
from .apps import apps
from .build import build
from .execute import execute, shell
from .export import export
from .help import helpcmd
from .inspect import inspect
from .instances import list_instances, stopall # global instance commands
from .pull import pull
from .run import run
# Actions
client.apps = apps
client.build = build
client.execute = execute
client.export = export
client.help = helpcmd
client.inspect = inspect
client.instances = list_instances
client.run = run
client.shell = shell
client.pull = pull
# Commands Groups, Instances
from spython.instance.cmd import ( # instance level commands
generate_instance_commands,
)
client.instance = generate_instance_commands()
client.instance_stopall = stopall
client.instance.version = client.version
# Commands Groups, OCI (Singularity version 3 and up)
from spython.oci.cmd import generate_oci_commands
client.oci = generate_oci_commands()() # first () runs function, second
# initializes OciImage class
client.oci.debug = client.debug
client.oci.quiet = client.quiet
client.oci.OciImage.quiet = client.quiet
client.oci.OciImage.debug = client.debug
# Initialize
cli = client()
# Pass on verbosity
cli.instance.debug = cli.debug
cli.instance.quiet = cli.quiet
cli.instance.version = cli.version
return cli
Client = get_client()
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/apps.py 0000644 0001751 0000177 00000002301 14536152223 016653 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
def apps(self, image=None, full_path=False, root=""):
"""
return list of SCIF apps in image. The Singularity software serves
a scientific filesystem integration that will install apps to
/scif/apps and associated data to /scif/data. For more information
about SCIF, see https://sci-f.github.io. Note that this seems
to be deprecated in Singularity 3.x.
Parameters
==========
full_path: if True, return relative to scif base folder
image_path: full path to the image
"""
from spython.utils import check_install
check_install()
# No image provided, default to use the client's loaded image
if image is None:
image = self._get_uri()
cmd = self._init_command("apps") + [image]
output = self._run_command(cmd)
if full_path:
root = "/scif/apps/"
if output:
output = "".join(output).split("\n")
output = ["%s%s" % (root, x) for x in output if x]
return output
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.041557
spython-0.3.13/spython/main/base/ 0000755 0001751 0000177 00000000000 14536152245 016260 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/base/Dockerfile 0000644 0001751 0000177 00000000507 14536152223 020250 0 ustar 00runner docker FROM continuumio/miniconda3
#########################################
# The Robot Namer
#
# docker build -t vanessa/robotname .
# docker run vanessa/robotname
#########################################
LABEL maintainer vsochat@stanford.edu
ADD generate.py /
ENV PATH /usr/local/bin:$PATH
ENTRYPOINT ["python", "/generate.py"]
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/base/README.md 0000644 0001751 0000177 00000001440 14536152223 017532 0 ustar 00runner docker # Robot Generator
This folder contains a (sub-application) for a robot name generator.
## Docker
It's built on Docker Hub, so you can run as:
```
docker run vanessa/robotname
```
or build locally first, with the present working directory as this folder.
```
docker build -t vanessa/robotname .
```
```
for i in `seq 1 10`;
do
docker run vanessa/robotname
done
boopy-peanut-butter-7311
blank-snack-0334
hello-buttface-6320
chocolate-bicycle-9982
frigid-frito-9511
doopy-soup-7712
phat-pancake-4952
wobbly-kitty-3213
lovely-mango-1987
milky-poo-7960
```
## Singularity
To build your image:
```
sudo singularity build robotname Singularity
```
or pull from Docker Hub :)
```
singularity pull --name robotname docker://vanessa/robotname
sregistry pull docker://vanessa/robotname
```
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/base/__init__.py 0000644 0001751 0000177 00000003516 14536152223 020372 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
from spython.logger import bot
from spython.utils import check_install, get_singularity_version
from .command import generate_bind_list, init_command, run_command
from .flags import parse_verbosity
from .generate import RobotNamer
from .logger import init_level, println
from .sutils import get_filename, get_uri, load, setenv
class Client:
def __str__(self):
base = "[singularity-python]"
if hasattr(self, "simage"):
if self.simage.image not in [None, ""]:
base = "%s[%s]" % (base, self.simage)
return base
def __repr__(self):
return self.__str__()
def __init__(self):
"""
The base client for singularity, will have commands added to it.
upon init, store verbosity requested in environment MESSAGELEVEL.
"""
self._init_level()
def version(self):
"""
Shortcut to get_singularity_version, takes no arguments.
"""
return get_singularity_version()
def _check_install(self):
"""
Ensure that singularity is installed, and exit if not.
"""
if check_install() is not True:
bot.exit("Cannot find Singularity! Is it installed?")
# Image Utils
Client.load = load
Client._get_filename = get_filename
Client._get_uri = get_uri
Client.setenv = setenv
# Commands
Client._generate_bind_list = generate_bind_list
Client._init_command = init_command
Client._run_command = run_command
# Flags and Logger
Client._parse_verbosity = parse_verbosity
Client._println = println
Client._init_level = init_level
Client.RobotNamer = RobotNamer()
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/base/command.py 0000644 0001751 0000177 00000010514 14536152223 020245 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import subprocess
import sys
from spython.logger import bot
from spython.utils import run_command as run_cmd
def init_command(self, action, flags=None):
"""
Return the initial Singularity command with any added flags.
Parameters
==========
action: the main action to perform (e.g., build)
flags: one or more additional singularity options
"""
flags = flags or []
if not isinstance(action, list):
action = [action]
cmd = ["singularity"] + flags + action
if self.quiet:
cmd.insert(1, "--quiet")
if self.debug:
cmd.insert(1, "--debug")
return cmd
def generate_bind_list(self, bindlist=None):
"""
Generate bind string will take a single string or list of binds, and
return a list that can be added to an exec or run command. For example,
the following map as follows:
['/host:/container', '/both'] --> ["--bind", "/host:/container","--bind","/both" ]
['/both'] --> ["--bind", "/both"]
'/host:container' --> ["--bind", "/host:container"]
None --> []
An empty bind or otherwise value of None should return an empty list.
The binds are also checked on the host.
Parameters
==========
bindlist: a string or list of bind mounts
"""
binds = []
# Case 1: No binds provided
if not bindlist:
return binds
# Case 2: provides a long string or non list, and must be split
if not isinstance(bindlist, list):
bindlist = bindlist.split(" ")
for bind in bindlist:
# Still cannot be None
if bind:
bot.debug("Adding bind %s" % bind)
binds += ["--bind", bind]
# Check that exists on host
host = bind.split(":")[0]
if not os.path.exists(host):
bot.error("%s does not exist on host." % bind)
sys.exit(1)
return binds
def send_command(self, cmd, sudo=False, stderr=None, stdout=None):
"""
Send command is a non interactive version of run_command, meaning
that we execute the command and return the return value, but don't
attempt to stream any content (text from the screen) back to the
user. This is useful for commands interacting with OCI bundles.
Parameters
==========
cmd: the list of commands to send to the terminal
sudo: use sudo (or not)
"""
if sudo:
cmd = ["sudo"] + cmd
process = subprocess.Popen(cmd, stderr=stderr, stdout=stdout)
result = process.communicate()
return result
def run_command(
self,
cmd,
sudo=False,
capture=True,
quiet=None,
return_result=False,
sudo_options=None,
environ=None,
background=False,
):
"""
Run_command is a wrapper for the global run_command, checking first
for sudo and exiting on error if needed. The message is returned as
a list of lines for the calling function to parse, and stdout uses
the parent process so it appears for the user.
Parameters
==========
cmd: the command to run
sudo: does the command require sudo?
quiet: if quiet set by function, overrides client setting.
return_result: return the result, if not successful (default False).
sudo_options: string or list of strings that will be passed as options to sudo
On success, returns result.
background: run the instance in the background (just Popen)
"""
# First preference to function, then to client setting
if quiet is None:
quiet = self.quiet
result = run_cmd(
cmd,
sudo=sudo,
capture=capture,
quiet=quiet,
sudo_options=sudo_options,
environ=environ,
background=background,
)
if background:
return
# If one line is returned, squash dimension
if len(result["message"]) == 1:
result["message"] = result["message"][0]
# If the user wants to return the result, just return it
if return_result:
return result
# On success, return result
if result["return_code"] == 0:
return result["message"]
return result
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/base/flags.py 0000644 0001751 0000177 00000004232 14536152223 017723 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
"""
GLOBAL OPTIONS:
-d|--debug Print debugging information
-h|--help Display usage summary
-s|--silent Only print errors
-q|--quiet Suppress all normal output
--version Show application version
-v|--verbose Increase verbosity +1
-x|--sh-debug Print shell wrapper debugging information
GENERAL COMMANDS:
help Show additional help for a command or container
selftest Run some self tests for singularity install
CONTAINER USAGE COMMANDS:
exec Execute a command within container
run Launch a runscript within container
shell Run a Bourne shell within container
test Launch a testscript within container
CONTAINER MANAGEMENT COMMANDS:
apps List available apps within a container
bootstrap *Deprecated* use build instead
build Build a new Singularity container
check Perform container lint checks
inspect Display container's metadata
mount Mount a Singularity container image
pull Pull a Singularity/Docker container to $PWD
siflist list data object descriptors of a SIF container image
sign Sign a group of data objects in container
verify Verify the crypto signature of group of data objects in container
COMMAND GROUPS:
capability User's capabilities management command group
image Container image command group
instance Persistent instance command group
"""
def parse_verbosity(self, args):
"""parse_verbosity will take an argument object, and return the args
passed (from a dictionary) to a list
Parameters
==========
args: the argparse argument objects
"""
flags = []
if args.silent:
flags.append("--silent")
elif args.quiet:
flags.append("--quiet")
elif args.debug:
flags.append("--debug")
elif args.verbose:
flags.append("-" + "v" * args.verbose)
return flags
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/base/generate.py 0000644 0001751 0000177 00000010333 14536152223 020420 0 ustar 00runner docker #!/usr/bin/env python
# Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
from random import choice
class RobotNamer:
_descriptors = [
"chunky",
"buttery",
"delicious",
"scruptious",
"dinosaur",
"boopy",
"lovely",
"carnivorous",
"hanky",
"loopy",
"doopy",
"astute",
"gloopy",
"outstanding",
"stinky",
"conspicuous",
"fugly",
"frigid",
"angry",
"adorable",
"sticky",
"moolicious",
"cowy",
"spicy",
"grated",
"crusty",
"stanky",
"blank",
"bumfuzzled",
"fuzzy",
"hairy",
"peachy",
"tart",
"creamy",
"arid",
"strawberry",
"butterscotch",
"wobbly",
"persnickety",
"nerdy",
"dirty",
"placid",
"bloated",
"swampy",
"pusheena",
"hello",
"goodbye",
"milky",
"purple",
"rainbow",
"bricky",
"muffled",
"anxious",
"misunderstood",
"eccentric",
"quirky",
"lovable",
"reclusive",
"faux",
"evasive",
"confused",
"crunchy",
"expensive",
"ornery",
"fat",
"phat",
"joyous",
"expressive",
"psycho",
"chocolate",
"salted",
"gassy",
"red",
"blue",
]
_nouns = [
"squidward",
"hippo",
"butter",
"animal",
"peas",
"lettuce",
"carrot",
"onion",
"peanut",
"cupcake",
"muffin",
"buttface",
"leopard",
"parrot",
"parsnip",
"poodle",
"itch",
"punk",
"kerfuffle",
"soup",
"noodle",
"avocado",
"peanut-butter",
"latke",
"milkshake",
"banana",
"lizard",
"lemur",
"lentil",
"bits",
"house",
"leader",
"toaster",
"signal",
"pancake",
"kitty",
"cat",
"cattywampus",
"poo",
"malarkey",
"general",
"rabbit",
"chair",
"staircase",
"underoos",
"snack",
"lamp",
"eagle",
"hobbit",
"diablo",
"earthworm",
"pot",
"plant",
"leg",
"arm",
"bike",
"citrus",
"dog",
"puppy",
"blackbean",
"ricecake",
"gato",
"nalgas",
"lemon",
"caramel",
"fudge",
"cherry",
"sundae",
"truffle",
"cinnamonbun",
"pastry",
"egg",
"omelette",
"fork",
"knife",
"spoon",
"salad",
"train",
"car",
"motorcycle",
"bicycle",
"platanos",
"mango",
"taco",
"pedo",
"nunchucks",
"destiny",
"hope",
"despacito",
"frito",
"chip",
]
def generate(self, delim="-", length=4, chars="0123456789"):
"""
Generate a robot name. Inspiration from Haikunator, but much more
poorly implemented ;)
Parameters
==========
delim: Delimiter
length: TokenLength
chars: TokenChars
"""
descriptor = self._select(self._descriptors)
noun = self._select(self._nouns)
numbers = "".join((self._select(chars) for _ in range(length)))
return delim.join([descriptor, noun, numbers])
def _select(self, select_from):
"""select an element from a list using random.choice
Parameters
==========
should be a list of things to select from
"""
if not select_from:
return ""
return choice(select_from)
def main():
bot = RobotNamer()
print(bot.generate())
if __name__ == "__main__":
main()
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/base/logger.py 0000644 0001751 0000177 00000002010 14536152223 020076 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
from spython.logger import decodeUtf8String
def init_level(self, quiet=False):
"""set the logging level based on the environment
Parameters
==========
quiet: boolean if True, set to quiet. Gets overridden by environment
setting, and only exists to define default
"""
if os.environ.get("MESSAGELEVEL") == "QUIET":
quiet = True
self.quiet = quiet
def println(self, output, quiet=False):
"""print will print the output, given that quiet is not True. This
function also serves to convert output in bytes to utf-8
Parameters
==========
output: the string to print
quiet: a runtime variable to over-ride the default.
"""
if not self.quiet and not quiet:
print(decodeUtf8String(output))
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/base/sutils.py 0000644 0001751 0000177 00000003563 14536152223 020160 0 ustar 00runner docker # Singularity Image utils for interacting with the Image/Instance
# classes from the client
# Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import re
from spython.logger import bot
def load(self, image=None):
"""load an image, either an actual path on the filesystem or a uri.
Parameters
==========
image: the image path or uri to load (e.g., docker://ubuntu
"""
from spython.image import Image
from spython.instance import Instance
self.simage = Image(image)
if image is not None:
if image.startswith("instance://"):
self.simage = Instance(image)
bot.info(self.simage)
def setenv(self, variable, value):
"""set an environment variable for Singularity
Parameters
==========
variable: the variable to set
value: the value to set
"""
os.environ[variable] = value
os.putenv(variable, value)
bot.debug("%s set to %s" % (variable, value))
def get_filename(self, image, ext="sif", pwd=True):
"""return an image filename based on the image uri.
Parameters
==========
ext: the extension to use
pwd: derive a filename for the pwd
"""
if pwd:
image = os.path.basename(image)
image = re.sub("^.*://", "", image)
if not image.endswith(ext):
image = "%s.%s" % (image, ext)
return image
def get_uri(self):
"""check if the loaded image object (self.simage) has an associated uri
return if yes, None if not.
"""
if hasattr(self, "simage"):
if self.simage is not None:
if self.simage.image not in ["", None]:
# Concatenates the ://
return str(self.simage)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/build.py 0000644 0001751 0000177 00000007535 14536152223 017025 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import re
from spython.logger import bot
from spython.utils import stream_command
def build(
self,
recipe=None,
image=None,
isolated=False,
sandbox=False,
writable=False,
build_folder=None,
robot_name=False,
ext="sif",
sudo=True,
stream=False,
force=False,
options=None,
quiet=False,
return_result=False,
sudo_options=None,
singularity_options=None,
):
"""build a singularity image, optionally for an isolated build
(requires sudo). If you specify to stream, expect the image name
and an iterator to be returned.
image, builder = Client.build(...)
Parameters
==========
recipe: the path to the recipe file (or source to build from). If not
defined, we look for "Singularity" file in $PWD
image: the image to build (if None, will use arbitrary name
isolated: if True, run build with --isolated flag
sandbox: if True, create a writable sandbox
writable: if True, use writable ext3 (sandbox takes preference)
build_folder: where the container should be built.
ext: the image extension to use.
robot_name: boolean, default False. if you don't give your image a
name (with "image") then a fun robot name will be generated
instead. Highly recommended :)
sudo: give sudo to the command (or not) default is True for build
sudo_options: options to pass to sudo (e.g. --preserve-env=SINGULARITY_CACHEDIR,SINGULARITY_TMPDIR)
options: for all other options, specify them in this list.
singularity_options: a list of options to provide to the singularity client
quiet: quiet verbose printing from the client.
return_result: if True, return complete error code / message dictionary
"""
from spython.utils import check_install
check_install()
cmd = self._init_command("build", singularity_options)
# If no extra options
options = options or []
# Force the build if the image / sandbox exists
if force:
cmd.append("--force")
# No image provided, default to use the client's loaded image
if recipe is None:
recipe = self._get_uri()
# If it's still None, try default build recipe
if recipe is None:
recipe = "Singularity"
if not os.path.exists(recipe):
bot.exit("Cannot find %s, exiting." % image)
if image is None:
if re.search("(docker|shub|library)://", recipe) and not robot_name:
image = self._get_filename(recipe, ext)
else:
image = "%s.%s" % (self.RobotNamer.generate(), ext)
# Does the user want a custom build folder?
if build_folder is not None:
if not os.path.exists(build_folder):
bot.exit("%s does not exist!" % build_folder)
image = os.path.join(build_folder, image)
# The user wants to run an isolated build
if isolated:
cmd.append("--isolated")
if sandbox:
cmd.append("--sandbox")
elif writable:
cmd.append("--writable")
cmd = cmd + options + [image, recipe]
# Does the user want to see the command printed?
if not (quiet or self.quiet):
bot.info(" ".join(cmd))
if not stream:
self._run_command(
cmd,
sudo=sudo,
sudo_options=sudo_options,
quiet=quiet,
return_result=return_result,
capture=False,
)
else:
# Here we return the expected image, and an iterator!
# The caller must iterate over
return image, stream_command(cmd, sudo=sudo, sudo_options=sudo_options)
if os.path.exists(image):
return image
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/execute.py 0000644 0001751 0000177 00000013242 14536152223 017360 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import shutil
from spython.logger import bot
from spython.utils import stream_command
def execute(
self,
image=None,
command=None,
app=None,
writable=False,
contain=False,
bind=None,
stream=False,
nv=False,
return_result=False,
options=None,
singularity_options=None,
sudo=False,
sudo_options=None,
quiet=True,
environ=None,
stream_type="stdout",
):
"""execute: send a command to a container
Parameters
==========
image: full path to singularity image
command: command to send to container
app: if not None, execute a command in context of an app
writable: This option makes the file system accessible as read/write
contain: This option disables the automatic sharing of writable
filesystems on your host
options: an optional list of options to provide to execute.
singularity_options: a list of options to provide to the singularity client
bind: list or single string of bind paths.
This option allows you to map directories on your host system to
directories within your container using bind mounts
nv: if True, load Nvidia Drivers in runtime (default False)
return_result: if True, return entire json object with return code
and message result not (default)
quiet: Do not print verbose output.
environ: extra environment to add.
stream_type: Sets which output stream from the singularity command should be return. Values are 'stdout', 'stderr', 'both'.
"""
from spython.utils import check_install
check_install()
cmd = self._init_command("exec", singularity_options)
# nv option leverages any GPU cards
if nv:
cmd += ["--nv"]
# If the image is given as a list, it's probably the command
if isinstance(image, list):
command = image
image = None
if command is not None:
# No image provided, default to use the client's loaded image
if image is None:
image = self._get_uri()
self.quiet = True
# If an instance is provided, grab it's name
if isinstance(image, self.instance):
image = image.get_uri()
# If image is still None, not defined by user or previously with client
if image is None:
bot.exit("Please load or provide an image.")
# Does the user want to use bind paths option?
if bind is not None:
cmd += self._generate_bind_list(bind)
# Does the user want to run an app?
if app is not None:
cmd = cmd + ["--app", app]
if writable:
cmd.append("--writable")
# Add additional options
if options is not None:
cmd = cmd + options
if not isinstance(command, list):
command = command.split(" ")
cmd = cmd + [image] + command
# Does the user want to see the command printed?
if not (quiet or self.quiet):
bot.info(" ".join(cmd))
if not stream:
return self._run_command(
cmd,
sudo=sudo,
sudo_options=sudo_options,
return_result=return_result,
quiet=quiet,
environ=environ,
)
return stream_command(
cmd, sudo=sudo, sudo_options=sudo_options, output_type=stream_type
)
bot.exit("Please include a command (list) to execute.")
def shell(
self,
image,
app=None,
writable=False,
contain=False,
bind=None,
nv=False,
options=None,
singularity_options=None,
sudo=False,
quiet=True,
):
"""shell into a container. A user is advised to use singularity to do
this directly, however this function is useful for supporting tools.
Parameters
==========
image: full path to singularity image
app: if not None, execute a shell in context of an app
writable: This option makes the file system accessible as read/write
contain: This option disables the automatic sharing of writable
filesystems on your host
options: an optional list of options to provide to shell.
singularity_options: a list of options to provide to the singularity client
bind: list or single string of bind paths.
This option allows you to map directories on your host system to
directories within your container using bind mounts
nv: if True, load Nvidia Drivers in runtime (default False)
"""
from spython.utils import check_install
check_install()
cmd = self._init_command("shell", singularity_options)
# nv option leverages any GPU cards
if nv:
cmd += ["--nv"]
# Does the user want to use bind paths option?
if bind is not None:
cmd += self._generate_bind_list(bind)
# Does the user want to run an app?
if app is not None:
cmd = cmd + ["--app", app]
# Add additional options
if options is not None:
cmd = cmd + options
if writable:
cmd.append("--writable")
# Finally, add the image or uri
cmd.append(image)
singularity = shutil.which("singularity")
if not singularity:
raise ValueError("Cannot find singularity executable.")
# Does the user want to see the command printed?
if not (quiet or self.quiet):
bot.info(" ".join(cmd))
if writable or sudo:
os.execvp("sudo", ["sudo"] + cmd)
else:
os.execvp(singularity, cmd)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/export.py 0000644 0001751 0000177 00000002677 14536152223 017251 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
from spython.logger import bot
def export(
self,
image_path,
pipe=False,
output_file=None,
command=None,
sudo=False,
singularity_options=None,
):
"""export will export an image, sudo must be used. Since we have Singularity
versions after 3, export is replaced with building into a sandbox.
Parameters
==========
image_path: full path to image
pipe: export to pipe and not file (default, False)
singularity_options: a list of options to provide to the singularity client
output_file: if pipe=False, export tar to this file. If not specified,
will generate temporary directory.
"""
from spython.utils import check_install
check_install()
# If export is deprecated, we run a build
bot.warning(
"Export is not supported for Singularity 3.x. Building to sandbox instead."
)
if output_file is None:
basename, _ = os.path.splitext(image_path)
output_file = self._get_filename(basename, "sandbox", pwd=False)
return self.build(
recipe=image_path,
image=output_file,
sandbox=True,
force=True,
sudo=sudo,
singularity_options=singularity_options,
)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/help.py 0000644 0001751 0000177 00000001221 14536152223 016640 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
def helpcmd(self, command=None):
"""help prints the general function help, or help for a specific command
Parameters
==========
command: the command to get help for, if none, prints general help
"""
from spython.utils import check_install
check_install()
cmd = ["singularity", "--help"]
if command is not None:
cmd.append(command)
return self._run_command(cmd)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/inspect.py 0000644 0001751 0000177 00000004415 14536152223 017365 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import json as jsonp
from spython.logger import bot
from spython.utils import check_install, run_command
def inspect(
self, image=None, json=True, app=None, quiet=True, singularity_options=None
):
"""inspect will show labels, defile, runscript, and tests for an image
Parameters
==========
image: path of image to inspect
json: print json instead of raw text (default True)
quiet: Don't print result to the screen (default True)
app: if defined, return help in context of an app
singularity_options: a list of options to provide to the singularity client
"""
check_install()
# No image provided, default to use the client's loaded image
if not image:
image = self._get_uri()
# If there still isn't an image, exit on error
if not image:
bot.exit("Please provide an image to inspect.")
cmd = self._init_command("inspect", singularity_options)
if app:
cmd = cmd + ["--app", app]
options = ["e", "d", "l", "r", "H", "t"]
for x in options:
cmd.append("-%s" % x)
if json:
cmd.append("--json")
cmd.append(image)
# Does the user want to see the command printed?
if not (quiet or self.quiet):
bot.info(" ".join(cmd))
result = run_command(cmd, quiet=quiet)
if result["return_code"] == 0:
result = jsonp.loads(result["message"][0])
# Unify output to singularity 3 format
if "data" in result:
result = result["data"]
# Fix up labels
result = parse_labels(result)
if not quiet:
print(jsonp.dumps(result, indent=4))
return result
def parse_labels(result):
"""fix up the labels, meaning parse to json if needed, and return
original updated object
Parameters
==========
result: the json object to parse from inspect
"""
labels = result["attributes"].get("labels") or {}
try:
labels = jsonp.loads(labels)
except Exception:
pass
result["attributes"]["labels"] = labels
return result
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/instances.py 0000644 0001751 0000177 00000010064 14536152223 017704 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import json
from spython.logger import bot
from spython.utils import run_command
def list_instances(
self,
name=None,
return_json=False,
quiet=False,
sudo=False,
sudo_options=None,
singularity_options=None,
):
"""
List instances. For Singularity, this is provided as a command sub
group.
singularity instance list
Return codes provided are different from standard linux:
see https://github.com/singularityware/singularity/issues/1706
Since we expect json output, we don't support older versions of Singularity.
Parameters
==========
return_json: return a json list of instances instead of objects (False)
name: if defined, return the list for just one instance (used to ged pid)
singularity_options: a list of options to provide to the singularity client
Return Code -- Reason
0 -- Instances Found
1 -- No Instances, libexecdir value not found, functions file not found
255 -- Couldn't get UID
"""
from spython.utils import check_install
check_install()
subgroup = ["instance", "list", "--json"]
cmd = self._init_command(subgroup, singularity_options)
# If the user has provided a name, we want to see a particular instance
if name is not None:
cmd.append(name)
# Does the user want to see the command printed?
if not (quiet or self.quiet):
bot.info(" ".join(cmd))
output = run_command(cmd, quiet=True, sudo=sudo, sudo_options=sudo_options)
instances = []
# Success, we have instances
if output["return_code"] == 0:
instances = json.loads(output["message"][0]).get("instances", {})
# Does the user want instance objects instead?
listing = []
if not return_json:
for i in instances:
# If the user has provided a name, only add instance matches
if name is not None:
if name != i["instance"]:
continue
# Otherwise, add instances to the listing
new_instance = self.instance(
pid=i.get("pid"),
ip_address=i.get("ip"),
name=i.get("instance") or i.get("daemon_name"),
log_err_path=i.get("logErrPath"),
log_out_path=i.get("logOutPath"),
image=i.get("img") or i.get("container_image"),
start=False,
)
listing.append(new_instance)
instances = listing
# Couldn't get UID
elif output["return_code"] == 255:
bot.error("Couldn't get UID")
# Return code of 0
else:
bot.info("No instances found.")
# If we are given a name, return just one
if name is not None and instances and len(instances) == 1:
instances = instances[0]
return instances
def stopall(self, sudo=False, quiet=True, singularity_options=None):
"""
Stop ALL instances. This command is only added to the command group
as it doesn't make sense to call from a single instance
Parameters
==========
sudo: if the command should be done with sudo (exposes different set of
instances)
"""
from spython.utils import check_install
check_install()
subgroup = "instance.stop"
if "version 3" in self.version():
subgroup = ["instance", "stop"]
cmd = self._init_command(subgroup, singularity_options)
cmd = cmd + ["--all"]
# Does the user want to see the command printed?
if not (quiet or self.quiet):
bot.info(" ".join(cmd))
output = run_command(cmd, sudo=sudo, quiet=quiet)
if output["return_code"] != 0:
message = "%s : return code %s" % (output["message"], output["return_code"])
bot.error(message)
return output["return_code"]
return output["return_code"]
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.041557
spython-0.3.13/spython/main/parse/ 0000755 0001751 0000177 00000000000 14536152245 016460 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/parse/__init__.py 0000644 0001751 0000177 00000000000 14536152223 020553 0 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.041557
spython-0.3.13/spython/main/parse/parsers/ 0000755 0001751 0000177 00000000000 14536152245 020137 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/parse/parsers/README.md 0000644 0001751 0000177 00000001470 14536152223 021414 0 ustar 00runner docker # Parsers
A parser class is intended to read in a container recipe file, and parse
sections into a spython.main.recipe Recipe object. To create a new subclass
of parser, you can copy one of the current (Docker or Singularity) as an
example, and keep in mind the following:
- The base class, `ParserBase` in [base.py](base.py) has already added an instantiated (and empty) Recipe() for the subclass to interact with (fill with content).
- The subclass is encouraged to define the name (self.name) attribute for printing to the user.
- The subclass should take the input file as an argument to pass the the ParserBase, which will handle reading in lines to a list self.lines.
- The subclass should have a main method, parse, that when called will read the input file and populate the recipe (and return it to the user).
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/parse/parsers/__init__.py 0000644 0001751 0000177 00000001422 14536152223 022243 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
from .docker import DockerParser
from .singularity import SingularityParser
def get_parser(name):
"""get_parser is a simple helper function to return a parser based on it's
name, if it exists. If there is no writer defined, we return None.
Parameters
==========
name: the name of the parser to return.
"""
name = name.lower()
parsers = {
"docker": DockerParser,
"singularity": SingularityParser,
"dockerfile": DockerParser,
}
if name in parsers:
return parsers[name]
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/parse/parsers/base.py 0000644 0001751 0000177 00000012466 14536152223 021430 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import abc
import os
import re
from copy import deepcopy
from spython.logger import bot
from spython.utils import read_file
from ..recipe import Recipe
class ParserBase:
"""a parser Base is intended to provide helper functions for a parser,
namely to read lines in files, and otherwise interact with outputs.
Input should be some recipe (text file to describe a container build)
and output of parse() is a Recipe (spython.main.parse.recipe.Recipe)
object, which can be used to write to file, etc.
"""
def __init__(self, filename, load=True):
"""a generic recipe parser holds the original file, and provides
shared functions for interacting with files. If the subclass has
a parse function defined, we parse the filename
Parameters
==========
filename: the recipe file to parse.
load: if True, load the filename into the Recipe. If not loaded,
the user can call self.parse() at a later time.
"""
self.filename = filename
self._run_checks()
self.lines = []
# Arguments can be used internally, active layer name and number
self.args = {}
self.active_layer = "spython-base"
self.active_layer_num = 1
# Support multistage builds
self.recipe = {"spython-base": Recipe(self.filename)}
if self.filename:
# Read in the raw lines of the file
self.lines = read_file(self.filename)
# If parsing function defined, parse the recipe
if load:
self.parse()
@abc.abstractmethod
def parse(self):
"""parse is the base function for parsing an input filename, and
extracting elements into the correct Recipe sections. The exact
logic and supporting functions will vary based on the recipe type.
"""
return
def _run_checks(self):
"""basic sanity checks for the file name (and others if needed) before
attempting parsing.
"""
if self.filename is not None:
# Does the recipe provided exist?
if not os.path.exists(self.filename):
bot.exit("Cannot find %s, is the path correct?" % self.filename)
# Ensure we carry fullpath
self.filename = os.path.abspath(self.filename)
# Printing
def __str__(self):
"""show the user the recipe object, along with the type. E.g.,
[spython-parser][docker]
[spython-parser][singularity]
"""
base = "[spython-parser]"
if hasattr(self, "name"):
base = "%s[%s]" % (base, self.name)
return base
def __repr__(self):
return self.__str__()
# Lines
def _split_line(self, line):
"""clean a line to prepare it for parsing, meaning separation
of commands. We remove newlines (from ends) along with extra spaces.
Parameters
==========
line: the string to parse into parts
Returns
=======
parts: a list of line pieces, the command is likely first
"""
return [x.strip() for x in line.split(" ", 1)]
# Multistage
def _multistage(self, fromHeader):
"""Given a from header, determine if we have a multistage build, and
update the recipe parser active in case that we do. If we are dealing
with the first layer and it's named, we also update the default
name "spython-base" to be what the recipe intended.
Parameters
==========
fromHeader: the fromHeader parsed from self.from, possibly with AS
"""
# Derive if there is a named layer
match = re.search("AS (?P.+)", fromHeader, flags=re.I)
if match:
layer = match.groups("layer")[0].strip()
# If it's the first layer named incorrectly, we need to rename
if len(self.recipe) == 1 and list(self.recipe)[0] == "spython-base":
self.recipe[layer] = deepcopy(self.recipe[self.active_layer])
del self.recipe[self.active_layer]
else:
self.active_layer_num += 1
self.recipe[layer] = Recipe(self.filename, self.active_layer_num)
self.active_layer = layer
bot.debug(
"Active layer #%s updated to %s"
% (self.active_layer_num, self.active_layer)
)
def _replace_from_dict(self, string, args):
"""Given a lookup of arguments, args, replace any that are found in
the given string. This is intended to be used to substitute ARGs
provided in a Dockerfile into other sections, e.g., FROM $BASE
Parameters
==========
string: an input string to look for replacements
args: a dictionary to make lookups from
Returns
=======
string: the string with replacements made
"""
for key, value in args.items():
if re.search("([$]" + key + "|[$][{]" + key + "[}])", string):
string = re.sub("([$]" + key + "|[$]{" + key + "[}])", value, string)
return string
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/parse/parsers/docker.py 0000644 0001751 0000177 00000043253 14536152223 021763 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import json
import os
import re
from spython.logger import bot
from .base import ParserBase
class DockerParser(ParserBase):
name = "docker"
def __init__(self, filename="Dockerfile", load=True):
"""a docker parser will read in a Dockerfile and put it into a Recipe
object.
Parameters
==========
filename: the Dockerfile to parse. If not defined, defaults to
Dockerfile assumed to be in the $PWD.
load: whether to load the recipe file (default True)
"""
super(DockerParser, self).__init__(filename, load)
def parse(self):
"""parse is the base function for parsing the Dockerfile, and extracting
elements into the correct data structures. Everything is parsed into
lists or dictionaries that can be assembled again on demand.
Environment: Since Docker also exports environment as we go,
we add environment to the environment section and
install
Labels: include anything that is a LABEL, ARG, or (deprecated)
maintainer.
Add/Copy: are treated the same
"""
parser = None
previous = None
for line in self.lines:
parser = self._get_mapping(line, parser, previous)
# Parse it, if appropriate
if parser:
parser(line)
previous = line
# Instantiated by ParserBase
return self.recipe
# Setup for each Parser
def _setup(self, action, line):
"""replace the command name from the group, alert the user of content,
and clean up empty spaces
"""
bot.debug("[in] %s" % line)
# Replace ACTION at beginning
line = re.sub("^%s" % action, "", line)
# Handle continuation lines without ACTION by padding with leading space
line = " " + line
# Split into components
return [x for x in self._split_line(line) if x not in ["", None]]
# From Parser
def _from(self, line):
"""get the FROM container image name from a FROM line. If we have
already seen a FROM statement, this is indicative of adding
another image (multistage build).
Parameters
==========
line: the line from the recipe file to parse for FROM
recipe: the recipe object to populate.
"""
fromHeader = self._setup("FROM", line)
# Do we have a multistge build to update the active layer?
self._multistage(fromHeader[0])
# Now extract the from header, make args replacements
self.recipe[self.active_layer].fromHeader = self._replace_from_dict(
re.sub("AS .+", "", fromHeader[0], flags=re.I), self.args
)
if "scratch" in self.recipe[self.active_layer].fromHeader:
bot.warning("scratch is no longer available on Docker Hub.")
bot.debug("FROM %s" % self.recipe[self.active_layer].fromHeader)
# Run and Test Parser
def _run(self, line):
"""everything from RUN goes into the install list
Parameters
==========
line: the line from the recipe file to parse for FROM
"""
line = self._setup("RUN", line)
self.recipe[self.active_layer].install += line
def _test(self, line):
"""A healthcheck is generally a test command
Parameters
==========
line: the line from the recipe file to parse for FROM
"""
self.recipe[self.active_layer].test = self._setup("HEALTHCHECK", line)
# Arg Parser
def _arg(self, line):
"""singularity doesn't have support for ARG, so instead will issue
a warning to the console for the user to export the variable
with SINGULARITY prefixed at build.
Parameters
==========
line: the line from the recipe file to parse for ARG
"""
line = self._setup("ARG", line)
# Args are treated like envars, so we add them to install
environ = self.parse_env([x for x in line if "=" in x])
self.recipe[self.active_layer].install += environ
# Try to extract arguments from the line
for arg in line:
# An undefined arg cannot be used
if "=" not in arg:
bot.warning(
"ARG is not supported for Singularity, and must be defined with "
"a default to be parsed. Skipping %s" % arg
)
continue
arg, value = arg.split("=", 1)
arg = arg.strip()
value = value.strip()
bot.debug("Updating ARG %s to %s" % (arg, value))
self.args[arg] = value
# Env Parser
def _env(self, line):
"""env will parse a line that beings with ENV, indicative of one or
more environment variables.
Parameters
==========
line: the line from the recipe file to parse for ADD
"""
line = self._setup("ENV", line)
# Extract environment (list) from the line
environ = self.parse_env(line)
# Add to global environment, run during install
self.recipe[self.active_layer].install += environ
# Also define for global environment
self.recipe[self.active_layer].environ += environ
def parse_env(self, envlist):
"""parse_env will parse a single line (with prefix like ENV removed) to
a list of commands in the format KEY=VALUE For example:
ENV PYTHONBUFFER 1 --> [PYTHONBUFFER=1]
Docker: https://docs.docker.com/engine/reference/builder/#env
"""
if not isinstance(envlist, list):
envlist = [envlist]
exports = []
for env in envlist:
pieces = re.split("( |\\\".*?\\\"|'.*?')", env)
pieces = [p for p in pieces if p.strip()]
while pieces:
current = pieces.pop(0)
if current.endswith("="):
# Case 1: ['A='] --> A=
nextone = ""
# Case 2: ['A=', '"1 2"'] --> A=1 2
if pieces:
nextone = pieces.pop(0)
exports.append("%s%s" % (current, nextone))
# Case 3: ['A=B'] --> A=B
elif "=" in current:
exports.append(current)
# Case 4: ENV \\
elif current.endswith("\\"):
continue
# Case 5: ['A', 'B'] --> A=B
else:
nextone = pieces.pop(0)
exports.append("%s=%s" % (current, nextone))
return exports
# Add and Copy Parser
def _copy(self, lines):
"""parse_add will copy multiple files from one location to another.
This likely will need tweaking, as the files might need to be
mounted from some location before adding to the image.
The add command is done for an entire directory. It is also
possible to have more than one file copied to a destination:
https://docs.docker.com/engine/reference/builder/#copy
e.g.: /
"""
lines = self._setup("COPY", lines)
for line in lines:
# Take into account multistage builds
layer = None
if line.startswith("--from"):
layer = line.strip("--from").split(" ")[0].lstrip("=")
if layer not in self.recipe:
bot.warning(
"COPY requested from layer %s, but layer not previously defined."
% layer
)
continue
# Remove the --from from the line
line = " ".join([word for word in line.split(" ")[1:] if word])
values = line.split(" ")
topath = values.pop()
for frompath in values:
self._add_files(frompath, topath, layer)
def _add(self, lines):
"""Add can also handle https, and compressed files.
Parameters
==========
line: the line from the recipe file to parse for ADD
"""
lines = self._setup("ADD", lines)
for line in lines:
values = line.split(" ")
frompath = values.pop(0)
# Custom parsing for frompath
# If it's a web address, add to install routine to get
if frompath.startswith("http"):
for topath in values:
self._parse_http(frompath, topath)
# Add the file, and decompress in install
elif re.search("[.](gz|gzip|bz2|xz)$", frompath.strip()):
for topath in values:
self._parse_archive(frompath, topath)
# Just add the files
else:
for topath in values:
self._add_files(frompath, topath)
# File Handling
def _add_files(self, source, dest, layer=None):
"""add files is the underlying function called to add files to the
list, whether originally called from the functions to parse archives,
or https. We make sure that any local references are changed to
actual file locations before adding to the files list.
Parameters
==========
source: the source
dest: the destiation
"""
# Warn the user Singularity doesn't support expansion
if "*" in source:
bot.warning("Singularity doesn't support expansion, * found in %s" % source)
# Warning if file/folder (src) doesn't exist
if not os.path.exists(source) and layer is None:
bot.warning("%s doesn't exist, ensure exists for build" % source)
# The pair is added to the files as a list
if not layer:
self.recipe[self.active_layer].files.append([source, dest])
# Unless the file is to be copied from a particular layer
else:
if layer not in self.recipe[self.active_layer].layer_files:
self.recipe[self.active_layer].layer_files[layer] = []
self.recipe[self.active_layer].layer_files[layer].append([source, dest])
def _parse_http(self, url, dest):
"""will get the filename of an http address, and return a statement
to download it to some location
Parameters
==========
url: the source url to retrieve with curl
dest: the destination folder to put it in the image
"""
file_name = os.path.basename(url)
download_path = "%s/%s" % (dest, file_name)
command = "curl %s -o %s" % (url, download_path)
self.recipe[self.active_layer].install.append(command)
def _parse_archive(self, targz, dest):
"""parse_targz will add a line to the install script to extract a
targz to a location, and also add it to the files.
Parameters
==========
targz: the targz to extract
dest: the location to extract it to
"""
# Add command to extract it
self.recipe[self.active_layer].install.append("tar -zvf %s %s" % (targz, dest))
# Ensure added to container files
return self._add_files(targz, dest)
# Comments and Default
def _comment(self, line):
"""Simply add the line to the install as a comment. This function is
equivalent to default, but added in the case we need future custom
parsing (meaning a comment is different from a line.
Parameters
==========
line: the line from the recipe file to parse to INSTALL
"""
self.recipe[self.active_layer].install.append(line)
def _default(self, line):
"""the default action assumes a line that is either a command (a
continuation of a previous, for example) or a comment.
Parameters
==========
line: the line from the recipe file to parse to INSTALL
"""
if line.strip().startswith("#"):
return self._comment(line)
self.recipe[self.active_layer].install.append(line)
# Ports and Volumes
def _volume(self, line):
"""We don't have logic for volume for Singularity, so we add as
a comment in the install, and a metadata value for the recipe
object
Parameters
==========
line: the line from the recipe file to parse to INSTALL
"""
volumes = self._setup("VOLUME", line)
if volumes:
self.recipe[self.active_layer].volumes += volumes
return self._comment("# %s" % line)
def _expose(self, line):
"""Again, just add to metadata, and comment in install.
Parameters
==========
line: the line from the recipe file to parse to INSTALL
"""
ports = self._setup("EXPOSE", line)
if ports:
self.recipe[self.active_layer].ports += ports
return self._comment("# %s" % line)
def _stopsignal(self, line):
"""Again, just add to metadata, and comment in install.
Parameters
==========
line: the line from the recipe file to parse STOPSIGNAL
"""
return self._comment("# %s" % line)
# Working Directory
def _workdir(self, line):
"""A Docker WORKDIR command simply implies to cd to that location
Parameters
==========
line: the line from the recipe file to parse for WORKDIR
"""
# Save the last working directory to add to the runscript
workdir = self._setup("WORKDIR", line)
workdir_mkdir = "mkdir -p %s" % ("".join(workdir))
self.recipe[self.active_layer].install.append(workdir_mkdir)
workdir_cd = "cd %s" % ("".join(workdir))
self.recipe[self.active_layer].install.append(workdir_cd)
self.recipe[self.active_layer].workdir = workdir[0]
# Entrypoint and Command
def _cmd(self, line):
"""_cmd will parse a Dockerfile CMD command
eg: CMD /code/run_uwsgi.sh --> /code/run_uwsgi.sh.
If a list is provided, it's parsed to a list.
Parameters
==========
line: the line from the recipe file to parse for CMD
"""
cmd = self._setup("CMD", line)[0]
self.recipe[self.active_layer].cmd = self._load_list(cmd)
def _load_list(self, line):
"""load an entrypoint or command, meaning it can be wrapped in a list
or a regular string. We try loading as json to return an actual
list. E.g., after _setup, we might go from 'ENTRYPOINT ["one", "two"]'
to '["one", "two"]', and this function loads as json and returns
["one", "two"]
"""
try:
line = json.loads(line)
except Exception:
pass
return line
def _entry(self, line):
"""_entrypoint will parse a Dockerfile ENTRYPOINT command
Parameters
==========
line: the line from the recipe file to parse for CMD
"""
entrypoint = self._setup("ENTRYPOINT", line)[0]
self.recipe[self.active_layer].entrypoint = self._load_list(entrypoint)
# Labels
def _label(self, line):
"""_label will parse a Dockerfile label
Parameters
==========
line: the line from the recipe file to parse for CMD
"""
label = self._setup("LABEL", line)
self.recipe[self.active_layer].labels += [label]
# Main Parsing Functions
def _get_mapping(self, line, parser=None, previous=None):
"""mapping will take the command from a Dockerfile and return a map
function to add it to the appropriate place. Any lines that don't
cleanly map are assumed to be comments.
Parameters
==========
line: the list that has been parsed into parts with _split_line
parser: the previously used parser, for context
Returns
=======
function: to map a line to its command group
"""
# Split the command into cleanly the command and rest
if not isinstance(line, list):
line = self._split_line(line)
# No line we will give function to handle empty line
if not line:
return None
cmd = line[0].upper()
mapping = {
"ADD": self._add,
"ARG": self._arg,
"COPY": self._copy,
"CMD": self._cmd,
"ENTRYPOINT": self._entry,
"ENV": self._env,
"EXPOSE": self._expose,
"FROM": self._from,
"HEALTHCHECK": self._test,
"RUN": self._run,
"WORKDIR": self._workdir,
"MAINTAINER": self._label,
"VOLUME": self._volume,
"LABEL": self._label,
"STOPSIGNAL": self._stopsignal,
}
# If it's a command line, return correct function
if cmd in mapping:
return mapping[cmd]
# If it's a continued line, return previous
cleaned = self._clean_line(line[-1])
previous = self._clean_line(previous)
# if we are continuing from last
if cleaned.endswith("\\") and parser or previous.endswith("\\"):
return parser
return self._default
def _clean_line(self, line):
"""clean line will remove comments, and strip the line of newlines
or spaces.
Parameters
==========
line: the string to parse into parts
Returns
=======
line: a cleaned line
"""
# A line that is None should return empty string
line = line or ""
return line.split("#")[0].strip()
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/parse/parsers/singularity.py 0000644 0001751 0000177 00000027312 14536152223 023064 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import re
from spython.logger import bot
from .base import ParserBase
class SingularityParser(ParserBase):
name = "singularity"
def __init__(self, filename="Singularity", load=True):
"""a SingularityParser parses a Singularity file into expected fields of
labels, environment, and install/runtime commands. The base class
ParserBase will instantiate an empty Recipe() object to populate,
and call parse() here on the recipe.
Parameters
==========
filename: the recipe file (Singularity) to parse
load: load and parse the recipe (defaults to True)
"""
super(SingularityParser, self).__init__(filename, load)
def parse(self):
"""parse is the base function for parsing the recipe, and extracting
elements into the correct data structures. Everything is parsed into
lists or dictionaries that can be assembled again on demand.
Singularity: we parse files/labels first, then install.
cd first in a line is parsed as WORKDIR
"""
self.load_recipe()
return self.recipe
# Setup for each Parser
def _setup(self, lines):
"""setup required adding content from the host to the rootfs,
so we try to capture with with ADD.
"""
bot.warning("SETUP is error prone, please check output.")
for line in lines:
# For all lines, replace rootfs with actual root /
line = re.sub("[$]{?SINGULARITY_ROOTFS}?", "", "$SINGULARITY_ROOTFS")
# If we have nothing left, don't continue
if line in ["", None]:
continue
# If the line starts with copy or move, assume is file from host
if re.search("(^cp|^mv)", line):
line = re.sub("(^cp|^mv)", "", line)
self.files.append(line)
# If it's a general command, add to install routine
else:
self.install.append(line)
# From Parser
def _from(self, line):
"""get the FROM container image name from a FROM line!
Parameters
==========
line: the line from the recipe file to parse for FROM
"""
self.recipe[self.active_layer].fromHeader = line
bot.debug("FROM %s" % self.recipe[self.active_layer].fromHeader)
# Run and Test Parser
def _test(self, lines):
"""A healthcheck is generally a test command
Parameters
==========
line: the line from the recipe file to parse for FROM
"""
self._write_script("/tests.sh", lines)
self.recipe[self.active_layer].test = "/bin/bash /tests.sh"
# Env Parser
def _env(self, lines):
"""env will parse a list of environment lines and simply remove any
blank lines, and exports. Dockerfiles don't usually
have exports.
Parameters
==========
lines: A list of environment pair lines.
"""
environ = [re.sub("^export", "", x).strip() for x in lines if "=" in x]
self.recipe[self.active_layer].environ += environ
# Files for container
def _files(self, lines, layer=None):
"""parse_files will simply add the list of files to the correct object
Parameters
==========
lines: pairs of files, one pair per line
"""
if not layer:
self.recipe[self.active_layer].files += lines
else:
if layer not in self.recipe[self.active_layer].layer_files:
self.recipe[self.active_layer].layer_files[layer] = []
self.recipe[self.active_layer].layer_files[layer] += lines
# Comments and Help
def _comments(self, lines):
"""comments is a wrapper for comment, intended to be given a list
of comments.
Parameters
==========
lines: the list of lines to parse
"""
for line in lines:
comment = self._comment(line)
if comment not in self.recipe[self.active_layer].comments:
self.recipe[self.active_layer].comments.append(comment)
def _comment(self, line):
"""Simply add the line to the install as a comment. Add an extra # to be
extra careful.
Parameters
==========
line: the line from the recipe file to parse to INSTALL
"""
return "# %s" % line.strip().strip("#")
# Runscript Command
def _run(self, lines):
"""_parse the runscript to be the Docker CMD. If we have one line,
call it directly. If not, write the entrypoint into a script.
Parameters
==========
lines: the line from the recipe file to parse for CMD
"""
lines = [x for x in lines if x not in ["", None]]
# Default runscript is first index
runscript = lines[0]
# Multiple line runscript needs multiple lines written to script
if len(lines) > 1:
bot.warning("More than one line detected for runscript!")
bot.warning("These will be echoed into a single script to call.")
self._write_script("/entrypoint.sh", lines)
runscript = "/bin/bash /entrypoint.sh"
self.recipe[self.active_layer].cmd = runscript
# Labels
def _labels(self, lines):
"""_labels simply adds the labels to the list to save.
Parameters
==========
lines: the lines from the recipe with key,value pairs
"""
self.recipe[self.active_layer].labels += lines
def _post(self, lines):
"""the main core of commands, to be added to the install section
Parameters
==========
lines: the lines from the recipe with install commands
"""
self.recipe[self.active_layer].install += lines
# Main Parsing Functions
def _get_mapping(self, section):
"""mapping will take the section name from a Singularity recipe
and return a map function to add it to the appropriate place.
Any lines that don't cleanly map are assumed to be comments.
Parameters
==========
section: the name of the Singularity recipe section
Returns
=======
function: to map a line to its command group (e.g., install)
"""
# Ensure section is lowercase
section = section.lower()
mapping = {
"environment": self._env,
"comments": self._comments,
"runscript": self._run,
"labels": self._labels,
"setup": self._setup,
"files": self._files,
"from": self._from,
"post": self._post,
"test": self._test,
"help": self._comments,
}
if section in mapping:
return mapping[section]
return self._comments
# Loading Functions
def _load_from(self, line):
"""load the From section of the recipe for the Dockerfile."""
# Remove any comments
line = line.split("#", 1)[0]
line = re.sub("from:", "", line.lower()).strip()
self.recipe[self.active_layer].fromHeader = line
def _check_bootstrap(self, line):
"""checks that the bootstrap is Docker, otherwise we exit on fail."""
if not re.search("docker", line, re.IGNORECASE):
raise NotImplementedError("Only docker is supported.")
def _load_section(self, lines, section, layer=None):
"""read in a section to a list, and stop when we hit the next section"""
members = []
while True:
if not lines:
break
next_line = lines[0]
# We have a start of another bootstrap
if re.search("bootstrap:", next_line, re.IGNORECASE):
break
# The end of a section
if next_line.strip().startswith("%"):
break
# Still in current section!
else:
new_member = lines.pop(0).strip()
if new_member not in ["", None]:
members.append(new_member)
# Add the list to the config
if members and section is not None:
# Get the correct parsing function
parser = self._get_mapping(section)
# Parse it, if appropriate
if not parser:
bot.warning("%s is an unrecognized section, skipping." % section)
else:
if section == "files":
parser(members, layer)
else:
parser(members)
def load_recipe(self):
"""load_recipe will return a loaded in singularity recipe. The idea
is that these sections can then be parsed into a Dockerfile,
or printed back into their original form.
Returns
=======
config: a parsed recipe Singularity recipe
"""
# Comments between sections, add to top of file
lines = self.lines[:]
fromHeader = None
stage = None
section = None
comments = []
while lines:
# Clean up white trailing/leading space
line = lines.pop(0)
stripped = line.strip()
# Bootstrap Line
if re.search("bootstrap", line, re.IGNORECASE):
self._check_bootstrap(stripped)
# From Line
elif re.search("from:", stripped, re.IGNORECASE):
fromHeader = stripped
if stage is None:
self._load_from(fromHeader)
# Identify stage
elif re.search("stage:", stripped, re.IGNORECASE):
stage = re.sub("stage:", "", stripped.lower()).strip()
self._multistage("as %s" % stage)
self._load_from(fromHeader)
# Comment
elif stripped.startswith("#") and stripped not in comments:
comments.append(stripped)
# Section
elif stripped.startswith("%"):
section, layer = self._get_section(stripped)
bot.debug("Found section %s" % section)
# If we have a section, and are adding it
elif section is not None:
lines = [line] + lines
self._load_section(lines=lines, section=section, layer=layer)
self._comments(comments)
def _get_section(self, line):
"""parse a line for a section, and return the name of the section
Parameters
==========
line: the line to parse
"""
# Remove any comments
line = line.split("#", 1)[0].strip()
# Is there a section name?
parts = [word.strip() for word in line.split(" ") if word]
section = re.sub(r"[%]|(\s+)", "", parts[0]).lower()
# Is there a named layer?
layer = None
if len(parts) == 3 and parts[1].lower() == "from":
layer = parts[2]
return section, layer
def _write_script(self, path, lines, chmod=True):
"""write a script with some lines content to path in the image. This
is done by way of adding echo statements to the install section.
Parameters
==========
path: the path to the file to write
lines: the lines to echo to the file
chmod: If true, change permission to make u+x
"""
for line in lines:
self.recipe[self.active_layer].install.append(
'echo "%s" >> %s' % (line, path)
)
if chmod:
self.recipe[self.active_layer].install.append("chmod u+x %s" % path)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/parse/recipe.py 0000644 0001751 0000177 00000005055 14536152223 020302 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
class Recipe:
"""
a recipe includes an environment, labels, runscript or command,
and install sequence. This object is interacted with by a Parser
(intended to popualte the recipe with content) and a Writer (intended
to write a recipe to file). The parsers and writers are located in
parsers.py, and writers.py, respectively. The user is also free to use
the recipe class to build recipes.
Parameters
==========
recipe: the original recipe file, parsed by the subclass either
DockerParser or SingularityParser
layer: the count of the layer, for human readability
"""
def __init__(self, recipe=None, layer=1):
self.cmd = None
self.comments = []
self.entrypoint = None
self.environ = []
self.files = []
self.layer_files = {}
self.install = []
self.labels = []
self.ports = []
self.test = None
self.volumes = []
self.workdir = None
self.layer = layer
self.fromHeader = None
self.source = recipe
def __str__(self):
"""show the user the recipe object, along with the type. E.g.,
[spython-recipe][source:Singularity]
[spython-recipe][source:Dockerfile]
"""
base = "[spython-recipe]"
if self.source:
base = "%s[source:%s]" % (base, self.source)
return base
def json(self):
"""return a dictionary version of the recipe, intended to be parsed
or printed as json.
Returns: a dictionary of attributes including cmd, comments,
entrypoint, environ, files, install, labels, ports,
test, volumes, and workdir, organized by layer for
multistage builds.
"""
attributes = [
"cmd",
"comments",
"entrypoint",
"environ",
"files",
"fromHeader",
"layer_files",
"install",
"labels",
"ports",
"test",
"volumes",
"workdir",
]
result = {}
for attrib in attributes:
value = getattr(self, attrib)
if value:
result[attrib] = value
return result
def __repr__(self):
return self.__str__()
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.041557
spython-0.3.13/spython/main/parse/writers/ 0000755 0001751 0000177 00000000000 14536152245 020157 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/parse/writers/__init__.py 0000644 0001751 0000177 00000001423 14536152223 022264 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
from .docker import DockerWriter
from .singularity import SingularityWriter
def get_writer(name):
"""get_writer is a simple helper function to return a writer based on it's
name, if it exists. If there is no writer defined, we return None.
Parameters
==========
name: the name of the writer to return.
"""
name = name.lower()
writers = {
"docker": DockerWriter,
"singularity": SingularityWriter,
"dockerfile": DockerWriter,
}
if name in writers:
return writers[name]
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/parse/writers/base.py 0000644 0001751 0000177 00000004651 14536152223 021445 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import tempfile
from spython.logger import bot
from spython.utils import write_file
class WriterBase:
def __init__(self, recipe=None):
"""a writer base will take a recipe object (parser.base.Recipe) and
provide helpers for writing to file.
Parameters
==========
recipe: the recipe instance to parse
"""
self.recipe = recipe
def write(self, output_file=None, force=False):
"""convert a recipe to a specified format, and write to file, meaning
we use the loaded recipe to write to an output file.
If the output file is not specified, a temporary file is used.
Parameters
==========
output_file: the file to save to, not required (estimates default)
force: if True, if file exists, over-write existing file
"""
if output_file is None:
output_file = self._get_conversion_outfile()
# Cut out early if file exists and we aren't overwriting
if os.path.exists(output_file) and not force:
bot.exit("%s exists, and force is False." % output_file)
# Do the conversion if function is provided by subclass
if hasattr(self, "convert"):
converted = self.convert()
bot.info("Saving to %s" % output_file)
write_file(output_file, converted)
def _get_conversion_outfile(self):
"""a helper function to return a conversion temporary output file
based on kind of conversion
Parameters
==========
convert_to: a string either docker or singularity, if a different
"""
prefix = "spythonRecipe"
if hasattr(self, "name"):
prefix = self.name
suffix = next(tempfile._get_candidate_names())
return "%s.%s" % (prefix, suffix)
# Printing
def __str__(self):
"""show the user the recipe object, along with the type. E.g.,
[spython-writer][docker]
[spython-writer][singularity]
"""
base = "[spython-writer]"
if hasattr(self, "name"):
base = "%s[%s]" % (base, self.name)
return base
def __repr__(self):
return self.__str__()
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/parse/writers/docker.py 0000644 0001751 0000177 00000012001 14536152223 021766 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import re
from spython.logger import bot
from .base import WriterBase
# FROM Validation
# Regular expressions to parse registry, collection, repo, tag and version
_docker_uri = re.compile(
"(?:(?P[^/@]+[.:][^/@]*)/)?"
"(?P(?:[^:@/]+/)+)?"
"(?P[^:@/]+)"
"(?::(?P[^:@]+))?"
"(?:@(?P.+))?"
"$"
)
# Reduced to match registry:port/repo or registry.com/repo
_reduced_uri = re.compile(
"(?:(?P[^/@]+[.:][^/@]*)/)?"
"(?P[^:@/]+)"
"(?::(?P[^:@]+))?"
"(?:@(?P.+))?"
"$"
"(?P.)?"
)
# Default
_default_uri = re.compile(
"(?:(?P[^/@]+)/)?"
"(?P(?:[^:@/]+/)+)"
"(?P[^:@/]+)"
"(?::(?P[^:@]+))?"
"(?:@(?P.+))?"
"$"
)
class DockerWriter(WriterBase):
name = "docker"
def __init__(self, recipe=None): # pylint: disable=useless-super-delegation
"""a DockerWriter will take a Recipe as input, and write
to a Dockerfile.
Parameters
==========
recipe: the Recipe object to write to file.
"""
super(DockerWriter, self).__init__(recipe)
def validate(self):
"""validate that all (required) fields are included for the Docker
recipe. We minimimally just need a FROM image, and must ensure
it's in a valid format. If anything is missing, we exit with error.
"""
if self.recipe is None:
bot.exit("Please provide a Recipe() to the writer first.")
def validate_stage(self, parser):
"""Given a recipe parser for a stage, ensure that the recipe is valid"""
if parser.fromHeader is None:
bot.exit("Dockerfile requires a fromHeader.")
# Parse the provided name
uri_regexes = [_reduced_uri, _default_uri, _docker_uri]
for r in uri_regexes:
match = r.match(parser.fromHeader)
if match:
break
if not match:
bot.exit("FROM header %s not valid." % parser.fromHeader)
def convert(self, runscript="/bin/bash", force=False):
"""convert is called by the parent class to convert the recipe object
(at self.recipe) to the output file content to write to file.
"""
self.validate()
recipe = []
for stage, parser in self.recipe.items():
self.validate_stage(parser)
recipe += ["FROM %s AS %s" % (parser.fromHeader, stage)]
# First add files, labels, environment
recipe += write_files("ADD", parser.files)
recipe += write_lines("LABEL", parser.labels)
recipe += write_lines("ENV", parser.environ)
# Handle custom files from other layers
for layer, files in parser.layer_files.items():
for pair in files:
recipe += ["COPY --from=%s %s" % (layer, pair)]
# Install routine is added as RUN commands
# TODO: this needs some work
recipe += write_lines("RUN", parser.install)
# Expose ports
recipe += write_lines("EXPOSE", parser.ports)
if parser.workdir is not None:
recipe.append("WORKDIR %s" % parser.workdir)
# write the command, and entrypoint as is
if parser.cmd is not None:
recipe.append("CMD %s" % parser.cmd)
if parser.entrypoint is not None:
recipe.append("ENTRYPOINT %s" % parser.entrypoint)
if parser.test is not None:
recipe += write_lines("HEALTHCHECK", parser.test)
# Clean up extra white spaces
recipe = "\n".join(recipe).replace("\n\n", "\n")
return recipe.rstrip()
def write_files(label, lines):
"""write a list of lines with a header for a section.
Parameters
==========
lines: one or more lines to write, with header appended
"""
result = []
for line in lines:
if isinstance(line, list):
result.append("%s %s %s" % (label, line[0], line[1]))
else:
result.append("%s %s" % (label, line))
return result
def write_lines(label, lines):
"""write a list of lines with a header for a section.
Parameters
==========
lines: one or more lines to write, with header appended
"""
if not isinstance(lines, list):
lines = [lines]
result = []
continued = False
for line in lines:
# Skip comments and empty lines
if line.strip() == "" or line.strip().startswith("#"):
continue
if continued or "USER" in line:
result.append(line)
else:
result.append("%s %s" % (label, line))
continued = False
if line.endswith("\\"):
continued = True
return result
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/parse/writers/singularity.py 0000644 0001751 0000177 00000017710 14536152223 023105 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import re
from spython.logger import bot
from .base import WriterBase
class SingularityWriter(WriterBase):
name = "singularity"
def __init__(self, recipe=None): # pylint: disable=useless-super-delegation
"""a SingularityWriter will take a Recipe as input, and write
to a Singularity recipe file.
Parameters
==========
recipe: the Recipe object to write to file.
"""
super(SingularityWriter, self).__init__(recipe)
def validate(self):
"""validate that all (required) fields are included for the Docker
recipe. We minimimally just need a FROM image, and must ensure
it's in a valid format. If anything is missing, we exit with error.
"""
if self.recipe is None:
bot.exit("Please provide a Recipe() to the writer first.")
def convert(self, runscript="/bin/bash", force=False):
"""docker2singularity will return a Singularity build recipe based on
a the loaded recipe object. It doesn't take any arguments as the
recipe object contains the sections, and the calling function
determines saving / output logic.
"""
self.validate()
# Write single recipe that includes all layer
recipe = []
# Number of layers
num_layers = len(self.recipe)
count = 0
# Write each layer to new file
for stage, parser in self.recipe.items():
# Set the first and active stage
self.stage = stage
# From header is required
if parser.fromHeader is None:
bot.exit("Singularity recipe requires a from header.")
recipe += ["\n\n\nBootstrap: docker"]
recipe += ["From: %s" % parser.fromHeader]
recipe += ["Stage: %s\n\n\n" % stage]
# TODO: stopped here - bug with files being found
# Add global files, and then layer files
recipe += self._create_section("files")
for layer, files in parser.layer_files.items():
recipe += create_keyval_section(files, "files", layer)
# Sections with key value pairs
recipe += self._create_section("labels")
recipe += self._create_section("install", "post")
recipe += self._create_section("environ", "environment")
# If we are at the last layer, write the runscript
if count == num_layers - 1:
runscript = self._create_runscript(runscript, force)
# If a working directory was used, add it as a cd
if parser.workdir is not None:
runscript = ["cd " + parser.workdir] + [runscript]
# Finish the recipe, also add as startscript
recipe += finish_section(runscript, "runscript")
recipe += finish_section(runscript, "startscript")
if parser.test is not None:
recipe += finish_section(parser.test, "test")
count += 1
# Clean up extra white spaces
recipe = "\n".join(recipe).replace("\n\n", "\n").strip("\n")
return recipe.rstrip()
def _create_runscript(self, default="/bin/bash", force=False):
"""create_entrypoint is intended to create a singularity runscript
based on a Docker entrypoint or command. We first use the Docker
ENTRYPOINT, if defined. If not, we use the CMD. If neither is found,
we use function default.
Parameters
==========
default: set a default entrypoint, if the container does not have
an entrypoint or cmd.
force: If true, use default and ignore Dockerfile settings
"""
entrypoint = default
# Only look at Docker if not enforcing default
if not force:
if self.recipe[self.stage].entrypoint is not None:
# The provided entrypoint can be a string or a list
if isinstance(self.recipe[self.stage].entrypoint, list):
entrypoint = " ".join(self.recipe[self.stage].entrypoint)
else:
entrypoint = "".join(self.recipe[self.stage].entrypoint)
if self.recipe[self.stage].cmd is not None:
if isinstance(self.recipe[self.stage].cmd, list):
entrypoint = (
entrypoint + " " + " ".join(self.recipe[self.stage].cmd)
)
else:
entrypoint = entrypoint + " " + "".join(self.recipe[self.stage].cmd)
# Entrypoint should use exec
if not entrypoint.startswith("exec"):
entrypoint = "exec %s" % entrypoint
# Should take input arguments into account
if not re.search('"?[$]@"?', entrypoint):
entrypoint = '%s "$@"' % entrypoint
return entrypoint
def _create_section(self, attribute, name=None, stage=None):
"""create a section based on key, value recipe pairs,
This is used for files or label
Parameters
==========
attribute: the name of the data section, either labels or files
name: the name to write to the recipe file (e.g., %name).
if not defined, the attribute name is used.
"""
# Default section name is the same as attribute
if name is None:
name = attribute
# Put a space between sections
section = ["\n"]
# Only continue if we have the section and it's not empty
try:
section = getattr(self.recipe[self.stage], attribute)
except AttributeError:
bot.debug("Recipe does not have section for %s" % attribute)
return section
# if the section is empty, don't print it
if not section:
return section
# Files
if attribute in ["files", "labels"]:
return create_keyval_section(section, name, stage)
# An environment section needs exports
if attribute in ["environ"]:
return create_env_section(section, name)
# Post, Setup
return finish_section(section, name)
def finish_section(section, name):
"""finish_section will add the header to a section, to finish the recipe
take a custom command or list and return a section.
Parameters
==========
section: the section content, without a header
name: the name of the section for the header
"""
if not isinstance(section, list):
section = [section]
# Convert USER lines to change user
lines = []
for line in section:
if re.search("^USER", line):
username = line.replace("USER", "", 1).rstrip()
line = "su - %s" % username + " # " + line
lines.append(line)
header = ["%" + name]
return header + lines
def create_keyval_section(pairs, name, layer):
"""create a section based on key, value recipe pairs,
This is used for files or label
Parameters
==========
section: the list of values to return as a parsed list of lines
name: the name of the section to write (e.g., files)
layer: if a layer name is provided, name section
"""
if layer:
section = ["%" + name + " from %s" % layer]
else:
section = ["%" + name]
for pair in pairs:
section.append(" ".join(pair).strip().strip("\\"))
return section
def create_env_section(pairs, name):
"""environment key value pairs need to be joined by an equal, and
exported at the end.
Parameters
==========
section: the list of values to return as a parsed list of lines
name: the name of the section to write (e.g., files)
"""
section = ["%" + name]
for pair in pairs:
section.append("export %s" % pair)
return section
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/pull.py 0000644 0001751 0000177 00000005650 14536152223 016676 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import re
from spython.logger import bot
from spython.utils import ScopedEnvVar, stream_command
def pull(
self,
image=None,
name=None,
pull_folder="",
ext="sif",
force=False,
capture=False,
stream=False,
quiet=False,
singularity_options=None,
):
"""pull will pull a singularity hub or Docker image
Parameters
==========
image: the complete image uri. If not provided, the client loaded is used
singularity_options: a list of options to provide to the singularity client
pull_folder: if not defined, pulls to $PWD (''). If defined, pulls to
user specified location instead.
Docker and Singularity Hub Naming
---------------------------------
name: a custom name to use, to override default
ext: if no name specified, the default extension to use.
"""
from spython.utils import check_install
check_install()
cmd = self._init_command("pull", singularity_options)
# Quiet is honored if set by the client, or user
quiet = quiet or self.quiet
# No image provided, default to use the client's loaded image
if image is None:
image = self._get_uri()
# If it's still None, no go!
if image is None:
bot.exit("You must provide an image uri, or use client.load() first.")
# Singularity Only supports shub, docker and library pull
if not re.search("^(shub|docker|library|https|oras)://", image):
bot.exit("pull only valid for docker, oras, https, shub and library.")
# If we still don't have a custom name, base off of image uri.
if name is None:
name = self._get_filename(image, ext)
if pull_folder:
final_image = os.path.join(pull_folder, os.path.basename(name))
# Regression Singularity 3.* onward, PULLFOLDER not honored
# https://github.com/sylabs/singularity/issues/2788
name = final_image
pull_folder = None # Don't use pull_folder
else:
final_image = name
cmd = cmd + ["--name", name]
if force:
cmd = cmd + ["--force"]
cmd.append(image)
if not quiet:
bot.info(" ".join(cmd))
with ScopedEnvVar("SINGULARITY_PULLFOLDER", pull_folder):
# Option 1: Streaming we just run to show user
if not stream:
self._run_command(cmd, capture=capture, quiet=quiet)
# Option 3: A custom name we can predict (not commit/hash) and can also show
else:
# As of Singularity 3.x (at least 3.8) output goes to stderr
return final_image, stream_command(cmd, sudo=False, output_type="stderr")
if os.path.exists(final_image) and not quiet:
bot.info(final_image)
return final_image
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/main/run.py 0000644 0001751 0000177 00000006673 14536152223 016534 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import json
from spython.logger import bot
from spython.utils import stream_command
def run(
self,
image=None,
args=None,
app=None,
sudo=False,
writable=False,
contain=False,
bind=None,
stream=False,
nv=False,
options=None,
singularity_options=None,
return_result=False,
quiet=False,
background=False,
):
"""
run will run the container, with or withour arguments (which
should be provided in a list)
Parameters
==========
image: full path to singularity image
args: args to include with the run
app: if not None, execute a command in context of an app
writable: This option makes the file system accessible as read/write
options: an optional list of options to provide to run.
singularity_options: a list of options to provide to the singularity client
contain: This option disables the automatic sharing of writable
filesystems on your host
bind: list or single string of bind paths.
This option allows you to map directories on your host system to
directories within your container using bind mounts
stream: if True, return for the user to run
nv: if True, load Nvidia Drivers in runtime (default False)
return_result: if True, return entire json object with return code
and message result (default is False)
quiet: print the command to the user
"""
from spython.utils import check_install
check_install()
cmd = self._init_command("run", singularity_options)
# Does the user want to see the command printed?
quiet = quiet or self.quiet
# nv option leverages any GPU cards
if nv:
cmd += ["--nv"]
# No image provided, default to use the client's loaded image
if image is None:
image = self._get_uri()
# If an instance is provided, grab it's name
if isinstance(image, self.instance):
image = image.get_uri()
# If image is still None, not defined by user or previously with client
if image is None:
bot.exit("Please load or provide an image.")
# Does the user want to use bind paths option?
if bind is not None:
cmd += self._generate_bind_list(bind)
# Does the user want to run an app?
if app is not None:
cmd = cmd + ["--app", app]
# Does the user want writable?
if writable:
cmd.append("--writable")
# Add options
if options is not None:
cmd = cmd + options
cmd = cmd + [image]
if args is not None:
if not isinstance(args, list):
args = args.split(" ")
cmd = cmd + args
if not quiet:
bot.info(" ".join(cmd))
if background:
return self._run_command(cmd, sudo=sudo, background=True)
elif not stream:
result = self._run_command(cmd, sudo=sudo, return_result=return_result)
else:
return stream_command(cmd, sudo=sudo)
# If the user wants the raw result object
if return_result:
return result
# Otherwise, we parse the result if it was successful
if result:
result = result.strip("\n")
try:
result = json.loads(result)
except Exception:
pass
return result
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.041557
spython-0.3.13/spython/oci/ 0000755 0001751 0000177 00000000000 14536152245 015174 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/oci/README.md 0000644 0001751 0000177 00000002171 14536152223 016450 0 ustar 00runner docker # OCI Development
Here I'll write how I created an OCI bundle using Singularity to help with
development of the client. First, notice the [config.json](config.json)
in the present working directory - it's a general configuration for OCI
runtime specification (version 1.0) that we can put into a bundle (a folder
that will serve as the root of a container). Here is how I did that.
First, we are developing with the first release of Singularity that supports
OCI:
```bash
$ singularity --version
singularity version 3.1.0-rc2.28.ga72e427
```
Next, we are going to create a bundle. A bundle is a folder that we will
treat as the root of our container filesystem. The easiest way to do
this is to dump a filesystem there from another container.
```bash
$ singularity build --sandbox /tmp/bundle docker://ubuntu:18.04
$ cp config.json /tmp/bundle
```
The purpose of the build is only to dump a complete operating system into the
bundle folder. The configuration file then is to conform to the Oci
Runtime specification.
We can then test interaction with the OCI client of Singularity python
by providing the bundle directory at /tmp/bundle.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/oci/__init__.py 0000644 0001751 0000177 00000010707 14536152223 017306 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
from spython.image import ImageBase
from spython.logger import bot
class OciImage(ImageBase):
# Default functions of client don't use sudo
sudo = False
def __init__(
self, container_id=None, bundle=None, create=True, sudo=True, **kwargs
):
"""An Oci Image is an Image Base with OCI functions appended
Parameters
==========
container_id: image uri to parse (required)
bundle: a bundle directory to create a container from.
the bundle should have a config.json at the root
create: if the bundle is provided, create a container (default True)
sudo: if init is called with or without sudo, keep a record and use
for following commands unless sudo is provided to function.
"""
super(OciImage, self).__init__()
# Will typically be None, unless used outside of Client
self.container_id = container_id
self.protocol = "oci"
self.sudo = sudo
# If bundle is provided, create it
if bundle is not None and container_id is not None and create:
self.bundle = bundle
self.create(bundle, container_id, **kwargs)
# Unique resource identifier
def get_container_id(self, container_id=None):
"""a helper function shared between functions that will return a
container_id. First preference goes to a container_id provided by
the user at runtime. Second preference goes to the container_id
instantiated with the client.
Parameters
==========
container_id: image uri to parse (required)
"""
# The user must provide a container_id, or have one with the client
if container_id is None and self.container_id is None:
bot.exit("You must provide a container_id.")
# Choose whichever is not None, with preference for function provided
container_id = container_id or self.container_id
return container_id
def get_uri(self):
"""return the image uri (oci://) along with it's name"""
return self.__str__()
# Naming
def __str__(self):
if self.container_id is not None:
return "[singularity-python-oci:%s]" % self.container_id
return "[singularity-python-oci]"
def __repr__(self):
return self.__str__()
# Commands
def _get_sudo(self, sudo=None):
"""if the client was initialized with sudo, remember this choice for
later communication with the Oci Images. However, if the user provides
a sudo argument (True or False) and not the default None, take
preference to this argument.
Parameters
==========
sudo: if None, use self.sudo. Otherwise return sudo.
"""
if sudo is None:
sudo = self.sudo
return sudo
def _run_and_return(self, cmd, sudo=None):
"""Run a command, show the message to the user if quiet isn't set,
and return the return code. This is a wrapper for the OCI client
to run a command and easily return the return code value (what
the user is ultimately interested in).
Parameters
==========
cmd: the command (list) to run.
sudo: whether to add sudo or not.
"""
sudo = self._get_sudo(sudo)
result = self._run_command(cmd, sudo=sudo, quiet=True, return_result=True)
# Successful return with no output
if not result:
return
# Show the response to the user, only if not quiet.
elif not self.quiet:
bot.println(result["message"])
# Return the state object to the user
return result["return_code"]
def _init_command(self, action, flags=None):
"""a wrapper to the base init_command, ensuring that "oci" is added
to each command
Parameters
==========
action: the main action to perform (e.g., build)
flags: one or more additional flags (e.g, volumes)
not implemented yet.
"""
from spython.main.base.command import init_command
if not isinstance(action, list):
action = [action]
cmd = ["oci"] + action
return init_command(self, cmd, flags)
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.041557
spython-0.3.13/spython/oci/cmd/ 0000755 0001751 0000177 00000000000 14536152245 015737 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/oci/cmd/__init__.py 0000644 0001751 0000177 00000003013 14536152223 020041 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
def generate_oci_commands():
"""The oci command group will allow interaction with an image using
OCI commands.
"""
# run_command uses run_cmd, but wraps to catch error
from spython.main.base.command import run_command, send_command
from spython.main.base.generate import RobotNamer
from spython.main.base.logger import println
from spython.oci import OciImage
from .actions import _run, attach, create, delete, execute, run, update
# Oci Command Groups
from .mounts import mount, umount
from .states import _state_command, kill, pause, resume, start, state
# Oci Commands
OciImage.start = start
OciImage.mount = mount
OciImage.umount = umount
OciImage.state = state
OciImage.resume = resume
OciImage.pause = pause
OciImage.attach = attach
OciImage.create = create
OciImage.delete = delete
OciImage.execute = execute
OciImage.update = update
OciImage.kill = kill
OciImage.run = run
OciImage._run = _run
OciImage._state_command = _state_command
OciImage.RobotNamer = RobotNamer()
OciImage._send_command = send_command # send and disregard stderr, stdout
OciImage._run_command = run_command
OciImage._println = println
OciImage.OciImage = OciImage
return OciImage
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/oci/cmd/actions.py 0000644 0001751 0000177 00000022740 14536152223 017752 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
from spython.logger import bot
from spython.utils import stream_command
def run(
self,
bundle,
container_id=None,
log_path=None,
pid_file=None,
singularity_options=None,
log_format="kubernetes",
):
"""run is a wrapper to create, start, attach, and delete a container.
Equivalent command line example:
singularity oci run -b ~/bundle mycontainer
Parameters
==========
bundle: the full path to the bundle folder
container_id: an optional container_id. If not provided, use same
container_id used to generate OciImage instance
log_path: the path to store the log.
pid_file: specify the pid file path to use
log_format: defaults to kubernetes. Can also be "basic" or "json"
singularity_options: a list of options to provide to the singularity client
"""
return self._run(
bundle,
container_id=container_id,
log_path=log_path,
pid_file=pid_file,
command="run",
log_format=log_format,
singularity_options=singularity_options,
)
def create(
self,
bundle,
container_id=None,
empty_process=False,
log_path=None,
pid_file=None,
sync_socket=None,
log_format="kubernetes",
singularity_options=None,
):
"""use the client to create a container from a bundle directory. The bundle
directory should have a config.json. You must be the root user to
create a runtime.
Equivalent command line example:
singularity oci create [create options...]
Parameters
==========
bundle: the full path to the bundle folder
container_id: an optional container_id. If not provided, use same
container_id used to generate OciImage instance
empty_process: run container without executing container process (for
example, for a pod container waiting for signals). This
is a specific use case for tools like Kubernetes
log_path: the path to store the log.
pid_file: specify the pid file path to use
sync_socket: the path to the unix socket for state synchronization.
log_format: defaults to kubernetes. Can also be "basic" or "json"
singularity_options: a list of options to provide to the singularity client
"""
return self._run(
bundle,
container_id=container_id,
empty_process=empty_process,
log_path=log_path,
pid_file=pid_file,
sync_socket=sync_socket,
command="create",
log_format=log_format,
singularity_options=singularity_options,
)
def _run(
self,
bundle,
container_id=None,
empty_process=False,
log_path=None,
pid_file=None,
sync_socket=None,
command="run",
log_format="kubernetes",
singularity_options=None,
):
"""_run is the base function for run and create, the only difference
between the two being that run does not have an option for sync_socket.
Equivalent command line example:
singularity oci create [create options...]
Parameters
==========
bundle: the full path to the bundle folder
container_id: an optional container_id. If not provided, use same
container_id used to generate OciImage instance
empty_process: run container without executing container process (for
example, for a pod container waiting for signals). This
is a specific use case for tools like Kubernetes
log_path: the path to store the log.
pid_file: specify the pid file path to use
sync_socket: the path to the unix socket for state synchronization.
command: the command (run or create) to use (default is run)
log_format: defaults to kubernetes. Can also be "basic" or "json"
singularity_options: a list of options to provide to the singularity client
"""
container_id = self.get_container_id(container_id)
# singularity oci create
cmd = self._init_command(command, singularity_options)
# Check that the bundle exists
if not os.path.exists(bundle):
bot.exit("Bundle not found at %s" % bundle)
# Add the bundle
cmd = cmd + ["--bundle", bundle]
# Additional Logging Files
cmd = cmd + ["--log-format", log_format]
if log_path is not None:
cmd = cmd + ["--log-path", log_path]
if pid_file is not None:
cmd = cmd + ["--pid-file", pid_file]
if sync_socket is not None:
cmd = cmd + ["--sync-socket", sync_socket]
if empty_process:
cmd.append("--empty-process")
# Finally, add the container_id
cmd.append(container_id)
# Generate the instance
self._send_command(cmd, sudo=True)
# Get the status to report to the user!
return self.state(container_id, sudo=True, sync_socket=sync_socket)
def delete(self, container_id=None, sudo=None, singularity_options=None):
"""delete an instance based on container_id.
Parameters
==========
container_id: the container_id to delete
singularity_options: a list of options to provide to the singularity client
sudo: whether to issue the command with sudo (or not)
a container started with sudo will belong to the root user
If started by a user, the user needs to control deleting it
if the user doesn't set to True/False, we use client self.sudo
Returns
=======
return_code: the return code from the delete command. 0 indicates a
successful delete, 255 indicates not.
"""
sudo = self._get_sudo(sudo)
container_id = self.get_container_id(container_id)
# singularity oci delete
cmd = self._init_command("delete", singularity_options)
# Add the container_id
cmd.append(container_id)
# Delete the container, return code goes to user (message to screen)
return self._run_and_return(cmd, sudo=sudo)
def attach(self, container_id=None, sudo=False, singularity_options=None):
"""attach to a container instance based on container_id
Parameters
==========
container_id: the container_id to delete
singularity_options: a list of options to provide to the singularity client
sudo: whether to issue the command with sudo (or not)
a container started with sudo will belong to the root user
If started by a user, the user needs to control deleting it
Returns
=======
return_code: the return code from the delete command. 0 indicates a
successful delete, 255 indicates not.
"""
sudo = self._get_sudo(sudo)
container_id = self.get_container_id(container_id)
# singularity oci attach
cmd = self._init_command("attach", singularity_options)
# Add the container_id
cmd.append(container_id)
# Delete the container, return code goes to user (message to screen)
return self._run_and_return(cmd, sudo)
def execute(
self,
command=None,
container_id=None,
sudo=False,
stream=False,
singularity_options=None,
):
"""execute a command to a container instance based on container_id
Parameters
==========
container_id: the container_id to delete
command: the command to execute to the container
singularity_options: a list of options to provide to the singularity client
sudo: whether to issue the command with sudo (or not)
a container started with sudo will belong to the root user
If started by a user, the user needs to control deleting it
stream: if True, return an iterate to iterate over results of exec.
default is False, will return full output as string.
Returns
=======
return_code: the return code from the delete command. 0 indicates a
successful delete, 255 indicates not.
"""
sudo = self._get_sudo(sudo)
container_id = self.get_container_id(container_id)
# singularity oci delete
cmd = self._init_command("exec", singularity_options)
# Add the container_id
cmd.append(container_id)
if command is not None:
if not isinstance(command, list):
command = [command]
cmd = cmd + command
# Execute the command, return response to user
if stream:
return stream_command(cmd, sudo=sudo)
return self._run_command(cmd, sudo=sudo, quiet=True)
def update(self, container_id, from_file=None, sudo=False, singularity_options=None):
"""update container cgroup resources for a specific container_id,
The container must have state "running" or "created."
Singularity Example:
singularity oci update [update options...]
singularity oci update --from-file cgroups-update.json mycontainer
Parameters
==========
container_id: the container_id to update cgroups for
from_file: a path to an OCI JSON resource file to update from.
singularity_options: a list of options to provide to the singularity client
"""
sudo = self._get_sudo(sudo)
container_id = self.get_container_id(container_id)
# singularity oci delete
cmd = self._init_command("update", singularity_options)
if from_file is not None:
cmd = cmd + ["--from-file", from_file]
# Add the container_id
cmd.append(container_id)
# Delete the container, return code goes to user (message to screen)
return self._run_and_return(cmd, sudo)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/oci/cmd/mounts.py 0000644 0001751 0000177 00000001276 14536152223 017640 0 ustar 00runner docker # Copyright (C) 2019-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
def mount(self, image, sudo=None):
"""create an OCI bundle from SIF image
Parameters
==========
image: the container (sif) to mount
"""
return self._state_command(image, command="mount", sudo=sudo)
def umount(self, image, sudo=None):
"""delete an OCI bundle created from SIF image
Parameters
==========
image: the container (sif) to mount
"""
return self._state_command(image, command="umount", sudo=sudo)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/oci/cmd/states.py 0000644 0001751 0000177 00000013672 14536152223 017621 0 ustar 00runner docker # Copyright (C) 2017-2022 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import json
def state(
self, container_id=None, sudo=None, sync_socket=None, singularity_options=None
):
"""get the state of an OciImage, if it exists. The optional states that
can be returned are created, running, stopped or (not existing).
Equivalent command line example:
singularity oci state
Parameters
==========
container_id: the id to get the state of.
sudo: Add sudo to the command. If the container was created by root,
you need sudo to interact and get its state.
sync_socket: the path to the unix socket for state synchronization
singularity_options: a list of options to provide to the singularity client
Returns
=======
state: a parsed json of the container state, if exists. If the
container is not found, None is returned.
"""
sudo = self._get_sudo(sudo)
container_id = self.get_container_id(container_id)
# singularity oci state
cmd = self._init_command("state", singularity_options)
if sync_socket is not None:
cmd = cmd + ["--sync-socket", sync_socket]
# Finally, add the container_id
cmd.append(container_id)
# Get the instance state
result = self._run_command(cmd, sudo=sudo, quiet=True)
if result is not None:
# If successful, a string is returned to parse
if isinstance(result, str):
return json.loads(result)
def _state_command(
self, container_id=None, command="start", sudo=None, singularity_options=None
):
"""A generic state command to wrap pause, resume, kill, etc., where the
only difference is the command. This function will be unwrapped if the
child functions get more complicated (with additional arguments).
Equivalent command line example:
singularity oci
Parameters
==========
container_id: the id to start.
command: one of start, resume, pause, kill, defaults to start.
singularity_options: a list of options to provide to the singularity client
sudo: Add sudo to the command. If the container was created by root,
you need sudo to interact and get its state.
Returns
=======
return_code: the return code to indicate if the container was started.
"""
sudo = self._get_sudo(sudo)
container_id = self.get_container_id(container_id)
# singularity oci state
cmd = self._init_command(command, singularity_options)
# Finally, add the container_id
cmd.append(container_id)
# Run the command, return return code
return self._run_and_return(cmd, sudo)
def start(self, container_id=None, sudo=None, singularity_options=None):
"""start a previously invoked OciImage, if it exists.
Equivalent command line example:
singularity oci start
Parameters
==========
container_id: the id to start.
sudo: Add sudo to the command. If the container was created by root,
you need sudo to interact and get its state.
Returns
=======
return_code: the return code to indicate if the container was started.
"""
return self._state_command(
container_id, sudo=sudo, singularity_options=singularity_options
)
def kill(self, container_id=None, sudo=None, signal=None, singularity_options=None):
"""stop (kill) a started OciImage container, if it exists
Equivalent command line example:
singularity oci kill
Parameters
==========
container_id: the id to stop.
signal: signal sent to the container (default SIGTERM)
singularity_options: a list of options to provide to the singularity client
sudo: Add sudo to the command. If the container was created by root,
you need sudo to interact and get its state.
Returns
=======
return_code: the return code to indicate if the container was killed.
"""
sudo = self._get_sudo(sudo)
container_id = self.get_container_id(container_id)
# singularity oci state
cmd = self._init_command("kill", singularity_options)
# Finally, add the container_id
cmd.append(container_id)
# Add the signal, if defined
if signal is not None:
cmd = cmd + ["--signal", signal]
# Run the command, return return code
return self._run_and_return(cmd, sudo)
def resume(self, container_id=None, sudo=None, singularity_options=None):
"""resume a stopped OciImage container, if it exists
Equivalent command line example:
singularity oci resume
Parameters
==========
container_id: the id to stop.
singularity_options: a list of options to provide to the singularity client
sudo: Add sudo to the command. If the container was created by root,
you need sudo to interact and get its state.
Returns
=======
return_code: the return code to indicate if the container was resumed.
"""
return self._state_command(
container_id,
command="resume",
sudo=sudo,
singularity_options=singularity_options,
)
def pause(self, container_id=None, sudo=None, singularity_options=None):
"""pause a running OciImage container, if it exists
Equivalent command line example:
singularity oci pause
Parameters
==========
container_id: the id to stop.
singularity_options: a list of options to provide to the singularity client
sudo: Add sudo to the command. If the container was created by root,
you need sudo to interact and get its state.
Returns
=======
return_code: the return code to indicate if the container was paused.
"""
return self._state_command(
container_id,
command="pause",
sudo=sudo,
singularity_options=singularity_options,
)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/oci/config.json 0000644 0001751 0000177 00000004736 14536152223 017342 0 ustar 00runner docker {
"ociVersion": "1.0.1",
"process": {
"terminal": true,
"user": {
"uid": 1,
"gid": 1
},
"args": [
"sh"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm"
],
"cwd": "/",
"noNewPrivileges": true
},
"root": {
"path": ".",
"readonly": true
},
"hostname": "slartibartfast",
"mounts": [
{
"destination": "/proc",
"type": "proc",
"source": "proc"
},
{
"destination": "/dev/pts",
"type": "devpts",
"source": "devpts"
},
{
"destination": "/dev/shm",
"type": "tmpfs",
"source": "shm"
},
{
"destination": "/dev/mqueue",
"type": "mqueue",
"source": "mqueue"
},
{
"destination": "/sys",
"type": "sysfs",
"source": "sysfs"
},
{
"destination": "/sys/fs/cgroup",
"type": "cgroup",
"source": "cgroup"
}
],
"linux": {
"uidMappings": [
{
"containerID": 0,
"hostID": 1000,
"size": 32000
}
],
"gidMappings": [
{
"containerID": 0,
"hostID": 1000,
"size": 32000
}
],
"rootfsPropagation": "slave",
"namespaces": [
{
"type": "pid"
},
{
"type": "network"
},
{
"type": "ipc"
},
{
"type": "uts"
},
{
"type": "mount"
},
{
"type": "user"
},
{
"type": "cgroup"
}
],
"maskedPaths": [
"/proc/kcore",
"/proc/latency_stats",
"/proc/timer_stats",
"/proc/sched_debug"
],
"readonlyPaths": [
"/proc/asound",
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"annotations": {
"com.example.key1": "value1",
"com.example.key2": "value2"
}
}
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.045557
spython-0.3.13/spython/tests/ 0000755 0001751 0000177 00000000000 14536152245 015564 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/Xtest_oci.py 0000644 0001751 0000177 00000006145 14536152223 020101 0 ustar 00runner docker #!/usr/bin/python
# Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import shutil
import pytest
from spython.main import Client
from spython.main.base.generate import RobotNamer
from spython.utils import get_installdir
@pytest.fixture
def sandbox(tmp_path):
image = Client.build(
"docker://busybox:1.30.1",
image=str(tmp_path / "sandbox"),
sandbox=True,
sudo=False,
)
assert os.path.exists(image)
config = os.path.join(get_installdir(), "oci", "config.json")
shutil.copyfile(config, os.path.join(image, "config.json"))
return image
def test_oci_image():
image = Client.oci.OciImage("oci://imagename")
assert image.get_uri() == "[singularity-python-oci:oci://imagename]"
def test_oci(sandbox): # pylint: disable=redefined-outer-name
image = sandbox
container_id = RobotNamer().generate()
# A non existing process should not have a state
print("...Case 1. Check status of non-existing bundle.")
state = Client.oci.state("mycontainer")
assert state is None
# This will use sudo
print("...Case 2: Create OCI image from bundle")
result = Client.oci.create(bundle=image, container_id=container_id)
print(result)
assert result["status"] == "created"
print("...Case 3. Execute command to non running bundle.")
result = Client.oci.execute(
container_id=container_id, sudo=True, command=["ls", "/"]
)
print(result)
assert "bin" in result
print("...Case 4. Start container return value 0.")
state = Client.oci.start(container_id, sudo=True)
assert state == 0
print("...Case 5. Execute command to running bundle.")
result = Client.oci.execute(
container_id=container_id, sudo=True, command=["ls", "/"]
)
print(result)
assert "bin" in result
print("...Case 6. Check status of existing bundle.")
state = Client.oci.state(container_id, sudo=True)
assert state["status"] == "running"
print("...Case 7. Pause running container return value 0.")
state = Client.oci.pause(container_id, sudo=True)
assert state == 0
# State was still reported as running
print("...check status of paused bundle.")
state = Client.oci.state(container_id, sudo=True)
assert state["status"] == "paused"
print("...Case 8. Resume paused container return value 0.")
state = Client.oci.resume(container_id, sudo=True)
assert state == 0
print("...check status of resumed bundle.")
state = Client.oci.state(container_id, sudo=True)
assert state["status"] == "running"
print("...Case 9. Kill container.")
state = Client.oci.kill(container_id, sudo=True)
assert state == 0
# Clean up the image (should still use sudo)
# Bug in singularity that kill doesn't kill completely - this returns
# 255. When testsupdated to 3.1.* add signal=K to run
result = Client.oci.delete(container_id, sudo=True)
assert result in [0, 255]
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/__init__.py 0000644 0001751 0000177 00000000000 14536152223 017657 0 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/conftest.py 0000644 0001751 0000177 00000002227 14536152223 017762 0 ustar 00runner docker import os
from glob import glob
import pytest
from spython.main import Client
from spython.utils import get_installdir
@pytest.fixture
def installdir():
return get_installdir()
@pytest.fixture
def test_data(installdir): # pylint: disable=redefined-outer-name
root = os.path.join(installdir, "tests", "testdata")
dockerFiles = glob(os.path.join(root, "docker2singularity", "*.docker"))
singularityFiles = glob(os.path.join(root, "singularity2docker", "*.def"))
return {
"root": root,
"d2s": [(file, os.path.splitext(file)[0] + ".def") for file in dockerFiles],
"s2d": [
(file, os.path.splitext(file)[0] + ".docker") for file in singularityFiles
],
}
@pytest.fixture(scope="session")
def oras_container(tmp_path_factory):
folder = tmp_path_factory.mktemp("oras-img")
return folder, Client.pull(
"oras://ghcr.io/singularityhub/github-ci:latest", pull_folder=str(folder)
)
@pytest.fixture(scope="session")
def docker_container(tmp_path_factory):
folder = tmp_path_factory.mktemp("docker-img")
return folder, Client.pull("docker://busybox:1.30.1", pull_folder=str(folder))
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/helpers.sh 0000644 0001751 0000177 00000001053 14536152223 017555 0 ustar 00runner docker runTest() {
# The first argument is the code we should get
ERROR="${1:-}"
shift
OUTPUT=${1:-}
shift
"$@" > "${OUTPUT}" 2>&1
RETVAL="$?"
if [ "$ERROR" = "0" -a "$RETVAL" != "0" ]; then
echo "$@ (retval=$RETVAL) ERROR"
cat ${OUTPUT}
echo "Output in ${OUTPUT}"
exit 1
elif [ "$ERROR" != "0" -a "$RETVAL" = "0" ]; then
echo "$@ (retval=$RETVAL) ERROR"
echo "Output in ${OUTPUT}"
cat ${OUTPUT}
exit 1
else
echo "$@ (retval=$RETVAL) OK"
fi
}
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/test_base.py 0000644 0001751 0000177 00000000723 14536152223 020105 0 ustar 00runner docker #!/usr/bin/python
# Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
from spython.image import Image
def test_image():
image = Image("docker://ubuntu")
assert str(image) == "docker://ubuntu"
assert image.protocol == "docker"
assert image.image == "ubuntu"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/test_client.py 0000644 0001751 0000177 00000007075 14536152223 020460 0 ustar 00runner docker #!/usr/bin/python
# Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import shutil
from subprocess import CalledProcessError
import pytest
from spython.main import Client
from spython.utils import write_file
def test_build_from_docker(tmp_path):
container = str(tmp_path / "container.sif")
created_container = Client.build(
"docker://busybox:1.30.1", image=container, sudo=False
)
assert created_container == container
assert os.path.exists(created_container)
def test_export():
sandbox = "busybox:1.30.sandbox"
created_sandbox = Client.export("docker://busybox:1.30.1")
assert created_sandbox == sandbox
assert os.path.exists(created_sandbox)
shutil.rmtree(created_sandbox)
def test_docker_pull(docker_container):
tmp_path, container = docker_container
print(container)
assert container == str(tmp_path / ("busybox:1.30.1.sif"))
assert os.path.exists(container)
def test_oras_pull(oras_container):
tmp_path, container = oras_container
print(container)
assert container == str(tmp_path / ("github-ci:latest.sif"))
assert os.path.exists(container)
def test_execute(docker_container):
result = Client.execute(docker_container[1], "ls /")
print(result)
if isinstance(result, list):
result = "".join(result)
assert "tmp\nusr\nvar" in result
def test_execute_with_return_code(docker_container):
result = Client.execute(docker_container[1], "ls /", return_result=True)
print(result)
if isinstance(result["message"], list):
result["message"] = "".join(result["message"])
assert "tmp\nusr\nvar" in result["message"]
assert result["return_code"] == 0
def test_execute_with_stream(docker_container):
output = Client.execute(docker_container[1], "ls /", stream=True)
message = "".join(list(output))
assert "tmp\nusr\nvar" in message
output = Client.execute(
docker_container[1], "ls /", stream=True, stream_type="both"
)
message = "".join(list(output))
assert "tmp\nusr\nvar" in message
# ls / should be successful, so there will be no stderr
output = Client.execute(
docker_container[1], "ls /", stream=True, stream_type="stderr"
)
message = "".join(list(output))
assert "tmp\nusr\nvar" not in message
@pytest.mark.parametrize("return_code", [True, False])
def test_execute_with_called_process_error(
capsys, docker_container, return_code, tmp_path
):
tmp_file = os.path.join(tmp_path, "CalledProcessError.sh")
# "This is stdout" to stdout, "This is stderr" to stderr
script = f"""#!/bin/bash
echo "This is stdout"
>&2 echo "This is stderr"
{"exit 1" if return_code else ""}
"""
write_file(tmp_file, script)
if return_code:
with pytest.raises(CalledProcessError):
for line in Client.execute(
docker_container[1], f"/bin/sh {tmp_file}", stream=True
):
print(line, "")
else:
for line in Client.execute(
docker_container[1], f"/bin/sh {tmp_file}", stream=True
):
print(line, "")
captured = capsys.readouterr()
assert "stdout" in captured.out
if return_code:
assert "stderr" in captured.err
else:
assert "stderr" not in captured.err
def test_inspect(docker_container):
result = Client.inspect(docker_container[1])
assert "attributes" in result or "data" in result
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/test_client.sh 0000755 0001751 0000177 00000004204 14536152223 020434 0 ustar 00runner docker #!/bin/bash
# Include help functions
. helpers.sh
echo
echo "************** START: test_client.sh **********************"
# Create temporary testing directory
echo "Creating temporary directory to work in."
tmpdir=$(mktemp -d)
output=$(mktemp ${tmpdir:-/tmp}/spython_test.XXXXXX)
here="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
echo "Testing help commands..."
# Test help for all commands
for command in recipe shell;
do
runTest 0 $output spython $command --help
done
echo "#### Testing recipe auto generation"
runTest 1 $output spython recipe $here/testdata/Dockerfile | grep "FROM"
runTest 0 $output spython recipe $here/testdata/Dockerfile | grep "%post"
runTest 1 $output spython recipe $here/testdata/Singularity | grep "%post"
runTest 0 $output spython recipe $here/testdata/Singularity | grep "FROM"
echo "#### Testing recipe targeted generation"
runTest 0 $output spython recipe --writer docker $here/testdata/Dockerfile | grep "FROM"
runTest 1 $output spython recipe --writer docker $here/testdata/Dockerfile | grep "%post"
runTest 0 $output spython recipe --writer singularity $here/testdata/Singularity | grep "%post"
runTest 1 $output spython recipe --writer singularity $here/testdata/Singularity | grep "FROM"
echo "#### Testing recipe file generation"
outfile=$(mktemp ${tmpdir:-/tmp}/spython_recipe.XXXXXX)
runTest 0 $output spython recipe $here/testdata/Dockerfile $outfile
runTest 0 $output test -f "$outfile"
runTest 0 $output cat $outfile | grep "%post"
rm $outfile
echo "#### Testing recipe json export"
runTest 0 $output spython recipe --json $here/testdata/Dockerfile | grep "ports"
runTest 0 $output spython recipe $here/testdata/Dockerfile $outfile
runTest 0 $output test -f "$outfile"
runTest 0 $output cat $outfile | grep "%post"
# Force is false, should fail
echo "#### Testing recipe json export, writing to file"
runTest 0 $output spython recipe --json $here/testdata/Dockerfile $outfile
runTest 0 $output spython recipe --force --json $here/testdata/Dockerfile $outfile
runTest 0 $output test -f "$outfile"
runTest 0 $output cat $outfile | grep "ports"
echo "Finish testing basic client"
rm -rf ${tmpdir}
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/test_conversion.py 0000644 0001751 0000177 00000003555 14536152223 021366 0 ustar 00runner docker #!/usr/bin/python
# Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
from glob import glob
def read_file(file):
with open(file) as fd:
content = fd.read().strip("\n")
return content
def test_other_recipe_exists(test_data):
# Have any example
assert test_data["d2s"]
assert test_data["s2d"]
for _, outFile in test_data["d2s"] + test_data["s2d"]:
assert os.path.exists(outFile), outFile + " is missing"
dockerfiles = glob(
os.path.join(os.path.dirname(test_data["s2d"][0][0]), "*.docker")
)
singularityfiles = glob(
os.path.join(os.path.dirname(test_data["d2s"][0][0]), "*.def")
)
for file in dockerfiles:
assert file in [out for _, out in test_data["s2d"]]
for file in singularityfiles:
assert file in [out for _, out in test_data["d2s"]]
def test_docker2singularity(test_data, tmp_path):
from spython.main.parse.parsers import DockerParser
from spython.main.parse.writers import SingularityWriter
for dockerfile, recipe in test_data["d2s"]:
parser = DockerParser(dockerfile)
writer = SingularityWriter(parser.recipe)
result = read_file(recipe).strip()
assert writer.convert().replace("\n", "") == result.replace("\n", "")
def test_singularity2docker(test_data, tmp_path):
print("Testing spython conversion from singularity2docker")
from spython.main.parse.parsers import SingularityParser
from spython.main.parse.writers import DockerWriter
for recipe, dockerfile in test_data["s2d"]:
parser = SingularityParser(recipe)
writer = DockerWriter(parser.recipe)
assert writer.convert() == read_file(dockerfile)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/test_instances.py 0000644 0001751 0000177 00000004745 14536152223 021172 0 ustar 00runner docker #!/usr/bin/python
# Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
# name instance based on Python version in case running in parallel
import sys
import pytest
from spython.main import Client
version_string = "%s_%s_%s" % (
sys.version_info[0],
sys.version_info[1],
sys.version_info[2],
)
def test_instance_class():
instance = Client.instance("docker://ubuntu", start=False)
assert instance.get_uri() == "instance://" + instance.name
assert instance.name != ""
name = "coolName"
instance = Client.instance("docker://busybox:1.30.1", start=False, name=name)
assert instance.get_uri() == "instance://" + instance.name
assert instance.name == name
def test_has_no_instances():
instances = Client.instances()
assert instances == []
class TestInstanceFuncs:
@pytest.fixture(autouse=True)
def test_instance_cmds(self, docker_container):
image = docker_container[1]
instance_name = "instance1_" + version_string
myinstance = Client.instance(image, name=instance_name)
assert myinstance.get_uri().startswith("instance://")
print("...Case 2: List instances")
instances = Client.instances()
assert len(instances) == 1
instances = Client.instances(return_json=True)
assert len(instances) == 1
assert isinstance(instances[0], dict)
print("...Case 3: Commands to instances")
result = Client.execute(myinstance, ["echo", "hello"])
assert result == "hello\n"
print("...Case 4: Return value from instance")
result = Client.execute(myinstance, "ls /", return_result=True)
print(result)
assert "tmp\nusr\nvar" in result["message"]
assert result["return_code"] == 0
print("...Case 5: Stop instances")
myinstance.stop()
instances = Client.instances()
assert instances == []
myinstance1 = Client.instance(image, name="instance1_" + version_string)
myinstance2 = Client.instance(image, name="instance2_" + version_string)
assert myinstance1 is not None
assert myinstance2 is not None
instances = Client.instances()
assert len(instances) == 2
myinstance1.stop()
myinstance2.stop()
instances = Client.instances()
assert instances == []
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/test_parsers.py 0000644 0001751 0000177 00000003731 14536152223 020654 0 ustar 00runner docker #!/usr/bin/python
# Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
from spython.main.parse.parsers import DockerParser, SingularityParser
def test_get_parser():
from spython.main.parse.parsers import get_parser
parser = get_parser("docker")
assert parser == DockerParser
parser = get_parser("Dockerfile")
assert parser == DockerParser
parser = get_parser("Singularity")
assert parser == SingularityParser
def test_docker_parser(test_data):
dockerfile = os.path.join(test_data["root"], "Dockerfile")
parser = DockerParser(dockerfile)
assert str(parser) == "[spython-parser][docker]"
assert "spython-base" in parser.recipe
recipe = parser.recipe["spython-base"]
# Test all fields from recipe
assert recipe.fromHeader == "python:3.5.1"
assert recipe.cmd == "/code/run_uwsgi.sh"
assert recipe.entrypoint is None
assert recipe.workdir == "/code"
assert recipe.volumes == []
assert recipe.ports == ["3031"]
assert recipe.files[0] == ["requirements.txt", "/tmp/requirements.txt"]
assert recipe.environ == ["PYTHONUNBUFFERED=1"]
assert recipe.source == dockerfile
def test_singularity_parser(test_data):
recipefile = os.path.join(test_data["root"], "Singularity")
parser = SingularityParser(recipefile)
assert str(parser) == "[spython-parser][singularity]"
assert "spython-base" in parser.recipe
recipe = parser.recipe["spython-base"]
# Test all fields from recipe
assert recipe.fromHeader == "continuumio/miniconda3"
assert recipe.cmd == 'exec /opt/conda/bin/spython "$@"'
assert recipe.entrypoint is None
assert recipe.workdir is None
assert recipe.volumes == []
assert recipe.files == []
assert recipe.environ == []
assert recipe.source == recipefile
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/test_recipe.py 0000644 0001751 0000177 00000002515 14536152223 020443 0 ustar 00runner docker #!/usr/bin/python
# Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
def test_recipe_base():
from spython.main.parse.recipe import Recipe
recipe = Recipe()
assert str(recipe) == "[spython-recipe]"
attributes = [
"cmd",
"comments",
"entrypoint",
"environ",
"files",
"install",
"labels",
"ports",
"test",
"volumes",
"workdir",
]
for att in attributes:
assert hasattr(recipe, att)
print("Checking that empty recipe returns empty")
result = recipe.json()
assert not result
print("Checking that non-empty recipe returns values")
recipe.cmd = ["echo", "hello"]
recipe.entrypoint = "/bin/bash"
recipe.comments = ["This recipe is great", "Yes it is!"]
recipe.environ = ["PANCAKES=WITHSYRUP"]
recipe.files = [["one", "two"]]
recipe.test = ["true"]
recipe.install = ["apt-get update"]
recipe.labels = ["Maintainer vanessasaur"]
recipe.ports = ["3031"]
recipe.volumes = ["/data"]
recipe.workdir = "/code"
result = recipe.json()
for att in attributes:
assert att in result
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/test_utils.py 0000644 0001751 0000177 00000010727 14536152223 020340 0 ustar 00runner docker #!/usr/bin/python
# Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import pytest
from spython.utils import ScopedEnvVar
def test_write_read_files(tmp_path):
"""
test_write_read_files will test the functions write_file and read_file
"""
print("Testing utils.write_file...")
from spython.utils import write_file
tmpfile = str(tmp_path / "written_file.txt")
assert not os.path.exists(tmpfile)
write_file(tmpfile, "hello!")
assert os.path.exists(tmpfile)
print("Testing utils.read_file...")
from spython.utils import read_file
content = read_file(tmpfile)[0]
assert content == "hello!"
def test_write_bad_json(tmp_path):
from spython.utils import write_json
bad_json = {"Wakkawakkawakka'}": [{True}, "2", 3]}
tmpfile = str(tmp_path / "json_file.txt")
assert not os.path.exists(tmpfile)
with pytest.raises(TypeError):
write_json(bad_json, tmpfile)
def test_write_json(tmp_path):
import json
from spython.utils import write_json
good_json = {"Wakkawakkawakka": [True, "2", 3]}
tmpfile = str(tmp_path / "good_json_file.txt")
assert not os.path.exists(tmpfile)
write_json(good_json, tmpfile)
with open(tmpfile, "r") as f:
content = json.loads(f.read())
assert isinstance(content, dict)
assert "Wakkawakkawakka" in content
def test_check_install():
"""check install is used to check if a particular software is installed.
If no command is provided, singularity is assumed to be the test case"""
print("Testing utils.check_install")
from spython.utils import check_install
is_installed = check_install()
assert is_installed
is_not_installed = check_install("fakesoftwarename")
assert not is_not_installed
def test_check_get_singularity_version():
"""check that the singularity version is found to be that installed"""
from spython.utils import get_singularity_version
version = get_singularity_version()
assert version != ""
with ScopedEnvVar("SPYTHON_SINGULARITY_VERSION", "3.0"):
version = get_singularity_version()
assert version == "3.0"
def test_get_installdir():
"""get install directory should return the base of where singularity
is installed
"""
print("Testing utils.get_installdir")
from spython.utils import get_installdir
whereami = get_installdir()
print(whereami)
assert whereami.endswith("spython")
def test_split_uri():
from spython.utils import split_uri
protocol, image = split_uri("docker://ubuntu")
assert protocol == "docker"
assert image == "ubuntu"
protocol, image = split_uri("http://image/path/with/slash/")
assert protocol == "http"
assert image == "image/path/with/slash"
protocol, image = split_uri("no/proto/")
assert protocol == ""
assert image == "no/proto"
def test_remove_uri():
print("Testing utils.remove_uri")
from spython.utils import remove_uri
assert remove_uri("docker://ubuntu") == "ubuntu"
assert (
remove_uri("shub://vanessa/singularity-images") == "vanessa/singularity-images"
)
assert remove_uri("library://library/default/alpine") == "library/default/alpine"
assert remove_uri("vanessa/singularity-images") == "vanessa/singularity-images"
def test_decode():
from spython.logger import decodeUtf8String
out = decodeUtf8String(str("Hello"))
assert isinstance(out, str)
assert out == "Hello"
out = decodeUtf8String(bytes(b"Hello"))
assert isinstance(out, str)
assert out == "Hello"
def test_ScopedEnvVar():
assert "FOO" not in os.environ
with ScopedEnvVar("FOO", "bar") as e:
assert e.name == "FOO"
assert e.value == "bar"
assert os.environ["FOO"] == "bar"
with ScopedEnvVar("FOO", "baz"):
assert os.environ["FOO"] == "baz"
assert os.environ["FOO"] == "bar"
# None removes it
with ScopedEnvVar("FOO", None):
assert "FOO" not in os.environ
# But empty string is allowed
with ScopedEnvVar("FOO", ""):
assert os.environ["FOO"] == ""
assert os.environ["FOO"] == "bar"
assert "FOO" not in os.environ
# Unset a non-existing variable
with ScopedEnvVar("FOO", None):
assert "FOO" not in os.environ
assert "FOO" not in os.environ
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/test_writers.py 0000644 0001751 0000177 00000002417 14536152223 020674 0 ustar 00runner docker #!/usr/bin/python
# Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
from spython.main.parse.writers import DockerWriter, SingularityWriter
def test_writers():
from spython.main.parse.writers import get_writer
writer = get_writer("docker")
assert writer == DockerWriter
writer = get_writer("Dockerfile")
assert writer == DockerWriter
writer = get_writer("Singularity")
assert writer == SingularityWriter
def test_docker_writer(test_data):
from spython.main.parse.parsers import DockerParser
dockerfile = os.path.join(test_data["root"], "Dockerfile")
parser = DockerParser(dockerfile)
writer = DockerWriter(parser.recipe)
assert str(writer) == "[spython-writer][docker]"
print(writer.convert())
def test_singularity_writer(test_data):
from spython.main.parse.parsers import SingularityParser
recipe = os.path.join(test_data["root"], "Singularity")
parser = SingularityParser(recipe)
writer = SingularityWriter(parser.recipe)
assert str(writer) == "[spython-writer][singularity]"
print(writer.convert())
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.045557
spython-0.3.13/spython/tests/testdata/ 0000755 0001751 0000177 00000000000 14536152245 017375 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/Dockerfile 0000644 0001751 0000177 00000004203 14536152223 021362 0 ustar 00runner docker FROM python:3.5.1
ENV PYTHONUNBUFFERED 1
################################################################################
# CORE
# Do not modify this section
RUN apt-get update && apt-get install -y \
pkg-config \
cmake \
openssl \
wget \
git \
vim
RUN apt-get update && apt-get install -y \
anacron \
autoconf \
automake \
libarchive-dev \
libtool \
libopenblas-dev \
libglib2.0-dev \
gfortran \
libxml2-dev \
libxmlsec1-dev \
libhdf5-dev \
libgeos-dev \
libsasl2-dev \
libldap2-dev \
squashfs-tools \
build-essential
# Install Singularity
RUN git clone -b vault/release-2.5 https://www.github.com/sylabs/singularity.git
WORKDIR singularity
RUN ./autogen.sh && ./configure --prefix=/usr/local && make && make install
# Install Python requirements out of /tmp so not triggered if other contents of /code change
ADD requirements.txt /tmp/requirements.txt
RUN pip install --upgrade pip
RUN pip install -r /tmp/requirements.txt
ADD . /code/
################################################################################
# PLUGINS
# You are free to comment out those plugins that you don't want to use
# Install LDAP (uncomment if wanted)
# RUN pip install python3-ldap
# RUN pip install django-auth-ldap
# Install Globus (uncomment if wanted)
# RUN /bin/bash /code/scripts/globus/globus-install.sh
# Install SAML (uncomment if wanted)
# RUN pip install python3-saml
# RUN pip install social-auth-core[saml]
################################################################################
# BASE
RUN mkdir -p /code && mkdir -p /code/images
RUN mkdir -p /var/www/images && chmod -R 0755 /code/images/
USER tacos
WORKDIR /code
RUN apt-get remove -y gfortran
RUN apt-get autoremove -y
RUN apt-get clean
RUN rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Install crontab to setup job
RUN echo "0 0 * * * /usr/bin/python /code/manage.py generate_tree" >> /code/cronjob
RUN crontab /code/cronjob
RUN rm /code/cronjob
# Create hashed temporary upload locations
RUN mkdir -p /var/www/images/_upload/{0..9} && chmod 777 -R /var/www/images/_upload
CMD /code/run_uwsgi.sh
EXPOSE 3031
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/README.md 0000644 0001751 0000177 00000001416 14536152223 020652 0 ustar 00runner docker # Test Data
This folder contains test data for Singularity Python.
- [Singularity](Singularity) and [Docker](Docker) are generic recipes used to run the tests in [one folder up](../). They are not inended to be converted between one another.
- [singularity2docker](singularity2docker) is a folder of docker recipes (`*.docker`) and Singularity recipes (`*.def`) that are tested for conversion *from* Singularity to Docker.
- [docker2singularity](docker2singularity) is a folder of Singularity recipes (`*.def`) and docker recipes (`*.docker`) that are tested for conversion *from* Docker to Singularity.
To add a new pair of recipes to either folder, simply write a .def and .docker file with the same name. They will be tested by [test_conversion.py](../test_conversion.py).
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/Singularity 0000644 0001751 0000177 00000000615 14536152223 021630 0 ustar 00runner docker Bootstrap: docker
From: continuumio/miniconda3
%runscript
exec /opt/conda/bin/spython "$@"
%labels
maintainer vsochat@stanford.edu
%post
apt-get update && apt-get install -y git
# Dependencies
cd /opt
git clone https://www.github.com/singularityhub/singularity-cli
cd singularity-cli
/opt/conda/bin/pip install setuptools
/opt/conda/bin/python setup.py install
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.049557
spython-0.3.13/spython/tests/testdata/docker2singularity/ 0000755 0001751 0000177 00000000000 14536152245 023221 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/add.def 0000644 0001751 0000177 00000000212 14536152223 024420 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%files
. /opt
%runscript
exec /bin/bash "$@"
%startscript
exec /bin/bash "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/add.docker 0000644 0001751 0000177 00000000037 14536152223 025136 0 ustar 00runner docker FROM busybox:latest
ADD . /opt
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/argsub.def 0000644 0001751 0000177 00000001353 14536152223 025162 0 ustar 00runner docker Bootstrap: docker
From: nvidia/cuda:11.1.1-cudnn8-devel-ubuntu20.04
Stage: spython-base
%files
./requirements.txt /workspace
%labels
maintainer="Dong Wang"
%post
CUDA_VERSION=11.1.1
OS_VERSION=20.04
PATH="/root/miniconda3/bin:${PATH}"
PATH="/root/miniconda3/bin:${PATH}"
DEBIAN_FRONTEND=noninteractive
SHELL ["/bin/bash", "-c"]
apt-get update && apt-get upgrade -y &&\
apt-get install -y wget python3-pip
python3 -m pip install --upgrade pip
mkdir -p /workspace
cd /workspace
python3 -m pip install -r /workspace/requirements.txt and &&\
rm /workspace/requirements.txt
%environment
export PATH="/root/miniconda3/bin:${PATH}"
%runscript
cd /workspace
exec /bin/bash /bin/bash "$@"
%startscript
cd /workspace
exec /bin/bash /bin/bash "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/argsub.docker 0000644 0001751 0000177 00000001104 14536152223 025665 0 ustar 00runner docker ARG CUDA_VERSION=11.1.1
ARG OS_VERSION=20.04
FROM nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu${OS_VERSION}
LABEL maintainer="Dong Wang"
ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
ARG DEBIAN_FRONTEND=noninteractive
SHELL ["/bin/bash", "-c"]
RUN apt-get update && apt-get upgrade -y &&\
apt-get install -y wget python3-pip
RUN python3 -m pip install --upgrade pip
WORKDIR /workspace
ADD ./requirements.txt /workspace
RUN python3 -m pip install -r /workspace/requirements.txt and &&\
rm /workspace/requirements.txt
CMD ["/bin/bash"]
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/cmd.def 0000644 0001751 0000177 00000000222 14536152223 024434 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%runscript
exec /bin/bash echo hello "$@"
%startscript
exec /bin/bash echo hello "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/cmd.docker 0000644 0001751 0000177 00000000052 14536152223 025146 0 ustar 00runner docker FROM busybox:latest
CMD ["echo", "hello"]
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/comments.def 0000644 0001751 0000177 00000000333 14536152223 025521 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%post
# This is a really important line
cp /bin/echo /opt/echo
# I'm sure you agree with me?
%runscript
exec /bin/bash "$@"
%startscript
exec /bin/bash "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/comments.docker 0000644 0001751 0000177 00000000161 14536152223 026231 0 ustar 00runner docker FROM busybox:latest
# This is a really important line
RUN cp /bin/echo /opt/echo
# I'm sure you agree with me?
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/copy.def 0000644 0001751 0000177 00000000212 14536152223 024642 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%files
. /opt
%runscript
exec /bin/bash "$@"
%startscript
exec /bin/bash "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/copy.docker 0000644 0001751 0000177 00000000040 14536152223 025352 0 ustar 00runner docker FROM busybox:latest
COPY . /opt
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/entrypoint-cmd.def 0000644 0001751 0000177 00000000226 14536152223 026651 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%runscript
exec python /code/script.py "$@"
%startscript
exec python /code/script.py "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/entrypoint-cmd.docker 0000644 0001751 0000177 00000000102 14536152223 027353 0 ustar 00runner docker FROM busybox:latest
CMD ["/code/script.py"]
ENTRYPOINT ["python"]
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/entrypoint.def 0000644 0001751 0000177 00000000226 14536152223 026110 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%runscript
exec /bin/bash run_uwsgi.sh "$@"
%startscript
exec /bin/bash run_uwsgi.sh "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/entrypoint.docker 0000644 0001751 0000177 00000000066 14536152223 026623 0 ustar 00runner docker FROM busybox:latest
ENTRYPOINT /bin/bash run_uwsgi.sh
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/expose.def 0000644 0001751 0000177 00000000236 14536152223 025201 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%post
# EXPOSE 3031
# EXPOSE 9000
%runscript
exec /bin/bash "$@"
%startscript
exec /bin/bash "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/expose.docker 0000644 0001751 0000177 00000000054 14536152223 025710 0 ustar 00runner docker FROM busybox:latest
EXPOSE 3031
EXPOSE 9000
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/from.def 0000644 0001751 0000177 00000000174 14536152223 024642 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%runscript
exec /bin/bash "$@"
%startscript
exec /bin/bash "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/from.docker 0000644 0001751 0000177 00000000024 14536152223 025345 0 ustar 00runner docker FROM busybox:latest
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/healthcheck.def 0000644 0001751 0000177 00000000207 14536152223 026137 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%runscript
exec /bin/bash "$@"
%startscript
exec /bin/bash "$@"
%test
true
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/healthcheck.docker 0000644 0001751 0000177 00000000045 14536152223 026650 0 ustar 00runner docker FROM busybox:latest
HEALTHCHECK true
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/label.def 0000644 0001751 0000177 00000000230 14536152223 024747 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%labels
maintainer dinosaur
%runscript
exec /bin/bash "$@"
%startscript
exec /bin/bash "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/label.docker 0000644 0001751 0000177 00000000056 14536152223 025466 0 ustar 00runner docker FROM busybox:latest
LABEL maintainer dinosaur
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/multiple-lines.def 0000644 0001751 0000177 00000000315 14536152223 026637 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%post
apt-get update && \
apt-get install -y git \
wget \
curl \
squashfs-tools
%runscript
exec /bin/bash "$@"
%startscript
exec /bin/bash "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/multiple-lines.docker 0000644 0001751 0000177 00000000254 14536152223 027352 0 ustar 00runner docker FROM busybox:latest
RUN apt-get update && \
apt-get install -y git \
wget \
curl \
squashfs-tools
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/multistage.def 0000644 0001751 0000177 00000000471 14536152223 026055 0 ustar 00runner docker Bootstrap: docker
From: golang:1.12.3-alpine3.9
Stage: devel
%post
export PATH="/go/bin:/usr/local/go/bin:$PATH"
export HOME="/root"
cd /root
touch hello
Bootstrap: docker
From: alpine:3.9
Stage: final
%files from devel
/root/hello /bin/hello
%runscript
exec /bin/bash "$@"
%startscript
exec /bin/bash "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/multistage.docker 0000644 0001751 0000177 00000000317 14536152223 026565 0 ustar 00runner docker FROM golang:1.12.3-alpine3.9 AS devel
RUN export PATH="/go/bin:/usr/local/go/bin:$PATH"
RUN export HOME="/root"
RUN cd /root
RUN touch hello
FROM alpine:3.9 AS final
COPY --from=devel /root/hello /bin/hello
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/user.def 0000644 0001751 0000177 00000000325 14536152223 024653 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%post
echo "cloud"
su - rainman # USER rainman
echo "makeitrain"
su - root # USER root
%runscript
exec /bin/bash "$@"
%startscript
exec /bin/bash "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/user.docker 0000644 0001751 0000177 00000000122 14536152223 025357 0 ustar 00runner docker FROM busybox:latest
RUN echo "cloud"
USER rainman
RUN echo "makeitrain"
USER root
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/workdir.def 0000644 0001751 0000177 00000000254 14536152223 025357 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
Stage: spython-base
%post
mkdir -p /code
cd /code
%runscript
cd /code
exec /bin/bash "$@"
%startscript
cd /code
exec /bin/bash "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/docker2singularity/workdir.docker 0000644 0001751 0000177 00000000042 14536152223 026063 0 ustar 00runner docker FROM busybox:latest
WORKDIR /code
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.049557
spython-0.3.13/spython/tests/testdata/singularity2docker/ 0000755 0001751 0000177 00000000000 14536152245 023221 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/files.def 0000644 0001751 0000177 00000000137 14536152223 025000 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
%files
file.txt /opt/file.txt
/path/to/thing /opt/thing
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/files.docker 0000644 0001751 0000177 00000000135 14536152223 025507 0 ustar 00runner docker FROM busybox:latest AS spython-base
ADD file.txt /opt/file.txt
ADD /path/to/thing /opt/thing
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/from.def 0000644 0001751 0000177 00000000047 14536152223 024641 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/from.docker 0000644 0001751 0000177 00000000044 14536152223 025347 0 ustar 00runner docker FROM busybox:latest AS spython-base
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/labels.def 0000644 0001751 0000177 00000000121 14536152223 025131 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
%labels
Maintainer dinosaur
Version 1.0.0
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/labels.docker 0000644 0001751 0000177 00000000122 14536152223 025643 0 ustar 00runner docker FROM busybox:latest AS spython-base
LABEL Maintainer dinosaur
LABEL Version 1.0.0
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/multiple-lines.def 0000644 0001751 0000177 00000000261 14536152223 026637 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
%post
apt-get update && \
apt-get install -y git \
wget \
curl \
squashfs-tools
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/multiple-lines.docker 0000644 0001751 0000177 00000000162 14536152223 027350 0 ustar 00runner docker FROM busybox:latest AS spython-base
RUN apt-get update && \
apt-get install -y git \
wget \
curl \
squashfs-tools
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/multistage.def 0000644 0001751 0000177 00000000527 14536152223 026057 0 ustar 00runner docker Bootstrap: docker
From: golang:1.12.3-alpine3.9
Stage: devel
%post
# prep environment
export PATH="/go/bin:/usr/local/go/bin:$PATH"
export HOME="/root"
cd /root
touch hello
# Install binary into final image
Bootstrap: docker
From: alpine:3.9
Stage: final
# install binary from stage one
%files from devel
/root/hello /bin/hello
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/multistage.docker 0000644 0001751 0000177 00000000317 14536152223 026565 0 ustar 00runner docker FROM golang:1.12.3-alpine3.9 AS devel
RUN export PATH="/go/bin:/usr/local/go/bin:$PATH"
RUN export HOME="/root"
RUN cd /root
RUN touch hello
FROM alpine:3.9 AS final
COPY --from=devel /root/hello /bin/hello
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/post.def 0000644 0001751 0000177 00000000155 14536152223 024663 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
%post
apt-get update
apt-get install -y git \
wget
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/post.docker 0000644 0001751 0000177 00000000131 14536152223 025366 0 ustar 00runner docker FROM busybox:latest AS spython-base
RUN apt-get update
RUN apt-get install -y git \
wget
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/runscript.def 0000644 0001751 0000177 00000000121 14536152223 025720 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
%runscript
exec /bin/bash echo hello "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/runscript.docker 0000644 0001751 0000177 00000000107 14536152223 026435 0 ustar 00runner docker FROM busybox:latest AS spython-base
CMD exec /bin/bash echo hello "$@"
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/test.def 0000644 0001751 0000177 00000000062 14536152223 024652 0 ustar 00runner docker Bootstrap: docker
From: busybox:latest
%test
true
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/tests/testdata/singularity2docker/test.docker 0000644 0001751 0000177 00000000171 14536152223 025364 0 ustar 00runner docker FROM busybox:latest AS spython-base
RUN echo "true" >> /tests.sh
RUN chmod u+x /tests.sh
HEALTHCHECK /bin/bash /tests.sh
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.049557
spython-0.3.13/spython/utils/ 0000755 0001751 0000177 00000000000 14536152245 015562 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/utils/__init__.py 0000644 0001751 0000177 00000000512 14536152223 017665 0 ustar 00runner docker from .fileio import mkdir_p, read_file, read_json, write_file, write_json
from .misc import ScopedEnvVar
from .terminal import (
check_install,
format_container_name,
get_installdir,
get_singularity_version,
get_userhome,
get_username,
remove_uri,
run_command,
split_uri,
stream_command,
)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/utils/fileio.py 0000644 0001751 0000177 00000004761 14536152223 017407 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import errno
import json
import os
import sys
from spython.logger import bot
################################################################################
## FOLDER OPERATIONS ###########################################################
################################################################################
def mkdir_p(path):
"""mkdir_p attempts to get the same functionality as mkdir -p
:param path: the path to create.
"""
try:
os.makedirs(path)
except OSError as e:
if e.errno == errno.EEXIST and os.path.isdir(path):
pass
else:
bot.error("Error creating path %s, exiting." % path)
sys.exit(1)
################################################################################
## FILE OPERATIONS #############################################################
################################################################################
def write_file(filename, content, mode="w"):
"""write_file will open a file, "filename" and write content, "content"
and properly close the file
"""
with open(filename, mode) as filey:
filey.writelines(content)
return filename
def write_json(json_obj, filename, mode="w", print_pretty=True):
"""
write_json will (optionally,pretty print) a json object to file
:param json_obj: the dict to print to json
:param filename: the output file to write to
:param pretty_print: if True, will use nicer formatting
"""
with open(filename, mode) as filey:
if print_pretty:
filey.writelines(json.dumps(json_obj, indent=4, separators=(",", ": ")))
else:
filey.writelines(json.dumps(json_obj))
return filename
def read_file(filename, mode="r", readlines=True):
"""
write_file will open a file, "filename" and write content, "content"
and properly close the file
"""
with open(filename, mode) as filey:
if readlines:
content = filey.readlines()
else:
content = filey.read()
return content
def read_json(filename, mode="r"):
"""
read_json reads in a json file and returns
the data structure as dict.
"""
with open(filename, mode) as filey:
data = json.load(filey)
return data
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/utils/misc.py 0000644 0001751 0000177 00000002410 14536152223 017060 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
def setEnvVar(name, value):
"""Set or unset an environment variable
name -- Name of the variable to set
value -- Value to use or None to clear
"""
if value is None:
if name in os.environ:
del os.environ[name]
else:
os.environ[name] = value
class ScopedEnvVar:
"""Temporarily change an environment variable
Usage:
with ScopedEnvVar("FOO", "bar"):
print(os.environ["FOO"]) # "bar"
print(os.environ["FOO"]) #
"""
def __init__(self, name, value):
"""Create the scoped environment variable object
name -- Name of the variable to set
value -- Value to use or None to clear
"""
self.name = name
self.value = value
self.oldValue = None
def __enter__(self):
self.oldValue = os.environ.get(self.name)
setEnvVar(self.name, self.value)
return self
def __exit__(self, ex_type, ex_value, traceback):
setEnvVar(self.name, self.oldValue)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/utils/terminal.py 0000644 0001751 0000177 00000015767 14536152223 017763 0 ustar 00runner docker """
# Copyright (C) 2017-2022 Vanessa Sochat.
This Source Code Form is subject to the terms of the
Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
"""
import os
import pwd
import re
import shlex
import subprocess
import sys
from spython.logger import bot, decodeUtf8String
def _process_sudo_cmd(cmd, sudo, sudo_options):
"""
Process the sudo command and honor adding environment (or not)
"""
if sudo and sudo_options is not None:
if isinstance(sudo_options, str):
sudo_options = shlex.split(sudo_options)
cmd = ["sudo"] + sudo_options + cmd
elif sudo:
cmd = ["sudo"] + cmd
return [x for x in cmd if x]
def check_install(software="singularity", quiet=True):
"""
check_install will attempt to run the singularity command, and
return True if installed. The command line utils will not run
without this check.
"""
cmd = [software, "--version"]
found = False
try:
version = run_command(cmd, quiet=True)
except Exception: # FileNotFoundError
return found
if version is not None:
if version["return_code"] == 0:
found = True
if not quiet:
version = version["message"]
bot.info("Found %s version %s" % (software.upper(), version))
return found
def get_singularity_version():
"""
get the full singularity client version as reported by
singularity --version [...]. For Singularity 3.x, this means:
"singularity version 3.0.1-1"
"""
version = os.environ.get("SPYTHON_SINGULARITY_VERSION", "")
if version == "":
try:
version = run_command(["singularity", "--version"], quiet=True)
except Exception: # FileNotFoundError
return version
if version["return_code"] == 0:
if version["message"]:
version = version["message"][0].strip("\n")
return version
def get_userhome():
"""
Get the user home based on the effective uid
"""
return pwd.getpwuid(os.getuid())[5]
def get_username():
"""
Get the user name based on the effective uid
"""
return pwd.getpwuid(os.getuid())[0]
def get_installdir():
"""
Get_installdir returns the installation directory of the application
"""
return os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
def stream_command(
cmd,
no_newline_regexp="Progess",
sudo=False,
sudo_options=None,
output_type="stdout",
):
"""
Stream a command (yield) back to the user, as each line is available.
# Example usage:
results = []
for line in stream_command(cmd):
print(line, end="")
results.append(line)
Parameters
==========
cmd: the command to send, should be a list for subprocess
no_newline_regexp: the regular expression to determine skipping a
newline. Defaults to finding Progress
sudo_options: string or list of strings that will be passed as options to sudo
"""
if output_type not in ["stdout", "stderr", "both"]:
bot.exit(
"Invalid output type %s. Must be stderr, stdout or both." % output_type
)
cmd = _process_sudo_cmd(cmd, sudo, sudo_options)
stderr_pipe = subprocess.STDOUT if output_type == "both" else subprocess.PIPE
process = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=stderr_pipe, universal_newlines=True
)
# Allow the runner to choose streaming output or error
stream = process.stdout.readline
if output_type == "stderr":
stream = process.stderr.readline
# Stream lines back to the caller
for line in iter(stream, ""):
if not re.search(no_newline_regexp, line):
yield line
# If there is an error, raise.
process.stdout.close()
return_code = process.wait()
if return_code:
# Some situations may return process without an attached stderr object
# to read from
if process.stderr:
print(process.stderr.read(), file=sys.stderr)
raise subprocess.CalledProcessError(return_code, cmd)
def run_command(
cmd,
sudo=False,
capture=True,
no_newline_regexp="Progess",
quiet=False,
sudo_options=None,
environ=None,
background=False,
):
"""
run_command uses subprocess to send a command to the terminal. If
capture is True, we use the parent stdout, so the progress bar (and
other commands of interest) are piped to the user. This means we
don't return the output to parse.
Parameters
==========
cmd: the command to send, should be a list for subprocess
sudo: if needed, add to start of command
no_newline_regexp: the regular expression to determine skipping a
newline. Defaults to finding Progress
capture: if True, don't set stdout and have it go to console. This
option can print a progress bar, but won't return the lines
as output.
sudo_options: string or list of strings that will be passed as options to sudo
background: run in background and don't try to get output.
"""
cmd = _process_sudo_cmd(cmd, sudo, sudo_options)
stdout = None
if capture:
stdout = subprocess.PIPE
# Use the parent stdout and stderr
if background:
subprocess.Popen(cmd, env=environ)
return
process = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=stdout, env=environ)
lines = []
found_match = False
for line in process.communicate():
if line:
line = decodeUtf8String(line)
lines.append(line)
if re.search(no_newline_regexp, line) and found_match:
if not quiet:
sys.stdout.write(line)
found_match = True
else:
if not quiet:
sys.stdout.write(line)
print(line.rstrip())
found_match = False
output = {"message": lines, "return_code": process.returncode}
return output
def format_container_name(name, special_characters=None):
"""
format_container_name will take a name supplied by the user,
remove all special characters (except for those defined by "special-characters"
and return the new image name.
"""
if special_characters is None:
special_characters = []
return "".join(e.lower() for e in name if e.isalnum() or e in special_characters)
def split_uri(container):
"""
Split the uri of a container into the protocol and image part
An empty protocol is returned if none found.
A trailing slash is removed from the image part.
"""
parts = container.split("://", 1)
if len(parts) == 2:
protocol, image = parts
else:
protocol = ""
image = parts[0]
return protocol, image.rstrip("/")
def remove_uri(container):
"""
remove_uri will remove docker:// or shub:// or library:// from the uri
"""
return split_uri(container)[1]
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417555.0
spython-0.3.13/spython/version.py 0000644 0001751 0000177 00000001204 14536152223 016452 0 ustar 00runner docker # Copyright (C) 2017-2024 Vanessa Sochat.
# This Source Code Form is subject to the terms of the
# Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
# with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
__version__ = "0.3.13"
AUTHOR = "Vanessa Sochat"
AUTHOR_EMAIL = "vsoch@users.noreply.github.com"
NAME = "spython"
PACKAGE_URL = "https://github.com/singularityhub/singularity-cli"
KEYWORDS = "singularity python client (spython)"
DESCRIPTION = "Command line python tool for working with singularity."
LICENSE = "LICENSE"
INSTALL_REQUIRES = ()
TESTS_REQUIRES = (("pytest", {"min_version": "4.6.2"}),)
././@PaxHeader 0000000 0000000 0000000 00000000033 00000000000 010211 x ustar 00 27 mtime=1702417573.037557
spython-0.3.13/spython.egg-info/ 0000755 0001751 0000177 00000000000 14536152245 016114 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417572.0
spython-0.3.13/spython.egg-info/PKG-INFO 0000644 0001751 0000177 00000020625 14536152244 017215 0 ustar 00runner docker Metadata-Version: 2.1
Name: spython
Version: 0.3.13
Summary: Command line python tool for working with singularity.
Home-page: https://github.com/singularityhub/singularity-cli
Author: Vanessa Sochat
Author-email: vsoch@users.noreply.github.com
Maintainer: Vanessa Sochat
Maintainer-email: vsoch@users.noreply.github.com
License: LICENSE
Keywords: singularity python client (spython)
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development
Classifier: Topic :: Scientific/Engineering
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python :: 3
Description-Content-Type: text/markdown
License-File: LICENSE
# Singularity Python
[](https://travis-ci.org/singularityhub/singularity-cli)
[](https://github.com/singularityhub/singularity-cli/actions?query=branch%3Amaster+workflow%3Aspython-ci)
Singularity Python (spython) is the Python API for working with Singularity containers. See
the [documentation](https://singularityhub.github.io/singularity-cli) for installation and usage, and
the [install instructions](https://singularityhub.github.io/singularity-cli/install) for a quick start.
**This library does not support Singularity 2.x! It won't work and we no longer support it.**
We provide a [Singularity](Singularity) recipe for you to use if more convenient, along with the [full modules docstring](https://singularityhub.github.io/singularity-cli/api/source/spython.main.base.html#module-spython.main.base).
As of version 0.1.0, we only support Singularity > 3.5.2. This is done to encourage using
newer versions of Singularity with security fixes. If you want to use an older version of Singularity,
you will need to use version 0.0.85 or earlier.
## 😁️ Contributors 😁️
We use the [all-contributors](https://github.com/all-contributors/all-contributors)
tool to generate a contributors graphic below.
## License
This code is licensed under the MPL 2.0 [LICENSE](LICENSE).
## Help and Contribution
Please contribute to the package, or post feedback and questions as issues. For points that require discussion of the larger group, please use the Singularity List
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417573.0
spython-0.3.13/spython.egg-info/SOURCES.txt 0000644 0001751 0000177 00000011420 14536152245 017776 0 ustar 00runner docker LICENSE
MANIFEST.in
README.md
pyproject.toml
setup.cfg
setup.py
spython/README.md
spython/__init__.py
spython/image.py
spython/version.py
spython.egg-info/PKG-INFO
spython.egg-info/SOURCES.txt
spython.egg-info/dependency_links.txt
spython.egg-info/entry_points.txt
spython.egg-info/not-zip-safe
spython.egg-info/top_level.txt
spython/client/__init__.py
spython/client/recipe.py
spython/client/shell.py
spython/client/test.py
spython/instance/__init__.py
spython/instance/cmd/__init__.py
spython/instance/cmd/logs.py
spython/instance/cmd/start.py
spython/instance/cmd/stop.py
spython/logger/__init__.py
spython/logger/compatibility.py
spython/logger/message.py
spython/logger/progress.py
spython/logger/spinner.py
spython/main/__init__.py
spython/main/apps.py
spython/main/build.py
spython/main/execute.py
spython/main/export.py
spython/main/help.py
spython/main/inspect.py
spython/main/instances.py
spython/main/pull.py
spython/main/run.py
spython/main/base/Dockerfile
spython/main/base/README.md
spython/main/base/__init__.py
spython/main/base/command.py
spython/main/base/flags.py
spython/main/base/generate.py
spython/main/base/logger.py
spython/main/base/sutils.py
spython/main/parse/__init__.py
spython/main/parse/recipe.py
spython/main/parse/parsers/README.md
spython/main/parse/parsers/__init__.py
spython/main/parse/parsers/base.py
spython/main/parse/parsers/docker.py
spython/main/parse/parsers/singularity.py
spython/main/parse/writers/__init__.py
spython/main/parse/writers/base.py
spython/main/parse/writers/docker.py
spython/main/parse/writers/singularity.py
spython/oci/README.md
spython/oci/__init__.py
spython/oci/config.json
spython/oci/cmd/__init__.py
spython/oci/cmd/actions.py
spython/oci/cmd/mounts.py
spython/oci/cmd/states.py
spython/tests/Xtest_oci.py
spython/tests/__init__.py
spython/tests/conftest.py
spython/tests/helpers.sh
spython/tests/test_base.py
spython/tests/test_client.py
spython/tests/test_client.sh
spython/tests/test_conversion.py
spython/tests/test_instances.py
spython/tests/test_parsers.py
spython/tests/test_recipe.py
spython/tests/test_utils.py
spython/tests/test_writers.py
spython/tests/testdata/Dockerfile
spython/tests/testdata/README.md
spython/tests/testdata/Singularity
spython/tests/testdata/docker2singularity/add.def
spython/tests/testdata/docker2singularity/add.docker
spython/tests/testdata/docker2singularity/argsub.def
spython/tests/testdata/docker2singularity/argsub.docker
spython/tests/testdata/docker2singularity/cmd.def
spython/tests/testdata/docker2singularity/cmd.docker
spython/tests/testdata/docker2singularity/comments.def
spython/tests/testdata/docker2singularity/comments.docker
spython/tests/testdata/docker2singularity/copy.def
spython/tests/testdata/docker2singularity/copy.docker
spython/tests/testdata/docker2singularity/entrypoint-cmd.def
spython/tests/testdata/docker2singularity/entrypoint-cmd.docker
spython/tests/testdata/docker2singularity/entrypoint.def
spython/tests/testdata/docker2singularity/entrypoint.docker
spython/tests/testdata/docker2singularity/expose.def
spython/tests/testdata/docker2singularity/expose.docker
spython/tests/testdata/docker2singularity/from.def
spython/tests/testdata/docker2singularity/from.docker
spython/tests/testdata/docker2singularity/healthcheck.def
spython/tests/testdata/docker2singularity/healthcheck.docker
spython/tests/testdata/docker2singularity/label.def
spython/tests/testdata/docker2singularity/label.docker
spython/tests/testdata/docker2singularity/multiple-lines.def
spython/tests/testdata/docker2singularity/multiple-lines.docker
spython/tests/testdata/docker2singularity/multistage.def
spython/tests/testdata/docker2singularity/multistage.docker
spython/tests/testdata/docker2singularity/user.def
spython/tests/testdata/docker2singularity/user.docker
spython/tests/testdata/docker2singularity/workdir.def
spython/tests/testdata/docker2singularity/workdir.docker
spython/tests/testdata/singularity2docker/files.def
spython/tests/testdata/singularity2docker/files.docker
spython/tests/testdata/singularity2docker/from.def
spython/tests/testdata/singularity2docker/from.docker
spython/tests/testdata/singularity2docker/labels.def
spython/tests/testdata/singularity2docker/labels.docker
spython/tests/testdata/singularity2docker/multiple-lines.def
spython/tests/testdata/singularity2docker/multiple-lines.docker
spython/tests/testdata/singularity2docker/multistage.def
spython/tests/testdata/singularity2docker/multistage.docker
spython/tests/testdata/singularity2docker/post.def
spython/tests/testdata/singularity2docker/post.docker
spython/tests/testdata/singularity2docker/runscript.def
spython/tests/testdata/singularity2docker/runscript.docker
spython/tests/testdata/singularity2docker/test.def
spython/tests/testdata/singularity2docker/test.docker
spython/utils/__init__.py
spython/utils/fileio.py
spython/utils/misc.py
spython/utils/terminal.py ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417572.0
spython-0.3.13/spython.egg-info/dependency_links.txt 0000644 0001751 0000177 00000000001 14536152244 022161 0 ustar 00runner docker
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417572.0
spython-0.3.13/spython.egg-info/entry_points.txt 0000644 0001751 0000177 00000000060 14536152244 021405 0 ustar 00runner docker [console_scripts]
spython = spython.client:main
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417572.0
spython-0.3.13/spython.egg-info/not-zip-safe 0000644 0001751 0000177 00000000001 14536152244 020341 0 ustar 00runner docker
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1702417572.0
spython-0.3.13/spython.egg-info/top_level.txt 0000644 0001751 0000177 00000000010 14536152244 020634 0 ustar 00runner docker spython
