pax_global_header00006660000000000000000000000064136274601570014525gustar00rootroot0000000000000052 comment=094bc84fdabff81c2eb2017d32caad2582835f90 pmemkv-python-1.0/000077500000000000000000000000001362746015700142035ustar00rootroot00000000000000pmemkv-python-1.0/.clang-format000066400000000000000000000017021362746015700165560ustar00rootroot00000000000000AccessModifierOffset: -8 AlignOperands: false AllowShortBlocksOnASingleLine: false AllowShortFunctionsOnASingleLine: false AllowShortIfStatementsOnASingleLine: false AlwaysBreakAfterDefinitionReturnType: false AlwaysBreakTemplateDeclarations: true BasedOnStyle: LLVM BraceWrapping: AfterClass: false AfterControlStatement: false AfterEnum: false AfterFunction: true AfterNamespace: true AfterObjCDeclaration: false AfterStruct: false AfterUnion: false BeforeCatch: false BeforeElse: false IndentBraces: false BreakBeforeBraces: Custom BreakStringLiterals: false ColumnLimit: 90 ConstructorInitializerAllOnOneLineOrOnePerLine: true ContinuationIndentWidth: 8 FixNamespaceComments: false IndentCaseLabels: true IndentWidth: 8 PointerAlignment: Right SpaceBeforeParens: ControlStatements SpacesBeforeTrailingComments: 1 SpacesInContainerLiterals: false SpacesInCStyleCastParentheses: false UseTab: Always SortIncludes: false pmemkv-python-1.0/.gitattributes000066400000000000000000000000721362746015700170750ustar00rootroot00000000000000* text=auto eol=lf *.jpg binary *.png binary *.gif binary pmemkv-python-1.0/.gitignore000066400000000000000000000003711362746015700161740ustar00rootroot00000000000000.* !.gitignore !.gitattributes !.travis.yml __pycache__/ /doc/_build/ # Distribution / packaging build/ develop-eggs/ dist/ downloads/ eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ pip-wheel-metadata/ share/python-wheels/ *.egg-info/ *.egg MANIFEST pmemkv-python-1.0/.travis.yml000077500000000000000000000016641362746015700163260ustar00rootroot00000000000000dist: trusty sudo: required language: python services: - docker env: matrix: - TYPE=pmemkv-master OS=ubuntu OS_VER=19.10 PUSH_IMAGE=1 - TYPE=pmemkv-stable-1.0 OS=ubuntu OS_VER=19.10 - TYPE=pmemkv-stable-1.1 OS=ubuntu OS_VER=19.10 - TYPE=pmemkv-master OS=fedora OS_VER=31 PUSH_IMAGE=1 - TYPE=pmemkv-stable-1.0 OS=fedora OS_VER=31 - TYPE=pmemkv-stable-1.1 OS=fedora OS_VER=31 before_install: - echo $TRAVIS_COMMIT_RANGE - export HOST_WORKDIR=`pwd` - export GITHUB_REPO=pmem/pmemkv-python - export DOCKERHUB_REPO=pmem/pmemkv-python - cd utils/docker - ./pull-or-rebuild-image.sh - if [[ -f push_image_to_repo_flag ]]; then PUSH_THE_IMAGE=1; fi - if [[ -f skip_build_package_check ]]; then export SKIP_CHECK=1; fi - rm -f push_image_to_repo_flag skip_build_package_check script: - ./build.sh after_success: - if [[ $PUSH_THE_IMAGE -eq 1 ]]; then images/push-image.sh $OS-$OS_VER; fi pmemkv-python-1.0/CONTRIBUTING.md000066400000000000000000000042401362746015700164340ustar00rootroot00000000000000# Contributing to pmemkv-python ## Opening New Issues Please log bugs or suggestions as [GitHub issues](https://github.com/pmem/pmemkv-python/issues). Details such as OS and pmemkv version are always appreciated. ## Submitting Pull Requests We take outside code contributions to `pmemkv-python` through GitHub pull requests. **NOTE: If you do decide to implement code changes and contribute them, please make sure you agree your contribution can be made available under the [BSD-style License used for PMEMKV-PYTHON](https://github.com/pmem/pmemkv-python/blob/master/LICENSE).** **NOTE: Submitting your changes also means that you certify the following:** ``` Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. ``` In case of any doubt, the gatekeeper may ask you to certify the above in writing, i.e. via email or by including a `Signed-off-by:` line at the bottom of your commit comments. To improve tracking of who is the author of a contribution, we kindly ask you to use your real name (not an alias) when committing your changes to pmemkv-python: ``` Author: Random J Developer ``` pmemkv-python-1.0/ChangeLog000066400000000000000000000017271362746015700157640ustar00rootroot00000000000000Tue Mar 03 2020 Szymon Romik * Version 1.0 This is the first release of pmemkv Python bindings which guarantees backward compatibility and supports all of the pmemkv 1.0 features. API changes: - Introduced interface for direct memory access via Buffer Protocol. - Introduced dictionary interface for indirect access to data. - Introduced context manager interface. - Switched from return value based error handling to exception based. - Switched from text representation of engine's JSON configuration to dictionary. Other changes: - Added example for exposing pmemkv as REST API service. - Switched from distutils to setuptools. - Added docstring documentation. Fri Oct 04 2019 Szymon Romik * Version 0.9 This is the first official release of pmemkv-python project. It provides Python bindings to libpmemkv 1.0 with simplified API and not functionally equal to its native C/C++ counterpart. pmemkv-python-1.0/LICENSE000066400000000000000000000027431362746015700152160ustar00rootroot00000000000000Copyright 2019, Intel Corporation Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. pmemkv-python-1.0/README.md000066400000000000000000000037011362746015700154630ustar00rootroot00000000000000[![Build Status](https://travis-ci.org/pmem/pmemkv-python.svg?branch=master)](https://travis-ci.org/pmem/pmemkv-python) # pmemkv-python Python bindings for pmemkv. Currently functionally equal to pmemkv in version 1.0. Some of the new functionalities (from pmemkv 1.1) are not yet available. All known issues and limitations are logged as GitHub issues or are described in pmemkv's man pages. ## Dependencies * Python 3.6 or later * along with python3-setuptools * python3-dev(el) - header files and a static library for Python * libpmemkv-dev(el) - at least in version 1.0 - native key/value library ## Installation Start by installing [pmemkv](https://github.com/pmem/pmemkv/blob/master/INSTALLING.md) (currently at best in version **1.0.2** or **1.1**) in your system. ```sh git clone https://github.com/pmem/pmemkv-python cd pmemkv-python ``` If pmemkv is installed in default directory (e.g. /usr): ```sh sudo python3 setup.py install ``` or to rather install it locally (in '/home/user_name/.local/lib/python3.X/site-packages'): ```sh python3 setup.py install --user ``` If pmemkv is in some other directory: ```sh python3 setup.py build_ext --library-dirs= --include-dirs= python3 setup.py install --user ``` ## Testing Python bindings includes automated test cases. Use following command to run test cases: ```sh cd tests python3 -m pytest -v pmemkv_tests.py ``` ## Examples We are using `/dev/shm` to [emulate persistent memory](https://pmem.io/2016/02/22/pm-emulation.html) in examples. They can be found within this repository in [examples directory](https://github.com/pmem/pmemkv-python/tree/master/examples). To execute examples: ```bash PMEM_IS_PMEM_FORCE=1 python3 basic_example.py PMEM_IS_PMEM_FORCE=1 python3 restAPI/run_example.py ``` ## Documentation After installation, docs can be generated using sphinx (to install, run: `pip3 install sphinx`) by executing commands: ```sh cd doc make html ``` pmemkv-python-1.0/doc/000077500000000000000000000000001362746015700147505ustar00rootroot00000000000000pmemkv-python-1.0/doc/Makefile000066400000000000000000000042231362746015700164110ustar00rootroot00000000000000# Copyright 2020, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # Minimal makefile for Sphinx documentation # # You can set these variables from the command line, and also # from the environment for the first two. SPHINXOPTS ?= SPHINXBUILD ?= sphinx-build SOURCEDIR = . BUILDDIR = _build # Put it first so that "make" without argument is like "make help". help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) pmemkv-python-1.0/doc/conf.py000066400000000000000000000061711362746015700162540ustar00rootroot00000000000000# Copyright 2020, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # Generated using command: # sphinx-apidoc -F -M -e -H "pmemkv-python" -A "Intel Corporation" -V `git describe` --ext-doctest -o . ../pmemkv # with minor updates in conf.py and index.rst # -- Project information ----------------------------------------------------- project = 'pmemkv-python' copyright = '2020, Intel Corporation' author = 'Intel Corporation' # The short X.Y version version = '1.0' # The full version, including alpha/beta/rc tags release = '1.0' # -- General configuration --------------------------------------------------- master_doc = 'index' language = 'en' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.viewcode', 'sphinx.ext.todo', 'sphinx.ext.doctest', 'sphinx.ext.napoleon', ] # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # html_theme = 'alabaster' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". #html_static_path = ['_static'] pmemkv-python-1.0/doc/index.rst000066400000000000000000000005601362746015700166120ustar00rootroot00000000000000Welcome to pmemkv-python's documentation! ========================================= .. toctree:: :maxdepth: 4 :caption: Contents: pmemkv Readme file =========== Readme file can be found here_. .. _here: https://github.com/pmem/pmemkv-python/blob/master/README.md Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` pmemkv-python-1.0/doc/pmemkv.pmemkv.rst000066400000000000000000000003111362746015700202720ustar00rootroot00000000000000pmemkv.pmemkv module ==================== .. automodule:: pmemkv.pmemkv :members: :undoc-members: :show-inheritance: :private-members: :inherited-members: :special-members: __init__ pmemkv-python-1.0/doc/pmemkv.rst000066400000000000000000000003551362746015700170040ustar00rootroot00000000000000pmemkv package ============== .. automodule:: pmemkv :members: :undoc-members: :show-inheritance: :private-members: :inherited-members: :special-members: __init__ Submodules ---------- .. toctree:: pmemkv.pmemkv pmemkv-python-1.0/examples/000077500000000000000000000000001362746015700160215ustar00rootroot00000000000000pmemkv-python-1.0/examples/README.md000077500000000000000000000000661362746015700173050ustar00rootroot00000000000000This directory contains an example for pmemkv-python. pmemkv-python-1.0/examples/basic_example.py000066400000000000000000000061241362746015700211720ustar00rootroot00000000000000# Copyright 2019-2020, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. import pmemkv import json def callback(key): mem_view = memoryview(key) print(mem_view.tobytes().decode()) print("Loading config from json file") config = None with open("vsmap_conf.json") as f: config = json.load(f) print(f"Starting engine with config: {config}") db = pmemkv.Database("vsmap", config) print("Put new key") db.put("key1", "value1") assert db.count_all() == 1 print("Reading key back") assert db.get_string("key1") == "value1" print("Iterating existing keys") db.put("key2", "value2") db.put("key3", "value3") db.get_keys(lambda k: print(f"visited: {memoryview(k).tobytes().decode()}")) print("Get single value") db.get("key1", callback) print("Get single value and key in lambda expression") key = "key1" db.get( key, lambda v, k=key: print( f"key: {k} with value: " f"{memoryview(v).tobytes().decode()}" ), ) print("Get first 2 bytes of the value (without copying the entire value) and the key in a lambda expression") db.get( key, lambda v, k=key: print( f"key: {k} with first 2 bytes of the value: " f"{(memoryview(v)[0:2]).tobytes().decode()}" ), ) print("Removing existing key") db.remove("key1") assert not db.exists("key1") print("Stopping engine") db.stop() print("Configuring engine") config = {} config["path"] = "/dev/shm" config["size"] = 1073741824 print(f"Starting engine with config: {config}") db = pmemkv.Database("vsmap", config) print("Put new key") db.put("key1", "value1") print("Get single value") db.get("key1", callback) print("Stopping engine") db.stop() pmemkv-python-1.0/examples/restAPI/000077500000000000000000000000001362746015700173305ustar00rootroot00000000000000pmemkv-python-1.0/examples/restAPI/pmemkvREST.py000066400000000000000000000110661362746015700217030ustar00rootroot00000000000000#!/usr/bin/env python3 # Copyright 2020, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. """ This example shows how to use pmemkv-python binding as data store backend for simple REST API service based on falcon framework.""" import pmemkv import falcon import atexit import os import json from wsgiref import simple_server class Pmemkv(): def __init__(self): print( f"PATH: {os.environ['PMEMKV_POOL_PATH']}") try: config = {"path" : os.environ['PMEMKV_POOL_PATH']} except KeyError: print("To configure path please set PMEMKV_POOL_PATH environment variable") exit(1) print("Open datastore") try: """ Open pmemkv datastore with parameters passed by config dictionary, and concurrent hash map as storage engine. """ self.db = pmemkv.Database("cmap", config) except pmemkv.Error as e: print(f"Cannot open datastore: {e}") exit(1) def teardown(self): print("Close datastore") """ When access is no more needed, storage engine should be closed.""" self.db.stop() class ListResource: def __init__(self, storage): self.db = storage.db self.response = [] def on_get(self, req, resp): print(req) """ Pass callback function, which is run on every key in the datastore. Function have to accept key parameter, which is Buffer Object. It's possible to directly access its memory through read-only memoryview() interface, or copy to volatile memory through bytes(). To provide data to higher layer of framework, it has to be copied to object in volatile memory. It's needed as an access to the key object outside of a callback function may cause application crashes """ resp.media = [] self.db.get_keys(lambda key: resp.media.append(bytes(key).decode())) def on_put(self, req, resp): print(req) doc = json.load(req.bounded_stream) """ Save data to the datastore using dictionary interface """ for key in doc: self.db[key] = doc[key] class ElementResource: def __init__(self, storage): self.db = storage.db def on_get(self, req, resp, key): print(req) """ For use cases, where zero-copy property is not needed, access to pmemkv may be done using dictionary interface """ try: resp.media = self.db[key] except(KeyError): resp.status = falcon.HTTP_NOT_FOUND def on_delete(self, req, resp, key): print(req) """ Remove element by key """ try: self.db.remove(key) except: resp.status = falcon.HTTP_NOT_FOUND def main(): pm = Pmemkv() atexit.register(pm.teardown) app = falcon.API() app.add_route('/db', ListResource(pm)) app.add_route('/db/{key}', ElementResource(pm)) httpd = simple_server.make_server('0.0.0.0', 8000, app) print("Starting server") httpd.serve_forever() if __name__ == '__main__': main() pmemkv-python-1.0/examples/restAPI/run_example.py000066400000000000000000000074321362746015700222270ustar00rootroot00000000000000#!/usr/bin/env python3 # Copyright 2020, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. """ CI integration script for pmemkvREST example. """ import pexpect import os import sys import requests class RestApiRunner: def __init__(self, pool_path): self.server = None self.pool_path = pool_path self.script_dir = os.path.dirname(os.path.realpath(__file__)) self.python_path = sys.executable def run(self): self.server = pexpect.spawn(f"{self.python_path} pmemkvREST.py", env={"PMEMKV_POOL_PATH" : self.pool_path}, cwd=self.script_dir) self.server.expect("Starting server") def stop(self): self.server.close() def create_pool(pool_path, pool_size): cmd = f"pmempool create -s {pool_size} -l pmemkv obj {pool_path}" if not os.path.exists(pool_path): out, exit_status = pexpect.run(cmd, withexitstatus=True) if exit_status != 0: raise Exception(f"Cannot create pool: {out}") if __name__ == "__main__": default_pool_size = 1024 * 1024 *1024 default_pool_path = "/dev/shm/pmemkvREST_pool" pool_path = os.environ.get("PMEMKV_POOL_PATH") or default_pool_path pool_size = os.environ.get("POOL_SIZE") or default_pool_size create_pool(pool_path, pool_size) server = RestApiRunner(pool_path) print("Run server with cmap engine for storage") server.run() print("Put data into database") requests.put("http://localhost:8000/db", json={"message": "Hello Data"}) print("Get list of all elements") all_keys = requests.get("http://localhost:8000/db") print(f"{'': <8}{all_keys}, {all_keys.content}") print("Read data back:") response1 = requests.get("http://localhost:8000/db/message") print(f"{'': <8}{response1}, {response1.content}") print("Shutdown server") server.stop() print("Rerun server") server.run() print("Read data written in previous session:") response2 = requests.get("http://localhost:8000/db/message") print(f"{'': <8}{response2}, {response2.content}") print("Remove data") response2 = requests.delete("http://localhost:8000/db/message") print("Data was removed:") response3 = requests.get("http://localhost:8000/db/message") print(f"{'': <8}{response3}, {response3.content}") print("Shutdown server") server.stop() pmemkv-python-1.0/examples/vsmap_conf.json000066400000000000000000000000531362746015700210450ustar00rootroot00000000000000{ "path":"/dev/shm", "size":1073741824 } pmemkv-python-1.0/pmemkv/000077500000000000000000000000001362746015700155025ustar00rootroot00000000000000pmemkv-python-1.0/pmemkv/__init__.py000066400000000000000000000037201362746015700176150ustar00rootroot00000000000000# Copyright 2019-2020, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. """ Python bindings for pmemkv pmemkv is a local/embedded key-value datastore optimized for persistent memory. For more information, see https://pmem.io/pmemkv. """ from pmemkv.pmemkv import Database from _pmemkv import ( Error, UnknownError, NotSupported, InvalidArgument, ConfigParsingError, ConfigTypeError, StoppedByCallback, WrongEngineName, TransactionScopeError, ) pmemkv-python-1.0/pmemkv/kvengine.cc000066400000000000000000000517251362746015700176310ustar00rootroot00000000000000/* * Copyright 2019-2020, Intel Corporation * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * * Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * * Neither the name of the copyright holder nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #define PY_SSIZE_T_CLEAN #include #include "structmember.h" #include #include #include #include #include #ifdef __cplusplus extern "C" { #endif static PyObject *PmemkvException; typedef struct { PyObject *exception; const char *object_name; const char *exception_name; const char *docstring; } Exception; static std::unordered_map ExceptionDispatcher = { {PMEMKV_STATUS_UNKNOWN_ERROR, Exception{NULL, "UnknownError", "pmemkv_NI.UnknownError", "Something unexpected happened"}}, {PMEMKV_STATUS_NOT_FOUND, Exception{PyExc_KeyError, NULL, NULL, "Database entry or config item not found"}}, {PMEMKV_STATUS_NOT_SUPPORTED, Exception{NULL, "NotSupported", "pmemkv_NI.NotSupported", "Function is not implemented by current engine"}}, {PMEMKV_STATUS_INVALID_ARGUMENT, Exception{NULL, "InvalidArgument", "pmemkv_NI.InvalidArgument", "Argument to function has wrong value"}}, {PMEMKV_STATUS_CONFIG_PARSING_ERROR, Exception{NULL, "ConfigParsingError", "pmemkv_NI.ConfigParsingError", "Processing config failed"}}, {PMEMKV_STATUS_CONFIG_TYPE_ERROR, Exception{NULL, "ConfigTypeError", "pmemkv_NI.ConfigTypeError", "Config item has different type than expected"}}, {PMEMKV_STATUS_STOPPED_BY_CB, Exception{NULL, "StoppedByCallback", "pmemkv_NI.StoppedByCallback", "Callback function aborted in an unexpected way"}}, {PMEMKV_STATUS_OUT_OF_MEMORY, Exception{ PyExc_MemoryError, NULL, NULL, "Operation failed because there is not enough memory (or space on the device)"}}, {PMEMKV_STATUS_WRONG_ENGINE_NAME, Exception{NULL, "WrongEngineName", "pmemkv_NI.WrongEngineName", "Engine name does not match any available engine"}}, {PMEMKV_STATUS_TRANSACTION_SCOPE_ERROR, Exception{ NULL, "TransactionScopeError", "pmemkv_NI.TransactionScopeError", "An error with the scope of the libpmemobj transaction. This exception is defined for compatibility with pmemkv API and probably will never occur"}}, }; static const char *memory_exception_msg = "Cannot allocate memory for internal objects"; typedef struct { PyObject_HEAD const char *value; Py_ssize_t length; } PmemkvValueBufferObject; static PyMemberDef PmemvValueBuffer_members[] = { {"value", T_INT, offsetof(PmemkvValueBufferObject, value), 0, "Pointer to data"}, {"length", T_INT, offsetof(PmemkvValueBufferObject, length), 0, "Length of underlying data"}, {NULL}}; static int PmemkvValueBufferObject_getbuffer(PyObject *obj, Py_buffer *view, int flags) { if (view == NULL) { PyErr_SetString(PyExc_ValueError, "NULL view in getbuffer"); return -1; } PmemkvValueBufferObject *self = (PmemkvValueBufferObject *)obj; view->obj = obj; view->buf = (void *)self->value; view->len = self->length; view->readonly = 1; view->itemsize = 1; view->format = 0; // unsigned bytes view->ndim = 1; view->shape = NULL; view->strides = NULL; view->suboffsets = NULL; view->internal = NULL; Py_INCREF(self); return 0; } static PyBufferProcs PmemkvValueBuffer_as_buffer = { // this definition is only compatible with Python 3.3 and above (getbufferproc)PmemkvValueBufferObject_getbuffer, (releasebufferproc)0, // we do not require any special release function }; static PyObject *PmemkvValueBuffer_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { PmemkvValueBufferObject *self = (PmemkvValueBufferObject *)type->tp_alloc(type, 0); return (PyObject *)self; } static int PmemkvValueBuffer_init(PmemkvValueBufferObject *self) { if (self != NULL) { self->value = NULL; self->length = 0; } else { return -1; } return 0; } static void PmemkvValueBuffer_dealloc(PmemkvValueBufferObject *self) { PyObject_Del(self); } /* * Configuration of PmemkvValueBuffer object. */ static PyTypeObject PmemkvValueBufferType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) .tp_name = "pmemkv.pmemkv_NI", .tp_basicsize = sizeof(PmemkvValueBufferObject), .tp_dealloc = (destructor)PmemkvValueBuffer_dealloc, .tp_as_buffer = &PmemkvValueBuffer_as_buffer, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, .tp_doc = "Pmemkv value type", .tp_members = PmemvValueBuffer_members, .tp_init = (initproc)PmemkvValueBuffer_init, .tp_new = PmemkvValueBuffer_new, }; typedef struct { PyObject_HEAD pmemkv_db *db; } PmemkvObject; static PyMemberDef pmemkv_NI_members[] = { {"db", T_INT, offsetof(PmemkvObject, db), 0, "Engine instance"}, {NULL} }; static PyObject * Pmemkv_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { PmemkvObject *self = (PmemkvObject *) type->tp_alloc(type, 0); return (PyObject *) self; } // Turn on/off operations. static PyObject * pmemkv_NI_Start(PmemkvObject *self, PyObject* args) { Py_buffer engine, json_config; if (!PyArg_ParseTuple(args, "s*s*", &engine, &json_config)) { return NULL; } pmemkv_config *config = pmemkv_config_new(); if (config == nullptr) { // "Allocating a new pmemkv config failed" PyErr_SetString(PmemkvException, pmemkv_errormsg()); return NULL; } int rv = pmemkv_config_from_json(config, (const char*) json_config.buf); if (rv != PMEMKV_STATUS_OK) { pmemkv_config_delete(config); // "Creating a pmemkv config from JSON string failed" PyErr_SetString(ExceptionDispatcher[rv].exception, pmemkv_config_from_json_errormsg()); return NULL; } rv = pmemkv_open((const char*) engine.buf, config, &self->db); if (rv != PMEMKV_STATUS_OK) { // "pmemkv_open failed" PyErr_SetString(ExceptionDispatcher[rv].exception, pmemkv_errormsg()); return NULL; } Py_RETURN_NONE; } static PyObject * pmemkv_NI_Stop(PmemkvObject *self) { if( self->db != NULL) pmemkv_close(self->db); self->db = NULL; Py_RETURN_NONE; } static void Pmemkv_dealloc(PmemkvObject *self) { pmemkv_NI_Stop(self); Py_TYPE(self)->tp_free((PyObject *) self); } void value_callback(const char *value, size_t valuebyte, void *context) { PmemkvValueBufferObject *entry = PyObject_New(PmemkvValueBufferObject, &PmemkvValueBufferType); if (entry == NULL) { PyErr_SetString(PyExc_MemoryError, memory_exception_msg); return; } entry->value = value; entry->length = valuebyte; PyObject *args = PyTuple_New(1); if (args == NULL) { Py_DECREF(entry); PyErr_SetString(PyExc_MemoryError, memory_exception_msg); return; } // PyTuple_SetItem sets en exception on failure on its own if (PyTuple_SetItem(args, 0, (PyObject *)entry) == 0) { PyObject *res = PyObject_CallObject((PyObject *)context, args); Py_XDECREF(res); } Py_XDECREF(args); // args is the owner of the entry reference counter } int key_callback(const char *key, size_t keybytes, const char *value, size_t valuebyte, void *context) { value_callback(key, keybytes, context); if (PyErr_Occurred() != NULL) return -1; return 0; } int key_value_callback(const char *key, size_t keybytes, const char *value, size_t valuebyte, void *context) { PmemkvValueBufferObject *value_buffer = PyObject_New(PmemkvValueBufferObject, &PmemkvValueBufferType); PmemkvValueBufferObject *key_buffer = PyObject_New(PmemkvValueBufferObject, &PmemkvValueBufferType); if ((value_buffer == NULL) || (key_buffer == NULL)) { Py_XDECREF(value_buffer); Py_XDECREF(key_buffer); PyErr_SetString(PyExc_MemoryError, memory_exception_msg); return -1; } value_buffer->value = value; value_buffer->length = valuebyte; key_buffer->value = key; key_buffer->length = keybytes; PyObject *args = PyTuple_New(2); if (args == NULL) { Py_DECREF(value_buffer); Py_DECREF(key_buffer); return -1; } // PyTuple_SetItem sets an exception on failure on its own if ((PyTuple_SetItem(args, 0, (PyObject *)key_buffer) == 0) && (PyTuple_SetItem(args, 1, (PyObject *)value_buffer) == 0)) { PyObject *res = PyObject_CallObject((PyObject *)context, args); Py_XDECREF(res); } key_buffer->value = NULL; key_buffer->length = 0; value_buffer->value = NULL; value_buffer->length = 0; Py_DECREF(args); // args is the owner of the contained objects reference counters if (PyErr_Occurred() != NULL) return -1; return 0; } // "All" Methods. static PyObject * pmemkv_NI_GetKeys(PmemkvObject *self, PyObject* args) { PyObject* python_callback; if (!PyArg_ParseTuple(args, "O:set_callback", &python_callback)) { return NULL; } int result = pmemkv_get_all(self->db, key_callback, python_callback); if (PyErr_Occurred() != NULL) return NULL; if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } Py_RETURN_NONE; } static PyObject * pmemkv_NI_GetKeysAbove(PmemkvObject *self, PyObject* args) { Py_buffer key; PyObject* python_callback; if (!PyArg_ParseTuple(args, "s*O:set_callback", &key, &python_callback)) { return NULL; } int result = pmemkv_get_above(self->db, (const char *)key.buf, key.len, key_callback, python_callback); if (PyErr_Occurred() != NULL) return NULL; if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } Py_RETURN_NONE; } static PyObject * pmemkv_NI_GetKeysBelow(PmemkvObject *self, PyObject* args) { Py_buffer key; PyObject* python_callback; if (!PyArg_ParseTuple(args, "s*O:set_callback", &key, &python_callback)) { return NULL; } int result = pmemkv_get_below(self->db, (const char *)key.buf, key.len, key_callback, python_callback); if (PyErr_Occurred() != NULL) return NULL; if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } Py_RETURN_NONE; } static PyObject * pmemkv_NI_GetKeysBetween(PmemkvObject *self, PyObject* args) { Py_buffer key1, key2; PyObject* python_callback; if (!PyArg_ParseTuple(args, "s*s*O:set_callback", &key1, &key2, &python_callback)) { return NULL; } int result = pmemkv_get_between(self->db, (const char *)key1.buf, key1.len, (const char *)key2.buf, key2.len, key_callback, python_callback); if (PyErr_Occurred() != NULL) return NULL; if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } Py_RETURN_NONE; } // "Count" Methods. static PyObject * pmemkv_NI_CountAll(PmemkvObject *self) { size_t cnt; int result = pmemkv_count_all(self->db, &cnt); if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } return Py_BuildValue("i", cnt); } static PyObject * pmemkv_NI_CountAbove(PmemkvObject *self, PyObject* args) { Py_buffer key; if (!PyArg_ParseTuple(args, "s*", &key)) { return NULL; } size_t cnt; int result = pmemkv_count_above(self->db, (const char*) key.buf, key.len, &cnt); if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } return Py_BuildValue("i", cnt); } static PyObject * pmemkv_NI_CountBelow(PmemkvObject *self, PyObject* args) { Py_buffer key; if (!PyArg_ParseTuple(args, "s*", &key)) { return NULL; } size_t cnt; int result = pmemkv_count_below(self->db, (const char*) key.buf, key.len, &cnt); if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } return Py_BuildValue("i", cnt); } static PyObject * pmemkv_NI_CountBetween(PmemkvObject *self, PyObject* args) { Py_buffer key1, key2; if (!PyArg_ParseTuple(args, "s*s*", &key1, &key2)) { return NULL; } size_t cnt; int result = pmemkv_count_between(self->db, (const char*) key1.buf, key1.len, (const char*) key2.buf, key2.len, &cnt); if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } return Py_BuildValue("i", cnt); } // "Each" Methods. static PyObject * pmemkv_NI_GetAll(PmemkvObject *self, PyObject* args) { PyObject* python_callback; if (!PyArg_ParseTuple(args, "O:set_callback", &python_callback)) { return NULL; } int result = pmemkv_get_all(self->db, key_value_callback, python_callback); if (PyErr_Occurred() != NULL) return NULL; if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } Py_RETURN_NONE; } static PyObject * pmemkv_NI_GetAbove(PmemkvObject *self, PyObject* args) { Py_buffer key; PyObject* python_callback; if (!PyArg_ParseTuple(args, "s*O:set_callback", &key, &python_callback)) { return NULL; } int result = pmemkv_get_above(self->db, (const char *)key.buf, key.len, key_value_callback, python_callback); if (PyErr_Occurred() != NULL) return NULL; if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } Py_RETURN_NONE; } static PyObject * pmemkv_NI_GetBelow(PmemkvObject *self, PyObject* args) { Py_buffer key; PyObject* python_callback; if (!PyArg_ParseTuple(args, "s*O:set_callback", &key, &python_callback)) { return NULL; } int result = pmemkv_get_below(self->db, (const char *)key.buf, key.len, key_value_callback, python_callback); if (PyErr_Occurred() != NULL) return NULL; if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } Py_RETURN_NONE; } static PyObject * pmemkv_NI_GetBetween(PmemkvObject *self, PyObject* args) { Py_buffer key1, key2; PyObject* python_callback; if (!PyArg_ParseTuple(args, "s*s*O:set_callback", &key1, &key2, &python_callback)) { return NULL; } int result = pmemkv_get_between(self->db, (const char *)key1.buf, key1.len, (const char *)key2.buf, key2.len, key_value_callback, python_callback); if (PyErr_Occurred() != NULL) return NULL; if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } Py_RETURN_NONE; } // "Exists" Method. static PyObject * pmemkv_NI_Exists(PmemkvObject *self, PyObject* args) { Py_buffer key; if (!PyArg_ParseTuple(args, "s*", &key)) { return NULL; } int result = pmemkv_exists(self->db, (const char*) key.buf, key.len); if (result != PMEMKV_STATUS_OK && result != PMEMKV_STATUS_NOT_FOUND) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } return PyBool_FromLong(result == PMEMKV_STATUS_OK); } // "CRUD" Operations. static PyObject * pmemkv_NI_Put(PmemkvObject *self, PyObject* args) { Py_buffer key, value; if (!PyArg_ParseTuple(args, "s*s*", &key, &value)) { return NULL; } int result = pmemkv_put(self->db, (const char*) key.buf, key.len, (const char*) value.buf, value.len); if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } Py_RETURN_NONE; } static PyObject *pmemkv_NI_GetString(PmemkvObject *self, PyObject *args) { Py_buffer key; if (!PyArg_ParseTuple(args, "s*", &key)) { return NULL; } struct GetCallbackContext { int status; std::string value; }; GetCallbackContext cxt = {PMEMKV_STATUS_NOT_FOUND, ""}; auto callback = [](const char* v, size_t vb, void* context) { const auto c = ((GetCallbackContext*) context); c->status = PMEMKV_STATUS_OK; c->value.append(v, vb); }; int result = pmemkv_get(self->db, (const char*) key.buf, key.len, callback, &cxt); if (PyErr_Occurred() != NULL) return NULL; if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } else if (cxt.status == PMEMKV_STATUS_OK) { return Py_BuildValue("s#", cxt.value.data(), cxt.value.size()); } Py_RETURN_NONE; } static PyObject *pmemkv_NI_Get(PmemkvObject *self, PyObject *args) { Py_buffer key; PyObject *python_callback; if (!PyArg_ParseTuple(args, "s*O:set_callback", &key, &python_callback)) { return NULL; } int result = pmemkv_get(self->db, (const char *)key.buf, key.len, value_callback, python_callback); if (PyErr_Occurred() != NULL) return NULL; if (result != PMEMKV_STATUS_OK) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } Py_RETURN_NONE; } static PyObject * pmemkv_NI_Remove(PmemkvObject *self, PyObject* args) { Py_buffer key; if (!PyArg_ParseTuple(args, "s*", &key)) { return NULL; } int result = pmemkv_remove(self->db, (const char*) key.buf, key.len); if (result != PMEMKV_STATUS_OK && result != PMEMKV_STATUS_NOT_FOUND) { PyErr_SetString(ExceptionDispatcher[result].exception, pmemkv_errormsg()); return NULL; } return PyBool_FromLong(result == PMEMKV_STATUS_OK); } // Functions declarations. static PyMethodDef pmemkv_NI_methods[] = { {"start", (PyCFunction)pmemkv_NI_Start, METH_VARARGS, NULL}, {"stop", (PyCFunction)pmemkv_NI_Stop, METH_NOARGS, NULL}, {"put", (PyCFunction)pmemkv_NI_Put, METH_VARARGS, NULL}, {"get_string", (PyCFunction)pmemkv_NI_GetString, METH_VARARGS, NULL}, {"get", (PyCFunction)pmemkv_NI_Get, METH_VARARGS, NULL}, {"get_keys", (PyCFunction)pmemkv_NI_GetKeys, METH_VARARGS, NULL}, {"get_keys_above", (PyCFunction)pmemkv_NI_GetKeysAbove, METH_VARARGS, NULL}, {"get_keys_below", (PyCFunction)pmemkv_NI_GetKeysBelow, METH_VARARGS, NULL}, {"get_keys_between", (PyCFunction)pmemkv_NI_GetKeysBetween, METH_VARARGS, NULL}, {"count_all", (PyCFunction)pmemkv_NI_CountAll, METH_NOARGS, NULL}, {"count_above", (PyCFunction)pmemkv_NI_CountAbove, METH_VARARGS, NULL}, {"count_below", (PyCFunction)pmemkv_NI_CountBelow, METH_VARARGS, NULL}, {"count_between", (PyCFunction)pmemkv_NI_CountBetween, METH_VARARGS, NULL}, {"get_all", (PyCFunction)pmemkv_NI_GetAll, METH_VARARGS, NULL}, {"get_above", (PyCFunction)pmemkv_NI_GetAbove, METH_VARARGS, NULL}, {"get_below", (PyCFunction)pmemkv_NI_GetBelow, METH_VARARGS, NULL}, {"get_between", (PyCFunction)pmemkv_NI_GetBetween, METH_VARARGS, NULL}, {"exists", (PyCFunction)pmemkv_NI_Exists, METH_VARARGS, NULL}, {"remove", (PyCFunction)pmemkv_NI_Remove, METH_VARARGS, NULL}, {NULL, NULL, 0, NULL}}; /* * Configuration of pmemkv_NI object. */ static PyTypeObject PmemkvType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "pmemkv.pmemkv_NI", .tp_basicsize = sizeof(PmemkvObject), .tp_dealloc = (destructor)Pmemkv_dealloc, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, .tp_doc = "Pmemkv binding", .tp_methods = pmemkv_NI_methods, .tp_members = pmemkv_NI_members, .tp_new = Pmemkv_new, }; // Module definition. static struct PyModuleDef pmemkv_NI_module = { PyModuleDef_HEAD_INIT, "_pmemkv", /* name of the module */ NULL, /* module documentation, may be NULL */ -1, /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */ }; // Creating dynamic module. PyMODINIT_FUNC PyInit__pmemkv(void) { PyObject *m; if (PyType_Ready(&PmemkvType) < 0) return NULL; m = PyModule_Create(&pmemkv_NI_module); if (m == NULL) return NULL; try { Py_INCREF(&PmemkvType); if (PyModule_AddObject(m, "pmemkv_NI", (PyObject *)&PmemkvType) < 0) { throw; } PmemkvException = PyErr_NewException("pmemkv_NI.PmemkvException", NULL, NULL); if (PyModule_AddObject(m, "Error", PmemkvException) < 0) { throw; } for (auto &e : ExceptionDispatcher) { if (e.second.exception == NULL) { e.second.exception = PyErr_NewExceptionWithDoc( e.second.exception_name, e.second.docstring, PmemkvException, NULL); if (PyModule_AddObject(m, e.second.object_name, e.second.exception) < 0) { throw; } } } } catch (...) { Py_XDECREF(&PmemkvType); Py_XDECREF(m); Py_XDECREF(PmemkvException); Py_XDECREF(m); for (auto &e : ExceptionDispatcher) { Py_XDECREF(e.second.exception); Py_XDECREF(m); } return NULL; } return m; } #ifdef __cplusplus } #endif pmemkv-python-1.0/pmemkv/pmemkv.py000066400000000000000000000331621362746015700173600ustar00rootroot00000000000000# Copyright 2019-2020, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. """ Python bindings for pmemkv. """ import _pmemkv import json class Database(): """ Main Python pmemkv class, it provides functions to operate on data in database. This class can be used dict-like, i.a. accessing and assigning data using '[]'. If an error/exception is thrown from any method it will contain pmemkv's status and error message. Currently returned statuses are described in libpmemkv manpage: https://pmem.io/pmemkv/master/manpages/libpmemkv.3.html#errors Possible exceptions to be thrown in Python binding are as follows: - Error, - UnknownError, - NotSupported, - InvalidArgument, - ConfigParsingError, - ConfigTypeError, - StoppedByCallback, - WrongEngineName, - TransactionScopeError. """ def __init__(self, engine, config): """ Parameters ---------- engine : str Name of the engine to work with. config : dict Dictionary with parameters specified for the engine. Required configuration parameters are dependent on particular engine. For more information on engine configuration please look into pmemkv man pages. """ if not isinstance(config, dict): raise TypeError("Config should be dictionary") self.config = json.dumps(config) self.db = _pmemkv.pmemkv_NI() self.db.start(engine, self.config) def __setitem__(self, key, value): self.put(key,value) def __getitem__(self, key): if key not in self: raise KeyError(key) return self.db.get_string(key) def __len__(self): return self.count_all() def __contains__(self, key): return self.exists(key) def __delitem__(self, key): if key not in self: raise KeyError(key) self.remove(key) def __enter__(self): return self def __exit__(self, exception_type, exception_value, traceback): self.stop() def stop(self): """ Stops the running engine. """ self.db.stop() def put(self, key, value): """ Inserts the key/value pair into the pmemkv datastore. This method accepts Unicode objects as well as bytes-like objects. Unicode objects are stored using 'utf-8' encoding. Parameters ---------- key : str or byte-like object record's key; record will be put into database under its name. value : str or byte-like object data to be inserted into this new datastore record. """ self.db.put(key, value) def get_keys(self, func): """ Executes callback function for every key stored in the pmemkv datastore. Parameters ---------- func : function (may be lambda) Function to be called for each key. Key passed to func is read-only buffer and may be accessed by memoryview function. Callback function should accept one positional argument, which is key. For more information please look into Buffer Protocol documentation. """ self.db.get_keys(func) def get_keys_above(self, key, func): """ Executes callback function for every key stored in the pmemkv datastore, whose keys are greater than the given key. Parameters ---------- key : str or byte-like object Sets the lower bound for querying. func : function (may be lambda) Function to be called for each key above one specified in key parameter. Key passed to func is read-only buffer and may be accessed by memoryview function. Callback function should accept one positional argument, which is key. For more information please look into Buffer Protocol documentation. """ self.db.get_keys_above(key, func) def get_keys_below(self, key, func): """ Executes callback function for every key stored in the pmemkv datastore, whose keys are lower than the given key. Parameters ---------- key : str or byte-like object Sets the upper bound for querying. func : function (may be lambda) Function to be called for each key below one specified in key parameter. Key passed to func is read-only buffer and may be accessed by memoryview function. Callback function should accept one positional argument, which is key. For more information please look into Buffer Protocol documentation. """ self.db.get_keys_below(key, func) def get_keys_between(self, key1, key2, func): """ Executes callback function for every key stored in pmemkv datastore, whose keys are greater than the key1 and less than the key2. Parameters ---------- key1 : str or byte-like object Sets the lower bound for querying. key2 : str Sets the upper bound for querying. func : function (may be lambda) Function to be called for each key between key1 and key2. Key passed to func is read-only buffer and may be accessed by memoryview function. Callback function should accept one positional argument, which is key. For more information please look into Buffer Protocol documentation. """ self.db.get_keys_between(key1, key2, func) def count_all(self): """ Returns number of currently stored key/value pairs in the pmemkv datastore. Returns ------- number : int Total number of elements in the datastore. """ return self.db.count_all() def count_above(self, key): """ Returns number of currently stored key/value pairs in the pmemkv datastore, whose keys are greater than the given key. Parameters ---------- key : str Sets the lower bound for querying. Returns ------- number: int Number of key/value pairs in the datastore, whose keys are greater than the given key. """ return self.db.count_above(key) def count_below(self, key): """ Returns number of currently stored key/value pairs in the pmemkv datastore, whose keys are less than the given key. Parameters ---------- key : str Sets the upper bound for querying. Returns ------- number : int Number of key/value pairs in the datastore, whose keys are lower than the given key. """ return self.db.count_below(key) def count_between(self, key1, key2): """ Returns number of currently stored key/value pairs in the pmemkv datastore, whose keys are greater than the key1 and less than the key2. Parameters ---------- key1 : str Sets the lower bound for querying. key2 : str Sets the upper bound for querying. Returns ------- number : int Number of key/value pairs in the datastore, between given keys. """ return self.db.count_between(key1, key2) def get_all(self, func): """ Executes callback function for every key/value pair stored in the pmemkv datastore. Parameters ---------- func : function (may be lambda) Function to be called for each key/value pair in the datastore. Key and value passed to func are read-only buffers and may be accessed by memoryview function. Callback function should accept two positional arguments, which are key and value. For more information please look into Buffer Protocol documentation. """ self.db.get_all(func) def get_above(self, key, func): """ Executes callback function for every key/value pair stored in the pmemkv datastore, whose keys are greater than the given key. Parameters ---------- key : str Sets the lower bound for querying. func : function (may be lambda) Function to be called for each specified key/value pair. Key and value passed to func are read-only buffers and may be accessed by memoryview function. Callback function should accept two positional arguments, which are key and value. For more information please look into Buffer Protocol documentation. """ self.db.get_above(key, func) def get_below(self, key, func): """ Executes callback function for every key/value pair stored in the pmemkv datastore, whose keys are lower than the given key. Parameters ---------- key : str Sets the upper bound for querying. func : function (may be lambda) Function to be called for each specified key/value pair. Key and value passed to func are read-only buffers and may be accessed by memoryview function. Callback function should accept two positional arguments, which are key and value. For more information please look into Buffer Protocol documentation. """ self.db.get_below(key, func) def get_between(self, key1, key2, func): """ Executes callback function for every key/value pair stored in the pmemkv datastore, whose keys are greater than the key1 and less than the key2. Parameters ---------- key1 : str Sets the lower bound for querying. key2 : str Sets the upper bound for querying. func : function (may be lambda) Function to be called for each specified key/value pair. Key and value passed to func are read-only buffers and may be accessed by memoryview function. Callback function should accept two positional arguments, which are key and value. For more information please look into Buffer Protocol documentation. """ self.db.get_between(key1, key2, func) def exists(self, key): """ Verifies the presence key/value pair in the pmemkv datastore. Parameters ---------- key : str key to query for. Returns ------- exists : bool true if element with given key exists in the datastore, false if not. """ return self.db.exists(key) def get(self, key, func): """ Executes callback function for value for given key. Parameters ---------- key : str key to query for. func : function (may be lambda) Function to be called for specified key/value pair. Value passed to func is read-only buffer and may be accessed by memoryview function. Callback function should accept one positional argument, which is value. Please notice, key is not passed to callback function. For more information please look into Buffer Protocol documentation. """ self.db.get(key, func) def get_string(self, key): """ Gets copy (as a string) of value for given key. Value returned by get_string() is still accessible after removal of element from datastore. Parameters ---------- key : str key to query for. Returns ------- value : str or byte-like object Copy of value associated with the given key. """ return self.db.get_string(key) def remove(self, key): """ Removes key/value pair from the pmemkv datastore for given key. Parameters ---------- key : str Record's key to query for, to be removed. Returns ------- removed : bool true if element was removed, false if element didn't exist before removal. """ return self.db.remove(key) pmemkv-python-1.0/setup.py000066400000000000000000000053761362746015700157300ustar00rootroot00000000000000""" * Copyright 2019-2020, Intel Corporation * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * * Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * * Neither the name of the copyright holder nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. """ import setuptools from os import path project_dir = path.abspath(path.dirname(__file__)) with open(path.join(project_dir, "README.md"), encoding="utf-8") as f: readme = f.read() link_modules = setuptools.Extension( "_pmemkv", ["pmemkv/kvengine.cc"], libraries=["pmemkv", "pmemkv_json_config"] ) setuptools.setup( name="pmemkv", version="1.0", description="Python bindings for PMEMKV library", long_description=readme, long_description_content_type="text/markdown", url="https://pmem.io/pmemkv-python", author="Pmemkv development team", packages=setuptools.find_packages(), classifiers=[ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: BSD License", "Operating System :: POSIX :: Linux", "Programming Language :: Python :: 3", "Topic :: Software Development :: Libraries", ], python_requires=">=3.6", project_urls={ "Source Code": "https://github.com/pmem/pmemkv-python", "Bug Reports": "https://github.com/pmem/pmemkv-python/issues", "Pmemkv": "https://github.com/pmem/pmemkv", }, ext_modules=[link_modules], ) pmemkv-python-1.0/tests/000077500000000000000000000000001362746015700153455ustar00rootroot00000000000000pmemkv-python-1.0/tests/nontrivial_data_tests.py000066400000000000000000000052141362746015700223210ustar00rootroot00000000000000''' * Copyright 2019-2020, Intel Corporation * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * * Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * * Neither the name of the copyright holder nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ''' import unittest import json import urllib.request import os.path import pmemkv class TestNaughtyStrings(unittest.TestCase): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.engine = r"vsmap" self.config = {"path":"/dev/shm","size":1073741824} self.strings = self._get_naughty_strings("https://raw.githubusercontent.com/minimaxir/big-list-of-naughty-strings/master/blns.json") def _get_naughty_strings(self, url): file_name = "blns.json" data = {} if not os.path.exists(file_name): urllib.request.urlretrieve(url, file_name) with open(file_name) as f: data = json.load(f) return data def test_puts_tricky_keys_and_values(self): db = pmemkv.Database(self.engine, self.config) data = {} with open("blns.json") as f: data = json.load(f) for val in data: db.put(val, val) for val in data: self.assertEqual(db[val], val) db.stop() if __name__ == '__main__': unittest.main() pmemkv-python-1.0/tests/pmemkv_tests.py000066400000000000000000000515611362746015700204500ustar00rootroot00000000000000''' * Copyright 2019-2020, Intel Corporation * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * * Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * * Neither the name of the copyright holder nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ''' import unittest from pmemkv import Database import pmemkv class TestKVEngine(unittest.TestCase): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.engine = r"vsmap" self.config = {"path":"/dev/shm", "size":1073741824} self.key_and_value = r"" self.formatter = r"{}," def all_and_each(self, key = b'', value = b''): value_copy = bytes(value) key_copy = bytes(key) value_text_representation = value_copy.decode('utf-8') key_text_representation = key_copy.decode('utf-8') self.key_and_value += self.formatter.format(key_text_representation, value_text_representation) def all_and_each_strings(self, key = b'', value = b''): if value != b'': self.key_and_value += self.formatter.format(key.decode(), value.decode()) else: self.key += self.formatter.format(key.decode()) def test_uses_module_to_publish_type(self): import pmemkv self.assertEqual(Database.__class__, pmemkv.Database.__class__) def test_blackhole_engine(self): db = Database(r"blackhole", self.config) self.assertEqual(db.count_all(), 0) self.assertFalse(db.exists(r"key1")) with self.assertRaises(KeyError): db.get_string(r"key1") db.put(r"key1", r"value123") self.assertEqual(db.count_all(), 0) self.assertFalse(db.exists(r"key1")) with self.assertRaises(KeyError): db.get_string(r"key1") self.assertTrue(db.remove(r"key1")) self.assertFalse(db.exists(r"key1")) with self.assertRaises(KeyError): db.get_string(r"key1") db.stop() def test_stop_engine_multiple_times(self): """ In case of failure, this test cause segmentation fault. As there is no way to catch segmentation fault in python, just do not assert anything. """ db = Database(self.engine, self.config) db.stop() db.stop() db.stop() def test_gets_missing_key(self): db = Database(self.engine, self.config) self.assertFalse(db.exists(r"key1")) with self.assertRaises(KeyError): db.get_string(r"key1") db.stop() def test_puts_basic_values(self): db = Database(self.engine, self.config) self.assertFalse(db.exists(r"key1")) db.put(r"key1", r"value1") self.assertTrue(db.exists(r"key1")) self.assertEqual(db.get_string(r"key1"), r"value1") db.stop() def test_puts_binary_keys(self): db = Database(self.engine, self.config) db.put("A\0B\0\0C", r"value1") self.assertTrue(db.exists("A\0B\0\0C")) self.assertEqual(db.get_string("A\0B\0\0C"), r"value1") db.stop() def test_puts_binary_values(self): db = Database(self.engine, self.config) db.put(r"key1", "A\0B\0\0C") self.assertEqual(db.get_string(r"key1"), "A\0B\0\0C") db.stop() def test_puts_complex_value(self): db = Database(self.engine, self.config) val = r"one\ttwo or

three

\n {four} and ^five" db.put(r"key1", val) self.assertEqual(db.get_string(r"key1"), val) db.stop() def test_puts_empty_key(self): db = Database(self.engine, self.config) db.put(r"", r"empty") db.put(r" ", r"single-space") db.put(r"\t\t", r"two-tab") self.assertTrue(db.exists(r"")) self.assertEqual(db.get_string(r""), r"empty") self.assertTrue(db.exists(r" ")) self.assertEqual(db.get_string(r" "), r"single-space") self.assertTrue(db.exists(r"\t\t")) self.assertEqual(db.get_string(r"\t\t"), r"two-tab") db.stop() def test_puts_empty_values(self): db = Database(self.engine, self.config) db.put(r"empty", r"") db.put(r"single-space", r" ") db.put(r"two-tab", r"\t\t") self.assertEqual(db.get_string(r"empty"), r"") self.assertEqual(db.get_string(r"single-space"), r" ") self.assertEqual(db.get_string(r"two-tab"), r"\t\t") db.stop() def test_puts_multiple_values(self): db = Database(self.engine, self.config) db.put(r"key1", r"value1") db.put(r"key2", r"value2") db.put(r"key3", r"value3") self.assertTrue(db.exists(r"key1")) self.assertEqual(db.get_string(r"key1"), r"value1") self.assertTrue(db.exists(r"key2")) self.assertEqual(db.get_string(r"key2"), r"value2") self.assertTrue(db.exists(r"key3")) self.assertEqual(db.get_string(r"key3"), r"value3") db.stop() def test_puts_overwriting_existing_value(self): db = Database(self.engine, self.config) db.put(r"key1", r"value1") self.assertEqual(db.get_string(r"key1"), r"value1") db.put(r"key1", r"value123") self.assertEqual(db.get_string(r"key1"), r"value123") db.put(r"key1", r"asdf") self.assertEqual(db.get_string(r"key1"), r"asdf") db.stop() def test_puts_utf8_key(self): db = Database(self.engine, self.config) val = r"to remember, note, record" db.put(r"记", val) self.assertTrue(db.exists(r"记")) self.assertEqual(db.get_string(r"记"), val) db.stop() def test_puts_utf8_value(self): db = Database(self.engine, self.config) val = r"记 means to remember, note, record" db.put(r"key1", val) self.assertEqual(db.get_string(r"key1"), val) db.stop() def test_removes_key_and_value(self): db = Database(self.engine, self.config) db.put(r"key1", r"value1") self.assertTrue(db.exists(r"key1")) self.assertEqual(db.get_string(r"key1"), r"value1") self.assertTrue(db.remove(r"key1")) self.assertFalse(db.remove(r"key1")) self.assertFalse(db.exists(r"key1")) with self.assertRaises(KeyError): db.get_string(r"key1") db.stop() def test_exceptions_hierarchy(self): exceptions = [pmemkv.Error, pmemkv.UnknownError, pmemkv.NotSupported, pmemkv.InvalidArgument, pmemkv.ConfigParsingError, pmemkv.ConfigTypeError, pmemkv.StoppedByCallback, pmemkv.WrongEngineName, pmemkv.TransactionScopeError] with self.assertRaises(Exception): raise(pmemkv.Error) for ex in exceptions: with self.assertRaises(pmemkv.Error): raise(ex) def test_throws_exception_on_start_when_config_is_empty(self): db = None with self.assertRaises(pmemkv.Error): db = Database(self.engine, {}) """ InvalidArgument is for consistency with pmemkv interface reference in pmemkv test: basic_tests/PmemkvCApiTest.NullConfig """ with self.assertRaises(pmemkv.InvalidArgument): db = Database(self.engine, {}) def test_exception_on_start_when_config_is_wrong_type(self): db = None with self.assertRaises(TypeError): db = Database(self.engine, "{}") self.assertEqual(db, None) def test_throws_exception_on_start_when_engine_is_invalid(self): db = None with self.assertRaises(pmemkv.Error): db = Database(r"nope.nope", self.config) with self.assertRaises(pmemkv.WrongEngineName): db = Database(r"nope.nope", self.config) self.assertEqual(db, None) def test_throws_exception_on_start_when_path_is_invalid(self): db = None with self.assertRaises(pmemkv.Error): db = Database(self.engine, {"path":"/tmp/123/234/345/456/567/678/nope.nope", "size": 1073741824}) """ This part need to be commented out due to pmemkv issue https://github.com/pmem/pmemkv/issues/565 with self.assertRaises(pmemkv.InvalidArgument): db = Database(self.engine, {"path":"/tmp/123/234/345/456/567/678/nope.nope", "size": 1073741824}) """ self.assertEqual(db, None) def test_throws_exception_on_start_when_path_is_wrong_type(self): db = None with self.assertRaises(pmemkv.Error): db = Database(self.engine, {"path":1234, "size": 1073741824}) with self.assertRaises(pmemkv.ConfigTypeError): db = Database(self.engine, {"path":1234, "size": 1073741824}) self.assertEqual(db, None) def test_uses_get_keys(self): db = Database(self.engine, self.config) db.put(r"1", r"one") db.put(r"2", r"two") self.formatter = r"<{}>," self.key = r"" db.get_keys(self.all_and_each) self.assertEqual(self.key_and_value, r"<1>,<2>,") db.stop() def test_uses_get_keys_above(self): db = Database(self.engine, self.config) db.put(r"A", r"1") db.put(r"AB", r"2") db.put(r"AC", r"3") db.put(r"B", r"4") db.put(r"BB", r"5") db.put(r"BC", r"6") self.formatter = r"{}," self.key = r"" db.get_keys_above(r"B", self.all_and_each) self.assertEqual(self.key_and_value, r"BB,BC,") db.stop() def test_uses_get_keys_below(self): db = Database(self.engine, self.config) db.put(r"A", r"1") db.put(r"AB", r"2") db.put(r"AC", r"3") db.put(r"B", r"4") db.put(r"BB", r"5") db.put(r"BC", r"6") self.formatter = r"{}," self.key_and_value = r"" db.get_keys_below(r"B", self.all_and_each) self.assertEqual(self.key_and_value, r"A,AB,AC,") db.stop() def test_uses_get_keys_between(self): db = Database(self.engine, self.config) db.put(r"A", r"1") db.put(r"AB", r"2") db.put(r"AC", r"3") db.put(r"B", r"4") db.put(r"BB", r"5") db.put(r"BC", r"6") self.formatter = r"{}," self.key_and_value = r"" db.get_keys_between(r"A", r"B", self.all_and_each) self.assertEqual(self.key_and_value, r"AB,AC,") self.key_and_value = r"" db.get_keys_between(r"", r"", self.all_and_each) db.get_keys_between(r"A", r"A", self.all_and_each) db.get_keys_between(r"B", r"A", self.all_and_each) self.assertEqual(self.key_and_value, r"") db.stop() def test_uses_count_all(self): db = Database(self.engine, self.config) db.put(r"A", r"1") db.put(r"AB", r"2") db.put(r"AC", r"3") db.put(r"B", r"4") db.put(r"BB", r"5") db.put(r"BC", r"6") db.put(r"BD", r"7") self.assertEqual(db.count_all(), 7) self.assertEqual(db.count_above(r""), 7) self.assertEqual(db.count_above(r"A"), 6) self.assertEqual(db.count_above(r"B"), 3) self.assertEqual(db.count_above(r"BC"), 1) self.assertEqual(db.count_above(r"BD"), 0) self.assertEqual(db.count_above(r"Z"), 0) self.assertEqual(db.count_below(r""), 0) self.assertEqual(db.count_below(r"A"), 0) self.assertEqual(db.count_below(r"B"), 3) self.assertEqual(db.count_below(r"BD"), 6) self.assertEqual(db.count_below(r"ZZZZZ"), 7) self.assertEqual(db.count_between(r"", r"ZZZZZ"), 7) self.assertEqual(db.count_between(r"", r"A"), 0) self.assertEqual(db.count_between(r"", r"B"), 3) self.assertEqual(db.count_between(r"A", r"B"), 2) self.assertEqual(db.count_between(r"B", r"ZZZZZ"), 3) self.assertEqual(db.count_between(r"", r""), 0) self.assertEqual(db.count_between(r"A", r"A"), 0) self.assertEqual(db.count_between(r"AC", r"A"), 0) self.assertEqual(db.count_between(r"B", r"A"), 0) self.assertEqual(db.count_between(r"BD", r"A"), 0) self.assertEqual(db.count_between(r"ZZZ", r"B"), 0) db.stop() def test_uses_get_all(self): db = Database(self.engine, self.config) db.put(r"1", r"one") db.put(r"2", r"two") self.formatter = r"<{}>,<{}>|" self.key_and_value = r"" db.get_all(self.all_and_each) self.assertEqual(self.key_and_value, r"<1>,|<2>,|") db.stop() def test_uses_get_above(self): db = Database(self.engine, self.config) db.put(r"A", "1") db.put(r"AB", r"2") db.put(r"AC", r"3") db.put(r"B", r"4") db.put(r"BB", r"5") db.put(r"BC", r"6") self.formatter = r"{},{}|" self.key_and_value = r"" db.get_above(r"B", self.all_and_each) self.assertEqual(self.key_and_value, r"BB,5|BC,6|") db.stop() def test_uses_get_below(self): db = Database(self.engine, self.config) db.put(r"A", r"1") db.put(r"AB", r"2") db.put(r"AC", r"3") db.put(r"B", r"4") db.put(r"BB", r"5") db.put(r"BC", r"6") self.formatter = r"{},{}|" self.key_and_value = r"" db.get_below(r"AC", self.all_and_each) self.assertEqual(self.key_and_value, r"A,1|AB,2|") db.put(r"记!", r"RR") db.stop() def test_each_between(self): db = Database(self.engine, self.config) db.put(r"A", r"1") db.put(r"AB", r"2") db.put(r"AC", r"3") db.put(r"B", r"4") db.put(r"BB", r"5") db.put(r"BC", r"6") self.formatter = r"{},{}|" self.key_and_value = r"" db.get_between(r"A", r"B", self.all_and_each) self.assertEqual(self.key_and_value, r"AB,2|AC,3|") db.stop() def test_dict_set_item(self): db = Database(self.engine, self.config) db['string_value'] = "test" self.assertEqual(db['string_value'], "test") db.stop() def test_dict_get_item(self): db = Database(self.engine, self.config) key = "dict_test" db[key] = "123" temp = db[key] self.assertEqual(temp, "123") db.stop() def test_get_copy_to_class_member(self): class Callback: def __init__(self): self.result = None def __call__(self, key): self.result = bytes(key) callback = Callback() db = Database(self.engine, self.config) key = "dict_test" val = "123" db[key] = val db.get(key, callback) db.remove(key) db[key] = "Some other string" self.assertEqual(callback.result.decode(), val) db.stop() def test_get_assert_in_callback(self): def callback (key): self.assertEqual(memoryview(key).tobytes(), "123".encode('utf-8')) key = "dict_test" val = "123" db = Database(self.engine, self.config) db[key] = val db.get(key, callback) db.stop() def test_get_exception_in_callback(self): class LocalException(Exception): pass def callback(key): raise LocalException('TestException') db = Database(self.engine, self.config) key = "dict_test" val = "123" db[key] = val try: db.get(key, callback) except LocalException as e: db.stop() self.assertEqual(type(e).__name__ , "LocalException") db.stop() def test_get_AttributeError_in_callback(self): def callback (key): self.assertEqual(key.NonexistentMethod(), "123".encode('utf-8')) key = "dict_test" val = "123" db = Database(self.engine, self.config) db[key] = val try: db.get(key, callback) except Exception as e: assert type(e).__name__ == "AttributeError" db.stop() def test_get_out_of_bound_access_in_callback(self): key = "dict_test" val = "123" db = Database(self.engine, self.config) db[key] = val with self.assertRaises(IndexError): db.get(key, lambda v: memoryview(v).tobytes()[4]) with self.assertRaises(IndexError): db.get(key, lambda v: memoryview(v)[4]) db.stop() def test_get_lambda_in_callback(self): key = "dict_test" val = "123" db = Database(self.engine, self.config) db[key] = val db.get(key, lambda v, k=key: self.assertEqual(memoryview(v).tobytes(), "123".encode('utf-8'))) db.get(key, lambda v, k=key: self.assertEqual(k, "dict_test")) db.stop() def test_get_with_value_indexing(self): key = "dict_test" val = "123" db = Database(self.engine, self.config) db[key] = val db.get(key, lambda v, k=key: self.assertEqual((memoryview(v)[0:2]) .tobytes(), "12".encode('utf-8'))) db.stop() def test_dict_len(self): db = Database(self.engine, self.config) db['dict_test'] = "123" db['Aa'] = "42" self.assertEqual(len(db), 2) db.stop() def test_dict_item_in(self): db = Database(self.engine, self.config) db['dict_test'] = "123" self.assertIn('dict_test', db) self.assertNotIn('Aa', db) db.stop() def test_dict_item_del(self): db = Database(self.engine, self.config) db['dict_test'] = "123" del db['dict_test'] with self.assertRaises(KeyError): temp = db['dict_test'] db.stop() def test_databases_interference(self): db1 = Database(self.engine, self.config) db2 = Database(self.engine, self.config) db1['1'] = "A" db2['2'] = "B" with self.assertRaises(KeyError): temp = db2['1'] db1.stop() db2.stop() def test_get_same_element_two_times(self): db = Database(self.engine, self.config) db['dict_test'] = "123" val1 = db['dict_test'] val2 = db['dict_test'] self.assertEqual(val1, val2) db.stop() def test_delete_same_element_two_times(self): db = Database(self.engine, self.config) db['dict_test'] = "123" del db['dict_test'] with self.assertRaises(KeyError): del db['dict_test'] db.stop() def test_call_del_inside_callback(self): def callback(val): del(val) # check if buffer protocol object was properly removed with self.assertRaises(UnboundLocalError): del(val) key = "dict_test" val = "123" db = Database(self.engine, self.config) db[key] = val db.get(key, callback) # check if key is accessable self.assertEqual(db[key], val) db.stop() def test_context_manager(self): key = "dict_test" val = "123" with Database(self.engine, self.config) as db: db[key] = val self.assertEqual(db[key], val) def test_throws_exception_in_context_manager(self): class TestException(Exception): pass def callback(val): raise TestException() key = "dict_test" val = "123" with Database(self.engine, self.config) as db: db[key] = val with self.assertRaises(TestException): db.get(key, callback) with self.assertRaises(TestException): with Database(self.engine, self.config) as db: db[key] = val db.get(key, callback) if __name__ == '__main__': unittest.main() pmemkv-python-1.0/utils/000077500000000000000000000000001362746015700153435ustar00rootroot00000000000000pmemkv-python-1.0/utils/docker/000077500000000000000000000000001362746015700166125ustar00rootroot00000000000000pmemkv-python-1.0/utils/docker/build.sh000077500000000000000000000077131362746015700202600ustar00rootroot00000000000000#!/usr/bin/env bash # # Copyright 2017-2019, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # build.sh - runs a Docker container from a Docker image with environment # prepared for running pmemkv-python build and tests. # # # Notes: # - run this script from its location or set the variable 'HOST_WORKDIR' to # where the root of this project is on the host machine, # - set variables 'OS' and 'OS_VER' properly to a system you want to build this # repo on (for proper values take a look on the list of Dockerfiles at the # utils/docker/images directory), eg. OS=ubuntu, OS_VER=16.04. # set -e if [[ -z "$OS" || -z "$OS_VER" ]]; then echo "ERROR: The variables OS and OS_VER have to be set " \ "(eg. OS=fedora, OS_VER=30)." exit 1 fi if [[ -z "$HOST_WORKDIR" ]]; then HOST_WORKDIR=$(readlink -f ../..) fi chmod -R a+w $HOST_WORKDIR if [[ "$TRAVIS_EVENT_TYPE" == "cron" || "$TRAVIS_BRANCH" == "coverity_scan" ]]; then if [[ "$TYPE" != "coverity" ]]; then echo "Skipping non-Coverity job for cron/Coverity build" exit 0 fi else if [[ "$TYPE" == "coverity" ]]; then echo "Skipping Coverity job for non cron/Coverity build" exit 0 fi fi imageName=${DOCKERHUB_REPO}:${OS}-${OS_VER} containerName=pmemkv-python-${OS}-${OS_VER} pmemkv_version=$(echo $TYPE | cut -d'-' -f 2-) if [[ "$command" == "" ]]; then command="./run-build.sh $pmemkv_version"; fi if [ "$COVERAGE" == "1" ]; then docker_opts="${docker_opts} `bash <(curl -s https://codecov.io/env)`"; ci_env=`bash <(curl -s https://codecov.io/env)` fi if [ -n "$DNS_SERVER" ]; then DNS_SETTING=" --dns=$DNS_SERVER "; fi WORKDIR=/pmemkv-python SCRIPTSDIR=$WORKDIR/utils/docker echo Building ${OS}-${OS_VER} # Run a container with # - environment variables set (--env) # - host directory containing source mounted (-v) # - working directory set (-w) docker run --network="bridge" --name=$containerName -ti \ $DNS_SETTING \ ${docker_opts} \ $ci_env \ --env http_proxy=$http_proxy \ --env https_proxy=$https_proxy \ --env GITHUB_TOKEN=$GITHUB_TOKEN \ --env WORKDIR=$WORKDIR \ --env SCRIPTSDIR=$SCRIPTSDIR \ --env COVERAGE=$COVERAGE \ --env TRAVIS_REPO_SLUG=$TRAVIS_REPO_SLUG \ --env TRAVIS_BRANCH=$TRAVIS_BRANCH \ --env TRAVIS_EVENT_TYPE=$TRAVIS_EVENT_TYPE \ --env COVERITY_SCAN_TOKEN=$COVERITY_SCAN_TOKEN \ --env COVERITY_SCAN_NOTIFICATION_EMAIL=$COVERITY_SCAN_NOTIFICATION_EMAIL \ --env TEST_BUILD=$TEST_BUILD \ --shm-size=4G \ -v $HOST_WORKDIR:$WORKDIR \ -v /etc/localtime:/etc/localtime \ -w $SCRIPTSDIR \ $imageName $command pmemkv-python-1.0/utils/docker/images/000077500000000000000000000000001362746015700200575ustar00rootroot00000000000000pmemkv-python-1.0/utils/docker/images/Dockerfile.fedora-31000077500000000000000000000055111362746015700235360ustar00rootroot00000000000000# # Copyright 2016-2020, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # Dockerfile - a 'recipe' for Docker to build an image of fedora-based # environment prepared for running pmemkv-python build and tests. # # Pull base image FROM fedora:31 MAINTAINER szymon.romik@intel.com # Set required environment variables ENV OS fedora ENV OS_VER 31 ENV PACKAGE_MANAGER rpm ENV NOTTY 1 # Install basic tools RUN dnf update -y \ && dnf install -y \ autoconf \ automake \ clang \ cmake \ daxctl-devel \ gcc \ gcc-c++ \ gdb \ git \ hub \ libtool \ make \ man \ memkind-devel \ ndctl-devel \ numactl-devel \ pandoc \ python3-devel \ python3-sphinx \ rapidjson-devel \ rpm-build \ sudo \ tbb-devel \ unzip \ wget \ which \ python3-pip \ && dnf clean all RUN pip3 install pytest setuptools wheel #Install examples dependencies RUN pip3 install falcon pexpect requests # Install glibc-debuginfo RUN dnf debuginfo-install -y glibc # Install pmdk COPY install-pmdk.sh install-pmdk.sh RUN ./install-pmdk.sh rpm # Install pmdk c++ bindings COPY install-libpmemobj-cpp.sh install-libpmemobj-cpp.sh RUN ./install-libpmemobj-cpp.sh RPM # Prepare pmemkv COPY prepare-pmemkv.sh prepare-pmemkv.sh RUN ./prepare-pmemkv.sh RPM # Add user ENV USER user ENV USERPASS pass RUN useradd -m $USER RUN echo "$USER:$USERPASS" | chpasswd RUN gpasswd wheel -a $USER USER $USER pmemkv-python-1.0/utils/docker/images/Dockerfile.ubuntu-19.10000077500000000000000000000056771362746015700240620ustar00rootroot00000000000000# # Copyright 2016-2020, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # Dockerfile - a 'recipe' for Docker to build an image of ubuntu-based # environment prepared for running pmemkv-python build and tests. # # Pull base image FROM ubuntu:19.10 MAINTAINER szymon.romik@intel.com # Set required environment variables ENV OS ubuntu ENV OS_VER 19.10 ENV PACKAGE_MANAGER deb ENV NOTTY 1 # Update the Apt cache and install basic tools RUN apt-get update RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ autoconf \ automake \ build-essential \ clang \ cmake \ curl \ debhelper \ devscripts \ fakeroot \ git \ libc6-dbg \ libdaxctl-dev \ libmemkind-dev \ libndctl-dev \ libnode-dev \ libnuma-dev \ libtbb-dev \ libtext-diff-perl \ libtool \ libunwind8-dev \ numactl \ pandoc \ pkg-config \ python3-dev \ python3-distutils \ python3-sphinx \ rapidjson-dev \ sudo \ wget \ whois \ python3-pip \ && rm -rf /var/lib/apt/lists/* RUN pip3 install pytest setuptools wheel #Install examples dependencies RUN pip3 install falcon pexpect requests # Install pmdk COPY install-pmdk.sh install-pmdk.sh RUN ./install-pmdk.sh dpkg # Install pmdk c++ bindings COPY install-libpmemobj-cpp.sh install-libpmemobj-cpp.sh RUN ./install-libpmemobj-cpp.sh DEB # Prepare pmemkv COPY prepare-pmemkv.sh prepare-pmemkv.sh RUN ./prepare-pmemkv.sh DEB # Add user ENV USER user ENV USERPASS pass RUN useradd -m $USER -g sudo -p `mkpasswd $USERPASS` USER $USER pmemkv-python-1.0/utils/docker/images/build-image.sh000077500000000000000000000050021362746015700225720ustar00rootroot00000000000000#!/usr/bin/env bash # # Copyright 2016-2019, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # build-image.sh - prepares a Docker image with -based # environment for testing pmemkv-python, # according to the Dockerfile. file # located in the same directory. # # The script can be run locally. # set -e function usage { echo "Usage:" echo " build-image.sh " echo "where , for example, can be 'fedora-30', provided " \ "a Dockerfile named 'Dockerfile.fedora-30' exists in the " \ "current directory." } # Check if the first and second argument is nonempty if [[ -z "$1" || -z "$2" ]]; then usage exit 1 fi # Check if the file Dockerfile.OS-VER exists if [[ ! -f "Dockerfile.$2" ]]; then echo "ERROR: wrong argument." usage exit 1 fi # Build a Docker image tagged with ${DOCKERHUB_REPO}:OS-VER docker build -t $1:$2 \ --build-arg http_proxy=$http_proxy \ --build-arg https_proxy=$https_proxy \ -f Dockerfile.$2 . pmemkv-python-1.0/utils/docker/images/install-libpmemobj-cpp.sh000077500000000000000000000045111362746015700247630ustar00rootroot00000000000000#!/usr/bin/env bash # # Copyright 2019, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # install-libpmemobj-cpp.sh # - installs PMDK C++ bindings (libpmemobj-cpp) # set -e PREFIX=/usr PACKAGE_TYPE=$1 # master: Merge pull request #652 from ldorau/Create-separate-Docker-images-for-v1.10-branch LIBPMEMOBJ_CPP_VERSION="006137044243981f4760ae220101a43cf97bcf2d" git clone https://github.com/pmem/libpmemobj-cpp --shallow-since=2019-10-02 cd libpmemobj-cpp git checkout $LIBPMEMOBJ_CPP_VERSION mkdir build cd build cmake .. -DCPACK_GENERATOR="$PACKAGE_TYPE" -DCMAKE_INSTALL_PREFIX=$PREFIX if [ "$PACKAGE_TYPE" = "" ]; then make -j$(nproc) install else make -j$(nproc) package if [ "$PACKAGE_TYPE" = "DEB" ]; then sudo dpkg -i libpmemobj++*.deb elif [ "$PACKAGE_TYPE" = "RPM" ]; then sudo rpm -i libpmemobj++*.rpm fi fi cd ../.. rm -r libpmemobj-cpp pmemkv-python-1.0/utils/docker/images/install-pmdk.sh000077500000000000000000000046241362746015700230230ustar00rootroot00000000000000#!/usr/bin/env bash # # Copyright 2019, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # install-pmdk.sh - installs PMDK # set -e PREFIX=/usr PACKAGE_TYPE=$1 # stable-1.8: common: fix build on GitHub Actions PMDK_VERSION="eeb8fd86c6c72f4607a22a1e9d9f7da31be8af2a" git clone https://github.com/pmem/pmdk --shallow-since=2019-09-26 cd pmdk git checkout $PMDK_VERSION if [ "$PACKAGE_TYPE" = "" ]; then make -j$(nproc) install prefix=$PREFIX else make -j$(nproc) BUILD_PACKAGE_CHECK=n $PACKAGE_TYPE if [ "$PACKAGE_TYPE" = "dpkg" ]; then sudo dpkg -i dpkg/libpmem_*.deb dpkg/libpmem-dev_*.deb sudo dpkg -i dpkg/libpmemobj_*.deb dpkg/libpmemobj-dev_*.deb sudo dpkg -i dpkg/libpmemblk_*.deb dpkg/libpmemlog_*.deb dpkg/libpmempool_*.deb dpkg/pmempool_*.deb elif [ "$PACKAGE_TYPE" = "rpm" ]; then sudo rpm -i rpm/*/pmdk-debuginfo-*.rpm sudo rpm -i rpm/*/libpmem*-*.rpm sudo rpm -i rpm/*/pmempool-*.rpm fi fi cd .. rm -r pmdk pmemkv-python-1.0/utils/docker/images/prepare-pmemkv.sh000077500000000000000000000052331362746015700233540ustar00rootroot00000000000000#!/usr/bin/env bash # # Copyright 2019, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # prepare-pmemkv.sh - prepare pmemkv packages # set -e PREFIX=/usr PACKAGE_TYPE=$1 # master: Merge pull request #602 from ldorau/Fix-setting-compilers--do-not-mix-gcc-with-clang current_pmemkv_version="5aff9f197e2e3eaae7f419c04ab9e2fc98309eed" # stable-1.0: Merge pull request #528 from ldorau/Do-not-add-pmemkv_config...; 14.11.2019 stable_1_pmemkv_version="a3735b5393f0d5411ef8a2468b36d2a1ed00c0a1" # stable-1.1: Version 1.1 stable_1_1_pmemkv_version="2f719305afb0f44103734851cfe825e1b1d73dbf" prepare_pmemkv () { pmemkv_version="$1" version_name="$2" git checkout "$pmemkv_version" mkdir build cd build cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCPACK_GENERATOR=$PACKAGE_TYPE \ -DBUILD_TESTS=OFF make -j$(nproc) package cd .. mkdir /opt/"$version_name" mv build/* /opt/"$version_name" rm -rf build } git clone https://github.com/pmem/pmemkv cd pmemkv prepare_pmemkv "$current_pmemkv_version" "pmemkv-master" prepare_pmemkv "$stable_1_pmemkv_version" "pmemkv-stable-1.0" prepare_pmemkv "$stable_1_pmemkv_version" "pmemkv-stable-1.1" cd .. rm -r pmemkv pmemkv-python-1.0/utils/docker/images/push-image.sh000077500000000000000000000050361362746015700224610ustar00rootroot00000000000000#!/usr/bin/env bash # # Copyright 2016-2019, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # push-image.sh - pushes the Docker image tagged with OS-VER # to the Docker Hub. # # The script utilizes $DOCKERHUB_USER and $DOCKERHUB_PASSWORD variables to log in to # Docker Hub. The variables can be set in the Travis project's configuration # for automated builds. # set -e function usage { echo "Usage:" echo " push-image.sh " echo "where , for example, can be 'fedora-30', provided " \ "a Docker image tagged with ${DOCKERHUB_REPO}:fedora-30 exists " \ "locally." } # Check if the first argument is nonempty if [[ -z "$1" ]]; then usage exit 1 fi # Check if the image tagged with ${DOCKERHUB_REPO}:OS-VER exists locally if [[ ! $(docker images -a | awk -v pattern="^${DOCKERHUB_REPO}:$1\$" \ '$1":"$2 ~ pattern') ]] then echo "ERROR: wrong argument." usage exit 1 fi # Log in to the Docker Hub docker login -u="$DOCKERHUB_USER" -p="$DOCKERHUB_PASSWORD" # Push the image to the repository docker push ${DOCKERHUB_REPO}:$1 pmemkv-python-1.0/utils/docker/pull-or-rebuild-image.sh000077500000000000000000000132541362746015700232540ustar00rootroot00000000000000#!/usr/bin/env bash # # Copyright 2016-2019, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # pull-or-rebuild-image.sh - rebuilds the Docker image used in the # current Travis build if necessary. # # The script rebuilds the Docker image if the Dockerfile for the current # OS version (Dockerfile.${OS}-${OS_VER}) or any .sh script from the directory # with Dockerfiles were modified and committed. # # If the Travis build is not of the "pull_request" type (i.e. in case of # merge after pull_request) and it succeed, the Docker image should be pushed # to the Docker Hub repository. An empty file is created to signal that to # further scripts. # # If the Docker image does not have to be rebuilt, it will be pulled from # Docker Hub. # set -e if [[ "$TRAVIS_EVENT_TYPE" != "cron" && "$TRAVIS_BRANCH" != "coverity_scan" \ && "$TYPE" == "coverity" ]]; then echo "INFO: Skip Coverity scan job if build is triggered neither by " \ "'cron' nor by a push to 'coverity_scan' branch" exit 0 fi if [[ ( "$TRAVIS_EVENT_TYPE" == "cron" || "$TRAVIS_BRANCH" == "coverity_scan" )\ && "$TYPE" != "coverity" ]]; then echo "INFO: Skip regular jobs if build is triggered either by 'cron'" \ " or by a push to 'coverity_scan' branch" exit 0 fi if [[ -z "$OS" || -z "$OS_VER" ]]; then echo "ERROR: The variables OS and OS_VER have to be set properly " \ "(eg. OS=ubuntu, OS_VER=16.04)." exit 1 fi if [[ -z "$HOST_WORKDIR" ]]; then echo "ERROR: The variable HOST_WORKDIR has to contain a path to " \ "the root of this project on the host machine" exit 1 fi # TRAVIS_COMMIT_RANGE is usually invalid for force pushes - fix it when used # with a non-upstream repository if [ -n "$TRAVIS_COMMIT_RANGE" -a "$TRAVIS_REPO_SLUG" != "${GITHUB_REPO}" ]; then if ! git rev-list $TRAVIS_COMMIT_RANGE; then # get commit id of the last merge LAST_MERGE=$(git log --merges --pretty=%H -1) if [ "$LAST_MERGE" == "" ]; then # possible in case of shallow clones TRAVIS_COMMIT_RANGE="" else TRAVIS_COMMIT_RANGE="$LAST_MERGE..HEAD" # make sure it works now if ! git rev-list $TRAVIS_COMMIT_RANGE; then TRAVIS_COMMIT_RANGE="" fi fi fi fi # Find all the commits for the current build if [[ -n "$TRAVIS_COMMIT_RANGE" ]]; then # $TRAVIS_COMMIT_RANGE contains "..." instead of ".." # https://github.com/travis-ci/travis-ci/issues/4596 PR_COMMIT_RANGE="${TRAVIS_COMMIT_RANGE/.../..}" commits=$(git rev-list $PR_COMMIT_RANGE) else commits=$TRAVIS_COMMIT fi echo "Commits in the commit range:" for commit in $commits; do echo $commit; done # Get the list of files modified by the commits files=$(for commit in $commits; do git diff-tree --no-commit-id --name-only \ -r $commit; done | sort -u) echo "Files modified within the commit range:" for file in $files; do echo $file; done # Path to directory with Dockerfiles and image building scripts images_dir_name=images base_dir=utils/docker/$images_dir_name # Check if committed file modifications require the Docker image to be rebuilt for file in $files; do # Check if modified files are relevant to the current build if [[ $file =~ ^($base_dir)\/Dockerfile\.($OS)-($OS_VER)$ ]] \ || [[ $file =~ ^($base_dir)\/.*\.sh$ ]] then # Rebuild Docker image for the current OS version echo "Rebuilding the Docker image for the Dockerfile.$OS-$OS_VER" pushd $images_dir_name ./build-image.sh ${DOCKERHUB_REPO} ${OS}-${OS_VER} popd # Check if the image has to be pushed to Docker Hub # (i.e. the build is triggered by commits to the ${GITHUB_REPO} # repository's master branch, and the Travis build is not # of the "pull_request" type). In that case, create the empty # file. if [[ $TRAVIS_REPO_SLUG == "${GITHUB_REPO}" \ && $TRAVIS_BRANCH == "master" \ && $TRAVIS_EVENT_TYPE != "pull_request" && $PUSH_IMAGE == "1" ]] then echo "The image will be pushed to Docker Hub" touch push_image_to_repo_flag else echo "Skip pushing the image to Docker Hub" fi if [[ $PUSH_IMAGE == "1" ]] then echo "Skip build package check if image has to be pushed" touch skip_build_package_check fi exit 0 fi done # Getting here means rebuilding the Docker image is not required. # Pull the image from Docker Hub. docker pull ${DOCKERHUB_REPO}:${OS}-${OS_VER} pmemkv-python-1.0/utils/docker/run-build.sh000077500000000000000000000057571362746015700210700ustar00rootroot00000000000000#!/usr/bin/env bash # # Copyright 2019-2020, Intel Corporation # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # * Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # run-build.sh - checks binding's building and installation # with pmemkv installed in given version # set -e # build and install pmemkv pmemkv_version=$1 cd /opt/pmemkv-$pmemkv_version/ if [ "${PACKAGE_MANAGER}" = "deb" ]; then echo $USERPASS | sudo -S dpkg -i libpmemkv*.deb elif [ "${PACKAGE_MANAGER}" = "rpm" ]; then echo $USERPASS | sudo -S rpm -i libpmemkv*.rpm else echo "PACKAGE_MANAGER env variable not set or set improperly ('deb' or 'rpm' supported)." exit 1 fi echo echo "##########################################################" echo "### Verifying building, installation and tests execution " echo "##########################################################" cd $WORKDIR python3 setup.py bdist_wheel pip3 install dist/pmemkv-*.whl --user cd $WORKDIR/tests python3 -X faulthandler -m pytest -v pmemkv_tests.py python3 -X faulthandler -m pytest -v nontrivial_data_tests.py echo echo "##########################################################" echo "### Checking examples" echo "##########################################################" cd $WORKDIR/examples PMEM_IS_PMEM_FORCE=1 python3 basic_example.py REST_API_EXAMPLE_DIR="${WORKDIR}/examples/restAPI" python3 ${REST_API_EXAMPLE_DIR}/run_example.py echo echo "########################################################" echo "### Generating doc" echo "########################################################" cd $WORKDIR/doc make html