pmdk-1.13.1/0000775000000000000000000000000014435627653011247 5ustar rootrootpmdk-1.13.1/.gitignore0000664000000000000000000000040314435627501013224 0ustar rootroot.* !.github !.gitignore !.gitattributes !.cirrus.yml !.clang-format !.travis.yml !.mailmap !.cstyleignore !.codecov.yml *~ *.swp *.o make.out core a.out nbproject/ /rpmbuild/ /dpkgbuild/ /rpm/ /dpkg/ /user.mk *.user ~* *.db *.htmp *.hpptmp *.aps tags *.link pmdk-1.13.1/CONTRIBUTING.md0000664000000000000000000001451714435627501013500 0ustar rootroot# Contributing to the Persistent Memory Development Kit Down below you'll find instructions on how to contribute to the Persistent Memory Development Kit. Your contributions are most welcome! You'll find it is best to begin with a conversation about your changes, rather than just writing a bunch of code and contributing it out of the blue. There are several good ways to suggest new features, offer to add a feature, or just begin a dialog about the Persistent Memory Development Kit: * Open an issue in our [GitHub Issues Database](https://github.com/pmem/pmdk/issues) * Suggest a feature, ask a question, start a discussion, etc. in our [pmem Google group](https://groups.google.com/group/pmem) * Chat with members of the PMDK team real-time on the **#pmem** IRC channel on [OFTC](https://www.oftc.net) **NOTE: If you do decide to implement code changes and contribute them, please make sure you agree your contribution can be made available under the [BSD-style License used for the Persistent Memory Development Kit](https://github.com/pmem/pmdk/blob/master/LICENSE).** **NOTE: Submitting your changes also means that you certify the following:** ``` Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. ``` In case of any doubt, the gatekeeper may ask you to certify the above in writing, i.e. via email or by including a `Signed-off-by:` line at the bottom of your commit comments. To improve tracking of who is the author of the contribution, we kindly ask you to use your real name (not an alias) when committing your changes to the Persistent Memory Development Kit: ``` Author: Random J Developer ``` ### Code Contributions Please feel free to use the forums mentioned above to ask for comments & questions on your code before submitting a pull request. The Persistent Memory Development Kit project uses the common *fork and merge* workflow used by most GitHub-hosted projects. The [Git Workflow blog article](https://pmem.io/2014/09/09/git-workflow.html) describes our workflow in more detail. #### Linux/FreeBSD Before contributing please remember to run: ``` $ make cstyle ``` This will check all C/C++ files in the tree for style issues. To check C++ files you have to have clang-format version 9.0, otherwise they will be skipped. If you want to run this target automatically at build time, you can pass CSTYLEON=1 to make. If you want cstyle to be run, but not fail the build, pass CSTYLEON=2 to make. There is also a target for automatic C++ code formatting, to do this run: ``` $ make format ``` There are cases, when you might have several clang-format-X.Y binaries and either no clang-format or it pointing to an older version. In such case run: ``` $ make CLANG_FORMAT=/path/to/clang-format cstyle|format ``` #### Windows On Windows to check the code for style issues, please run: ``` $ pmdk\utils\CSTYLE.ps1 ``` To check or format C++ files, you may use a standalone Visual Studio plugin for clang-format. The plugin installer can be downloaded from [LLVM Builds](https://llvm.org/builds) page. If you are actively working on an PMDK feature, please let other developers know by [creating an issue](https://github.com/pmem/pmdk/issues). Use the template `Feature` and assign it to yourself (due to the way GitHub permissions work, you may have to ask a team member to assign it to you). ### Bug Reports Bugs for the PMDK project are tracked in our [GitHub Issues Database](https://github.com/pmem/pmdk/issues). When reporting a new bug, please use `New issue` button, pick proper template and fill in all fields. Provide as much information as possible, including the product version: #### PMDK version Put the release name of the version of PMDK running when the bug was discovered in a bug comment. If you saw this bug in multiple PMDK versions, please put at least the most recent version and list the others if necessary. - Stable release names are in the form `#.#` (where `#` represents an integer); for example `0.3`. - Release names from working versions look like `#.#+b#` (adding a build #) or `#.#-rc#` (adding a release candidate number) If PMDK was built from source, the version number can be retrieved from git using this command: `git describe` For binary PMDK releases, use the entire package name. For RPMs, use `rpm -q pmdk` to display the name. For Deb packages, run `dpkg-query -W pmdk` and use the second (version) string. #### Priority Requested priority describes the urgency to resolve a defect and establishes the time frame for providing a verified resolution. Priorities are defined as: * **P1**: Showstopper bug, requiring a resolution before the next release of the library. * **P2**: High-priority bug, requiring a resolution although it may be decided that the bug does not prevent the next release of the library. * **P3**: Medium-priority bug. The expectation is that the bug will be evaluated and a plan will be made for when the bug will be resolved. * **P4**: Low-priority bug, the least urgent. Fixed when the resources are available. ### Other issues On our issues page we also gather feature requests and questions. Templates to use are `Feature` and `Question`, respectively. They should help deliver a meaningful description of a feature or ask a question to us (remember though we have different means of communication, as described at the top of the page). pmdk-1.13.1/.gitattributes0000664000000000000000000000017114435627501014131 0ustar rootroot* text=auto eol=lf *.jpg binary *.png binary *.gif binary *.ico binary *.match text -whitespace GIT_VERSION export-subst pmdk-1.13.1/.travis.yml0000664000000000000000000000166614435627501013361 0ustar rootrootos: linux dist: bionic arch: - ppc64le - arm64-graviton2 language: c services: - docker env: global: - GITHUB_REPO=pmem/pmdk - DOCKER_REPO=ghcr.io/pmem/pmdk - OS=ubuntu - OS_VER=22.04 - MAKE_PKG=0 - PMDK_CC=gcc - PMDK_CXX=g++ - VALGRIND=1 - SRC_CHECKERS=0 - EXPERIMENTAL=n jobs: - FAULT_INJECTION=1 TEST_BUILD=debug PUSH_IMAGE=1 - OS=fedora OS_VER=31 PMDK_CC=clang PMDK_CXX=clang++ TEST_BUILD=nondebug PUSH_IMAGE=1 - MAKE_PKG=1 VALGRIND=0 - MAKE_PKG=1 VALGRIND=0 OS=fedora OS_VER=31 - COVERAGE=1 FAULT_INJECTION=1 TEST_BUILD=debug jobs: allow_failures: - arch: ppc64le - arch: arm64-graviton2 before_install: - echo $TRAVIS_COMMIT_RANGE - export HOST_WORKDIR=`pwd` - cd utils/docker - ./pull-or-rebuild-image.sh script: - ./build-CI.sh after_success: - source ./set-vars.sh - if [[ -f $CI_FILE_PUSH_IMAGE_TO_REPO ]]; then ./images/push-image.sh; fi pmdk-1.13.1/.codecov.yml0000664000000000000000000000053014435627501013460 0ustar rootrootignore: - '.*windows.*' - '.*test.*' - '.*valgrind.*' - '.*benchmarks.*' comment: layout: "diff" behavior: default require_changes: yes coverage: status: project: default: threshold: 1% parsers: gcov: branch_detection: conditional: false loop: false method: false macro: false pmdk-1.13.1/Makefile0000664000000000000000000000602414435627501012701 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2023, Intel Corporation # # Makefile -- top-level Makefile for PMDK # # Use "make" to build the library. # # Use "make doc" to build documentation. # # Use "make test" to build unit tests. # # Use "make check" to run unit tests. # # Use "make clean" to delete all intermediate files (*.o, etc). # # Use "make clobber" to delete everything re-buildable (binaries, etc.). # # Use "make gitclean" for a complete tree clean, save for test configs. # # Use "make cstyle" to run cstyle on all C source files # # Use "make check-license" to check copyright and license in all source files # # Use "make rpm" to build rpm packages # # Use "make dpkg" to build dpkg packages # # Use "make source DESTDIR=path_to_dir" to copy source files # from HEAD to 'path_to_dir/pmdk' directory. # # As root, use "make install" to install the library in the usual # locations (/usr/local/lib, /usr/local/include, and /usr/local/share/man). # You can provide custom directory prefix for installation using # DESTDIR variable e.g.: "make install DESTDIR=/opt" # You can override the prefix within DESTDIR using prefix variable # e.g.: "make install prefix=/usr" include src/common.inc RPM_BUILDDIR=rpmbuild DPKG_BUILDDIR=dpkgbuild EXPERIMENTAL ?= n BUILD_PACKAGE_CHECK ?= y TEST_CONFIG_FILE ?= "$(CURDIR)"/src/test/testconfig.sh DOC ?= y rpm : override DESTDIR="$(CURDIR)/$(RPM_BUILDDIR)" dpkg: override DESTDIR="$(CURDIR)/$(DPKG_BUILDDIR)" rpm dpkg: override prefix=/usr all: doc $(MAKE) -C src $@ doc: ifeq ($(DOC),y) test -f .skip-doc || $(MAKE) -C doc all endif clean: $(MAKE) -C src $@ ifeq ($(DOC),y) test -f .skip-doc || $(MAKE) -C doc $@ endif $(RM) -r $(RPM_BUILDDIR) $(DPKG_BUILDDIR) $(RM) -f $(GIT_VERSION) clobber: $(MAKE) -C src $@ ifeq ($(DOC),y) test -f .skip-doc || $(MAKE) -C doc $@ endif $(RM) -r $(RPM_BUILDDIR) $(DPKG_BUILDDIR) rpm dpkg $(RM) -f $(GIT_VERSION) test check pcheck pycheck: all $(MAKE) -C src $@ check pcheck pycheck: check-doc cstyle: test -d .git && utils/check-commits.sh $(MAKE) -C src $@ $(MAKE) -C utils $@ @echo Checking files for whitespace issues... @utils/check_whitespace -g @echo Done. format: $(MAKE) -C src $@ @echo Done. check-license: @utils/check_license/check-headers.sh $(TOP) BSD-3-Clause @echo Done. check-doc: doc ./utils/check-manpages sparse: $(MAKE) -C src sparse gitclean: git clean -dfx -etestconfig.sh -etestconfig.py source: clobber $(if "$(DESTDIR)", , $(error Please provide DESTDIR variable)) +utils/copy-source.sh "$(DESTDIR)" $(SRCVERSION) pkg-clean: $(RM) -r "$(DESTDIR)" rpm dpkg: pkg-clean $(MAKE) source DESTDIR="$(DESTDIR)" +utils/build-$@.sh -t $(SRCVERSION) -s "$(DESTDIR)"/pmdk -w "$(DESTDIR)" -o $(CURDIR)/$@\ -e $(EXPERIMENTAL) -c $(BUILD_PACKAGE_CHECK)\ -f $(TEST_CONFIG_FILE) -n $(NDCTL_ENABLE) install: all install uninstall: $(MAKE) -C src $@ ifeq ($(DOC),y) $(MAKE) -C doc $@ endif .PHONY: all clean clobber test check cstyle check-license install uninstall\ source rpm dpkg pkg-clean pcheck format doc\ $(SUBDIRS) pmdk-1.13.1/utils/0000775000000000000000000000000014435627501012377 5ustar rootrootpmdk-1.13.1/utils/.gitignore0000664000000000000000000000000614435627501014363 0ustar rootroot*.zip pmdk-1.13.1/utils/pmdk.spec.in0000664000000000000000000004763214435627501014627 0ustar rootroot # rpmbuild options: # --with | --without ndctl # --define _testconfig # --define _skip_check 1 # do not terminate build if files in the $RPM_BUILD_ROOT # directory are not found in the %files %define _unpackaged_files_terminate_build 0 # disable 'make check' on suse %if %{defined suse_version} %define _skip_check 1 %define dist .suse%{suse_version} %endif %bcond_without ndctl %define min_ndctl_ver __NDCTL_MIN_VER__ Name: pmdk Version: __VERSION__ Release: 1%{?dist} Summary: __PACKAGE_SUMMARY__ Packager: __PACKAGE_MAINTAINER__ Group: __GROUP_SYS_LIBS__ License: __LICENSE__ URL: https://pmem.io/pmdk Source0: %{name}-%{version}.tar.gz BuildRequires: gcc BuildRequires: make BuildRequires: glibc-devel BuildRequires: autoconf BuildRequires: automake BuildRequires: man BuildRequires: pkgconfig BuildRequires: gdb # fdupes package is available only on 'openSUSE Tumbleweed' and 'openSUSE Leap 15.1' %if (0%{?suse_version} > 1500) || (0%{?sles_version} >= 150100 && 0%{?is_opensuse}) BuildRequires: fdupes %endif %if %{with ndctl} %if %{defined suse_version} BuildRequires: libndctl-devel >= %{min_ndctl_ver} %else BuildRequires: ndctl-devel >= %{min_ndctl_ver} BuildRequires: daxctl-devel >= %{min_ndctl_ver} %endif %endif # Debug variants of the libraries should be filtered out of the provides. %global __provides_exclude_from ^%{_libdir}/pmdk_debug/.*\\.so.*$ # By design, PMDK does not support any 32-bit architecture. # Due to dependency on xmmintrin.h and some inline assembly, it can be # compiled only for x86_64 at the moment. # Other 64-bit architectures could also be supported, if only there is # a request for that, and if somebody provides the arch-specific # implementation of the low-level routines for flushing to persistent # memory. # https://bugzilla.redhat.com/show_bug.cgi?id=1340634 # https://bugzilla.redhat.com/show_bug.cgi?id=1340635 # https://bugzilla.redhat.com/show_bug.cgi?id=1340636 # https://bugzilla.redhat.com/show_bug.cgi?id=1340637 ExclusiveArch: x86_64 ppc64le %description The Persistent Memory Development Kit is a collection of libraries for using memory-mapped persistence, optimized specifically for persistent memory. %package -n libpmem2__PKG_NAME_SUFFIX__ Summary: Low-level persistent memory support library Group: __GROUP_SYS_LIBS__ %description -n libpmem2__PKG_NAME_SUFFIX__ The libpmem2 provides low level persistent memory support. In particular, support for the persistent memory instructions for flushing changes to pmem is provided. %files -n libpmem2__PKG_NAME_SUFFIX__ %defattr(-,root,root,-) %dir %{_datadir}/pmdk %{_libdir}/libpmem2.so.* %{_datadir}/pmdk/pmdk.magic %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmem2-devel Summary: Development files for the low-level persistent memory library Group: __GROUP_DEV_LIBS__ Requires: libpmem2__PKG_NAME_SUFFIX__ = %{version}-%{release} %description -n libpmem2-devel The libpmem2 provides low level persistent memory support. In particular, support for the persistent memory instructions for flushing changes to pmem is provided. This library is provided for software which tracks every store to pmem and needs to flush those changes to durability. Most developers will find higher level libraries like libpmemobj to be much more convenient. %files -n libpmem2-devel %defattr(-,root,root,-) %{_libdir}/libpmem2.so %{_libdir}/pkgconfig/libpmem2.pc %{_includedir}/libpmem2.h %{_includedir}/libpmem2/*.h %{_mandir}/man7/libpmem2.7.gz %{_mandir}/man3/pmem2_*.3.gz %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmem2-debug Summary: Debug variant of the low-level persistent memory library Group: __GROUP_DEV_LIBS__ Requires: libpmem2__PKG_NAME_SUFFIX__ = %{version}-%{release} %description -n libpmem2-debug The libpmem provides low level persistent memory support. In particular, support for the persistent memory instructions for flushing changes to pmem is provided. This sub-package contains debug variant of the library, providing run-time assertions and trace points. The typical way to access the debug version is to set the environment variable LD_LIBRARY_PATH to /usr/lib64/pmdk_debug. %files -n libpmem2-debug %defattr(-,root,root,-) %dir %{_libdir}/pmdk_debug %{_libdir}/pmdk_debug/libpmem2.so %{_libdir}/pmdk_debug/libpmem2.so.* %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmem__PKG_NAME_SUFFIX__ Summary: Low-level persistent memory support library Group: __GROUP_SYS_LIBS__ %description -n libpmem__PKG_NAME_SUFFIX__ The libpmem provides low level persistent memory support. In particular, support for the persistent memory instructions for flushing changes to pmem is provided. %files -n libpmem__PKG_NAME_SUFFIX__ %defattr(-,root,root,-) %dir %{_datadir}/pmdk %{_libdir}/libpmem.so.* %{_datadir}/pmdk/pmdk.magic %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmem-devel Summary: Development files for the low-level persistent memory library Group: __GROUP_DEV_LIBS__ Requires: libpmem__PKG_NAME_SUFFIX__ = %{version}-%{release} %description -n libpmem-devel The libpmem provides low level persistent memory support. In particular, support for the persistent memory instructions for flushing changes to pmem is provided. This library is provided for software which tracks every store to pmem and needs to flush those changes to durability. Most developers will find higher level libraries like libpmemobj to be much more convenient. %files -n libpmem-devel %defattr(-,root,root,-) %{_libdir}/libpmem.so %{_libdir}/pkgconfig/libpmem.pc %{_includedir}/libpmem.h %{_mandir}/man7/libpmem.7.gz %{_mandir}/man3/pmem_*.3.gz %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmem-debug Summary: Debug variant of the low-level persistent memory library Group: __GROUP_DEV_LIBS__ Requires: libpmem__PKG_NAME_SUFFIX__ = %{version}-%{release} %description -n libpmem-debug The libpmem provides low level persistent memory support. In particular, support for the persistent memory instructions for flushing changes to pmem is provided. This sub-package contains debug variant of the library, providing run-time assertions and trace points. The typical way to access the debug version is to set the environment variable LD_LIBRARY_PATH to /usr/lib64/pmdk_debug. %files -n libpmem-debug %defattr(-,root,root,-) %dir %{_libdir}/pmdk_debug %{_libdir}/pmdk_debug/libpmem.so %{_libdir}/pmdk_debug/libpmem.so.* %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmemblk__PKG_NAME_SUFFIX__ Summary: Persistent Memory Resident Array of Blocks library Group: __GROUP_SYS_LIBS__ Requires: libpmem__PKG_NAME_SUFFIX__ >= %{version}-%{release} %description -n libpmemblk__PKG_NAME_SUFFIX__ The libpmemblk implements a pmem-resident array of blocks, all the same size, where a block is updated atomically with respect to power failure or program interruption (no torn blocks). %files -n libpmemblk__PKG_NAME_SUFFIX__ %defattr(-,root,root,-) %{_libdir}/libpmemblk.so.* %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmemblk-devel Summary: Development files for the Persistent Memory Resident Array of Blocks library Group: __GROUP_DEV_LIBS__ Requires: libpmemblk__PKG_NAME_SUFFIX__ = %{version}-%{release} Requires: libpmem-devel = %{version}-%{release} %description -n libpmemblk-devel The libpmemblk implements a pmem-resident array of blocks, all the same size, where a block is updated atomically with respect to power failure or program interruption (no torn blocks). For example, a program keeping a cache of fixed-size objects in pmem might find this library useful. This library is provided for cases requiring large arrays of objects at least 512 bytes each. Most developers will find higher level libraries like libpmemobj to be more generally useful. %files -n libpmemblk-devel %defattr(-,root,root,-) %{_libdir}/libpmemblk.so %{_libdir}/pkgconfig/libpmemblk.pc %{_includedir}/libpmemblk.h %{_mandir}/man7/libpmemblk.7.gz %{_mandir}/man5/poolset.5.gz %{_mandir}/man3/pmemblk_*.3.gz %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmemblk-debug Summary: Debug variant of the Persistent Memory Resident Array of Blocks library Group: __GROUP_DEV_LIBS__ Requires: libpmemblk__PKG_NAME_SUFFIX__ = %{version}-%{release} %description -n libpmemblk-debug The libpmemblk implements a pmem-resident array of blocks, all the same size, where a block is updated atomically with respect to power failure or program interruption (no torn blocks). This sub-package contains debug variant of the library, providing run-time assertions and trace points. The typical way to access the debug version is to set the environment variable LD_LIBRARY_PATH to /usr/lib64/pmdk_debug. %files -n libpmemblk-debug %defattr(-,root,root,-) %dir %{_libdir}/pmdk_debug %{_libdir}/pmdk_debug/libpmemblk.so %{_libdir}/pmdk_debug/libpmemblk.so.* %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmemlog__PKG_NAME_SUFFIX__ Summary: Persistent Memory Resident Log File library Group: __GROUP_SYS_LIBS__ Requires: libpmem__PKG_NAME_SUFFIX__ >= %{version}-%{release} %description -n libpmemlog__PKG_NAME_SUFFIX__ The libpmemlog library provides a pmem-resident log file. This is useful for programs like databases that append frequently to a log file. %files -n libpmemlog__PKG_NAME_SUFFIX__ %defattr(-,root,root,-) %{_libdir}/libpmemlog.so.* %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmemlog-devel Summary: Development files for the Persistent Memory Resident Log File library Group: __GROUP_DEV_LIBS__ Requires: libpmemlog__PKG_NAME_SUFFIX__ = %{version}-%{release} Requires: libpmem-devel = %{version}-%{release} %description -n libpmemlog-devel The libpmemlog library provides a pmem-resident log file. This library is provided for cases requiring an append-mostly file to record variable length entries. Most developers will find higher level libraries like libpmemobj to be more generally useful. %files -n libpmemlog-devel %defattr(-,root,root,-) %{_libdir}/libpmemlog.so %{_libdir}/pkgconfig/libpmemlog.pc %{_includedir}/libpmemlog.h %{_mandir}/man7/libpmemlog.7.gz %{_mandir}/man5/poolset.5.gz %{_mandir}/man3/pmemlog_*.3.gz %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmemlog-debug Summary: Debug variant of the Persistent Memory Resident Log File library Group: __GROUP_DEV_LIBS__ Requires: libpmemlog__PKG_NAME_SUFFIX__ = %{version}-%{release} %description -n libpmemlog-debug The libpmemlog library provides a pmem-resident log file. This library is provided for cases requiring an append-mostly file to record variable length entries. Most developers will find higher level libraries like libpmemobj to be more generally useful. This sub-package contains debug variant of the library, providing run-time assertions and trace points. The typical way to access the debug version is to set the environment variable LD_LIBRARY_PATH to /usr/lib64/pmdk_debug. %files -n libpmemlog-debug %defattr(-,root,root,-) %dir %{_libdir}/pmdk_debug %{_libdir}/pmdk_debug/libpmemlog.so %{_libdir}/pmdk_debug/libpmemlog.so.* %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmemobj__PKG_NAME_SUFFIX__ Summary: Persistent Memory Transactional Object Store library Group: __GROUP_SYS_LIBS__ Requires: libpmem__PKG_NAME_SUFFIX__ >= %{version}-%{release} %description -n libpmemobj__PKG_NAME_SUFFIX__ The libpmemobj library provides a transactional object store, providing memory allocation, transactions, and general facilities for persistent memory programming. %files -n libpmemobj__PKG_NAME_SUFFIX__ %defattr(-,root,root,-) %{_libdir}/libpmemobj.so.* %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmemobj-devel Summary: Development files for the Persistent Memory Transactional Object Store library Group: __GROUP_DEV_LIBS__ Requires: libpmemobj__PKG_NAME_SUFFIX__ = %{version}-%{release} Requires: libpmem-devel = %{version}-%{release} %description -n libpmemobj-devel The libpmemobj library provides a transactional object store, providing memory allocation, transactions, and general facilities for persistent memory programming. Developers new to persistent memory probably want to start with this library. %files -n libpmemobj-devel %defattr(-,root,root,-) %{_libdir}/libpmemobj.so %{_libdir}/pkgconfig/libpmemobj.pc %{_includedir}/libpmemobj.h %{_includedir}/libpmemobj/*.h %{_mandir}/man7/libpmemobj.7.gz %{_mandir}/man5/poolset.5.gz %{_mandir}/man3/pmemobj_*.3.gz %{_mandir}/man3/pobj_*.3.gz %{_mandir}/man3/oid_*.3.gz %{_mandir}/man3/toid*.3.gz %{_mandir}/man3/direct_*.3.gz %{_mandir}/man3/d_r*.3.gz %{_mandir}/man3/tx_*.3.gz %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmemobj-debug Summary: Debug variant of the Persistent Memory Transactional Object Store library Group: __GROUP_DEV_LIBS__ Requires: libpmemobj__PKG_NAME_SUFFIX__ = %{version}-%{release} %description -n libpmemobj-debug The libpmemobj library provides a transactional object store, providing memory allocation, transactions, and general facilities for persistent memory programming. Developers new to persistent memory probably want to start with this library. This sub-package contains debug variant of the library, providing run-time assertions and trace points. The typical way to access the debug version is to set the environment variable LD_LIBRARY_PATH to /usr/lib64/pmdk_debug. %files -n libpmemobj-debug %defattr(-,root,root,-) %dir %{_libdir}/pmdk_debug %{_libdir}/pmdk_debug/libpmemobj.so %{_libdir}/pmdk_debug/libpmemobj.so.* %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmempool__PKG_NAME_SUFFIX__ Summary: Persistent Memory pool management library Group: __GROUP_SYS_LIBS__ Requires: libpmem__PKG_NAME_SUFFIX__ >= %{version}-%{release} %description -n libpmempool__PKG_NAME_SUFFIX__ The libpmempool library provides a set of utilities for off-line administration, analysis, diagnostics and repair of persistent memory pools created by libpmemlog, libpemblk and libpmemobj libraries. %files -n libpmempool__PKG_NAME_SUFFIX__ %defattr(-,root,root,-) %{_libdir}/libpmempool.so.* %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmempool-devel Summary: Development files for Persistent Memory pool management library Group: __GROUP_DEV_LIBS__ Requires: libpmempool__PKG_NAME_SUFFIX__ = %{version}-%{release} Requires: libpmem-devel = %{version}-%{release} %description -n libpmempool-devel The libpmempool library provides a set of utilities for off-line administration, analysis, diagnostics and repair of persistent memory pools created by libpmemlog, libpemblk and libpmemobj libraries. %files -n libpmempool-devel %defattr(-,root,root,-) %{_libdir}/libpmempool.so %{_libdir}/pkgconfig/libpmempool.pc %{_includedir}/libpmempool.h %{_mandir}/man7/libpmempool.7.gz %{_mandir}/man5/poolset.5.gz %{_mandir}/man3/pmempool_*.3.gz %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n libpmempool-debug Summary: Debug variant of the Persistent Memory pool management library Group: __GROUP_DEV_LIBS__ Requires: libpmempool__PKG_NAME_SUFFIX__ = %{version}-%{release} %description -n libpmempool-debug The libpmempool library provides a set of utilities for off-line administration, analysis, diagnostics and repair of persistent memory pools created by libpmemlog, libpemblk and libpmemobj libraries. This sub-package contains debug variant of the library, providing run-time assertions and trace points. The typical way to access the debug version is to set the environment variable LD_LIBRARY_PATH to /usr/lib64/pmdk_debug. %files -n libpmempool-debug %defattr(-,root,root,-) %dir %{_libdir}/pmdk_debug %{_libdir}/pmdk_debug/libpmempool.so %{_libdir}/pmdk_debug/libpmempool.so.* %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n pmempool Summary: Utilities for Persistent Memory Group: __GROUP_SYS_BASE__ Requires: libpmem__PKG_NAME_SUFFIX__ >= %{version}-%{release} Requires: libpmemlog__PKG_NAME_SUFFIX__ >= %{version}-%{release} Requires: libpmemblk__PKG_NAME_SUFFIX__ >= %{version}-%{release} Requires: libpmemobj__PKG_NAME_SUFFIX__ >= %{version}-%{release} Requires: libpmempool__PKG_NAME_SUFFIX__ >= %{version}-%{release} Obsoletes: nvml-tools < %{version}-%{release} %description -n pmempool The pmempool is a standalone utility for management and off-line analysis of Persistent Memory pools created by PMDK libraries. It provides a set of utilities for administration and diagnostics of Persistent Memory pools. The pmempool may be useful for troubleshooting by system administrators and users of the applications based on PMDK libraries. %files -n pmempool %{_bindir}/pmempool %{_mandir}/man1/pmempool.1.gz %{_mandir}/man1/pmempool-*.1.gz %config(noreplace) %{_sysconfdir}/bash_completion.d/pmempool %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %package -n pmreorder Summary: Consistency Checker for Persistent Memory Group: __GROUP_SYS_BASE__ %description -n pmreorder The pmreorder tool is a collection of python scripts designed to parse and replay operations logged by pmemcheck - a persistent memory checking tool. Pmreorder performs the store reordering between persistent memory barriers - a sequence of flush-fence operations. It uses a consistency checking routine provided in the command line options to check whether files are in a consistent state. %files -n pmreorder %{_bindir}/pmreorder %{_datadir}/pmreorder/*.py %{_mandir}/man1/pmreorder.1.gz %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md %if %{with ndctl} %package -n daxio Summary: Perform I/O on Device DAX devices or zero a Device DAX device Group: __GROUP_SYS_BASE__ Requires: libpmem__PKG_NAME_SUFFIX__ >= %{version}-%{release} %description -n daxio The daxio utility performs I/O on Device DAX devices or zero a Device DAX device. Since the standard I/O APIs (read/write) cannot be used with Device DAX, data transfer is performed on a memory-mapped device. The daxio may be used to dump Device DAX data to a file, restore data from a backup copy, move/copy data to another device or to erase data from a device. %files -n daxio %{_bindir}/daxio %{_mandir}/man1/daxio.1.gz %license LICENSE %doc ChangeLog CONTRIBUTING.md README.md # end of "if _with_ndctl" %endif %prep %setup -q -n %{name}-%{version} %build %define _lto_cflags %{nil} # For debug build default flags may be overridden to disable compiler # optimizations. CFLAGS="%{optflags}" \ LDFLAGS="%{?__global_ldflags}" \ make %{?_smp_mflags} \ %if %{without ndctl} NDCTL_ENABLE=n \ %endif __MAKE_FLAGS__ # Override LIB_AR with empty string to skip installation of static libraries %install make install DESTDIR=%{buildroot} \ %if %{without ndctl} NDCTL_ENABLE=n \ %endif LIB_AR= \ prefix=%{_prefix} \ libdir=%{_libdir} \ includedir=%{_includedir} \ mandir=%{_mandir} \ bindir=%{_bindir} \ sysconfdir=%{_sysconfdir} \ docdir=%{_docdir} mkdir -p %{buildroot}%{_datadir}/pmdk cp utils/pmdk.magic %{buildroot}%{_datadir}/pmdk/ __MAKE_INSTALL_FDUPES__ %check %if 0%{?_skip_check} == 1 echo "Check skipped" %else %if %{defined _testconfig} cp %{_testconfig} src/test/testconfig.sh %else echo "PMEM_FS_DIR=/tmp" > src/test/testconfig.sh echo "PMEM_FS_DIR_FORCE_PMEM=1" >> src/test/testconfig.sh echo 'TEST_BUILD="debug nondebug"' >> src/test/testconfig.sh echo 'TEST_FS="pmem any none"' >> src/test/testconfig.sh %endif make \ %if %{without ndctl} NDCTL_ENABLE=n \ %endif check %endif %post -n libpmem__PKG_NAME_SUFFIX__ -p /sbin/ldconfig %postun -n libpmem__PKG_NAME_SUFFIX__ -p /sbin/ldconfig %post -n libpmemblk__PKG_NAME_SUFFIX__ -p /sbin/ldconfig %postun -n libpmemblk__PKG_NAME_SUFFIX__ -p /sbin/ldconfig %post -n libpmemlog__PKG_NAME_SUFFIX__ -p /sbin/ldconfig %postun -n libpmemlog__PKG_NAME_SUFFIX__ -p /sbin/ldconfig %post -n libpmemobj__PKG_NAME_SUFFIX__ -p /sbin/ldconfig %postun -n libpmemobj__PKG_NAME_SUFFIX__ -p /sbin/ldconfig %post -n libpmempool__PKG_NAME_SUFFIX__ -p /sbin/ldconfig %postun -n libpmempool__PKG_NAME_SUFFIX__ -p /sbin/ldconfig %if 0%{?__debug_package} == 0 %debug_package %endif %changelog pmdk-1.13.1/utils/pkg-config.sh0000664000000000000000000000074614435627501014766 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2020, Intel Corporation # Name of package PACKAGE_NAME="pmdk" # Name and email of package maintainer PACKAGE_MAINTAINER="Piotr Balcer " # Brief description of the package PACKAGE_SUMMARY="Persistent Memory Development Kit" # Full description of the package PACKAGE_DESCRIPTION="The collection of libraries and utilities for Persistent Memory Programming" # Website PACKAGE_URL="https://pmem.io/pmdk" pmdk-1.13.1/utils/libpmempool.pc.in0000664000000000000000000000037514435627501015654 0ustar rootrootincludedir=${prefix}/include Name: libpmempool Description: libpmempool library from PMDK project Version: ${version} URL: https://pmem.io/pmdk Requires.private: libpmem${rasdeps} Libs: -L${libdir} -lpmempool Libs.private: -ldl Cflags: -I${includedir} pmdk-1.13.1/utils/docker/0000775000000000000000000000000014435627501013646 5ustar rootrootpmdk-1.13.1/utils/docker/images/0000775000000000000000000000000014435627501015113 5ustar rootrootpmdk-1.13.1/utils/docker/images/set-images-version.sh0000775000000000000000000000067014435627501021176 0ustar rootroot#!/usr/bin/env bash # # SPDX-License-Identifier: BSD-3-Clause # Copyright 2020-2023, Intel Corporation # # # set-images-version.sh -- set value of the 'IMG_VER' variable # containing the current version of Docker images # # This file has to be located in the "utils/docker/images" # subdirectory, because every change of a value of IMG_VER # has to trigger the rebuild of all Docker images. # export IMG_VER=1.13 pmdk-1.13.1/utils/docker/images/Dockerfile.ubuntu-22.040000664000000000000000000000451114435627501021072 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # Dockerfile - a 'recipe' for Docker to build an image of ubuntu-based # environment for building the PMDK project. # # Pull base image FROM ubuntu:22.04 MAINTAINER tomasz.gromadzki@intel.com # Set required environment variables ENV OS ubuntu ENV OS_VER 22.04 ENV PACKAGE_MANAGER dpkg ENV NOTTY 1 ENV DEBIAN_FRONTEND noninteractive # Additional parameters to build docker without building components ARG SKIP_SCRIPTS_DOWNLOAD # PMDK basic dependencies ENV BASE_DEPS "build-essential \ cmake \ git \ libdaxctl-dev \ libndctl-dev \ pkg-config" # Valgrind dependencies ENV VALGRIND_DEPS "autoconf \ automake \ build-essential \ git" # benchmarks dependencies (optional) ENV BENCH_DEPS libglib2.0-dev # examples dependencies (optional) ENV EXAMPLES_DEPS "libfuse-dev \ libncurses5-dev \ libuv1-dev" # documentation dependencies (optional) ENV DOC_DEPS pandoc # tests dependencies ENV TESTS_DEPS "bc \ gdb \ libc6-dbg \ libunwind-dev \ ndctl \ python3 \ ssh \ strace \ bandit " # packaging dependencies ENV PACKAGING_DEPS "debhelper \ devscripts \ fakeroot" # Codecov (coverage measurement) dependencies (optional) ENV CODECOV_DEPS curl perl # Coverity (static analysis) dependencies (optional) ENV COVERITY_DEPS ruby gcc g++ wget # miscellaneous dependencies (mostly for CI) ENV MISC_DEPS "clang \ hub \ rsync \ clang-format \ flake8 \ python3-pip \ sudo \ whois" # python scripts dependencies ENV PYTHON_DEPS "\ distro \ xmlrunner" # Copy install valgrind script COPY install-valgrind.sh install-valgrind.sh # Copy script to download codecov script required in run-coverage.sh COPY download-scripts.sh download-scripts.sh # Update the Apt cache and install basic tools RUN apt-get update && apt-get dist-upgrade -y \ && apt-get install -y --no-install-recommends \ $BASE_DEPS \ $VALGRIND_DEPS \ $BENCH_DEPS \ $EXAMPLES_DEPS \ $DOC_DEPS \ $TESTS_DEPS \ $PACKAGING_DEPS \ $CODECOV_DEPS \ $COVERITY_DEPS \ $MISC_DEPS \ && ./install-valgrind.sh ubuntu \ && ./download-scripts.sh \ && rm -rf /var/lib/apt/lists/* RUN pip3 install $PYTHON_DEPS # Add and switch user ENV USER pmdkuser ENV USERPASS pmdkpass RUN useradd -m $USER -g sudo -p `mkpasswd $USERPASS` RUN echo "%sudo ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers USER $USER pmdk-1.13.1/utils/docker/images/install-valgrind.sh0000775000000000000000000000210714435627501020724 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # install-valgrind.sh - installs valgrind for persistent memory # set -e OS=$1 install_upstream_from_distro() { case "$OS" in rockylinux) dnf install -y valgrind ;; fedora) dnf install -y valgrind ;; ubuntu) apt-get install -y --no-install-recommends valgrind ;; *) return 1 ;; esac } install_upstream_3_16_1() { git clone git://sourceware.org/git/valgrind.git cd valgrind # valgrind v3.16.1 upstream git checkout VALGRIND_3_16_BRANCH ./autogen.sh ./configure make -j$(nproc) make -j$(nproc) install cd .. rm -rf valgrind } install_custom-pmem_from_source() { git clone https://github.com/pmem/valgrind.git cd valgrind # valgrind v3.20 + fixes for ppc64; 01.02.2023 git checkout c0abd814ff955c7eb2850bd3827167a6b084e975 ./autogen.sh ./configure make -j$(nproc) make -j$(nproc) install cd .. rm -rf valgrind } ARCH=$(uname -m) case "$ARCH" in aarch64) install_upstream_3_16_1 ;; *) install_custom-pmem_from_source ;; esac pmdk-1.13.1/utils/docker/images/Dockerfile.fedora-370000664000000000000000000000350114435627501020572 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # Dockerfile - a 'recipe' for Docker to build an image of fedora-based # environment for building the PMDK project. # # Pull base image FROM fedora:37 MAINTAINER tomasz.gromadzki@intel.com # Set required environment variables ENV OS fedora ENV OS_VER 37 ENV PACKAGE_MANAGER rpm ENV NOTTY 1 # PMDK basic dependencies ENV BASE_DEPS "\ cmake \ git \ daxctl-devel \ make \ ndctl-devel \ pkgconfig" # Valgrind dependencies ENV VALGRIND_DEPS "\ autoconf \ automake \ file \ findutils \ git" # benchmarks dependencies (optional) ENV BENCH_DEPS "\ glib2-devel" # examples dependencies (optional) ENV EXAMPLES_DEPS "\ fuse \ fuse-devel \ ncurses-devel \ libuv-devel" # documentation dependencies (optional) ENV DOC_DEPS "\ pandoc" # tests dependencies ENV TESTS_DEPS "\ bc \ gdb \ libunwind-devel \ ndctl \ openssh-server \ python3 \ strace" # packaging dependencies ENV PACKAGING_DEPS "\ rpm-build \ rpm-build-libs \ rpmdevtools" # miscellaneous dependencies (mostly for CI) ENV MISC_DEPS "\ clang \ hub \ lbzip2 \ man \ python3-flake8 \ python3-pip \ rsync \ shadow-utils \ sudo \ tar \ which \ xmlto" # python scripts dependencies ENV PYTHON_DEPS "\ distro \ xmlrunner" # Copy install valgrind script COPY install-valgrind.sh install-valgrind.sh RUN dnf update -y && dnf install -y \ $BASE_DEPS \ $VALGRIND_DEPS \ $BENCH_DEPS \ $EXAMPLES_DEPS \ $DOC_DEPS \ $TESTS_DEPS \ $PACKAGING_DEPS \ $MISC_DEPS \ && ./install-valgrind.sh fedora \ && dnf clean all RUN pip3 install $PYTHON_DEPS # Add and switch user ENV USER pmdkuser ENV USERPASS pmdkpass RUN useradd -m $USER RUN echo "$USER:$USERPASS" | chpasswd RUN gpasswd wheel -a $USER RUN echo "%wheel ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers USER $USER pmdk-1.13.1/utils/docker/images/download-scripts.sh0000775000000000000000000000176414435627501020756 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2020-2023, Intel Corporation # # download-scripts.sh - downloads latest version of # codecov's uploader to generate and upload reports. # set -e if [ "${SKIP_SCRIPTS_DOWNLOAD}" ]; then echo "Variable 'SKIP_SCRIPTS_DOWNLOAD' is set; skipping scripts' download" exit fi mkdir -p /opt/scripts if ! [ -x "$(command -v curl)" ]; then echo "Error: curl is not installed." return 1 fi # Download codecov and check integrity mkdir -p codecov-tmp pushd codecov-tmp curl https://keybase.io/codecovsecurity/pgp_keys.asc | gpg --no-default-keyring --keyring trustedkeys.gpg --import curl -Os https://uploader.codecov.io/latest/linux/codecov curl -Os https://uploader.codecov.io/latest/linux/codecov.SHA256SUM curl -Os https://uploader.codecov.io/latest/linux/codecov.SHA256SUM.sig gpgv codecov.SHA256SUM.sig codecov.SHA256SUM shasum -a 256 -c codecov.SHA256SUM chmod +x codecov mv -v codecov /opt/scripts/codecov popd rm -rf codecov-tmp pmdk-1.13.1/utils/docker/images/push-image.sh0000775000000000000000000000307114435627501017512 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # push-image.sh - pushes the Docker image to $DOCKER_REPO. # # The script utilizes $GH_CR_USER and $GH_CR_PAT variables # to log in to $DOCKER_REPO. These variables can be set # in the project's CI configuration for automated builds. # set -e source $(dirname $0)/../set-ci-vars.sh if [[ -z "$OS" ]]; then echo "OS environment variable is not set" exit 1 fi if [[ -z "$OS_VER" ]]; then echo "OS_VER environment variable is not set" exit 1 fi if [[ -z "$CI_CPU_ARCH" ]]; then echo "CI_CPU_ARCH environment variable is not set" exit 1 fi if [[ -z "${DOCKER_REPO}" ]]; then echo "DOCKER_REPO environment variable is not set" exit 1 fi if [[ -z "$IMG_VER" ]]; then echo "IMG_VER environment variable is not set" exit 1 fi if [[ -z "${GH_CR_USER}" || -z "${GH_CR_PAT}" ]]; then echo "ERROR: variables GH_CR_USER=\"${GH_CR_USER}\" and GH_CR_PAT=\"${GH_CR_PAT}\"" \ "have to be set properly to allow login to the $DOCKER_REPO." exit 1 fi TAG="${IMG_VER}-${OS}-${OS_VER}-${CI_CPU_ARCH}" # Check if the image tagged with $TAG exists locally if [[ ! $(docker images -a | awk -v pattern="^${DOCKER_REPO}:${TAG}\$" \ '$1":"$2 ~ pattern') ]] then echo "ERROR: Docker image tagged ${DOCKER_REPO}:${TAG} does not exists locally." exit 1 fi # Log in to $DOCKER_REPO echo "${GH_CR_PAT}" | docker login "${DOCKER_REPO}" -u="${GH_CR_USER}" --password-stdin echo "Push the image '${DOCKER_REPO}:${TAG}' to the $DOCKER_REPO." docker push ${DOCKER_REPO}:${TAG} echo "Image pushed." pmdk-1.13.1/utils/docker/images/README0000664000000000000000000000142414435627501015774 0ustar rootrootPersistent Memory Development Kit This is utils/docker/images/README. Dockerfiles and scripts placed in this directory are intended to be used as development process vehicles and part of continuous integration process. Images built out of those recipes may by used with Docker or podman as development environment. To manually build docker image using docker(1) execute, e.g.: docker build --build-arg https_proxy=http://proxy.com:port --build-arg http_proxy=http://proxy.com:port \ -t pmdk:debian-unstable -f ./Dockerfile.ubuntu-22.04 . To run build and tests within a (previously built) Docker container execute, e.g.: docker run --network=bridge --shm-size=4G -v /your/workspace/path/:/opt/workspace:z -w /opt/workspace/ \ -e PMDK_CC=gcc -it pmdk:ubuntu-22.04 /bin/bash pmdk-1.13.1/utils/docker/images/Dockerfile.rockylinux-80000664000000000000000000000373614435627501021471 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2023, Intel Corporation # # Dockerfile - a 'recipe' for Docker to build an image of rockylinux-based # environment for building the PMDK project. # # Pull base image FROM rockylinux/rockylinux:8 MAINTAINER tomasz.gromadzki@intel.com # Set required environment variables ENV OS rockylinux/rockylinux ENV OS_VER 8 ENV PACKAGE_MANAGER rpm ENV NOTTY 1 # PMDK basic dependencies ENV BASE_DEPS "\ cmake \ git \ daxctl-devel \ make \ ndctl-devel \ passwd \ pkg-config" # Valgrind dependencies ENV VALGRIND_DEPS "\ autoconf \ automake \ file \ findutils \ git" # benchmarks dependencies (optional) ENV BENCH_DEPS "\ glib2-devel" # examples dependencies (optional) ENV EXAMPLES_DEPS "\ fuse \ fuse-devel \ ncurses-devel \ libuv-devel" # documentation dependencies (optional) ENV DOC_DEPS "\ pandoc" # tests dependencies ENV TESTS_DEPS "\ bc \ gdb \ libunwind-devel \ ndctl \ openssh-server \ strace" # packaging dependencies ENV PACKAGING_DEPS "\ rpm-build \ rpm-build-libs \ rpmdevtools" # miscellaneous dependencies (mostly for CI) ENV MISC_DEPS "\ clang \ lbzip2 \ man \ python3 \ python3-flake8 \ python3-pip \ rsync \ shadow-utils \ sudo \ tar \ which \ xmlto" # python scripts dependencies ENV PYTHON_DEPS "\ distro \ xmlrunner" # Copy install valgrind script COPY install-valgrind.sh install-valgrind.sh RUN dnf update -y && dnf install -y epel-release \ 'dnf-command(config-manager)' \ && dnf config-manager --set-enabled powertools \ && dnf install -y \ $BASE_DEPS \ $VALGRIND_DEPS \ $BENCH_DEPS \ $EXAMPLES_DEPS \ $DOC_DEPS \ $TESTS_DEPS \ $PACKAGING_DEPS \ $MISC_DEPS \ && ./install-valgrind.sh rockylinux \ && dnf clean all RUN pip3 install $PYTHON_DEPS # Add ans switch user ENV USER pmdkuser ENV USERPASS pmdkpass RUN useradd -m $USER RUN echo $USERPASS | passwd $USER --stdin RUN gpasswd wheel -a $USER RUN echo "%wheel ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers USER $USER pmdk-1.13.1/utils/docker/images/Dockerfile.debian-110000664000000000000000000000335314435627501020551 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # Dockerfile - a 'recipe' for Docker to build an image of Debian-based # environment for building the PMDK project. # # Pull base image FROM debian:11 MAINTAINER tomasz.gromadzki@intel.com # Set required environment variables ENV OS debian ENV OS_VER 11 ENV PACKAGE_MANAGER dpkg ENV NOTTY 1 ENV DEBIAN_FRONTEND noninteractive # PMDK basic dependencies ENV BASE_DEPS "build-essential \ cmake \ git \ libdaxctl-dev \ libndctl-dev \ pkg-config" # Valgrind dependencies ENV VALGRIND_DEPS "autoconf \ automake \ build-essential \ git" # benchmarks dependencies (optional) ENV BENCH_DEPS libglib2.0-dev # examples dependencies (optional) ENV EXAMPLES_DEPS "libfuse-dev \ libncurses5-dev \ libuv1-dev" # documentation dependencies (optional) ENV DOC_DEPS pandoc # tests dependencies ENV TESTS_DEPS "bc \ gdb \ libc6-dbg \ libunwind-dev \ ndctl \ python3 \ strace" # packaging dependencies ENV PACKAGING_DEPS "debhelper \ devscripts \ fakeroot" # miscellaneous dependencies (mostly for CI) ENV MISC_DEPS "clang \ clang-format \ flake8 \ sudo \ whois" # Copy install valgrind script COPY install-valgrind.sh install-valgrind.sh # Update the Apt cache and install basic tools RUN apt-get update && apt-get dist-upgrade -y \ && apt-get install -y --no-install-recommends \ $BASE_DEPS \ $VALGRIND_DEPS \ $BENCH_DEPS \ $EXAMPLES_DEPS \ $DOC_DEPS \ $TESTS_DEPS \ $PACKAGING_DEPS \ $MISC_DEPS \ && ./install-valgrind.sh ubuntu \ && rm -rf /var/lib/apt/lists/* # Add and switch user ENV USER pmdkuser ENV USERPASS pmdkpass RUN useradd -m $USER -g sudo -p `mkpasswd $USERPASS` RUN echo "%sudo ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers USER $USER pmdk-1.13.1/utils/docker/images/Dockerfile.opensuse-leap-150000664000000000000000000000415414435627501022113 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # Dockerfile - a 'recipe' for Docker to build an image of openSUSE-based # environment for building the PMDK project. # # Pull base image FROM opensuse/leap:15 MAINTAINER tomasz.gromadzki@intel.com # Set required environment variables ENV OS opensuse/leap ENV OS_VER 15 ENV PACKAGE_MANAGER rpm ENV NOTTY 1 ENV LD_LIBRARY_PATH /usr/lib:$LD_LIBRARY_PATH ENV PKG_CONFIG_PATH /usr/lib/pkgconfig:$PKG_CONFIG_PATH # PMDK basic dependencies ENV BASE_DEPS "\ cmake \ gcc \ gcc-c++ \ git \ libndctl-devel \ make \ pkg-config \ systemd" # Valgrind dependencies ENV VALGRIND_DEPS "\ autoconf \ automake \ file \ findutils \ git" # benchmarks dependencies (optional) ENV BENCH_DEPS "\ glib2-devel" # examples dependencies (optional) ENV EXAMPLES_DEPS "\ fuse \ fuse-devel \ ncurses-devel \ libuv-devel" # documentation dependencies (optional) ENV DOC_DEPS "\ pandoc" # tests dependencies ENV TESTS_DEPS "\ bc \ gdb \ libunwind-devel \ ndctl \ strace" # packaging dependencies ENV PACKAGING_DEPS "\ fdupes \ rpm-build \ rpmdevtools" # miscellaneous dependencies (mostly for CI) ENV MISC_DEPS "\ clang \ hub \ lbzip2 \ man \ python3-flake8 \ python3-pip \ rsync \ sudo \ tar \ which \ xmlto" # python scripts dependencies ENV PYTHON_DEPS "\ distro \ xmlrunner" # Copy install valgrind script COPY install-valgrind.sh install-valgrind.sh # add openSUSE Leap 15.4 Oss repo RUN zypper ar -f http://download.opensuse.org/distribution/leap/15.4/repo/oss/ oss RUN zypper update -y && zypper install -y \ $BASE_DEPS \ $VALGRIND_DEPS \ $BENCH_DEPS \ $EXAMPLES_DEPS \ $DOC_DEPS \ $TESTS_DEPS \ $PACKAGING_DEPS \ $MISC_DEPS \ && ./install-valgrind.sh fedora \ && zypper clean all RUN pip3 install $PYTHON_DEPS # Add and switch user ENV USER pmdkuser ENV USERPASS pmdkpass RUN useradd -m $USER ENV PFILE ./password RUN echo $USERPASS > $PFILE RUN echo $USERPASS >> $PFILE RUN passwd $USER < $PFILE RUN rm -f $PFILE RUN sed -i 's/# %wheel/%wheel/g' /etc/sudoers RUN groupadd wheel RUN gpasswd wheel -a $USER USER $USER pmdk-1.13.1/utils/docker/images/build-image.sh0000775000000000000000000000261214435627501017632 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # build-image.sh - prepares a Docker image with -based # environment intended for the CPU architecture # designed for building PMDK project, according to # the Dockerfile. file located in the same directory. # # The script can be run locally. # set -e OS_VER=$1 CPU_ARCH=$2 function usage { echo "Usage:" echo " build-image.sh " echo "where:" echo " - can be for example 'ubuntu-22.04' provided "\ "a Dockerfile named 'Dockerfile.ubuntu-22.04' "\ "exists in the current directory and" echo " - is a CPU architecture, for example 'x86_64'" } # Check if two first arguments are not empty if [[ -z "$2" ]]; then usage exit 1 fi # Check if the file Dockerfile.OS-VER exists if [[ ! -f "Dockerfile.$OS_VER" ]]; then echo "Error: Dockerfile.$OS_VER does not exist." echo usage exit 1 fi if [[ -z "${DOCKER_REPO}" || -z "${IMG_VER}" ]]; then echo "Error: DOCKER_REPO (${DOCKER_REPO}) or IMG_VER (${IMG_VER}) " \ "environment variables are not set." exit 1 fi # Build a Docker image tagged as follows: tag=${DOCKER_REPO}:${IMG_VER}-${OS_VER}-${CPU_ARCH} docker build -t $tag \ --build-arg http_proxy=$http_proxy \ --build-arg https_proxy=$https_proxy \ -f Dockerfile.$OS_VER . pmdk-1.13.1/utils/docker/images/Dockerfile.rockylinux-90000664000000000000000000000363314435627501021466 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2023, Intel Corporation # # Dockerfile - a 'recipe' for Docker to build an image of rockylinux-based # environment for building the PMDK project. # # Pull base image FROM rockylinux/rockylinux:9 MAINTAINER tomasz.gromadzki@intel.com # Set required environment variables ENV OS rockylinux/rockylinux ENV OS_VER 9 ENV PACKAGE_MANAGER rpm ENV NOTTY 1 # PMDK basic dependencies ENV BASE_DEPS "\ cmake \ git \ daxctl-devel \ make \ ndctl-devel \ passwd \ pkgconfig" # Valgrind dependencies ENV VALGRIND_DEPS "\ autoconf \ automake \ file \ findutils \ git" # benchmarks dependencies (optional) ENV BENCH_DEPS "\ glib2-devel" # examples dependencies (optional) ENV EXAMPLES_DEPS "\ fuse \ fuse-devel \ ncurses-devel \ libuv-devel" # documentation dependencies (optional) ENV DOC_DEPS "\ pandoc" # tests dependencies ENV TESTS_DEPS "\ bc \ gdb \ libunwind-devel \ ndctl \ openssh-server \ strace" # packaging dependencies ENV PACKAGING_DEPS "\ rpm-build \ rpm-build-libs \ rpmdevtools" # miscellaneous dependencies (mostly for CI) ENV MISC_DEPS "\ clang \ lbzip2 \ man \ python3 \ python3-flake8 \ python3-pip \ rsync \ shadow-utils \ sudo \ tar \ which \ xmlto" # python scripts dependencies ENV PYTHON_DEPS "\ distro \ xmlrunner" # Copy install valgrind script COPY install-valgrind.sh install-valgrind.sh RUN dnf update -y && dnf install -y epel-release \ && dnf --enablerepo devel install -y \ $BASE_DEPS \ $VALGRIND_DEPS \ $BENCH_DEPS \ $EXAMPLES_DEPS \ $DOC_DEPS \ $TESTS_DEPS \ $PACKAGING_DEPS \ $MISC_DEPS \ && ./install-valgrind.sh rockylinux \ && dnf clean all RUN pip3 install $PYTHON_DEPS # Add ans switch user ENV USER pmdkuser ENV USERPASS pmdkpass RUN useradd -m $USER RUN echo $USERPASS | passwd $USER --stdin RUN gpasswd wheel -a $USER RUN echo "%wheel ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers USER $USER pmdk-1.13.1/utils/docker/images/Dockerfile.fedora-310000664000000000000000000000327114435627501020570 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # Dockerfile - a 'recipe' for Docker to build an image of fedora-based # environment for building the PMDK project. # # Pull base image FROM fedora:31 MAINTAINER tomasz.gromadzki@intel.com # Set required environment variables ENV OS fedora ENV OS_VER 31 ENV PACKAGE_MANAGER rpm ENV NOTTY 1 # PMDK basic dependencies ENV BASE_DEPS "\ cmake \ git \ daxctl-devel \ make \ ndctl-devel \ pkgconfig" # Valgrind dependencies ENV VALGRIND_DEPS "\ autoconf \ automake \ file \ findutils \ git" # benchmarks dependencies (optional) ENV BENCH_DEPS "\ glib2-devel" # examples dependencies (optional) ENV EXAMPLES_DEPS "\ fuse \ fuse-devel \ ncurses-devel \ libuv-devel" # documentation dependencies (optional) ENV DOC_DEPS "\ pandoc" # tests dependencies ENV TESTS_DEPS "\ bc \ gdb \ libunwind-devel \ ndctl \ openssh-server \ strace" # packaging dependencies ENV PACKAGING_DEPS "\ rpm-build \ rpm-build-libs \ rpmdevtools" # miscellaneous dependencies (mostly for CI) ENV MISC_DEPS "\ clang \ lbzip2 \ man \ python3-flake8 \ rsync \ shadow-utils \ sudo \ tar \ which \ xmlto" # Copy install valgrind script COPY install-valgrind.sh install-valgrind.sh RUN dnf update -y && dnf install -y \ $BASE_DEPS \ $VALGRIND_DEPS \ $BENCH_DEPS \ $EXAMPLES_DEPS \ $DOC_DEPS \ $TESTS_DEPS \ $PACKAGING_DEPS \ $MISC_DEPS \ && ./install-valgrind.sh fedora \ && dnf clean all # Add and switch user ENV USER pmdkuser ENV USERPASS pmdkpass RUN useradd -m $USER RUN echo "$USER:$USERPASS" | chpasswd RUN gpasswd wheel -a $USER RUN echo "%wheel ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers USER $USER pmdk-1.13.1/utils/docker/run-build.sh0000775000000000000000000000201214435627501016101 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # run-build.sh - is called inside a Docker container; prepares the environment # and starts a build of PMDK project. # set -e # Prepare build environment ./prepare-for-build.sh # Build all and run tests cd $WORKDIR if [ "$SRC_CHECKERS" != "0" ]; then make -j$(nproc) check-license make -j$(nproc) cstyle fi echo "## Running make" make -j$(nproc) echo "" echo "## Running make test" make -j$(nproc) test echo "" echo "## Running make pcheck" # do not change -j1 to -j$(nproc) in case of tests (make check/pycheck) make -j1 pcheck TEST_BUILD=$TEST_BUILD echo "" echo "## Running make pycheck" # do not change -j1 to -j$(nproc) in case of tests (make check/pycheck) make -j1 pycheck echo "" echo "## Running make source" make -j$(nproc) DESTDIR=/tmp source # Create PR with generated docs if [ "${AUTO_DOC_UPDATE}" == "1" ]; then echo "" echo "## Running auto doc update" ./utils/docker/run-doc-update.sh fi pmdk-1.13.1/utils/docker/configure-tests.sh0000775000000000000000000000336614435627501017336 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # configure-tests.sh - is called inside a Docker container; configures tests # for use during build of PMDK project. # set -e # Configure tests cat << EOF > $WORKDIR/src/test/testconfig.sh LONGDIR=LoremipsumdolorsitametconsecteturadipiscingelitVivamuslacinianibhattortordictumsollicitudinNullamvariusvestibulumligulaetegestaselitsemperidMaurisultriciesligulaeuipsumtinciduntluctusMorbimaximusvariusdolorid # this path is ~3000 characters long DIRSUFFIX="$LONGDIR/$LONGDIR/$LONGDIR/$LONGDIR/$LONGDIR" NON_PMEM_FS_DIR=/tmp PMEM_FS_DIR=/tmp PMEM_FS_DIR_FORCE_PMEM=1 TEST_BUILD="debug nondebug" ENABLE_SUDO_TESTS=y TM=1 EOF if [[ "${1}" == "PKG" ]]; then #Append variables exclusively for PKG tests: if [ $OS = opensuse/leap ] || [ $OS = rockylinux/rockylinux ]; then echo "PMDK_LIB_PATH_NONDEBUG=/usr/lib64" >> $WORKDIR/src/test/testconfig.sh echo "PMDK_LIB_PATH_DEBUG=/usr/lib64/pmdk_debug" >> $WORKDIR/src/test/testconfig.sh elif [ $OS = ubuntu ]; then echo "PMDK_LIB_PATH_NONDEBUG=/lib/x86_64-linux-gnu" >> $WORKDIR/src/test/testconfig.sh echo "PMDK_LIB_PATH_DEBUG=/lib/x86_64-linux-gnu/pmdk_dbg" >> $WORKDIR/src/test/testconfig.sh fi fi # Configure python tests cat << EOF >> $WORKDIR/src/test/testconfig.py config = { 'unittest_log_level': 1, 'cacheline_fs_dir': '/tmp', 'force_cacheline': True, 'page_fs_dir': '/tmp', 'force_page': False, 'byte_fs_dir': '/tmp', 'force_byte': True, 'tm': True, 'test_type': 'check', 'granularity': 'all', 'fs_dir_force_pmem': 0, 'keep_going': False, 'timeout': '3m', 'build': ['debug', 'release'], 'force_enable': None, 'device_dax_path': [], 'fail_on_skip': False, 'enable_admin_tests': True } EOF pmdk-1.13.1/utils/docker/run-coverage.sh0000775000000000000000000000320314435627501016600 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2017-2023, Intel Corporation # # run-coverage.sh - is called inside a Docker container; runs tests # to measure code coverage and sends report to codecov.io # set -e # Get and prepare PMDK source ./prepare-for-build.sh # Hush error messages, mainly from Valgrind export UT_DUMP_LINES=0 # Skip printing mismatched files for tests with Valgrind export UT_VALGRIND_SKIP_PRINT_MISMATCHED=1 # Build all and run tests pushd ${WORKDIR} make -j$(nproc) COVERAGE=1 make -j$(nproc) test COVERAGE=1 # XXX: unfortunately valgrind reports issues in coverage instrumentation # which we have to ignore (-k flag) pushd src/test # do not change -j2 to -j$(nproc) in case of tests (make check/pycheck) make -kj2 pcheck-local-quiet TEST_BUILD=debug || true # do not change -j2 to -j$(nproc) in case of tests (make check/pycheck) make -kj2 pycheck TEST_BUILD=debug || true popd # prepare flag for codecov report to differentiate builds flag=tests [ -n "${GITHUB_ACTIONS}" ] && flag=GHA [ -n "${TRAVIS}" ] && flag=Travis # validate codecov.yaml file cat "${WORKDIR}/.codecov.yml" | curl --data-binary @- https://codecov.io/validate # run codecov's uploader in current dir (WORKDIR), with gcov executable # (clean parsed coverage files, set flag and exit 1 if not successful) /opt/scripts/codecov --rootDir . --gcov --clean --flags ${flag} --nonZero --verbose echo "Check for any leftover gcov files" leftover_files=$(find . -name "*.gcov") if [[ -n "${leftover_files}" ]]; then # display found files and exit with error (they all should be parsed) echo "${leftover_files}" return 1 fi pmdk-1.13.1/utils/docker/run-coverity.sh0000775000000000000000000000422414435627501016655 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2017-2020, Intel Corporation # # run-coverity.sh - runs the Coverity scan build # set -e if [[ "$CI_REPO_SLUG" != "$GITHUB_REPO" \ && ( "$COVERITY_SCAN_NOTIFICATION_EMAIL" == "" \ || "$COVERITY_SCAN_TOKEN" == "" ) ]]; then echo echo "Skipping Coverity build:"\ "COVERITY_SCAN_TOKEN=\"$COVERITY_SCAN_TOKEN\" or"\ "COVERITY_SCAN_NOTIFICATION_EMAIL="\ "\"$COVERITY_SCAN_NOTIFICATION_EMAIL\" is not set" exit 0 fi # Prepare build environment ./prepare-for-build.sh CERT_FILE=/etc/ssl/certs/ca-certificates.crt TEMP_CF=$(mktemp) cp $CERT_FILE $TEMP_CF # Download Coverity certificate echo -n | openssl s_client -connect scan.coverity.com:443 | \ sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | \ tee -a $TEMP_CF echo $USERPASS | sudo -S mv $TEMP_CF $CERT_FILE export COVERITY_SCAN_PROJECT_NAME="$CI_REPO_SLUG" [[ "$CI_EVENT_TYPE" == "cron" ]] \ && export COVERITY_SCAN_BRANCH_PATTERN="master" \ || export COVERITY_SCAN_BRANCH_PATTERN="coverity_scan" export COVERITY_SCAN_BUILD_COMMAND="make -j$(nproc) all" cd $WORKDIR # # Run the Coverity scan # # The 'travisci_build_coverity_scan.sh' script requires the following # environment variables to be set: # - TRAVIS_BRANCH - has to contain the name of the current branch # - TRAVIS_PULL_REQUEST - has to be set to 'true' in case of pull requests # export TRAVIS_BRANCH=${CI_BRANCH} [ "${CI_EVENT_TYPE}" == "pull_request" ] && export TRAVIS_PULL_REQUEST="true" # XXX: Patch the Coverity script. # Recently, this script regularly exits with an error, even though # the build is successfully submitted. Probably because the status code # is missing in response, or it's not 201. # Changes: # 1) change the expected status code to 200 and # 2) print the full response string. # # This change should be reverted when the Coverity script is fixed. # # The previous version was: # curl -s https://scan.coverity.com/scripts/travisci_build_coverity_scan.sh | bash wget https://scan.coverity.com/scripts/travisci_build_coverity_scan.sh patch < utils/docker/0001-travis-fix-travisci_build_coverity_scan.sh.patch bash ./travisci_build_coverity_scan.sh pmdk-1.13.1/utils/docker/ppc64le.blacklist0000664000000000000000000000015714435627501017020 0ustar rootrootobj_basic_integration obj_ctl_debug obj_mem obj_memcheck_register obj_pmalloc_mt obj_ulog_size pmempool_create pmdk-1.13.1/utils/docker/README0000664000000000000000000000150014435627501014522 0ustar rootrootPersistent Memory Development Kit This is utils/docker/README. Scripts in this directory let Travis or Github Actions CIs run a Docker container with ubuntu- or fedora-based environment and build PMDK project inside it. 'build-local.sh' can be used to build PMDK locally. 'build-CI.sh' is used for building PMDK on Travis and GitHub Actions CIs NOTE: If you commit changes to any Dockerfile or shell script in the 'images' subdirectory and then do git-rebase before pushing your commits to the repository, make sure that you do not squash the commit which is the head in your repository. This will let Travis and GitHub Actions CIs recreate Docker images used during the build before the build. Otherwise the not-updated Docker image will be pulled from $DOCKER_REPO and used during the build on Travis and GitHub Actions CIs. pmdk-1.13.1/utils/docker/arm64.blacklist0000664000000000000000000000011714435627501016470 0ustar rootrootblk_pool_lock log_pool_lock obj_basic_integration obj_pool_lock libpmempool_rm pmdk-1.13.1/utils/docker/build-CI.sh0000775000000000000000000001131014435627501015571 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # build-CI.sh - runs a Docker container from a Docker image with environment # prepared for building PMDK project and starts building PMDK. # # This script is used for building PMDK on project's CIs. # set -e source $(dirname $0)/set-ci-vars.sh source $(dirname $0)/set-vars.sh if [[ -z "$OS" || -z "$OS_VER" || -z "$IMG_VER" ]]; then echo "ERROR: The variables OS, OS_VER and IMG_VER have to be set properly " \ "(eg. OS=ubuntu, OS_VER=16.04, IMG_VER=1.10)." exit 1 fi if [[ -z "$HOST_WORKDIR" ]]; then echo "ERROR: The variable HOST_WORKDIR has to contain a path to " \ "the root of the PMDK project on the host machine" exit 1 fi if [[ -z "$TEST_BUILD" ]]; then TEST_BUILD=all fi imageName=${DOCKER_REPO}:${IMG_VER}-${OS}-${OS_VER}-${CI_CPU_ARCH} containerName=pmdk-${OS}-${OS_VER} if [[ $MAKE_PKG -eq 0 ]] ; then command="./run-build.sh"; fi if [[ $MAKE_PKG -eq 1 ]] ; then command="./run-build-package.sh"; fi if [[ $COVERAGE -eq 1 ]] ; then command="./run-coverage.sh"; fi if [[ $BANDIT -eq 1 ]] ; then command="./run-bandit.sh"; fi if [[ "$COVERITY" -eq 1 ]]; then command="./run-coverity.sh"; fi if [ -n "$DNS_SERVER" ]; then DNS_SETTING=" --dns=$DNS_SERVER "; fi if [[ -f $CI_FILE_SKIP_BUILD_PKG_CHECK ]]; then BUILD_PACKAGE_CHECK=n; else BUILD_PACKAGE_CHECK=y; fi if [ -z "$NDCTL_ENABLE" ]; then ndctl_enable=; else ndctl_enable="--env NDCTL_ENABLE=$NDCTL_ENABLE"; fi if [[ $UBSAN -eq 1 ]]; then for x in C CPP LD; do declare EXTRA_${x}FLAGS=-fsanitize=undefined; done; fi # Only run auto doc update on push events on "upstream" repo if [[ "${CI_EVENT_TYPE}" != "push" || "${CI_REPO_SLUG}" != "${GITHUB_REPO}" ]]; then AUTO_DOC_UPDATE=0 echo "Skipping auto doc update" fi # Check if we are running on a CI (Travis or GitHub Actions) [ -n "$GITHUB_ACTIONS" -o -n "$TRAVIS" ] && CI_RUN="YES" || CI_RUN="NO" # We have a blacklist only for ppc64le and aarch64 arch if [[ "$CI_CPU_ARCH" == ppc64le ]] ; then BLACKLIST_FILE=../../utils/docker/ppc64le.blacklist; fi if [[ "$CI_CPU_ARCH" == arm64 ]] ; then BLACKLIST_FILE=../../utils/docker/arm64.blacklist; fi # docker on travis + ppc64le runs inside an LXD container and for security # limits what can be done inside it, and as such, `docker run` fails with # > the input device is not a TTY # when using -t because of limited permissions to /dev imposed by LXD. if [[ -n "$TRAVIS" && "$CI_CPU_ARCH" == ppc64le ]] || [[ -n "$GITHUB_ACTIONS" ]]; then TTY='' else TTY='-t' fi WORKDIR=/pmdk SCRIPTSDIR=$WORKDIR/utils/docker # Run a container with # - environment variables set (--env) # - host directory containing PMDK source mounted (-v) # - a tmpfs /tmp with the necessary size and permissions (--tmpfs)* # - working directory set (-w) # # * We need a tmpfs /tmp inside docker but we cannot run it with --privileged # and do it from inside, so we do using this docker-run option. # By default --tmpfs add nosuid,nodev,noexec to the mount flags, we don't # want that and just to make sure we add the usually default rw,relatime just # in case docker change the defaults. docker run --rm --name=$containerName -i $TTY \ --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \ $DNS_SETTING \ --env http_proxy=$http_proxy \ --env https_proxy=$https_proxy \ --env AUTO_DOC_UPDATE=$AUTO_DOC_UPDATE \ --env CC=$PMDK_CC \ --env CXX=$PMDK_CXX \ --env VALGRIND=$VALGRIND \ --env EXTRA_CFLAGS=$EXTRA_CFLAGS \ --env EXTRA_CXXFLAGS=$EXTRA_CXXFLAGS \ --env EXTRA_LDFLAGS=$EXTRA_LDFLAGS \ --env TEST_BUILD=$TEST_BUILD \ --env WORKDIR=$WORKDIR \ --env EXPERIMENTAL=$EXPERIMENTAL \ --env BUILD_PACKAGE_CHECK=$BUILD_PACKAGE_CHECK \ --env SCRIPTSDIR=$SCRIPTSDIR \ --env TRAVIS=$TRAVIS \ --env CI_COMMIT_RANGE=$CI_COMMIT_RANGE \ --env CI_COMMIT=$CI_COMMIT \ --env CI_REPO_SLUG=$CI_REPO_SLUG \ --env CI_BRANCH=$CI_BRANCH \ --env CI_EVENT_TYPE=$CI_EVENT_TYPE \ --env DOC_UPDATE_GITHUB_TOKEN=$DOC_UPDATE_GITHUB_TOKEN \ --env COVERITY_SCAN_TOKEN=$COVERITY_SCAN_TOKEN \ --env COVERITY_SCAN_NOTIFICATION_EMAIL=$COVERITY_SCAN_NOTIFICATION_EMAIL \ --env FAULT_INJECTION=$FAULT_INJECTION \ --env GITHUB_ACTIONS=$GITHUB_ACTIONS \ --env GITHUB_HEAD_REF=$GITHUB_HEAD_REF \ --env GITHUB_REPO=$GITHUB_REPO \ --env GITHUB_REPOSITORY=$GITHUB_REPOSITORY \ --env GITHUB_REF=$GITHUB_REF \ --env GITHUB_RUN_ID=$GITHUB_RUN_ID \ --env GITHUB_SHA=$GITHUB_SHA \ --env GITHUB_SERVER_URL=$GITHUB_SERVER_URL \ --env CI_RUN=$CI_RUN \ --env SRC_CHECKERS=$SRC_CHECKERS \ --env BLACKLIST_FILE=$BLACKLIST_FILE \ --env BANDIT=$BANDIT \ $ndctl_enable \ --tmpfs /tmp:rw,relatime,suid,dev,exec,size=6G \ -v $HOST_WORKDIR:$WORKDIR \ -v /etc/localtime:/etc/localtime \ -w $SCRIPTSDIR \ $imageName $command pmdk-1.13.1/utils/docker/test_package/0000775000000000000000000000000014435627501016300 5ustar rootrootpmdk-1.13.1/utils/docker/test_package/.gitignore0000664000000000000000000000001514435627501020264 0ustar rootroottest_package pmdk-1.13.1/utils/docker/test_package/test_package.c0000664000000000000000000000133714435627501021102 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2018, Intel Corporation */ #include #include #include #define LAYOUT_NAME "test" struct my_root { int foo; }; int main(int argc, char *argv[]) { if (argc < 2) { printf("usage: %s file-name\n", argv[0]); return 1; } const char *path = argv[1]; PMEMobjpool *pop = pmemobj_create(path, LAYOUT_NAME, PMEMOBJ_MIN_POOL, S_IWUSR | S_IRUSR); if (pop == NULL) { printf("failed to create pool\n"); return 1; } PMEMoid root = pmemobj_root(pop, sizeof(struct my_root)); struct my_root *rootp = pmemobj_direct(root); rootp->foo = 10; pmemobj_persist(pop, &rootp->foo, sizeof(rootp->foo)); pmemobj_close(pop); return 0; } pmdk-1.13.1/utils/docker/test_package/Makefile0000664000000000000000000000107114435627501017737 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018, Intel Corporation check_package = $(shell pkg-config $(1) && echo y || echo n) HAS_LIBPMEMOBJ := $(call check_package, libpmemobj --atleast-version $(LIBPMEMOBJ_MIN_VERSION) --print-errors) LIBS += $(shell pkg-config --libs libpmemobj) CFLAGS += $(shell pkg-config --cflags libpmemobj) ifeq ($(HAS_LIBPMEMOBJ),n) $(error libpmemobj(version >= $(LIBPMEMOBJ_MIN_VERSION)) is missing) endif test_package: test_package.c $(CC) test_package.c $(LIBS) $(CFLAGS) -o $@ clean: rm -f test_package .PHONY: clean pmdk-1.13.1/utils/docker/test_package/test-packages-installation.py0000664000000000000000000001413714435627501024112 0ustar rootroot#!usr/bin/env python3 # # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2023, Intel Corporation """ This module includes tests for PMDK packages installation. Tests check: -compatibility of the version of installed packages from PMDK library with the current version of PMDK library, -if all packages from PMDK library are installed. Required arguments: -r the PMDK library root path. """ from os import listdir, path, linesep, walk from subprocess import check_output import distro import unittest import xmlrunner import sys import re NO_PKG_CONFIGS = ('pmdk', 'pmempool', 'daxio', 'pmreorder', 'rpmemd') PMDK_TOOLS = ('pmempool', 'daxio', 'pmreorder', 'rpmemd') PMDK_VERSION = '' SYSTEM_ARCHITECTURE = '' PMDK_PATH = '' def get_package_version_and_system_architecture(): """ Returns packages version and system architecture from names of directories from pkg directory. """ global PMDK_VERSION global SYSTEM_ARCHITECTURE os_distro = distro.id() if os_distro != 'ubuntu': pkg_directory = path.join(PMDK_PATH, 'rpm') for elem in listdir(pkg_directory): if '.src.rpm' in elem: # looks for the version number of package in package name PMDK_VERSION = re.search( r'[\s]*pmdk-([\S]+).src.rpm', elem).group(1) else: SYSTEM_ARCHITECTURE = elem elif os_distro == 'ubuntu': pkg_directory = path.join(PMDK_PATH, 'dpkg') for elem in listdir(pkg_directory): if '.changes' in elem: # looks for the version number of package in package name PMDK_VERSION = re.search(r'pmdk_*(.+)_(.+).changes', elem).group(1) SYSTEM_ARCHITECTURE = re.search(r'pmdk_*(.+)_(.+).changes', elem).group(2) def get_libraries_names(packages_path, split_param): """ Returns names of elements, for which are installed packages from PMDK library. """ libraries_names = [re.split(split_param, elem)[0] for elem in listdir(packages_path) if PMDK_VERSION in elem] for i in range(len(libraries_names)): if libraries_names[i].endswith('1'): libraries_names[i] = libraries_names[i].replace('1', '') return set(libraries_names) def get_not_installed_packages(packages_path, so_path, split_param): """ Returns names of packages from PMDK library, which are not installed. """ def is_installed(elem): return elem in PMDK_TOOLS and elem in listdir('/usr/bin/') or\ elem == "pmdk" or elem + '.so' in listdir(so_path) elements = get_libraries_names(packages_path, split_param) not_installed_packages = [] for elem in elements: if not is_installed(elem): not_installed_packages.append(elem) return not_installed_packages def get_incompatible_packages(packages_path, pkgconfig_directory, split_param): """ Returns names of packages from PMDK library, which are not compatible with the current version of PMDK library. """ incompatibe_packages = [] libraries = get_libraries_names(packages_path, split_param) - set(NO_PKG_CONFIGS) for library in libraries: with open(pkgconfig_directory + library + '.pc') as f: out = f.readlines() for line in out: if 'version=' in line: version = line.split('=')[1].strip(linesep) if not version in PMDK_VERSION.replace('~', '-'): incompatibe_packages.append(library) return incompatibe_packages class TestBuildPackages(unittest.TestCase): def test_compatibility_of_version_of_installed_packages(self): """ Checks if the version of installed packages is correct. """ incompatible_packages = get_incompatible_packages(packages_path, pkgconfig_directory, split_param) error_msg = linesep + 'List of incompatible packages: ' for package in incompatible_packages: error_msg += linesep + package self.assertFalse(incompatible_packages, error_msg) def test_correctness_of_installed_packages(self): """ Checks if all packages from PMDK library are installed. """ not_installed_packages = get_not_installed_packages(packages_path, so_path, split_param) error_msg = linesep + 'List of not installed packages: ' for package in not_installed_packages: error_msg += linesep + package self.assertFalse(not_installed_packages, error_msg) def parse_argument(argument_option): """ Parses an option from the command line. """ index = sys.argv.index(argument_option) try: argument = sys.argv[index+1] except IndexError: print('ERROR: Invalid argument!') print(__doc__) print(unittest.main.__doc__) else: sys.argv.pop(index) sys.argv.pop(index) return argument if __name__ == '__main__': if '-h' in sys.argv or '--help' in sys.argv: print(__doc__) unittest.main() elif '-r' in sys.argv: PMDK_PATH = parse_argument('-r') get_package_version_and_system_architecture() os_distro = distro.id() if os_distro != 'ubuntu': packages_path = path.join(PMDK_PATH, 'rpm', SYSTEM_ARCHITECTURE) pkgconfig_directory = '/usr/lib64/pkgconfig/' so_path = '/usr/lib64/' split_param = '-' elif os_distro == 'ubuntu': packages_path = path.join(PMDK_PATH, 'dpkg') pkgconfig_directory = '/lib/x86_64-linux-gnu/pkgconfig/' so_path = '/lib/x86_64-linux-gnu/' split_param = '[-_]' if PMDK_VERSION == '' or SYSTEM_ARCHITECTURE == '': sys.exit("FATAL ERROR: command 'make rpm/dpkg' was not done correctly") unittest.main( testRunner=xmlrunner.XMLTestRunner(output='test-reports'), # these make sure that some options that are not applicable # remain hidden from the help menu. failfast=False, buffer=False, catchbreak=False) else: print(__doc__) print(unittest.main.__doc__) pmdk-1.13.1/utils/docker/test_package/README0000664000000000000000000000052714435627501017164 0ustar rootrootPersistent Memory Development Kit This is utils/docker/test_package/README. This directory contains: - Simple application which uses libpmemobj. It can be used to test whether libpmemobj was installed properly. - Test python modules with various checks for PMDK packages. Each module contains a detailed description of its contents. pmdk-1.13.1/utils/docker/test_package/test-built-packages.py0000664000000000000000000002426614435627501022534 0ustar rootroot#!usr/bin/env python3 # # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2023, Intel Corporation """ This module includes tests that check if all packages in PMDK library are built correctly. Tests check: -if all required packages are built, -if built packages are consistent with names of libraries read from .so files and other elements (tools and "PMDK"). Required arguments: -r the PMDK library root path. Optional arguments: --skip-daxio to be set if daxio was not built (e.g., if NDCTL_ENABLE was set to 'n') """ from os import listdir, path, linesep from collections import namedtuple import distro import unittest import xmlrunner import sys import re PACKAGES_INFO = namedtuple('packages_info', 'basic devel debug debuginfo debug_debuginfo') PMDK_VERSION = '' SYSTEM_ARCHITECTURE = '' def get_package_version_and_system_architecture(pmdk_path): """ Returns packages version and system architecture from names of directories from packages directory. """ os_distro=distro.id() if os_distro != 'ubuntu': pkg_directory = path.join(pmdk_path, 'rpm') else: pkg_directory = path.join(pmdk_path, 'dpkg') version = '' architecture = '' for elem in listdir(pkg_directory): if os_distro != 'ubuntu': if '.src.rpm' in elem: # looks for the version number of rpm package in rpm package name version = re.search(r'[\s]*pmdk-([\S]+).src.rpm', elem).group(1) else: architecture = elem else: if '.changes' in elem: # looks for the version number of packages in package name version = re.search(r'pmdk_*(.+)_(.+).changes', elem).group(1) architecture = re.search(r'pmdk_*(.+)_(.+).changes', elem).group(2) return version, architecture def get_built_packages(pmdk_pkg_path): """ Returns built packages. """ packages = listdir(pmdk_pkg_path) return packages def get_libraries_names_from_so_files(pmdk_path, is_pmdk_debuginfo): """ Returns names of libraries from .so files, and information which packages should be built for individual libraries. """ libraries_from_so_files = dict() path_to_so_files = path.join(pmdk_path, 'src', 'nondebug') for elem in listdir(path_to_so_files): if elem.endswith('.so') and elem.startswith('lib'): library_name = elem.split('.')[0] if is_pmdk_debuginfo: libraries_from_so_files[library_name] =\ PACKAGES_INFO(basic=True, devel=True, debug=True, debuginfo=False, debug_debuginfo=False) else: libraries_from_so_files[library_name] =\ PACKAGES_INFO(basic=True, devel=True, debug=True, debuginfo=True, debug_debuginfo=True) return libraries_from_so_files def get_names_of_packages(packages_info): """ Returns names of packages, that should be built. """ packages = [] pkg_ext = '' separator = '' os_distro=distro.id() if os_distro != 'ubuntu': types = ['-', '-debug-', '-devel-', '-debuginfo-', '-debug-debuginfo-'] pkg_ext = '.rpm' separator ='.' elif os_distro == 'ubuntu': types = ['_', '-dev_'] pkg_ext = '.deb' separator = '_' for elem in packages_info: sets_of_information = zip(packages_info[elem], types) for kit in sets_of_information: if kit[0] and os_distro == 'opensuse': if elem.startswith('lib') and (kit[1] == '-' or kit[1] == '-debuginfo-'): package_name = elem + str(1) + kit[1] + PMDK_VERSION + separator + \ SYSTEM_ARCHITECTURE + pkg_ext packages.append(package_name) elif kit[0]: package_name = elem + kit[1] + PMDK_VERSION + separator + \ SYSTEM_ARCHITECTURE + pkg_ext packages.append(package_name) elif kit[0] and not os_distro == 'opensuse': package_name = elem + kit[1] + PMDK_VERSION + separator +\ SYSTEM_ARCHITECTURE + pkg_ext packages.append(package_name) return packages def check_existence_of_pmdk_debuginfo_package(pmdk_debuginfo_package_name, built_packages): """ Checks if 'pmdk-debuginfo' package is built """ is_pmdk_debuginfo_package = False if pmdk_debuginfo_package_name in built_packages: is_pmdk_debuginfo_package = True return is_pmdk_debuginfo_package def find_missing_packages(packages_path, pmdk_path, pmdk_debuginfo_package_name): """ Checks if names of built packages are the same as names of packages, which should be built and returns missing packages. Tools are taken into account. """ built_packages = get_built_packages(packages_path) is_pmdk_debuginfo =\ check_existence_of_pmdk_debuginfo_package(pmdk_debuginfo_package_name, built_packages) tools = { 'pmempool': PACKAGES_INFO(basic=True, devel=False, debug=False, debuginfo=True, debug_debuginfo=False), 'pmreorder': PACKAGES_INFO(basic=True, devel=False, debug=False, debuginfo=False, debug_debuginfo=False) } if not skip_daxio: tools['daxio'] = PACKAGES_INFO(basic=True, devel=False, debug=False, debuginfo=True, debug_debuginfo=False) tools_packages = get_names_of_packages(tools) missing_tools_packages = [ elem for elem in tools_packages if elem not in built_packages] libraries = get_libraries_names_from_so_files(pmdk_path, is_pmdk_debuginfo) library_packages = get_names_of_packages(libraries) missing_library_packages = [ elem for elem in library_packages if elem not in built_packages] missing_packages = missing_library_packages + missing_tools_packages return missing_packages def find_missing_libraries_and_tools(packages_path, pmdk_path, pmdk_debuginfo_package_name, library_name_pattern): """ Checks if names of functions from .so files are the same as names of functions extracted from built packages and returns missing functions. Others pkg (tools and "PMDK") are taken into account. """ others_pkg = ['pmempool', 'daxio', 'pmdk', 'pmreorder'] built_packages = get_built_packages(packages_path) is_pmdk_debuginfo =\ check_existence_of_pmdk_debuginfo_package(pmdk_debuginfo_package_name, built_packages) libraries = get_libraries_names_from_so_files(pmdk_path, is_pmdk_debuginfo) missing_elements = [] # looks for the name of library/others in package name for elem in listdir(packages_path): library_name = re.search(library_name_pattern, elem).group(1) if library_name.endswith('1'): library_name = library_name.replace('1','') if library_name not in libraries.keys() and library_name not in\ others_pkg and library_name not in missing_elements: missing_elements.append(library_name) return missing_elements def parse_argument(argument_option): """ Parses an option from the command line. """ index = sys.argv.index(argument_option) try: argument = sys.argv[index+1] except IndexError: print('ERROR: Invalid argument!') print(__doc__) print(unittest.main.__doc__) else: sys.argv.pop(index) sys.argv.pop(index) return argument class TestBuildPackages(unittest.TestCase): def test_completeness_of_built_packages(self): """ Checks if all packages are built. """ missing_packages =\ find_missing_packages(packages_path, pmdk_path, pmdk_debuginfo_package_name) error_msg = linesep + 'List of missing packages:' for package in missing_packages: error_msg += linesep + package self.assertFalse(missing_packages, error_msg) def test_completeness_of_name_of_libraries_and_tools(self): """ Checks if names of functions from .so files and other elements (tools and "PMDK") are the same as functions/other elements extracted from the name of built packages. """ os_distro=distro.id() missing_elements =\ find_missing_libraries_and_tools(packages_path, pmdk_path, pmdk_debuginfo_package_name,library_name_pattern) error_msg = linesep +\ 'List of missing libraries and other elements (tools and "PMDK"):' for elem in missing_elements: error_msg += linesep + elem self.assertFalse(missing_elements, error_msg) if __name__ == '__main__': path_argument = '-r' daxio_build_argument = "--skip-daxio" skip_daxio = False if '-h' in sys.argv or '--help' in sys.argv: print(__doc__) unittest.main() elif path_argument in sys.argv: pmdk_path = parse_argument(path_argument) if daxio_build_argument in sys.argv: skip_daxio = True index = sys.argv.index(daxio_build_argument) sys.argv.pop(index) if pmdk_path: PMDK_VERSION, SYSTEM_ARCHITECTURE =\ get_package_version_and_system_architecture(pmdk_path) os_distro=distro.id() if os_distro != 'ubuntu': packages_path = path.join(pmdk_path, 'rpm', SYSTEM_ARCHITECTURE) pmdk_debuginfo_package_name = 'pmdk-debuginfo-' + PMDK_VERSION + '.' + SYSTEM_ARCHITECTURE + '.rpm' library_name_pattern = r'[\s]*([2a-zA-Z+\d]+)-' else: packages_path = path.join(pmdk_path, 'dpkg') pmdk_debuginfo_package_name = 'pmdk-debuginfo-' + PMDK_VERSION + '.' + '.deb' library_name_pattern = r'^([2a-zA-Z]+)[-_].*$' unittest.main( testRunner=xmlrunner.XMLTestRunner(output='test-reports'), # these make sure that some options that are not applicable # remain hidden from the help menu. failfast=False, buffer=False, catchbreak=False) else: print(__doc__) print(unittest.main.__doc__) pmdk-1.13.1/utils/docker/run-build-package.sh0000775000000000000000000000406514435627501017504 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # run-build-package.sh - is called inside a Docker container; prepares # the environment and starts a build of PMDK project. # set -e # Prepare build environment ./prepare-for-build.sh # Create fake tag, so that package has proper 'version' field git config user.email "test@package.com" git config user.name "test package" git tag -a 1.4.99 -m "1.4" HEAD~1 || true echo "## Build package (and run basic tests)" pushd $WORKDIR export PCHECK_OPTS="-j2 BLACKLIST_FILE=${BLACKLIST_FILE}" make -j$(nproc) $PACKAGE_MANAGER echo "## Build PMDK once more (clobber from packaging process cleared out some required files)" make -j$(nproc) echo "## Test built packages" [ "$NDCTL_ENABLE" == "n" ] && extra_params="--skip-daxio" || extra_params="" python3 $SCRIPTSDIR/test_package/test-built-packages.py -r $(pwd) ${extra_params} echo "## Install packages" if [[ "$PACKAGE_MANAGER" == "dpkg" ]]; then pushd $PACKAGE_MANAGER echo $USERPASS | sudo -S dpkg --install *.deb else RPM_ARCH=$(uname -m) pushd $PACKAGE_MANAGER/$RPM_ARCH echo $USERPASS | sudo -S rpm --install *.rpm fi popd echo "## Test installed packages" python3 $SCRIPTSDIR/test_package/test-packages-installation.py -r $(pwd) echo "## Compile and run standalone test" pushd $SCRIPTSDIR/test_package make -j$(nproc) LIBPMEMOBJ_MIN_VERSION=1.4 ./test_package testfile1 echo "## Use pmreorder installed in the system" pmreorder_version="$(pmreorder -v)" pmreorder_pattern="pmreorder\.py .+$" (echo "$pmreorder_version" | grep -Ev "$pmreorder_pattern") && echo "pmreorder version failed" && exit 1 touch testfile2 touch logfile1 pmreorder -p testfile2 -l logfile1 popd echo "## Run tests (against PMDK installed in the system)" pushd $WORKDIR/src/test make -j$(nproc) clobber make -j$(nproc) # Prepare test config once more. Now, with path to PMDK set in the OS # (rather than in the git tree) - for testing packages installed in the system. $SCRIPTSDIR/configure-tests.sh PKG ./RUNTESTS -t check popd popd pmdk-1.13.1/utils/docker/set-ci-vars.sh0000775000000000000000000000511214435627501016341 0ustar rootroot#!/usr/bin/env bash # # SPDX-License-Identifier: BSD-3-Clause # Copyright 2020, Intel Corporation # # set-ci-vars.sh -- set CI variables common for both: # Travis and GitHub Actions CIs # set -e # set version of Docker images (IMG_VER) source $(dirname ${BASH_SOURCE[0]})/images/set-images-version.sh function get_commit_range_from_last_merge { # get commit id of the last merge LAST_MERGE=$(git log --merges --pretty=%H -1) LAST_COMMIT=$(git log --pretty=%H -1) if [ "$LAST_MERGE" == "$LAST_COMMIT" ]; then # GitHub Actions commits its own merge in case of pull requests # so the first merge commit has to be skipped. LAST_MERGE=$(git log --merges --pretty=%H -2 | tail -n1) fi if [ "$LAST_MERGE" == "" ]; then # possible in case of shallow clones # or new repos with no merge commits yet # - pick up the first commit LAST_MERGE=$(git log --pretty=%H | tail -n1) fi COMMIT_RANGE="$LAST_MERGE..HEAD" # make sure it works now if ! git rev-list $COMMIT_RANGE >/dev/null; then COMMIT_RANGE="" fi echo $COMMIT_RANGE } COMMIT_RANGE_FROM_LAST_MERGE=$(get_commit_range_from_last_merge) if [ -n "$TRAVIS" ]; then CI_COMMIT=$TRAVIS_COMMIT CI_COMMIT_RANGE="${TRAVIS_COMMIT_RANGE/.../..}" CI_BRANCH=$TRAVIS_BRANCH CI_EVENT_TYPE=$TRAVIS_EVENT_TYPE CI_REPO_SLUG=$TRAVIS_REPO_SLUG # CI_COMMIT_RANGE is usually invalid for force pushes - fix it when used # with non-upstream repository if [ -n "$CI_COMMIT_RANGE" -a "$CI_REPO_SLUG" != "$GITHUB_REPO" ]; then if ! git rev-list $CI_COMMIT_RANGE; then CI_COMMIT_RANGE=$COMMIT_RANGE_FROM_LAST_MERGE fi fi case "$TRAVIS_CPU_ARCH" in "amd64") CI_CPU_ARCH="x86_64" ;; *) CI_CPU_ARCH=$TRAVIS_CPU_ARCH ;; esac elif [ -n "$GITHUB_ACTIONS" ]; then CI_COMMIT=$GITHUB_SHA CI_COMMIT_RANGE=$COMMIT_RANGE_FROM_LAST_MERGE CI_BRANCH=$(echo $GITHUB_REF | cut -d'/' -f3) CI_REPO_SLUG=$GITHUB_REPOSITORY CI_CPU_ARCH="x86_64" # GitHub Actions supports only x86_64 case "$GITHUB_EVENT_NAME" in "schedule") CI_EVENT_TYPE="cron" ;; *) CI_EVENT_TYPE=$GITHUB_EVENT_NAME ;; esac else CI_COMMIT=$(git log --pretty=%H -1) CI_COMMIT_RANGE=$COMMIT_RANGE_FROM_LAST_MERGE CI_CPU_ARCH="x86_64" fi export CI_COMMIT=$CI_COMMIT export CI_COMMIT_RANGE=$CI_COMMIT_RANGE export CI_BRANCH=$CI_BRANCH export CI_EVENT_TYPE=$CI_EVENT_TYPE export CI_REPO_SLUG=$CI_REPO_SLUG export CI_CPU_ARCH=$CI_CPU_ARCH export IMG_VER=$IMG_VER echo CI_COMMIT=$CI_COMMIT echo CI_COMMIT_RANGE=$CI_COMMIT_RANGE echo CI_BRANCH=$CI_BRANCH echo CI_EVENT_TYPE=$CI_EVENT_TYPE echo CI_REPO_SLUG=$CI_REPO_SLUG echo CI_CPU_ARCH=$CI_CPU_ARCH echo IMG_VER=$IMG_VER pmdk-1.13.1/utils/docker/pull-or-rebuild-image.sh0000775000000000000000000000737514435627501020317 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # pull-or-rebuild-image.sh - rebuilds the Docker image used in the # current build (if necessary) or pulls it from the $DOCKER_REPO. # # Usage: pull-or-rebuild-image.sh [rebuild|pull] # # If Docker was rebuilt and all requirements are fulfilled (more details in # push_image function below) image will be pushed to the $DOCKER_REPO. # # The script rebuilds the Docker image if: # 1. the Dockerfile for the current OS version (Dockerfile.${OS}-${OS_VER}) # or any .sh script in the Dockerfiles directory were modified and committed, or # 2. "rebuild" param was passed as a first argument to this script. # # The script pulls the Docker image if: # 1. it does not have to be rebuilt (based on committed changes), or # 2. "pull" param was passed as a first argument to this script. # set -e source $(dirname $0)/set-ci-vars.sh source $(dirname $0)/set-vars.sh # Path to directory with Dockerfiles and image building scripts images_dir_name=images base_dir=utils/docker/$images_dir_name if [[ -z "$OS" || -z "$OS_VER" ]]; then echo "ERROR: The variables OS and OS_VER have to be set properly " \ "(eg. OS=ubuntu, OS_VER=22.04)." exit 1 fi if [[ -z "${DOCKER_REPO}" ]]; then echo "ERROR: DOCKER_REPO environment variable is not set " \ "(e.g. \"//\")." exit 1 fi function build_image() { echo "Building the Docker image for the Dockerfile.${OS}-${OS_VER}" pushd $images_dir_name ./build-image.sh ${OS}-${OS_VER} ${CI_CPU_ARCH} popd } function pull_image() { echo "Pull the image from the DOCKER_REPO." docker pull ${DOCKER_REPO}:${IMG_VER}-${OS}-${OS_VER}-${CI_CPU_ARCH} } function push_image { # Check if the image has to be pushed to the DOCKER_REPO: # - only upstream (not forked) repository, # - stable-*, devel-*, or master branch, # - not a pull_request event, # - and PUSH_IMAGE flag was set for current build. if [[ "${CI_REPO_SLUG}" == "${GITHUB_REPO}" \ && (${CI_BRANCH} == stable-* || ${CI_BRANCH} == devel-* || ${CI_BRANCH} == master) \ && ${CI_EVENT_TYPE} != "pull_request" \ && ${PUSH_IMAGE} == "1" ]] then echo "The image will be pushed to the Container Registry: ${DOCKER_REPO}" pushd ${images_dir_name} ./push-image.sh popd else echo "Skip pushing the image to the ${DOCKER_REPO}." fi } # If "rebuild" or "pull" are passed to the script as param, force rebuild/pull. if [[ "${1}" == "rebuild" ]]; then build_image push_image exit 0 elif [[ "${1}" == "pull" ]]; then pull_image exit 0 fi # # Determine if we need to rebuild the image or just pull it from # the DOCKER_REPO, based on committed changes. # # Find all the commits for the current build if [ -n "$CI_COMMIT_RANGE" ]; then commits=$(git rev-list $CI_COMMIT_RANGE) else commits=$CI_COMMIT fi echo "Commits in the commit range:" for commit in $commits; do echo $commit; done echo "Files modified within the commit range:" files=$(for commit in $commits; do git diff-tree --no-commit-id --name-only \ -r $commit; done | sort -u) for file in $files; do echo $file; done # Check if committed file modifications require the Docker image to be rebuilt for file in $files; do # Check if modified files are relevant to the current build if [[ $file =~ ^($base_dir)\/Dockerfile\.($OS)-($OS_VER)$ ]] \ || [[ $file =~ ^($base_dir)\/.*\.sh$ ]] then build_image push_image if [[ $PUSH_IMAGE == "1" ]] then echo "Skip build package check if image has to be pushed" touch $CI_FILE_SKIP_BUILD_PKG_CHECK fi exit 0 fi done # Getting here means rebuilding the Docker image isn't required (based on changed files). # Pull the image from the $DOCKER_REPO or rebuild anyway, if pull fails. if ! pull_image; then build_image push_image fi pmdk-1.13.1/utils/docker/build-local.sh0000775000000000000000000000761714435627501016407 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2017-2023, Intel Corporation # # build-local.sh - runs a Docker container from a Docker image with environment # prepared for building PMDK project and starts building PMDK. # # This script is for building PMDK locally (not on CI). # # Notes: # - run this script from its location or set the variable 'HOST_WORKDIR' to # where the root of the PMDK project is on the host machine. # - set variables 'OS' and 'OS_VER' properly to a system you want to build PMDK # on (for proper values take a look on the list of Dockerfiles at the # utils/docker/images directory), eg. OS=ubuntu, OS_VER=16.04. # - set 'KEEP_TEST_CONFIG' variable to 1 if you do not want the tests to be # reconfigured (your current test configuration will be preserved and used). # - tests with Device Dax are not supported by pcheck yet, so do not provide # these devices in your configuration. # set -e # Environment variables that can be customized (default values are after dash): export KEEP_CONTAINER=${KEEP_CONTAINER:-0} export KEEP_TEST_CONFIG=${KEEP_TEST_CONFIG:-0} export TEST_BUILD=${TEST_BUILD:-all} export MAKE_PKG=${MAKE_PKG:-0} export EXTRA_CFLAGS=${EXTRA_CFLAGS} export EXTRA_CXXFLAGS=${EXTRA_CXXFLAGS:-} export PMDK_CC=${PMDK_CC:-gcc} export PMDK_CXX=${PMDK_CXX:-g++} export EXPERIMENTAL=${EXPERIMENTAL:-n} export VALGRIND=${VALGRIND:-1} export DOCKER_REPO=${DOCKER_REPO:-ghcr.io/pmem/pmdk} export GITHUB_REPO=${GITHUB_REPO:-pmem/pmdk} if [[ -z "$IMG_VER" ]]; then # set the IMG_VER variable - version of Docker images source $(dirname $0)/images/set-images-version.sh fi if [[ -z "$OS" || -z "$OS_VER" || -z "$IMG_VER" ]]; then echo "ERROR: The variables OS, OS_VER and IMG_VER have to be set " \ "(eg. OS=ubuntu, OS_VER=16.04, IMG_VER=1.10)." exit 1 fi if [[ -z "$HOST_WORKDIR" ]]; then HOST_WORKDIR=$(readlink -f ../..) fi if [[ "$KEEP_CONTAINER" != "1" ]]; then RM_SETTING=" --rm" fi imageName=${DOCKER_REPO}:${IMG_VER}-${OS}-${OS_VER}-${CI_CPU_ARCH} containerName=pmdk-${OS}-${OS_VER} if [[ $MAKE_PKG -eq 1 ]] ; then command="./run-build-package.sh" else command="./run-build.sh" fi if [ -n "$DNS_SERVER" ]; then DNS_SETTING=" --dns=$DNS_SERVER "; fi if [ -z "$NDCTL_ENABLE" ]; then ndctl_enable=; else ndctl_enable="--env NDCTL_ENABLE=$NDCTL_ENABLE"; fi WORKDIR=/pmdk SCRIPTSDIR=$WORKDIR/utils/docker # Check if we are running on a CI (Travis or GitHub Actions) [ -n "$GITHUB_ACTIONS" -o -n "$TRAVIS" ] && CI_RUN="YES" || CI_RUN="NO" echo Building ${OS}-${OS_VER} # Run a container with # - environment variables set (--env) # - host directory containing PMDK source mounted (-v) # - a tmpfs /tmp with the necessary size and permissions (--tmpfs)* # - working directory set (-w) # # * We need a tmpfs /tmp inside docker but we cannot run it with --privileged # and do it from inside, so we do using this docker-run option. # By default --tmpfs add nosuid,nodev,noexec to the mount flags, we don't # want that and just to make sure we add the usually default rw,relatime just # in case docker change the defaults. docker run --name=$containerName -ti \ --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \ $RM_SETTING \ $DNS_SETTING \ --env http_proxy=$http_proxy \ --env https_proxy=$https_proxy \ --env CC=$PMDK_CC \ --env CXX=$PMDK_CXX \ --env VALGRIND=$VALGRIND \ --env EXTRA_CFLAGS=$EXTRA_CFLAGS \ --env EXTRA_CXXFLAGS=$EXTRA_CXXFLAGS \ --env EXTRA_LDFLAGS=$EXTRA_LDFLAGS \ --env CONFIGURE_TESTS=$CONFIGURE_TESTS \ --env TEST_BUILD=$TEST_BUILD \ --env WORKDIR=$WORKDIR \ --env EXPERIMENTAL=$EXPERIMENTAL \ --env SCRIPTSDIR=$SCRIPTSDIR \ --env KEEP_TEST_CONFIG=$KEEP_TEST_CONFIG \ --env CI_RUN=$CI_RUN \ --env BLACKLIST_FILE=$BLACKLIST_FILE \ $ndctl_enable \ --tmpfs /tmp:rw,relatime,suid,dev,exec,size=6G \ -v $HOST_WORKDIR:$WORKDIR \ -v /etc/localtime:/etc/localtime \ $DAX_SETTING \ -w $SCRIPTSDIR \ $imageName $command pmdk-1.13.1/utils/docker/0001-travis-fix-travisci_build_coverity_scan.sh.patch0000664000000000000000000000162514435627501025707 0ustar rootrootFrom b5179dc4822eaab192361da05aa95d98f523960f Mon Sep 17 00:00:00 2001 From: Lukasz Dorau Date: Mon, 7 May 2018 12:05:40 +0200 Subject: [PATCH] travis: fix travisci_build_coverity_scan.sh --- travisci_build_coverity_scan.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/travisci_build_coverity_scan.sh b/travisci_build_coverity_scan.sh index ad9d4afcf..562b08bcc 100644 --- a/travisci_build_coverity_scan.sh +++ b/travisci_build_coverity_scan.sh @@ -92,8 +92,8 @@ response=$(curl \ --form description="Travis CI build" \ $UPLOAD_URL) status_code=$(echo "$response" | sed -n '$p') -if [ "$status_code" != "201" ]; then +if [ "$status_code" != "200" ]; then TEXT=$(echo "$response" | sed '$d') - echo -e "\033[33;1mCoverity Scan upload failed: $TEXT.\033[0m" + echo -e "\033[33;1mCoverity Scan upload failed: $response.\033[0m" exit 1 fi -- 2.13.6 pmdk-1.13.1/utils/docker/set-vars.sh0000775000000000000000000000035014435627501015747 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2019-2023, Intel Corporation # # set-vars.sh - set required environment variables # set -e export CI_FILE_SKIP_BUILD_PKG_CHECK=/tmp/skip_build_package_check pmdk-1.13.1/utils/docker/get-system-info.sh0000775000000000000000000000321014435627501017233 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2023, Intel Corporation # # get-system-info.sh - Script for printing system info # function system_info { echo "********** system_info **********" cat /etc/os-release | grep -oP "PRETTY_NAME=\K.*" uname -r echo "libndctl: $(pkg-config --modversion libndctl || echo 'libndctl not found')" echo "valgrind: $(pkg-config --modversion valgrind || echo 'valgrind not found')" echo "*************** installed-packages ***************" # Instructions below will return some minor errors, as they are dependent on the Linux distribution. zypper se --installed-only 2>/dev/null || true apt list --installed 2>/dev/null || true yum list installed 2>/dev/null || true echo "**********/proc/cmdline**********" cat /proc/cmdline echo "**********/proc/modules**********" cat /proc/modules echo "**********/proc/cpuinfo**********" cat /proc/cpuinfo echo "**********/proc/meminfo**********" cat /proc/meminfo echo "**********/proc/swaps**********" cat /proc/swaps echo "**********/proc/version**********" cat /proc/version echo "**********check-updates**********" # Instructions below will return some minor errors, as they are dependent on the Linux distribution. zypper list-updates 2>/dev/null || true apt-get update 2>/dev/null || true apt upgrade --dry-run 2>/dev/null || true dnf check-update 2>/dev/null || true echo "**********list-enviroment**********" env echo "******list-build-system-versions*******" gcc --version 2>/dev/null || true clang --version 2>/dev/null || true make --version 2>/dev/null || true } # Call the function above to print system info. system_info pmdk-1.13.1/utils/docker/run-doc-update.sh0000775000000000000000000000652214435627501017041 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2019-2022, Intel Corporation # # run-doc-update.sh - is called inside a Docker container to build docs in the current repository, # it checks if current branch is a 'valid' one (to only publish "merged" content, not from a PR), # and it creates a pull request with an update of our docs (on 'main' branch of pmem.github.io repo). # set -e if [[ -z "${DOC_UPDATE_GITHUB_TOKEN}" ]]; then echo "ERROR: To build documentation and upload it as a Github pull request, " \ "variable 'DOC_UPDATE_GITHUB_TOKEN' has to be provided." exit 1 fi if [[ -z "${WORKDIR}" ]]; then echo "ERROR: The variable WORKDIR has to contain a path to the root " \ "of this project." exit 1 fi BOT_NAME="pmem-bot" USER_NAME="pmem" PAGES_REPO_NAME="pmem.github.io" DOC_REPO_DIR=$(mktemp -d -t pmem_io-XXX) ARTIFACTS_DIR=$(mktemp -d -t ARTIFACTS-XXX) ORIGIN="https://${DOC_UPDATE_GITHUB_TOKEN}@github.com/${BOT_NAME}/${PAGES_REPO_NAME}" UPSTREAM="https://github.com/${USER_NAME}/${PAGES_REPO_NAME}" # Only 'master' or 'stable-*' branches are valid; determine docs location dir on gh-pages branch TARGET_BRANCH=${CI_BRANCH} if [[ "${TARGET_BRANCH}" == "master" ]]; then TARGET_DOCS_DIR="master" elif [[ ${TARGET_BRANCH} == stable-* ]]; then TARGET_DOCS_DIR=v$(echo ${TARGET_BRANCH} | cut -d"-" -f2 -s) else echo "Skipping docs build, this script should be run only on master or stable-* branches." echo "TARGET_BRANCH is set to: \'${TARGET_BRANCH}\'." exit 0 fi if [ -z "${TARGET_DOCS_DIR}" ]; then echo "ERROR: Target docs location for branch: ${TARGET_BRANCH} is not set." exit 1 fi pushd ${WORKDIR}/doc echo "Build docs and copy man & web md" make -j$(nproc) web mv ./web_linux ${ARTIFACTS_DIR} mv ./web_windows ${ARTIFACTS_DIR} mv ./generated/libs_map.yml ${ARTIFACTS_DIR} popd echo "Clone bot's pmem.io repo" git clone --depth=1 ${ORIGIN} ${DOC_REPO_DIR} pushd ${DOC_REPO_DIR} git remote add upstream ${UPSTREAM} git fetch upstream git config --local user.name ${BOT_NAME} git config --local user.email "${BOT_NAME}@intel.com" hub config --global hub.protocol https echo "Checkout new branch (based on 'main') for PR" DOCS_BRANCH_NAME="pmdk-${TARGET_DOCS_DIR}-docs-update" git checkout -B ${DOCS_BRANCH_NAME} upstream/main git clean -dfx echo "Copy content" rsync -a ${ARTIFACTS_DIR}/web_linux/ ./content/pmdk/manpages/linux/${TARGET_DOCS_DIR}/ --delete rsync -a ${ARTIFACTS_DIR}/web_windows/ ./content/pmdk/manpages/windows/${TARGET_DOCS_DIR}/ --delete \ --exclude='librpmem' \ --exclude='rpmemd' --exclude='pmreorder' \ --exclude='daxio' if [ ${TARGET_BRANCH} = "master" ]; then cp ${ARTIFACTS_DIR}/libs_map.yml data/ fi echo "Add and push changes" # git commit command may fail if there is nothing to commit. # In that case we want to force push anyway (there might be open pull request # with changes which were reverted). git add -A git commit -m "pmdk: automatic docs update for '${TARGET_BRANCH}'" && true git push -f ${ORIGIN} ${DOCS_BRANCH_NAME} echo "Make a Pull Request" # When there is already an open PR or there are no changes an error is thrown, which we ignore. GITHUB_TOKEN=${DOC_UPDATE_GITHUB_TOKEN} hub pull-request -f \ -b ${USER_NAME}:main \ -h ${BOT_NAME}:${DOCS_BRANCH_NAME} \ -m "pmdk: automatic docs update for '${TARGET_BRANCH}'" && true popd rm -rf ${DOC_REPO_DIR} rm -rf ${ARTIFACTS_DIR} exit 0 pmdk-1.13.1/utils/docker/prepare-for-build.sh0000775000000000000000000000133614435627501017527 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # # prepare-for-build.sh - is called inside a Docker container; prepares # the environment inside a Docker container for # running build of PMDK project. # set -e # This should be run only on CIs if [ "$CI_RUN" == "YES" ]; then # Make sure $WORKDIR has correct access rights # - set them to the current UID and GID echo $USERPASS | sudo -S chown -R $(id -u).$(id -g) $WORKDIR fi # Configure tests (e.g. python tests) unless the current configuration # should be preserved KEEP_TEST_CONFIG=${KEEP_TEST_CONFIG:-0} if [[ "$KEEP_TEST_CONFIG" == 0 ]]; then ./configure-tests.sh $@ fi pmdk-1.13.1/utils/docker/run-bandit.sh0000775000000000000000000000077714435627501016263 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2022, Intel Corporation # # run-bandit.sh - is called inside a Docker container; runs bandit # security checker for code written in python # set -e # Get and prepare PMDK source ./prepare-for-build.sh cd $WORKDIR # set path to pmreorder tool # at the moment pmreorder is the only python tool # released in the PMDK SCAN_DIR=src/tools/pmreorder echo "Start Bandit scan" bandit --version bandit -r "$SCAN_DIR" echo "End Bandit scan" pmdk-1.13.1/utils/md2man.sh0000775000000000000000000000364314435627501014122 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2020, Intel Corporation # # # md2man.sh -- convert markdown to groff man pages # # usage: md2man.sh file template outfile # # This script converts markdown file into groff man page using pandoc. # It performs some pre- and post-processing for better results: # - uses m4 to preprocess OS-specific directives. See doc/macros.man. # - parse input file for YAML metadata block and read man page title, # section and version # - cut-off metadata block and license # - unindent code blocks # - cut-off windows and web specific parts of documentation # # If the TESTOPTS variable is set, generates a preprocessed markdown file # with the header stripped off for testing purposes. # set -e set -o pipefail filename=$1 template=$2 outfile=$3 title=`sed -n 's/^title:\ _MP(*\([A-Za-z0-9_-]*\).*$/\1/p' $filename` section=`sed -n 's/^title:.*\([0-9]\))$/\1/p' $filename` version=`sed -n 's/^date:\ *\(.*\)$/\1/p' $filename` if [ "$TESTOPTS" != "" ]; then m4 $TESTOPTS macros.man $filename | sed -n -e '/# NAME #/,$p' > $outfile else OPTS= if [ "$WIN32" == 1 ]; then OPTS="$OPTS -DWIN32" else OPTS="$OPTS -UWIN32" fi if [ "$(uname -s)" == "FreeBSD" ]; then OPTS="$OPTS -DFREEBSD" else OPTS="$OPTS -UFREEBSD" fi if [ "$WEB" == 1 ]; then OPTS="$OPTS -DWEB" mkdir -p "$(dirname $outfile)" m4 $OPTS macros.man $filename | sed -n -e '/---/,$p' > $outfile else SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(date +%s)}" COPYRIGHT=$(grep -rwI "\[comment]: <> (Copyright" $filename |\ sed "s/\[comment\]: <> (\([^)]*\))/\1/") dt=$(date -u -d "@$SOURCE_DATE_EPOCH" +%F 2>/dev/null || date -u -r "$SOURCE_DATE_EPOCH" +%F 2>/dev/null || date -u +%F) m4 $OPTS macros.man $filename | sed -n -e '/# NAME #/,$p' |\ pandoc -s -t man -o $outfile --template=$template \ -V title=$title -V section=$section \ -V date="$dt" -V version="$version" \ -V copyright="$COPYRIGHT" fi fi pmdk-1.13.1/utils/gha-runners/0000775000000000000000000000000014435627501014630 5ustar rootrootpmdk-1.13.1/utils/gha-runners/build-pmdk.sh0000775000000000000000000000107614435627501017223 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2022-2023, Intel Corporation # # build-pmdk.sh - Script for building pmdk project # FULL_PATH=$(readlink -f $(dirname ${BASH_SOURCE[0]})) PMDK_PATH="${FULL_PATH}/../.." set -eo pipefail # # build_pmdk -- build pmdk from source # function build_pmdk { echo "********** make pmdk **********" cd ${PMDK_PATH} && make -j$(nproc) clean cd ${PMDK_PATH} && make -j$(nproc) EXTRA_CFLAGS=-DUSE_VALGRIND echo "********** make pmdk test **********" cd ${PMDK_PATH}/ && make -j$(nproc) test } build_pmdk pmdk-1.13.1/utils/gha-runners/run-ras-linux.yml0000664000000000000000000002625514435627501020111 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2022, Intel Corporation # Ansible playbook which is executing PMDK RAS test: Unsafe Shutdown Local on Linux. # Requirements: # 1. Two bare metal platforms with pmem memory installed. First machine is a controller, the second one is a runner. # 2. The runner platform requires a specific HW setup: at least 4 functional DIMMs have to be present in the system. # This is due to the tests using multiple regions with namespaces to store data, # whose health is later compared and verified after an error injection and an unsafe shutdown of the system. # 3. ErrorInjectionEnabled option must be enabled in the systems BIOS (the runner only). # 4. Ssh keys exchanged between machines (optional but highly recommended). # 5. List of required packages: # - ansible ( minimum 2.10 ) - controller platform only. # - ipmitool # - ipmctl # - ndctl # - make # - cmake # - lxml # To launch the playbook, use ansible specific command on the controller machine: # ansible-playbook -i /path/to/inventory /path/to/playbook.yml -e "host=some_host_or_group ansible_user=some_user" # Where: # - -i inventory - file which defines the hosts and groups of hosts upon which commands, modules, and tasks in a playbook operate. # If no inventory file is present, the user may simply use an IP address of the host; ansible-playbook -i # - run-ras-linux.yml - name of this script. # Adding path to the script file is optional if the playbook is launched from the same directory. # - '-e' or --extra-vars - is a section where the user may add several variables used by ansible or override the existing ones on the host platform. # - host - Name of the runner platform as specified in an inventory file or an IP address. # - ansible_user - specify which user ansible will use to connect to the runner and run tasks. # Warning! If the user won't exchange ssh keys between platforms, an additional variable will be needed to successfully # run the playbook: 'ansible_password' with the password of the ansible_user. # Examples: # 1. ansible-playbook -i /home/test_user/hosts.yml run-ras-linux.yml -e "host=all ansible_user=test-user" # 2. ansible-playbook -i 192.168.0.1, /home/test_user/run-ras-linux.yml -e "host=192.168.0.1 ansible_user=test-user" # Note, that the comma is needed only if no inventory file is used, as shown in the example no.2 # More information about ansible playbook command and its options can be found here: # https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html#ansible-playbook # More information about tests can be found here: # https://github.com/pmem/pmdk-tests/blob/master/README.md # More information about the config file can be found here: # https://github.com/pmem/pmdk-tests/blob/master/etc/config/README.md # The general workflow of the test is as follows: # 1. Create `AppDirectNotInterleaved` region. # 2. Create fsdaxes with filesystem for test. # 3. Clone and build `pmdk` and `pmdk-tests` repositories. # 4. Mount fsdaxes by UUID to proper directories. # 5. Create config file for RAS USL. # 6. Run phase I of the test. # 7. Perform power cycle by `ipmitool power cycle`. # 8. Mount fsdaxes after server goes up. # 9. Run phase II of the test. # 10. Check if phase I and phase II were successful. # # Cleanup stage contains: # 1. Remove temporary files. # # The test execution should result in two XML log files having been created on the DUT platform. # Those XMLs will be located in the /var/tmp/raslogs directory. # - hosts: '{{ host }}' vars: bash_path: /bin/bash working_dir_path: "/var/tmp/rasUnsafeShutdown" logs_dir_path: "/var/tmp/raslogs" test_dir_path: "{{ working_dir_path }}/testDir" pmdk_install_dir_path: "{{ working_dir_path }}/pmdkInstallTarget" pmdk_tests_build_dir_path: "{{ working_dir_path }}/pmdkTestsBuild" workspace_dirs: - "{{ working_dir_path }}" - "{{ test_dir_path }}" - "{{ pmdk_install_dir_path }}" - "{{ pmdk_tests_build_dir_path }}" - "{{ logs_dir_path }}" repo: pmdk: url: https://github.com/pmem/pmdk.git branch: master target_dir: "{{ working_dir_path }}/pmdk" pmdk_tests: url: https://github.com/pmem/pmdk-tests branch: master target_dir: "{{ working_dir_path }}/pmdk-tests" tasks: - name: "Namespace and pmem configuration" block: - name: "Unmount all pmems" shell: umount /dev/pmem* || true - name: "Destroy current namespaces" shell: | ndctl disable-namespace all ndctl destroy-namespace all # Create AppDirectNotInterleaved regions on all available capacity. # Force option is necessary to avoid needing user confirmation. - name: "Create goal in AppDirectNotInterleaved mode" shell: ipmctl create -f -goal PersistentMemoryType=AppDirectNotInterleaved - name: "Reboot machine in order to apply new AppDirectNotInterleaved goal" reboot: # Create namespaces with options: # -m (mode): fsdax. # -f (force): force to create namespaces if previous steps would fail for any reason. # -c (continue): allows to create multiple namespaces, one for each region. - name: "Create namespaces: one for each existing region" shell: ndctl create-namespace -m fsdax -f -c - name: "Store list of current namespaces" shell: ndctl list register: namespaces_list # Create variables: pmem_devices & uuids. # pmem_devices - list of pmems as seen in /dev directory or lsblk command. Example: pmem0, pmem1. # uuids - list of namespace's uuid string chains. Example: cbf59e18-ae10-499d-b681-53275c1c3706. # These lists will be later used to create filesystems, directories and mounting pmems. - name: "Use namespaces list to get pmem devices names and uuids" set_fact: pmem_devices="{{ namespaces_list.stdout | regex_findall('(pmem\d)') }}" uuids="{{ namespaces_list.stdout | regex_findall('([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})') }}" - name: "Create fs on namespaces" filesystem: fstype: ext4 dev: "/dev/{{ item }}" force: yes with_items: "{{ pmem_devices }}" - name: "Create directories for namespaces based on their uuid numbers" file: path: "/mnt/{{ item }}" state: directory with_items: "{{ uuids }}" - name: "Mount namespaces to previously added directories" mount: path: '/mnt/{{ item.0 }}' src: '/dev/{{ item.1 }}' fstype: ext4 state: mounted fstab: /tmp/tmp.fstab with_together: - "{{ uuids }}" - "{{ pmem_devices }}" - name: "Workspace configuration and test running" block: - name: "Create workspace directories" file: path: "{{ item }}" state: directory with_items: "{{ workspace_dirs }}" # The 'Main' branch is used for both repositories. - name: "Clone repositories" git: repo: "{{ item.value.url }}" dest: "{{ item.value.target_dir }}" version: "{{ item.value.branch }}" with_dict: "{{ repo }}" - name: "Build PMDK" shell: cmd: "make -j install prefix={{ pmdk_install_dir_path }}" chdir: "{{ repo.pmdk.target_dir }}" executable: "{{ bash_path }}" - name: "Configure & build pmdk-tests" shell: cmd: "{{ item }}" chdir: "{{ pmdk_tests_build_dir_path }}" executable: "{{ bash_path }}" environment: PKG_CONFIG_PATH: "{{ pmdk_install_dir_path }}/lib/pkgconfig/" with_items: - "cmake '{{ repo.pmdk_tests.target_dir }}'" - "make -j" - name: "Create template for config.xml file" lineinfile: path: '{{ pmdk_tests_build_dir_path }}/config.xml' state: present create: yes line: | {{ test_dir_path }} - name: "Add the namespace mount points to the config file" xml: path: '{{ pmdk_tests_build_dir_path }}/config.xml' xpath: /configuration/localConfiguration/dimmConfiguration pretty_print: yes add_children: - mountPoint: /mnt/{{item}} with_items: "{{ uuids }}" - name: "Run the first phase of tests" shell: cmd: PATH=$PATH:{{ pmdk_install_dir_path }}/bin ./UNSAFE_SHUTDOWN_LOCAL 1 inject all --gtest_output=xml:{{ logs_dir_path }}/phase1.xml chdir: "{{ pmdk_tests_build_dir_path }}" executable: "{{ bash_path }}" environment: LD_LIBRARY_PATH: "{{ pmdk_install_dir_path }}/lib/" register: phase_1 ignore_errors: true - name: "Print phase I output" debug: var=phase_1 - name: "Flush filesystems before power cycle" shell: "sync" # Combination of "async:1, poll:0 and ignore_errors: true" means more or less "fire and forget"; # "fire and forget" (and later waiting for connection) approach is required here because Ansible # was not handling gracefully when SSH connection was broken while restarting the platform. - name: "Perform power cycle" shell: "ipmitool power cycle" async: 1 poll: 0 ignore_errors: true # 30 seconds delay before start checking; # This is required because SSH connection is not always broken immediately which leads to false-positives while checking availability. - name: "Wait for server to come back" wait_for_connection: delay: 30 - name: "Remount namespaces after reboot" mount: path: '/mnt/{{ item.0 }}' src: '/dev/{{ item.1 }}' fstype: ext4 state: mounted fstab: /tmp/tmp.fstab with_together: - "{{ uuids }}" - "{{ pmem_devices }}" - name: "Run the second phase of tests" shell: cmd: PATH=$PATH:{{ pmdk_install_dir_path }}/bin ./UNSAFE_SHUTDOWN_LOCAL 2 cleanup all --gtest_output=xml:{{ logs_dir_path }}/phase2.xml chdir: "{{ pmdk_tests_build_dir_path }}" executable: "{{ bash_path }}" environment: LD_LIBRARY_PATH: "{{ pmdk_install_dir_path }}/lib/" register: phase_2 ignore_errors: true - name: "Print phase II output" debug: var=phase_2 # 'always' section makes sure that the temporary files are deleted regardless of the results. always: - name: "Delete temporary files after the job" file: path: "{{ item }}" state: absent with_items: - "/tmp/tmp.fstab" - "{{ working_dir_path }}" pmdk-1.13.1/utils/gha-runners/get-system-info.sh0000775000000000000000000000436614435627501020232 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2022-2023, Intel Corporation # # get-system-info.sh - Script for printing system info # function system_info { echo "********** system_info **********" cat /etc/os-release | grep -oP "PRETTY_NAME=\K.*" uname -r echo "libndctl: $(pkg-config --modversion libndctl || echo 'libndctl not found')" echo "valgrind: $(pkg-config --modversion valgrind || echo 'valgrind not found')" echo "******************** memory-info *******************" ipmctl show -dimm || true ipmctl show -topology || true echo "*************** list-existing-namespaces ***************" ndctl list -M -N echo "*************** installed-packages ***************" # Instructions below will return some minor errors, as they are dependent on the Linux distribution. zypper se --installed-only 2>/dev/null || true apt list --installed 2>/dev/null || true yum list installed 2>/dev/null || true echo "**********/proc/cmdline**********" cat /proc/cmdline echo "**********/proc/modules**********" cat /proc/modules echo "**********/proc/cpuinfo**********" cat /proc/cpuinfo echo "**********/proc/meminfo**********" cat /proc/meminfo echo "**********/proc/swaps**********" cat /proc/swaps echo "**********/proc/version**********" cat /proc/version echo "**********check-updates**********" # Instructions below will return some minor errors, as they are dependent on the Linux distribution. zypper list-updates 2>/dev/null || true apt-get update 2>/dev/null || true apt upgrade --dry-run 2>/dev/null || true dnf check-update 2>/dev/null || true echo "**********list-enviroment**********" env echo "**********list-avaialble-pmem-devices**********" ls -la /dev/dax* ls -la /dev/pmem* echo "**********list-nd-resources**********" ls -la /sys/bus/nd/devices/ndbus*/region*/resource ls -la /sys/bus/nd/devices/ndbus*/region*/dax*/resource ls -la /sys/bus/nd/devices/ndbus*/region*/pfn*/resource ls -la /sys/bus/nd/devices/ndbus*/region*/namespace*/resource ls -la /sys/bus/nd/devices/region*/deep_flush echo "******list-build-system-versions*******" gcc --version 2>/dev/null || true clang --version 2>/dev/null || true make --version 2>/dev/null || true } # Call the function above to print system info. system_info pmdk-1.13.1/utils/gha-runners/create-testconfig.sh0000775000000000000000000000307214435627501020577 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2022-2023, Intel Corporation # # create-testconfig.sh - Script for creating testconfig files for test execution. # # Default location for testconfig.sh CONF_PATH="src/test" MOUNT_POINT=${PMDK_MOUNT_POINT:-"/mnt/pmem0"} NON_PMEM_DIR="/dev/shm" # Create config file for unittests. # We are using ndctl command to gather information about devdaxes, in form known from namespace configuration. cat >${CONF_PATH}/testconfig.sh <${CONF_PATH}/testconfig.py <> /etc/magic else echo "PMDK magic already exists" fi pmdk-1.13.1/utils/libpmemobj.pc.in0000664000000000000000000000037214435627501015452 0ustar rootrootincludedir=${prefix}/include Name: libpmemobj Description: libpmemobj library from PMDK project Version: ${version} URL: https://pmem.io/pmdk Requires.private: libpmem${rasdeps} Libs: -L${libdir} -lpmemobj Libs.private: -ldl Cflags: -I${includedir} pmdk-1.13.1/utils/check-manpages0000775000000000000000000000065714435627501015203 0ustar rootroot#!/bin/bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2019-2023, Intel Corporation # check-manpages -- a tool to test against some manpage errors MANS="$*" [ -n "$MANS" ] || MANS="$(find doc -name '*.1' -o -name '*.3' -o -name '*.5' -o -name '*.7')" [ -n "$MANS" ] || { echo >&2 "No man pages given, and none found in doc/"; exit 1;} for page in $MANS;do echo $page done | xargs -P `nproc` -n1 -- utils/check-manpage pmdk-1.13.1/utils/ps_analyze.ps10000664000000000000000000000153314435627501015173 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2017, Intel Corporation # # ps_analyze -- script to analyze ps1 files # Write-Output "Starting PSScript analyzing ..." $scriptdir = Split-Path -Parent $PSCommandPath $rootdir = $scriptdir + "\.." $detected = 0 $include = @("*.ps1" ) Get-ChildItem -Path $rootdir -Recurse -Include $include | ` Where-Object { $_.FullName -notlike "*test*" } | ` ForEach-Object { $analyze_result = Invoke-ScriptAnalyzer -Path $_.FullName if ($analyze_result) { $detected = $detected + $analyze_result.Count Write-Output $_.FullName Write-Output $analyze_result } } if ($detected) { Write-Output "PSScriptAnalyzer FAILED. Issues detected: $detected" Exit 1 } else { Write-Output "PSScriptAnalyzer PASSED. No issue detected." Exit 0 } pmdk-1.13.1/utils/Makefile0000664000000000000000000000042514435627501014040 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2020, Intel Corporation rwildcard=$(strip $(foreach d,$(wildcard $1*),$(call rwildcard,$d/,$2)\ $(filter $(subst *,%,$2),$d))) SCRIPTS = $(call rwildcard,,*.sh) cstyle: ./check-shebang.sh $(SCRIPTS) .PHONY: cstyle pmdk-1.13.1/utils/check_license/0000775000000000000000000000000014435627501015156 5ustar rootrootpmdk-1.13.1/utils/check_license/.gitignore0000664000000000000000000000001614435627501017143 0ustar rootrootcheck-license pmdk-1.13.1/utils/check_license/check-headers.sh0000775000000000000000000001225314435627501020206 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2023, Intel Corporation # check-headers.sh - check copyright and license in source files SELF=$0 function usage() { echo "Usage: $SELF [-h|-v|-a]" echo " -h, --help this help message" echo " -v, --verbose verbose mode" echo " -a, --all check all files (only modified files are checked by default)" } if [ "$#" -lt 2 ]; then usage >&2 exit 2 fi SOURCE_ROOT=$1 shift LICENSE=$1 shift PATTERN=`mktemp` TMP=`mktemp` TMP2=`mktemp` TEMPFILE=`mktemp` rm -f $PATTERN $TMP $TMP2 if [ "$1" == "-h" -o "$1" == "--help" ]; then usage exit 0 fi export GIT="git -C ${SOURCE_ROOT}" $GIT rev-parse || exit 1 if [ -f $SOURCE_ROOT/.git/shallow ]; then SHALLOW_CLONE=1 echo echo "Warning: This is a shallow clone. Checking dates in copyright headers" echo " will be skipped in case of files that have no history." echo else SHALLOW_CLONE=0 fi VERBOSE=0 CHECK_ALL=0 while [ "$1" != "" ]; do case $1 in -v|--verbose) VERBOSE=1 ;; -a|--all) CHECK_ALL=1 ;; esac shift done if [ $CHECK_ALL -eq 0 ]; then CURRENT_COMMIT=$($GIT log --pretty=%H -1) MERGE_BASE=$($GIT merge-base HEAD origin/master 2>/dev/null) [ -z $MERGE_BASE ] && \ MERGE_BASE=$($GIT log --pretty="%cN:%H" | grep GitHub | head -n1 | cut -d: -f2) [ -z $MERGE_BASE -o "$CURRENT_COMMIT" = "$MERGE_BASE" ] && \ CHECK_ALL=1 fi if [ $CHECK_ALL -eq 1 ]; then echo "Checking copyright headers of all files..." GIT_COMMAND="ls-tree -r --name-only HEAD" else if [ $VERBOSE -eq 1 ]; then echo echo "Warning: will check copyright headers of modified files only," echo " in order to check all files issue the following command:" echo " $ $SELF -a" echo " (e.g.: $ $SELF $SOURCE_ROOT $LICENSE -a)" echo fi echo "Checking copyright headers of modified files only..." GIT_COMMAND="diff --name-only $MERGE_BASE $CURRENT_COMMIT" fi FILES=$($GIT $GIT_COMMAND | ${SOURCE_ROOT}/utils/check_license/file-exceptions.sh | \ grep -E -e '\.[chs]$' -e '\.[ch]pp$' -e '\.sh$' \ -e '\.py$' -e '\.link$' -e 'Makefile*' -e 'TEST*' \ -e '/common.inc$' -e '/match$' -e '/check_whitespace$' \ -e 'LICENSE$' -e 'CMakeLists.txt$' -e '\.cmake$' | \ xargs) RV=0 for file in $FILES ; do # The src_path is a path which should be used in every command except git. # git is called with -C flag so filepaths should be relative to SOURCE_ROOT src_path="${SOURCE_ROOT}/$file" [ ! -f $src_path ] && continue # ensure that file is UTF-8 encoded ENCODING=`file -b --mime-encoding $src_path` iconv -f $ENCODING -t "UTF-8" $src_path > $TEMPFILE if ! grep -q "SPDX-License-Identifier: $LICENSE" $src_path; then echo "$src_path:1: no $LICENSE SPDX tag found " >&2 RV=1 elif [[ $file == *.c ]] || [[ $file == *.cpp ]]; then if ! grep -q -e "\/\/ SPDX-License-Identifier: $LICENSE" $src_path; then echo "$src_path:1: wrong format of $LICENSE SPDX tag" >&2 RV=1 fi elif [[ $file == *.h ]] || [[ $file == *.hpp ]]; then if ! grep -q -e "\/\* SPDX-License-Identifier: $LICENSE \*\/" $src_path; then echo "$src_path:1: wrong format of $LICENSE SPDX tag" >&2 RV=1 fi elif [[ $file != LICENSE ]]; then if ! grep -q -e "# SPDX-License-Identifier: $LICENSE" $src_path; then echo "$src_path:1: wrong format of $LICENSE SPDX tag" >&2 RV=1 fi fi if [ $SHALLOW_CLONE -eq 0 ]; then $GIT log --no-merges --format="%ai %aE" -- $file | sort > $TMP else # mark the grafted commits (commits with no parents) $GIT log --no-merges --format="%ai %aE grafted-%p-commit" -- $file | sort > $TMP fi # skip checking dates for non-Intel commits [[ ! $(tail -n1 $TMP) =~ "@intel.com" ]] && continue # skip checking dates for new files [ $(cat $TMP | wc -l) -le 1 ] && continue # grep out the grafted commits (commits with no parents) # and skip checking dates for non-Intel commits grep -v -e "grafted--commit" $TMP | grep -e "@intel.com" > $TMP2 [ $(cat $TMP2 | wc -l) -eq 0 ] && continue FIRST=`head -n1 $TMP2` LAST=` tail -n1 $TMP2` YEARS=`sed ' /Copyright [0-9-]\+.*, Intel Corporation/!d s/.*Copyright \([0-9]\+\)-\([0-9]\+\),.*/\1-\2/ s/.*Copyright \([0-9]\+\),.*/\1-\1/' $src_path` if [ -z "$YEARS" ]; then echo >&2 "$src_path:1: No copyright years found" RV=1 continue fi HEADER_FIRST=`echo $YEARS | cut -d"-" -f1` HEADER_LAST=` echo $YEARS | cut -d"-" -f2` COMMIT_FIRST=`echo $FIRST | cut -d"-" -f1` COMMIT_LAST=` echo $LAST | cut -d"-" -f1` if [ "$COMMIT_FIRST" != "" -a "$COMMIT_LAST" != "" ]; then if [ $HEADER_LAST -lt $COMMIT_LAST ]; then if [ $HEADER_FIRST -lt $COMMIT_FIRST ]; then COMMIT_FIRST=$HEADER_FIRST fi COMMIT_LAST=`date +%G` if [ $COMMIT_FIRST -eq $COMMIT_LAST ]; then NEW=$COMMIT_LAST else NEW=$COMMIT_FIRST-$COMMIT_LAST fi echo "$file:1: error: wrong copyright date: (is: $YEARS, should be: $NEW)" >&2 RV=1 fi else echo "$file:1: unknown commit dates" >&2 RV=1 fi done rm -f $TMP $TMP2 $TEMPFILE $(dirname "$0")/check-ms-license.pl $FILES # check if error found if [ $RV -eq 0 ]; then echo "Copyright headers are OK." else echo "Error(s) in copyright headers found!" >&2 fi exit $RV pmdk-1.13.1/utils/check_license/check-ms-license.pl0000775000000000000000000000425014435627501020631 0ustar rootroot#!/usr/bin/perl -w # SPDX-License-Identifier: BSD-3-Clause # Copyright 2020, Intel Corporation use Digest::MD5 "md5_hex"; my $BSD3 = <; close F; next unless /Copyright.*(Microsoft Corporation|FUJITSU)/; s/^ \*//mg; s/^#//mg; if (index($_, $BSD3) == -1) { $err = 1; print STDERR "Outside copyright but no/wrong license text in $f\n"; } } exit $err pmdk-1.13.1/utils/check_license/file-exceptions.sh0000775000000000000000000000047114435627501020615 0ustar rootroot#!/bin/sh -e # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2021, Intel Corporation # file-exceptions.sh - filter out files not checked for copyright and license grep -v -E -e '/queue.h$' -e '/getopt.h$' -e '/getopt.c$' -e 'src/core/valgrind/' \ -e '/testconfig\...$' -e'src/deps/miniasync/LICENSE' pmdk-1.13.1/utils/cstyle0000775000000000000000000006610414435627501013637 0ustar rootroot#!/usr/bin/env perl # # CDDL HEADER START # # The contents of this file are subject to the terms of the # Common Development and Distribution License (the "License"). # You may not use this file except in compliance with the License. # # You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE # or http://www.opensolaris.org/os/licensing. # See the License for the specific language governing permissions # and limitations under the License. # # When distributing Covered Code, include this CDDL HEADER in each # file and include the License file at usr/src/OPENSOLARIS.LICENSE. # If applicable, add the following below this CDDL HEADER, with the # fields enclosed by brackets "[]" replaced with your own identifying # information: Portions Copyright [yyyy] [name of copyright owner] # # CDDL HEADER END # # # Copyright 2008 Sun Microsystems, Inc. All rights reserved. # Use is subject to license terms. # # Portions copyright 2017, Intel Corporation. # # @(#)cstyle 1.58 98/09/09 (from shannon) #ident "%Z%%M% %I% %E% SMI" # # cstyle - check for some common stylistic errors. # # cstyle is a sort of "lint" for C coding style. # It attempts to check for the style used in the # kernel, sometimes known as "Bill Joy Normal Form". # # There's a lot this can't check for, like proper indentation # of code blocks. There's also a lot more this could check for. # # A note to the non perl literate: # # perl regular expressions are pretty much like egrep # regular expressions, with the following special symbols # # \s any space character # \S any non-space character # \w any "word" character [a-zA-Z0-9_] # \W any non-word character # \d a digit [0-9] # \D a non-digit # \b word boundary (between \w and \W) # \B non-word boundary # require 5.0; use IO::File; use Getopt::Std; use strict; use warnings; my $usage = "usage: cstyle [-chpvCP] [-o constructs] file ... -c check continuation indentation inside functions -h perform heuristic checks that are sometimes wrong -p perform some of the more picky checks -v verbose -C don't check anything in header block comments -P check for use of non-POSIX types -o constructs allow a comma-separated list of optional constructs: doxygen allow doxygen-style block comments (/** /*!) splint allow splint-style lint comments (/*@ ... @*/) "; my %opts; if (!getopts("cho:pvCP", \%opts)) { print $usage; exit 2; } my $check_continuation = $opts{'c'}; my $heuristic = $opts{'h'}; my $picky = $opts{'p'}; my $verbose = $opts{'v'}; my $ignore_hdr_comment = $opts{'C'}; my $check_posix_types = $opts{'P'}; my $doxygen_comments = 0; my $splint_comments = 0; if (defined($opts{'o'})) { for my $x (split /,/, $opts{'o'}) { if ($x eq "doxygen") { $doxygen_comments = 1; } elsif ($x eq "splint") { $splint_comments = 1; } else { print "cstyle: unrecognized construct \"$x\"\n"; print $usage; exit 2; } } } my ($filename, $line, $prev); # shared globals my $fmt; my $hdr_comment_start; if ($verbose) { $fmt = "%s:%d: %s\n%s\n"; } else { $fmt = "%s:%d: %s\n"; } if ($doxygen_comments) { # doxygen comments look like "/*!" or "/**"; allow them. $hdr_comment_start = qr/^\s*\/\*[\!\*]?$/; } else { $hdr_comment_start = qr/^\s*\/\*$/; } # Note, following must be in single quotes so that \s and \w work right. my $typename = '(int|char|short|long|unsigned|float|double' . '|\w+_t|struct\s+\w+|union\s+\w+|FILE|BOOL)'; # mapping of old types to POSIX compatible types my %old2posix = ( 'unchar' => 'uchar_t', 'ushort' => 'ushort_t', 'uint' => 'uint_t', 'ulong' => 'ulong_t', 'u_int' => 'uint_t', 'u_short' => 'ushort_t', 'u_long' => 'ulong_t', 'u_char' => 'uchar_t', 'quad' => 'quad_t' ); my $lint_re = qr/\/\*(?: ARGSUSED[0-9]*|NOTREACHED|LINTLIBRARY|VARARGS[0-9]*| CONSTCOND|CONSTANTCOND|CONSTANTCONDITION|EMPTY| FALLTHRU|FALLTHROUGH|LINTED.*?|PRINTFLIKE[0-9]*| PROTOLIB[0-9]*|SCANFLIKE[0-9]*|CSTYLED.*? )\*\//x; my $splint_re = qr/\/\*@.*?@\*\//x; my $warlock_re = qr/\/\*\s*(?: VARIABLES\ PROTECTED\ BY| MEMBERS\ PROTECTED\ BY| ALL\ MEMBERS\ PROTECTED\ BY| READ-ONLY\ VARIABLES:| READ-ONLY\ MEMBERS:| VARIABLES\ READABLE\ WITHOUT\ LOCK:| MEMBERS\ READABLE\ WITHOUT\ LOCK:| LOCKS\ COVERED\ BY| LOCK\ UNNEEDED\ BECAUSE| LOCK\ NEEDED:| LOCK\ HELD\ ON\ ENTRY:| READ\ LOCK\ HELD\ ON\ ENTRY:| WRITE\ LOCK\ HELD\ ON\ ENTRY:| LOCK\ ACQUIRED\ AS\ SIDE\ EFFECT:| READ\ LOCK\ ACQUIRED\ AS\ SIDE\ EFFECT:| WRITE\ LOCK\ ACQUIRED\ AS\ SIDE\ EFFECT:| LOCK\ RELEASED\ AS\ SIDE\ EFFECT:| LOCK\ UPGRADED\ AS\ SIDE\ EFFECT:| LOCK\ DOWNGRADED\ AS\ SIDE\ EFFECT:| FUNCTIONS\ CALLED\ THROUGH\ POINTER| FUNCTIONS\ CALLED\ THROUGH\ MEMBER| LOCK\ ORDER: )/x; my $err_stat = 0; # exit status if ($#ARGV >= 0) { foreach my $arg (@ARGV) { my $fh = new IO::File $arg, "r"; if (!defined($fh)) { printf "%s: can not open\n", $arg; } else { &cstyle($arg, $fh); close $fh; } } } else { &cstyle("", *STDIN); } exit $err_stat; my $no_errs = 0; # set for CSTYLED-protected lines sub err($) { my ($error) = @_; unless ($no_errs) { if ($verbose) { printf $fmt, $filename, $., $error, $line; } else { printf $fmt, $filename, $., $error; } $err_stat = 1; } } sub err_prefix($$) { my ($prevline, $error) = @_; my $out = $prevline."\n".$line; unless ($no_errs) { printf $fmt, $filename, $., $error, $out; $err_stat = 1; } } sub err_prev($) { my ($error) = @_; unless ($no_errs) { printf $fmt, $filename, $. - 1, $error, $prev; $err_stat = 1; } } sub cstyle($$) { my ($fn, $filehandle) = @_; $filename = $fn; # share it globally my $in_cpp = 0; my $next_in_cpp = 0; my $in_comment = 0; my $in_header_comment = 0; my $comment_done = 0; my $in_warlock_comment = 0; my $in_function = 0; my $in_function_header = 0; my $in_declaration = 0; my $note_level = 0; my $nextok = 0; my $nocheck = 0; my $in_string = 0; my ($okmsg, $comment_prefix); $line = ''; $prev = ''; reset_indent(); line: while (<$filehandle>) { s/\r?\n$//; # strip return and newline # save the original line, then remove all text from within # double or single quotes, we do not want to check such text. $line = $_; # # C allows strings to be continued with a backslash at the end of # the line. We translate that into a quoted string on the previous # line followed by an initial quote on the next line. # # (we assume that no-one will use backslash-continuation with character # constants) # $_ = '"' . $_ if ($in_string && !$nocheck && !$in_comment); # # normal strings and characters # s/'([^\\']|\\[^xX0]|\\0[0-9]*|\\[xX][0-9a-fA-F]*)'/''/g; s/"([^\\"]|\\.)*"/\"\"/g; # # detect string continuation # if ($nocheck || $in_comment) { $in_string = 0; } else { # # Now that all full strings are replaced with "", we check # for unfinished strings continuing onto the next line. # $in_string = (s/([^"](?:"")*)"([^\\"]|\\.)*\\$/$1""/ || s/^("")*"([^\\"]|\\.)*\\$/""/); } # # figure out if we are in a cpp directive # $in_cpp = $next_in_cpp || /^\s*#/; # continued or started $next_in_cpp = $in_cpp && /\\$/; # only if continued # strip off trailing backslashes, which appear in long macros s/\s*\\$//; # an /* END CSTYLED */ comment ends a no-check block. if ($nocheck) { if (/\/\* *END *CSTYLED *\*\//) { $nocheck = 0; } else { reset_indent(); next line; } } # a /*CSTYLED*/ comment indicates that the next line is ok. if ($nextok) { if ($okmsg) { err($okmsg); } $nextok = 0; $okmsg = 0; if (/\/\* *CSTYLED.*\*\//) { /^.*\/\* *CSTYLED *(.*) *\*\/.*$/; $okmsg = $1; $nextok = 1; } $no_errs = 1; } elsif ($no_errs) { $no_errs = 0; } # check length of line. # first, a quick check to see if there is any chance of being too long. if (($line =~ tr/\t/\t/) * 7 + length($line) > 80) { # yes, there is a chance. # replace tabs with spaces and check again. my $eline = $line; 1 while $eline =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e; if (length($eline) > 80) { # allow long line if it is user visible string # find if line start from " or L" and ends # with " + 2 optional characters # (these characters can be i.e. '");' '" \' or '",' etc...) if($eline =~ /^ *L?".*"[^"]{0,2}$/) { # check if entire line is one string literal $eline =~ s/^ *L?"//; $eline =~ s/"[^"]{0,2}$//; if($eline =~ /[^\\]"|[^\\](\\\\)+"/) { err("line > 80 characters"); } } else { err("line > 80 characters"); } } } # ignore NOTE(...) annotations (assumes NOTE is on lines by itself). if ($note_level || /\b_?NOTE\s*\(/) { # if in NOTE or this is NOTE s/[^()]//g; # eliminate all non-parens $note_level += s/\(//g - length; # update paren nest level next; } # a /* BEGIN CSTYLED */ comment starts a no-check block. if (/\/\* *BEGIN *CSTYLED *\*\//) { $nocheck = 1; } # a /*CSTYLED*/ comment indicates that the next line is ok. if (/\/\* *CSTYLED.*\*\//) { /^.*\/\* *CSTYLED *(.*) *\*\/.*$/; $okmsg = $1; $nextok = 1; } if (/\/\/ *CSTYLED/) { /^.*\/\/ *CSTYLED *(.*)$/; $okmsg = $1; $nextok = 1; } # universal checks; apply to everything if (/\t +\t/) { err("spaces between tabs"); } if (/ \t+ /) { err("tabs between spaces"); } if (/\s$/) { err("space or tab at end of line"); } if (/[^ \t(]\/\*/ && !/\w\(\/\*.*\*\/\);/) { err("comment preceded by non-blank"); } # is this the beginning or ending of a function? # (not if "struct foo\n{\n") if (/^{$/ && $prev =~ /\)\s*(const\s*)?(\/\*.*\*\/\s*)?\\?$/) { $in_function = 1; $in_declaration = 1; $in_function_header = 0; $prev = $line; next line; } if (/^}\s*(\/\*.*\*\/\s*)*$/) { if ($prev =~ /^\s*return\s*;/) { err_prev("unneeded return at end of function"); } $in_function = 0; reset_indent(); # we don't check between functions $prev = $line; next line; } if (/^\w*\($/) { $in_function_header = 1; } if ($in_warlock_comment && /\*\//) { $in_warlock_comment = 0; $prev = $line; next line; } # a blank line terminates the declarations within a function. # XXX - but still a problem in sub-blocks. if ($in_declaration && /^$/) { $in_declaration = 0; } if ($comment_done) { $in_comment = 0; $in_header_comment = 0; $comment_done = 0; } # does this looks like the start of a block comment? if (/$hdr_comment_start/) { if (!/^\t*\/\*/) { err("block comment not indented by tabs"); } $in_comment = 1; /^(\s*)\//; $comment_prefix = $1; if ($comment_prefix eq "") { $in_header_comment = 1; } $prev = $line; next line; } # are we still in the block comment? if ($in_comment) { if (/^$comment_prefix \*\/$/) { $comment_done = 1; } elsif (/\*\//) { $comment_done = 1; err("improper block comment close") unless ($ignore_hdr_comment && $in_header_comment); } elsif (!/^$comment_prefix \*[ \t]/ && !/^$comment_prefix \*$/) { err("improper block comment") unless ($ignore_hdr_comment && $in_header_comment); } } if ($in_header_comment && $ignore_hdr_comment) { $prev = $line; next line; } # check for errors that might occur in comments and in code. # allow spaces to be used to draw pictures in header and block comments. if (/[^ ] / && !/".* .*"/ && !$in_header_comment && !$in_comment) { err("spaces instead of tabs"); } if (/^ / && !/^ \*[ \t\/]/ && !/^ \*$/ && (!/^ \w/ || $in_function != 0)) { err("indent by spaces instead of tabs"); } if (/^\t+ [^ \t\*]/ || /^\t+ \S/ || /^\t+ \S/) { err("continuation line not indented by 4 spaces"); } if (/$warlock_re/ && !/\*\//) { $in_warlock_comment = 1; $prev = $line; next line; } if (/^\s*\/\*./ && !/^\s*\/\*.*\*\// && !/$hdr_comment_start/) { err("improper first line of block comment"); } if ($in_comment) { # still in comment, don't do further checks $prev = $line; next line; } if ((/[^(]\/\*\S/ || /^\/\*\S/) && !(/$lint_re/ || ($splint_comments && /$splint_re/))) { err("missing blank after open comment"); } if (/\S\*\/[^)]|\S\*\/$/ && !(/$lint_re/ || ($splint_comments && /$splint_re/))) { err("missing blank before close comment"); } if (/\/\/\S/) { # C++ comments err("missing blank after start comment"); } # check for unterminated single line comments, but allow them when # they are used to comment out the argument list of a function # declaration. if (/\S.*\/\*/ && !/\S.*\/\*.*\*\// && !/\(\/\*/) { err("unterminated single line comment"); } if (/^(#else|#endif|#include)(.*)$/) { $prev = $line; if ($picky) { my $directive = $1; my $clause = $2; # Enforce ANSI rules for #else and #endif: no noncomment # identifiers are allowed after #endif or #else. Allow # C++ comments since they seem to be a fact of life. if ((($1 eq "#endif") || ($1 eq "#else")) && ($clause ne "") && (!($clause =~ /^\s+\/\*.*\*\/$/)) && (!($clause =~ /^\s+\/\/.*$/))) { err("non-comment text following " . "$directive (or malformed $directive " . "directive)"); } } next line; } # # delete any comments and check everything else. Note that # ".*?" is a non-greedy match, so that we don't get confused by # multiple comments on the same line. # s/\/\*.*?\*\//\x01/g; s/\/\/.*$/\x01/; # C++ comments # delete any trailing whitespace; we have already checked for that. s/\s*$//; # following checks do not apply to text in comments. if (/[^ \t\+]\+[^\+=]/ || /[^\+]\+[^ \+=]/) { err("missing space around + operator"); } if (/[^ \t]\+=/ || /\+=[^ ]/) { err("missing space around += operator"); } if (/[^ \t\-]\-[^\->]/ && !/\(\w+\)\-\w/ && !/[\(\[]\-[\w \t]+[\)\],]/) { err("missing space before - operator"); } if (/[^\-]\-[^ \-=>]/ && !/\(\-\w+\)/ && !/(return|case|=|>|<|\?|:|,|^[ \t]+)[ \t]+\-[\w\(]/ && !/(\([^\)]+\)|\[|\()\-[\w\(\]]/) { err("missing space after - operator"); } if (/(return|case|=|\?|:|,|\[)[ \t]+\-[ \t]/ || /[\(\[]\-[ \t]/) { err("extra space after - operator"); } if (/[ \t]\+\+ /) { err("extra space before or after ++ operator"); } if (/[ \t]\-\- /) { err("extra space before or after -- operator"); } if (/[^ \t]\-=/ || /\-=[^ ]/) { err("missing space around -= operator"); } if (/[^ \t][\%\/]/ || /[\%\/][^ =]/ || /[\%\/]=[^ ]/) { err("missing space around one of operators: % %= / /="); } if (/[^ \t]\*=/ || /\*=[^ ]/) { err("missing space around *= operator"); } if (/[^ \t\(\)\*\[]\*/) { err("missing space before * operator"); } if (/\*[^ =\*\w\(,]/ && !/\(.+ \*+\)/ && !/\*\[\]/ && !/\*\-\-\w/ && !/\*\+\+\w/ && !/\*\)/) { err("missing space after * operator"); } if (/[^<>\s][!<>=]=/ || /[^<>][!<>=]=[^\s,]/ || (/[^->]>[^,=>\s]/ && !/[^->]>$/) || (/[^<]<[^,=<\s]/ && !/[^<]<$/) || /[^<\s]<[^<]/ || /[^->\s]>[^>]/) { err("missing space around relational operator"); } if (/\S>>=/ || /\S<<=/ || />>=\S/ || /<<=\S/ || /\S[-+*\/&|^%]=/ || (/[^-+*\/&|^%!<>=\s]=[^=]/ && !/[^-+*\/&|^%!<>=\s]=$/) || (/[^!<>=]=[^=\s]/ && !/[^!<>=]=$/)) { # XXX - should only check this for C++ code # XXX - there are probably other forms that should be allowed if (!/\soperator=/) { err("missing space around assignment operator"); } } if (/[,;]\S/ && !/\bfor \(;;\)/) { err("comma or semicolon followed by non-blank"); } # allow "for" statements to have empty "while" clauses if (/\s[,;]/ && !/^[\t]+;$/ && !/^\s*for \([^;]*; ;[^;]*\)/) { err("comma or semicolon preceded by blank"); } if (/^\s*(&&|\|\|)/) { err("improper boolean continuation"); } if (/\S *(&&|\|\|)/ || /(&&|\|\|) *\S/) { err("more than one space around boolean operator"); } if (/\b(for|if|while|switch|return|case)\(/) { err("missing space between keyword and paren"); } if (/(\b(for|if|while|switch|return)\b.*){2,}/ && !/^#define/) { # multiple "case" and "sizeof" allowed err("more than one keyword on line"); } if (/\b(for|if|while|switch|return|case)\s\s+\(/ && !/^#if\s+\(/) { err("extra space between keyword and paren"); } # try to detect "func (x)" but not "if (x)" or # "#define foo (x)" or "int (*func)();" if (/\w\s\(/) { my $s = $_; # strip off all keywords on the line s/\b(for|if|while|switch|return|case)\s\(/XXX(/g; s/\b(sizeof|typeof|__typeof__)\s*\(/XXX(/g; s/#elif\s\(/XXX(/g; s/^#define\s+\w+\s+\(/XXX(/; # do not match things like "void (*f)();" # or "typedef void (func_t)();" s/\w\s\(+\*/XXX(*/g; s/\b($typename|void)\s+\(+/XXX(/og; s/\btypedef\s($typename|void)\s+\(+/XXX(/og; # do not match "__attribute__ ((format (...)))" s/\b__attribute__\s*\(\(format\s*\(/__attribute__((XXX(/g; if (/\w\s\(/) { err("extra space between function name and left paren"); } $_ = $s; } # try to detect "int foo(x)", but not "extern int foo(x);" # XXX - this still trips over too many legitimate things, # like "int foo(x,\n\ty);" # if (/^(\w+(\s|\*)+)+\w+\(/ && !/\)[;,](\s|\x01)*$/ && # !/^(extern|static)\b/) { # err("return type of function not on separate line"); # } # this is a close approximation if (/^(\w+(\s|\*)+)+\w+\(.*\)(\s|\x01)*$/ && !/^(extern|static)\b/) { err("return type of function not on separate line"); } if (/^#define\t/ || /^#ifdef\t/ || /^#ifndef\t/) { err("#define/ifdef/ifndef followed by tab instead of space"); } if (/^#define\s\s+/ || /^#ifdef\s\s+/ || /^#ifndef\s\s+/) { err("#define/ifdef/ifndef followed by more than one space"); } # AON C-style doesn't require this. #if (/^\s*return\W[^;]*;/ && !/^\s*return\s*\(.*\);/) { # err("unparenthesized return expression"); #} if (/\bsizeof\b/ && !/\bsizeof\s*\(.*\)/) { err("unparenthesized sizeof expression"); } if (/\b(sizeof|typeof)\b/ && /\b(sizeof|typeof)\s+\(.*\)/) { err("spaces between sizeof/typeof expression and paren"); } if (/\(\s/) { err("whitespace after left paren"); } # allow "for" statements to have empty "continue" clauses if (/\s\)/ && !/^\s*for \([^;]*;[^;]*; \)/) { err("whitespace before right paren"); } if (/^\s*\(void\)[^ ]/) { err("missing space after (void) cast"); } if (/\S\{/ && !/\{\{/ && !/\(struct \w+\)\{/) { err("missing space before left brace"); } if ($in_function && /^\s+{/ && ($prev =~ /\)\s*$/ || $prev =~ /\bstruct\s+\w+$/)) { err("left brace starting a line"); } if (/}(else|while)/) { err("missing space after right brace"); } if (/}\s\s+(else|while)/) { err("extra space after right brace"); } if (/\b_VOID\b|\bVOID\b|\bSTATIC\b/) { err("obsolete use of VOID or STATIC"); } if (/\b($typename|void)\*/o) { err("missing space between type name and *"); } if (/^\s+#/) { err("preprocessor statement not in column 1"); } if (/^#\s/) { err("blank after preprocessor #"); } if (/!\s*(strcmp|strncmp|bcmp)\s*\(/) { err("don't use boolean ! with comparison functions"); } if (/^\S+\([\S\s]*\)\s*{/) { err("brace of function definition not at beginning of line"); } if (/static\s+\S+\s*=\s*(0|NULL)\s*;/) { err("static variable initialized with 0 or NULL"); } if (/typedef[\S\s]+\*\s*\w+\s*;/) { err("typedefed pointer type"); } if (/unsigned\s+int\s/) { err("'unsigned int' instead of just 'unsigned'"); } if (/long\s+long\s+int\s/) { err("'long long int' instead of just 'long long'"); } elsif (/long\s+int\s/) { err("'long int' instead of just 'long'"); } # # We completely ignore, for purposes of indentation: # * lines outside of functions # * preprocessor lines # if ($check_continuation && $in_function && !$in_cpp) { process_indent($_); } if ($picky) { # try to detect spaces after casts, but allow (e.g.) # "sizeof (int) + 1", "void (*funcptr)(int) = foo;", and # "int foo(int) __NORETURN;" if ((/^\($typename( \*+)?\)\s/o || /\W\($typename( \*+)?\)\s/o) && !/sizeof\($typename( \*)?\)\s/o && !/\($typename( \*+)?\)\s+=[^=]/o) { err("space after cast"); } if (/\b($typename|void)\s*\*\s/o && !/\b($typename|void)\s*\*\s+const\b/o) { err("unary * followed by space"); } } if ($check_posix_types) { # try to detect old non-POSIX types. # POSIX requires all non-standard typedefs to end in _t, # but historically these have been used. if (/\b(unchar|ushort|uint|ulong|u_int|u_short|u_long|u_char|quad)\b/) { err("non-POSIX typedef $1 used: use $old2posix{$1} instead"); } } if ($heuristic) { # cannot check this everywhere due to "struct {\n...\n} foo;" if ($in_function && !$in_declaration && /}./ && !/}\s+=/ && !/{.*}[;,]$/ && !/}(\s|\x01)*$/ && !/} (else|while)/ && !/}}/) { err("possible bad text following right brace"); } # cannot check this because sub-blocks in # the middle of code are ok if ($in_function && /^\s+{/) { err("possible left brace starting a line"); } } if (/^\s*else\W/) { if ($prev =~ /^\s*}$/) { err_prefix($prev, "else and right brace should be on same line"); } } $prev = $line; } if ($prev eq "") { err("last line in file is blank"); } } # # Continuation-line checking # # The rest of this file contains the code for the continuation checking # engine. It's a pretty simple state machine which tracks the expression # depth (unmatched '('s and '['s). # # Keep in mind that the argument to process_indent() has already been heavily # processed; all comments have been replaced by control-A, and the contents of # strings and character constants have been elided. # my $cont_in; # currently inside of a continuation my $cont_off; # skipping an initializer or definition my $cont_noerr; # suppress cascading errors my $cont_start; # the line being continued my $cont_base; # the base indentation my $cont_first; # this is the first line of a statement my $cont_multiseg; # this continuation has multiple segments my $cont_special; # this is a C statement (if, for, etc.) my $cont_macro; # this is a macro my $cont_case; # this is a multi-line case my @cont_paren; # the stack of unmatched ( and [s we've seen sub reset_indent() { $cont_in = 0; $cont_off = 0; } sub delabel($) { # # replace labels with tabs. Note that there may be multiple # labels on a line. # local $_ = $_[0]; while (/^(\t*)( *(?:(?:\w+\s*)|(?:case\b[^:]*)): *)(.*)$/) { my ($pre_tabs, $label, $rest) = ($1, $2, $3); $_ = $pre_tabs; while ($label =~ s/^([^\t]*)(\t+)//) { $_ .= "\t" x (length($2) + length($1) / 8); } $_ .= ("\t" x (length($label) / 8)).$rest; } return ($_); } sub process_indent($) { require strict; local $_ = $_[0]; # preserve the global $_ s/\x01//g; # No comments s/\s+$//; # Strip trailing whitespace return if (/^$/); # skip empty lines # regexps used below; keywords taking (), macros, and continued cases my $special = '(?:(?:\}\s*)?else\s+)?(?:if|for|while|switch)\b'; my $macro = '[A-Z_][A-Z_0-9]*\('; my $case = 'case\b[^:]*$'; # skip over enumerations, array definitions, initializers, etc. if ($cont_off <= 0 && !/^\s*$special/ && (/(?:(?:\b(?:enum|struct|union)\s*[^\{]*)|(?:\s+=\s*))\{/ || (/^\s*{/ && $prev =~ /=\s*(?:\/\*.*\*\/\s*)*$/))) { $cont_in = 0; $cont_off = tr/{/{/ - tr/}/}/; return; } if ($cont_off) { $cont_off += tr/{/{/ - tr/}/}/; return; } if (!$cont_in) { $cont_start = $line; if (/^\t* /) { err("non-continuation indented 4 spaces"); $cont_noerr = 1; # stop reporting } $_ = delabel($_); # replace labels with tabs # check if the statement is complete return if (/^\s*\}?$/); return if (/^\s*\}?\s*else\s*\{?$/); return if (/^\s*do\s*\{?$/); return if (/{$/); return if (/}[,;]?$/); # Allow macros on their own lines return if (/^\s*[A-Z_][A-Z_0-9]*$/); # cases we don't deal with, generally non-kosher if (/{/) { err("stuff after {"); return; } # Get the base line, and set up the state machine /^(\t*)/; $cont_base = $1; $cont_in = 1; @cont_paren = (); $cont_first = 1; $cont_multiseg = 0; # certain things need special processing $cont_special = /^\s*$special/? 1 : 0; $cont_macro = /^\s*$macro/? 1 : 0; $cont_case = /^\s*$case/? 1 : 0; } else { $cont_first = 0; # Strings may be pulled back to an earlier (half-)tabstop unless ($cont_noerr || /^$cont_base / || (/^\t*(?: )?(?:gettext\()?\"/ && !/^$cont_base\t/)) { err_prefix($cont_start, "continuation should be indented 4 spaces"); } } my $rest = $_; # keeps the remainder of the line # # The split matches 0 characters, so that each 'special' character # is processed separately. Parens and brackets are pushed and # popped off the @cont_paren stack. For normal processing, we wait # until a ; or { terminates the statement. "special" processing # (if/for/while/switch) is allowed to stop when the stack empties, # as is macro processing. Case statements are terminated with a : # and an empty paren stack. # foreach $_ (split /[^\(\)\[\]\{\}\;\:]*/) { next if (length($_) == 0); # rest contains the remainder of the line my $rxp = "[^\Q$_\E]*\Q$_\E"; $rest =~ s/^$rxp//; if (/\(/ || /\[/) { push @cont_paren, $_; } elsif (/\)/ || /\]/) { my $cur = $_; tr/\)\]/\(\[/; my $old = (pop @cont_paren); if (!defined($old)) { err("unexpected '$cur'"); $cont_in = 0; last; } elsif ($old ne $_) { err("'$cur' mismatched with '$old'"); $cont_in = 0; last; } # # If the stack is now empty, do special processing # for if/for/while/switch and macro statements. # next if (@cont_paren != 0); if ($cont_special) { if ($rest =~ /^\s*{?$/) { $cont_in = 0; last; } if ($rest =~ /^\s*;$/) { err("empty if/for/while body ". "not on its own line"); $cont_in = 0; last; } if (!$cont_first && $cont_multiseg == 1) { err_prefix($cont_start, "multiple statements continued ". "over multiple lines"); $cont_multiseg = 2; } elsif ($cont_multiseg == 0) { $cont_multiseg = 1; } # We've finished this section, start # processing the next. goto section_ended; } if ($cont_macro) { if ($rest =~ /^$/) { $cont_in = 0; last; } } } elsif (/\;/) { if ($cont_case) { err("unexpected ;"); } elsif (!$cont_special) { err("unexpected ;") if (@cont_paren != 0); if (!$cont_first && $cont_multiseg == 1) { err_prefix($cont_start, "multiple statements continued ". "over multiple lines"); $cont_multiseg = 2; } elsif ($cont_multiseg == 0) { $cont_multiseg = 1; } if ($rest =~ /^$/) { $cont_in = 0; last; } if ($rest =~ /^\s*special/) { err("if/for/while/switch not started ". "on its own line"); } goto section_ended; } } elsif (/\{/) { err("{ while in parens/brackets") if (@cont_paren != 0); err("stuff after {") if ($rest =~ /[^\s}]/); $cont_in = 0; last; } elsif (/\}/) { err("} while in parens/brackets") if (@cont_paren != 0); if (!$cont_special && $rest !~ /^\s*(while|else)\b/) { if ($rest =~ /^$/) { err("unexpected }"); } else { err("stuff after }"); } $cont_in = 0; last; } } elsif (/\:/ && $cont_case && @cont_paren == 0) { err("stuff after multi-line case") if ($rest !~ /$^/); $cont_in = 0; last; } next; section_ended: # End of a statement or if/while/for loop. Reset # cont_special and cont_macro based on the rest of the # line. $cont_special = ($rest =~ /^\s*$special/)? 1 : 0; $cont_macro = ($rest =~ /^\s*$macro/)? 1 : 0; $cont_case = 0; next; } $cont_noerr = 0 if (!$cont_in); } pmdk-1.13.1/utils/CSTYLE.ps10000664000000000000000000000200314435627501014022 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2017, Intel Corporation # # CSTYLE.ps1 -- script to check coding style # # XXX - integrate with VS projects and execute for each build # $scriptdir = Split-Path -Parent $PSCommandPath $rootdir = $scriptdir + "\.." $cstyle = $rootdir + "\utils\cstyle" $checkdir = $rootdir # XXX - *.cpp/*.hpp files not supported yet $include = @( "*.c", "*.h" ) If ( Get-Command -Name perl -ErrorAction SilentlyContinue ) { Get-ChildItem -Path $checkdir -Recurse -Include $include | ` Where-Object { $_.FullName -notlike "*jemalloc*" } | ` ForEach-Object { $IGNORE = $_.DirectoryName + "\.cstyleignore" if(Test-Path $IGNORE) { if((Select-String $_.Name $IGNORE)) { return } } $_ } | ForEach-Object { Write-Output $_.FullName & perl $cstyle $_.FullName if ($LASTEXITCODE -ne 0) { Exit $LASTEXITCODE } } } else { Write-Output "Cannot execute cstyle - perl is missing" } pmdk-1.13.1/utils/README0000664000000000000000000000107014435627501013255 0ustar rootrootPersistent Memory Development Kit This is utils/README. Scripts found in this directory are used during library development. Sub-directory 'docker' contains scripts for running builds, tests and checks inside a Docker containers, among with prepared images' recipes and instruction on using them. Sub-directory 'gha-runners' contains scripts used for execution specific sets of tests, on a self-hosted runners in GitHub Actions. Both of these sub-directories are intended to be used as development process vehicles and are part of continuous integration process. pmdk-1.13.1/utils/version.sh0000775000000000000000000000272114435627501014425 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2017-2022, Intel Corporation # # utils/version.sh -- determine project's version # set -e if [ -f "$1/VERSION" ]; then cat "$1/VERSION" exit 0 fi if [ -f $1/GIT_VERSION ]; then echo -n "\$Format:%h\$" | cmp -s $1/GIT_VERSION - && true if [ $? -eq 0 ]; then PARSE_GIT_VERSION=0 else PARSE_GIT_VERSION=1 fi else PARSE_GIT_VERSION=0 fi LATEST_RELEASE=$(cat $1/ChangeLog | grep "* Version" | cut -d " " -f 3 | sort -rV | head -n1) if [ $PARSE_GIT_VERSION -eq 1 ]; then GIT_VERSION_HASH=$(cat $1/GIT_VERSION) if [ -n "$GIT_VERSION_HASH" ]; then echo "$LATEST_RELEASE+git.$GIT_VERSION_HASH" exit 0 fi fi cd "$1" GIT_DESCRIBE=$(git describe 2>/dev/null) && true if [ -n "$GIT_DESCRIBE" ]; then # 1.5-19-gb8f78a329 -> 1.5+git19.gb8f78a329 # 1.5-rc1-19-gb8f78a329 -> 1.5-rc1+git19.gb8f78a329 echo "$GIT_DESCRIBE" | sed "s/\([0-9.]*\)-rc\([0-9]*\)-\([0-9]*\)-\([0-9a-g]*\)/\1-rc\2+git\3.\4/" | sed "s/\([0-9.]*\)-\([0-9]*\)-\([0-9a-g]*\)/\1+git\2.\3/" exit 0 fi # try commit it, git describe can fail when there are no tags (e.g. with shallow clone, like on Travis) GIT_COMMIT=$(git log -1 --format=%h) && true if [ -n "$GIT_COMMIT" ]; then echo "$LATEST_RELEASE+git.$GIT_COMMIT" exit 0 fi cd - >/dev/null # If nothing works, try to get version from directory name VER=$(basename `realpath "$1"` | sed 's/pmdk[-]*\([0-9a-z.+-]*\).*/\1/') if [ -n "$VER" ]; then echo "$VER" exit 0 fi exit 1 pmdk-1.13.1/utils/check_docs/0000775000000000000000000000000014435627501014464 5ustar rootrootpmdk-1.13.1/utils/check_docs/docs_tests.py0000664000000000000000000002466014435627501017220 0ustar rootroot#!/usr/bin/env python3 # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2023, Intel Corporation """ This module includes functions to check if documentation of PMDK library is built correctly and tests which use these functions. Tests check: -if all macros from header files are included in the doc directory, -if all functions from .so files are included in the doc directory, -if all functions and macros from the doc directory, are included in headers files and .so files, -if all functions and macros have direct access to man page through use command 'man function_name/macro_name'. Required argument: -r the PMDK library root path. -i PMDK library installation path. """ import unittest from subprocess import check_output, call, DEVNULL from os import listdir, path, linesep, walk import re import sys # definitions of regexes EXPRESSION_AT_THE_LINE_START = r'[\s]*([a-zA-Z_]+)[\\(\s]+' EXPRESSION_AFTER_DEFINE_PHRASE = r'[\s]*#define[\s]*([a-zA-Z_]+)[\\(\s]+' PMDK_LIBRARIES = ['libpmem', 'libpmemblk', 'libpmemlog', 'libpmemobj', 'libpmempool', 'libpmem2'] def get_exceptions(pmdk_path): """ Returns exceptions in a list. File 'exceptions.txt' contains all macros, which do not have a description in a man page by design. """ exceptions_file_path =\ path.join(path.dirname(path.realpath(__file__)), 'exceptions.txt') with open(exceptions_file_path, 'r') as f: exceptions = [line.strip().lower() for line in f] return exceptions def parse_macro_name(exceptions, line, const_expression, macros): """ Parses the name of the macro and adds it to existing list. """ macro_name = re.search(const_expression, line).group(1).lower() if macro_name not in exceptions: macros.append(macro_name) def get_macros(pmdk_path): """ Parses all headers from the libpmemobj library, extracts all macros, and returns them in a list. """ macros = [] exceptions = get_exceptions(pmdk_path) is_macro_in_next_line = False includes_path = path.join(pmdk_path, 'src', 'include', 'libpmemobj') # Definition of macro occurs after the phrases: 'static inline' or '#define' for header_file in listdir(includes_path): with open(path.join(includes_path, header_file), 'r') as f: file_lines = f.readlines() for line in file_lines: if line.startswith('static inline'): is_macro_in_next_line = True elif is_macro_in_next_line: parse_macro_name(exceptions, line, EXPRESSION_AT_THE_LINE_START, macros) is_macro_in_next_line = False elif '#define' in line: parse_macro_name(exceptions, line, EXPRESSION_AFTER_DEFINE_PHRASE, macros) return macros def get_functions_from_so_files(lib_path): """ Returns names of functions in a list based on symbols 'T' from .so files. """ functions_from_so_files = [] path_to_so_files = lib_path for elem in listdir(path_to_so_files): if elem.endswith('.so') and elem.split('.')[0] in PMDK_LIBRARIES: process = check_output( 'nm ' + elem + ' | grep " T "', cwd=path_to_so_files, shell=True) out = process.decode('UTF-8') for line in out.split(linesep): if line: name = line.split(' ')[2].strip() # Exclude'_pobj_debug_notice', because it is a name of # a function which is not described in the man page # by design. if name != '_pobj_debug_notice': functions_from_so_files.append(name) return functions_from_so_files def get_functions_windows_only(): """ Returns list of PMDK functions that are specific to Windows OS """ return ['pmem2_source_from_handle', 'pmem2_source_get_handle'] def get_functions_and_macros_from_doc(pmdk_path): """ Returns names of functions and macros in a list based on names of files from the doc directory. """ path_to_functions_and_macros = path.join(pmdk_path, 'doc') functions_and_macros_from_doc = [] for _, _, files in walk(path_to_functions_and_macros): for f in files: if not f.endswith('.3'): continue # Files with extension '.3' have the same name as functions and # macros of PMDK library. 'pmemobj_action' is excluded, because # it is not a name of the function. if f.startswith('pmemobj_action'): continue functions_and_macros_from_doc.append(f.split('.')[0]) return functions_and_macros_from_doc def check_completeness_of_macros_in_doc(pmdk_path): """ Returns missing macros in the doc directory in a list, based on macros extracted from header files. """ macros = get_macros(pmdk_path) functions_from_doc = get_functions_and_macros_from_doc(pmdk_path) missing_macros = [macro for macro in macros if macro not in functions_from_doc] return missing_macros def check_completeness_of_functions_in_doc(pmdk_path, lib_path): """ Returns list of missing macros in the doc directory, based on macros extracted from .so files. """ functions_from_so_files = get_functions_from_so_files(lib_path) functions_from_doc = get_functions_and_macros_from_doc(pmdk_path) missing_functions_in_doc = [function for function in functions_from_so_files if function not in functions_from_doc] return missing_functions_in_doc def check_completeness_of_extracted_functions_and_macros(pmdk_path, lib_path): """ Returns list of missing macros in header files and missing functions in .so files, based on functions and macros from the doc directory. """ functions_from_doc = get_functions_and_macros_from_doc(pmdk_path) macros = get_macros(pmdk_path) functions_from_so_files = get_functions_from_so_files(lib_path) functions_windows_specific = get_functions_windows_only() pmem2_async_group_name = 'pmem2_async' missing_functions_and_macros_in_doc = [ item for item in functions_from_doc if (item not in macros and item not in functions_from_so_files and item not in functions_windows_specific and item not in pmem2_async_group_name)] return missing_functions_and_macros_in_doc def check_linking_manpages(pmdk_path, doc_path): """ Checks if macros and functions from the doc directory have direct access to man page through command 'man function_name/macro_name' and returns list with elements which are not linked with the man page. """ functions_from_doc = get_functions_and_macros_from_doc(pmdk_path) functions_without_manpage = [] for function in functions_from_doc: if doc_path: process_find_in_manpage = call( 'man -M ' + doc_path + ' ' + function, shell=True, stdout=DEVNULL, stderr=DEVNULL) else: process_find_in_manpage = call( 'man ' + function, shell=True, stdout=DEVNULL, stderr=DEVNULL) if process_find_in_manpage: functions_without_manpage.append(function) return functions_without_manpage def parse_argument(argument_option): """ Parses an option from the command line. """ index = sys.argv.index(argument_option) try: argument = sys.argv[index+1] except IndexError: print('ERROR: Invalid argument!') print(__doc__) print(unittest.main.__doc__) else: sys.argv.pop(index) sys.argv.pop(index) return argument class TestDocumentation(unittest.TestCase): def test_completeness_of_macros_in_doc(self): """ Checks if all extracted macros from header files are in the doc directory. """ missing_macros = check_completeness_of_macros_in_doc(pmdk_path) error_msg = linesep + 'List of missing macros in the doc directory:' for macro in missing_macros: error_msg += linesep + macro self.assertFalse(missing_macros, error_msg) def test_completeness_of_functions_in_doc(self): """ Checks if all extracted functions from .so files are in the doc directory. """ missing_functions =\ check_completeness_of_functions_in_doc(pmdk_path, lib_path) error_msg = linesep + 'List of missing functions in the doc directory:' for function in missing_functions: error_msg += linesep + function self.assertFalse(missing_functions, error_msg) def test_completeness_of_extracted_functions_and_macros(self): """ Checks if all functions and macros from the doc directory, are extracted from header files and .so files. """ missing_functions_and_macros =\ check_completeness_of_extracted_functions_and_macros( pmdk_path, lib_path) error_msg = linesep + 'List of missing macros and functions:' for elem in missing_functions_and_macros: error_msg += linesep + elem self.assertFalse(missing_functions_and_macros, error_msg) def test_linking_manpage_for_functions_and_macros(self): """ Checks if all functions and macros have direct access to the man page through use command 'man function_name/macro_name'. """ no_linked_functions_and_macros =\ check_linking_manpages(pmdk_path, doc_path) error_msg =\ linesep + 'List of macros and functions without the linked man\ page:' for elem in no_linked_functions_and_macros: error_msg += linesep + elem self.assertFalse(no_linked_functions_and_macros, error_msg) if __name__ == '__main__': doc_path = '' lib_path = '' if '-h' in sys.argv or '--help' in sys.argv: print(__doc__) unittest.main() elif '-r' in sys.argv: pmdk_path = parse_argument('-r') if '-i' in sys.argv: installation_path = parse_argument('-i') doc_path = path.join(installation_path, 'share', 'man') lib_path = path.join(installation_path, 'lib') else: print(__doc__) print(unittest.main.__doc__) exit(1) unittest.main() else: print(__doc__) print(unittest.main.__doc__) pmdk-1.13.1/utils/check_docs/exceptions.txt0000664000000000000000000000315014435627501017405 0ustar rootroottoid_null pmemobj_max_layout offsetof pobj_root_type_num _toid_struct _toid_union _toid_enum _pobj_layout_ref _toid_constr _toid_constr _toid_declare libpmemobj_ctl_h libpmemobj_tx_h tx_onabort_check tx_onabort_check _pobj_tx_begin tx_begin_lock _pobj_validate_cb_sig libpmemobj_lists_atomic_h pobj_list_dest_head pobj_list_dest_tail pobj_list_dest_before pobj_list_dest_after libpmemobj_lists_atomic_base_h libpmemobj_iterator_h libpmemobj_atomic_base_h pobj_xalloc_valid_flags libpmemobj_atomic_h libpmemobj_iterator_base_h libpmemobj_pool_base_h pmemobj_min_pool pmemobj_min_part libpmemobj_pool_h libpmemobj_action_h libpmemobj_base_h __stdc_limit_macros pmemobj_max_alloc_size pobj_flag_zero pobj_flag_no_flush pobj_class_id pobj_xalloc_class_mask pobj_xalloc_zero pobj_xalloc_no_flush pmemobj_direct_inline pmemobj_major_version pmemobj_minor_version libpmemobj_tx_base_h _has_deprecated_with_message _has_deprecated_with_message tx_lock_deprecated tx_lock_deprecated pobj_tx_xalloc_valid_flags pobj_xadd_no_flush pobj_xadd_valid_flags libpmemobj_thread_h _pobj_cl_size libpmemobj_action_base_h pobj_max_actions pobj_action_xreserve_valid_flags libpmemobj_types_h pmemvlt pobj_arena_id pobj_xalloc_arena_mask pmdk_use_attr_deprec_with_msg pobj_flag_no_snapshot pobj_xadd_no_snapshot pobj_flag_assume_initialized pobj_xadd_assume_initialized pobj_flag_tx_no_abort pobj_xalloc_no_abort pobj_xadd_no_abort pobj_xlock_no_abort pobj_xlock_valid_flags pobj_xfree_no_abort pobj_xfree_valid_flags pobj_xpublish_no_abort pobj_xpublish_valid_flags pobj_xlog_append_buffer_no_abort pobj_xlog_append_buffer_valid_flags win_depr_str win_depr_attr pmdk-1.13.1/utils/git-years0000775000000000000000000000041614435627501014232 0ustar rootroot#!/bin/sh # SPDX-License-Identifier: BSD-3-Clause # Copyright 2019, Intel Corporation # git-years -- calculate the range of years for a given file from git git log --pretty='%aI %aE' "$@"|grep '@intel\.com'|cut -d- -f1|sort| sed '$p;2,$d'|uniq|tr '\n' -|sed 's/-$//' pmdk-1.13.1/utils/ansible/0000775000000000000000000000000014435627501014014 5ustar rootrootpmdk-1.13.1/utils/ansible/README0000664000000000000000000000031014435627501014666 0ustar rootrootPersistent Memory Development Kit This is utils/ansible/README. The scripts in this directory allow you to set up an RockyLinux and OpenSuSe environment on a real HW and build a PMDK project in it. pmdk-1.13.1/utils/build-dpkg.sh0000775000000000000000000005146614435627501014774 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2023, Intel Corporation # # build-dpkg.sh - Script for building deb packages # set -e SCRIPT_DIR=$(dirname $0) source $SCRIPT_DIR/pkg-common.sh # # usage -- print usage message and exit # usage() { [ "$1" ] && echo Error: $1 cat >&2 < debian/libpmem2.install $LIB_DIR/libpmem2.so.* EOF cat << EOF > debian/libpmem2.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug libpmem2: package-name-doesnt-match-sonames EOF cat << EOF > debian/libpmem2-dev.install $LIB_DIR/pmdk_debug/libpmem2.a $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmem2.so $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmem2.so.* $LIB_DIR/pmdk_dbg/ $LIB_DIR/libpmem2.so $LIB_DIR/pkgconfig/libpmem2.pc $INC_DIR/libpmem2.h $INC_DIR/libpmem2/*.h $MAN7_DIR/libpmem2.7 $MAN3_DIR/pmem2_*.3 EOF cat << EOF > debian/libpmem2-dev.triggers interest man-db EOF cat << EOF > debian/libpmem2-dev.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug # The following warnings are triggered by a bug in debhelper: # https://bugs.debian.org/204975 postinst-has-useless-call-to-ldconfig postrm-has-useless-call-to-ldconfig # We do not want to compile with -O2 for debug version hardening-no-fortify-functions $LIB_DIR/pmdk_dbg/* EOF } function append_libpmem2_control() { cat << EOF >> $CONTROL_FILE Package: libpmem2 Architecture: any Depends: \${shlibs:Depends}, \${misc:Depends} Description: Persistent Memory low level support library libpmem2 provides low level persistent memory support. In particular, support for the persistent memory instructions for flushing changes to pmem is provided. Package: libpmem2-dev Section: libdevel Architecture: any Depends: libpmem2 (=\${binary:Version}), \${shlibs:Depends}, \${misc:Depends} Description: Development files for libpmem2 libpmem2 provides low level persistent memory support. In particular, support for the persistent memory instructions for flushing changes to pmem is provided. EOF } function daxio_install_triggers_overrides() { cat << EOF > debian/daxio.install usr/bin/daxio $MAN1_DIR/daxio.1 EOF cat << EOF > debian/daxio.triggers interest man-db EOF cat << EOF > debian/daxio.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug EOF } function append_daxio_control() { cat << EOF >> $CONTROL_FILE Package: daxio Section: misc Architecture: any Priority: optional Depends: libpmem (=\${binary:Version}), \${shlibs:Depends}, \${misc:Depends} Description: dd-like tool to read/write to a devdax device The daxio utility performs I/O on Device DAX devices or zeroes a Device DAX device. Since the standard I/O APIs (read/write) cannot be used with Device DAX, data transfer is performed on a memory-mapped device. The daxio may be used to dump Device DAX data to a file, restore data from a backup copy, move/copy data to another device or to erase data from a device. EOF } if [ "${BUILD_PACKAGE_CHECK}" == "y" ] then CHECK_CMD=" override_dh_auto_test: dh_auto_test if [ -f $TEST_CONFIG_FILE ]; then\ cp $TEST_CONFIG_FILE src/test/testconfig.sh;\ else\ echo 'PMEM_FS_DIR=/tmp' > src/test/testconfig.sh; \ echo 'PMEM_FS_DIR_FORCE_PMEM=1' >> src/test/testconfig.sh; \ echo 'TEST_BUILD=\"debug nondebug\"' >> src/test/testconfig.sh; \ echo 'TEST_FS=\"pmem any none\"' >> src/test/testconfig.sh; \ fi make pcheck ${PCHECK_OPTS} " else CHECK_CMD=" override_dh_auto_test: " fi check_tool debuild check_tool dch check_file $SCRIPT_DIR/pkg-config.sh source $SCRIPT_DIR/pkg-config.sh PACKAGE_VERSION=$(get_version $PACKAGE_VERSION_TAG) PACKAGE_RELEASE=1 PACKAGE_SOURCE=${PACKAGE_NAME}-${PACKAGE_VERSION} PACKAGE_TARBALL_ORIG=${PACKAGE_NAME}_${PACKAGE_VERSION}.orig.tar.gz MAGIC_INSTALL=utils/magic-install.sh MAGIC_UNINSTALL=utils/magic-uninstall.sh CONTROL_FILE=debian/control [ -d $WORKING_DIR ] || mkdir $WORKING_DIR [ -d $OUT_DIR ] || mkdir $OUT_DIR OLD_DIR=$PWD cd $WORKING_DIR check_dir $SOURCE mv $SOURCE $PACKAGE_SOURCE tar zcf $PACKAGE_TARBALL_ORIG $PACKAGE_SOURCE cd $PACKAGE_SOURCE rm -rf debian mkdir debian # Generate compat file cat << EOF > debian/compat 10 EOF # Generate control file cat << EOF > $CONTROL_FILE Source: $PACKAGE_NAME Maintainer: $PACKAGE_MAINTAINER Section: libs Priority: optional Standards-version: 4.1.4 Build-Depends: debhelper (>= 9) Homepage: https://pmem.io/pmdk/ Package: libpmem Architecture: any Depends: \${shlibs:Depends}, \${misc:Depends} Description: Persistent Memory low level support library libpmem provides low level persistent memory support. In particular, support for the persistent memory instructions for flushing changes to pmem is provided. Package: libpmem-dev Section: libdevel Architecture: any Depends: libpmem (=\${binary:Version}), \${shlibs:Depends}, \${misc:Depends} Description: Development files for libpmem libpmem provides low level persistent memory support. In particular, support for the persistent memory instructions for flushing changes to pmem is provided. Package: libpmemblk Architecture: any Depends: libpmem (=\${binary:Version}), \${shlibs:Depends}, \${misc:Depends} Description: Persistent Memory block array support library libpmemblk implements a pmem-resident array of blocks, all the same size, where a block is updated atomically with respect to power failure or program interruption (no torn blocks). Package: libpmemblk-dev Section: libdevel Architecture: any Depends: libpmemblk (=\${binary:Version}), libpmem-dev, \${shlibs:Depends}, \${misc:Depends} Description: Development files for libpmemblk libpmemblk implements a pmem-resident array of blocks, all the same size, where a block is updated atomically with respect to power failure or program interruption (no torn blocks). Package: libpmemlog Architecture: any Depends: libpmem (=\${binary:Version}), \${shlibs:Depends}, \${misc:Depends} Description: Persistent Memory log file support library libpmemlog implements a pmem-resident log file. Package: libpmemlog-dev Section: libdevel Architecture: any Depends: libpmemlog (=\${binary:Version}), libpmem-dev, \${shlibs:Depends}, \${misc:Depends} Description: Development files for libpmemlog libpmemlog implements a pmem-resident log file. Package: libpmemobj Architecture: any Depends: libpmem (=\${binary:Version}), \${shlibs:Depends}, \${misc:Depends} Description: Persistent Memory object store support library libpmemobj turns a persistent memory file into a flexible object store, supporting transactions, memory management, locking, lists, and a number of other features. Package: libpmemobj-dev Section: libdevel Architecture: any Depends: libpmemobj (=\${binary:Version}), libpmem-dev, \${shlibs:Depends}, \${misc:Depends} Description: Development files for libpmemobj libpmemobj turns a persistent memory file into a flexible object store, supporting transactions, memory management, locking, lists, and a number of other features. . This package contains libraries and header files used for linking programs against libpmemobj. Package: libpmempool Architecture: any Depends: libpmem (=\${binary:Version}), \${shlibs:Depends}, \${misc:Depends} Description: Persistent Memory pool management support library libpmempool provides a set of utilities for management, diagnostics and repair of persistent memory pools. A pool in this context means a pmemobj pool, pmemblk pool, pmemlog pool or BTT layout, independent of the underlying storage. The libpmempool is for applications that need high reliability or built-in troubleshooting. It may be useful for testing and debugging purposes also. Package: libpmempool-dev Section: libdevel Architecture: any Depends: libpmempool (=\${binary:Version}), libpmem-dev, \${shlibs:Depends}, \${misc:Depends} Description: Development files for libpmempool libpmempool provides a set of utilities for management, diagnostics and repair of persistent memory pools. . This package contains libraries and header files used for linking programs against libpmempool. Package: $PACKAGE_NAME-dbg Section: debug Priority: optional Architecture: any Depends: libpmem (=\${binary:Version}), libpmemblk (=\${binary:Version}), libpmemlog (=\${binary:Version}), libpmemobj (=\${binary:Version}), libpmempool (=\${binary:Version}), \${misc:Depends} Description: Debug symbols for PMDK libraries Debug symbols for all PMDK libraries. Package: pmempool Section: misc Architecture: any Priority: optional Depends: \${shlibs:Depends}, \${misc:Depends} Description: utility for management and off-line analysis of PMDK memory pools This utility is a standalone tool that manages Persistent Memory pools created by PMDK libraries. It provides a set of utilities for administration and diagnostics of Persistent Memory pools. Pmempool may be useful for troubleshooting by system administrators and users of the applications based on PMDK libraries. Package: pmreorder Section: misc Architecture: any Priority: optional Depends: \${shlibs:Depends}, \${misc:Depends} Description: tool to parse and replay pmemcheck logs Pmreorder is tool that parses and replays log of operations collected by pmemcheck -- a atandalone tool which is a collection of python scripts designed to parse and replay operations logged by pmemcheck - a persistent memory checking tool. Pmreorder performs the store reordering between persistent memory barriers - a sequence of flush-fence operations. It uses a consistency checking routine provided in the command line options to check whether files are in a consistent state. EOF cp LICENSE debian/copyright if [ -n "$NDCTL_ENABLE" ]; then pass_ndctl_enable="NDCTL_ENABLE=$NDCTL_ENABLE" else pass_ndctl_enable="" fi cat << EOF > debian/rules #!/usr/bin/make -f #export DH_VERBOSE=1 %: dh \$@ override_dh_strip: dh_strip --dbg-package=$PACKAGE_NAME-dbg override_dh_auto_build: dh_auto_build -- EXPERIMENTAL=${EXPERIMENTAL} prefix=/$PREFIX libdir=/$LIB_DIR includedir=/$INC_DIR docdir=/$DOC_DIR man1dir=/$MAN1_DIR man3dir=/$MAN3_DIR man5dir=/$MAN5_DIR man7dir=/$MAN7_DIR sysconfdir=/etc bashcompdir=/usr/share/bash-completion/completions NORPATH=1 ${pass_ndctl_enable} SRCVERSION=$SRCVERSION PMEM2_INSTALL=${PMEM2_INSTALL} override_dh_auto_install: dh_auto_install -- EXPERIMENTAL=${EXPERIMENTAL} prefix=/$PREFIX libdir=/$LIB_DIR includedir=/$INC_DIR docdir=/$DOC_DIR man1dir=/$MAN1_DIR man3dir=/$MAN3_DIR man5dir=/$MAN5_DIR man7dir=/$MAN7_DIR sysconfdir=/etc bashcompdir=/usr/share/bash-completion/completions NORPATH=1 ${pass_ndctl_enable} SRCVERSION=$SRCVERSION PMEM2_INSTALL=${PMEM2_INSTALL} find -path './debian/*usr/share/man/man*/*.gz' -exec gunzip {} \; override_dh_install: mkdir -p debian/tmp/usr/share/pmdk/ cp utils/pmdk.magic debian/tmp/usr/share/pmdk/ dh_install ${CHECK_CMD} EOF chmod +x debian/rules mkdir debian/source ITP_BUG_EXCUSE="# This is our first package but we do not want to upload it yet. # Please refer to Debian Developer's Reference section 5.1 (New packages) for details: # https://www.debian.org/doc/manuals/developers-reference/pkgs.html#newpackage" cat << EOF > debian/source/format 3.0 (quilt) EOF cat << EOF > debian/libpmem.install $LIB_DIR/libpmem.so.* usr/share/pmdk/pmdk.magic $MAN5_DIR/poolset.5 EOF cat $MAGIC_INSTALL > debian/libpmem.postinst sed -i '1s/.*/\#\!\/bin\/bash/' debian/libpmem.postinst echo $'\n#DEBHELPER#\n' >> debian/libpmem.postinst cat $MAGIC_UNINSTALL > debian/libpmem.prerm sed -i '1s/.*/\#\!\/bin\/bash/' debian/libpmem.prerm echo $'\n#DEBHELPER#\n' >> debian/libpmem.prerm cat << EOF > debian/libpmem.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug libpmem: package-name-doesnt-match-sonames EOF cat << EOF > debian/libpmem-dev.install $LIB_DIR/pmdk_debug/libpmem.a $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmem.so $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmem.so.* $LIB_DIR/pmdk_dbg/ $LIB_DIR/libpmem.so $LIB_DIR/pkgconfig/libpmem.pc $INC_DIR/libpmem.h $MAN7_DIR/libpmem.7 $MAN3_DIR/pmem_*.3 EOF cat << EOF > debian/libpmem-dev.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug # The following warnings are triggered by a bug in debhelper: # https://bugs.debian.org/204975 postinst-has-useless-call-to-ldconfig postrm-has-useless-call-to-ldconfig # We do not want to compile with -O2 for debug version hardening-no-fortify-functions $LIB_DIR/pmdk_dbg/* # pmdk provides second set of libraries for debugging. # These are in /usr/lib/$arch/pmdk_dbg/, but still trigger ldconfig. # Related issue: https://github.com/pmem/issues/issues/841 libpmem-dev: package-has-unnecessary-activation-of-ldconfig-trigger EOF cat << EOF > debian/libpmemblk.install $LIB_DIR/libpmemblk.so.* EOF cat << EOF > debian/libpmemblk.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug libpmemblk: package-name-doesnt-match-sonames EOF cat << EOF > debian/libpmemblk-dev.install $LIB_DIR/pmdk_debug/libpmemblk.a $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmemblk.so $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmemblk.so.* $LIB_DIR/pmdk_dbg/ $LIB_DIR/libpmemblk.so $LIB_DIR/pkgconfig/libpmemblk.pc $INC_DIR/libpmemblk.h $MAN7_DIR/libpmemblk.7 $MAN3_DIR/pmemblk_*.3 EOF cat << EOF > debian/libpmemblk-dev.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug # The following warnings are triggered by a bug in debhelper: # https://bugs.debian.org/204975 postinst-has-useless-call-to-ldconfig postrm-has-useless-call-to-ldconfig # We do not want to compile with -O2 for debug version hardening-no-fortify-functions $LIB_DIR/pmdk_dbg/* # pmdk provides second set of libraries for debugging. # These are in /usr/lib/$arch/pmdk_dbg/, but still trigger ldconfig. # Related issue: https://github.com/pmem/issues/issues/841 libpmemblk-dev: package-has-unnecessary-activation-of-ldconfig-trigger EOF cat << EOF > debian/libpmemlog.install $LIB_DIR/libpmemlog.so.* EOF cat << EOF > debian/libpmemlog.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug libpmemlog: package-name-doesnt-match-sonames EOF cat << EOF > debian/libpmemlog-dev.install $LIB_DIR/pmdk_debug/libpmemlog.a $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmemlog.so $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmemlog.so.* $LIB_DIR/pmdk_dbg/ $LIB_DIR/libpmemlog.so $LIB_DIR/pkgconfig/libpmemlog.pc $INC_DIR/libpmemlog.h $MAN7_DIR/libpmemlog.7 $MAN3_DIR/pmemlog_*.3 EOF cat << EOF > debian/libpmemlog-dev.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug # The following warnings are triggered by a bug in debhelper: # https://bugs.debian.org/204975 postinst-has-useless-call-to-ldconfig postrm-has-useless-call-to-ldconfig # We do not want to compile with -O2 for debug version hardening-no-fortify-functions $LIB_DIR/pmdk_dbg/* # pmdk provides second set of libraries for debugging. # These are in /usr/lib/$arch/pmdk_dbg/, but still trigger ldconfig. # Related issue: https://github.com/pmem/issues/issues/841 libpmemlog-dev: package-has-unnecessary-activation-of-ldconfig-trigger EOF cat << EOF > debian/libpmemobj.install $LIB_DIR/libpmemobj.so.* EOF cat << EOF > debian/libpmemobj.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug libpmemobj: package-name-doesnt-match-sonames EOF cat << EOF > debian/libpmemobj-dev.install $LIB_DIR/pmdk_debug/libpmemobj.a $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmemobj.so $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmemobj.so.* $LIB_DIR/pmdk_dbg/ $LIB_DIR/libpmemobj.so $LIB_DIR/pkgconfig/libpmemobj.pc $INC_DIR/libpmemobj.h $INC_DIR/libpmemobj/*.h $MAN7_DIR/libpmemobj.7 $MAN3_DIR/pmemobj_*.3 $MAN3_DIR/pobj_*.3 $MAN3_DIR/oid_*.3 $MAN3_DIR/toid*.3 $MAN3_DIR/direct_*.3 $MAN3_DIR/d_r*.3 $MAN3_DIR/tx_*.3 EOF cat << EOF > debian/libpmemobj-dev.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug # The following warnings are triggered by a bug in debhelper: # https://bugs.debian.org/204975 postinst-has-useless-call-to-ldconfig postrm-has-useless-call-to-ldconfig # We do not want to compile with -O2 for debug version hardening-no-fortify-functions $LIB_DIR/pmdk_dbg/* # pmdk provides second set of libraries for debugging. # These are in /usr/lib/$arch/pmdk_dbg/, but still trigger ldconfig. # Related issue: https://github.com/pmem/issues/issues/841 libpmemobj-dev: package-has-unnecessary-activation-of-ldconfig-trigger EOF cat << EOF > debian/libpmempool.install $LIB_DIR/libpmempool.so.* EOF cat << EOF > debian/libpmempool.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug libpmempool: package-name-doesnt-match-sonames EOF cat << EOF > debian/libpmempool-dev.install $LIB_DIR/pmdk_debug/libpmempool.a $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmempool.so $LIB_DIR/pmdk_dbg/ $LIB_DIR/pmdk_debug/libpmempool.so.* $LIB_DIR/pmdk_dbg/ $LIB_DIR/libpmempool.so $LIB_DIR/pkgconfig/libpmempool.pc $INC_DIR/libpmempool.h $MAN7_DIR/libpmempool.7 $MAN3_DIR/pmempool_*.3 EOF cat << EOF > debian/libpmempool-dev.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug # The following warnings are triggered by a bug in debhelper: # https://bugs.debian.org/204975 postinst-has-useless-call-to-ldconfig postrm-has-useless-call-to-ldconfig # We do not want to compile with -O2 for debug version hardening-no-fortify-functions $LIB_DIR/pmdk_dbg/* # pmdk provides second set of libraries for debugging. # These are in /usr/lib/$arch/pmdk_dbg/, but still trigger ldconfig. # Related issue: https://github.com/pmem/issues/issues/841 libpmempool-dev: package-has-unnecessary-activation-of-ldconfig-trigger EOF cat << EOF > debian/$PACKAGE_NAME-dbg.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug EOF cat << EOF > debian/pmempool.install usr/bin/pmempool $MAN1_DIR/pmempool.1 $MAN1_DIR/pmempool-*.1 usr/share/bash-completion/completions/pmempool EOF cat << EOF > debian/pmempool.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug EOF cat << EOF > debian/pmreorder.install usr/bin/pmreorder usr/share/pmreorder/*.py $MAN1_DIR/pmreorder.1 EOF cat << EOF > debian/pmreorder.lintian-overrides $ITP_BUG_EXCUSE new-package-should-close-itp-bug EOF # libpmem2 if [ "${PMEM2_INSTALL}" == "y" ] then append_libpmem2_control; libpmem2_install_triggers_overrides; fi # daxio if [ "${NDCTL_ENABLE}" != "n" ] then append_daxio_control; daxio_install_triggers_overrides; fi # Convert ChangeLog to debian format CHANGELOG_TMP=changelog.tmp dch --create --empty --package $PACKAGE_NAME -v $PACKAGE_VERSION-$PACKAGE_RELEASE -M -c $CHANGELOG_TMP touch debian/changelog head -n1 $CHANGELOG_TMP >> debian/changelog echo "" >> debian/changelog convert_changelog ChangeLog >> debian/changelog echo "" >> debian/changelog tail -n1 $CHANGELOG_TMP >> debian/changelog rm $CHANGELOG_TMP # This is our first release but we do debuild --preserve-envvar=EXTRA_CFLAGS_RELEASE \ --preserve-envvar=EXTRA_CFLAGS_DEBUG \ --preserve-envvar=EXTRA_CFLAGS \ --preserve-envvar=EXTRA_CXXFLAGS \ --preserve-envvar=EXTRA_LDFLAGS \ --preserve-envvar=NDCTL_ENABLE \ -us -uc -b cd $OLD_DIR find $WORKING_DIR -name "*.deb"\ -or -name "*.dsc"\ -or -name "*.changes"\ -or -name "*.orig.tar.gz"\ -or -name "*.debian.tar.gz" | while read FILE do mv -v $FILE $OUT_DIR/ done exit 0 pmdk-1.13.1/utils/check_whitespace0000775000000000000000000000722114435627501015620 0ustar rootroot#!/usr/bin/env perl # SPDX-License-Identifier: BSD-3-Clause # Copyright 2015-2022, Intel Corporation # # check_whitespace -- scrub source tree for whitespace errors # use strict; use warnings; use File::Basename; use File::Find; use Encode; use v5.16; my $Me = $0; $Me =~ s,.*/,,; $SIG{HUP} = $SIG{INT} = $SIG{TERM} = $SIG{__DIE__} = sub { die @_ if $^S; my $errstr = shift; die "$Me: ERROR: $errstr"; }; my $Errcount = 0; # # err -- emit error, keep total error count # sub err { warn @_, "\n"; $Errcount++; } # # decode_file_as_string -- slurp an entire file into memory and decode # sub decode_file_as_string { my ($full, $file) = @_; my $fh; open($fh, '<', $full) or die "$full $!\n"; local $/; $_ = <$fh>; close $fh; # check known encodings or die my $decoded; my @encodings = ("UTF-8", "UTF-16", "UTF-16LE", "UTF-16BE"); foreach my $enc (@encodings) { eval { $decoded = decode( $enc, $_, Encode::FB_CROAK ) }; if (!$@) { $decoded =~ s/\R/\n/g; return $decoded; } } die "$Me: ERROR: Unknown file encoding"; } # # check_whitespace -- run the checks on the given file # sub check_whitespace { my ($full, $file) = @_; my $line = 0; my $eol; my $nf = 0; my $fstr = decode_file_as_string($full, $file); my $empty = 0; my $is_python = $full =~ /\.py$/; for (split /^/, $fstr) { $line++; if (!$is_python && /^$/) { $empty++; if ($empty > 1) { err("$full:$line: ERROR duplicated empty line"); } } else { $empty = 0; } $eol = /[\n]/s; if (/^\.nf$/) { err("$full:$line: ERROR: nested .nf") if $nf; $nf = 1; } elsif (/^\.fi$/) { $nf = 0; } elsif ($nf == 0) { chomp; err("$full:$line: ERROR: trailing whitespace") if /\s$/; err("$full:$line: ERROR: spaces before tabs") if / \t/; } } err("$full:$line: .nf without .fi") if $nf; err("$full:$line: noeol") unless $eol; } sub check_whitespace_with_exc { my ($full) = @_; $_ = $full; return 0 if /^[.\/]*src\/common\/queue\.h/; return 0 if /^[.\/]*src\/core\/valgrind\/.*\.h/; return 0 if /^[.\/]*src\/deps\/miniasync\/src\/core\/valgrind\/.*\.h/; $_ = basename($full); return 0 unless /^(README.*|LICENSE.*|Makefile.*|CMakeLists.txt|.gitignore|TEST.*|RUNTESTS|check_whitespace|.*\.([chp13s]|sh|map|cpp|hpp|inc|PS1|ps1|py|md|cmake))$/; return 0 if -z; check_whitespace($full, $_); return 1; } my $verbose = 0; my $force = 0; my $recursive = 0; sub check { my ($file) = @_; my $r; if ($force) { $r = check_whitespace($file, basename($file)); } else { $r = check_whitespace_with_exc($file); } if ($verbose) { if ($r == 0) { printf("skipped $file\n"); } else { printf("checked $file\n"); } } } my @files = (); foreach my $arg (@ARGV) { if ($arg eq '-v') { $verbose = 1; next; } if ($arg eq '-f') { $force = 1; next; } if ($arg eq '-r') { $recursive = 1; next; } if ($arg eq '-g') { @files = `git ls-tree -r --name-only HEAD`; chomp(@files); next; } if ($arg eq '-h') { printf "Options: -g - check all files tracked by git -r dir - recursively check all files in specified directory -v verbose - print whether file was checked or not -f force - disable blacklist\n"; exit 1; } if ($recursive == 1) { find(sub { my $full = $File::Find::name; if (!$force && ($full eq './.git' || $full eq './src/debug' || $full eq './src/nondebug' || $full eq './rpmbuild' || $full eq './dpkgbuild')) { $File::Find::prune = 1; return; } return unless -f; push @files, $full; }, $arg); $recursive = 0; next; } push @files, $arg; } if (!@files) { printf "Empty file list!\n"; } foreach (@files) { check($_); } exit $Errcount; pmdk-1.13.1/utils/pkg-common.sh0000664000000000000000000000205414435627501015003 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2021, Intel Corporation # # pkg-common.sh - common functions and variables for building packages # export LC_ALL="C" function error() { echo -e "error: $@" } function check_dir() { if [ ! -d $1 ] then error "Directory '$1' does not exist." exit 1 fi } function check_file() { if [ ! -f $1 ] then error "File '$1' does not exist." exit 1 fi } function check_tool() { local tool=$1 if [ -z "$(which $tool 2>/dev/null)" ] then error "'${tool}' not installed or not in PATH" exit 1 fi } function get_version() { echo -n $1 | sed "s/-rc/~rc/" } function get_os() { if [ -f /etc/os-release ] then local OS=$(cat /etc/os-release | grep -m1 -o -P '(?<=NAME=).*($)') [[ "$OS" =~ SLES|openSUSE ]] && echo -n "SLES_like" || ([[ "$OS" =~ "Fedora"|"Red Hat"|"CentOS"|"AlmaLinux"|"Rocky Linux" ]] && echo -n "RHEL_like" || echo 1) else echo 1 fi } REGEX_DATE_AUTHOR="([a-zA-Z]{3} [a-zA-Z]{3} [0-9]{2} [0-9]{4})\s*(.*)" REGEX_MESSAGE_START="\s*\*\s*(.*)" REGEX_MESSAGE="\s*(\S.*)" pmdk-1.13.1/utils/os-banned0000664000000000000000000000167014435627501014174 0ustar rootrootpthread_once pthread_key_create pthread_key_delete pthread_setspecific pthread_getspecific pthread_mutex_init pthread_mutex_destroy pthread_mutex_lock pthread_mutex_trylock pthread_mutex_unlock pthread_mutex_timedlock pthread_rwlock_init pthread_rwlock_destroy pthread_rwlock_rdlock pthread_rwlock_wrlock pthread_rwlock_tryrdlock pthread_rwlock_trywrlock pthread_rwlock_unlock pthread_rwlock_timedrdlock pthread_rwlock_timedwrlock pthread_spin_init pthread_spin_destroy pthread_spin_lock pthread_spin_unlock pthread_spin_trylock pthread_cond_init pthread_cond_destroy pthread_cond_broadcast pthread_cond_signal pthread_cond_timedwait pthread_cond_wait pthread_create pthread_join cpu_zero cpu_set pthread_setaffinity_np pthread_atfork sem_init sem_destroy sem_wait sem_trywait sem_post fsync fsync_dir open stat unlink access fopen fdopen chmod mkstemp posix_fallocate ftruncate flock writev clock_gettime rand_r unsetenv setenv getenv strsignal execv pmdk-1.13.1/utils/magic-uninstall.sh0000664000000000000000000000125014435627501016020 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2017, Intel Corporation # # magic-uninstall.sh -- Script for uninstalling magic script # set -e HDR_LOCAL=$(grep "File: pmdk" /etc/magic) HDR_PKG=$(grep "File: pmdk" /usr/share/pmdk/pmdk.magic) if [[ $HDR_LOCAL == $HDR_PKG ]] then echo "Removing PMDK magic from /etc/magic" HDR_LINE=$(grep -n "File: pmdk" /etc/magic | cut -f1 -d:) HDR_PKG_LINE=$(grep -n "File: pmdk" /usr/share/pmdk/pmdk.magic | cut -f1 -d:) HDR_LINES=$(cat /usr/share/pmdk/pmdk.magic | wc -l) HDR_FIRST=$(($HDR_LINE - $HDR_PKG_LINE + 1)) HDR_LAST=$(($HDR_FIRST + $HDR_LINES)) sed -i "${HDR_FIRST},${HDR_LAST}d" /etc/magic fi pmdk-1.13.1/utils/CREATE-ZIP.PS10000664000000000000000000000477314435627501014342 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2022, Intel Corporation # # CREATE-ZIP.PS1 -- script to create release zip package # # # parameter handling # [CmdletBinding(PositionalBinding=$false)] Param( [alias("b")] $build = "debug", [alias("v")] $version = "0", [alias("e")] $extended = "0" ) $scriptdir = Split-Path -Parent $PSCommandPath $rootdir = $scriptdir + "\..\" $builddir = $rootdir + "\src\x64\" $zipdir = $builddir + "\pmdk\" if ($version -eq "0") { $git = Get-Command -Name git -ErrorAction SilentlyContinue if ($git) { $version = $(git describe) } else { $version = "0" } } $zipfile = $builddir + "\pmdk-" + $version + "-win-x64-" + $build + ".zip" Remove-Item $zipdir -Force -Recurse -ea si Get-ChildItem | Where-Object {$_.Name -Match "pmdk-.*-win-x64.zip"} | Remove-Item -Force -ea si New-Item -ItemType directory -Path ( $zipdir) -Force | Out-Null New-Item -ItemType directory -Path ( $zipdir + "\bin\") -Force | Out-Null New-Item -ItemType directory -Path ( $zipdir + "\lib\") -Force | Out-Null $libs = @("libpmem", "libpmemblk", "libpmemlog", "libpmemobj", "libpmempool") $apps = @("pmempool") $apps_extended = @("pmempool", "pmemalloc", "pmemdetect", "pmemspoil", "pmemwrite") if ($extended -eq "1") { $apps = $apps_extended } foreach ($lib in $libs) { Copy-Item ($builddir + $build + "\libs\" + $lib + ".dll") ($zipdir + "\bin\") foreach ($ex in @(".lib", ".pdb")) { Copy-Item ($builddir + $build + "\libs\" + $lib + $ex) ($zipdir + "\lib\") } } foreach ($app in $apps) { if ($app -eq "pmempool") { Copy-Item ($builddir + $build + "\libs\" + $app + ".exe") ($zipdir + "\bin\") Copy-Item ($builddir + $build + "\libs\" + $app + ".pdb") ($zipdir + "\lib\") } else { Copy-Item ($builddir + $build + "\tests\" + $app + ".exe") ($zipdir + "\bin\") Copy-Item ($builddir + $build + "\tests\" + $app + ".pdb") ($zipdir + "\lib\") } } Copy-Item -Recurse ($rootdir + "src\include") ($zipdir) Remove-Item -Force ($zipdir + "include\.cstyleignore") Remove-Item -Force ($zipdir + "include\README") Copy-Item ($rootdir + "README.md") ($zipdir) Copy-Item ($rootdir + "LICENSE") ($zipdir) Copy-Item ($rootdir + "ChangeLog") ($zipdir) Add-Type -Assembly System.IO.Compression.FileSystem $comprlevel = [System.IO.Compression.CompressionLevel]::Optimal if (Test-Path ($zipdir)) { [System.IO.Compression.ZipFile]::CreateFromDirectory($zipdir, $zipfile, $comprlevel, $true) } Remove-Item $zipdir -Force -Recurse -ea si pmdk-1.13.1/utils/libpmemblk.pc.in0000664000000000000000000000034714435627501015452 0ustar rootrootincludedir=${prefix}/include Name: libpmemblk Description: libpmemblk library from PMDK project Version: ${version} URL: https://pmem.io/pmdk Requires.private: libpmem${rasdeps} Libs: -L${libdir} -lpmemblk Cflags: -I${includedir} pmdk-1.13.1/utils/build-rpm.sh0000775000000000000000000001212314435627501014630 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2023, Intel Corporation # # build-rpm.sh - Script for building rpm packages # set -e SCRIPT_DIR=$(dirname $0) source $SCRIPT_DIR/pkg-common.sh check_tool rpmbuild check_file $SCRIPT_DIR/pkg-config.sh source $SCRIPT_DIR/pkg-config.sh # # usage -- print usage message and exit # usage() { [ "$1" ] && echo Error: $1 cat >&2 < $RPM_SPEC_FILE if [ "$DISTRO" = "SLES_like" ] then sed -i '/^#.*bugzilla.redhat/d' $RPM_SPEC_FILE fi # do not split on space IFS=$'\n' # experimental features if [ "${EXPERIMENTAL}" = "y" ] then # no experimental features for now RPMBUILD_OPTS+=( ) fi # libpmem2 if [ "${PMEM2_INSTALL}" == "y" ] then RPMBUILD_OPTS+=(--define "_pmem2_install 1") fi # daxio & RAS if [ "${NDCTL_ENABLE}" = "n" ] then RPMBUILD_OPTS+=(--without ndctl) else RPMBUILD_OPTS+=(--with ndctl) fi # use specified testconfig file or default if [[( -n "${TEST_CONFIG_FILE}") && ( -f "$TEST_CONFIG_FILE" ) ]] then echo "Test config file: $TEST_CONFIG_FILE" RPMBUILD_OPTS+=(--define "_testconfig $TEST_CONFIG_FILE") else echo -e "Test config file $TEST_CONFIG_FILE does not exist.\n"\ "Default test config will be used." fi # run make check or not if [ "${BUILD_PACKAGE_CHECK}" == "n" ] then RPMBUILD_OPTS+=(--define "_skip_check 1") fi tar zcf $PACKAGE_TARBALL $PACKAGE_SOURCE # Create directory structure for rpmbuild mkdir -v BUILD SPECS echo "opts: ${RPMBUILD_OPTS[@]}" rpmbuild --define "_topdir `pwd`"\ --define "_rpmdir ${OUT_DIR}"\ --define "_srcrpmdir ${OUT_DIR}"\ -ta $PACKAGE_TARBALL \ ${RPMBUILD_OPTS[@]} echo "Building rpm packages done" exit 0 pmdk-1.13.1/utils/check-commits.sh0000775000000000000000000000222614435627501015466 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2020, Intel Corporation # # Used to check whether all the commit messages in a pull request # follow the GIT/PMDK guidelines. # # usage: ./check-commits.sh [range] # if [ -z "$1" ]; then # on CI run this check only for pull requests if [ -n "$CI_REPO_SLUG" ]; then if [[ "$CI_REPO_SLUG" != "$GITHUB_REPO" \ || $CI_EVENT_TYPE != "pull_request" ]]; then echo "SKIP: $0 can only be executed for pull requests to $GITHUB_REPO" exit 0 fi fi # CI_COMMIT_RANGE can be invalid for force pushes - use another # method to determine the list of commits if [[ $(git rev-list $CI_COMMIT_RANGE 2>/dev/null) || -n "$CI_COMMIT_RANGE" ]]; then MERGE_BASE=$(echo $CI_COMMIT_RANGE | cut -d. -f1) [ -z $MERGE_BASE ] && \ MERGE_BASE=$(git log --pretty="%cN:%H" | grep GitHub | head -n1 | cut -d: -f2) RANGE=$MERGE_BASE..$CI_COMMIT else MERGE_BASE=$(git log --pretty="%cN:%H" | grep GitHub | head -n1 | cut -d: -f2) RANGE=$MERGE_BASE..HEAD fi else RANGE="$1" fi COMMITS=$(git log --pretty=%H $RANGE) set -e for commit in $COMMITS; do `dirname $0`/check-commit.sh $commit done pmdk-1.13.1/utils/pmdk.magic0000664000000000000000000000076014435627501014337 0ustar rootroot #------------------------------------------------------------------------------ # $File: pmdk,v 1.2 2017/12/11 20:00:00 # pmdk: file(1) magic for Persistent Memory Development Kit pool files # # The PMDK specific format of pool files. # # PMEM signature 0 string PMEM >4 string POOLSET Persistent Memory Poolset file >>11 search REPLICA with replica # Pool type signature >4 regex LOG|BLK|OBJ Persistent Memory Pool file, type: %s, # Major version number >>8 lelong >0 version 0x%x pmdk-1.13.1/utils/check-manpage0000775000000000000000000000313114435627501015006 0ustar rootroot#!/bin/bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2019, Intel Corporation # check-manpage -- a tool to check a single man page against errors # # While it can handle multiple files, it's recommended to use # check-manpages instead. set -e check_link() { [ $(wc -l <"$file") = 1 ] || { echo ".so link isn't the only line" && return; } link=$(cat "$file") link=${link#.so } [ "${link##*/}" = "$link" ] || grep -q '^\.so man\([0-9]\)/[a-z0-9+_-]\+\.\1$' "$file" || { echo ".so link directory is not matching manX" && return; } [ -e "${link##*/}" ] || { echo ".so link target doesn't exist: ${link##*/}" && return; } } for m in "$@"; do dir="$(dirname $m)" file="$(basename $m)" [ -n "$dir" ] && pushd "$dir" >/dev/null if grep -q '^\.so' "$file"; then err=$(check_link) [ -z "$err" ] || { echo >&2 "$file: $err" FAILED=1 } popd >/dev/null 2>/dev/null continue fi # man can emit warnings and errors. Even non-fatal errors are normally # suppressed if a pager is in use (ie, all interactive usage). Common # messages include an unknown macro, an unbreakable line, etc. err=$(MANWIDTH=80 man --warnings -E UTF-8 -l -Tutf8 -Z "$file" 2>&1 >/dev/null| grep -v 'cannot adjust line' || true) [ -z "$err" ] || { echo >&2 "$file: $err" FAILED=1 } # If a "NAME" section exists, call lexgrog to see if it's properly # formatted. if grep -q '^\.SH NAME' "$file"; then if ! lexgrog "$file" >/dev/null; then # lexgrog doesn't give any interesting messages. echo 2>&1 "lexgrog failed on $file" FAILED=1 fi fi popd >/dev/null 2>/dev/null done exit $FAILED pmdk-1.13.1/utils/check-commit.sh0000775000000000000000000000230314435627501015277 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2022, Intel Corporation # # Used to check whether all the commit messages in a pull request # follow the GIT/PMDK guidelines. # # usage: ./check-commit.sh commit # if [ -z "$1" ]; then echo "Usage: check-commit.sh commit-id" exit 1 fi echo "Checking $1" subject=$(git log --format="%s" -n 1 $1) body=$(git log --format="%b" -n 1 $1) if [[ $subject =~ ^Merge.* ]]; then # skip exit 0 fi if [[ $subject =~ ^Revert.* ]]; then # skip exit 0 fi if [[ $body =~ "git-subtree-dir: src/deps/miniasync" ]]; then # skip exit 0 fi # valid area names AREAS="pmem\|pmem2\|log\|blk\|obj\|pool\|set\|test\|benchmark\|examples\|doc\|core\|common\|daxio\|pmreorder" prefix=$(echo $subject | sed -n "s/^\($AREAS\)\:.*/\1/p") if [ "$prefix" = "" ]; then echo "FAIL: subject line in commit message does not contain valid area name" echo `dirname $0`/check-area.sh $1 exit 1 fi ignore_long_link_lines="!/^http/" commit_len=$(git log --format="%s%n%b" -n 1 $1 | awk ${ignore_long_link_lines} | wc -L) if [ $commit_len -gt 73 ]; then echo "FAIL: commit message exceeds 72 chars per line (commit_len)" echo git log -n 1 $1 | cat exit 1 fi pmdk-1.13.1/utils/check-shebang.sh0000775000000000000000000000142314435627501015420 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2017-2019, Intel Corporation # # utils/check-shebang.sh -- interpreter directive check script # set -e err_count=0 for file in $@ ; do [ ! -f $file ] && continue SHEBANG=`head -n1 $file | cut -d" " -f1` [ "${SHEBANG:0:2}" != "#!" ] && continue if [ "$SHEBANG" != "#!/usr/bin/env" -a $SHEBANG != "#!/bin/sh" ]; then INTERP=`echo $SHEBANG | rev | cut -d"/" -f1 | rev` echo "$file:1: error: invalid interpreter directive:" >&2 echo " (is: \"$SHEBANG\", should be: \"#!/usr/bin/env $INTERP\")" >&2 ((err_count+=1)) fi done if [ "$err_count" == "0" ]; then echo "Interpreter directives are OK." else echo "Found $err_count errors in interpreter directives!" >&2 err_count=1 fi exit $err_count pmdk-1.13.1/utils/check-area.sh0000775000000000000000000000430714435627501014725 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2023, Intel Corporation # # Finds applicable area name for specified commit id. # if [ -z "$1" ]; then echo "Missing commit id argument." exit 1 fi files=$(git show $1 --format=oneline --name-only | grep -v -e "$1") git show -q $1 | cat echo echo "Modified files:" echo "$files" function categorize() { category=$1 shift cat_files=`echo "$files" | grep $*` if [ -n "${cat_files}" ]; then echo "$category" files=`echo "$files" | grep -v $*` fi } echo echo "Areas computed basing on the list of modified files: (see utils/check-area.sh for full algorithm)" categorize core -e "^src/core/" categorize pmem -e "^src/libpmem/" -e "^src/include/libpmem.h" categorize pmem2 -e "^src/libpmem2/" -e "^src/include/libpmem2.h" -e "^src/include/libpmem2/" categorize log -e "^src/libpmemlog/" -e "^src/include/libpmemlog.h" categorize blk -e "^src/libpmemblk/" -e "^src/include/libpmemblk.h" categorize obj -e "^src/libpmemobj/" -e "^src/include/libpmemobj.h" -e "^src/include/libpmemobj/" categorize pool -e "^src/libpmempool/" -e "^src/include/libpmempool.h" -e "^src/tools/pmempool/" categorize benchmark -e "^src/benchmarks/" categorize examples -e "^src/examples/" categorize daxio -e "^src/tools/daxio/" categorize pmreorder -e "^src/tools/pmreorder/" categorize test -e "^src/test/" categorize doc -e "^doc/" -e ".md\$" -e "^ChangeLog" -e "README" categorize common -e "^src/common/" \ -e "^utils/" \ -e ".inc\$" \ -e ".yml\$" \ -e ".gitattributes" \ -e ".gitignore" \ -e "^.mailmap\$" \ -e "^src/PMDK.sln\$" \ -e "Makefile\$" \ -e "^src/freebsd/" \ -e "^src/windows/" \ -e "^src/include/pmemcompat.h" echo echo "If the above list contains more than 1 entry, please consider splitting" echo "your change into more commits, unless those changes don't make sense " echo "individually (they do not build, tests do not pass, etc)." echo "For example, it's perfectly fine to use 'obj' prefix for one commit that" echo "changes libpmemobj source code, its tests and documentation." if [ -n "$files" ]; then echo echo "Uncategorized files:" echo "$files" fi pmdk-1.13.1/utils/get_aliases.sh0000775000000000000000000000527214435627501015224 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2017-2020, Intel Corporation # # # get_aliases.sh -- generate map of manuals functions and libraries # # usage: run from /pmdk/doc/generated location without parameters: # ./../../utils/get_aliases.sh # # This script searches manpages from section 7 then # takes all functions from each section using specified pattern # and at the end to every function it assign real markdown file # representation based on *.gz file content # # Generated libs_map.yml file is used on gh-pages # to handle functions and their aliases # list=("$@") man_child=("$@") function search_aliases { children=$1 parent=$2 for i in ${children[@]} do if [ -e ../$parent/$i ] then echo "Man: $i" content=$(head -c 150 ../$parent/$i) if [[ "$content" == ".so "* ]] ; then content=$(basename ${content#".so"}) i="${i%.*}" echo " $i: $content" >> $map_file else r="${i%.*}" echo " $r: $i" >> $map_file fi fi done } function list_pages { parent="${1%.*}" list=("$@") man_child=("$@") if [ "$parent" == "libpmem" ]; then man_child=($(ls -1 ../libpmem | grep -e ".*\.3$")) echo -n "- $parent: " >> $map_file echo "${man_child[@]}" >> $map_file fi if [ "$parent" == "libpmem2" ]; then man_child=($(ls -1 ../libpmem2 | grep -e ".*\.3$")) echo -n "- $parent: " >> $map_file echo "${man_child[@]}" >> $map_file fi if [ "$parent" == "libpmemset" ]; then man_child=($(ls -1 ../libpmemset | grep -e ".*\.3$")) echo -n "- $parent: " >> $map_file echo "${man_child[@]}" >> $map_file fi if [ "$parent" == "libpmemblk" ]; then man_child=($(ls -1 ../libpmemblk | grep -e ".*\.3$")) echo -n "- $parent: " >> $map_file echo "${man_child[@]}" >> $map_file fi if [ "$parent" == "libpmemlog" ]; then man_child=($(ls -1 ../libpmemlog | grep -e ".*\.3$")) echo -n "- $parent: " >> $map_file echo "${man_child[@]}" >> $map_file fi if [ "$parent" == "libpmemobj" ]; then man_child=($(ls -1 ../libpmemobj | grep -e ".*\.3$")) echo -n "- $parent: " >> $map_file echo "${man_child[@]}" >> $map_file fi if [ "$parent" == "libpmempool" ]; then man_child=($(ls -1 ../libpmempool | grep -e ".*\.3$")) echo -n "- $parent: " >> $map_file echo "${man_child[@]}" >> $map_file fi if [ "$parent" == "librpmem" ]; then man_child=($(ls -1 ../librpmem | grep -e ".*\.3$")) echo -n "- $parent: " >> $map_file echo "${man_child[@]}" >> $map_file fi if [ ${#man_child[@]} -ne 0 ] then list=${man_child[@]} search_aliases "${list[@]}" "$parent" fi } man7=($(ls -1 ../*/ | grep -e ".*\.7$")) map_file=libs_map.yml [ -e $map_file ] && rm $map_file touch $map_file for i in "${man7[@]}" do echo "Library: $i" list_pages $i done pmdk-1.13.1/utils/style_check.sh0000775000000000000000000000461414435627501015240 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2022, Intel Corporation # # utils/style_check.sh -- common style checking script # set -e ARGS=("$@") CSTYLE_ARGS=() CLANG_ARGS=() FLAKE8_ARGS=() CHECK_TYPE=$1 # When updating, please search for all references to "clang-format" and update # them as well; at this time these are CONTRIBUTING.md src/common.inc and # docker images. [ -z "$clang_format_bin" ] && which clang-format-9 >/dev/null && clang_format_bin=clang-format-9 [ -z "$clang_format_bin" ] && which clang-format >/dev/null && clang_format_bin=clang-format [ -z "$clang_format_bin" ] && clang_format_bin=clang-format # # print script usage # function usage() { echo "$0 [C/C++ files]" } # # require clang-format version 9.0 # function check_clang_version() { set +e which ${clang_format_bin} &> /dev/null && ${clang_format_bin} --version |\ grep "version 9\.0"\ &> /dev/null if [ $? -ne 0 ]; then echo "SKIP: requires clang-format version 9.0" exit 0 fi set -e } # # run old cstyle check # function run_cstyle() { if [ $# -eq 0 ]; then return fi ${cstyle_bin} -pP $@ } # # generate diff with clang-format rules # function run_clang_check() { if [ $# -eq 0 ]; then return fi check_clang_version for file in $@ do LINES=$(${clang_format_bin} -style=file $file |\ git diff --no-index $file - | wc -l) if [ $LINES -ne 0 ]; then ${clang_format_bin} -style=file $file | git diff --no-index $file - fi done } # # in-place format according to clang-format rules # function run_clang_format() { if [ $# -eq 0 ]; then return fi check_clang_version ${clang_format_bin} -style=file -i $@ } function run_flake8() { if [ $# -eq 0 ]; then return fi ${flake8_bin} --exclude=testconfig.py,envconfig.py $@ } for ((i=1; i<$#; i++)) { IGNORE="$(dirname ${ARGS[$i]})/.cstyleignore" if [ -e $IGNORE ]; then if grep -q ${ARGS[$i]} $IGNORE ; then echo "SKIP ${ARGS[$i]}" continue fi fi case ${ARGS[$i]} in *.[ch]pp) CLANG_ARGS+="${ARGS[$i]} " ;; *.[ch]) CSTYLE_ARGS+="${ARGS[$i]} " ;; *.py) FLAKE8_ARGS+="${ARGS[$i]} " ;; *) echo "Unknown argument" exit 1 ;; esac } case $CHECK_TYPE in check) run_cstyle ${CSTYLE_ARGS} run_clang_check ${CLANG_ARGS} run_flake8 ${FLAKE8_ARGS} ;; format) run_clang_format ${CLANG_ARGS} ;; *) echo "Invalid parameters" usage exit 1 ;; esac pmdk-1.13.1/utils/CHECK_WHITESPACE.PS10000664000000000000000000000104214435627501015252 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2017, Intel Corporation # # CHECK_WHITESPACE.PS1 -- script to check coding style # # XXX - integrate with VS projects and execute for each build # $scriptdir = Split-Path -Parent $PSCommandPath $rootdir = $scriptdir + "\.." $whitepace = $rootdir + "\utils\check_whitespace" If ( Get-Command -Name perl -ErrorAction SilentlyContinue ) { &perl $whitepace -g if ($LASTEXITCODE -ne 0) { Exit $LASTEXITCODE } } else { Write-Output "Cannot execute check_whitespace - perl is missing" } pmdk-1.13.1/utils/copy-source.sh0000775000000000000000000000146214435627501015211 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018, Intel Corporation # # utils/copy-source.sh -- copy source files (from HEAD) to 'path_to_dir/pmdk' # directory whether in git repository or not. # # usage: ./copy-source.sh [path_to_dir] [srcversion] set -e DESTDIR="$1" SRCVERSION=$2 if [ -d .git ]; then if [ -n "$(git status --porcelain)" ]; then echo "Error: Working directory is dirty: $(git status --porcelain)" exit 1 fi else echo "Warning: You are not in git repository, working directory might be dirty." fi mkdir -p "$DESTDIR"/pmdk echo -n $SRCVERSION > "$DESTDIR"/pmdk/.version if [ -d .git ]; then git archive HEAD | tar -x -C "$DESTDIR"/pmdk else find . \ -maxdepth 1 \ -not -name $(basename "$DESTDIR") \ -not -name . \ -exec cp -r "{}" "$DESTDIR"/pmdk \; fi pmdk-1.13.1/utils/check-os.sh0000775000000000000000000000135614435627501014437 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2017-2019, Intel Corporation # # Used to check if there are no banned functions in .o file # # usage: ./check-os.sh [os.h path] [.o file] [.c file] EXCLUDE="os_posix|os_thread_posix" if [[ $2 =~ $EXCLUDE ]]; then echo "skip $2" exit 0 fi symbols=$(nm --demangle --undefined-only --format=posix $2 | sed 's/ U *//g') functions=$(cat $1 | tr '\n' '|') functions=${functions%?} # remove trailing | character out=$( for sym in $symbols do grep -wE $functions <<<"$sym" done | sed 's/$/\(\)/g') [[ ! -z $out ]] && echo -e "`pwd`/$3:1: non wrapped function(s):\n$out\nplease use os wrappers" && rm -f $2 && # remove .o file as it don't match requirements exit 1 exit 0 pmdk-1.13.1/utils/check_sdk_version.py0000775000000000000000000000450014435627501016436 0ustar rootroot#!/usr/bin/env python3 # SPDX-License-Identifier: BSD-3-Clause # Copyright 2019-2020, Intel Corporation import argparse import os from subprocess import check_output, CalledProcessError import sys import shlex from xml.dom import minidom from xml.parsers.expat import ExpatError VALID_SDK_VERSION = '10.0.17134.0' def get_vcxproj_files(root_dir, ignored): """Get a list ".vcxproj" files under PMDK directory.""" to_format = [] command = 'git ls-files *.vcxproj' try: output = check_output(shlex.split(command), cwd=root_dir).decode("UTF-8") except CalledProcessError as e: sys.exit('Error: "' + command + '" failed with returncode: ' + str(e.returncode)) for line in output.splitlines(): if not line: continue file_path = os.path.join(root_dir, line) if os.path.isfile(file_path): to_format.append(file_path) return to_format def get_sdk_version(file): """ Get Windows SDK version from modified/new files from the current pull request. """ tag = 'WindowsTargetPlatformVersion' try: xml_file = minidom.parse(file) except ExpatError as e: sys.exit('Error: "' + file + '" is incorrect.\n' + str(e)) version_list = xml_file.getElementsByTagName(tag) if len(version_list) != 1: sys.exit('Error: the amount of tags "' + tag + '" is other than 1.') version = version_list[0].firstChild.data return version def main(): parser = argparse.ArgumentParser(prog='check_sdk_version.py', description='The script checks Windows SDK version in .vcxproj files.') parser.add_argument('-d', '--directory', help='Directory of PMDK tree.', required=True) args = parser.parse_args() current_directory = args.directory if not os.path.isdir(current_directory): sys.exit('"' + current_directory + '" is not a directory.') files = get_vcxproj_files(current_directory, '') if not files: sys.exit(0) for file in files: sdk_version = get_sdk_version(file) if sdk_version != VALID_SDK_VERSION: sys.exit('Wrong Windows SDK version: ' + sdk_version + ' in file: "' + file + '". Please use: ' + VALID_SDK_VERSION) if __name__ == '__main__': main() pmdk-1.13.1/utils/libpmemlog.pc.in0000664000000000000000000000034714435627501015463 0ustar rootrootincludedir=${prefix}/include Name: libpmemlog Description: libpmemlog library from PMDK project Version: ${version} URL: https://pmem.io/pmdk Requires.private: libpmem${rasdeps} Libs: -L${libdir} -lpmemlog Cflags: -I${includedir} pmdk-1.13.1/utils/libpmem2.pc.in0000664000000000000000000000030714435627501015037 0ustar rootrootincludedir=${prefix}/include Name: libpmem2 Description: libpmem2 library from PMDK project Version: ${version} URL: https://pmem.io/pmdk Requires: Libs: -L${libdir} -lpmem2 Cflags: -I${includedir} pmdk-1.13.1/utils/sort_solution0000775000000000000000000000515414435627501015255 0ustar rootroot#!/usr/bin/perl # SPDX-License-Identifier: BSD-3-Clause # Copyright 2016, Intel Corporation # # sort_solution -- sort visual studio solution projects lists # use strict; use warnings; # install libtext-diff-perl or perl-Text-Diff use Text::Diff; use Cwd 'abs_path'; use File::Basename; use File::Compare; sub help { print "Usage: sort_solution [check|sort]\n"; exit; } sub sort_global_section { my ($solution_fh, $temp_fh, $section_name) = @_; my $line = ""; my @array; while (defined($line = <$solution_fh>) && ($line !~ $section_name)) { print $temp_fh $line; } print $temp_fh $line; while (defined($line = <$solution_fh>) && ($line !~ "EndGlobalSection")) { push @array, $line; } @array = sort @array; foreach (@array) { print $temp_fh $_; } print $temp_fh $line; # print EndGlobalSection line } my $num_args = $#ARGV + 1; if ($num_args != 1) { help; } my $arg = $ARGV[0]; if($arg ne "check" && $arg ne "sort") { help; } my $filename = dirname(abs_path($0)).'/../src/PMDK.sln'; my $tempfile = dirname(abs_path($0)).'/../src/temp.sln'; open(my $temp_fh, '>', $tempfile) or die "Could not open file '$tempfile' $!"; open(my $solution_fh, '<:crlf', $filename) or die "Could not open file '$filename' $!"; my $line; # Read a header of file while (defined($line = <$solution_fh>) && ($line !~ "^Project")) { print $temp_fh $line; } my @part1; my $buff; my $guid; # Read the projects list with project dependencies do { if($line =~ "^Project") { $buff = $line; $guid = (split(/\,/, $line))[2]; } elsif($line =~ "^EndProject") { $buff .= $line; my %table = ( guid => $guid, buff => $buff, ); push @part1, \%table; } else { $buff .= $line; } } while (defined($line = <$solution_fh>) && $line ne "Global\n"); # sort the project list by a project GIUD and write to the tempfile @part1 = sort { $a->{guid} cmp $b->{guid} } @part1; foreach (@part1) { my %hash = %$_; print $temp_fh $hash{"buff"}; } print $temp_fh $line; # EndProject line sort_global_section $solution_fh, $temp_fh, "ProjectConfigurationPlatforms"; sort_global_section $solution_fh, $temp_fh, "NestedProjects"; # read solution file to the end and copy it to the temp file while (defined($line = <$solution_fh>)){ print $temp_fh $line; } close($temp_fh); close($solution_fh); if($arg eq "check") { my $diff = diff $filename => $tempfile; if ($diff eq "") { unlink $tempfile; exit; } print "PMDK solution file is not sorted, " . "please use sort_solution script before pushing your changes\n"; unlink $tempfile; exit 1; } else { unlink $filename or die "Cannot replace solution file $!"; rename $tempfile, $filename; } pmdk-1.13.1/utils/libpmem.pc.in0000664000000000000000000000030414435627501014752 0ustar rootrootincludedir=${prefix}/include Name: libpmem Description: libpmem library from PMDK project Version: ${version} URL: https://pmem.io/pmdk Requires: Libs: -L${libdir} -lpmem Cflags: -I${includedir} pmdk-1.13.1/utils/SRCVERSION.ps10000664000000000000000000001164114435627501014524 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2020, Intel Corporation # # SRCVERSION.PS1 -- script to create SCRVERSION macro and generate srcversion.h # # # Windows dll versioning supports only fixed number of fields. The most # important are MAJOR, MINOR and REVISION. We have 3-compoment releases # (e.g. 1.5.1) with release candidates, so we have to encode this information # into this fixed number of fields. That's why we abuse REVISION to encode both # 3rd component and rc status. # REVISION = 3RDCOMP * 1000 + (!is_rc) * 100 + rc. # # Examples: # +---------------------+-----+-----+--------+-----+------+-------+----------+ # |git describe --long |MAJOR|MINOR|REVISION|BUILD|BUGFIX|PRIVATE|PRERELEASE| # +---------------------+-----+-----+--------+-----+------+-------+----------+ # |1.5-rc2-0-12345678 | 1| 5| 2| 0| false| false| true| # |1.5-rc3-6-12345678 | 1| 5| 3| 6| false| true| true| # |1.5-0-12345678 | 1| 5| 100| 0| false| false| false| # |1.5-6-123345678 | 1| 5| 100| 6| false| true| false| # |1.5.2-rc1-0-12345678 | 1| 5| 2001| 0| true| false| true| # |1.5.2-rc4-6-12345678 | 1| 5| 2004| 6| true| true| true| # |1.5.2-0-12345678 | 1| 5| 2100| 0| true| false| false| # |1.5.2-6-12345678 | 1| 5| 2100| 6| true| true| false| # +---------------------+-----+-----+--------+-----+------+-------+----------+ # $scriptPath = Split-Path -parent $MyInvocation.MyCommand.Definition $file_path = $scriptPath + "\..\src\windows\include\srcversion.h" $git_version_file = $scriptPath + "\..\GIT_VERSION" $version_file = $scriptPath + "\..\VERSION" $git = Get-Command -Name git -ErrorAction SilentlyContinue if (Test-Path $file_path) { $old_src_version = Get-Content $file_path | ` Where-Object { $_ -like '#define SRCVERSION*' } } else { $old_src_version = "" } $git_version = "" $git_version_hash = "" if (Test-Path $git_version_file) { $git_version = Get-Content $git_version_file if ($git_version -eq "`$Format:%h`$") { $git_version = "" } else { $git_version_hash = $git_version } } $PRERELEASE = $false $BUGFIX = $false $PRIVATE = $true $CUSTOM = $false if ($null -ne $args[0]) { $version = $args[0] $ver_array = $version.split("-+") } elseif (Test-Path $version_file) { $version = Get-Content $version_file $ver_array = $version.split("-+") } elseif ($git_version_hash -ne "") { $MAJOR = 0 $MINOR = 0 $REVISION = 0 $BUILD = 0 $version = $git_version_hash $CUSTOM = $true $version_custom_msg = "#define VERSION_CUSTOM_MSG `"$git_version_hash`"" } elseif ($null -ne $git) { $version = $(git describe) $ver_array = $(git describe --long).split("-+") } else { $MAJOR = 0 $MINOR = 0 $REVISION = 0 $BUILD = 0 $version = "UNKNOWN_VERSION" $CUSTOM = $true $version_custom_msg = "#define VERSION_CUSTOM_MSG `"UNKNOWN_VERSION`"" } if ($null -ne $ver_array) { $ver_dots = $ver_array[0].split(".") $MAJOR = $ver_dots[0] $MINOR = $ver_dots[1] if ($ver_dots.length -ge 3) { $REV = $ver_dots[2] $BUGFIX = $true } else { $REV = 0 } $REVISION = 1000 * $REV $BUILD = $ver_array[$ver_array.length - 2] if ($ver_array.length -eq 4) { # .[.]--- if ($ver_array[1].StartsWith("rc")) { # .[.]-rc-- $REVISION += $ver_array[1].Substring("rc".Length) $PRERELEASE = $true $version = "$($ver_array[0])-$($ver_array[1])+git$($ver_array[2]).$($ver_array[3])" } else { # .[.]--- throw "Unknown version format" } } else { # .[.]-- $REVISION += 100 $version = "$($ver_array[0])+git$($ver_array[1]).$($ver_array[2])" } if ($BUILD -eq 0) { # it is not a (pre)release build $PRIVATE = $false } } $src_version = "#define SRCVERSION `"$version`"" if ($old_src_version -eq $src_version) { exit 0 } Write-Output "updating source version: $version" Write-Output $src_version > $file_path Write-Output "#ifdef RC_INVOKED" >> $file_path Write-Output "#define MAJOR $MAJOR" >> $file_path Write-Output "#define MINOR $MINOR" >> $file_path Write-Output "#define REVISION $REVISION" >> $file_path Write-Output "#define BUILD $BUILD" >> $file_path if ($PRERELEASE) { Write-Output "#define PRERELEASE 1" >> $file_path } if ($BUGFIX) { Write-Output "#define BUGFIX 1" >> $file_path } if ($PRIVATE) { Write-Output "#define PRIVATE 1" >> $file_path } if ($CUSTOM) { Write-Output "#define CUSTOM 1" >> $file_path Write-Output $version_custom_msg >> $file_path } Write-Output "#endif" >> $file_path pmdk-1.13.1/.cirrus.yml0000664000000000000000000000044514435627501013352 0ustar rootrootfreebsd_instance: image: freebsd-12-2-release-amd64 task: install_script: ASSUME_ALWAYS_YES=yes pkg bootstrap -f; pkg install -y autoconf bash binutils cmake coreutils e2fsprogs-libuuid git gmake libunwind ncurses pkgconf hs-pandoc script: CFLAGS="-Wno-unused-value" gmake pmdk-1.13.1/.github/0000775000000000000000000000000014435627501012577 5ustar rootrootpmdk-1.13.1/.github/ISSUE_TEMPLATE.md0000664000000000000000000000175414435627501015313 0ustar rootroot# GENERAL ISSUE: ## Bug Report - PMDK package version(s): - OS(es) version(s): - ndctl version(s): - kernel version(s): - compiler, libraries, packaging and other related tools version(s): ## Describe the issue: ## Actual behavior: ## Expected behavior: ## Additional information about Priority and Help Requested: Are you willing to submit a pull request with a proposed change? (Yes, No) Requested priority: (Showstopper, High, Medium, Low) pmdk-1.13.1/.github/workflows/0000775000000000000000000000000014435627501014634 5ustar rootrootpmdk-1.13.1/.github/workflows/codeql.yml0000664000000000000000000000273214435627501016632 0ustar rootroot# Run the CodeQL scanner for finding vulnerabilities in the code (both C and Python). # # This workflow does not use docker containers. name: CodeQL on: [push, pull_request, workflow_dispatch] env: GITHUB_REPO: pmem/pmdk HOST_WORKDIR: /home/runner/work/pmdk/pmdk WORKDIR: utils/gha-runners PMDK_CC: gcc PMDK_CXX: g++ jobs: analyze: name: Analyze runs-on: ubuntu-latest permissions: actions: read contents: read security-events: write strategy: fail-fast: false matrix: language: [ 'cpp', 'python' ] steps: - name: Clone the git repo uses: actions/checkout@v3 - name: Install required packages run: | export VALGRIND_DEPS="autoconf \ automake" export BASE_DEPS="build-essential \ cmake \ git \ libdaxctl-dev \ libndctl-dev \ pandoc \ pkg-config" sudo apt-get update \ && sudo apt-get install -y --no-install-recommends \ ${VALGRIND_DEPS} \ ${BASE_DEPS} \ && sudo ./utils/docker/images/install-valgrind.sh - name: Initialize CodeQL uses: github/codeql-action/init@v2 with: languages: ${{ matrix.language }} - name: Run the build run: cd ${WORKDIR} && ./build-pmdk.sh - name: Perform CodeQL Analysis uses: github/codeql-action/analyze@v2 with: category: "/language:${{matrix.language}}" pmdk-1.13.1/.github/workflows/unittest.yml0000664000000000000000000000225314435627501017240 0ustar rootroot# Run short and medium tests (as defined in test frameworks). # # This workflow is run on 'self-hosted' runners. name: PMDK_runners on: [push, pull_request, workflow_dispatch] jobs: linux: name: PMDK_runners if: github.repository == 'pmem/pmdk' runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: os: [[self-hosted, rhel],[self-hosted, opensuse]] config: ['check TEST_BUILD=debug', 'check TEST_BUILD=nondebug', 'check TEST_BUILD=static-debug', 'check TEST_BUILD=static-nondebug', 'pycheck TEST_BUILD=debug', 'pycheck TEST_BUILD=release', 'pycheck TEST_BUILD=static_debug', 'pycheck TEST_BUILD=static_release'] env: WORKDIR: utils/gha-runners steps: - name: Clone the git repo uses: actions/checkout@v3 - name: Get system information run: ./$WORKDIR/get-system-info.sh - name: Build run: ./$WORKDIR/build-pmdk.sh - name: Create testconfig files run: ./$WORKDIR/create-testconfig.sh - name: Run tests run: make ${{ matrix.config }} pmdk-1.13.1/.github/workflows/documentation.yml0000664000000000000000000000175014435627501020233 0ustar rootroot# Run checks around PMDK's documentation (like missing manpages, etc.). # # This workflow does not use docker containers. name: Documentation on: workflow_dispatch: schedule: # run this job at 00:00 UTC every Saturday - cron: '0 0 * * 6' jobs: linux: name: Documentation runs-on: ubuntu-latest env: TEST_SCRIPT: utils/check_docs/docs_tests.py steps: - name: Clone the git repo uses: actions/checkout@v3 - name: Install required packages run: | export BASE_DEPS="build-essential \ cmake \ libdaxctl-dev \ libndctl-dev \ pandoc \ pkg-config" sudo apt-get update \ && sudo apt-get install -y --no-install-recommends ${BASE_DEPS} - name: Build and install PMDK with docs run: make doc install prefix=$(pwd)/../install -j$(nproc) - name: Run docs' tests run: python3 ${TEST_SCRIPT} -r $(pwd) -i $(pwd)/../install -v pmdk-1.13.1/.github/workflows/ras_linux.yml0000664000000000000000000000535414435627501017372 0ustar rootroot# Run RAS test: Unsafe Shutdown Local. # # This workflow is run on 'self-hosted' runners. # # RAS tests require a different approach compared to the standard tests - they need to # reboot the runner during the test. Normally, rebooting and continuing the job on GHA # is not possible, due to losing connection with the runner. To work around this issue, # an additional runner (not connected to the GH) runs the tests instead. # # The general idea of the solution is: # - First platform [self-hosted runner] functions as the controller [ras_controller], # - Second platform functions as the test runner [ras_runner], # - The workflow launches its steps on the controller, # - The controller will then run an ansible playbook on the second platform [ras_runner], # with options provided by the workflow, # - The test runner follows the steps given by the controller, # running the tests in the process and providing results as output, # - The controller gathers this output and prints it in GHA job. # # The only drawback of this idea is that workflow would always finish successfully. # The solution was added as an additional step, at the end of the workflow, parsing the output. # # More detailed information about the ansible playbook and tests themselves can be found in: # utils/gha-runners/run-ras-linux.yml name: RAS_Linux on: [push, pull_request, workflow_dispatch] jobs: linux: name: RAS_Linux if: github.repository == 'pmem/pmdk' runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: os: [[self-hosted, ras_controller]] env: WORKDIR: utils/gha-runners steps: - name: Clone the git repo uses: actions/checkout@v3 # Variables, such as $ras_runner are hidden on the controller platform as environmental variables. # 'sed' command is used to filter out IP addresses from the ansible output, it will show up as the 'ras_runner' instead. # 'tee' command is used to save the overall output to the file. This file is needed for the next step. - name: Prepare and run RAS Linux tests via ansible-playbook run: | cd $WORKDIR ansible-playbook -i $ras_runner, run-ras-linux.yml -e "host=all ansible_user=$ras_user" | sed 's/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/ras_runner/' | tee playbook_output.txt # This simple step will look through the output in search of specific fail strings. # If any phrase is found in the file, the workflow will fail. - name: Fail the workflow if the playbook finished with a failure run: | cd $WORKDIR if grep -E 'fatal: \[ras_runner\]: FAILED!|failed: \[ras_runner\]' "playbook_output.txt"; then exit 1 else exit 0 fi pmdk-1.13.1/.github/workflows/gha.yml0000664000000000000000000001302614435627501016120 0ustar rootroot# Run default OSes for each PR, push event or a new tag, # checks basic builds with various compilers and executes all sets of tests. name: PMDK on: [push, pull_request, workflow_dispatch] env: GITHUB_REPO: pmem/pmdk DOCKER_REPO: ghcr.io/pmem/pmdk jobs: linux: name: Linux runs-on: ubuntu-latest env: # use org's Private Access Token to log in to GitHub Container Registry GH_CR_USER: ${{ secrets.GH_CR_USER }} GH_CR_PAT: ${{ secrets.GH_CR_PAT }} DOC_UPDATE_GITHUB_TOKEN: ${{ secrets.DOC_UPDATE_GITHUB_TOKEN }} HOST_WORKDIR: /home/runner/work/pmdk/pmdk WORKDIR: utils/docker PMDK_CC: gcc PMDK_CXX: g++ MAKE_PKG: 0 VALGRIND: 1 SRC_CHECKERS: 0 strategy: matrix: CONFIG: [ "N=1 OS=ubuntu OS_VER=22.04 FAULT_INJECTION=1 TEST_BUILD=debug PUSH_IMAGE=1", "N=2 OS=ubuntu OS_VER=22.04 FAULT_INJECTION=1 TEST_BUILD=nondebug UBSAN=1", "N=3 OS=ubuntu OS_VER=22.04 PMDK_CC=clang PMDK_CXX=clang++ TEST_BUILD=debug SRC_CHECKERS=1", "N=4 OS=ubuntu OS_VER=22.04 PMDK_CC=clang PMDK_CXX=clang++ TEST_BUILD=nondebug AUTO_DOC_UPDATE=1", "N=5 OS=ubuntu OS_VER=22.04 COVERAGE=1 FAULT_INJECTION=1 TEST_BUILD=debug", ] steps: - name: Clone the git repo uses: actions/checkout@v3 with: fetch-depth: 50 - name: Get system information run: ./$WORKDIR/get-system-info.sh - name: Pull or rebuild the image run: cd $WORKDIR && ${{ matrix.CONFIG }} ./pull-or-rebuild-image.sh - name: Run the build run: cd $WORKDIR && ${{ matrix.CONFIG }} ./build-CI.sh windows: name: Windows runs-on: windows-2022 env: solutionname: PMDK.sln ex_solutionname: Examples.sln perl: "C:\\Strawberry\\perl\\bin" strategy: matrix: CONFIG: [Debug, Release] steps: - name: Update Path run: | echo "${env:perl}" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append - name: Clone the git repo uses: actions/checkout@v3 - name: Setup MSBuild.exe uses: microsoft/setup-msbuild@v1.1 with: msbuild-architecture: x64 - name: Unshallow it run: git fetch --prune --unshallow - name: Various debug checks (cstyle, whitespace etc.) run: | if ("${{ matrix.CONFIG }}" -eq "Release") { Install-Module PsScriptAnalyzer -Force utils/CSTYLE.ps1 if ($LASTEXITCODE -ne 0) { exit 1 } utils/CHECK_WHITESPACE.ps1 if ($LASTEXITCODE -ne 0) { exit 1 } utils/ps_analyze.ps1 if ($LASTEXITCODE -ne 0) { exit 1 } perl utils/sort_solution check if ($LASTEXITCODE -ne 0) { exit 1 } ./utils/check_sdk_version.py -d . if ($LASTEXITCODE -ne 0) { exit 1 } } - name: Build run: | msbuild src\$Env:solutionname -property:Configuration=${{ matrix.CONFIG }} -m -v:m msbuild src\examples\$Env:ex_solutionname -property:Configuration=${{ matrix.CONFIG }} -m -v:m - name: Create ZIP archive run: utils/CREATE-ZIP.ps1 -b ${{ matrix.CONFIG }} - name: Run tests shell: powershell run: | if ($true) { cd src\test echo "`$Env:NON_PMEM_FS_DIR = `"C:\temp`"" >> testconfig.ps1 echo "`$Env:PMEM_FS_DIR = `"C:\temp`"" >> testconfig.ps1 echo "`$Env:PMEM_FS_DIR_FORCE_PMEM = `"1`"" >> testconfig.ps1 echo "`$Env:PMDK_NO_ABORT_MSG = `"1`"" >> testconfig.ps1 echo "`$Env:TM = `"1`"" >> testconfig.ps1 write-output "config = { 'unittest_log_level': 1, 'cacheline_fs_dir': 'C:\\temp', 'force_cacheline': True, 'page_fs_dir': 'C:\\temp', 'force_page': False, 'byte_fs_dir': 'C:\\temp', 'force_byte': True, 'tm': True, 'test_type': 'check', 'granularity': 'all', 'fs_dir_force_pmem': 1, 'keep_going': False, 'timeout': '4m', 'build': 'debug', 'force_enable': None, 'fail_on_skip': False, 'enable_admin_tests': False, }" | out-file "testconfig.py" -encoding utf8 if ("${{ matrix.CONFIG }}" -eq "Debug") { ./RUNTESTS.ps1 -b debug -o 4m if ($?) { python ./RUNTESTS.py -b debug } } if ("${{ matrix.CONFIG }}" -eq "Release") { ./RUNTESTS.ps1 -b nondebug -o 4m if ($?) { python ./RUNTESTS.py -b release } } } pmdk-1.13.1/.github/workflows/unittest_long.yml0000664000000000000000000000242214435627501020255 0ustar rootroot# Run tests of type 'long' (as defined in test frameworks). # # This workflow is run on 'self-hosted' runners. name: Long_run_tests on: workflow_dispatch: schedule: - cron: '5 0 * * 6' # At 00:05 on Saturday. jobs: linux: name: long if: github.repository == 'pmem/pmdk' runs-on: ${{ matrix.os }} timeout-minutes: 960 strategy: fail-fast: false matrix: os: [[self-hosted, rhel],[self-hosted, opensuse]] config: ['RUNTESTS -t long -b debug', 'RUNTESTS -t long -b nondebug', 'RUNTESTS -t long -b static-debug', 'RUNTESTS -t long -b static-nondebug', 'RUNTESTS.py -t long -b debug', 'RUNTESTS.py -t long -b release', 'RUNTESTS.py -t long -b static_debug', 'RUNTESTS.py -t long -b static_release'] env: WORKDIR: utils/gha-runners steps: - name: Clone the git repo uses: actions/checkout@v3 - name: Get system information run: ./$WORKDIR/get-system-info.sh - name: Build run: ./$WORKDIR/build-pmdk.sh - name: Create testconfig file run: ./$WORKDIR/create-testconfig.sh - name: Run tests run: cd src/test/ && ./${{ matrix.config }} pmdk-1.13.1/.github/workflows/coverity.yml0000664000000000000000000000173714435627501017233 0ustar rootroot# Run the public instance of Coverity static analysis # for finding vulnerabilities in the code. name: Coverity on: schedule: # run this job at 00:00 UTC every other day - cron: '0 0 */2 * *' env: GITHUB_REPO: pmem/pmdk DOCKER_REPO: ghcr.io/pmem/pmdk COVERITY_SCAN_NOTIFICATION_EMAIL: ${{ secrets.COVERITY_SCAN_NOTIFICATION_EMAIL }} COVERITY_SCAN_TOKEN: ${{ secrets.COVERITY_SCAN_TOKEN }} HOST_WORKDIR: /home/runner/work/pmdk/pmdk WORKDIR: utils/docker PMDK_CC: gcc PMDK_CXX: g++ VALGRIND: 1 COVERITY: 1 jobs: linux: name: Linux runs-on: ubuntu-latest strategy: matrix: CONFIG: ["OS=ubuntu OS_VER=22.04"] steps: - name: Clone the git repo uses: actions/checkout@v3 - name: Pull or rebuild the image run: cd $WORKDIR && ${{ matrix.CONFIG }} ./pull-or-rebuild-image.sh - name: Run the build run: cd $WORKDIR && ${{ matrix.CONFIG }} ./build-CI.sh pmdk-1.13.1/.github/workflows/unittest_valgrind_sh.yml0000664000000000000000000000205714435627501021622 0ustar rootroot# Run all tests (in bash test framework) with 'force-enabled' valgrind. # # This workflow is run on 'self-hosted' runners. name: Valgrind bash on: workflow_dispatch: schedule: - cron: '0 18 * * 5' # At 18:00 on Friday. jobs: linux: name: valgrind if: github.repository == 'pmem/pmdk' runs-on: ${{ matrix.os }} timeout-minutes: 4320 # wait max 3 days strategy: fail-fast: false matrix: config: ['-d', '-p', '-m', '-e'] build: ['debug', 'nondebug', 'static-debug', 'static-nondebug'] os: [[self-hosted, rhel],[self-hosted, opensuse]] env: WORKDIR: utils/gha-runners steps: - name: Clone the git repo uses: actions/checkout@v3 - name: Get system information run: ./$WORKDIR/get-system-info.sh - name: Build run: ./$WORKDIR/build-pmdk.sh - name: Create testconfig file run: ./$WORKDIR/create-testconfig.sh - name: Run tests run: cd src/test/ && ./RUNTESTS -b ${{ matrix.build }} ${{ matrix.config }} force-enable pmdk-1.13.1/.github/workflows/nightly.yml0000664000000000000000000000457314435627501017046 0ustar rootroot# Run less frequently used OSes and checks. name: Nightly on: workflow_dispatch: schedule: # run this job at 00:00 UTC every other day - cron: '0 0 */2 * *' push: tags: - '*' env: GITHUB_REPO: pmem/pmdk DOCKER_REPO: ghcr.io/pmem/pmdk # use org's Private Access Token to log in to GitHub Container Registry GH_CR_USER: ${{ secrets.GH_CR_USER }} GH_CR_PAT: ${{ secrets.GH_CR_PAT }} HOST_WORKDIR: /home/runner/work/pmdk/pmdk WORKDIR: utils/docker PMDK_CC: gcc PMDK_CXX: g++ MAKE_PKG: 0 VALGRIND: 1 SRC_CHECKERS: 0 jobs: linux: name: Linux runs-on: ubuntu-latest strategy: matrix: CONFIG: [ "OS=ubuntu OS_VER=22.04 MAKE_PKG=1 EXPERIMENTAL=y VALGRIND=0 NDCTL_ENABLE=n", "OS=ubuntu OS_VER=22.04 MAKE_PKG=1 EXPERIMENTAL=y VALGRIND=0 NDCTL_ENABLE=n PMDK_CC=clang PMDK_CXX=clang++", "OS=debian OS_VER=11 FAULT_INJECTION=1 TEST_BUILD=debug", "OS=debian OS_VER=11 FAULT_INJECTION=1 TEST_BUILD=nondebug PUSH_IMAGE=1", "OS=opensuse-leap OS_VER=15 FAULT_INJECTION=1 TEST_BUILD=debug", "OS=opensuse-leap OS_VER=15 FAULT_INJECTION=1 TEST_BUILD=nondebug", "OS=opensuse-leap OS_VER=15 MAKE_PKG=1 EXPERIMENTAL=y VALGRIND=0 PUSH_IMAGE=1", "OS=rockylinux OS_VER=8 TEST_BUILD=debug", "OS=rockylinux OS_VER=8 TEST_BUILD=nondebug", "OS=rockylinux OS_VER=8 MAKE_PKG=1 EXPERIMENTAL=y VALGRIND=0 PUSH_IMAGE=1", "OS=rockylinux OS_VER=9 TEST_BUILD=debug", "OS=rockylinux OS_VER=9 TEST_BUILD=nondebug", "OS=rockylinux OS_VER=9 MAKE_PKG=1 EXPERIMENTAL=y VALGRIND=0 PUSH_IMAGE=1", "OS=fedora OS_VER=37 PMDK_CC=clang PMDK_CXX=clang++ TEST_BUILD=debug PUSH_IMAGE=1", "OS=fedora OS_VER=37 PMDK_CC=clang PMDK_CXX=clang++ TEST_BUILD=nondebug", ] steps: - name: Clone the git repo uses: actions/checkout@v3 with: fetch-depth: 50 - name: Get system information run: ./$WORKDIR/get-system-info.sh - name: Pull or rebuild the image run: cd $WORKDIR && ${{ matrix.CONFIG }} ./pull-or-rebuild-image.sh rebuild - name: Run the build run: cd $WORKDIR && ${{ matrix.CONFIG }} ./build-CI.sh pmdk-1.13.1/.github/workflows/bandit.yml0000664000000000000000000000127014435627501016620 0ustar rootroot# Run the Bandit scanner for finding vulnerabilities in the python code. name: bandit on: [push, pull_request, workflow_dispatch] env: GITHUB_REPO: pmem/pmdk DOCKER_REPO: ghcr.io/pmem/pmdk HOST_WORKDIR: /home/runner/work/pmdk/pmdk WORKDIR: utils/docker jobs: linux: name: Linux runs-on: ubuntu-latest strategy: matrix: CONFIG: ["BANDIT=1 OS=ubuntu OS_VER=22.04"] steps: - name: Clone the git repo uses: actions/checkout@v3 - name: Pull or rebuild the image run: cd $WORKDIR && ${{ matrix.CONFIG }} ./pull-or-rebuild-image.sh - name: Run the build run: cd $WORKDIR && ${{ matrix.CONFIG }} ./build-CI.sh pmdk-1.13.1/.github/workflows/unittest_valgrind_py.yml0000664000000000000000000000211414435627501021632 0ustar rootroot# Run all tests (in python test framework) with 'force-enabled' valgrind. # # This workflow is run on 'self-hosted' runners. name: Valgrind python on: workflow_dispatch: schedule: - cron: '0 23 * * 6' # At 23:00 on Saturday. jobs: linux: name: valgrind if: github.repository == 'pmem/pmdk' runs-on: ${{ matrix.os }} timeout-minutes: 4320 # wait max 3 days strategy: fail-fast: false matrix: config: ['drd', 'pmemcheck', 'memcheck', 'helgrind'] build: ['debug', 'release', 'static_debug', 'static_release'] os: [[self-hosted, rhel],[self-hosted, opensuse]] env: WORKDIR: utils/gha-runners steps: - name: Clone the git repo uses: actions/checkout@v3 - name: Get system information run: ./$WORKDIR/get-system-info.sh - name: Build run: ./$WORKDIR/build-pmdk.sh - name: Create testconfig file run: ./$WORKDIR/create-testconfig.sh - name: Run tests run: cd src/test/ && ./RUNTESTS.py --force-enable ${{ matrix.config }} -b ${{ matrix.build }} pmdk-1.13.1/.github/ISSUE_TEMPLATE/0000775000000000000000000000000014435627501014762 5ustar rootrootpmdk-1.13.1/.github/ISSUE_TEMPLATE/feature.md0000664000000000000000000000051314435627501016736 0ustar rootroot--- name: Feature about: Feature your request labels: "Type: Feature" --- # FEAT: ## Rationale ## Description ## API Changes ## Implementation details ## Meta pmdk-1.13.1/.github/ISSUE_TEMPLATE/bug_report.md0000664000000000000000000000317214435627501017457 0ustar rootroot--- name: Bug report about: Did you find a bug in PMDK? Please let us know. labels: "Type: Bug" --- # ISSUE: ## Environment Information - PMDK package version(s): - OS(es) version(s): - ndctl version(s): - kernel version(s): - compiler, libraries, packaging and other related tools version(s): ## Please provide a reproduction of the bug: ## How often bug is revealed: (always, often, rare): ## Actual behavior: ## Expected behavior: ## Details ## Additional information about Priority and Help Requested: Are you willing to submit a pull request with a proposed change? (Yes, No) Requested priority: (Showstopper, High, Medium, Low) pmdk-1.13.1/.github/ISSUE_TEMPLATE/question.md0000664000000000000000000000060214435627501017151 0ustar rootroot--- name: Question about: Do you have question regarding PMDK? Don't hesitate to ask. labels: "Type: Question" --- # QUESTION: ## Details pmdk-1.13.1/LICENSE0000664000000000000000000000357314435627501012254 0ustar rootrootSPDX-License-Identifier: BSD-3-Clause Copyright 2014-2020, Intel Corporation Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Everything in this source tree is covered by the previous license with the following exceptions: * src/core/valgrind/valgrind.h, src/core/valgrind/memcheck.h, src/core/valgrind/helgrind.h, src/core/valgrind/drd.h are covered by another similar BSD license variant, contained in those files. * utils/cstyle (used only during development) licensed under CDDL. pmdk-1.13.1/ChangeLog0000664000000000000000000012771014435627501013021 0ustar rootrootWed May 31 2023 Oksana Sałyk * Version 1.13.1 This release : - Identifies and temporarily disables tests permanently or sporadically failing in some configurations: - ex_libpmem2 (#5593, #5635, #5638, #5641) - obj_defrag_advanced (#5690) - obj_sync (#5643, #5646) - pmem2_future (#5596) - pmem2_map (#5600) - pmem2_map_prot (#5599) - pmem2_mem_ext (#5640) - pmem2_mover (partially fixed) (#5595, #5686, #5694) - pmem2_vm_reservation (#5592) - Fixes failing tests: - obj_tx_alloc_mt (#5680) - pmem2_map_from_existing (#5594) - Temporarily disable pmem2_badblock test (#5650) due to missing infrastructure (#5636, #5652) Other changes and notable bug fixes: - pmem2: fix valgrind instrumentation in pmem2_map_from_exisiting (#5605) - pmem2: remove data_mover memory leak (#5681, #5637) - test: update permissions of DAX devices after clearing bad blocks (#5611) - test: disable avx512f tests (pmem2_mem_ext) under Valgrind (#5658) Wed Apr 26 2023 Łukasz Stolarczuk * Version 1.13.0 This release: - Removes (previously deprecated) librpmem. Those interested in remote persistent memory support should use the rpma library (https://github.com/pmem/rpma). - Removes libpmemset - it was an experimental library. - Sets libpmemblk as deprecated, to be removed in the next release. - Sets libpmemlog as deprecated, to be removed in the next release. - Sets btt support as deprecated, to be removed in the next release. - Sets async as deprecated, to be removed in the next release. - Sets FreeBSD support as deprecated, to be removed in the next release. - Sets Windows support as deprecated, to be removed in the next release. Deprecations and removals are done accordingly to the support strategy outlined in the blog post: https://pmem.io/blog/2022/11/update-on-pmdk-and-our-long-term-support-strategy. Other changes and notable bug fixes: - core: handle ENOMEM gracefully during logging (#5515) - pool: fix dependencies with pmempool (#5521) - test: add more security checks for pmempool input - pmreorder: share markers from an application through env variable (#5419) - examples: fix potential buffer overflow in pmemobjfs example (#5472) - several fixes to support ppc64le architecture - several fixes for issues found by Coverity tool Known issues: - The latest validation cycle exposed previously unknown persistency issues related to pmem2 async functions (DEPRECATED) and miniasync (EXPERIMENTAL): #5596 and #5597. Both will be removed in the PMDK 1.14 release. These issues do NOT affect any other PMDK library. Tue Aug 25 2022 Łukasz Plewa * Version 1.12.1 This release allows usings the movdir64b instruction by setting an environment variable, and fixes a number of smaller bugs. Detailed list of bug fixes: - obj: always perform cleanup of logs with leftover state - pmem2: actually skip msync with noflush on non-pmem - pmem2: make sure async functions use proper vdm flags - common: fix default cflags in makefiles - obj: ensure zones are reclaimed prior to free - pmem2: enable movdir64b compilation but disable it in the runtime Tue May 24 2022 Łukasz Plewa * Version 1.12.0 This release: - Deprecates librpmem library and rpmemd tool. Using the librpmem library API will result in warnings and is no longer recommended. Those interested in a remote persistent memory support should use the new rpma (https://github.com/pmem/rpma) library. - Introduces a new set of APIs in libpmem2 to perform asynchronous data movement operations. To use this feature, software needs to include an optional miniasync(7) dependency. - Adds new API to machine safe read/write operations in the pmem2 library. - Introduces support for movdir64b instruction for memory operation functions in the libpmem2 library. - Adds experimental support for RISC-V. Other changes and notable bug fixes: - common: supress false positve of '-Wunused-parameter' - examples: remove unnecessary persists in pminvaders - obj: fix invalid type when setting cache size ctl (pmem/issues#5291) - pmem: fix eADR memmove and memset (pmem/issues#5364) - pmem2: add addr alignment prediction to vm reservation - pmreorder: update docs with required last arg - file_name - pmreorder: add more debug/info logs Fri Sep 24 2020 Łukasz Plewa * Version 1.11.1 This release fixes a missing sfence in non-temporal version of memcpy function (https://github.com/pmem/pmdk/issues/5292), and fixes a number of smaller bugs. Detailed list of bug fixes: - doc: remove exprimental moniker from libpmem2(7) - common: fix missing sfence in non-temporal memcpy - common: fix a mismatch between prototype and body - common: the yearly spellchecker run - common: force no LTO for rpm build - common: fix mismatched function args - obj: rename vars clashing with those of a containing block - pmem2: don't force smaller alignment for fsdax mappings - pool: don't trample upon users of localtime() - rpmem: Fix RPMEM_RAW_BUFF_SIZE and LANE_ALIGN_SIZE for powerpc64le - rpmem: drop a redundant check Mon Sep 24 2021 Łukasz Plewa * Version 1.10.1 This release fixes a missing sfence in non-temporal version of memcpy function (https://github.com/pmem/pmdk/issues/5292), and fixes a number of smaller bugs. Detailed list of bug fixes: - common: add disable lto to cflags - doc: remove exprimental moniker from libpmem2(7) - common: fix missing sfence in non-temporal memcpy - common: fix a mismatch between prototype and body - common: the yearly spellchecker run - common: force no LTO for rpm build - common: fix mismatched function args - obj: rename vars clashing with those of a containing block - pmem2: don't force smaller alignment for fsdax mappings - pool: don't trample upon users of localtime() - rpmem: Fix RPMEM_RAW_BUFF_SIZE and LANE_ALIGN_SIZE for powerpc64le - rpmem: drop a redundant check Fri Sep 24 2021 Łukasz Plewa * Version 1.9.3 This release fixes a missing sfence in non-temporal version of memcpy function (https://github.com/pmem/pmdk/issues/5292), and fixes a number of smaller bugs. Detailed list of bug fixes: - common: add disable lto to cflags - common: fix missing sfence in non-temporal memcpy - common: fix mismatched function args - common: fix a mismatch between prototype and body - common: the yearly spellchecker run - common: force no LTO for rpm build - rpmem: Fix RPMEM_RAW_BUFF_SIZE and LANE_ALIGN_SIZE for powerpc64le Fri Jul 2 2021 Piotr Balcer * Version 1.11.0 This release: - Adds new APIs for libpmem2, most notably there are new functions to shrink and extend an existing reservation and a new iterator API for mappings contained within an existing reservation. There's also a new function to retrieve a numa node for a source. - Makes the pmemobj_open() and pmemobj_close() functions from libpmemobj thread-safe. It's now easier to correctly manage persistent memory pools in a parallel environment. - Introduces a new API in libpmemobj to globally change the method of assigning arenas to threads. The default is to rely on a OS per-thread key to store arena information. This release introduces an option to avoid the use of thread-local keys by simply using one global arena for all threads in a pool. Other changes and notable bug fixes: - pmem2: don't force smaller alignment for fsdax mappings - rpmem: various fixes for powerpc64le - doc: fix documentation of pmem_is_pmem() - common: fix various minor problems found by static analysis - pmem2: arm64: fix possible data loss on ARMv8.2+ (improper flushing) This release introduces no changes to the on-media layout and is fully compatible with the previous version of PMDK. Wed Oct 28 2020 Piotr Balcer * Version 1.10 This release introduces a new stable PMDK library, libpmem2, which is the next major release of libpmem. This library has an entirely new, but familiar, API that addresses many shortcomings of the previous version, while retaining all of its functionality. To learn more, see https://pmem.io/pmdk/libpmem2/ or libpmem2(7). The old library, libpmem, is still going to be maintained for the foreseeable future, but we'd like to encourage any new applications to leverage libpmem2. Wed Oct 28 2020 Piotr Balcer * Version 1.9.2 This release reverts an incorrect change in SDS handling "pool: disable SDS check if not supported", and introduces a proper fix for the issues that patch attempted to correct. Wed Sep 16 2020 Piotr Balcer * Version 1.9.1 Detailed list of bug fixes: - common: fix LIBFABRIC flags - common: Add runtime SDS check and disable - pool: disable SDS check if not supported - obj: fix failure atomicity bug in huge allocs - obj: add missing drain after ulog processing Fri Jul 3 2020 Piotr Balcer * Version 1.9 This release: - Switches the default instruction set for memcpy, memmove and memset implementations on x86 to AVX512, and introduces numerous performance improvements for those operations on AVX and SSE2 fallback paths. - Optimizes transactional allocations in libpmemobj by avoiding one extraneous cache miss and reducing the amount of work required to perform a reservation. - Introduces a new API in libpmemobj, pmemobj_tx_set_failure_behavior, that enables the application to control the behavior of aborting transactions. - Improves performance of pool creation on Windows by avoiding expensive physical page allocation during file allocation. - Stabilizes support for ppc64. Other changes: - pmem: mem[cpy|set] optimization when eADR is available - obj: detect msync failures in non-pmem variants of mem[cpy|move|set] Notable bug fixes: - core: fix Last_errormsg_get when NO_LIBPTHREAD is defined - pmem: read Unsafe Shutdown Count from region instead of interleave set - common: fix deep_flushes failing on platforms that don't need them - pmem: fix data cache flush on ppc64 - obj: fix run allocated recalculation Fri Jan 31 2020 Marcin Ślusarz * Version 1.8 This release: - Introduces new API in libpmemobj for user-assisted defragmentation (see pmemobj_defrag man page for details). - Introduces experimental support for PPC64LE architecture. - Introduces new flag in libpmemobj for opting-out of transaction aborts on failure (POBJ_TX_NO_ABORT), along with new variants of existing APIs that didn't accept flags (pmemobj_tx_xfree, pmemobj_tx_xpublish, pmemobj_tx_xlock, pmemobj_tx_xlog_append_buffer, pmemobj_tx_xstrdup, pmemobj_tx_xwcsdup). - Moves out libvmem and libvmmalloc to the new repository (https://github.com/pmem/vmem). Other changes: - obj: introduce new statistics useful for defrag - obj: introduce transient/persistent statistics enabling - obj: introduce pmemobj_tx_(set/get)_user_data funcs - obj: introduce pmemobj_(set/get)_user_data funcs - obj: disable workaround for offsetof() since VS 15.5 in public header - common: drop support for libndctl < 63 on Linux - pool: rename -N --no-exec to -d --dry-run Notable bug fixes: - obj: fix zone size calculations - obj: fix potential NULL-dereference in ulog_store - obj: fix unintended recursive locking during palloc - obj: fix lock release order in palloc publish - obj: fix transient redo log of more than 64 entries - obj: fix capacity ulog calculation - obj: fix check of unaligned capacity size - rpmem: add a missing case for GPSPM + FLUSH_STRICT - pmem: fix pmemcheck support on platforms with eADR - pool: fix possible memory leak - rpmem: fix possible memory leak in rpmemd_config_read - rpmem: fix possible memory leak in rpmemd_log_init - rpmem: fix possible use-after-free Mon Sep 30 2019 Marcin Ślusarz * Version 1.7 This release: - Introduces new APIs in libpmemobj for managing space used by transactions. (see pmemobj_tx_log_append_buffer man page for details) - Introduces new APIs in librpmem, splitting rpmem_persist into rpmem_flush and rpmem_drain, allowing applications to use the flush + drain model already known from libpmem. (libpmemobj does not use this feature yet) - Optimizes large libpmemobj transactions by significantly reducing the amount of memory modified at the commit phase. - Optimizes tracking of libpmemobj reservations. - Adds new flags for libpmemobj's pmemobj_tx_xadd_range[_direct] API: POBJ_XADD_NO_SNAPSHOT and POBJ_XADD_ASSUME_INITIALIZED, allowing applications to optimize how memory is tracked by the library. To support some of the above changes the libpmemobj on-media layout had to be changed, which means that old pools have to be converted using pmdk-convert >= 1.7. Other changes: - obj: fix merging of ranges when NOFLUSH flag is used (pmem/issues#1100) - rpmem: fix closing of ssh connection (pmem/issues#995, pmem/issues#1060) - obj: abort transaction on pmemobj_tx_publish failure Internal changes: - test: fault injection tests for pmemblk, pmemlog, and pmemobj - test: improved Python testing framework - test: support real pmem in bad blocks tests - common: allow not building examples and benchmarks Tue Aug 27 2019 Marcin Ślusarz * Version 1.6.1 This release fixes possible pool corruptions on Windows (see https://github.com/pmem/pmdk/pull/3728 for details), improves compatibility with newer Linux kernels with respect to Device DAX detection, fixes pmemobj space management for large pools, improves compatibility with newer toolchains, incorporates build fixes for FreeBSD and fixes a number of smaller bugs. Detailed list of bug fixes: - common: (win) fix possible pool file coruption (pmem/issues#972, pmem/issues#715, pmem/issues#603) - common: implement correct / robust device_dax_alignment (pmem/issues#1071) - obj: fix recycler not locating unused chunks - doc: update pmemobj_tx_lock documentation wrt behavior on fail - common: fix persistent domain detection (pmem/issues#1093) - common: vecq: fix a pointer-to-struct aliasing violation (crash on arm64) - common: fix minor issues related to ndctl linking - obj: drop recursion from pmemobj_next - common: fix bug in badblock file error handling - obj: fix handling of malloc failures - common: fix handling of malloc failures (ctl) - jemalloc: fix build with gcc 9 - obj: don't overwrite errno when palloc_heap_check_remote fails - doc: fix pmreorder emit log macro - rpmem: change order of rpmem init (pmem/issues#1043) - common: Fix build failure due to unused macro PAGE_SIZE - common: support older versions of pkg-config - tools: link with release variant of pmemcommon - common: add PMDK prefix to local copy of queue.h (pmem/issues#990) - rpmem: switch to using an open coded basename (FreeBSD) - common: posix_fallocate: guard against integer underflow in check (FreeBSD) - test: support Valgrind 3.15 - test: skip if fi_info is missing - test: (win) fix sparsefile error handling - test: fix libpmempool_feature tests that match logs - test: remove vmem_delete test (pmem/issues#1074) - test: adjust matchfiles in vmem_valgrind_region test (pmem/issues#1087) - test: remove old log files for windows (pmem/issues#1013) - test: remove invalid expect_normal_exit (pmem/issues#1092) - test: suppress ld leak (pmem/issues#1098) - test: Expose necessary symbols in libvmmalloc_dummy_funcs (FreeBSD) - test: fix tests failing because `tput` fails (FreeBSD) - test: avoid obj_critnib_mt taking very long on many-core machines - test: deal with libndctl's path without build system - test: overwrite old log in pmempool_create/TEST14.PS1 - test: fix match files in tests which use dax devices - test: fix match file in rpmem_addr_ext test - test: fix pmempool_check test Wed Aug 28 2019 Marcin Ślusarz * Version 1.5.2 This release fixes possible pool corruptions on Windows (see https://github.com/pmem/pmdk/pull/3728 for details), improves compatibility with newer Linux kernels with respect to Device DAX detection, fixes pmemobj space management for large pools, improves compatibility with newer toolchains and fixes a number of smaller bugs. Detailed list of bug fixes: - common: (win) fix possible pool file coruption (pmem/issues#972, pmem/issues#715, pmem/issues#603) - common: implement correct / robust device_dax_alignment (pmem/issues#1071) - obj: fix crash after large undo log recovery - obj: fix recycler not locating unused chunks - doc: update pmemobj_tx_lock documentation wrt behavior on fail - common: fix build of rpm packages on suse (pmem/issues#1023) - common: fix persistent domain detection (pmem/issues#1093) - common: vecq: fix a pointer-to-struct aliasing violation (crash on arm64) - rpmem: lock file prior to unlink (pmem/issues#833) - common: fix for pool_set error handling (pmem/issues#1036) - pmreorder: fix handling of store drain flush drain pattern - obj: fix possible memory leak in tx_add_lock - pool: free bad_block vector - common: fix bug in badblock file error handling - obj: fix handling of malloc failures - common: fix handling of malloc failures (ctl) - jemalloc: fix build with gcc 9 - obj: don't overwrite errno when palloc_heap_check_remote fails - doc: fix typos in pmreorder configuration - doc: fix pmreorder emit log macro - tools: link with release variant of pmemcommon - test: support Valgrind 3.15 - test: skip if fi_info is missing - test: split test obj_tx_lock into two test cases (pmem/issues#1027) - test: (win) fix sparsefile error handling - test: fix libpmempool_feature tests that match logs - test: remove vmem_delete test (pmem/issues#1074) - test: adjust matchfiles in vmem_valgrind_region test (pmem/issues#1087) - test: remove old log files for windows (pmem/issues#1013) - test: remove invalid expect_normal_exit (pmem/issues#1092) - test: suppress ld leak (pmem/issues#1098) - test: fix failing pmemdetect on Windows - test: fix match files in tests which use dax devices - test: fix pmempool_check test Fri Aug 30 2019 Marcin Ślusarz * Version 1.4.3 This release fixes possible pool corruptions on Windows (see https://github.com/pmem/pmdk/pull/3728 for details) and improves compatibility with newer Linux kernels with respect to Device DAX detection. Bug fixes: - common: (win) fix possible pool file coruption (pmem/issues#972, pmem/issues#715, pmem/issues#603) - common: implement correct / robust device_dax_alignment (pmem/issues#1071) - common: fix device dax detection - obj: fix pmemobj_check for pools with some sizes (pmem/issues#975) - obj: fix type numbers for pmemobj_list_insert_new - obj: fix pmemobj_tx_lock error handling - obj: fix possible memory leak in tx_add_lock - common: fix ctl_load_config during libpmemobj initialization (pmem/issues#917) - common: win: fix getopt returning "option is ambiguous" - common: fix persistent domain detection (pmem/issues#1093) - pool: do not copy same regions in update_uuids - test: split test obj_tx_lock into two test cases - test: remove checking errno in obj_tx_add_range_direct - test: remove invalid expect_normal_exit - test: fix int overflow in pmem_deep_persist test - test: fix pmempool_check test - test: (win) fix a few issues related to long paths Tue Aug 27 2019 Marcin Ślusarz * Version 1.3.3 Bug fixes: - pmem: fix clflush bit position - common: implement correct / robust device_dax_alignment - common: fix device dax detection - common: fix library dependencies (pmem/issues#767) - common: use rpm-config CFLAGS/LDFLAGS when building packages (pmem/issues#768) - test: fix vmmalloc_malloc_hooks (pmem/issues#773) - test: fix compilation with clang-5.0 (pmem/issues#783) - pool: fix set convert of v3 -> v4 - common: generate pkg-config files on make install (pmem/issues#610) - common: fix dependencies for Debian's dev packages - test: add missing include in unittest.h - common: (win) fix timed locks - common: provide src version in GitHub tarballs - common: fix free function in tls Tue Aug 27 2019 Marcin Ślusarz * Version 1.2.4 Bug fixes: - common: fix device dax detection (compatibility with newer kernels) Tue Mar 26 2019 Marcin Ślusarz * Version 1.6 This release: - Enables unsafe shutdown and bad block detection on Linux on systems with libndctl >= 63. It is expected that systems with libndctl >= 63 has necessary kernel support (Linux >= 4.20). However, due to bugs in libndctl = 63 and Linux = 4.20, it is recommended to use libndctl >= 64.1 and Linux >= 5.0.4. On systems with libndctl < 63, PMDK uses old superuser-only interfaces. Support for old or new interfaces is chosen at BUILD time. - Introduces arena control interface in pmemobj, allowing applications to tweak performance and scalability of heap operations. See pmemobj_ctl_get man page ("heap" namespace) for details. - Introduces copy_on_write mode, which allows testing applications using pmemobj with pmreorder. See pmemobj_ctl_get man page ("copy_on_write" namespace) for details. Other changes: - allocate file space when creating a pool on existing file (pmem/issues#167) - initial support for testing using fault injection - initial Python test framework - improve performance of pmemobj_pool_by_ptr Bug fixes: - common: work around tmpfs bug during pool creation (pmem/issues#1018) - pool: race-free pmempool create --max-size - obj: don't modify remote pools in pmemobj_check Tue Feb 19 2019 Marcin Ślusarz * Version 1.5.1 This release fixes minor bugs and improves compatibility with newer tool chains. Notable bug fixes: - common: make detection of device-dax instances more robust - obj: fix pmemobj_check for pools with some sizes - obj: don't use anon struct in an union (public header) - obj: fix pmemobj_tx_lock error handling - obj: don't use braces in an expression with clang (public header) - obj: suppress pmemcheck warnings for statistics - pmreorder: fix markers nontype issue Fri Oct 26 2018 Marcin Ślusarz * Version 1.5 This release has had two major focus areas - performance and RAS (Reliability, Availability and Serviceability). Beyond that, it introduces new APIs, new tools and many other improvements. As a side effect of performance optimizations, the libpmemobj on-media layout had to be changed, which means that old pools have to be converted using pmdk-convert. libpmemcto experiment has been finished and removed from the tree. For more details, please see https://pmem.io/2018/10/22/release-1-5.html. New features: - common: unsafe shutdown detection (SDS) - common: detection and repair of uncorrectable memory errors (bad blocks) - pool: new "feature" subcommand for enabling and disabling detection of unsafe shutdown and uncorrectable memory errors - common: auto flush detection on Windows (on Linux since 1.4) - pmreorder: new tool for verification of persistent memory algorithms - obj: new on media layout - pmem/obj: new flexible memcpy|memmove|memset API - obj: new flushing APIs: pmemobj_xpersist, pmemobj_xflush (PMEMOBJ_F_RELAXED) - rpmem: new flag RPMEM_PERSIST_RELAXED for rpmem_persist - obj: lazily initialized volatile variables (pmemobj_volatile) (EXPERIMENTAL) - obj: allocation classes with alignment - obj: new action APIs: pmemobj_defer_free, POBJ_XRESERVE_NEW, POBJ_XRESERVE_ALLOC - blk/log: new "ctl" API Optimizations: - obj: major performance improvements for AEP NVDIMMs - obj: better space utilization for small allocations - common: call msync only on one page for deep drain Other changes: - cto: removed - obj: remove actions limit - common: new dependency on libndctl on Linux - pmempool: "convert" subcommand is now a wrapper around pmdk-convert (please see https://github.com/pmem/pmdk-convert) - obj: C++ bindings have been moved to a new repository (please see https://github.com/pmem/libpmemobj-cpp) Bug fixes: - obj: fix type numbers for pmemobj_list_insert_new - pmem: fix inconsistency in pmem_is_pmem - common: fix windows mmap destruction - daxio: fix checking and adjusting length - common: fix long paths support on Windows Thu Aug 16 2018 Marcin Ślusarz * Version 1.4.2 This release fixes the way PMDK reports its version via pkg-config files. Bug fixes: - common: fix reported version - doc: use single "-" in NAME section (pmem/issues#914) Fri Jun 29 2018 Marcin Ślusarz * Version 1.4.1 In 1.4 development cycle, we created new daxio utility (command line tool for performing I/O on Device-DAX), but due to some complications we had to disable it just before the 1.4 release. In 1.4.1 we finally enable it. Daxio depends on ndctl v60.1. Bug fixes: - pmem: fix clflush bit position - obj: fix invalid OOMs when zones are fully packed - obj: don't register undo logs twice in memcheck - pool: fix bash completion script - pool: fix incorrect errno after transform - obj: fix clang-7 compilation - obj: test for msync failures in non-pmem path - doc: add missing field to alloc class entry point - common: (win) fix timed locks - common: provide src version in GitHub tarballs - common: fix free function in tls - common: fix double close - test: allow testing installed libraries - test: fix Valgrind vs stripped libraries issue - test: fix dependencies between tests and tools - test: fix races on make pcheck -jN - test: use libvmmalloc.so.1 - test: fix incorrect number of required dax devices - test: add suppression for leak in ld.so - test: fail if memcheck detects overlapping chunks - test: simplify time measurements in obj_sync - benchmark: check lseek() return value - examples: catch exceptions in map_cli Thu Mar 29 2018 Krzysztof Czurylo * Version 1.4 This is the first release of PMDK under a new name. The NVML project has been renamed to PMDK (Persistent Memory Development Kit). This is only the project/repo name change and it does not affect the names of the PMDK packages. See this blog article for more details on the reasons and impact of the name change: https://pmem.io/2017/12/11/NVML-is-now-PMDK.html New features: - common: support for concatenated Device-DAX devices with 2M/1G alignment - common: add support for MAP_SYNC flag - common: always enable Valgrind instrumentation (pmem/issues#292) - common: pool set options / headerless pools - pmem: add support for "deep flush" operation - rpmem: add rpmem_deep_persist - doc: split man pages and add per-function aliases (pmem/issues#385) Optimizations: - pmem: skip CPU cache flushing when eADR is available (no Windows support yet) - pmem: add AVX512F support in pmem_memcpy/memset (pmem/issues#656) Bug fixes: - common: fix library dependencies (pmem/issues#767, RHBZ #1539564) - common: use rpm-config CFLAGS/LDFLAGS when building packages (pmem/issues#768, RHBZ #1539564) - common: do not unload librpmem on close (pmem/issues#776) - common: fix NULL check in os_fopen (pmem/issues#813) - common: fix missing version in .pc files - obj: fix cancel of huge allocations (pmem/issues#726) - obj: fix error handling in pmemobj_open (pmem/issues#750) - obj: validate pe_offset in pmemobj_list_* APIs (pmem/issues#772) - obj: fix add_range with size == 0 (pmem/issues#781) - log: add check for negative iovcnt (pmem/issues#690) - rpmem: limit maximum number of lanes (pmem/issues#609) - rpmem: change order of memory registration (pmem/issues#655) - rpmem: fix removing remote pools (pmem/issues#721) - pool: fix error handling (pmem/issues#643) - pool: fix sync with switched parts (pmem/issues#730) - pool: fix sync with missing replica (pmem/issues#731) - pool: fix detection of Device DAX size (pmem/issues#805) - pool: fail pmempool_sync if there are no replicas (pmem/issues#816) - benchmark: fix calculating standard deviation (pmem/issues#318) - doc: clarify pmem_is_pmem behavior (pmem/issues#719) - doc: clarify pmemobj_root behavior (pmem/issues#733) Experimental features: - common: port PMDK to FreeBSD - common: add experimental support for aarch64 - obj: introduce allocation classes - obj: introduce two-phase heap ops (reserve/publish) (pmem/issues#380, pmem/issues#415) - obj: provide basic heap statistics (pmem/issues#676) - obj: implement run-time pool extending (pmem/issues#382) - cto: add close-to-open persistence library (pmem/issues#192) The following features are disabled by default, until ndctl v60.0 is available: - daxio: add utility to perform I/O on Device-DAX - RAS: unsafe shutdown detection/handling Wed Dec 20 2017 Krzysztof Czurylo * Version 1.3.1 Bug fixes: - rpmem: fix issues reported by Coverity - rpmem: fix read error handling - rpmem: add fip monitor (pmem/issues#597) - test: add rpmemd termination handling test - cpp: fix pop.persist function in obj_cpp_ptr - rpmem: return failure for a failed allocation - rpmem: fix potential memory leak - common: fix available rm options msg (pmem/issues#651) - pool: fix pmempool_get_max_size - obj: fix potential deadlock during realloc (pmem/issues#635, pmem/issues#636, pmem/issues#637) - obj: initialize TLS data - rpmem: fix cleanup if fork() failed (pmem/issues#634) - obj: fix bogus OOM after exhausting first zone Thu Jul 13 2017 Krzysztof Czurylo * Version 1.3 This release introduces some useful features and optimizations in libpmemobj. Most of them are experimental and controlled by the new pmemobj_ctl APIs. For details, please check the feature requests identified by the issue numbers listed next to the items below. Other important changes are related to performance tuning and stabilization of librpmem library, which is used by libpmemobj to get remote access to persistent memory and to provide basic data replication over RDMA. The librpmem is still considered experimental. NVML for Windows is feature complete (except for libvmmalloc). This release includes the support for Unicode, long paths and the NVML installer. New features: - common: add support for concatenated DAX Devices - common: add Unicode support on Windows - common: add long path support on Windows - common: add NVML installer for Windows - pmem: make pmem_is_pmem() true for Device DAX only - obj: add pmemobj_wcsdup()/pmemobj_tx_wcsdup() APIs - obj: export non-inlined pmemobj_direct() - obj: add PMEMOBJ_NLANES env variable - cpp: introduce the allocator - cpp: add wstring version of C++ entry points - vmem: add vmem_wcsdup() API entry - pool: add pmempool_rm() function (pmem/issues#307) - pool: add --force flag for create command (pmem/issues#529) - benchmark: add a minimal execution time option - benchmark: add thread affinity option - benchmark: print 99% and 99.9% percentiles - doc: separate Linux/Windows version of web-based man pages Optimizations: - obj: cache _pobj_cached_pool in pmemobj_direct() - obj: optimize thread utilization of buckets - obj: stop grabbing a lock when querying pool ptr - rpmem: use multiple endpoints Bug fixes: - common: fix issues reported by static code analyzers - pmem: fix mmap() implementation on Windows - pmem: fix mapping addr/length alignment on Windows - pmem: fix PMEM_MMAP_HINT implementation on Windows - pmem: fix pmem_is_pmem() on invalid memory ranges - pmem: fix wrong is_pmem returned by pmem_map_file() - pmem: fix mprotect() for private mappings on Windows - pmem: modify pmem_is_pmem() behavior for len==0 - obj: add failsafe to prevent allocs in constructor - cpp: fix swap implementation - cpp: fix sync primitives' constructors - cpp: fix wrong pointer type in the allocator - cpp: return persistent_ptr::swap to being public - pool: treat invalid answer as 'n' - pool: unify flags value for dry run - pool: transform for remote replicas - rpmem: persistency method detection - benchmark: fix time measurement Experimental features/optimizations: - obj: pmemobjctl - statistics and control submodule (pmem/issues#194, pmem/issues#211) - obj: zero-overhead allocations - customizable alloc header (pmem/issues#347) - obj: flexible run size index (pmem/issues#377) - obj: dynamic range cache (pmem/issues#378) - obj: asynchronous post-commit (pmem/issues#381) - obj: configurable object cache (pmem/issues#515) - obj: add cache size and threshold tx params - obj: add CTL var for suppressing expensive checks - rpmem: add rpmem_set_attr() API entry - rpmem: switch to libfabric v1.4.2 Thu May 18 2017 Krzysztof Czurylo * Version 1.2.3 Bug fixes: - test: extend timeout for selected tests - test: reduce number of operations in obj_tx_mt - test: define cfree() as free() in vmmalloc_calloc Other changes: - common: move Docker images to new repo Sat Apr 15 2017 Krzysztof Czurylo * Version 1.2.2 Bug fixes: - pmempool: fix mapping type in pool_params_parse - test: limit number of arenas in vmem_stats - test: do not run pool_lock test as root - common: fix pkg-config files - common: fix building packages for Debian Tue Feb 21 2017 Krzysztof Czurylo * Version 1.2.1 This NVML release changes the behavior of pmem_is_pmem() on Linux. The pmem_is_pmem() function will now return true only if the entire range is mapped directly from Device DAX (/dev/daxX.Y) without an intervening file system, and only if the corresponding file mapping was created with pmem_map_file(). See libpmem(7) for details. Bug fixes: - jemalloc: fix test compilation on Fedora 26 (rawhide) - test: fix cpp test compilation on Fedora 26 (rawhide) - common: use same queue.h on linux and windows - common: queue.h clang static analyzer fix - common: fix path handling in build-dpkg.sh - test: fix match files in pmempool_transform/TEST8 Fri Dec 30 2016 Krzysztof Czurylo * Version 1.2 - Windows Technical Preview #1 This is the first Technical Preview release of NVML for Windows. It is based on NVML 1.2 version, but not all the 1.2 features are ported to Windows. In particular, Device DAX and remote access to persistent memory (librpmem) are not supported by design. NOTE: This release has not gone through the full validation cycle, but only through some basic tests on Travis and AppVeyor. Thus, it cannot be assumed "Production quality" and should not be used in production environments. Besides several minor improvements and bug fixes, all the other changes since NVML 1.2 release were related to Windows support: - win: port libvmem (and jemalloc) - win: benchmarks Windows port - win: fix mapping files of unaligned length - win: clean up possible race condition in mmap_init() - win: enable QueryVirtualMemoryInformation() in pmem_is_pmem() - test: check open handles at START/DONE - test: port all the remaining unit tests (scope, pmem_map, obj_debug, util_poolset, pmempool_*) - win: add resource files for versioning Known issues and limitations of Windows version of NVML: - Unicode support is missing. The UTF/USC-encoded file paths or pool set files may not be handled correctly. - The libvmmalloc library is not ported yet. - The on-media format of pmem pools is not portable at the moment. The pmem pools created using Windows version of NVM libraries cannot be open on Linux and vice versa. - Despite the fact the current version of NVML would work with any recent version of Windows OS, to take full advantage of PMEM and NVML features and to benefit from the PMEM performance, the recommended platforms needs be equipped with the real NVDIMMs hardware and should support the native, Microsoft's implementation of DAX-enabled file system (i.e. Windows Server 2016 or later). In case of using NVML with older versions of Windows or with the custom implementation of PMEM/DAX drivers, the performance might not be satisfactory. Please, contact the provider of PMEM/DAX drivers for your platform to get the customized version of NVML in such case. Thu Dec 15 2016 Krzysztof Czurylo * Version 1.2 This NVML release causes a "flag day" for libpmemobj. The pmemobj pools built under NVML 1.1 are incompatible with pools built under NVML 1.2 and later. This is because an issue was discovered with the alignment of locks (pmem/issues#358) and, although rare, the issue potentially impacts program correctness, making the fix mandatory. The major version number of the pmemobj pool layout and the version of the libpmemobj API is changed to prevent the use of the potentially incorrect layout. Other key changes introduced in this release: - Add Device DAX support, providing that "optimized flush" mechanism defined in SNIA NVM Programming Model can safely be used, even if PMEM-aware file system supporting that model is not available, or if the user does not want to use the file system for some reason. - Add a package for libpmemobj C++ bindings. C++ API is no longer considered experimental. Web-based documentation for C++ API is available on https://pmem.io. - Add "sync" and "transform" commands to pmempool utility. The "sync" command allows one to recover missing or corrupted part(s) of a pool set from a healthy replica, while the "transform" command is a convenient way for modifying the structure of an existing pool set, i.e. by adding or removing replicas. - Add experimental support for remote access to persistent memory and basic remote data replication over RDMA (librpmem). Experimental support for remote replicas is also provided by libpmemobj library. New features: - common: add Device DAX support (pmem/issues#197) - obj: add C++ bindings package (libpmemobj++-devel) - obj: add TOID_OFFSETOF macro - pmempool: add "sync" and "transform" commands (pmem/issues#172, pmem/issues#196) Bug fixes: - obj: force alignment of pmem lock structures (pmem/issues#358) - blk: cast translation entry to uint64_t when calculating data offset - obj: fix Valgrind instrumentation of chunk headers and cancelled allocations - obj: set error message when user called pmemobj_tx_abort() - obj: fix status returned by pmemobj_list_insert() (pmem/issues#226) - obj: defer allocation of global structures Optimizations: - obj: fast path for pmemobj_pool_by_ptr() when inside a transaction - obj: simplify and optimize allocation class generation Experimental features: - rpmem: add support for remote access to persistent memory and basic remote data replication over RDMA - libpmempool: add pmempool_sync() and pmempool_transform() (pmem/issues#196) - obj: introduce pmemobj_oid() - obj: add pmemobj_tx_xalloc()/pmemobj_tx_xadd_range() APIs and the corresponding macros - obj: add transaction stage transition callbacks Thu Jun 23 2016 Krzysztof Czurylo * Version 1.1 This NVML release introduces a new version of libpmemobj pool layout. Internal undo log structure has been modified to improve performance of pmemobj transactions. Memory pools created with older versions of the libpmemobj library must be converted to the new format using "pmempool convert" command. See pmempool-convert(1) for details. A new "libpmempool" library is available, providing support for off-line pool management and diagnostics. Initially it provides only "check" and "repair" operations for log and blk memory pools, and for BTT devices. Other changes: - pmem: deprecate PCOMMIT - blk: match BTT Flog initialization with Linux NVDIMM BTT - pmem: defer pmem_is_pmem() initialization (pmem/issues#158) - obj: add TOID_TYPEOF macro Bug fixes: - doc: update description of valid file size units (pmem/issues#133) - pmempool: fix --version short option in man page (pmem/issues#135) - pmempool: print usage when running rm without arg (pmem/issues#136) - cpp: clarify polymorphism in persistent_ptr (pmem/issues#150) - obj: let the before flag be any non-zero value (pmem/issues#151) - obj: fix compare array pptr to nullptr (pmem/issues#152) - obj: cpp pool.get_root() fix (pmem/issues#156) - log/blk: set errno if replica section is specified (pmem/issues#161) - cpp: change exception message (pmem/issues#163) - doc: remove duplicated words in man page (pmem/issues#164) - common: always append EXTRA_CFLAGS after our CFLAGS Experimental features: - Implementation of C++ bindings for libpmempobj is complete. Web-based documentation for C++ API is available on https://pmem.io. Note that C++ API is still considered experimental. Do not use it in production environments. - Porting NVML to Windows is in progress. There are MS Visual Studio solution/projects available, allowing to compile libpmem, libpmemlog, libpmemblk and libpmemobj on Windows, but the libraries are not fully functional and most of the test are not enabled yet. Thu Apr 07 2016 Krzysztof Czurylo * Version 1.0 The API of six libraries (libpmem, libpmemblk, libpmemlog, libpmemobj, libvmem, libvmmalloc) is complete and stable. The on-media layout of persistent memory pools will be maintained from this point, and if changed it will be backward compatible. Man pages are all complete. This release has been validated to "Production quality". For the purpose of new features planned for next releases of NVML there have been some API modifications made: - pmem: pmem_map replaced with pmem_map_file - log/blk: 'off_t' substituted with 'long long' - obj: type numbers extended to 64-bit - obj: new entry points and macros added: pmemobj_tx_errno, pmemobj_tx_lock, pmemobj_mutex_timedlock, TX_ADD_DIRECT, TX_ADD_FIELD_DIRECT, TX_SET_DIRECT Other key changes since version 0.4 include: - common: updated/fixed installation scripts - common: eliminated dependency on libuuid - pmem: CPU features/ISA detection using CPUID - obj: improved error handling - obj: atomic allocation fails if constructor returns error - obj: multiple performance optimizations - obj: object store refactoring - obj: additional examples and benchmarks This release also introduces a prototype implementation of C++ bindings for libpmemobj. Note that C++ API is still experimental and should not be used in production environments. Fri Dec 04 2015 Krzysztof Czurylo * Version 0.4 This NVML version primarily focuses on improving code quality and reliability. In addition to a couple of bug fixes, the changes include: - benchmarks for libpmemobj, libpmemblk and libvmem - additional pmemobj tests and examples - pool mapping address randomization - added pmempool "rm" command - eliminated libpmem dependency on libpthread - enabled extra warnings - minor performance improvements Man pages are all complete. This release is considered "Beta quality" by the team, having been thoroughly validated, including significant performance analysis. The pmempool command does not yet support "check" and "repair" operations for pmemobj type pools. Sun Sep 13 2015 Andy Rudoff * Version 0.3 NVML is now feature complete, adding support for: - pool sets - pmemobj local replication (active/passive) - experimental valgrind support - pmempool support for all pool types Man pages are all complete. This release is considered "Alpha quality" by the team, having gone through significant validation but only some performance analysis at this point. Tue Jun 30 2015 Andy Rudoff * Version 0.2 NVML now consists of six libraries: - libpmem (basic flushing, etc) - libpmemblk, libpmemlog, libpmemobj (transactions) - libvmem, libvmmalloc (volatile use of pmem) The "pmempool" command is available for managing pmem files. Man pages for all the above are complete. The only things documented in man pages but not implemented are: - pmem sets (ability to spread a pool over a set of files) - replication (coming for libpmemobj) The pmempool command does not yet support pmemobj type pools. Thu Sep 11 2014 Andy Rudoff * Version 0.1 Initial development done in 0.1 builds pmdk-1.13.1/.skip-doc0000664000000000000000000000000014435627653012747 0ustar rootrootpmdk-1.13.1/README.md0000664000000000000000000004407514435627501012530 0ustar rootroot# **PMDK: Persistent Memory Development Kit** [![Travis build status](https://travis-ci.org/pmem/pmdk.svg?branch=master)](https://travis-ci.org/pmem/pmdk) [![GHA build status](https://github.com/pmem/pmdk/workflows/PMDK/badge.svg?branch=master)](https://github.com/pmem/pmdk/actions) [![Cirrus build status](https://api.cirrus-ci.com/github/pmem/pmdk.svg)](https://cirrus-ci.com/github/pmem/pmdk/master) [![Coverity Scan Build Status](https://img.shields.io/coverity/scan/3015.svg)](https://scan.coverity.com/projects/pmem-pmdk) [![Coverage Status](https://codecov.io/github/pmem/pmdk/coverage.svg?branch=master)](https://codecov.io/gh/pmem/pmdk/branch/master) [![PMDK release version](https://img.shields.io/github/release/pmem/pmdk.svg?sort=semver)](https://github.com/pmem/pmdk/releases/latest) [![Packaging status](https://repology.org/badge/tiny-repos/pmdk.svg)](https://repology.org/project/pmdk/versions) [![CodeQL status](https://github.com/pmem/pmemstream/actions/workflows/codeql.yml/badge.svg?branch=master)](https://github.com/pmem/pmemstream/actions/workflows/codeql.yml) [![Security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg?branch=master)](https://github.com/pmem/pmdk/actions/workflows/bandit.yml) The **Persistent Memory Development Kit (PMDK)** is a collection of libraries and tools for System Administrators and Application Developers to simplify managing and accessing persistent memory devices. For more information, see https://pmem.io. To install PMDK libraries, either install pre-built packages, which we build for every stable release, or clone the tree and build it yourself. **Pre-built** packages can be found in popular Linux distribution package repositories, or you can check out our recent stable releases on our [github release page](https://github.com/pmem/pmdk/releases). Specific installation instructions are outlined below. Bugs and feature requests for this repo are tracked in our [GitHub Issues Database](https://github.com/pmem/pmdk/issues). ## Contents 1. [Libraries and Utilities](#libraries-and-utilities) 2. [Getting Started](#getting-started) 3. [Version Conventions](#version-conventions) 4. [Pre-Built Packages for Windows](#pre-built-packages-for-windows) 5. [Dependencies](#dependencies) * [Linux](#linux) * [Windows](#windows) * [FreeBSD](#freebsd) 6. [Building PMDK on Linux or FreeBSD](#building-pmdk-on-linux-or-freebsd) * [Make Options](#make-options) * [Testing Libraries](#testing-libraries-on-linux-and-freebsd) * [Memory Management Tools](#memory-management-tools) 7. [Building PMDK on Windows](#building-pmdk-on-windows) * [Testing Libraries](#testing-libraries-on-windows) 8. [Debugging](#debugging) 9. [Experimental Packages](#experimental-packages) * [Experimental support for 64-bit ARM](#experimental-support-for-64-bit-arm) 10. [Contact Us](#contact-us) ## Libraries and Utilities All PMDK related libraries are described in detail on [pmem.io/pmdk](https://pmem.io/pmdk/). Libraries available in this repository: - [libpmem](https://pmem.io/pmdk/libpmem/): provides low level persistent memory support. - [libpmem2](https://pmem.io/pmdk/libpmem2/): provides low level persistent memory support, is a new version of libpmem. > NOTICE: Support for async functions is deprecated since PMDK 1.13.0 release and will be removed in the PMDK 1.14.0 release along with the miniasync dependency. - [libpmemobj](https://pmem.io/pmdk/libpmemobj/): provides a transactional object store, providing memory allocation, transactions, and general facilities for persistent memory programming. - [libpmemblk](https://pmem.io/pmdk/libpmemblk/): supports arrays of pmem-resident blocks, all the same size, that are atomically updated. (DEPRECATED) > NOTICE: The **libpmemblk** library is deprecated since PMDK 1.13.0 release and will be removed in the PMDK 1.14.0 release. - [libpmemlog](https://pmem.io/pmdk/libpmemlog/): provides a pmem-resident log file. (DEPRECATED) > NOTICE: The **libpmemlog** library is deprecated since PMDK 1.13.0 release and will be removed in the PMDK 1.14.0 release. - [libpmempool](https://pmem.io/pmdk/libpmempool/): provides support for off-line pool management and diagnostics. **Libpmemset** has been removed from PMDK repository. **Librpmem** library has been removed from PMDK repository. If you are interested in a remote persistent memory support please look at new [librpma](https://github.com/pmem/rpma). If you're looking for *libvmem* and *libvmmalloc*, they have been moved to a [separate repository](https://github.com/pmem/vmem). Available Utilities: - [pmempool](https://pmem.io/pmdk/pmempool/): Manage and analyze persistent memory pools with this stand-alone utility - [pmemcheck](https://pmem.io/2015/07/17/pmemcheck-basic.html): Use dynamic runtime analysis with an enhanced version of Valgrind for use with persistent memory. Currently these libraries only work on 64-bit Linux, Windows2, and 64-bit FreeBSD 11+3. For information on how these libraries are licensed, see our [LICENSE](LICENSE) file. > NOTICE: Support for Windows and FreeBSD are deprecated since PMDK 1.13.0 release and will be removed in the PMDK 1.14.0 release. >1 Not supported on Windows. > >2 PMDK for Windows is feature complete, but not yet considered production quality. > >3 DAX is not yet supported in FreeBSD, so at this time PMDK is available as a technical preview release for development purposes. ## Getting Started Getting Started with Persistent Memory Programming is a tutorial series created by Intel architect, Andy Rudoff. In this tutorial, you will be introduced to persistent memory programming and learn how to apply it to your applications. - Part 1: [What is Persistent Memory?](https://software.intel.com/en-us/persistent-memory/get-started/series) - Part 2: [Describing The SNIA Programming Model](https://software.intel.com/en-us/videos/the-nvm-programming-model-persistent-memory-programming-series) - Part 3: [Introduction to PMDK Libraries](https://software.intel.com/en-us/videos/intro-to-the-nvm-libraries-persistent-memory-programming-series) - Part 4: [Thinking Transactionally](https://software.intel.com/en-us/videos/thinking-transactionally-persistent-memory-programming-series) - Part 5: [A C++ Example](https://software.intel.com/en-us/videos/a-c-example-persistent-memory-programming-series) Additionally, we recommend reading [Introduction to Programming with Persistent Memory from Intel](https://software.intel.com/en-us/articles/introduction-to-programming-with-persistent-memory-from-intel) ## Version Conventions - **Builds** are tagged something like `0.2+b1`, which means _Build 1 on top of version 0.2_ - **Release Candidates** have a '-rc{version}' tag, e.g. `0.2-rc3`, meaning _Release Candidate 3 for version 0.2_ - **Stable Releases** use a _major.minor_ tag like `0.2` ## Pre-Built Packages for Windows > NOTICE: Support for Windows is deprecated since PMDK 1.13.0 release and will be removed in the PMDK 1.14.0 release. The recommended and easiest way to install PMDK on Windows is to use Microsoft vcpkg. Vcpkg is an open source tool and ecosystem created for library management. To install the latest PMDK release and link it to your Visual Studio solution you first need to clone and set up vcpkg on your machine as described on the [vcpkg github page](https://github.com/Microsoft/vcpkg) in **Quick Start** section. In brief: ``` > git clone https://github.com/Microsoft/vcpkg > cd vcpkg > .\bootstrap-vcpkg.bat > .\vcpkg integrate install > .\vcpkg install pmdk:x64-windows ``` The last command can take a while - it is PMDK building and installation time. After a successful completion of all of the above steps, the libraries are ready to be used in Visual Studio and no additional configuration is required. Just open VS with your already existing project or create a new one (remember to use platform **x64**) and then include headers to project as you always do. ## Dependencies Required packages for each supported OS are listed below. It is important to note that some tests and example applications require additional packages, but they do not interrupt building if they are missing. An appropriate message is displayed instead. For details please read the DEPENDENCIES section in the appropriate README file (in tests/ or examples/ sub-directories). See our **[Dockerfiles](utils/docker/images)** (used e.g. on our CI systems) to get an idea what packages are required to build the entire PMDK, with all the tests and examples. ### Linux You will need to install the following required packages on the build system: * **autoconf** * **pkg-config** * **libndctl-devel** (v63 or later)1 * **libdaxctl-devel** (v63 or later) * **pandoc** (for documentation, required during install) The following packages are required only by selected PMDK components or features: >1 PMDK depends on libndctl to support RAS features. It is possible to disable this support by passing NDCTL_ENABLE=n to "make", but we strongly discourage users from doing that. Disabling NDCTL strips PMDK from ability to detect hardware failures, which may lead to silent data corruption. For information how to disable RAS at runtime for kernels prior to 5.0.4 please see https://github.com/pmem/pmdk/issues/4207. ### Windows > NOTICE: Support for Windows is deprecated since PMDK 1.13.0 release and will be removed in the PMDK 1.14.0 release. * **MS Visual Studio 2022** * [Windows SDK 10.0.22000.0](https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/) * **Windows, version >= 1803** * **perl** (e.g. [StrawberryPerl](http://strawberryperl.com/)) * **PowerShell 5** ### FreeBSD > NOTICE: Support for FreeBSD is deprecated since PMDK 1.13.0 release and will be removed in the PMDK 1.14.0 release. * **autoconf** * **bash** * **binutils** * **coreutils** * **e2fsprogs-libuuid** * **gmake** * **libunwind** * **ncurses**4 * **pkgconf** >4 The pkg version of ncurses is required for proper operation; the base version included in FreeBSD is not sufficient. ## Building PMDK on Linux or FreeBSD > NOTICE: Support for FreeBSD is deprecated since PMDK 1.13.0 release and will be removed in the PMDK 1.14.0 release. To build from source, clone this tree: ``` $ git clone https://github.com/pmem/pmdk $ cd pmdk ``` For a stable version, checkout a [release tag](https://github.com/pmem/pmdk/releases) as follows. Otherwise skip this step to build the latest development release. ``` $ git checkout tags/1.13.1 ``` Once the build system is setup, the Persistent Memory Development Kit is built using the `make` command at the top level: ``` $ make ``` For FreeBSD, use `gmake` rather than `make`. By default, all code is built with the `-Werror` flag, which fails the whole build when the compiler emits any warning. This is very useful during development, but can be annoying in deployment. If you want to **disable -Werror**, use the EXTRA_CFLAGS variable: ``` $ make EXTRA_CFLAGS="-Wno-error" ``` >or ``` $ make EXTRA_CFLAGS="-Wno-error=$(type-of-warning)" ``` ### Make Options There are many options that follow `make`. If you want to invoke make with the same variables multiple times, you can create a user.mk file in the top level directory and put all variables there. For example: ``` $ cat user.mk EXTRA_CFLAGS_RELEASE = -ggdb -fno-omit-frame-pointer PATH += :$HOME/valgrind/bin ``` This feature is intended to be used only by developers and it may not work for all variables. Please do not file bug reports about it. Just fix it and make a PR. **Built-in tests:** can be compiled and ran with different compiler. To do this, you must provide the `CC` and `CXX` variables. These variables are independent and setting `CC=clang` does not set `CXX=clang++`. For example: ``` $ make CC=clang CXX=clang++ ``` Once make completes, all the libraries and examples are built. You can play with the library within the build tree, or install it locally on your machine. For information about running different types of tests, please refer to the [src/test/README](src/test/README). **Installing the library** is convenient since it installs man pages and libraries in the standard system locations: ``` (as root...) # make install ``` To install this library into **other locations**, you can use the `prefix` variable, e.g.: ``` $ make install prefix=/usr/local ``` This will install files to /usr/local/lib, /usr/local/include /usr/local/share/man. **Prepare library for packaging** can be done using the DESTDIR variable, e.g.: ``` $ make install DESTDIR=/tmp ``` This will install files to /tmp/usr/lib, /tmp/usr/include /tmp/usr/share/man. **Man pages** (groff files) are generated as part of the `install` rule. To generate the documentation separately, run: ``` $ make doc ``` This call requires the following dependencies: **pandoc**. Pandoc is provided by the hs-pandoc package on FreeBSD. **Install copy of source tree** can be done by specifying the path where you want it installed. ``` $ make source DESTDIR=some_path ``` For this example, it will be installed at $(DESTDIR)/pmdk. **Build rpm packages** on rpm-based distributions is done by: ``` $ make rpm ``` To build rpm packages without running tests: ``` $ make BUILD_PACKAGE_CHECK=n rpm ``` This requires **rpmbuild** to be installed. **Build dpkg packages** on Debian-based distributions is done by: ``` $ make dpkg ``` To build dpkg packages without running tests: ``` $ make BUILD_PACKAGE_CHECK=n dpkg ``` This requires **devscripts** to be installed. ### Testing Libraries on Linux and FreeBSD > NOTICE: Support for FreeBSD is deprecated since PMDK 1.13.0 release and will be removed in the PMDK 1.14.0 release. You will need to install the following package to run unit tests: * **ndctl** Before running the tests, you may need to prepare a test configuration file (src/test/testconfig.sh). Please see the available configuration settings in the example file [src/test/testconfig.sh.example](src/test/testconfig.sh.example). To build and run the **unit tests**: ``` $ make check ``` To run a specific **subset of tests**, run for example: ``` $ make check TEST_TYPE=short TEST_BUILD=debug TEST_FS=pmem ``` To **modify the timeout** which is available for **check** type tests, run: ``` $ make check TEST_TIME=1m ``` This will set the timeout to 1 minute. Please refer to the **src/test/README** for more details on how to run different types of tests. ### Memory Management Tools The PMDK libraries support standard Valgrind DRD, Helgrind and Memcheck, as well as a PM-aware version of [Valgrind](https://github.com/pmem/valgrind) (not yet available for FreeBSD). By default, support for all tools is enabled. If you wish to disable it, supply the compiler with **VG_\_ENABLED** flag set to 0, for example: ``` $ make EXTRA_CFLAGS=-DVG_MEMCHECK_ENABLED=0 ``` **VALGRIND_ENABLED** flag, when set to 0, disables all Valgrind tools (drd, helgrind, memcheck and pmemcheck). The **SANITIZE** flag allows the libraries to be tested with various sanitizers. For example, to test the libraries with AddressSanitizer and UndefinedBehaviorSanitizer, run: ``` $ make SANITIZE=address,undefined clobber check ``` ## Building PMDK on Windows > NOTICE: Support for Windows is deprecated since PMDK 1.13.0 release and will be removed in the PMDK 1.14.0 release. Clone the PMDK tree and open the solution: ``` > git clone https://github.com/pmem/pmdk > cd pmdk/src > devenv PMDK.sln ``` Select the desired configuration (Debug or Release) and build the solution (i.e. by pressing Ctrl-Shift-B). ### Testing Libraries on Windows > NOTICE: Support for Windows is deprecated since PMDK 1.13.0 release and will be removed in the PMDK 1.14.0 release. Before running the tests, you may need to prepare a test configuration file (src/test/testconfig.ps1). Please see the available configuration settings in the example file [src/test/testconfig.ps1.example](src/test/testconfig.ps1.example). To **run the unit tests**, open the PowerShell console and type: ``` > cd pmdk/src/test > RUNTESTS.ps1 ``` To run a specific **subset of tests**, run for example: ``` > RUNTESTS.ps1 -b debug -t short ``` To run **just one test**, run for example: ``` > RUNTESTS.ps1 -b debug -i pmem_is_pmem ``` To **modify the timeout**, run: ``` > RUNTESTS.ps1 -o 3m ``` This will set the timeout to 3 minutes. To **display all the possible options**, run: ``` > RUNTESTS.ps1 -h ``` Please refer to the **[src/test/README](src/test/README)** for more details on how to run different types of tests. ## Debugging To enable logging of debug information, use debug version of a library and set desired log level using (library-specific) variable, e.g. `PMEM_LOG_LEVEL=`. For more details see appropriate manpage (debbuging section), e.g. [libpmem(7)](https://pmem.io/pmdk/manpages/linux/master/libpmem/libpmem.7.html#debugging-and-error-handling). ## Experimental Packages Some components in the source tree are treated as experimental. By default, those components are built but not installed (and thus not included in packages). If you want to build/install experimental packages run: ``` $ make EXPERIMENTAL=y [install,rpm,dpkg] ``` ### Experimental Support for 64-bit ARM and RISC-V There is an initial support for 64-bit ARM and RISC-V processors provided. While PMDK's internal testsuite passes on DRAM-only systems, support for neither of these architectures has been validated on any persistent memory hardware, nor has the code received review from any person with professional knowledge of either of these platforms. Thus, these architectures should not be used in a production environment. ### PowerPC support PowerPC support is ppc64le only and includes all libraries. They should build and pass all tests. The on-media pool layout is tightly attached to the page size of 64KiB used by default on ppc64le, so it is not interchangeable with different page sizes, includes those on other architectures. For more information on this port, contact Rajalakshmi Srinivasaraghavan (rajis@linux.ibm.com) or Lucas Magalhães (lucmaga@gmail.com). ## Contact Us For more information on this library, contact Piotr Balcer (piotr.balcer@intel.com), Andy Rudoff (andy.rudoff@intel.com), or post to our [Google group](https://groups.google.com/group/pmem). pmdk-1.13.1/src/0000775000000000000000000000000014435627501012026 5ustar rootrootpmdk-1.13.1/src/.gitignore0000664000000000000000000000027414435627501014021 0ustar rootroot!/core/ *.so *.so.* *.a *.pc tags TAGS cscope.in.out cscope.out cscope.po.out debug/ nondebug/ *.sdf *.opensdf *.opendb *.log *.suo *.vcxproj.user .vs/ x64/ Generated files/ srcversion.h pmdk-1.13.1/src/libpmem/0000775000000000000000000000000014435627501013453 5ustar rootrootpmdk-1.13.1/src/libpmem/pmem_windows.c0000664000000000000000000001405214435627501016331 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016-2019, Intel Corporation */ /* * Copyright (c) 2016, Microsoft Corporation. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * * Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * * Neither the name of the copyright holder nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ /* * pmem_windows.c -- pmem utilities with OS-specific implementation */ #include #include "pmem.h" #include "out.h" #include "mmap.h" #include "win_mmap.h" #include "sys/mman.h" #if (NTDDI_VERSION >= NTDDI_WIN10_RS1) typedef BOOL (WINAPI *PQVM)( HANDLE, const void *, enum WIN32_MEMORY_INFORMATION_CLASS, PVOID, SIZE_T, PSIZE_T); static PQVM Func_qvmi = NULL; #endif /* * is_direct_mapped -- (internal) for each page in the given region * checks with MM, if it's direct mapped. */ static int is_direct_mapped(const void *begin, const void *end) { LOG(3, "begin %p end %p", begin, end); #if (NTDDI_VERSION >= NTDDI_WIN10_RS1) int retval = 1; WIN32_MEMORY_REGION_INFORMATION region_info; SIZE_T bytes_returned; if (Func_qvmi == NULL) { LOG(4, "QueryVirtualMemoryInformation not supported, " "assuming non-DAX."); return 0; } const void *begin_aligned = (const void *)rounddown((intptr_t)begin, Pagesize); const void *end_aligned = (const void *)roundup((intptr_t)end, Pagesize); for (const void *page = begin_aligned; page < end_aligned; page = (const void *)((char *)page + Pagesize)) { if (Func_qvmi(GetCurrentProcess(), page, MemoryRegionInfo, ®ion_info, sizeof(region_info), &bytes_returned)) { retval = region_info.DirectMapped; } else { LOG(4, "QueryVirtualMemoryInformation failed, assuming " "non-DAX. Last error: %08x", GetLastError()); retval = 0; } if (retval == 0) { LOG(4, "page %p is not direct mapped", page); break; } } return retval; #else /* if the MM API is not available the safest answer is NO */ return 0; #endif /* NTDDI_VERSION >= NTDDI_WIN10_RS1 */ } /* * is_pmem_detect -- implement pmem_is_pmem() * * This function returns true only if the entire range can be confirmed * as being direct access persistent memory. Finding any part of the * range is not direct access, or failing to look up the information * because it is unmapped or because any sort of error happens, just * results in returning false. */ int is_pmem_detect(const void *addr, size_t len) { LOG(3, "addr %p len %zu", addr, len); if (len == 0) return 0; if (len > UINTPTR_MAX - (uintptr_t)addr) { len = UINTPTR_MAX - (uintptr_t)addr; LOG(4, "limit len to %zu to not get beyond address space", len); } int retval = 1; const void *begin = addr; const void *end = (const void *)((char *)addr + len); LOG(4, "begin %p end %p", begin, end); AcquireSRWLockShared(&FileMappingQLock); PFILE_MAPPING_TRACKER mt; PMDK_SORTEDQ_FOREACH(mt, &FileMappingQHead, ListEntry) { if (mt->BaseAddress >= end) { LOG(4, "ignoring all mapped ranges beyond given range"); break; } if (mt->EndAddress <= begin) { LOG(4, "skipping all mapped ranges before given range"); continue; } if (!(mt->Flags & FILE_MAPPING_TRACKER_FLAG_DIRECT_MAPPED)) { LOG(4, "tracked range [%p, %p) is not direct mapped", mt->BaseAddress, mt->EndAddress); retval = 0; break; } /* * If there is a gap between the given region that we process * currently and the mapped region in our tracking list, we * need to process the gap by taking the long route of asking * MM for each page in that range. */ if (begin < mt->BaseAddress && !is_direct_mapped(begin, mt->BaseAddress)) { LOG(4, "untracked range [%p, %p) is not direct mapped", begin, mt->BaseAddress); retval = 0; break; } /* push our begin to reflect what we have already processed */ begin = mt->EndAddress; } /* * If we still have a range to verify, check with MM if the entire * region is direct mapped. */ if (begin < end && !is_direct_mapped(begin, end)) { LOG(4, "untracked end range [%p, %p) is not direct mapped", begin, end); retval = 0; } ReleaseSRWLockShared(&FileMappingQLock); LOG(4, "returning %d", retval); return retval; } /* * pmem_map_register -- memory map file and register mapping */ void * pmem_map_register(int fd, size_t len, const char *path, int is_dev_dax) { /* there is no device dax on windows */ ASSERTeq(is_dev_dax, 0); return util_map(fd, 0, len, MAP_SHARED, 0, 0, NULL); } /* * pmem_os_init -- os-dependent part of pmem initialization */ void pmem_os_init(is_pmem_func *func) { LOG(3, NULL); *func = is_pmem_detect; #if NTDDI_VERSION >= NTDDI_WIN10_RS1 Func_qvmi = (PQVM)GetProcAddress( GetModuleHandle(TEXT("KernelBase.dll")), "QueryVirtualMemoryInformation"); #endif } pmdk-1.13.1/src/libpmem/pmem.h0000664000000000000000000000214614435627501014565 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2014-2022, Intel Corporation */ /* * pmem.h -- internal definitions for libpmem */ #ifndef PMEM_H #define PMEM_H #include #include "alloc.h" #include "fault_injection.h" #include "util.h" #include "valgrind_internal.h" #ifdef __cplusplus extern "C" { #endif #define PMEM_LOG_PREFIX "libpmem" #define PMEM_LOG_LEVEL_VAR "PMEM_LOG_LEVEL" #define PMEM_LOG_FILE_VAR "PMEM_LOG_FILE" typedef int (*is_pmem_func)(const void *addr, size_t len); void pmem_init(void); void pmem_os_init(is_pmem_func *func); int is_pmem_detect(const void *addr, size_t len); void *pmem_map_register(int fd, size_t len, const char *path, int is_dev_dax); #if FAULT_INJECTION void pmem_inject_fault_at(enum pmem_allocation_type type, int nth, const char *at); int pmem_fault_injection_enabled(void); #else static inline void pmem_inject_fault_at(enum pmem_allocation_type type, int nth, const char *at) { SUPPRESS_UNUSED(type, nth, at); abort(); } static inline int pmem_fault_injection_enabled(void) { return 0; } #endif #ifdef __cplusplus } #endif #endif pmdk-1.13.1/src/libpmem/libpmem.vcxproj.filters0000664000000000000000000002044614435627501020172 0ustar rootroot {16473205-8f12-4d4c-b1e9-e14ea3013e70} h {17275273-f923-45ff-9b7e-b2ea76561168} c;def Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Source Files Source Files pmdk-1.13.1/src/libpmem/libpmem.rc0000664000000000000000000000044214435627501015426 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016, Intel Corporation */ /* * libpmem.rc -- libpmem resource file */ #include #define FILE_NAME "libpmem.dll" #define DESCRIPTION "libpmem - persistent memory support library" #define TYPE VFT_DLL #include pmdk-1.13.1/src/libpmem/libpmem.vcxproj0000664000000000000000000002201114435627501016511 0ustar rootroot Debug x64 Release x64 AdvancedVectorExtensions AdvancedVectorExtensions AdvancedVectorExtensions AdvancedVectorExtensions AdvancedVectorExtensions AdvancedVectorExtensions AdvancedVectorExtensions AdvancedVectorExtensions {901f04db-e1a5-4a41-8b81-9d31c19acd59} {9e9e3d25-2139-4a5d-9200-18148ddead45} DynamicLibrary libpmem libpmem en-US 14.0 10.0.10240.0 10.0.22000.0 DynamicLibrary true v143 DynamicLibrary false false v143 $(VC_IncludePath);$(WindowsSDK_IncludePath);..\..\src\libpmem2\x86_64\;$(solutionDir)deps\miniasync\src\include $(VC_IncludePath);$(WindowsSDK_IncludePath);..\..\src\libpmem2\x86_64\;$(solutionDir)deps\miniasync\src\include $(SolutionDir)libpmem2;%(AdditionalIncludeDirectories) $(SolutionDir)libpmem2;%(AdditionalIncludeDirectories) pmdk-1.13.1/src/libpmem/Makefile0000664000000000000000000000254414435627501015120 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2023, Intel Corporation # # src/libpmem/Makefile -- Makefile for libpmem # include ../common.inc LIBRARY_NAME = pmem LIBRARY_SO_VERSION = 1 LIBRARY_VERSION = 0.0 SOURCE =\ $(CORE)/alloc.c\ $(CORE)/fs_posix.c\ $(CORE)/os_posix.c\ $(CORE)/os_thread_posix.c\ $(CORE)/out.c\ $(CORE)/util.c\ $(CORE)/util_posix.c\ $(COMMON)/file.c\ $(COMMON)/file_posix.c\ $(COMMON)/mmap.c\ $(COMMON)/mmap_posix.c\ $(COMMON)/os_deep_linux.c\ libpmem.c\ memops_generic.c\ pmem.c\ pmem_posix.c\ $(PMEM2)/pmem2_utils.c\ $(PMEM2)/config.c\ $(PMEM2)/persist_posix.c\ $(PMEM2)/source.c\ $(PMEM2)/source_posix.c ifeq ($(OS_KERNEL_NAME),Linux) SOURCE +=\ $(PMEM2)/pmem2_utils_linux.c\ $(PMEM2)/pmem2_utils_$(OS_DIMM).c\ $(PMEM2)/auto_flush_linux.c\ $(PMEM2)/deep_flush_linux.c else SOURCE +=\ $(PMEM2)/pmem2_utils_other.c\ $(PMEM2)/auto_flush_none.c\ $(PMEM2)/deep_flush_other.c endif ifeq ($(OS_DIMM),ndctl) SOURCE +=\ region_namespace_ndctl.c\ numa_ndctl.c CFLAGS += $(LIBNDCTL_CFLAGS) LIBS += $(LIBNDCTL_LIBS) else SOURCE +=\ region_namespace_none.c\ numa_none.c endif INCS += -I$(TOP)/src/libpmem2 include ../libpmem2/$(ARCH)/sources.inc SOURCE += $(LIBPMEM2_ARCH_SOURCE) include ../Makefile.inc include $(PMEM2)/$(ARCH)/flags.inc CFLAGS += -I. $(MINIASYNC_CFLAGS) -DPMEM2_USE_MINIASYNC=1 LIBS += -pthread pmdk-1.13.1/src/libpmem/libpmem.link.in0000664000000000000000000000116514435627501016367 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2019, Intel Corporation # # # src/libpmem.link -- linker link file for libpmem # LIBPMEM_1.0 { global: pmem_map_file; pmem_unmap; pmem_is_pmem; pmem_persist; pmem_msync; pmem_has_auto_flush; pmem_deep_persist; pmem_flush; pmem_deep_flush; pmem_deep_drain; pmem_drain; pmem_has_hw_drain; pmem_check_version; pmem_errormsg; pmem_memmove_persist; pmem_memcpy_persist; pmem_memset_persist; pmem_memmove_nodrain; pmem_memcpy_nodrain; pmem_memset_nodrain; pmem_memmove; pmem_memcpy; pmem_memset; fault_injection; local: *; }; pmdk-1.13.1/src/libpmem/pmem_posix.c0000664000000000000000000000320714435627501016001 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2019, Intel Corporation */ /* * pmem_posix.c -- pmem utilities with Posix implementation */ #include #include #include "pmem.h" #include "out.h" #include "mmap.h" /* * is_pmem_detect -- implement pmem_is_pmem() * * This function returns true only if the entire range can be confirmed * as being direct access persistent memory. Finding any part of the * range is not direct access, or failing to look up the information * because it is unmapped or because any sort of error happens, just * results in returning false. */ int is_pmem_detect(const void *addr, size_t len) { LOG(3, "addr %p len %zu", addr, len); if (len == 0) return 0; int retval = util_range_is_pmem(addr, len); LOG(4, "returning %d", retval); return retval; } /* * pmem_map_register -- memory map file and register mapping */ void * pmem_map_register(int fd, size_t len, const char *path, int is_dev_dax) { LOG(3, "fd %d len %zu path %s id_dev_dax %d", fd, len, path, is_dev_dax); void *addr; int map_sync; addr = util_map(fd, 0, len, MAP_SHARED, 0, 0, &map_sync); if (!addr) return NULL; enum pmem_map_type type = MAX_PMEM_TYPE; if (is_dev_dax) type = PMEM_DEV_DAX; else if (map_sync) type = PMEM_MAP_SYNC; if (type != MAX_PMEM_TYPE) { if (util_range_register(addr, len, path, type)) { LOG(1, "can't track mapped region"); goto err_unmap; } } return addr; err_unmap: util_unmap(addr, len); return NULL; } /* * pmem_os_init -- os-dependent part of pmem initialization */ void pmem_os_init(is_pmem_func *func) { LOG(3, NULL); *func = is_pmem_detect; } pmdk-1.13.1/src/libpmem/libpmem_main.c0000664000000000000000000000131014435627501016243 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2018, Intel Corporation */ /* * libpmem_main.c -- entry point for libpmem.dll * * XXX - This is a placeholder. All the library initialization/cleanup * that is done in library ctors/dtors, as well as TLS initialization * should be moved here. */ #include "win_mmap.h" void libpmem_init(void); void libpmem_fini(void); int APIENTRY DllMain(HINSTANCE hInstance, DWORD dwReason, LPVOID lpReserved) { switch (dwReason) { case DLL_PROCESS_ATTACH: libpmem_init(); win_mmap_init(); break; case DLL_THREAD_ATTACH: case DLL_THREAD_DETACH: break; case DLL_PROCESS_DETACH: win_mmap_fini(); libpmem_fini(); break; } return TRUE; } pmdk-1.13.1/src/libpmem/pmem.c0000664000000000000000000005345514435627501014571 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2023, Intel Corporation */ /* * pmem.c -- pmem entry points for libpmem * * * PERSISTENT MEMORY INSTRUCTIONS ON X86 * * The primary feature of this library is to provide a way to flush * changes to persistent memory as outlined below (note that many * of the decisions below are made at initialization time, and not * repeated every time a flush is requested). * * To flush a range to pmem when CLWB is available: * * CLWB for each cache line in the given range. * * SFENCE to ensure the CLWBs above have completed. * * To flush a range to pmem when CLFLUSHOPT is available and CLWB is not * (same as above but issue CLFLUSHOPT instead of CLWB): * * CLFLUSHOPT for each cache line in the given range. * * SFENCE to ensure the CLWBs above have completed. * * To flush a range to pmem when neither CLFLUSHOPT or CLWB are available * (same as above but fences surrounding CLFLUSH are not required): * * CLFLUSH for each cache line in the given range. * * To memcpy a range of memory to pmem when MOVNT is available: * * Copy any non-64-byte portion of the destination using MOV. * * Use the flush flow above without the fence for the copied portion. * * Copy using MOVNTDQ, up to any non-64-byte aligned end portion. * (The MOVNT instructions bypass the cache, so no flush is required.) * * Copy any unaligned end portion using MOV. * * Use the flush flow above for the copied portion (including fence). * * To memcpy a range of memory to pmem when MOVNT is not available: * * Just pass the call to the normal memcpy() followed by pmem_persist(). * * To memset a non-trivial sized range of memory to pmem: * * Same as the memcpy cases above but store the given value instead * of reading values from the source. * * These features are supported for ARM AARCH64 using equivalent ARM * assembly instruction. Please refer to (arm_cacheops.h) for more details. * * INTERFACES FOR FLUSHING TO PERSISTENT MEMORY * * Given the flows above, three interfaces are provided for flushing a range * so that the caller has the ability to separate the steps when necessary, * but otherwise leaves the detection of available instructions to the libpmem: * * pmem_persist(addr, len) * * This is the common case, which just calls the two other functions: * * pmem_flush(addr, len); * pmem_drain(); * * pmem_flush(addr, len) * * CLWB or CLFLUSHOPT or CLFLUSH for each cache line * * pmem_drain() * * SFENCE unless using CLFLUSH * * * INTERFACES FOR COPYING/SETTING RANGES OF MEMORY * * Given the flows above, the following interfaces are provided for the * memmove/memcpy/memset operations to persistent memory: * * pmem_memmove_nodrain() * * Checks for overlapped ranges to determine whether to copy from * the beginning of the range or from the end. If MOVNT instructions * are available, uses the memory copy flow described above, otherwise * calls the libc memmove() followed by pmem_flush(). Since no conditional * compilation and/or architecture specific CFLAGS are in use at the * moment, SSE2 ( thus movnt ) is just assumed to be available. * * pmem_memcpy_nodrain() * * Just calls pmem_memmove_nodrain(). * * pmem_memset_nodrain() * * If MOVNT instructions are available, uses the memset flow described * above, otherwise calls the libc memset() followed by pmem_flush(). * * pmem_memmove_persist() * pmem_memcpy_persist() * pmem_memset_persist() * * Calls the appropriate _nodrain() function followed by pmem_drain(). * * * DECISIONS MADE AT INITIALIZATION TIME * * As much as possible, all decisions described above are made at library * initialization time. This is achieved using function pointers that are * setup by pmem_init() when the library loads. * * Func_fence is used by pmem_drain() to call one of: * fence_empty() * memory_barrier() * * Func_flush is used by pmem_flush() to call one of: * flush_dcache() * flush_dcache_invalidate_opt() * flush_dcache_invalidate() * * Func_memmove_nodrain is used by memmove_nodrain() to call one of: * memmove_nodrain_libc() * memmove_nodrain_movnt() * * Func_memset_nodrain is used by memset_nodrain() to call one of: * memset_nodrain_libc() * memset_nodrain_movnt() * * DEBUG LOGGING * * Many of the functions here get called hundreds of times from loops * iterating over ranges, making the usual LOG() calls at level 3 * impractical. The call tracing log for those functions is set at 15. */ #include #include #include #include #include "libpmem.h" #include "pmem.h" #include "pmem2_arch.h" #include "out.h" #include "os.h" #include "mmap.h" #include "file.h" #include "valgrind_internal.h" #include "os_deep.h" #include "auto_flush.h" struct pmem_funcs { struct memmove_nodrain memmove_funcs; struct memset_nodrain memset_funcs; memmove_nodrain_func memmove_nodrain; memset_nodrain_func memset_nodrain; flush_func deep_flush; flush_func flush; fence_func fence; }; static struct pmem_funcs Funcs; static is_pmem_func Is_pmem = NULL; /* * pmem_has_hw_drain -- return whether or not HW drain was found * * Always false for x86: HW drain is done by HW with no SW involvement. */ int pmem_has_hw_drain(void) { LOG(3, NULL); return 0; } /* * pmem_drain -- wait for any PM stores to drain from HW buffers */ void pmem_drain(void) { LOG(15, NULL); Funcs.fence(); } /* * pmem_has_auto_flush -- check if platform supports eADR */ int pmem_has_auto_flush() { LOG(3, NULL); return pmem2_auto_flush(); } /* * pmem_deep_flush -- flush processor cache for the given range * regardless of eADR support on platform */ void pmem_deep_flush(const void *addr, size_t len) { LOG(15, "addr %p len %zu", addr, len); VALGRIND_DO_CHECK_MEM_IS_ADDRESSABLE(addr, len); Funcs.deep_flush(addr, len); } /* * pmem_flush -- flush processor cache for the given range */ void pmem_flush(const void *addr, size_t len) { LOG(15, "addr %p len %zu", addr, len); VALGRIND_DO_CHECK_MEM_IS_ADDRESSABLE(addr, len); Funcs.flush(addr, len); } /* * pmem_persist -- make any cached changes to a range of pmem persistent */ void pmem_persist(const void *addr, size_t len) { LOG(15, "addr %p len %zu", addr, len); pmem_flush(addr, len); pmem_drain(); } /* * pmem_msync -- flush to persistence via msync * * Using msync() means this routine is less optimal for pmem (but it * still works) but it also works for any memory mapped file, unlike * pmem_persist() which is only safe where pmem_is_pmem() returns true. */ int pmem_msync(const void *addr, size_t len) { LOG(15, "addr %p len %zu", addr, len); VALGRIND_DO_CHECK_MEM_IS_ADDRESSABLE(addr, len); /* * msync requires addr to be a multiple of pagesize but there are no * requirements for len. Align addr down and change len so that * [addr, addr + len) still contains initial range. */ /* increase len by the amount we gain when we round addr down */ len += (uintptr_t)addr & (Pagesize - 1); /* round addr down to page boundary */ uintptr_t uptr = (uintptr_t)addr & ~((uintptr_t)Pagesize - 1); /* * msync accepts addresses aligned to page boundary, so we may sync * more and part of it may have been marked as undefined/inaccessible * Msyncing such memory is not a bug, so as a workaround temporarily * disable error reporting. */ VALGRIND_DO_DISABLE_ERROR_REPORTING; int ret; if ((ret = msync((void *)uptr, len, MS_SYNC)) < 0) ERR("!msync"); VALGRIND_DO_ENABLE_ERROR_REPORTING; /* full flush */ VALGRIND_DO_PERSIST(uptr, len); return ret; } /* * is_pmem_always -- (internal) always true (for meaningful parameters) version * of pmem_is_pmem() */ static int is_pmem_always(const void *addr, size_t len) { LOG(3, "addr %p len %zu", addr, len); if (len == 0) return 0; return 1; } /* * is_pmem_never -- (internal) never true version of pmem_is_pmem() */ static int is_pmem_never(const void *addr, size_t len) { LOG(3, "addr %p len %zu", addr, len); return 0; } /* * pmem_is_pmem_init -- (internal) initialize Func_is_pmem pointer * * This should be done only once - on the first call to pmem_is_pmem(). * If PMEM_IS_PMEM_FORCE is set, it would override the default behavior * of pmem_is_pmem(). */ static void pmem_is_pmem_init(void) { LOG(3, NULL); static volatile unsigned init; while (init != 2) { if (!util_bool_compare_and_swap32(&init, 0, 1)) continue; /* * For debugging/testing, allow pmem_is_pmem() to be forced * to always true or never true using environment variable * PMEM_IS_PMEM_FORCE values of zero or one. * * This isn't #ifdef DEBUG because it has a trivial performance * impact and it may turn out to be useful as a "chicken bit" * for systems where pmem_is_pmem() isn't correctly detecting * true persistent memory. */ char *ptr = os_getenv("PMEM_IS_PMEM_FORCE"); if (ptr) { int val = atoi(ptr); if (val == 0) Is_pmem = is_pmem_never; else if (val == 1) Is_pmem = is_pmem_always; VALGRIND_ANNOTATE_HAPPENS_BEFORE(&Is_pmem); LOG(4, "PMEM_IS_PMEM_FORCE=%d", val); } if (Funcs.deep_flush == NULL) Is_pmem = is_pmem_never; if (!util_bool_compare_and_swap32(&init, 1, 2)) FATAL("util_bool_compare_and_swap32"); } } /* * pmem_is_pmem -- return true if entire range is persistent memory */ int pmem_is_pmem(const void *addr, size_t len) { LOG(10, "addr %p len %zu", addr, len); static int once; /* This is not thread-safe, but pmem_is_pmem_init() is. */ if (once == 0) { pmem_is_pmem_init(); util_fetch_and_add32(&once, 1); } VALGRIND_ANNOTATE_HAPPENS_AFTER(&Is_pmem); return Is_pmem(addr, len); } #define PMEM_FILE_ALL_FLAGS\ (PMEM_FILE_CREATE|PMEM_FILE_EXCL|PMEM_FILE_SPARSE|PMEM_FILE_TMPFILE) #define PMEM_DAX_VALID_FLAGS\ (PMEM_FILE_CREATE|PMEM_FILE_SPARSE) /* * pmem_map_fileU -- create or open the file and map it to memory */ #ifndef _WIN32 static inline #endif void * pmem_map_fileU(const char *path, size_t len, int flags, mode_t mode, size_t *mapped_lenp, int *is_pmemp) { LOG(3, "path \"%s\" size %zu flags %x mode %o mapped_lenp %p " "is_pmemp %p", path, len, flags, mode, mapped_lenp, is_pmemp); int oerrno; int fd; int open_flags = O_RDWR; int delete_on_err = 0; int file_type = util_file_get_type(path); #ifdef _WIN32 open_flags |= O_BINARY; #endif if (file_type == OTHER_ERROR) return NULL; if (flags & ~(PMEM_FILE_ALL_FLAGS)) { ERR("invalid flag specified %x", flags); errno = EINVAL; return NULL; } if (file_type == TYPE_DEVDAX) { if (flags & ~(PMEM_DAX_VALID_FLAGS)) { ERR("flag unsupported for Device DAX %x", flags); errno = EINVAL; return NULL; } else { /* we are ignoring all of the flags */ flags = 0; ssize_t actual_len = util_file_get_size(path); if (actual_len < 0) { ERR("unable to read Device DAX size"); errno = EINVAL; return NULL; } if (len != 0 && len != (size_t)actual_len) { ERR("Device DAX length must be either 0 or " "the exact size of the device: %zu", actual_len); errno = EINVAL; return NULL; } len = 0; } } if (flags & PMEM_FILE_CREATE) { if ((os_off_t)len < 0) { ERR("invalid file length %zu", len); errno = EINVAL; return NULL; } open_flags |= O_CREAT; } if (flags & PMEM_FILE_EXCL) open_flags |= O_EXCL; if ((len != 0) && !(flags & PMEM_FILE_CREATE)) { ERR("non-zero 'len' not allowed without PMEM_FILE_CREATE"); errno = EINVAL; return NULL; } if ((len == 0) && (flags & PMEM_FILE_CREATE)) { ERR("zero 'len' not allowed with PMEM_FILE_CREATE"); errno = EINVAL; return NULL; } if ((flags & PMEM_FILE_TMPFILE) && !(flags & PMEM_FILE_CREATE)) { ERR("PMEM_FILE_TMPFILE not allowed without PMEM_FILE_CREATE"); errno = EINVAL; return NULL; } if (flags & PMEM_FILE_TMPFILE) { if ((fd = util_tmpfile(path, OS_DIR_SEP_STR"pmem.XXXXXX", open_flags & O_EXCL)) < 0) { LOG(2, "failed to create temporary file at \"%s\"", path); return NULL; } } else { if ((fd = os_open(path, open_flags, mode)) < 0) { ERR("!open %s", path); return NULL; } if ((flags & PMEM_FILE_CREATE) && (flags & PMEM_FILE_EXCL)) delete_on_err = 1; } if (flags & PMEM_FILE_CREATE) { /* * Always set length of file to 'len'. * (May either extend or truncate existing file.) */ if (os_ftruncate(fd, (os_off_t)len) != 0) { ERR("!ftruncate"); goto err; } if ((flags & PMEM_FILE_SPARSE) == 0) { if ((errno = os_posix_fallocate(fd, 0, (os_off_t)len)) != 0) { ERR("!posix_fallocate"); goto err; } } } else { ssize_t actual_size = util_fd_get_size(fd); if (actual_size < 0) { ERR("stat %s: negative size", path); errno = EINVAL; goto err; } len = (size_t)actual_size; } void *addr = pmem_map_register(fd, len, path, file_type == TYPE_DEVDAX); if (addr == NULL) goto err; if (mapped_lenp != NULL) *mapped_lenp = len; if (is_pmemp != NULL) *is_pmemp = pmem_is_pmem(addr, len); LOG(3, "returning %p", addr); VALGRIND_REGISTER_PMEM_MAPPING(addr, len); VALGRIND_REGISTER_PMEM_FILE(fd, addr, len, 0); (void) os_close(fd); return addr; err: oerrno = errno; (void) os_close(fd); if (delete_on_err) (void) os_unlink(path); errno = oerrno; return NULL; } #ifndef _WIN32 /* * pmem_map_file -- create or open the file and map it to memory */ void * pmem_map_file(const char *path, size_t len, int flags, mode_t mode, size_t *mapped_lenp, int *is_pmemp) { return pmem_map_fileU(path, len, flags, mode, mapped_lenp, is_pmemp); } #else /* * pmem_map_fileW -- create or open the file and map it to memory */ void * pmem_map_fileW(const wchar_t *path, size_t len, int flags, mode_t mode, size_t *mapped_lenp, int *is_pmemp) { char *upath = util_toUTF8(path); if (upath == NULL) return NULL; void *ret = pmem_map_fileU(upath, len, flags, mode, mapped_lenp, is_pmemp); util_free_UTF8(upath); return ret; } #endif /* * pmem_unmap -- unmap the specified region */ int pmem_unmap(void *addr, size_t len) { LOG(3, "addr %p len %zu", addr, len); #ifndef _WIN32 util_range_unregister(addr, len); #endif VALGRIND_REMOVE_PMEM_MAPPING(addr, len); return util_unmap(addr, len); } /* * pmem_memmove -- memmove to pmem */ void * pmem_memmove(void *pmemdest, const void *src, size_t len, unsigned flags) { LOG(15, "pmemdest %p src %p len %zu flags 0x%x", pmemdest, src, len, flags); #ifdef DEBUG if (flags & ~PMEM_F_MEM_VALID_FLAGS) ERR("invalid flags 0x%x", flags); #endif PMEM_API_START(); Funcs.memmove_nodrain(pmemdest, src, len, flags & ~PMEM_F_MEM_NODRAIN, Funcs.flush, &Funcs.memmove_funcs); if ((flags & (PMEM_F_MEM_NODRAIN | PMEM_F_MEM_NOFLUSH)) == 0) pmem_drain(); PMEM_API_END(); return pmemdest; } /* * pmem_memcpy -- memcpy to pmem */ void * pmem_memcpy(void *pmemdest, const void *src, size_t len, unsigned flags) { LOG(15, "pmemdest %p src %p len %zu flags 0x%x", pmemdest, src, len, flags); #ifdef DEBUG if (flags & ~PMEM_F_MEM_VALID_FLAGS) ERR("invalid flags 0x%x", flags); #endif PMEM_API_START(); Funcs.memmove_nodrain(pmemdest, src, len, flags & ~PMEM_F_MEM_NODRAIN, Funcs.flush, &Funcs.memmove_funcs); if ((flags & (PMEM_F_MEM_NODRAIN | PMEM_F_MEM_NOFLUSH)) == 0) pmem_drain(); PMEM_API_END(); return pmemdest; } /* * pmem_memset -- memset to pmem */ void * pmem_memset(void *pmemdest, int c, size_t len, unsigned flags) { LOG(15, "pmemdest %p c 0x%x len %zu flags 0x%x", pmemdest, c, len, flags); #ifdef DEBUG if (flags & ~PMEM_F_MEM_VALID_FLAGS) ERR("invalid flags 0x%x", flags); #endif PMEM_API_START(); Funcs.memset_nodrain(pmemdest, c, len, flags & ~PMEM_F_MEM_NODRAIN, Funcs.flush, &Funcs.memset_funcs); if ((flags & (PMEM_F_MEM_NODRAIN | PMEM_F_MEM_NOFLUSH)) == 0) pmem_drain(); PMEM_API_END(); return pmemdest; } /* * pmem_memmove_nodrain -- memmove to pmem without hw drain */ void * pmem_memmove_nodrain(void *pmemdest, const void *src, size_t len) { LOG(15, "pmemdest %p src %p len %zu", pmemdest, src, len); PMEM_API_START(); Funcs.memmove_nodrain(pmemdest, src, len, 0, Funcs.flush, &Funcs.memmove_funcs); PMEM_API_END(); return pmemdest; } /* * pmem_memcpy_nodrain -- memcpy to pmem without hw drain */ void * pmem_memcpy_nodrain(void *pmemdest, const void *src, size_t len) { LOG(15, "pmemdest %p src %p len %zu", pmemdest, src, len); PMEM_API_START(); Funcs.memmove_nodrain(pmemdest, src, len, 0, Funcs.flush, &Funcs.memmove_funcs); PMEM_API_END(); return pmemdest; } /* * pmem_memmove_persist -- memmove to pmem */ void * pmem_memmove_persist(void *pmemdest, const void *src, size_t len) { LOG(15, "pmemdest %p src %p len %zu", pmemdest, src, len); PMEM_API_START(); Funcs.memmove_nodrain(pmemdest, src, len, 0, Funcs.flush, &Funcs.memmove_funcs); pmem_drain(); PMEM_API_END(); return pmemdest; } /* * pmem_memcpy_persist -- memcpy to pmem */ void * pmem_memcpy_persist(void *pmemdest, const void *src, size_t len) { LOG(15, "pmemdest %p src %p len %zu", pmemdest, src, len); PMEM_API_START(); Funcs.memmove_nodrain(pmemdest, src, len, 0, Funcs.flush, &Funcs.memmove_funcs); pmem_drain(); PMEM_API_END(); return pmemdest; } /* * pmem_memset_nodrain -- memset to pmem without hw drain */ void * pmem_memset_nodrain(void *pmemdest, int c, size_t len) { LOG(15, "pmemdest %p c %d len %zu", pmemdest, c, len); PMEM_API_START(); Funcs.memset_nodrain(pmemdest, c, len, 0, Funcs.flush, &Funcs.memset_funcs); PMEM_API_END(); return pmemdest; } /* * pmem_memset_persist -- memset to pmem */ void * pmem_memset_persist(void *pmemdest, int c, size_t len) { LOG(15, "pmemdest %p c %d len %zu", pmemdest, c, len); PMEM_API_START(); Funcs.memset_nodrain(pmemdest, c, len, 0, Funcs.flush, &Funcs.memset_funcs); pmem_drain(); PMEM_API_END(); return pmemdest; } /* * memmove_nodrain_libc -- (internal) memmove to pmem using libc */ static void * memmove_nodrain_libc(void *pmemdest, const void *src, size_t len, unsigned flags, flush_func flush, const struct memmove_nodrain *memmove_funcs) { LOG(15, "pmemdest %p src %p len %zu flags 0x%x", pmemdest, src, len, flags); SUPPRESS_UNUSED(memmove_funcs); memmove(pmemdest, src, len); if (!(flags & PMEM_F_MEM_NOFLUSH)) flush(pmemdest, len); return pmemdest; } /* * memset_nodrain_libc -- (internal) memset to pmem using libc */ static void * memset_nodrain_libc(void *pmemdest, int c, size_t len, unsigned flags, flush_func flush, const struct memset_nodrain *memset_funcs) { LOG(15, "pmemdest %p c 0x%x len %zu flags 0x%x", pmemdest, c, len, flags); SUPPRESS_UNUSED(memset_funcs); memset(pmemdest, c, len); if (!(flags & PMEM_F_MEM_NOFLUSH)) flush(pmemdest, len); return pmemdest; } /* * flush_empty -- (internal) do not flush the CPU cache */ static void flush_empty(const void *addr, size_t len) { LOG(15, "addr %p len %zu", addr, len); flush_empty_nolog(addr, len); } /* * fence_empty -- (internal) issue the fence instruction */ static void fence_empty(void) { LOG(15, NULL); VALGRIND_DO_FENCE; } /* * pmem_init -- load-time initialization for pmem.c */ void pmem_init(void) { LOG(3, NULL); struct pmem2_arch_info info; info.memmove_nodrain = NULL; info.memset_nodrain = NULL; info.memmove_nodrain_eadr = NULL; info.memset_nodrain_eadr = NULL; info.flush = NULL; info.fence = NULL; info.flush_has_builtin_fence = 0; pmem2_arch_init(&info); int flush; char *e = os_getenv("PMEM_NO_FLUSH"); if (e && (strcmp(e, "1") == 0)) { flush = 0; LOG(3, "Forced not flushing CPU_cache"); } else if (e && (strcmp(e, "0") == 0)) { flush = 1; LOG(3, "Forced flushing CPU_cache"); } else if (pmem2_auto_flush() == 1) { flush = 0; LOG(3, "Not flushing CPU_cache, eADR detected"); } else { flush = 1; LOG(3, "Flushing CPU cache"); } Funcs.memmove_funcs = info.memmove_funcs; Funcs.memset_funcs = info.memset_funcs; Funcs.deep_flush = info.flush; if (flush) { Funcs.flush = info.flush; Funcs.memmove_nodrain = info.memmove_nodrain; Funcs.memset_nodrain = info.memset_nodrain; if (info.flush_has_builtin_fence) Funcs.fence = fence_empty; else Funcs.fence = info.fence; } else { Funcs.memmove_nodrain = info.memmove_nodrain_eadr; Funcs.memset_nodrain = info.memset_nodrain_eadr; Funcs.flush = flush_empty; Funcs.fence = info.fence; } char *ptr = os_getenv("PMEM_NO_GENERIC_MEMCPY"); long long no_generic = 0; if (ptr) no_generic = atoll(ptr); if (Funcs.memmove_nodrain == NULL) { if (no_generic) { Funcs.memmove_nodrain = memmove_nodrain_libc; LOG(3, "using libc memmove"); } else { Funcs.memmove_nodrain = memmove_nodrain_generic; LOG(3, "using generic memmove"); } } if (Funcs.memset_nodrain == NULL) { if (no_generic) { Funcs.memset_nodrain = memset_nodrain_libc; LOG(3, "using libc memset"); } else { Funcs.memset_nodrain = memset_nodrain_generic; LOG(3, "using generic memset"); } } if (Funcs.flush == flush_empty) LOG(3, "not flushing CPU cache"); else if (Funcs.flush != Funcs.deep_flush) FATAL("invalid flush function address"); pmem_os_init(&Is_pmem); } /* * pmem_deep_persist -- perform deep persist on a memory range * * It merely acts as wrapper around an msync call in most cases, the only * exception is the case of an mmap'ed DAX device on Linux. */ int pmem_deep_persist(const void *addr, size_t len) { LOG(3, "addr %p len %zu", addr, len); pmem_deep_flush(addr, len); return pmem_deep_drain(addr, len); } /* * pmem_deep_drain -- perform deep drain on a memory range */ int pmem_deep_drain(const void *addr, size_t len) { LOG(3, "addr %p len %zu", addr, len); return os_range_deep_common((uintptr_t)addr, len); } #if VG_PMEMCHECK_ENABLED /* * pmem_emit_log -- logs library and function names to pmemcheck store log */ void pmem_emit_log(const char *func, int order) { util_emit_log("libpmem", func, order); } #endif #if FAULT_INJECTION void pmem_inject_fault_at(enum pmem_allocation_type type, int nth, const char *at) { core_inject_fault_at(type, nth, at); } int pmem_fault_injection_enabled(void) { return core_fault_injection_enabled(); } #endif pmdk-1.13.1/src/libpmem/libpmem.def0000664000000000000000000000342314435627501015562 0ustar rootroot;;;; Begin Copyright Notice ; SPDX-License-Identifier: BSD-3-Clause ; Copyright 2015-2018, Intel Corporation ;;;; End Copyright Notice ; ; XXX - libpmem exports mmap/munmap/msync/mprotect functions ; ; This is a _temporary_ solution to make those function available for all ; the other PMDK libraries and to have only one instance of a file mapping ; list (owned by libpmem). Otherwise, each library would have its own ; instance of the file mapping list, resulting in libpmem being not able ; to find a file handle associated with the mapping address passed to ; pmem_msync(), pmem_memcpy(), etc. causing those functions to fail. ; ; The proposed target solution would include: ; - implementation of pmem_mmap, pmem_unmap, pmem_msync and ; pmem_mprotect functions in libpmem (pmem_unmap and pmem_msync are ; already there); ; - making sure that all the PMDK libraries never call mmap, munmap, ; msync and mprotect directly, but only through their libpmem counterparts; ; - new pmem_mmap() function must provide similar functionality to ; mmap(), i.e. it must take 'offset' argument, but should not take ; file descriptor argument. Perhaps it could be an opaque handle ; to the file, that is internally casted to a file descriptor ; or a HANDLE, depending on the OS. ; LIBRARY libpmem VERSION 1.0 EXPORTS pmem_map_fileU pmem_map_fileW pmem_unmap pmem_is_pmem pmem_persist pmem_msync pmem_has_auto_flush pmem_deep_persist pmem_flush pmem_deep_flush pmem_deep_drain pmem_drain pmem_has_hw_drain pmem_memmove_persist pmem_memcpy_persist pmem_memset_persist pmem_memmove_nodrain pmem_memcpy_nodrain pmem_memset_nodrain pmem_memmove pmem_memcpy pmem_memset pmem_check_versionU pmem_check_versionW pmem_errormsgU pmem_errormsgW mmap munmap msync mprotect DllMain pmdk-1.13.1/src/libpmem/libpmem.c0000664000000000000000000000452314435627501015250 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2017, Intel Corporation */ /* * libpmem.c -- pmem entry points for libpmem */ #include #include #include "libpmem.h" #include "pmem.h" #include "pmemcommon.h" /* * libpmem_init -- load-time initialization for libpmem * * Called automatically by the run-time loader. */ ATTR_CONSTRUCTOR void libpmem_init(void) { common_init(PMEM_LOG_PREFIX, PMEM_LOG_LEVEL_VAR, PMEM_LOG_FILE_VAR, PMEM_MAJOR_VERSION, PMEM_MINOR_VERSION); LOG(3, NULL); pmem_init(); } /* * libpmem_fini -- libpmem cleanup routine * * Called automatically when the process terminates. */ ATTR_DESTRUCTOR void libpmem_fini(void) { LOG(3, NULL); common_fini(); } /* * pmem_check_versionU -- see if library meets application version requirements */ #ifndef _WIN32 static inline #endif const char * pmem_check_versionU(unsigned major_required, unsigned minor_required) { LOG(3, "major_required %u minor_required %u", major_required, minor_required); if (major_required != PMEM_MAJOR_VERSION) { ERR("libpmem major version mismatch (need %u, found %u)", major_required, PMEM_MAJOR_VERSION); return out_get_errormsg(); } if (minor_required > PMEM_MINOR_VERSION) { ERR("libpmem minor version mismatch (need %u, found %u)", minor_required, PMEM_MINOR_VERSION); return out_get_errormsg(); } return NULL; } #ifndef _WIN32 /* * pmem_check_version -- see if library meets application version requirements */ const char * pmem_check_version(unsigned major_required, unsigned minor_required) { return pmem_check_versionU(major_required, minor_required); } #else /* * pmem_check_versionW -- see if library meets application version requirements */ const wchar_t * pmem_check_versionW(unsigned major_required, unsigned minor_required) { if (pmem_check_versionU(major_required, minor_required) != NULL) return out_get_errormsgW(); else return NULL; } #endif /* * pmem_errormsgU -- return last error message */ #ifndef _WIN32 static inline #endif const char * pmem_errormsgU(void) { return out_get_errormsg(); } #ifndef _WIN32 /* * pmem_errormsg -- return last error message */ const char * pmem_errormsg(void) { return pmem_errormsgU(); } #else /* * pmem_errormsgW -- return last error message as wchar_t */ const wchar_t * pmem_errormsgW(void) { return out_get_errormsgW(); } #endif pmdk-1.13.1/src/libvmem/0000775000000000000000000000000014435627501013461 5ustar rootrootpmdk-1.13.1/src/libvmem/README.md0000664000000000000000000000012614435627501014737 0ustar rootrootThis library has been moved to a [separate repository](https://github.com/pmem/vmem). pmdk-1.13.1/src/freebsd/0000775000000000000000000000000014435627501013440 5ustar rootrootpmdk-1.13.1/src/freebsd/include/0000775000000000000000000000000014435627501015063 5ustar rootrootpmdk-1.13.1/src/freebsd/include/linux/0000775000000000000000000000000014435627501016222 5ustar rootrootpmdk-1.13.1/src/freebsd/include/linux/kdev_t.h0000664000000000000000000000021214435627501017642 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2017-2020, Intel Corporation */ /* * linux/kdev_t.h -- Empty file redirect */ pmdk-1.13.1/src/freebsd/include/linux/limits.h0000664000000000000000000000021214435627501017667 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2017-2020, Intel Corporation */ /* * linux/limits.h -- Empty file redirect */ pmdk-1.13.1/src/freebsd/include/endian.h0000664000000000000000000000025514435627501016474 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2017-2020, Intel Corporation */ /* * endian.h -- redirect for FreeBSD */ #include pmdk-1.13.1/src/freebsd/include/features.h0000664000000000000000000000020614435627501017050 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2017-2020, Intel Corporation */ /* * features.h -- Empty file redirect */ pmdk-1.13.1/src/freebsd/include/sys/0000775000000000000000000000000014435627501015701 5ustar rootrootpmdk-1.13.1/src/freebsd/include/sys/sysmacros.h0000664000000000000000000000021314435627501020071 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2017-2020, Intel Corporation */ /* * sys/sysmacros.h -- Empty file redirect */ pmdk-1.13.1/src/freebsd/README0000664000000000000000000000115714435627501014324 0ustar rootrootPersistent Memory Development Kit This is src/freebsd/README. This directory contains FreeBSD-specific files for the Persistent Memory Development Kit. The subdirectory "include" contains header files that have no equivalents on FreeBSD. Most of these files are empty, which is a cheap trick to avoid preprocessor errors when including non-existing files. Others are redirects for files that are in different locations on FreeBSD. This way we don't need a lot of preprocessor conditionals in all the source code files, although it does require conditionals in the Makefiles (which could be addressed by using autoconf). pmdk-1.13.1/src/libpmemlog/0000775000000000000000000000000014435627501014155 5ustar rootrootpmdk-1.13.1/src/libpmemlog/libpmemlog.rc0000664000000000000000000000045714435627501016640 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016, Intel Corporation */ /* * libpmemlog.rc -- libpmemlog resource file */ #include #define FILE_NAME "libpmemlog.dll" #define DESCRIPTION "libpmemlog - persistent memory resident log file" #define TYPE VFT_DLL #include pmdk-1.13.1/src/libpmemlog/log.c0000664000000000000000000004635714435627501015121 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2020, Intel Corporation */ /* * log.c -- log memory pool entry points for libpmem */ #include #include #include #include #include #include #include #include #include #include #include "libpmem.h" #include "libpmemlog.h" #include "ctl_global.h" #include "os.h" #include "set.h" #include "out.h" #include "log.h" #include "mmap.h" #include "sys_util.h" #include "util_pmem.h" #include "valgrind_internal.h" static const struct pool_attr Log_create_attr = { LOG_HDR_SIG, LOG_FORMAT_MAJOR, LOG_FORMAT_FEAT_DEFAULT, {0}, {0}, {0}, {0}, {0} }; static const struct pool_attr Log_open_attr = { LOG_HDR_SIG, LOG_FORMAT_MAJOR, LOG_FORMAT_FEAT_CHECK, {0}, {0}, {0}, {0}, {0} }; /* * log_descr_create -- (internal) create log memory pool descriptor */ static void log_descr_create(PMEMlogpool *plp, size_t poolsize) { LOG(3, "plp %p poolsize %zu", plp, poolsize); ASSERTeq(poolsize % Pagesize, 0); /* create required metadata */ plp->start_offset = htole64(roundup(sizeof(*plp), LOG_FORMAT_DATA_ALIGN)); plp->end_offset = htole64(poolsize); plp->write_offset = plp->start_offset; /* store non-volatile part of pool's descriptor */ util_persist(plp->is_pmem, &plp->start_offset, 3 * sizeof(uint64_t)); } /* * log_descr_check -- (internal) validate log memory pool descriptor */ static int log_descr_check(PMEMlogpool *plp, size_t poolsize) { LOG(3, "plp %p poolsize %zu", plp, poolsize); struct pmemlog hdr = *plp; log_convert2h(&hdr); if ((hdr.start_offset != roundup(sizeof(*plp), LOG_FORMAT_DATA_ALIGN)) || (hdr.end_offset != poolsize) || (hdr.start_offset > hdr.end_offset)) { ERR("wrong start/end offsets " "(start: %" PRIu64 " end: %" PRIu64 "), " "pool size %zu", hdr.start_offset, hdr.end_offset, poolsize); errno = EINVAL; return -1; } if ((hdr.write_offset > hdr.end_offset) || (hdr.write_offset < hdr.start_offset)) { ERR("wrong write offset (start: %" PRIu64 " end: %" PRIu64 " write: %" PRIu64 ")", hdr.start_offset, hdr.end_offset, hdr.write_offset); errno = EINVAL; return -1; } LOG(3, "start: %" PRIu64 ", end: %" PRIu64 ", write: %" PRIu64 "", hdr.start_offset, hdr.end_offset, hdr.write_offset); return 0; } /* * log_runtime_init -- (internal) initialize log memory pool runtime data */ static int log_runtime_init(PMEMlogpool *plp, int rdonly) { LOG(3, "plp %p rdonly %d", plp, rdonly); /* remove volatile part of header */ VALGRIND_REMOVE_PMEM_MAPPING(&plp->addr, sizeof(struct pmemlog) - sizeof(struct pool_hdr) - 3 * sizeof(uint64_t)); /* * Use some of the memory pool area for run-time info. This * run-time state is never loaded from the file, it is always * created here, so no need to worry about byte-order. */ plp->rdonly = rdonly; if ((plp->rwlockp = Malloc(sizeof(*plp->rwlockp))) == NULL) { ERR("!Malloc for a RW lock"); return -1; } util_rwlock_init(plp->rwlockp); /* * If possible, turn off all permissions on the pool header page. * * The prototype PMFS doesn't allow this when large pages are in * use. It is not considered an error if this fails. */ RANGE_NONE(plp->addr, sizeof(struct pool_hdr), plp->is_dev_dax); /* the rest should be kept read-only (debug version only) */ RANGE_RO((char *)plp->addr + sizeof(struct pool_hdr), plp->size - sizeof(struct pool_hdr), plp->is_dev_dax); return 0; } /* * pmemlog_createU -- create a log memory pool */ #ifndef _WIN32 static inline #endif PMEMlogpool * pmemlog_createU(const char *path, size_t poolsize, mode_t mode) { LOG(3, "path %s poolsize %zu mode %d", path, poolsize, mode); struct pool_set *set; struct pool_attr adj_pool_attr = Log_create_attr; /* force set SDS feature */ if (SDS_at_create) adj_pool_attr.features.incompat |= POOL_FEAT_SDS; else adj_pool_attr.features.incompat &= ~POOL_FEAT_SDS; if (util_pool_create(&set, path, poolsize, PMEMLOG_MIN_POOL, PMEMLOG_MIN_PART, &adj_pool_attr, NULL, REPLICAS_DISABLED) != 0) { LOG(2, "cannot create pool or pool set"); return NULL; } ASSERT(set->nreplicas > 0); struct pool_replica *rep = set->replica[0]; PMEMlogpool *plp = rep->part[0].addr; VALGRIND_REMOVE_PMEM_MAPPING(&plp->addr, sizeof(struct pmemlog) - ((uintptr_t)&plp->addr - (uintptr_t)&plp->hdr)); plp->addr = plp; plp->size = rep->repsize; plp->set = set; plp->is_pmem = rep->is_pmem; plp->is_dev_dax = rep->part[0].is_dev_dax; /* is_dev_dax implies is_pmem */ ASSERT(!plp->is_dev_dax || plp->is_pmem); /* create pool descriptor */ log_descr_create(plp, rep->repsize); /* initialize runtime parts */ if (log_runtime_init(plp, 0) != 0) { ERR("pool initialization failed"); goto err; } if (util_poolset_chmod(set, mode)) goto err; util_poolset_fdclose(set); LOG(3, "plp %p", plp); return plp; err: LOG(4, "error clean up"); int oerrno = errno; util_poolset_close(set, DELETE_CREATED_PARTS); errno = oerrno; return NULL; } #ifndef _WIN32 /* * pmemlog_create -- create a log memory pool */ PMEMlogpool * pmemlog_create(const char *path, size_t poolsize, mode_t mode) { return pmemlog_createU(path, poolsize, mode); } #else /* * pmemlog_createW -- create a log memory pool */ PMEMlogpool * pmemlog_createW(const wchar_t *path, size_t poolsize, mode_t mode) { char *upath = util_toUTF8(path); if (upath == NULL) return NULL; PMEMlogpool *ret = pmemlog_createU(upath, poolsize, mode); util_free_UTF8(upath); return ret; } #endif /* * log_open_common -- (internal) open a log memory pool * * This routine does all the work, but takes a cow flag so internal * calls can map a read-only pool if required. */ static PMEMlogpool * log_open_common(const char *path, unsigned flags) { LOG(3, "path %s flags 0x%x", path, flags); struct pool_set *set; if (util_pool_open(&set, path, PMEMLOG_MIN_PART, &Log_open_attr, NULL, NULL, flags) != 0) { LOG(2, "cannot open pool or pool set"); return NULL; } ASSERT(set->nreplicas > 0); struct pool_replica *rep = set->replica[0]; PMEMlogpool *plp = rep->part[0].addr; VALGRIND_REMOVE_PMEM_MAPPING(&plp->addr, sizeof(struct pmemlog) - ((uintptr_t)&plp->addr - (uintptr_t)&plp->hdr)); plp->addr = plp; plp->size = rep->repsize; plp->set = set; plp->is_pmem = rep->is_pmem; plp->is_dev_dax = rep->part[0].is_dev_dax; /* is_dev_dax implies is_pmem */ ASSERT(!plp->is_dev_dax || plp->is_pmem); if (set->nreplicas > 1) { errno = ENOTSUP; ERR("!replicas not supported"); goto err; } /* validate pool descriptor */ if (log_descr_check(plp, rep->repsize) != 0) { LOG(2, "descriptor check failed"); goto err; } /* initialize runtime parts */ if (log_runtime_init(plp, set->rdonly) != 0) { ERR("pool initialization failed"); goto err; } util_poolset_fdclose(set); LOG(3, "plp %p", plp); return plp; err: LOG(4, "error clean up"); int oerrno = errno; util_poolset_close(set, DO_NOT_DELETE_PARTS); errno = oerrno; return NULL; } /* * pmemlog_openU -- open an existing log memory pool */ #ifndef _WIN32 static inline #endif PMEMlogpool * pmemlog_openU(const char *path) { LOG(3, "path %s", path); return log_open_common(path, COW_at_open ? POOL_OPEN_COW : 0); } #ifndef _WIN32 /* * pmemlog_open -- open an existing log memory pool */ PMEMlogpool * pmemlog_open(const char *path) { return pmemlog_openU(path); } #else /* * pmemlog_openW -- open an existing log memory pool */ PMEMlogpool * pmemlog_openW(const wchar_t *path) { char *upath = util_toUTF8(path); if (upath == NULL) return NULL; PMEMlogpool *ret = pmemlog_openU(upath); util_free_UTF8(upath); return ret; } #endif /* * pmemlog_close -- close a log memory pool */ void pmemlog_close(PMEMlogpool *plp) { LOG(3, "plp %p", plp); util_rwlock_destroy(plp->rwlockp); Free((void *)plp->rwlockp); util_poolset_close(plp->set, DO_NOT_DELETE_PARTS); } /* * pmemlog_nbyte -- return usable size of a log memory pool */ size_t pmemlog_nbyte(PMEMlogpool *plp) { LOG(3, "plp %p", plp); util_rwlock_rdlock(plp->rwlockp); size_t size = le64toh(plp->end_offset) - le64toh(plp->start_offset); LOG(4, "plp %p nbyte %zu", plp, size); util_rwlock_unlock(plp->rwlockp); return size; } /* * log_persist -- (internal) persist data, then metadata * * On entry, the write lock should be held. */ static void log_persist(PMEMlogpool *plp, uint64_t new_write_offset) { uint64_t old_write_offset = le64toh(plp->write_offset); size_t length = new_write_offset - old_write_offset; /* unprotect the log space range (debug version only) */ RANGE_RW((char *)plp->addr + old_write_offset, length, plp->is_dev_dax); /* persist the data */ if (plp->is_pmem) pmem_drain(); /* data already flushed */ else pmem_msync((char *)plp->addr + old_write_offset, length); /* protect the log space range (debug version only) */ RANGE_RO((char *)plp->addr + old_write_offset, length, plp->is_dev_dax); /* unprotect the pool descriptor (debug version only) */ RANGE_RW((char *)plp->addr + sizeof(struct pool_hdr), LOG_FORMAT_DATA_ALIGN, plp->is_dev_dax); /* write the metadata */ plp->write_offset = htole64(new_write_offset); /* persist the metadata */ if (plp->is_pmem) pmem_persist(&plp->write_offset, sizeof(plp->write_offset)); else pmem_msync(&plp->write_offset, sizeof(plp->write_offset)); /* set the write-protection again (debug version only) */ RANGE_RO((char *)plp->addr + sizeof(struct pool_hdr), LOG_FORMAT_DATA_ALIGN, plp->is_dev_dax); } /* * pmemlog_append -- add data to a log memory pool */ int pmemlog_append(PMEMlogpool *plp, const void *buf, size_t count) { int ret = 0; LOG(3, "plp %p buf %p count %zu", plp, buf, count); if (plp->rdonly) { ERR("can't append to read-only log"); errno = EROFS; return -1; } util_rwlock_wrlock(plp->rwlockp); /* get the current values */ uint64_t end_offset = le64toh(plp->end_offset); uint64_t write_offset = le64toh(plp->write_offset); if (write_offset >= end_offset) { /* no space left */ errno = ENOSPC; ERR("!pmemlog_append"); ret = -1; goto end; } /* make sure we don't write past the available space */ if (count > (end_offset - write_offset)) { errno = ENOSPC; ERR("!pmemlog_append"); ret = -1; goto end; } char *data = plp->addr; /* * unprotect the log space range, where the new data will be stored * (debug version only) */ RANGE_RW(&data[write_offset], count, plp->is_dev_dax); if (plp->is_pmem) pmem_memcpy_nodrain(&data[write_offset], buf, count); else memcpy(&data[write_offset], buf, count); /* protect the log space range (debug version only) */ RANGE_RO(&data[write_offset], count, plp->is_dev_dax); write_offset += count; /* persist the data and the metadata */ log_persist(plp, write_offset); end: util_rwlock_unlock(plp->rwlockp); return ret; } /* * pmemlog_appendv -- add gathered data to a log memory pool */ int pmemlog_appendv(PMEMlogpool *plp, const struct iovec *iov, int iovcnt) { LOG(3, "plp %p iovec %p iovcnt %d", plp, iov, iovcnt); int ret = 0; int i; if (iovcnt < 0) { errno = EINVAL; ERR("iovcnt is less than zero: %d", iovcnt); return -1; } if (plp->rdonly) { ERR("can't append to read-only log"); errno = EROFS; return -1; } util_rwlock_wrlock(plp->rwlockp); /* get the current values */ uint64_t end_offset = le64toh(plp->end_offset); uint64_t write_offset = le64toh(plp->write_offset); if (write_offset >= end_offset) { /* no space left */ errno = ENOSPC; ERR("!pmemlog_appendv"); ret = -1; goto end; } char *data = plp->addr; uint64_t count = 0; char *buf; /* calculate required space */ for (i = 0; i < iovcnt; ++i) count += iov[i].iov_len; /* check if there is enough free space */ if (count > (end_offset - write_offset)) { errno = ENOSPC; ret = -1; goto end; } /* append the data */ for (i = 0; i < iovcnt; ++i) { buf = iov[i].iov_base; count = iov[i].iov_len; /* * unprotect the log space range, where the new data will be * stored (debug version only) */ RANGE_RW(&data[write_offset], count, plp->is_dev_dax); if (plp->is_pmem) pmem_memcpy_nodrain(&data[write_offset], buf, count); else memcpy(&data[write_offset], buf, count); /* * protect the log space range (debug version only) */ RANGE_RO(&data[write_offset], count, plp->is_dev_dax); write_offset += count; } /* persist the data and the metadata */ log_persist(plp, write_offset); end: util_rwlock_unlock(plp->rwlockp); return ret; } /* * pmemlog_tell -- return current write point in a log memory pool */ long long pmemlog_tell(PMEMlogpool *plp) { LOG(3, "plp %p", plp); util_rwlock_rdlock(plp->rwlockp); ASSERT(le64toh(plp->write_offset) >= le64toh(plp->start_offset)); long long wp = (long long)(le64toh(plp->write_offset) - le64toh(plp->start_offset)); LOG(4, "write offset %lld", wp); util_rwlock_unlock(plp->rwlockp); return wp; } /* * pmemlog_rewind -- discard all data, resetting a log memory pool to empty */ void pmemlog_rewind(PMEMlogpool *plp) { LOG(3, "plp %p", plp); if (plp->rdonly) { ERR("can't rewind read-only log"); errno = EROFS; return; } util_rwlock_wrlock(plp->rwlockp); /* unprotect the pool descriptor (debug version only) */ RANGE_RW((char *)plp->addr + sizeof(struct pool_hdr), LOG_FORMAT_DATA_ALIGN, plp->is_dev_dax); plp->write_offset = plp->start_offset; if (plp->is_pmem) pmem_persist(&plp->write_offset, sizeof(uint64_t)); else pmem_msync(&plp->write_offset, sizeof(uint64_t)); /* set the write-protection again (debug version only) */ RANGE_RO((char *)plp->addr + sizeof(struct pool_hdr), LOG_FORMAT_DATA_ALIGN, plp->is_dev_dax); util_rwlock_unlock(plp->rwlockp); } /* * pmemlog_walk -- walk through all data in a log memory pool * * chunksize of 0 means process_chunk gets called once for all data * as a single chunk. */ void pmemlog_walk(PMEMlogpool *plp, size_t chunksize, int (*process_chunk)(const void *buf, size_t len, void *arg), void *arg) { LOG(3, "plp %p chunksize %zu", plp, chunksize); /* * We are assuming that the walker doesn't change the data it's reading * in place. We prevent everyone from changing the data behind our back * until we are done with processing it. */ util_rwlock_rdlock(plp->rwlockp); char *data = plp->addr; uint64_t write_offset = le64toh(plp->write_offset); uint64_t data_offset = le64toh(plp->start_offset); size_t len; if (chunksize == 0) { /* most common case: process everything at once */ len = write_offset - data_offset; LOG(3, "length %zu", len); (*process_chunk)(&data[data_offset], len, arg); } else { /* * Walk through the complete record, chunk by chunk. * The callback returns 0 to terminate the walk. */ while (data_offset < write_offset) { len = MIN(chunksize, write_offset - data_offset); if (!(*process_chunk)(&data[data_offset], len, arg)) break; data_offset += chunksize; } } util_rwlock_unlock(plp->rwlockp); } /* * pmemlog_checkU -- log memory pool consistency check * * Returns true if consistent, zero if inconsistent, -1/error if checking * cannot happen due to other errors. */ #ifndef _WIN32 static inline #endif int pmemlog_checkU(const char *path) { LOG(3, "path \"%s\"", path); PMEMlogpool *plp = log_open_common(path, POOL_OPEN_COW); if (plp == NULL) return -1; /* errno set by log_open_common() */ int consistent = 1; /* validate pool descriptor */ uint64_t hdr_start = le64toh(plp->start_offset); uint64_t hdr_end = le64toh(plp->end_offset); uint64_t hdr_write = le64toh(plp->write_offset); if (hdr_start != roundup(sizeof(*plp), LOG_FORMAT_DATA_ALIGN)) { ERR("wrong value of start_offset"); consistent = 0; } if (hdr_end != plp->size) { ERR("wrong value of end_offset"); consistent = 0; } if (hdr_start > hdr_end) { ERR("start_offset greater than end_offset"); consistent = 0; } if (hdr_start > hdr_write) { ERR("start_offset greater than write_offset"); consistent = 0; } if (hdr_write > hdr_end) { ERR("write_offset greater than end_offset"); consistent = 0; } pmemlog_close(plp); if (consistent) LOG(4, "pool consistency check OK"); return consistent; } #ifndef _WIN32 /* * pmemlog_check -- log memory pool consistency check * * Returns true if consistent, zero if inconsistent, -1/error if checking * cannot happen due to other errors. */ int pmemlog_check(const char *path) { return pmemlog_checkU(path); } #else /* * pmemlog_checkW -- log memory pool consistency check */ int pmemlog_checkW(const wchar_t *path) { char *upath = util_toUTF8(path); if (upath == NULL) return -1; int ret = pmemlog_checkU(upath); util_free_UTF8(upath); return ret; } #endif /* * pmemlog_ctl_getU -- programmatically executes a read ctl query */ #ifndef _WIN32 static inline #endif int pmemlog_ctl_getU(PMEMlogpool *plp, const char *name, void *arg) { LOG(3, "plp %p name %s arg %p", plp, name, arg); return ctl_query(plp == NULL ? NULL : plp->ctl, plp, CTL_QUERY_PROGRAMMATIC, name, CTL_QUERY_READ, arg); } /* * pmemblk_ctl_setU -- programmatically executes a write ctl query */ #ifndef _WIN32 static inline #endif int pmemlog_ctl_setU(PMEMlogpool *plp, const char *name, void *arg) { LOG(3, "plp %p name %s arg %p", plp, name, arg); return ctl_query(plp == NULL ? NULL : plp->ctl, plp, CTL_QUERY_PROGRAMMATIC, name, CTL_QUERY_WRITE, arg); } /* * pmemlog_ctl_execU -- programmatically executes a runnable ctl query */ #ifndef _WIN32 static inline #endif int pmemlog_ctl_execU(PMEMlogpool *plp, const char *name, void *arg) { LOG(3, "plp %p name %s arg %p", plp, name, arg); return ctl_query(plp == NULL ? NULL : plp->ctl, plp, CTL_QUERY_PROGRAMMATIC, name, CTL_QUERY_RUNNABLE, arg); } #ifndef _WIN32 /* * pmemlog_ctl_get -- programmatically executes a read ctl query */ int pmemlog_ctl_get(PMEMlogpool *plp, const char *name, void *arg) { return pmemlog_ctl_getU(plp, name, arg); } /* * pmemlog_ctl_set -- programmatically executes a write ctl query */ int pmemlog_ctl_set(PMEMlogpool *plp, const char *name, void *arg) { return pmemlog_ctl_setU(plp, name, arg); } /* * pmemlog_ctl_exec -- programmatically executes a runnable ctl query */ int pmemlog_ctl_exec(PMEMlogpool *plp, const char *name, void *arg) { return pmemlog_ctl_execU(plp, name, arg); } #else /* * pmemlog_ctl_getW -- programmatically executes a read ctl query */ int pmemlog_ctl_getW(PMEMlogpool *plp, const wchar_t *name, void *arg) { char *uname = util_toUTF8(name); if (uname == NULL) return -1; int ret = pmemlog_ctl_getU(plp, uname, arg); util_free_UTF8(uname); return ret; } /* * pmemlog_ctl_setW -- programmatically executes a write ctl query */ int pmemlog_ctl_setW(PMEMlogpool *plp, const wchar_t *name, void *arg) { char *uname = util_toUTF8(name); if (uname == NULL) return -1; int ret = pmemlog_ctl_setU(plp, uname, arg); util_free_UTF8(uname); return ret; } /* * pmemlog_ctl_execW -- programmatically executes a runnable ctl query */ int pmemlog_ctl_execW(PMEMlogpool *plp, const wchar_t *name, void *arg) { char *uname = util_toUTF8(name); if (uname == NULL) return -1; int ret = pmemlog_ctl_execU(plp, uname, arg); util_free_UTF8(uname); return ret; } #endif #if FAULT_INJECTION void pmemlog_inject_fault_at(enum pmem_allocation_type type, int nth, const char *at) { core_inject_fault_at(type, nth, at); } int pmemlog_fault_injection_enabled(void) { return core_fault_injection_enabled(); } #endif pmdk-1.13.1/src/libpmemlog/libpmemlog.c0000664000000000000000000001031514435627501016450 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2018, Intel Corporation */ /* * libpmemlog.c -- pmem entry points for libpmemlog */ #include #include #include "libpmemlog.h" #include "ctl_global.h" #include "pmemcommon.h" #include "log.h" /* * The variable from which the config is directly loaded. The string * cannot contain any comments or extraneous white characters. */ #define LOG_CONFIG_ENV_VARIABLE "PMEMLOG_CONF" /* * The variable that points to a config file from which the config is loaded. */ #define LOG_CONFIG_FILE_ENV_VARIABLE "PMEMLOG_CONF_FILE" /* * log_ctl_init_and_load -- (static) initializes CTL and loads configuration * from env variable and file */ static int log_ctl_init_and_load(PMEMlogpool *plp) { LOG(3, "plp %p", plp); if (plp != NULL && (plp->ctl = ctl_new()) == NULL) { LOG(2, "!ctl_new"); return -1; } char *env_config = os_getenv(LOG_CONFIG_ENV_VARIABLE); if (env_config != NULL) { if (ctl_load_config_from_string(plp ? plp->ctl : NULL, plp, env_config) != 0) { LOG(2, "unable to parse config stored in %s " "environment variable", LOG_CONFIG_ENV_VARIABLE); goto err; } } char *env_config_file = os_getenv(LOG_CONFIG_FILE_ENV_VARIABLE); if (env_config_file != NULL && env_config_file[0] != '\0') { if (ctl_load_config_from_file(plp ? plp->ctl : NULL, plp, env_config_file) != 0) { LOG(2, "unable to parse config stored in %s " "file (from %s environment variable)", env_config_file, LOG_CONFIG_FILE_ENV_VARIABLE); goto err; } } return 0; err: if (plp) ctl_delete(plp->ctl); return -1; } /* * log_init -- load-time initialization for log * * Called automatically by the run-time loader. */ ATTR_CONSTRUCTOR void libpmemlog_init(void) { ctl_global_register(); if (log_ctl_init_and_load(NULL)) FATAL("error: %s", pmemlog_errormsg()); common_init(PMEMLOG_LOG_PREFIX, PMEMLOG_LOG_LEVEL_VAR, PMEMLOG_LOG_FILE_VAR, PMEMLOG_MAJOR_VERSION, PMEMLOG_MINOR_VERSION); LOG(3, NULL); } /* * libpmemlog_fini -- libpmemlog cleanup routine * * Called automatically when the process terminates. */ ATTR_DESTRUCTOR void libpmemlog_fini(void) { LOG(3, NULL); common_fini(); } /* * pmemlog_check_versionU -- see if lib meets application version requirements */ #ifndef _WIN32 static inline #endif const char * pmemlog_check_versionU(unsigned major_required, unsigned minor_required) { LOG(3, "major_required %u minor_required %u", major_required, minor_required); if (major_required != PMEMLOG_MAJOR_VERSION) { ERR("libpmemlog major version mismatch (need %u, found %u)", major_required, PMEMLOG_MAJOR_VERSION); return out_get_errormsg(); } if (minor_required > PMEMLOG_MINOR_VERSION) { ERR("libpmemlog minor version mismatch (need %u, found %u)", minor_required, PMEMLOG_MINOR_VERSION); return out_get_errormsg(); } return NULL; } #ifndef _WIN32 /* * pmemlog_check_version -- see if lib meets application version requirements */ const char * pmemlog_check_version(unsigned major_required, unsigned minor_required) { return pmemlog_check_versionU(major_required, minor_required); } #else /* * pmemlog_check_versionW -- see if lib meets application version requirements */ const wchar_t * pmemlog_check_versionW(unsigned major_required, unsigned minor_required) { if (pmemlog_check_versionU(major_required, minor_required) != NULL) return out_get_errormsgW(); else return NULL; } #endif /* * pmemlog_set_funcs -- allow overriding libpmemlog's call to malloc, etc. */ void pmemlog_set_funcs( void *(*malloc_func)(size_t size), void (*free_func)(void *ptr), void *(*realloc_func)(void *ptr, size_t size), char *(*strdup_func)(const char *s)) { LOG(3, NULL); util_set_alloc_funcs(malloc_func, free_func, realloc_func, strdup_func); } /* * pmemlog_errormsgU -- return last error message */ #ifndef _WIN32 static inline #endif const char * pmemlog_errormsgU(void) { return out_get_errormsg(); } #ifndef _WIN32 /* * pmemlog_errormsg -- return last error message */ const char * pmemlog_errormsg(void) { return pmemlog_errormsgU(); } #else /* * pmemlog_errormsgW -- return last error message as wchar_t */ const wchar_t * pmemlog_errormsgW(void) { return out_get_errormsgW(); } #endif pmdk-1.13.1/src/libpmemlog/libpmemlog.def0000664000000000000000000000114414435627501016764 0ustar rootroot;;;; Begin Copyright Notice ; SPDX-License-Identifier: BSD-3-Clause ; Copyright 2016-2018, Intel Corporation ;;;; End Copyright Notice LIBRARY libpmemlog VERSION 1.0 EXPORTS pmemlog_check_versionU pmemlog_check_versionW pmemlog_ctl_execU; pmemlog_ctl_execW; pmemlog_ctl_getU; pmemlog_ctl_getW; pmemlog_ctl_setU; pmemlog_ctl_setW; pmemlog_set_funcs pmemlog_errormsgU pmemlog_errormsgW pmemlog_createU pmemlog_createW pmemlog_openU pmemlog_openW pmemlog_close pmemlog_checkU pmemlog_checkW pmemlog_nbyte pmemlog_append pmemlog_appendv pmemlog_rewind pmemlog_tell pmemlog_walk DllMain pmdk-1.13.1/src/libpmemlog/Makefile0000664000000000000000000000115314435627501015615 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2023, Intel Corporation # # src/libpmemlog/Makefile -- Makefile for libpmemlog # LIBRARY_NAME = pmemlog LIBRARY_SO_VERSION = 1 LIBRARY_VERSION = 0.0 include ../core/pmemcore.inc include ../common/pmemcommon.inc SOURCE +=\ libpmemlog.c\ log.c include ../Makefile.inc # Libpmemlog is deprecated. # This flag allows to build tests, examples and benchmarks # using pmemlog despite the deprecated state. CFLAGS += -Wno-deprecated-declarations CFLAGS += $(LIBNDCTL_CFLAGS) $(MINIASYNC_CFLAGS) -DPMEM2_USE_MINIASYNC=1 LIBS += -pthread -lpmem $(LIBNDCTL_LIBS) pmdk-1.13.1/src/libpmemlog/libpmemlog.link.in0000664000000000000000000000076514435627501017600 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2019, Intel Corporation # # # src/libpmemlog.link -- linker link file for libpmemlog # LIBPMEMLOG_1.0 { global: pmemlog_check_version; pmemlog_ctl_exec; pmemlog_ctl_get; pmemlog_ctl_set; pmemlog_set_funcs; pmemlog_errormsg; pmemlog_create; pmemlog_open; pmemlog_close; pmemlog_check; pmemlog_nbyte; pmemlog_append; pmemlog_appendv; pmemlog_tell; pmemlog_rewind; pmemlog_walk; fault_injection; local: *; }; pmdk-1.13.1/src/libpmemlog/libpmemlog.vcxproj0000664000000000000000000001457714435627501017737 0ustar rootroot Debug x64 Release x64 {9e9e3d25-2139-4a5d-9200-18148ddead45} {901f04db-e1a5-4a41-8b81-9d31c19acd59} {0B1818EB-BDC8-4865-964F-DB8BF05CFD86} DynamicLibrary libpmemlog libpmemlog en-US 14.0 10.0.22000.0 10.0.10240.0 DynamicLibrary true v143 DynamicLibrary false false v143 pmdk-1.13.1/src/libpmemlog/libpmemlog.vcxproj.filters0000664000000000000000000001542214435627501021374 0ustar rootroot Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files {49cfa2b4-cfcb-4c02-928a-c04d1cceffb8} {ac09c2fe-a24b-4a86-8763-d4e06d996ef3} Source Files Source Files pmdk-1.13.1/src/libpmemlog/libpmemlog_main.c0000664000000000000000000000123514435627501017455 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016-2017, Intel Corporation */ /* * libpmemlog_main.c -- entry point for libpmemlog.dll * * XXX - This is a placeholder. All the library initialization/cleanup * that is done in library ctors/dtors, as well as TLS initialization * should be moved here. */ void libpmemlog_init(void); void libpmemlog_fini(void); int APIENTRY DllMain(HINSTANCE hInstance, DWORD dwReason, LPVOID lpReserved) { switch (dwReason) { case DLL_PROCESS_ATTACH: libpmemlog_init(); break; case DLL_THREAD_ATTACH: case DLL_THREAD_DETACH: break; case DLL_PROCESS_DETACH: libpmemlog_fini(); break; } return TRUE; } pmdk-1.13.1/src/libpmemlog/log.h0000664000000000000000000000553514435627501015117 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2014-2021, Intel Corporation */ /* * log.h -- internal definitions for libpmem log module */ #ifndef LOG_H #define LOG_H 1 #include #include #include #include "ctl.h" #include "util.h" #include "os_thread.h" #include "pool_hdr.h" #include "page_size.h" #ifdef __cplusplus extern "C" { #endif #include "alloc.h" #include "fault_injection.h" #define PMEMLOG_LOG_PREFIX "libpmemlog" #define PMEMLOG_LOG_LEVEL_VAR "PMEMLOG_LOG_LEVEL" #define PMEMLOG_LOG_FILE_VAR "PMEMLOG_LOG_FILE" /* attributes of the log memory pool format for the pool header */ #define LOG_HDR_SIG "PMEMLOG" /* must be 8 bytes including '\0' */ #define LOG_FORMAT_MAJOR 1 #define LOG_FORMAT_FEAT_DEFAULT \ {POOL_FEAT_COMPAT_DEFAULT, POOL_FEAT_INCOMPAT_DEFAULT, 0x0000} #define LOG_FORMAT_FEAT_CHECK \ {POOL_FEAT_COMPAT_VALID, POOL_FEAT_INCOMPAT_VALID, 0x0000} static const features_t log_format_feat_default = LOG_FORMAT_FEAT_DEFAULT; struct pmemlog { struct pool_hdr hdr; /* memory pool header */ /* root info for on-media format... */ uint64_t start_offset; /* start offset of the usable log space */ uint64_t end_offset; /* maximum offset of the usable log space */ uint64_t write_offset; /* current write point for the log */ /* some run-time state, allocated out of memory pool... */ void *addr; /* mapped region */ size_t size; /* size of mapped region */ int is_pmem; /* true if pool is PMEM */ int rdonly; /* true if pool is opened read-only */ os_rwlock_t *rwlockp; /* pointer to RW lock */ int is_dev_dax; /* true if mapped on device dax */ struct ctl *ctl; /* top level node of the ctl tree structure */ struct pool_set *set; /* pool set info */ }; /* data area starts at this alignment after the struct pmemlog above */ #define LOG_FORMAT_DATA_ALIGN ((uintptr_t)PMEM_PAGESIZE) /* * log_convert2h -- convert pmemlog structure to host byte order */ static inline void log_convert2h(struct pmemlog *plp) { plp->start_offset = le64toh(plp->start_offset); plp->end_offset = le64toh(plp->end_offset); plp->write_offset = le64toh(plp->write_offset); } /* * log_convert2le -- convert pmemlog structure to LE byte order */ static inline void log_convert2le(struct pmemlog *plp) { plp->start_offset = htole64(plp->start_offset); plp->end_offset = htole64(plp->end_offset); plp->write_offset = htole64(plp->write_offset); } #if FAULT_INJECTION void pmemlog_inject_fault_at(enum pmem_allocation_type type, int nth, const char *at); int pmemlog_fault_injection_enabled(void); #else static inline void pmemlog_inject_fault_at(enum pmem_allocation_type type, int nth, const char *at) { /* suppress unused-parameter errors */ SUPPRESS_UNUSED(type, nth, at); abort(); } static inline int pmemlog_fault_injection_enabled(void) { return 0; } #endif #ifdef __cplusplus } #endif #endif pmdk-1.13.1/src/tools/0000775000000000000000000000000014435627501013166 5ustar rootrootpmdk-1.13.1/src/tools/.gitignore0000664000000000000000000000012514435627501015154 0ustar rootrootTAGS cscope.in.out cscope.po.out cscope.out *.static-debug *.static-nondebug .synced pmdk-1.13.1/src/tools/Makefile0000664000000000000000000000162314435627501014630 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2022, Intel Corporation # # Makefile -- top Makefile for tools # TOP = ../.. TESTCONFIG=$(TOP)/src/test/testconfig.sh TARGETS = pmempool daxio pmreorder SCOPEDIRS=$(TARGETS) SCOPEFILES=$(foreach dir, $(SCOPEDIRS), $(shell find $(dir) -name *.[ch] )) all : TARGET = all check : TARGET = check test : TARGET = test clean : TARGET = clean clobber: TARGET = clobber cstyle : TARGET = cstyle format : TARGET = format install: TARGET = install uninstall: TARGET = uninstall sparse: TARGET = sparse all clean clobber cstyle install uninstall check format test sparse: $(TARGETS) $(TESTCONFIG): $(TARGETS): $(MAKE) -C $@ $(TARGET) clean: $(RM) TAGS cscope.in.out cscope.out cscope.po.out clobber: clean cscope: cscope -q -b $(SCOPEFILES) ctags -e $(SCOPEFILES) .PHONY: all clean clobber cstyle format install uninstall common cscope $(TARGETS) pmdk-1.13.1/src/tools/pmreorder/0000775000000000000000000000000014435627501015165 5ustar rootrootpmdk-1.13.1/src/tools/pmreorder/pmreorder.py0000664000000000000000000000667514435627501017554 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2023, Intel Corporation import argparse import statemachine import opscontext import consistencycheckwrap import loggingfacility import markerparser import sys import reorderengines import signal # Prevent printing python stacktrace on SIGINT. To achieve that throw # "SystemExit exception" instead of "KeyboardInterrupt exception". signal.signal(signal.SIGINT, lambda signum, frame: exit(signum)) def main(): pmreorder_version = "unknown" """ Argv[1] should be given in order to use -v or --version flag. It is passed from the installation script. We check whether argv[1] was given. If it's not any of regular parameters - we use it as a version of pmreorder and remove it from the arguments list. """ if len(sys.argv) > 1 and sys.argv[1][0] != "-": pmreorder_version = sys.argv[1] del sys.argv[1] # TODO unicode support # TODO parameterize reorder engine type parser = argparse.ArgumentParser(description="Store reordering tool") parser.add_argument( "-l", "--logfile", required=True, help="the pmemcheck log file to process", ) parser.add_argument( "-c", "--checker", choices=consistencycheckwrap.checkers, default=consistencycheckwrap.checkers[0], help="choose consistency checker type", ) parser.add_argument( "-p", "--path", required=True, help="path to the consistency checker and arguments. " + "Note: If program is given, the program has to take " + "a file name as the last parameter.", nargs="+", ) parser.add_argument( "-n", "--name", help="consistency check function for the 'lib' checker. Note: " + "The function has to take a file name as the only parameter.", ) parser.add_argument("-o", "--output", help="set the logger output file") parser.add_argument( "-e", "--output-level", choices=loggingfacility.log_levels, help="set the output log level", ) parser.add_argument( "-x", "--extended-macros", help="list of pairs MARKER=ENGINE or json config file", ) parser.add_argument( "-v", "--version", help="print version of the pmreorder", action="version", version="%(prog)s " + pmreorder_version, ) engines_keys = list(reorderengines.engines.keys()) parser.add_argument( "-r", "--default-engine", help="set default reorder engine default=NoReorderNoChecker", choices=engines_keys, default=engines_keys[0], ) args = parser.parse_args() logger = loggingfacility.get_logger(args.output, args.output_level) checker = consistencycheckwrap.get_checker( args.checker, " ".join(args.path), args.name, logger ) markers = markerparser.MarkerParser().get_markers(args.extended_macros) # create the script context context = opscontext.OpsContext( args.logfile, checker, logger, args.default_engine, markers ) logger.debug("Input parameters: {}".format(context.__dict__)) # init and run the state machine a = statemachine.StateMachine(statemachine.InitState(context)) operations, operation_ids, markers = context.extract_operations() if a.run_all(operations, operation_ids, markers) is False: sys.exit(1) if __name__ == "__main__": main() pmdk-1.13.1/src/tools/pmreorder/.gitignore0000664000000000000000000000003414435627501017152 0ustar rootroot__pycache__ *.pyc pmreorder pmdk-1.13.1/src/tools/pmreorder/reorderengines.py0000664000000000000000000002523514435627501020561 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2021, Intel Corporation from itertools import combinations from itertools import permutations from itertools import islice from itertools import chain from random import sample from functools import partial from reorderexceptions import NotSupportedOperationException import collections class FullReorderEngine: def __init__(self): self.test_on_barrier = True """ Realizes a full reordering of stores within a given list. Example: input: (a, b, c) output: () ('a',) ('b',) ('c',) ('a', 'b') ('a', 'c') ('b', 'a') ('b', 'c') ('c', 'a') ('c', 'b') ('a', 'b', 'c') ('a', 'c', 'b') ('b', 'a', 'c') ('b', 'c', 'a') ('c', 'a', 'b') ('c', 'b', 'a') """ def generate_sequence(self, store_list): """ Generates all possible combinations of all possible lengths, based on the operations in the list. :param store_list: The list of stores to be reordered. :type store_list: list of :class:`memoryoperations.Store` :return: Yields all combinations of stores. :rtype: iterable """ for length in range(0, len(store_list) + 1): for permutation in permutations(store_list, length): yield permutation class AccumulativeReorderEngine: def __init__(self): self.test_on_barrier = True """ Realizes an accumulative reorder of stores within a given list. Example: input: (a, b, c) output: () ('a') ('a', 'b') ('a', 'b', 'c') """ def generate_sequence(self, store_list): """ Generates all accumulative lists, based on the operations in the store list. :param store_list: The list of stores to be reordered. :type store_list: list of :class:`memoryoperations.Store` :return: Yields all accumulative combinations of stores. :rtype: iterable """ for i in range(0, len(store_list) + 1): out_list = [store_list[i] for i in range(0, i)] yield out_list class AccumulativeReverseReorderEngine: def __init__(self): self.test_on_barrier = True """ Realizes an accumulative reorder of stores within a given list in reverse order. Example: input: (a, b, c) output: () ('c') ('c', 'b') ('c', 'b', 'a') """ def generate_sequence(self, store_list): """ Reverse all elements order and generates all accumulative lists. :param store_list: The list of stores to be reordered. :type store_list: list of :class:`memoryoperations.Store` :return: Yields all accumulative combinations of stores. :rtype: iterable """ store_list = list(reversed(store_list)) for i in range(len(store_list) + 1): yield [store_list[j] for j in range(i)] class SlicePartialReorderEngine: """ Generates a slice of the full reordering of stores within a given list. Example: input: (a, b, c), start = 2, stop = None, step = 2 output: ('b') ('a', 'b') ('b', 'c') """ def __init__(self, start, stop, step=1): """ Initializes the generator with the provided parameters. :param start: Number of preceding elements to be skipped. :param stop: The element at which the slice is to stop. :param step: How many values are skipped between successive calls. """ self._start = start self._stop = stop self._step = step self.test_on_barrier = True def generate_sequence(self, store_list): """ This generator yields a slice of all possible combinations. The result may be a set of combinations of different lengths, depending on the slice parameters provided at object creation. :param store_list: The list of stores to be reordered. :type store_list: list of :class:`memoryoperations.Store` :return: Yields a slice of all combinations of stores. :rtype: iterable """ for sl in islice( chain( *map( lambda x: combinations(store_list, x), range(0, len(store_list) + 1), ) ), self._start, self._stop, self._step, ): yield sl class FilterPartialReorderEngine: """ Generates a filtered set of the combinations without duplication of stores within a given list. Example: input: (a, b, c), filter = filter_min_elem, kwarg1 = 2 output: (a, b) (a, c) (b, c) (a, b, c) input: (a, b, c), filter = filter_max_elem, kwarg1 = 2 output: () (a) (b) (c) (a, b) (a, c) (b, c) input: (a, b, c), filter = filter_between_elem, kwarg1 = 2, kwarg2 = 2 output: (a, b) (a, c) (b, c) """ def __init__(self, func, **kwargs): """ Initializes the generator with the provided parameters. :param func: The filter function. :param **kwargs: Arguments to the filter function. """ self._filter = func self._filter_kwargs = kwargs self.test_on_barrier = True @staticmethod def filter_min_elem(store_list, **kwargs): """ Filter stores list if number of element is less than kwarg1 """ if len(store_list) < kwargs["kwarg1"]: return False return True @staticmethod def filter_max_elem(store_list, **kwargs): """ Filter stores list if number of element is greater than kwarg1. """ if len(store_list) > kwargs["kwarg1"]: return False return True @staticmethod def filter_between_elem(store_list, **kwargs): """ Filter stores list if number of element is greater or equal kwarg1 and less or equal kwarg2. """ store_len = len(store_list) if store_len >= kwargs["kwarg1"] and store_len <= kwargs["kwarg2"]: return True return False def generate_sequence(self, store_list): """ This generator yields a filtered set of combinations. :param store_list: The list of stores to be reordered. :type store_list: list of :class:`memoryoperations.Store` :return: Yields a filtered set of combinations. :rtype: iterable """ filter_fun = getattr(self, self._filter, None) for elem in filter( partial(filter_fun, **self._filter_kwargs), chain( *map( lambda x: combinations(store_list, x), range(0, len(store_list) + 1), ) ), ): yield elem class RandomPartialReorderEngine: """ Generates a random sequence of combinations of stores. Example: input: (a, b, c), max_seq = 3 output: ('b', 'c') ('b',) ('a', 'b', 'c') """ def __init__(self, max_seq=3): """ Initializes the generator with the provided parameters. :param max_seq: The number of combinations to be generated. """ self.test_on_barrier = True self._max_seq = max_seq def generate_sequence(self, store_list): """ This generator yields a random sequence of combinations. Number of combinations without replacement has to be limited to 1000 because of exponential growth of elements. Example: for 10 element from 80 -> 1646492110120 combinations for 20 element from 80 -> 3.5353161422122E+18 combinations for 40 element from 80 -> 1.0750720873334E+23 combinations :param store_list: The list of stores to be reordered. :type store_list: list of :class:`memoryoperations.Store` :return: Yields a random sequence of combinations. :rtype: iterable """ population = list( chain( *map( lambda x: islice(combinations(store_list, x), 1000), range(0, len(store_list) + 1), ) ) ) population_size = len(population) for elem in sample( population, self._max_seq if self._max_seq <= population_size else population_size, ): yield elem class NoReorderEngine: def __init__(self): self.test_on_barrier = True """ A NULL reorder engine. Example: input: (a, b, c) output: (a, b, c) """ def generate_sequence(self, store_list): """ This generator does not modify the provided store list. :param store_list: The list of stores to be reordered. :type store_list: The list of :class:`memoryoperations.Store` :return: The unmodified list of stores. :rtype: iterable """ return [store_list] class NoCheckerEngine: def __init__(self): self.test_on_barrier = False """ A NULL reorder engine. Example: input: (a, b, c) output: (a, b, c) """ def generate_sequence(self, store_list): """ This generator does not modify the provided store list and does not do the check. :param store_list: The list of stores to be reordered. :type store_list: The list of :class:`memoryoperations.Store` :return: The unmodified list of stores. :rtype: iterable """ return [store_list] def get_engine(engine): if engine in engines: reorder_engine = engines[engine]() else: raise NotSupportedOperationException( "Not supported reorder engine: {}".format(engine) ) return reorder_engine engines = collections.OrderedDict( [ ("NoReorderNoCheck", NoCheckerEngine), ("ReorderFull", FullReorderEngine), ("NoReorderDoCheck", NoReorderEngine), ("ReorderAccumulative", AccumulativeReorderEngine), ("ReorderReverseAccumulative", AccumulativeReverseReorderEngine), ("ReorderPartial", RandomPartialReorderEngine), ] ) pmdk-1.13.1/src/tools/pmreorder/opscontext.py0000664000000000000000000000563714435627501017760 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2022, Intel Corporation from operationfactory import OperationFactory from binaryoutputhandler import BinaryOutputHandler import reorderengines import memoryoperations from itertools import repeat class OpsContext: """ Holds the context of the performed operations. :ivar _operations: The operations to be performed, based on the log file. :type _operations: list of strings :ivar reorder_engine: The reordering engine used at the moment. :type one of the reorderengine Class :ivar default_engine: The default reordering engine. :type default_engine: One of the reorderengines Class :ivar test_on_barrier: Check consistency on barrier. :type test_on_barrier: bool :ivar default_barrier: Default consistency barrier status. :type default_barrier: bool :ivar file_handler: The file handler used. """ def __init__(self, log_file, checker, logger, arg_engine, markers): """ Splits the operations in the log file and sets the instance variables to default values. :param log_file: The full name of the log file. :type log_file: str :return: None """ # TODO reading the whole file at once is rather naive # change in the future self._operations = open(log_file).read().split("|") engine = reorderengines.get_engine(arg_engine) self.reorder_engine = engine self.test_on_barrier = engine.test_on_barrier self.default_engine = self.reorder_engine self.default_barrier = self.default_engine.test_on_barrier self.file_handler = BinaryOutputHandler(checker, logger) self.checker = checker self.logger = logger self.markers = markers self.stack_engines = [("START", getattr(memoryoperations, arg_engine))] # TODO this should probably be made a generator def extract_operations(self): """ Creates specific operation objects based on the labels available in the split log file. :return: list of subclasses of :class:`memoryoperations.BaseOperation` """ enumerated_ops = list(enumerate(self._operations)) markers = list( filter( lambda e: e[1].endswith(".BEGIN") or e[1].endswith(".END"), enumerated_ops, ) ) operation_ids = list(enumerated_ops) stop_index = start_index = 0 for i, elem in enumerated_ops: if "START" in elem: start_index = i elif "STOP" in elem: stop_index = i operations = list( map( OperationFactory.create_operation, self._operations[start_index + 1: stop_index], repeat(self.markers), repeat(self.stack_engines), ) ) return (operations, operation_ids, markers) pmdk-1.13.1/src/tools/pmreorder/loggingfacility.py0000664000000000000000000000330214435627501020710 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2021, Intel Corporation import logging log_levels = ["debug", "info", "warning", "error", "critical"] class LoggingBase: def debug(self, text): pass def info(self, text): pass def warning(self, text): pass def error(self, text): pass def critical(self, text): pass class DefaultFileLogger(LoggingBase): def __init__(self, name="pmreorder", **kwargs): logging.basicConfig(**kwargs) self.__logger = logging.getLogger(name) def debug(self, text): self.__logger.debug(text) def info(self, text): self.__logger.info(text) def warning(self, text): self.__logger.warning(text) def error(self, text): self.__logger.error(text) def critical(self, text): self.__logger.critical(text) class DefaultPrintLogger(LoggingBase): def debug(self, text): print("DEBUG:", text) def info(self, text): print("INFO:", text) def warning(self, text): print("WARNING:", text) def error(self, text): print("ERROR:", text) def critical(self, text): print("CRITICAL:", text) def get_logger(log_output, log_level=None): logger = None # check if log_level is valid log_level = "warning" if log_level is None else log_level numeric_level = getattr(logging, log_level.upper()) if not isinstance(numeric_level, int): raise ValueError("Invalid log level: {}".format(log_level.upper())) if log_output is None: logger = DefaultPrintLogger() else: logger = DefaultFileLogger(filename=log_output, level=numeric_level) return logger pmdk-1.13.1/src/tools/pmreorder/binaryoutputhandler.py0000664000000000000000000001745014435627501021651 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2021, Intel Corporation from loggingfacility import LoggingBase from reorderexceptions import InconsistentFileException from sys import byteorder import utils import os class BinaryOutputHandler: """ Handle :class:`BinaryFile` objects. Creates and aggregates :class:`BinaryFile` objects for ease of use. Implements methods for batch handling of aggregated files. :ivar _files: A list of registered files, most recent last. :type _files: list """ def __init__(self, checker, logger=None): """ Binary handler constructor. :param checker: consistency checker object :type checker: ConsistencyCheckerBase :param logger: logger handle, default: empty logger (LoggingBase) :type logger: subclass of :class:`LoggingBase` """ self._files = [] self._checker = checker self._logger = logger or LoggingBase() def add_file(self, file, map_base, size): """ Create and append a mapped file to :attr:`_files`. :param file: Full path of the mapped file to be added. :type file: str :param map_base: Base address of the mapped file. :type map_base: int :param size: Size of the file. :type size: int :return: None """ self._files.append( BinaryFile(file, map_base, size, self._checker, self._logger) ) def remove_file(self, file): """Remove file from :attr:`_files`. :param file: File to be removed. :type file: str :return: None """ for bf in self._files: if bf.file_name is file: self._files.remove(bf) def do_store(self, store_op): """ Perform a store to the given file. The file is chosen based on the address and size of the store. :param store_op: The store operation to be performed. :type store_op: Store :return: None :raises: Generic exception - to be precised later. """ store_ok = False for i, bf in enumerate(self._files): if utils.range_cmp(store_op, bf) == 0: self._logger.debug( "Doing store in file no. {0}: {1}".format(i, bf) ) bf.do_store(store_op) store_ok = True if not store_ok: raise OSError( "No suitable file found for store {}".format(store_op) ) def do_revert(self, store_op): """ Reverts a store made to a file. Performing a revert on a store that has not been made previously yields undefined behavior. :param store_op: The store to be reverted. :type store_op: Store :return: None :raises: Generic exception - to be precised later. """ revert_ok = False for bf in self._files: if utils.range_cmp(store_op, bf) == 0: bf.do_revert(store_op) revert_ok = True if not revert_ok: raise OSError( "No suitable file found for store {}".format(store_op) ) def check_consistency(self): """ Checks consistency of each registered file. :return: None :raises: Generic exception - to be precised later. """ for bf in self._files: if not bf.check_consistency(): raise InconsistentFileException( "File {} inconsistent".format(bf) ) class BinaryFile(utils.Rangeable): """Binary file handler. It is a handler for binary file operations. Internally it uses mmap to write to and read from the file. :ivar _file_name: Full path of the mapped file. :type _file_name: str :ivar _map_base: Base address of the mapped file. :type _map_base: int :ivar _map_max: Max address of the mapped file. :type _map_max: int :ivar _file_map: Memory mapped from the file. :type _file_map: mmap.mmap :ivar _checker: consistency checker object :type _checker: ConsistencyCheckerBase """ def __init__(self, file_name, map_base, size, checker, logger=None): """ Initializes the binary file handler. :param file_name: Full path of the mapped file to be added. :type file_name: str :param map_base: Base address of the mapped file. :type map_base: int :param size: Size of the file. :type size: int :param checker: consistency checker object :type checker: ConsistencyCheckerBase :param logger: logger handle, default: empty logger (LoggingBase) :type logger: subclass of :class:`LoggingBase` :return: None """ self._file_name = file_name self._map_base = map_base self._map_max = map_base + size # TODO consider mmaping only necessary parts on demand self._file_map = utils.memory_map(file_name) self._checker = checker self._logger = logger or LoggingBase() def __str__(self): return "{0} (base: {1}, size: {2})".format( self._file_name, hex(self._map_base), hex(self._map_max - self._map_base) ) def do_store(self, store_op): """ Perform the store on the file. The store records the old value for reverting. :param store_op: The store to be performed. :type store_op: Store :return: None """ base_off = store_op.get_base_address() - self._map_base max_off = store_op.get_max_address() - self._map_base # read and save old value store_op.old_value = bytes(self._file_map[base_off:max_off]) self._logger.debug( "do_store: old_value: {0}, new_value: {1}".format( hex(int.from_bytes(store_op.old_value, byteorder=byteorder)), hex(int.from_bytes(store_op.new_value, byteorder=byteorder)) ) ) # write out the new value pagesize = os.sysconf("SC_PAGE_SIZE") self._file_map[base_off:max_off] = store_op.new_value self._file_map.flush(base_off & ~(pagesize - 1), pagesize) def do_revert(self, store_op): """ Reverts the store. Write back the old value recorded while doing the store. Reverting a store which has not been made previously has undefined behavior. :param store_op: The store to be reverted. :type store_op: Store :return: None """ base_off = store_op.get_base_address() - self._map_base max_off = store_op.get_max_address() - self._map_base self._logger.debug( "do_revert: old_value: {}".format( hex(int.from_bytes(store_op.old_value, byteorder=byteorder)) ) ) # write out the old value pagesize = os.sysconf("SC_PAGE_SIZE") self._file_map[base_off:max_off] = store_op.old_value self._file_map.flush(base_off & ~(pagesize - 1), pagesize) def check_consistency(self): """ Check consistency of the file. :return: True if consistent, False otherwise. :rtype: bool """ return self._checker.check_consistency(self._file_name) == 0 def get_base_address(self): """ Returns the base address of the file. Overrides from :class:`utils.Rangeable`. :return: The base address of the mapping passed to the constructor. :rtype: int """ return self._map_base def get_max_address(self): """ Get max address of the file mapping. Overrides from :class:`utils.Rangeable`. :return: The max address of the mapping. :rtype: int """ return self._map_max pmdk-1.13.1/src/tools/pmreorder/Makefile0000664000000000000000000000044114435627501016624 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018, Intel Corporation # # Makefile -- Makefile for pmreorder # include ../Makefile.inc FLAKE8 := $(shell flake8 --version 2>/dev/null) cstyle: ifdef FLAKE8 flake8 . else @echo "Flake8 not found. Python files check skipped." endif pmdk-1.13.1/src/tools/pmreorder/markerparser.py0000664000000000000000000000326014435627501020236 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2021, Intel Corporation import os import json class MarkerParser: """ Parse marker config file and command line arg provided by user via -x parameter. """ def marker_file_parser(self, macros): """ Parse markers passed by file. They should be in json format: { "MARKER_NAME"="ENGINE_TYPE" } and separated by commas. """ markers = {} try: with open(macros) as config_file: markers = json.load(config_file) except json.decoder.JSONDecodeError: print( "Invalid config macros file format: ", macros, 'Use: {"MARKER_NAME1"="ENGINE_TYPE1",' '"MARKER_NAME2"="ENGINE_TYPE2"}' ) return markers def marker_cli_parser(self, macros): """ Parse markers passed by cli. They should be in specific format: MARKER_NAME=ENGINE_TYPE and separated by commas. """ try: markers_array = macros.split(",") return dict(pair.split("=") for pair in markers_array) except ValueError: print( "Invalid extended macros format: ", macros, "Use: MARKER_NAME1=ENGINE_TYPE1,MARKER_NAME2=ENGINE_TYPE2" ) def get_markers(self, markerset): """ Parse markers based on their format. """ if markerset is not None: if os.path.exists(markerset): return self.marker_file_parser(markerset) else: return self.marker_cli_parser(markerset) pmdk-1.13.1/src/tools/pmreorder/consistencycheckwrap.py0000664000000000000000000001334214435627501021773 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2022, Intel Corporation import subprocess # nosec from ctypes import cdll, c_char_p, c_int from loggingfacility import LoggingBase from os import path from sys import exit checkers = ["prog", "lib"] class ConsistencyCheckerBase: """ Base class for consistency checker classes. Checker of each type should implement check_consistency method. """ def check_consistency(self, filename): pass class LibChecker(ConsistencyCheckerBase): """ Allows registration of a consistency checking function and verifying the consistency of a file. The function has to be in a shared library. It is then used to check consistency of an arbitrary file. The function has to take a file name as the only parameter and return an int: 0 for inconsistent, 1 for consistent. The prototype of the function: int func_name(const char* file_name) """ def __init__(self, library_name, func_name, logger=None): """ Loads the consistency checking function from the given library. :param library_name: The full name of the library. :type library_name: str :param func_name: The name of the consistency checking function within the library. :type func_name: str :param logger: logger handle, default: empty logger (LoggingBase) :type logger: subclass of :class:`LoggingBase` :return: None """ self._lib_name = library_name self._lib_func_name = func_name self._lib_func = getattr(cdll.LoadLibrary(library_name), func_name) self._lib_func.argtypes = [c_char_p] self._lib_func.restype = c_int self._logger = logger or LoggingBase() def check_consistency(self, filename): """ Checks the consistency of a given file using the previously loaded function. :param filename: The full name of the file to be checked. :type filename: str :return: 1 if file is consistent, 0 otherwise. :rtype: int :raises: Generic exception, when no function has been loaded. """ if self._lib_func is None: raise RuntimeError( "Consistency check function {} not loaded".format( self._lib_func_name ) ) self._logger.debug( "Consistency check function: {0}.{1}({2})".format( self._lib_name, self._lib_func_name, filename ) ) return self._lib_func(filename) class ProgChecker(ConsistencyCheckerBase): """ Allows registration of a consistency checking program and verifying the consistency of a file. The binary executed with its argument is used to check consistency of an arbitrary file. The program has to take a file name as the last parameter and return an int: 0 for inconsistent, 1 for consistent. """ def __init__(self, bin_path, bin_args, logger=None): """ Loads the consistency checking binary and its arguments required to verify the consistency of a file. :param bin_path: The full path of the binary. :type bin_path: str :param bin_args: Binary's arguments to run consistency check. :type bin_args: str :param logger: logger handle, default: empty logger (LoggingBase) :type logger: subclass of :class:`LoggingBase` :return: None """ self._bin_path = bin_path self._bin_cmd = bin_args self._logger = logger or LoggingBase() def check_consistency(self, filename): """ Checks the consistency of a given file using the previously loaded function. :param filename: The full name of the file to be checked. :type filename: str :return: 1 if file is consistent, 0 otherwise. :rtype: int :raises: Generic exception, when no function has been loaded. """ if self._bin_path is None or self._bin_cmd is None: raise RuntimeError("consistency check handle not set") cmd = "{0} {1} {2}".format(self._bin_path, self._bin_cmd, filename) self._logger.debug("Consistency check program command: {}".format(cmd)) """ We mark the call of this command as 'nosec' (for Bandit scan) because pmreorder entirely relies on the execution of checkers, which are user-developed programs. Therefore, it is the user's responsibility to provide safe input as a consistency checker. """ return subprocess.call(cmd, shell=True) # nosec def get_checker(checker_type, checker_path_args, func_name, logger=None): """ Returns checker instance, based on the checker type. :param checker_type: Type of the checker, supported types: "prog", "lib". :type checker_type: str :param checker_path_args: Checker's arguments for consistency check. :type checker_path_args: str :param func_name: Name of the checker function, given only if "lib" type. :type func_name: str :param logger: logger handle, default: None :type logger: subclass of :class:`LoggingBase` :return: subclass of :class:`ConsistencyCheckerBase` """ checker_path_args = checker_path_args.split(" ", 1) checker_path = checker_path_args[0] # check for params if len(checker_path_args) > 1: args = checker_path_args[1] else: args = "" if not path.exists(checker_path): print("Invalid path: " + checker_path) exit(1) checker = None if checker_type == "prog": checker = ProgChecker(checker_path, args, logger) elif checker_type == "lib": checker = LibChecker(checker_path, func_name, logger) return checker pmdk-1.13.1/src/tools/pmreorder/utils.py0000664000000000000000000000467514435627501016713 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2021, Intel Corporation import os import mmap class Rangeable: """ Interface for all rangeable objects. All rangeable objects must be able to return their base and max addresses. """ def get_base_address(self): """ Getter for the base address of the object. :return: The base address of the object. :rtype: int """ raise NotImplementedError def get_max_address(self): """ Getter for the max address of the object. :return: The max address of the object. :rtype: int """ raise NotImplementedError class StackTrace: def __init__(self, trace=None): self.trace = trace def __str__(self): ret = "" if self.trace is not None: for line in self.trace: ret += " by\t{}\n".format(line) return ret def memory_map(filename, size=0, access=mmap.ACCESS_WRITE, offset=0): """ Memory map a file. :Warning: `offset` has to be a non-negative multiple of PAGESIZE or ALLOCATIONGRANULARITY :param filename: The file to be mapped. :type filename: str :param size: Number of bytes to be mapped. If is equal 0, the whole file at the moment of the call will be mapped. :type size: int :param offset: The offset within the file to be mapped. :type offset: int :param access: The type of access provided to mmap. :return: The mapped file. :rtype: mmap.mmap """ fd = os.open(filename, os.O_RDWR) m_file = mmap.mmap(fd, size, access=access, offset=offset) os.close(fd) return m_file def range_cmp(lhs, rhs): """ A range compare function. :param lhs: The left hand side of the comparison. :type lhs: Rangeable :param rhs: The right hand side of the comparison. :type rhs: Rangeable :return: -1 if lhs is before rhs, 1 when after and 0 on overlap. :rtype: int The comparison function may be explained as:: Will return -1: |___lhs___| |___rhs___| Will return +1: |___rhs___| |___lhs___| Will return 0: |___lhs___| |___rhs___| """ if lhs.get_max_address() <= rhs.get_base_address(): return -1 elif lhs.get_base_address() >= rhs.get_max_address(): return 1 else: return 0 pmdk-1.13.1/src/tools/pmreorder/reorderexceptions.py0000664000000000000000000000027714435627501021311 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018, Intel Corporation class InconsistentFileException(Exception): pass class NotSupportedOperationException(Exception): pass pmdk-1.13.1/src/tools/pmreorder/operationfactory.py0000664000000000000000000001224714435627501021135 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2021, Intel Corporation import memoryoperations from reorderexceptions import NotSupportedOperationException class OperationFactory: """ An abstract memory operation factory. This object factory puts special constraints on names of classes. It creates objects based on log in string format, as such the classes have to start with a capital letter and the rest of the name has to be in lowercase. For example:: STORE -> Store FULL_REORDER -> Full_reorder The object to be created has to have and internal **Factory** class with a :func:`create` method taking a string parameter. For example see :class:`memoryoperations.Store`. :cvar __factories: The registered object factories. :type __factories: dict """ __factories = {} __suffix = [".BEGIN", ".END"] memoryoperations.BaseOperation() @staticmethod def add_factory(id_, operation_factory): """ Explicitly register an object factory. This method should be used when the factory cannot be inferred from the name of the object to be created. :param id_: The id under which this factory is to be registered in the dictionary. :type id_: str :param operation_factory: The operation factory to be registered. :return: None """ OperationFactory.__factories[id_] = operation_factory @staticmethod def create_operation(string_operation, markers, stack): def check_marker_format(marker): """ Checks if marker has proper suffix. """ for s in OperationFactory.__suffix: if marker.endswith(s): return raise NotSupportedOperationException( "Incorrect marker format {}, suffix is missing.".format(marker) ) def check_pair_consistency(stack, marker): """ Checks if markers do not cross. You can pop from stack only if end marker match previous one. Example OK: MACRO1.BEGIN MACRO2.BEGIN MACRO2.END MACRO1.END Example NOT OK: MACRO1.BEGIN MACRO2.BEGIN MACRO1.END MACRO2.END """ top = stack[-1][0] if top.endswith(OperationFactory.__suffix[0]): top = top[: -len(OperationFactory.__suffix[0])] if marker.endswith(OperationFactory.__suffix[-1]): marker = marker[: -len(OperationFactory.__suffix[-1])] if top != marker: raise NotSupportedOperationException( "Cannot cross markers: {0}, {1}".format(top, marker) ) """ Creates the object based on the pre-formatted string. The string needs to be in the specific format. Each specific value in the string has to be separated with a `;`. The first field has to be the name of the operation, the rest are operation specific values. :param string_operation: The string describing the operation. :param markers: The dict describing the pair marker-engine. :param stack: The stack describing the order of engine changes. :return: The specific object instantiated based on the string. """ id_ = string_operation.split(";")[0] id_case_sensitive = id_.lower().capitalize() # checks if id_ is one of memoryoperation classes mem_ops = getattr(memoryoperations, id_case_sensitive, None) # if class is not one of memoryoperations # it means it can be user defined marker if mem_ops is None: check_marker_format(id_) # if id_ is section BEGIN if id_.endswith(OperationFactory.__suffix[0]): # BEGIN defined by user marker_name = id_.partition(".")[0] if markers is not None and marker_name in markers: engine = markers[marker_name] try: mem_ops = getattr(memoryoperations, engine) except AttributeError: raise NotSupportedOperationException( "Not supported reorder engine: {}".format(engine) ) # BEGIN but not defined by user else: mem_ops = stack[-1][1] if issubclass(mem_ops, memoryoperations.ReorderBase): stack.append((id_, mem_ops)) # END section elif id_.endswith(OperationFactory.__suffix[-1]): check_pair_consistency(stack, id_) stack.pop() mem_ops = stack[-1][1] # here we have proper memory operation to perform, # it can be Store, Fence, ReorderDefault etc. id_ = mem_ops.__name__ if id_ not in OperationFactory.__factories: OperationFactory.__factories[id_] = mem_ops.Factory() return OperationFactory.__factories[id_].create(string_operation) pmdk-1.13.1/src/tools/pmreorder/memoryoperations.py0000664000000000000000000003126314435627501021160 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2021, Intel Corporation from utils import Rangeable from utils import range_cmp from utils import StackTrace from sys import byteorder class BaseOperation: """ Base class for all memory operations. """ pass class Fence(BaseOperation): """ Describes a fence operation. The exact type of the memory barrier is not important, it is interpreted as an SFENCE or MFENCE. """ def __str__(self): return "Fence" class Factory: """ Internal factory class to be used in dynamic object creation. """ def create(self, values): """ Factory object creation method. :param values: Ignored. :type values: str :return: New Fence object. :rtype: Fence """ return Fence() class Store(BaseOperation, Rangeable): """ Describes a store operation. :ivar address: The virtual address at which to store the new value. :type address: int :ivar new_value: The new value to be written. :type new_value: bytearray :ivar size: The size of the store in bytes. :type size: int :ivar old_value: The old value read from the file. :type old_value: bytearray :ivar flushed: Indicates whether the store has been flushed. :type flushed: bool """ def __init__(self, values): """ Initializes the object based on the describing string. :param values: Pre-formatted string describing the store. :type values: str :return: None """ params = values.split(";") # calculate the offset given the registered file mapping self.address = int(params[1], 16) self.size = int(params[3], 16) self.new_value = int(params[2], 16).to_bytes( self.size, byteorder=byteorder ) if len(params) > 4: self.trace = StackTrace(params[4:]) else: self.trace = StackTrace( [ "No trace available", ] ) self.old_value = None self.flushed = False def __str__(self): return ( "Store: addr: {0}, size: {1}, val: {2}, stack trace: {3}".format( hex(self.address), hex(self.size), hex(int.from_bytes(self.new_value, byteorder=byteorder)), self.trace, ) ) def get_base_address(self): """ Override from :class:`utils.Rangeable`. :return: Virtual address of the store. :rtype: int """ return self.address def get_max_address(self): """ Override from :class:`utils.Rangeable`. :return: Virtual address of the first byte after the store. :rtype: int """ return self.address + self.size class Factory: """ Internal factory class to be used in dynamic object creation. """ def create(self, values): """ Factory object creation method. :param values: Pre-formatted string describing the store. :type values: str :return: New Store object. :rtype: Store """ return Store(values) class FlushBase(BaseOperation, Rangeable): """ Base class for flush operations. """ def is_in_flush(self, store_op): """ Check if a given store is within the flush. :param store_op: Store operation to check. :return: True if store is in flush, false otherwise. :rtype: bool """ raise NotImplementedError class Flush(FlushBase): """ Describes a flush operation. Examples of flush instructions are CLFLUSH, CLFLUSHOPT or CLWB. :ivar _address: Virtual address of the flush. :type _address: int :ivar _size: The size of the flush in bytes (should be cache line aligned). :type _size: int """ def __init__(self, values): """ Initializes the object based on the describing string. :param values: Pre-formatted string describing the flush. :type values: str :return: None """ params = values.split(";") self._address = int(params[1], 16) self._size = int(params[2], 16) def __str__(self): return "Flush: addr: {0} size: {1}".format( hex(self._address), hex(self._size) ) def is_in_flush(self, store_op): """ Override from :class:`FlushBase`. :param store_op: Store operation to check. :return: True if store is in flush, false otherwise. :rtype: bool """ if range_cmp(store_op, self) == 0: return True else: return False def get_base_address(self): """ Override from :class:`utils.Rangeable`. :return: Virtual address of the flush. :rtype: int """ return self._address def get_max_address(self): """ Override from :class:`utils.Rangeable`. :return: Virtual address of the first byte after the flush. :rtype: int """ return self._address + self._size class Factory: """ Internal factory class to be used in dynamic object creation. """ def create(self, values): """ Factory object creation method. :param values: Pre-formatted string describing the flush. :type values: str :return: New Flush object. :rtype: Flush """ return Flush(values) class ReorderBase(BaseOperation): """ Base class for all reorder type classes. """ def __init__(self, values): """ Initializes the object based on the describing string. :param values: Pre-formatted string describing values for current op. :type values: str :return: None """ # first value is the op name; for reorder op, it's marker name self._marker_name = values.split(";")[0] def __str__(self): name = self.__class__.__name__ if self._marker_name is not None: name += " -- " + self._marker_name return name class NoReorderDoCheck(ReorderBase): """ Describes the type of reordering engine to be used. This marker class triggers writing the whole sequence of stores between barriers. """ class Factory: """ Internal factory class to be used in dynamic object creation. """ def create(self, values): """ Factory object creation method. :param values: Pre-formatted string describing values for current op. :type values: str :return: New NoReorderDoCheck object. :rtype: NoReorderDoCheck """ return NoReorderDoCheck(values) class ReorderFull(ReorderBase): """ Describes the type of reordering engine to be used. This marker class triggers writing all possible sequences of stores between barriers. """ class Factory: """ Internal factory class to be used in dynamic object creation. """ def create(self, values): """ Factory object creation method. :param values: Pre-formatted string describing values for current op. :type values: str :return: New ReorderFull object. :rtype: ReorderFull """ return ReorderFull(values) class ReorderAccumulative(ReorderBase): """ Describes the type of reordering engine to be used. This marker class triggers writing all possible accumulative sequences of stores between barriers. """ class Factory: """ Internal factory class to be used in dynamic object creation. """ def create(self, values): """ Factory object creation method. :param values: Pre-formatted string describing values for current op. :type values: str :return: New ReorderAccumulative object. :rtype: ReorderAccumulative """ return ReorderAccumulative(values) class ReorderReverseAccumulative(ReorderBase): """ Describes the type of reordering engine to be used. This marker class triggers writing all possible reverted accumulative sequences of stores between barriers. """ class Factory: """ Internal factory class to be used in dynamic object creation. """ def create(self, values): """ Factory object creation method. :param values: Pre-formatted string describing values for current op. :type values: str :return: New ReorderReverseAccumulative object. :rtype: ReorderReverseAccumulative """ return ReorderReverseAccumulative(values) class NoReorderNoCheck(ReorderBase): """ Describes the type of reordering engine to be used. This marker class triggers writing the whole sequence of stores between barriers. It additionally marks that no consistency checking is to be made. """ class Factory: """ Internal factory class to be used in dynamic object creation. """ def create(self, values): """ Factory object creation method. :param values: Pre-formatted string describing values for current op. :type values: str :return: New NoReorderNoCheck object. :rtype: NoReorderNoCheck """ return NoReorderNoCheck(values) class ReorderDefault(ReorderBase): """ Describes the default reordering engine to be used. This marker class triggers default reordering. """ class Factory: """ Internal factory class to be used in dynamic object creation. """ def create(self, values): """ Factory object creation method. :param values: Pre-formatted string describing values for current op. :type values: str :return: ReorderDefault object. :rtype: ReorderDefault """ return ReorderDefault(values) class ReorderPartial(ReorderBase): """ Describes the type of reordering engine to be used. This marker class triggers writing a subset of all possible sequences of stores between barriers. The type of partial reordering is chosen at runtime. Not yet implemented. """ class Factory: """ Internal factory class to be used in dynamic object creation. """ def create(self, values): """ Factory object creation method. :param values: Pre-formatted string describing values for current op. :type values: str :return: New ReorderPartial object. :rtype: ReorderPartial """ return ReorderPartial(values) class Register_file(BaseOperation): """ Describes the file to be mapped into processes address space. :ivar name: The full name of the file. :type name: str :ivar address: The base address where the file was mapped. :type address: int :ivar size: The size of the mapping. :type size: int :ivar offset: The start offset of the mapping within the file. :type offset: int """ def __init__(self, values): """ Initializes the object based on the describing string. :param values: Pre-formatted string describing the flush. :type values: str :return: None """ params = values.split(";") self.name = params[1] self.address = int(params[2], 16) self.size = int(params[3], 16) self.offset = int(params[4], 16) def __str__(self): return ( "Register_file: name: {0} addr: {1} size: {2} offset: {3}".format( self.name, hex(self.address), hex(self.size), hex(self.offset) ) ) class Factory: """ Internal factory class to be used in dynamic object creation. """ def create(self, values): """ Factory object creation method. :param values: Pre-formatted string describing the file registration. :type values: str :return: New Register_file object. :rtype: Register_file """ return Register_file(values) pmdk-1.13.1/src/tools/pmreorder/statemachine.py0000664000000000000000000003201714435627501020207 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2022, Intel Corporation import memoryoperations as memops import reorderengines from reorderexceptions import InconsistentFileException from reorderexceptions import NotSupportedOperationException import os class State: """ The base class of all states. :ivar _context: The reordering context. :type _context: opscontext.OpsContext :ivar trans_stores: The list of unflushed stores. :type trans_stores: list of :class:`memoryoperations.Store` """ trans_stores = [] def __init__(self, context): """ Default state constructor. :param context: The context of the reordering. :type context: opscontext.OpsContext """ self._context = context def next(self, in_op): """ Go to the next state based on the input. :Note: The next state might in fact be the same state. :param in_op: The state switch trigger operation. :type in_op: subclass of :class:`memoryoperations.BaseOperation` :return: The next state. :rtype: subclass of :class:`State` """ raise NotImplementedError def run(self, in_op): """ Perform the required operation in this state. :param in_op: The operation to be performed in this state. :type in_op: subclass of :class:`memoryoperations.BaseOperation` :return: None """ raise NotImplementedError class InitState(State): """ The initial no-op state. """ def __init__(self, context): """ Saves the reordering context. :param context: The reordering context. :type context: opscontext.OpsContext """ super(InitState, self).__init__(context) def next(self, in_op): """ Switch to the next valid state. :param in_op: Ignored. :return: The next valid state. :rtype: CollectingState """ return CollectingState(self._context) def run(self, in_op): """ Does nothing. :param in_op: Ignored. :return: always True """ return True class CollectingState(State): """ Collects appropriate operations. This state mostly aggregates stores and flushes. It also validates which stores will be made persistent and passes them on to the next state. :ivar _ops_list: The list of collected stores. :type _ops_list: list of :class:`memoryoperations.Store` :ivar _inner_state: The internal state of operations. :type _inner_state: str """ def __init__(self, context): """ Saves the reordering context. :param context: The reordering context. :type context: opscontext.OpsContext """ super(CollectingState, self).__init__(context) self._ops_list = [] self._ops_list += State.trans_stores self._inner_state = "init" def next(self, in_op): """ Switch to the next valid state. :param in_op: The state switch trigger operation. :type in_op: subclass of :class:`memoryoperations.BaseOperation` :return: The next state. :rtype: subclass of :class:`State` """ if isinstance(in_op, memops.Fence) and self._inner_state == "flush": return ReplayingState(self._ops_list, self._context) else: return self def run(self, in_op): """ Perform operations in this state. Based on the type of operation, different handling is employed. The recognized and handled types of operations are: * :class:`memoryoperations.ReorderBase` * :class:`memoryoperations.FlushBase` * :class:`memoryoperations.Store` * :class:`memoryoperations.Register_file` :param in_op: The operation to be performed in this state. :type in_op: subclass of :class:`memoryoperations.BaseOperation` :return: always True """ self.move_inner_state(in_op) if isinstance(in_op, memops.ReorderBase): self.substitute_reorder(in_op) elif isinstance(in_op, memops.FlushBase): self.flush_stores(in_op) elif isinstance(in_op, memops.Store): self._ops_list.append(in_op) elif isinstance(in_op, memops.Register_file): self.reg_file(in_op) return True def substitute_reorder(self, order_ops): """ Changes the reordering engine based on the log marker class. :param order_ops: The reordering marker class. :type order_ops: subclass of :class:`memoryoperations.ReorderBase` :return: None """ if isinstance(order_ops, memops.ReorderFull): self._context.reorder_engine = reorderengines.FullReorderEngine() self._context.test_on_barrier = ( self._context.reorder_engine.test_on_barrier ) elif isinstance(order_ops, memops.ReorderPartial): # TODO add macro in valgrind or # parameter inside the tool to support parameters? self._context.reorder_engine = ( reorderengines.RandomPartialReorderEngine(3) ) self._context.test_on_barrier = ( self._context.reorder_engine.test_on_barrier ) elif isinstance(order_ops, memops.ReorderAccumulative): self._context.reorder_engine = ( reorderengines.AccumulativeReorderEngine() ) self._context.test_on_barrier = ( self._context.reorder_engine.test_on_barrier ) elif isinstance(order_ops, memops.ReorderReverseAccumulative): self._context.reorder_engine = ( reorderengines.AccumulativeReverseReorderEngine() ) self._context.test_on_barrier = ( self._context.reorder_engine.test_on_barrier ) elif isinstance(order_ops, memops.NoReorderDoCheck): self._context.reorder_engine = reorderengines.NoReorderEngine() self._context.test_on_barrier = ( self._context.reorder_engine.test_on_barrier ) elif isinstance(order_ops, memops.NoReorderNoCheck): self._context.reorder_engine = reorderengines.NoCheckerEngine() self._context.test_on_barrier = ( self._context.reorder_engine.test_on_barrier ) elif isinstance(order_ops, memops.ReorderDefault): self._context.reorder_engine = self._context.default_engine self._context.test_on_barrier = self._context.default_barrier else: raise NotSupportedOperationException( "Not supported reorder engine: {}".format(order_ops) ) def flush_stores(self, flush_op): """ Marks appropriate stores as flushed. Does not align the flush, the log is expected to have the flushes properly aligned. :param flush_op: The flush operation marker. :type flush_op: subclass of :class:`memoryoperations.FlushBase` :return: None """ for st in self._ops_list: if flush_op.is_in_flush(st): st.flushed = True def reg_file(self, file_op): """ Register a new file mapped into virtual memory. :param file_op: File registration operation marker. :type file_op: memoryoperations.Register_file :return: None """ self._context.file_handler.add_file(file_op.name, file_op.address, file_op.size) def move_inner_state(self, in_op): """ Tracks the internal state of the collection. The collected stores need to be processed only at specific moments - after full persistent memory barriers (flush-fence). :param in_op: The performed operation. :type in_op: subclass of :class:`memoryoperations.BaseOperation` :return: None """ if isinstance(in_op, memops.Store) and self._inner_state == "init": self._inner_state = "dirty" elif isinstance(in_op, memops.FlushBase) and \ self._inner_state == "dirty": self._inner_state = "flush" elif isinstance(in_op, memops.Fence) and self._inner_state == "flush": self._inner_state = "fence" elif isinstance(in_op, memops.Flush) and self._inner_state == "init": self._inner_state = "flush" class ReplayingState(State): """ Replays all collected stores according to the reordering context. :ivar _ops_list: The list of stores to be reordered and replayed. :type _ops_list: list of :class:`memoryoperations.Store` """ def __init__(self, in_ops_list, context): """ :param in_ops_list: :param context: :return: """ super(ReplayingState, self).__init__(context) self._ops_list = in_ops_list def next(self, in_op): """ Switches to the collecting state regardless of the input. :param in_op: Ignored. :type in_op: subclass of :class:`memoryoperations.BaseOperation` :return: The next state. :rtype: CollectingState """ return CollectingState(self._context) def run(self, in_op): """ Perform operations in this state. The replaying state performs reordering and if necessary checks the consistency of the registered files. The decisions and type of reordering to be used is defined by the context. :param in_op: The operation to be performed in this state. :type in_op: subclass of :class:`memoryoperations.BaseOperation` :return: State of consistency check. """ # specifies consistency state of sequence consistency = True # consider only flushed stores flushed_stores = list(filter(lambda x: x.flushed, self._ops_list)) # not-flushed stores should be passed to next state State.trans_stores = list(filter(lambda x: x.flushed is False, self._ops_list)) if self._context.test_on_barrier: self._context.logger.debug("Current reorder engine: {}" .format(self._context.reorder_engine)) for i, seq in enumerate( self._context.reorder_engine.generate_sequence(flushed_stores) ): self._context.logger.debug( "NEXT Sequence (no. {0}) with length: \ {1}".format(i, len(seq)) ) for j, op in enumerate(seq): self._context.logger.debug( "NEXT Operation (no. {0}): {1}".format(j, op) ) # do stores self._context.file_handler.do_store(op) # check consistency of all files try: self._context.file_handler.check_consistency() except InconsistentFileException as e: consistency = False self._context.logger.warning(e) stacktrace = "Call trace:\n" for num, op in enumerate(seq): stacktrace += "Store [{}]:\n".format(num) stacktrace += str(op.trace) self._context.logger.warning(stacktrace) for op in reversed(seq): # revert the changes self._context.file_handler.do_revert(op) # write all flushed stores for op in flushed_stores: self._context.file_handler.do_store(op) return consistency class StateMachine: """ The state machine driver. :ivar _curr_state: The current state. :type _curr_state: subclass of :class:`State` """ def __init__(self, init_state): """ Initialize the state machine with a specified state. :param init_state: The initial state to be used. :type init_state: subclass of :class:`State` """ self._curr_state = init_state def run_all(self, operations, operation_ids, markers): """ Starts the state machine. :param operations: The operations to be performed by the state machine. :type operations: list of :class:`memoryoperations.BaseOperation` :return: None """ all_consistent = True for ops, ops_id in zip(operations, operation_ids): self._curr_state._context.logger.info(ops) markers_to_pass = (m[1] for m in markers if m[0] < ops_id[0]) markers_to_pass = "|".join(markers_to_pass) self._curr_state._context.logger.info(markers_to_pass) os.environ["PMREORDER_MARKERS"] = markers_to_pass self._curr_state = self._curr_state.next(ops) check = self._curr_state.run(ops) if check is False: all_consistent = check return all_consistent pmdk-1.13.1/src/tools/pmempool/0000775000000000000000000000000014435627501015016 5ustar rootrootpmdk-1.13.1/src/tools/pmempool/.gitignore0000664000000000000000000000001114435627501016776 0ustar rootrootpmempool pmdk-1.13.1/src/tools/pmempool/output.h0000664000000000000000000000671514435627501016540 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2014-2023, Intel Corporation */ /* * output.h -- declarations of output printing related functions */ #include #include #include #define BLK_DEPR_STR "Libpmemblk is deprecated." #ifdef _WIN32 #define PMEMBLK_DEPR_ATTR __declspec(deprecated(BLK_DEPR_STR)) #define WIN_DEPR_STR "Windows support is deprecated." #define WIN_DEPR_ATTR __declspec(deprecated(WIN_DEPR_STR)) #else #define PMEMBLK_DEPR_ATTR __attribute__((deprecated(BLK_DEPR_STR))) #endif #ifdef _WIN32 WIN_DEPR_ATTR #endif void out_set_vlevel(int vlevel); #ifdef _WIN32 WIN_DEPR_ATTR #endif void out_set_stream(FILE *stream); #ifdef _WIN32 WIN_DEPR_ATTR #endif void out_set_prefix(const char *prefix); #ifdef _WIN32 WIN_DEPR_ATTR #endif void out_set_col_width(unsigned col_width); #ifdef _WIN32 WIN_DEPR_ATTR #endif void outv_err(const char *fmt, ...) FORMAT_PRINTF(1, 2); #ifdef _WIN32 WIN_DEPR_ATTR #endif void out_err(const char *file, int line, const char *func, const char *fmt, ...) FORMAT_PRINTF(4, 5); #ifdef _WIN32 WIN_DEPR_ATTR #endif void outv_err_vargs(const char *fmt, va_list ap); #ifdef _WIN32 WIN_DEPR_ATTR #endif void outv_indent(int vlevel, int i); #ifdef _WIN32 WIN_DEPR_ATTR #endif void outv(int vlevel, const char *fmt, ...) FORMAT_PRINTF(2, 3); #ifdef _WIN32 WIN_DEPR_ATTR #endif void outv_nl(int vlevel); #ifdef _WIN32 WIN_DEPR_ATTR #endif int outv_check(int vlevel); #ifdef _WIN32 WIN_DEPR_ATTR #endif void outv_title(int vlevel, const char *fmt, ...) FORMAT_PRINTF(2, 3); #ifdef _WIN32 WIN_DEPR_ATTR #endif void outv_field(int vlevel, const char *field, const char *fmt, ...) FORMAT_PRINTF(3, 4); #ifdef _WIN32 WIN_DEPR_ATTR #endif void outv_hexdump(int vlevel, const void *addr, size_t len, size_t offset, int sep); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_uuid_str(uuid_t uuid); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_time_str(time_t time); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_size_str(uint64_t size, int human); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_percentage(double percentage); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_checksum(void *addr, size_t len, uint64_t *csump, uint64_t skip_off); PMEMBLK_DEPR_ATTR const char *out_get_btt_map_entry(uint32_t map); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_pool_type_str(pmem_pool_type_t type); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_pool_signature(pmem_pool_type_t type); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_tx_state_str(uint64_t state); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_chunk_type_str(enum chunk_type type); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_chunk_flags(uint16_t flags); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_zone_magic_str(uint32_t magic); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_pmemoid_str(PMEMoid oid, uint64_t uuid_lo); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_arch_machine_class_str(uint8_t machine_class); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_arch_data_str(uint8_t data); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_arch_machine_str(uint16_t machine); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_last_shutdown_str(uint8_t dirty); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_alignment_desc_str(uint64_t ad, uint64_t cur_ad); #ifdef _WIN32 WIN_DEPR_ATTR #endif const char *out_get_incompat_features_str(uint32_t incompat); pmdk-1.13.1/src/tools/pmempool/common.h0000664000000000000000000001630614435627501016465 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2014-2023, Intel Corporation */ /* * common.h -- declarations of common functions */ #include #include #include #include #include "queue.h" #include "log.h" #include "blk.h" #include "libpmemobj.h" #include "lane.h" #include "ulog.h" #include "memops.h" #include "pmalloc.h" #include "list.h" #include "obj.h" #include "memblock.h" #include "heap_layout.h" #include "tx.h" #include "heap.h" #include "btt_layout.h" #include "page_size.h" /* XXX - modify Linux makefiles to generate srcversion.h and remove #ifdef */ #ifdef _WIN32 #include "srcversion.h" #define WIN_DEPR_STR "Windows support is deprecated." #define WIN_DEPR_ATTR __declspec(deprecated(WIN_DEPR_STR)) #endif #define COUNT_OF(x) (sizeof(x) / sizeof(0[x])) #define OPT_SHIFT 12 #define OPT_MASK (~((1 << OPT_SHIFT) - 1)) #define OPT_LOG (1 << (PMEM_POOL_TYPE_LOG + OPT_SHIFT)) /* deprecated */ #define OPT_BLK (1 << (PMEM_POOL_TYPE_BLK + OPT_SHIFT)) /* deprecated */ #define OPT_OBJ (1 << (PMEM_POOL_TYPE_OBJ + OPT_SHIFT)) #define OPT_BTT (1 << (PMEM_POOL_TYPE_BTT + OPT_SHIFT)) /* deprecated */ #define OPT_ALL (OPT_LOG | OPT_BLK | OPT_OBJ | OPT_BTT) #define OPT_REQ_SHIFT 8 #define OPT_REQ_MASK ((1 << OPT_REQ_SHIFT) - 1) #define _OPT_REQ(c, n) ((c) << (OPT_REQ_SHIFT * (n))) #define OPT_REQ0(c) _OPT_REQ(c, 0) #define OPT_REQ1(c) _OPT_REQ(c, 1) #define OPT_REQ2(c) _OPT_REQ(c, 2) #define OPT_REQ3(c) _OPT_REQ(c, 3) #define OPT_REQ4(c) _OPT_REQ(c, 4) #define OPT_REQ5(c) _OPT_REQ(c, 5) #define OPT_REQ6(c) _OPT_REQ(c, 6) #define OPT_REQ7(c) _OPT_REQ(c, 7) #ifndef min #define min(a, b) ((a) < (b) ? (a) : (b)) #endif #define FOREACH_RANGE(range, ranges)\ PMDK_LIST_FOREACH(range, &(ranges)->head, next) #define PLIST_OFF_TO_PTR(pop, off)\ ((off) == 0 ? NULL : (void *)((uintptr_t)(pop) + (off) - OBJ_OOB_SIZE)) #define ENTRY_TO_ALLOC_HDR(entry)\ ((void *)((uintptr_t)(entry) - sizeof(struct allocation_header))) #define OBJH_FROM_PTR(ptr)\ ((void *)((uintptr_t)(ptr) - sizeof(struct legacy_object_header))) #define DEFAULT_HDR_SIZE PMEM_PAGESIZE #define DEFAULT_DESC_SIZE PMEM_PAGESIZE #define POOL_HDR_DESC_SIZE (DEFAULT_HDR_SIZE + DEFAULT_DESC_SIZE) #define PTR_TO_ALLOC_HDR(ptr)\ ((void *)((uintptr_t)(ptr) -\ sizeof(struct legacy_object_header))) #define OBJH_TO_PTR(objh)\ ((void *)((uintptr_t)(objh) + sizeof(struct legacy_object_header))) /* invalid answer for ask_* functions */ #define INV_ANS '\0' #define FORMAT_PRINTF(a, b) __attribute__((__format__(__printf__, (a), (b)))) /* * pmem_pool_type_t -- pool types */ typedef enum { PMEM_POOL_TYPE_LOG = 0x01, /* deprecated */ PMEM_POOL_TYPE_BLK = 0x02, /* deprecated */ PMEM_POOL_TYPE_OBJ = 0x04, PMEM_POOL_TYPE_BTT = 0x08, /* deprecated */ PMEM_POOL_TYPE_ALL = 0x0f, PMEM_POOL_TYPE_UNKNOWN = 0x80, } pmem_pool_type_t; struct option_requirement { int opt; pmem_pool_type_t type; uint64_t req; }; struct options { const struct option *opts; size_t noptions; char *bitmap; const struct option_requirement *req; }; struct pmem_pool_params { pmem_pool_type_t type; char signature[POOL_HDR_SIG_LEN]; uint64_t size; mode_t mode; int is_poolset; int is_part; int is_checksum_ok; union { struct { uint64_t bsize; } blk; struct { char layout[PMEMOBJ_MAX_LAYOUT]; } obj; }; }; struct pool_set_file { int fd; char *fname; void *addr; size_t size; struct pool_set *poolset; size_t replica; time_t mtime; mode_t mode; bool fileio; }; #ifdef _WIN32 WIN_DEPR_ATTR #endif struct pool_set_file *pool_set_file_open(const char *fname, int rdonly, int check); #ifdef _WIN32 WIN_DEPR_ATTR #endif void pool_set_file_close(struct pool_set_file *file); #ifdef _WIN32 WIN_DEPR_ATTR #endif int pool_set_file_read(struct pool_set_file *file, void *buff, size_t nbytes, uint64_t off); #ifdef _WIN32 WIN_DEPR_ATTR #endif int pool_set_file_write(struct pool_set_file *file, void *buff, size_t nbytes, uint64_t off); #ifdef _WIN32 WIN_DEPR_ATTR #endif int pool_set_file_set_replica(struct pool_set_file *file, size_t replica); #ifdef _WIN32 WIN_DEPR_ATTR #endif size_t pool_set_file_nreplicas(struct pool_set_file *file); #ifdef _WIN32 WIN_DEPR_ATTR #endif void *pool_set_file_map(struct pool_set_file *file, uint64_t offset); #ifdef _WIN32 WIN_DEPR_ATTR #endif void pool_set_file_persist(struct pool_set_file *file, const void *addr, size_t len); struct range { PMDK_LIST_ENTRY(range) next; uint64_t first; uint64_t last; }; struct ranges { PMDK_LIST_HEAD(rangeshead, range) head; }; #ifdef _WIN32 WIN_DEPR_ATTR #endif pmem_pool_type_t pmem_pool_type_parse_hdr(const struct pool_hdr *hdrp); #ifdef _WIN32 WIN_DEPR_ATTR #endif pmem_pool_type_t pmem_pool_type(const void *base_pool_addr); #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmem_pool_checksum(const void *base_pool_addr); #ifdef _WIN32 WIN_DEPR_ATTR #endif pmem_pool_type_t pmem_pool_type_parse_str(const char *str); #ifdef _WIN32 WIN_DEPR_ATTR #endif uint64_t pmem_pool_get_min_size(pmem_pool_type_t type); #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmem_pool_parse_params(const char *fname, struct pmem_pool_params *paramsp, int check); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_poolset_map(const char *fname, struct pool_set **poolset, int rdonly); #ifdef _WIN32 WIN_DEPR_ATTR #endif struct options *util_options_alloc(const struct option *options, size_t nopts, const struct option_requirement *req); #ifdef _WIN32 WIN_DEPR_ATTR #endif void util_options_free(struct options *opts); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_options_verify(const struct options *opts, pmem_pool_type_t type); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_options_getopt(int argc, char *argv[], const char *optstr, const struct options *opts); #ifdef _WIN32 WIN_DEPR_ATTR #endif pmem_pool_type_t util_get_pool_type_second_page(const void *pool_base_addr); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_parse_mode(const char *str, mode_t *mode); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_parse_ranges(const char *str, struct ranges *rangesp, struct range entire); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_ranges_add(struct ranges *rangesp, struct range range); #ifdef _WIN32 WIN_DEPR_ATTR #endif void util_ranges_clear(struct ranges *rangesp); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_ranges_contain(const struct ranges *rangesp, uint64_t n); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_ranges_empty(const struct ranges *rangesp); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_check_memory(const uint8_t *buff, size_t len, uint8_t val); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_parse_chunk_types(const char *str, uint64_t *types); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_parse_lane_sections(const char *str, uint64_t *types); #ifdef _WIN32 WIN_DEPR_ATTR #endif char ask(char op, char *answers, char def_ans, const char *fmt, va_list ap); #ifdef _WIN32 WIN_DEPR_ATTR #endif char ask_Yn(char op, const char *fmt, ...) FORMAT_PRINTF(2, 3); #ifdef _WIN32 WIN_DEPR_ATTR #endif char ask_yN(char op, const char *fmt, ...) FORMAT_PRINTF(2, 3); #ifdef _WIN32 WIN_DEPR_ATTR #endif unsigned util_heap_max_zone(size_t size); #ifdef _WIN32 WIN_DEPR_ATTR #endif int util_pool_clear_badblocks(const char *path, int create); static const struct range ENTIRE_UINT64 = { { NULL, NULL }, /* range */ 0, /* first */ UINT64_MAX /* last */ }; pmdk-1.13.1/src/tools/pmempool/create.c0000664000000000000000000003565314435627501016441 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2023, Intel Corporation */ /* * create.c -- pmempool create command source file */ #include #include #include #include #include #include #include #include #include #include #include #include #include "common.h" #include "file.h" #include "create.h" #include "os.h" #include "set.h" #include "output.h" #include "libpmemblk.h" #include "libpmemlog.h" #include "libpmempool.h" #define DEFAULT_MODE 0664 /* * pmempool_create -- context and args for create command */ struct pmempool_create { int verbose; char *fname; int fexists; char *inherit_fname; int max_size; char *str_type; struct pmem_pool_params params; struct pmem_pool_params inherit_params; char *str_size; char *str_mode; char *str_bsize; uint64_t csize; int write_btt_layout; int force; char *layout; struct options *opts; int clearbadblocks; }; /* * pmempool_create_default -- default args for create command */ static const struct pmempool_create pmempool_create_default = { .verbose = 0, .fname = NULL, .fexists = 0, .inherit_fname = NULL, .max_size = 0, .str_type = NULL, .str_bsize = NULL, .csize = 0, .write_btt_layout = 0, .force = 0, .layout = NULL, .clearbadblocks = 0, .params = { .type = PMEM_POOL_TYPE_UNKNOWN, .size = 0, .mode = DEFAULT_MODE, } }; /* * help_str -- string for help message */ static const char * const help_str = "Create pmem pool of specified size, type and name\n" "NOTE: pmem blk and log pools are deprecated\n" "\n" "Common options:\n" " -s, --size size of pool\n" " -M, --max-size use maximum available space on file system\n" " -m, --mode set permissions to (the default is 0664)\n" " -i, --inherit take required parameters from specified pool file\n" " -b, --clear-bad-blocks clear bad blocks in existing files\n" " -f, --force remove the pool first\n" " -v, --verbose increase verbosity level\n" " -h, --help display this help and exit\n" "\n" "Options for PMEMBLK: (DEPRECATED)\n" " -w, --write-layout force writing the BTT layout\n" "\n" "Options for PMEMOBJ:\n" " -l, --layout layout name stored in pool's header\n" "\n" "For complete documentation see %s-create(1) manual page.\n" ; /* * long_options -- command line options */ static const struct option long_options[] = { {"size", required_argument, NULL, 's' | OPT_ALL}, {"verbose", no_argument, NULL, 'v' | OPT_ALL}, {"help", no_argument, NULL, 'h' | OPT_ALL}, {"max-size", no_argument, NULL, 'M' | OPT_ALL}, {"inherit", required_argument, NULL, 'i' | OPT_ALL}, {"mode", required_argument, NULL, 'm' | OPT_ALL}, {"write-layout", no_argument, NULL, 'w' | OPT_BLK}, {"layout", required_argument, NULL, 'l' | OPT_OBJ}, {"force", no_argument, NULL, 'f' | OPT_ALL}, {"clear-bad-blocks", no_argument, NULL, 'b' | OPT_ALL}, {NULL, 0, NULL, 0 }, }; /* * print_usage -- print application usage short description */ static void print_usage(const char *appname) { printf("NOTE: pmem blk and log pools are deprecated\n"); printf("Usage: %s create [] [] \n", appname); } /* * print_version -- print version string */ static void print_version(const char *appname) { printf("NOTE: pmem blk and log pools are deprecated\n"); printf("%s %s\n", appname, SRCVERSION); } /* * pmempool_create_help -- print help message for create command */ void pmempool_create_help(const char *appname) { print_usage(appname); print_version(appname); printf(help_str, appname); } /* * pmempool_create_obj -- create pmem obj pool */ static int pmempool_create_obj(struct pmempool_create *pcp) { PMEMobjpool *pop = pmemobj_create(pcp->fname, pcp->layout, pcp->params.size, pcp->params.mode); if (!pop) { outv_err("'%s' -- %s\n", pcp->fname, pmemobj_errormsg()); return -1; } pmemobj_close(pop); return 0; } /* * pmempool_create_blk (DEPRECATED) -- create pmem blk pool */ static int pmempool_create_blk(struct pmempool_create *pcp) { ASSERTne(pcp->params.blk.bsize, 0); int ret = 0; PMEMblkpool *pbp = pmemblk_create(pcp->fname, pcp->params.blk.bsize, pcp->params.size, pcp->params.mode); if (!pbp) { outv_err("'%s' -- %s\n", pcp->fname, pmemblk_errormsg()); return -1; } if (pcp->write_btt_layout) { outv(1, "Writing BTT layout using block %d.\n", pcp->write_btt_layout); if (pmemblk_set_error(pbp, 0) || pmemblk_set_zero(pbp, 0)) { outv_err("writing BTT layout to block 0 failed\n"); ret = -1; } } pmemblk_close(pbp); return ret; } /* * pmempool_create_log (DEPRECATED) -- create pmem log pool */ static int pmempool_create_log(struct pmempool_create *pcp) { PMEMlogpool *plp = pmemlog_create(pcp->fname, pcp->params.size, pcp->params.mode); if (!plp) { outv_err("'%s' -- %s\n", pcp->fname, pmemlog_errormsg()); return -1; } pmemlog_close(plp); return 0; } /* * pmempool_get_max_size -- return maximum allowed size of file */ #ifndef _WIN32 static int pmempool_get_max_size(const char *fname, uint64_t *sizep) { struct statvfs buf; int ret = 0; char *name = strdup(fname); if (name == NULL) { return -1; } char *dir = dirname(name); if (statvfs(dir, &buf)) ret = -1; else *sizep = buf.f_bsize * buf.f_bavail; free(name); return ret; } #else static int pmempool_get_max_size(const char *fname, uint64_t *sizep) { int ret = 0; ULARGE_INTEGER freespace; char *name = strdup(fname); if (name == NULL) { return -1; } char *dir = dirname(name); wchar_t *str = util_toUTF16(dir); if (str == NULL) { free(name); return -1; } if (GetDiskFreeSpaceExW(str, &freespace, NULL, NULL) == 0) ret = -1; else *sizep = freespace.QuadPart; free(str); free(name); return ret; } #endif /* * print_pool_params -- print some parameters of a pool */ static void print_pool_params(struct pmem_pool_params *params) { outv(1, "\ttype : %s\n", out_get_pool_type_str(params->type)); outv(1, "\tsize : %s\n", out_get_size_str(params->size, 2)); outv(1, "\tmode : 0%o\n", params->mode); switch (params->type) { case PMEM_POOL_TYPE_BLK: /* deprecated */ outv(1, "\tbsize : %s\n", out_get_size_str(params->blk.bsize, 0)); break; case PMEM_POOL_TYPE_OBJ: outv(1, "\tlayout: '%s'\n", params->obj.layout); break; default: break; } } /* * inherit_pool_params -- inherit pool parameters from specified file */ static int inherit_pool_params(struct pmempool_create *pcp) { outv(1, "Parsing pool: '%s'\n", pcp->inherit_fname); /* * If no type string passed, --inherit option must be passed * so parse file and get required parameters. */ if (pmem_pool_parse_params(pcp->inherit_fname, &pcp->inherit_params, 1)) { if (errno) perror(pcp->inherit_fname); else outv_err("%s: cannot determine type of pool\n", pcp->inherit_fname); return -1; } if (PMEM_POOL_TYPE_UNKNOWN == pcp->inherit_params.type) { outv_err("'%s' -- unknown pool type\n", pcp->inherit_fname); return -1; } print_pool_params(&pcp->inherit_params); return 0; } /* * pmempool_create_parse_args -- parse command line args */ static int pmempool_create_parse_args(struct pmempool_create *pcp, const char *appname, int argc, char *argv[], struct options *opts) { int opt, ret; while ((opt = util_options_getopt(argc, argv, "vhi:s:Mm:l:wfb", opts)) != -1) { switch (opt) { case 'v': pcp->verbose = 1; break; case 'h': pmempool_create_help(appname); exit(EXIT_SUCCESS); case 's': pcp->str_size = optarg; ret = util_parse_size(optarg, (size_t *)&pcp->params.size); if (ret || pcp->params.size == 0) { outv_err("invalid size value specified '%s'\n", optarg); return -1; } break; case 'M': pcp->max_size = 1; break; case 'm': pcp->str_mode = optarg; if (util_parse_mode(optarg, &pcp->params.mode)) { outv_err("invalid mode value specified '%s'\n", optarg); return -1; } break; case 'i': pcp->inherit_fname = optarg; break; case 'w': pcp->write_btt_layout = 1; break; case 'l': pcp->layout = optarg; break; case 'f': pcp->force = 1; break; case 'b': pcp->clearbadblocks = 1; break; default: print_usage(appname); return -1; } } /* check for , and strings */ if (optind + 2 < argc) { pcp->str_type = argv[optind]; pcp->str_bsize = argv[optind + 1]; pcp->fname = argv[optind + 2]; } else if (optind + 1 < argc) { pcp->str_type = argv[optind]; pcp->fname = argv[optind + 1]; } else if (optind < argc) { pcp->fname = argv[optind]; pcp->str_type = NULL; } else { print_usage(appname); return -1; } return 0; } static int allocate_max_size_available_file(const char *name_of_file, mode_t mode, os_off_t max_size) { int fd = os_open(name_of_file, O_CREAT | O_EXCL | O_RDWR, mode); if (fd == -1) { outv_err("!open '%s' failed", name_of_file); return -1; } os_off_t offset = 0; os_off_t length = max_size - (max_size % (os_off_t)Pagesize); int ret; do { ret = os_posix_fallocate(fd, offset, length); if (ret == 0) offset += length; else if (ret != ENOSPC) { os_close(fd); if (os_unlink(name_of_file) == -1) outv_err("!unlink '%s' failed", name_of_file); errno = ret; outv_err("!space allocation for '%s' failed", name_of_file); return -1; } length /= 2; length -= (length % (os_off_t)Pagesize); } while (length > (os_off_t)Pagesize); os_close(fd); return 0; } /* * pmempool_create_func -- main function for create command */ int pmempool_create_func(const char *appname, int argc, char *argv[]) { int ret = 0; struct pmempool_create pc = pmempool_create_default; pc.opts = util_options_alloc(long_options, sizeof(long_options) / sizeof(long_options[0]), NULL); /* parse command line arguments */ ret = pmempool_create_parse_args(&pc, appname, argc, argv, pc.opts); if (ret) exit(EXIT_FAILURE); /* set verbosity level */ out_set_vlevel(pc.verbose); umask(0); int exists = util_file_exists(pc.fname); if (exists < 0) return -1; pc.fexists = exists; int is_poolset = util_is_poolset_file(pc.fname) == 1; if (pc.inherit_fname) { if (inherit_pool_params(&pc)) { outv_err("parsing pool '%s' failed\n", pc.inherit_fname); return -1; } } /* * Parse pool type and other parameters if --inherit option * passed. It is possible to either pass --inherit option * or pool type string in command line arguments. This is * validated here. */ if (pc.str_type) { /* parse pool type string if passed in command line arguments */ pc.params.type = pmem_pool_type_parse_str(pc.str_type); if (PMEM_POOL_TYPE_UNKNOWN == pc.params.type) { outv_err("'%s' -- unknown pool type\n", pc.str_type); return -1; } if (PMEM_POOL_TYPE_BLK == pc.params.type) { /* deprecated */ if (pc.str_bsize == NULL) { outv_err("blk pool requires " "argument\n"); return -1; } if (util_parse_size(pc.str_bsize, (size_t *)&pc.params.blk.bsize)) { outv_err("cannot parse '%s' as block size\n", pc.str_bsize); return -1; } } if (PMEM_POOL_TYPE_OBJ == pc.params.type && pc.layout != NULL) { size_t max_layout = PMEMOBJ_MAX_LAYOUT; if (strlen(pc.layout) >= max_layout) { outv_err( "Layout name is too long, maximum number of characters (including the terminating null byte) is %zu\n", max_layout); return -1; } size_t len = sizeof(pc.params.obj.layout); strncpy(pc.params.obj.layout, pc.layout, len); pc.params.obj.layout[len - 1] = '\0'; } } else if (pc.inherit_fname) { pc.params.type = pc.inherit_params.type; } else { /* neither pool type string nor --inherit options passed */ print_usage(appname); return -1; } if (util_options_verify(pc.opts, pc.params.type)) return -1; if (pc.params.type != PMEM_POOL_TYPE_BLK && pc.str_bsize != NULL) { outv_err("invalid option specified for %s pool type" " -- block size\n", out_get_pool_type_str(pc.params.type)); return -1; } if (is_poolset) { if (pc.params.size) { outv_err("-s|--size cannot be used with " "poolset file\n"); return -1; } if (pc.max_size) { outv_err("-M|--max-size cannot be used with " "poolset file\n"); return -1; } } if (pc.params.size && pc.max_size) { outv_err("-M|--max-size option cannot be used with -s|--size" " option\n"); return -1; } if (pc.inherit_fname) { if (!pc.str_size && !pc.max_size) pc.params.size = pc.inherit_params.size; if (!pc.str_mode) pc.params.mode = pc.inherit_params.mode; switch (pc.params.type) { case PMEM_POOL_TYPE_BLK: /* deprecated */ if (!pc.str_bsize) pc.params.blk.bsize = pc.inherit_params.blk.bsize; break; case PMEM_POOL_TYPE_OBJ: if (!pc.layout) { memcpy(pc.params.obj.layout, pc.inherit_params.obj.layout, sizeof(pc.params.obj.layout)); } else { size_t len = sizeof(pc.params.obj.layout); strncpy(pc.params.obj.layout, pc.layout, len - 1); pc.params.obj.layout[len - 1] = '\0'; } break; default: break; } } /* * If neither --size nor --inherit options passed, check * for --max-size option - if not passed use minimum pool size. */ uint64_t min_size = pmem_pool_get_min_size(pc.params.type); if (pc.params.size == 0) { if (pc.max_size) { outv(1, "Maximum size option passed " "- getting available space of file system.\n"); ret = pmempool_get_max_size(pc.fname, &pc.params.size); if (ret) { outv_err("cannot get available space of fs\n"); return -1; } if (pc.params.size == 0) { outv_err("No space left on device\n"); return -1; } outv(1, "Available space is %s\n", out_get_size_str(pc.params.size, 2)); if (allocate_max_size_available_file(pc.fname, pc.params.mode, (os_off_t)pc.params.size)) return -1; /* * We are going to create pool based * on file size instead of the pc.params.size. */ pc.params.size = 0; } else { if (!pc.fexists) { outv(1, "No size option passed " "- picking minimum pool size.\n"); pc.params.size = min_size; } } } else { if (pc.params.size < min_size) { outv_err("size must be >= %lu bytes\n", min_size); return -1; } } if (pc.force) pmempool_rm(pc.fname, PMEMPOOL_RM_FORCE); outv(1, "Creating pool: %s\n", pc.fname); print_pool_params(&pc.params); if (pc.clearbadblocks) { int ret = util_pool_clear_badblocks(pc.fname, 1 /* ignore non-existing */); if (ret) { outv_err("'%s' -- clearing bad blocks failed\n", pc.fname); return -1; } } switch (pc.params.type) { case PMEM_POOL_TYPE_BLK: /* deprecated */ ret = pmempool_create_blk(&pc); break; case PMEM_POOL_TYPE_LOG: /* deprecated */ ret = pmempool_create_log(&pc); break; case PMEM_POOL_TYPE_OBJ: ret = pmempool_create_obj(&pc); break; default: ret = -1; break; } if (ret) { outv_err("creating pool file failed\n"); if (!pc.fexists) util_unlink(pc.fname); } util_options_free(pc.opts); return ret; } pmdk-1.13.1/src/tools/pmempool/synchronize.c0000664000000000000000000000665314435627501017547 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016-2018, Intel Corporation */ /* * synchronize.c -- pmempool sync command source file */ #include "synchronize.h" #include #include #include #include #include #include #include #include #include #include "common.h" #include "output.h" #include "libpmempool.h" /* * pmempool_sync_context -- context and arguments for sync command */ struct pmempool_sync_context { unsigned flags; /* flags which modify the command execution */ char *poolset_file; /* a path to a poolset file */ }; /* * pmempool_sync_default -- default arguments for sync command */ static const struct pmempool_sync_context pmempool_sync_default = { .flags = 0, .poolset_file = NULL, }; /* * help_str -- string for help message */ static const char * const help_str = "Check consistency of a pool\n" "\n" "Common options:\n" " -b, --bad-blocks fix bad blocks - it requires creating or reading special recovery files\n" " -d, --dry-run do not apply changes, only check for viability of synchronization\n" " -v, --verbose increase verbosity level\n" " -h, --help display this help and exit\n" "\n" "For complete documentation see %s-sync(1) manual page.\n" ; /* * long_options -- command line options */ static const struct option long_options[] = { {"bad-blocks", no_argument, NULL, 'b'}, {"dry-run", no_argument, NULL, 'd'}, {"help", no_argument, NULL, 'h'}, {"verbose", no_argument, NULL, 'v'}, {NULL, 0, NULL, 0 }, }; /* * print_usage -- (internal) print application usage short description */ static void print_usage(const char *appname) { printf("usage: %s sync [] \n", appname); } /* * print_version -- (internal) print version string */ static void print_version(const char *appname) { printf("%s %s\n", appname, SRCVERSION); } /* * pmempool_sync_help -- print help message for the sync command */ void pmempool_sync_help(const char *appname) { print_usage(appname); print_version(appname); printf(help_str, appname); } /* * pmempool_sync_parse_args -- (internal) parse command line arguments */ static int pmempool_sync_parse_args(struct pmempool_sync_context *ctx, const char *appname, int argc, char *argv[]) { int opt; while ((opt = getopt_long(argc, argv, "bdhv", long_options, NULL)) != -1) { switch (opt) { case 'd': ctx->flags |= PMEMPOOL_SYNC_DRY_RUN; break; case 'b': ctx->flags |= PMEMPOOL_SYNC_FIX_BAD_BLOCKS; break; case 'h': pmempool_sync_help(appname); exit(EXIT_SUCCESS); case 'v': out_set_vlevel(1); break; default: print_usage(appname); exit(EXIT_FAILURE); } } if (optind < argc) { ctx->poolset_file = argv[optind]; } else { print_usage(appname); exit(EXIT_FAILURE); } return 0; } /* * pmempool_sync_func -- main function for the sync command */ int pmempool_sync_func(const char *appname, int argc, char *argv[]) { int ret = 0; struct pmempool_sync_context ctx = pmempool_sync_default; /* parse command line arguments */ if ((ret = pmempool_sync_parse_args(&ctx, appname, argc, argv))) return ret; ret = pmempool_sync(ctx.poolset_file, ctx.flags); if (ret) { outv_err("failed to synchronize: %s\n", pmempool_errormsg()); if (errno) outv_err("%s\n", strerror(errno)); return -1; } else { outv(1, "%s: synchronized\n", ctx.poolset_file); return 0; } } pmdk-1.13.1/src/tools/pmempool/pmempool.rc0000664000000000000000000000721214435627501017176 0ustar rootroot/* * Copyright 2016, Intel Corporation * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * * Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * * Neither the name of the copyright holder nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ /* * pmempool.rc -- pmempool resource file */ #include <windows.h> #define FILE_NAME "pmempool.exe" #define DESCRIPTION "pmempool - Persistent Memory Pool Management Tool" #define TYPE VFT_APP MAINICON ICON "../../../res/PMDK.ico" #include <common.rc>pmdk-1.13.1/src/tools/pmempool/transform.c0000664000000000000000000000715114435627501017201 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016-2018, Intel Corporation */ /* * transform.c -- pmempool transform command source file */ #include #include #include #include #include #include #include #include #include #include "common.h" #include "output.h" #include "transform.h" #include "libpmempool.h" /* * pmempool_transform_context -- context and arguments for transform command */ struct pmempool_transform_context { unsigned flags; /* flags which modify the command execution */ char *poolset_file_src; /* a path to a source poolset file */ char *poolset_file_dst; /* a path to a target poolset file */ }; /* * pmempool_transform_default -- default arguments for transform command */ static const struct pmempool_transform_context pmempool_transform_default = { .flags = 0, .poolset_file_src = NULL, .poolset_file_dst = NULL, }; /* * help_str -- string for help message */ static const char * const help_str = "Modify internal structure of a poolset\n" "\n" "Common options:\n" " -d, --dry-run do not apply changes, only check for viability of" " transformation\n" " -v, --verbose increase verbosity level\n" " -h, --help display this help and exit\n" "\n" "For complete documentation see %s-transform(1) manual page.\n" ; /* * long_options -- command line options */ static const struct option long_options[] = { {"dry-run", no_argument, NULL, 'd'}, {"help", no_argument, NULL, 'h'}, {"verbose", no_argument, NULL, 'v'}, {NULL, 0, NULL, 0 }, }; /* * print_usage -- print application usage short description */ static void print_usage(const char *appname) { printf("usage: %s transform [] " " \n", appname); } /* * print_version -- print version string */ static void print_version(const char *appname) { printf("%s %s\n", appname, SRCVERSION); } /* * pmempool_transform_help -- print help message for the transform command */ void pmempool_transform_help(const char *appname) { print_usage(appname); print_version(appname); printf(help_str, appname); } /* * pmempool_check_parse_args -- parse command line arguments */ static int pmempool_transform_parse_args(struct pmempool_transform_context *ctx, const char *appname, int argc, char *argv[]) { int opt; while ((opt = getopt_long(argc, argv, "dhv", long_options, NULL)) != -1) { switch (opt) { case 'd': ctx->flags = PMEMPOOL_TRANSFORM_DRY_RUN; break; case 'h': pmempool_transform_help(appname); exit(EXIT_SUCCESS); case 'v': out_set_vlevel(1); break; default: print_usage(appname); exit(EXIT_FAILURE); } } if (optind + 1 < argc) { ctx->poolset_file_src = argv[optind]; ctx->poolset_file_dst = argv[optind + 1]; } else { print_usage(appname); exit(EXIT_FAILURE); } return 0; } /* * pmempool_transform_func -- main function for the transform command */ int pmempool_transform_func(const char *appname, int argc, char *argv[]) { int ret; struct pmempool_transform_context ctx = pmempool_transform_default; /* parse command line arguments */ if ((ret = pmempool_transform_parse_args(&ctx, appname, argc, argv))) return ret; ret = pmempool_transform(ctx.poolset_file_src, ctx.poolset_file_dst, ctx.flags); if (ret) { if (errno) outv_err("%s\n", strerror(errno)); outv_err("failed to transform %s -> %s: %s\n", ctx.poolset_file_src, ctx.poolset_file_dst, pmempool_errormsg()); return -1; } else { outv(1, "%s -> %s: transformed\n", ctx.poolset_file_src, ctx.poolset_file_dst); return 0; } } pmdk-1.13.1/src/tools/pmempool/info.c0000664000000000000000000006346214435627501016130 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2023, Intel Corporation */ /* * info.c -- pmempool info command main source file */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include "common.h" #include "output.h" #include "out.h" #include "info.h" #include "set.h" #include "file.h" #include "badblocks.h" #include "set_badblocks.h" #define DEFAULT_CHUNK_TYPES\ ((1<> (ALIGNMENT_DESC_BITS * (x))) & ((1 << ALIGNMENT_DESC_BITS) - 1))) #define UNDEF_REPLICA UINT_MAX #define UNDEF_PART UINT_MAX /* * Default arguments */ static const struct pmempool_info_args pmempool_info_args_default = { /* * Picked experimentally based on used fields names. * This should be at least the number of characters of * the longest field name. */ .col_width = 24, .human = false, .force = false, .badblocks = PRINT_BAD_BLOCKS_NOT_SET, .type = PMEM_POOL_TYPE_UNKNOWN, .vlevel = VERBOSE_DEFAULT, .vdata = VERBOSE_SILENT, .vhdrdump = VERBOSE_SILENT, .vstats = VERBOSE_SILENT, .log = { /* deprecated */ .walk = 0, }, .blk = { /* deprecated */ .vmap = VERBOSE_SILENT, .vflog = VERBOSE_SILENT, .vbackup = VERBOSE_SILENT, .skip_zeros = false, .skip_error = false, .skip_no_flag = false, }, .obj = { .vlanes = VERBOSE_SILENT, .vroot = VERBOSE_SILENT, .vobjects = VERBOSE_SILENT, .valloc = VERBOSE_SILENT, .voobhdr = VERBOSE_SILENT, .vheap = VERBOSE_SILENT, .vzonehdr = VERBOSE_SILENT, .vchunkhdr = VERBOSE_SILENT, .vbitmap = VERBOSE_SILENT, .lanes_recovery = false, .ignore_empty_obj = false, .chunk_types = DEFAULT_CHUNK_TYPES, .replica = 0, }, }; /* * long-options -- structure holding long options. */ static const struct option long_options[] = { {"version", no_argument, NULL, 'V' | OPT_ALL}, {"verbose", no_argument, NULL, 'v' | OPT_ALL}, {"help", no_argument, NULL, 'h' | OPT_ALL}, {"human", no_argument, NULL, 'n' | OPT_ALL}, {"force", required_argument, NULL, 'f' | OPT_ALL}, {"data", no_argument, NULL, 'd' | OPT_ALL}, {"headers-hex", no_argument, NULL, 'x' | OPT_ALL}, {"stats", no_argument, NULL, 's' | OPT_ALL}, {"range", required_argument, NULL, 'r' | OPT_ALL}, {"bad-blocks", required_argument, NULL, 'k' | OPT_ALL}, {"walk", required_argument, NULL, 'w' | OPT_LOG}, {"skip-zeros", no_argument, NULL, 'z' | OPT_BLK | OPT_BTT}, {"skip-error", no_argument, NULL, 'e' | OPT_BLK | OPT_BTT}, {"skip-no-flag", no_argument, NULL, 'u' | OPT_BLK | OPT_BTT}, {"map", no_argument, NULL, 'm' | OPT_BLK | OPT_BTT}, {"flog", no_argument, NULL, 'g' | OPT_BLK | OPT_BTT}, {"backup", no_argument, NULL, 'B' | OPT_BLK | OPT_BTT}, {"lanes", no_argument, NULL, 'l' | OPT_OBJ}, {"recovery", no_argument, NULL, 'R' | OPT_OBJ}, {"section", required_argument, NULL, 'S' | OPT_OBJ}, {"object-store", no_argument, NULL, 'O' | OPT_OBJ}, {"types", required_argument, NULL, 't' | OPT_OBJ}, {"no-empty", no_argument, NULL, 'E' | OPT_OBJ}, {"alloc-header", no_argument, NULL, 'A' | OPT_OBJ}, {"oob-header", no_argument, NULL, 'a' | OPT_OBJ}, {"root", no_argument, NULL, 'o' | OPT_OBJ}, {"heap", no_argument, NULL, 'H' | OPT_OBJ}, {"zones", no_argument, NULL, 'Z' | OPT_OBJ}, {"chunks", no_argument, NULL, 'C' | OPT_OBJ}, {"chunk-type", required_argument, NULL, 'T' | OPT_OBJ}, {"bitmap", no_argument, NULL, 'b' | OPT_OBJ}, {"replica", required_argument, NULL, 'p' | OPT_OBJ}, {NULL, 0, NULL, 0 }, }; static const struct option_requirement option_requirements[] = { { .opt = 'r', .type = PMEM_POOL_TYPE_LOG, .req = OPT_REQ0('d') }, { .opt = 'r', .type = PMEM_POOL_TYPE_BLK | PMEM_POOL_TYPE_BTT, .req = OPT_REQ0('d') | OPT_REQ1('m') }, { .opt = 'z', .type = PMEM_POOL_TYPE_BLK | PMEM_POOL_TYPE_BTT, .req = OPT_REQ0('d') | OPT_REQ1('m') }, { .opt = 'e', .type = PMEM_POOL_TYPE_BLK | PMEM_POOL_TYPE_BTT, .req = OPT_REQ0('d') | OPT_REQ1('m') }, { .opt = 'u', .type = PMEM_POOL_TYPE_BLK | PMEM_POOL_TYPE_BTT, .req = OPT_REQ0('d') | OPT_REQ1('m') }, { .opt = 'r', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('O') | OPT_REQ1('Z') | OPT_REQ2('C') | OPT_REQ3('l'), }, { .opt = 'R', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('l') }, { .opt = 'S', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('l') }, { .opt = 'E', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('O') }, { .opt = 'T', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('C') }, { .opt = 'b', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('H') }, { .opt = 'b', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('C') }, { .opt = 'A', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('O') | OPT_REQ1('l') | OPT_REQ2('o') }, { .opt = 'a', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('O') | OPT_REQ1('l') | OPT_REQ2('o') }, { .opt = 't', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('O') | OPT_REQ1('s'), }, { .opt = 'C', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('O') | OPT_REQ1('H') | OPT_REQ2('s'), }, { .opt = 'Z', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('O') | OPT_REQ1('H') | OPT_REQ2('s'), }, { .opt = 'd', .type = PMEM_POOL_TYPE_OBJ, .req = OPT_REQ0('O') | OPT_REQ1('o'), }, { 0, 0, 0} }; /* * help_str -- string for help message */ static const char * const help_str = "Show information about pmem pool from specified file.\n" "NOTE: pmem blk and log pools are deprecated\n" "\n" "Common options:\n" " -h, --help Print this help and exit.\n" " -V, --version Print version and exit.\n" " -v, --verbose Increase verbisity level.\n" " -f, --force blk|log|obj|btt Force parsing a pool of specified type.\n" " -n, --human Print sizes in human readable format.\n" " -x, --headers-hex Hexdump all headers.\n" " -d, --data Dump log data and blocks.\n" " -s, --stats Print statistics.\n" " -r, --range Range of blocks/chunks/objects.\n" " -k, --bad-blocks= Print bad blocks.\n" "\n" "Options for PMEMLOG: (DEPRECATED)\n" " -w, --walk Chunk size.\n" "\n" "Options for PMEMBLK: (DEPRECATED)\n" " -m, --map Print BTT Map entries.\n" " -g, --flog Print BTT FLOG entries.\n" " -B, --backup Print BTT Info header backup.\n" " -z, --skip-zeros Skip blocks marked with zero flag.\n" " -e, --skip-error Skip blocks marked with error flag.\n" " -u, --skip-no-flag Skip blocks not marked with any flag.\n" "\n" "Options for PMEMOBJ:\n" " -l, --lanes [] Print lanes from specified range.\n" " -R, --recovery Print only lanes which need recovery.\n" " -S, --section tx,allocator,list Print only specified sections.\n" " -O, --object-store Print object store.\n" " -t, --types Specify objects' type numbers range.\n" " -E, --no-empty Print only non-empty object store lists.\n" " -o, --root Print root object information\n" " -A, --alloc-header Print allocation header for objects in\n" " object store.\n" " -a, --oob-header Print OOB header\n" " -H, --heap Print heap header.\n" " -Z, --zones [] Print zones header. If range is specified\n" " and --object|-O option is specified prints\n" " objects from specified zones only.\n" " -C, --chunks [] Print zones header. If range is specified\n" " and --object|-O option is specified prints\n" " objects from specified zones only.\n" " -T, --chunk-type used,free,run,footer\n" " Print only specified type(s) of chunk.\n" " [requires --chunks|-C]\n" " -b, --bitmap Print chunk run's bitmap in graphical\n" " format. [requires --chunks|-C]\n" " -p, --replica Print info from specified replica\n" "For complete documentation see %s-info(1) manual page.\n" ; /* * print_usage -- print application usage short description */ static void print_usage(const char *appname) { printf("Usage: %s info [] \n", appname); } /* * print_version -- print version string */ static void print_version(const char *appname) { printf("NOTE: pmem blk and log pools are deprecated\n"); printf("%s %s\n", appname, SRCVERSION); } /* * pmempool_info_help -- print application usage detailed description */ void pmempool_info_help(const char *appname) { print_usage(appname); print_version(appname); printf(help_str, appname); } /* * parse_args -- parse command line arguments * * Parse command line arguments and store them in pmempool_info_args * structure. * Terminates process if invalid arguments passed. */ static int parse_args(const char *appname, int argc, char *argv[], struct pmempool_info_args *argsp, struct options *opts) { int opt; if (argc == 1) { print_usage(appname); return -1; } struct ranges *rangesp = &argsp->ranges; while ((opt = util_options_getopt(argc, argv, "vhnf:ezuF:L:c:dmxVw:gBsr:lRS:OECZHT:bot:aAp:k:", opts)) != -1) { switch (opt) { case 'v': argsp->vlevel = VERBOSE_MAX; break; case 'V': print_version(appname); exit(EXIT_SUCCESS); case 'h': pmempool_info_help(appname); exit(EXIT_SUCCESS); case 'n': argsp->human = true; break; case 'f': argsp->type = pmem_pool_type_parse_str(optarg); if (argsp->type == PMEM_POOL_TYPE_UNKNOWN) { outv_err("'%s' -- unknown pool type\n", optarg); return -1; } argsp->force = true; break; case 'k': if (strcmp(optarg, "no") == 0) { argsp->badblocks = PRINT_BAD_BLOCKS_NO; } else if (strcmp(optarg, "yes") == 0) { argsp->badblocks = PRINT_BAD_BLOCKS_YES; } else { outv_err( "'%s' -- invalid argument of the '-k/--bad-blocks' option\n", optarg); return -1; } break; case 'e': argsp->blk.skip_error = true; break; case 'z': argsp->blk.skip_zeros = true; break; case 'u': argsp->blk.skip_no_flag = true; break; case 'r': if (util_parse_ranges(optarg, rangesp, ENTIRE_UINT64)) { outv_err("'%s' -- cannot parse range(s)\n", optarg); return -1; } if (rangesp == &argsp->ranges) argsp->use_range = 1; break; case 'd': argsp->vdata = VERBOSE_DEFAULT; break; case 'm': argsp->blk.vmap = VERBOSE_DEFAULT; break; case 'g': argsp->blk.vflog = VERBOSE_DEFAULT; break; case 'B': argsp->blk.vbackup = VERBOSE_DEFAULT; break; case 'x': argsp->vhdrdump = VERBOSE_DEFAULT; break; case 's': argsp->vstats = VERBOSE_DEFAULT; break; case 'w': argsp->log.walk = (size_t)atoll(optarg); if (argsp->log.walk == 0) { outv_err("'%s' -- invalid chunk size\n", optarg); return -1; } break; case 'l': argsp->obj.vlanes = VERBOSE_DEFAULT; rangesp = &argsp->obj.lane_ranges; break; case 'R': argsp->obj.lanes_recovery = true; break; case 'O': argsp->obj.vobjects = VERBOSE_DEFAULT; rangesp = &argsp->ranges; break; case 'a': argsp->obj.voobhdr = VERBOSE_DEFAULT; break; case 'A': argsp->obj.valloc = VERBOSE_DEFAULT; break; case 'E': argsp->obj.ignore_empty_obj = true; break; case 'Z': argsp->obj.vzonehdr = VERBOSE_DEFAULT; rangesp = &argsp->obj.zone_ranges; break; case 'C': argsp->obj.vchunkhdr = VERBOSE_DEFAULT; rangesp = &argsp->obj.chunk_ranges; break; case 'H': argsp->obj.vheap = VERBOSE_DEFAULT; break; case 'T': argsp->obj.chunk_types = 0; if (util_parse_chunk_types(optarg, &argsp->obj.chunk_types) || (argsp->obj.chunk_types & (1 << CHUNK_TYPE_UNKNOWN))) { outv_err("'%s' -- cannot parse chunk type(s)\n", optarg); return -1; } break; case 'o': argsp->obj.vroot = VERBOSE_DEFAULT; break; case 't': if (util_parse_ranges(optarg, &argsp->obj.type_ranges, ENTIRE_UINT64)) { outv_err("'%s' -- cannot parse range(s)\n", optarg); return -1; } break; case 'b': argsp->obj.vbitmap = VERBOSE_DEFAULT; break; case 'p': { char *endptr; int olderrno = errno; errno = 0; long long ll = strtoll(optarg, &endptr, 10); if ((endptr && *endptr != '\0') || errno) { outv_err("'%s' -- invalid replica number", optarg); return -1; } errno = olderrno; argsp->obj.replica = (size_t)ll; break; } default: print_usage(appname); return -1; } } if (optind < argc) { argsp->file = argv[optind]; } else { print_usage(appname); return -1; } if (!argsp->use_range) util_ranges_add(&argsp->ranges, ENTIRE_UINT64); if (util_ranges_empty(&argsp->obj.type_ranges)) util_ranges_add(&argsp->obj.type_ranges, ENTIRE_UINT64); if (util_ranges_empty(&argsp->obj.lane_ranges)) util_ranges_add(&argsp->obj.lane_ranges, ENTIRE_UINT64); if (util_ranges_empty(&argsp->obj.zone_ranges)) util_ranges_add(&argsp->obj.zone_ranges, ENTIRE_UINT64); if (util_ranges_empty(&argsp->obj.chunk_ranges)) util_ranges_add(&argsp->obj.chunk_ranges, ENTIRE_UINT64); return 0; } /* * pmempool_info_read -- read data from file */ int pmempool_info_read(struct pmem_info *pip, void *buff, size_t nbytes, uint64_t off) { return pool_set_file_read(pip->pfile, buff, nbytes, off); } /* * pmempool_info_badblocks -- (internal) prints info about file badblocks */ static int pmempool_info_badblocks(struct pmem_info *pip, const char *file_name, int v) { int ret; if (pip->args.badblocks != PRINT_BAD_BLOCKS_YES) return 0; struct badblocks *bbs = badblocks_new(); if (bbs == NULL) return -1; ret = badblocks_get(file_name, bbs); if (ret) { if (errno == ENOTSUP) { outv(v, BB_NOT_SUPP "\n"); ret = -1; goto exit_free; } outv_err("checking bad blocks failed -- '%s'", file_name); goto exit_free; } if (bbs->bb_cnt == 0 || bbs->bbv == NULL) goto exit_free; outv(v, "bad blocks:\n"); outv(v, "\toffset\t\tlength\n"); unsigned b; for (b = 0; b < bbs->bb_cnt; b++) { outv(v, "\t%zu\t\t%zu\n", B2SEC(bbs->bbv[b].offset), B2SEC(bbs->bbv[b].length)); } exit_free: badblocks_delete(bbs); return ret; } /* * pmempool_info_part -- (internal) print info about poolset part */ static int pmempool_info_part(struct pmem_info *pip, unsigned repn, unsigned partn, int v) { /* get path of the part file */ const char *path = NULL; if (repn != UNDEF_REPLICA && partn != UNDEF_PART) { outv(v, "part %u:\n", partn); struct pool_set_part *part = &pip->pfile->poolset->replica[repn]->part[partn]; path = part->path; } else { outv(v, "Part file:\n"); path = pip->file_name; } outv_field(v, "path", "%s", path); enum file_type type = util_file_get_type(path); if (type < 0) return -1; const char *type_str = type == TYPE_DEVDAX ? "device dax" : "regular file"; outv_field(v, "type", "%s", type_str); /* get size of the part file */ ssize_t size = util_file_get_size(path); if (size < 0) { outv_err("couldn't get size of %s", path); return -1; } outv_field(v, "size", "%s", out_get_size_str((size_t)size, pip->args.human)); /* get alignment of device dax */ if (type == TYPE_DEVDAX) { size_t alignment = util_file_device_dax_alignment(path); outv_field(v, "alignment", "%s", out_get_size_str(alignment, pip->args.human)); } /* look for bad blocks */ if (pmempool_info_badblocks(pip, path, VERBOSE_DEFAULT)) { outv_err("Unable to retrieve badblock info"); return -1; } return 0; } /* * pmempool_info_directory -- (internal) print information about directory */ static void pmempool_info_directory(struct pool_set_directory *d, int v) { outv(v, "Directory %s:\n", d->path); outv_field(v, "reservation size", "%lu", d->resvsize); } /* * pmempool_info_replica -- (internal) print info about replica */ static int pmempool_info_replica(struct pmem_info *pip, unsigned repn, int v) { struct pool_replica *rep = pip->pfile->poolset->replica[repn]; outv(v, "Replica %u%s - local", repn, repn == 0 ? " (master)" : ""); outv(v, ", %u part(s):\n", rep->nparts); for (unsigned p = 0; p < rep->nparts; ++p) { if (pmempool_info_part(pip, repn, p, v)) return -1; } if (pip->pfile->poolset->directory_based) { size_t nd = VEC_SIZE(&rep->directory); outv(v, "%lu %s:\n", nd, nd == 1 ? "Directory" : "Directories"); struct pool_set_directory *d; VEC_FOREACH_BY_PTR(d, &rep->directory) { pmempool_info_directory(d, v); } } return 0; } /* * pmempool_info_poolset -- (internal) print info about poolset structure */ static int pmempool_info_poolset(struct pmem_info *pip, int v) { ASSERTeq(pip->params.is_poolset, 1); if (pip->pfile->poolset->directory_based) outv(v, "Directory-based Poolset structure:\n"); else outv(v, "Poolset structure:\n"); outv_field(v, "Number of replicas", "%u", pip->pfile->poolset->nreplicas); for (unsigned r = 0; r < pip->pfile->poolset->nreplicas; ++r) { if (pmempool_info_replica(pip, r, v)) return -1; } if (pip->pfile->poolset->options > 0) { outv_title(v, "Poolset options"); if (pip->pfile->poolset->options & OPTION_SINGLEHDR) outv(v, "%s", "SINGLEHDR\n"); } return 0; } /* * pmempool_info_pool_hdr -- (internal) print pool header information */ static int pmempool_info_pool_hdr(struct pmem_info *pip, int v) { static const char *alignment_desc_str[] = { " char", " short", " int", " long", " long long", " size_t", " os_off_t", " float", " double", " long double", " void *", }; static const size_t alignment_desc_n = sizeof(alignment_desc_str) / sizeof(alignment_desc_str[0]); int ret = 0; struct pool_hdr *hdr = malloc(sizeof(struct pool_hdr)); if (!hdr) err(1, "Cannot allocate memory for pool_hdr"); if (pmempool_info_read(pip, hdr, sizeof(*hdr), 0)) { outv_err("cannot read pool header\n"); free(hdr); return -1; } struct arch_flags arch_flags; util_get_arch_flags(&arch_flags); outv_title(v, "POOL Header"); outv_hexdump(pip->args.vhdrdump, hdr, sizeof(*hdr), 0, 1); util_convert2h_hdr_nocheck(hdr); outv_field(v, "Signature", "%.*s%s", POOL_HDR_SIG_LEN, hdr->signature, pip->params.is_part ? " [part file]" : ""); outv_field(v, "Major", "%d", hdr->major); outv_field(v, "Mandatory features", "%s", out_get_incompat_features_str(hdr->features.incompat)); outv_field(v, "Not mandatory features", "0x%x", hdr->features.compat); outv_field(v, "Forced RO", "0x%x", hdr->features.ro_compat); outv_field(v, "Pool set UUID", "%s", out_get_uuid_str(hdr->poolset_uuid)); outv_field(v, "UUID", "%s", out_get_uuid_str(hdr->uuid)); outv_field(v, "Previous part UUID", "%s", out_get_uuid_str(hdr->prev_part_uuid)); outv_field(v, "Next part UUID", "%s", out_get_uuid_str(hdr->next_part_uuid)); outv_field(v, "Previous replica UUID", "%s", out_get_uuid_str(hdr->prev_repl_uuid)); outv_field(v, "Next replica UUID", "%s", out_get_uuid_str(hdr->next_repl_uuid)); outv_field(v, "Creation Time", "%s", out_get_time_str((time_t)hdr->crtime)); uint64_t ad = hdr->arch_flags.alignment_desc; uint64_t cur_ad = arch_flags.alignment_desc; outv_field(v, "Alignment Descriptor", "%s", out_get_alignment_desc_str(ad, cur_ad)); for (size_t i = 0; i < alignment_desc_n; i++) { uint64_t a = GET_ALIGNMENT(ad, i); if (ad == cur_ad) { outv_field(v + 1, alignment_desc_str[i], "%2lu", a); } else { uint64_t av = GET_ALIGNMENT(cur_ad, i); if (a == av) { outv_field(v + 1, alignment_desc_str[i], "%2lu [OK]", a); } else { outv_field(v + 1, alignment_desc_str[i], "%2lu [wrong! should be %2lu]", a, av); } } } outv_field(v, "Class", "%s", out_get_arch_machine_class_str( hdr->arch_flags.machine_class)); outv_field(v, "Data", "%s", out_get_arch_data_str(hdr->arch_flags.data)); outv_field(v, "Machine", "%s", out_get_arch_machine_str(hdr->arch_flags.machine)); outv_field(v, "Last shutdown", "%s", out_get_last_shutdown_str(hdr->sds.dirty)); outv_field(v, "Checksum", "%s", out_get_checksum(hdr, sizeof(*hdr), &hdr->checksum, POOL_HDR_CSUM_END_OFF(hdr))); free(hdr); return ret; } /* * pmempool_info_file -- print info about single file */ static int pmempool_info_file(struct pmem_info *pip, const char *file_name) { int ret = 0; pip->file_name = file_name; /* * If force flag is set 'types' fields _must_ hold * single pool type - this is validated when processing * command line arguments. */ if (pip->args.force) { pip->type = pip->args.type; } else { if (pmem_pool_parse_params(file_name, &pip->params, 1)) { if (errno) perror(file_name); else outv_err("%s: cannot determine type of pool\n", file_name); return -1; } pip->type = pip->params.type; } if (PMEM_POOL_TYPE_UNKNOWN == pip->type) { outv_err("%s: unknown pool type -- '%s'\n", file_name, pip->params.signature); return -1; } else if (!pip->args.force && !pip->params.is_checksum_ok) { outv_err("%s: invalid checksum\n", file_name); return -1; } else { if (util_options_verify(pip->opts, pip->type)) return -1; pip->pfile = pool_set_file_open(file_name, 0, !pip->args.force); if (!pip->pfile) { perror(file_name); return -1; } /* check if we should check and print bad blocks */ if (pip->args.badblocks == PRINT_BAD_BLOCKS_NOT_SET) { struct pool_hdr hdr; if (pmempool_info_read(pip, &hdr, sizeof(hdr), 0)) { outv_err("cannot read pool header\n"); goto out_close; } util_convert2h_hdr_nocheck(&hdr); if (hdr.features.compat & POOL_FEAT_CHECK_BAD_BLOCKS) pip->args.badblocks = PRINT_BAD_BLOCKS_YES; else pip->args.badblocks = PRINT_BAD_BLOCKS_NO; } if (pip->type != PMEM_POOL_TYPE_BTT) { struct pool_set *ps = pip->pfile->poolset; for (unsigned r = 0; r < ps->nreplicas; ++r) { if (mprotect(ps->replica[r]->part[0].addr, ps->replica[r]->repsize, PROT_READ) < 0) { outv_err( "%s: failed to change pool protection", pip->pfile->fname); ret = -1; goto out_close; } } } if (pip->args.obj.replica) { size_t nreplicas = pool_set_file_nreplicas(pip->pfile); if (nreplicas == 1) { outv_err("only master replica available"); ret = -1; goto out_close; } if (pip->args.obj.replica >= nreplicas) { outv_err("replica number out of range" " (valid range is: 0-%" PRIu64 ")", nreplicas - 1); ret = -1; goto out_close; } if (pool_set_file_set_replica(pip->pfile, pip->args.obj.replica)) { outv_err("setting replica number failed"); ret = -1; goto out_close; } } /* hdr info is not present in btt device */ if (pip->type != PMEM_POOL_TYPE_BTT) { if (pip->params.is_poolset && pmempool_info_poolset(pip, VERBOSE_DEFAULT)) { ret = -1; goto out_close; } if (!pip->params.is_poolset && pmempool_info_part(pip, UNDEF_REPLICA, UNDEF_PART, VERBOSE_DEFAULT)) { ret = -1; goto out_close; } if (pmempool_info_pool_hdr(pip, VERBOSE_DEFAULT)) { ret = -1; goto out_close; } } if (pip->params.is_part) { ret = 0; goto out_close; } switch (pip->type) { case PMEM_POOL_TYPE_LOG: ret = pmempool_info_log(pip); break; case PMEM_POOL_TYPE_BLK: ret = pmempool_info_blk(pip); break; case PMEM_POOL_TYPE_OBJ: ret = pmempool_info_obj(pip); break; case PMEM_POOL_TYPE_BTT: ret = pmempool_info_btt(pip); break; case PMEM_POOL_TYPE_UNKNOWN: default: ret = -1; break; } out_close: pool_set_file_close(pip->pfile); } return ret; } /* * pmempool_info_alloc -- allocate pmem info context */ static struct pmem_info * pmempool_info_alloc(void) { struct pmem_info *pip = malloc(sizeof(struct pmem_info)); if (!pip) err(1, "Cannot allocate memory for pmempool info context"); if (pip) { memset(pip, 0, sizeof(*pip)); /* set default command line parameters */ memcpy(&pip->args, &pmempool_info_args_default, sizeof(pip->args)); pip->opts = util_options_alloc(long_options, sizeof(long_options) / sizeof(long_options[0]), option_requirements); PMDK_LIST_INIT(&pip->args.ranges.head); PMDK_LIST_INIT(&pip->args.obj.type_ranges.head); PMDK_LIST_INIT(&pip->args.obj.lane_ranges.head); PMDK_LIST_INIT(&pip->args.obj.zone_ranges.head); PMDK_LIST_INIT(&pip->args.obj.chunk_ranges.head); PMDK_TAILQ_INIT(&pip->obj.stats.type_stats); } return pip; } /* * pmempool_info_free -- free pmem info context */ static void pmempool_info_free(struct pmem_info *pip) { if (pip->obj.stats.zone_stats) { for (uint64_t i = 0; i < pip->obj.stats.n_zones; ++i) VEC_DELETE(&pip->obj.stats.zone_stats[i].class_stats); free(pip->obj.stats.zone_stats); } util_options_free(pip->opts); util_ranges_clear(&pip->args.ranges); util_ranges_clear(&pip->args.obj.type_ranges); util_ranges_clear(&pip->args.obj.zone_ranges); util_ranges_clear(&pip->args.obj.chunk_ranges); util_ranges_clear(&pip->args.obj.lane_ranges); while (!PMDK_TAILQ_EMPTY(&pip->obj.stats.type_stats)) { struct pmem_obj_type_stats *type = PMDK_TAILQ_FIRST(&pip->obj.stats.type_stats); PMDK_TAILQ_REMOVE(&pip->obj.stats.type_stats, type, next); free(type); } free(pip); } int pmempool_info_func(const char *appname, int argc, char *argv[]) { int ret = 0; struct pmem_info *pip = pmempool_info_alloc(); /* read command line arguments */ if ((ret = parse_args(appname, argc, argv, &pip->args, pip->opts)) == 0) { /* set some output format values */ out_set_vlevel(pip->args.vlevel); out_set_col_width(pip->args.col_width); ret = pmempool_info_file(pip, pip->args.file); } pmempool_info_free(pip); return ret; } pmdk-1.13.1/src/tools/pmempool/check.c0000664000000000000000000001577314435627501016254 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2019, Intel Corporation */ /* * check.c -- pmempool check command source file */ #include #include #include "common.h" #include "check.h" #include "output.h" #include "set.h" #include "file.h" #include "libpmempool.h" typedef enum { CHECK_RESULT_CONSISTENT, CHECK_RESULT_NOT_CONSISTENT, CHECK_RESULT_REPAIRED, CHECK_RESULT_CANNOT_REPAIR, CHECK_RESULT_SYNC_REQ, CHECK_RESULT_ERROR } check_result_t; /* * pmempool_check_context -- context and arguments for check command */ struct pmempool_check_context { int verbose; /* verbosity level */ char *fname; /* file name */ struct pool_set_file *pfile; bool repair; /* do repair */ bool backup; /* do backup */ bool advanced; /* do advanced repairs */ char *backup_fname; /* backup file name */ bool exec; /* do execute */ char ans; /* default answer on all questions or '?' */ }; /* * pmempool_check_default -- default arguments for check command */ static const struct pmempool_check_context pmempool_check_default = { .verbose = 1, .fname = NULL, .repair = false, .backup = false, .backup_fname = NULL, .advanced = false, .exec = true, .ans = '?', }; /* * help_str -- string for help message */ static const char * const help_str = "Check consistency of a pool\n" "\n" "Common options:\n" " -r, --repair try to repair a pool file if possible\n" " -y, --yes answer yes to all questions\n" " -d, --dry-run don't execute, just show what would be done\n" " -b, --backup create backup of a pool file before executing\n" " -a, --advanced perform advanced repairs\n" " -q, --quiet be quiet and don't print any messages\n" " -v, --verbose increase verbosity level\n" " -h, --help display this help and exit\n" "\n" "For complete documentation see %s-check(1) manual page.\n" ; /* * long_options -- command line options */ static const struct option long_options[] = { {"repair", no_argument, NULL, 'r'}, {"yes", no_argument, NULL, 'y'}, {"dry-run", no_argument, NULL, 'd'}, {"no-exec", no_argument, NULL, 'N'}, /* deprecated */ {"backup", required_argument, NULL, 'b'}, {"advanced", no_argument, NULL, 'a'}, {"quiet", no_argument, NULL, 'q'}, {"verbose", no_argument, NULL, 'v'}, {"help", no_argument, NULL, 'h'}, {NULL, 0, NULL, 0 }, }; /* * print_usage -- print short description of application's usage */ static void print_usage(const char *appname) { printf("Usage: %s check [] \n", appname); } /* * print_version -- print version string */ static void print_version(const char *appname) { printf("%s %s\n", appname, SRCVERSION); } /* * pmempool_check_help -- print help message for check command */ void pmempool_check_help(const char *appname) { print_usage(appname); print_version(appname); printf(help_str, appname); } /* * pmempool_check_parse_args -- parse command line arguments */ static int pmempool_check_parse_args(struct pmempool_check_context *pcp, const char *appname, int argc, char *argv[]) { int opt; while ((opt = getopt_long(argc, argv, "ahvrdNb:qy", long_options, NULL)) != -1) { switch (opt) { case 'r': pcp->repair = true; break; case 'y': pcp->ans = 'y'; break; case 'd': case 'N': pcp->exec = false; break; case 'b': pcp->backup = true; pcp->backup_fname = optarg; break; case 'a': pcp->advanced = true; break; case 'q': pcp->verbose = 0; break; case 'v': pcp->verbose = 2; break; case 'h': pmempool_check_help(appname); exit(EXIT_SUCCESS); default: print_usage(appname); exit(EXIT_FAILURE); } } if (optind < argc) { pcp->fname = argv[optind]; } else { print_usage(appname); exit(EXIT_FAILURE); } if (!pcp->repair && !pcp->exec) { outv_err("'-N' option requires '-r'\n"); exit(EXIT_FAILURE); } if (!pcp->repair && pcp->backup) { outv_err("'-b' option requires '-r'\n"); exit(EXIT_FAILURE); } return 0; } static check_result_t pmempool_check_2_check_res_t[] = { [PMEMPOOL_CHECK_RESULT_CONSISTENT] = CHECK_RESULT_CONSISTENT, [PMEMPOOL_CHECK_RESULT_NOT_CONSISTENT] = CHECK_RESULT_NOT_CONSISTENT, [PMEMPOOL_CHECK_RESULT_REPAIRED] = CHECK_RESULT_REPAIRED, [PMEMPOOL_CHECK_RESULT_CANNOT_REPAIR] = CHECK_RESULT_CANNOT_REPAIR, [PMEMPOOL_CHECK_RESULT_SYNC_REQ] = CHECK_RESULT_SYNC_REQ, [PMEMPOOL_CHECK_RESULT_ERROR] = CHECK_RESULT_ERROR, }; static const char * check_ask(const char *msg) { char answer = ask_Yn('?', "%s", msg); switch (answer) { case 'y': return "yes"; case 'n': return "no"; default: return "?"; } } static check_result_t pmempool_check_perform(struct pmempool_check_context *pc) { struct pmempool_check_args args = { .path = pc->fname, .backup_path = pc->backup_fname, .pool_type = PMEMPOOL_POOL_TYPE_DETECT, .flags = PMEMPOOL_CHECK_FORMAT_STR }; if (pc->repair) args.flags |= PMEMPOOL_CHECK_REPAIR; if (!pc->exec) args.flags |= PMEMPOOL_CHECK_DRY_RUN; if (pc->advanced) args.flags |= PMEMPOOL_CHECK_ADVANCED; if (pc->ans == 'y') args.flags |= PMEMPOOL_CHECK_ALWAYS_YES; if (pc->verbose == 2) args.flags |= PMEMPOOL_CHECK_VERBOSE; PMEMpoolcheck *ppc = pmempool_check_init(&args, sizeof(args)); if (ppc == NULL) return CHECK_RESULT_ERROR; struct pmempool_check_status *status = NULL; while ((status = pmempool_check(ppc)) != NULL) { switch (status->type) { case PMEMPOOL_CHECK_MSG_TYPE_ERROR: outv(1, "%s\n", status->str.msg); break; case PMEMPOOL_CHECK_MSG_TYPE_INFO: outv(2, "%s\n", status->str.msg); break; case PMEMPOOL_CHECK_MSG_TYPE_QUESTION: status->str.answer = check_ask(status->str.msg); break; default: pmempool_check_end(ppc); exit(EXIT_FAILURE); } } enum pmempool_check_result ret = pmempool_check_end(ppc); return pmempool_check_2_check_res_t[ret]; } /* * pmempool_check_func -- main function for check command */ int pmempool_check_func(const char *appname, int argc, char *argv[]) { int ret = 0; check_result_t res = CHECK_RESULT_CONSISTENT; struct pmempool_check_context pc = pmempool_check_default; /* parse command line arguments */ ret = pmempool_check_parse_args(&pc, appname, argc, argv); if (ret) return ret; /* set verbosity level */ out_set_vlevel(pc.verbose); res = pmempool_check_perform(&pc); switch (res) { case CHECK_RESULT_CONSISTENT: outv(2, "%s: consistent\n", pc.fname); ret = 0; break; case CHECK_RESULT_NOT_CONSISTENT: outv(1, "%s: not consistent\n", pc.fname); ret = -1; break; case CHECK_RESULT_REPAIRED: outv(1, "%s: repaired\n", pc.fname); ret = 0; break; case CHECK_RESULT_CANNOT_REPAIR: outv(1, "%s: cannot repair\n", pc.fname); ret = -1; break; case CHECK_RESULT_SYNC_REQ: outv(1, "%s: sync required\n", pc.fname); ret = 0; break; case CHECK_RESULT_ERROR: if (errno) outv_err("%s\n", strerror(errno)); if (pc.repair) outv_err("repairing failed\n"); else outv_err("checking consistency failed\n"); ret = -1; break; default: outv_err("status unknown\n"); ret = -1; break; } return ret; } pmdk-1.13.1/src/tools/pmempool/bash_completion/0000775000000000000000000000000014435627501020164 5ustar rootrootpmdk-1.13.1/src/tools/pmempool/bash_completion/pmempool0000664000000000000000000000660014435627501021741 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2017, Intel Corporation # # pmempool -- bash completion script for pmempool # # # _pmempool_gen -- generates result for completion # Arguments: # $1 - values # $2 - current string # $3 - prefix for results _pmempool_gen() { COMPREPLAY=() local values=$1 local cur=$2 local prefix=$3 local i=${#COMPREPLY[@]} for v in $values do [[ "$v" == "$cur"* ]] && COMPREPLY[i++]="$prefix$v" done } # # _pmempool_get_cmds -- returns available pmempool commands # _pmempool_get_cmds() { echo -n $(pmempool --help | grep -e '^\S\+\s\+-' |\ grep -o '^\S\+' | sed '/help/d') } # # _pmempool_get_opts -- returns available options for specified command # Arguments: # $1 - command # _pmempool_get_opts() { local c=$1 local opts=$(pmempool ${c} --help | grep -o -e "-., --\S\+" |\ grep -o -e "--\S\+") echo "$opts" } # # _pmempool_get_values -- returns available values for specified option # Arguments: # $1 - command # $2 - option # $3 - values delimiter # $4 - current values, will be removed from result # _pmempool_get_values() { local cmd=$1 local opt=$2 local delim=$3 local curvals=$4 local vals=$(pmempool ${cmd} --help |\ grep -o -e "${opt}\s\+\S\+${delim}\S\+" |\ sed "s/${opt}\s\+\(\S\+${delim}\S\+\)/\1/" |\ sed "s/${delim}/ /g") if [ -n "$curvals" ] then local OLD_IFS=$IFS IFS="," for v in $curvals do vals=$(echo $vals | sed "s/$v//g") done IFS=$OLD_IFS fi echo "${vals}" } # # _pmempool_get_cmd -- returns command name if exist in specified array # Arguments: # $1 - command name # $2 - list of available commands # _pmempool_get_cmd() { local cmd=$1 local cmds=$2 [[ ${cmds} =~ ${cmd} ]] && echo -n ${cmd} } # # _pmempool_get_used_values -- returns already used values # Arguments: # $1 - current string # $2 - values delimiter # _pmempool_get_used_values() { local cur=$1 local delim=$2 local used=$(echo $cur | rev | cut -d $delim -s -f1 --complement | rev) [ -n "$used" ] && used="$used$delim" echo "$used" } # # _pmempool_get_current_value -- returns current value string # Arguments: # $1 - current string # $2 - values delimiter # _pmempool_get_current_value() { local cur=$1 local delim=$2 echo $cur | rev | cut -d $delim -f1 | rev } _pmempool() { local cur prev opts cur="${COMP_WORDS[COMP_CWORD]}" prev="${COMP_WORDS[COMP_CWORD-1]}" cmds=$(_pmempool_get_cmds) cmds_all="$cmds help" opts_pool_types="blk log obj" cmd=$(_pmempool_get_cmd "${COMP_WORDS[1]}" "$cmds_all") if [[ ${cur} == -* ]] then local opts=$(_pmempool_get_opts $cmd) _pmempool_gen "$opts" "$cur" elif [[ ${prev} == --* ]] then local used=$(_pmempool_get_used_values "$cur" ",") local _cur=$(_pmempool_get_current_value "$cur" ",") local values=$(_pmempool_get_values ${cmd} ${prev} "," $used) if [ -n "${values}" ] then # values separated by ',' may contain multiple values _pmempool_gen "$values" "$_cur" "$used" else # values separated by '|' may contain only one value values=$(_pmempool_get_values $cmd $prev "|") _pmempool_gen "$values" "$cur" fi elif [[ $cmd == create ]] then case "${COMP_WORDS[@]}" in *blk*|*log*|*obj*|*--inherit*) ;; *) _pmempool_gen "$opts_pool_types" "$cur" ;; esac elif [[ ${prev} == help ]] then _pmempool_gen "$cmds" "$cur" elif [[ ${prev} == pmempool ]] then _pmempool_gen "$cmds_all" "$cur" fi } complete -o default -F _pmempool pmempool pmdk-1.13.1/src/tools/pmempool/synchronize.h0000664000000000000000000000073014435627501017542 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2016-2023, Intel Corporation */ /* * synchronize.h -- pmempool sync command header file */ #ifdef _WIN32 #define WIN_DEPR_STR "Windows support is deprecated." #define WIN_DEPR_ATTR __declspec(deprecated(WIN_DEPR_STR)) #endif #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmempool_sync_func(const char *appname, int argc, char *argv[]); #ifdef _WIN32 WIN_DEPR_ATTR #endif void pmempool_sync_help(const char *appname); pmdk-1.13.1/src/tools/pmempool/output.c0000664000000000000000000004114514435627501016527 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2023, Intel Corporation */ /* * output.c -- definitions of output printing related functions */ #include #include #include #include #include #include #include #include #include #include #include #include "feature.h" #include "common.h" #include "output.h" #define _STR(s) #s #define STR(s) _STR(s) #define TIME_STR_FMT "%a %b %d %Y %H:%M:%S" #define UUID_STR_MAX 37 #define HEXDUMP_ROW_WIDTH 16 /* * 2 chars + space per byte + * space after 8 bytes and terminating NULL */ #define HEXDUMP_ROW_HEX_LEN (HEXDUMP_ROW_WIDTH * 3 + 1 + 1) /* 1 printable char per byte + terminating NULL */ #define HEXDUMP_ROW_ASCII_LEN (HEXDUMP_ROW_WIDTH + 1) #define SEPARATOR_CHAR '-' #define MAX_INDENT 32 #define INDENT_CHAR ' ' static char out_indent_str[MAX_INDENT + 1]; static int out_indent_level; static int out_vlevel; static unsigned out_column_width = 20; static FILE *out_fh; static const char *out_prefix; #define STR_MAX 256 /* * outv_check -- verify verbosity level */ int outv_check(int vlevel) { return vlevel && (out_vlevel >= vlevel); } /* * out_set_col_width -- set column width * * See: outv_field() function */ void out_set_col_width(unsigned col_width) { out_column_width = col_width; } /* * out_set_vlevel -- set verbosity level */ void out_set_vlevel(int vlevel) { out_vlevel = vlevel; if (out_fh == NULL) out_fh = stdout; } /* * out_set_prefix -- set prefix to output format */ void out_set_prefix(const char *prefix) { out_prefix = prefix; } /* * out_set_stream -- set output stream */ void out_set_stream(FILE *stream) { out_fh = stream; memset(out_indent_str, INDENT_CHAR, MAX_INDENT); } /* * outv_err -- print error message */ void outv_err(const char *fmt, ...) { va_list ap; va_start(ap, fmt); outv_err_vargs(fmt, ap); va_end(ap); } /* * outv_err_vargs -- print error message */ void outv_err_vargs(const char *fmt, va_list ap) { char *_str = strdup(fmt); if (!_str) err(1, "strdup"); char *str = _str; fprintf(stderr, "error: "); int errstr = str[0] == '!'; if (errstr) str++; char *nl = strchr(str, '\n'); if (nl) *nl = '\0'; vfprintf(stderr, str, ap); if (errstr) fprintf(stderr, ": %s", strerror(errno)); fprintf(stderr, "\n"); free(_str); } /* * outv_indent -- change indentation level by factor */ void outv_indent(int vlevel, int i) { if (!outv_check(vlevel)) return; out_indent_str[out_indent_level] = INDENT_CHAR; out_indent_level += i; if (out_indent_level < 0) out_indent_level = 0; if (out_indent_level > MAX_INDENT) out_indent_level = MAX_INDENT; out_indent_str[out_indent_level] = '\0'; } /* * _out_prefix -- print prefix if defined */ static void _out_prefix(void) { if (out_prefix) fprintf(out_fh, "%s: ", out_prefix); } /* * _out_indent -- print indent */ static void _out_indent(void) { fprintf(out_fh, "%s", out_indent_str); } /* * outv -- print message taking into account verbosity level */ void outv(int vlevel, const char *fmt, ...) { va_list ap; if (!outv_check(vlevel)) return; _out_prefix(); _out_indent(); va_start(ap, fmt); vfprintf(out_fh, fmt, ap); va_end(ap); } /* * outv_nl -- print new line without indentation */ void outv_nl(int vlevel) { if (!outv_check(vlevel)) return; _out_prefix(); fprintf(out_fh, "\n"); } void outv_title(int vlevel, const char *fmt, ...) { va_list ap; if (!outv_check(vlevel)) return; fprintf(out_fh, "\n"); _out_prefix(); _out_indent(); va_start(ap, fmt); vfprintf(out_fh, fmt, ap); va_end(ap); fprintf(out_fh, ":\n"); } /* * outv_field -- print field name and value in specified format * * Field name will have fixed width which can be changed by * out_set_column_width() function. * vlevel - verbosity level * field - field name * fmt - format form value */ void outv_field(int vlevel, const char *field, const char *fmt, ...) { va_list ap; if (!outv_check(vlevel)) return; _out_prefix(); _out_indent(); va_start(ap, fmt); fprintf(out_fh, "%-*s : ", out_column_width, field); vfprintf(out_fh, fmt, ap); fprintf(out_fh, "\n"); va_end(ap); } /* * out_get_percentage -- return percentage string */ const char * out_get_percentage(double perc) { static char str_buff[STR_MAX] = {0, }; int ret = 0; if (perc > 0.0 && perc < 0.0001) { ret = util_snprintf(str_buff, STR_MAX, "%e %%", perc); if (ret < 0) return ""; } else { int decimal = 0; if (perc >= 100.0 || perc < DBL_EPSILON) decimal = 0; else decimal = 6; ret = util_snprintf(str_buff, STR_MAX, "%.*f %%", decimal, perc); if (ret < 0) return ""; } return str_buff; } /* * out_get_size_str -- return size string * * human - if 1 return size in human-readable format * if 2 return size in bytes and human-readable format * otherwise return size in bytes. */ const char * out_get_size_str(uint64_t size, int human) { static char str_buff[STR_MAX] = {0, }; char units[] = { 'K', 'M', 'G', 'T', '\0' }; const int nunits = sizeof(units) / sizeof(units[0]); int ret = 0; if (!human) { ret = util_snprintf(str_buff, STR_MAX, "%"PRIu64, size); } else { int i = -1; double dsize = (double)size; uint64_t csize = size; while (csize >= 1024 && i < nunits) { csize /= 1024; dsize /= 1024.0; i++; } if (i >= 0 && i < nunits) if (human == 1) ret = util_snprintf(str_buff, STR_MAX, "%.1f%c", dsize, units[i]); else ret = util_snprintf(str_buff, STR_MAX, "%.1f%c [%" PRIu64"]", dsize, units[i], size); else ret = util_snprintf(str_buff, STR_MAX, "%"PRIu64, size); } if (ret < 0) return ""; return str_buff; } /* * out_get_uuid_str -- returns uuid in human readable format */ const char * out_get_uuid_str(uuid_t uuid) { static char uuid_str[UUID_STR_MAX] = {0, }; int ret = util_uuid_to_string(uuid, uuid_str); if (ret != 0) { outv(2, "failed to convert uuid to string"); return NULL; } return uuid_str; } /* * out_get_time_str -- returns time in human readable format */ const char * out_get_time_str(time_t time) { static char str_buff[STR_MAX] = {0, }; struct tm tm; if (util_localtime(&time, &tm)) { strftime(str_buff, STR_MAX, TIME_STR_FMT, &tm); } else { int ret = util_snprintf(str_buff, STR_MAX, "unknown"); if (ret < 0) return ""; } return str_buff; } /* * out_get_ascii_str -- get string with printable ASCII dump buffer * * Convert non-printable ASCII characters to dot '.' * See: util_get_printable_ascii() function. */ static int out_get_ascii_str(char *str, size_t str_len, const uint8_t *datap, size_t len) { int c = 0; size_t i; char pch; if (str_len < len) return -1; for (i = 0; i < len; i++) { pch = util_get_printable_ascii((char)datap[i]); int t = util_snprintf(str + c, str_len - (size_t)c, "%c", pch); if (t < 0) return -1; c += t; } return c; } /* * out_get_hex_str -- get string with hexadecimal dump of buffer * * Hexadecimal bytes in format %02x, each one followed by space, * additional space after every 8th byte. */ static int out_get_hex_str(char *str, size_t str_len, const uint8_t *datap, size_t len) { int c = 0; size_t i; int t; if (str_len < (3 * len + 1)) return -1; for (i = 0; i < len; i++) { /* add space after n*8 byte */ if (i && (i % 8) == 0) { t = util_snprintf(str + c, str_len - (size_t)c, " "); if (t < 0) return -1; c += t; } t = util_snprintf(str + c, str_len - (size_t)c, "%02x ", datap[i]); if (t < 0) return -1; c += t; } return c; } /* * outv_hexdump -- print buffer in canonical hex+ASCII format * * Print offset in hexadecimal, * sixteen space-separated, two column, hexadecimal bytes, * followed by the same sixteen bytes converted to printable ASCII characters * enclosed in '|' characters. */ void outv_hexdump(int vlevel, const void *addr, size_t len, size_t offset, int sep) { if (!outv_check(vlevel) || len <= 0) return; const uint8_t *datap = (uint8_t *)addr; uint8_t row_hex_str[HEXDUMP_ROW_HEX_LEN] = {0, }; uint8_t row_ascii_str[HEXDUMP_ROW_ASCII_LEN] = {0, }; size_t curr = 0; size_t prev = 0; int repeated = 0; int n = 0; while (len) { size_t curr_len = min(len, HEXDUMP_ROW_WIDTH); /* * Check if current row is the same as the previous one * don't check it for first and last rows. */ if (len != curr_len && curr && !memcmp(datap + prev, datap + curr, curr_len)) { if (!repeated) { /* print star only for the first repeated */ fprintf(out_fh, "*\n"); repeated = 1; } } else { repeated = 0; /* row with hexadecimal bytes */ int rh = out_get_hex_str((char *)row_hex_str, HEXDUMP_ROW_HEX_LEN, datap + curr, curr_len); /* row with printable ascii chars */ int ra = out_get_ascii_str((char *)row_ascii_str, HEXDUMP_ROW_ASCII_LEN, datap + curr, curr_len); if (ra && rh) n = fprintf(out_fh, "%08zx %-*s|%-*s|\n", curr + offset, HEXDUMP_ROW_HEX_LEN, row_hex_str, HEXDUMP_ROW_WIDTH, row_ascii_str); prev = curr; } len -= curr_len; curr += curr_len; } if (sep && n) { while (--n) fprintf(out_fh, "%c", SEPARATOR_CHAR); fprintf(out_fh, "\n"); } } /* * out_get_checksum -- return checksum string with result */ const char * out_get_checksum(void *addr, size_t len, uint64_t *csump, size_t skip_off) { static char str_buff[STR_MAX] = {0, }; int ret = 0; uint64_t csum = util_checksum_compute(addr, len, csump, skip_off); if (*csump == htole64(csum)) ret = util_snprintf(str_buff, STR_MAX, "0x%" PRIx64" [OK]", le64toh(csum)); else ret = util_snprintf(str_buff, STR_MAX, "0x%" PRIx64 " [wrong! should be: 0x%" PRIx64 "]", le64toh(*csump), le64toh(csum)); if (ret < 0) return ""; return str_buff; } /* * out_get_btt_map_entry -- return BTT map entry with flags strings (DEPRECATED) */ const char * out_get_btt_map_entry(uint32_t map) { static char str_buff[STR_MAX] = {0, }; int is_init = (map & ~BTT_MAP_ENTRY_LBA_MASK) == 0; int is_zero = (map & ~BTT_MAP_ENTRY_LBA_MASK) == BTT_MAP_ENTRY_ZERO; int is_error = (map & ~BTT_MAP_ENTRY_LBA_MASK) == BTT_MAP_ENTRY_ERROR; int is_normal = (map & ~BTT_MAP_ENTRY_LBA_MASK) == BTT_MAP_ENTRY_NORMAL; uint32_t lba = map & BTT_MAP_ENTRY_LBA_MASK; int ret = util_snprintf(str_buff, STR_MAX, "0x%08x state: %s", lba, is_init ? "init" : is_zero ? "zero" : is_error ? "error" : is_normal ? "normal" : "unknown"); if (ret < 0) return ""; return str_buff; } /* * out_get_pool_type_str -- get pool type string */ const char * out_get_pool_type_str(pmem_pool_type_t type) { switch (type) { case PMEM_POOL_TYPE_LOG: /* deprecated */ return "log"; case PMEM_POOL_TYPE_BLK: /* deprecated */ return "blk"; case PMEM_POOL_TYPE_OBJ: return "obj"; case PMEM_POOL_TYPE_BTT: /* deprecated */ return "btt"; default: return "unknown"; } } /* * out_get_pool_signature -- return signature of specified pool type */ const char * out_get_pool_signature(pmem_pool_type_t type) { switch (type) { case PMEM_POOL_TYPE_LOG: /* deprecated */ return LOG_HDR_SIG; case PMEM_POOL_TYPE_BLK: /* deprecated */ return BLK_HDR_SIG; case PMEM_POOL_TYPE_OBJ: return OBJ_HDR_SIG; default: return NULL; } } /* * out_get_chunk_type_str -- get chunk type string */ const char * out_get_chunk_type_str(enum chunk_type type) { switch (type) { case CHUNK_TYPE_FOOTER: return "footer"; case CHUNK_TYPE_FREE: return "free"; case CHUNK_TYPE_USED: return "used"; case CHUNK_TYPE_RUN: return "run"; case CHUNK_TYPE_UNKNOWN: default: return "unknown"; } } /* * out_get_chunk_flags -- get names of set flags for chunk header */ const char * out_get_chunk_flags(uint16_t flags) { if (flags & CHUNK_FLAG_COMPACT_HEADER) return "compact header"; else if (flags & CHUNK_FLAG_HEADER_NONE) return "header none"; return ""; } /* * out_get_zone_magic_str -- get zone magic string with additional * information about correctness of the magic value */ const char * out_get_zone_magic_str(uint32_t magic) { static char str_buff[STR_MAX] = {0, }; const char *correct = NULL; switch (magic) { case 0: correct = "uninitialized"; break; case ZONE_HEADER_MAGIC: correct = "OK"; break; default: correct = "wrong! should be " STR(ZONE_HEADER_MAGIC); break; } int ret = util_snprintf(str_buff, STR_MAX, "0x%08x [%s]", magic, correct); if (ret < 0) return ""; return str_buff; } /* * out_get_pmemoid_str -- get PMEMoid string */ const char * out_get_pmemoid_str(PMEMoid oid, uint64_t uuid_lo) { static char str_buff[STR_MAX] = {0, }; int free_cor = 0; int ret = 0; char *correct = "OK"; if (oid.pool_uuid_lo && oid.pool_uuid_lo != uuid_lo) { ret = util_snprintf(str_buff, STR_MAX, "wrong! should be 0x%016"PRIx64, uuid_lo); if (ret < 0) err(1, "snprintf: %d", ret); correct = strdup(str_buff); if (!correct) err(1, "Cannot allocate memory for PMEMoid string\n"); free_cor = 1; } ret = util_snprintf(str_buff, STR_MAX, "off: 0x%016"PRIx64" pool_uuid_lo: 0x%016" PRIx64" [%s]", oid.off, oid.pool_uuid_lo, correct); if (free_cor) free(correct); if (ret < 0) err(1, "snprintf: %d", ret); return str_buff; } /* * out_get_arch_machine_class_str -- get a string representation of the machine * class */ const char * out_get_arch_machine_class_str(uint8_t machine_class) { switch (machine_class) { case PMDK_MACHINE_CLASS_64: return "64"; default: return "unknown"; } } /* * out_get_arch_data_str -- get a string representation of the data endianness */ const char * out_get_arch_data_str(uint8_t data) { switch (data) { case PMDK_DATA_LE: return "2's complement, little endian"; case PMDK_DATA_BE: return "2's complement, big endian"; default: return "unknown"; } } /* * out_get_arch_machine_str -- get a string representation of the machine type */ const char * out_get_arch_machine_str(uint16_t machine) { static char str_buff[STR_MAX] = {0, }; switch (machine) { case PMDK_MACHINE_X86_64: return "AMD X86-64"; case PMDK_MACHINE_AARCH64: return "Aarch64"; case PMDK_MACHINE_PPC64: return "PPC64"; case PMDK_MACHINE_RISCV64: return "RISCV"; default: break; } int ret = util_snprintf(str_buff, STR_MAX, "unknown %u", machine); if (ret < 0) return "unknown"; return str_buff; } /* * out_get_last_shutdown_str -- get a string representation of the finish state */ const char * out_get_last_shutdown_str(uint8_t dirty) { if (dirty) return "dirty"; else return "clean"; } /* * out_get_alignment_descr_str -- get alignment descriptor string */ const char * out_get_alignment_desc_str(uint64_t ad, uint64_t valid_ad) { static char str_buff[STR_MAX] = {0, }; int ret = 0; if (ad == valid_ad) ret = util_snprintf(str_buff, STR_MAX, "0x%016"PRIx64"[OK]", ad); else ret = util_snprintf(str_buff, STR_MAX, "0x%016"PRIx64" " "[wrong! should be 0x%016"PRIx64"]", ad, valid_ad); if (ret < 0) return ""; return str_buff; } /* * out_concat -- concatenate the new element to the list of strings * * If concatenation is successful it increments current position in the output * string and number of elements in the list. Elements are separated with ", ". */ static int out_concat(char *str_buff, int *curr, int *count, const char *str) { ASSERTne(str_buff, NULL); ASSERTne(curr, NULL); ASSERTne(str, NULL); const char *separator = (count != NULL && *count > 0) ? ", " : ""; int ret = util_snprintf(str_buff + *curr, (size_t)(STR_MAX - *curr), "%s%s", separator, str); if (ret < 0) return -1; *curr += ret; if (count) ++(*count); return 0; } /* * out_get_incompat_features_str -- (internal) get a string with names of * incompatibility flags */ const char * out_get_incompat_features_str(uint32_t incompat) { static char str_buff[STR_MAX] = {0}; features_t features = {POOL_FEAT_ZERO, incompat, POOL_FEAT_ZERO}; int ret = 0; if (incompat == 0) { /* print the value only */ return "0x0"; } else { /* print the value and the left square bracket */ ret = util_snprintf(str_buff, STR_MAX, "0x%x [", incompat); if (ret < 0) { ERR("snprintf for incompat features: %d", ret); return ""; } /* print names of known options */ int count = 0; int curr = ret; features_t found; const char *feat; while (((feat = util_feature2str(features, &found))) != NULL) { util_feature_disable(&features, found); ret = out_concat(str_buff, &curr, &count, feat); if (ret < 0) return ""; } /* check if any unknown flags are set */ if (!util_feature_is_zero(features)) { if (out_concat(str_buff, &curr, &count, "?UNKNOWN_FLAG?")) return ""; } /* print the right square bracket */ if (out_concat(str_buff, &curr, NULL, "]")) return ""; } return str_buff; } pmdk-1.13.1/src/tools/pmempool/dump.c0000664000000000000000000002006714435627501016134 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2023, Intel Corporation */ /* * create.c -- pmempool create command source file */ #include #include #include #include #include #include #include #include #include "common.h" #include "dump.h" #include "output.h" #include "os.h" #include "libpmemblk.h" #include "libpmemlog.h" #define VERBOSE_DEFAULT 1 /* * pmempool_dump -- context and arguments for dump command */ struct pmempool_dump { char *fname; char *ofname; char *range; FILE *ofh; int hex; uint64_t bsize; struct ranges ranges; size_t chunksize; uint64_t chunkcnt; }; /* * pmempool_dump_default -- default arguments and context values */ static const struct pmempool_dump pmempool_dump_default = { .fname = NULL, .ofname = NULL, .range = NULL, .ofh = NULL, .hex = 1, .bsize = 0, .chunksize = 0, .chunkcnt = 0, }; /* * long_options -- command line options */ static const struct option long_options[] = { {"output", required_argument, NULL, 'o' | OPT_ALL}, {"binary", no_argument, NULL, 'b' | OPT_ALL}, {"range", required_argument, NULL, 'r' | OPT_ALL}, {"chunk", required_argument, NULL, 'c' | OPT_LOG}, {"help", no_argument, NULL, 'h' | OPT_ALL}, {NULL, 0, NULL, 0 }, }; /* * help_str -- string for help message */ static const char * const help_str = "Dump user data from pool\n" "NOTE: pmem blk and log pools are deprecated\n" "\n" "Available options:\n" " -o, --output output file name\n" " -b, --binary dump data in binary format\n" " -r, --range range of bytes/blocks/data chunks\n" " -c, --chunk size of chunk for PMEMLOG pool\n" " -h, --help display this help and exit\n" "\n" "For complete documentation see %s-dump(1) manual page.\n" ; /* * print_usage -- print application usage short description */ static void print_usage(const char *appname) { printf("Usage: %s dump [] \n", appname); } /* * print_version -- print version string */ static void print_version(const char *appname) { printf("NOTE: pmem blk and log pools are deprecated\n"); printf("%s %s\n", appname, SRCVERSION); } /* * pmempool_dump_help -- print help message for dump command */ void pmempool_dump_help(const char *appname) { print_usage(appname); print_version(appname); printf(help_str, appname); } /* * pmempool_dump_log_process_chunk -- callback for pmemlog_walk */ static int pmempool_dump_log_process_chunk(const void *buf, size_t len, void *arg) { struct pmempool_dump *pdp = (struct pmempool_dump *)arg; if (len == 0) return 0; struct range *curp = NULL; if (pdp->chunksize) { PMDK_LIST_FOREACH(curp, &pdp->ranges.head, next) { if (pdp->chunkcnt >= curp->first && pdp->chunkcnt <= curp->last && pdp->chunksize <= len) { if (pdp->hex) { outv_hexdump(VERBOSE_DEFAULT, buf, pdp->chunksize, pdp->chunksize * pdp->chunkcnt, 0); } else { if (fwrite(buf, pdp->chunksize, 1, pdp->ofh) != 1) err(1, "%s", pdp->ofname); } } } pdp->chunkcnt++; } else { PMDK_LIST_FOREACH(curp, &pdp->ranges.head, next) { if (curp->first >= len) continue; uint8_t *ptr = (uint8_t *)buf + curp->first; if (curp->last >= len) curp->last = len - 1; uint64_t count = curp->last - curp->first + 1; if (pdp->hex) { outv_hexdump(VERBOSE_DEFAULT, ptr, count, curp->first, 0); } else { if (fwrite(ptr, count, 1, pdp->ofh) != 1) err(1, "%s", pdp->ofname); } } } return 1; } /* * pmempool_dump_parse_range -- parse range passed by arguments */ static int pmempool_dump_parse_range(struct pmempool_dump *pdp, size_t max) { struct range entire; memset(&entire, 0, sizeof(entire)); entire.last = max; if (util_parse_ranges(pdp->range, &pdp->ranges, entire)) { outv_err("invalid range value specified" " -- '%s'\n", pdp->range); return -1; } if (PMDK_LIST_EMPTY(&pdp->ranges.head)) util_ranges_add(&pdp->ranges, entire); return 0; } /* * pmempool_dump_log (DEPRECATED) -- dump data from pmem log pool */ static int pmempool_dump_log(struct pmempool_dump *pdp) { PMEMlogpool *plp = pmemlog_open(pdp->fname); if (!plp) { warn("%s", pdp->fname); return -1; } os_off_t off = pmemlog_tell(plp); if (off < 0) { warn("%s", pdp->fname); pmemlog_close(plp); return -1; } if (off == 0) goto end; size_t max = (size_t)off - 1; if (pdp->chunksize) max /= pdp->chunksize; if (pmempool_dump_parse_range(pdp, max)) return -1; pdp->chunkcnt = 0; pmemlog_walk(plp, pdp->chunksize, pmempool_dump_log_process_chunk, pdp); end: pmemlog_close(plp); return 0; } /* * pmempool_dump_blk (DEPRECATED) -- dump data from pmem blk pool */ static int pmempool_dump_blk(struct pmempool_dump *pdp) { PMEMblkpool *pbp = pmemblk_open(pdp->fname, pdp->bsize); if (!pbp) { warn("%s", pdp->fname); return -1; } if (pmempool_dump_parse_range(pdp, pmemblk_nblock(pbp) - 1)) return -1; uint8_t *buff = malloc(pdp->bsize); if (!buff) err(1, "Cannot allocate memory for pmemblk block buffer"); int ret = 0; uint64_t i; struct range *curp = NULL; PMDK_LIST_FOREACH(curp, &pdp->ranges.head, next) { assert((os_off_t)curp->last >= 0); for (i = curp->first; i <= curp->last; i++) { if (pmemblk_read(pbp, buff, (os_off_t)i)) { ret = -1; outv_err("reading block number %lu " "failed\n", i); break; } if (pdp->hex) { uint64_t offset = i * pdp->bsize; outv_hexdump(VERBOSE_DEFAULT, buff, pdp->bsize, offset, 0); } else { if (fwrite(buff, pdp->bsize, 1, pdp->ofh) != 1) { warn("write"); ret = -1; break; } } } } free(buff); pmemblk_close(pbp); return ret; } static const struct option_requirement option_requirements[] = { { 0, 0, 0} }; /* * pmempool_dump_func -- dump command main function */ int pmempool_dump_func(const char *appname, int argc, char *argv[]) { struct pmempool_dump pd = pmempool_dump_default; PMDK_LIST_INIT(&pd.ranges.head); out_set_vlevel(VERBOSE_DEFAULT); struct options *opts = util_options_alloc(long_options, sizeof(long_options) / sizeof(long_options[0]), option_requirements); int ret = 0; long long chunksize; int opt; while ((opt = util_options_getopt(argc, argv, "ho:br:c:", opts)) != -1) { switch (opt) { case 'o': pd.ofname = optarg; break; case 'b': pd.hex = 0; break; case 'r': pd.range = optarg; break; case 'c': chunksize = atoll(optarg); if (chunksize <= 0) { outv_err("invalid chunk size specified '%s'\n", optarg); exit(EXIT_FAILURE); } pd.chunksize = (size_t)chunksize; break; case 'h': pmempool_dump_help(appname); exit(EXIT_SUCCESS); default: print_usage(appname); exit(EXIT_FAILURE); } } if (optind < argc) { pd.fname = argv[optind]; } else { print_usage(appname); exit(EXIT_FAILURE); } if (pd.ofname == NULL) { /* use standard output by default */ pd.ofh = stdout; } else { pd.ofh = os_fopen(pd.ofname, "wb"); if (!pd.ofh) { warn("%s", pd.ofname); exit(EXIT_FAILURE); } } /* set output stream - stdout or file passed by -o option */ out_set_stream(pd.ofh); struct pmem_pool_params params; /* parse pool type and block size for pmem blk pool */ pmem_pool_parse_params(pd.fname, ¶ms, 1); ret = util_options_verify(opts, params.type); if (ret) goto out; switch (params.type) { case PMEM_POOL_TYPE_LOG: /* deprecated */ ret = pmempool_dump_log(&pd); break; case PMEM_POOL_TYPE_BLK: /* deprecated */ pd.bsize = params.blk.bsize; ret = pmempool_dump_blk(&pd); break; case PMEM_POOL_TYPE_OBJ: outv_err("%s: PMEMOBJ pool not supported\n", pd.fname); ret = -1; goto out; case PMEM_POOL_TYPE_UNKNOWN: outv_err("%s: unknown pool type -- '%s'\n", pd.fname, params.signature); ret = -1; goto out; default: outv_err("%s: cannot determine type of pool\n", pd.fname); ret = -1; goto out; } if (ret) outv_err("%s: dumping pool file failed\n", pd.fname); out: if (pd.ofh != stdout) fclose(pd.ofh); util_ranges_clear(&pd.ranges); util_options_free(opts); return ret; } pmdk-1.13.1/src/tools/pmempool/rm.h0000664000000000000000000000071114435627501015604 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2014-2023, Intel Corporation */ /* * rm.h -- pmempool rm command header file */ #ifdef _WIN32 #define WIN_DEPR_STR "Windows support is deprecated." #define WIN_DEPR_ATTR __declspec(deprecated(WIN_DEPR_STR)) #endif #ifdef _WIN32 WIN_DEPR_ATTR #endif void pmempool_rm_help(const char *appname); #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmempool_rm_func(const char *appname, int argc, char *argv[]); pmdk-1.13.1/src/tools/pmempool/rm.c0000664000000000000000000001254114435627501015603 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2022, Intel Corporation */ /* * rm.c -- pmempool rm command main source file */ #include #include #include #include #include #include #include "os.h" #include "out.h" #include "common.h" #include "output.h" #include "file.h" #include "rm.h" #include "set.h" enum ask_type { ASK_SOMETIMES, /* ask before removing write-protected files */ ASK_ALWAYS, /* always ask */ ASK_NEVER, /* never ask */ }; /* verbosity level */ static int vlevel; /* force remove and ignore errors */ static int force; /* poolset files options */ #define RM_POOLSET_NONE (0) #define RM_POOLSET_LOCAL (1 << 0) #define RM_POOLSET_ALL (RM_POOLSET_LOCAL) static int rm_poolset_mode; /* mode of interaction */ static enum ask_type ask_mode; /* help message */ static const char * const help_str = "Remove pool file or all files from poolset\n" "\n" "Available options:\n" " -h, --help Print this help message.\n" " -v, --verbose Be verbose.\n" " -s, --only-pools Remove only pool files (default).\n" " -a, --all Remove all poolset files.\n" " -l, --local Remove local poolset files\n" " -f, --force Ignore nonexisting files.\n" " -i, --interactive Prompt before every single removal.\n" "\n" "For complete documentation see %s-rm(1) manual page.\n"; /* short options string */ static const char *optstr = "hvsfial"; /* long options */ static const struct option long_options[] = { {"help", no_argument, NULL, 'h'}, {"verbose", no_argument, NULL, 'v'}, {"only-pools", no_argument, NULL, 's'}, {"all", no_argument, NULL, 'a'}, {"local", no_argument, NULL, 'l'}, {"force", no_argument, NULL, 'f'}, {"interactive", no_argument, NULL, 'i'}, {NULL, 0, NULL, 0 }, }; /* * print_usage -- print usage message */ static void print_usage(const char *appname) { printf("Usage: %s rm [] \n", appname); } /* * pmempool_rm_help -- print help message */ void pmempool_rm_help(const char *appname) { print_usage(appname); printf(help_str, appname); } /* * rm_file -- remove single file */ static int rm_file(const char *file) { int write_protected = os_access(file, W_OK) != 0; char cask = 'y'; switch (ask_mode) { case ASK_ALWAYS: cask = '?'; break; case ASK_NEVER: cask = 'y'; break; case ASK_SOMETIMES: cask = write_protected ? '?' : 'y'; break; default: outv_err("unknown state"); return 1; } const char *pre_msg = write_protected ? "write-protected " : ""; char ans = ask_Yn(cask, "remove %sfile '%s' ?", pre_msg, file); if (ans == 'y') { if (util_unlink(file)) { outv_err("cannot remove file '%s'", file); return 1; } outv(1, "removed '%s'\n", file); } return 0; } /* * rm_poolset_cb -- (internal) callback for removing replicas */ static int rm_poolset_cb(struct part_file *pf, void *arg) { int *error = (int *)arg; int ret; const char *part_file = pf->part->path; outv(2, "part file : %s\n", part_file); int exists = util_file_exists(part_file); if (exists < 0) ret = 1; else if (!exists) { /* * Ignore not accessible file if force * flag is set. */ if (force) return 0; ret = 1; outv_err("!cannot remove file '%s'", part_file); } else { ret = rm_file(part_file); } if (ret) *error = ret; return 0; } /* * rm_poolset -- remove files parsed from poolset file */ static int rm_poolset(const char *file) { int error = 0; int ret = util_poolset_foreach_part(file, rm_poolset_cb, &error); if (ret == -1) { outv_err("parsing poolset failed: %s\n", out_get_errormsg()); return ret; } if (error && !force) { outv_err("!removing '%s' failed\n", file); return error; } return 0; } /* * pmempool_rm_func -- main function for rm command */ int pmempool_rm_func(const char *appname, int argc, char *argv[]) { /* by default do not remove any poolset files */ rm_poolset_mode = RM_POOLSET_NONE; int opt; while ((opt = getopt_long(argc, argv, optstr, long_options, NULL)) != -1) { switch (opt) { case 'h': pmempool_rm_help(appname); return 0; case 'v': vlevel++; break; case 's': rm_poolset_mode = RM_POOLSET_NONE; break; case 'a': rm_poolset_mode |= RM_POOLSET_ALL; break; case 'l': rm_poolset_mode |= RM_POOLSET_LOCAL; break; case 'f': force = 1; ask_mode = ASK_NEVER; break; case 'i': ask_mode = ASK_ALWAYS; break; default: print_usage(appname); return 1; } } out_set_vlevel(vlevel); if (optind == argc) { print_usage(appname); return 1; } int lret = 0; for (int i = optind; i < argc; i++) { char *file = argv[i]; /* check if file exists and we can read it */ int exists = os_access(file, F_OK | R_OK) == 0; if (!exists) { /* ignore not accessible file if force flag is set */ if (force) continue; outv_err("!cannot remove '%s'", file); lret = 1; continue; } int is_poolset = util_is_poolset_file(file); if (is_poolset < 0) { outv(1, "%s: cannot determine type of file", file); if (force) continue; } if (is_poolset) outv(2, "poolset file: %s\n", file); else outv(2, "pool file : %s\n", file); int ret; if (is_poolset) { ret = rm_poolset(file); if (!ret && (rm_poolset_mode & RM_POOLSET_LOCAL)) ret = rm_file(file); } else { ret = rm_file(file); } if (ret) lret = ret; } return lret; } pmdk-1.13.1/src/tools/pmempool/check.h0000664000000000000000000000072514435627501016250 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2014-2023, Intel Corporation */ /* * check.h -- pmempool check command header file */ #ifdef _WIN32 #define WIN_DEPR_STR "Windows support is deprecated." #define WIN_DEPR_ATTR __declspec(deprecated(WIN_DEPR_STR)) #endif #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmempool_check_func(const char *appname, int argc, char *argv[]); #ifdef _WIN32 WIN_DEPR_ATTR #endif void pmempool_check_help(const char *appname); pmdk-1.13.1/src/tools/pmempool/Makefile0000664000000000000000000000272214435627501016461 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2023, Intel Corporation # # Makefile -- top Makefile for pmempool # vpath %.c ../../libpmemobj/ TARGET = pmempool OBJS = pmempool.o\ info.o info_blk.o info_log.o info_obj.o ulog.o\ create.o dump.o check.o rm.o convert.o synchronize.o transform.o feature.o LIBPMEM=y LIBPMEMBLK=y LIBPMEMOBJ=y LIBPMEMLOG=y LIBPMEMPOOL=y TOOLS_COMMON=y TOOLS_PMEMPOOL_COMMON=y LIBPMEMOBJ_PRIV=memblock_from_offset alloc_class_by_id\ memblock_rebuild_state alloc_class_by_run\ heap_run_foreach_object alloc_class_collection_new\ alloc_class_collection_delete LIBPMEMBLK_PRIV=btt_init btt_write btt_fini btt_info_convert2h\ btt_info_convert2le btt_flog_convert2h btt_flog_convert2le # Libpmemblk and libpmemlog are deprecated. # This flag allows to build tests, examples and benchmarks # using pmemblk/pmemlog despite the deprecated state. CFLAGS += -Wno-deprecated-declarations INCS += -I$(TOP)/src/common INCS += -I$(TOP)/src/libpmem2 INCS += -I$(TOP)/src/libpmemlog INCS += -I$(TOP)/src/libpmemblk INCS += -I$(TOP)/src/libpmemobj MANPAGES = $(TOP)/doc/pmempool.1\ $(TOP)/doc/pmempool-info.1\ $(TOP)/doc/pmempool-create.1\ $(TOP)/doc/pmempool-check.1\ $(TOP)/doc/pmempool-dump.1\ $(TOP)/doc/pmempool-rm.1\ $(TOP)/doc/pmempool-convert.1\ $(TOP)/doc/pmempool-sync.1\ $(TOP)/doc/pmempool-transform.1 BASH_COMP_FILES = bash_completion/pmempool include ../Makefile.inc .PHONY: test check pmdk-1.13.1/src/tools/pmempool/convert.h0000664000000000000000000000076514435627501016657 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2014-2023, Intel Corporation */ /* * convert.h -- pmempool convert command header file */ #include #ifdef _WIN32 #define WIN_DEPR_STR "Windows support is deprecated." #define WIN_DEPR_ATTR __declspec(deprecated(WIN_DEPR_STR)) #endif #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmempool_convert_func(const char *appname, int argc, char *argv[]); #ifdef _WIN32 WIN_DEPR_ATTR #endif void pmempool_convert_help(const char *appname); pmdk-1.13.1/src/tools/pmempool/dump.h0000664000000000000000000000072114435627501016134 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2014-2023, Intel Corporation */ /* * dump.h -- pmempool dump command header file */ #ifdef _WIN32 #define WIN_DEPR_STR "Windows support is deprecated." #define WIN_DEPR_ATTR __declspec(deprecated(WIN_DEPR_STR)) #endif #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmempool_dump_func(const char *appname, int argc, char *argv[]); #ifdef _WIN32 WIN_DEPR_ATTR #endif void pmempool_dump_help(const char *appname); pmdk-1.13.1/src/tools/pmempool/README0000664000000000000000000002607214435627501015705 0ustar rootrootPersistent Memory Development Kit This is src/tools/pmempool/README. This file contains the high-level description of pmempool utility. 1. Introduction 2. Subcommands 2.1. info 2.2. check 2.3. create 2.4. dump 2.5. rm 2.6. sync 2.7. transform 2.8. convert 3. Source code 4. Packaging 5. Versioning 1. Introduction --------------- The main purpose of pmempool is to provide a user with set of utilities for off-line analysis and manipulation of pools created by pmem libraries. The pmempool is a generic command which consists of subcommands for specific purposes. Some of commands are required to work without any impact on processed pool, but some of them may create new or modify existing one. The pmempool may be useful for troubleshooting and may be used by system administrators and by software developers work on applications based on Persistent Memory Development Kit. The latter may find these tools useful for testing and debugging purposes also. Currently there is a following set of commands available: * info - Prints information and statistics in human-readable format about specified pool. * check - Checks pool's consistency and repairs pool if it is not consistent. * create - Creates a pool of specified type with additional properties specific for this type of pool. * dump - Dumps usable data from pool in hexadecimal or binary format. * rm - Removes pool file or all pool files listed in poolset configuration file. * sync - Synchronizes replicas within a poolset. * transform - Modifies internal structure of a poolset. * convert - Updates the pool to the latest available layout version. This file contains high-level description of available commands and their features. For details about usage and available command line arguments please refer to specific manual pages. There is one common manual page with description of all commands - pmempool(1) and manual pages which describe all commands in detail: pmempool-info(1) pmempool-check(1) pmempool-create(1) pmempool-dump(1) pmempool-rm(1) pmempool-sync(1) pmempool-transform(1) pmempool-convert(1) Subsequent sections contain detailed description of each command, information about the source code, packaging and versioning scheme. 2. Subcommands -------------- The pmempool application contains number of commands which perform specific operations on pool. The following subsections contain detailed description of existing commands. 2.1. info --------- The pmempool invoked with *info* command analyzes the existing pool created by PMDK libraries. The main task of this command is to print all usable information from pool headers and user data in human readable format. It automatically recognizes pool type by parsing and analyzing pool header. The recognition is done by checking the signature in pool header. The main intention of *info* command is to present internal data structures as they are stored in file - not for checking consistency. For this purpose there is *check* command available. The pmempool with *info* command analyzes a pool file as long as it is possible regarding correctness of internal meta-data (correct offsets, sizes etc.). If it is not possible to analyze rest of file, pmempool exits with an error code and prints an appropriate error message. Currently there is lack of interprocess synchronization for pool files, so the pmempool with *info* command should be invoked off-line. Using pmempool on pool file which may be modified by another process may lead to stopping processing the file. There is a set of common features for all pool types and a set of features specific for particular pool type. All features are described below. ** Common features * The basic function of *info* command is to print information about the most important internal data structures from specific pool. By default this is done by invoking pmempool with *info* command and one or more files. * It is possible to print basic statistics about the pool by passing appropriate command line argument. * The type of pool is recognized automatically. The exact list of headers and internal meta-data depends on pool's type. All information is displayed in human-readable format. * The pool header may be corrupted and automatic recognition of pool's type will fail. In order to analyze a pool file as a pool of specific type it is possible to force that by specifying the desired pool type using appropriate command line argument. * Headers and internal meta-data are displayed in human-readable format by default. However it is possible to display them in mixed format which consists of hexadecimal dump of headers and parsed data in human-readable format. * By default only non-volatile fields from internal structures are displayed. In order to display volatile fields you should increase the verbosity level. * By default all sizes are displayed in bytes unit. It is possible to print them in more human-readable formats with appropriate units (e.g. 4k, 8M, 16G). ** Features for *log* pool type (DEPRECATED) * By default pmempool with *info* command displays the pool header, log pool type specific header and statistics. It is possible to print data in hexadecimal format by passing appropriate command line option. * It is possible to walk through the usable data using fixed data chunk size. This feature uses similar approach as pmemlog_walk() function. For details please refer to libpmemlog(7). ** Features for *blk* pool type (DEPRECATED) * By default pmempool with *info* command displays the pool header, blk pool type specific header, BTT Info header and statistics. * It is possible to print more headers and internal data by passing specific command line options. It is possible to print the following sections: BTT Map entries, BTT FLOG, BTT Info backup and data blocks. * It is possible to print specific range of blocks in both absolute or relative manner (e.g. display blocks from 10 to 1000, display 10 blocks starting from block number 1000) * By default when displaying data blocks all blocks are displayed. However it is possible to skip blocks marked with zero or error flags, or to skip blocks which are not marked by any flag. Skipping blocks has impact on blocks ranges (e.g. display 10 blocks marked with error flag in the range from 0 to 10000) 2.2. check ---------- The pmempool invoked with *check* command checks existing pool's consistency. If the pool is consistent pmempool exits with exit code 0. Otherwise nonzero error code is returned and appropriate message is displayed. In addition it may also try to fix some common known errors upon explicit demand of user. In this case a pool file will be opened in read-write mode so the user should be aware of modifications made by pmempool application. Below is the description of available features: * By default pmempool with *check* command prints brief description about encountered error(s) and proper error value is returned. If there is no error nothing is printed and exit code is 0. * If an error is encountered while checking a consistency of a pool it is possible to try to fix all errors. In this case the pool file will be opened in read-write mode. * User may request to _not_ modify pool's file when trying to repair it but just to report what would be done if the repair was performed. * When repairing a pool user may request to create backup before any modification is made. If it is not possible to create full backup of existing pool, the process will terminate. 2.3. create ----------- The pmempool invoked with *create* command creates a pool file of specific type and size. Depending on pool's type it is possible to provide more desired properties of a pool. Below is the description of available features: * The main feature is to create pool of specified type and size. Therefore it is required to pass at least two command line arguments. * User may want to create a pool file with size of the whole partition. This is possible by passing proper command line argument. * It is possible to create a pool with the same parameters as another pool passed in command line arguments - it may be considered as cloning the pool. 2.4. dump --------- The pmempool invoked with *dump* command dumps usable data from specified pool. This may be dumped either in hexadecimal or binary format. Below is the description of available features: * The main feature is to dump all data from a pool file. In case of dumping data to terminal by default data is dumped in hexadecimal format. In case of redirecting standard output to a file data will be dumped in binary format. * It is possible to specify the format of dumped data to either hexadecimal or binary. * By default data is dumped to standard output. However it is possible to specify a file name to dump data into. * In case of pmem blk pool type it is possible to set range of blocks in either absolute or relative manner. (DEPRECATED) * In case of pmem log pool type it is possible to set size of chunk and range of chunks to dump in either absolute or relative manner. (DEPRECATED) 2.5. rm ------- The pmempool *rm* command is a simple helper utility which removes pool files created using either PMDK libraries or pmempool *create* command. * The main feature is to parse the poolset configuration file and remove all listed pool files. * It is possible to run the pmempool *rm* command in interactive mode, where before removing each file the user must confirm the removal operation. * The command line interface is similar to interface provided by standard, system *rm* command. 2.6. sync --------- The pmempool *sync* synchronize data between replicas within a poolset. The command has the following features: * Metadata in a poolset are checked for consistency. * Missing or damaged parts are recreated. 2.7. transform -------------- The pmempool *transform* command modifies internal structure of a poolset. Available features of the command: * Adding replicas. * Removing replicas. 2.8. convert -------------- The pmempool invoked with the *convert* command performs a conversion of the specified pool to the newest layout supported by this tool. Currently only libpmemobj pools are supported. It is advised to have a backup of the pool before conversion. 3. Source code -------------- The source code of pmempool is located in pmempool directory. By default pmempool is installed in $(DESTDIR)/usr/bin directory. You can change it by passing $(TOOLSDIR) variable to "make install". For example, the following command will install pmempool in ~/bin directory: $ make install DESTDIR=~ TOOLSDIR=/bin See the top-level README file for detailed information about building and installation. 4. Packaging ------------ The pmempool application is provided in separate packages. Both rpm and dpkg packages are built automatically with other packages. See the top-level README file for detailed information about building packages. 5. Versioning ------------- The versioning of pmempool application is the same as all PMDK libraries. pmdk-1.13.1/src/tools/pmempool/pmempool.c0000664000000000000000000001352314435627501017016 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2023, Intel Corporation */ /* * pmempool.c -- pmempool main source file */ #include #include #include #include #include #include #include #include "common.h" #include "output.h" #include "info.h" #include "create.h" #include "dump.h" #include "check.h" #include "rm.h" #include "convert.h" #include "synchronize.h" #include "transform.h" #include "feature.h" #include "set.h" #include "pmemcommon.h" #define APPNAME "pmempool" #define PMEMPOOL_TOOL_LOG_PREFIX "pmempool" #define PMEMPOOL_TOOL_LOG_LEVEL_VAR "PMEMPOOL_TOOL_LOG_LEVEL" #define PMEMPOOL_TOOL_LOG_FILE_VAR "PMEMPOOL_TOOL_LOG_FILE" /* * command -- struct for pmempool commands definition */ struct command { const char *name; const char *brief; int (*func)(const char *, int, char *[]); void (*help)(const char *); }; static const struct command *get_command(const char *cmd_str); static void print_help(const char *appname); /* * long_options -- pmempool command line arguments */ static const struct option long_options[] = { {"version", no_argument, NULL, 'V'}, {"help", no_argument, NULL, 'h'}, {NULL, 0, NULL, 0 }, }; /* * help_help -- prints help message for help command */ static void help_help(const char *appname) { printf("NOTE: pmem blk and log pools are deprecated\n"); printf("Usage: %s help \n", appname); } /* * help_func -- prints help message for specified command */ static int help_func(const char *appname, int argc, char *argv[]) { if (argc > 1) { char *cmd_str = argv[1]; const struct command *cmdp = get_command(cmd_str); if (cmdp && cmdp->help) { cmdp->help(appname); return 0; } else { outv_err("No help text for '%s' command\n", cmd_str); return -1; } } else { print_help(appname); return -1; } } /* * commands -- definition of all pmempool commands */ static const struct command commands[] = { { .name = "info", .brief = "print information and statistics about a pool", .func = pmempool_info_func, .help = pmempool_info_help, }, { .name = "create", .brief = "create a pool", .func = pmempool_create_func, .help = pmempool_create_help, }, { .name = "dump", .brief = "dump user data from a pool", .func = pmempool_dump_func, .help = pmempool_dump_help, }, { .name = "check", .brief = "check consistency of a pool", .func = pmempool_check_func, .help = pmempool_check_help, }, { .name = "rm", .brief = "remove pool or poolset", .func = pmempool_rm_func, .help = pmempool_rm_help, }, { .name = "convert", .brief = "perform pool layout conversion", .func = pmempool_convert_func, .help = pmempool_convert_help, }, { .name = "sync", .brief = "synchronize data between replicas", .func = pmempool_sync_func, .help = pmempool_sync_help, }, { .name = "transform", .brief = "modify internal structure of a poolset", .func = pmempool_transform_func, .help = pmempool_transform_help, }, { .name = "feature", .brief = "toggle / query pool features", .func = pmempool_feature_func, .help = pmempool_feature_help, }, { .name = "help", .brief = "print help text about a command", .func = help_func, .help = help_help, }, }; /* * number of pmempool commands */ #define COMMANDS_NUMBER (sizeof(commands) / sizeof(commands[0])) /* * print_version -- prints pmempool version message */ static void print_version(const char *appname) { printf("NOTE: pmem blk and log pools are deprecated\n"); printf("%s %s\n", appname, SRCVERSION); } /* * print_usage -- prints pmempool usage message */ static void print_usage(const char *appname) { printf("usage: %s [--version] [--help] []\n", appname); } /* * print_help -- prints pmempool help message */ static void print_help(const char *appname) { print_usage(appname); print_version(appname); printf("\n"); printf("Options:\n"); printf(" -V, --version display version\n"); printf(" -h, --help display this help and exit\n"); printf("\n"); printf("The available commands are:\n"); unsigned i; for (i = 0; i < COMMANDS_NUMBER; i++) { const char *format = (strlen(commands[i].name) / 8) ? "%s\t- %s\n" : "%s\t\t- %s\n"; printf(format, commands[i].name, commands[i].brief); } printf("\n"); printf("For complete documentation see %s(1) manual page.\n", appname); } /* * get_command -- returns command for specified command name */ static const struct command * get_command(const char *cmd_str) { unsigned i; for (i = 0; i < COMMANDS_NUMBER; i++) { if (strcmp(cmd_str, commands[i].name) == 0) return &commands[i]; } return NULL; } int main(int argc, char *argv[]) { int opt; int option_index; int ret = 0; #ifdef _WIN32 util_suppress_errmsg(); wchar_t **wargv = CommandLineToArgvW(GetCommandLineW(), &argc); for (int i = 0; i < argc; i++) { argv[i] = util_toUTF8(wargv[i]); if (argv[i] == NULL) { for (i--; i >= 0; i--) free(argv[i]); outv_err("Error during arguments conversion\n"); return 1; } } #endif common_init(PMEMPOOL_TOOL_LOG_PREFIX, PMEMPOOL_TOOL_LOG_LEVEL_VAR, PMEMPOOL_TOOL_LOG_FILE_VAR, 0 /* major version */, 0 /* minor version */); if (argc < 2) { print_usage(APPNAME); goto end; } while ((opt = getopt_long(2, argv, "Vh", long_options, &option_index)) != -1) { switch (opt) { case 'V': print_version(APPNAME); goto end; case 'h': print_help(APPNAME); goto end; default: print_usage(APPNAME); ret = 1; goto end; } } char *cmd_str = argv[optind]; const struct command *cmdp = get_command(cmd_str); if (cmdp) { ret = cmdp->func(APPNAME, argc - 1, argv + 1); } else { outv_err("'%s' -- unknown command\n", cmd_str); ret = 1; } end: common_fini(); #ifdef _WIN32 for (int i = argc; i > 0; i--) free(argv[i - 1]); #endif if (ret) return 1; return 0; } pmdk-1.13.1/src/tools/pmempool/info.h0000664000000000000000000001220714435627501016124 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2014-2023, Intel Corporation */ /* * info.h -- pmempool info command header file */ #include "vec.h" #define BLK_DEPR_STR "Libpmemblk is deprecated." #define LOG_DEPR_STR "Libpmemlog is deprecated." #ifdef _WIN32 #define PMEMBLK_DEPR_ATTR __declspec(deprecated(BLK_DEPR_STR)) #define PMEMLOG_DEPR_ATTR __declspec(deprecated(LOG_DEPR_STR)) #define WIN_DEPR_STR "Windows support is deprecated." #define WIN_DEPR_ATTR __declspec(deprecated(WIN_DEPR_STR)) #else #define PMEMBLK_DEPR_ATTR __attribute__((deprecated(BLK_DEPR_STR))) #define PMEMLOG_DEPR_ATTR __attribute__((deprecated(LOG_DEPR_STR))) #endif /* * Verbose levels used in application: * * VERBOSE_DEFAULT: * Default value for application's verbosity level. * This is also set for data structures which should be * printed without any command line argument. * * VERBOSE_MAX: * Maximum value for application's verbosity level. * This value is used when -v command line argument passed. * * VERBOSE_SILENT: * This value is higher than VERBOSE_MAX and it is used only * for verbosity levels of data structures which should _not_ be * printed without specified command line arguments. */ #define VERBOSE_SILENT 0 #define VERBOSE_DEFAULT 1 #define VERBOSE_MAX 2 /* * print_bb_e -- printing bad blocks options */ enum print_bb_e { PRINT_BAD_BLOCKS_NOT_SET, PRINT_BAD_BLOCKS_NO, PRINT_BAD_BLOCKS_YES, PRINT_BAD_BLOCKS_MAX }; /* * pmempool_info_args -- structure for storing command line arguments */ struct pmempool_info_args { char *file; /* input file */ unsigned col_width; /* column width for printing fields */ bool human; /* sizes in human-readable formats */ bool force; /* force parsing pool */ enum print_bb_e badblocks; /* print bad blocks */ pmem_pool_type_t type; /* forced pool type */ bool use_range; /* use range for blocks */ struct ranges ranges; /* range of block/chunks to dump */ int vlevel; /* verbosity level */ int vdata; /* verbosity level for data dump */ int vhdrdump; /* verbosity level for headers hexdump */ int vstats; /* verbosity level for statistics */ struct { size_t walk; /* data chunk size */ } log; /* deprecated */ struct { int vmap; /* verbosity level for BTT Map */ int vflog; /* verbosity level for BTT FLOG */ int vbackup; /* verbosity level for BTT Info backup */ bool skip_zeros; /* skip blocks marked with zero flag */ bool skip_error; /* skip blocks marked with error flag */ bool skip_no_flag; /* skip blocks not marked with any flag */ } blk; /* deprecated */ struct { int vlanes; /* verbosity level for lanes */ int vroot; int vobjects; int valloc; int voobhdr; int vheap; int vzonehdr; int vchunkhdr; int vbitmap; bool lanes_recovery; bool ignore_empty_obj; uint64_t chunk_types; size_t replica; struct ranges lane_ranges; struct ranges type_ranges; struct ranges zone_ranges; struct ranges chunk_ranges; } obj; }; /* * pmem_blk_stats -- structure with statistics for pmemblk (DEPRECATED) */ struct pmem_blk_stats { uint32_t total; /* number of processed blocks */ uint32_t zeros; /* number of blocks marked by zero flag */ uint32_t errors; /* number of blocks marked by error flag */ uint32_t noflag; /* number of blocks not marked with any flag */ }; struct pmem_obj_class_stats { uint64_t n_units; uint64_t n_used; uint64_t unit_size; uint64_t alignment; uint32_t nallocs; uint16_t flags; }; struct pmem_obj_zone_stats { uint64_t n_chunks; uint64_t n_chunks_type[MAX_CHUNK_TYPE]; uint64_t size_chunks; uint64_t size_chunks_type[MAX_CHUNK_TYPE]; VEC(, struct pmem_obj_class_stats) class_stats; }; struct pmem_obj_type_stats { PMDK_TAILQ_ENTRY(pmem_obj_type_stats) next; uint64_t type_num; uint64_t n_objects; uint64_t n_bytes; }; struct pmem_obj_stats { uint64_t n_total_objects; uint64_t n_total_bytes; uint64_t n_zones; uint64_t n_zones_used; struct pmem_obj_zone_stats *zone_stats; PMDK_TAILQ_HEAD(obj_type_stats_head, pmem_obj_type_stats) type_stats; }; /* * pmem_info -- context for pmeminfo application */ struct pmem_info { const char *file_name; /* current file name */ struct pool_set_file *pfile; struct pmempool_info_args args; /* arguments parsed from command line */ struct options *opts; struct pool_set *poolset; pmem_pool_type_t type; struct pmem_pool_params params; struct { struct pmem_blk_stats stats; } blk; /* deprecated */ struct { struct pmemobjpool *pop; struct palloc_heap *heap; struct alloc_class_collection *alloc_classes; size_t size; struct pmem_obj_stats stats; uint64_t uuid_lo; uint64_t objid; } obj; }; #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmempool_info_func(const char *appname, int argc, char *argv[]); #ifdef _WIN32 WIN_DEPR_ATTR #endif void pmempool_info_help(const char *appname); #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmempool_info_read(struct pmem_info *pip, void *buff, size_t nbytes, uint64_t off); PMEMBLK_DEPR_ATTR int pmempool_info_blk(struct pmem_info *pip); PMEMLOG_DEPR_ATTR int pmempool_info_log(struct pmem_info *pip); #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmempool_info_obj(struct pmem_info *pip); PMEMBLK_DEPR_ATTR int pmempool_info_btt(struct pmem_info *pip); pmdk-1.13.1/src/tools/pmempool/transform.h0000664000000000000000000000074514435627501017210 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2016-2023, Intel Corporation */ /* * transform.h -- pmempool transform command header file */ #ifdef _WIN32 #define WIN_DEPR_STR "Windows support is deprecated." #define WIN_DEPR_ATTR __declspec(deprecated(WIN_DEPR_STR)) #endif #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmempool_transform_func(const char *appname, int argc, char *argv[]); #ifdef _WIN32 WIN_DEPR_ATTR #endif void pmempool_transform_help(const char *appname); pmdk-1.13.1/src/tools/pmempool/pmempool.vcxproj0000664000000000000000000002112014435627501020257 0ustar rootroot Debug x64 Release x64 {492baa3d-0d5d-478e-9765-500463ae69aa} {f7c6c6b6-4142-4c82-8699-4a9d8183181b} {0b1818eb-bdc8-4865-964f-db8bf05cfd86} {1baa1617-93ae-4196-8a1a-bd492fb18aef} {cf9a0883-6334-44c7-ac29-349468c78e27} {9e9e3d25-2139-4a5d-9200-18148ddead45} {9186eac4-2f34-4f17-b940-6585d7869bcd} {7DC3B3DD-73ED-4602-9AF3-8D7053620DEA} Win32Proj pmempool 10.0.22000.0 Application true v143 NotSet Application false v143 false NotSet $(SolutionDir)\core;$(SolutionDir)\common;$(SolutionDir)\test\unittest;$(SolutionDir)\windows\include;$(SolutionDir)\include;$(SolutionDir)\windows\getopt;$(SolutionDir)\libpmemlog;$(SolutionDir)\libpmemblk;$(SolutionDir)\libpmemobj;$(SolutionDir)\libpmem2;$(IncludePath) $(SolutionDir)$(Platform)\$(Configuration)\libs\ $(SolutionDir)\core;$(SolutionDir)\common;$(SolutionDir)\test\unittest;$(SolutionDir)\windows\include;$(SolutionDir)\include;$(SolutionDir)\windows\getopt;$(SolutionDir)\libpmemlog;$(SolutionDir)\libpmemblk;$(SolutionDir)\libpmemobj;$(SolutionDir)\libpmem2;$(IncludePath) $(SolutionDir)$(Platform)\$(Configuration)\libs\ NotUsing Level3 PMDK_UTF8_API;SDS_ENABLED; NTDDI_VERSION=NTDDI_WIN10_RS1;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) platform.h CompileAsC false true Console Shlwapi.lib;%(AdditionalDependencies) true Debug _DEBUG Level3 NotUsing true PMDK_UTF8_API;SDS_ENABLED; NTDDI_VERSION=NTDDI_WIN10_RS1;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) CompileAsC platform.h false true Console Shlwapi.lib;%(AdditionalDependencies) true DebugFastLink pmdk-1.13.1/src/tools/pmempool/info_obj.c0000664000000000000000000005716114435627501016761 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2019, Intel Corporation */ /* * info_obj.c -- pmempool info command source file for obj pool */ #include #include #include #include #include #include #include #include #include "alloc_class.h" #include "set.h" #include "common.h" #include "output.h" #include "info.h" #include "util.h" #define BITMAP_BUFF_SIZE 1024 #define OFF_TO_PTR(pop, off) ((void *)((uintptr_t)(pop) + (off))) #define PTR_TO_OFF(pop, ptr) ((uintptr_t)(ptr) - (uintptr_t)(pop)) /* * lane_need_recovery -- return 1 if lane section needs recovery */ static int lane_need_recovery(struct pmem_info *pip, struct lane_layout *lane) { return ulog_recovery_needed((struct ulog *)&lane->external, 1) || ulog_recovery_needed((struct ulog *)&lane->internal, 1) || ulog_recovery_needed((struct ulog *)&lane->undo, 0); } #define RUN_BITMAP_SEPARATOR_DISTANCE 8 /* * get_bitmap_str -- get bitmap single value string */ static const char * get_bitmap_str(uint64_t val, unsigned values) { static char buff[BITMAP_BUFF_SIZE]; unsigned j = 0; for (unsigned i = 0; i < values && j < BITMAP_BUFF_SIZE - 3; i++) { buff[j++] = ((val & ((uint64_t)1 << i)) ? 'x' : '.'); if ((i + 1) % RUN_BITMAP_SEPARATOR_DISTANCE == 0) buff[j++] = ' '; } buff[j] = '\0'; return buff; } /* * pmem_obj_stats_get_type -- get stats for specified type number */ static struct pmem_obj_type_stats * pmem_obj_stats_get_type(struct pmem_obj_stats *stats, uint64_t type_num) { struct pmem_obj_type_stats *type; struct pmem_obj_type_stats *type_dest = NULL; PMDK_TAILQ_FOREACH(type, &stats->type_stats, next) { if (type->type_num == type_num) return type; if (!type_dest && type->type_num > type_num) type_dest = type; } type = calloc(1, sizeof(*type)); if (!type) { outv_err("cannot allocate memory for type stats\n"); exit(EXIT_FAILURE); } type->type_num = type_num; if (type_dest) PMDK_TAILQ_INSERT_BEFORE(type_dest, type, next); else PMDK_TAILQ_INSERT_TAIL(&stats->type_stats, type, next); return type; } struct info_obj_redo_args { int v; size_t i; struct pmem_info *pip; }; /* * info_obj_redo_entry - print redo log entry info */ static int info_obj_redo_entry(struct ulog_entry_base *e, void *arg, const struct pmem_ops *p_ops) { struct info_obj_redo_args *a = arg; struct ulog_entry_val *ev; struct ulog_entry_buf *eb; switch (ulog_entry_type(e)) { case ULOG_OPERATION_AND: case ULOG_OPERATION_OR: case ULOG_OPERATION_SET: ev = (struct ulog_entry_val *)e; outv(a->v, "%010zu: " "Offset: 0x%016jx " "Value: 0x%016jx ", a->i++, ulog_entry_offset(e), ev->value); break; case ULOG_OPERATION_BUF_CPY: case ULOG_OPERATION_BUF_SET: eb = (struct ulog_entry_buf *)e; outv(a->v, "%010zu: " "Offset: 0x%016jx " "Size: %s ", a->i++, ulog_entry_offset(e), out_get_size_str(eb->size, a->pip->args.human)); break; default: ASSERT(0); /* unreachable */ } return 0; } /* * info_obj_redo -- print ulog log entries */ static void info_obj_ulog(struct pmem_info *pip, int v, struct ulog *ulog, const struct pmem_ops *ops) { outv_title(v, "Log entries"); struct info_obj_redo_args args = {v, 0, pip}; ulog_foreach_entry(ulog, info_obj_redo_entry, &args, ops); } /* * info_obj_alloc_hdr -- print allocation header */ static void info_obj_alloc_hdr(struct pmem_info *pip, int v, const struct memory_block *m) { outv_title(v, "Allocation Header"); outv_field(v, "Size", "%s", out_get_size_str(m->m_ops->get_user_size(m), pip->args.human)); outv_field(v, "Extra", "%lu", m->m_ops->get_extra(m)); outv_field(v, "Flags", "0x%x", m->m_ops->get_flags(m)); } /* * info_obj_object_hdr -- print object headers and data */ static void info_obj_object_hdr(struct pmem_info *pip, int v, int vid, const struct memory_block *m, uint64_t id) { struct pmemobjpool *pop = pip->obj.pop; void *data = m->m_ops->get_user_data(m); outv_nl(vid); outv_field(vid, "Object", "%lu", id); outv_field(vid, "Offset", "0x%016lx", PTR_TO_OFF(pop, data)); int vahdr = v && pip->args.obj.valloc; int voobh = v && pip->args.obj.voobhdr; outv_indent(vahdr || voobh, 1); info_obj_alloc_hdr(pip, vahdr, m); outv_hexdump(v && pip->args.vdata, data, m->m_ops->get_real_size(m), PTR_TO_OFF(pip->obj.pop, data), 1); outv_indent(vahdr || voobh, -1); } /* * info_obj_lane_section -- print lane's section */ static void info_obj_lane(struct pmem_info *pip, int v, struct lane_layout *lane) { struct pmem_ops p_ops; p_ops.base = pip->obj.pop; outv_title(v, "Undo Log"); outv_indent(v, 1); info_obj_ulog(pip, v, (struct ulog *)&lane->undo, &p_ops); outv_indent(v, -1); outv_nl(v); outv_title(v, "Internal Undo Log"); outv_indent(v, 1); info_obj_ulog(pip, v, (struct ulog *)&lane->internal, &p_ops); outv_indent(v, -1); outv_title(v, "External Undo Log"); outv_indent(v, 1); info_obj_ulog(pip, v, (struct ulog *)&lane->external, &p_ops); outv_indent(v, -1); } /* * info_obj_lanes -- print lanes structures */ static void info_obj_lanes(struct pmem_info *pip) { int v = pip->args.obj.vlanes; if (!outv_check(v)) return; struct pmemobjpool *pop = pip->obj.pop; /* * Iterate through all lanes from specified range and print * specified sections. */ struct lane_layout *lanes = (void *)((char *)pip->obj.pop + pop->lanes_offset); struct range *curp = NULL; FOREACH_RANGE(curp, &pip->args.obj.lane_ranges) { for (uint64_t i = curp->first; i <= curp->last && i < pop->nlanes; i++) { /* For -R check print lane only if needs recovery */ if (pip->args.obj.lanes_recovery && !lane_need_recovery(pip, &lanes[i])) continue; outv_title(v, "Lane %" PRIu64, i); outv_indent(v, 1); info_obj_lane(pip, v, &lanes[i]); outv_indent(v, -1); } } } /* * info_obj_heap -- print pmemobj heap headers */ static void info_obj_heap(struct pmem_info *pip) { int v = pip->args.obj.vheap; struct pmemobjpool *pop = pip->obj.pop; struct heap_layout *layout = OFF_TO_PTR(pop, pop->heap_offset); struct heap_header *heap = &layout->header; outv(v, "\nPMEMOBJ Heap Header:\n"); outv_hexdump(v && pip->args.vhdrdump, heap, sizeof(*heap), pop->heap_offset, 1); outv_field(v, "Signature", "%s", heap->signature); outv_field(v, "Major", "%ld", heap->major); outv_field(v, "Minor", "%ld", heap->minor); outv_field(v, "Chunk size", "%s", out_get_size_str(heap->chunksize, pip->args.human)); outv_field(v, "Chunks per zone", "%ld", heap->chunks_per_zone); outv_field(v, "Checksum", "%s", out_get_checksum(heap, sizeof(*heap), &heap->checksum, 0)); } /* * info_obj_zone -- print information about zone */ static void info_obj_zone_hdr(struct pmem_info *pip, int v, struct zone_header *zone) { outv_hexdump(v && pip->args.vhdrdump, zone, sizeof(*zone), PTR_TO_OFF(pip->obj.pop, zone), 1); outv_field(v, "Magic", "%s", out_get_zone_magic_str(zone->magic)); outv_field(v, "Size idx", "%u", zone->size_idx); } /* * info_obj_object -- print information about object */ static void info_obj_object(struct pmem_info *pip, const struct memory_block *m, uint64_t objid) { if (!util_ranges_contain(&pip->args.ranges, objid)) return; uint64_t type_num = m->m_ops->get_extra(m); if (!util_ranges_contain(&pip->args.obj.type_ranges, type_num)) return; uint64_t real_size = m->m_ops->get_real_size(m); pip->obj.stats.n_total_objects++; pip->obj.stats.n_total_bytes += real_size; struct pmem_obj_type_stats *type_stats = pmem_obj_stats_get_type(&pip->obj.stats, type_num); type_stats->n_objects++; type_stats->n_bytes += real_size; int vid = pip->args.obj.vobjects; int v = pip->args.obj.vobjects; outv_indent(v, 1); info_obj_object_hdr(pip, v, vid, m, objid); outv_indent(v, -1); } /* * info_obj_run_bitmap -- print chunk run's bitmap */ static void info_obj_run_bitmap(int v, struct run_bitmap *b) { /* print only used values for lower verbosity */ uint32_t i; for (i = 0; i < b->nbits / RUN_BITS_PER_VALUE; i++) outv(v, "%s\n", get_bitmap_str(b->values[i], RUN_BITS_PER_VALUE)); unsigned mod = b->nbits % RUN_BITS_PER_VALUE; if (mod != 0) { outv(v, "%s\n", get_bitmap_str(b->values[i], mod)); } } /* * info_obj_memblock_is_root -- (internal) checks whether the object is root */ static int info_obj_memblock_is_root(struct pmem_info *pip, const struct memory_block *m) { uint64_t roff = pip->obj.pop->root_offset; if (roff == 0) return 0; struct memory_block rm = memblock_from_offset(pip->obj.heap, roff); return MEMORY_BLOCK_EQUALS(*m, rm); } /* * info_obj_run_cb -- (internal) run object callback */ static int info_obj_run_cb(const struct memory_block *m, void *arg) { struct pmem_info *pip = arg; if (info_obj_memblock_is_root(pip, m)) return 0; info_obj_object(pip, m, pip->obj.objid++); return 0; } static struct pmem_obj_class_stats * info_obj_class_stats_get_or_insert(struct pmem_obj_zone_stats *stats, uint64_t unit_size, uint64_t alignment, uint32_t nallocs, uint16_t flags) { struct pmem_obj_class_stats *cstats; VEC_FOREACH_BY_PTR(cstats, &stats->class_stats) { if (cstats->alignment == alignment && cstats->flags == flags && cstats->nallocs == nallocs && cstats->unit_size == unit_size) return cstats; } struct pmem_obj_class_stats s = {0, 0, unit_size, alignment, nallocs, flags}; if (VEC_PUSH_BACK(&stats->class_stats, s) != 0) return NULL; return &VEC_BACK(&stats->class_stats); } /* * info_obj_chunk -- print chunk info */ static void info_obj_chunk(struct pmem_info *pip, uint64_t c, uint64_t z, struct chunk_header *chunk_hdr, struct chunk *chunk, struct pmem_obj_zone_stats *stats) { int v = pip->args.obj.vchunkhdr; outv(v, "\n"); outv_field(v, "Chunk", "%lu", c); struct pmemobjpool *pop = pip->obj.pop; outv_hexdump(v && pip->args.vhdrdump, chunk_hdr, sizeof(*chunk_hdr), PTR_TO_OFF(pop, chunk_hdr), 1); outv_field(v, "Type", "%s", out_get_chunk_type_str(chunk_hdr->type)); outv_field(v, "Flags", "0x%x %s", chunk_hdr->flags, out_get_chunk_flags(chunk_hdr->flags)); outv_field(v, "Size idx", "%u", chunk_hdr->size_idx); struct memory_block m = MEMORY_BLOCK_NONE; m.zone_id = (uint32_t)z; m.chunk_id = (uint32_t)c; m.size_idx = (uint32_t)chunk_hdr->size_idx; memblock_rebuild_state(pip->obj.heap, &m); if (chunk_hdr->type == CHUNK_TYPE_USED || chunk_hdr->type == CHUNK_TYPE_FREE) { VEC_FRONT(&stats->class_stats).n_units += chunk_hdr->size_idx; if (chunk_hdr->type == CHUNK_TYPE_USED) { VEC_FRONT(&stats->class_stats).n_used += chunk_hdr->size_idx; /* skip root object */ if (!info_obj_memblock_is_root(pip, &m)) { info_obj_object(pip, &m, pip->obj.objid++); } } } else if (chunk_hdr->type == CHUNK_TYPE_RUN) { struct chunk_run *run = (struct chunk_run *)chunk; outv_hexdump(v && pip->args.vhdrdump, run, sizeof(run->hdr.block_size) + sizeof(run->hdr.alignment), PTR_TO_OFF(pop, run), 1); struct run_bitmap bitmap; m.m_ops->get_bitmap(&m, &bitmap); struct pmem_obj_class_stats *cstats = info_obj_class_stats_get_or_insert(stats, run->hdr.block_size, run->hdr.alignment, bitmap.nbits, chunk_hdr->flags); if (cstats == NULL) { outv_err("out of memory, can't allocate statistics"); return; } outv_field(v, "Block size", "%s", out_get_size_str(run->hdr.block_size, pip->args.human)); uint32_t units = bitmap.nbits; uint32_t free_space = 0; uint32_t max_free_block = 0; m.m_ops->calc_free(&m, &free_space, &max_free_block); uint32_t used = units - free_space; cstats->n_units += units; cstats->n_used += used; outv_field(v, "Bitmap", "%u / %u", used, units); info_obj_run_bitmap(v && pip->args.obj.vbitmap, &bitmap); m.m_ops->iterate_used(&m, info_obj_run_cb, pip); } } /* * info_obj_zone_chunks -- print chunk headers from specified zone */ static void info_obj_zone_chunks(struct pmem_info *pip, struct zone *zone, uint64_t z, struct pmem_obj_zone_stats *stats) { VEC_INIT(&stats->class_stats); struct pmem_obj_class_stats default_class_stats = {0, 0, CHUNKSIZE, 0, 0, 0}; VEC_PUSH_BACK(&stats->class_stats, default_class_stats); uint64_t c = 0; while (c < zone->header.size_idx) { enum chunk_type type = zone->chunk_headers[c].type; uint64_t size_idx = zone->chunk_headers[c].size_idx; if (util_ranges_contain(&pip->args.obj.chunk_ranges, c)) { if (pip->args.obj.chunk_types & (1ULL << type)) { stats->n_chunks++; stats->n_chunks_type[type]++; stats->size_chunks += size_idx; stats->size_chunks_type[type] += size_idx; info_obj_chunk(pip, c, z, &zone->chunk_headers[c], &zone->chunks[c], stats); } if (size_idx > 1 && type != CHUNK_TYPE_RUN && pip->args.obj.chunk_types & (1 << CHUNK_TYPE_FOOTER)) { size_t f = c + size_idx - 1; info_obj_chunk(pip, f, z, &zone->chunk_headers[f], &zone->chunks[f], stats); } } c += size_idx; } } /* * info_obj_root_obj -- print root object */ static void info_obj_root_obj(struct pmem_info *pip) { int v = pip->args.obj.vroot; struct pmemobjpool *pop = pip->obj.pop; if (!pop->root_offset) { outv(v, "\nNo root object...\n"); return; } outv_title(v, "Root object"); outv_field(v, "Offset", "0x%016zx", pop->root_offset); uint64_t root_size = pop->root_size; outv_field(v, "Size", "%s", out_get_size_str(root_size, pip->args.human)); struct memory_block m = memblock_from_offset( pip->obj.heap, pop->root_offset); /* do not print object id and offset for root object */ info_obj_object_hdr(pip, v, VERBOSE_SILENT, &m, 0); } /* * info_obj_zones -- print zones and chunks */ static void info_obj_zones_chunks(struct pmem_info *pip) { if (!outv_check(pip->args.obj.vheap) && !outv_check(pip->args.vstats) && !outv_check(pip->args.obj.vobjects)) return; struct pmemobjpool *pop = pip->obj.pop; struct heap_layout *layout = OFF_TO_PTR(pop, pop->heap_offset); size_t maxzone = util_heap_max_zone(pop->heap_size); pip->obj.stats.n_zones = maxzone; pip->obj.stats.zone_stats = calloc(maxzone, sizeof(struct pmem_obj_zone_stats)); if (!pip->obj.stats.zone_stats) err(1, "Cannot allocate memory for zone stats"); for (size_t i = 0; i < maxzone; i++) { struct zone *zone = ZID_TO_ZONE(layout, i); if (util_ranges_contain(&pip->args.obj.zone_ranges, i)) { int vvv = pip->args.obj.vheap && (pip->args.obj.vzonehdr || pip->args.obj.vchunkhdr); outv_title(vvv, "Zone %zu", i); if (zone->header.magic == ZONE_HEADER_MAGIC) pip->obj.stats.n_zones_used++; info_obj_zone_hdr(pip, pip->args.obj.vheap && pip->args.obj.vzonehdr, &zone->header); outv_indent(vvv, 1); info_obj_zone_chunks(pip, zone, i, &pip->obj.stats.zone_stats[i]); outv_indent(vvv, -1); } } } /* * info_obj_descriptor -- print pmemobj descriptor */ static void info_obj_descriptor(struct pmem_info *pip) { int v = VERBOSE_DEFAULT; if (!outv_check(v)) return; outv(v, "\nPMEM OBJ Header:\n"); struct pmemobjpool *pop = pip->obj.pop; uint8_t *hdrptr = (uint8_t *)pop + sizeof(pop->hdr); size_t hdrsize = sizeof(*pop) - sizeof(pop->hdr); size_t hdroff = sizeof(pop->hdr); outv_hexdump(pip->args.vhdrdump, hdrptr, hdrsize, hdroff, 1); /* check if layout is zeroed */ char *layout = util_check_memory((uint8_t *)pop->layout, sizeof(pop->layout), 0) ? pop->layout : "(null)"; /* address for checksum */ void *dscp = (void *)((uintptr_t)(pop) + sizeof(struct pool_hdr)); outv_field(v, "Layout", "%s", layout); outv_field(v, "Lanes offset", "0x%lx", pop->lanes_offset); outv_field(v, "Number of lanes", "%lu", pop->nlanes); outv_field(v, "Heap offset", "0x%lx", pop->heap_offset); outv_field(v, "Heap size", "%lu", pop->heap_size); outv_field(v, "Checksum", "%s", out_get_checksum(dscp, OBJ_DSC_P_SIZE, &pop->checksum, 0)); outv_field(v, "Root offset", "0x%lx", pop->root_offset); /* run id with -v option */ outv_field(v + 1, "Run id", "%lu", pop->run_id); } /* * info_obj_stats_objjects -- print objects' statistics */ static void info_obj_stats_objects(struct pmem_info *pip, int v, struct pmem_obj_stats *stats) { outv_field(v, "Number of objects", "%lu", stats->n_total_objects); outv_field(v, "Number of bytes", "%s", out_get_size_str( stats->n_total_bytes, pip->args.human)); outv_title(v, "Objects by type"); outv_indent(v, 1); struct pmem_obj_type_stats *type_stats; PMDK_TAILQ_FOREACH(type_stats, &pip->obj.stats.type_stats, next) { if (!type_stats->n_objects) continue; double n_objects_perc = 100.0 * (double)type_stats->n_objects / (double)stats->n_total_objects; double n_bytes_perc = 100.0 * (double)type_stats->n_bytes / (double)stats->n_total_bytes; outv_nl(v); outv_field(v, "Type number", "%lu", type_stats->type_num); outv_field(v, "Number of objects", "%lu [%s]", type_stats->n_objects, out_get_percentage(n_objects_perc)); outv_field(v, "Number of bytes", "%s [%s]", out_get_size_str( type_stats->n_bytes, pip->args.human), out_get_percentage(n_bytes_perc)); } outv_indent(v, -1); } /* * info_boj_stats_alloc_classes -- print allocation classes' statistics */ static void info_obj_stats_alloc_classes(struct pmem_info *pip, int v, struct pmem_obj_zone_stats *stats) { uint64_t total_bytes = 0; uint64_t total_used = 0; outv_indent(v, 1); struct pmem_obj_class_stats *cstats; VEC_FOREACH_BY_PTR(cstats, &stats->class_stats) { if (cstats->n_units == 0) continue; double used_perc = 100.0 * (double)cstats->n_used / (double)cstats->n_units; outv_nl(v); outv_field(v, "Unit size", "%s", out_get_size_str( cstats->unit_size, pip->args.human)); outv_field(v, "Units", "%lu", cstats->n_units); outv_field(v, "Used units", "%lu [%s]", cstats->n_used, out_get_percentage(used_perc)); uint64_t bytes = cstats->unit_size * cstats->n_units; uint64_t used = cstats->unit_size * cstats->n_used; total_bytes += bytes; total_used += used; double used_bytes_perc = 100.0 * (double)used / (double)bytes; outv_field(v, "Bytes", "%s", out_get_size_str(bytes, pip->args.human)); outv_field(v, "Used bytes", "%s [%s]", out_get_size_str(used, pip->args.human), out_get_percentage(used_bytes_perc)); } outv_indent(v, -1); double used_bytes_perc = total_bytes ? 100.0 * (double)total_used / (double)total_bytes : 0.0; outv_nl(v); outv_field(v, "Total bytes", "%s", out_get_size_str(total_bytes, pip->args.human)); outv_field(v, "Total used bytes", "%s [%s]", out_get_size_str(total_used, pip->args.human), out_get_percentage(used_bytes_perc)); } /* * info_obj_stats_chunks -- print chunks' statistics */ static void info_obj_stats_chunks(struct pmem_info *pip, int v, struct pmem_obj_zone_stats *stats) { outv_field(v, "Number of chunks", "%lu", stats->n_chunks); outv_indent(v, 1); for (unsigned type = 0; type < MAX_CHUNK_TYPE; type++) { double type_perc = 100.0 * (double)stats->n_chunks_type[type] / (double)stats->n_chunks; if (stats->n_chunks_type[type]) { outv_field(v, out_get_chunk_type_str(type), "%lu [%s]", stats->n_chunks_type[type], out_get_percentage(type_perc)); } } outv_indent(v, -1); outv_nl(v); outv_field(v, "Total chunks size", "%s", out_get_size_str( stats->size_chunks, pip->args.human)); outv_indent(v, 1); for (unsigned type = 0; type < MAX_CHUNK_TYPE; type++) { double type_perc = 100.0 * (double)stats->size_chunks_type[type] / (double)stats->size_chunks; if (stats->size_chunks_type[type]) { outv_field(v, out_get_chunk_type_str(type), "%lu [%s]", stats->size_chunks_type[type], out_get_percentage(type_perc)); } } outv_indent(v, -1); } /* * info_obj_add_zone_stats -- add stats to total */ static void info_obj_add_zone_stats(struct pmem_obj_zone_stats *total, struct pmem_obj_zone_stats *stats) { total->n_chunks += stats->n_chunks; total->size_chunks += stats->size_chunks; for (int type = 0; type < MAX_CHUNK_TYPE; type++) { total->n_chunks_type[type] += stats->n_chunks_type[type]; total->size_chunks_type[type] += stats->size_chunks_type[type]; } struct pmem_obj_class_stats *cstats; VEC_FOREACH_BY_PTR(cstats, &stats->class_stats) { struct pmem_obj_class_stats *ctotal = info_obj_class_stats_get_or_insert(total, cstats->unit_size, cstats->alignment, cstats->nallocs, cstats->flags); if (ctotal == NULL) { outv_err("out of memory, can't allocate statistics"); return; } ctotal->n_units += cstats->n_units; ctotal->n_used += cstats->n_used; } } /* * info_obj_stats_zones -- print zones' statistics */ static void info_obj_stats_zones(struct pmem_info *pip, int v, struct pmem_obj_stats *stats, struct pmem_obj_zone_stats *total) { double used_zones_perc = 100.0 * (double)stats->n_zones_used / (double)stats->n_zones; outv_field(v, "Number of zones", "%lu", stats->n_zones); outv_field(v, "Number of used zones", "%lu [%s]", stats->n_zones_used, out_get_percentage(used_zones_perc)); outv_indent(v, 1); for (uint64_t i = 0; i < stats->n_zones_used; i++) { outv_title(v, "Zone %" PRIu64, i); struct pmem_obj_zone_stats *zstats = &stats->zone_stats[i]; info_obj_stats_chunks(pip, v, zstats); outv_title(v, "Zone's allocation classes"); info_obj_stats_alloc_classes(pip, v, zstats); info_obj_add_zone_stats(total, zstats); } outv_indent(v, -1); } /* * info_obj_stats -- print statistics */ static void info_obj_stats(struct pmem_info *pip) { int v = pip->args.vstats; if (!outv_check(v)) return; struct pmem_obj_stats *stats = &pip->obj.stats; struct pmem_obj_zone_stats total; memset(&total, 0, sizeof(total)); outv_title(v, "Statistics"); outv_title(v, "Objects"); info_obj_stats_objects(pip, v, stats); outv_title(v, "Heap"); info_obj_stats_zones(pip, v, stats, &total); if (stats->n_zones_used > 1) { outv_title(v, "Total zone's statistics"); outv_title(v, "Chunks statistics"); info_obj_stats_chunks(pip, v, &total); outv_title(v, "Allocation classes"); info_obj_stats_alloc_classes(pip, v, &total); } VEC_DELETE(&total.class_stats); } static struct pmem_info *Pip; #ifndef _WIN32 static void info_obj_sa_sigaction(int signum, siginfo_t *info, void *context) { uintptr_t offset = (uintptr_t)info->si_addr - (uintptr_t)Pip->obj.pop; outv_err("Invalid offset 0x%lx\n", offset); exit(EXIT_FAILURE); } static struct sigaction info_obj_sigaction = { .sa_sigaction = info_obj_sa_sigaction, .sa_flags = SA_SIGINFO }; #else #define CALL_FIRST 1 static LONG CALLBACK exception_handler(_In_ PEXCEPTION_POINTERS ExceptionInfo) { PEXCEPTION_RECORD record = ExceptionInfo->ExceptionRecord; if (record->ExceptionCode != EXCEPTION_ACCESS_VIOLATION) { return EXCEPTION_CONTINUE_SEARCH; } uintptr_t offset = (uintptr_t)record->ExceptionInformation[1] - (uintptr_t)Pip->obj.pop; outv_err("Invalid offset 0x%lx\n", offset); exit(EXIT_FAILURE); } #endif /* * info_obj -- print information about obj pool type */ int pmempool_info_obj(struct pmem_info *pip) { pip->obj.pop = pool_set_file_map(pip->pfile, 0); if (pip->obj.pop == NULL) return -1; pip->obj.size = pip->pfile->size; struct palloc_heap *heap = calloc(1, sizeof(*heap)); if (heap == NULL) err(1, "Cannot allocate memory for heap data"); heap->layout = OFF_TO_PTR(pip->obj.pop, pip->obj.pop->heap_offset); heap->base = pip->obj.pop; pip->obj.alloc_classes = alloc_class_collection_new(); pip->obj.heap = heap; Pip = pip; #ifndef _WIN32 if (sigaction(SIGSEGV, &info_obj_sigaction, NULL)) { #else if (AddVectoredExceptionHandler(CALL_FIRST, exception_handler) == NULL) { #endif perror("sigaction"); return -1; } pip->obj.uuid_lo = pmemobj_get_uuid_lo(pip->obj.pop); info_obj_descriptor(pip); info_obj_lanes(pip); info_obj_root_obj(pip); info_obj_heap(pip); info_obj_zones_chunks(pip); info_obj_stats(pip); free(heap); alloc_class_collection_delete(pip->obj.alloc_classes); return 0; } pmdk-1.13.1/src/tools/pmempool/info_blk.c0000664000000000000000000003436214435627501016755 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2023, Intel Corporation */ /* * Copyright (c) 2016, Microsoft Corporation. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * * Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * * Neither the name of the copyright holder nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ /* * info_blk.c -- pmempool info command source file for blk pool (DEPRECATED) */ #include #include #include #include #include #include #include "os.h" #include "common.h" #include "output.h" #include "info.h" #include "btt.h" /* * pmempool_info_get_range -- get blocks/data chunk range * * Get range based on command line arguments and maximum value. * Return value: * 0 - range is empty * 1 - range is not empty */ static int pmempool_info_get_range(struct pmem_info *pip, struct range *rangep, struct range *curp, uint32_t max, uint64_t offset) { /* not using range */ if (!pip->args.use_range) { rangep->first = 0; rangep->last = max; return 1; } if (curp->first > offset + max) return 0; if (curp->first >= offset) rangep->first = curp->first - offset; else rangep->first = 0; if (curp->last < offset) return 0; if (curp->last <= offset + max) rangep->last = curp->last - offset; else rangep->last = max; return 1; } /* * info_blk_skip_block -- get action type for block/data chunk * * Return value indicating whether processing block/data chunk * should be skipped. * * Return values: * 0 - continue processing * 1 - skip current block */ static int info_blk_skip_block(struct pmem_info *pip, int is_zero, int is_error) { if (pip->args.blk.skip_no_flag && !is_zero && !is_error) return 1; if (is_zero && pip->args.blk.skip_zeros) return 1; if (is_error && pip->args.blk.skip_error) return 1; return 0; } /* * info_btt_data -- print block data and corresponding flags from map */ static int info_btt_data(struct pmem_info *pip, int v, struct btt_info *infop, uint64_t arena_off, uint64_t offset, uint64_t *countp) { if (!outv_check(v)) return 0; int ret = 0; size_t mapsize = infop->external_nlba * BTT_MAP_ENTRY_SIZE; uint32_t *map = malloc(mapsize); if (!map) err(1, "Cannot allocate memory for BTT map"); uint8_t *block_buff = malloc(infop->external_lbasize); if (!block_buff) err(1, "Cannot allocate memory for pmemblk block buffer"); /* read btt map area */ if (pmempool_info_read(pip, (uint8_t *)map, mapsize, arena_off + infop->mapoff)) { outv_err("wrong BTT Map size or offset\n"); ret = -1; goto error; } uint64_t i; struct range *curp = NULL; struct range range; FOREACH_RANGE(curp, &pip->args.ranges) { if (pmempool_info_get_range(pip, &range, curp, infop->external_nlba - 1, offset) == 0) continue; for (i = range.first; i <= range.last; i++) { uint32_t map_entry = le32toh(map[i]); int is_init = (map_entry & ~BTT_MAP_ENTRY_LBA_MASK) == 0; int is_zero = (map_entry & ~BTT_MAP_ENTRY_LBA_MASK) == BTT_MAP_ENTRY_ZERO || is_init; int is_error = (map_entry & ~BTT_MAP_ENTRY_LBA_MASK) == BTT_MAP_ENTRY_ERROR; uint64_t blockno = is_init ? i : map_entry & BTT_MAP_ENTRY_LBA_MASK; if (info_blk_skip_block(pip, is_zero, is_error)) continue; /* compute block's data address */ uint64_t block_off = arena_off + infop->dataoff + blockno * infop->internal_lbasize; if (pmempool_info_read(pip, block_buff, infop->external_lbasize, block_off)) { outv_err("cannot read %lu block\n", i); ret = -1; goto error; } if (*countp == 0) outv_title(v, "PMEM BLK blocks data"); /* * Print block number, offset and flags * from map entry. */ outv(v, "Block %10lu: offset: %s\n", offset + i, out_get_btt_map_entry(map_entry)); /* dump block's data */ outv_hexdump(v, block_buff, infop->external_lbasize, block_off, 1); *countp = *countp + 1; } } error: free(map); free(block_buff); return ret; } /* * info_btt_map -- print all map entries */ static int info_btt_map(struct pmem_info *pip, int v, struct btt_info *infop, uint64_t arena_off, uint64_t offset, uint64_t *count) { if (!outv_check(v) && !outv_check(pip->args.vstats)) return 0; int ret = 0; size_t mapsize = infop->external_nlba * BTT_MAP_ENTRY_SIZE; uint32_t *map = malloc(mapsize); if (!map) err(1, "Cannot allocate memory for BTT map"); /* read btt map area */ if (pmempool_info_read(pip, (uint8_t *)map, mapsize, arena_off + infop->mapoff)) { outv_err("wrong BTT Map size or offset\n"); ret = -1; goto error; } uint32_t arena_count = 0; uint64_t i; struct range *curp = NULL; struct range range; FOREACH_RANGE(curp, &pip->args.ranges) { if (pmempool_info_get_range(pip, &range, curp, infop->external_nlba - 1, offset) == 0) continue; for (i = range.first; i <= range.last; i++) { uint32_t entry = le32toh(map[i]); int is_zero = (entry & ~BTT_MAP_ENTRY_LBA_MASK) == BTT_MAP_ENTRY_ZERO || (entry & ~BTT_MAP_ENTRY_LBA_MASK) == 0; int is_error = (entry & ~BTT_MAP_ENTRY_LBA_MASK) == BTT_MAP_ENTRY_ERROR; if (info_blk_skip_block(pip, is_zero, is_error) == 0) { if (arena_count == 0) outv_title(v, "PMEM BLK BTT Map"); if (is_zero) pip->blk.stats.zeros++; if (is_error) pip->blk.stats.errors++; if (!is_zero && !is_error) pip->blk.stats.noflag++; pip->blk.stats.total++; arena_count++; (*count)++; outv(v, "%010lu: %s\n", offset + i, out_get_btt_map_entry(entry)); } } } error: free(map); return ret; } /* * info_btt_flog -- print all flog entries */ static int info_btt_flog(struct pmem_info *pip, int v, struct btt_info *infop, uint64_t arena_off) { if (!outv_check(v)) return 0; int ret = 0; struct btt_flog *flogp = NULL; struct btt_flog *flogpp = NULL; uint64_t flog_size = infop->nfree * roundup(2 * sizeof(struct btt_flog), BTT_FLOG_PAIR_ALIGN); flog_size = roundup(flog_size, BTT_ALIGNMENT); uint8_t *buff = malloc(flog_size); if (!buff) err(1, "Cannot allocate memory for FLOG entries"); if (pmempool_info_read(pip, buff, flog_size, arena_off + infop->flogoff)) { outv_err("cannot read BTT FLOG"); ret = -1; goto error; } outv_title(v, "PMEM BLK BTT FLOG"); uint8_t *ptr = buff; uint32_t i; for (i = 0; i < infop->nfree; i++) { flogp = (struct btt_flog *)ptr; flogpp = flogp + 1; btt_flog_convert2h(flogp); btt_flog_convert2h(flogpp); outv(v, "%010d:\n", i); outv_field(v, "LBA", "0x%08x", flogp->lba); outv_field(v, "Old map", "0x%08x: %s", flogp->old_map, out_get_btt_map_entry(flogp->old_map)); outv_field(v, "New map", "0x%08x: %s", flogp->new_map, out_get_btt_map_entry(flogp->new_map)); outv_field(v, "Seq", "0x%x", flogp->seq); outv_field(v, "LBA'", "0x%08x", flogpp->lba); outv_field(v, "Old map'", "0x%08x: %s", flogpp->old_map, out_get_btt_map_entry(flogpp->old_map)); outv_field(v, "New map'", "0x%08x: %s", flogpp->new_map, out_get_btt_map_entry(flogpp->new_map)); outv_field(v, "Seq'", "0x%x", flogpp->seq); ptr += BTT_FLOG_PAIR_ALIGN; } error: free(buff); return ret; } /* * info_btt_stats -- print btt related statistics */ static void info_btt_stats(struct pmem_info *pip, int v) { if (pip->blk.stats.total > 0) { outv_title(v, "PMEM BLK Statistics"); double perc_zeros = (double)pip->blk.stats.zeros / (double)pip->blk.stats.total * 100.0; double perc_errors = (double)pip->blk.stats.errors / (double)pip->blk.stats.total * 100.0; double perc_noflag = (double)pip->blk.stats.noflag / (double)pip->blk.stats.total * 100.0; outv_field(v, "Total blocks", "%u", pip->blk.stats.total); outv_field(v, "Zeroed blocks", "%u [%s]", pip->blk.stats.zeros, out_get_percentage(perc_zeros)); outv_field(v, "Error blocks", "%u [%s]", pip->blk.stats.errors, out_get_percentage(perc_errors)); outv_field(v, "Blocks without flag", "%u [%s]", pip->blk.stats.noflag, out_get_percentage(perc_noflag)); } } /* * info_btt_info -- print btt_info structure fields */ static int info_btt_info(struct pmem_info *pip, int v, struct btt_info *infop) { outv_field(v, "Signature", "%.*s", BTTINFO_SIG_LEN, infop->sig); outv_field(v, "UUID of container", "%s", out_get_uuid_str(infop->parent_uuid)); outv_field(v, "Flags", "0x%x", infop->flags); outv_field(v, "Major", "%d", infop->major); outv_field(v, "Minor", "%d", infop->minor); outv_field(v, "External LBA size", "%s", out_get_size_str(infop->external_lbasize, pip->args.human)); outv_field(v, "External LBA count", "%u", infop->external_nlba); outv_field(v, "Internal LBA size", "%s", out_get_size_str(infop->internal_lbasize, pip->args.human)); outv_field(v, "Internal LBA count", "%u", infop->internal_nlba); outv_field(v, "Free blocks", "%u", infop->nfree); outv_field(v, "Info block size", "%s", out_get_size_str(infop->infosize, pip->args.human)); outv_field(v, "Next arena offset", "0x%lx", infop->nextoff); outv_field(v, "Arena data offset", "0x%lx", infop->dataoff); outv_field(v, "Area map offset", "0x%lx", infop->mapoff); outv_field(v, "Area flog offset", "0x%lx", infop->flogoff); outv_field(v, "Info block backup offset", "0x%lx", infop->infooff); outv_field(v, "Checksum", "%s", out_get_checksum(infop, sizeof(*infop), &infop->checksum, 0)); return 0; } /* * info_btt_layout -- print information about BTT layout */ static int info_btt_layout(struct pmem_info *pip, os_off_t btt_off) { int ret = 0; if (btt_off <= 0) { outv_err("wrong BTT layout offset\n"); return -1; } struct btt_info *infop = NULL; infop = malloc(sizeof(struct btt_info)); if (!infop) err(1, "Cannot allocate memory for BTT Info structure"); int narena = 0; uint64_t cur_lba = 0; uint64_t count_data = 0; uint64_t count_map = 0; uint64_t offset = (uint64_t)btt_off; uint64_t nextoff = 0; do { /* read btt info area */ if (pmempool_info_read(pip, infop, sizeof(*infop), offset)) { ret = -1; outv_err("cannot read BTT Info header\n"); goto err; } if (util_check_memory((uint8_t *)infop, sizeof(*infop), 0) == 0) { outv(1, "\n\n"); break; } outv(1, "\n[ARENA %d]", narena); outv_title(1, "PMEM BLK BTT Info Header"); outv_hexdump(pip->args.vhdrdump, infop, sizeof(*infop), offset, 1); btt_info_convert2h(infop); nextoff = infop->nextoff; /* print btt info fields */ if (info_btt_info(pip, 1, infop)) { ret = -1; goto err; } /* dump blocks data */ if (info_btt_data(pip, pip->args.vdata, infop, offset, cur_lba, &count_data)) { ret = -1; goto err; } /* print btt map entries and get statistics */ if (info_btt_map(pip, pip->args.blk.vmap, infop, offset, cur_lba, &count_map)) { ret = -1; goto err; } /* print flog entries */ if (info_btt_flog(pip, pip->args.blk.vflog, infop, offset)) { ret = -1; goto err; } /* increment LBA's counter before reading info backup */ cur_lba += infop->external_nlba; /* read btt info backup area */ if (pmempool_info_read(pip, infop, sizeof(*infop), offset + infop->infooff)) { outv_err("wrong BTT Info Backup size or offset\n"); ret = -1; goto err; } outv_title(pip->args.blk.vbackup, "PMEM BLK BTT Info Header Backup"); if (outv_check(pip->args.blk.vbackup)) outv_hexdump(pip->args.vhdrdump, infop, sizeof(*infop), offset + infop->infooff, 1); btt_info_convert2h(infop); info_btt_info(pip, pip->args.blk.vbackup, infop); offset += nextoff; narena++; } while (nextoff > 0); info_btt_stats(pip, pip->args.vstats); err: if (infop) free(infop); return ret; } /* * info_blk_descriptor -- print pmemblk descriptor */ static void info_blk_descriptor(struct pmem_info *pip, int v, struct pmemblk *pbp) { size_t pmemblk_size; #ifdef DEBUG pmemblk_size = offsetof(struct pmemblk, write_lock); #else pmemblk_size = sizeof(*pbp); #endif outv_title(v, "PMEM BLK Header"); /* dump pmemblk header without pool_hdr */ outv_hexdump(pip->args.vhdrdump, (uint8_t *)pbp + sizeof(pbp->hdr), pmemblk_size - sizeof(pbp->hdr), sizeof(pbp->hdr), 1); outv_field(v, "Block size", "%s", out_get_size_str(pbp->bsize, pip->args.human)); outv_field(v, "Is zeroed", pbp->is_zeroed ? "true" : "false"); } /* * pmempool_info_blk -- print information about block type pool */ int pmempool_info_blk(struct pmem_info *pip) { int ret; struct pmemblk *pbp = malloc(sizeof(struct pmemblk)); if (!pbp) err(1, "Cannot allocate memory for pmemblk structure"); if (pmempool_info_read(pip, pbp, sizeof(struct pmemblk), 0)) { outv_err("cannot read pmemblk header\n"); free(pbp); return -1; } info_blk_descriptor(pip, VERBOSE_DEFAULT, pbp); ssize_t btt_off = (char *)pbp->data - (char *)pbp->addr; ret = info_btt_layout(pip, btt_off); free(pbp); return ret; } /* * pmempool_info_btt -- print information about btt device */ int pmempool_info_btt(struct pmem_info *pip) { int ret; outv(1, "\nBTT Device"); ret = info_btt_layout(pip, DEFAULT_HDR_SIZE); return ret; } pmdk-1.13.1/src/tools/pmempool/info_log.c0000664000000000000000000000762114435627501016764 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2023, Intel Corporation */ /* * info_log.c -- pmempool info command source file for log pool (DEPRECATED) */ #include #include #include #include #include "common.h" #include "output.h" #include "info.h" /* * info_log_data -- print used data from log pool */ static int info_log_data(struct pmem_info *pip, int v, struct pmemlog *plp) { if (!outv_check(v)) return 0; uint64_t size_used = plp->write_offset - plp->start_offset; if (size_used == 0) return 0; uint8_t *addr = pool_set_file_map(pip->pfile, plp->start_offset); if (addr == MAP_FAILED) { warn("%s", pip->file_name); outv_err("cannot read pmem log data\n"); return -1; } if (pip->args.log.walk == 0) { outv_title(v, "PMEMLOG data"); struct range *curp = NULL; PMDK_LIST_FOREACH(curp, &pip->args.ranges.head, next) { uint8_t *ptr = addr + curp->first; if (curp->last >= size_used) curp->last = size_used - 1; uint64_t count = curp->last - curp->first + 1; outv_hexdump(v, ptr, count, curp->first + plp->start_offset, 1); size_used -= count; if (!size_used) break; } } else { /* * Walk through used data with fixed chunk size * passed by user. */ uint64_t nchunks = size_used / pip->args.log.walk; outv_title(v, "PMEMLOG data [chunks: total = %lu size = %ld]", nchunks, pip->args.log.walk); struct range *curp = NULL; PMDK_LIST_FOREACH(curp, &pip->args.ranges.head, next) { uint64_t i; for (i = curp->first; i <= curp->last && i < nchunks; i++) { outv(v, "Chunk %10lu:\n", i); outv_hexdump(v, addr + i * pip->args.log.walk, pip->args.log.walk, plp->start_offset + i * pip->args.log.walk, 1); } } } return 0; } /* * info_logs_stats -- print log type pool statistics */ static void info_log_stats(struct pmem_info *pip, int v, struct pmemlog *plp) { uint64_t size_total = plp->end_offset - plp->start_offset; uint64_t size_used = plp->write_offset - plp->start_offset; uint64_t size_avail = size_total - size_used; if (size_total == 0) return; double perc_used = (double)size_used / (double)size_total * 100.0; double perc_avail = 100.0 - perc_used; outv_title(v, "PMEM LOG Statistics"); outv_field(v, "Total", "%s", out_get_size_str(size_total, pip->args.human)); outv_field(v, "Available", "%s [%s]", out_get_size_str(size_avail, pip->args.human), out_get_percentage(perc_avail)); outv_field(v, "Used", "%s [%s]", out_get_size_str(size_used, pip->args.human), out_get_percentage(perc_used)); } /* * info_log_descriptor -- print pmemlog descriptor and return 1 if * write offset is valid */ static int info_log_descriptor(struct pmem_info *pip, int v, struct pmemlog *plp) { outv_title(v, "PMEM LOG Header"); /* dump pmemlog header without pool_hdr */ outv_hexdump(pip->args.vhdrdump, (uint8_t *)plp + sizeof(plp->hdr), sizeof(*plp) - sizeof(plp->hdr), sizeof(plp->hdr), 1); log_convert2h(plp); int write_offset_valid = plp->write_offset >= plp->start_offset && plp->write_offset <= plp->end_offset; outv_field(v, "Start offset", "0x%lx", plp->start_offset); outv_field(v, "Write offset", "0x%lx [%s]", plp->write_offset, write_offset_valid ? "OK":"ERROR"); outv_field(v, "End offset", "0x%lx", plp->end_offset); return write_offset_valid; } /* * pmempool_info_log -- print information about log type pool */ int pmempool_info_log(struct pmem_info *pip) { int ret = 0; struct pmemlog *plp = malloc(sizeof(struct pmemlog)); if (!plp) err(1, "Cannot allocate memory for pmemlog structure"); if (pmempool_info_read(pip, plp, sizeof(struct pmemlog), 0)) { outv_err("cannot read pmemlog header\n"); free(plp); return -1; } if (info_log_descriptor(pip, VERBOSE_DEFAULT, plp)) { info_log_stats(pip, pip->args.vstats, plp); ret = info_log_data(pip, pip->args.vdata, plp); } free(plp); return ret; } pmdk-1.13.1/src/tools/pmempool/common.c0000664000000000000000000007311414435627501016460 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2023, Intel Corporation */ /* * common.c -- definitions of common functions */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "common.h" #include "output.h" #include "libpmem.h" #include "libpmemblk.h" #include "libpmemlog.h" #include "libpmemobj.h" #include "btt.h" #include "file.h" #include "os.h" #include "set.h" #include "out.h" #include "mmap.h" #include "util_pmem.h" #include "set_badblocks.h" #include "util.h" #define REQ_BUFF_SIZE 2048U #define Q_BUFF_SIZE 8192 typedef const char *(*enum_to_str_fn)(enum chunk_type); /* * pmem_pool_type -- return pool type based on first two pages. * If pool header's content suggests that pool may be BTT device * (first page zeroed and no correct signature for pool header), * signature from second page is checked to prove that it's BTT device layout. */ pmem_pool_type_t pmem_pool_type(const void *base_pool_addr) { struct pool_hdr *hdrp = (struct pool_hdr *)base_pool_addr; if (util_is_zeroed(hdrp, DEFAULT_HDR_SIZE)) { return util_get_pool_type_second_page(base_pool_addr); } pmem_pool_type_t type = pmem_pool_type_parse_hdr(hdrp); if (type != PMEM_POOL_TYPE_UNKNOWN) return type; else return util_get_pool_type_second_page(base_pool_addr); } /* * pmem_pool_checksum -- return true if checksum is correct * based on first two pages */ int pmem_pool_checksum(const void *base_pool_addr) { /* check whether it's btt device -> first page zeroed */ if (util_is_zeroed(base_pool_addr, DEFAULT_HDR_SIZE)) { struct btt_info bttinfo; void *sec_page_addr = (char *)base_pool_addr + DEFAULT_HDR_SIZE; memcpy(&bttinfo, sec_page_addr, sizeof(bttinfo)); btt_info_convert2h(&bttinfo); return util_checksum(&bttinfo, sizeof(bttinfo), &bttinfo.checksum, 0, 0); } else { /* it's not btt device - first page contains header */ struct pool_hdr hdrp; memcpy(&hdrp, base_pool_addr, sizeof(hdrp)); return util_checksum(&hdrp, sizeof(hdrp), &hdrp.checksum, 0, POOL_HDR_CSUM_END_OFF(&hdrp)); } } /* * pmem_pool_type_parse_hdr -- return pool type based only on signature */ pmem_pool_type_t pmem_pool_type_parse_hdr(const struct pool_hdr *hdrp) { if (memcmp(hdrp->signature, LOG_HDR_SIG, POOL_HDR_SIG_LEN) == 0) return PMEM_POOL_TYPE_LOG; else if (memcmp(hdrp->signature, BLK_HDR_SIG, POOL_HDR_SIG_LEN) == 0) return PMEM_POOL_TYPE_BLK; else if (memcmp(hdrp->signature, OBJ_HDR_SIG, POOL_HDR_SIG_LEN) == 0) return PMEM_POOL_TYPE_OBJ; else return PMEM_POOL_TYPE_UNKNOWN; } /* * pmem_pool_type_parse_str -- returns pool type from command line arg */ pmem_pool_type_t pmem_pool_type_parse_str(const char *str) { if (strcmp(str, "blk") == 0) { return PMEM_POOL_TYPE_BLK; } else if (strcmp(str, "log") == 0) { return PMEM_POOL_TYPE_LOG; } else if (strcmp(str, "obj") == 0) { return PMEM_POOL_TYPE_OBJ; } else if (strcmp(str, "btt") == 0) { return PMEM_POOL_TYPE_BTT; } else { return PMEM_POOL_TYPE_UNKNOWN; } } /* * util_get_pool_type_second_page -- return type based on second page content */ pmem_pool_type_t util_get_pool_type_second_page(const void *pool_base_addr) { struct btt_info bttinfo; void *sec_page_addr = (char *)pool_base_addr + DEFAULT_HDR_SIZE; memcpy(&bttinfo, sec_page_addr, sizeof(bttinfo)); btt_info_convert2h(&bttinfo); if (util_is_zeroed(&bttinfo, sizeof(bttinfo))) return PMEM_POOL_TYPE_UNKNOWN; if (memcmp(bttinfo.sig, BTTINFO_SIG, BTTINFO_SIG_LEN) == 0) return PMEM_POOL_TYPE_BTT; return PMEM_POOL_TYPE_UNKNOWN; } /* * util_parse_mode -- parse file mode from octal string */ int util_parse_mode(const char *str, mode_t *mode) { mode_t m = 0; int digits = 0; /* skip leading zeros */ while (*str == '0') str++; /* parse at most 3 octal digits */ while (digits < 3 && *str != '\0') { if (*str < '0' || *str > '7') return -1; m = (mode_t)(m << 3) | (mode_t)(*str - '0'); digits++; str++; } /* more than 3 octal digits */ if (digits == 3 && *str != '\0') return -1; if (mode) *mode = m; return 0; } static void util_range_limit(struct range *rangep, struct range limit) { if (rangep->first < limit.first) rangep->first = limit.first; if (rangep->last > limit.last) rangep->last = limit.last; } /* * util_parse_range_from -- parse range string as interval from specified number */ static int util_parse_range_from(char *str, struct range *rangep, struct range entire) { size_t str_len = strlen(str); if (str[str_len - 1] != '-') return -1; str[str_len - 1] = '\0'; if (util_parse_size(str, (size_t *)&rangep->first)) return -1; rangep->last = entire.last; util_range_limit(rangep, entire); return 0; } /* * util_parse_range_to -- parse range string as interval to specified number */ static int util_parse_range_to(char *str, struct range *rangep, struct range entire) { if (str[0] != '-' || str[1] == '\0') return -1; if (util_parse_size(str + 1, (size_t *)&rangep->last)) return -1; rangep->first = entire.first; util_range_limit(rangep, entire); return 0; } /* * util_parse_range_number -- parse range string as a single number */ static int util_parse_range_number(char *str, struct range *rangep, struct range entire) { if (util_parse_size(str, (size_t *)&rangep->first) != 0) return -1; rangep->last = rangep->first; if (rangep->first > entire.last || rangep->last < entire.first) return -1; util_range_limit(rangep, entire); return 0; } /* * util_parse_range -- parse single range string */ static int util_parse_range(char *str, struct range *rangep, struct range entire) { char *dash = strchr(str, '-'); if (!dash) return util_parse_range_number(str, rangep, entire); /* '-' at the beginning */ if (dash == str) return util_parse_range_to(str, rangep, entire); /* '-' at the end */ if (dash[1] == '\0') return util_parse_range_from(str, rangep, entire); *dash = '\0'; dash++; if (util_parse_size(str, (size_t *)&rangep->first)) return -1; if (util_parse_size(dash, (size_t *)&rangep->last)) return -1; if (rangep->first > rangep->last) { uint64_t tmp = rangep->first; rangep->first = rangep->last; rangep->last = tmp; } util_range_limit(rangep, entire); return 0; } /* * util_ranges_overlap -- return 1 if two ranges are overlapped */ static int util_ranges_overlap(struct range *rangep1, struct range *rangep2) { if (rangep1->last + 1 < rangep2->first || rangep2->last + 1 < rangep1->first) return 0; else return 1; } /* * util_ranges_add -- create and add range */ int util_ranges_add(struct ranges *rangesp, struct range range) { struct range *rangep = malloc(sizeof(struct range)); if (!rangep) err(1, "Cannot allocate memory for range\n"); memcpy(rangep, &range, sizeof(*rangep)); struct range *curp, *next; uint64_t first = rangep->first; uint64_t last = rangep->last; curp = PMDK_LIST_FIRST(&rangesp->head); while (curp) { next = PMDK_LIST_NEXT(curp, next); if (util_ranges_overlap(curp, rangep)) { PMDK_LIST_REMOVE(curp, next); if (curp->first < first) first = curp->first; if (curp->last > last) last = curp->last; free(curp); } curp = next; } rangep->first = first; rangep->last = last; PMDK_LIST_FOREACH(curp, &rangesp->head, next) { if (curp->first < rangep->first) { PMDK_LIST_INSERT_AFTER(curp, rangep, next); return 0; } } PMDK_LIST_INSERT_HEAD(&rangesp->head, rangep, next); return 0; } /* * util_ranges_contain -- return 1 if ranges contain the number n */ int util_ranges_contain(const struct ranges *rangesp, uint64_t n) { struct range *curp = NULL; PMDK_LIST_FOREACH(curp, &rangesp->head, next) { if (curp->first <= n && n <= curp->last) return 1; } return 0; } /* * util_ranges_empty -- return 1 if ranges are empty */ int util_ranges_empty(const struct ranges *rangesp) { return PMDK_LIST_EMPTY(&rangesp->head); } /* * util_ranges_clear -- clear list of ranges */ void util_ranges_clear(struct ranges *rangesp) { while (!PMDK_LIST_EMPTY(&rangesp->head)) { struct range *rangep = PMDK_LIST_FIRST(&rangesp->head); PMDK_LIST_REMOVE(rangep, next); free(rangep); } } /* * util_parse_ranges -- parser ranges from string * * The valid formats of range are: * - 'n-m' -- from n to m * - '-m' -- from minimum passed in entirep->first to m * - 'n-' -- from n to maximum passed in entirep->last * - 'n' -- n'th byte/block * Multiple ranges may be separated by comma: * 'n1-m1,n2-,-m3,n4' */ int util_parse_ranges(const char *ptr, struct ranges *rangesp, struct range entire) { if (ptr == NULL) return util_ranges_add(rangesp, entire); char *dup = strdup(ptr); if (!dup) err(1, "Cannot allocate memory for ranges"); char *str = dup; int ret = 0; char *next = str; do { str = next; next = strchr(str, ','); if (next != NULL) { *next = '\0'; next++; } struct range range; if (util_parse_range(str, &range, entire)) { ret = -1; goto out; } else if (util_ranges_add(rangesp, range)) { ret = -1; goto out; } } while (next != NULL); out: free(dup); return ret; } /* * pmem_pool_get_min_size -- return minimum size of pool for specified type */ uint64_t pmem_pool_get_min_size(pmem_pool_type_t type) { switch (type) { case PMEM_POOL_TYPE_LOG: return PMEMLOG_MIN_POOL; case PMEM_POOL_TYPE_BLK: return PMEMBLK_MIN_POOL; case PMEM_POOL_TYPE_OBJ: return PMEMOBJ_MIN_POOL; default: break; } return 0; } /* * util_poolset_map -- map poolset */ int util_poolset_map(const char *fname, struct pool_set **poolset, int rdonly) { if (util_is_poolset_file(fname) != 1) { int ret = util_poolset_create_set(poolset, fname, 0, 0, true); if (ret < 0) { outv_err("cannot open pool set -- '%s'", fname); return -1; } unsigned flags = (rdonly ? POOL_OPEN_COW : 0) | POOL_OPEN_IGNORE_BAD_BLOCKS; return util_pool_open_nocheck(*poolset, flags); } /* open poolset file */ int fd = util_file_open(fname, NULL, 0, O_RDONLY); if (fd < 0) return -1; struct pool_set *set; /* parse poolset file */ if (util_poolset_parse(&set, fname, fd)) { outv_err("parsing poolset file failed\n"); os_close(fd); return -1; } set->ignore_sds = true; os_close(fd); /* read the pool header from first pool set file */ const char *part0_path = PART(REP(set, 0), 0)->path; struct pool_hdr hdr; if (util_file_pread(part0_path, &hdr, sizeof(hdr), 0) != sizeof(hdr)) { outv_err("cannot read pool header from poolset\n"); goto err_pool_set; } util_poolset_free(set); util_convert2h_hdr_nocheck(&hdr); /* parse pool type from first pool set file */ pmem_pool_type_t type = pmem_pool_type_parse_hdr(&hdr); if (type == PMEM_POOL_TYPE_UNKNOWN) { outv_err("cannot determine pool type from poolset\n"); return -1; } /* * Just use one thread. */ unsigned nlanes = 1; /* * Open the poolset, the values passed to util_pool_open are read * from the first poolset file, these values are then compared with * the values from all headers of poolset files. */ struct pool_attr attr; util_pool_hdr2attr(&attr, &hdr); unsigned flags = (rdonly ? POOL_OPEN_COW : 0) | POOL_OPEN_IGNORE_SDS | POOL_OPEN_IGNORE_BAD_BLOCKS; if (util_pool_open(poolset, fname, 0 /* minpartsize */, &attr, &nlanes, NULL, flags)) { outv_err("opening poolset failed\n"); return -1; } return 0; err_pool_set: util_poolset_free(set); return -1; } /* * pmem_pool_parse_params -- parse pool type, file size and block size */ int pmem_pool_parse_params(const char *fname, struct pmem_pool_params *paramsp, int check) { paramsp->type = PMEM_POOL_TYPE_UNKNOWN; char pool_str_addr[POOL_HDR_DESC_SIZE]; enum file_type type = util_file_get_type(fname); if (type < 0) return -1; int is_poolset = util_is_poolset_file(fname); if (is_poolset < 0) return -1; paramsp->is_poolset = is_poolset; int fd = util_file_open(fname, NULL, 0, O_RDONLY); if (fd < 0) return -1; /* get file size and mode */ os_stat_t stat_buf; if (os_fstat(fd, &stat_buf)) { os_close(fd); return -1; } int ret = 0; assert(stat_buf.st_size >= 0); paramsp->size = (uint64_t)stat_buf.st_size; paramsp->mode = stat_buf.st_mode; void *addr = NULL; struct pool_set *set = NULL; if (paramsp->is_poolset) { /* close the file */ os_close(fd); fd = -1; if (check) { if (util_poolset_map(fname, &set, 0)) { ret = -1; goto out_close; } } else { ret = util_poolset_create_set(&set, fname, 0, 0, true); if (ret < 0) { outv_err("cannot open pool set -- '%s'", fname); ret = -1; goto out_close; } if (util_pool_open_nocheck(set, POOL_OPEN_IGNORE_BAD_BLOCKS)) { ret = -1; goto out_close; } } paramsp->size = set->poolsize; addr = set->replica[0]->part[0].addr; /* * XXX mprotect for device dax with length not aligned to its * page granularity causes SIGBUS on the next page fault. * The length argument of this call should be changed to * set->poolsize once the kernel issue is solved. */ if (mprotect(addr, set->replica[0]->repsize, PROT_READ) < 0) { outv_err("!mprotect"); goto out_close; } } else { /* read first two pages */ if (type == TYPE_DEVDAX) { addr = util_file_map_whole(fname); if (addr == NULL) { ret = -1; goto out_close; } } else { ssize_t num = read(fd, pool_str_addr, POOL_HDR_DESC_SIZE); if (num < (ssize_t)POOL_HDR_DESC_SIZE) { outv_err("!read"); ret = -1; goto out_close; } addr = pool_str_addr; } } struct pool_hdr hdr; memcpy(&hdr, addr, sizeof(hdr)); util_convert2h_hdr_nocheck(&hdr); memcpy(paramsp->signature, hdr.signature, sizeof(paramsp->signature)); /* * Check if file is a part of pool set by comparing * the UUID with the next part UUID. If it is the same * it means the pool consist of a single file. */ paramsp->is_part = !paramsp->is_poolset && (memcmp(hdr.uuid, hdr.next_part_uuid, POOL_HDR_UUID_LEN) || memcmp(hdr.uuid, hdr.prev_part_uuid, POOL_HDR_UUID_LEN) || memcmp(hdr.uuid, hdr.next_repl_uuid, POOL_HDR_UUID_LEN) || memcmp(hdr.uuid, hdr.prev_repl_uuid, POOL_HDR_UUID_LEN)); if (check) paramsp->type = pmem_pool_type(addr); else paramsp->type = pmem_pool_type_parse_hdr(addr); paramsp->is_checksum_ok = pmem_pool_checksum(addr); if (paramsp->type == PMEM_POOL_TYPE_BLK) { struct pmemblk *pbp = addr; paramsp->blk.bsize = le32toh(pbp->bsize); } else if (paramsp->type == PMEM_POOL_TYPE_OBJ) { struct pmemobjpool *pop = addr; memcpy(paramsp->obj.layout, pop->layout, PMEMOBJ_MAX_LAYOUT); } if (paramsp->is_poolset) util_poolset_close(set, DO_NOT_DELETE_PARTS); out_close: if (fd >= 0) (void) os_close(fd); return ret; } /* * util_check_memory -- check if memory contains single value */ int util_check_memory(const uint8_t *buff, size_t len, uint8_t val) { size_t i; for (i = 0; i < len; i++) { if (buff[i] != val) return -1; } return 0; } /* * pmempool_ask_yes_no -- prints the question, * takes user answer and returns validated value */ static char pmempool_ask_yes_no(char def_ans, const char *answers, const char *qbuff) { char ret = INV_ANS; printf("%s", qbuff); size_t len = strlen(answers); size_t i; char def_anslo = (char)tolower(def_ans); printf(" ["); for (i = 0; i < len; i++) { char anslo = (char)tolower(answers[i]); printf("%c", anslo == def_anslo ? toupper(anslo) : anslo); if (i != len - 1) printf("/"); } printf("] "); char *line_of_answer = util_readline(stdin); if (line_of_answer == NULL) { outv_err("input is empty"); return '?'; } char first_letter = line_of_answer[0]; line_of_answer[0] = (char)tolower(first_letter); if (strcmp(line_of_answer, "yes\n") == 0) { if (strchr(answers, 'y') != NULL) ret = 'y'; } if (strcmp(line_of_answer, "no\n") == 0) { if (strchr(answers, 'n') != NULL) ret = 'n'; } if (strlen(line_of_answer) == 2 && line_of_answer[1] == '\n') { if (strchr(answers, line_of_answer[0]) != NULL) ret = line_of_answer[0]; } if (strlen(line_of_answer) == 1 && line_of_answer[0] == '\n') { ret = def_ans; } Free(line_of_answer); return ret; } /* * ask -- keep asking for answer until it gets valid input */ char ask(char op, char *answers, char def_ans, const char *fmt, va_list ap) { char qbuff[Q_BUFF_SIZE]; char ret = INV_ANS; int is_tty = 0; if (op != '?') return op; int p = vsnprintf(qbuff, Q_BUFF_SIZE, fmt, ap); if (p < 0) { outv_err("vsnprintf"); exit(EXIT_FAILURE); } if (p >= Q_BUFF_SIZE) { outv_err("vsnprintf: output was truncated"); exit(EXIT_FAILURE); } is_tty = isatty(fileno(stdin)); while ((ret = pmempool_ask_yes_no(def_ans, answers, qbuff)) == INV_ANS) ; if (!is_tty) printf("%c\n", ret); return ret; } char ask_Yn(char op, const char *fmt, ...) { va_list ap; va_start(ap, fmt); char ret = ask(op, "yn", 'y', fmt, ap); va_end(ap); return ret; } char ask_yN(char op, const char *fmt, ...) { va_list ap; va_start(ap, fmt); char ret = ask(op, "yn", 'n', fmt, ap); va_end(ap); return ret; } /* * util_parse_enum -- parse single enum and store to bitmap */ static int util_parse_enum(const char *str, int first, int max, uint64_t *bitmap, enum_to_str_fn enum_to_str) { for (int i = first; i < max; i++) { if (strcmp(str, enum_to_str((enum chunk_type)i)) == 0) { *bitmap |= (uint64_t)1<opts = options; opts->noptions = nopts; opts->req = req; size_t bitmap_size = howmany(nopts, 8); opts->bitmap = calloc(bitmap_size, 1); if (!opts->bitmap) err(1, "Cannot allocate memory for options bitmap"); return opts; } /* * util_options_free -- free options structure */ void util_options_free(struct options *opts) { free(opts->bitmap); free(opts); } /* * util_opt_get_index -- return index of specified option in global * array of options */ static int util_opt_get_index(const struct options *opts, int opt) { const struct option *lopt = &opts->opts[0]; int ret = 0; while (lopt->name) { if ((lopt->val & ~OPT_MASK) == opt) return ret; lopt++; ret++; } return -1; } /* * util_opt_get_req -- get required option for specified option */ static struct option_requirement * util_opt_get_req(const struct options *opts, int opt, pmem_pool_type_t type) { size_t n = 0; struct option_requirement *ret = NULL; struct option_requirement *tmp = NULL; const struct option_requirement *req = &opts->req[0]; while (req->opt) { if (req->opt == opt && (req->type & type)) { n++; tmp = realloc(ret, n * sizeof(*ret)); if (!tmp) err(1, "Cannot allocate memory for" " option requirements"); ret = tmp; ret[n - 1] = *req; } req++; } if (ret) { tmp = realloc(ret, (n + 1) * sizeof(*ret)); if (!tmp) err(1, "Cannot allocate memory for" " option requirements"); ret = tmp; memset(&ret[n], 0, sizeof(*ret)); } return ret; } /* * util_opt_check_requirements -- check if requirements has been fulfilled */ static int util_opt_check_requirements(const struct options *opts, const struct option_requirement *req) { int count = 0; int set = 0; uint64_t tmp; while ((tmp = req->req) != 0) { while (tmp) { int req_idx = util_opt_get_index(opts, tmp & OPT_REQ_MASK); if (req_idx >= 0 && util_isset(opts->bitmap, req_idx)) { set++; break; } tmp >>= OPT_REQ_SHIFT; } req++; count++; } return count != set; } /* * util_opt_print_requirements -- print requirements for specified option */ static void util_opt_print_requirements(const struct options *opts, const struct option_requirement *req) { char buff[REQ_BUFF_SIZE]; unsigned n = 0; uint64_t tmp; const struct option *opt = &opts->opts[util_opt_get_index(opts, req->opt)]; int sn; sn = util_snprintf(&buff[n], REQ_BUFF_SIZE - n, "option [-%c|--%s] requires: ", opt->val, opt->name); assert(sn >= 0); n += (unsigned)sn; if (n >= REQ_BUFF_SIZE) goto exit; size_t rc = 0; while ((tmp = req->req) != 0) { if (rc != 0) { sn = util_snprintf(&buff[n], REQ_BUFF_SIZE - n, " and "); assert(sn >= 0); n += (unsigned)sn; if (n >= REQ_BUFF_SIZE) break; } size_t c = 0; while (tmp) { sn = util_snprintf(&buff[n], REQ_BUFF_SIZE - n, c == 0 ? "[" : "|"); assert(sn >= 0); n += (unsigned)sn; if (n >= REQ_BUFF_SIZE) break; int req_opt_ind = util_opt_get_index(opts, tmp & OPT_REQ_MASK); const struct option *req_option = &opts->opts[req_opt_ind]; sn = util_snprintf(&buff[n], REQ_BUFF_SIZE - n, "-%c|--%s", req_option->val, req_option->name); assert(sn >= 0); n += (unsigned)sn; if (n >= REQ_BUFF_SIZE) break; tmp >>= OPT_REQ_SHIFT; c++; } sn = util_snprintf(&buff[n], REQ_BUFF_SIZE - n, "]"); assert(sn >= 0); n += (unsigned)sn; if (n >= REQ_BUFF_SIZE) break; req++; rc++; } exit: outv_err("%s\n", buff); } /* * util_opt_verify_requirements -- verify specified requirements for options */ static int util_opt_verify_requirements(const struct options *opts, size_t index, pmem_pool_type_t type) { const struct option *opt = &opts->opts[index]; int val = opt->val & ~OPT_MASK; struct option_requirement *req; if ((req = util_opt_get_req(opts, val, type)) == NULL) return 0; int ret = 0; if (util_opt_check_requirements(opts, req)) { ret = -1; util_opt_print_requirements(opts, req); } free(req); return ret; } /* * util_opt_verify_type -- check if used option matches pool type */ static int util_opt_verify_type(const struct options *opts, pmem_pool_type_t type, size_t index) { const struct option *opt = &opts->opts[index]; int val = opt->val & ~OPT_MASK; int opt_type = opt->val; opt_type >>= OPT_SHIFT; if (!(opt_type & (1<name, val, out_get_pool_type_str(type)); return -1; } return 0; } /* * util_options_getopt -- wrapper for getopt_long which sets bitmap */ int util_options_getopt(int argc, char *argv[], const char *optstr, const struct options *opts) { int opt = getopt_long(argc, argv, optstr, opts->opts, NULL); if (opt == -1 || opt == '?') return opt; opt &= ~OPT_MASK; int option_index = util_opt_get_index(opts, opt); assert(option_index >= 0); util_setbit((uint8_t *)opts->bitmap, (unsigned)option_index); return opt; } /* * util_options_verify -- verify options */ int util_options_verify(const struct options *opts, pmem_pool_type_t type) { for (size_t i = 0; i < opts->noptions; i++) { if (util_isset(opts->bitmap, i)) { if (util_opt_verify_type(opts, type, i)) return -1; if (opts->req) if (util_opt_verify_requirements(opts, i, type)) return -1; } } return 0; } /* * util_heap_max_zone -- get number of zones */ unsigned util_heap_max_zone(size_t size) { unsigned max_zone = 0; size -= sizeof(struct heap_header); while (size >= ZONE_MIN_SIZE) { max_zone++; size -= size <= ZONE_MAX_SIZE ? size : ZONE_MAX_SIZE; } return max_zone; } /* * pool_set_file_open -- opens pool set file or regular file */ struct pool_set_file * pool_set_file_open(const char *fname, int rdonly, int check) { struct pool_set_file *file = calloc(1, sizeof(*file)); if (!file) return NULL; file->replica = 0; file->fname = strdup(fname); if (!file->fname) goto err; os_stat_t buf; if (os_stat(fname, &buf)) { warn("%s", fname); goto err_free_fname; } file->mtime = buf.st_mtime; file->mode = buf.st_mode; if (S_ISBLK(file->mode)) file->fileio = true; if (file->fileio) { /* Simple file open for BTT device */ int fd = util_file_open(fname, NULL, 0, O_RDONLY); if (fd < 0) { outv_err("util_file_open failed\n"); goto err_free_fname; } os_off_t seek_size = os_lseek(fd, 0, SEEK_END); if (seek_size == -1) { outv_err("lseek SEEK_END failed\n"); os_close(fd); goto err_free_fname; } file->size = (size_t)seek_size; file->fd = fd; } else { /* * The check flag indicates whether the headers from each pool * set file part should be checked for valid values. */ if (check) { if (util_poolset_map(file->fname, &file->poolset, rdonly)) goto err_free_fname; } else { int ret = util_poolset_create_set(&file->poolset, file->fname, 0, 0, true); if (ret < 0) { outv_err("cannot open pool set -- '%s'", file->fname); goto err_free_fname; } unsigned flags = (rdonly ? POOL_OPEN_COW : 0) | POOL_OPEN_IGNORE_BAD_BLOCKS; if (util_pool_open_nocheck(file->poolset, flags)) goto err_free_fname; } /* get modification time from the first part of first replica */ const char *path = file->poolset->replica[0]->part[0].path; if (os_stat(path, &buf)) { warn("%s", path); goto err_close_poolset; } file->size = file->poolset->poolsize; file->addr = file->poolset->replica[0]->part[0].addr; } return file; err_close_poolset: util_poolset_close(file->poolset, DO_NOT_DELETE_PARTS); err_free_fname: free(file->fname); err: free(file); return NULL; } /* * pool_set_file_close -- closes pool set file or regular file */ void pool_set_file_close(struct pool_set_file *file) { if (!file->fileio) { if (file->poolset) util_poolset_close(file->poolset, DO_NOT_DELETE_PARTS); else if (file->addr) { munmap(file->addr, file->size); os_close(file->fd); } } free(file->fname); free(file); } /* * pool_set_file_read -- read from pool set file or regular file * * 'buff' has to be a buffer at least 'nbytes' long * 'off' is an offset from the beginning of the file */ int pool_set_file_read(struct pool_set_file *file, void *buff, size_t nbytes, uint64_t off) { if (off + nbytes > file->size) return -1; if (file->fileio) { ssize_t num = pread(file->fd, buff, nbytes, (os_off_t)off); if (num < (ssize_t)nbytes) return -1; } else { memcpy(buff, (char *)file->addr + off, nbytes); } return 0; } /* * pool_set_file_write -- write to pool set file or regular file * * 'buff' has to be a buffer at least 'nbytes' long * 'off' is an offset from the beginning of the file */ int pool_set_file_write(struct pool_set_file *file, void *buff, size_t nbytes, uint64_t off) { enum file_type type = util_file_get_type(file->fname); if (type < 0) return -1; if (off + nbytes > file->size) return -1; if (file->fileio) { ssize_t num = pwrite(file->fd, buff, nbytes, (os_off_t)off); if (num < (ssize_t)nbytes) return -1; } else { memcpy((char *)file->addr + off, buff, nbytes); util_persist_auto(type == TYPE_DEVDAX, (char *)file->addr + off, nbytes); } return 0; } /* * pool_set_file_set_replica -- change replica for pool set file */ int pool_set_file_set_replica(struct pool_set_file *file, size_t replica) { if (!replica) return 0; if (!file->poolset) return -1; if (replica >= file->poolset->nreplicas) return -1; file->replica = replica; file->addr = file->poolset->replica[replica]->part[0].addr; return 0; } /* * pool_set_file_nreplicas -- return number of replicas */ size_t pool_set_file_nreplicas(struct pool_set_file *file) { return file->poolset->nreplicas; } /* * pool_set_file_map -- return mapped address at given offset */ void * pool_set_file_map(struct pool_set_file *file, uint64_t offset) { if (file->addr == MAP_FAILED) return NULL; return (char *)file->addr + offset; } /* * pool_set_file_persist -- propagates and persists changes to a memory range * * 'addr' points to the beginning of data in the master replica that has to be * propagated * 'len' is the number of bytes to be propagated to other replicas */ void pool_set_file_persist(struct pool_set_file *file, const void *addr, size_t len) { uintptr_t offset = (uintptr_t)((char *)addr - (char *)file->poolset->replica[0]->part[0].addr); for (unsigned r = 1; r < file->poolset->nreplicas; ++r) { struct pool_replica *rep = file->poolset->replica[r]; void *dst = (char *)rep->part[0].addr + offset; memcpy(dst, addr, len); util_persist(rep->is_pmem, dst, len); } struct pool_replica *rep = file->poolset->replica[0]; util_persist(rep->is_pmem, (void *)addr, len); } /* * util_pool_clear_badblocks -- clear badblocks in a pool (set or a single file) */ int util_pool_clear_badblocks(const char *path, int create) { LOG(3, "path %s create %i", path, create); struct pool_set *setp; /* do not check minsize */ int ret = util_poolset_create_set(&setp, path, 0, 0, POOL_OPEN_IGNORE_SDS); if (ret < 0) { LOG(2, "cannot open pool set -- '%s'", path); return -1; } if (badblocks_clear_poolset(setp, create)) { outv_err("clearing bad blocks in the pool set failed -- '%s'", path); errno = EIO; ret = -1; goto err; } ret = 0; err: util_poolset_free(setp); return ret; } pmdk-1.13.1/src/tools/pmempool/create.h0000664000000000000000000000073114435627501016433 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2014-2023, Intel Corporation */ /* * create.h -- pmempool create command header file */ #ifdef _WIN32 #define WIN_DEPR_STR "Windows support is deprecated." #define WIN_DEPR_ATTR __declspec(deprecated(WIN_DEPR_STR)) #endif #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmempool_create_func(const char *appname, int argc, char *argv[]); #ifdef _WIN32 WIN_DEPR_ATTR #endif void pmempool_create_help(const char *appname); pmdk-1.13.1/src/tools/pmempool/pmempool.vcxproj.filters0000664000000000000000000001152714435627501021740 0ustar rootroot {4FC737F1-C7A5-4376-A066-2A32D752A2FF} cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx {93995380-89BD-4b04-88EB-625FBE52EBFB} h;hh;hpp;hxx;hm;inl;inc;xsd {c91552dc-7579-447b-ad7f-7b2307c52502} Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files libs libs libs libs libs libs libs libs libs libs libs libs libs libs libs libs Source Files Source Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Source Files pmdk-1.13.1/src/tools/pmempool/feature.h0000664000000000000000000000073514435627501016627 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2018-2023, Intel Corporation */ /* * feature.h -- pmempool feature command header file */ #ifdef _WIN32 #define WIN_DEPR_STR "Windows support is deprecated." #define WIN_DEPR_ATTR __declspec(deprecated(WIN_DEPR_STR)) #endif #ifdef _WIN32 WIN_DEPR_ATTR #endif int pmempool_feature_func(const char *appname, int argc, char *argv[]); #ifdef _WIN32 WIN_DEPR_ATTR #endif void pmempool_feature_help(const char *appname); pmdk-1.13.1/src/tools/pmempool/convert.c0000664000000000000000000000407014435627501016643 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2018, Intel Corporation */ /* * convert.c -- pmempool convert command source file */ #include #include #include #include #include #include #include "convert.h" #include "os.h" #ifdef _WIN32 static const char *delimiter = ";"; static const char *convert_bin = "\\pmdk-convert.exe"; #else static const char *delimiter = ":"; static const char *convert_bin = "/pmdk-convert"; #endif // _WIN32 static int pmempool_convert_get_path(char *p, size_t max_len) { char *path = strdup(os_getenv("PATH")); if (!path) { perror("strdup"); return -1; } char *dir = strtok(path, delimiter); while (dir) { size_t length = strlen(dir) + strlen(convert_bin) + 1; if (length > max_len) { fprintf(stderr, "very long dir in PATH, ignoring\n"); continue; } strcpy(p, dir); strcat(p, convert_bin); if (os_access(p, F_OK) == 0) { free(path); return 0; } dir = strtok(NULL, delimiter); } free(path); return -1; } /* * pmempool_convert_help -- print help message for convert command. This is * help message from pmdk-convert tool. */ void pmempool_convert_help(const char *appname) { char path[4096]; if (pmempool_convert_get_path(path, sizeof(path))) { fprintf(stderr, "pmdk-convert is not installed. Please install it.\n"); exit(1); } char *args[] = { path, "-h", NULL }; os_execv(path, args); perror("execv"); exit(1); } /* * pmempool_convert_func -- main function for convert command. * It invokes pmdk-convert tool. */ int pmempool_convert_func(const char *appname, int argc, char *argv[]) { char path[4096]; if (pmempool_convert_get_path(path, sizeof(path))) { fprintf(stderr, "pmdk-convert is not installed. Please install it.\n"); exit(1); } char **args = malloc(((size_t)argc + 1) * sizeof(*args)); if (!args) { perror("malloc"); exit(1); } args[0] = path; for (int i = 1; i < argc; ++i) args[i] = argv[i]; args[argc] = NULL; os_execv(args[0], args); perror("execv"); free(args); exit(1); } pmdk-1.13.1/src/tools/pmempool/feature.c0000664000000000000000000000773114435627501016625 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2018-2019, Intel Corporation */ /* * feature.c -- pmempool feature command source file */ #include #include #include "common.h" #include "feature.h" #include "output.h" #include "libpmempool.h" /* operations over features */ enum feature_op { undefined, enable, disable, query }; /* * feature_ctx -- context and arguments for feature command */ struct feature_ctx { int verbose; const char *fname; enum feature_op op; enum pmempool_feature feature; unsigned flags; }; /* * pmempool_feature_default -- default arguments for feature command */ static const struct feature_ctx pmempool_feature_default = { .verbose = 0, .fname = NULL, .op = undefined, .feature = UINT32_MAX, .flags = 0 }; /* * help_str -- string for help message */ static const char * const help_str = "Toggle or query a pool feature\n" "\n" "For complete documentation see %s-feature(1) manual page.\n" ; /* * long_options -- command line options */ static const struct option long_options[] = { {"enable", required_argument, NULL, 'e'}, {"disable", required_argument, NULL, 'd'}, {"query", required_argument, NULL, 'q'}, {"verbose", no_argument, NULL, 'v'}, {"help", no_argument, NULL, 'h'}, {NULL, 0, NULL, 0 }, }; /* * print_usage -- print short description of application's usage */ static void print_usage(const char *appname) { printf("Usage: %s feature [] \n", appname); printf( "feature: SINGLEHDR, CKSUM_2K, SHUTDOWN_STATE, CHECK_BAD_BLOCKS\n"); } /* * print_version -- print version string */ static void print_version(const char *appname) { printf("%s %s\n", appname, SRCVERSION); } /* * pmempool_feature_help -- print help message for feature command */ void pmempool_feature_help(const char *appname) { print_usage(appname); print_version(appname); printf(help_str, appname); } /* * feature_perform -- perform operation over function */ static int feature_perform(struct feature_ctx *pfp) { int ret; switch (pfp->op) { case enable: return pmempool_feature_enable(pfp->fname, pfp->feature, pfp->flags); case disable: return pmempool_feature_disable(pfp->fname, pfp->feature, pfp->flags); case query: ret = pmempool_feature_query(pfp->fname, pfp->feature, pfp->flags); if (ret < 0) return 1; printf("%d", ret); return 0; default: outv_err("Invalid option."); return -1; } } /* * set_op -- set operation */ static void set_op(const char *appname, struct feature_ctx *pfp, enum feature_op op, const char *feature) { /* only one operation allowed */ if (pfp->op != undefined) goto misuse; pfp->op = op; /* parse feature name */ uint32_t fval = util_str2pmempool_feature(feature); if (fval == UINT32_MAX) goto misuse; pfp->feature = (enum pmempool_feature)fval; return; misuse: print_usage(appname); exit(EXIT_FAILURE); } /* * parse_args -- parse command line arguments */ static int parse_args(struct feature_ctx *pfp, const char *appname, int argc, char *argv[]) { int opt; while ((opt = getopt_long(argc, argv, "vhe:d:q:h", long_options, NULL)) != -1) { switch (opt) { case 'e': set_op(appname, pfp, enable, optarg); break; case 'd': set_op(appname, pfp, disable, optarg); break; case 'q': set_op(appname, pfp, query, optarg); break; case 'v': pfp->verbose = 2; break; case 'h': pmempool_feature_help(appname); exit(EXIT_SUCCESS); default: print_usage(appname); exit(EXIT_FAILURE); } } if (optind >= argc) { print_usage(appname); exit(EXIT_FAILURE); } pfp->fname = argv[optind]; return 0; } /* * pmempool_feature_func -- main function for feature command */ int pmempool_feature_func(const char *appname, int argc, char *argv[]) { struct feature_ctx pf = pmempool_feature_default; int ret = 0; /* parse command line arguments */ ret = parse_args(&pf, appname, argc, argv); if (ret) return ret; /* set verbosity level */ out_set_vlevel(pf.verbose); return feature_perform(&pf); } pmdk-1.13.1/src/tools/daxio/0000775000000000000000000000000014435627501014272 5ustar rootrootpmdk-1.13.1/src/tools/daxio/.gitignore0000664000000000000000000000000614435627501016256 0ustar rootrootdaxio pmdk-1.13.1/src/tools/daxio/Makefile0000664000000000000000000000102614435627501015731 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018-2022, Intel Corporation # # Makefile -- top Makefile for daxio # TOP = ../../.. include $(TOP)/src/common.inc INCS += -I$(TOP)/src/libpmem2 ifeq ($(NDCTL_ENABLE),y) TARGET = daxio OBJS = daxio.o LIBPMEM=y LIBPMEMCOMMON=y CFLAGS += $(LIBNDCTL_CFLAGS) LIBS += $(LIBNDCTL_LIBS) MANPAGES = $(TOP)/doc/daxio.1 # XXX: to be done # BASH_COMP_FILES = daxio.sh else $(info NOTE: Skipping daxio because ndctl is not available) endif include ../Makefile.inc .PHONY: test check pmdk-1.13.1/src/tools/daxio/daxio.c0000664000000000000000000003352114435627501015546 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2018-2022, Intel Corporation */ /* * daxio.c -- simple app for reading and writing data from/to * Device DAX device using mmap instead of file I/O API */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "util.h" #include "os.h" #include "badblocks.h" #define ALIGN_UP(size, align) (((size) + (align) - 1) & ~((align) - 1)) #define ALIGN_DOWN(size, align) ((size) & ~((align) - 1)) #define ERR(fmt, ...)\ do {\ fprintf(stderr, "daxio: " fmt, ##__VA_ARGS__);\ } while (0) #define FAIL(func)\ do {\ fprintf(stderr, "daxio: %s:%d: %s: %s\n",\ __func__, __LINE__, func, strerror(errno));\ } while (0) #define USAGE_MESSAGE \ "Usage: daxio [option] ...\n"\ "Valid options:\n"\ " -i, --input=FILE - input device/file (default stdin)\n"\ " -o, --output=FILE - output device/file (default stdout)\n"\ " -k, --skip=BYTES - skip offset for input (default 0)\n"\ " -s, --seek=BYTES - seek offset for output (default 0)\n"\ " -l, --len=BYTES - total length to perform the I/O\n"\ " -b, --clear-bad-blocks= - clear bad blocks (default: yes)\n"\ " -z, --zero - zeroing the device\n"\ " -h. --help - print this help\n"\ " -V, --version - display version of daxio\n" struct daxio_device { char *path; int fd; size_t size; /* actual file/device size */ int is_devdax; /* Device DAX only */ size_t align; /* internal device alignment */ char *addr; /* mapping base address */ size_t maplen; /* mapping length */ size_t offset; /* seek or skip */ unsigned major; unsigned minor; struct ndctl_ctx *ndctl_ctx; struct ndctl_region *region; /* parent region */ }; /* * daxio_context -- context and arguments */ struct daxio_context { size_t len; /* total length of I/O */ int zero; int clear_bad_blocks; struct daxio_device src; struct daxio_device dst; }; /* * default context */ static struct daxio_context Ctx = { SIZE_MAX, /* len */ 0, /* zero */ 1, /* clear_bad_blocks */ { NULL, -1, SIZE_MAX, 0, 0, NULL, 0, 0, 0, 0, NULL, NULL }, { NULL, -1, SIZE_MAX, 0, 0, NULL, 0, 0, 0, 0, NULL, NULL }, }; /* * print_version -- print daxio version */ static void print_version(void) { printf("%s\n", SRCVERSION); } /* * print_usage -- print short description of usage */ static void print_usage(void) { fprintf(stderr, USAGE_MESSAGE); } /* * long_options -- command line options */ static const struct option long_options[] = { {"input", required_argument, NULL, 'i'}, {"output", required_argument, NULL, 'o'}, {"skip", required_argument, NULL, 'k'}, {"seek", required_argument, NULL, 's'}, {"len", required_argument, NULL, 'l'}, {"clear-bad-blocks", required_argument, NULL, 'b'}, {"zero", no_argument, NULL, 'z'}, {"help", no_argument, NULL, 'h'}, {"version", no_argument, NULL, 'V'}, {NULL, 0, NULL, 0 }, }; /* * parse_args -- (internal) parse command line arguments */ static int parse_args(struct daxio_context *ctx, int argc, char * const argv[]) { int opt; size_t offset; size_t len; while ((opt = getopt_long(argc, argv, "i:o:k:s:l:b:zhV", long_options, NULL)) != -1) { switch (opt) { case 'i': ctx->src.path = optarg; break; case 'o': ctx->dst.path = optarg; break; case 'k': if (util_parse_size(optarg, &offset)) { ERR("'%s' -- invalid input offset\n", optarg); return -1; } ctx->src.offset = offset; break; case 's': if (util_parse_size(optarg, &offset)) { ERR("'%s' -- invalid output offset\n", optarg); return -1; } ctx->dst.offset = offset; break; case 'l': if (util_parse_size(optarg, &len)) { ERR("'%s' -- invalid length\n", optarg); return -1; } ctx->len = len; break; case 'z': ctx->zero = 1; break; case 'b': if (strcmp(optarg, "no") == 0) { ctx->clear_bad_blocks = 0; } else if (strcmp(optarg, "yes") == 0) { ctx->clear_bad_blocks = 1; } else { ERR( "'%s' -- invalid argument of the '--clear-bad-blocks' option\n", optarg); return -1; } break; case 'h': print_usage(); exit(EXIT_SUCCESS); case 'V': print_version(); exit(EXIT_SUCCESS); default: print_usage(); exit(EXIT_FAILURE); } } return 0; } /* * validate_args -- (internal) validate command line arguments */ static int validate_args(struct daxio_context *ctx) { if (ctx->zero && ctx->dst.path == NULL) { ERR("zeroing flag specified but no output file provided\n"); return -1; } if (!ctx->zero && ctx->src.path == NULL && ctx->dst.path == NULL) { ERR("an input file and/or an output file must be provided\n"); return -1; } /* if no input file provided, use stdin */ if (ctx->src.path == NULL) { if (ctx->src.offset != 0) { ERR( "skip offset specified but no input file provided\n"); return -1; } ctx->src.fd = STDIN_FILENO; ctx->src.path = "STDIN"; } /* if no output file provided, use stdout */ if (ctx->dst.path == NULL) { if (ctx->dst.offset != 0) { ERR( "seek offset specified but no output file provided\n"); return -1; } ctx->dst.fd = STDOUT_FILENO; ctx->dst.path = "STDOUT"; } return 0; } /* * match_dev_dax -- (internal) find Device DAX by major/minor device number */ static int match_dev_dax(struct daxio_device *dev, struct daxctl_region *dax_region) { struct daxctl_dev *d; daxctl_dev_foreach(dax_region, d) { if (dev->major == (unsigned)daxctl_dev_get_major(d) && dev->minor == (unsigned)daxctl_dev_get_minor(d)) { dev->size = daxctl_dev_get_size(d); return 1; } } return 0; } /* * find_dev_dax -- (internal) check if device is Device DAX * * If there is matching Device DAX, find its region, size and alignment. */ static int find_dev_dax(struct ndctl_ctx *ndctl_ctx, struct daxio_device *dev) { struct ndctl_bus *bus = NULL; struct ndctl_region *region = NULL; struct ndctl_dax *dax = NULL; struct daxctl_region *dax_region = NULL; ndctl_bus_foreach(ndctl_ctx, bus) { ndctl_region_foreach(bus, region) { ndctl_dax_foreach(region, dax) { dax_region = ndctl_dax_get_daxctl_region(dax); if (match_dev_dax(dev, dax_region)) { dev->is_devdax = 1; dev->align = ndctl_dax_get_align(dax); dev->region = region; return 1; } } } } /* try with dax regions */ struct daxctl_ctx *daxctl_ctx; if (daxctl_new(&daxctl_ctx)) return 0; int ret = 0; daxctl_region_foreach(daxctl_ctx, dax_region) { if (match_dev_dax(dev, dax_region)) { dev->is_devdax = 1; dev->align = daxctl_region_get_align(dax_region); dev->region = region; ret = 1; goto end; } } end: daxctl_unref(daxctl_ctx); return ret; } /* * setup_device -- (internal) open/mmap file/device */ static int setup_device(struct ndctl_ctx *ndctl_ctx, struct daxio_device *dev, int is_dst, int clear_bad_blocks) { int ret; int flags = O_RDWR; int prot = is_dst ? PROT_WRITE : PROT_READ; if (dev->fd != -1) { dev->size = SIZE_MAX; return 0; /* stdin/stdout */ } /* try to open file/device (if exists) */ dev->fd = os_open(dev->path, flags, S_IRUSR|S_IWUSR); if (dev->fd == -1) { ret = errno; if (ret == ENOENT && is_dst) { /* file does not exist - create it */ flags = O_CREAT|O_WRONLY|O_TRUNC; dev->size = SIZE_MAX; dev->fd = os_open(dev->path, flags, S_IRUSR|S_IWUSR); if (dev->fd == -1) { FAIL("open"); return -1; } return 0; } else { ERR("failed to open '%s': %s\n", dev->path, strerror(errno)); return -1; } } struct stat stbuf; ret = fstat(dev->fd, &stbuf); if (ret == -1) { FAIL("stat"); return -1; } /* check if this is regular file or device */ if (S_ISREG(stbuf.st_mode)) { if (is_dst) dev->size = SIZE_MAX; else dev->size = (size_t)stbuf.st_size; } else if (S_ISBLK(stbuf.st_mode)) { dev->size = (size_t)stbuf.st_size; } else if (S_ISCHR(stbuf.st_mode)) { dev->size = SIZE_MAX; dev->major = major(stbuf.st_rdev); dev->minor = minor(stbuf.st_rdev); } else { return -1; } /* check if this is Device DAX */ if (S_ISCHR(stbuf.st_mode)) find_dev_dax(ndctl_ctx, dev); if (!dev->is_devdax) return 0; if (is_dst && clear_bad_blocks) { /* XXX - clear only badblocks in range bound by offset/len */ if (badblocks_clear_all(dev->path)) { ERR("failed to clear bad blocks on \"%s\"\n" " Probably you have not enough permissions to do that.\n" " You can choose one of three options now:\n" " 1) run 'daxio' with 'sudo' or as 'root',\n" " 2) turn off clearing bad blocks using\n" " the '-b/--clear-bad-blocks=no' option or\n" " 3) change permissions of some resource files -\n" " - for details see the description of the CHECK_BAD_BLOCKS\n" " compat feature in the pmempool-feature(1) man page.\n", dev->path); return -1; } } if (dev->align == ULONG_MAX) { ERR("cannot determine device alignment for \"%s\"\n", dev->path); return -1; } if (dev->offset > dev->size) { ERR("'%zu' -- offset beyond device size (%zu)\n", dev->offset, dev->size); return -1; } /* align len/offset to the internal device alignment */ dev->maplen = ALIGN_UP(dev->size, dev->align); size_t offset = ALIGN_DOWN(dev->offset, dev->align); dev->offset = dev->offset - offset; dev->maplen = dev->maplen - offset; dev->addr = mmap(NULL, dev->maplen, prot, MAP_SHARED, dev->fd, (off_t)offset); if (dev->addr == MAP_FAILED) { FAIL("mmap"); return -1; } return 0; } /* * setup_devices -- (internal) open/mmap input and output */ static int setup_devices(struct ndctl_ctx *ndctl_ctx, struct daxio_context *ctx) { if (!ctx->zero && setup_device(ndctl_ctx, &ctx->src, 0, ctx->clear_bad_blocks)) return -1; return setup_device(ndctl_ctx, &ctx->dst, 1, ctx->clear_bad_blocks); } /* * adjust_io_len -- (internal) calculate I/O length if not specified */ static void adjust_io_len(struct daxio_context *ctx) { size_t src_len = ctx->src.maplen - ctx->src.offset; size_t dst_len = ctx->dst.maplen - ctx->dst.offset; size_t max_len = SIZE_MAX; if (ctx->zero) assert(ctx->dst.is_devdax); else assert(ctx->src.is_devdax || ctx->dst.is_devdax); if (ctx->src.is_devdax) max_len = src_len; if (ctx->dst.is_devdax) max_len = max_len < dst_len ? max_len : dst_len; /* if length is specified and is not bigger than mmapped region */ if (ctx->len != SIZE_MAX && ctx->len <= max_len) return; /* adjust len to device size */ ctx->len = max_len; } /* * cleanup_device -- (internal) unmap/close file/device */ static void cleanup_device(struct daxio_device *dev) { if (dev->addr) (void) munmap(dev->addr, dev->maplen); if (dev->path && dev->fd != -1) (void) close(dev->fd); } /* * cleanup_devices -- (internal) unmap/close input and output */ static void cleanup_devices(struct daxio_context *ctx) { cleanup_device(&ctx->dst); if (!ctx->zero) cleanup_device(&ctx->src); } /* * do_io -- (internal) write data to device/file */ static int do_io(struct ndctl_ctx *ndctl_ctx, struct daxio_context *ctx) { ssize_t cnt = 0; assert(ctx->src.is_devdax || ctx->dst.is_devdax); if (ctx->zero) { if (ctx->dst.offset > ctx->dst.maplen) { ERR("output offset larger than device size"); return -1; } if (ctx->dst.offset + ctx->len > ctx->dst.maplen) { ERR("output offset beyond device size"); return -1; } char *dst_addr = ctx->dst.addr + ctx->dst.offset; pmem_memset_persist(dst_addr, 0, ctx->len); cnt = (ssize_t)ctx->len; } else if (ctx->src.is_devdax && ctx->dst.is_devdax) { /* memcpy between src and dst */ char *src_addr = ctx->src.addr + ctx->src.offset; char *dst_addr = ctx->dst.addr + ctx->dst.offset; pmem_memcpy_persist(dst_addr, src_addr, ctx->len); cnt = (ssize_t)ctx->len; } else if (ctx->src.is_devdax) { /* write to file directly from mmap'ed src */ char *src_addr = ctx->src.addr + ctx->src.offset; if (ctx->dst.offset) { if (lseek(ctx->dst.fd, (off_t)ctx->dst.offset, SEEK_SET) < 0) { FAIL("lseek"); goto err; } } do { ssize_t wcnt = write(ctx->dst.fd, src_addr + cnt, ctx->len - (size_t)cnt); if (wcnt == -1) { FAIL("write"); goto err; } cnt += wcnt; } while ((size_t)cnt < ctx->len); } else if (ctx->dst.is_devdax) { /* read from file directly to mmap'ed dst */ char *dst_addr = ctx->dst.addr + ctx->dst.offset; if (ctx->src.offset) { if (lseek(ctx->src.fd, (off_t)ctx->src.offset, SEEK_SET) < 0) { FAIL("lseek"); return -1; } } do { ssize_t rcnt = read(ctx->src.fd, dst_addr + cnt, ctx->len - (size_t)cnt); if (rcnt == -1) { FAIL("read"); goto err; } /* end of file */ if (rcnt == 0) break; cnt = cnt + rcnt; } while ((size_t)cnt < ctx->len); pmem_persist(dst_addr, (size_t)cnt); if ((size_t)cnt != ctx->len) ERR("requested size %zu larger than source\n", ctx->len); } ERR("copied %zd bytes to device \"%s\"\n", cnt, ctx->dst.path); return 0; err: ERR("failed to perform I/O\n"); return -1; } int main(int argc, char **argv) { struct ndctl_ctx *ndctl_ctx; int ret = EXIT_SUCCESS; if (parse_args(&Ctx, argc, argv)) return EXIT_FAILURE; if (validate_args(&Ctx)) return EXIT_FAILURE; if (ndctl_new(&ndctl_ctx)) return EXIT_FAILURE; if (setup_devices(ndctl_ctx, &Ctx)) { ret = EXIT_FAILURE; goto err; } if (!Ctx.src.is_devdax && !Ctx.dst.is_devdax) { ERR("neither input nor output is device dax\n"); ret = EXIT_FAILURE; goto err; } adjust_io_len(&Ctx); if (do_io(ndctl_ctx, &Ctx)) ret = EXIT_FAILURE; err: cleanup_devices(&Ctx); ndctl_unref(ndctl_ctx); return ret; } pmdk-1.13.1/src/tools/daxio/README0000664000000000000000000000320114435627501015146 0ustar rootrootPersistent Memory Development Kit This is src/tools/daxio/README. This file contains the high-level description of daxio utility. The main purpose of daxio is to perform I/O on Device DAX devices or zero a Device DAX device. Since the standard I/O APIs (read/write) cannot be used with Device DAX, data transfer is performed on a memory-mapped device. The daxio may be used to dump Device DAX data to a file, restore data from a backup copy, or move/copy data to another device. There must be at least one Device DAX device involved either as the input or output. If input or output is not specified, it will default to stdin or stdout respectively. No length specified will default to input file/device length or to the output file/device length, if input is a special char file or stdin. For a Device DAX device, daxio will attempt to clear badblocks within range of writes before performing the I/O. 3. Source code -------------- The source code of daxio is located in daxio directory. By default daxio is installed in $(DESTDIR)/usr/bin directory. You can change it by passing $(TOOLSDIR) variable to "make install". For example, the following command will install daxio in ~/bin directory: $ make install DESTDIR=~ TOOLSDIR=/bin See the top-level README file for detailed information about building and installation. 4. Packaging ------------ The daxio utility is provided in separate packages. Both rpm and dpkg packages are built automatically with other packages. See the top-level README file for detailed information about building packages. 5. Versioning ------------- The versioning of daxio utility is the same as all PMDK libraries. pmdk-1.13.1/src/tools/Makefile.inc0000664000000000000000000001627514435627501015411 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2023, Intel Corporation # # src/tools/Makefile.inc -- Makefile include for all tools # TOP := $(dir $(lastword $(MAKEFILE_LIST)))../.. include $(TOP)/src/common.inc INSTALL_TARGET ?= y INCS += -I. INCS += -I$(TOP)/src/include INCS += $(OS_INCS) CFLAGS += -std=gnu99 CFLAGS += -Wall CFLAGS += -Werror CFLAGS += -Wmissing-prototypes CFLAGS += -Wpointer-arith CFLAGS += -Wsign-conversion CFLAGS += -Wsign-compare ifeq ($(WCONVERSION_AVAILABLE), y) CFLAGS += -Wconversion endif CFLAGS += -fno-common CFLAGS += -DSRCVERSION='"$(SRCVERSION)"' ifeq ($(OS_DIMM),ndctl) CFLAGS += -DSDS_ENABLED endif ifeq ($(IS_ICC), n) CFLAGS += -Wunused-macros CFLAGS += -Wmissing-field-initializers endif ifeq ($(WUNREACHABLE_CODE_RETURN_AVAILABLE), y) CFLAGS += -Wunreachable-code-return endif ifeq ($(WMISSING_VARIABLE_DECLARATIONS_AVAILABLE), y) CFLAGS += -Wmissing-variable-declarations endif ifeq ($(WFLOAT_EQUAL_AVAILABLE), y) CFLAGS += -Wfloat-equal endif ifeq ($(WSWITCH_DEFAULT_AVAILABLE), y) CFLAGS += -Wswitch-default endif ifeq ($(WCAST_FUNCTION_TYPE_AVAILABLE), y) CFLAGS += -Wcast-function-type endif ifeq ($(DEBUG),1) CFLAGS += -ggdb $(EXTRA_CFLAGS_DEBUG) else CFLAGS += -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 $(EXTRA_CFLAGS_RELEASE) endif ifeq ($(VALGRIND),0) CFLAGS += -DVALGRIND_ENABLED=0 CXXFLAGS += -DVALGRIND_ENABLED=0 endif ifeq ($(FAULT_INJECTION),1) CFLAGS += -DFAULT_INJECTION=1 CXXFLAGS += -DFAULT_INJECTION=1 endif ifneq ($(SANITIZE),) CFLAGS += -fsanitize=$(SANITIZE) LDFLAGS += -fsanitize=$(SANITIZE) endif LDFLAGS += $(OS_LIBS) CFLAGS += $(EXTRA_CFLAGS) LDFLAGS += -Wl,-z,relro -Wl,--warn-common -Wl,--fatal-warnings $(EXTRA_LDFLAGS) ifeq ($(DEBUG),1) LDFLAGS += -L$(TOP)/src/debug else LDFLAGS += -L$(TOP)/src/nondebug endif TARGET_DIR=$(DESTDIR)$(bindir) BASH_COMP_FILES ?= BASH_COMP_DESTDIR = $(DESTDIR)$(bashcompdir) ifneq ($(DEBUG),1) TARGET_STATIC_NONDEBUG=$(TARGET).static-nondebug endif TARGET_STATIC_DEBUG=$(TARGET).static-debug LIBSDIR=$(TOP)/src LIBSDIR_DEBUG=$(LIBSDIR)/debug LIBSDIR_NONDEBUG=$(LIBSDIR)/nondebug ifneq ($(DEBUG),) LIBSDIR_PRIV=$(LIBSDIR_DEBUG) else LIBSDIR_PRIV=$(LIBSDIR_NONDEBUG) endif PMEMLOG_PRIV_OBJ=$(LIBSDIR_PRIV)/libpmemlog/libpmemlog_unscoped.o PMEMOBJ_PRIV_OBJ=$(LIBSDIR_PRIV)/libpmemobj/libpmemobj_unscoped.o PMEMBLK_PRIV_OBJ=$(LIBSDIR_PRIV)/libpmemblk/libpmemblk_unscoped.o LIBS += $(LIBUUID) ifeq ($(LIBRT_NEEDED), y) LIBS += -lrt endif ifeq ($(TOOLS_COMMON), y) LIBPMEMCOMMON=y endif ifeq ($(LIBPMEMCOMMON), y) DYNAMIC_LIBS += -lpmemcommon STATIC_DEBUG_LIBS += $(LIBSDIR_DEBUG)/libpmemcommon.a STATIC_NONDEBUG_LIBS += $(LIBSDIR_NONDEBUG)/libpmemcommon.a CFLAGS += -I$(TOP)/src/common LIBPMEMCORE=y endif ifeq ($(LIBPMEMCORE), y) DYNAMIC_LIBS += -lpmemcore STATIC_DEBUG_LIBS += $(LIBSDIR_DEBUG)/libpmemcore.a STATIC_NONDEBUG_LIBS += $(LIBSDIR_NONDEBUG)/libpmemcore.a CFLAGS += -I$(TOP)/src/core CFLAGS += $(LIBNDCTL_CFLAGS) LIBS += $(LIBNDCTL_LIBS) endif ifeq ($(LIBPMEMPOOL), y) LIBPMEM=y DYNAMIC_LIBS += -lpmempool STATIC_DEBUG_LIBS += $(LIBSDIR_DEBUG)/libpmempool.a STATIC_NONDEBUG_LIBS += $(LIBSDIR_NONDEBUG)/libpmempool.a endif ifeq ($(LIBPMEMBLK), y) LIBPMEM=y DYNAMIC_LIBS += -lpmemblk STATIC_DEBUG_LIBS += $(LIBSDIR_DEBUG)/libpmemblk.a STATIC_NONDEBUG_LIBS += $(LIBSDIR_NONDEBUG)/libpmemblk.a endif ifeq ($(LIBPMEMLOG), y) LIBPMEM=y DYNAMIC_LIBS += -lpmemlog STATIC_DEBUG_LIBS += $(LIBSDIR_DEBUG)/libpmemlog.a STATIC_NONDEBUG_LIBS += $(LIBSDIR_NONDEBUG)/libpmemlog.a endif ifeq ($(LIBPMEMOBJ), y) LIBPMEM=y DYNAMIC_LIBS += -lpmemobj STATIC_DEBUG_LIBS += $(LIBSDIR_DEBUG)/libpmemobj.a STATIC_NONDEBUG_LIBS += $(LIBSDIR_NONDEBUG)/libpmemobj.a endif ifeq ($(LIBPMEM),y) DYNAMIC_LIBS += -lpmem STATIC_DEBUG_LIBS += $(LIBSDIR_DEBUG)/libpmem.a STATIC_NONDEBUG_LIBS += $(LIBSDIR_NONDEBUG)/libpmem.a endif ifeq ($(LIBPMEM2),y) DYNAMIC_LIBS += -lpmem2 STATIC_DEBUG_LIBS += $(LIBSDIR_DEBUG)/libpmem2.a STATIC_NONDEBUG_LIBS += $(LIBSDIR_NONDEBUG)/libpmem2.a CFLAGS += $(LIBNDCTL_CFLAGS) $(MINIASYNC_CFLAGS) LIBS += $(LIBNDCTL_LIBS) endif # If any of these libraries is required, we need to link libpthread ifneq ($(LIBPMEMCORE)$(LIBPMEMCOMMON)$(LIBPMEM)$(LIBPMEM2)$(LIBPMEMPOOL)$(LIBPMEMBLK)$(LIBPMEMLOG)$(LIBPMEMOBJ),) LIBS += -pthread endif # If any of these libraries is required, we need to link libdl ifneq ($(LIBPMEMCOMMON)$(LIBPMEMPOOL)$(LIBPMEMOBJ),) LIBS += $(LIBDL) endif ifeq ($(TOOLS_COMMON), y) CFLAGS += -I$(TOP)/src/core CFLAGS += -I$(TOP)/src/common CFLAGS += -I$(TOP)/src/libpmemlog CFLAGS += -I$(TOP)/src/libpmemblk CFLAGS += -I$(TOP)/src/libpmemobj CFLAGS += $(UNIX98_CFLAGS) endif ifeq ($(TOOLS_PMEMPOOL_COMMON), y) vpath %.c $(TOP)/src/tools/pmempool CFLAGS += -I$(TOP)/src/tools/pmempool OBJS += common.o output.o endif ifneq ($(LIBPMEMLOG_PRIV),) OBJS += pmemlog_priv.o endif ifneq ($(LIBPMEMOBJ_PRIV),) OBJS += pmemobj_priv.o endif ifneq ($(LIBPMEMBLK_PRIV),) OBJS += pmemblk_priv.o endif ifneq ($(HEADERS),) ifneq ($(filter 1 2, $(CSTYLEON)),) TMP_HEADERS := $(addsuffix tmp, $(HEADERS)) endif endif ifeq ($(COVERAGE),1) CFLAGS += $(GCOV_CFLAGS) LDFLAGS += $(GCOV_LDFLAGS) LIBS += $(GCOV_LIBS) endif MAKEFILE_DEPS=$(TOP)/src/tools/Makefile.inc $(TOP)/src/common.inc ifneq ($(TARGET),) all: $(TARGET) $(TARGET_STATIC_NONDEBUG) $(TARGET_STATIC_DEBUG) else all: endif clean: $(RM) $(OBJS) $(CLEAN_FILES) $(TMP_HEADERS) clobber: clean ifneq ($(TARGET),) $(RM) $(TARGET) $(RM) $(TARGET_STATIC_NONDEBUG) $(RM) $(TARGET_STATIC_DEBUG) $(RM) -r .deps endif install: all ifeq ($(INSTALL_TARGET),y) ifneq ($(TARGET),) install -d $(TARGET_DIR) install -p -m 0755 $(TARGET) $(TARGET_DIR) endif ifneq ($(BASH_COMP_FILES),) install -d $(BASH_COMP_DESTDIR) install -p -m 0644 $(BASH_COMP_FILES) $(BASH_COMP_DESTDIR) endif endif uninstall: ifeq ($(INSTALL_TARGET),y) ifneq ($(TARGET),) $(RM) $(TARGET_DIR)/$(TARGET) endif ifneq ($(BASH_COMP_FILES),) $(RM) $(BASH_COMP_DESTDIR)/$(BASH_COMP_FILES) endif endif %.gz: % gzip -nc ./$< > $@ %.txt: % man ./$< > $@ $(TARGET) $(TARGET_STATIC_DEBUG) $(TARGET_STATIC_NONDEBUG): $(TMP_HEADERS) $(OBJS) $(MAKEFILE_DEPS) $(TARGET_STATIC_DEBUG): $(STATIC_DEBUG_LIBS) $(CC) $(LDFLAGS) -o $@ $(OBJS) $(STATIC_DEBUG_LIBS) $(LIBS) $(TARGET_STATIC_NONDEBUG): $(STATIC_NONDEBUG_LIBS) $(CC) $(LDFLAGS) -o $@ $(OBJS) $(STATIC_NONDEBUG_LIBS) $(LIBS) $(TARGET): $(CC) $(LDFLAGS) -o $@ $(OBJS) $(DYNAMIC_LIBS) $(LIBS) $(PMEMLOG_PRIV_OBJ): $(MAKE) -C $(LIBSDIR) libpmemlog pmemlog_priv.o: $(PMEMLOG_PRIV_OBJ) $(OBJCOPY) --localize-hidden $(addprefix -G, $(LIBPMEMLOG_PRIV)) $< $@ $(PMEMOBJ_PRIV_OBJ): $(MAKE) -C $(LIBSDIR) libpmemobj pmemobj_priv.o: $(PMEMOBJ_PRIV_OBJ) $(OBJCOPY) --localize-hidden $(addprefix -G, $(LIBPMEMOBJ_PRIV)) $< $@ $(PMEMBLK_PRIV_OBJ): $(MAKE) -C $(LIBSDIR) libpmemblk pmemblk_priv.o: $(PMEMBLK_PRIV_OBJ) $(OBJCOPY) --localize-hidden $(addprefix -G, $(LIBPMEMBLK_PRIV)) $< $@ objdir=. %.o: %.c $(MAKEFILE_DEPS) $(call check-cstyle, $<) @mkdir -p .deps $(CC) -MD $(CFLAGS) $(INCS) -c -o $@ $(call coverage-path, $<) $(call check-os, $@, $<) $(create-deps) %.htmp: %.h $(call check-cstyle, $<, $@) test check pcheck: all TESTCONFIG=$(TOP)/src/test/testconfig.sh $(TESTCONFIG): sparse: $(if $(TARGET), $(sparse-c)) .PHONY: all clean clobber install uninstall test check pcheck -include .deps/*.P pmdk-1.13.1/src/examples/0000775000000000000000000000000014435627501013644 5ustar rootrootpmdk-1.13.1/src/examples/.gitignore0000664000000000000000000000000414435627501015626 0ustar rootroot*.o pmdk-1.13.1/src/examples/libpmem/0000775000000000000000000000000014435627501015271 5ustar rootrootpmdk-1.13.1/src/examples/libpmem/.gitignore0000664000000000000000000000003614435627501017260 0ustar rootrootmanpage simple_copy full_copy pmdk-1.13.1/src/examples/libpmem/simple_copy.vcxproj.filters0000664000000000000000000000065014435627501022701 0ustar rootroot {3bc86b19-55f2-4b90-9ccd-1470361ca84c} Source Files pmdk-1.13.1/src/examples/libpmem/full_copy.vcxproj0000664000000000000000000000453214435627501020706 0ustar rootroot Debug x64 Release x64 {0287C3DC-AE03-4714-AAFF-C52F062ECA6F} pmem 10.0.22000.0 Application true v143 Application false v143 ..\..\LongPath.manifest libpmem.lib;%(AdditionalDependencies) {9e9e3d25-2139-4a5d-9200-18148ddead45} pmdk-1.13.1/src/examples/libpmem/simple_copy.vcxproj0000664000000000000000000000453414435627501021237 0ustar rootroot Debug x64 Release x64 {D062166F-0EC7-4C13-A772-0C7157EEFE41} pmem 10.0.22000.0 Application true v143 Application false v143 ..\..\LongPath.manifest libpmem.lib;%(AdditionalDependencies) {9e9e3d25-2139-4a5d-9200-18148ddead45} pmdk-1.13.1/src/examples/libpmem/Makefile0000664000000000000000000000046014435627501016731 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2016, Intel Corporation # # examples/libpmem/Makefile -- build the libpmem examples # PROGS = manpage simple_copy full_copy LIBS = -lpmem -pthread include ../Makefile.inc manpage: manpage.o simple_copy: simple_copy.o full_copy: full_copy.o pmdk-1.13.1/src/examples/libpmem/README0000664000000000000000000000140414435627501016150 0ustar rootrootPersistent Memory Development Kit This is examples/libpmem/README. This directory contains examples for libpmem, the library containing low-level persistent memory support. A detailed explanation of these examples can be found here: https://pmem.io/pmdk/libpmem manpage.c is the example used in the libpmem man page. simple_copy.c is a simple pmem_memcpy() example. full_copy.c shows how to use pmem_memcpy_nodrain(). To build these examples: make These examples can be built against an installed system using: make LIBDIR=/usr/lib INCDIR=/usr/include If you're looking for documentation to get you started using PMDK, start here: https://pmem.io/pmdk and follow the links to examples and man pages. Developers new to PMDK are probably looking for libpmemobj. pmdk-1.13.1/src/examples/libpmem/simple_copy.c0000664000000000000000000000266414435627501017770 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2017, Intel Corporation */ /* * simple_copy.c -- show how to use pmem_memcpy_persist() * * usage: simple_copy src-file dst-file * * Reads 4k from src-file and writes it to dst-file. */ #include #include #include #include #include #include #ifndef _WIN32 #include #else #include #endif #include #include /* just copying 4k to pmem for this example */ #define BUF_LEN 4096 int main(int argc, char *argv[]) { int srcfd; char buf[BUF_LEN]; char *pmemaddr; size_t mapped_len; int is_pmem; int cc; if (argc != 3) { fprintf(stderr, "usage: %s src-file dst-file\n", argv[0]); exit(1); } /* open src-file */ if ((srcfd = open(argv[1], O_RDONLY)) < 0) { perror(argv[1]); exit(1); } /* create a pmem file and memory map it */ if ((pmemaddr = pmem_map_file(argv[2], BUF_LEN, PMEM_FILE_CREATE|PMEM_FILE_EXCL, 0666, &mapped_len, &is_pmem)) == NULL) { perror("pmem_map_file"); exit(1); } /* read up to BUF_LEN from srcfd */ if ((cc = read(srcfd, buf, BUF_LEN)) < 0) { pmem_unmap(pmemaddr, mapped_len); perror("read"); exit(1); } /* write it to the pmem */ if (is_pmem) { pmem_memcpy_persist(pmemaddr, buf, cc); } else { memcpy(pmemaddr, buf, cc); pmem_msync(pmemaddr, cc); } close(srcfd); pmem_unmap(pmemaddr, mapped_len); exit(0); } pmdk-1.13.1/src/examples/libpmem/full_copy.vcxproj.filters0000664000000000000000000000064614435627501022357 0ustar rootroot {0f7bab61-c1ad-4a74-8663-fe40f393b2eb} Source Files pmdk-1.13.1/src/examples/libpmem/full_copy.c0000664000000000000000000000451314435627501017434 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2017, Intel Corporation */ /* * full_copy.c -- show how to use pmem_memcpy_nodrain() * * usage: full_copy src-file dst-file * * Copies src-file to dst-file in 4k chunks. */ #include #include #include #include #include #include #ifndef _WIN32 #include #else #include #endif #include #include /* copying 4k at a time to pmem for this example */ #define BUF_LEN 4096 /* * do_copy_to_pmem -- copy to pmem, postponing drain step until the end */ static void do_copy_to_pmem(char *pmemaddr, int srcfd, off_t len) { char buf[BUF_LEN]; int cc; /* copy the file, saving the last flush step to the end */ while ((cc = read(srcfd, buf, BUF_LEN)) > 0) { pmem_memcpy_nodrain(pmemaddr, buf, cc); pmemaddr += cc; } if (cc < 0) { perror("read"); exit(1); } /* perform final flush step */ pmem_drain(); } /* * do_copy_to_non_pmem -- copy to a non-pmem memory mapped file */ static void do_copy_to_non_pmem(char *addr, int srcfd, off_t len) { char *startaddr = addr; char buf[BUF_LEN]; int cc; /* copy the file, saving the last flush step to the end */ while ((cc = read(srcfd, buf, BUF_LEN)) > 0) { memcpy(addr, buf, cc); addr += cc; } if (cc < 0) { perror("read"); exit(1); } /* flush it */ if (pmem_msync(startaddr, len) < 0) { perror("pmem_msync"); exit(1); } } int main(int argc, char *argv[]) { int srcfd; struct stat stbuf; char *pmemaddr; size_t mapped_len; int is_pmem; if (argc != 3) { fprintf(stderr, "usage: %s src-file dst-file\n", argv[0]); exit(1); } /* open src-file */ if ((srcfd = open(argv[1], O_RDONLY)) < 0) { perror(argv[1]); exit(1); } /* find the size of the src-file */ if (fstat(srcfd, &stbuf) < 0) { perror("fstat"); exit(1); } /* create a pmem file and memory map it */ if ((pmemaddr = pmem_map_file(argv[2], stbuf.st_size, PMEM_FILE_CREATE|PMEM_FILE_EXCL, 0666, &mapped_len, &is_pmem)) == NULL) { perror("pmem_map_file"); exit(1); } /* determine if range is true pmem, call appropriate copy routine */ if (is_pmem) do_copy_to_pmem(pmemaddr, srcfd, stbuf.st_size); else do_copy_to_non_pmem(pmemaddr, srcfd, stbuf.st_size); close(srcfd); pmem_unmap(pmemaddr, mapped_len); exit(0); } pmdk-1.13.1/src/examples/libpmem/manpage.vcxproj0000664000000000000000000000453014435627501020320 0ustar rootroot Debug x64 Release x64 {FCD0587A-4504-4F5E-8E9C-468CC03D250A} pmem 10.0.22000.0 Application true v143 Application false v143 ..\..\LongPath.manifest libpmem.lib;%(AdditionalDependencies) {9e9e3d25-2139-4a5d-9200-18148ddead45} pmdk-1.13.1/src/examples/libpmem/manpage.vcxproj.filters0000664000000000000000000000075514435627501021774 0ustar rootroot {4FC737F1-C7A5-4376-A066-2A32D752A2FF} cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx Source Files pmdk-1.13.1/src/examples/libpmem/manpage.c0000664000000000000000000000220014435627501017037 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2019, Intel Corporation */ /* * manpage.c -- simple example for the libpmem man page */ #include #include #include #include #include #include #ifndef _WIN32 #include #else #include #endif #include #include /* using 4k of pmem for this example */ #define PMEM_LEN 4096 #define PATH "/pmem-fs/myfile" int main(int argc, char *argv[]) { char *pmemaddr; size_t mapped_len; int is_pmem; /* create a pmem file and memory map it */ if ((pmemaddr = pmem_map_file(PATH, PMEM_LEN, PMEM_FILE_CREATE, 0666, &mapped_len, &is_pmem)) == NULL) { perror("pmem_map_file"); exit(1); } /* store a string to the persistent memory */ strcpy(pmemaddr, "hello, persistent memory"); /* flush above strcpy to persistence */ if (is_pmem) pmem_persist(pmemaddr, mapped_len); else pmem_msync(pmemaddr, mapped_len); /* * Delete the mappings. The region is also * automatically unmapped when the process is * terminated. */ pmem_unmap(pmemaddr, mapped_len); return 0; } pmdk-1.13.1/src/examples/libpmemlog/0000775000000000000000000000000014435627501015773 5ustar rootrootpmdk-1.13.1/src/examples/libpmemlog/.gitignore0000664000000000000000000000001014435627501017752 0ustar rootrootmanpage pmdk-1.13.1/src/examples/libpmemlog/Makefile0000664000000000000000000000066114435627501017436 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2023, Intel Corporation # # examples/libpmemlog/Makefile -- build the libpmemlog examples # PROGS = manpage DIRS = logfile LIBS = -lpmemlog -lpmem -pthread include ../Makefile.inc # Libpmemlog is deprecated. # This flag allows to build tests, examples and benchmarks # using pmemlog despite the deprecated state. CFLAGS += -Wno-deprecated-declarations manpage: manpage.o pmdk-1.13.1/src/examples/libpmemlog/README0000664000000000000000000000131514435627501016653 0ustar rootrootPersistent Memory Development Kit This is examples/libpmemlog/README. This directory contains examples for libpmemlog, the library providing pmem-resident log files. Some of these examples are explained in more detail here: https://pmem.io/pmdk/libpmemlog manpage.c is the example used in the libpmemlog man page. logfile implements a simple log using libpmemlog. To build these examples: make These examples can be built against an installed system using: make LIBDIR=/usr/lib INCDIR=/usr/include If you're looking for documentation to get you started using PMDK, start here: https://pmem.io/pmdk and follow the links to examples and man pages. Developers new to PMDK are probably looking for libpmemobj. pmdk-1.13.1/src/examples/libpmemlog/logfile/0000775000000000000000000000000014435627501017414 5ustar rootrootpmdk-1.13.1/src/examples/libpmemlog/logfile/.gitignore0000664000000000000000000000002014435627501021374 0ustar rootrootaddlog printlog pmdk-1.13.1/src/examples/libpmemlog/logfile/addlog.vcxproj.filters0000664000000000000000000000144214435627501023733 0ustar rootroot {d3d73d42-d99e-4314-97df-e9d6289423a5} cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx {416df294-73b9-4a41-b319-8d0c1922b01b} h;hh;hpp;hxx;hm;inl;inc;xsd Source Files Header Files pmdk-1.13.1/src/examples/libpmemlog/logfile/printlog.c0000664000000000000000000000311614435627501021417 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2019, Intel Corporation */ /* * printlog -- given a log file, print the entries * * Usage: * printlog [-t] /path/to/pm-aware/file * * -t option means truncate the file after printing it. */ #include #include #include #include #include #include #include #include "logentry.h" /* * printlog -- callback function called when walking the log */ static int printlog(const void *buf, size_t len, void *arg) { /* first byte after log contents */ const void *endp = (char *)buf + len; /* for each entry in the log... */ while (buf < endp) { struct logentry *headerp = (struct logentry *)buf; buf = (char *)buf + sizeof(struct logentry); /* print the header */ printf("Entry from pid: %d\n", headerp->pid); printf(" Created: %s", ctime(&headerp->timestamp)); printf(" Contents:\n"); /* print the log data itself, it is NUL-terminated */ printf("%s", (char *)buf); buf = (char *)buf + headerp->len; } return 0; } int main(int argc, char *argv[]) { int ind = 1; int tflag = 0; PMEMlogpool *plp; if (argc > 2) { if (strcmp(argv[1], "-t") == 0) { tflag = 1; ind++; } else { fprintf(stderr, "usage: %s [-t] file\n", argv[0]); exit(1); } } const char *path = argv[ind]; if ((plp = pmemlog_open(path)) == NULL) { perror(path); exit(1); } /* the rest of the work happens in printlog() above */ pmemlog_walk(plp, 0, printlog, NULL); if (tflag) pmemlog_rewind(plp); pmemlog_close(plp); return 0; } pmdk-1.13.1/src/examples/libpmemlog/logfile/Makefile0000664000000000000000000000076614435627501021065 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2023, Intel Corporation # # examples/logfile/Makefile -- build the Persistent Memory Development Kit examples # PROGS = addlog printlog LIBS = -lpmemlog -lpmem -pthread include ../../Makefile.inc # Libpmemlog is deprecated. # This flag allows to build tests, examples and benchmarks # using pmemlog despite the deprecated state. CFLAGS += -Wno-deprecated-declarations addlog: addlog.o printlog: printlog.o addlog.o printlog.o: logentry.h pmdk-1.13.1/src/examples/libpmemlog/logfile/README0000664000000000000000000000171314435627501020276 0ustar rootrootPersistent Memory Development Kit This is examples/libpmemlog/logfile/README. The example in this directory uses persistent memory to implement a simple log file using libpmemlog. To run this example, follow these steps: 0. Build the example with "make". The libraries must be built first (i.e. by running "make" in ../../..). 1. Create the log file. This can be anywhere but the point is it will be a much faster log file if it is created on a pmem-aware file system. For example, if /pmem is the mount point for a pmem-aware file system: $ fallocate -l 1G /pmem/logfile 2. Append to the log file as many times as you like: $ addlog /pmem/logfile "Hello there." $ addlog /pmem/logfile "First line." "Second line." ... 3. Print the contents of the log any time using: $ printlog /pmem/logfile 4. The printlog command will throw away the current log file contents after printing it if given the -t argument: $ printlog -t /pmem/logfile pmdk-1.13.1/src/examples/libpmemlog/logfile/printlog.vcxproj0000664000000000000000000000544614435627501022700 0ustar rootroot Debug x64 Release x64 {C3CEE34C-29E0-4A22-B258-3FBAF662AA19} pmemlog 10.0.22000.0 {0b1818eb-bdc8-4865-964f-db8bf05cfd86} {9e9e3d25-2139-4a5d-9200-18148ddead45} Application true v143 Application false v143 true ..\..\..\LongPath.manifest libpmem.lib;libpmemlog.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies) pmdk-1.13.1/src/examples/libpmemlog/logfile/printlog.vcxproj.filters0000664000000000000000000000144414435627501024341 0ustar rootroot {a0af7281-6084-422f-946c-e05d18224229} cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx {9eebac3f-530b-4b3b-b0c7-68f43650d681} h;hh;hpp;hxx;hm;inl;inc;xsd Source Files Header Files pmdk-1.13.1/src/examples/libpmemlog/logfile/printlog.filters0000664000000000000000000000124014435627501022641 0ustar rootroot {7b7d2f5a-464e-4d55-894b-3110e15303f1} {c2e783a4-4f6d-419d-bbf6-1056eddfdd8f} Source Files Header Files pmdk-1.13.1/src/examples/libpmemlog/logfile/logentry.h0000664000000000000000000000043314435627501021430 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2014-2020, Intel Corporation */ /* * info prepended to each log entry... */ struct logentry { size_t len; /* length of the rest of the log entry */ time_t timestamp; #ifndef _WIN32 pid_t pid; #else int pid; #endif }; pmdk-1.13.1/src/examples/libpmemlog/logfile/addlog.filters0000664000000000000000000000123614435627501022242 0ustar rootroot {7b7d2f5a-464e-4d55-894b-3110e15303f1} {c2e783a4-4f6d-419d-bbf6-1056eddfdd8f} Source Files Header Files pmdk-1.13.1/src/examples/libpmemlog/logfile/addlog.vcxproj0000664000000000000000000000551414435627501022270 0ustar rootroot Debug x64 Release x64 {A7CA7975-CEDB-48E6-9AEB-1209DCBD07F2} pmemlog 10.0.22000.0 {0b1818eb-bdc8-4865-964f-db8bf05cfd86} {9e9e3d25-2139-4a5d-9200-18148ddead45} Application true v143 NotSet Application false v143 true ..\..\..\LongPath.manifest libpmem.lib;libpmemlog.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies) pmdk-1.13.1/src/examples/libpmemlog/logfile/addlog.c0000664000000000000000000000471714435627501021023 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2019, Intel Corporation */ /* * addlog -- given a log file, append a log entry * * Usage: * fallocate -l 1G /path/to/pm-aware/file * addlog /path/to/pm-aware/file "first line of entry" "second line" */ #include #include #include #include #include #include #include #include #include "logentry.h" int main(int argc, char *argv[]) { PMEMlogpool *plp; struct logentry header; struct iovec *iovp; struct iovec *next_iovp; int iovcnt; if (argc < 3) { fprintf(stderr, "usage: %s filename lines...\n", argv[0]); exit(1); } const char *path = argv[1]; /* create the log in the given file, or open it if already created */ plp = pmemlog_create(path, 0, CREATE_MODE_RW); if (plp == NULL && (plp = pmemlog_open(path)) == NULL) { perror(path); exit(1); } /* fill in the header */ time(&header.timestamp); header.pid = getpid(); /* * Create an iov for pmemlog_appendv(). For each argument given, * allocate two entries (one for the string, one for the newline * appended to the string). Allocate 1 additional entry for the * header that gets prepended to the entry. */ iovcnt = (argc - 2) * 2 + 2; if ((iovp = malloc(sizeof(*iovp) * iovcnt)) == NULL) { perror("malloc"); exit(1); } next_iovp = iovp; /* put the header into iov first */ next_iovp->iov_base = &header; next_iovp->iov_len = sizeof(header); next_iovp++; /* * Now put each arg in, following it with the string "\n". * Calculate a total character count in header.len along the way. */ header.len = 0; for (int arg = 2; arg < argc; arg++) { /* add the string given */ next_iovp->iov_base = argv[arg]; next_iovp->iov_len = strlen(argv[arg]); header.len += next_iovp->iov_len; next_iovp++; /* add the newline */ next_iovp->iov_base = "\n"; next_iovp->iov_len = 1; header.len += 1; next_iovp++; } /* * pad with NULs (at least one) to align next entry to sizeof(long long) * bytes */ int a = sizeof(long long); int len_to_round = 1 + (a - (header.len + 1) % a) % a; char *buf[sizeof(long long)] = {0}; next_iovp->iov_base = buf; next_iovp->iov_len = len_to_round; header.len += len_to_round; next_iovp++; /* atomically add it all to the log */ if (pmemlog_appendv(plp, iovp, iovcnt) < 0) { perror("pmemlog_appendv"); free(iovp); exit(1); } free(iovp); pmemlog_close(plp); return 0; } pmdk-1.13.1/src/examples/libpmemlog/manpage.vcxproj0000664000000000000000000000532314435627501021023 0ustar rootroot Debug x64 Release x64 {9FF51F3E-AF36-4F45-A797-C5F03A090298} pememlog 10.0.22000.0 Application true v143 Application false v143 true ..\..\LongPath.manifest libpmem.lib;libpmemlog.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies) {0b1818eb-bdc8-4865-964f-db8bf05cfd86} {9e9e3d25-2139-4a5d-9200-18148ddead45} pmdk-1.13.1/src/examples/libpmemlog/manpage.vcxproj.filters0000664000000000000000000000125414435627501022471 0ustar rootroot {4FC737F1-C7A5-4376-A066-2A32D752A2FF} cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx {93995380-89BD-4b04-88EB-625FBE52EBFB} h;hh;hpp;hxx;hm;inl;inc;xsd Source Files pmdk-1.13.1/src/examples/libpmemlog/manpage.c0000664000000000000000000000266014435627501017553 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2014-2019, Intel Corporation */ /* * manpage.c -- simple example for the libpmemlog man page */ #include #include #include #include #ifndef _WIN32 #include #endif #include #include /* size of the pmemlog pool -- 1 GB */ #define POOL_SIZE ((size_t)(1 << 30)) /* * printit -- log processing callback for use with pmemlog_walk() */ static int printit(const void *buf, size_t len, void *arg) { fwrite(buf, len, 1, stdout); return 0; } int main(int argc, char *argv[]) { const char path[] = "/pmem-fs/myfile"; PMEMlogpool *plp; size_t nbyte; char *str; /* create the pmemlog pool or open it if it already exists */ plp = pmemlog_create(path, POOL_SIZE, 0666); if (plp == NULL) plp = pmemlog_open(path); if (plp == NULL) { perror(path); exit(1); } /* how many bytes does the log hold? */ nbyte = pmemlog_nbyte(plp); printf("log holds %zu bytes\n", nbyte); /* append to the log... */ str = "This is the first string appended\n"; if (pmemlog_append(plp, str, strlen(str)) < 0) { perror("pmemlog_append"); exit(1); } str = "This is the second string appended\n"; if (pmemlog_append(plp, str, strlen(str)) < 0) { perror("pmemlog_append"); exit(1); } /* print the log contents */ printf("log contains:\n"); pmemlog_walk(plp, 0, printit, NULL); pmemlog_close(plp); return 0; } pmdk-1.13.1/src/examples/libpmempool/0000775000000000000000000000000014435627501016163 5ustar rootrootpmdk-1.13.1/src/examples/libpmempool/.gitignore0000664000000000000000000000001014435627501020142 0ustar rootrootmanpage pmdk-1.13.1/src/examples/libpmempool/Makefile0000664000000000000000000000035514435627501017626 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016, Intel Corporation # # examples/libpmempool/Makefile -- build the libpmempool examples # PROGS = manpage LIBS = -lpmempool -pthread include ../Makefile.inc manpage: manpage.o pmdk-1.13.1/src/examples/libpmempool/README0000664000000000000000000000130214435627501017037 0ustar rootrootPersistent Memory Development Kit This is examples/libpmempool/README. This directory contains examples for libpmempool, the library providing support for off-line pool management, diagnostics and repair. Some of these examples are explained in more detail here: https://pmem.io/pmdk/libpmempool manpage.c is the example used in the libpmempool man page. To build these examples: make These examples can be built against an installed system using: make LIBDIR=/usr/lib INCDIR=/usr/include If you're looking for documentation to get you started using PMDK, start here: https://pmem.io/pmdk and follow the links to examples and man pages. Developers new to PMDK are probably looking for libpmemobj. pmdk-1.13.1/src/examples/libpmempool/manpage.c0000664000000000000000000000302314435627501017735 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016, Intel Corporation */ /* * manpage.c -- simple example for the libpmempool man page */ #include #include #include #include #include #define PATH "./pmem-fs/myfile" #define CHECK_FLAGS (PMEMPOOL_CHECK_FORMAT_STR|PMEMPOOL_CHECK_REPAIR|\ PMEMPOOL_CHECK_VERBOSE) int main(int argc, char *argv[]) { PMEMpoolcheck *ppc; struct pmempool_check_status *status; enum pmempool_check_result ret; /* arguments for check */ struct pmempool_check_args args = { .path = PATH, .backup_path = NULL, .pool_type = PMEMPOOL_POOL_TYPE_DETECT, .flags = CHECK_FLAGS }; /* initialize check context */ if ((ppc = pmempool_check_init(&args, sizeof(args))) == NULL) { perror("pmempool_check_init"); exit(EXIT_FAILURE); } /* perform check and repair, answer 'yes' for each question */ while ((status = pmempool_check(ppc)) != NULL) { switch (status->type) { case PMEMPOOL_CHECK_MSG_TYPE_ERROR: printf("%s\n", status->str.msg); break; case PMEMPOOL_CHECK_MSG_TYPE_INFO: printf("%s\n", status->str.msg); break; case PMEMPOOL_CHECK_MSG_TYPE_QUESTION: printf("%s\n", status->str.msg); status->str.answer = "yes"; break; default: pmempool_check_end(ppc); exit(EXIT_FAILURE); } } /* finalize the check and get the result */ ret = pmempool_check_end(ppc); switch (ret) { case PMEMPOOL_CHECK_RESULT_CONSISTENT: case PMEMPOOL_CHECK_RESULT_REPAIRED: return 0; default: return 1; } } pmdk-1.13.1/src/examples/examples_debug.props0000664000000000000000000000246714435627501017726 0ustar rootroot $(Platform)\$(Configuration)\$(TargetName)\ ex_$(RootNamespace)_$(ProjectName) $(SolutionDir)$(Platform)\$(Configuration)\examples\ .;$(solutionDir)include;$(solutionDir)..\include;$(ProjectDir)..\..\;$(ProjectDir)..\;$(IncludePath);$(WindowsSDK_IncludePath) $(ProjectDir)..\..\x64\$(Configuration)\libs;$(ProjectDir)..\..\..\x64\$(Configuration)\libs;$(VC_LibraryPath_x64);$(WindowsSDK_LibraryPath_x64);$(NETFXKitsDir)Lib\um\x64 Level3 Disabled true 4996 PMDK_UTF8_API;SDS_ENABLED;NTDDI_VERSION=NTDDI_WIN10_RS1;_MBCS;%(PreprocessorDefinitions) true true pmdk-1.13.1/src/examples/examples_release.props0000664000000000000000000000301514435627501020246 0ustar rootroot $(Platform)\$(Configuration)\$(TargetName)\ ex_$(RootNamespace)_$(ProjectName) $(SolutionDir)$(Platform)\$(Configuration)\examples\ $(ProjectDir)..\..\x64\$(Configuration)\libs;$(ProjectDir)..\..\..\x64\$(Configuration)\libs;$(VC_LibraryPath_x64);$(WindowsSDK_LibraryPath_x64);$(NETFXKitsDir)Lib\um\x64 .;$(solutionDir)include;$(solutionDir)..\include;$(ProjectDir)..\..\;$(ProjectDir)..\;$(IncludePath);$(WindowsSDK_IncludePath) Level3 MaxSpeed true 4996 true true PMDK_UTF8_API;SDS_ENABLED;NTDDI_VERSION=NTDDI_WIN10_RS1;_MBCS;%(PreprocessorDefinitions) true true true true pmdk-1.13.1/src/examples/libpmemobj/0000775000000000000000000000000014435627501015764 5ustar rootrootpmdk-1.13.1/src/examples/libpmemobj/.gitignore0000664000000000000000000000007614435627501017757 0ustar rootroot*~ pi manpage btree rtree lists setjmp buffons_needle_problem pmdk-1.13.1/src/examples/libpmemobj/pi.vcxproj.filters0000664000000000000000000000063714435627501021466 0ustar rootroot {d1b32241-e048-49ce-904f-d7f06f71f38c} Source Files pmdk-1.13.1/src/examples/libpmemobj/pi.c0000664000000000000000000001277714435627501016556 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2019, Intel Corporation */ /* * pi.c -- example usage of user lists * * Calculates pi number with multiple threads using Leibniz formula. */ #include #include #include #include #include #include #include #include #ifndef _WIN32 #include #endif /* * Layout definition */ POBJ_LAYOUT_BEGIN(pi); POBJ_LAYOUT_ROOT(pi, struct pi); POBJ_LAYOUT_TOID(pi, struct pi_task); POBJ_LAYOUT_END(pi); static PMEMobjpool *pop; struct pi_task_proto { uint64_t start; uint64_t stop; long double result; }; struct pi_task { struct pi_task_proto proto; POBJ_LIST_ENTRY(struct pi_task) todo; POBJ_LIST_ENTRY(struct pi_task) done; }; struct pi { POBJ_LIST_HEAD(todo, struct pi_task) todo; POBJ_LIST_HEAD(done, struct pi_task) done; }; /* * pi_task_construct -- task constructor */ static int pi_task_construct(PMEMobjpool *_pop, void *ptr, void *arg) { struct pi_task *t = (struct pi_task *)ptr; struct pi_task_proto *p = (struct pi_task_proto *)arg; t->proto = *p; pmemobj_persist(_pop, t, sizeof(*t)); return 0; } /* * calc_pi -- worker for pi calculation */ #ifndef _WIN32 static void * calc_pi(void *arg) #else static DWORD WINAPI calc_pi(LPVOID arg) #endif { TOID(struct pi) pi = POBJ_ROOT(pop, struct pi); TOID(struct pi_task) task = *((TOID(struct pi_task) *)arg); long double result = 0; for (uint64_t i = D_RO(task)->proto.start; i < D_RO(task)->proto.stop; ++i) { result += (pow(-1, (double)i) / (2 * i + 1)); } D_RW(task)->proto.result = result; pmemobj_persist(pop, &D_RW(task)->proto.result, sizeof(result)); POBJ_LIST_MOVE_ELEMENT_HEAD(pop, &D_RW(pi)->todo, &D_RW(pi)->done, task, todo, done); return NULL; } /* * calc_pi_mt -- calculate all the pending to-do tasks */ static void calc_pi_mt(void) { TOID(struct pi) pi = POBJ_ROOT(pop, struct pi); int pending = 0; TOID(struct pi_task) iter; POBJ_LIST_FOREACH(iter, &D_RO(pi)->todo, todo) pending++; if (pending == 0) return; int i = 0; TOID(struct pi_task) *tasks = (TOID(struct pi_task) *)malloc( sizeof(TOID(struct pi_task)) * pending); if (tasks == NULL) { fprintf(stderr, "failed to allocate tasks\n"); return; } POBJ_LIST_FOREACH(iter, &D_RO(pi)->todo, todo) tasks[i++] = iter; #ifndef _WIN32 pthread_t workers[pending]; for (i = 0; i < pending; ++i) if (pthread_create(&workers[i], NULL, calc_pi, &tasks[i]) != 0) break; for (i = i - 1; i >= 0; --i) pthread_join(workers[i], NULL); #else HANDLE *workers = (HANDLE *) malloc(sizeof(HANDLE) * pending); for (i = 0; i < pending; ++i) { workers[i] = CreateThread(NULL, 0, calc_pi, &tasks[i], 0, NULL); if (workers[i] == NULL) break; } WaitForMultipleObjects(i, workers, TRUE, INFINITE); for (i = i - 1; i >= 0; --i) CloseHandle(workers[i]); free(workers); #endif free(tasks); } /* * prep_todo_list -- create tasks to be done */ static int prep_todo_list(int threads, int ops) { TOID(struct pi) pi = POBJ_ROOT(pop, struct pi); if (!POBJ_LIST_EMPTY(&D_RO(pi)->todo)) return -1; int ops_per_thread = ops / threads; uint64_t last = 0; /* last calculated denominator */ TOID(struct pi_task) iter; POBJ_LIST_FOREACH(iter, &D_RO(pi)->done, done) { if (last < D_RO(iter)->proto.stop) last = D_RO(iter)->proto.stop; } int i; for (i = 0; i < threads; ++i) { uint64_t start = last + (i * ops_per_thread); struct pi_task_proto proto; proto.start = start; proto.stop = start + ops_per_thread; proto.result = 0; POBJ_LIST_INSERT_NEW_HEAD(pop, &D_RW(pi)->todo, todo, sizeof(struct pi_task), pi_task_construct, &proto); } return 0; } int main(int argc, char *argv[]) { if (argc < 3) { printf("usage: %s file-name " "[print|done|todo|finish|calc <# of threads> ]\n", argv[0]); return 1; } const char *path = argv[1]; pop = NULL; if (file_exists(path) != 0) { if ((pop = pmemobj_create(path, POBJ_LAYOUT_NAME(pi), PMEMOBJ_MIN_POOL, CREATE_MODE_RW)) == NULL) { printf("failed to create pool\n"); return 1; } } else { if ((pop = pmemobj_open(path, POBJ_LAYOUT_NAME(pi))) == NULL) { printf("failed to open pool\n"); return 1; } } TOID(struct pi) pi = POBJ_ROOT(pop, struct pi); char op = argv[2][0]; switch (op) { case 'p': { /* print pi */ long double pi_val = 0; TOID(struct pi_task) iter; POBJ_LIST_FOREACH(iter, &D_RO(pi)->done, done) { pi_val += D_RO(iter)->proto.result; } printf("pi: %.10Lf\n", pi_val * 4); } break; case 'd': { /* print done list */ TOID(struct pi_task) iter; POBJ_LIST_FOREACH(iter, &D_RO(pi)->done, done) { printf("(%" PRIu64 " - %" PRIu64 ") = %.10Lf\n", D_RO(iter)->proto.start, D_RO(iter)->proto.stop, D_RO(iter)->proto.result); } } break; case 't': { /* print to-do list */ TOID(struct pi_task) iter; POBJ_LIST_FOREACH(iter, &D_RO(pi)->todo, todo) { printf("(%" PRIu64 " - %" PRIu64 ") = %.10Lf\n", D_RO(iter)->proto.start, D_RO(iter)->proto.stop, D_RO(iter)->proto.result); } } break; case 'c': { /* calculate pi */ if (argc < 5) { printf("usage: %s file-name " "calc <# of threads> \n", argv[0]); return 1; } int threads = atoi(argv[3]); int ops = atoi(argv[4]); assert((threads > 0) && (ops > 0)); if (prep_todo_list(threads, ops) == -1) printf("pending todo tasks\n"); else calc_pi_mt(); } break; case 'f': { /* finish to-do tasks */ calc_pi_mt(); } break; } pmemobj_close(pop); return 0; } pmdk-1.13.1/src/examples/libpmemobj/lists.vcxproj0000664000000000000000000000551414435627501020544 0ustar rootroot Debug x64 Release x64 {2CD7408E-2F60-43C3-ACEB-C7D58CDD8462} pmemobj 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} Application true v143 Application false v143 true ..\..\LongPath.manifest CompileAsCpp libpmem.lib;libpmemobj.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies) pmdk-1.13.1/src/examples/libpmemobj/queue/0000775000000000000000000000000014435627501017110 5ustar rootrootpmdk-1.13.1/src/examples/libpmemobj/queue/.gitignore0000664000000000000000000000000614435627501021074 0ustar rootrootqueue pmdk-1.13.1/src/examples/libpmemobj/queue/Makefile0000664000000000000000000000033514435627501020551 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2018, Intel Corporation # # examples/libpmemobj/queue/Makefile -- build the queue example # PROGS = queue LIBS = -lpmemobj include ../../Makefile.inc queue: queue.o pmdk-1.13.1/src/examples/libpmemobj/queue/README0000664000000000000000000000221514435627501017770 0ustar rootrootPersistent Memory Development Kit This is examples/libpmemobj/queue/README. This directory contains an array-based implementation of a persistent queue. Usage: ./queue The file must exist and must be a libpmemobj pool with a layout name "queue". This file can be created using pmempool: $ pmempool create obj --layout queue The queue supports 4 operations: * new - create a new queue with the given capacity. * enqueue - inserts a new entry into the queue with the given data string * dequeue - removes the entry at the front of the queue * show - lists all entries in the queue For example, the following series of commands will create a new queue in the pool, and then will push and pop two entries. $ ./queue /mnt/pmem/queue.pool new 16 $ ./queue /mnt/pmem/queue.pool enqueue hello inserting 0: hello $ ./queue /mnt/pmem/queue.pool enqueue world inserting 0: world $ ./queue /mnt/pmem/queue.pool show Entries 2/16 0: hello 1: world $ ./queue /mnt/pmem/queue.pool dequeue removing 0: hello $ ./queue /mnt/pmem/queue.pool dequeue removing 1: world $ ./queue /mnt/pmem/queue.pool show Entries 0/16 pmdk-1.13.1/src/examples/libpmemobj/queue/queue.c0000664000000000000000000001342514435627501020405 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2018, Intel Corporation */ /* * queue.c -- array based queue example */ #include #include #include #include #include POBJ_LAYOUT_BEGIN(queue); POBJ_LAYOUT_ROOT(queue, struct root); POBJ_LAYOUT_TOID(queue, struct entry); POBJ_LAYOUT_TOID(queue, struct queue); POBJ_LAYOUT_END(queue); struct entry { /* queue entry that contains arbitrary data */ size_t len; /* length of the data buffer */ char data[]; }; struct queue { /* array-based queue container */ size_t front; /* position of the first entry */ size_t back; /* position of the last entry */ size_t capacity; /* size of the entries array */ TOID(struct entry) entries[]; }; struct root { TOID(struct queue) queue; }; /* * queue_constructor -- constructor of the queue container */ static int queue_constructor(PMEMobjpool *pop, void *ptr, void *arg) { struct queue *q = ptr; size_t *capacity = arg; q->front = 0; q->back = 0; q->capacity = *capacity; /* atomic API requires that objects are persisted in the constructor */ pmemobj_persist(pop, q, sizeof(*q)); return 0; } /* * queue_new -- allocates a new queue container using the atomic API */ static int queue_new(PMEMobjpool *pop, TOID(struct queue) *q, size_t nentries) { return POBJ_ALLOC(pop, q, struct queue, sizeof(struct queue) + sizeof(TOID(struct entry)) * nentries, queue_constructor, &nentries); } /* * queue_nentries -- returns the number of entries */ static size_t queue_nentries(struct queue *queue) { return queue->back - queue->front; } /* * queue_enqueue -- allocates and inserts a new entry into the queue */ static int queue_enqueue(PMEMobjpool *pop, struct queue *queue, const char *data, size_t len) { if (queue->capacity - queue_nentries(queue) == 0) return -1; /* at capacity */ /* back is never decreased, need to calculate the real position */ size_t pos = queue->back % queue->capacity; int ret = 0; printf("inserting %zu: %s\n", pos, data); TX_BEGIN(pop) { /* let's first reserve the space at the end of the queue */ TX_ADD_DIRECT(&queue->back); queue->back += 1; /* now we can safely allocate and initialize the new entry */ TOID(struct entry) entry = TX_ALLOC(struct entry, sizeof(struct entry) + len); D_RW(entry)->len = len; memcpy(D_RW(entry)->data, data, len); /* and then snapshot the queue entry that we will modify */ TX_ADD_DIRECT(&queue->entries[pos]); queue->entries[pos] = entry; } TX_ONABORT { /* don't forget about error handling! ;) */ ret = -1; } TX_END return ret; } /* * queue_dequeue - removes and frees the first element from the queue */ static int queue_dequeue(PMEMobjpool *pop, struct queue *queue) { if (queue_nentries(queue) == 0) return -1; /* no entries to remove */ /* front is also never decreased */ size_t pos = queue->front % queue->capacity; int ret = 0; printf("removing %zu: %s\n", pos, D_RO(queue->entries[pos])->data); TX_BEGIN(pop) { /* move the queue forward */ TX_ADD_DIRECT(&queue->front); queue->front += 1; /* and since this entry is now unreachable, free it */ TX_FREE(queue->entries[pos]); /* notice that we do not change the PMEMoid itself */ } TX_ONABORT { ret = -1; } TX_END return ret; } /* * queue_show -- prints all queue entries */ static void queue_show(PMEMobjpool *pop, struct queue *queue) { size_t nentries = queue_nentries(queue); printf("Entries %zu/%zu\n", nentries, queue->capacity); for (size_t i = queue->front; i < queue->back; ++i) { size_t pos = i % queue->capacity; printf("%zu: %s\n", pos, D_RO(queue->entries[pos])->data); } } /* available queue operations */ enum queue_op { UNKNOWN_QUEUE_OP, QUEUE_NEW, QUEUE_ENQUEUE, QUEUE_DEQUEUE, QUEUE_SHOW, MAX_QUEUE_OP, }; /* queue operations strings */ static const char *ops_str[MAX_QUEUE_OP] = {"", "new", "enqueue", "dequeue", "show"}; /* * parse_queue_op -- parses the operation string and returns matching queue_op */ static enum queue_op queue_op_parse(const char *str) { for (int i = 0; i < MAX_QUEUE_OP; ++i) if (strcmp(str, ops_str[i]) == 0) return (enum queue_op)i; return UNKNOWN_QUEUE_OP; } /* * fail -- helper function to exit the application in the event of an error */ static void __attribute__((noreturn)) /* this function terminates */ fail(const char *msg) { fprintf(stderr, "%s\n", msg); exit(EXIT_FAILURE); } int main(int argc, char *argv[]) { enum queue_op op; if (argc < 3 || (op = queue_op_parse(argv[2])) == UNKNOWN_QUEUE_OP) fail("usage: file-name [new |show|enqueue |dequeue]"); PMEMobjpool *pop = pmemobj_open(argv[1], POBJ_LAYOUT_NAME(queue)); if (pop == NULL) fail("failed to open the pool"); TOID(struct root) root = POBJ_ROOT(pop, struct root); struct root *rootp = D_RW(root); size_t capacity; switch (op) { case QUEUE_NEW: if (argc != 4) fail("missing size of the queue"); char *end; errno = 0; capacity = strtoull(argv[3], &end, 0); if (errno == ERANGE || *end != '\0') fail("invalid size of the queue"); if (queue_new(pop, &rootp->queue, capacity) != 0) fail("failed to create a new queue"); break; case QUEUE_ENQUEUE: if (argc != 4) fail("missing new entry data"); if (D_RW(rootp->queue) == NULL) fail("queue must exist"); if (queue_enqueue(pop, D_RW(rootp->queue), argv[3], strlen(argv[3]) + 1) != 0) fail("failed to insert new entry"); break; case QUEUE_DEQUEUE: if (D_RW(rootp->queue) == NULL) fail("queue must exist"); if (queue_dequeue(pop, D_RW(rootp->queue)) != 0) fail("failed to remove entry"); break; case QUEUE_SHOW: if (D_RW(rootp->queue) == NULL) fail("queue must exist"); queue_show(pop, D_RW(rootp->queue)); break; default: assert(0); /* unreachable */ break; } pmemobj_close(pop); return 0; } pmdk-1.13.1/src/examples/libpmemobj/pmemobjfs/0000775000000000000000000000000014435627501017746 5ustar rootrootpmdk-1.13.1/src/examples/libpmemobj/pmemobjfs/.gitignore0000664000000000000000000000014114435627501021732 0ustar rootrootpmemobjfs pmemobjfs.tx_begin pmemobjfs.tx_commit pmemobjfs.tx_abort mkfs.pmemobjfs pmemobjfs.log pmdk-1.13.1/src/examples/libpmemobj/pmemobjfs/Makefile0000664000000000000000000000226414435627501021412 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2015-2019, Intel Corporation # # examples/libpmemobj/pmemobjfs/Makefile -- build simple fuse base on libpmemobj # TOP := $(dir $(lastword $(MAKEFILE_LIST)))../../../../ include $(TOP)/src/common.inc vpath %.c ../tree_map/ vpath %.c ../map/ FUSE_REQ_VER = 2.9.1 FUSE := $(call check_package, fuse --atleast-version $(FUSE_REQ_VER)) ifeq ($(FUSE), y) LINKS = mkfs.pmemobjfs\ pmemobjfs.tx_begin\ pmemobjfs.tx_commit\ pmemobjfs.tx_abort PROGS = pmemobjfs else $(info NOTE: Skipping pmemobjfs because fuse (version >= $(FUSE_REQ_VER)) is missing \ -- see src/examples/libpmemobj/pmemobjfs/README for details.) endif all-progs: $(LINKS) clean-progs: clean-links include ../../Makefile.inc LIBS += -lpmemobj -lpmem -pthread ifeq ($(FUSE), y) LIBS += $(shell $(PKG_CONFIG) --libs fuse) CFLAGS += $(shell $(PKG_CONFIG) --cflags fuse) endif CFLAGS += -I../tree_map CFLAGS += -I../map CFLAGS += -DFUSE_USE_VERSION=28 CFLAGS += -DUSE_CTREE $(LINKS): pmemobjfs $(LN) -sf pmemobjfs $@ clean-links: $(RM) $(LINKS) pmemobjfs: pmemobjfs.o ../map/libmap_ctree.a ../map/libmap_ctree.a: $(MAKE) -C ../map map_ctree .PHONY: clean-links pmdk-1.13.1/src/examples/libpmemobj/pmemobjfs/README0000664000000000000000000000361514435627501020633 0ustar rootrootPersistent Memory Development Kit This is examples/libpmemobj/pmemobjfs/README. This directory contains an example application implemented using libpmemobj. The pmemobjfs is a simple implementation of FUSE file system using the transactional API of libpmemobj. To create a file system layout run the following command: $ mkfs.pmemobjfs [-s ] [-b ] To mount the filesystem run the following command: $ pmemobjfs -s -odefault_permissions,hard_remove,allow_other The -o default_permissions option enables permission checking by the FUSE implementation. The -s option runs FUSE in single threaded mode. The current implementation does not use any synchronization mechanisms. The -o hard_remove forces FUSE to not use the .fuse_hiddenXXX files when deleting a file. The -o allow_other option allows other users to access the file system. To begin a transaction run the following command: $ pmemobjfs.tx_begin To commit a transaction run the following command: $ pmemobjfs.tx_commit To abort a transaction run the following command: $ pmemobjfs.tx_abort The transactions works across multiple operations performed on pmemobjfs file system including modification of files. Example usage of transactions: $ mkfs.pmemobjfs /mnt/pmem/pmemobjfs.obj $ pmemobjfs -s -odefault_permissions,hard_remove,allow_other\ /mnt/pmem/pmemobjfs.obj /mnt/pmemobjfs $ ls /mnt/pmemobjfs $ pmemobjfs.tx_begin /mnt/pmemobjfs $ mkdir /mnt/pmemobjfs/dir1 $ echo "The pmemobjfs filesystem" > /mnt/pmemobjfs/dir1/file1 $ pmemobjfs.tx_commit $ ls /mnt/pmemobjfs/dir1 file1 $ pmemobjfs.tx_begin /mnt/pmemobjfs $ rm /mnt/pmemobjfs/dir1/file1 $ ls /mnt/pmemobjfs/dir1 $ pmemobjfs.tx_abort $ ls /mnt/pmemobjfs/dir1 file1 ** DEPENDENCIES: ** In order to build pmemobjfs you need to install fuse (version >= 2.9.1) development package. rpm-based systems : fuse-devel dpkg-based systems: libfuse-dev pmdk-1.13.1/src/examples/libpmemobj/pmemobjfs/pmemobjfs.c0000664000000000000000000015204214435627501022100 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2023, Intel Corporation */ /* * pmemobjfs.c -- simple filesystem based on libpmemobj tx API */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifndef PMEMOBJFS_TRACK_BLOCKS #define PMEMOBJFS_TRACK_BLOCKS 1 #endif #if DEBUG static FILE *log_fh; static uint64_t log_cnt; #define log(fmt, args...) do {\ if (log_fh) {\ fprintf(log_fh, "[%016lx] %s: " fmt "\n", log_cnt, __func__, ## args);\ log_cnt++;\ fflush(log_fh);\ }\ } while (0) #else #define log(fmt, args...) do {} while (0) #endif #define PMEMOBJFS_MOUNT "pmemobjfs" #define PMEMOBJFS_MKFS "mkfs.pmemobjfs" #define PMEMOBJFS_TX_BEGIN "pmemobjfs.tx_begin" #define PMEMOBJFS_TX_COMMIT "pmemobjfs.tx_commit" #define PMEMOBJFS_TX_ABORT "pmemobjfs.tx_abort" #define PMEMOBJFS_TMP_TEMPLATE "/.tx_XXXXXX" #define PMEMOBJFS_CTL 'I' #define PMEMOBJFS_CTL_TX_BEGIN _IO(PMEMOBJFS_CTL, 1) #define PMEMOBJFS_CTL_TX_COMMIT _IO(PMEMOBJFS_CTL, 2) #define PMEMOBJFS_CTL_TX_ABORT _IO(PMEMOBJFS_CTL, 3) /* * struct pmemobjfs -- volatile state of pmemobjfs */ struct pmemobjfs { PMEMobjpool *pop; struct map_ctx *mapc; uint64_t pool_uuid_lo; int ioctl_cmd; uint64_t ioctl_off; uint64_t block_size; uint64_t max_name; }; #define PMEMOBJFS (struct pmemobjfs *)fuse_get_context()->private_data #define PDLL_ENTRY(type)\ struct {\ TOID(type) next;\ TOID(type) prev;\ } #define PDLL_HEAD(type)\ struct {\ TOID(type) first;\ TOID(type) last;\ } #define PDLL_HEAD_INIT(head) do {\ ((head)).first = ((typeof((head).first))OID_NULL);\ ((head)).last = ((typeof((head).first))OID_NULL);\ } while (0) #define PDLL_FOREACH(entry, head, field)\ for ((entry) = ((head)).first; !TOID_IS_NULL(entry);\ (entry) = D_RO(entry)->field.next) #define PDLL_FOREACH_SAFE(entry, next, head, field)\ for ((entry) = ((head)).first; !TOID_IS_NULL(entry) &&\ ((next) = D_RO(entry)->field.next, 1);\ (entry) = (next)) #define PDLL_INSERT_HEAD(head, entry, field) do {\ pmemobj_tx_add_range_direct(&(head).first, sizeof((head).first));\ TX_ADD_FIELD(entry, field);\ D_RW(entry)->field.next = (head).first;\ D_RW(entry)->field.prev =\ (typeof(D_RW(entry)->field.prev))OID_NULL;\ (head).first = entry;\ if (TOID_IS_NULL((head).last)) {\ pmemobj_tx_add_range_direct(&(head).last, sizeof((head).last));\ (head).last = entry;\ }\ typeof(entry) next = D_RO(entry)->field.next;\ if (!TOID_IS_NULL(next)) {\ pmemobj_tx_add_range_direct(&D_RW(next)->field.prev,\ sizeof(D_RW(next)->field.prev));\ D_RW(next)->field.prev = entry;\ }\ } while (0) #define PDLL_REMOVE(head, entry, field) do {\ if (TOID_EQUALS((head).first, entry) &&\ TOID_EQUALS((head).last, entry)) {\ pmemobj_tx_add_range_direct(&(head).first,\ sizeof((head).first));\ pmemobj_tx_add_range_direct(&(head).last, sizeof((head).last));\ (head).first = (typeof(D_RW(entry)->field.prev))OID_NULL;\ (head).last = (typeof(D_RW(entry)->field.prev))OID_NULL;\ } else if (TOID_EQUALS((head).first, entry)) {\ typeof(entry) next = D_RW(entry)->field.next;\ pmemobj_tx_add_range_direct(&D_RW(next)->field.prev,\ sizeof(D_RW(next)->field.prev));\ pmemobj_tx_add_range_direct(&(head).first,\ sizeof((head).first));\ (head).first = D_RO(entry)->field.next;\ D_RW(next)->field.prev.oid = OID_NULL;\ } else if (TOID_EQUALS((head).last, entry)) {\ typeof(entry) prev = D_RW(entry)->field.prev;\ pmemobj_tx_add_range_direct(&D_RW(prev)->field.next,\ sizeof(D_RW(prev)->field.next));\ pmemobj_tx_add_range_direct(&(head).last, sizeof((head).last));\ (head).last = D_RO(entry)->field.prev;\ D_RW(prev)->field.next.oid = OID_NULL;\ } else {\ typeof(entry) prev = D_RW(entry)->field.prev;\ typeof(entry) next = D_RW(entry)->field.next;\ pmemobj_tx_add_range_direct(&D_RW(prev)->field.next,\ sizeof(D_RW(prev)->field.next));\ pmemobj_tx_add_range_direct(&D_RW(next)->field.prev,\ sizeof(D_RW(next)->field.prev));\ D_RW(prev)->field.next = D_RO(entry)->field.next;\ D_RW(next)->field.prev = D_RO(entry)->field.prev;\ }\ } while (0) typedef uint8_t objfs_block_t; /* * pmemobjfs persistent layout */ POBJ_LAYOUT_BEGIN(pmemobjfs); POBJ_LAYOUT_ROOT(pmemobjfs, struct objfs_super); POBJ_LAYOUT_TOID(pmemobjfs, struct objfs_inode); POBJ_LAYOUT_TOID(pmemobjfs, struct objfs_dir_entry); POBJ_LAYOUT_TOID(pmemobjfs, objfs_block_t); POBJ_LAYOUT_TOID(pmemobjfs, char); POBJ_LAYOUT_END(pmemobjfs); #define PMEMOBJFS_MIN_BLOCK_SIZE ((size_t)(512 - 64)) /* * struct objfs_super -- pmemobjfs super (root) object */ struct objfs_super { TOID(struct objfs_inode) root_inode; /* root dir inode */ TOID(struct map) opened; /* map of opened files / dirs */ uint64_t block_size; /* size of data block */ }; /* * struct objfs_dir_entry -- pmemobjfs directory entry structure */ struct objfs_dir_entry { PDLL_ENTRY(struct objfs_dir_entry) pdll; /* list entry */ TOID(struct objfs_inode) inode; /* pointer to inode */ char name[]; /* name */ }; /* * struct objfs_dir -- pmemobjfs directory structure */ struct objfs_dir { PDLL_HEAD(struct objfs_dir_entry) entries; /* directory entries */ }; /* * key == 0 for ctree_map is not allowed */ #define GET_KEY(off) ((off) + 1) /* * struct objfs_file -- pmemobjfs file structure */ struct objfs_file { TOID(struct map) blocks; /* blocks map */ }; /* * struct objfs_symlink -- symbolic link */ struct objfs_symlink { uint64_t len; /* length of symbolic link */ TOID(char) name; /* symbolic link data */ }; /* * struct objfs_inode -- pmemobjfs inode structure */ struct objfs_inode { uint64_t size; /* size of file */ uint64_t flags; /* file flags */ uint64_t dev; /* device info */ uint64_t ctime; /* time of last status change */ uint64_t mtime; /* time of last modification */ uint64_t atime; /* time of last access */ uint32_t uid; /* user ID */ uint32_t gid; /* group ID */ uint32_t ref; /* reference counter */ struct objfs_file file; /* file specific data */ struct objfs_dir dir; /* directory specific data */ struct objfs_symlink symlink; /* symlink specific data */ }; /* * pmemobjfs_ioctl -- do the ioctl command */ static void pmemobjfs_ioctl(struct pmemobjfs *objfs) { switch (objfs->ioctl_cmd) { case PMEMOBJFS_CTL_TX_BEGIN: (void) pmemobj_tx_begin(objfs->pop, NULL, TX_PARAM_NONE); break; case PMEMOBJFS_CTL_TX_ABORT: pmemobj_tx_abort(-1); (void) pmemobj_tx_end(); break; case PMEMOBJFS_CTL_TX_COMMIT: pmemobj_tx_commit(); (void) pmemobj_tx_end(); break; default: break; } /* clear deferred inode offset and command */ objfs->ioctl_cmd = 0; objfs->ioctl_off = 0; } /* * pmemobjfs_inode_alloc -- allocate inode structure */ static TOID(struct objfs_inode) pmemobjfs_inode_alloc(struct pmemobjfs *objfs, uint64_t flags, uint32_t uid, uint32_t gid, uint64_t dev) { TOID(struct objfs_inode) inode = TOID_NULL(struct objfs_inode); TX_BEGIN(objfs->pop) { inode = TX_ZNEW(struct objfs_inode); time_t cur_time = time(NULL); D_RW(inode)->flags = flags; D_RW(inode)->dev = dev; D_RW(inode)->ctime = cur_time; D_RW(inode)->mtime = cur_time; D_RW(inode)->atime = cur_time; D_RW(inode)->uid = uid; D_RW(inode)->gid = gid; D_RW(inode)->ref = 0; } TX_ONABORT { inode = TOID_NULL(struct objfs_inode); } TX_END return inode; } /* * pmemobjfs_inode_init_dir -- initialize directory in inode */ static void pmemobjfs_inode_init_dir(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode) { TX_BEGIN(objfs->pop) { PDLL_HEAD_INIT(D_RW(inode)->dir.entries); } TX_END; } /* * pmemobjfs_inode_destroy_dir -- destroy directory from inode */ static void pmemobjfs_inode_destroy_dir(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode) { /* nothing to do */ } /* * pmemobjfs_file_alloc -- allocate file structure */ static void pmemobjfs_inode_init_file(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode) { TX_BEGIN(objfs->pop) { map_create(objfs->mapc, &D_RW(inode)->file.blocks, NULL); } TX_END } /* * pmemobjfs_file_free -- free file structure */ static void pmemobjfs_inode_destroy_file(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode) { TX_BEGIN(objfs->pop) { map_destroy(objfs->mapc, &D_RW(inode)->file.blocks); } TX_END } /* * pmemobjfs_inode_hold -- increase reference counter of inode */ static void pmemobjfs_inode_hold(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode) { if (TOID_IS_NULL(inode)) return; TX_BEGIN(objfs->pop) { /* update number of references */ TX_ADD_FIELD(inode, ref); D_RW(inode)->ref++; /* update status change time */ TX_ADD_FIELD(inode, ctime); D_RW(inode)->ctime = time(NULL); } TX_END } /* * pmemobjfs_dir_entry_alloc -- allocate directory entry structure */ static TOID(struct objfs_dir_entry) pmemobjfs_dir_entry_alloc(struct pmemobjfs *objfs, const char *name, TOID(struct objfs_inode) inode) { TOID(struct objfs_dir_entry) entry = TOID_NULL(struct objfs_dir_entry); TX_BEGIN(objfs->pop) { size_t len = strlen(name) + 1; entry = TX_ALLOC(struct objfs_dir_entry, objfs->block_size); memcpy(D_RW(entry)->name, name, len); D_RW(entry)->inode = inode; pmemobjfs_inode_hold(objfs, inode); } TX_ONABORT { entry = TOID_NULL(struct objfs_dir_entry); } TX_END return entry; } /* * pmemobjfs_dir_entry_free -- free dir entry structure */ static void pmemobjfs_dir_entry_free(struct pmemobjfs *objfs, TOID(struct objfs_dir_entry) entry) { TX_BEGIN(objfs->pop) { TX_FREE(entry); } TX_END } /* * pmemobjfs_inode_init_symlink -- initialize symbolic link */ static void pmemobjfs_inode_init_symlink(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, const char *name) { TX_BEGIN(objfs->pop) { size_t len = strlen(name) + 1; D_RW(inode)->symlink.len = len; TOID_ASSIGN(D_RW(inode)->symlink.name, TX_STRDUP(name, TOID_TYPE_NUM(char))); } TX_END } /* * pmemobjfs_inode_destroy_symlink -- destroy symbolic link */ static void pmemobjfs_inode_destroy_symlink(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode) { TX_BEGIN(objfs->pop) { TX_FREE(D_RO(inode)->symlink.name); } TX_END } /* * pmemobjfs_symlink_read -- read symlink to buffer */ static int pmemobjfs_symlink_read(TOID(struct objfs_inode) inode, char *buff, size_t size) { /* check inode type */ switch (D_RO(inode)->flags & S_IFMT) { case S_IFLNK: break; case S_IFDIR: return -EISDIR; case S_IFREG: default: return -EINVAL; } char *name = D_RW(D_RW(inode)->symlink.name); strncpy(buff, name, size); return 0; } /* * pmemobjfs_symlink_size -- get size of symlink */ static size_t pmemobjfs_symlink_size(TOID(struct objfs_inode) inode) { return D_RO(inode)->symlink.len - 1; } /* * pmemobjfs_inode_free -- free inode structure */ static void pmemobjfs_inode_free(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode) { TX_BEGIN(objfs->pop) { /* release data specific for inode type */ if (S_ISREG(D_RO(inode)->flags)) { pmemobjfs_inode_destroy_file(objfs, inode); } else if (S_ISDIR(D_RO(inode)->flags)) { pmemobjfs_inode_destroy_dir(objfs, inode); } else if (S_ISLNK(D_RO(inode)->flags)) { pmemobjfs_inode_destroy_symlink(objfs, inode); } TX_FREE(inode); } TX_END } /* * pmemobjfs_inode_put -- decrease reference counter of inode and free */ static void pmemobjfs_inode_put(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode) { if (TOID_IS_NULL(inode)) return; TX_BEGIN(objfs->pop) { /* update number of references */ TX_ADD_FIELD(inode, ref); D_RW(inode)->ref--; /* update status change time */ TX_ADD_FIELD(inode, ctime); D_RW(inode)->ctime = time(NULL); if (!D_RO(inode)->ref) pmemobjfs_inode_free(objfs, inode); } TX_END } /* * pmemobjfs_dir_get_inode -- get inode from dir of given name */ static TOID(struct objfs_inode) pmemobjfs_dir_get_inode(TOID(struct objfs_inode) inode, const char *name) { log("%s", name); TOID(struct objfs_dir_entry) entry; PDLL_FOREACH(entry, D_RW(inode)->dir.entries, pdll) { if (strcmp(name, D_RO(entry)->name) == 0) return D_RO(entry)->inode; } return TOID_NULL(struct objfs_inode); } /* * pmemobjfs_get_dir_entry -- get dir entry from dir of given name */ static TOID(struct objfs_dir_entry) pmemobjfs_get_dir_entry(TOID(struct objfs_inode) inode, const char *name) { log("%s", name); TOID(struct objfs_dir_entry) entry; PDLL_FOREACH(entry, D_RW(inode)->dir.entries, pdll) { if (strcmp(name, D_RO(entry)->name) == 0) return entry; } return TOID_NULL(struct objfs_dir_entry); } /* * pmemobjfs_inode_lookup_parent -- lookup for parent inode and child name */ static int pmemobjfs_inode_lookup_parent(struct pmemobjfs *objfs, const char *path, TOID(struct objfs_inode) *inodep, const char **child) { log("%s", path); TOID(struct objfs_super) super = POBJ_ROOT(objfs->pop, struct objfs_super); TOID(struct objfs_inode) cur = D_RO(super)->root_inode; TOID(struct objfs_inode) par = TOID_NULL(struct objfs_inode); if (path[0] == '/') path++; int ret = 0; char *p = strdup(path); char *name = p; char *ch = NULL; while (name && *name != '\0' && !TOID_IS_NULL(cur)) { char *slash = strchr(name, '/'); if (slash) { *slash = '\0'; slash++; } if (!S_ISDIR(D_RO(cur)->flags)) { ret = -ENOTDIR; goto out; } if (strlen(name) > objfs->max_name) { ret = -ENAMETOOLONG; goto out; } par = cur; cur = pmemobjfs_dir_get_inode(cur, name); ch = name; name = slash; } if (child) { if (strchr(ch, '/')) { ret = -ENOENT; goto out; } if (TOID_IS_NULL(par)) { ret = -ENOENT; goto out; } cur = par; size_t parent_len = ch - p; *child = path + parent_len; } else { if (TOID_IS_NULL(cur)) ret = -ENOENT; } if (inodep) *inodep = cur; out: free(p); return ret; } /* * pmemobjfs_inode_lookup -- get inode for given path */ static int pmemobjfs_inode_lookup(struct pmemobjfs *objfs, const char *path, TOID(struct objfs_inode) *inodep) { log("%s", path); return pmemobjfs_inode_lookup_parent(objfs, path, inodep, NULL); } /* * pmemobjfs_file_get_block -- get block at given offset */ static TOID(objfs_block_t) pmemobjfs_file_get_block(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, uint64_t offset) { TOID(objfs_block_t) block; PMEMoid block_oid = map_get(objfs->mapc, D_RO(inode)->file.blocks, GET_KEY(offset)); TOID_ASSIGN(block, block_oid); return block; } /* * pmemobjfs_file_get_block_for_write -- get or allocate block at given offset */ static TOID(objfs_block_t) pmemobjfs_file_get_block_for_write(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, uint64_t offset) { TOID(objfs_block_t) block = pmemobjfs_file_get_block(objfs, inode, offset); if (TOID_IS_NULL(block)) { TX_BEGIN(objfs->pop) { block = TX_ALLOC(objfs_block_t, objfs->block_size); map_insert(objfs->mapc, D_RW(inode)->file.blocks, GET_KEY(offset), block.oid); } TX_ONABORT { block = TOID_NULL(objfs_block_t); } TX_END } else { #if PMEMOBJFS_TRACK_BLOCKS TX_ADD(block); #endif } return block; } /* * pmemobjfs_truncate -- truncate file */ static int pmemobjfs_truncate(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, off_t off) { /* check inode type */ switch (D_RO(inode)->flags & S_IFMT) { case S_IFREG: break; case S_IFDIR: return -EISDIR; default: return -EINVAL; } int ret = 0; TX_BEGIN(objfs->pop) { off_t old_off = D_RO(inode)->size; if (old_off > off) { /* release blocks */ uint64_t old_boff = (old_off - 1) / objfs->block_size; uint64_t boff = (off + 1) / objfs->block_size; for (uint64_t o = boff; o <= old_boff; o++) { map_remove_free(objfs->mapc, D_RW(inode)->file.blocks, GET_KEY(o)); } } time_t t = time(NULL); /* update modification time */ TX_ADD_FIELD(inode, mtime); D_RW(inode)->mtime = t; /* update status change time */ TX_ADD_FIELD(inode, ctime); D_RW(inode)->ctime = t; /* update size */ TX_ADD_FIELD(inode, size); D_RW(inode)->size = off; } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_read -- read from file */ static int pmemobjfs_read(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, char *buff, size_t size, off_t offset) { /* check inode type */ switch (D_RO(inode)->flags & S_IFMT) { case S_IFREG: break; case S_IFDIR: return -EISDIR; default: return -EINVAL; } uint64_t fsize = D_RO(inode)->size; size_t sz = size; size_t off = offset; while (sz > 0) { if (off >= fsize) break; uint64_t block_id = off / objfs->block_size; uint64_t block_off = off % objfs->block_size; uint64_t block_size = sz < objfs->block_size ? sz : objfs->block_size; TOID(objfs_block_t) block = pmemobjfs_file_get_block(objfs, inode, block_id); if (block_off + block_size > objfs->block_size) block_size = objfs->block_size - block_off; if (TOID_IS_NULL(block)) { memset(buff, 0, block_size); } else { memcpy(buff, &D_RW(block)[block_off], block_size); } buff += block_size; off += block_size; sz -= block_size; } return size - sz; } /* * pmemobjfs_write -- write to file */ static int pmemobjfs_write(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, const char *buff, size_t size, off_t offset) { /* check inode type */ switch (D_RO(inode)->flags & S_IFMT) { case S_IFREG: break; case S_IFDIR: return -EISDIR; default: return -EINVAL; } int ret = 0; TX_BEGIN(objfs->pop) { size_t sz = size; off_t off = offset; while (sz > 0) { uint64_t block_id = off / objfs->block_size; uint64_t block_off = off % objfs->block_size; uint64_t block_size = sz < objfs->block_size ? sz : objfs->block_size; TOID(objfs_block_t) block = pmemobjfs_file_get_block_for_write(objfs, inode, block_id); if (TOID_IS_NULL(block)) return -ENOSPC; if (block_off + block_size > objfs->block_size) block_size = objfs->block_size - block_off; memcpy(&D_RW(block)[block_off], buff, block_size); buff += block_size; off += block_size; sz -= block_size; } time_t t = time(NULL); if (offset + size > D_RO(inode)->size) { /* update size */ TX_ADD_FIELD(inode, size); D_RW(inode)->size = offset + size; /* update status change time */ TX_ADD_FIELD(inode, ctime); D_RW(inode)->ctime = t; } /* update modification time */ TX_ADD_FIELD(inode, mtime); D_RW(inode)->mtime = t; } TX_ONCOMMIT { ret = size; } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_fallocate -- allocate blocks for file */ static int pmemobjfs_fallocate(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, off_t offset, off_t size) { /* check inode type */ switch (D_RO(inode)->flags & S_IFMT) { case S_IFREG: break; case S_IFDIR: return -EISDIR; default: return -EINVAL; } int ret = 0; TX_BEGIN(objfs->pop) { /* allocate blocks from requested range */ uint64_t b_off = offset / objfs->block_size; uint64_t e_off = (offset + size) / objfs->block_size; for (uint64_t off = b_off; off <= e_off; off++) pmemobjfs_file_get_block_for_write(objfs, inode, off); time_t t = time(NULL); /* update modification time */ TX_ADD_FIELD(inode, mtime); D_RW(inode)->mtime = t; /* update status change time */ TX_ADD_FIELD(inode, ctime); D_RW(inode)->ctime = t; /* update inode size */ D_RW(inode)->size = offset + size; TX_ADD_FIELD(inode, size); } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_remove_dir_entry -- remove dir entry from directory */ static void pmemobjfs_remove_dir_entry(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, TOID(struct objfs_dir_entry) entry) { TX_BEGIN(objfs->pop) { pmemobjfs_inode_put(objfs, D_RO(entry)->inode); PDLL_REMOVE(D_RW(inode)->dir.entries, entry, pdll); pmemobjfs_dir_entry_free(objfs, entry); } TX_END } /* * pmemobjfs_remove_dir_entry_name -- remove dir entry of given name */ static void pmemobjfs_remove_dir_entry_name(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, const char *name) { TX_BEGIN(objfs->pop) { TOID(struct objfs_dir_entry) entry = pmemobjfs_get_dir_entry(inode, name); pmemobjfs_remove_dir_entry(objfs, inode, entry); } TX_END } /* * pmemobjfs_add_dir_entry -- add new directory entry */ static int pmemobjfs_add_dir_entry(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, TOID(struct objfs_dir_entry) entry) { /* check inode type */ if (!S_ISDIR(D_RO(inode)->flags)) return -ENOTDIR; int ret = 0; TX_BEGIN(objfs->pop) { /* insert new dir entry to list */ PDLL_INSERT_HEAD(D_RW(inode)->dir.entries, entry, pdll); /* update dir size */ TX_ADD_FIELD(inode, size); D_RW(inode)->size++; } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_unlink_dir_entry -- unlink directory entry */ static int pmemobjfs_unlink_dir_entry(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, TOID(struct objfs_dir_entry) entry) { /* check inode type */ if (!S_ISDIR(D_RO(inode)->flags)) return -ENOTDIR; int ret = 0; TX_BEGIN(objfs->pop) { pmemobjfs_remove_dir_entry(objfs, inode, entry); /* update dir size */ TX_ADD_FIELD(inode, size); D_RW(inode)->size--; } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_new_dir -- create new directory */ static TOID(struct objfs_inode) pmemobjfs_new_dir(struct pmemobjfs *objfs, TOID(struct objfs_inode) parent, const char *name, uint64_t flags, uint32_t uid, uint32_t gid) { TOID(struct objfs_inode) inode = TOID_NULL(struct objfs_inode); TX_BEGIN(objfs->pop) { inode = pmemobjfs_inode_alloc(objfs, flags, uid, gid, 0); pmemobjfs_inode_init_dir(objfs, inode); /* add . and .. to new directory */ TOID(struct objfs_dir_entry) dot = pmemobjfs_dir_entry_alloc(objfs, ".", inode); TOID(struct objfs_dir_entry) dotdot = pmemobjfs_dir_entry_alloc(objfs, "..", parent); pmemobjfs_add_dir_entry(objfs, inode, dot); pmemobjfs_add_dir_entry(objfs, inode, dotdot); } TX_ONABORT { inode = TOID_NULL(struct objfs_inode); } TX_END return inode; } /* * pmemobjfs_mkdir -- make new directory */ static int pmemobjfs_mkdir(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, const char *name, uint64_t flags, uint32_t uid, uint32_t gid) { /* check inode type */ if (!S_ISDIR(D_RO(inode)->flags)) return -ENOTDIR; int ret = 0; TX_BEGIN(objfs->pop) { TOID(struct objfs_inode) new_inode = pmemobjfs_new_dir(objfs, inode, name, flags, uid, gid); TOID(struct objfs_dir_entry) entry = pmemobjfs_dir_entry_alloc(objfs, name, new_inode); pmemobjfs_add_dir_entry(objfs, inode, entry); /* update modification time */ TX_ADD_FIELD(inode, mtime); D_RW(inode)->mtime = time(NULL); } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_remove_dir -- remove directory from directory */ static void pmemobjfs_remove_dir(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, TOID(struct objfs_dir_entry) entry) { /* removing entry inode */ TOID(struct objfs_inode) rinode = D_RO(entry)->inode; TX_BEGIN(objfs->pop) { /* remove . and .. from removing dir */ pmemobjfs_remove_dir_entry_name(objfs, rinode, "."); pmemobjfs_remove_dir_entry_name(objfs, rinode, ".."); /* remove dir entry from parent */ pmemobjfs_remove_dir_entry(objfs, inode, entry); } TX_END } /* * pmemobjfs_rmdir -- remove directory of given name */ static int pmemobjfs_rmdir(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, const char *name) { /* check parent inode type */ if (!S_ISDIR(D_RO(inode)->flags)) return -ENOTDIR; TOID(struct objfs_dir_entry) entry = pmemobjfs_get_dir_entry(inode, name); if (TOID_IS_NULL(entry)) return -ENOENT; TOID(struct objfs_inode) entry_inode = D_RO(entry)->inode; /* check removing dir type */ if (!S_ISDIR(D_RO(entry_inode)->flags)) return -ENOTDIR; /* check if dir is empty (contains only . and ..) */ if (D_RO(entry_inode)->size > 2) return -ENOTEMPTY; int ret = 0; TX_BEGIN(objfs->pop) { pmemobjfs_remove_dir(objfs, inode, entry); /* update dir size */ TX_ADD_FIELD(inode, size); D_RW(inode)->size--; /* update modification time */ TX_ADD_FIELD(inode, mtime); D_RW(inode)->mtime = time(NULL); } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_create -- create new file in directory */ static int pmemobjfs_create(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, const char *name, mode_t mode, uid_t uid, gid_t gid, TOID(struct objfs_inode) *inodep) { int ret = 0; uint64_t flags = mode | S_IFREG; TOID(struct objfs_dir_entry) entry = TOID_NULL(struct objfs_dir_entry); TX_BEGIN(objfs->pop) { TOID(struct objfs_inode) new_file= pmemobjfs_inode_alloc(objfs, flags, uid, gid, 0); pmemobjfs_inode_init_file(objfs, new_file); entry = pmemobjfs_dir_entry_alloc(objfs, name, new_file); pmemobjfs_add_dir_entry(objfs, inode, entry); time_t t = time(NULL); /* update modification time */ TX_ADD_FIELD(inode, mtime); D_RW(inode)->mtime = t; /* update status change time */ TX_ADD_FIELD(inode, ctime); D_RW(inode)->ctime = t; } TX_ONABORT { ret = -ECANCELED; } TX_ONCOMMIT { if (inodep) *inodep = D_RO(entry)->inode; } TX_END return ret; } /* * pmemobjfs_open -- open inode */ static int pmemobjfs_open(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode) { TOID(struct objfs_super) super = POBJ_ROOT(objfs->pop, struct objfs_super); int ret = 0; TX_BEGIN(objfs->pop) { /* insert inode to opened inodes map */ map_insert(objfs->mapc, D_RW(super)->opened, inode.oid.off, inode.oid); /* hold inode */ pmemobjfs_inode_hold(objfs, inode); } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_close -- release inode */ static int pmemobjfs_close(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode) { TOID(struct objfs_super) super = POBJ_ROOT(objfs->pop, struct objfs_super); int ret = 0; TX_BEGIN(objfs->pop) { /* remove inode from opened inodes map */ map_remove(objfs->mapc, D_RW(super)->opened, inode.oid.off); /* release inode */ pmemobjfs_inode_put(objfs, inode); } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_rename -- rename/move inode */ static int pmemobjfs_rename(struct pmemobjfs *objfs, TOID(struct objfs_inode) src_parent, const char *src_name, TOID(struct objfs_inode) dst_parent, const char *dst_name) { /* check source and destination inodes type */ if (!S_ISDIR(D_RO(src_parent)->flags)) return -ENOTDIR; if (!S_ISDIR(D_RO(dst_parent)->flags)) return -ENOTDIR; /* get source dir entry */ TOID(struct objfs_dir_entry) src_entry = pmemobjfs_get_dir_entry(src_parent, src_name); TOID(struct objfs_inode) src_inode = D_RO(src_entry)->inode; if (TOID_IS_NULL(src_entry)) return -ENOENT; int ret = 0; TX_BEGIN(objfs->pop) { /* * Allocate new dir entry with destination name * and source inode. * NOTE: * This *must* be called before removing dir entry from * source directory because otherwise the source inode * could be released before inserting to new dir entry. */ TOID(struct objfs_dir_entry) dst_entry = pmemobjfs_dir_entry_alloc(objfs, dst_name, src_inode); /* remove old dir entry from source */ pmemobjfs_unlink_dir_entry(objfs, src_parent, src_entry); /* add new dir entry to destination */ pmemobjfs_add_dir_entry(objfs, dst_parent, dst_entry); } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_symlink -- create symbolic link */ static int pmemobjfs_symlink(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, const char *name, const char *path, uid_t uid, gid_t gid) { /* check inode type */ if (!S_ISDIR(D_RO(inode)->flags)) return -ENOTDIR; /* set 0777 permissions for symbolic links */ uint64_t flags = 0777 | S_IFLNK; int ret = 0; TX_BEGIN(objfs->pop) { TOID(struct objfs_inode) symlink = pmemobjfs_inode_alloc(objfs, flags, uid, gid, 0); pmemobjfs_inode_init_symlink(objfs, symlink, path); D_RW(symlink)->size = pmemobjfs_symlink_size(symlink); TOID(struct objfs_dir_entry) entry = pmemobjfs_dir_entry_alloc(objfs, name, symlink); pmemobjfs_add_dir_entry(objfs, inode, entry); time_t t = time(NULL); /* update modification time */ TX_ADD_FIELD(inode, mtime); D_RW(inode)->mtime = t; /* update status change time */ TX_ADD_FIELD(inode, ctime); D_RW(inode)->ctime = t; } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_mknod -- create node */ static int pmemobjfs_mknod(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, const char *name, mode_t mode, uid_t uid, gid_t gid, dev_t dev) { /* check inode type */ if (!S_ISDIR(D_RO(inode)->flags)) return -ENOTDIR; int ret = 0; TX_BEGIN(objfs->pop) { TOID(struct objfs_inode) node = pmemobjfs_inode_alloc(objfs, mode, uid, gid, dev); D_RW(node)->size = 0; TOID(struct objfs_dir_entry) entry = pmemobjfs_dir_entry_alloc(objfs, name, node); pmemobjfs_add_dir_entry(objfs, inode, entry); time_t t = time(NULL); /* update modification time */ TX_ADD_FIELD(inode, mtime); D_RW(inode)->mtime = t; /* update status change time */ TX_ADD_FIELD(inode, ctime); D_RW(inode)->ctime = t; } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_chmod -- change mode of inode */ static int pmemobjfs_chmod(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, mode_t mode) { int ret = 0; TX_BEGIN(objfs->pop) { TX_ADD_FIELD(inode, flags); /* mask file type bit fields */ uint64_t flags = D_RO(inode)->flags; flags = flags & S_IFMT; D_RW(inode)->flags = flags | (mode & ~S_IFMT); /* update status change time */ TX_ADD_FIELD(inode, ctime); D_RW(inode)->ctime = time(NULL); } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_chown -- change owner and group of inode */ static int pmemobjfs_chown(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, uid_t uid, gid_t gid) { int ret = 0; TX_BEGIN(objfs->pop) { TX_ADD_FIELD(inode, uid); D_RW(inode)->uid = uid; TX_ADD_FIELD(inode, gid); D_RW(inode)->gid = gid; /* update status change time */ TX_ADD_FIELD(inode, ctime); D_RW(inode)->ctime = time(NULL); } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_getattr -- get inode's attributes */ static int pmemobjfs_getattr(TOID(struct objfs_inode) inode, struct stat *statbuf) { memset(statbuf, 0, sizeof(*statbuf)); statbuf->st_size = D_RO(inode)->size; statbuf->st_ctime = D_RO(inode)->ctime; statbuf->st_mtime = D_RO(inode)->mtime; statbuf->st_atime = D_RO(inode)->atime; statbuf->st_mode = D_RO(inode)->flags; statbuf->st_uid = D_RO(inode)->uid; statbuf->st_gid = D_RO(inode)->gid; statbuf->st_rdev = D_RO(inode)->dev; return 0; } /* * pmemobjfs_utimens -- set atime and mtime */ static int pmemobjfs_utimens(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, const struct timespec tv[2]) { int ret = 0; TX_BEGIN(objfs->pop) { TX_ADD_FIELD(inode, atime); D_RW(inode)->atime = tv[0].tv_sec; TX_ADD_FIELD(inode, mtime); D_RW(inode)->mtime = tv[0].tv_sec; } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_unlink -- unlink file from inode */ static int pmemobjfs_unlink(struct pmemobjfs *objfs, TOID(struct objfs_inode) inode, const char *name) { /* check inode type */ if (!S_ISDIR(D_RO(inode)->flags)) return -ENOTDIR; TOID(struct objfs_dir_entry) entry = pmemobjfs_get_dir_entry(inode, name); if (TOID_IS_NULL(entry)) return -ENOENT; TOID(struct objfs_inode) entry_inode = D_RO(entry)->inode; /* check unlinking inode type */ if (S_ISDIR(D_RO(entry_inode)->flags)) return -EISDIR; int ret = 0; TX_BEGIN(objfs->pop) { pmemobjfs_remove_dir_entry(objfs, inode, entry); TX_ADD_FIELD(inode, size); D_RW(inode)->size--; } TX_ONABORT { ret = -ECANCELED; } TX_END return ret; } /* * pmemobjfs_put_opened_cb -- release all opened inodes */ static int pmemobjfs_put_opened_cb(uint64_t key, PMEMoid value, void *arg) { struct pmemobjfs *objfs = arg; TOID(struct objfs_inode) inode; TOID_ASSIGN(inode, value); TOID(struct objfs_super) super = POBJ_ROOT(objfs->pop, struct objfs_super); /* * Set current value to OID_NULL so the tree_map_clear won't * free this inode and release the inode. */ map_insert(objfs->mapc, D_RW(super)->opened, key, OID_NULL); pmemobjfs_inode_put(objfs, inode); return 0; } /* * pmemobjfs_fuse_getattr -- (FUSE) get file attributes */ static int pmemobjfs_fuse_getattr(const char *path, struct stat *statbuf) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; int ret = pmemobjfs_inode_lookup(objfs, path, &inode); if (ret) return ret; return pmemobjfs_getattr(inode, statbuf); } /* * pmemobjfs_fuse_opendir -- (FUSE) open directory */ static int pmemobjfs_fuse_opendir(const char *path, struct fuse_file_info *fi) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; int ret = pmemobjfs_inode_lookup(objfs, path, &inode); if (ret) return ret; /* check inode type */ switch (D_RO(inode)->flags & S_IFMT) { case S_IFDIR: break; case S_IFREG: return -ENOTDIR; default: return -EINVAL; } /* add inode to opened inodes map */ ret = pmemobjfs_open(objfs, inode); if (!ret) fi->fh = inode.oid.off; return ret; } /* * pmemobjfs_fuse_releasedir -- (FUSE) release opened dir */ static int pmemobjfs_fuse_releasedir(const char *path, struct fuse_file_info *fi) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; if (!fi->fh) return -EINVAL; TOID(struct objfs_inode) inode; inode.oid.off = fi->fh; inode.oid.pool_uuid_lo = objfs->pool_uuid_lo; /* remove inode from opened inodes map */ int ret = pmemobjfs_close(objfs, inode); fi->fh = 0; return ret; } /* * pmemobjfs_fuse_readdir -- (FUSE) read directory entries */ static int pmemobjfs_fuse_readdir(const char *path, void *buff, fuse_fill_dir_t fill, off_t off, struct fuse_file_info *fi) { log("%s off = %lu", path, off); struct pmemobjfs *objfs = PMEMOBJFS; if (!fi->fh) return -EINVAL; TOID(struct objfs_inode) inode; inode.oid.off = fi->fh; inode.oid.pool_uuid_lo = objfs->pool_uuid_lo; if (!TOID_VALID(inode)) return -EINVAL; /* check inode type */ if (!S_ISDIR(D_RO(inode)->flags)) return -ENOTDIR; /* walk through all dir entries and fill fuse buffer */ int ret; TOID(struct objfs_dir_entry) entry; PDLL_FOREACH(entry, D_RW(inode)->dir.entries, pdll) { ret = fill(buff, D_RW(entry)->name, NULL, 0); if (ret) return ret; } return 0; } /* * pmemobjfs_fuse_mkdir -- (FUSE) create directory */ static int pmemobjfs_fuse_mkdir(const char *path, mode_t mode) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; const char *name; int ret = pmemobjfs_inode_lookup_parent(objfs, path, &inode, &name); if (ret) return ret; uid_t uid = fuse_get_context()->uid; uid_t gid = fuse_get_context()->gid; return pmemobjfs_mkdir(objfs, inode, name, mode | S_IFDIR, uid, gid); } /* * pmemobjfs_fuse_rmdir -- (FUSE) remove directory */ static int pmemobjfs_fuse_rmdir(const char *path) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; const char *name; int ret = pmemobjfs_inode_lookup_parent(objfs, path, &inode, &name); if (ret) return ret; return pmemobjfs_rmdir(objfs, inode, name); } /* * pmemobjfs_fuse_chmod -- (FUSE) change file permissions */ static int pmemobjfs_fuse_chmod(const char *path, mode_t mode) { log("%s 0%o", path, mode); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; int ret = pmemobjfs_inode_lookup(objfs, path, &inode); if (ret) return ret; return pmemobjfs_chmod(objfs, inode, mode); } /* * pmemobjfs_fuse_chown -- (FUSE) change owner */ static int pmemobjfs_fuse_chown(const char *path, uid_t uid, gid_t gid) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; int ret = pmemobjfs_inode_lookup(objfs, path, &inode); if (ret) return ret; return pmemobjfs_chown(objfs, inode, uid, gid); } /* * pmemobjfs_fuse_create -- (FUSE) create file */ static int pmemobjfs_fuse_create(const char *path, mode_t mode, struct fuse_file_info *fi) { log("%s mode %o", path, mode); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; const char *name; int ret = pmemobjfs_inode_lookup_parent(objfs, path, &inode, &name); if (ret) return ret; if (!S_ISDIR(D_RO(inode)->flags)) return -EINVAL; uid_t uid = fuse_get_context()->uid; uid_t gid = fuse_get_context()->gid; TOID(struct objfs_inode) new_file; ret = pmemobjfs_create(objfs, inode, name, mode, uid, gid, &new_file); if (ret) return ret; /* add new inode to opened inodes */ ret = pmemobjfs_open(objfs, new_file); if (ret) return ret; fi->fh = new_file.oid.off; return 0; } /* * pmemobjfs_fuse_utimens -- (FUSE) update access and modification times */ static int pmemobjfs_fuse_utimens(const char *path, const struct timespec tv[2]) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; int ret = pmemobjfs_inode_lookup(objfs, path, &inode); if (ret) return ret; return pmemobjfs_utimens(objfs, inode, tv); } /* * pmemobjfs_fuse_open -- (FUSE) open file */ static int pmemobjfs_fuse_open(const char *path, struct fuse_file_info *fi) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; int ret = pmemobjfs_inode_lookup(objfs, path, &inode); if (ret) return ret; /* check inode type */ switch (D_RO(inode)->flags & S_IFMT) { case S_IFREG: break; case S_IFDIR: return -EISDIR; default: return -EINVAL; } ret = pmemobjfs_open(objfs, inode); if (!ret) fi->fh = inode.oid.off; return ret; } /* * pmemobjfs_fuse_release -- (FUSE) release opened file */ static int pmemobjfs_fuse_release(const char *path, struct fuse_file_info *fi) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; if (!fi->fh) return -EINVAL; TOID(struct objfs_inode) inode; inode.oid.off = fi->fh; inode.oid.pool_uuid_lo = objfs->pool_uuid_lo; int ret = pmemobjfs_close(objfs, inode); /* perform deferred ioctl operation */ if (!ret && objfs->ioctl_off && objfs->ioctl_off == fi->fh) pmemobjfs_ioctl(objfs); fi->fh = 0; return ret; } /* * pmemobjfs_fuse_write -- (FUSE) write to file */ static int pmemobjfs_fuse_write(const char *path, const char *buff, size_t size, off_t offset, struct fuse_file_info *fi) { log("%s size = %zu off = %lu", path, size, offset); struct pmemobjfs *objfs = PMEMOBJFS; if (!fi->fh) return -EINVAL; TOID(struct objfs_inode) inode; inode.oid.off = fi->fh; inode.oid.pool_uuid_lo = objfs->pool_uuid_lo; if (!TOID_VALID(inode)) return -EINVAL; return pmemobjfs_write(objfs, inode, buff, size, offset); } /* * pmemobjfs_fuse_read -- (FUSE) read from file */ static int pmemobjfs_fuse_read(const char *path, char *buff, size_t size, off_t off, struct fuse_file_info *fi) { log("%s size = %zu off = %lu", path, size, off); struct pmemobjfs *objfs = PMEMOBJFS; if (!fi->fh) return -EINVAL; TOID(struct objfs_inode) inode; inode.oid.off = fi->fh; inode.oid.pool_uuid_lo = objfs->pool_uuid_lo; if (!TOID_VALID(inode)) return -EINVAL; return pmemobjfs_read(objfs, inode, buff, size, off); } /* * pmemobjfs_fuse_truncate -- (FUSE) truncate file */ static int pmemobjfs_fuse_truncate(const char *path, off_t off) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; int ret = pmemobjfs_inode_lookup(objfs, path, &inode); if (ret) return ret; return pmemobjfs_truncate(objfs, inode, off); } /* * pmemobjfs_fuse_ftruncate -- (FUSE) truncate file */ static int pmemobjfs_fuse_ftruncate(const char *path, off_t off, struct fuse_file_info *fi) { log("%s off = %lu", path, off); struct pmemobjfs *objfs = PMEMOBJFS; if (!fi->fh) return -EINVAL; TOID(struct objfs_inode) inode; inode.oid.off = fi->fh; inode.oid.pool_uuid_lo = objfs->pool_uuid_lo; if (!TOID_VALID(inode)) return -EINVAL; return pmemobjfs_truncate(objfs, inode, off); } /* * pmemobjfs_fuse_unlink -- (FUSE) unlink inode */ static int pmemobjfs_fuse_unlink(const char *path) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; const char *name; int ret = pmemobjfs_inode_lookup_parent(objfs, path, &inode, &name); if (ret) return ret; return pmemobjfs_unlink(objfs, inode, name); } /* * pmemobjfs_fuse_flush -- (FUSE) flush file */ static int pmemobjfs_fuse_flush(const char *path, struct fuse_file_info *fi) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; if (!fi->fh) return -EINVAL; TOID(struct objfs_inode) inode; inode.oid.off = fi->fh; inode.oid.pool_uuid_lo = objfs->pool_uuid_lo; if (!TOID_VALID(inode)) return -EINVAL; /* check inode type */ switch (D_RO(inode)->flags & S_IFMT) { case S_IFREG: break; case S_IFDIR: return -EISDIR; default: return -EINVAL; } /* nothing to do */ return 0; } /* * pmemobjfs_fuse_ioctl -- (FUSE) ioctl for file */ #ifdef __FreeBSD__ #define EBADFD EBADF /* XXX */ #endif static int pmemobjfs_fuse_ioctl(const char *path, int cmd, void *arg, struct fuse_file_info *fi, unsigned flags, void *data) { log("%s cmd %d", path, _IOC_NR(cmd)); struct pmemobjfs *objfs = PMEMOBJFS; /* check transaction stage */ switch (cmd) { case PMEMOBJFS_CTL_TX_BEGIN: if (pmemobj_tx_stage() != TX_STAGE_NONE) return -EINPROGRESS; break; case PMEMOBJFS_CTL_TX_ABORT: if (pmemobj_tx_stage() != TX_STAGE_WORK) return -EBADFD; break; case PMEMOBJFS_CTL_TX_COMMIT: if (pmemobj_tx_stage() != TX_STAGE_WORK) return -EBADFD; break; default: return -EINVAL; } /* * Store the inode offset and command and defer ioctl * execution to releasing the file. This is required * to avoid unlinking .tx_XXXXXX file inside transaction * because one would be rolled back if transaction abort * would occur. */ objfs->ioctl_off = fi->fh; objfs->ioctl_cmd = cmd; return 0; } /* * pmemobjfs_fuse_rename -- (FUSE) rename file or directory */ static int pmemobjfs_fuse_rename(const char *path, const char *dest) { log("%s dest %s\n", path, dest); struct pmemobjfs *objfs = PMEMOBJFS; int ret; /* get source inode's parent and name */ TOID(struct objfs_inode) src_parent; const char *src_name; ret = pmemobjfs_inode_lookup_parent(objfs, path, &src_parent, &src_name); if (ret) return ret; /* get destination inode's parent and name */ TOID(struct objfs_inode) dst_parent; const char *dst_name; ret = pmemobjfs_inode_lookup_parent(objfs, dest, &dst_parent, &dst_name); if (ret) return ret; return pmemobjfs_rename(objfs, src_parent, src_name, dst_parent, dst_name); } /* * pmemobjfs_fuse_symlink -- (FUSE) create symbolic link */ static int pmemobjfs_fuse_symlink(const char *path, const char *link) { log("%s link %s", path, link); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; const char *name; int ret = pmemobjfs_inode_lookup_parent(objfs, link, &inode, &name); if (ret) return ret; uid_t uid = fuse_get_context()->uid; uid_t gid = fuse_get_context()->gid; return pmemobjfs_symlink(objfs, inode, name, path, uid, gid); } /* * pmemobjfs_fuse_readlink -- (FUSE) read symbolic link */ static int pmemobjfs_fuse_readlink(const char *path, char *buff, size_t size) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; int ret = pmemobjfs_inode_lookup(objfs, path, &inode); if (ret) return ret; return pmemobjfs_symlink_read(inode, buff, size); } /* * pmemobjfs_fuse_mknod -- (FUSE) create node */ static int pmemobjfs_fuse_mknod(const char *path, mode_t mode, dev_t dev) { log("%s mode %o major %u minor %u", path, mode, (unsigned)MAJOR(dev), (unsigned)MINOR(dev)); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_inode) inode; const char *name; int ret = pmemobjfs_inode_lookup_parent(objfs, path, &inode, &name); if (ret) return ret; uid_t uid = fuse_get_context()->uid; uid_t gid = fuse_get_context()->gid; return pmemobjfs_mknod(objfs, inode, name, mode, uid, gid, dev); } /* * pmemobjfs_fuse_fallocate -- (FUSE) allocate blocks for file */ static int pmemobjfs_fuse_fallocate(const char *path, int mode, off_t offset, off_t size, struct fuse_file_info *fi) { log("%s mode %d offset %lu size %lu", path, mode, offset, size); struct pmemobjfs *objfs = PMEMOBJFS; if (!fi->fh) return -EINVAL; TOID(struct objfs_inode) inode; inode.oid.off = fi->fh; inode.oid.pool_uuid_lo = objfs->pool_uuid_lo; if (!TOID_VALID(inode)) return -EINVAL; return pmemobjfs_fallocate(objfs, inode, offset, size); } /* * pmemobjfs_fuse_statvfs -- (FUSE) get filesystem info */ static int pmemobjfs_fuse_statvfs(const char *path, struct statvfs *buff) { log("%s", path); struct pmemobjfs *objfs = PMEMOBJFS; memset(buff, 0, sizeof(*buff)); /* * Some fields are ignored by FUSE. * Some fields cannot be set due to the nature of pmemobjfs. */ buff->f_bsize = objfs->block_size; /* ignored buff->f_frsize */ /* unknown buff->f_blocks */ /* unknown buff->f_bfree */ /* unknown buff->f_bavail */ /* unknown buff->f_files */ /* unknown buff->f_ffree */ /* ignored buff->f_favail */ /* ignored buff->f_fsid */ /* ignored buff->f_flag */ buff->f_namemax = objfs->max_name; return 0; } /* * pmemobjfs_fuse_init -- (FUSE) initialization */ static void * pmemobjfs_fuse_init(struct fuse_conn_info *conn) { log(""); struct pmemobjfs *objfs = PMEMOBJFS; TOID(struct objfs_super) super = POBJ_ROOT(objfs->pop, struct objfs_super); /* fill some runtime information */ objfs->block_size = D_RO(super)->block_size; objfs->max_name = objfs->block_size - sizeof(struct objfs_dir_entry); objfs->pool_uuid_lo = super.oid.pool_uuid_lo; TX_BEGIN(objfs->pop) { /* release all opened inodes */ map_foreach(objfs->mapc, D_RW(super)->opened, pmemobjfs_put_opened_cb, objfs); /* clear opened inodes map */ map_clear(objfs->mapc, D_RW(super)->opened); } TX_ONABORT { objfs = NULL; } TX_END return objfs; } /* * pmemobjfs_ops -- fuse operations */ static struct fuse_operations pmemobjfs_ops = { /* filesystem operations */ .init = pmemobjfs_fuse_init, .statfs = pmemobjfs_fuse_statvfs, /* inode operations */ .getattr = pmemobjfs_fuse_getattr, .chmod = pmemobjfs_fuse_chmod, .chown = pmemobjfs_fuse_chown, .utimens = pmemobjfs_fuse_utimens, .ioctl = pmemobjfs_fuse_ioctl, /* directory operations */ .opendir = pmemobjfs_fuse_opendir, .releasedir = pmemobjfs_fuse_releasedir, .readdir = pmemobjfs_fuse_readdir, .mkdir = pmemobjfs_fuse_mkdir, .rmdir = pmemobjfs_fuse_rmdir, .rename = pmemobjfs_fuse_rename, .mknod = pmemobjfs_fuse_mknod, .symlink = pmemobjfs_fuse_symlink, .create = pmemobjfs_fuse_create, .unlink = pmemobjfs_fuse_unlink, /* regular file operations */ .open = pmemobjfs_fuse_open, .release = pmemobjfs_fuse_release, .write = pmemobjfs_fuse_write, .read = pmemobjfs_fuse_read, .flush = pmemobjfs_fuse_flush, .truncate = pmemobjfs_fuse_truncate, .ftruncate = pmemobjfs_fuse_ftruncate, .fallocate = pmemobjfs_fuse_fallocate, /* symlink operations */ .readlink = pmemobjfs_fuse_readlink, }; /* * pmemobjfs_mkfs -- create pmemobjfs filesystem */ static int pmemobjfs_mkfs(const char *fname, size_t size, size_t bsize, mode_t mode) { struct pmemobjfs *objfs = calloc(1, sizeof(*objfs)); if (!objfs) return -1; int ret = 0; objfs->block_size = bsize; objfs->pop = pmemobj_create(fname, POBJ_LAYOUT_NAME(pmemobjfs), size, mode); if (!objfs->pop) { fprintf(stderr, "error: %s\n", pmemobj_errormsg()); ret = -1; goto out_free_objfs; } objfs->mapc = map_ctx_init(MAP_CTREE, objfs->pop); if (!objfs->mapc) { perror("map_ctx_init"); ret = -1; goto out_close_pop; } TOID(struct objfs_super) super = POBJ_ROOT(objfs->pop, struct objfs_super); uid_t uid = getuid(); gid_t gid = getgid(); mode_t mask = umask(0); umask(mask); TX_BEGIN(objfs->pop) { /* inherit permissions from umask */ uint64_t root_flags = S_IFDIR | (~mask & 0777); TX_ADD(super); /* create an opened files map */ map_create(objfs->mapc, &D_RW(super)->opened, NULL); /* create root inode, inherit uid and gid from current user */ D_RW(super)->root_inode = pmemobjfs_new_dir(objfs, TOID_NULL(struct objfs_inode), "/", root_flags, uid, gid); D_RW(super)->block_size = bsize; } TX_ONABORT { fprintf(stderr, "error: creating pmemobjfs aborted\n"); ret = -ECANCELED; } TX_END map_ctx_free(objfs->mapc); out_close_pop: pmemobj_close(objfs->pop); out_free_objfs: free(objfs); return ret; } /* * parse_size -- parse size from string */ static int parse_size(const char *str, uint64_t *sizep) { uint64_t size = 0; int shift = 0; char unit[4] = {0}; int ret = sscanf(str, "%lu%3s", &size, unit); if (ret <= 0) return -1; if (ret == 2) { if ((unit[1] != '\0' && unit[1] != 'B') || unit[2] != '\0') return -1; switch (unit[0]) { case 'K': case 'k': shift = 10; break; case 'M': shift = 20; break; case 'G': shift = 30; break; case 'T': shift = 40; break; case 'P': shift = 50; break; default: return -1; } } if (sizep) *sizep = size << shift; return 0; } /* * pmemobjfs_mkfs_main -- parse arguments and create pmemobjfs */ static int pmemobjfs_mkfs_main(int argc, char *argv[]) { static const char *usage_str = "usage: %s " "[-h] " "[-s ] " "[-b ] " "\n"; if (argc < 2) { fprintf(stderr, usage_str, argv[0]); return -1; } uint64_t size = PMEMOBJ_MIN_POOL; uint64_t bsize = PMEMOBJFS_MIN_BLOCK_SIZE; int opt; const char *optstr = "hs:b:"; int size_used = 0; while ((opt = getopt(argc, argv, optstr)) != -1) { switch (opt) { case 'h': printf(usage_str, argv[0]); return 0; case 'b': if (parse_size(optarg, &bsize)) { fprintf(stderr, "error: invalid block size " "value specified -- '%s'\n", optarg); return -1; } break; case 's': if (parse_size(optarg, &size)) { fprintf(stderr, "error: invalid size " "value specified -- '%s'\n", optarg); return -1; } size_used = 1; break; } } if (optind >= argc) { fprintf(stderr, usage_str, argv[0]); return -1; } const char *path = argv[optind]; if (!access(path, F_OK)) { if (size_used) { fprintf(stderr, "error: cannot use size option " "for existing file\n"); return -1; } size = 0; } else { if (size < PMEMOBJ_MIN_POOL) { fprintf(stderr, "error: minimum size is %lu\n", PMEMOBJ_MIN_POOL); return -1; } } if (bsize < PMEMOBJFS_MIN_BLOCK_SIZE) { fprintf(stderr, "error: minimum block size is %zu\n", PMEMOBJFS_MIN_BLOCK_SIZE); return -1; } return pmemobjfs_mkfs(path, size, bsize, 0777); } /* * pmemobjfs_tx_ioctl -- transaction ioctl * * In order to call the ioctl we need to create a temporary file in * specified directory and call the ioctl on that file. After calling the * ioctl the file is unlinked. The actual action is performed after unlinking * the file so if the operation was to start a transaction the temporary file * won't be unlinked within the transaction. */ static int pmemobjfs_tx_ioctl(const char *dir, int req) { int ret = 0; /* append temporary file template to specified path */ size_t dirlen = strlen(dir); size_t tmpllen = strlen(PMEMOBJFS_TMP_TEMPLATE); char *path = malloc(dirlen + tmpllen + 1); if (!path) return -1; memcpy(path, dir, dirlen); strcpy(path + dirlen, PMEMOBJFS_TMP_TEMPLATE); /* create temporary file */ mode_t prev_umask = umask(S_IRWXG | S_IRWXO); int fd = mkstemp(path); umask(prev_umask); if (fd < 0) { perror(path); ret = -1; goto out_free; } /* perform specified ioctl command */ ret = ioctl(fd, req); if (ret) { perror(path); goto out_unlink; } out_unlink: /* unlink temporary file */ ret = unlink(path); if (ret) perror(path); close(fd); out_free: free(path); return ret; } int main(int argc, char *argv[]) { char *bname = basename(argv[0]); if (strcmp(PMEMOBJFS_MKFS, bname) == 0) { return pmemobjfs_mkfs_main(argc, argv); } else if (strcmp(PMEMOBJFS_TX_BEGIN, bname) == 0) { if (argc != 2) { fprintf(stderr, "usage: %s \n", bname); return -1; } char *arg = argv[1]; return pmemobjfs_tx_ioctl(arg, PMEMOBJFS_CTL_TX_BEGIN); } else if (strcmp(PMEMOBJFS_TX_COMMIT, bname) == 0) { if (argc != 2) { fprintf(stderr, "usage: %s \n", bname); return -1; } char *arg = argv[1]; return pmemobjfs_tx_ioctl(arg, PMEMOBJFS_CTL_TX_COMMIT); } else if (strcmp(PMEMOBJFS_TX_ABORT, bname) == 0) { if (argc != 2) { fprintf(stderr, "usage: %s \n", bname); return -1; } char *arg = argv[1]; return pmemobjfs_tx_ioctl(arg, PMEMOBJFS_CTL_TX_ABORT); } #if DEBUG log_fh = fopen("pmemobjfs.log", "w+"); if (!log_fh) err(-1, "pmemobjfs.log"); log("\n\n\nPMEMOBJFS\n"); #endif const char *fname = argv[argc - 2]; struct pmemobjfs *objfs = calloc(1, sizeof(*objfs)); if (!objfs) { perror("malloc"); return -1; } int ret = 0; objfs->pop = pmemobj_open(fname, POBJ_LAYOUT_NAME(pmemobjfs)); if (objfs->pop == NULL) { perror("pmemobj_open"); ret = -1; goto out; } objfs->mapc = map_ctx_init(MAP_CTREE, objfs->pop); if (!objfs->mapc) { perror("map_ctx_init"); ret = -1; goto out; } argv[argc - 2] = argv[argc - 1]; argv[argc - 1] = NULL; argc--; ret = fuse_main(argc, argv, &pmemobjfs_ops, objfs); pmemobj_close(objfs->pop); out: free(objfs); log("ret = %d", ret); return ret; } pmdk-1.13.1/src/examples/libpmemobj/buffons_needle_problem.c0000664000000000000000000000605714435627501022636 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2019, Intel Corporation */ /* * buffons_needle_problem.c [] -- example illustrating * usage of libpmemobj * * Calculates pi number by solving Buffon's needle problem. * Takes one/two arguments -- path of the file and integer amount of trials * or only path when continuing simulation after interruption. * The greater number of trials, the higher calculation precision. */ #include #include #include #include #ifdef _WIN32 #define _USE_MATH_DEFINES #endif #include #include #include /* * Layout definition */ POBJ_LAYOUT_BEGIN(pi); POBJ_LAYOUT_ROOT(pi, struct my_root) POBJ_LAYOUT_END(pi) /* * Used for changing degrees into radians */ #define RADIAN_CALCULATE M_PI / 180.0 static PMEMobjpool *pop; struct my_root { double x; /* coordinate of the needle's center */ double angle; /* angle between vertical position and the needle */ double l; /* length of the needle */ double sin_angle_l; /* sin(angle) * l */ double pi; /* calculated pi number */ double d; /* distance between lines on the board */ uint64_t i; /* variable used in for loop */ uint64_t p; /* amount of the positive trials */ uint64_t n; /* amount of the trials */ }; static void print_usage(char *argv_main[]) { printf("usage: %s []\n", argv_main[0]); } /* * random_number -- randomizes number in range [0,1] */ static double random_number(void) { return (double)rand() / (double)RAND_MAX; } int main(int argc, char *argv[]) { if (argc < 2 || argc > 3) { print_usage(argv); return 1; } const char *path = argv[1]; if (file_exists(path) != 0) { if ((pop = pmemobj_create(path, POBJ_LAYOUT_NAME(pi), PMEMOBJ_MIN_POOL, 0666)) == NULL) { perror("failed to create pool\n"); return 1; } } else { if ((pop = pmemobj_open(path, POBJ_LAYOUT_NAME(pi))) == NULL) { perror("failed to open pool\n"); return 1; } } srand((unsigned int)time(NULL)); TOID(struct my_root) root = POBJ_ROOT(pop, struct my_root); struct my_root *const rootp_rw = D_RW(root); if (argc == 3) { const char *n = argv[2]; char *endptr; errno = 0; uint64_t ull_n = strtoull(n, &endptr, 10); if (*endptr != '\0' || (ull_n == ULLONG_MAX && errno == ERANGE)) { perror("wrong n parameter\n"); print_usage(argv); pmemobj_close(pop); return 1; } TX_BEGIN(pop) { TX_ADD(root); rootp_rw->l = 0.9; rootp_rw->d = 1.0; rootp_rw->i = 0; rootp_rw->p = 0; rootp_rw->n = ull_n; } TX_END } for (; rootp_rw->i < rootp_rw->n; ) { TX_BEGIN(pop) { TX_ADD(root); rootp_rw->angle = random_number() * 90 * RADIAN_CALCULATE; rootp_rw->x = random_number() * rootp_rw->d / 2; rootp_rw->sin_angle_l = rootp_rw->l / 2 * sin(rootp_rw->angle); if (rootp_rw->x <= rootp_rw->sin_angle_l) { rootp_rw->p++; } rootp_rw->pi = (2 * rootp_rw->l * rootp_rw->n) / (rootp_rw->p * rootp_rw->d); rootp_rw->i++; } TX_END } printf("%f\n", D_RO(root)->pi); pmemobj_close(pop); return 0; } pmdk-1.13.1/src/examples/libpmemobj/buffons_needle_problem.vcxproj0000664000000000000000000000553514435627501024107 0ustar rootroot Debug x64 Release x64 {BA0EF7F5-BE6C-4B61-9D5F-1480462EE001} pmemobj 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} Application true v143 Application false v143 true ..\..\LongPath.manifest CompileAsCpp libpmem.lib;libpmemobj.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies) pmdk-1.13.1/src/examples/libpmemobj/Makefile0000664000000000000000000000117414435627501017427 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2014-2021, Intel Corporation # # examples/libpmemobj/Makefile -- build the libpmemobj examples # PROGS = manpage btree pi lists setjmp buffons_needle_problem DIRS = pminvaders pmemlog pmemblk string_store string_store_tx\ string_store_tx_type hashmap tree_map pmemobjfs map libart\ linkedlist list_map slab_allocator queue LIBS = -lpmemobj -lpmem -pthread -lm -pthread include ../Makefile.inc map: hashmap tree_map pmemobjfs: map manpage: manpage.o btree: btree.o pi: pi.o lists: lists.o buffons_needle_problem: buffons_needle_problem.o setjmp: CFLAGS += -O2 setjmp: setjmp.o pmdk-1.13.1/src/examples/libpmemobj/buffons_needle_problem.vcxproj.filters0000664000000000000000000000066314435627501025553 0ustar rootroot {69963480-7711-4463-a2fb-38be4d57760e} Source Files pmdk-1.13.1/src/examples/libpmemobj/map/0000775000000000000000000000000014435627501016541 5ustar rootrootpmdk-1.13.1/src/examples/libpmemobj/map/map_rbtree.c0000664000000000000000000001047314435627501021032 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2021, Intel Corporation */ /* * map_rbtree.c -- common interface for maps */ #include #include #include "map_rbtree.h" /* * map_rbtree_check -- wrapper for rbtree_map_check */ static int map_rbtree_check(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct rbtree_map) rbtree_map; TOID_ASSIGN(rbtree_map, map.oid); return rbtree_map_check(pop, rbtree_map); } /* * map_rbtree_create -- wrapper for rbtree_map_new */ static int map_rbtree_create(PMEMobjpool *pop, TOID(struct map) *map, void *arg) { TOID(struct rbtree_map) *rbtree_map = (TOID(struct rbtree_map) *)map; return rbtree_map_create(pop, rbtree_map, arg); } /* * map_rbtree_destroy -- wrapper for rbtree_map_delete */ static int map_rbtree_destroy(PMEMobjpool *pop, TOID(struct map) *map) { TOID(struct rbtree_map) *rbtree_map = (TOID(struct rbtree_map) *)map; return rbtree_map_destroy(pop, rbtree_map); } /* * map_rbtree_insert -- wrapper for rbtree_map_insert */ static int map_rbtree_insert(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, PMEMoid value) { TOID(struct rbtree_map) rbtree_map; TOID_ASSIGN(rbtree_map, map.oid); return rbtree_map_insert(pop, rbtree_map, key, value); } /* * map_rbtree_insert_new -- wrapper for rbtree_map_insert_new */ static int map_rbtree_insert_new(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg) { TOID(struct rbtree_map) rbtree_map; TOID_ASSIGN(rbtree_map, map.oid); return rbtree_map_insert_new(pop, rbtree_map, key, size, type_num, constructor, arg); } /* * map_rbtree_remove -- wrapper for rbtree_map_remove */ static PMEMoid map_rbtree_remove(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct rbtree_map) rbtree_map; TOID_ASSIGN(rbtree_map, map.oid); return rbtree_map_remove(pop, rbtree_map, key); } /* * map_rbtree_remove_free -- wrapper for rbtree_map_remove_free */ static int map_rbtree_remove_free(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct rbtree_map) rbtree_map; TOID_ASSIGN(rbtree_map, map.oid); return rbtree_map_remove_free(pop, rbtree_map, key); } /* * map_rbtree_clear -- wrapper for rbtree_map_clear */ static int map_rbtree_clear(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct rbtree_map) rbtree_map; TOID_ASSIGN(rbtree_map, map.oid); return rbtree_map_clear(pop, rbtree_map); } /* * map_rbtree_get -- wrapper for rbtree_map_get */ static PMEMoid map_rbtree_get(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct rbtree_map) rbtree_map; TOID_ASSIGN(rbtree_map, map.oid); return rbtree_map_get(pop, rbtree_map, key); } /* * map_rbtree_lookup -- wrapper for rbtree_map_lookup */ static int map_rbtree_lookup(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct rbtree_map) rbtree_map; TOID_ASSIGN(rbtree_map, map.oid); return rbtree_map_lookup(pop, rbtree_map, key); } /* * map_rbtree_foreach -- wrapper for rbtree_map_foreach */ static int map_rbtree_foreach(PMEMobjpool *pop, TOID(struct map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { TOID(struct rbtree_map) rbtree_map; TOID_ASSIGN(rbtree_map, map.oid); return rbtree_map_foreach(pop, rbtree_map, cb, arg); } /* * map_rbtree_is_empty -- wrapper for rbtree_map_is_empty */ static int map_rbtree_is_empty(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct rbtree_map) rbtree_map; TOID_ASSIGN(rbtree_map, map.oid); return rbtree_map_is_empty(pop, rbtree_map); } /* * map_rbtree_init -- recovers map state * Since there is no need for recovery for rbtree, function is dummy. */ static int map_rbtree_init(PMEMobjpool *pop, TOID(struct map) map) { return 0; } struct map_ops rbtree_map_ops = { /* .check = */ map_rbtree_check, /* .create = */ map_rbtree_create, /* .destroy = */ map_rbtree_destroy, /* .init = */ map_rbtree_init, /* .insert = */ map_rbtree_insert, /* .insert_new = */ map_rbtree_insert_new, /* .remove = */ map_rbtree_remove, /* .remove_free = */ map_rbtree_remove_free, /* .clear = */ map_rbtree_clear, /* .get = */ map_rbtree_get, /* .lookup = */ map_rbtree_lookup, /* .foreach = */ map_rbtree_foreach, /* .is_empty = */ map_rbtree_is_empty, /* .count = */ NULL, /* .cmd = */ NULL, }; pmdk-1.13.1/src/examples/libpmemobj/map/.gitignore0000664000000000000000000000003414435627501020526 0ustar rootrootmapcli data_store kv_server pmdk-1.13.1/src/examples/libpmemobj/map/kv_server_test.sh0000775000000000000000000000103114435627501022140 0ustar rootroot#!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # Copyright 2015-2016, Intel Corporation set -euo pipefail MAP=ctree PORT=9100 POOL=$1 # start a new server instance ./kv_server $MAP $POOL $PORT & # wait for the server to properly start sleep 1 # insert a new key value pair and disconnect RESP=`echo -e "INSERT foo bar\nGET foo\nBYE" | nc 127.0.0.1 $PORT` echo $RESP # remove previously inserted key value pair and shutdown the server RESP=`echo -e "GET foo\nREMOVE foo\nGET foo\nKILL" | nc 127.0.0.1 $PORT` echo $RESP pmdk-1.13.1/src/examples/libpmemobj/map/map_ctree.h0000664000000000000000000000056114435627501020653 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2015-2020, Intel Corporation */ /* * map_ctree.h -- common interface for maps */ #ifndef MAP_CTREE_H #define MAP_CTREE_H #include "map.h" #ifdef __cplusplus extern "C" { #endif extern struct map_ops ctree_map_ops; #define MAP_CTREE (&ctree_map_ops) #ifdef __cplusplus } #endif #endif /* MAP_CTREE_H */ pmdk-1.13.1/src/examples/libpmemobj/map/map_hashmap_rp.c0000664000000000000000000000636314435627501021674 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2018, Intel Corporation */ /* * map_hashmap_rp.c -- common interface for maps */ #include #include #include "map_hashmap_rp.h" /* * map_hm_rp_check -- wrapper for hm_rp_check */ static int map_hm_rp_check(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct hashmap_rp) hashmap_rp; TOID_ASSIGN(hashmap_rp, map.oid); return hm_rp_check(pop, hashmap_rp); } /* * map_hm_rp_count -- wrapper for hm_rp_count */ static size_t map_hm_rp_count(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct hashmap_rp) hashmap_rp; TOID_ASSIGN(hashmap_rp, map.oid); return hm_rp_count(pop, hashmap_rp); } /* * map_hm_rp_init -- wrapper for hm_rp_init */ static int map_hm_rp_init(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct hashmap_rp) hashmap_rp; TOID_ASSIGN(hashmap_rp, map.oid); return hm_rp_init(pop, hashmap_rp); } /* * map_hm_rp_create -- wrapper for hm_rp_create */ static int map_hm_rp_create(PMEMobjpool *pop, TOID(struct map) *map, void *arg) { TOID(struct hashmap_rp) *hashmap_rp = (TOID(struct hashmap_rp) *)map; return hm_rp_create(pop, hashmap_rp, arg); } /* * map_hm_rp_insert -- wrapper for hm_rp_insert */ static int map_hm_rp_insert(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, PMEMoid value) { TOID(struct hashmap_rp) hashmap_rp; TOID_ASSIGN(hashmap_rp, map.oid); return hm_rp_insert(pop, hashmap_rp, key, value); } /* * map_hm_rp_remove -- wrapper for hm_rp_remove */ static PMEMoid map_hm_rp_remove(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct hashmap_rp) hashmap_rp; TOID_ASSIGN(hashmap_rp, map.oid); return hm_rp_remove(pop, hashmap_rp, key); } /* * map_hm_rp_get -- wrapper for hm_rp_get */ static PMEMoid map_hm_rp_get(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct hashmap_rp) hashmap_rp; TOID_ASSIGN(hashmap_rp, map.oid); return hm_rp_get(pop, hashmap_rp, key); } /* * map_hm_rp_lookup -- wrapper for hm_rp_lookup */ static int map_hm_rp_lookup(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct hashmap_rp) hashmap_rp; TOID_ASSIGN(hashmap_rp, map.oid); return hm_rp_lookup(pop, hashmap_rp, key); } /* * map_hm_rp_foreach -- wrapper for hm_rp_foreach */ static int map_hm_rp_foreach(PMEMobjpool *pop, TOID(struct map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { TOID(struct hashmap_rp) hashmap_rp; TOID_ASSIGN(hashmap_rp, map.oid); return hm_rp_foreach(pop, hashmap_rp, cb, arg); } /* * map_hm_rp_cmd -- wrapper for hm_rp_cmd */ static int map_hm_rp_cmd(PMEMobjpool *pop, TOID(struct map) map, unsigned cmd, uint64_t arg) { TOID(struct hashmap_rp) hashmap_rp; TOID_ASSIGN(hashmap_rp, map.oid); return hm_rp_cmd(pop, hashmap_rp, cmd, arg); } struct map_ops hashmap_rp_ops = { /* .check = */ map_hm_rp_check, /* .create = */ map_hm_rp_create, /* .destroy = */ NULL, /* .init = */ map_hm_rp_init, /* .insert = */ map_hm_rp_insert, /* .insert_new = */ NULL, /* .remove = */ map_hm_rp_remove, /* .remove_free = */ NULL, /* .clear = */ NULL, /* .get = */ map_hm_rp_get, /* .lookup = */ map_hm_rp_lookup, /* .foreach = */ map_hm_rp_foreach, /* .is_empty = */ NULL, /* .count = */ map_hm_rp_count, /* .cmd = */ map_hm_rp_cmd, }; pmdk-1.13.1/src/examples/libpmemobj/map/mapcli.vcxproj0000664000000000000000000000600114435627501021420 0ustar rootroot Debug x64 Release x64 {BB248BAC-6E1B-433C-A254-75140A273AB5} pmemobj 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} Application true v143 Application false v143 true ..\..\..\LongPath.manifest CompileAsCpp libpmem.lib;libpmemobj.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies) {49a7cc5a-d5e7-4a07-917f-c6918b982be8} pmdk-1.13.1/src/examples/libpmemobj/map/kv_protocol.h0000664000000000000000000000404514435627501021256 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2015-2020, Intel Corporation */ /* * kv_protocol.h -- kv store text protocol */ #ifndef KV_PROTOCOL_H #define KV_PROTOCOL_H #include #define MAX_KEY_LEN 255 /* * All client messages must start with a valid message token and be terminated * by a newline character ('\n'). The message parser is case-sensitive. * * Server responds with newline terminated string literals. * If invalid message token is received RESP_MSG_UNKNOWN is sent. */ enum kv_cmsg { /* * INSERT client message * Syntax: INSERT [key] [value]\n * * The key is limited to 255 characters, the size of a value is limited * by the pmemobj maximum allocation size (~16 gigabytes). * * Operation adds a new key value pair to the map. * Returns RESP_MSG_SUCCESS if successful or RESP_MSG_FAIL otherwise. */ CMSG_INSERT, /* * REMOVE client message * Syntax: REMOVE [key]\n * * Operation removes a key value pair from the map. * Returns RESP_MSG_SUCCESS if successful or RESP_MSG_FAIL otherwise. */ CMSG_REMOVE, /* * GET client message * Syntax: GET [key]\n * * Operation retrieves a key value pair from the map. * Returns the value if found or RESP_MSG_NULL otherwise. */ CMSG_GET, /* * BYE client message * Syntax: BYE\n * * Operation terminates the client connection. * No return value. */ CMSG_BYE, /* * KILL client message * Syntax: KILL\n * * Operation terminates the client connection and gracefully shutdowns * the server. * No return value. */ CMSG_KILL, MAX_CMSG }; enum resp_messages { RESP_MSG_SUCCESS, RESP_MSG_FAIL, RESP_MSG_NULL, RESP_MSG_UNKNOWN, MAX_RESP_MSG }; static const char *resp_msg[MAX_RESP_MSG] = { [RESP_MSG_SUCCESS] = "SUCCESS\n", [RESP_MSG_FAIL] = "FAIL\n", [RESP_MSG_NULL] = "NULL\n", [RESP_MSG_UNKNOWN] = "UNKNOWN\n" }; static const char *kv_cmsg_token[MAX_CMSG] = { [CMSG_INSERT] = "INSERT", [CMSG_REMOVE] = "REMOVE", [CMSG_GET] = "GET", [CMSG_BYE] = "BYE", [CMSG_KILL] = "KILL" }; #endif /* KV_PROTOCOL_H */ pmdk-1.13.1/src/examples/libpmemobj/map/data_store.c0000664000000000000000000001332714435627501021040 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2020, Intel Corporation */ /* * data_store.c -- tree_map example usage */ #include #include #include #include #include #include #include #include "map.h" #include "map_ctree.h" #include "map_btree.h" #include "map_rbtree.h" #include "map_hashmap_atomic.h" #include "map_hashmap_tx.h" #include "map_hashmap_rp.h" #include "map_skiplist.h" POBJ_LAYOUT_BEGIN(data_store); POBJ_LAYOUT_ROOT(data_store, struct store_root); POBJ_LAYOUT_TOID(data_store, struct store_item); POBJ_LAYOUT_END(data_store); #define MAX_INSERTS 500 static uint64_t nkeys; static uint64_t keys[MAX_INSERTS]; struct store_item { uint64_t item_data; }; struct store_root { TOID(struct map) map; }; typedef int (*insert_rand_fn)(struct map_ctx *mapc, TOID(struct store_root) *root, int nops); /* * new_store_item_transact -- transactionally creates and initializes new item */ static TOID(struct store_item) new_store_item_transact(void) { TOID(struct store_item) item = TX_NEW(struct store_item); D_RW(item)->item_data = rand(); return item; } /* * store_item_construct -- initializes data of the new store item */ static int store_item_construct(PMEMobjpool *pop, void *ptr, void *arg) { struct store_item *item = (struct store_item *)ptr; item->item_data = rand(); pmemobj_persist(pop, item, sizeof(*item)); return 0; } /* * new_store_item -- creates and initializes new item */ static TOID(struct store_item) new_store_item(PMEMobjpool *pop) { TOID(struct store_item) item; POBJ_NEW(pop, &item, struct store_item, store_item_construct, NULL); return item; } /* * insert_rand_items -- insert new random items */ static int insert_rand_items(struct map_ctx *mapc, TOID(struct store_root) *root, int nops) { map_create(mapc, &D_RW(*root)->map, NULL); /* insert random items */ for (int i = 0; i < nops; ++i) map_insert(mapc, D_RW(*root)->map, rand(), new_store_item(mapc->pop).oid); return 0; } /* * insert_rand_items_transact -- insert new random items transactionally */ static int insert_rand_items_transact(struct map_ctx *mapc, TOID(struct store_root) *root, int nops) { int aborted = 0; /* transactional APIs should be inside transaction block */ TX_BEGIN(mapc->pop) { map_create(mapc, &D_RW(*root)->map, NULL); /* insert random items in transaction */ for (int i = 0; i < nops; ++i) map_insert(mapc, D_RW(*root)->map, rand(), new_store_item_transact().oid); } TX_ONABORT { perror("transaction aborted\n"); map_ctx_free(mapc); aborted = 1; } TX_END return aborted; } /* * get_keys -- inserts the keys of the items by key order (sorted, descending) */ static int get_keys(uint64_t key, PMEMoid value, void *arg) { keys[nkeys++] = key; return 0; } /* * dec_keys -- decrements the keys count for every item */ static int dec_keys(uint64_t key, PMEMoid value, void *arg) { nkeys--; return 0; } /* * parse_map_type -- parse type of map */ static const struct map_ops * parse_map_type(const char *type, insert_rand_fn *insert) { if (strcmp(type, "ctree") == 0) { *insert = insert_rand_items_transact; return MAP_CTREE; } else if (strcmp(type, "btree") == 0) { *insert = insert_rand_items_transact; return MAP_BTREE; } else if (strcmp(type, "rbtree") == 0) { *insert = insert_rand_items_transact; return MAP_RBTREE; } else if (strcmp(type, "hashmap_atomic") == 0) { *insert = insert_rand_items; return MAP_HASHMAP_ATOMIC; } else if (strcmp(type, "hashmap_tx") == 0) { *insert = insert_rand_items_transact; return MAP_HASHMAP_TX; } else if (strcmp(type, "hashmap_rp") == 0) { *insert = insert_rand_items; return MAP_HASHMAP_RP; } else if (strcmp(type, "skiplist") == 0) { *insert = insert_rand_items_transact; return MAP_SKIPLIST; } return NULL; } int main(int argc, const char *argv[]) { if (argc < 3) { printf("usage: %s " " file-name [nops]\n", argv[0]); return 1; } const char *type = argv[1]; const char *path = argv[2]; insert_rand_fn insert_items; const struct map_ops *map_ops = parse_map_type(type, &insert_items); if (!map_ops) { fprintf(stderr, "invalid container type -- '%s'\n", type); return 1; } int nops = MAX_INSERTS; if (argc > 3) { nops = atoi(argv[3]); if (nops <= 0 || nops > MAX_INSERTS) { fprintf(stderr, "number of operations must be " "in range 1..%d\n", MAX_INSERTS); return 1; } } PMEMobjpool *pop; srand((unsigned)time(NULL)); if (file_exists(path) != 0) { if ((pop = pmemobj_create(path, POBJ_LAYOUT_NAME(data_store), PMEMOBJ_MIN_POOL, 0666)) == NULL) { perror("failed to create pool\n"); return 1; } } else { if ((pop = pmemobj_open(path, POBJ_LAYOUT_NAME(data_store))) == NULL) { perror("failed to open pool\n"); return 1; } } TOID(struct store_root) root = POBJ_ROOT(pop, struct store_root); struct map_ctx *mapc = map_ctx_init(map_ops, pop); if (!mapc) { perror("cannot allocate map context\n"); return 1; } /* delete the map if it exists */ if (!map_check(mapc, D_RW(root)->map)) map_destroy(mapc, &D_RW(root)->map); /* insert random items */ if (insert_items(mapc, &root, nops)) return 1; /* count the items */ map_foreach(mapc, D_RW(root)->map, get_keys, NULL); /* remove the items without outer transaction */ for (uint64_t i = 0; i < nkeys; ++i) { PMEMoid item = map_remove(mapc, D_RW(root)->map, keys[i]); assert(!OID_IS_NULL(item)); assert(OID_INSTANCEOF(item, struct store_item)); } uint64_t old_nkeys = nkeys; /* tree should be empty */ map_foreach(mapc, D_RW(root)->map, dec_keys, NULL); assert(old_nkeys == nkeys); map_ctx_free(mapc); pmemobj_close(pop); return 0; } pmdk-1.13.1/src/examples/libpmemobj/map/map_hashmap_tx.c0000664000000000000000000000636714435627501021712 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2017, Intel Corporation */ /* * map_hashmap_tx.c -- common interface for maps */ #include #include #include "map_hashmap_tx.h" /* * map_hm_tx_check -- wrapper for hm_tx_check */ static int map_hm_tx_check(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct hashmap_tx) hashmap_tx; TOID_ASSIGN(hashmap_tx, map.oid); return hm_tx_check(pop, hashmap_tx); } /* * map_hm_tx_count -- wrapper for hm_tx_count */ static size_t map_hm_tx_count(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct hashmap_tx) hashmap_tx; TOID_ASSIGN(hashmap_tx, map.oid); return hm_tx_count(pop, hashmap_tx); } /* * map_hm_tx_init -- wrapper for hm_tx_init */ static int map_hm_tx_init(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct hashmap_tx) hashmap_tx; TOID_ASSIGN(hashmap_tx, map.oid); return hm_tx_init(pop, hashmap_tx); } /* * map_hm_tx_create -- wrapper for hm_tx_create */ static int map_hm_tx_create(PMEMobjpool *pop, TOID(struct map) *map, void *arg) { TOID(struct hashmap_tx) *hashmap_tx = (TOID(struct hashmap_tx) *)map; return hm_tx_create(pop, hashmap_tx, arg); } /* * map_hm_tx_insert -- wrapper for hm_tx_insert */ static int map_hm_tx_insert(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, PMEMoid value) { TOID(struct hashmap_tx) hashmap_tx; TOID_ASSIGN(hashmap_tx, map.oid); return hm_tx_insert(pop, hashmap_tx, key, value); } /* * map_hm_tx_remove -- wrapper for hm_tx_remove */ static PMEMoid map_hm_tx_remove(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct hashmap_tx) hashmap_tx; TOID_ASSIGN(hashmap_tx, map.oid); return hm_tx_remove(pop, hashmap_tx, key); } /* * map_hm_tx_get -- wrapper for hm_tx_get */ static PMEMoid map_hm_tx_get(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct hashmap_tx) hashmap_tx; TOID_ASSIGN(hashmap_tx, map.oid); return hm_tx_get(pop, hashmap_tx, key); } /* * map_hm_tx_lookup -- wrapper for hm_tx_lookup */ static int map_hm_tx_lookup(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct hashmap_tx) hashmap_tx; TOID_ASSIGN(hashmap_tx, map.oid); return hm_tx_lookup(pop, hashmap_tx, key); } /* * map_hm_tx_foreach -- wrapper for hm_tx_foreach */ static int map_hm_tx_foreach(PMEMobjpool *pop, TOID(struct map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { TOID(struct hashmap_tx) hashmap_tx; TOID_ASSIGN(hashmap_tx, map.oid); return hm_tx_foreach(pop, hashmap_tx, cb, arg); } /* * map_hm_tx_cmd -- wrapper for hm_tx_cmd */ static int map_hm_tx_cmd(PMEMobjpool *pop, TOID(struct map) map, unsigned cmd, uint64_t arg) { TOID(struct hashmap_tx) hashmap_tx; TOID_ASSIGN(hashmap_tx, map.oid); return hm_tx_cmd(pop, hashmap_tx, cmd, arg); } struct map_ops hashmap_tx_ops = { /* .check = */ map_hm_tx_check, /* .create = */ map_hm_tx_create, /* .delete = */ NULL, /* .init = */ map_hm_tx_init, /* .insert = */ map_hm_tx_insert, /* .insert_new = */ NULL, /* .remove = */ map_hm_tx_remove, /* .remove_free = */ NULL, /* .clear = */ NULL, /* .get = */ map_hm_tx_get, /* .lookup = */ map_hm_tx_lookup, /* .foreach = */ map_hm_tx_foreach, /* .is_empty = */ NULL, /* .count = */ map_hm_tx_count, /* .cmd = */ map_hm_tx_cmd, }; pmdk-1.13.1/src/examples/libpmemobj/map/libmap.vcxproj.filters0000664000000000000000000000403214435627501023070 0ustar rootroot {b263d2a8-4e00-4310-ae46-f24fc6aa8f4f} {f7a2eae5-3cc9-452a-801d-73ca2d879931} Header Files Header Files Header Files Header Files Header Files Header Files Header Files Header Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files Source Files pmdk-1.13.1/src/examples/libpmemobj/map/Makefile0000664000000000000000000000446114435627501020206 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2015-2018, Intel Corporation # # examples/libpmemobj/map/Makefile -- build the map example # TOP := $(dir $(lastword $(MAKEFILE_LIST)))../../../../ include $(TOP)/src/common.inc PROGS = mapcli data_store LIBRARIES = map_ctree map_btree map_rbtree map_skiplist\ map_hashmap_atomic map_hashmap_tx map_hashmap_rp\ map_rtree map LIBUV := $(call check_package, libuv --atleast-version 1.0) ifeq ($(LIBUV),y) PROGS += kv_server else $(info NOTE: Skipping kv_server because libuv is missing \ -- see src/examples/libpmemobj/map/README for details.) endif LIBS = -lpmemobj -pthread ifeq ($(LIBUV),y) LIBS += $(shell $(PKG_CONFIG) --libs libuv) endif include ../../Makefile.inc CFLAGS += -I../ CFLAGS += -I../hashmap CFLAGS += -I../tree_map CFLAGS += -I../list_map mapcli: mapcli.o libmap.a data_store: data_store.o libmap.a kv_server: kv_server.o libmap.a libmap_ctree.o: map_ctree.o map.o ../tree_map/libctree_map.a libmap_btree.o: map_btree.o map.o ../tree_map/libbtree_map.a libmap_rtree.o: map_rtree.o map.o ../tree_map/librtree_map.a libmap_rbtree.o: map_rbtree.o map.o ../tree_map/librbtree_map.a libmap_hashmap_atomic.o: map_hashmap_atomic.o map.o ../hashmap/libhashmap_atomic.a libmap_hashmap_tx.o: map_hashmap_tx.o map.o ../hashmap/libhashmap_tx.a libmap_hashmap_rp.o: map_hashmap_rp.o map.o ../hashmap/libhashmap_rp.a libmap_skiplist.o: map_skiplist.o map.o ../list_map/libskiplist_map.a libmap.o: map.o map_ctree.o map_btree.o map_rtree.o map_rbtree.o map_skiplist.o\ map_hashmap_atomic.o map_hashmap_tx.o map_hashmap_rp.o\ ../tree_map/libctree_map.a\ ../tree_map/libbtree_map.a\ ../tree_map/librtree_map.a\ ../tree_map/librbtree_map.a\ ../list_map/libskiplist_map.a\ ../hashmap/libhashmap_atomic.a\ ../hashmap/libhashmap_tx.a\ ../hashmap/libhashmap_rp.a ../tree_map/libctree_map.a: $(MAKE) -C ../tree_map ctree_map ../tree_map/libbtree_map.a: $(MAKE) -C ../tree_map btree_map ../tree_map/librtree_map.a: $(MAKE) -C ../tree_map rtree_map ../tree_map/librbtree_map.a: $(MAKE) -C ../tree_map rbtree_map ../list_map/libskiplist_map.a: $(MAKE) -C ../list_map skiplist_map ../hashmap/libhashmap_atomic.a: $(MAKE) -C ../hashmap hashmap_atomic ../hashmap/libhashmap_tx.a: $(MAKE) -C ../hashmap hashmap_tx ../hashmap/libhashmap_rp.a: $(MAKE) -C ../hashmap hashmap_rp pmdk-1.13.1/src/examples/libpmemobj/map/map_ctree.c0000664000000000000000000001031314435627501020642 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2021, Intel Corporation */ /* * map_ctree.c -- common interface for maps */ #include #include #include "map_ctree.h" /* * map_ctree_check -- wrapper for ctree_map_check */ static int map_ctree_check(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct ctree_map) ctree_map; TOID_ASSIGN(ctree_map, map.oid); return ctree_map_check(pop, ctree_map); } /* * map_ctree_create -- wrapper for ctree_map_create */ static int map_ctree_create(PMEMobjpool *pop, TOID(struct map) *map, void *arg) { TOID(struct ctree_map) *ctree_map = (TOID(struct ctree_map) *)map; return ctree_map_create(pop, ctree_map, arg); } /* * map_ctree_destroy -- wrapper for ctree_map_destroy */ static int map_ctree_destroy(PMEMobjpool *pop, TOID(struct map) *map) { TOID(struct ctree_map) *ctree_map = (TOID(struct ctree_map) *)map; return ctree_map_destroy(pop, ctree_map); } /* * map_ctree_insert -- wrapper for ctree_map_insert */ static int map_ctree_insert(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, PMEMoid value) { TOID(struct ctree_map) ctree_map; TOID_ASSIGN(ctree_map, map.oid); return ctree_map_insert(pop, ctree_map, key, value); } /* * map_ctree_insert_new -- wrapper for ctree_map_insert_new */ static int map_ctree_insert_new(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg) { TOID(struct ctree_map) ctree_map; TOID_ASSIGN(ctree_map, map.oid); return ctree_map_insert_new(pop, ctree_map, key, size, type_num, constructor, arg); } /* * map_ctree_remove -- wrapper for ctree_map_remove */ static PMEMoid map_ctree_remove(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct ctree_map) ctree_map; TOID_ASSIGN(ctree_map, map.oid); return ctree_map_remove(pop, ctree_map, key); } /* * map_ctree_remove_free -- wrapper for ctree_map_remove_free */ static int map_ctree_remove_free(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct ctree_map) ctree_map; TOID_ASSIGN(ctree_map, map.oid); return ctree_map_remove_free(pop, ctree_map, key); } /* * map_ctree_clear -- wrapper for ctree_map_clear */ static int map_ctree_clear(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct ctree_map) ctree_map; TOID_ASSIGN(ctree_map, map.oid); return ctree_map_clear(pop, ctree_map); } /* * map_ctree_get -- wrapper for ctree_map_get */ static PMEMoid map_ctree_get(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct ctree_map) ctree_map; TOID_ASSIGN(ctree_map, map.oid); return ctree_map_get(pop, ctree_map, key); } /* * map_ctree_lookup -- wrapper for ctree_map_lookup */ static int map_ctree_lookup(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct ctree_map) ctree_map; TOID_ASSIGN(ctree_map, map.oid); return ctree_map_lookup(pop, ctree_map, key); } /* * map_ctree_foreach -- wrapper for ctree_map_foreach */ static int map_ctree_foreach(PMEMobjpool *pop, TOID(struct map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { TOID(struct ctree_map) ctree_map; TOID_ASSIGN(ctree_map, map.oid); return ctree_map_foreach(pop, ctree_map, cb, arg); } /* * map_ctree_is_empty -- wrapper for ctree_map_is_empty */ static int map_ctree_is_empty(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct ctree_map) ctree_map; TOID_ASSIGN(ctree_map, map.oid); return ctree_map_is_empty(pop, ctree_map); } /* * map_ctree_init -- recovers map state * Since there is no need for recovery for ctree, function is dummy. */ static int map_ctree_init(PMEMobjpool *pop, TOID(struct map) map) { return 0; } struct map_ops ctree_map_ops = { /* .check = */ map_ctree_check, /* .create = */ map_ctree_create, /* .destroy = */ map_ctree_destroy, /* .init = */ map_ctree_init, /* .insert = */ map_ctree_insert, /* .insert_new = */ map_ctree_insert_new, /* .remove = */ map_ctree_remove, /* .remove_free = */ map_ctree_remove_free, /* .clear = */ map_ctree_clear, /* .get = */ map_ctree_get, /* .lookup = */ map_ctree_lookup, /* .foreach = */ map_ctree_foreach, /* .is_empty = */ map_ctree_is_empty, /* .count = */ NULL, /* .cmd = */ NULL, }; pmdk-1.13.1/src/examples/libpmemobj/map/map_rtree.h0000664000000000000000000000056114435627501020672 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2016-2020, Intel Corporation */ /* * map_rtree.h -- common interface for maps */ #ifndef MAP_RTREE_H #define MAP_RTREE_H #include "map.h" #ifdef __cplusplus extern "C" { #endif extern struct map_ops rtree_map_ops; #define MAP_RTREE (&rtree_map_ops) #ifdef __cplusplus } #endif #endif /* MAP_RTREE_H */ pmdk-1.13.1/src/examples/libpmemobj/map/map_hashmap_rp.h0000664000000000000000000000061414435627501021672 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2018-2020, Intel Corporation */ /* * map_hashmap_rp.h -- common interface for maps */ #ifndef MAP_HASHMAP_RP_H #define MAP_HASHMAP_RP_H #include "map.h" #ifdef __cplusplus extern "C" { #endif extern struct map_ops hashmap_rp_ops; #define MAP_HASHMAP_RP (&hashmap_rp_ops) #ifdef __cplusplus } #endif #endif /* MAP_HASHMAP_RP_H */ pmdk-1.13.1/src/examples/libpmemobj/map/map.h0000664000000000000000000000570514435627501017476 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2015-2020, Intel Corporation */ /* * map.h -- common interface for maps */ #ifndef MAP_H #define MAP_H #include #ifdef __cplusplus extern "C" { #endif #ifndef MAP_TYPE_OFFSET #define MAP_TYPE_OFFSET 1000 #endif TOID_DECLARE(struct map, MAP_TYPE_OFFSET + 0); struct map; struct map_ctx; struct map_ops { int(*check)(PMEMobjpool *pop, TOID(struct map) map); int(*create)(PMEMobjpool *pop, TOID(struct map) *map, void *arg); int(*destroy)(PMEMobjpool *pop, TOID(struct map) *map); int(*init)(PMEMobjpool *pop, TOID(struct map) map); int(*insert)(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, PMEMoid value); int(*insert_new)(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, size_t size, unsigned type_num, void(*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg); PMEMoid(*remove)(PMEMobjpool *pop, TOID(struct map) map, uint64_t key); int(*remove_free)(PMEMobjpool *pop, TOID(struct map) map, uint64_t key); int(*clear)(PMEMobjpool *pop, TOID(struct map) map); PMEMoid(*get)(PMEMobjpool *pop, TOID(struct map) map, uint64_t key); int(*lookup)(PMEMobjpool *pop, TOID(struct map) map, uint64_t key); int(*foreach)(PMEMobjpool *pop, TOID(struct map) map, int(*cb)(uint64_t key, PMEMoid value, void *arg), void *arg); int(*is_empty)(PMEMobjpool *pop, TOID(struct map) map); size_t(*count)(PMEMobjpool *pop, TOID(struct map) map); int(*cmd)(PMEMobjpool *pop, TOID(struct map) map, unsigned cmd, uint64_t arg); }; struct map_ctx { PMEMobjpool *pop; const struct map_ops *ops; }; struct map_ctx *map_ctx_init(const struct map_ops *ops, PMEMobjpool *pop); void map_ctx_free(struct map_ctx *mapc); int map_check(struct map_ctx *mapc, TOID(struct map) map); int map_create(struct map_ctx *mapc, TOID(struct map) *map, void *arg); int map_destroy(struct map_ctx *mapc, TOID(struct map) *map); int map_init(struct map_ctx *mapc, TOID(struct map) map); int map_insert(struct map_ctx *mapc, TOID(struct map) map, uint64_t key, PMEMoid value); int map_insert_new(struct map_ctx *mapc, TOID(struct map) map, uint64_t key, size_t size, unsigned type_num, void(*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg); PMEMoid map_remove(struct map_ctx *mapc, TOID(struct map) map, uint64_t key); int map_remove_free(struct map_ctx *mapc, TOID(struct map) map, uint64_t key); int map_clear(struct map_ctx *mapc, TOID(struct map) map); PMEMoid map_get(struct map_ctx *mapc, TOID(struct map) map, uint64_t key); int map_lookup(struct map_ctx *mapc, TOID(struct map) map, uint64_t key); int map_foreach(struct map_ctx *mapc, TOID(struct map) map, int(*cb)(uint64_t key, PMEMoid value, void *arg), void *arg); int map_is_empty(struct map_ctx *mapc, TOID(struct map) map); size_t map_count(struct map_ctx *mapc, TOID(struct map) map); int map_cmd(struct map_ctx *mapc, TOID(struct map) map, unsigned cmd, uint64_t arg); #ifdef __cplusplus } #endif #endif /* MAP_H */ pmdk-1.13.1/src/examples/libpmemobj/map/map_rtree.c0000664000000000000000000001146114435627501020666 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016-2021, Intel Corporation */ /* * map_rtree.c -- common interface for maps */ #include #include "map_rtree.h" /* * map_rtree_check -- wrapper for rtree_map_check */ static int map_rtree_check(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct rtree_map) rtree_map; TOID_ASSIGN(rtree_map, map.oid); return rtree_map_check(pop, rtree_map); } /* * map_rtree_create -- wrapper for rtree_map_new */ static int map_rtree_create(PMEMobjpool *pop, TOID(struct map) *map, void *arg) { TOID(struct rtree_map) *rtree_map = (TOID(struct rtree_map) *)map; return rtree_map_create(pop, rtree_map, arg); } /* * map_rtree_destroy -- wrapper for rtree_map_delete */ static int map_rtree_destroy(PMEMobjpool *pop, TOID(struct map) *map) { TOID(struct rtree_map) *rtree_map = (TOID(struct rtree_map) *)map; return rtree_map_destroy(pop, rtree_map); } /* * map_rtree_insert -- wrapper for rtree_map_insert */ static int map_rtree_insert(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, PMEMoid value) { TOID(struct rtree_map) rtree_map; TOID_ASSIGN(rtree_map, map.oid); return rtree_map_insert(pop, rtree_map, (unsigned char *)&key, sizeof(key), value); } /* * map_rtree_insert_new -- wrapper for rtree_map_insert_new */ static int map_rtree_insert_new(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg) { TOID(struct rtree_map) rtree_map; TOID_ASSIGN(rtree_map, map.oid); return rtree_map_insert_new(pop, rtree_map, (unsigned char *)&key, sizeof(key), size, type_num, constructor, arg); } /* * map_rtree_remove -- wrapper for rtree_map_remove */ static PMEMoid map_rtree_remove(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct rtree_map) rtree_map; TOID_ASSIGN(rtree_map, map.oid); return rtree_map_remove(pop, rtree_map, (unsigned char *)&key, sizeof(key)); } /* * map_rtree_remove_free -- wrapper for rtree_map_remove_free */ static int map_rtree_remove_free(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct rtree_map) rtree_map; TOID_ASSIGN(rtree_map, map.oid); return rtree_map_remove_free(pop, rtree_map, (unsigned char *)&key, sizeof(key)); } /* * map_rtree_clear -- wrapper for rtree_map_clear */ static int map_rtree_clear(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct rtree_map) rtree_map; TOID_ASSIGN(rtree_map, map.oid); return rtree_map_clear(pop, rtree_map); } /* * map_rtree_get -- wrapper for rtree_map_get */ static PMEMoid map_rtree_get(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct rtree_map) rtree_map; TOID_ASSIGN(rtree_map, map.oid); return rtree_map_get(pop, rtree_map, (unsigned char *)&key, sizeof(key)); } /* * map_rtree_lookup -- wrapper for rtree_map_lookup */ static int map_rtree_lookup(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct rtree_map) rtree_map; TOID_ASSIGN(rtree_map, map.oid); return rtree_map_lookup(pop, rtree_map, (unsigned char *)&key, sizeof(key)); } struct cb_arg2 { int (*cb)(uint64_t key, PMEMoid value, void *arg); void *arg; }; /* * map_rtree_foreach_cb -- wrapper for callback */ static int map_rtree_foreach_cb(const unsigned char *key, uint64_t key_size, PMEMoid value, void *arg2) { const struct cb_arg2 *const a2 = (const struct cb_arg2 *)arg2; const uint64_t *const k2 = (uint64_t *)key; return a2->cb(*k2, value, a2->arg); } /* * map_rtree_foreach -- wrapper for rtree_map_foreach */ static int map_rtree_foreach(PMEMobjpool *pop, TOID(struct map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { struct cb_arg2 arg2 = {cb, arg}; TOID(struct rtree_map) rtree_map; TOID_ASSIGN(rtree_map, map.oid); return rtree_map_foreach(pop, rtree_map, map_rtree_foreach_cb, &arg2); } /* * map_rtree_is_empty -- wrapper for rtree_map_is_empty */ static int map_rtree_is_empty(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct rtree_map) rtree_map; TOID_ASSIGN(rtree_map, map.oid); return rtree_map_is_empty(pop, rtree_map); } /* * map_rtree_init -- recovers map state * Since there is no need for recovery for rtree, function is dummy. */ static int map_rtree_init(PMEMobjpool *pop, TOID(struct map) map) { return 0; } struct map_ops rtree_map_ops = { /* .check = */map_rtree_check, /* .create = */map_rtree_create, /* .destroy = */map_rtree_destroy, /* .init = */map_rtree_init, /* .insert = */map_rtree_insert, /* .insert_new = */map_rtree_insert_new, /* .remove = */map_rtree_remove, /* .remove_free = */map_rtree_remove_free, /* .clear = */map_rtree_clear, /* .get = */map_rtree_get, /* .lookup = */map_rtree_lookup, /* .foreach = */map_rtree_foreach, /* .is_empty = */map_rtree_is_empty, /* .count = */NULL, /* .cmd = */NULL, }; pmdk-1.13.1/src/examples/libpmemobj/map/map_skiplist.h0000664000000000000000000000050114435627501021405 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2016-2020, Intel Corporation */ /* * map_skiplist.h -- common interface for maps */ #ifndef MAP_SKIPLIST_H #define MAP_SKIPLIST_H #include "map.h" extern struct map_ops skiplist_map_ops; #define MAP_SKIPLIST (&skiplist_map_ops) #endif /* MAP_SKIPLIST_H */ pmdk-1.13.1/src/examples/libpmemobj/map/README0000664000000000000000000000303014435627501017415 0ustar rootrootThis directory contains an example application implemented using libpmemobj. The *mapcli* application is a simple CLI application which uses: * three implementations of hashmap: ** hashmap_atomic - hashmap using atomic API of libpmemobj ** hashmap_tx - hashmap using tx API of libpmemobj ** hashmap_rp - hashmap using action API of libpmemobj * four implementations of tree maps: ** ctree - Crit-Bit using tx API of libpmemobj ** btree - B-tree using tx API of libpmemobj ** rtree - Radix-tree using tx API of libpmemobj ** rbtree - red-black tree using tx API of libpmemobj Usage: $ ./mapcli ctree|btree|rtree|rbtree|hashmap_atomic|hashmap_tx|hashmap_rp [] The first argument specifies which map should be used. The file will either be created if it doesn't exist or opened if it contains a valid pool. The third argument specifies seed for RNG - the seed is utilized by three of hashmaps implementations. The application expects one of the below commands on standard input: h - help i $value - insert $value r $value - remove $value c $value - check $value, returns 0/1 n $value - insert $value random values p - print all values d - print debug info b - rebuild q - quit ** NOTE: ** Please note that some of functions may not be implemented by all types of map. In such case the application will abort with proper message. ** DEPENDENCIES: ** In order to build kv_server you need to install libuv development package. rpm-based systems : libuv-devel dpkg-based systems: libuvX-dev (where X is the API/ABI version) pmdk-1.13.1/src/examples/libpmemobj/map/map_btree.c0000664000000000000000000001031314435627501020641 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2021, Intel Corporation */ /* * map_btree.c -- common interface for maps */ #include #include #include "map_btree.h" /* * map_btree_check -- wrapper for btree_map_check */ static int map_btree_check(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct btree_map) btree_map; TOID_ASSIGN(btree_map, map.oid); return btree_map_check(pop, btree_map); } /* * map_btree_create -- wrapper for btree_map_create */ static int map_btree_create(PMEMobjpool *pop, TOID(struct map) *map, void *arg) { TOID(struct btree_map) *btree_map = (TOID(struct btree_map) *)map; return btree_map_create(pop, btree_map, arg); } /* * map_btree_destroy -- wrapper for btree_map_destroy */ static int map_btree_destroy(PMEMobjpool *pop, TOID(struct map) *map) { TOID(struct btree_map) *btree_map = (TOID(struct btree_map) *)map; return btree_map_destroy(pop, btree_map); } /* * map_btree_insert -- wrapper for btree_map_insert */ static int map_btree_insert(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, PMEMoid value) { TOID(struct btree_map) btree_map; TOID_ASSIGN(btree_map, map.oid); return btree_map_insert(pop, btree_map, key, value); } /* * map_btree_insert_new -- wrapper for btree_map_insert_new */ static int map_btree_insert_new(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg) { TOID(struct btree_map) btree_map; TOID_ASSIGN(btree_map, map.oid); return btree_map_insert_new(pop, btree_map, key, size, type_num, constructor, arg); } /* * map_btree_remove -- wrapper for btree_map_remove */ static PMEMoid map_btree_remove(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct btree_map) btree_map; TOID_ASSIGN(btree_map, map.oid); return btree_map_remove(pop, btree_map, key); } /* * map_btree_remove_free -- wrapper for btree_map_remove_free */ static int map_btree_remove_free(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct btree_map) btree_map; TOID_ASSIGN(btree_map, map.oid); return btree_map_remove_free(pop, btree_map, key); } /* * map_btree_clear -- wrapper for btree_map_clear */ static int map_btree_clear(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct btree_map) btree_map; TOID_ASSIGN(btree_map, map.oid); return btree_map_clear(pop, btree_map); } /* * map_btree_get -- wrapper for btree_map_get */ static PMEMoid map_btree_get(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct btree_map) btree_map; TOID_ASSIGN(btree_map, map.oid); return btree_map_get(pop, btree_map, key); } /* * map_btree_lookup -- wrapper for btree_map_lookup */ static int map_btree_lookup(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct btree_map) btree_map; TOID_ASSIGN(btree_map, map.oid); return btree_map_lookup(pop, btree_map, key); } /* * map_btree_foreach -- wrapper for btree_map_foreach */ static int map_btree_foreach(PMEMobjpool *pop, TOID(struct map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { TOID(struct btree_map) btree_map; TOID_ASSIGN(btree_map, map.oid); return btree_map_foreach(pop, btree_map, cb, arg); } /* * map_btree_is_empty -- wrapper for btree_map_is_empty */ static int map_btree_is_empty(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct btree_map) btree_map; TOID_ASSIGN(btree_map, map.oid); return btree_map_is_empty(pop, btree_map); } /* * map_btree_init -- recovers map state * Since there is no need for recovery for btree, function is dummy. */ static int map_btree_init(PMEMobjpool *pop, TOID(struct map) map) { return 0; } struct map_ops btree_map_ops = { /* .check = */ map_btree_check, /* .create = */ map_btree_create, /* .destroy = */ map_btree_destroy, /* .init = */ map_btree_init, /* .insert = */ map_btree_insert, /* .insert_new = */ map_btree_insert_new, /* .remove = */ map_btree_remove, /* .remove_free = */ map_btree_remove_free, /* .clear = */ map_btree_clear, /* .get = */ map_btree_get, /* .lookup = */ map_btree_lookup, /* .foreach = */ map_btree_foreach, /* .is_empty = */ map_btree_is_empty, /* .count = */ NULL, /* .cmd = */ NULL, }; pmdk-1.13.1/src/examples/libpmemobj/map/map_hashmap_atomic.c0000664000000000000000000000715514435627501022527 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2017, Intel Corporation */ /* * map_hashmap_atomic.c -- common interface for maps */ #include #include #include "map_hashmap_atomic.h" /* * map_hm_atomic_check -- wrapper for hm_atomic_check */ static int map_hm_atomic_check(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct hashmap_atomic) hashmap_atomic; TOID_ASSIGN(hashmap_atomic, map.oid); return hm_atomic_check(pop, hashmap_atomic); } /* * map_hm_atomic_count -- wrapper for hm_atomic_count */ static size_t map_hm_atomic_count(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct hashmap_atomic) hashmap_atomic; TOID_ASSIGN(hashmap_atomic, map.oid); return hm_atomic_count(pop, hashmap_atomic); } /* * map_hm_atomic_init -- wrapper for hm_atomic_init */ static int map_hm_atomic_init(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct hashmap_atomic) hashmap_atomic; TOID_ASSIGN(hashmap_atomic, map.oid); return hm_atomic_init(pop, hashmap_atomic); } /* * map_hm_atomic_new -- wrapper for hm_atomic_create */ static int map_hm_atomic_create(PMEMobjpool *pop, TOID(struct map) *map, void *arg) { TOID(struct hashmap_atomic) *hashmap_atomic = (TOID(struct hashmap_atomic) *)map; return hm_atomic_create(pop, hashmap_atomic, arg); } /* * map_hm_atomic_insert -- wrapper for hm_atomic_insert */ static int map_hm_atomic_insert(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, PMEMoid value) { TOID(struct hashmap_atomic) hashmap_atomic; TOID_ASSIGN(hashmap_atomic, map.oid); return hm_atomic_insert(pop, hashmap_atomic, key, value); } /* * map_hm_atomic_remove -- wrapper for hm_atomic_remove */ static PMEMoid map_hm_atomic_remove(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct hashmap_atomic) hashmap_atomic; TOID_ASSIGN(hashmap_atomic, map.oid); return hm_atomic_remove(pop, hashmap_atomic, key); } /* * map_hm_atomic_get -- wrapper for hm_atomic_get */ static PMEMoid map_hm_atomic_get(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct hashmap_atomic) hashmap_atomic; TOID_ASSIGN(hashmap_atomic, map.oid); return hm_atomic_get(pop, hashmap_atomic, key); } /* * map_hm_atomic_lookup -- wrapper for hm_atomic_lookup */ static int map_hm_atomic_lookup(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct hashmap_atomic) hashmap_atomic; TOID_ASSIGN(hashmap_atomic, map.oid); return hm_atomic_lookup(pop, hashmap_atomic, key); } /* * map_hm_atomic_foreach -- wrapper for hm_atomic_foreach */ static int map_hm_atomic_foreach(PMEMobjpool *pop, TOID(struct map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { TOID(struct hashmap_atomic) hashmap_atomic; TOID_ASSIGN(hashmap_atomic, map.oid); return hm_atomic_foreach(pop, hashmap_atomic, cb, arg); } /* * map_hm_atomic_cmd -- wrapper for hm_atomic_cmd */ static int map_hm_atomic_cmd(PMEMobjpool *pop, TOID(struct map) map, unsigned cmd, uint64_t arg) { TOID(struct hashmap_atomic) hashmap_atomic; TOID_ASSIGN(hashmap_atomic, map.oid); return hm_atomic_cmd(pop, hashmap_atomic, cmd, arg); } struct map_ops hashmap_atomic_ops = { /* .check = */ map_hm_atomic_check, /* .create = */ map_hm_atomic_create, /* .destroy = */ NULL, /* .init = */ map_hm_atomic_init, /* .insert = */ map_hm_atomic_insert, /* .insert_new = */ NULL, /* .remove = */ map_hm_atomic_remove, /* .remove_free = */ NULL, /* .clear = */ NULL, /* .get = */ map_hm_atomic_get, /* .lookup = */ map_hm_atomic_lookup, /* .foreach = */ map_hm_atomic_foreach, /* .is_empty = */ NULL, /* .count = */ map_hm_atomic_count, /* .cmd = */ map_hm_atomic_cmd, }; pmdk-1.13.1/src/examples/libpmemobj/map/map_hashmap_atomic.h0000664000000000000000000000065014435627501022525 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2015-2020, Intel Corporation */ /* * map_hashmap_atomic.h -- common interface for maps */ #ifndef MAP_HASHMAP_ATOMIC_H #define MAP_HASHMAP_ATOMIC_H #include "map.h" #ifdef __cplusplus extern "C" { #endif extern struct map_ops hashmap_atomic_ops; #define MAP_HASHMAP_ATOMIC (&hashmap_atomic_ops) #ifdef __cplusplus } #endif #endif /* MAP_HASHMAP_ATOMIC_H */ pmdk-1.13.1/src/examples/libpmemobj/map/data_store.vcxproj.filters0000664000000000000000000000064714435627501023761 0ustar rootroot {5b0d30df-c510-411d-8529-c3695816137d} Source Files pmdk-1.13.1/src/examples/libpmemobj/map/map_hashmap_tx.h0000664000000000000000000000061414435627501021704 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2015-2020, Intel Corporation */ /* * map_hashmap_tx.h -- common interface for maps */ #ifndef MAP_HASHMAP_TX_H #define MAP_HASHMAP_TX_H #include "map.h" #ifdef __cplusplus extern "C" { #endif extern struct map_ops hashmap_tx_ops; #define MAP_HASHMAP_TX (&hashmap_tx_ops) #ifdef __cplusplus } #endif #endif /* MAP_HASHMAP_TX_H */ pmdk-1.13.1/src/examples/libpmemobj/map/map_rbtree.h0000664000000000000000000000057014435627501021034 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2015-2020, Intel Corporation */ /* * map_rbtree.h -- common interface for maps */ #ifndef MAP_RBTREE_H #define MAP_RBTREE_H #include "map.h" #ifdef __cplusplus extern "C" { #endif extern struct map_ops rbtree_map_ops; #define MAP_RBTREE (&rbtree_map_ops) #ifdef __cplusplus } #endif #endif /* MAP_RBTREE_H */ pmdk-1.13.1/src/examples/libpmemobj/map/map_btree.h0000664000000000000000000000056114435627501020652 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2015-2020, Intel Corporation */ /* * map_ctree.h -- common interface for maps */ #ifndef MAP_BTREE_H #define MAP_BTREE_H #include "map.h" #ifdef __cplusplus extern "C" { #endif extern struct map_ops btree_map_ops; #define MAP_BTREE (&btree_map_ops) #ifdef __cplusplus } #endif #endif /* MAP_BTREE_H */ pmdk-1.13.1/src/examples/libpmemobj/map/mapcli.c0000664000000000000000000001434114435627501020155 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2021, Intel Corporation */ #include #include #include #include #include #include #include #include #include "map.h" #include "map_ctree.h" #include "map_btree.h" #include "map_rtree.h" #include "map_rbtree.h" #include "map_hashmap_atomic.h" #include "map_hashmap_tx.h" #include "map_hashmap_rp.h" #include "map_skiplist.h" #include "hashmap/hashmap.h" #define PM_HASHSET_POOL_SIZE (160 * 1024 * 1024) POBJ_LAYOUT_BEGIN(map); POBJ_LAYOUT_ROOT(map, struct root); POBJ_LAYOUT_END(map); struct root { TOID(struct map) map; }; static PMEMobjpool *pop; static struct map_ctx *mapc; static TOID(struct root) root; static TOID(struct map) map; /* * str_insert -- hs_insert wrapper which works on strings */ static void str_insert(const char *str) { uint64_t key; if (sscanf(str, "%" PRIu64, &key) > 0) map_insert(mapc, map, key, OID_NULL); else fprintf(stderr, "insert: invalid syntax\n"); } /* * str_remove -- hs_remove wrapper which works on strings */ static void str_remove(const char *str) { uint64_t key; if (sscanf(str, "%" PRIu64, &key) > 0) { int l = map_lookup(mapc, map, key); if (l) map_remove(mapc, map, key); else fprintf(stderr, "no such value\n"); } else fprintf(stderr, "remove: invalid syntax\n"); } /* * str_check -- hs_check wrapper which works on strings */ static void str_check(const char *str) { uint64_t key; if (sscanf(str, "%" PRIu64, &key) > 0) { int r = map_lookup(mapc, map, key); printf("%d\n", r); } else { fprintf(stderr, "check: invalid syntax\n"); } } /* * str_insert_random -- inserts specified (as string) number of random numbers */ static void str_insert_random(const char *str) { uint64_t val; if (sscanf(str, "%" PRIu64, &val) > 0) for (uint64_t i = 0; i < val; ) { uint64_t r = ((uint64_t)rand()) << 32 | rand(); int ret = map_insert(mapc, map, r, OID_NULL); if (ret < 0) break; if (ret == 0) i += 1; } else fprintf(stderr, "random insert: invalid syntax\n"); } /* * rebuild -- rebuilds hashmap and measures execution time */ static void rebuild(void) { printf("rebuild "); fflush(stdout); time_t t1 = time(NULL); map_cmd(mapc, map, HASHMAP_CMD_REBUILD, 0); printf("%" PRIu64"s\n", (uint64_t)(time(NULL) - t1)); } /* * str_rebuild -- hs_rebuild wrapper which executes specified number of times */ static void str_rebuild(const char *str) { uint64_t val; if (sscanf(str, "%" PRIu64, &val) > 0) { for (uint64_t i = 0; i < val; ++i) { printf("%2" PRIu64 " ", i); rebuild(); } } else { rebuild(); } } static void help(void) { printf("h - help\n"); printf("i $value - insert $value\n"); printf("r $value - remove $value\n"); printf("c $value - check $value, returns 0/1\n"); printf("n $value - insert $value random values\n"); printf("p - print all values\n"); printf("d - print debug info\n"); printf("b [$value] - rebuild $value (default: 1) times\n"); printf("q - quit\n"); } static void unknown_command(const char *str) { fprintf(stderr, "unknown command '%c', use 'h' for help\n", str[0]); } static int hashmap_print(uint64_t key, PMEMoid value, void *arg) { printf("%" PRIu64 " ", key); return 0; } static void print_all(void) { if (mapc->ops->count) printf("count: %zu\n", map_count(mapc, map)); map_foreach(mapc, map, hashmap_print, NULL); printf("\n"); } #define INPUT_BUF_LEN 1000 int main(int argc, char *argv[]) { if (argc < 3 || argc > 4) { printf("usage: %s " "hashmap_tx|hashmap_atomic|hashmap_rp|" "ctree|btree|rtree|rbtree|skiplist" " file-name []\n", argv[0]); return 1; } const struct map_ops *ops = NULL; const char *path = argv[2]; const char *type = argv[1]; if (strcmp(type, "hashmap_tx") == 0) { ops = MAP_HASHMAP_TX; } else if (strcmp(type, "hashmap_atomic") == 0) { ops = MAP_HASHMAP_ATOMIC; } else if (strcmp(type, "hashmap_rp") == 0) { ops = MAP_HASHMAP_RP; } else if (strcmp(type, "ctree") == 0) { ops = MAP_CTREE; } else if (strcmp(type, "btree") == 0) { ops = MAP_BTREE; } else if (strcmp(type, "rtree") == 0) { ops = MAP_RTREE; } else if (strcmp(type, "rbtree") == 0) { ops = MAP_RBTREE; } else if (strcmp(type, "skiplist") == 0) { ops = MAP_SKIPLIST; } else { fprintf(stderr, "invalid container type -- '%s'\n", type); return 1; } struct hashmap_args args; if (argc > 3) args.seed = atoi(argv[3]); else args.seed = (uint32_t)time(NULL); srand(args.seed); if (file_exists(path) != 0) { pop = pmemobj_create(path, POBJ_LAYOUT_NAME(map), PM_HASHSET_POOL_SIZE, CREATE_MODE_RW); if (pop == NULL) { fprintf(stderr, "failed to create pool: %s\n", pmemobj_errormsg()); return 1; } mapc = map_ctx_init(ops, pop); if (!mapc) { pmemobj_close(pop); perror("map_ctx_init"); return 1; } root = POBJ_ROOT(pop, struct root); printf("seed: %u\n", args.seed); } else { pop = pmemobj_open(path, POBJ_LAYOUT_NAME(map)); if (pop == NULL) { fprintf(stderr, "failed to open pool: %s\n", pmemobj_errormsg()); return 1; } mapc = map_ctx_init(ops, pop); if (!mapc) { pmemobj_close(pop); perror("map_ctx_init"); return 1; } root = POBJ_ROOT(pop, struct root); map = D_RO(root)->map; } if (TOID_IS_NULL(map)) { map_create(mapc, &D_RW(root)->map, &args); map = D_RO(root)->map; } /* Manual recovery */ if (ops) { map_init(mapc, D_RW(root)->map); } char buf[INPUT_BUF_LEN]; if (isatty(fileno(stdout))) printf("Type 'h' for help\n$ "); while (fgets(buf, sizeof(buf), stdin)) { if (buf[0] == 0 || buf[0] == '\n') continue; switch (buf[0]) { case 'i': str_insert(buf + 1); break; case 'r': str_remove(buf + 1); break; case 'c': str_check(buf + 1); break; case 'n': str_insert_random(buf + 1); break; case 'p': print_all(); break; case 'd': map_cmd(mapc, map, HASHMAP_CMD_DEBUG, (uint64_t)stdout); break; case 'b': str_rebuild(buf + 1); break; case 'q': fclose(stdin); break; case 'h': help(); break; default: unknown_command(buf); break; } if (isatty(fileno(stdout))) printf("$ "); } map_ctx_free(mapc); pmemobj_close(pop); return 0; } pmdk-1.13.1/src/examples/libpmemobj/map/kv_server.c0000664000000000000000000002416214435627501020720 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2018, Intel Corporation */ /* * kv_server.c -- persistent tcp key-value store server */ #include #include #include #include #include #include #include "libpmemobj.h" #include "map.h" #include "map_ctree.h" #include "map_btree.h" #include "map_rtree.h" #include "map_rbtree.h" #include "map_hashmap_atomic.h" #include "map_hashmap_tx.h" #include "map_hashmap_rp.h" #include "map_skiplist.h" #include "kv_protocol.h" #define COUNT_OF(x) (sizeof(x) / sizeof(0[x])) #define COMPILE_ERROR_ON(cond) ((void)sizeof(char[(cond) ? -1 : 1])) POBJ_LAYOUT_BEGIN(kv_server); POBJ_LAYOUT_ROOT(kv_server, struct root); POBJ_LAYOUT_TOID(kv_server, struct map_value); POBJ_LAYOUT_TOID(kv_server, uint64_t); POBJ_LAYOUT_END(kv_server); struct map_value { uint64_t len; char buf[]; }; struct root { TOID(struct map) map; }; static struct map_ctx *mapc; static PMEMobjpool *pop; static TOID(struct map) map; static uv_tcp_t server; static uv_loop_t *loop; typedef int (*msg_handler)(uv_stream_t *client, const char *msg, size_t len); struct write_req { uv_write_t req; uv_buf_t buf; }; struct client_data { char *buf; /* current message, always NULL terminated */ size_t buf_len; /* sizeof(buf) */ size_t len; /* actual length of the message (while parsing) */ }; /* * djb2_hash -- string hashing function by Dan Bernstein */ static uint32_t djb2_hash(const char *str) { uint32_t hash = 5381; int c; while ((c = *str++)) hash = ((hash << 5) + hash) + c; return hash; } /* * write_done_cb -- callback after message write completes */ static void write_done_cb(uv_write_t *req, int status) { struct write_req *wr = (struct write_req *)req; free(wr); if (status == -1) { printf("response failed"); } } /* * client_close_cb -- callback after client tcp connection closes */ static void client_close_cb(uv_handle_t *handle) { struct client_data *d = handle->data; free(d->buf); free(handle->data); free(handle); } /* * response_write -- response writing helper */ static void response_write(uv_stream_t *client, char *resp, size_t len) { struct write_req *wr = malloc(sizeof(struct write_req)); assert(wr != NULL); wr->buf = uv_buf_init(resp, len); uv_write(&wr->req, client, &wr->buf, 1, write_done_cb); } /* * response_msg -- predefined message writing helper */ static void response_msg(uv_stream_t *client, enum resp_messages msg) { response_write(client, (char *)resp_msg[msg], strlen(resp_msg[msg])); } /* * cmsg_insert_handler -- handler of INSERT client message */ static int cmsg_insert_handler(uv_stream_t *client, const char *msg, size_t len) { int result = 0; TX_BEGIN(pop) { /* * For simplicity sake the length of the value buffer is just * a length of the message. */ TOID(struct map_value) val = TX_ZALLOC(struct map_value, sizeof(struct map_value) + len); char key[MAX_KEY_LEN]; int ret = sscanf(msg, "INSERT %254s %s\n", key, D_RW(val)->buf); assert(ret == 2); D_RW(val)->len = len; /* properly terminate the value */ D_RW(val)->buf[strlen(D_RO(val)->buf)] = '\n'; map_insert(mapc, map, djb2_hash(key), val.oid); } TX_ONABORT { result = 1; } TX_END response_msg(client, result); return 0; } /* * cmsg_remove_handler -- handler of REMOVE client message */ static int cmsg_remove_handler(uv_stream_t *client, const char *msg, size_t len) { char key[MAX_KEY_LEN] = {0}; /* check if the constant used in sscanf() below has the correct value */ COMPILE_ERROR_ON(MAX_KEY_LEN - 1 != 254); int ret = sscanf(msg, "REMOVE %254s\n", key); assert(ret == 1); int result = map_remove_free(mapc, map, djb2_hash(key)); response_msg(client, result); return 0; } /* * cmsg_get_handler -- handler of GET client message */ static int cmsg_get_handler(uv_stream_t *client, const char *msg, size_t len) { char key[MAX_KEY_LEN]; /* check if the constant used in sscanf() below has the correct value */ COMPILE_ERROR_ON(MAX_KEY_LEN - 1 != 254); int ret = sscanf(msg, "GET %254s\n", key); assert(ret == 1); TOID(struct map_value) value; TOID_ASSIGN(value, map_get(mapc, map, djb2_hash(key))); if (TOID_IS_NULL(value)) { response_msg(client, RESP_MSG_NULL); } else { response_write(client, D_RW(value)->buf, D_RO(value)->len); } return 0; } /* * cmsg_bye_handler -- handler of BYE client message */ static int cmsg_bye_handler(uv_stream_t *client, const char *msg, size_t len) { uv_close((uv_handle_t *)client, client_close_cb); return 0; } /* * cmsg_bye_handler -- handler of KILL client message */ static int cmsg_kill_handler(uv_stream_t *client, const char *msg, size_t len) { uv_close((uv_handle_t *)client, client_close_cb); uv_close((uv_handle_t *)&server, NULL); return 0; } /* kv protocol implementation */ static msg_handler protocol_impl[MAX_CMSG] = { cmsg_insert_handler, cmsg_remove_handler, cmsg_get_handler, cmsg_bye_handler, cmsg_kill_handler }; /* * cmsg_handle -- handles current client message */ static int cmsg_handle(uv_stream_t *client, struct client_data *data) { int ret = 0; int i; for (i = 0; i < MAX_CMSG; ++i) if (strncmp(kv_cmsg_token[i], data->buf, strlen(kv_cmsg_token[i])) == 0) break; if (i == MAX_CMSG) { response_msg(client, RESP_MSG_UNKNOWN); } else { ret = protocol_impl[i](client, data->buf, data->len); } data->len = 0; /* reset the message length */ return ret; } /* * cmsg_handle_stream -- handle incoming tcp stream from clients */ static int cmsg_handle_stream(uv_stream_t *client, struct client_data *data, const char *buf, ssize_t nread) { char *last; int ret; size_t len; /* * A single read operation can contain zero or more operations, so this * has to be handled appropriately. Client messages are terminated by * newline character. */ while ((last = memchr(buf, '\n', nread)) != NULL) { len = last - buf + 1; nread -= len; assert(data->len + len <= data->buf_len); memcpy(data->buf + data->len, buf, len); data->len += len; if ((ret = cmsg_handle(client, data)) != 0) return ret; buf = last + 1; } if (nread != 0) { memcpy(data->buf + data->len, buf, nread); data->len += nread; } return 0; } static uv_buf_t msg_buf = {0}; /* * get_read_buf_cb -- returns buffer for incoming client message */ static void get_read_buf_cb(uv_handle_t *handle, size_t size, uv_buf_t *buf) { buf->base = msg_buf.base; buf->len = msg_buf.len; } /* * read_cb -- async tcp read from clients */ static void read_cb(uv_stream_t *client, ssize_t nread, const uv_buf_t *buf) { if (nread <= 0) { printf("client connection closed\n"); uv_close((uv_handle_t *)client, client_close_cb); return; } struct client_data *d = client->data; if (d->buf_len < (d->len + nread + 1)) { char *cbuf = realloc(d->buf, d->buf_len + nread + 1); assert(cbuf != NULL); /* zero only the new memory */ memset(cbuf + d->buf_len, 0, nread + 1); d->buf_len += nread + 1; d->buf = cbuf; } if (cmsg_handle_stream(client, client->data, buf->base, nread)) { printf("client disconnect\n"); uv_close((uv_handle_t *)client, client_close_cb); } } /* * connection_cb -- async incoming client request */ static void connection_cb(uv_stream_t *_server, int status) { if (status != 0) { printf("client connect error\n"); return; } printf("new client\n"); uv_tcp_t *client = malloc(sizeof(uv_tcp_t)); assert(client != NULL); client->data = calloc(1, sizeof(struct client_data)); assert(client->data != NULL); uv_tcp_init(loop, client); if (uv_accept(_server, (uv_stream_t *)client) == 0) { uv_read_start((uv_stream_t *)client, get_read_buf_cb, read_cb); } else { uv_close((uv_handle_t *)client, client_close_cb); } } static const struct { struct map_ops *ops; const char *name; } maps[] = { {MAP_HASHMAP_TX, "hashmap_tx"}, {MAP_HASHMAP_ATOMIC, "hashmap_atomic"}, {MAP_HASHMAP_RP, "hashmap_rp"}, {MAP_CTREE, "ctree"}, {MAP_BTREE, "btree"}, {MAP_RTREE, "rtree"}, {MAP_RBTREE, "rbtree"}, {MAP_SKIPLIST, "skiplist"} }; /* * get_map_ops_by_string -- parse the type string and return the associated ops */ static const struct map_ops * get_map_ops_by_string(const char *type) { for (int i = 0; i < COUNT_OF(maps); ++i) if (strcmp(maps[i].name, type) == 0) return maps[i].ops; return NULL; } #define KV_SIZE (PMEMOBJ_MIN_POOL) #define MAX_READ_LEN (64 * 1024) /* 64 kilobytes */ int main(int argc, char *argv[]) { if (argc < 4) { printf("usage: %s hashmap_tx|hashmap_atomic|hashmap_rp|" "ctree|btree|rtree|rbtree|skiplist file-name port\n", argv[0]); return 1; } const char *path = argv[2]; const char *type = argv[1]; int port = atoi(argv[3]); /* use only a single buffer for all incoming data */ void *read_buf = malloc(MAX_READ_LEN); assert(read_buf != NULL); msg_buf = uv_buf_init(read_buf, MAX_READ_LEN); if (access(path, F_OK) != 0) { pop = pmemobj_create(path, POBJ_LAYOUT_NAME(kv_server), KV_SIZE, 0666); if (pop == NULL) { fprintf(stderr, "failed to create pool: %s\n", pmemobj_errormsg()); return 1; } } else { pop = pmemobj_open(path, POBJ_LAYOUT_NAME(kv_server)); if (pop == NULL) { fprintf(stderr, "failed to open pool: %s\n", pmemobj_errormsg()); return 1; } } /* map context initialization */ mapc = map_ctx_init(get_map_ops_by_string(type), pop); if (!mapc) { pmemobj_close(pop); fprintf(stderr, "map_ctx_init failed (wrong type?)\n"); return 1; } /* initialize the actual map */ TOID(struct root) root = POBJ_ROOT(pop, struct root); if (TOID_IS_NULL(D_RO(root)->map)) { /* create new if it doesn't exist (a fresh pool) */ map_create(mapc, &D_RW(root)->map, NULL); } map = D_RO(root)->map; loop = uv_default_loop(); /* tcp server initialization */ uv_tcp_init(loop, &server); struct sockaddr_in bind_addr; uv_ip4_addr("0.0.0.0", port, &bind_addr); int ret = uv_tcp_bind(&server, (const struct sockaddr *)&bind_addr, 0); assert(ret == 0); ret = uv_listen((uv_stream_t *)&server, SOMAXCONN, connection_cb); assert(ret == 0); ret = uv_run(loop, UV_RUN_DEFAULT); assert(ret == 0); /* no more events in the loop, release resources and quit */ uv_loop_delete(loop); map_ctx_free(mapc); pmemobj_close(pop); free(read_buf); return 0; } pmdk-1.13.1/src/examples/libpmemobj/map/libmap.vcxproj0000664000000000000000000001347514435627501021434 0ustar rootroot Debug x64 Release x64 {49A7CC5A-D5E7-4A07-917F-C6918B982BE8} $(ProjectName) pmemobj 10.0.22000.0 ..\..\..\LongPath.manifest {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} {f5e2f6c4-19ba-497a-b754-232e469be647} {F5E2F6C4-19BA-497A-B754-232E4666E647} {d93a2683-6d99-4f18-b378-91195d23e007} {3799ba67-3c4f-4ae0-85dc-5baaea01a180} {79d37ffe-ff76-44b3-bb27-3dcaeff2ebe9} {be18f227-a9f0-4b38-b689-4e2f9f09ca5f} {17a4b817-68b1-4719-a9ef-bd8fab747de6} {3ed56e55-84a6-422c-a8d4-a8439fb8f245} StaticLibrary true v143 StaticLibrary false v143 true $(solutionDir)include;$(solutionDir)..\include;$(ProjectDir)..\..\;$(ProjectDir)..\list_map;$(ProjectDir)..\hashmap;$(ProjectDir)..\tree_map;$(IncludePath);$(WindowsSDK_IncludePath);$(SolutionDir)..\$(Platform)\$(Configuration);$(ProjectDir) CompileAsCpp {f5e2f6c4-19ba-497a-b754-232e469be647} {F5E2F6C4-19BA-497A-B754-232E4666E647} {d93a2683-6d99-4f18-b378-91195d23e007} {3799ba67-3c4f-4ae0-85dc-5baaea01a180} {79d37ffe-ff76-44b3-bb27-3dcaeff2ebe9} {be18f227-a9f0-4b38-b689-4e2f9f09ca5f} {17a4b817-68b1-4719-a9ef-bd8fab747de6} {3ed56e55-84a6-422c-a8d4-a8439fb8f245} pmdk-1.13.1/src/examples/libpmemobj/map/map_skiplist.c0000664000000000000000000001115114435627501021403 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016-2021, Intel Corporation */ /* * map_skiplist.c -- common interface for maps */ #include #include #include "map_skiplist.h" /* * map_skiplist_check -- wrapper for skiplist_map_check */ static int map_skiplist_check(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct skiplist_map_node) skiplist_map; TOID_ASSIGN(skiplist_map, map.oid); return skiplist_map_check(pop, skiplist_map); } /* * map_skiplist_create -- wrapper for skiplist_map_new */ static int map_skiplist_create(PMEMobjpool *pop, TOID(struct map) *map, void *arg) { TOID(struct skiplist_map_node) *skiplist_map = (TOID(struct skiplist_map_node) *)map; return skiplist_map_create(pop, skiplist_map, arg); } /* * map_skiplist_destroy -- wrapper for skiplist_map_delete */ static int map_skiplist_destroy(PMEMobjpool *pop, TOID(struct map) *map) { TOID(struct skiplist_map_node) *skiplist_map = (TOID(struct skiplist_map_node) *)map; return skiplist_map_destroy(pop, skiplist_map); } /* * map_skiplist_insert -- wrapper for skiplist_map_insert */ static int map_skiplist_insert(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, PMEMoid value) { TOID(struct skiplist_map_node) skiplist_map; TOID_ASSIGN(skiplist_map, map.oid); return skiplist_map_insert(pop, skiplist_map, key, value); } /* * map_skiplist_insert_new -- wrapper for skiplist_map_insert_new */ static int map_skiplist_insert_new(PMEMobjpool *pop, TOID(struct map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg) { TOID(struct skiplist_map_node) skiplist_map; TOID_ASSIGN(skiplist_map, map.oid); return skiplist_map_insert_new(pop, skiplist_map, key, size, type_num, constructor, arg); } /* * map_skiplist_remove -- wrapper for skiplist_map_remove */ static PMEMoid map_skiplist_remove(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct skiplist_map_node) skiplist_map; TOID_ASSIGN(skiplist_map, map.oid); return skiplist_map_remove(pop, skiplist_map, key); } /* * map_skiplist_remove_free -- wrapper for skiplist_map_remove_free */ static int map_skiplist_remove_free(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct skiplist_map_node) skiplist_map; TOID_ASSIGN(skiplist_map, map.oid); return skiplist_map_remove_free(pop, skiplist_map, key); } /* * map_skiplist_clear -- wrapper for skiplist_map_clear */ static int map_skiplist_clear(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct skiplist_map_node) skiplist_map; TOID_ASSIGN(skiplist_map, map.oid); return skiplist_map_clear(pop, skiplist_map); } /* * map_skiplist_get -- wrapper for skiplist_map_get */ static PMEMoid map_skiplist_get(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct skiplist_map_node) skiplist_map; TOID_ASSIGN(skiplist_map, map.oid); return skiplist_map_get(pop, skiplist_map, key); } /* * map_skiplist_lookup -- wrapper for skiplist_map_lookup */ static int map_skiplist_lookup(PMEMobjpool *pop, TOID(struct map) map, uint64_t key) { TOID(struct skiplist_map_node) skiplist_map; TOID_ASSIGN(skiplist_map, map.oid); return skiplist_map_lookup(pop, skiplist_map, key); } /* * map_skiplist_foreach -- wrapper for skiplist_map_foreach */ static int map_skiplist_foreach(PMEMobjpool *pop, TOID(struct map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { TOID(struct skiplist_map_node) skiplist_map; TOID_ASSIGN(skiplist_map, map.oid); return skiplist_map_foreach(pop, skiplist_map, cb, arg); } /* * map_skiplist_is_empty -- wrapper for skiplist_map_is_empty */ static int map_skiplist_is_empty(PMEMobjpool *pop, TOID(struct map) map) { TOID(struct skiplist_map_node) skiplist_map; TOID_ASSIGN(skiplist_map, map.oid); return skiplist_map_is_empty(pop, skiplist_map); } /* * map_skiplist_init -- recovers map state * Since there is no need for recovery for skiplist, function is dummy. */ static int map_skiplist_init(PMEMobjpool *pop, TOID(struct map) map) { return 0; } struct map_ops skiplist_map_ops = { /* .check = */ map_skiplist_check, /* .create = */ map_skiplist_create, /* .destroy = */ map_skiplist_destroy, /* .init = */ map_skiplist_init, /* .insert = */ map_skiplist_insert, /* .insert_new = */ map_skiplist_insert_new, /* .remove = */ map_skiplist_remove, /* .remove_free = */ map_skiplist_remove_free, /* .clear = */ map_skiplist_clear, /* .get = */ map_skiplist_get, /* .lookup = */ map_skiplist_lookup, /* .foreach = */ map_skiplist_foreach, /* .is_empty = */ map_skiplist_is_empty, /* .count = */ NULL, /* .cmd = */ NULL, }; pmdk-1.13.1/src/examples/libpmemobj/map/mapcli.vcxproj.filters0000664000000000000000000000064314435627501023075 0ustar rootroot {eec7f2ae-38c2-47d2-90b9-3ef11235f61e} Source Files pmdk-1.13.1/src/examples/libpmemobj/map/map.c0000664000000000000000000001015014435627501017457 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2017, Intel Corporation */ /* * map.c -- common interface for maps */ #include #include #include #include "map.h" #define ABORT_NOT_IMPLEMENTED(mapc, func)\ if ((mapc)->ops->func == NULL) {\ fprintf(stderr, "error: '%s'"\ " function not implemented\n", #func);\ exit(1);\ } /* * map_ctx_init -- initialize map context */ struct map_ctx * map_ctx_init(const struct map_ops *ops, PMEMobjpool *pop) { if (!ops) return NULL; struct map_ctx *mapc = (struct map_ctx *)calloc(1, sizeof(*mapc)); if (!mapc) return NULL; mapc->ops = ops; mapc->pop = pop; return mapc; } /* * map_ctx_free -- free map context */ void map_ctx_free(struct map_ctx *mapc) { free(mapc); } /* * map_create -- create new map */ int map_create(struct map_ctx *mapc, TOID(struct map) *map, void *arg) { ABORT_NOT_IMPLEMENTED(mapc, create); return mapc->ops->create(mapc->pop, map, arg); } /* * map_destroy -- free the map */ int map_destroy(struct map_ctx *mapc, TOID(struct map) *map) { ABORT_NOT_IMPLEMENTED(mapc, destroy); return mapc->ops->destroy(mapc->pop, map); } /* * map_init -- initialize map */ int map_init(struct map_ctx *mapc, TOID(struct map) map) { ABORT_NOT_IMPLEMENTED(mapc, init); return mapc->ops->init(mapc->pop, map); } /* * map_check -- check if persistent object is a valid map object */ int map_check(struct map_ctx *mapc, TOID(struct map) map) { ABORT_NOT_IMPLEMENTED(mapc, check); return mapc->ops->check(mapc->pop, map); } /* * map_insert -- insert key value pair */ int map_insert(struct map_ctx *mapc, TOID(struct map) map, uint64_t key, PMEMoid value) { ABORT_NOT_IMPLEMENTED(mapc, insert); return mapc->ops->insert(mapc->pop, map, key, value); } /* * map_insert_new -- allocate and insert key value pair */ int map_insert_new(struct map_ctx *mapc, TOID(struct map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg) { ABORT_NOT_IMPLEMENTED(mapc, insert_new); return mapc->ops->insert_new(mapc->pop, map, key, size, type_num, constructor, arg); } /* * map_remove -- remove key value pair */ PMEMoid map_remove(struct map_ctx *mapc, TOID(struct map) map, uint64_t key) { ABORT_NOT_IMPLEMENTED(mapc, remove); return mapc->ops->remove(mapc->pop, map, key); } /* * map_remove_free -- remove and free key value pair */ int map_remove_free(struct map_ctx *mapc, TOID(struct map) map, uint64_t key) { ABORT_NOT_IMPLEMENTED(mapc, remove_free); return mapc->ops->remove_free(mapc->pop, map, key); } /* * map_clear -- remove all key value pairs from map */ int map_clear(struct map_ctx *mapc, TOID(struct map) map) { ABORT_NOT_IMPLEMENTED(mapc, clear); return mapc->ops->clear(mapc->pop, map); } /* * map_get -- get value of specified key */ PMEMoid map_get(struct map_ctx *mapc, TOID(struct map) map, uint64_t key) { ABORT_NOT_IMPLEMENTED(mapc, get); return mapc->ops->get(mapc->pop, map, key); } /* * map_lookup -- check if specified key exists in map */ int map_lookup(struct map_ctx *mapc, TOID(struct map) map, uint64_t key) { ABORT_NOT_IMPLEMENTED(mapc, lookup); return mapc->ops->lookup(mapc->pop, map, key); } /* * map_foreach -- iterate through all key value pairs in a map */ int map_foreach(struct map_ctx *mapc, TOID(struct map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { ABORT_NOT_IMPLEMENTED(mapc, foreach); return mapc->ops->foreach(mapc->pop, map, cb, arg); } /* * map_is_empty -- check if map is empty */ int map_is_empty(struct map_ctx *mapc, TOID(struct map) map) { ABORT_NOT_IMPLEMENTED(mapc, is_empty); return mapc->ops->is_empty(mapc->pop, map); } /* * map_count -- get number of key value pairs in map */ size_t map_count(struct map_ctx *mapc, TOID(struct map) map) { ABORT_NOT_IMPLEMENTED(mapc, count); return mapc->ops->count(mapc->pop, map); } /* * map_cmd -- execute command specific for map type */ int map_cmd(struct map_ctx *mapc, TOID(struct map) map, unsigned cmd, uint64_t arg) { ABORT_NOT_IMPLEMENTED(mapc, cmd); return mapc->ops->cmd(mapc->pop, map, cmd, arg); } pmdk-1.13.1/src/examples/libpmemobj/map/data_store.vcxproj0000664000000000000000000000600514435627501022304 0ustar rootroot Debug x64 Release x64 {5B2B9C0D-1B6D-4357-8307-6DE1EE0A41A3} pmemobj 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} {49a7cc5a-d5e7-4a07-917f-c6918b982be8} Application true v143 Application false v143 true ..\..\..\LongPath.manifest CompileAsCpp libpmem.lib;libpmemobj.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies) pmdk-1.13.1/src/examples/libpmemobj/README0000664000000000000000000000117214435627501016645 0ustar rootrootPersistent Memory Development Kit This is examples/libpmemobj/README. This directory contains examples for libpmemobj, the library providing a transactional object store for pmem. Some of these examples are explained in more detail here: https://pmem.io/pmdk/libpmemobj manpage.c and setjmp.c are the examples used in the libpmemobj man page. To build these examples: make These examples can be built against an installed system using: make LIBDIR=/usr/lib INCDIR=/usr/include If you're looking for documentation to get you started using PMDK, start here: https://pmem.io/pmdk and follow the links to examples and man pages. pmdk-1.13.1/src/examples/libpmemobj/setjmp.c0000664000000000000000000000327314435627501017437 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016-2019, Intel Corporation */ /* * setjmp.c -- example illustrating an issue with indeterminate value * of non-volatile automatic variables after transaction abort. * See libpmemobj(7) for details. * * NOTE: To observe the problem (likely segfault on a second call to free()), * the example program should be compiled with optimizations enabled (-O2). */ #include #include #include /* name of our layout in the pool */ #define LAYOUT_NAME "setjmp_example" int main(int argc, const char *argv[]) { const char path[] = "/pmem-fs/myfile"; PMEMobjpool *pop; /* create the pmemobj pool */ pop = pmemobj_create(path, LAYOUT_NAME, PMEMOBJ_MIN_POOL, 0666); if (pop == NULL) { perror(path); exit(1); } /* initialize pointer variables with invalid addresses */ int *bad_example_1 = (int *)0xBAADF00D; int *bad_example_2 = (int *)0xBAADF00D; int *bad_example_3 = (int *)0xBAADF00D; int *volatile good_example = (int *)0xBAADF00D; TX_BEGIN(pop) { bad_example_1 = malloc(sizeof(int)); bad_example_2 = malloc(sizeof(int)); bad_example_3 = malloc(sizeof(int)); good_example = malloc(sizeof(int)); /* manual or library abort called here */ pmemobj_tx_abort(EINVAL); } TX_ONCOMMIT { /* * This section is longjmp-safe */ } TX_ONABORT { /* * This section is not longjmp-safe */ free(good_example); /* OK */ free(bad_example_1); /* undefined behavior */ } TX_FINALLY { /* * This section is not longjmp-safe on transaction abort only */ free(bad_example_2); /* undefined behavior */ } TX_END free(bad_example_3); /* undefined behavior */ pmemobj_close(pop); return 0; } pmdk-1.13.1/src/examples/libpmemobj/list_map/0000775000000000000000000000000014435627501017574 5ustar rootrootpmdk-1.13.1/src/examples/libpmemobj/list_map/Makefile0000664000000000000000000000040614435627501021234 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2016-2017, Intel Corporation # # examples/libpmemobj/list_map/Makefile -- build the list map example # LIBRARIES = skiplist_map LIBS = -lpmemobj include ../../Makefile.inc libskiplist_map.o: skiplist_map.o pmdk-1.13.1/src/examples/libpmemobj/list_map/list_map.vcxproj0000664000000000000000000000523614435627501023027 0ustar rootroot Debug x64 Release x64 {3799BA67-3C4F-4AE0-85DC-5BAAEA01A180} pmemobj 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} StaticLibrary true v143 StaticLibrary false v143 true ..\..\..\LongPath.manifest CompileAsCpp pmdk-1.13.1/src/examples/libpmemobj/list_map/skiplist_map.c0000664000000000000000000001547714435627501022455 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016, Intel Corporation */ /* * skiplist_map.c -- Skiplist implementation */ #include #include #include #include #include "skiplist_map.h" #define SKIPLIST_LEVELS_NUM 4 #define NULL_NODE TOID_NULL(struct skiplist_map_node) struct skiplist_map_entry { uint64_t key; PMEMoid value; }; struct skiplist_map_node { TOID(struct skiplist_map_node) next[SKIPLIST_LEVELS_NUM]; struct skiplist_map_entry entry; }; /* * skiplist_map_create -- allocates a new skiplist instance */ int skiplist_map_create(PMEMobjpool *pop, TOID(struct skiplist_map_node) *map, void *arg) { int ret = 0; TX_BEGIN(pop) { pmemobj_tx_add_range_direct(map, sizeof(*map)); *map = TX_ZNEW(struct skiplist_map_node); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * skiplist_map_clear -- removes all elements from the map */ int skiplist_map_clear(PMEMobjpool *pop, TOID(struct skiplist_map_node) map) { while (!TOID_EQUALS(D_RO(map)->next[0], NULL_NODE)) { TOID(struct skiplist_map_node) next = D_RO(map)->next[0]; skiplist_map_remove_free(pop, map, D_RO(next)->entry.key); } return 0; } /* * skiplist_map_destroy -- cleanups and frees skiplist instance */ int skiplist_map_destroy(PMEMobjpool *pop, TOID(struct skiplist_map_node) *map) { int ret = 0; TX_BEGIN(pop) { skiplist_map_clear(pop, *map); pmemobj_tx_add_range_direct(map, sizeof(*map)); TX_FREE(*map); *map = TOID_NULL(struct skiplist_map_node); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * skiplist_map_insert_new -- allocates a new object and inserts it into * the list */ int skiplist_map_insert_new(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg) { int ret = 0; TX_BEGIN(pop) { PMEMoid n = pmemobj_tx_alloc(size, type_num); constructor(pop, pmemobj_direct(n), arg); skiplist_map_insert(pop, map, key, n); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * skiplist_map_insert_node -- (internal) adds new node in selected place */ static void skiplist_map_insert_node(TOID(struct skiplist_map_node) new_node, TOID(struct skiplist_map_node) path[SKIPLIST_LEVELS_NUM]) { unsigned current_level = 0; do { TX_ADD_FIELD(path[current_level], next[current_level]); D_RW(new_node)->next[current_level] = D_RO(path[current_level])->next[current_level]; D_RW(path[current_level])->next[current_level] = new_node; } while (++current_level < SKIPLIST_LEVELS_NUM && rand() % 2 == 0); } /* * skiplist_map_map_find -- (internal) returns path to searched node, or if * node doesn't exist, it will return path to place where key should be. */ static void skiplist_map_find(uint64_t key, TOID(struct skiplist_map_node) map, TOID(struct skiplist_map_node) *path) { int current_level; TOID(struct skiplist_map_node) active = map; for (current_level = SKIPLIST_LEVELS_NUM - 1; current_level >= 0; current_level--) { for (TOID(struct skiplist_map_node) next = D_RO(active)->next[current_level]; !TOID_EQUALS(next, NULL_NODE) && D_RO(next)->entry.key < key; next = D_RO(active)->next[current_level]) { active = next; } path[current_level] = active; } } /* * skiplist_map_insert -- inserts a new key-value pair into the map */ int skiplist_map_insert(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key, PMEMoid value) { int ret = 0; TOID(struct skiplist_map_node) new_node; TOID(struct skiplist_map_node) path[SKIPLIST_LEVELS_NUM]; TX_BEGIN(pop) { new_node = TX_ZNEW(struct skiplist_map_node); D_RW(new_node)->entry.key = key; D_RW(new_node)->entry.value = value; skiplist_map_find(key, map, path); skiplist_map_insert_node(new_node, path); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * skiplist_map_remove_free -- removes and frees an object from the list */ int skiplist_map_remove_free(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key) { int ret = 0; TX_BEGIN(pop) { PMEMoid val = skiplist_map_remove(pop, map, key); pmemobj_tx_free(val); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * skiplist_map_remove_node -- (internal) removes selected node */ static void skiplist_map_remove_node( TOID(struct skiplist_map_node) path[SKIPLIST_LEVELS_NUM]) { TOID(struct skiplist_map_node) to_remove = D_RO(path[0])->next[0]; int i; for (i = 0; i < SKIPLIST_LEVELS_NUM; i++) { if (TOID_EQUALS(D_RO(path[i])->next[i], to_remove)) { TX_ADD_FIELD(path[i], next[i]); D_RW(path[i])->next[i] = D_RO(to_remove)->next[i]; } } } /* * skiplist_map_remove -- removes key-value pair from the map */ PMEMoid skiplist_map_remove(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key) { PMEMoid ret = OID_NULL; TOID(struct skiplist_map_node) path[SKIPLIST_LEVELS_NUM]; TOID(struct skiplist_map_node) to_remove; TX_BEGIN(pop) { skiplist_map_find(key, map, path); to_remove = D_RO(path[0])->next[0]; if (!TOID_EQUALS(to_remove, NULL_NODE) && D_RO(to_remove)->entry.key == key) { ret = D_RO(to_remove)->entry.value; skiplist_map_remove_node(path); } } TX_ONABORT { ret = OID_NULL; } TX_END return ret; } /* * skiplist_map_get -- searches for a value of the key */ PMEMoid skiplist_map_get(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key) { PMEMoid ret = OID_NULL; TOID(struct skiplist_map_node) path[SKIPLIST_LEVELS_NUM], found; skiplist_map_find(key, map, path); found = D_RO(path[0])->next[0]; if (!TOID_EQUALS(found, NULL_NODE) && D_RO(found)->entry.key == key) { ret = D_RO(found)->entry.value; } return ret; } /* * skiplist_map_lookup -- searches if a key exists */ int skiplist_map_lookup(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key) { int ret = 0; TOID(struct skiplist_map_node) path[SKIPLIST_LEVELS_NUM], found; skiplist_map_find(key, map, path); found = D_RO(path[0])->next[0]; if (!TOID_EQUALS(found, NULL_NODE) && D_RO(found)->entry.key == key) { ret = 1; } return ret; } /* * skiplist_map_foreach -- calls function for each node on a list */ int skiplist_map_foreach(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { TOID(struct skiplist_map_node) next = map; while (!TOID_EQUALS(D_RO(next)->next[0], NULL_NODE)) { next = D_RO(next)->next[0]; cb(D_RO(next)->entry.key, D_RO(next)->entry.value, arg); } return 0; } /* * skiplist_map_is_empty -- checks whether the list map is empty */ int skiplist_map_is_empty(PMEMobjpool *pop, TOID(struct skiplist_map_node) map) { return TOID_IS_NULL(D_RO(map)->next[0]); } /* * skiplist_map_check -- check if given persistent object is a skiplist */ int skiplist_map_check(PMEMobjpool *pop, TOID(struct skiplist_map_node) map) { return TOID_IS_NULL(map) || !TOID_VALID(map); } pmdk-1.13.1/src/examples/libpmemobj/list_map/list_map.vcxproj.filters0000664000000000000000000000145414435627501024474 0ustar rootroot {4FC737F1-C7A5-4376-A066-2A32D752A2FF} cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx {93995380-89BD-4b04-88EB-625FBE52EBFB} h;hh;hpp;hxx;hm;inl;inc;xsd Header Files Source Files pmdk-1.13.1/src/examples/libpmemobj/list_map/skiplist_map.h0000664000000000000000000000324014435627501022443 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2016-2020, Intel Corporation */ /* * skiplist_map.h -- sorted list collection implementation */ #ifndef SKIPLIST_MAP_H #define SKIPLIST_MAP_H #include #ifndef SKIPLIST_MAP_TYPE_OFFSET #define SKIPLIST_MAP_TYPE_OFFSET 2020 #endif struct skiplist_map_node; TOID_DECLARE(struct skiplist_map_node, SKIPLIST_MAP_TYPE_OFFSET + 0); int skiplist_map_check(PMEMobjpool *pop, TOID(struct skiplist_map_node) map); int skiplist_map_create(PMEMobjpool *pop, TOID(struct skiplist_map_node) *map, void *arg); int skiplist_map_destroy(PMEMobjpool *pop, TOID(struct skiplist_map_node) *map); int skiplist_map_insert(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key, PMEMoid value); int skiplist_map_insert_new(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg); PMEMoid skiplist_map_remove(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key); int skiplist_map_remove_free(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key); int skiplist_map_clear(PMEMobjpool *pop, TOID(struct skiplist_map_node) map); PMEMoid skiplist_map_get(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key); int skiplist_map_lookup(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, uint64_t key); int skiplist_map_foreach(PMEMobjpool *pop, TOID(struct skiplist_map_node) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg); int skiplist_map_is_empty(PMEMobjpool *pop, TOID(struct skiplist_map_node) map); #endif /* SKIPLIST_MAP_H */ pmdk-1.13.1/src/examples/libpmemobj/tree_map/0000775000000000000000000000000014435627501017560 5ustar rootrootpmdk-1.13.1/src/examples/libpmemobj/tree_map/ctree_map.vcxproj0000664000000000000000000000523014435627501023134 0ustar rootroot Debug x64 Release x64 {BE18F227-A9F0-4B38-B689-4E2F9F09CA5F} pmemobj 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} StaticLibrary true v143 StaticLibrary false v143 true ..\..\..\LongPath.manifest CompileAsCpp pmdk-1.13.1/src/examples/libpmemobj/tree_map/btree_map.c0000664000000000000000000004112014435627501021660 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2019, Intel Corporation */ /* * btree_map.c -- textbook implementation of btree /w preemptive splitting */ #include #include #include #include "btree_map.h" TOID_DECLARE(struct tree_map_node, BTREE_MAP_TYPE_OFFSET + 1); #define BTREE_ORDER 8 /* can't be odd */ #define BTREE_MIN ((BTREE_ORDER / 2) - 1) /* min number of keys per node */ struct tree_map_node_item { uint64_t key; PMEMoid value; }; struct tree_map_node { int n; /* number of occupied slots */ struct tree_map_node_item items[BTREE_ORDER - 1]; TOID(struct tree_map_node) slots[BTREE_ORDER]; }; struct btree_map { TOID(struct tree_map_node) root; }; /* * set_empty_item -- (internal) sets null to the item */ static void set_empty_item(struct tree_map_node_item *item) { item->key = 0; item->value = OID_NULL; } /* * btree_map_create -- allocates a new btree instance */ int btree_map_create(PMEMobjpool *pop, TOID(struct btree_map) *map, void *arg) { int ret = 0; TX_BEGIN(pop) { pmemobj_tx_add_range_direct(map, sizeof(*map)); *map = TX_ZNEW(struct btree_map); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * btree_map_clear_node -- (internal) removes all elements from the node */ static void btree_map_clear_node(TOID(struct tree_map_node) node) { if (TOID_IS_NULL(node)) return; for (int i = 0; i < D_RO(node)->n; ++i) { btree_map_clear_node(D_RO(node)->slots[i]); } TX_FREE(node); } /* * btree_map_clear -- removes all elements from the map */ int btree_map_clear(PMEMobjpool *pop, TOID(struct btree_map) map) { int ret = 0; TX_BEGIN(pop) { btree_map_clear_node(D_RO(map)->root); TX_ADD_FIELD(map, root); D_RW(map)->root = TOID_NULL(struct tree_map_node); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * btree_map_destroy -- cleanups and frees btree instance */ int btree_map_destroy(PMEMobjpool *pop, TOID(struct btree_map) *map) { int ret = 0; TX_BEGIN(pop) { btree_map_clear(pop, *map); pmemobj_tx_add_range_direct(map, sizeof(*map)); TX_FREE(*map); *map = TOID_NULL(struct btree_map); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * btree_map_insert_item_at -- (internal) inserts an item at position */ static void btree_map_insert_item_at(TOID(struct tree_map_node) node, int pos, struct tree_map_node_item item) { D_RW(node)->items[pos] = item; D_RW(node)->n += 1; } /* * btree_map_insert_empty -- (internal) inserts an item into an empty node */ static void btree_map_insert_empty(TOID(struct btree_map) map, struct tree_map_node_item item) { TX_ADD_FIELD(map, root); D_RW(map)->root = TX_ZNEW(struct tree_map_node); btree_map_insert_item_at(D_RO(map)->root, 0, item); } /* * btree_map_insert_node -- (internal) inserts and makes space for new node */ static void btree_map_insert_node(TOID(struct tree_map_node) node, int p, struct tree_map_node_item item, TOID(struct tree_map_node) left, TOID(struct tree_map_node) right) { TX_ADD(node); if (D_RO(node)->items[p].key != 0) { /* move all existing data */ memmove(&D_RW(node)->items[p + 1], &D_RW(node)->items[p], sizeof(struct tree_map_node_item) * ((BTREE_ORDER - 2 - p))); memmove(&D_RW(node)->slots[p + 1], &D_RW(node)->slots[p], sizeof(TOID(struct tree_map_node)) * ((BTREE_ORDER - 1 - p))); } D_RW(node)->slots[p] = left; D_RW(node)->slots[p + 1] = right; btree_map_insert_item_at(node, p, item); } /* * btree_map_create_split_node -- (internal) splits a node into two */ static TOID(struct tree_map_node) btree_map_create_split_node(TOID(struct tree_map_node) node, struct tree_map_node_item *m) { TOID(struct tree_map_node) right = TX_ZNEW(struct tree_map_node); int c = (BTREE_ORDER / 2); *m = D_RO(node)->items[c - 1]; /* select median item */ TX_ADD(node); set_empty_item(&D_RW(node)->items[c - 1]); /* move everything right side of median to the new node */ for (int i = c; i < BTREE_ORDER; ++i) { if (i != BTREE_ORDER - 1) { D_RW(right)->items[D_RW(right)->n++] = D_RO(node)->items[i]; set_empty_item(&D_RW(node)->items[i]); } D_RW(right)->slots[i - c] = D_RO(node)->slots[i]; D_RW(node)->slots[i] = TOID_NULL(struct tree_map_node); } D_RW(node)->n = c - 1; return right; } /* * btree_map_find_dest_node -- (internal) finds a place to insert the new key at */ static TOID(struct tree_map_node) btree_map_find_dest_node(TOID(struct btree_map) map, TOID(struct tree_map_node) n, TOID(struct tree_map_node) parent, uint64_t key, int *p) { if (D_RO(n)->n == BTREE_ORDER - 1) { /* node is full, perform a split */ struct tree_map_node_item m; TOID(struct tree_map_node) right = btree_map_create_split_node(n, &m); if (!TOID_IS_NULL(parent)) { btree_map_insert_node(parent, *p, m, n, right); if (key > m.key) /* select node to continue search */ n = right; } else { /* replacing root node, the tree grows in height */ TOID(struct tree_map_node) up = TX_ZNEW(struct tree_map_node); D_RW(up)->n = 1; D_RW(up)->items[0] = m; D_RW(up)->slots[0] = n; D_RW(up)->slots[1] = right; TX_ADD_FIELD(map, root); D_RW(map)->root = up; n = up; } } int i; for (i = 0; i < BTREE_ORDER - 1; ++i) { *p = i; /* * The key either fits somewhere in the middle or at the * right edge of the node. */ if (D_RO(n)->n == i || D_RO(n)->items[i].key > key) { return TOID_IS_NULL(D_RO(n)->slots[i]) ? n : btree_map_find_dest_node(map, D_RO(n)->slots[i], n, key, p); } } /* * The key is bigger than the last node element, go one level deeper * in the rightmost child. */ return btree_map_find_dest_node(map, D_RO(n)->slots[i], n, key, p); } /* * btree_map_insert_item -- (internal) inserts and makes space for new item */ static void btree_map_insert_item(TOID(struct tree_map_node) node, int p, struct tree_map_node_item item) { TX_ADD(node); if (D_RO(node)->items[p].key != 0) { memmove(&D_RW(node)->items[p + 1], &D_RW(node)->items[p], sizeof(struct tree_map_node_item) * ((BTREE_ORDER - 2 - p))); } btree_map_insert_item_at(node, p, item); } /* * btree_map_is_empty -- checks whether the tree map is empty */ int btree_map_is_empty(PMEMobjpool *pop, TOID(struct btree_map) map) { return TOID_IS_NULL(D_RO(map)->root) || D_RO(D_RO(map)->root)->n == 0; } /* * btree_map_insert -- inserts a new key-value pair into the map */ int btree_map_insert(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key, PMEMoid value) { struct tree_map_node_item item = {key, value}; TX_BEGIN(pop) { if (btree_map_is_empty(pop, map)) { btree_map_insert_empty(map, item); } else { int p; /* position at the dest node to insert */ TOID(struct tree_map_node) parent = TOID_NULL(struct tree_map_node); TOID(struct tree_map_node) dest = btree_map_find_dest_node(map, D_RW(map)->root, parent, key, &p); btree_map_insert_item(dest, p, item); } } TX_END return 0; } /* * btree_map_rotate_right -- (internal) takes one element from right sibling */ static void btree_map_rotate_right(TOID(struct tree_map_node) rsb, TOID(struct tree_map_node) node, TOID(struct tree_map_node) parent, int p) { /* move the separator from parent to the deficient node */ struct tree_map_node_item sep = D_RO(parent)->items[p]; btree_map_insert_item(node, D_RO(node)->n, sep); /* the first element of the right sibling is the new separator */ TX_ADD_FIELD(parent, items[p]); D_RW(parent)->items[p] = D_RO(rsb)->items[0]; /* the nodes are not necessarily leafs, so copy also the slot */ TX_ADD_FIELD(node, slots[D_RO(node)->n]); D_RW(node)->slots[D_RO(node)->n] = D_RO(rsb)->slots[0]; TX_ADD(rsb); D_RW(rsb)->n -= 1; /* it loses one element, but still > min */ /* move all existing elements back by one array slot */ memmove(D_RW(rsb)->items, D_RO(rsb)->items + 1, sizeof(struct tree_map_node_item) * (D_RO(rsb)->n)); memmove(D_RW(rsb)->slots, D_RO(rsb)->slots + 1, sizeof(TOID(struct tree_map_node)) * (D_RO(rsb)->n + 1)); } /* * btree_map_rotate_left -- (internal) takes one element from left sibling */ static void btree_map_rotate_left(TOID(struct tree_map_node) lsb, TOID(struct tree_map_node) node, TOID(struct tree_map_node) parent, int p) { /* move the separator from parent to the deficient node */ struct tree_map_node_item sep = D_RO(parent)->items[p - 1]; btree_map_insert_item(node, 0, sep); /* the last element of the left sibling is the new separator */ TX_ADD_FIELD(parent, items[p - 1]); D_RW(parent)->items[p - 1] = D_RO(lsb)->items[D_RO(lsb)->n - 1]; /* rotate the node children */ memmove(D_RW(node)->slots + 1, D_RO(node)->slots, sizeof(TOID(struct tree_map_node)) * (D_RO(node)->n)); /* the nodes are not necessarily leafs, so copy also the slot */ D_RW(node)->slots[0] = D_RO(lsb)->slots[D_RO(lsb)->n]; TX_ADD_FIELD(lsb, n); D_RW(lsb)->n -= 1; /* it loses one element, but still > min */ } /* * btree_map_merge -- (internal) merges node and right sibling */ static void btree_map_merge(TOID(struct btree_map) map, TOID(struct tree_map_node) rn, TOID(struct tree_map_node) node, TOID(struct tree_map_node) parent, int p) { struct tree_map_node_item sep = D_RO(parent)->items[p]; TX_ADD(node); /* add separator to the deficient node */ D_RW(node)->items[D_RW(node)->n++] = sep; /* copy right sibling data to node */ memcpy(&D_RW(node)->items[D_RO(node)->n], D_RO(rn)->items, sizeof(struct tree_map_node_item) * D_RO(rn)->n); memcpy(&D_RW(node)->slots[D_RO(node)->n], D_RO(rn)->slots, sizeof(TOID(struct tree_map_node)) * (D_RO(rn)->n + 1)); D_RW(node)->n += D_RO(rn)->n; TX_FREE(rn); /* right node is now empty */ TX_ADD(parent); D_RW(parent)->n -= 1; /* move everything to the right of the separator by one array slot */ memmove(D_RW(parent)->items + p, D_RW(parent)->items + p + 1, sizeof(struct tree_map_node_item) * (D_RO(parent)->n - p)); memmove(D_RW(parent)->slots + p + 1, D_RW(parent)->slots + p + 2, sizeof(TOID(struct tree_map_node)) * (D_RO(parent)->n - p + 1)); /* if the parent is empty then the tree shrinks in height */ if (D_RO(parent)->n == 0 && TOID_EQUALS(parent, D_RO(map)->root)) { TX_ADD(map); TX_FREE(D_RO(map)->root); D_RW(map)->root = node; } } /* * btree_map_rebalance -- (internal) performs tree rebalance */ static void btree_map_rebalance(TOID(struct btree_map) map, TOID(struct tree_map_node) node, TOID(struct tree_map_node) parent, int p) { TOID(struct tree_map_node) rsb = p >= D_RO(parent)->n ? TOID_NULL(struct tree_map_node) : D_RO(parent)->slots[p + 1]; TOID(struct tree_map_node) lsb = p == 0 ? TOID_NULL(struct tree_map_node) : D_RO(parent)->slots[p - 1]; if (!TOID_IS_NULL(rsb) && D_RO(rsb)->n > BTREE_MIN) btree_map_rotate_right(rsb, node, parent, p); else if (!TOID_IS_NULL(lsb) && D_RO(lsb)->n > BTREE_MIN) btree_map_rotate_left(lsb, node, parent, p); else if (TOID_IS_NULL(rsb)) /* always merge with rightmost node */ btree_map_merge(map, node, lsb, parent, p - 1); else btree_map_merge(map, rsb, node, parent, p); } /* * btree_map_get_leftmost_leaf -- (internal) searches for the successor */ static TOID(struct tree_map_node) btree_map_get_leftmost_leaf(TOID(struct btree_map) map, TOID(struct tree_map_node) n, TOID(struct tree_map_node) *p) { if (TOID_IS_NULL(D_RO(n)->slots[0])) return n; *p = n; return btree_map_get_leftmost_leaf(map, D_RO(n)->slots[0], p); } /* * btree_map_remove_from_node -- (internal) removes element from node */ static void btree_map_remove_from_node(TOID(struct btree_map) map, TOID(struct tree_map_node) node, TOID(struct tree_map_node) parent, int p) { if (TOID_IS_NULL(D_RO(node)->slots[0])) { /* leaf */ TX_ADD(node); if (D_RO(node)->n == 1 || p == BTREE_ORDER - 2) { set_empty_item(&D_RW(node)->items[p]); } else if (D_RO(node)->n != 1) { memmove(&D_RW(node)->items[p], &D_RW(node)->items[p + 1], sizeof(struct tree_map_node_item) * (D_RO(node)->n - p)); } D_RW(node)->n -= 1; return; } /* can't delete from non-leaf nodes, remove successor */ TOID(struct tree_map_node) rchild = D_RW(node)->slots[p + 1]; TOID(struct tree_map_node) lp = node; TOID(struct tree_map_node) lm = btree_map_get_leftmost_leaf(map, rchild, &lp); TX_ADD_FIELD(node, items[p]); D_RW(node)->items[p] = D_RO(lm)->items[0]; btree_map_remove_from_node(map, lm, lp, 0); if (D_RO(lm)->n < BTREE_MIN) /* right child can be deficient now */ btree_map_rebalance(map, lm, lp, TOID_EQUALS(lp, node) ? p + 1 : 0); } #define NODE_CONTAINS_ITEM(_n, _i, _k)\ ((_i) != D_RO(_n)->n && D_RO(_n)->items[_i].key == (_k)) #define NODE_CHILD_CAN_CONTAIN_ITEM(_n, _i, _k)\ ((_i) == D_RO(_n)->n || D_RO(_n)->items[_i].key > (_k)) &&\ !TOID_IS_NULL(D_RO(_n)->slots[_i]) /* * btree_map_remove_item -- (internal) removes item from node */ static PMEMoid btree_map_remove_item(TOID(struct btree_map) map, TOID(struct tree_map_node) node, TOID(struct tree_map_node) parent, uint64_t key, int p) { PMEMoid ret = OID_NULL; for (int i = 0; i <= D_RO(node)->n; ++i) { if (NODE_CONTAINS_ITEM(node, i, key)) { ret = D_RO(node)->items[i].value; btree_map_remove_from_node(map, node, parent, i); break; } else if (NODE_CHILD_CAN_CONTAIN_ITEM(node, i, key)) { ret = btree_map_remove_item(map, D_RO(node)->slots[i], node, key, i); break; } } /* check for deficient nodes walking up */ if (!TOID_IS_NULL(parent) && D_RO(node)->n < BTREE_MIN) btree_map_rebalance(map, node, parent, p); return ret; } /* * btree_map_remove -- removes key-value pair from the map */ PMEMoid btree_map_remove(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key) { PMEMoid ret = OID_NULL; TX_BEGIN(pop) { ret = btree_map_remove_item(map, D_RW(map)->root, TOID_NULL(struct tree_map_node), key, 0); } TX_END return ret; } /* * btree_map_get_in_node -- (internal) searches for a value in the node */ static PMEMoid btree_map_get_in_node(TOID(struct tree_map_node) node, uint64_t key) { for (int i = 0; i <= D_RO(node)->n; ++i) { if (NODE_CONTAINS_ITEM(node, i, key)) return D_RO(node)->items[i].value; else if (NODE_CHILD_CAN_CONTAIN_ITEM(node, i, key)) return btree_map_get_in_node(D_RO(node)->slots[i], key); } return OID_NULL; } /* * btree_map_get -- searches for a value of the key */ PMEMoid btree_map_get(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key) { if (TOID_IS_NULL(D_RO(map)->root)) return OID_NULL; return btree_map_get_in_node(D_RO(map)->root, key); } /* * btree_map_lookup_in_node -- (internal) searches for key if exists */ static int btree_map_lookup_in_node(TOID(struct tree_map_node) node, uint64_t key) { for (int i = 0; i <= D_RO(node)->n; ++i) { if (NODE_CONTAINS_ITEM(node, i, key)) return 1; else if (NODE_CHILD_CAN_CONTAIN_ITEM(node, i, key)) return btree_map_lookup_in_node( D_RO(node)->slots[i], key); } return 0; } /* * btree_map_lookup -- searches if key exists */ int btree_map_lookup(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key) { if (TOID_IS_NULL(D_RO(map)->root)) return 0; return btree_map_lookup_in_node(D_RO(map)->root, key); } /* * btree_map_foreach_node -- (internal) recursively traverses tree */ static int btree_map_foreach_node(const TOID(struct tree_map_node) p, int (*cb)(uint64_t key, PMEMoid, void *arg), void *arg) { if (TOID_IS_NULL(p)) return 0; for (int i = 0; i <= D_RO(p)->n; ++i) { if (btree_map_foreach_node(D_RO(p)->slots[i], cb, arg) != 0) return 1; if (i != D_RO(p)->n && D_RO(p)->items[i].key != 0) { if (cb(D_RO(p)->items[i].key, D_RO(p)->items[i].value, arg) != 0) return 1; } } return 0; } /* * btree_map_foreach -- initiates recursive traversal */ int btree_map_foreach(PMEMobjpool *pop, TOID(struct btree_map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { return btree_map_foreach_node(D_RO(map)->root, cb, arg); } /* * ctree_map_check -- check if given persistent object is a tree map */ int btree_map_check(PMEMobjpool *pop, TOID(struct btree_map) map) { return TOID_IS_NULL(map) || !TOID_VALID(map); } /* * btree_map_insert_new -- allocates a new object and inserts it into the tree */ int btree_map_insert_new(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg) { int ret = 0; TX_BEGIN(pop) { PMEMoid n = pmemobj_tx_alloc(size, type_num); constructor(pop, pmemobj_direct(n), arg); btree_map_insert(pop, map, key, n); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * btree_map_remove_free -- removes and frees an object from the tree */ int btree_map_remove_free(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key) { int ret = 0; TX_BEGIN(pop) { PMEMoid val = btree_map_remove(pop, map, key); pmemobj_tx_free(val); } TX_ONABORT { ret = 1; } TX_END return ret; } pmdk-1.13.1/src/examples/libpmemobj/tree_map/rtree_map.h0000664000000000000000000000332314435627501021710 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2016-2020, Intel Corporation */ /* * rtree_map.h -- Radix TreeMap collection implementation */ #ifndef RTREE_MAP_H #define RTREE_MAP_H #include #ifndef RTREE_MAP_TYPE_OFFSET #define RTREE_MAP_TYPE_OFFSET 1020 #endif struct rtree_map; TOID_DECLARE(struct rtree_map, RTREE_MAP_TYPE_OFFSET + 0); int rtree_map_check(PMEMobjpool *pop, TOID(struct rtree_map) map); int rtree_map_create(PMEMobjpool *pop, TOID(struct rtree_map) *map, void *arg); int rtree_map_destroy(PMEMobjpool *pop, TOID(struct rtree_map) *map); int rtree_map_insert(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size, PMEMoid value); int rtree_map_insert_new(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg); PMEMoid rtree_map_remove(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size); int rtree_map_remove_free(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size); int rtree_map_clear(PMEMobjpool *pop, TOID(struct rtree_map) map); PMEMoid rtree_map_get(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size); int rtree_map_lookup(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size); int rtree_map_foreach(PMEMobjpool *pop, TOID(struct rtree_map) map, int (*cb)(const unsigned char *key, uint64_t key_size, PMEMoid value, void *arg), void *arg); int rtree_map_is_empty(PMEMobjpool *pop, TOID(struct rtree_map) map); #endif /* RTREE_MAP_H */ pmdk-1.13.1/src/examples/libpmemobj/tree_map/btree_map.vcxproj.filters0000664000000000000000000000124214435627501024601 0ustar rootroot {e00bdf1b-1168-4521-8034-629bf8717652} {e34e9a85-44de-435d-815d-fd07b599fadd} Header Files Source Files pmdk-1.13.1/src/examples/libpmemobj/tree_map/Makefile0000664000000000000000000000056214435627501021223 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2015-2017, Intel Corporation # # examples/libpmemobj/tree_map/Makefile -- build the tree map example # LIBRARIES = ctree_map btree_map rtree_map rbtree_map LIBS = -lpmemobj include ../../Makefile.inc libctree_map.o: ctree_map.o libbtree_map.o: btree_map.o librtree_map.o: rtree_map.o librbtree_map.o: rbtree_map.o pmdk-1.13.1/src/examples/libpmemobj/tree_map/btree_map.h0000664000000000000000000000276614435627501021702 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2015-2020, Intel Corporation */ /* * btree_map.h -- TreeMap sorted collection implementation */ #ifndef BTREE_MAP_H #define BTREE_MAP_H #include #ifndef BTREE_MAP_TYPE_OFFSET #define BTREE_MAP_TYPE_OFFSET 1012 #endif struct btree_map; TOID_DECLARE(struct btree_map, BTREE_MAP_TYPE_OFFSET + 0); int btree_map_check(PMEMobjpool *pop, TOID(struct btree_map) map); int btree_map_create(PMEMobjpool *pop, TOID(struct btree_map) *map, void *arg); int btree_map_destroy(PMEMobjpool *pop, TOID(struct btree_map) *map); int btree_map_insert(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key, PMEMoid value); int btree_map_insert_new(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg); PMEMoid btree_map_remove(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key); int btree_map_remove_free(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key); int btree_map_clear(PMEMobjpool *pop, TOID(struct btree_map) map); PMEMoid btree_map_get(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key); int btree_map_lookup(PMEMobjpool *pop, TOID(struct btree_map) map, uint64_t key); int btree_map_foreach(PMEMobjpool *pop, TOID(struct btree_map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg); int btree_map_is_empty(PMEMobjpool *pop, TOID(struct btree_map) map); #endif /* BTREE_MAP_H */ pmdk-1.13.1/src/examples/libpmemobj/tree_map/rbtree_map.vcxproj.filters0000664000000000000000000000124414435627501024765 0ustar rootroot {6b3d53f9-3187-4f70-9d79-a6c29e123af5} {d5387ced-fc6e-4980-a4ba-5112aca37cc0} Source Files Header Files pmdk-1.13.1/src/examples/libpmemobj/tree_map/ctree_map.h0000664000000000000000000000276614435627501021703 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2015-2020, Intel Corporation */ /* * ctree_map.h -- TreeMap sorted collection implementation */ #ifndef CTREE_MAP_H #define CTREE_MAP_H #include #ifndef CTREE_MAP_TYPE_OFFSET #define CTREE_MAP_TYPE_OFFSET 1008 #endif struct ctree_map; TOID_DECLARE(struct ctree_map, CTREE_MAP_TYPE_OFFSET + 0); int ctree_map_check(PMEMobjpool *pop, TOID(struct ctree_map) map); int ctree_map_create(PMEMobjpool *pop, TOID(struct ctree_map) *map, void *arg); int ctree_map_destroy(PMEMobjpool *pop, TOID(struct ctree_map) *map); int ctree_map_insert(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key, PMEMoid value); int ctree_map_insert_new(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg); PMEMoid ctree_map_remove(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key); int ctree_map_remove_free(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key); int ctree_map_clear(PMEMobjpool *pop, TOID(struct ctree_map) map); PMEMoid ctree_map_get(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key); int ctree_map_lookup(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key); int ctree_map_foreach(PMEMobjpool *pop, TOID(struct ctree_map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg); int ctree_map_is_empty(PMEMobjpool *pop, TOID(struct ctree_map) map); #endif /* CTREE_MAP_H */ pmdk-1.13.1/src/examples/libpmemobj/tree_map/rbtree_map.vcxproj0000664000000000000000000000523214435627501023317 0ustar rootroot Debug x64 Release x64 {17A4B817-68B1-4719-A9EF-BD8FAB747DE6} pmemobj 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} StaticLibrary true v143 StaticLibrary false v143 true ..\..\..\LongPath.manifest CompileAsCpp pmdk-1.13.1/src/examples/libpmemobj/tree_map/ctree_map.c0000664000000000000000000001755014435627501021673 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2019, Intel Corporation */ /* * ctree_map.c -- Crit-bit trie implementation */ #include #include #include #include #include "ctree_map.h" #define BIT_IS_SET(n, i) (!!((n) & (1ULL << (i)))) TOID_DECLARE(struct tree_map_node, CTREE_MAP_TYPE_OFFSET + 1); struct tree_map_entry { uint64_t key; PMEMoid slot; }; struct tree_map_node { int diff; /* most significant differing bit */ struct tree_map_entry entries[2]; }; struct ctree_map { struct tree_map_entry root; }; /* * find_crit_bit -- (internal) finds the most significant differing bit */ static int find_crit_bit(uint64_t lhs, uint64_t rhs) { return find_last_set_64(lhs ^ rhs); } /* * ctree_map_create -- allocates a new crit-bit tree instance */ int ctree_map_create(PMEMobjpool *pop, TOID(struct ctree_map) *map, void *arg) { int ret = 0; TX_BEGIN(pop) { pmemobj_tx_add_range_direct(map, sizeof(*map)); *map = TX_ZNEW(struct ctree_map); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * ctree_map_clear_node -- (internal) clears this node and its children */ static void ctree_map_clear_node(PMEMoid p) { if (OID_IS_NULL(p)) return; if (OID_INSTANCEOF(p, struct tree_map_node)) { TOID(struct tree_map_node) node; TOID_ASSIGN(node, p); ctree_map_clear_node(D_RW(node)->entries[0].slot); ctree_map_clear_node(D_RW(node)->entries[1].slot); } pmemobj_tx_free(p); } /* * ctree_map_clear -- removes all elements from the map */ int ctree_map_clear(PMEMobjpool *pop, TOID(struct ctree_map) map) { TX_BEGIN(pop) { ctree_map_clear_node(D_RW(map)->root.slot); TX_ADD_FIELD(map, root); D_RW(map)->root.slot = OID_NULL; } TX_END return 0; } /* * ctree_map_destroy -- cleanups and frees crit-bit tree instance */ int ctree_map_destroy(PMEMobjpool *pop, TOID(struct ctree_map) *map) { int ret = 0; TX_BEGIN(pop) { ctree_map_clear(pop, *map); pmemobj_tx_add_range_direct(map, sizeof(*map)); TX_FREE(*map); *map = TOID_NULL(struct ctree_map); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * ctree_map_insert_leaf -- (internal) inserts a new leaf at the position */ static void ctree_map_insert_leaf(struct tree_map_entry *p, struct tree_map_entry e, int diff) { TOID(struct tree_map_node) new_node = TX_NEW(struct tree_map_node); D_RW(new_node)->diff = diff; int d = BIT_IS_SET(e.key, D_RO(new_node)->diff); /* insert the leaf at the direction based on the critical bit */ D_RW(new_node)->entries[d] = e; /* find the appropriate position in the tree to insert the node */ TOID(struct tree_map_node) node; while (OID_INSTANCEOF(p->slot, struct tree_map_node)) { TOID_ASSIGN(node, p->slot); /* the critical bits have to be sorted */ if (D_RO(node)->diff < D_RO(new_node)->diff) break; p = &D_RW(node)->entries[BIT_IS_SET(e.key, D_RO(node)->diff)]; } /* insert the found destination in the other slot */ D_RW(new_node)->entries[!d] = *p; pmemobj_tx_add_range_direct(p, sizeof(*p)); p->key = 0; p->slot = new_node.oid; } /* * ctree_map_insert_new -- allocates a new object and inserts it into the tree */ int ctree_map_insert_new(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg) { int ret = 0; TX_BEGIN(pop) { PMEMoid n = pmemobj_tx_alloc(size, type_num); constructor(pop, pmemobj_direct(n), arg); ctree_map_insert(pop, map, key, n); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * ctree_map_insert -- inserts a new key-value pair into the map */ int ctree_map_insert(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key, PMEMoid value) { struct tree_map_entry *p = &D_RW(map)->root; int ret = 0; /* descend the path until a best matching key is found */ TOID(struct tree_map_node) node; while (!OID_IS_NULL(p->slot) && OID_INSTANCEOF(p->slot, struct tree_map_node)) { TOID_ASSIGN(node, p->slot); p = &D_RW(node)->entries[BIT_IS_SET(key, D_RW(node)->diff)]; } struct tree_map_entry e = {key, value}; TX_BEGIN(pop) { if (p->key == 0 || p->key == key) { pmemobj_tx_add_range_direct(p, sizeof(*p)); *p = e; } else { ctree_map_insert_leaf(&D_RW(map)->root, e, find_crit_bit(p->key, key)); } } TX_ONABORT { ret = 1; } TX_END return ret; } /* * ctree_map_get_leaf -- (internal) searches for a leaf of the key */ static struct tree_map_entry * ctree_map_get_leaf(TOID(struct ctree_map) map, uint64_t key, struct tree_map_entry **parent) { struct tree_map_entry *n = &D_RW(map)->root; struct tree_map_entry *p = NULL; TOID(struct tree_map_node) node; while (!OID_IS_NULL(n->slot) && OID_INSTANCEOF(n->slot, struct tree_map_node)) { TOID_ASSIGN(node, n->slot); p = n; n = &D_RW(node)->entries[BIT_IS_SET(key, D_RW(node)->diff)]; } if (n->key == key) { if (parent) *parent = p; return n; } return NULL; } /* * ctree_map_remove_free -- removes and frees an object from the tree */ int ctree_map_remove_free(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key) { int ret = 0; TX_BEGIN(pop) { PMEMoid val = ctree_map_remove(pop, map, key); pmemobj_tx_free(val); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * ctree_map_remove -- removes key-value pair from the map */ PMEMoid ctree_map_remove(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key) { struct tree_map_entry *parent = NULL; struct tree_map_entry *leaf = ctree_map_get_leaf(map, key, &parent); if (leaf == NULL) return OID_NULL; PMEMoid ret = leaf->slot; if (parent == NULL) { /* root */ TX_BEGIN(pop) { pmemobj_tx_add_range_direct(leaf, sizeof(*leaf)); leaf->key = 0; leaf->slot = OID_NULL; } TX_END } else { /* * In this situation: * parent * / \ * LEFT RIGHT * there's no point in leaving the parent internal node * so it's swapped with the remaining node and then also freed. */ TX_BEGIN(pop) { struct tree_map_entry *dest = parent; TOID(struct tree_map_node) node; TOID_ASSIGN(node, parent->slot); pmemobj_tx_add_range_direct(dest, sizeof(*dest)); *dest = D_RW(node)->entries[ D_RO(node)->entries[0].key == leaf->key]; TX_FREE(node); } TX_END } return ret; } /* * ctree_map_get -- searches for a value of the key */ PMEMoid ctree_map_get(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key) { struct tree_map_entry *entry = ctree_map_get_leaf(map, key, NULL); return entry ? entry->slot : OID_NULL; } /* * ctree_map_lookup -- searches if a key exists */ int ctree_map_lookup(PMEMobjpool *pop, TOID(struct ctree_map) map, uint64_t key) { struct tree_map_entry *entry = ctree_map_get_leaf(map, key, NULL); return entry != NULL; } /* * ctree_map_foreach_node -- (internal) recursively traverses tree */ static int ctree_map_foreach_node(struct tree_map_entry e, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { int ret = 0; if (OID_INSTANCEOF(e.slot, struct tree_map_node)) { TOID(struct tree_map_node) node; TOID_ASSIGN(node, e.slot); if (ctree_map_foreach_node(D_RO(node)->entries[0], cb, arg) == 0) ctree_map_foreach_node(D_RO(node)->entries[1], cb, arg); } else { /* leaf */ ret = cb(e.key, e.slot, arg); } return ret; } /* * ctree_map_foreach -- initiates recursive traversal */ int ctree_map_foreach(PMEMobjpool *pop, TOID(struct ctree_map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { if (OID_IS_NULL(D_RO(map)->root.slot)) return 0; return ctree_map_foreach_node(D_RO(map)->root, cb, arg); } /* * ctree_map_is_empty -- checks whether the tree map is empty */ int ctree_map_is_empty(PMEMobjpool *pop, TOID(struct ctree_map) map) { return D_RO(map)->root.key == 0; } /* * ctree_map_check -- check if given persistent object is a tree map */ int ctree_map_check(PMEMobjpool *pop, TOID(struct ctree_map) map) { return TOID_IS_NULL(map) || !TOID_VALID(map); } pmdk-1.13.1/src/examples/libpmemobj/tree_map/btree_map.vcxproj0000664000000000000000000000523014435627501023133 0ustar rootroot Debug x64 Release x64 {79D37FFE-FF76-44B3-BB27-3DCAEFF2EBE9} pmemobj 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} StaticLibrary true v143 StaticLibrary false v143 true ..\..\..\LongPath.manifest CompileAsCpp pmdk-1.13.1/src/examples/libpmemobj/tree_map/rtree_map.vcxproj0000664000000000000000000000533314435627501023157 0ustar rootroot Debug x64 Release x64 {3ED56E55-84A6-422C-A8D4-A8439FB8F245} pmemobj 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} StaticLibrary true v143 StaticLibrary false v143 true ..\..\..\LongPath.manifest CompileAsCpp 4200;4996 pmdk-1.13.1/src/examples/libpmemobj/tree_map/rbtree_map.c0000664000000000000000000002776214435627501022062 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2019, Intel Corporation */ /* * rbtree.c -- red-black tree implementation /w sentinel nodes */ #include #include #include "rbtree_map.h" TOID_DECLARE(struct tree_map_node, RBTREE_MAP_TYPE_OFFSET + 1); #define NODE_P(_n)\ D_RW(_n)->parent #define NODE_GRANDP(_n)\ NODE_P(NODE_P(_n)) #define NODE_PARENT_AT(_n, _rbc)\ D_RW(NODE_P(_n))->slots[_rbc] #define NODE_PARENT_RIGHT(_n)\ NODE_PARENT_AT(_n, RB_RIGHT) #define NODE_IS(_n, _rbc)\ TOID_EQUALS(_n, NODE_PARENT_AT(_n, _rbc)) #define NODE_IS_RIGHT(_n)\ TOID_EQUALS(_n, NODE_PARENT_RIGHT(_n)) #define NODE_LOCATION(_n)\ NODE_IS_RIGHT(_n) #define RB_FIRST(_m)\ D_RW(D_RW(_m)->root)->slots[RB_LEFT] #define NODE_IS_NULL(_n)\ TOID_EQUALS(_n, s) enum rb_color { COLOR_BLACK, COLOR_RED, MAX_COLOR }; enum rb_children { RB_LEFT, RB_RIGHT, MAX_RB }; struct tree_map_node { uint64_t key; PMEMoid value; enum rb_color color; TOID(struct tree_map_node) parent; TOID(struct tree_map_node) slots[MAX_RB]; }; struct rbtree_map { TOID(struct tree_map_node) sentinel; TOID(struct tree_map_node) root; }; /* * rbtree_map_create -- allocates a new red-black tree instance */ int rbtree_map_create(PMEMobjpool *pop, TOID(struct rbtree_map) *map, void *arg) { int ret = 0; TX_BEGIN(pop) { pmemobj_tx_add_range_direct(map, sizeof(*map)); *map = TX_ZNEW(struct rbtree_map); TOID(struct tree_map_node) s = TX_ZNEW(struct tree_map_node); D_RW(s)->color = COLOR_BLACK; D_RW(s)->parent = s; D_RW(s)->slots[RB_LEFT] = s; D_RW(s)->slots[RB_RIGHT] = s; TOID(struct tree_map_node) r = TX_ZNEW(struct tree_map_node); D_RW(r)->color = COLOR_BLACK; D_RW(r)->parent = s; D_RW(r)->slots[RB_LEFT] = s; D_RW(r)->slots[RB_RIGHT] = s; D_RW(*map)->sentinel = s; D_RW(*map)->root = r; } TX_ONABORT { ret = 1; } TX_END return ret; } /* * rbtree_map_clear_node -- (internal) clears this node and its children */ static void rbtree_map_clear_node(TOID(struct rbtree_map) map, TOID(struct tree_map_node) p) { TOID(struct tree_map_node) s = D_RO(map)->sentinel; if (!NODE_IS_NULL(D_RO(p)->slots[RB_LEFT])) rbtree_map_clear_node(map, D_RO(p)->slots[RB_LEFT]); if (!NODE_IS_NULL(D_RO(p)->slots[RB_RIGHT])) rbtree_map_clear_node(map, D_RO(p)->slots[RB_RIGHT]); TX_FREE(p); } /* * rbtree_map_clear -- removes all elements from the map */ int rbtree_map_clear(PMEMobjpool *pop, TOID(struct rbtree_map) map) { TX_BEGIN(pop) { rbtree_map_clear_node(map, D_RW(map)->root); TX_ADD_FIELD(map, root); TX_ADD_FIELD(map, sentinel); TX_FREE(D_RW(map)->sentinel); D_RW(map)->root = TOID_NULL(struct tree_map_node); D_RW(map)->sentinel = TOID_NULL(struct tree_map_node); } TX_END return 0; } /* * rbtree_map_destroy -- cleanups and frees red-black tree instance */ int rbtree_map_destroy(PMEMobjpool *pop, TOID(struct rbtree_map) *map) { int ret = 0; TX_BEGIN(pop) { rbtree_map_clear(pop, *map); pmemobj_tx_add_range_direct(map, sizeof(*map)); TX_FREE(*map); *map = TOID_NULL(struct rbtree_map); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * rbtree_map_rotate -- (internal) performs a left/right rotation around a node */ static void rbtree_map_rotate(TOID(struct rbtree_map) map, TOID(struct tree_map_node) node, enum rb_children c) { TOID(struct tree_map_node) child = D_RO(node)->slots[!c]; TOID(struct tree_map_node) s = D_RO(map)->sentinel; TX_ADD(node); TX_ADD(child); D_RW(node)->slots[!c] = D_RO(child)->slots[c]; if (!TOID_EQUALS(D_RO(child)->slots[c], s)) TX_SET(D_RW(child)->slots[c], parent, node); NODE_P(child) = NODE_P(node); TX_SET(NODE_P(node), slots[NODE_LOCATION(node)], child); D_RW(child)->slots[c] = node; D_RW(node)->parent = child; } /* * rbtree_map_insert_bst -- (internal) inserts a node in regular BST fashion */ static void rbtree_map_insert_bst(TOID(struct rbtree_map) map, TOID(struct tree_map_node) n) { TOID(struct tree_map_node) parent = D_RO(map)->root; TOID(struct tree_map_node) *dst = &RB_FIRST(map); TOID(struct tree_map_node) s = D_RO(map)->sentinel; D_RW(n)->slots[RB_LEFT] = s; D_RW(n)->slots[RB_RIGHT] = s; while (!NODE_IS_NULL(*dst)) { parent = *dst; dst = &D_RW(*dst)->slots[D_RO(n)->key > D_RO(*dst)->key]; } TX_SET(n, parent, parent); pmemobj_tx_add_range_direct(dst, sizeof(*dst)); *dst = n; } /* * rbtree_map_recolor -- (internal) restores red-black tree properties */ static TOID(struct tree_map_node) rbtree_map_recolor(TOID(struct rbtree_map) map, TOID(struct tree_map_node) n, enum rb_children c) { TOID(struct tree_map_node) uncle = D_RO(NODE_GRANDP(n))->slots[!c]; if (D_RO(uncle)->color == COLOR_RED) { TX_SET(uncle, color, COLOR_BLACK); TX_SET(NODE_P(n), color, COLOR_BLACK); TX_SET(NODE_GRANDP(n), color, COLOR_RED); return NODE_GRANDP(n); } else { if (NODE_IS(n, !c)) { n = NODE_P(n); rbtree_map_rotate(map, n, c); } TX_SET(NODE_P(n), color, COLOR_BLACK); TX_SET(NODE_GRANDP(n), color, COLOR_RED); rbtree_map_rotate(map, NODE_GRANDP(n), (enum rb_children)!c); } return n; } /* * rbtree_map_insert -- inserts a new key-value pair into the map */ int rbtree_map_insert(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key, PMEMoid value) { int ret = 0; TX_BEGIN(pop) { TOID(struct tree_map_node) n = TX_ZNEW(struct tree_map_node); D_RW(n)->key = key; D_RW(n)->value = value; rbtree_map_insert_bst(map, n); D_RW(n)->color = COLOR_RED; while (D_RO(NODE_P(n))->color == COLOR_RED) n = rbtree_map_recolor(map, n, (enum rb_children) NODE_LOCATION(NODE_P(n))); TX_SET(RB_FIRST(map), color, COLOR_BLACK); } TX_END return ret; } /* * rbtree_map_successor -- (internal) returns the successor of a node */ static TOID(struct tree_map_node) rbtree_map_successor(TOID(struct rbtree_map) map, TOID(struct tree_map_node) n) { TOID(struct tree_map_node) dst = D_RO(n)->slots[RB_RIGHT]; TOID(struct tree_map_node) s = D_RO(map)->sentinel; if (!TOID_EQUALS(s, dst)) { while (!NODE_IS_NULL(D_RO(dst)->slots[RB_LEFT])) dst = D_RO(dst)->slots[RB_LEFT]; } else { dst = D_RO(n)->parent; while (TOID_EQUALS(n, D_RO(dst)->slots[RB_RIGHT])) { n = dst; dst = NODE_P(dst); } if (TOID_EQUALS(dst, D_RO(map)->root)) return s; } return dst; } /* * rbtree_map_find_node -- (internal) returns the node that contains the key */ static TOID(struct tree_map_node) rbtree_map_find_node(TOID(struct rbtree_map) map, uint64_t key) { TOID(struct tree_map_node) dst = RB_FIRST(map); TOID(struct tree_map_node) s = D_RO(map)->sentinel; while (!NODE_IS_NULL(dst)) { if (D_RO(dst)->key == key) return dst; dst = D_RO(dst)->slots[key > D_RO(dst)->key]; } return TOID_NULL(struct tree_map_node); } /* * rbtree_map_repair_branch -- (internal) restores red-black tree in one branch */ static TOID(struct tree_map_node) rbtree_map_repair_branch(TOID(struct rbtree_map) map, TOID(struct tree_map_node) n, enum rb_children c) { TOID(struct tree_map_node) sb = NODE_PARENT_AT(n, !c); /* sibling */ if (D_RO(sb)->color == COLOR_RED) { TX_SET(sb, color, COLOR_BLACK); TX_SET(NODE_P(n), color, COLOR_RED); rbtree_map_rotate(map, NODE_P(n), c); sb = NODE_PARENT_AT(n, !c); } if (D_RO(D_RO(sb)->slots[RB_RIGHT])->color == COLOR_BLACK && D_RO(D_RO(sb)->slots[RB_LEFT])->color == COLOR_BLACK) { TX_SET(sb, color, COLOR_RED); return D_RO(n)->parent; } else { if (D_RO(D_RO(sb)->slots[!c])->color == COLOR_BLACK) { TX_SET(D_RW(sb)->slots[c], color, COLOR_BLACK); TX_SET(sb, color, COLOR_RED); rbtree_map_rotate(map, sb, (enum rb_children)!c); sb = NODE_PARENT_AT(n, !c); } TX_SET(sb, color, D_RO(NODE_P(n))->color); TX_SET(NODE_P(n), color, COLOR_BLACK); TX_SET(D_RW(sb)->slots[!c], color, COLOR_BLACK); rbtree_map_rotate(map, NODE_P(n), c); return RB_FIRST(map); } return n; } /* * rbtree_map_repair -- (internal) restores red-black tree properties * after remove */ static void rbtree_map_repair(TOID(struct rbtree_map) map, TOID(struct tree_map_node) n) { /* if left, repair right sibling, otherwise repair left sibling. */ while (!TOID_EQUALS(n, RB_FIRST(map)) && D_RO(n)->color == COLOR_BLACK) n = rbtree_map_repair_branch(map, n, (enum rb_children) NODE_LOCATION(n)); TX_SET(n, color, COLOR_BLACK); } /* * rbtree_map_remove -- removes key-value pair from the map */ PMEMoid rbtree_map_remove(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key) { PMEMoid ret = OID_NULL; TOID(struct tree_map_node) n = rbtree_map_find_node(map, key); if (TOID_IS_NULL(n)) return ret; ret = D_RO(n)->value; TOID(struct tree_map_node) s = D_RO(map)->sentinel; TOID(struct tree_map_node) r = D_RO(map)->root; TOID(struct tree_map_node) y = (NODE_IS_NULL(D_RO(n)->slots[RB_LEFT]) || NODE_IS_NULL(D_RO(n)->slots[RB_RIGHT])) ? n : rbtree_map_successor(map, n); TOID(struct tree_map_node) x = NODE_IS_NULL(D_RO(y)->slots[RB_LEFT]) ? D_RO(y)->slots[RB_RIGHT] : D_RO(y)->slots[RB_LEFT]; TX_BEGIN(pop) { TX_SET(x, parent, NODE_P(y)); if (TOID_EQUALS(NODE_P(x), r)) { TX_SET(r, slots[RB_LEFT], x); } else { TX_SET(NODE_P(y), slots[NODE_LOCATION(y)], x); } if (D_RO(y)->color == COLOR_BLACK) rbtree_map_repair(map, x); if (!TOID_EQUALS(y, n)) { TX_ADD(y); D_RW(y)->slots[RB_LEFT] = D_RO(n)->slots[RB_LEFT]; D_RW(y)->slots[RB_RIGHT] = D_RO(n)->slots[RB_RIGHT]; D_RW(y)->parent = D_RO(n)->parent; D_RW(y)->color = D_RO(n)->color; TX_SET(D_RW(n)->slots[RB_LEFT], parent, y); TX_SET(D_RW(n)->slots[RB_RIGHT], parent, y); TX_SET(NODE_P(n), slots[NODE_LOCATION(n)], y); } TX_FREE(n); } TX_END return ret; } /* * rbtree_map_get -- searches for a value of the key */ PMEMoid rbtree_map_get(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key) { TOID(struct tree_map_node) node = rbtree_map_find_node(map, key); if (TOID_IS_NULL(node)) return OID_NULL; return D_RO(node)->value; } /* * rbtree_map_lookup -- searches if key exists */ int rbtree_map_lookup(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key) { TOID(struct tree_map_node) node = rbtree_map_find_node(map, key); if (TOID_IS_NULL(node)) return 0; return 1; } /* * rbtree_map_foreach_node -- (internal) recursively traverses tree */ static int rbtree_map_foreach_node(TOID(struct rbtree_map) map, TOID(struct tree_map_node) p, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { int ret = 0; if (TOID_EQUALS(p, D_RO(map)->sentinel)) return 0; if ((ret = rbtree_map_foreach_node(map, D_RO(p)->slots[RB_LEFT], cb, arg)) == 0) { if ((ret = cb(D_RO(p)->key, D_RO(p)->value, arg)) == 0) rbtree_map_foreach_node(map, D_RO(p)->slots[RB_RIGHT], cb, arg); } return ret; } /* * rbtree_map_foreach -- initiates recursive traversal */ int rbtree_map_foreach(PMEMobjpool *pop, TOID(struct rbtree_map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg) { return rbtree_map_foreach_node(map, RB_FIRST(map), cb, arg); } /* * rbtree_map_is_empty -- checks whether the tree map is empty */ int rbtree_map_is_empty(PMEMobjpool *pop, TOID(struct rbtree_map) map) { return TOID_IS_NULL(RB_FIRST(map)); } /* * rbtree_map_check -- check if given persistent object is a tree map */ int rbtree_map_check(PMEMobjpool *pop, TOID(struct rbtree_map) map) { return TOID_IS_NULL(map) || !TOID_VALID(map); } /* * rbtree_map_insert_new -- allocates a new object and inserts it into the tree */ int rbtree_map_insert_new(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg) { int ret = 0; TX_BEGIN(pop) { PMEMoid n = pmemobj_tx_alloc(size, type_num); constructor(pop, pmemobj_direct(n), arg); rbtree_map_insert(pop, map, key, n); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * rbtree_map_remove_free -- removes and frees an object from the tree */ int rbtree_map_remove_free(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key) { int ret = 0; TX_BEGIN(pop) { PMEMoid val = rbtree_map_remove(pop, map, key); pmemobj_tx_free(val); } TX_ONABORT { ret = 1; } TX_END return ret; } pmdk-1.13.1/src/examples/libpmemobj/tree_map/ctree_map.vcxproj.filters0000664000000000000000000000124214435627501024602 0ustar rootroot {ae1612b3-efcb-4228-8946-071051e0395a} {4dbe7d5b-3e5f-42cb-a55a-eba2f613e9c5} Source Files Header Files pmdk-1.13.1/src/examples/libpmemobj/tree_map/rbtree_map.h0000664000000000000000000000303014435627501022045 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2015-2020, Intel Corporation */ /* * rbtree_map.h -- TreeMap sorted collection implementation */ #ifndef RBTREE_MAP_H #define RBTREE_MAP_H #include #ifndef RBTREE_MAP_TYPE_OFFSET #define RBTREE_MAP_TYPE_OFFSET 1016 #endif struct rbtree_map; TOID_DECLARE(struct rbtree_map, RBTREE_MAP_TYPE_OFFSET + 0); int rbtree_map_check(PMEMobjpool *pop, TOID(struct rbtree_map) map); int rbtree_map_create(PMEMobjpool *pop, TOID(struct rbtree_map) *map, void *arg); int rbtree_map_destroy(PMEMobjpool *pop, TOID(struct rbtree_map) *map); int rbtree_map_insert(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key, PMEMoid value); int rbtree_map_insert_new(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg); PMEMoid rbtree_map_remove(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key); int rbtree_map_remove_free(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key); int rbtree_map_clear(PMEMobjpool *pop, TOID(struct rbtree_map) map); PMEMoid rbtree_map_get(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key); int rbtree_map_lookup(PMEMobjpool *pop, TOID(struct rbtree_map) map, uint64_t key); int rbtree_map_foreach(PMEMobjpool *pop, TOID(struct rbtree_map) map, int (*cb)(uint64_t key, PMEMoid value, void *arg), void *arg); int rbtree_map_is_empty(PMEMobjpool *pop, TOID(struct rbtree_map) map); #endif /* RBTREE_MAP_H */ pmdk-1.13.1/src/examples/libpmemobj/tree_map/rtree_map.c0000664000000000000000000003075514435627501021714 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016-2019, Intel Corporation */ /* * rtree_map.c -- implementation of rtree */ #include #include #include #include #include #include #include "rtree_map.h" TOID_DECLARE(struct tree_map_node, RTREE_MAP_TYPE_OFFSET + 1); /* Good values: 0x10 an 0x100, but implementation is bound to 0x100 */ #ifndef ALPHABET_SIZE #define ALPHABET_SIZE 0x100 #endif struct tree_map_node { TOID(struct tree_map_node) slots[ALPHABET_SIZE]; unsigned has_value; PMEMoid value; uint64_t key_size; unsigned char key[]; }; struct rtree_map { TOID(struct tree_map_node) root; }; /* * rtree_map_create -- allocates a new rtree instance */ int rtree_map_create(PMEMobjpool *pop, TOID(struct rtree_map) *map, void *arg) { int ret = 0; TX_BEGIN(pop) { TX_ADD_DIRECT(map); *map = TX_ZNEW(struct rtree_map); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * rtree_map_clear_node -- (internal) removes all elements from the node */ static void rtree_map_clear_node(TOID(struct tree_map_node) node) { for (unsigned i = 0; i < ALPHABET_SIZE; i++) { rtree_map_clear_node(D_RO(node)->slots[i]); } pmemobj_tx_add_range(node.oid, 0, sizeof(struct tree_map_node) + D_RO(node)->key_size); TX_FREE(node); } /* * rtree_map_clear -- removes all elements from the map */ int rtree_map_clear(PMEMobjpool *pop, TOID(struct rtree_map) map) { int ret = 0; TX_BEGIN(pop) { rtree_map_clear_node(D_RO(map)->root); TX_ADD_FIELD(map, root); D_RW(map)->root = TOID_NULL(struct tree_map_node); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * rtree_map_destroy -- cleanups and frees rtree instance */ int rtree_map_destroy(PMEMobjpool *pop, TOID(struct rtree_map) *map) { int ret = 0; TX_BEGIN(pop) { rtree_map_clear(pop, *map); TX_ADD_DIRECT(map); TX_FREE(*map); *map = TOID_NULL(struct rtree_map); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * rtree_new_node -- (internal) inserts a node into an empty map */ static TOID(struct tree_map_node) rtree_new_node(const unsigned char *key, uint64_t key_size, PMEMoid value, unsigned has_value) { TOID(struct tree_map_node) node; node = TX_ZALLOC(struct tree_map_node, sizeof(struct tree_map_node) + key_size); /* * !!! Here should be: D_RO(node)->value * ... because we don't change map */ D_RW(node)->value = value; D_RW(node)->has_value = has_value; D_RW(node)->key_size = key_size; memcpy(D_RW(node)->key, key, key_size); return node; } /* * rtree_map_insert_empty -- (internal) inserts a node into an empty map */ static void rtree_map_insert_empty(TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size, PMEMoid value) { TX_ADD_FIELD(map, root); D_RW(map)->root = rtree_new_node(key, key_size, value, 1); } /* * key_comm_len -- (internal) calculate the len of common part of keys */ static unsigned key_comm_len(TOID(struct tree_map_node) node, const unsigned char *key, uint64_t key_size) { unsigned i; for (i = 0; i < MIN(key_size, D_RO(node)->key_size) && key[i] == D_RO(node)->key[i]; i++) ; return i; } /* * rtree_map_insert_value -- (internal) inserts a pair into a tree */ static void rtree_map_insert_value(TOID(struct tree_map_node) *node, const unsigned char *key, uint64_t key_size, PMEMoid value) { unsigned i; if (TOID_IS_NULL(*node)) { TX_ADD_DIRECT(node); *node = rtree_new_node(key, key_size, value, 1); return; } i = key_comm_len(*node, key, key_size); if (i != D_RO(*node)->key_size) { /* Node does not exist. Let's add. */ TOID(struct tree_map_node) orig_node = *node; TX_ADD_DIRECT(node); if (i != key_size) { *node = rtree_new_node(D_RO(orig_node)->key, i, OID_NULL, 0); } else { *node = rtree_new_node(D_RO(orig_node)->key, i, value, 1); } D_RW(*node)->slots[D_RO(orig_node)->key[i]] = orig_node; TX_ADD_FIELD(orig_node, key_size); D_RW(orig_node)->key_size -= i; pmemobj_tx_add_range_direct(D_RW(orig_node)->key, D_RO(orig_node)->key_size); memmove(D_RW(orig_node)->key, D_RO(orig_node)->key + i, D_RO(orig_node)->key_size); if (i != key_size) { D_RW(*node)->slots[key[i]] = rtree_new_node(key + i, key_size - i, value, 1); } return; } if (i == key_size) { if (OID_IS_NULL(D_RO(*node)->value) || D_RO(*node)->has_value) { /* Just replace old value with new */ TX_ADD_FIELD(*node, value); TX_ADD_FIELD(*node, has_value); D_RW(*node)->value = value; D_RW(*node)->has_value = 1; } else { /* * Ignore. By the fact current value should be * removed in advance, or handled in a different way. */ } } else { /* Recurse deeply */ return rtree_map_insert_value(&D_RW(*node)->slots[key[i]], key + i, key_size - i, value); } } /* * rtree_map_is_empty -- checks whether the tree map is empty */ int rtree_map_is_empty(PMEMobjpool *pop, TOID(struct rtree_map) map) { return TOID_IS_NULL(D_RO(map)->root); } /* * rtree_map_insert -- inserts a new key-value pair into the map */ int rtree_map_insert(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size, PMEMoid value) { int ret = 0; TX_BEGIN(pop) { if (rtree_map_is_empty(pop, map)) { rtree_map_insert_empty(map, key, key_size, value); } else { rtree_map_insert_value(&D_RW(map)->root, key, key_size, value); } } TX_ONABORT { ret = 1; } TX_END return ret; } /* * rtree_map_insert_new -- allocates a new object and inserts it into the tree */ int rtree_map_insert_new(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size, size_t size, unsigned type_num, void (*constructor)(PMEMobjpool *pop, void *ptr, void *arg), void *arg) { int ret = 0; TX_BEGIN(pop) { PMEMoid n = pmemobj_tx_alloc(size, type_num); constructor(pop, pmemobj_direct(n), arg); rtree_map_insert(pop, map, key, key_size, n); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * is_leaf -- (internal) check a node for zero qty of children */ static bool is_leaf(TOID(struct tree_map_node) node) { unsigned j; for (j = 0; j < ALPHABET_SIZE && TOID_IS_NULL(D_RO(node)->slots[j]); j++) ; return (j == ALPHABET_SIZE); } /* * has_only_one_child -- (internal) check a node for qty of children */ static bool has_only_one_child(TOID(struct tree_map_node) node, unsigned *child_idx) { unsigned j, child_qty; for (j = 0, child_qty = 0; j < ALPHABET_SIZE; j++) if (!TOID_IS_NULL(D_RO(node)->slots[j])) { child_qty++; *child_idx = j; } return (1 == child_qty); } /* * remove_extra_node -- (internal) remove unneeded extra node */ static void remove_extra_node(TOID(struct tree_map_node) *node) { unsigned child_idx = UINT_MAX; TOID(struct tree_map_node) tmp, tmp_child; /* Our node has child with only one child. */ tmp = *node; has_only_one_child(tmp, &child_idx); assert(child_idx != UINT_MAX); tmp_child = D_RO(tmp)->slots[child_idx]; /* * That child's incoming label is appended to the ours incoming label * and the child is removed. */ uint64_t new_key_size = D_RO(tmp)->key_size + D_RO(tmp_child)->key_size; unsigned char *new_key = (unsigned char *)malloc(new_key_size); assert(new_key != NULL); memcpy(new_key, D_RO(tmp)->key, D_RO(tmp)->key_size); memcpy(new_key + D_RO(tmp)->key_size, D_RO(tmp_child)->key, D_RO(tmp_child)->key_size); TX_ADD_DIRECT(node); *node = rtree_new_node(new_key, new_key_size, D_RO(tmp_child)->value, D_RO(tmp_child)->has_value); free(new_key); TX_FREE(tmp); memcpy(D_RW(*node)->slots, D_RO(tmp_child)->slots, sizeof(D_RO(tmp_child)->slots)); TX_FREE(tmp_child); } /* * rtree_map_remove_node -- (internal) removes node from tree */ static PMEMoid rtree_map_remove_node(TOID(struct rtree_map) map, TOID(struct tree_map_node) *node, const unsigned char *key, uint64_t key_size, bool *check_for_child) { bool c4c; unsigned i, child_idx; PMEMoid ret = OID_NULL; *check_for_child = false; if (TOID_IS_NULL(*node)) return OID_NULL; i = key_comm_len(*node, key, key_size); if (i != D_RO(*node)->key_size) /* Node does not exist */ return OID_NULL; if (i == key_size) { if (0 == D_RO(*node)->has_value) return OID_NULL; /* Node is found */ ret = D_RO(*node)->value; /* delete node from tree */ TX_ADD_FIELD((*node), value); TX_ADD_FIELD((*node), has_value); D_RW(*node)->value = OID_NULL; D_RW(*node)->has_value = 0; if (is_leaf(*node)) { pmemobj_tx_add_range(node->oid, 0, sizeof(*node) + D_RO(*node)->key_size); TX_FREE(*node); TX_ADD_DIRECT(node); (*node) = TOID_NULL(struct tree_map_node); } return ret; } /* Recurse deeply */ ret = rtree_map_remove_node(map, &D_RW(*node)->slots[key[i]], key + i, key_size - i, &c4c); if (c4c) { /* Our node has child with only one child. Remove. */ remove_extra_node(&D_RW(*node)->slots[key[i]]); return ret; } if (has_only_one_child(*node, &child_idx) && (0 == D_RO(*node)->has_value)) { *check_for_child = true; } return ret; } /* * rtree_map_remove -- removes key-value pair from the map */ PMEMoid rtree_map_remove(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size) { PMEMoid ret = OID_NULL; bool check_for_child; if (TOID_IS_NULL(map)) return OID_NULL; TX_BEGIN(pop) { ret = rtree_map_remove_node(map, &D_RW(map)->root, key, key_size, &check_for_child); if (check_for_child) { /* Our root node has only one child. Remove. */ remove_extra_node(&D_RW(map)->root); } } TX_END return ret; } /* * rtree_map_remove_free -- removes and frees an object from the tree */ int rtree_map_remove_free(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size) { int ret = 0; if (TOID_IS_NULL(map)) return 1; TX_BEGIN(pop) { pmemobj_tx_free(rtree_map_remove(pop, map, key, key_size)); } TX_ONABORT { ret = 1; } TX_END return ret; } /* * rtree_map_get_in_node -- (internal) searches for a value in the node */ static PMEMoid rtree_map_get_in_node(TOID(struct tree_map_node) node, const unsigned char *key, uint64_t key_size) { unsigned i; if (TOID_IS_NULL(node)) return OID_NULL; i = key_comm_len(node, key, key_size); if (i != D_RO(node)->key_size) /* Node does not exist */ return OID_NULL; if (i == key_size) { /* Node is found */ return D_RO(node)->value; } else { /* Recurse deeply */ return rtree_map_get_in_node(D_RO(node)->slots[key[i]], key + i, key_size - i); } } /* * rtree_map_get -- searches for a value of the key */ PMEMoid rtree_map_get(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size) { if (TOID_IS_NULL(D_RO(map)->root)) return OID_NULL; return rtree_map_get_in_node(D_RO(map)->root, key, key_size); } /* * rtree_map_lookup_in_node -- (internal) searches for key if exists */ static int rtree_map_lookup_in_node(TOID(struct tree_map_node) node, const unsigned char *key, uint64_t key_size) { unsigned i; if (TOID_IS_NULL(node)) return 0; i = key_comm_len(node, key, key_size); if (i != D_RO(node)->key_size) /* Node does not exist */ return 0; if (i == key_size) { /* Node is found */ return 1; } /* Recurse deeply */ return rtree_map_lookup_in_node(D_RO(node)->slots[key[i]], key + i, key_size - i); } /* * rtree_map_lookup -- searches if key exists */ int rtree_map_lookup(PMEMobjpool *pop, TOID(struct rtree_map) map, const unsigned char *key, uint64_t key_size) { if (TOID_IS_NULL(D_RO(map)->root)) return 0; return rtree_map_lookup_in_node(D_RO(map)->root, key, key_size); } /* * rtree_map_foreach_node -- (internal) recursively traverses tree */ static int rtree_map_foreach_node(const TOID(struct tree_map_node) node, int (*cb)(const unsigned char *key, uint64_t key_size, PMEMoid, void *arg), void *arg) { unsigned i; if (TOID_IS_NULL(node)) return 0; for (i = 0; i < ALPHABET_SIZE; i++) { if (rtree_map_foreach_node(D_RO(node)->slots[i], cb, arg) != 0) return 1; } if (NULL != cb) { if (cb(D_RO(node)->key, D_RO(node)->key_size, D_RO(node)->value, arg) != 0) return 1; } return 0; } /* * rtree_map_foreach -- initiates recursive traversal */ int rtree_map_foreach(PMEMobjpool *pop, TOID(struct rtree_map) map, int (*cb)(const unsigned char *key, uint64_t key_size, PMEMoid value, void *arg), void *arg) { return rtree_map_foreach_node(D_RO(map)->root, cb, arg); } /* * ctree_map_check -- check if given persistent object is a tree map */ int rtree_map_check(PMEMobjpool *pop, TOID(struct rtree_map) map) { return TOID_IS_NULL(map) || !TOID_VALID(map); } pmdk-1.13.1/src/examples/libpmemobj/tree_map/rtree_map.vcxproj.filters0000664000000000000000000000124214435627501024621 0ustar rootroot {662f1b6f-2944-4a24-867d-1945f972ad8b} {ba8f3f5c-98e6-4dba-ac8f-fc44e1a45af8} Source Files Header Files pmdk-1.13.1/src/examples/libpmemobj/btree.vcxproj0000664000000000000000000000561714435627501020513 0ustar rootroot Debug x64 Release x64 {0FB8F0FD-276C-413B-97A8-67ABE0C9043B} pmemobj 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} Application true v143 Application false v143 true ..\..\LongPath.manifest 4996;4200 CompileAsCpp libpmem.lib;libpmemobj.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies) pmdk-1.13.1/src/examples/libpmemobj/string_store_tx/0000775000000000000000000000000014435627501021221 5ustar rootrootpmdk-1.13.1/src/examples/libpmemobj/string_store_tx/.gitignore0000664000000000000000000000001614435627501023206 0ustar rootrootwriter reader pmdk-1.13.1/src/examples/libpmemobj/string_store_tx/writer.vcxproj.filters0000664000000000000000000000123414435627501025621 0ustar rootroot {48ecfa7f-5d45-4014-970b-153e338a9a2f} {d54e716e-8b7f-4b37-b2a6-63c43209d4a6} Header Files Source Files pmdk-1.13.1/src/examples/libpmemobj/string_store_tx/reader.vcxproj.filters0000664000000000000000000000123414435627501025547 0ustar rootroot {1e16629a-714b-4f1f-a688-24b50a3e7937} {c8b38ed3-5b33-4f48-ad3f-51a95613669d} Header Files Source Files pmdk-1.13.1/src/examples/libpmemobj/string_store_tx/writer.vcxproj0000664000000000000000000000544514435627501024162 0ustar rootroot Debug x64 Release x64 {7337E34A-97B0-44FC-988B-7E6AE7E0FBBF} pmemobj_tx 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} Application true v143 Application false v143 true ..\..\..\LongPath.manifest libpmem.lib;libpmemobj.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies) pmdk-1.13.1/src/examples/libpmemobj/string_store_tx/layout.h0000664000000000000000000000036414435627501022712 0ustar rootroot/* SPDX-License-Identifier: BSD-3-Clause */ /* Copyright 2015-2020, Intel Corporation */ /* * layout.h -- example from introduction part 2 */ #define LAYOUT_NAME "intro_2" #define MAX_BUF_LEN 10 struct my_root { char buf[MAX_BUF_LEN]; }; pmdk-1.13.1/src/examples/libpmemobj/string_store_tx/Makefile0000664000000000000000000000043114435627501022657 0ustar rootroot# SPDX-License-Identifier: BSD-3-Clause # Copyright 2015-2016, Intel Corporation # # examples/libpmemobj/string_store_tx/Makefile -- build the intro 2 example # PROGS = writer reader LIBS = -lpmemobj -lpmem -pthread include ../../Makefile.inc writer: writer.o reader: reader.o pmdk-1.13.1/src/examples/libpmemobj/string_store_tx/reader.c0000664000000000000000000000116714435627501022634 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2017, Intel Corporation */ /* * reader.c -- example from introduction part 2 */ #include #include #include #include "layout.h" int main(int argc, char *argv[]) { if (argc != 2) { printf("usage: %s file-name\n", argv[0]); return 1; } PMEMobjpool *pop = pmemobj_open(argv[1], LAYOUT_NAME); if (pop == NULL) { perror("pmemobj_open"); return 1; } PMEMoid root = pmemobj_root(pop, sizeof(struct my_root)); struct my_root *rootp = pmemobj_direct(root); printf("%s\n", rootp->buf); pmemobj_close(pop); return 0; } pmdk-1.13.1/src/examples/libpmemobj/string_store_tx/reader.vcxproj0000664000000000000000000000544514435627501024110 0ustar rootroot Debug x64 Release x64 {59D7A9CD-9912-40E4-96E1-8A873F777F62} pmemobj_tx 10.0.22000.0 {1baa1617-93ae-4196-8a1a-bd492fb18aef} {9e9e3d25-2139-4a5d-9200-18148ddead45} Application true v143 Application false v143 true ..\..\..\LongPath.manifest libpmem.lib;libpmemobj.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies) pmdk-1.13.1/src/examples/libpmemobj/string_store_tx/writer.c0000664000000000000000000000154214435627501022703 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2015-2017, Intel Corporation */ /* * writer.c -- example from introduction part 2 */ #include #include #include #include "layout.h" int main(int argc, char *argv[]) { if (argc != 2) { printf("usage: %s file-name\n", argv[0]); return 1; } PMEMobjpool *pop = pmemobj_create(argv[1], LAYOUT_NAME, PMEMOBJ_MIN_POOL, 0666); if (pop == NULL) { perror("pmemobj_create"); return 1; } PMEMoid root = pmemobj_root(pop, sizeof(struct my_root)); struct my_root *rootp = pmemobj_direct(root); char buf[MAX_BUF_LEN] = {0}; if (scanf("%9s", buf) == EOF) { fprintf(stderr, "EOF\n"); return 1; } TX_BEGIN(pop) { pmemobj_tx_add_range(root, 0, sizeof(struct my_root)); memcpy(rootp->buf, buf, strlen(buf)); } TX_END pmemobj_close(pop); return 0; } pmdk-1.13.1/src/examples/libpmemobj/linkedlist/0000775000000000000000000000000014435627501020126 5ustar rootrootpmdk-1.13.1/src/examples/libpmemobj/linkedlist/.gitignore0000664000000000000000000000000514435627501022111 0ustar rootrootfifo pmdk-1.13.1/src/examples/libpmemobj/linkedlist/fifo.c0000664000000000000000000000435214435627501021221 0ustar rootroot// SPDX-License-Identifier: BSD-3-Clause /* Copyright 2016, Intel Corporation */ /* * fifo.c - example of tail queue usage */ #include #include #include #include #include "pmemobj_list.h" POBJ_LAYOUT_BEGIN(list); POBJ_LAYOUT_ROOT(list, struct fifo_root); POBJ_LAYOUT_TOID(list, struct tqnode); POBJ_LAYOUT_END(list); POBJ_TAILQ_HEAD(tqueuehead, struct tqnode); struct fifo_root { struct tqueuehead head; }; struct tqnode { char data; POBJ_TAILQ_ENTRY(struct tqnode) tnd; }; static void print_help(void) { printf("usage: fifo