././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1780354067.0904033 borgstore-0.5.1/0000755000076500000240000000000015207406023012077 5ustar00twstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777583326.0 borgstore-0.5.1/AUTHORS.rst0000644000076500000240000000043415174742336013774 0ustar00twstaffNote: E-mail addresses listed here are not intended for support, please use our Github site for that. Core developers ~~~~~~~~~~~~~~~ - Thomas Waldmann Contributors ~~~~~~~~~~~~ - Alexandru Bagu - Fabian Fröhlich - Mike Mason - Nick Craig-Wood - Zhuoyun Wei ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1773360264.0 borgstore-0.5.1/LICENSE.rst0000644000076500000240000000271215154652210013717 0ustar00twstaffCopyright (C) 2026 Thomas Waldmann All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1780354067.090189 borgstore-0.5.1/PKG-INFO0000644000076500000240000000772315207406023013205 0ustar00twstaffMetadata-Version: 2.4 Name: borgstore Version: 0.5.1 Summary: key/value store Author-email: Thomas Waldmann License-Expression: BSD-3-Clause Project-URL: Homepage, https://github.com/borgbackup/borgstore Keywords: kv,key/value,store Classifier: Development Status :: 3 - Alpha Classifier: Intended Audience :: Developers Classifier: Operating System :: POSIX Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Classifier: Programming Language :: Python :: 3.12 Classifier: Programming Language :: Python :: 3.13 Classifier: Programming Language :: Python :: 3.14 Classifier: Topic :: Software Development :: Libraries Classifier: Topic :: Software Development :: Libraries :: Python Modules Requires-Python: >=3.10 Description-Content-Type: text/x-rst License-File: LICENSE.rst Provides-Extra: rest Requires-Dist: requests>=2.25.1; extra == "rest" Provides-Extra: rclone Requires-Dist: requests>=2.25.1; extra == "rclone" Provides-Extra: sftp Requires-Dist: paramiko>=1.9.1; extra == "sftp" Provides-Extra: s3 Requires-Dist: boto3; extra == "s3" Provides-Extra: none Dynamic: license-file BorgStore ========= borgstore implements a general purpose key/value store in Python. Overview -------- Keys are simple strings like `config/main` or `data/0123456789abcdef` `[str]` (config and data are namespaces here). Values are binary objects `[bytes]`. The `Store` class is the high-level API, so you can comfortably work with the kv store without caring for low-level details. The `backends` package has misc. storage backend implementations. The `server` package has a REST server implementation, complementing the REST client functionality in the `rest` backend. To actually store stuff, the REST server can use any backend internally, e.g. the `posixfs` backend. Store features -------------- - supports URLs, like `file:///srv/borgstore` or `https://myserver/path` - easy to use, high-level `Store` API: create/destroy, open/close, list, load/store, delete, move, soft delete/undelete, hash, defrag, ... - uses a backend to implement the storage - optionally uses an additional caching backend, with a configurable cache policy per namespace - name nesting / unnesting, recursive directory listing - statistics collection - latency/bandwidth emulator Backend features ---------------- - existing backends for local filesystem, sftp, REST, S3 / B2 (native) and many other cloud storage protocols via rclone - new backends are simple to implement - key validation - partial loads / range requests - stored object hashing - stored object defragmentation - quota support (only `posixfs`) - permissions checking (only `posixfs`) REST server features -------------------- - server-side permissions/quota enforcement - server-side hashsum check of transferred objects before storing - network traffic optimization by doing stuff server-side: - stored object hashing - stored object defragmentation - the REST server can internally use any backend for storage, e.g. `posixfs` - for the REST server, we provide CI tested configs for: - an nginx-based reverse proxy - systemd-based on-demand `borgstore.server` process creation State of this project --------------------- **API is still unstable and expected to change as development goes on.** **As long as the API is unstable, there will be no data migration tools, such as tools for upgrading an existing store's data to a new release.** There are tests, and they pass for the basic functionality, so some functionality is already working well. There might be missing features or optimization potential. Feedback is welcome! Many possible backends are still missing. If you want to create and support one, pull requests are welcome. Borg? ----- Please note that this code is currently **not** used by the stable release of BorgBackup (also known as "borg"), but only by Borg 2 beta 10+ and the master branch. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1779896829.0 borgstore-0.5.1/README.rst0000644000076500000240000000535615205610775013610 0ustar00twstaffBorgStore ========= borgstore implements a general purpose key/value store in Python. Overview -------- Keys are simple strings like `config/main` or `data/0123456789abcdef` `[str]` (config and data are namespaces here). Values are binary objects `[bytes]`. The `Store` class is the high-level API, so you can comfortably work with the kv store without caring for low-level details. The `backends` package has misc. storage backend implementations. The `server` package has a REST server implementation, complementing the REST client functionality in the `rest` backend. To actually store stuff, the REST server can use any backend internally, e.g. the `posixfs` backend. Store features -------------- - supports URLs, like `file:///srv/borgstore` or `https://myserver/path` - easy to use, high-level `Store` API: create/destroy, open/close, list, load/store, delete, move, soft delete/undelete, hash, defrag, ... - uses a backend to implement the storage - optionally uses an additional caching backend, with a configurable cache policy per namespace - name nesting / unnesting, recursive directory listing - statistics collection - latency/bandwidth emulator Backend features ---------------- - existing backends for local filesystem, sftp, REST, S3 / B2 (native) and many other cloud storage protocols via rclone - new backends are simple to implement - key validation - partial loads / range requests - stored object hashing - stored object defragmentation - quota support (only `posixfs`) - permissions checking (only `posixfs`) REST server features -------------------- - server-side permissions/quota enforcement - server-side hashsum check of transferred objects before storing - network traffic optimization by doing stuff server-side: - stored object hashing - stored object defragmentation - the REST server can internally use any backend for storage, e.g. `posixfs` - for the REST server, we provide CI tested configs for: - an nginx-based reverse proxy - systemd-based on-demand `borgstore.server` process creation State of this project --------------------- **API is still unstable and expected to change as development goes on.** **As long as the API is unstable, there will be no data migration tools, such as tools for upgrading an existing store's data to a new release.** There are tests, and they pass for the basic functionality, so some functionality is already working well. There might be missing features or optimization potential. Feedback is welcome! Many possible backends are still missing. If you want to create and support one, pull requests are welcome. Borg? ----- Please note that this code is currently **not** used by the stable release of BorgBackup (also known as "borg"), but only by Borg 2 beta 10+ and the master branch. ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1780354067.076833 borgstore-0.5.1/contrib/0000755000076500000240000000000015207406023013537 5ustar00twstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1780354067.0768619 borgstore-0.5.1/contrib/server/0000755000076500000240000000000015207406023015045 5ustar00twstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1780354067.0799146 borgstore-0.5.1/contrib/server/nginx-systemd/0000755000076500000240000000000015207406023017656 5ustar00twstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780352745.0 borgstore-0.5.1/contrib/server/nginx-systemd/README.md0000644000076500000240000000573315207403351021146 0ustar00twstaff# borgstore systemd + nginx setup Runs one borgstore REST server per repository, started **on demand** by systemd socket activation, with nginx terminating TLS and routing clients to the right instance by URL path. ## How it works ``` borg client → http://backup.example.com/repos/myrepo/ (nginx) → strips /repos/myrepo prefix → http://unix:/run/borgstore/myrepo.sock:/ (borgstore process) ``` Note: The example nginx configuration uses HTTP for simplicity and CI compatibility. For production use, you **must** add SSL/TLS configuration. - **`borgstore@.socket`** — systemd creates `/run/borgstore/.sock` and starts the matching service on the first incoming connection. - **`borgstore@.service`** — runs `borgstore-server-rest --socket-activation`, adopting the pre-bound socket from systemd. - **`nginx-borgstore.conf`** — a single wildcard `location` block routes any `/repos//` URL to the matching Unix socket; nginx strips the path prefix so borgstore always sees requests rooted at `/`. - **`borgstore-proxy.conf`** — shared nginx snippet (proxy headers, buffering off, timeouts) included by the wildcard location. ## Files | File | Install to | |------|------------| | `borgstore@.service` | `/etc/systemd/system/` | | `borgstore@.socket` | `/etc/systemd/system/` | | `nginx-borgstore.conf` | `/etc/nginx/sites-available/` | | `borgstore-proxy.conf` | `/etc/nginx/snippets/` | | `repo1.env.example` | `/etc/borgstore/.env` (one per repo) | ## Adding a repository **1. Create the env file** (`chmod 600`, owned by `borgstore`): ```ini # /etc/borgstore/myrepo.env BORGSTORE_BACKEND=file:///srv/borgstore/myrepo BORGSTORE_USERNAME=myuser BORGSTORE_PASSWORD=secret ``` **2. Enable the socket unit:** ```bash systemctl enable --now borgstore@myrepo.socket ``` That's it. The wildcard nginx location picks up the new repo automatically — no nginx reload needed. **3. Use with borg:** ```bash borg -r http://myuser:secret@backup.example.com/repos/myrepo/ repo-create ... ``` ## Initial deployment ```bash # Install units cp borgstore@.service borgstore@.socket /etc/systemd/system/ systemctl daemon-reload # Install nginx config cp nginx-borgstore.conf /etc/nginx/sites-available/borgstore ln -s /etc/nginx/sites-available/borgstore /etc/nginx/sites-enabled/ cp borgstore-proxy.conf /etc/nginx/snippets/ # Create the borgstore system user (if not already present) useradd --system --home /srv/borgstore --shell /usr/sbin/nologin borgstore # Add repos as above, then test nginx config nginx -t && nginx -s reload ``` ## Notes - The borgstore process is started on the first connection and stays running while connections are open. Add `TimeoutStopSec=` to the service unit to shut it down after a period of inactivity. - The socket file at `/run/borgstore/.sock` is recreated automatically after a reboot by systemd (`RuntimeDirectory=borgstore` in the service unit). - TLS is handled entirely by nginx; the borgstore process never sees HTTPS. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777233091.0 borgstore-0.5.1/contrib/server/nginx-systemd/borgstore-proxy.conf0000644000076500000240000000163415173466303023727 0ustar00twstaff# /etc/nginx/snippets/borgstore-proxy.conf # # Common proxy settings for borgstore location blocks. # Timeouts for slow filesystem operations (create, destroy, etc.) proxy_read_timeout 300s; proxy_connect_timeout 60s; proxy_send_timeout 300s; # Disable retries to ensure idempotent-but-slow operations are handled correctly by the client. proxy_next_upstream off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # Essential: borgstore streams data; buffering causes timeouts on large chunks. proxy_buffering off; proxy_request_buffering off; # Allow persistent connections, borgstore uses chunked transfers. proxy_http_version 1.1; # Pass all headers from the client to borgstore proxy_pass_request_headers on; # Disable any potential caching proxy_cache off; ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780352745.0 borgstore-0.5.1/contrib/server/nginx-systemd/borgstore@.service0000644000076500000240000000357415207403351023360 0ustar00twstaff# Template service unit for borgstore REST server with socket activation. # # The instance name (%i) is the repo identifier, which must match a # borgstore@%i.socket unit and an environment file in /etc/borgstore/. # # Deploy: # cp borgstore@.service borgstore@.socket /etc/systemd/system/ # systemctl daemon-reload # systemctl enable --now borgstore@repo1.socket # # Connect (borg client, via nginx reverse proxy — see nginx-borgstore.conf): # borg -r http://backup.example.com/repos/borg/repo1/ repo-create ... [Unit] Description=BorgStore REST server (%i) Documentation=https://github.com/borgbackup/borgstore # Start after the socket unit activates us; also restart if socket is restarted Requires=borgstore@%i.socket After=borgstore@%i.socket [Service] Type=simple User=borgstore Group=borgstore # Credentials loaded from per-instance env file. # Minimum required: BORGSTORE_BACKEND, BORGSTORE_USERNAME, BORGSTORE_PASSWORD EnvironmentFile=/etc/borgstore/%i.env ExecStart=/usr/bin/borgstore-server-rest \ --backend ${BORGSTORE_BACKEND} \ --username ${BORGSTORE_USERNAME} \ --password ${BORGSTORE_PASSWORD} \ --socket-activation # Kill the process after 30 s of idle (no open connections). # Systemd will restart it on the next connection via the socket unit. # Remove these two lines if you prefer the process to stay resident. TimeoutStopSec=30 KillMode=control-group Restart=on-failure RestartSec=5 # Harden the service NoNewPrivileges=true ProtectSystem=strict ProtectHome=true ReadWritePaths=/srv/borgstore PrivateTmp=true # Keep /run/borgstore/ alive for the lifetime of the service. # The socket unit also declares RuntimeDirectory=borgstore, which creates # the directory (with correct ownership) before the socket is bound. RuntimeDirectory=borgstore RuntimeDirectoryMode=0755 [Install] # Not enabled directly; the .socket unit triggers this. WantedBy=multi-user.target ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777233091.0 borgstore-0.5.1/contrib/server/nginx-systemd/borgstore@.socket0000644000076500000240000000212715173466303023211 0ustar00twstaff# Template socket unit for borgstore REST server with Unix domain sockets. # # Using a Unix socket per repo avoids all port management: the socket path # encodes the repo name, and nginx proxies to it directly. # # Socket path: /run/borgstore/%i.sock # RuntimeDirectory= below ensures /run/borgstore/ exists before the socket # is created (required because the socket unit starts before the service unit). # # Enable for a repo: # systemctl enable --now borgstore@repo1.socket [Unit] Description=BorgStore REST socket (%i) Documentation=https://github.com/borgbackup/borgstore [Socket] ListenStream=/run/borgstore/%i.sock SocketUser=borgstore # Use the web server's group so nginx (www-data) can connect without extra # group membership changes. Adjust if your web server runs as a different user. SocketGroup=www-data SocketMode=0660 Accept=false # Create /run/borgstore/ before binding the socket. # This must be here (not only in the service unit) because the socket # unit starts before the service unit activates. RuntimeDirectory=borgstore RuntimeDirectoryMode=0755 [Install] WantedBy=sockets.target ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777233091.0 borgstore-0.5.1/contrib/server/nginx-systemd/nginx-borgstore.conf0000644000076500000240000000244415173466303023671 0ustar00twstaff# nginx config: reverse proxy to borgstore repos. # # A regex location extracts the repo name from the URL and maps it directly # to the Unix socket at /run/borgstore/.sock. # Adding a new repo requires only the env file and socket unit. # # Client usage (borg speaks to nginx, nginx proxies to borgstore): # borg -r http://backup.example.com/repos/repo1/ repo-create ... # borg -r http://backup.example.com/repos/repo2/ repo-create ... server { listen 80; server_name backup.example.com localhost; # Borgstore chunks can be large; 0 = no limit. client_max_body_size 0; # Route /repos//... to /run/borgstore/.sock # # Named captures: # $repo — repo name (restricted to safe characters to block traversal) # $rest — remainder of path forwarded to borgstore (always starts with /) # # Example: # POST /repos/myrepo/?cmd=create # → unix:/run/borgstore/myrepo.sock URI: /?cmd=create # GET /repos/myrepo/data/ab/cdef1234 # → unix:/run/borgstore/myrepo.sock URI: /data/ab/cdef1234 # location ~ ^/repos/(?[a-zA-Z0-9_.-]+) { rewrite ^/repos/[^/]+(/.*|)$ $1 break; proxy_pass http://unix:/run/borgstore/$repo.sock; include /etc/nginx/snippets/borgstore-proxy.conf; } } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777233091.0 borgstore-0.5.1/contrib/server/nginx-systemd/repo1.env.example0000644000076500000240000000045215173466303023062 0ustar00twstaff# Per-repo environment file for borgstore@repo1.service # Copy to /etc/borgstore/repo1.env and set correct values. # Permissions: chmod 600, owned by root or the borgstore service user. BORGSTORE_BACKEND=file:///srv/borgstore/repos/borg/repo1 BORGSTORE_USERNAME=myuser BORGSTORE_PASSWORD=changeme ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1780354067.0827346 borgstore-0.5.1/docs/0000755000076500000240000000000015207406023013027 5ustar00twstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777583326.0 borgstore-0.5.1/docs/Makefile0000644000076500000240000000117215174742336014505 0ustar00twstaff# Minimal makefile for Sphinx documentation # # You can set these variables from the command line, and also # from the environment for the first two. SPHINXOPTS ?= SPHINXBUILD ?= sphinx-build SOURCEDIR = . BUILDDIR = _build # Put it first so that "make" without argument is like "make help". help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1780354067.0829868 borgstore-0.5.1/docs/_templates/0000755000076500000240000000000015207406023015164 5ustar00twstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777583326.0 borgstore-0.5.1/docs/_templates/layout.html0000644000076500000240000000047415174742336017411 0ustar00twstaff{% extends "!layout.html" %} {% block footer %}
BorgStore {{ release }} | ©{{ copyright }} {%- if show_source and has_source and sourcename %} | {{ _('Page source') }} {%- endif %}
{% endblock %} ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777583326.0 borgstore-0.5.1/docs/authors.rst0000644000076500000240000000022215174742336015257 0ustar00twstaffAuthors and License =================== Authors ------- .. include:: ../AUTHORS.rst License ------- .. _license: .. include:: ../LICENSE.rst ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1779896829.0 borgstore-0.5.1/docs/backends.rst0000644000076500000240000001352515205610775015352 0ustar00twstaffBackends ======== The backend API is rather simple; one only needs to provide some very basic operations. Existing backends are listed below; more might come in the future. See also :doc:`store_caching` for optional Store-level caching with a secondary backend. posixfs ------- Use storage on a local POSIX filesystem: - URL: ``file:///absolute/path`` - It is the caller's responsibility to convert a relative path into an absolute filesystem path. - Namespaces: directories - Values: in key-named files - Quota: tracks backend storage size and rejects ``store`` if quota is exceeded. The current usage is persisted to a hidden file in the storage directory. When quota tracking is enabled on a backend that already contains data, the server automatically scans the directories at ``open`` time (that may take a while if there are many files). That scan can be avoided by always using quotas. - Permissions: This backend can enforce a simple, test-friendly permission system and raises ``PermissionDenied`` if access is not permitted by the configuration. You provide a mapping of names (paths) to granted permission letters. Permissions apply to the exact name and all of its descendants (inheritance). If a name is not present in the mapping, its nearest ancestor is consulted, up to the empty name "" (the store root). If no mapping is provided at all, all operations are allowed. Permission letters: - ``l``: allow listing object names (directory/namespace listing) - ``r``: allow reading objects (contents) - ``w``: allow writing new objects (must not already exist) - ``W``: allow writing objects including overwriting existing objects - ``D``: allow deleting objects Operation requirements: - create(): requires ``w`` or ``W`` on the store root (``wW``) - destroy(): requires ``D`` on the store root - mkdir(name): requires ``w`` - rmdir(name): requires ``w`` or ``D`` (``wD``) - list(name): requires ``l`` - info(name): requires ``l`` (``r`` also accepted) - load(name): requires ``r`` - store(name, value): requires ``w`` for new objects, ``W`` for overwrites (``wW``) - delete(name): requires ``D`` - move(src, dst): requires ``D`` for the source and ``w``/``W`` for the destination Examples: - Read-only store (recursively): ``permissions = {"": "lr"}`` - No-delete, no-overwrite (but allow adding new items): ``permissions = {"": "lrw"}`` - Hierarchical rules: only allow listing at root, allow read/write in "dir", but only read for "dir/file": :: permissions = { "": "l", "dir": "lrw", "dir/file": "r", } To use permissions with ``Store`` and ``posixfs``, pass the mapping to Store and it will be handed to the posixfs backend: :: from borgstore import Store store = Store(url="file:///abs/path", permissions={"": "lrwWD"}) store.create() store.open() # ... store.close() Note: When using posixfs as a caching backend, it needs to use a filesystem with ``atime`` support for ``max_age`` and LRU-based ``size`` limits to work as expected. For Linux that means you must not use ``noatime`` mount option. For Windows / NTFS, atime is disabled by default and you need: :: fsutil behavior set DisableLastAccess 0 # re-enable (requires admin, reboot) sftp ---- Use storage on an SFTP server: - URL: ``sftp://user@server:port/relative/path`` (strongly recommended) For users' and admins' convenience, the mapping of the URL path to the server filesystem path depends on the server configuration (home directory, sshd/sftpd config, ...). Usually the path is relative to the user's home directory. - URL: ``sftp://user@server:port//absolute/path`` As this uses an absolute path, some things become more difficult: - A user's configuration might break if a server admin moves a user's home to a new location. - Users must know the full absolute path of the space they are permitted to use. - Namespaces: directories - Values: in key-named files - hash: runs the hexdigest computation server-side (if server supports check-file). rclone ------ Use storage on any of the many cloud providers `rclone `_ supports: - URL: ``rclone:remote:path`` — we just prefix "rclone:" and pass everything to the right of that to rclone; see: https://rclone.org/docs/#syntax-of-remote-paths - The implementation primarily depends on the specific remote. - The rclone binary path can be set via the environment variable ``RCLONE_BINARY`` (default: "rclone"). s3 -- Use storage on an S3-compliant cloud service: - URL: ``(s3|b2):[profile|(access_key_id:access_key_secret)@][scheme://hostname[:port]]/bucket/path`` The underlying backend is based on ``boto3``, so all standard boto3 authentication methods are supported: - provide a named profile (from your boto3 config), - include access key ID and secret in the URL, - or use default credentials (e.g., environment variables, IAM roles, etc.). See the `boto3 credentials documentation `_ for more details. If you're connecting to **AWS S3**, the ``[schema://hostname[:port]]`` part is optional. Bucket and path are always required. .. note:: There is a known issue with some S3-compatible services (e.g., **Backblaze B2**). If you encounter problems, try using ``b2:`` instead of ``s3:`` in the URL. - Namespaces: directories - Values: in key-named files REST (http/https) ----------------- Use a storage backend running inside a BorgStore REST server process: - URL: ``http[s]://[user:password@]host:port/path`` - Namespaces: depends on backend used by the server - Values: depends on backend used by the server - Authentication: Optional Basic Auth is supported. - hash: runs the hexdigest computation server-side. - defrag: runs the defragmentation helper server-side. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780353831.0 borgstore-0.5.1/docs/changes.rst0000644000076500000240000001622015207405447015203 0ustar00twstaffChangelog ========= Version 0.5.1 (2026-06-02) -------------------------- New features: - borgstore.server.rest: support REST http via stdio (so one can, for example, invoke borgstore-server-rest --stdio via ssh). Speaks http over stdin/stdout, stderr is used for log output. - Store now supports "rest:" backend URLs: - rest://user@host:port/relative/path - creates an ssh connection to a remote system, starts a borgstore-server-rest process there with a posixfs backend at ./relative/path on the remote system. - rest:///relative/path - creates and connects to a borgstore-server-rest process with a posixfs backend at ./relative/path on the local system, without using ssh (good for testing). Version 0.5.0 (2026-05-28) -------------------------- New features: - implement an optional cache (modes: off, mirror, writethrough; max_age and size LRU eviction need atime support), see the store_caching.rst docs - rclone: add BORGSTORE_RCLONE_DEBUG env var Other changes: - Store: remove "levels" argument, replaced by new "config" argument - Store: optimize / speed up .find method (no backend.info calls when there is only 1 level, as usual) - docs: split README, real docs, sphinx Version 0.4.1 (2026-04-26) -------------------------- New features: - quota: implement quota tracking and enforcement (posixfs), #19 - load: implement tail loading support (negative offset) - store: hashsum content verification (REST server/client), #148 - hash: item content hashing, e.g. sha256 - defrag: defragmentation helper (copies blocks from source to target items) - sftp: try to use "check-file" for SFTP server-side hashing (not supported by OpenSSH and also not tested by us; please give feedback if you use it) - REST backend (client): support sub-paths, #155, #156 - REST server: - server-side implementation of defrag and hash - --socket-activation - add systemd socket activation support, enabling on-demand per-repo startup without port management. - contrib/server/nginx-systemd/ has a nginx reverse proxy setup example that can support multiple stores at different sub-paths. - support quotas (via posixfs backend) Fixes: - rclone: fix process leak in close() - REST server: - sanitize error messages to avoid leaking absolute storage paths - harden authentication against timing attacks - slightly optimize directory listing memory usage - sftp: - fix SSH connection leak - host_config["port"] is a str - Store paramiko.SSHClient in self.ssh and ensure it is closed in _disconnect, _connect calls _disconnect on any failure during setup. Version 0.4.0 (2026-03-15) -------------------------- New features: - REST (http/https) backend, REST server, #18 Fixes: - fix permissions check, #139 - posixfs/sftp: do not raise if base_path can not be deleted, #133 - list: do not yield invalid names, #130 - posixfs, s3, sftp: URL-unquote, #129 Other changes: - add "rclone" and "rest" extras, "requests" is now an optional requirement Version 0.3.1 (2026-02-09) -------------------------- Bug fixes: - s3 URL: ensure s3 endpoint is optional Other changes: - add support for Python 3.14, remove 3.9 - backends: have separate exceptions for invalid URL and dependency missing - posixfs: better exception message if not absolute path - use SPDX license identifier, require a recent setuptools - CI: - add sftp store testing, #64 - add s3 store testing - docs: - describe the posixfs permissions system - updates, typos and grammar fixes - mention the permissions system of posixfs backend Version 0.3.0 2025-05-22 ------------------------ New features: - posixfs: add a permissions system, #105 - Store: add permissions argument (only supported by posixfs) - Store: add logging for Store ops, #104. It logs: - operation - name(s) - parameters such as deleted - size and timing Please note: - logging is done at DEBUG level, so log output is not visible with a default logger. - borgstore does not configure logging; that is the task of the application that uses borgstore. Version 0.2.0 2025-04-21 ------------------------ Breaking changes: - Store.list: changed deleted argument semantics, #83: - True: list ONLY soft-deleted items - False: list ONLY non-deleted items New features: - new s3/b2 backend that uses the boto3 library, #96 - posixfs/sftp: create missing parent directories of the base path - rclone: add a way to specify the path to the rclone binary for custom installations Bug fixes: - rclone: fix discard thread issues, #92 - rclone: check rclone regex before raising rclone-related exceptions Other changes: - posixfs: also support Windows file:/// URLs, #82 - posixfs / sftp: optimize mkdir usage, add retries, #85 - posixfs / sftp: change .precreate_dirs default to False - rclone init: use a random port instead of relying on rclone to pick one Version 0.1.0 2024-10-15 ------------------------ Breaking changes: - accepted store URLs: see README - Store: require complete levels configuration, #46 Other changes: - sftp/posixfs backends: remove ad hoc mkdir calls, #46 - optimize Sftp._mkdir, #80 - sftp backend is now optional, avoiding dependency issues on some platforms, #74. Use pip install "borgstore[sftp]" to install with the sftp backend. Version 0.0.5 2024-10-01 ------------------------ Fixes: - backend.create: only reject non-empty storage, #57 - backends.sftp: fix _mkdir edge case - backends.sftp: raise BackendDoesNotExist if base path is not found - rclone backend: - don't error on create if source directory is empty, #57 - fix hang on termination, #54 New features: - rclone backend: retry errors on load and store 3 times Other changes: - remove MStore for now, see commit 6a6fb334. - refactor Store tests, add Store.set_levels method - move types-requests to tox.ini, only needed for development Version 0.0.4 2024-09-22 ------------------------ - rclone: new backend to access any of the 100s of cloud backends that rclone supports; needs rclone >= v1.57.0. See the rclone docs for installing rclone and creating remotes. After that, borgstore will support URLs like: - rclone://remote: - rclone://remote:path - rclone:///tmp/testdir (local fs, for testing) - Store.list: give up trying to do anything with a directory's "size" - .info / .list: return st.st_size for a directory "as-is" - tests: BORGSTORE_TEST_RCLONE_URL to set rclone test URL - tests: allow BORGSTORE_TEST_*_URL in the testenv to make tox work for testing sftp, rclone, or other URLs. Version 0.0.3 2024-09-17 ------------------------ - sftp: add support for ~/.ssh/config, #37 - sftp: username is optional, #27 - load known_hosts, remove AutoAddPolicy, #39 - store: raise backend-specific exceptions, #34 - add Store.stats property, #25 - bandwidth emulation via BORGSTORE_BANDWIDTH [bit/s], #24 - latency emulation via BORGSTORE_LATENCY [us], #24 - fix demo code, also output stats - tests: BORGSTORE_TEST_SFTP_URL to set sftp test URL Version 0.0.2 2024-09-10 ------------------------ - sftp backend: use paramiko's client.posix_rename, #17 - posixfs backend: hack: accept file://relative/path, #23 - support and test on Python 3.13, #21 Version 0.0.1 2024-08-23 ------------------------ First PyPI release. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777583326.0 borgstore-0.5.1/docs/conf.py0000644000076500000240000000213115174742336014340 0ustar00twstaff# Configuration file for the Sphinx documentation builder. # # For the full list of built-in configuration values, see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html # -- Project information ----------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information from setuptools_scm import get_version project = "BorgStore" copyright = "2026, Thomas Waldmann" author = "Thomas Waldmann" release = get_version(root="..", relative_to=__file__) version = ".".join(release.split(".")[:2]) # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration extensions = [] templates_path = ["_templates"] exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] # -- Options for HTML output ------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output html_theme = "alabaster" html_static_path = ["_static"] html_show_sphinx = False ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1779896829.0 borgstore-0.5.1/docs/index.rst0000644000076500000240000000024615205610775014703 0ustar00twstaff.. highlight:: none .. include:: ../README.rst .. toctree:: :maxdepth: 2 installation store store_caching backends servers changes authors ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777583326.0 borgstore-0.5.1/docs/installation.rst0000644000076500000240000000116515174742336016302 0ustar00twstaffInstallation ============ Minimal installation -------------------- .. code-block:: bash pip install 'borgstore' Only the `posixfs` (`file://...`) backend will be available. Installation with optional dependencies --------------------------------------- To also enable other backends or other optional features, use: .. code-block:: bash pip install 'borgstore[rest,rclone,sftp,s3]' For the available optional dependencies, see ``pyproject.toml``, section ``[project.optional-dependencies]``. Running the demo ---------------- Run this to get instructions on how to run the demo:: python3 -m borgstore ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780352745.0 borgstore-0.5.1/docs/servers.rst0000644000076500000240000001055615207403351015262 0ustar00twstaffREST Server =========== BorgStore includes a simple REST server that can be used to provide remote access to any BorgStore backend. It can do some stuff server-side, which is usually not possible when using other cloud storage servers: - enforcing permissions - server rejects store operation if content hashsum does not match expected hashsum (from http header X-Content-hash-sha256) - server-side hash computation (e.g. sha256) for item content - server-side defragmentation helper (copies blocks to new items) Running the server on host:port ------------------------------- Run a server with a file: backend (for a local directory), using HTTP Basic Authentication:: borgstore-server-rest --host 127.0.0.1 --port 5618 \ --username user --password pass \ --backend file:///tmp/teststore For production deployments, consider using systemd socket activation (see contrib/server/nginx-systemd/README.md). Running the server via stdio ---------------------------- Run a server with a file: backend (for a local directory), talking http via stdin/stdout, logging via stderr: borgstore-server-rest --stdio \ --backend file:///tmp/teststore Accessing the server from a client ---------------------------------- The borgstore REST client can then access via:: http://user:pass@127.0.0.1:5618/ or (when using http over stdio): rest://user@host:port//tmp/teststore # via ssh, abs. path rest://user@host:port/teststore # via ssh, rel. path rest:////tmp/teststore # locally, without ssh, abs. path rest:///teststore # locally, without ssh, rel. path Permissions ----------- The REST server, when used with the ``posixfs`` backend, supports the same permissions system as that backend (see above). If ``--permissions`` is omitted, all operations are allowed. To restrict permissions, pass a JSON-encoded permissions mapping via ``--permissions``. Examples: Read-only access:: borgstore-server-rest --host 127.0.0.1 --port 5618 \ --username user --password pass \ --backend file:///tmp/teststore \ --permissions '{"": "lr"}' No-delete, no-overwrite (allow adding new items):: borgstore-server-rest --host 127.0.0.1 --port 5618 \ --username user --password pass \ --backend file:///tmp/teststore \ --permissions '{"": "lrw"}' Full access:: borgstore-server-rest --host 127.0.0.1 --port 5618 \ --username user --password pass \ --backend file:///tmp/teststore \ --permissions '{"": "lrwWD"}' BorgBackup shortcuts ~~~~~~~~~~~~~~~~~~~~ Instead of hand-crafting a JSON mapping, you can use a named shortcut tailored for `BorgBackup `_ repositories: ``borgbackup-all`` No permission restrictions — all operations are allowed (equivalent to omitting ``--permissions``). ``borgbackup-no-delete`` Prevent deletion and overwriting of existing objects; new objects may still be added. ``borgbackup-write-only`` Clients may store new data but cannot read existing data back (except for caches and metadata that borg needs internally). ``borgbackup-read-only`` Clients may only list and read objects. Example — restrict a backup server to no-delete access: .. code-block:: bash borgstore-server-rest --host 127.0.0.1 --port 5618 \ --username user --password pass \ --backend file:///home/user/repos/repo1 \ --permissions borgbackup-no-delete Custom JSON permissions ~~~~~~~~~~~~~~~~~~~~~~~ You can also pass an arbitrary JSON-encoded permissions mapping directly. Hierarchical rules (list-only at root, read/write in ``data/``):: borgstore-server-rest --host 127.0.0.1 --port 5618 \ --username user --password pass \ --backend file:///tmp/teststore \ --permissions '{"": "l", "data": "lrw"}' Quota ----- The REST server, when used with the ``file:`` backend, optionally supports quota tracking and enforcement. Use the ``--quota`` argument to set a maximum storage size in bytes (default is no quota tracking and enforcement). When the quota is exceeded, ``store`` operations are rejected with HTTP 507 (Insufficient Storage). Example — limit storage to 1 GiB: .. code-block:: bash borgstore-server-rest --host 127.0.0.1 --port 5618 \ --username user --password pass \ --backend file:///tmp/teststore \ --quota 1073741824 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1779896829.0 borgstore-0.5.1/docs/store.rst0000644000076500000240000001166015205610775014732 0ustar00twstaffStore ===== Overview -------- The high-level Store API implementation transparently deals with nesting and soft deletion, so the caller doesn't need to care much about that, and the backend API can be much simpler: - create/destroy: initialize or remove the whole store. - list: flat list of the items in the given namespace (by default, only non-deleted items; optionally, only soft-deleted items). - store: write a new item into the store (providing its key/value pair). - load: read a value from the store (given its key); partial loads specifying an offset and/or size are supported. - info: get information about an item via its key (exists, size, ...). - hash: computes the hexdigest for the content of an item (given its key). - delete: immediately remove an item from the store (given its key). - move: implements renaming, soft delete/undelete, and moving to the current nesting level. - defrag: general purpose defragmentation helper (copies blocks to new items) - quota: return quota limit and usage (-1 if quotas not enabled or not supported) - stats: API call counters, time spent in API methods, data volume/throughput. - latency/bandwidth emulator: see :ref:`store-latency-bandwidth-emulator`. Store operations (and per-op timing and volume) are logged at DEBUG log level. See also :doc:`store_caching` for optional Store-level caching with a secondary backend. .. _store-latency-bandwidth-emulator: Latency and bandwidth emulator ------------------------------ The Store can emulate slower backend behavior using environment variables: - ``BORGSTORE_LATENCY``: per-primary-call latency in microseconds (``[us]``). - ``BORGSTORE_BANDWIDTH``: primary-call bandwidth limit in bits per second (``[bit/s]``). Current behavior with Store caching enabled: - Emulation applies to **primary backend** operations. - Emulation does **not** apply to **cache backend** operations. This means: - On cache miss paths (for example writethrough/mirror reads that load from the primary backend), emulation affects the primary backend calls. - On cache hit paths, cached reads avoid primary backend load operations and therefore do not incur emulated bandwidth delay for the cache backend read. - Name resolution for Store operations still uses primary backend lookups, therefore configured latency can still be visible even when data comes from cache. Keys ---- A key (str) can look like: - 0123456789abcdef... (usually a long, hex-encoded hash value) - Any other pure ASCII string without '/', '..', or spaces. Namespaces ---------- To keep things separate, keys should be prefixed with a namespace, such as: - config/settings - meta/0123456789abcdef... - data/0123456789abcdef... Please note: 1. You should always use namespaces. 2. Nested namespaces like namespace1/namespace2/key are not supported. 3. The code can work without a namespace (empty namespace ""), but then you can't add another namespace later, because that would create nested namespaces. Values ------ Values can be any arbitrary binary data (bytes). Automatic Nesting ----------------- For the Store user, items have names such as: - namespace/0123456789abcdef... - namespace/abcdef0123456789... If there are very many items in the namespace, this could lead to scalability issues in the backend. The Store implementation therefore offers transparent nesting, so that internally the backend API is called with names such as: - namespace/01/23/45/0123456789abcdef... - namespace/ab/cd/ef/abcdef0123456789... The nesting depth can be configured from 0 (= no nesting) to N levels and there can be different nesting configurations depending on the namespace. The Store supports operating at different nesting levels in the same namespace at the same time. When using nesting depth > 0, the backends assume that keys are hashes (contain hex digits) because some backends pre-create the nesting directories at initialization time to optimize backend performance. Soft deletion ------------- To soft-delete an item (so its value can still be read or it can be undeleted), the store just renames the item, appending ".del" to its name. Undelete reverses this by removing the ".del" suffix from the name. Some store operations provide a boolean flag "deleted" to control whether they consider soft-deleted items. Scalability ----------- - Count of key/value pairs stored in a namespace: automatic nesting is provided for keys to address common scalability issues. - Key size: there are no special provisions for extremely long keys (e.g., exceeding backend limitations). Usually this is not a problem, though. - Value size: there are no special provisions for dealing with large value sizes (e.g., more than available memory, more than backend storage limitations, etc.). If one deals with very large values, one usually cuts them into chunks before storing them in the store. - Partial loads improve performance by avoiding a full load if only part of the value is needed (e.g., a header with metadata). ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1779961472.0 borgstore-0.5.1/docs/store_caching.rst0000644000076500000240000001064715206007200016372 0ustar00twstaffStore caching ============= The ``Store`` can optionally use a second backend as a local cache for selected namespaces, which is especially useful when the primary backend is remote, slower or otherwise more "expensive" than the cache. Configuration ------------- - ``cache_url`` or ``cache_backend``: where cached data is stored - ``config``: mapping of namespace to its configuration dict, containing nesting levels and cache policy settings. Each namespace configuration dictionary can have: - ``levels``: a required list of integers specifying nesting levels. - ``cache``: optional cache mode, accepting ``CacheMode`` values or string aliases: - ``CacheMode.C_OFF`` or ``"off"``: bypass cache completely (default). - ``CacheMode.C_MIRROR`` or ``"mirror"``: always read from primary backend, but update the cache after successful primary backend reads and writes. - ``CacheMode.C_WRITETHROUGH`` or ``"writethrough"``: read-through + write-through. For now, only content-hash addressed namespaces should use this mode. - ``max_age``: optional maximum age expressed in seconds since last access. The default is ``None`` (no age limit). - ``size``: optional maximum size in bytes. It sets a per-namespace cache size budget enforced by evicting least-recently-used items until the namespace total size is within the configured budget. Example:: from borgstore.store import Store, CacheMode store = Store( url="sftp://user@host/repo", config={ "data": { "levels": [2], "cache": "writethrough", "max_age": 3600, "size": 4 * 1024**3, }, "meta": { "levels": [1], "cache": CacheMode.C_MIRROR, }, }, cache_url="file:///home/user/.cache/borgstore/repo", ) Behavior -------- - Cache keys are identical to primary backend keys (same nesting). - Soft-deleted items are cached under the same ``.del`` name as primary. - Soft delete/undelete renames cache entries as well. - On ``Store.open()`` and ``Store.close()``, cache-enabled namespaces are scanned to clean up the cache. Cleanup order per namespace is: 1. remove expired cache objects when ``max_age`` is configured, 2. if ``size`` is configured, evict the least-recently-used remaining items until the namespace total size is ``<= size``. Expired entries are always removed first, even if total size is already below the ``size`` limit. - Cache failures are non-fatal and logged as warnings. Manual Cache Invalidation ------------------------- If you need to programmatically clear or invalidate parts of the cache (for example, to resolve stale objects after primary backend deletes by other clients, or if cache corruption is suspected), you can use the ``cache_invalidate`` method: - To invalidate a single item:: store.cache_invalidate("data/00000000") - To invalidate all cached items in a specific namespace (e.g. ``"data/"``):: store.cache_invalidate("data/") - To invalidate all cached items across all configured namespaces, pass ``ROOTNS``:: from borgstore.constants import ROOTNS store.cache_invalidate(ROOTNS) Limitations ----------- - Eviction by ``max_age`` or ``size`` is open-time and close-time only (``Store.open()`` / ``Store.close()``), not continuous during ``store()``/``load()`` operations. - No proactive cache validation/revalidation. - If an object is deleted in the primary backend by another client, the local cache will still have a stale object. - ``max_age`` and LRU-by-``size`` depend on backend ``ItemInfo.atime`` support. If ``atime`` is 0 (not implemented): - using ``max_age`` would empty the cache on ``Store.open()`` or ``Store.close()`` - using ``size`` would not work in LRU order, because order can't be determined - If a partial range ``load`` call for an object in a cached namespace causes a cache miss, the full object will be read from the primary backend and the cache will be populated with the full object. Statistics ---------- ``Store.stats`` includes cache counters: - ``backend_load_volume`` - ``backend_store_volume`` - ``backend_load_calls`` - ``backend_store_calls`` - ``backend_delete_calls`` - ``cache_disabled`` - ``cache_hits`` - ``cache_misses`` - ``cache_hit_ratio`` - ``cache_errors`` - ``cache_load_volume`` - ``cache_store_volume`` - ``cache_load_calls`` - ``cache_store_calls`` - ``cache_delete_calls`` ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780352745.0 borgstore-0.5.1/pyproject.toml0000644000076500000240000000447015207403351015021 0ustar00twstaff[project] name = "borgstore" dynamic = ["version"] authors = [{name="Thomas Waldmann", email="tw@waldmann-edv.de"}, ] description = "key/value store" readme = "README.rst" keywords = ["kv", "key/value", "store"] classifiers = [ "Development Status :: 3 - Alpha", "Intended Audience :: Developers", "Operating System :: POSIX", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.14", "Topic :: Software Development :: Libraries", "Topic :: Software Development :: Libraries :: Python Modules", ] license = "BSD-3-Clause" license-files = ["LICENSE.rst"] requires-python = ">=3.10" dependencies = [ ] [project.optional-dependencies] rest = [ "requests >= 2.25.1", ] rclone = [ "requests >= 2.25.1", ] sftp = [ "paramiko >= 1.9.1", # 1.9.1+ supports multiple IdentityKey entries in .ssh/config ] s3 = [ "boto3", ] none = [] [project.scripts] borgstore-server-rest = "borgstore.server.rest:main" [project.urls] Homepage = "https://github.com/borgbackup/borgstore" [build-system] requires = ["setuptools>=78.1.1", "setuptools_scm[toml]>=6.2"] build-backend = "setuptools.build_meta" [tool.setuptools_scm] # make sure we have the same versioning scheme with all setuptools_scm versions, to avoid different autogenerated files # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1015052 # https://github.com/borgbackup/borg/issues/6875 write_to = "src/borgstore/_version.py" write_to_template = "__version__ = version = {version!r}\n" [tool.black] line-length = 120 skip-magic-trailing-comma = true target-version = ['py310'] [tool.pytest.ini_options] minversion = "6.0" testpaths = ["tests"] [tool.flake8] # Ignoring E203 due to https://github.com/PyCQA/pycodestyle/issues/373 ignore = ['E226', 'W503', 'E203', 'E402'] max_line_length = 120 exclude = ['build', 'dist', '.git', '.idea', '.mypy_cache', '.tox'] [tool.ruff] line-length = 120 [tool.ruff.lint] # E402: Module level import not at top of file ignore = ["E402"] [tool.mypy] python_version = '3.10' strict_optional = false local_partial_types = true show_error_codes = true files = 'src/borgstore/**/*.py' ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1780354067.090566 borgstore-0.5.1/setup.cfg0000644000076500000240000000004615207406023013720 0ustar00twstaff[egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1780354067.0771768 borgstore-0.5.1/src/0000755000076500000240000000000015207406023012666 5ustar00twstaff././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1780354067.0840807 borgstore-0.5.1/src/borgstore/0000755000076500000240000000000015207406023014674 5ustar00twstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1757407943.0 borgstore-0.5.1/src/borgstore/__init__.py0000644000076500000240000000013215057765307017021 0ustar00twstaff""" BorgStore: a key/value store. """ from ._version import __version__, version # noqa ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1779961472.0 borgstore-0.5.1/src/borgstore/__main__.py0000644000076500000240000000454515206007200016767 0ustar00twstaff""" Demo for BorgStore ================== Usage: python -m borgstore For example: python -m borgstore file:///tmp/borgstore_storage Please be careful: the given storage will be created, used, and **completely deleted**! """ def run_demo(storage_url): from .store import Store def id_key(data: bytes): from hashlib import new h = new("sha256", data) return f"data/{h.hexdigest()}" config = { "config/": {"levels": [0]}, # no nesting needed/wanted for the configs "data/": {"levels": [2]}, # 2 nesting levels wanted for the data } store = Store(url=storage_url, config=config) try: store.create() except FileExistsError: # Currently, we only have file:// storages, so this should be fine. print("Error: do not specify an existing directory.") return with store: print("Writing 2 items to config namespace...") settings1_key = "config/settings1" store.store(settings1_key, b"value1 = 42") settings2_key = "config/settings2" store.store(settings2_key, b"value2 = 23") print(f"Listing config namespace contents: {list(store.list('config'))}") settings1_value = store.load(settings1_key) print(f"Loaded from store: {settings1_key}: {settings1_value.decode()}") settings2_value = store.load(settings2_key) print(f"Loaded from store: {settings2_key}: {settings2_value.decode()}") print("Writing 2 items to data namespace...") data1 = b"some arbitrary binary data." key1 = id_key(data1) store.store(key1, data1) data2 = b"more arbitrary binary data. " * 2 key2 = id_key(data2) store.store(key2, data2) print(f"Soft-deleting item {key2} ...") store.move(key2, delete=True) print(f"Listing data namespace contents: {list(store.list('data', deleted=False))}") print(f"Listing data namespace contents (only deleted): {list(store.list('data', deleted=True))}") print(f"Stats: {store.stats}") answer = input("After you've inspected the storage, enter DESTROY to destroy the storage; anything else aborts: ") if answer == "DESTROY": store.destroy() if __name__ == "__main__": import sys if len(sys.argv) == 2: run_demo(sys.argv[1]) else: print(__doc__) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780354067.0 borgstore-0.5.1/src/borgstore/_version.py0000644000076500000240000000004015207406023017064 0ustar00twstaff__version__ = version = '0.5.1' ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1780354067.087864 borgstore-0.5.1/src/borgstore/backends/0000755000076500000240000000000015207406023016446 5ustar00twstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1757407943.0 borgstore-0.5.1/src/borgstore/backends/__init__.py0000644000076500000240000000013615057765307020577 0ustar00twstaff""" Package containing backend implementations. See borgstore.backends._base for details. """ ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1779896829.0 borgstore-0.5.1/src/borgstore/backends/_base.py0000644000076500000240000001605615205610775020112 0ustar00twstaff""" Base class and type definitions for all backend implementations in this package. Docs that are not backend-specific are also found here. """ import hashlib from abc import ABC, abstractmethod from collections import namedtuple from typing import Iterator from ..constants import MAX_NAME_LENGTH, TMP_SUFFIX, HID_SUFFIX # atime is the last read access UNIX timestamp [s] or 0 if not implemented ItemInfo = namedtuple("ItemInfo", "name exists size directory atime", defaults=(0,)) def validate_name(name): """Validate a backend key/name.""" # this is used before an object is accepted for storage and # it is also used before a name is returned by list method. # no crap in, no crap out (even if it is not from us). if not isinstance(name, str): raise TypeError(f"name must be str, but got: {type(name)}") # name must not be too long if len(name) > MAX_NAME_LENGTH: raise ValueError(f"name is too long (max: {MAX_NAME_LENGTH}): {name}") # avoid encoding issues try: name.encode("ascii") except UnicodeEncodeError: raise ValueError(f"name must encode to plain ascii, but failed with: {name}") # security: name must be relative - can be foo or foo/bar/baz, but must never be /foo or ../foo if name.startswith("/") or name.endswith("/") or ".." in name: raise ValueError(f"name must be relative and not contain '..': {name}") # names used here always have '/' as separator, never '\' - # this is to avoid confusion in case this is ported to e.g. Windows. # also: no blanks - simplifies usage via CLI / shell. if "\\" in name or " " in name: raise ValueError(f"name must not contain backslashes or blanks: {name}") # name must be lowercase - this is to avoid troubles in case this is ported to a non-case-sensitive backend. # also, guess we want to avoid that a key "config" would address a different item than a key "CONFIG" or # a key "1234CAFE5678BABE" would address a different item than a key "1234cafe5678babe". if name != name.lower(): raise ValueError(f"name must be lowercase, but got: {name}") if name.endswith(TMP_SUFFIX): # TMP_SUFFIX is used for temporary files internally, e.g. while files are uploading. raise ValueError(f"name must not end with {TMP_SUFFIX}, but got: {name}") if name.endswith(HID_SUFFIX): # HID_SUFFIX is used for hidden internal files, not accessible by users. raise ValueError(f"name must not end with {HID_SUFFIX}, but got: {name}") class BackendBase(ABC): # a backend can request all directories to be pre-created once at backend creation (initialization) time. # for some backends this will optimize the performance of store and move operation, because they won't # have to care for ad-hoc directory creation for every store or move call. of course, create will take # significantly longer, especially if nesting on levels > 1 is used. # otoh, for some backends this might be completely pointless, e.g. if mkdir is a NOP (is ignored). # for the unit tests, precreate_dirs should be set to False, otherwise they get slowed down too much. # for interactive usage, precreate_dirs = False is often the less annoying, quicker option. # code in .store and .move methods can deal with mkdir in the exception handler, after first just # assuming that the directory is usually already there. precreate_dirs: bool = False @abstractmethod def create(self): """create (initialize) a backend storage""" @abstractmethod def destroy(self): """completely remove the backend storage (and its contents)""" def __enter__(self): self.open() return self def __exit__(self, exc_type, exc_val, exc_tb): self.close() return False @abstractmethod def open(self): """open (start using) a backend storage""" @abstractmethod def close(self): """close (stop using) a backend storage""" @abstractmethod def mkdir(self, name: str) -> None: """create directory/namespace """ @abstractmethod def rmdir(self, name: str) -> None: """remove directory/namespace """ @abstractmethod def info(self, name) -> ItemInfo: """return information about """ @abstractmethod def load(self, name: str, *, size=None, offset=0) -> bytes: """load value from If offset is negative, it is counted from the end of the file. If size is None, the whole object starting at offset is loaded. """ @abstractmethod def store(self, name: str, value: bytes) -> None: """store into """ @abstractmethod def delete(self, name: str) -> None: """delete """ @abstractmethod def move(self, curr_name: str, new_name: str) -> None: """rename curr_name to new_name (overwrite target)""" def defrag(self, sources, *, target=None, algorithm=None, namespace=None, levels=0) -> str: """ Similar to the higher-level Store.defrag method, with these differences: - source and target item names are with namespace. - if levels > 0, source and target item names are nested. Returns the target item name. """ # default implementation: slow, but works for all backends. # might be overridden for performance. from ..utils.nesting import nest data = b"".join(self.load(source, offset=offset, size=size) for source, offset, size in sources) if target is None: if algorithm is None: raise ValueError("Either target or algorithm must be given for defrag") try: h = hashlib.new(algorithm) except (ValueError, TypeError): raise ValueError(f"Unsupported hash algorithm: {algorithm}") h.update(data) target = h.hexdigest() if namespace: target = namespace.rstrip("/") + "/" + target if levels: target = nest(target, levels) self.store(target, data) return target def hash(self, name: str, algorithm: str = "sha256") -> str: """compute full-file hex digest of content using """ # default implementation: slow, but works for all backends. # might be overridden for performance. try: h = hashlib.new(algorithm) except ValueError: raise ValueError(f"Unsupported hash algorithm: {algorithm}") from None h.update(self.load(name)) return h.hexdigest() def quota(self) -> dict: """Return quota information: limit and usage in bytes. -1 means not set / not tracked.""" return dict(limit=-1, usage=-1) @abstractmethod def list(self, name: str) -> Iterator[ItemInfo]: """list the contents of , non-recursively. Does not yield TMP_SUFFIX items - usually they are either not finished uploading or they are leftover crap from aborted uploads. The yielded ItemInfos are sorted alphabetically by name. """ ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1774648089.0 borgstore-0.5.1/src/borgstore/backends/_utils.py0000644000076500000240000000407115161575431020332 0ustar00twstaff""" Utilities for backend implementations. """ from typing import Tuple, Optional def make_range_header(offset: int, size: Optional[int] = None, total_size: Optional[int] = None) -> Optional[str]: """ Generate a standards compliant HTTP Range header. :param offset: offset in bytes. If negative, it is counted from the end of the file. :param size: number of bytes to load. If None, load until the end of the file. :param total_size: total size of the file. Required if offset < 0 and size is not None. :return: Range header value (e.g., "bytes=0-99") or None if no Range header is needed. """ if offset < 0: if size is None: return f"bytes={offset}" else: if total_size is None: raise ValueError("total_size is required for negative offset with a specific size") start = total_size + offset return f"bytes={start}-{start + size - 1}" else: if size is None: return f"bytes={offset}-" if offset > 0 else None else: return f"bytes={offset}-{offset + size - 1}" def parse_range_header(range_header: str) -> Tuple[int, Optional[int]]: """ Parse a standards compliant HTTP Range header. Only supports "bytes" unit and single range specs. :param range_header: Range header value (e.g., "bytes=0-99", "bytes=100-", "bytes=-500"). :return: A tuple (offset, size). offset is negative for suffix ranges. """ if not range_header or not range_header.startswith("bytes="): return 0, None try: range_val = range_header.split("=")[1] if range_val.startswith("-"): # bytes=-SUFFIX return int(range_val), None elif "-" in range_val: # bytes=OFFSET- or bytes=OFFSET-END start_str, end_str = range_val.split("-") offset = int(start_str) size = None if end_str: size = int(end_str) - offset + 1 return offset, size except (ValueError, IndexError): pass return 0, None ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1775307991.0 borgstore-0.5.1/src/borgstore/backends/errors.py0000644000076500000240000000171615164206327020350 0ustar00twstaff""" Generic exception classes used by all backends. """ class BackendError(Exception): """Base class for exceptions in this module.""" class BackendURLInvalid(BackendError): """Raised when trying to create a store using an invalid backend URL.""" class NoBackendGiven(BackendError): """Raised when trying to create a store and giving neither a backend nor a URL.""" class BackendAlreadyExists(BackendError): """Raised when a backend already exists.""" class BackendDoesNotExist(BackendError): """Raised when a backend does not exist.""" class BackendMustNotBeOpen(BackendError): """Backend must not be open.""" class BackendMustBeOpen(BackendError): """Backend must be open.""" class ObjectNotFound(BackendError): """Object not found.""" class PermissionDenied(BackendError): """Permission denied for the requested operation.""" class QuotaExceeded(BackendError): """Quota exceeded for the requested operation.""" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1779896829.0 borgstore-0.5.1/src/borgstore/backends/posixfs.py0000644000076500000240000004656615205610775020545 0ustar00twstaff""" Filesystem-based backend implementation - uses files in directories below a base path. """ import hashlib import os import re import sys import time from urllib.parse import unquote from pathlib import Path import shutil import stat import tempfile import types is_win32 = sys.platform == "win32" fcntl: types.ModuleType | None try: import fcntl except ImportError: fcntl = None # not available on Windows from ._base import BackendBase, ItemInfo, validate_name from .errors import BackendError, BackendAlreadyExists, BackendDoesNotExist, BackendMustNotBeOpen, BackendMustBeOpen from .errors import ObjectNotFound, PermissionDenied, QuotaExceeded from ..constants import TMP_SUFFIX, QUOTA_STORE_NAME, QUOTA_PERSIST_DELTA, QUOTA_PERSIST_INTERVAL def get_file_backend(url, permissions=None, quota=None): # file:///absolute/path # notes: # - we only support **local** fs **absolute** paths. # - there is no such thing as a "relative path" local fs file: URL # - the general URL syntax is proto://host/path # - // introduces the host part. it is empty here, meaning localhost / local fs. # - the third slash is NOT optional, it is the start of an absolute path as well # as the separator between the host and the path part. # - the caller is responsible to give an absolute path. # - Windows: see: https://en.wikipedia.org/wiki/File_URI_scheme windows_file_regex = r""" file:// # only empty host part is supported. / # 3rd slash is separator ONLY, not part of the path. (?P([a-zA-Z]:/.*)) # path must be an absolute path. """ file_regex = r""" file:// # only empty host part is supported. (?P(/.*)) # path must be an absolute path. 3rd slash is separator AND part of the path. """ # the path or drive_and_path could be URL-quoted and thus must be URL-unquoted if sys.platform in ("win32", "msys", "cygwin"): m = re.match(windows_file_regex, url, re.VERBOSE) if m: return PosixFS(path=unquote(m["drive_and_path"]), permissions=permissions, quota=quota) m = re.match(file_regex, url, re.VERBOSE) if m: return PosixFS(path=unquote(m["path"]), permissions=permissions, quota=quota) class PosixFS(BackendBase): # PosixFS implementation supports precreate = True as well as = False. precreate_dirs: bool = False def __init__(self, path, *, do_fsync=False, permissions=None, quota=None): self.base_path = Path(path) if not self.base_path.is_absolute(): raise BackendError(f"path must be an absolute path: {path}") self.opened = False self.do_fsync = do_fsync # False = 26x faster, see #10 self.permissions = permissions or {} # name [str] -> granted_permissions [str] self.quota_limit = quota # maximum allowed storage size in bytes, None means unlimited self._quota_use = 0 # current tracked storage usage in bytes self._quota_use_persisted = 0 # last persisted value self._quota_last_persist_time = 0.0 # monotonic time of last persist def _check_permission(self, name, required_permissions): """ Check in the self.permissions mapping if one of the required_permissions is granted for the given name or its parents. Permission characters: - l: allow listing object names ("namespace/directory listing") - r: allow reading objects (contents) - w: allow writing NEW objects (must not already exist) - W: allow writing objects (also overwrite existing objects) - D: allow deleting objects Move requires "D" (src) and "wW" (dst). Moves are used by the Store for soft-deletion/undeletion, level changes and generic renames. If permissions are granted for a directory like "foo", they also apply to objects below that directory, like "foo/bar". """ assert set(required_permissions).issubset("lrwWD") if not self.permissions: # If no permissions dict is provided, allow all operations. return # Check permissions, starting from full name (full path) going up to the root. path_parts = name.split("/") for i in range(len(path_parts), -1, -1): # i: LEN .. 0 path = "/".join(path_parts[:i]) # path: full path .. root if path in self.permissions: granted_permissions = self.permissions[path] # Check if any of the required permissions is present. if set(required_permissions) & set(granted_permissions): return # Permission granted # If path was found in permissions but didn't have required permission, we stop here # (more specific longer-path entry takes precedence over shorter-path entry). break # If we get here, none of the required permissions was found raise PermissionDenied(f"One of permissions '{required_permissions}' required for '{name}'") def create(self): if self.opened: raise BackendMustNotBeOpen() self._check_permission("", "wW") # we accept an already existing empty directory and we also optionally create # any missing parent dirs. the latter is important for repository hosters that # only offer limited access to their storage (e.g. only via borg/borgstore). # also, it is simpler than requiring users to create parent dirs separately. self.base_path.mkdir(exist_ok=True, parents=True) # avoid that users create a mess by using non-empty directories: contents = list(self.base_path.iterdir()) if contents: raise BackendAlreadyExists(f"posixfs storage base path is not empty: {self.base_path}") def destroy(self): if self.opened: raise BackendMustNotBeOpen() self._check_permission("", "D") if not self.base_path.exists(): raise BackendDoesNotExist(f"posixfs storage base path does not exist: {self.base_path}") def onexc(func, path, exc): # for rmtree, this is called if it can't remove a file or directory. # usually, this is because of missing permissions. if path != os.fspath(self.base_path): raise exc # do not raise if we can't remove the base path directory. # .create accepts an already existing base path, thus # .destroy may leave an existing base path behind. def onerror(func, path, excinfo): onexc(func, path, excinfo[1]) kw = {"onexc": onexc} if sys.version_info >= (3, 12) else {"onerror": onerror} shutil.rmtree(os.fspath(self.base_path), **kw) def open(self): if self.opened: raise BackendMustNotBeOpen() if not self.base_path.is_dir(): raise BackendDoesNotExist( f"posixfs storage base path does not exist or is not a directory: {self.base_path}" ) if self.quota_limit is not None: self._quota_persist(0) else: self._quota_delete() self.opened = True def close(self): if not self.opened: raise BackendMustBeOpen() if self.quota_limit is not None: self._quota_update(0, force=True) self.opened = False def _validate_join(self, name): validate_name(name) return self.base_path / name def mkdir(self, name): if not self.opened: raise BackendMustBeOpen() path = self._validate_join(name) # spamming a store with lots of random empty dirs == DoS, thus require w. self._check_permission(name, "w") path.mkdir(parents=True, exist_ok=True) def rmdir(self, name): if not self.opened: raise BackendMustBeOpen() path = self._validate_join(name) # path.rmdir only removes empty directories, thus no data can be lost. # thus, a granted "w" is already good enough, "D" is also ok. self._check_permission(name, "wD") try: path.rmdir() except FileNotFoundError: raise ObjectNotFound(name) from None def info(self, name): if not self.opened: raise BackendMustBeOpen() path = self._validate_join(name) # we do not read object content, so a granted "l" is enough, "r" is also ok. self._check_permission(name, "lr") try: st = path.stat() except FileNotFoundError: return ItemInfo(name=path.name, exists=False, directory=False, size=0) else: is_dir = stat.S_ISDIR(st.st_mode) return ItemInfo(name=path.name, exists=True, directory=is_dir, size=st.st_size, atime=st.st_atime) def load(self, name, *, size=None, offset=0): if not self.opened: raise BackendMustBeOpen() path = self._validate_join(name) self._check_permission(name, "r") try: with path.open("rb") as f: if offset != 0: f.seek(offset, os.SEEK_SET if offset >= 0 else os.SEEK_END) return f.read(-1 if size is None else size) except FileNotFoundError: raise ObjectNotFound(name) from None def _write_to_tempfile(self, path, value, suffix=TMP_SUFFIX, do_fsync=False): with tempfile.NamedTemporaryFile(suffix=suffix, dir=path, delete=False) as f: f.write(value) if do_fsync: f.flush() os.fsync(f.fileno()) tmp_path = Path(f.name) return tmp_path def store(self, name, value): if not self.opened: raise BackendMustBeOpen() path = self._validate_join(name) overwrite = path.exists() self._check_permission(name, "W" if overwrite else "wW") if self.quota_limit is not None: old_size = path.stat().st_size if overwrite else 0 new_size = len(value) delta = new_size - old_size if self._quota_use + delta > self.quota_limit: raise QuotaExceeded(f"Quota exceeded: {self._quota_use + delta} > {self.quota_limit}") tmp_dir = path.parent # write to a differently named temp file in same directory first, # so the store never sees partially written data. try: # try to do it quickly, not doing the mkdir. fs ops might be slow, esp. on network fs (latency). # this will frequently succeed, because the dir is already there. tmp_path = self._write_to_tempfile(tmp_dir, value, do_fsync=self.do_fsync) except FileNotFoundError: # retry, create potentially missing dirs first. this covers these cases: # - either the dirs were not precreated # - a previously existing directory was "lost" in the filesystem tmp_dir.mkdir(parents=True, exist_ok=True) tmp_path = self._write_to_tempfile(tmp_dir, value, do_fsync=self.do_fsync) # all written and synced to disk, rename it to the final name: try: tmp_path.replace(path) except OSError: tmp_path.unlink() raise if self.quota_limit is not None: self._quota_update(delta) def delete(self, name): if not self.opened: raise BackendMustBeOpen() path = self._validate_join(name) self._check_permission(name, "D") try: if self.quota_limit is not None: size = path.stat().st_size path.unlink() except FileNotFoundError: raise ObjectNotFound(name) from None if self.quota_limit is not None: self._quota_update(-size) def move(self, curr_name, new_name): def _rename_to_new_name(): curr_path.rename(new_path) if not self.opened: raise BackendMustBeOpen() curr_path = self._validate_join(curr_name) new_path = self._validate_join(new_name) # random moves could do a lot of harm in the store: # not finding an object anymore is similar to having it deleted. # also, the source object vanishes under its original name, thus we want D for the source. # as the move might replace the destination, we want W or wW for the destination. # move is also used for soft-deletion by the Store, that also hints to using D for the source. self._check_permission(curr_name, "D") self._check_permission(new_name, "W" if new_path.exists() else "wW") try: # try to do it quickly, not doing the mkdir. fs ops might be slow, esp. on network fs (latency). # this will frequently succeed, because the dir is already there. _rename_to_new_name() except FileNotFoundError: # retry, create potentially missing dirs first. this covers these cases: # - either the dirs were not precreated # - a previously existing directory was "lost" in the filesystem new_path.parent.mkdir(parents=True, exist_ok=True) try: _rename_to_new_name() except FileNotFoundError: raise ObjectNotFound(curr_name) from None def defrag(self, sources, *, target=None, algorithm=None, namespace=None, levels=0) -> str: if not self.opened: raise BackendMustBeOpen() # check all permissions before doing anything prefix = namespace.rstrip("/") + "/" if namespace else "" # if target is not given, an item named like content-hash is created in same namespace. check_target = target if target else prefix + "01234567" self._check_permission(check_target, "W") names = [prefix + source[0] for source in sources] for name in names: self._check_permission(name, "r") return super().defrag(sources, target=target, algorithm=algorithm, namespace=namespace, levels=levels) def hash(self, name: str, algorithm: str = "sha256") -> str: if not self.opened: raise BackendMustBeOpen() path = self._validate_join(name) self._check_permission(name, "r") try: h = hashlib.new(algorithm) except ValueError: raise ValueError(f"Unsupported hash algorithm: {algorithm}") from None try: with path.open("rb") as f: while True: data = f.read(1024 * 1024) if not data: break h.update(data) except FileNotFoundError: raise ObjectNotFound(name) from None return h.hexdigest() def list(self, name): if not self.opened: raise BackendMustBeOpen() path = self._validate_join(name) self._check_permission(name, "l") try: paths = sorted(path.iterdir()) except FileNotFoundError: raise ObjectNotFound(name) from None else: for p in paths: try: validate_name(p.name) except ValueError: pass # that file is likely not from us or is still uploading else: try: st = p.stat() except FileNotFoundError: pass else: is_dir = stat.S_ISDIR(st.st_mode) yield ItemInfo(name=p.name, exists=True, size=st.st_size, directory=is_dir, atime=st.st_atime) def quota(self) -> dict: """Return quota information: limit and usage in bytes. -1 means not set / not tracked.""" if self.quota_limit is None: return dict(limit=-1, usage=-1) return dict(limit=self.quota_limit, usage=self._quota_use) def _quota_path(self): return self.base_path / QUOTA_STORE_NAME def _quota_scan(self, path, skips): """Scan the filesystem to determine actual storage usage.""" total = 0 with os.scandir(path) as it: for entry in it: if entry.is_file(follow_symlinks=False): if os.path.abspath(entry.path) not in skips: total += entry.stat(follow_symlinks=False).st_size elif entry.is_dir(follow_symlinks=False): total += self._quota_scan(entry.path, skips) return total def _quota_persist(self, delta): """Persist quota usage to the on-disk quota file. To support concurrent sessions, this method applies the given *delta* to the current on-disk value under an exclusive file lock. This way, updates from other sessions are preserved. If the quota file does not exist or contains an invalid value, a filesystem scan is performed to determine the actual usage. The quota file itself is used as the lock file (opened and locked with flock) so no separate lock file is needed. """ quota_path = self._quota_path() try: fd = os.open(str(quota_path), os.O_RDONLY) except FileNotFoundError: # quota file missing, scan filesystem to determine usage skips = {os.path.abspath(quota_path)} quota_use = self._quota_scan(self.base_path, skips) quota_path.write_text(str(quota_use)) self._quota_use_persisted = quota_use self._quota_use = quota_use self._quota_last_persist_time = time.monotonic() return try: if fcntl is not None: fcntl.flock(fd, fcntl.LOCK_EX) # read current on-disk value (may have been updated by another session) try: on_disk = int(os.read(fd, 100)) except ValueError: # invalid content, scan filesystem to determine usage skips = {os.path.abspath(quota_path)} on_disk = self._quota_scan(self.base_path, skips) delta = 0 # scan already gives the true value if is_win32: # Close the file before replacing to avoid AccessDenied on Windows. os.close(fd) fd = -1 new_value = max(on_disk + delta, 0) quota_content = str(new_value).encode() tmp_path = self._write_to_tempfile(quota_path.parent, quota_content, do_fsync=True) try: tmp_path.replace(quota_path) # atomic update except OSError: tmp_path.unlink() raise self._quota_use_persisted = new_value self._quota_use = new_value # re-sync with on-disk truth self._quota_last_persist_time = time.monotonic() finally: if fcntl is not None: fcntl.flock(fd, fcntl.LOCK_UN) if fd >= 0: os.close(fd) def _quota_update(self, delta, force=False): """Update quota usage by delta and persist if the change is significant or enough time has elapsed.""" self._quota_use += delta persist_delta = self._quota_use - self._quota_use_persisted elapsed = time.monotonic() - self._quota_last_persist_time if force or abs(persist_delta) >= QUOTA_PERSIST_DELTA or elapsed >= QUOTA_PERSIST_INTERVAL: self._quota_persist(persist_delta) def _quota_delete(self): """Delete the quota file if it exists.""" try: self._quota_path().unlink() except FileNotFoundError: pass ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777263628.0 borgstore-0.5.1/src/borgstore/backends/rclone.py0000644000076500000240000002570715173562014020322 0ustar00twstaff""" BorgStore backend for rclone """ import os import re import subprocess import json import secrets from typing import Iterator import time import socket try: import requests except ImportError: requests = None from ._base import BackendBase, ItemInfo, validate_name from ._utils import make_range_header from .errors import ( BackendError, BackendDoesNotExist, BackendMustNotBeOpen, BackendMustBeOpen, BackendAlreadyExists, ObjectNotFound, ) # rclone binary - expected to be on the path RCLONE = os.environ.get("RCLONE_BINARY", "rclone") # Debug HTTP requests and responses RCLONE_DEBUG = os.environ.get("BORGSTORE_RCLONE_DEBUG", "0") if RCLONE_DEBUG.strip().lower() in ("1", "true", "yes", "y", "on"): import logging import http.client as http_client http_client.HTTPConnection.debuglevel = 1 logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) requests_log = logging.getLogger("requests.packages.urllib3") requests_log.setLevel(logging.DEBUG) requests_log.propagate = True def get_rclone_backend(url): """Get rclone backend from URL. rclone:remote: rclone:remote:path """ if not url.startswith("rclone:"): return None if requests is None: raise BackendDoesNotExist( "The rclone backend requires dependencies. Install them with: 'pip install borgstore[rclone]'" ) try: # Check rclone is on the path info = json.loads(subprocess.check_output([RCLONE, "rc", "--loopback", "core/version"])) except Exception: raise BackendDoesNotExist("rclone binary not found on the path or not working properly") if info["decomposed"] < [1, 57, 0]: raise BackendDoesNotExist(f"rclone version must be at least v1.57.0 - found {info['version']}") rclone_regex = r""" rclone: (?P(.*)) """ m = re.match(rclone_regex, url, re.VERBOSE) if m: # no URL-unquote here, we just pass through the rclone remote spec "as is" return Rclone(path=m["path"]) class Rclone(BackendBase): """BorgStore backend for rclone. This uses the rclone rc API to control an rclone rcd process. """ precreate_dirs: bool = False HOST = "127.0.0.1" TRIES = 3 # try failed load/store operations this many times def __init__(self, path, *, do_fsync=False): if not path.endswith(":") and not path.endswith("/"): path += "/" self.fs = path self.process = None self.url = None self.user = "borg" self.password = secrets.token_urlsafe(32) def find_available_port(self): with socket.socket() as s: s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.bind((self.HOST, 0)) return s.getsockname()[1] def check_port(self, port): with socket.socket() as s: try: s.connect((self.HOST, port)) return True except ConnectionRefusedError: return False def open(self): """ Start using the rclone server. """ if self.process: raise BackendMustNotBeOpen() while not self.process: port = self.find_available_port() # Open rclone rcd listening on a random port with random auth args = [ RCLONE, "rcd", "--rc-user", self.user, "--rc-addr", f"{self.HOST}:{port}", "--rc-serve", "--use-server-modtime", ] env = os.environ.copy() env["RCLONE_RC_PASS"] = self.password # pass password by env var so it isn't in process list self.process = subprocess.Popen( args, stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stdin=subprocess.DEVNULL, env=env ) self.url = f"http://{self.HOST}:{port}/" # Wait for rclone to start up while self.process.poll() is None and not self.check_port(port): time.sleep(0.01) if self.process.poll() is None: self.noop("noop") else: self.process = None def close(self): """ Stop using the rclone server. """ if not self.process: raise BackendMustBeOpen() self.process.terminate() try: self.process.wait(timeout=10) except subprocess.TimeoutExpired: self.process.kill() self.process.wait() self.process = None self.url = None def _requests(self, fn, *args, tries=1, **kwargs): """ Run a call to the requests function fn with *args and **kwargs. It adds auth and decodes errors in a consistent way. It returns the response object. This will retry any 500 errors received from rclone 'tries' times, as these correspond to backend, protocol, or Internet errors. Note that rclone will retry all operations internally except those which stream data. """ if not self.process or not self.url: raise BackendMustBeOpen() for try_number in range(tries): r = fn(*args, auth=(self.user, self.password), **kwargs) if r.status_code in (200, 206): return r elif r.status_code == 404: raise ObjectNotFound(f"Not Found: error {r.status_code}: {r.text}") err = BackendError(f"rclone rc command failed: error {r.status_code}: {r.text}") if r.status_code != 500: break raise err def _rpc(self, command, json_input, **kwargs): """ Run the rclone command over the rclone API. Additional kwargs may be passed to requests. """ if not self.url: raise BackendMustBeOpen() r = self._requests(requests.post, self.url + command, json=json_input, **kwargs) return r.json() def create(self): """Create (initialize) the rclone storage.""" if self.process: raise BackendMustNotBeOpen() with self: try: if any(self.list("")): raise BackendAlreadyExists(f"rclone storage base path exists and isn't empty: {self.fs}") except ObjectNotFound: pass self.mkdir("") def destroy(self): """Completely remove the rclone storage (and its contents).""" if self.process: raise BackendMustNotBeOpen() with self: info = self.info("") if not info.exists: raise BackendDoesNotExist(f"rclone storage base path does not exist: {self.fs}") self._rpc("operations/purge", {"fs": self.fs, "remote": ""}) def __enter__(self): self.open() return self def __exit__(self, exc_type, exc_val, exc_tb): self.close() return False def noop(self, value): """No-op request that returns the provided value.""" return self._rpc("rc/noop", {"value": value}) def mkdir(self, name: str) -> None: """Create directory/namespace .""" validate_name(name) self._rpc("operations/mkdir", {"fs": self.fs, "remote": name}) def rmdir(self, name: str) -> None: """Remove directory/namespace .""" validate_name(name) self._rpc("operations/rmdir", {"fs": self.fs, "remote": name}) def _to_item_info(self, remote, item): """Convert an rclone item at remote into a BorgStore ItemInfo.""" if item is None: return ItemInfo(name=os.path.basename(remote), exists=False, directory=False, size=0) name = item["Name"] size = item["Size"] directory = item["IsDir"] return ItemInfo(name=name, exists=True, size=size, directory=directory) def info(self, name) -> ItemInfo: """Return information about .""" validate_name(name) try: result = self._rpc( "operations/stat", {"fs": self.fs, "remote": name, "opt": {"recurse": False, "noModTime": True, "noMimeType": True}}, ) item = result["item"] except ObjectNotFound: item = None return self._to_item_info(name, item) def load(self, name: str, *, size=None, offset=0) -> bytes: """Load value from .""" validate_name(name) headers = {} if offset < 0 and size is not None: if -offset - size <= 1024: # Optimization: if the part of the tail we don't need is small, # we just request the last N bytes and truncate locally. range_header = make_range_header(offset, size=None) else: info = self.info(name) range_header = make_range_header(offset, size, info.size) else: range_header = make_range_header(offset, size) if range_header: headers["Range"] = range_header r = self._requests(requests.get, f"{self.url}[{self.fs}]/{name}", tries=self.TRIES, headers=headers) content = r.content if offset < 0 and size is not None and size < len(content): content = content[:size] return content def store(self, name: str, value: bytes) -> None: """Store into .""" validate_name(name) files = {"file": (os.path.basename(name), value, "application/octet-stream")} params = {"fs": self.fs, "remote": os.path.dirname(name)} self._rpc("operations/uploadfile", None, tries=self.TRIES, params=params, files=files) def delete(self, name: str) -> None: """Delete .""" validate_name(name) self._rpc("operations/deletefile", {"fs": self.fs, "remote": name}) def hash(self, name: str, algorithm: str = "sha256") -> str: validate_name(name) return super().hash(name, algorithm=algorithm) def move(self, curr_name: str, new_name: str) -> None: """Rename curr_name to new_name (overwrite target).""" validate_name(curr_name) validate_name(new_name) self._rpc( "operations/movefile", {"srcFs": self.fs, "srcRemote": curr_name, "dstFs": self.fs, "dstRemote": new_name} ) def list(self, name: str) -> Iterator[ItemInfo]: """List the contents of , non-recursively. The yielded ItemInfos are sorted alphabetically by name. """ validate_name(name) result = self._rpc( "operations/list", {"fs": self.fs, "remote": name, "opt": {"recurse": False, "noModTime": True, "noMimeType": True}}, ) for item in result["list"]: name = item["Name"] try: validate_name(name) except ValueError: pass # that file is likely not from us or is still uploading else: yield self._to_item_info(name, item) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780352745.0 borgstore-0.5.1/src/borgstore/backends/rest.py0000644000076500000240000004345615207403351020012 0ustar00twstaff""" REST http client based backend implementation (use with borgstore.server.rest). """ import collections import os import re import shlex import json import logging import hashlib import threading import subprocess from typing import Iterator, Dict, Optional from types import ModuleType from http import HTTPStatus as HTTP from urllib.parse import unquote requests: Optional[ModuleType] = None HTTPBasicAuth: Optional[type] = None try: import requests as requests_module from requests.auth import HTTPBasicAuth as HTTPBasicAuth_class requests = requests_module HTTPBasicAuth = HTTPBasicAuth_class except ImportError: pass from ._base import BackendBase, ItemInfo, validate_name from ._utils import make_range_header from .errors import ( ObjectNotFound, BackendAlreadyExists, BackendDoesNotExist, PermissionDenied, QuotaExceeded, BackendError, BackendMustBeOpen, BackendMustNotBeOpen, ) logger = logging.getLogger(__name__) class StdioSession: def __init__(self, command, auth=None, headers=None, timeout=30): self.command = command self.auth = auth self.headers = headers or {} self.timeout = timeout self.process = None self._stderr_thread = None self._stderr_lines: collections.deque = collections.deque(maxlen=10) # recent stderr for error messages def _drain_stderr(self): if self.process is None or self.process.stderr is None: return for line in self.process.stderr: decoded = line.decode("utf-8", errors="replace").rstrip() self._stderr_lines.append(decoded) logger.debug("Remote: %s", decoded) def open(self): if self.process is not None: return self.process = subprocess.Popen( self.command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE ) self._stderr_thread = threading.Thread(target=self._drain_stderr, daemon=True) self._stderr_thread.start() def close(self): if self.process is None: return returncode = None try: if self.process.stdin is not None: self.process.stdin.close() self.process.wait(timeout=self.timeout) returncode = self.process.returncode except subprocess.TimeoutExpired: self.process.kill() self.process.wait(timeout=self.timeout) finally: if self.process.stdout is not None: self.process.stdout.close() if self.process.stderr is not None: self.process.stderr.close() if self._stderr_thread is not None: self._stderr_thread.join(timeout=0.5) self.process = None self._stderr_thread = None if returncode: stderr_tail = "\n".join(self._stderr_lines) detail = f":\n{stderr_tail}" if stderr_tail else "" self._stderr_lines.clear() raise BackendError(f"stdio server exited with code {returncode}{detail}") self._stderr_lines.clear() def __enter__(self): self.open() return self def __exit__(self, exc_type, exc, tb): self.close() def request(self, method, url, params=None, data=None, headers=None, timeout=None): if self.process is None or self.process.stdin is None or self.process.stdout is None: raise BackendError("stdio session is not open") request_headers = dict(self.headers) if headers: request_headers.update(headers) request_headers["Connection"] = "keep-alive" prepared = requests.Request( method=method, url=url, params=params, data=data, headers=request_headers, auth=self.auth ).prepare() body = prepared.body if body is None: body = b"" elif isinstance(body, bytes): pass # ok elif isinstance(body, str): body = body.encode("utf-8") else: raise BackendError(f"unsupported body type: {type(body).__name__}") request_line = f"{prepared.method} {prepared.path_url} HTTP/1.1\r\n" header_lines = "".join(f"{k}: {v}\r\n" for k, v in prepared.headers.items()) self.process.stdin.write((request_line + header_lines + "\r\n").encode("ascii")) if body: self.process.stdin.write(body) self.process.stdin.flush() line = self.process.stdout.readline() if not line: if self._stderr_thread is not None: self._stderr_thread.join(timeout=0.5) stderr_tail = "\n".join(self._stderr_lines) detail = f":\n{stderr_tail}" if stderr_tail else "" raise BackendError(f"stdio server closed connection unexpectedly{detail}") status_line = line.decode("iso-8859-1").strip() parts = status_line.split(" ", 2) if len(parts) < 2: raise BackendError(f"invalid HTTP status line from stdio server: {status_line!r}") status_code = int(parts[1]) reason = parts[2] if len(parts) > 2 else "" response_headers = requests.structures.CaseInsensitiveDict() while True: line = self.process.stdout.readline() if line in (b"\r\n", b"\n", b""): break header_line = line.decode("iso-8859-1").strip() if ":" in header_line: key, value = header_line.split(":", 1) response_headers[key.strip()] = value.strip() content_length = int(response_headers.get("Content-Length", "0")) response_body = self.process.stdout.read(content_length) if content_length else b"" response = requests.Response() response.status_code = status_code response.headers = response_headers response._content = response_body response.url = prepared.url response.reason = reason response.encoding = requests.utils.get_encoding_from_headers(response_headers) response.request = prepared return response def ssh_cmd(user, host, port): """return an ssh command line that can be prefixed to another command line""" rsh = os.environ.get("BORGSTORE_RSH") if rsh: args = shlex.split(rsh) else: args = ["ssh"] if port: args += ["-p", str(port)] args += [f"{user}@{host}"] if user else [host] return args def get_rest_backend(base_url: str): if not base_url.startswith(("http:", "https:", "rest:")): return None if requests is None: raise BackendDoesNotExist( "The REST backend requires dependencies. Install them with: 'pip install borgstore[rest]'" ) # http(s)://username:password@hostname:port/sub/path or # http(s)://hostname:port/sub/path + authentication from environment # # note: borgstore.server.rest does not support sub-paths, but sub-paths are # supported in the rest client for use with reverse-proxy setups (see contrib/) # or custom REST servers. http_regex = r""" (?Phttp|https):// ((?P[^:]+):(?P[^@]+)@)? (?P[^:/]+)(:(?P\d+))? (?P/[^?#]*)? """ m = re.match(http_regex, base_url, re.VERBOSE) if m: scheme = m.group("scheme") host = m.group("host") port = m.group("port") path = m.group("path") or "" base_url = f"{scheme}://{host}{f':{port}' if port else ''}{path}" username, password = m.group("username"), m.group("password") if username and password: username, password = unquote(username), unquote(password) else: username, password = os.environ.get("BORGSTORE_REST_USERNAME"), os.environ.get("BORGSTORE_REST_PASSWORD") return REST(base_url, username=username, password=password) # rest protocol means: use stdio to talk to a borgstore.server.rest process, # either locally (empty host) or via ssh to the given host. The given path # is used to construct a "FILE:" (hack!) backend URI used by the rest server. # # rest:///path - talk to local rest server, path must be abs. fs path # rest://user@host:port/path - ssh to rest server on host, abs. fs path rest_regex = r""" rest:// ( (?:(?P[^@:/]+)@)? # optional user (?P( (?!\[)[^:/]+(?\d+))? # optional port )? / # separator always required (?P[^?#]+) # non-empty rel/path or /abs/path or even ~/path or ~user/path """ m = re.match(rest_regex, base_url, re.VERBOSE) if m: path = m.group("path") user = m.group("user") host = m.group("host") port = m.group("port") or "22" # empty host: don't use ssh, just run the rest server here command = [] if not host else ssh_cmd(user, host, port) # hack: we do NOT use a standards-compliant file:// URI here, because they only support absolute paths. # we just use FILE:path and that path can be relative or absolute or even have ~ or ~user. # borgstore.server.rest will translate it to an absolute file:// URI internally. command.extend(["borgstore-server-rest", "--stdio", "--backend", f"FILE:{path}"]) return REST(base_url="http://stdio-backend", command=command) class REST(BackendBase): def __init__( self, base_url: str, username: Optional[str] = None, password: Optional[str] = None, headers: Optional[Dict[str, str]] = None, timeout: Optional[int] = 30, command=None, ): self.base_url = base_url.rstrip("/") # _url method adds slash self.headers = headers or {} self.headers["Accept"] = "application/vnd.x.borgstore.rest.v1" self.timeout = timeout self.auth = HTTPBasicAuth(username, password) if username and password else None self.command = command self.session = None def _url(self, path: str) -> str: return f"{self.base_url}/{path.lstrip('/')}" def _assert_open(self): if self.session is None: raise BackendMustBeOpen() def _assert_closed(self): if self.session is not None: raise BackendMustNotBeOpen() def _request(self, method, url, *, headers=None, data=None, params=None): if self.session is not None: # between .open() and .close() return self.session.request(method, url, params=params, data=data, headers=headers, timeout=self.timeout) else: # .create() and .destroy() are called when backend is not opened if headers is not None: raise ValueError("custom headers are not supported outside of an open session") if self.command is not None: with StdioSession( command=self.command, auth=self.auth, headers=self.headers, timeout=self.timeout ) as session: return session.request(method, url, params=params, data=data, timeout=self.timeout) return requests.request( method, url, auth=self.auth, params=params, data=data, headers=self.headers, timeout=self.timeout ) def _handle_response(self, response, name=None): if response.status_code == HTTP.OK: return if response.status_code == HTTP.PARTIAL_CONTENT: return if response.status_code == HTTP.NOT_FOUND: raise ObjectNotFound(name or "unknown") if response.status_code == HTTP.GONE: raise BackendDoesNotExist(self.base_url) if response.status_code == HTTP.CONFLICT: raise BackendAlreadyExists(self.base_url) if response.status_code == HTTP.PRECONDITION_FAILED: # Precondition failed, used for state errors if "must be open" in response.text: raise BackendMustBeOpen() if "must not be open" in response.text: raise BackendMustNotBeOpen() raise BackendError(response.text) if response.status_code == HTTP.FORBIDDEN: raise PermissionDenied(name or self.base_url) if response.status_code == HTTP.INSUFFICIENT_STORAGE: raise QuotaExceeded(response.text) if response.status_code == HTTP.BAD_REQUEST: raise ValueError(response.text) response.raise_for_status() def create(self) -> None: self._assert_closed() response = self._request("post", self._url(""), params={"cmd": "create"}) self._handle_response(response, "backend") def destroy(self) -> None: self._assert_closed() response = self._request("delete", self._url(""), params={"cmd": "destroy"}) self._handle_response(response, "backend") def open(self): self._assert_closed() if self.command is not None: self.session = StdioSession( command=self.command, auth=self.auth, headers=self.headers, timeout=self.timeout ) self.session.open() else: self.session = requests.Session() self.session.auth = self.auth self.session.headers.update(self.headers) def close(self): self._assert_open() self.session.close() self.session = None def mkdir(self, name: str) -> None: self._assert_open() validate_name(name) response = self._request("post", self._url(name), params={"cmd": "mkdir"}) self._handle_response(response, name) def rmdir(self, name: str) -> None: self._assert_open() validate_name(name) response = self._request("delete", self._url(name), params={"cmd": "rmdir"}) self._handle_response(response, name) def info(self, name: str) -> ItemInfo: self._assert_open() validate_name(name) response = self._request("head", self._url(name)) if response.status_code not in (HTTP.OK, HTTP.NOT_FOUND): self._handle_response(response, name) # raises! exists = response.status_code == HTTP.OK is_dir = response.headers.get("X-BorgStore-Is-Directory") == "true" return ItemInfo(name=name, exists=exists, size=int(response.headers.get("Content-Length", 0)), directory=is_dir) def load(self, name: str, *, size=None, offset=0) -> bytes: self._assert_open() validate_name(name) if offset < 0 and size is not None: if -offset - size <= 1024: # Optimization: if the part of the tail we don't need is small, # we just request the last N bytes and truncate locally. range_header = make_range_header(offset, size=None) else: info = self.info(name) range_header = make_range_header(offset, size, info.size) else: range_header = make_range_header(offset, size) headers = self.headers.copy() if range_header: headers["Range"] = range_header response = self._request("get", self._url(name), headers=headers) self._handle_response(response, name) content = response.content if offset < 0 and size is not None and size < len(content): content = content[:size] return content def store(self, name: str, value: bytes) -> None: self._assert_open() validate_name(name) algorithm = "sha256" headers = {f"X-Content-hash-{algorithm}": hashlib.new(algorithm, value).hexdigest()} response = self._request("post", self._url(name), data=value, headers=headers) self._handle_response(response, name) def delete(self, name: str) -> None: self._assert_open() validate_name(name) response = self._request("delete", self._url(name)) self._handle_response(response, name) def move(self, curr_name: str, new_name: str) -> None: self._assert_open() validate_name(curr_name) validate_name(new_name) response = self._request("post", self._url(""), params={"cmd": "move", "current": curr_name, "new": new_name}) self._handle_response(response, f"{curr_name} -> {new_name}") def defrag(self, sources, *, target=None, algorithm=None, namespace=None, levels=0) -> str: self._assert_open() params = {"cmd": "defrag"} if target is not None: params["target"] = target if algorithm is not None: params["algorithm"] = algorithm if namespace is not None: params["namespace"] = namespace if levels: params["levels"] = levels data = json.dumps(sources).encode("utf-8") response = self._request("post", self._url(""), params=params, data=data) self._handle_response(response, "defrag") return response.text def quota(self) -> dict: self._assert_open() response = self._request("post", self._url(""), params={"cmd": "quota"}) self._handle_response(response, "quota") return response.json() def hash(self, name: str, algorithm: str = "sha256") -> str: self._assert_open() validate_name(name) response = self._request("post", self._url(name), params={"cmd": "hash", "algorithm": algorithm}) self._handle_response(response, name) return response.text def list(self, name: str) -> Iterator[ItemInfo]: self._assert_open() validate_name(name) response = self._request("get", self._url(name) + "/") # trailing "/" needed to get list self._handle_response(response, name) for entry in response.json(): yield ItemInfo(name=entry["name"], exists=True, size=entry["size"], directory=entry.get("directory", False)) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1774660792.0 borgstore-0.5.1/src/borgstore/backends/s3.py0000644000076500000240000003024215161626270017355 0ustar00twstaff""" BorgStore backend for S3-compatible services (including Backblaze B2) using boto3. """ try: import boto3 from botocore.client import Config except ImportError: boto3 = None import re from typing import Optional import urllib.parse from ._base import BackendBase, ItemInfo, validate_name from ._utils import make_range_header from .errors import BackendError, BackendMustBeOpen, BackendMustNotBeOpen, BackendDoesNotExist, BackendAlreadyExists from .errors import ObjectNotFound def get_s3_backend(url: str): """Get S3 backend from URL. Supports URLs of the form: (s3|b2):[profile|(access_key_id:access_key_secret)@][schema://hostname[:port]]/bucket/path """ if not url.startswith(("s3:", "b2:")): return None if boto3 is None: raise BackendDoesNotExist( "The S3 backend requires dependencies. Install them with: 'pip install borgstore[s3]'" ) # (s3|b2):[profile|(access_key_id:access_key_secret)@][schema://hostname[:port]]/bucket/path s3_regex = r""" (?P(s3|b2)): (( (?P[^@:]+) # profile (no colons allowed) | (?P[^:@]+):(?P[^@]+) # access key and secret )@)? # optional authentication ( (?P[^:/]+):// (?P[^:/]+) (:(?P\d+))? )? # optional endpoint / (?P[^/]+)/ # bucket name (?P.+) # path """ m = re.match(s3_regex, url, re.VERBOSE) if m: s3type = m["s3type"] profile = m["profile"] access_key_id = m["access_key_id"] access_key_secret = m["access_key_secret"] if profile is not None and access_key_id is not None: raise BackendError("S3: profile and access_key_id cannot be specified at the same time") if access_key_id is not None and access_key_secret is None: raise BackendError("S3: access_key_secret is mandatory when access_key_id is specified") if access_key_id is not None: access_key_id = urllib.parse.unquote(access_key_id) if access_key_secret is not None: access_key_secret = urllib.parse.unquote(access_key_secret) schema = m["schema"] hostname = m["hostname"] port = m["port"] bucket = m["bucket"] # no unquote: all valid bucket characters are URL-safe path = urllib.parse.unquote(m["path"]) endpoint_url = None if schema and hostname: endpoint_url = f"{schema}://{hostname}" if port: endpoint_url += f":{port}" return S3( bucket=bucket, path=path, is_b2=s3type == "b2", profile=profile, access_key_id=access_key_id, access_key_secret=access_key_secret, endpoint_url=endpoint_url, ) class S3(BackendBase): """BorgStore backend for S3 and Backblaze B2 (via boto3).""" def __init__( self, bucket: str, path: str, is_b2: bool, profile: Optional[str] = None, access_key_id: Optional[str] = None, access_key_secret: Optional[str] = None, endpoint_url: Optional[str] = None, ): self.delimiter = "/" self.bucket = bucket self.base_path = path.rstrip(self.delimiter) + self.delimiter # Ensure it ends with '/' self.opened = False if profile: session = boto3.Session(profile_name=profile) elif access_key_id and access_key_secret: session = boto3.Session(aws_access_key_id=access_key_id, aws_secret_access_key=access_key_secret) else: session = boto3.Session() config = None if is_b2: config = Config(request_checksum_calculation="when_required", response_checksum_validation="when_required") self.s3 = session.client("s3", endpoint_url=endpoint_url, config=config) if is_b2: event_system = self.s3.meta.events event_system.register_first("before-sign.*.*", self._fix_headers) def _fix_headers(self, request, **kwargs): if "x-amz-checksum-crc32" in request.headers: del request.headers["x-amz-checksum-crc32"] if "x-amz-sdk-checksum-algorithm" in request.headers: del request.headers["x-amz-sdk-checksum-algorithm"] def _mkdir(self, name): try: key = (self.base_path + name).rstrip(self.delimiter) + self.delimiter self.s3.put_object(Bucket=self.bucket, Key=key) except self.s3.exceptions.ClientError as e: raise BackendError(f"S3 error: {e}") def create(self): if self.opened: raise BackendMustNotBeOpen() try: objects = self.s3.list_objects_v2( Bucket=self.bucket, Prefix=self.base_path, Delimiter=self.delimiter, MaxKeys=1 ) if objects["KeyCount"] > 0: raise BackendAlreadyExists(f"Backend already exists: {self.base_path}") self._mkdir("") except self.s3.exceptions.NoSuchBucket: raise BackendDoesNotExist(f"S3 bucket does not exist: {self.bucket}") except self.s3.exceptions.ClientError as e: raise BackendError(f"S3 error: {e}") def destroy(self): if self.opened: raise BackendMustNotBeOpen() try: objects = self.s3.list_objects_v2( Bucket=self.bucket, Prefix=self.base_path, Delimiter=self.delimiter, MaxKeys=1 ) if objects["KeyCount"] == 0: raise BackendDoesNotExist(f"Backend does not exist: {self.base_path}") is_truncated = True while is_truncated: objects = self.s3.list_objects_v2(Bucket=self.bucket, Prefix=self.base_path, MaxKeys=1000) is_truncated = objects["IsTruncated"] if "Contents" in objects: self.s3.delete_objects( Bucket=self.bucket, Delete={"Objects": [{"Key": obj["Key"]} for obj in objects["Contents"]]} ) except self.s3.exceptions.ClientError as e: raise BackendError(f"S3 error: {e}") def open(self): if self.opened: raise BackendMustNotBeOpen() self.opened = True def close(self): if not self.opened: raise BackendMustBeOpen() self.opened = False def store(self, name, value): if not self.opened: raise BackendMustBeOpen() validate_name(name) key = self.base_path + name self.s3.put_object(Bucket=self.bucket, Key=key, Body=value) def load(self, name, *, size=None, offset=0): if not self.opened: raise BackendMustBeOpen() validate_name(name) key = self.base_path + name try: if offset < 0 and size is not None: if -offset - size <= 1024: # Optimization: if the part of the tail we don't need is small, # we just request the last N bytes and truncate locally. range_header = make_range_header(offset, size=None) else: info = self.info(name) range_header = make_range_header(offset, size, info.size) else: range_header = make_range_header(offset, size) if range_header: obj = self.s3.get_object(Bucket=self.bucket, Key=key, Range=range_header) else: obj = self.s3.get_object(Bucket=self.bucket, Key=key) content = obj["Body"].read() if offset < 0 and size is not None and size < len(content): content = content[:size] return content except self.s3.exceptions.NoSuchKey: raise ObjectNotFound(name) def delete(self, name): if not self.opened: raise BackendMustBeOpen() validate_name(name) key = self.base_path + name try: self.s3.head_object(Bucket=self.bucket, Key=key) self.s3.delete_object(Bucket=self.bucket, Key=key) except self.s3.exceptions.NoSuchKey: raise ObjectNotFound(name) except self.s3.exceptions.ClientError as e: if e.response["Error"]["Code"] == "404": raise ObjectNotFound(name) def hash(self, name: str, algorithm: str = "sha256") -> str: if not self.opened: raise BackendMustBeOpen() return super().hash(name, algorithm=algorithm) def move(self, curr_name, new_name): if not self.opened: raise BackendMustBeOpen() validate_name(curr_name) validate_name(new_name) src_key = self.base_path + curr_name dest_key = self.base_path + new_name try: self.s3.copy_object(Bucket=self.bucket, CopySource={"Bucket": self.bucket, "Key": src_key}, Key=dest_key) self.s3.delete_object(Bucket=self.bucket, Key=src_key) except self.s3.exceptions.NoSuchKey: raise ObjectNotFound(curr_name) def list(self, name): if not self.opened: raise BackendMustBeOpen() validate_name(name) base_prefix = (self.base_path + name).rstrip(self.delimiter) + self.delimiter try: start_after = "" is_truncated = True while is_truncated: objects = self.s3.list_objects_v2( Bucket=self.bucket, Prefix=base_prefix, Delimiter=self.delimiter, MaxKeys=1000, StartAfter=start_after, ) if objects["KeyCount"] == 0: raise ObjectNotFound(name) is_truncated = objects["IsTruncated"] for obj in objects.get("Contents", []): obj_name = obj["Key"][len(base_prefix) :] # Remove base_path prefix if obj_name == "": continue try: validate_name(obj_name) except ValueError: pass # that file is likely not from us or is still uploading else: start_after = obj["Key"] yield ItemInfo(name=obj_name, exists=True, size=obj["Size"], directory=False) for prefix in objects.get("CommonPrefixes", []): dir_name = prefix["Prefix"][len(base_prefix) : -1] # Remove base_path prefix and trailing slash yield ItemInfo(name=dir_name, exists=True, size=0, directory=True) except self.s3.exceptions.ClientError as e: raise BackendError(f"S3 error: {e}") def mkdir(self, name): if not self.opened: raise BackendMustBeOpen() validate_name(name) self._mkdir(name) def rmdir(self, name): if not self.opened: raise BackendMustBeOpen() validate_name(name) prefix = self.base_path + name.rstrip(self.delimiter) + self.delimiter objects = self.s3.list_objects_v2(Bucket=self.bucket, Prefix=prefix, Delimiter=self.delimiter, MaxKeys=2) if "Contents" in objects and len(objects["Contents"]) > 1: raise BackendError(f"Directory not empty: {name}") self.s3.delete_object(Bucket=self.bucket, Key=prefix) def info(self, name): if not self.opened: raise BackendMustBeOpen() validate_name(name) key = self.base_path + name try: obj = self.s3.head_object(Bucket=self.bucket, Key=key) return ItemInfo(name=name, exists=True, directory=False, size=obj["ContentLength"]) except self.s3.exceptions.ClientError as e: if e.response["Error"]["Code"] == "404": try: self.s3.head_object(Bucket=self.bucket, Key=key + self.delimiter) return ItemInfo(name=name, exists=True, directory=True, size=0) except self.s3.exceptions.ClientError: pass return ItemInfo(name=name, exists=False, directory=False, size=0) raise BackendError(f"S3 error: {e}") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777075926.0 borgstore-0.5.1/src/borgstore/backends/sftp.py0000644000076500000240000003420515173003326020001 0ustar00twstaff""" SFTP-based backend implementation — on an SFTP server, uses files in directories below a base path. """ from pathlib import Path from urllib.parse import unquote import random import re import stat from typing import Optional try: import paramiko except ImportError: paramiko = None from ._base import BackendBase, ItemInfo, validate_name from .errors import BackendError, BackendMustBeOpen, BackendMustNotBeOpen, BackendDoesNotExist, BackendAlreadyExists from .errors import ObjectNotFound from ..constants import TMP_SUFFIX def get_sftp_backend(url): """Get SFTP backend from URL.""" if not url.startswith("sftp:"): return None if paramiko is None: raise BackendDoesNotExist( "The SFTP backend requires dependencies. Install them with: 'pip install borgstore[sftp]'" ) # sftp://username@hostname:22/path # Notes: # - username and port are optional # - host must be a hostname (not an IP address) # - you must provide a path; by default it is a relative path (usually relative to the user's home directory — # this allows the SFTP server admin to move things without the user needing to know). # - giving an absolute path is also possible: sftp://username@hostname:22//home/username/borgstore sftp_regex = r""" sftp:// ((?P[^@]+)@)? (?P([^:/]+))(?::(?P\d+))?/ # slash as separator, not part of the path (?P(.+)) # path may or may not start with a slash, must not be empty """ m = re.match(sftp_regex, url, re.VERBOSE) if m: return Sftp( username=unquote(m["username"]) if m["username"] else None, hostname=m["hostname"], port=int(m["port"] or "0"), path=unquote(m["path"]), ) class Sftp(BackendBase): """BorgStore backend for SFTP.""" # Sftp implementation supports precreate = True as well as = False. precreate_dirs: bool = False def __init__(self, hostname: str, path: str, port: int = 0, username: Optional[str] = None): self.username = username self.hostname = hostname self.port = port self.base_path = path self.opened = False self.check_file_supported = True self.ssh: Optional[paramiko.SSHClient] = None self.client: Optional[paramiko.SFTPClient] = None if paramiko is None: raise BackendError("sftp backend unavailable: could not import paramiko!") def _get_host_config_from_file(self, path: str, hostname: str): """Look up the configuration for hostname in path (SSH config file).""" config_path = Path(path).expanduser() try: ssh_config = paramiko.SSHConfig.from_path(config_path) except FileNotFoundError: return paramiko.SSHConfigDict() # empty dict else: return ssh_config.lookup(hostname) def _get_host_config(self): """Assemble all provided and configured host configuration values.""" host_config = paramiko.SSHConfigDict() # self.hostname might be an alias/shortcut (with real hostname given in configuration), # but there might be also nothing in the configs at all for self.hostname: host_config["hostname"] = self.hostname # First process system-wide SSH config, then override with user SSH config: host_config.update(self._get_host_config_from_file("/etc/ssh/ssh_config", self.hostname)) # Note: no support yet for /etc/ssh/ssh_config.d/* host_config.update(self._get_host_config_from_file("~/.ssh/config", self.hostname)) # Now override configured values with provided values if self.username is not None: host_config["user"] = self.username if self.port != 0: host_config["port"] = str(self.port) # Make sure port is present. host_config["port"] = str(host_config.get("port") or "22") return host_config def _connect(self): try: self.ssh = paramiko.SSHClient() # Note: we do not deal with unknown hosts and ssh.set_missing_host_key_policy here. # The user should make the first contact to any new host using the ssh or sftp CLI command # and interactively verify remote host fingerprints. self.ssh.load_system_host_keys() # This is documented to load the user's known_hosts file host_config = self._get_host_config() self.ssh.connect( hostname=host_config["hostname"], username=host_config.get("user"), # if None, paramiko will use current user port=int(host_config["port"]), key_filename=host_config.get("identityfile"), # list of keys, ~ is already expanded allow_agent=True, ) self.client = self.ssh.open_sftp() except Exception: self._disconnect() raise def _disconnect(self): if self.client: self.client.close() self.client = None if self.ssh: self.ssh.close() self.ssh = None def create(self): if self.opened: raise BackendMustNotBeOpen() self._connect() try: # We accept an already existing empty directory and we also optionally create # any missing parent dirs. The latter is important for repository hosters that # only offer limited access to their storage (e.g., only via borg/borgstore). # It is also simpler than requiring users to create parent dirs separately. self._mkdir(self.base_path, exist_ok=True, parents=True) # Prevent users from creating a mess by using non-empty directories: contents = list(self.client.listdir(self.base_path)) if contents: raise BackendAlreadyExists(f"sftp storage base path is not empty: {self.base_path}") except IOError as err: raise BackendError(f"sftp storage I/O error: {err}") finally: self._disconnect() def destroy(self): def delete_recursive(path): parent = Path(path) for child_st in self.client.listdir_attr(str(parent)): child = parent / child_st.filename if stat.S_ISDIR(child_st.st_mode): delete_recursive(child) else: self.client.unlink(str(child)) try: self.client.rmdir(str(parent)) except OSError as e: # usually, this is because of missing permissions. if path != self.base_path: raise e from None # do not raise if we can't remove the base path directory. # .create accepts an already existing base path, thus # .destroy may leave an existing base path behind. if self.opened: raise BackendMustNotBeOpen() self._connect() try: try: self.client.stat(self.base_path) # check if this storage exists, fail early if not. except FileNotFoundError: raise BackendDoesNotExist(f"sftp storage base path does not exist: {self.base_path}") from None delete_recursive(self.base_path) finally: self._disconnect() def open(self): if self.opened: raise BackendMustNotBeOpen() self._connect() try: st = self.client.stat(self.base_path) # check if this storage exists, fail early if not. except FileNotFoundError: raise BackendDoesNotExist(f"sftp storage base path does not exist: {self.base_path}") from None if not stat.S_ISDIR(st.st_mode): raise BackendDoesNotExist(f"sftp storage base path is not a directory: {self.base_path}") self.client.chdir(self.base_path) # this sets the cwd we work in! self.opened = True def close(self): if not self.opened: raise BackendMustBeOpen() self._disconnect() self.opened = False def _mkdir(self, name, *, parents=False, exist_ok=False): # Path.mkdir, but via sftp p = Path(name) try: self.client.mkdir(str(p)) except FileNotFoundError: # the parent dir is missing if not parents: raise # first create parent dir(s), recursively: self._mkdir(p.parents[0], parents=parents, exist_ok=exist_ok) # then retry: self.client.mkdir(str(p)) except OSError: # maybe p already existed? if not exist_ok: raise def mkdir(self, name): if not self.opened: raise BackendMustBeOpen() validate_name(name) self._mkdir(name, parents=True, exist_ok=True) def rmdir(self, name): if not self.opened: raise BackendMustBeOpen() validate_name(name) try: self.client.rmdir(name) except FileNotFoundError: raise ObjectNotFound(name) from None def info(self, name): if not self.opened: raise BackendMustBeOpen() validate_name(name) try: st = self.client.stat(name) except FileNotFoundError: return ItemInfo(name=name, exists=False, directory=False, size=0) else: is_dir = stat.S_ISDIR(st.st_mode) return ItemInfo(name=name, exists=True, directory=is_dir, size=st.st_size) def load(self, name, *, size=None, offset=0): if not self.opened: raise BackendMustBeOpen() validate_name(name) try: with self.client.open(name) as f: f.seek(offset, 0 if offset >= 0 else 2) f.prefetch(size) # speeds up the following read() significantly! return f.read(size) except FileNotFoundError: raise ObjectNotFound(name) from None def store(self, name, value): def _write_to_tmpfile(): with self.client.open(tmp_name, mode="w") as f: f.set_pipelined(True) # speeds up the following write() significantly! f.write(value) if not self.opened: raise BackendMustBeOpen() validate_name(name) tmp_dir = Path(name).parent # write to a differently named temp file in same directory first, # so the store never sees partially written data. tmp_name = str(tmp_dir / ("".join(random.choices("abcdefghijklmnopqrstuvwxyz", k=8)) + TMP_SUFFIX)) try: # try to do it quickly, not doing the mkdir. each sftp op might be slow due to latency. # this will frequently succeed, because the dir is already there. _write_to_tmpfile() except FileNotFoundError: # retry, create potentially missing dirs first. this covers these cases: # - either the dirs were not precreated # - a previously existing directory was "lost" in the filesystem self._mkdir(str(tmp_dir), parents=True, exist_ok=True) _write_to_tmpfile() # rename it to the final name: try: self.client.posix_rename(tmp_name, name) except OSError: self.client.unlink(tmp_name) raise def delete(self, name): if not self.opened: raise BackendMustBeOpen() validate_name(name) try: self.client.unlink(name) except FileNotFoundError: raise ObjectNotFound(name) from None def _sftp_hash(self, name: str, algorithm: str) -> str | None: # Sadly, as of 2026-03-28, this is not supported by OpenSSH, # but by some less popular SFTP servers. if self.check_file_supported: try: with self.client.open(name) as f: digest = f.check(algorithm) return digest.hex() except FileNotFoundError: raise ObjectNotFound(name) from None except IOError: # check-file not supported or algorithm not supported self.check_file_supported = False return None def hash(self, name: str, algorithm: str = "sha256") -> str: if not self.opened: raise BackendMustBeOpen() validate_name(name) hexdigest = self._sftp_hash(name, algorithm) if hexdigest is not None: return hexdigest return super().hash(name, algorithm=algorithm) def move(self, curr_name, new_name): def _rename_to_new_name(): self.client.posix_rename(curr_name, new_name) if not self.opened: raise BackendMustBeOpen() validate_name(curr_name) validate_name(new_name) parent_dir = Path(new_name).parent try: # try to do it quickly, not doing the mkdir. each sftp op might be slow due to latency. # this will frequently succeed, because the dir is already there. _rename_to_new_name() except FileNotFoundError: # retry, create potentially missing dirs first. this covers these cases: # - either the dirs were not precreated # - a previously existing directory was "lost" in the filesystem self._mkdir(str(parent_dir), parents=True, exist_ok=True) try: _rename_to_new_name() except FileNotFoundError: raise ObjectNotFound(curr_name) from None def list(self, name): if not self.opened: raise BackendMustBeOpen() validate_name(name) try: infos = self.client.listdir_attr(name) except FileNotFoundError: raise ObjectNotFound(name) from None else: for info in sorted(infos, key=lambda i: i.filename): try: validate_name(info.filename) except ValueError: pass # that file is likely not from us or is still uploading else: is_dir = stat.S_ISDIR(info.st_mode) yield ItemInfo(name=info.filename, exists=True, size=info.st_size, directory=is_dir) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1777074484.0 borgstore-0.5.1/src/borgstore/constants.py0000644000076500000240000000141415173000464017263 0ustar00twstaff"""Constants used by BorgStore.""" # Namespace to pass to list() for the storage root: ROOTNS = "" # Filename suffixes used for special purposes TMP_SUFFIX = ".tmp" # Temporary file while being uploaded/written DEL_SUFFIX = ".del" # "Soft-deleted" item; can be undeleted HID_SUFFIX = ".hid" # Hidden internal file, not accessible by users # Maximum name length (not precise; suffixes might be added!) MAX_NAME_LENGTH = 100 # Being rather conservative here to improve portability between backends and platforms # Quota tracking QUOTA_STORE_NAME = "quota.hid" # Hidden file storing current quota usage QUOTA_PERSIST_DELTA = 10 * 1000 * 1000 # Persist quota if usage changed by at least 10MB QUOTA_PERSIST_INTERVAL = 300 # Persist quota if at least 5 minutes have elapsed ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1780354067.0886211 borgstore-0.5.1/src/borgstore/server/0000755000076500000240000000000015207406023016202 5ustar00twstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1773549086.0 borgstore-0.5.1/src/borgstore/server/__init__.py0000644000076500000240000000004415155433036020317 0ustar00twstaff""" BorgStore HTTP REST server. """ ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780352745.0 borgstore-0.5.1/src/borgstore/server/rest.py0000644000076500000240000006202615207403351017540 0ustar00twstaffimport secrets import hashlib import argparse import json import base64 import logging import os import socket import sys import itertools from http import HTTPStatus as HTTP from http.server import ThreadingHTTPServer, HTTPServer, BaseHTTPRequestHandler from pathlib import Path from urllib.parse import urlsplit, parse_qs from ..backends.errors import ( ObjectNotFound, BackendAlreadyExists, BackendDoesNotExist, PermissionDenied, QuotaExceeded, BackendError, BackendMustBeOpen, BackendMustNotBeOpen, ) from ..backends._utils import parse_range_header from ..store import get_backend logger = logging.getLogger(__name__) class BorgStoreRESTRequestHandler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" # https://en.wikipedia.org/wiki/List_of_Unicode_characters#Control_codes _control_char_table = str.maketrans({c: rf"\x{c:02x}" for c in itertools.chain(range(0x20), range(0x7F, 0xA0))}) _control_char_table[ord("\\")] = r"\\" def address_string(self): # Override to handle Unix domain sockets (AF_UNIX). # BaseHTTPRequestHandler.address_string() assumes client_address is a tuple (host, port). # For AF_UNIX, client_address is a string (the path), which can be empty. if isinstance(self.client_address, str): return self.client_address or "unix" return super().address_string() def _log(self, format, args, level=logging.INFO): addr = self.address_string() dt = self.log_date_time_string() user = self.server.username or "-" request_details = format % args msg = f"{addr} - {user} [{dt}] {request_details}" logger.log(level, msg.translate(self._control_char_table)) def log_message(self, format, *args): self._log(format, args, logging.INFO) def log_error(self, format, *args): # usually this is pretty useless and redundant, thus we only log it at debug level. self._log(format, args, logging.DEBUG) @staticmethod def checks_and_logging(func): def wrapper(self): if not self._check_accept(): return if not self._check_auth(): return self._send_unauthorized() return func(self) return wrapper def _check_auth(self): if not self.server.username or not self.server.password: return True auth_header = self.headers.get("Authorization") if not auth_header: return False scheme, _, encoded_credentials = auth_header.partition(" ") if scheme.lower() != "basic": return False try: decoded_credentials = base64.b64decode(encoded_credentials).decode("utf-8") username, _, password = decoded_credentials.partition(":") authorized = secrets.compare_digest(username, self.server.username) and secrets.compare_digest( password, self.server.password ) return authorized except Exception: logger.exception("Authentication code crashed, returning: unauthorized.") return False def respond(self, status=HTTP.OK, data=None, content_type=None, headers=None): self.send_response(status) if content_type: self.send_header("Content-Type", content_type) # Ensure no proxy or client caches our REST responses. self.send_header("Cache-Control", "no-store, no-cache, must-revalidate, max-age=0") if headers: for key, value in headers.items(): self.send_header(key, value) if data is not None: self.send_header("Content-Length", str(len(data))) elif not headers or "Content-Length" not in headers: self.send_header("Content-Length", "0") self.end_headers() if data is not None and self.command != "HEAD": self.wfile.write(data) def _send_unauthorized(self): self.respond( HTTP.UNAUTHORIZED, data=b"Unauthorized", headers={"WWW-Authenticate": 'Basic realm="BorgStore REST Server"'} ) def _check_accept(self): accept = self.headers.get("Accept") if accept != "application/vnd.x.borgstore.rest.v1": msg = "Not Acceptable: unsupported or missing Accept header" self.send_error(HTTP.NOT_ACCEPTABLE, msg) return False return True @property def split_url(self): return urlsplit(self.path) @property def query(self): return parse_qs(self.split_url.query) @property def name(self): return self.split_url.path.strip("/") def _handle_exception(self, e, name=None): msg = str(e) # Security: do not leak absolute paths in error messages for attr in ("base_path", "fs"): if self.server.backend and hasattr(self.server.backend, attr): path_val = str(getattr(self.server.backend, attr)) if path_val and path_val in msg: msg = msg.replace(path_val, "[STORAGE_BASE]") if isinstance(e, ObjectNotFound): self.send_error(HTTP.NOT_FOUND, msg) elif isinstance(e, BackendDoesNotExist): self.send_error(HTTP.GONE, msg) elif isinstance(e, BackendAlreadyExists): self.send_error(HTTP.CONFLICT, msg) elif isinstance(e, (BackendMustBeOpen, BackendMustNotBeOpen)): self.send_error(HTTP.PRECONDITION_FAILED, msg) elif isinstance(e, PermissionDenied): self.send_error(HTTP.FORBIDDEN, msg) elif isinstance(e, QuotaExceeded): self.send_error(HTTP.INSUFFICIENT_STORAGE, msg) elif isinstance(e, (ValueError, TypeError)): self.send_error(HTTP.BAD_REQUEST, msg) logger.exception("Exception for %s", name or self.path) elif isinstance(e, BackendError): self.send_error(HTTP.INTERNAL_SERVER_ERROR, msg) logger.exception("Exception for %s", name or self.path) else: self.send_error(HTTP.INTERNAL_SERVER_ERROR, "Internal Server Error") logger.exception("Exception for %s", name or self.path) @checks_and_logging def do_POST(self): cmd = self.query.get("cmd", [None])[0] if cmd == "create": try: self.server.backend.create() self.respond(HTTP.OK) except Exception as e: self._handle_exception(e, "create") return if cmd == "move": current = self.query.get("current", [None])[0] new = self.query.get("new", [None])[0] if current and new: try: with self.server.backend: self.server.backend.move(current, new) self.respond(HTTP.OK) except Exception as e: self._handle_exception(e, f"move {current} -> {new}") else: self.send_error(HTTP.BAD_REQUEST, "Missing current or new name for move") return if cmd == "mkdir": try: with self.server.backend: self.server.backend.mkdir(self.name) self.respond(HTTP.OK) except Exception as e: self._handle_exception(e, f"mkdir {self.name}") return if cmd == "hash": if not self.name: self.send_error(HTTP.BAD_REQUEST, "Missing name for hash") return algorithm = self.query.get("algorithm", ["sha256"])[0] try: with self.server.backend: digest = self.server.backend.hash(self.name, algorithm=algorithm) self.respond(HTTP.OK, data=digest.encode("ascii"), content_type="text/plain") except Exception as e: self._handle_exception(e, f"hash {self.name}") return if cmd == "quota": try: with self.server.backend: quota_info = self.server.backend.quota() response_data = json.dumps(quota_info).encode("utf-8") self.respond(HTTP.OK, data=response_data, content_type="application/json") except Exception as e: self._handle_exception(e, "quota") return if cmd == "defrag": target = self.query.get("target", [None])[0] algorithm = self.query.get("algorithm", [None])[0] namespace = self.query.get("namespace", [None])[0] levels = int(self.query.get("levels", [0])[0]) if not target and not algorithm: self.send_error(HTTP.BAD_REQUEST, "Missing target or algorithm for defrag") return try: content_length = int(self.headers.get("Content-Length", 0)) body = self.rfile.read(content_length) sources = json.loads(body) with self.server.backend: target = self.server.backend.defrag( sources, target=target, algorithm=algorithm, namespace=namespace, levels=levels ) self.respond(HTTP.OK, data=target.encode("ascii"), content_type="text/plain") except ValueError as e: self.send_error(HTTP.BAD_REQUEST, str(e)) except Exception as e: self._handle_exception(e, "defrag") return if self.name: try: content_length = int(self.headers.get("Content-Length", 0)) algorithm = "sha256" expected_hash = self.headers.get(f"X-Content-hash-{algorithm}") data = self.rfile.read(content_length) if expected_hash: got_hash = hashlib.new(algorithm, data).hexdigest() if got_hash != expected_hash: self.respond(HTTP.UNPROCESSABLE_ENTITY, b"Content hash verification failed, please retry") return with self.server.backend: self.server.backend.store(self.name, data) self.respond(HTTP.OK) except Exception as e: self._handle_exception(e, self.name) return self.send_error(HTTP.BAD_REQUEST, "Bad Request") @checks_and_logging def do_DELETE(self): cmd = self.query.get("cmd", [None])[0] if cmd == "rmdir": try: with self.server.backend: self.server.backend.rmdir(self.name) self.respond(HTTP.OK) except Exception as e: self._handle_exception(e, f"rmdir {self.name}") return if cmd == "destroy": try: self.server.backend.destroy() self.respond(HTTP.OK) except Exception as e: self._handle_exception(e, "destroy") return if not self.name: self.send_error(HTTP.BAD_REQUEST, "Bad Request") return try: with self.server.backend: self.server.backend.delete(self.name) self.respond(HTTP.OK) except Exception as e: self._handle_exception(e, self.name) @checks_and_logging def do_HEAD(self): if not self.name: self.send_error(HTTP.BAD_REQUEST, "Bad Request") return try: with self.server.backend: info = self.server.backend.info(self.name) if not info.exists: raise ObjectNotFound(self.name) self.respond( HTTP.OK, headers={ "Content-Length": str(info.size), "X-BorgStore-Is-Directory": "true" if info.directory else "false", }, ) except Exception as e: self._handle_exception(e, self.name) @checks_and_logging def do_GET(self): # List directory if self.split_url.path.endswith("/"): try: # send a JSON list of objects # [{"name": "...", "size": ...}, ...] with self.server.backend: items = ( {"name": item.name, "size": item.size, "directory": item.directory} for item in self.server.backend.list(self.name) ) json_data = json.dumps(list(items), indent=2) response_data = json_data.encode("utf-8") self.respond(HTTP.OK, data=response_data, content_type="application/json") except Exception as e: self._handle_exception(e, self.name) return # Load object if not self.name: self.send_error(HTTP.BAD_REQUEST, "Bad Request") return try: range_header = self.headers.get("Range") offset, size = parse_range_header(range_header) if range_header else (0, None) with self.server.backend: data = self.server.backend.load(self.name, offset=offset, size=size) self.respond( HTTP.PARTIAL_CONTENT if range_header else HTTP.OK, data=data, content_type="application/octet-stream" ) except Exception as e: self._handle_exception(e, self.name) def get_pre_bound_socket(): """Return pre-bound socket passed by systemd via socket activation. Reads LISTEN_FDS from the environment (set by systemd) and wraps each raw file descriptor (starting at fd 3) as a socket.socket object. See sd_listen_fds(3) for the protocol. """ n = int(os.environ.get("LISTEN_FDS", 0)) if n == 0: raise RuntimeError( "--socket-activation was requested but no sockets were passed by systemd (LISTEN_FDS not set or 0)" ) if n > 1: raise RuntimeError(f"--socket-activation expects exactly 1 socket from systemd, got {n}") # SD_LISTEN_FDS_START is always 3. The socket is a Unix domain socket # (as configured in borgstore@.socket), so use AF_UNIX / SOCK_STREAM. return socket.fromfd(3, socket.AF_UNIX, socket.SOCK_STREAM) class _UnclosableStream: """Wraps a binary stream and makes close() a no-op so that StreamRequestHandler.finish() cannot close sys.stdin/sys.stdout.""" def __init__(self, stream): self._stream = stream def close(self): pass # intentionally do nothing @property def closed(self): return self._stream.closed def read(self, *args, **kwargs): return self._stream.read(*args, **kwargs) def readline(self, *args, **kwargs): return self._stream.readline(*args, **kwargs) def peek(self, *args, **kwargs): return self._stream.peek(*args, **kwargs) def write(self, *args, **kwargs): return self._stream.write(*args, **kwargs) def flush(self, *args, **kwargs): return self._stream.flush(*args, **kwargs) class StdinStdoutSocket: """A mock socket that redirects reads to stdin and writes to stdout.""" def __init__(self): # Use .buffer to handle raw bytes instead of text strings self.rfile = sys.stdin.buffer self.wfile = sys.stdout.buffer def makefile(self, mode="r", buffering=None, encoding=None, errors=None, newline=None): """The HTTP request handler calls makefile() to get read/write streams. We wrap the underlying buffer in _UnclosableStream so that StreamRequestHandler.finish() cannot close sys.stdin/sys.stdout.""" if "r" in mode: return _UnclosableStream(self.rfile) elif "w" in mode: return _UnclosableStream(self.wfile) def sendall(self, data, flags=0): """Directly writes all data to stdout and flushes.""" self.wfile.write(data) self.wfile.flush() def send(self, data, flags=0): """Writes data to stdout and returns the number of bytes written.""" self.wfile.write(data) self.wfile.flush() return len(data) def recv(self, bufsize, flags=0): """Reads up to bufsize bytes from stdin.""" return self.rfile.read(bufsize) def getsockname(self): """Required by the server to log or bind addresses.""" return ("stdio", 0) def getpeername(self): """Required by the handler for logging client info.""" return ("stdio-client", 0) def close(self): """Prevent closing the actual sys.stdin/stdout prematurely.""" pass class StdIOHTTPServer(HTTPServer): """An HTTPServer variant that handles requests over stdin/stdout.""" def __init__(self, RequestHandlerClass): # Skip the base TCPServer __init__ entirely because we aren't binding to a network port host, port = "stdio", 0 self.server_name = host self.server_port = port self.server_address = (host, port) self.RequestHandlerClass = RequestHandlerClass # Instantiate our fake socket self.socket = StdinStdoutSocket() def serve_forever(self, poll_interval=0.5): """Continuously handle requests until stdin is empty/closed.""" while True: # Instantiate a fresh handler per request so that handler state # (close_connection, headers, etc.) never carries over between # requests. BaseHTTPRequestHandler.__init__ calls handle() which # calls handle_one_request() internally, so construction == handling. # # handle_one_request() sets raw_requestline=b"" and returns without # sending a response when readline() hits EOF (client closed stdin). # We detect that here and exit cleanly. # # Note: close_connection is NOT a reliable EOF signal — send_error() # and parse_request() both set it True for non-EOF reasons (e.g. 404). # _UnclosableStream ensures finish() cannot close sys.stdin/sys.stdout, # so the stream stays open across requests even after error responses. handler = self.RequestHandlerClass(self.socket, ("stdio-client", 0), self) if getattr(handler, "raw_requestline", b"") == b"": break class BorgStoreStdioRESTServer(StdIOHTTPServer): def __init__(self, backend, username=None, password=None): self.backend = backend self.username = username self.password = password super().__init__(BorgStoreRESTRequestHandler) class BorgStoreRESTServer(ThreadingHTTPServer): """ BorgStore REST Server. Security Warning: This server does not implement TLS. In a production environment, it SHOULD be run behind a reverse proxy (like Nginx or Caddy) that provides HTTPS. """ disable_nagle_algorithm = True # aka TCP_NODELAY, reduces latency def __init__(self, server_address, backend, username=None, password=None, adopted_socket=None): self.backend = backend self.username = username self.password = password if adopted_socket is not None: # Socket activation: systemd already bound and is listening on adopted_socket. # # TCPServer.__init__ unconditionally creates self.socket = socket.socket(...) # regardless of bind_and_activate, so we cannot set self.socket before calling # super().__init__. The correct sequence is: # 1. Call super().__init__ with bind_and_activate=False so it sets up # internal state (but also creates a fresh, unbound socket). # 2. Close and discard that fresh socket. # 3. Replace self.socket with the systemd-provided one. # 4. Read back server_address from the socket (getsockname()). # Do NOT call server_bind() or server_activate() — the socket is already # bound and listening; calling bind() again raises EADDRINUSE. self.address_family = socket.AF_UNIX # Unix sockets do not support TCP_NODELAY. self.disable_nagle_algorithm = False super().__init__(server_address, BorgStoreRESTRequestHandler, bind_and_activate=False) self.socket.close() # discard the socket super() created self.socket = adopted_socket # install the systemd socket self.server_address = self.socket.getsockname() # HTTPServer.server_bind usually sets these. We set them manually for AF_UNIX. self.server_name = "unix-socket" self.server_port = 0 else: super().__init__(server_address, BorgStoreRESTRequestHandler) def handle_error(self, request, client_address): # Ensure all errors are logged to the journal so we can see them in CI. logger.exception(f"Exception occurred during processing of request from {client_address}") super().handle_error(request, client_address) PERMISSION_SHORTCUTS = { # these are for borgbackup, see borg.repository.Repository.__init__ "borgbackup-all": None, # permissions system will not be used "borgbackup-no-delete": { # mostly no delete, no overwrite "": "lr", "archives": "lrw", "cache": "lrwWD", # WD for chunks., last-key-checked, ... "config": "lrW", # W for manifest "data": "lrw", "keys": "lr", "locks": "lrwD", # borg needs to create/delete a shared lock here }, "borgbackup-write-only": { # mostly no reading "": "l", "archives": "lw", "cache": "lrwWD", # read allowed, e.g. for chunks. cache "config": "lrW", # W for manifest "data": "lw", # no r! "keys": "lr", "locks": "lrwD", # borg needs to create/delete a shared lock here }, "borgbackup-read-only": {"": "lr", "locks": "lrwD"}, # mostly r/o } def resolve_permissions(permissions): """Resolve a permissions shortcut name or JSON string to a permissions dict (or None).""" if permissions is None: return None if permissions in PERMISSION_SHORTCUTS: return PERMISSION_SHORTCUTS[permissions] # Try to parse as JSON try: return json.loads(permissions) except json.JSONDecodeError: valid = ", ".join(PERMISSION_SHORTCUTS) raise ValueError(f"Invalid --permissions value: {permissions!r}. Use a shortcut ({valid}) or a JSON object.") def serve( host, port, backend_url, username=None, password=None, permissions=None, quota=None, socket_activation=False, stdio=False, ): if backend_url.startswith("FILE:"): # FILE: URIs are special: they are relative to the current working directory. path = backend_url[5:] # If path is relative, make it absolute. expand ~ and ~user. backend_url = Path(path).expanduser().resolve().as_uri() # Now we have a valid file:// URI! backend = get_backend(backend_url, permissions=permissions, quota=quota) if backend is None: raise ValueError(f"Invalid backend URL: {backend_url}") if stdio: server = BorgStoreStdioRESTServer(backend, username, password) logger.info("BorgStore REST server listening on stdin/stdout") elif socket_activation: adopted = get_pre_bound_socket() adopted.setblocking(True) server_address = adopted.getsockname() server = BorgStoreRESTServer(server_address, backend, username, password, adopted_socket=adopted) logger.info(f"BorgStore REST server using systemd-activated socket on {server_address}") else: server = BorgStoreRESTServer((host, port), backend, username, password) logger.info(f"BorgStore REST server listening on {host}:{port}") try: server.serve_forever() except KeyboardInterrupt: pass finally: server.server_close() def main(): logging.basicConfig(level=logging.INFO, format="%(message)s") logger.setLevel(logging.INFO) parser = argparse.ArgumentParser(description="BorgStore REST Server") parser.add_argument( "--host", default="127.0.0.1", help="Address/hostname to listen on (ignored with --socket-activation)" ) parser.add_argument("--port", type=int, default=5618, help="Port to listen on (ignored with --socket-activation)") parser.add_argument("--backend", required=True, help="Backend URL (e.g. file:///tmp/store)") parser.add_argument("--username", help="Basic Auth username") parser.add_argument("--password", help="Basic Auth password") parser.add_argument("--permissions", help="Permissions: a shortcut name or a JSON object string.") parser.add_argument("--quota", type=int, default=None, help="Quota in bytes.") parser.add_argument( "--socket-activation", action="store_true", help="Adopt pre-bound socket from systemd (SD_LISTEN_FDS)" ) parser.add_argument("--stdio", action="store_true", help="Serve on stdio") args = parser.parse_args() permissions = resolve_permissions(args.permissions) serve( args.host, args.port, args.backend, args.username, args.password, permissions, args.quota, args.socket_activation, args.stdio, ) if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1779964311.0 borgstore-0.5.1/src/borgstore/store.py0000644000076500000240000007527615206014627016427 0ustar00twstaff""" Key/value store implementation. The Store uses a backend to store key/value data and adds some functionality: - backend creation from a URL - configurable nesting - recursive list method - soft deletion """ from binascii import hexlify from collections import Counter from contextlib import contextmanager import enum import logging import os import time from typing import Iterator, NamedTuple, Optional from .utils.nesting import nest, unnest from .backends._base import ItemInfo, BackendBase from .backends.errors import ObjectNotFound, NoBackendGiven, BackendURLInvalid # noqa from .backends.posixfs import get_file_backend from .backends.rclone import get_rclone_backend from .backends.sftp import get_sftp_backend from .backends.s3 import get_s3_backend from .backends.rest import get_rest_backend from .constants import DEL_SUFFIX, ROOTNS logger = logging.getLogger(__name__) class CacheMode(enum.Enum): C_OFF = "off" C_MIRROR = "mirror" C_WRITETHROUGH = "writethrough" @classmethod def from_str(cls, value): if isinstance(value, cls): return value if isinstance(value, str): try: return cls(value.lower()) except ValueError as err: raise ValueError(f"unknown CacheMode: {value!r}") from err raise ValueError(f"unknown CacheMode: {value!r}") class CachePolicy(NamedTuple): mode: CacheMode max_age: Optional[float] size: Optional[int] def get_backend(url, permissions=None, quota=None): """Parse backend URL and return a backend instance (or None).""" backend = get_file_backend(url, permissions=permissions, quota=quota) if backend is not None: return backend if permissions is not None: raise ValueError("Permissions are only supported for the 'file:' backend.") if quota is not None: raise ValueError("Quota is only supported for the 'file:' backend.") backend = get_sftp_backend(url) if backend is not None: return backend backend = get_rclone_backend(url) if backend is not None: return backend backend = get_s3_backend(url) if backend is not None: return backend backend = get_rest_backend(url) if backend is not None: return backend class Store: def __init__( self, url: Optional[str] = None, backend: Optional[BackendBase] = None, config: Optional[dict] = None, permissions: Optional[dict] = None, *, cache_url: Optional[str] = None, cache_backend: Optional[BackendBase] = None, ): self.url = url if backend is None and url is not None: backend = get_backend(url, permissions=permissions) if backend is None: raise BackendURLInvalid(f"Invalid or unsupported Backend Storage URL: {url}") if backend is None: raise NoBackendGiven("You need to give a backend instance or a backend url.") self.backend = backend if not config or not isinstance(config, dict): raise ValueError("No or invalid config given.") levels_dict = {} cache_policies = {} for namespace, ns_config in config.items(): levels_list, policy = self._normalize_namespace_config(ns_config) levels_dict[namespace] = levels_list cache_policies[namespace] = policy self.set_levels(levels_dict) if cache_url is not None and cache_backend is not None: raise ValueError("Only one of cache_url and cache_backend can be given.") have_cache_enabled_namespaces = any(policy.mode != CacheMode.C_OFF for policy in cache_policies.values()) if have_cache_enabled_namespaces and cache_url is None and cache_backend is None: raise ValueError("cache_url or cache_backend is required for cache modes other than C_OFF.") self.cache_backend = cache_backend if cache_backend is not None else None if self.cache_backend is None and cache_url is not None: self.cache_backend = get_backend(cache_url) if self.cache_backend is None: raise BackendURLInvalid(f"Invalid or unsupported Cache Backend URL: {cache_url}") self._cache_disabled = False self.cache_namespaces = [ entry for entry in sorted( ((namespace, policy) for namespace, policy in cache_policies.items() if policy.mode != CacheMode.C_OFF), key=lambda item: len(item[0]), reverse=True, ) ] self._stats: Counter = Counter() # this is to emulate additional latency to what the backend actually offers: self.latency = float(os.environ.get("BORGSTORE_LATENCY", "0")) / 1e6 # [us] -> [s] # this is to emulate less bandwidth than what the backend actually offers: self.bandwidth = float(os.environ.get("BORGSTORE_BANDWIDTH", "0")) / 8 # [bits/s] -> [bytes/s] def __repr__(self): backend = self.backend.__class__.__name__ if self.backend is not None else None if self.cache_backend is not None: cache_backend = self.cache_backend.__class__.__name__ return f"" return f"" @staticmethod def _normalize_namespace_config(ns_config: dict) -> tuple[list[int], "CachePolicy"]: """Parse a per-namespace config dict into (levels, CachePolicy).""" if not isinstance(ns_config, dict): raise ValueError(f"Invalid namespace config: expected a dict, got {type(ns_config).__name__!r}.") unknown_keys = set(ns_config) - {"levels", "cache", "max_age", "size"} if unknown_keys: raise ValueError(f"Invalid namespace config keys: {sorted(unknown_keys)!r}") levels = ns_config.get("levels") if not levels or not isinstance(levels, list): raise ValueError("'levels' is required and must be a non-empty list of ints.") cache_val = ns_config.get("cache") if cache_val is None: policy = CachePolicy(mode=CacheMode.C_OFF, max_age=None, size=None) else: mode = CacheMode.from_str(cache_val) max_age = ns_config.get("max_age") if max_age is not None: if not isinstance(max_age, (int, float)) or max_age < 0: raise ValueError(f"Invalid cache max_age value: {max_age!r}") max_age = float(max_age) size = ns_config.get("size") if size is not None and (not isinstance(size, int) or size < 0): raise ValueError(f"Invalid cache size value: {size!r}") policy = CachePolicy(mode=mode, max_age=max_age, size=size) return levels, policy def _cache_policy_for(self, name: str) -> CachePolicy: for namespace, policy in self.cache_namespaces: if name.startswith(namespace): return policy return CachePolicy(mode=CacheMode.C_OFF, max_age=None, size=None) def set_levels(self, levels: dict, create: bool = False) -> None: if not levels or not isinstance(levels, dict): raise ValueError("No or invalid levels configuration given.") # we accept levels as a dict, but we rather want a list of (namespace, levels) tuples, longest namespace first: self.levels = [entry for entry in sorted(levels.items(), key=lambda item: len(item[0]), reverse=True)] if create: self.create_levels() def create_levels(self): """creating any needed namespaces / directory in advance""" # doing that saves a lot of ad-hoc mkdir calls, which is especially important # for backends with high latency or other noticeable costs of mkdir. with self: for namespace, levels in self.levels: namespace = namespace.rstrip("/") level = max(levels) cache_enabled = ( self.cache_backend is not None and not self._cache_disabled and self._cache_policy_for(f"{namespace}/").mode in {CacheMode.C_WRITETHROUGH, CacheMode.C_MIRROR} ) if level == 0: # flat, we just need to create the namespace directory: self.backend.mkdir(namespace) if cache_enabled: self.cache_backend.mkdir(namespace) elif level > 0: # nested, we only need to create the deepest nesting dir layer, # any missing parent dirs will be created as needed by backend.mkdir. limit = 2 ** (level * 8) for i in range(limit): dir = hexlify(i.to_bytes(length=level, byteorder="big")).decode("ascii") name = f"{namespace}/{dir}" if namespace else dir nested_name = nest(name, level) self.backend.mkdir(nested_name[: -2 * level - 1]) if cache_enabled: self.cache_backend.mkdir(nested_name[: -2 * level - 1]) else: raise ValueError(f"Invalid levels: {namespace}: {levels}") def create(self) -> None: self.backend.create() if self.cache_backend is not None and not self._cache_disabled: self.cache_backend.create() if self.backend.precreate_dirs: self.create_levels() def destroy(self) -> None: self.backend.destroy() if self.cache_backend is not None: self.cache_backend.destroy() def __enter__(self): self.open() return self def __exit__(self, exc_type, exc_val, exc_tb): self.close() return False def open(self) -> None: self.backend.open() if self.cache_backend is not None and not self._cache_disabled: try: self.cache_backend.open() except Exception as err: logger.warning(f"borgstore: cache open failed, disabling cache: {err!r}") self._cache_disabled = True else: self._cache_cleanup_expired() def close(self) -> None: self.backend.close() if self.cache_backend is not None: if not self._cache_disabled: self._cache_cleanup_expired() try: self.cache_backend.close() except Exception as err: logger.warning(f"borgstore: cache close failed: {err!r}") def quota(self) -> dict: return self.backend.quota() @contextmanager def _stats_updater(self, key, msg): """update call counters and overall times""" # do not use this in generators! volume_before = self._stats_get_volume(key) start = time.perf_counter_ns() yield end = time.perf_counter_ns() overall_time = end - start volume = self._stats_get_volume(key) - volume_before self._stats[f"{key}_calls"] += 1 self._stats[f"{key}_time"] += overall_time logger.debug(f"borgstore: {msg} -> {volume}B in {overall_time / 1e6:0.1f}ms") def _backend_call(self, operation, *, key=None, volume=0): # latency and bandwidth emulation is only applied to (primary) # backend calls, not to (secondary) cache backend calls. if key is not None: self._stats[f"backend_{key}_calls"] += 1 start = time.perf_counter_ns() result = operation() be_needed_ns = time.perf_counter_ns() - start volume = volume(result) if callable(volume) else volume if key is not None: self._stats[f"backend_{key}_volume"] += volume emulated_time = self.latency + (0 if not self.bandwidth else float(volume) / self.bandwidth) remaining_time = emulated_time - be_needed_ns / 1e9 if remaining_time > 0.0: time.sleep(remaining_time) return result def _stats_update_volume(self, key, amount): self._stats[f"{key}_volume"] += amount def _stats_get_volume(self, key): return self._stats.get(f"{key}_volume", 0) @property def stats(self): """ Return statistics such as method call counters, overall time [s], overall data volume, and overall throughput. Please note that the stats values only consider what is seen on the Store API: - There might be additional time spent by the caller, outside of Store, thus: - Real time is longer. - Real throughput is lower. - There are some overheads not accounted for, e.g., the volume only adds up the data size of load and store. - Write buffering or cached reads might give a wrong impression. """ st = dict(self._stats) # copy Counter -> generic dict for key in "info", "load", "store", "delete", "move", "list": # make sure key is present, even if method was not called st[f"{key}_calls"] = st.get(f"{key}_calls", 0) # convert integer ns timings to float s st[f"{key}_time"] = st.get(f"{key}_time", 0) / 1e9 for key in "load", "store": v = st.get(f"{key}_volume", 0) t = st.get(f"{key}_time", 0) st[f"{key}_throughput"] = v / t if t else 0 st["backend_load_calls"] = st.get("backend_load_calls", 0) st["backend_store_calls"] = st.get("backend_store_calls", 0) st["backend_delete_calls"] = st.get("backend_delete_calls", 0) st["backend_load_volume"] = st.get("backend_load_volume", 0) st["backend_store_volume"] = st.get("backend_store_volume", 0) st["cache_disabled"] = self._cache_disabled st["cache_hits"] = st.get("cache_hits", 0) st["cache_misses"] = st.get("cache_misses", 0) cache_total = st["cache_hits"] + st["cache_misses"] st["cache_hit_ratio"] = st["cache_hits"] / cache_total if cache_total else 0 st["cache_errors"] = st.get("cache_errors", 0) st["cache_load_calls"] = st.get("cache_load_calls", 0) st["cache_store_calls"] = st.get("cache_store_calls", 0) st["cache_delete_calls"] = st.get("cache_delete_calls", 0) st["cache_load_volume"] = st.get("cache_load_volume", 0) st["cache_store_volume"] = st.get("cache_store_volume", 0) return st def _get_levels(self, name): """Get levels from the configuration depending on the namespace.""" for namespace, levels in self.levels: if name.startswith(namespace): return levels # Store.create_levels requires all namespaces to be configured in self.levels. raise KeyError(f"no matching namespace found for: {name}") def find(self, name: str, *, deleted=False) -> str: """ Find an item checking all supported nesting levels and return its nested name: - item not in the store yet: we won't find it, but find will return a nested name for **last** level. - item is in the store already: find will return the same nested name as the already present item. If deleted is True, find will try to find a "deleted" item. """ nested_name = None suffix = DEL_SUFFIX if deleted else None levels = self._get_levels(name) if len(levels) == 1: # optimize the usual case: # the store is operating this namespace at a single specific level, # thus the item must be at that level, we do not need to search it. nested_name = nest(name, levels[0], add_suffix=suffix) else: # looks like the store is upgrading/downgrading levels, # items could be at old or new levels. for level in levels: nested_name = nest(name, level, add_suffix=suffix) info = self.backend.info(nested_name) if info.exists: break return nested_name def info(self, name: str, *, deleted=False) -> ItemInfo: with self._stats_updater("info", f"info({name!r}, deleted={deleted})"): return self._backend_call(lambda: self.backend.info(self.find(name, deleted=deleted)), volume=0) def _cache_load(self, nested_name: str) -> Optional[bytes]: if self.cache_backend is None or self._cache_disabled: return None self._stats["cache_load_calls"] += 1 try: value = self.cache_backend.load(nested_name) except ObjectNotFound: self._stats["cache_misses"] += 1 return None except Exception as err: logger.warning(f"borgstore: cache load failed for {nested_name!r}: {err!r}") self._stats["cache_errors"] += 1 return None self._stats["cache_hits"] += 1 self._stats["cache_load_volume"] += len(value) return value def load(self, name: str, *, size=None, offset=0, deleted=False) -> bytes: with self._stats_updater("load", f"load({name!r}, offset={offset}, size={size}, deleted={deleted})"): cache_policy = self._cache_policy_for(name) nested_name = self.find(name, deleted=deleted) if cache_policy.mode == CacheMode.C_WRITETHROUGH: full_value = self._cache_load(nested_name) if full_value is None: full_value = self._backend_call( lambda: self.backend.load(nested_name, size=None, offset=0), key="load", volume=lambda value: len(value), ) self._cache_store(nested_name, full_value) elif cache_policy.mode == CacheMode.C_MIRROR: full_value = self._backend_call( lambda: self.backend.load(nested_name, size=None, offset=0), key="load", volume=lambda value: len(value), ) self._cache_store(nested_name, full_value) else: result = self._backend_call( lambda: self.backend.load(nested_name, size=size, offset=offset), key="load", volume=lambda value: len(value), ) self._stats_update_volume("load", len(result)) return result result = full_value[offset : (None if size is None else offset + size)] self._stats_update_volume("load", len(result)) return result def _cache_store(self, nested_name: str, value: bytes) -> None: if self.cache_backend is None or self._cache_disabled: return self._stats["cache_store_calls"] += 1 try: self.cache_backend.store(nested_name, value) self._stats["cache_store_volume"] += len(value) except Exception as err: logger.warning(f"borgstore: cache store failed for {nested_name!r}: {err!r}") self._stats["cache_errors"] += 1 def store(self, name: str, value: bytes) -> None: # note: using .find here will: # - overwrite an existing item (level stays same) # - write to the last level if no existing item is found. with self._stats_updater("store", f"store({name!r})"): nested_name = self.find(name) self._backend_call(lambda: self.backend.store(nested_name, value), key="store", volume=len(value)) if self._cache_policy_for(name).mode in {CacheMode.C_WRITETHROUGH, CacheMode.C_MIRROR}: self._cache_store(nested_name, value) self._stats_update_volume("store", len(value)) def _cache_delete(self, nested_name: str) -> None: if self.cache_backend is None or self._cache_disabled: return self._stats["cache_delete_calls"] += 1 try: self.cache_backend.delete(nested_name) except ObjectNotFound: pass except Exception as err: logger.warning(f"borgstore: cache delete failed for {nested_name!r}: {err!r}") self._stats["cache_errors"] += 1 def delete(self, name: str, *, deleted=False) -> None: """ Really and immediately deletes an item. See also .move(name, delete=True) for "soft" deletion. """ with self._stats_updater("delete", f"delete({name!r}, deleted={deleted})"): nested_name = self.find(name, deleted=deleted) self._backend_call(lambda: self.backend.delete(nested_name), key="delete", volume=0) if self._cache_policy_for(name).mode in {CacheMode.C_WRITETHROUGH, CacheMode.C_MIRROR}: self._cache_delete(nested_name) def cache_invalidate(self, name: str, *, deleted: bool = False) -> None: """ Invalidate cached items. - If name is ROOTNS (""), invalidate caches of all cached namespaces. - If a namespace is given, invalidate all items in that namespace. - If an item name is given, invalidate only that single item. """ if self.cache_backend is None or self._cache_disabled: return if name == ROOTNS: # Root / all namespaces for namespace, policy in self.cache_namespaces: for info in self._cache_list(namespace.rstrip("/")): if not info.directory: self._cache_delete(info.name) else: # Check if name represents a namespace target_namespace = None for namespace, policy in self.cache_namespaces: if namespace.rstrip("/") == name.rstrip("/"): target_namespace = namespace break if target_namespace is not None: # Invalidate all items in the namespace for info in self._cache_list(target_namespace.rstrip("/")): if not info.directory: self._cache_delete(info.name) else: # Invalidate single item nested_name = self.find(name, deleted=deleted) self._cache_delete(nested_name) def _cache_move(self, old_nested: str, new_nested: str) -> None: if self.cache_backend is None or self._cache_disabled: return try: self.cache_backend.move(old_nested, new_nested) except ObjectNotFound: pass except Exception as err: logger.warning(f"borgstore: cache move failed for {old_nested!r}->{new_nested!r}: {err!r}") self._stats["cache_errors"] += 1 def move( self, name: str, new_name: Optional[str] = None, *, delete: bool = False, undelete: bool = False, change_level: bool = False, deleted: bool = False, ) -> None: if delete: # use case: keep name, but soft "delete" the item nested_name = self.find(name, deleted=False) nested_new_name = nested_name + DEL_SUFFIX msg = f"soft_delete({name!r}, deleted={deleted})" elif undelete: # use case: keep name, undelete a previously soft "deleted" item nested_name = self.find(name, deleted=True) nested_new_name = nested_name.removesuffix(DEL_SUFFIX) msg = f"soft_undelete({name!r}, deleted={deleted})" elif change_level: # use case: keep name, changing to another nesting level suffix = DEL_SUFFIX if deleted else None nested_name = self.find(name, deleted=deleted) nested_new_name = nest(name, self._get_levels(name)[-1], add_suffix=suffix) msg = f"change_level({name!r}, deleted={deleted})" else: # generic use (be careful!) if not new_name: raise ValueError("Generic move requires new_name to be given.") nested_name = self.find(name, deleted=deleted) nested_new_name = self.find(new_name, deleted=deleted) msg = f"rename({name!r}, {new_name!r}, deleted={deleted})" with self._stats_updater("move", msg + f" [{nested_name!r}, {nested_new_name!r}]"): self._backend_call(lambda: self.backend.move(nested_name, nested_new_name), volume=0) if self._cache_policy_for(name).mode in {CacheMode.C_WRITETHROUGH, CacheMode.C_MIRROR}: self._cache_move(nested_name, nested_new_name) def _cache_list(self, name: str) -> Iterator[ItemInfo]: if self.cache_backend is None: return for info in self.cache_backend.list(name): if info.directory: subdir_name = (name + "/" + info.name) if name else info.name yield from self._cache_list(subdir_name) else: full_name = (name + "/" + info.name) if name else info.name yield info._replace(name=full_name) def list(self, name: str, deleted: bool = False) -> Iterator[ItemInfo]: """ List all names in the namespace . If deleted is False (default), only non-deleted items are yielded. If deleted is True, only soft-deleted items are yielded. backend.list giving us sorted names implies Store.list is also sorted, if all items are stored on the same level. Note: list bypasses the cache and always queries the primary backend to ensure we only return items that really exist there, even if other clients have updated or deleted items directly in the primary backend. """ # we need this wrapper due to the recursion - we only want to increment list_calls once: logger.debug(f"borgstore: list_start({name!r}, deleted={deleted})") self._stats["list_calls"] += 1 count = 0 try: for info in self._list(name, deleted=deleted): count += 1 yield info finally: # note: as this is a generator, we do not measure the execution time because # that would include the time needed by the caller to process the infos. logger.debug(f"borgstore: list_end({name!r}, deleted={deleted}) -> {count}") def _list(self, name: str, deleted: bool = False) -> Iterator[ItemInfo]: # as the backend.list method only supports non-recursive listing and # also returns directories/namespaces we introduced for nesting, we do the # recursion here (and also we do not yield directory names from here). start = time.perf_counter_ns() backend_list_iterator = self.backend.list(name) if self.latency: # we add the simulated latency once per backend.list iteration, not per element. time.sleep(self.latency) end = time.perf_counter_ns() self._stats["list_time"] += end - start while True: start = time.perf_counter_ns() try: info = next(backend_list_iterator) except StopIteration: break finally: end = time.perf_counter_ns() self._stats["list_time"] += end - start if info.directory: # note: we only expect subdirectories from key nesting, but not namespaces nested into each other. subdir_name = (name + "/" + info.name) if name else info.name yield from self._list(subdir_name, deleted=deleted) else: is_deleted = info.name.endswith(DEL_SUFFIX) if deleted and is_deleted: yield info._replace(name=info.name.removesuffix(DEL_SUFFIX)) elif not deleted and not is_deleted: yield info def hash(self, name: str, algorithm: str = "sha256", *, deleted: bool = False) -> str: with self._stats_updater("hash", f"hash({name!r}, algorithm={algorithm!r}, deleted={deleted})"): return self._backend_call( lambda: self.backend.hash(self.find(name, deleted=deleted), algorithm=algorithm), volume=0 ) def defrag(self, sources, *, target=None, algorithm=None, namespace=None, deleted=False) -> str: """ efficiently create a new item (target) by combining blocks from existing items (sources) in the same namespace. item and target names are always without namespace. sources is a list of (name, block_offset, block_length) tuples. blocks will be processed in order of appearance in the list and their contents will be appended to the target item. if the target name is not given, algorithm must be given to compute the target name as hash(algorithm, target_content).hexdigest(). returns the target name. """ prefix = (namespace + "/") if namespace else "" mapped_sources = [ (self.find(prefix + source, deleted=deleted), offset, size) for source, offset, size in sources ] if target is not None: target = self.find(prefix + target, deleted=deleted) # Note: defrag does not interact with the cache. It creates a new item from # the chunks of the source items we want to keep. It does not delete the source # items; that is the task of the caller after defrag successfully returns the new # item name. If the caller subsequently deletes the source items, they will be # removed from the cache. levels = self._get_levels(prefix)[-1] if prefix else 0 backend_target = self.backend.defrag( mapped_sources, target=target, algorithm=algorithm, namespace=prefix.rstrip("/"), levels=levels ) return unnest(backend_target, namespace=prefix).removeprefix(prefix) def _cache_cleanup_expired(self) -> None: now = time.time() for namespace, policy in self.cache_namespaces: if policy.max_age is None and policy.size is None: continue try: items = [info for info in self._cache_list(namespace.rstrip("/")) if not info.directory] if policy.max_age is not None: remaining_items = [] for info in items: if not info.atime or (now - info.atime) > policy.max_age: self._cache_delete(info.name) else: remaining_items.append(info) items = remaining_items if policy.size is not None: total_size = sum(info.size for info in items) for info in sorted(items, key=lambda entry: (entry.atime, entry.name)): if total_size <= policy.size: break self._cache_delete(info.name) total_size -= info.size except Exception as err: logger.warning(f"borgstore: cache cleanup failed for namespace {namespace!r}: {err!r}") self._stats["cache_errors"] += 1 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1780354067.0892603 borgstore-0.5.1/src/borgstore/utils/0000755000076500000240000000000015207406023016034 5ustar00twstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1757407943.0 borgstore-0.5.1/src/borgstore/utils/__init__.py0000644000076500000240000000004515057765307020164 0ustar00twstaff"""Utility helpers for BorgStore.""" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1773543560.0 borgstore-0.5.1/src/borgstore/utils/nesting.py0000644000076500000240000000521615155420210020055 0ustar00twstaff""" Nest/un-nest names to address directory scalability issues and handle the suffix for deleted items. Many filesystem directory implementations do not cope well with extremely large numbers of entries, so we introduce intermediate directories to reduce the number of entries per directory. The name is expected to have the key as the last element, for example: name = "namespace/0123456789abcdef" # often, the key is hex(hash(content)) As we can have a huge number of keys, we could nest 2 levels deep: nested_name = nest(name, 2) nested_name == "namespace/01/23/0123456789abcdef" Note that the final element is the full key — this is better to deal with in case of errors (for example, a filesystem issue and items being pushed to lost+found) and also easier to handle (e.g., a directory listing directly yields keys without needing to reassemble the full key from parent directories and partial keys). Also, a sorted directory listing has the same order as a sorted key list. name = unnest(nested_name, namespace="namespace") # a namespace with a final slash is also supported name == "namespace/0123456789abcdef" Notes: - It works the same way without a namespace, but we recommend always using a namespace. - Always use nest/unnest, even if levels == 0 are desired, as they also perform some checks and handle adding/removing a suffix. """ from typing import Optional def split_key(name: str) -> tuple[Optional[str], str]: namespace_key = name.rsplit("/", 1) if len(namespace_key) == 2: namespace, key = namespace_key else: # == 1 (no slash in name) namespace, key = None, name return namespace, key def nest(name: str, levels: int, *, add_suffix: Optional[str] = None) -> str: """namespace/12345678 --2 levels--> namespace/12/34/12345678""" if levels > 0: namespace, key = split_key(name) parts = [key[2 * level : 2 * level + 2] for level in range(levels)] parts.append(key) if namespace is not None: parts.insert(0, namespace) name = "/".join(parts) return (name + add_suffix) if add_suffix else name def unnest(name: str, namespace: str, *, remove_suffix: Optional[str] = None) -> str: """namespace/12/34/12345678 --namespace=namespace--> namespace/12345678""" if namespace: if not namespace.endswith("/"): namespace += "/" if not name.startswith(namespace): raise ValueError(f"name {name} does not start with namespace {namespace}") name = name.removeprefix(namespace) key = name.rsplit("/", 1)[-1] if remove_suffix: key = key.removesuffix(remove_suffix) return namespace + key ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1780354067.0895479 borgstore-0.5.1/src/borgstore.egg-info/0000755000076500000240000000000015207406023016366 5ustar00twstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780354067.0 borgstore-0.5.1/src/borgstore.egg-info/PKG-INFO0000644000076500000240000000772315207406023017474 0ustar00twstaffMetadata-Version: 2.4 Name: borgstore Version: 0.5.1 Summary: key/value store Author-email: Thomas Waldmann License-Expression: BSD-3-Clause Project-URL: Homepage, https://github.com/borgbackup/borgstore Keywords: kv,key/value,store Classifier: Development Status :: 3 - Alpha Classifier: Intended Audience :: Developers Classifier: Operating System :: POSIX Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Classifier: Programming Language :: Python :: 3.12 Classifier: Programming Language :: Python :: 3.13 Classifier: Programming Language :: Python :: 3.14 Classifier: Topic :: Software Development :: Libraries Classifier: Topic :: Software Development :: Libraries :: Python Modules Requires-Python: >=3.10 Description-Content-Type: text/x-rst License-File: LICENSE.rst Provides-Extra: rest Requires-Dist: requests>=2.25.1; extra == "rest" Provides-Extra: rclone Requires-Dist: requests>=2.25.1; extra == "rclone" Provides-Extra: sftp Requires-Dist: paramiko>=1.9.1; extra == "sftp" Provides-Extra: s3 Requires-Dist: boto3; extra == "s3" Provides-Extra: none Dynamic: license-file BorgStore ========= borgstore implements a general purpose key/value store in Python. Overview -------- Keys are simple strings like `config/main` or `data/0123456789abcdef` `[str]` (config and data are namespaces here). Values are binary objects `[bytes]`. The `Store` class is the high-level API, so you can comfortably work with the kv store without caring for low-level details. The `backends` package has misc. storage backend implementations. The `server` package has a REST server implementation, complementing the REST client functionality in the `rest` backend. To actually store stuff, the REST server can use any backend internally, e.g. the `posixfs` backend. Store features -------------- - supports URLs, like `file:///srv/borgstore` or `https://myserver/path` - easy to use, high-level `Store` API: create/destroy, open/close, list, load/store, delete, move, soft delete/undelete, hash, defrag, ... - uses a backend to implement the storage - optionally uses an additional caching backend, with a configurable cache policy per namespace - name nesting / unnesting, recursive directory listing - statistics collection - latency/bandwidth emulator Backend features ---------------- - existing backends for local filesystem, sftp, REST, S3 / B2 (native) and many other cloud storage protocols via rclone - new backends are simple to implement - key validation - partial loads / range requests - stored object hashing - stored object defragmentation - quota support (only `posixfs`) - permissions checking (only `posixfs`) REST server features -------------------- - server-side permissions/quota enforcement - server-side hashsum check of transferred objects before storing - network traffic optimization by doing stuff server-side: - stored object hashing - stored object defragmentation - the REST server can internally use any backend for storage, e.g. `posixfs` - for the REST server, we provide CI tested configs for: - an nginx-based reverse proxy - systemd-based on-demand `borgstore.server` process creation State of this project --------------------- **API is still unstable and expected to change as development goes on.** **As long as the API is unstable, there will be no data migration tools, such as tools for upgrading an existing store's data to a new release.** There are tests, and they pass for the basic functionality, so some functionality is already working well. There might be missing features or optimization potential. Feedback is welcome! Many possible backends are still missing. If you want to create and support one, pull requests are welcome. Borg? ----- Please note that this code is currently **not** used by the stable release of BorgBackup (also known as "borg"), but only by Borg 2 beta 10+ and the master branch. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780354067.0 borgstore-0.5.1/src/borgstore.egg-info/SOURCES.txt0000644000076500000240000000242115207406023020251 0ustar00twstaffAUTHORS.rst LICENSE.rst README.rst pyproject.toml contrib/server/nginx-systemd/README.md contrib/server/nginx-systemd/borgstore-proxy.conf contrib/server/nginx-systemd/borgstore@.service contrib/server/nginx-systemd/borgstore@.socket contrib/server/nginx-systemd/nginx-borgstore.conf contrib/server/nginx-systemd/repo1.env.example docs/Makefile docs/authors.rst docs/backends.rst docs/changes.rst docs/conf.py docs/index.rst docs/installation.rst docs/servers.rst docs/store.rst docs/store_caching.rst docs/_templates/layout.html src/borgstore/__init__.py src/borgstore/__main__.py src/borgstore/_version.py src/borgstore/constants.py src/borgstore/store.py src/borgstore.egg-info/PKG-INFO src/borgstore.egg-info/SOURCES.txt src/borgstore.egg-info/dependency_links.txt src/borgstore.egg-info/entry_points.txt src/borgstore.egg-info/requires.txt src/borgstore.egg-info/top_level.txt src/borgstore/backends/__init__.py src/borgstore/backends/_base.py src/borgstore/backends/_utils.py src/borgstore/backends/errors.py src/borgstore/backends/posixfs.py src/borgstore/backends/rclone.py src/borgstore/backends/rest.py src/borgstore/backends/s3.py src/borgstore/backends/sftp.py src/borgstore/server/__init__.py src/borgstore/server/rest.py src/borgstore/utils/__init__.py src/borgstore/utils/nesting.py././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780354067.0 borgstore-0.5.1/src/borgstore.egg-info/dependency_links.txt0000644000076500000240000000000115207406023022434 0ustar00twstaff ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780354067.0 borgstore-0.5.1/src/borgstore.egg-info/entry_points.txt0000644000076500000240000000010515207406023021660 0ustar00twstaff[console_scripts] borgstore-server-rest = borgstore.server.rest:main ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780354067.0 borgstore-0.5.1/src/borgstore.egg-info/requires.txt0000644000076500000240000000014015207406023020761 0ustar00twstaff [none] [rclone] requests>=2.25.1 [rest] requests>=2.25.1 [s3] boto3 [sftp] paramiko>=1.9.1 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1780354067.0 borgstore-0.5.1/src/borgstore.egg-info/top_level.txt0000644000076500000240000000001215207406023021111 0ustar00twstaffborgstore