BTrees-4.3.1/0000775000175000017500000000000012716353165014062 5ustar tseavertseaver00000000000000BTrees-4.3.1/setup.cfg0000664000175000017500000000045212716353165015704 0ustar tseavertseaver00000000000000[nosetests] nocapture = 1 cover-package = BTrees cover-erase = 1 cover-branches = 1 cover-min-percentage = 100 with-doctest = 0 where = BTrees [aliases] dev = develop easy_install BTrees[testing] docs = develop easy_install BTrees[docs] [egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 BTrees-4.3.1/buildout.cfg0000664000175000017500000000150112510775554016371 0ustar tseavertseaver00000000000000[buildout] eggs = persistent zope.interface parts = w_persistent develop test test_w_zodb scripts [w_persistent] # Generate an interpreter w/ persistent installed, so that the # 'develop' phase doesn't faux-install the eggs in the root and # screw up the reset of the bulidout. recipe = zc.recipe.egg interpreter = w_persistent eggs = persistent [develop] recipe = zc.recipe.egg:develop setup = . executable = ${buildout:bin-directory}/w_persistent # Force the :develop recipe to prefer the excecutable defined in this section. python = develop [test] recipe = zc.recipe.testrunner eggs = BTrees [test] ${buildout:eggs} defaults = ['-sBTrees'] [test_w_zodb] <= test eggs = BTrees [test,ZODB] ${buildout:eggs} [scripts] recipe = zc.recipe.egg eggs = ${test_w_zodb:eggs} interpreter = py BTrees-4.3.1/appveyor.yml0000664000175000017500000000251012714130453016437 0ustar tseavertseaver00000000000000environment: password: secure: RtpeKCle25vCixaUcJBu6Q== matrix: - python : 27 - python : 27-x64 - python : 33 - python : 33-x64 - python : 34 - python : 34-x64 - python : 35 - python : 35-x64 install: - "SET PATH=C:\\Python%PYTHON%;c:\\Python%PYTHON%\\scripts;%PATH%" - echo "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 > "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\vcvars64.bat" - pip install -e .[ZODB] build: false test_script: - python setup.py -q test -q on_success: - echo Build succesful! deploy_script: - echo [distutils] > %USERPROFILE%\\.pypirc - echo index-servers = >> %USERPROFILE%\\.pypirc - echo pypi >> %USERPROFILE%\\.pypirc - echo [pypi] >> %USERPROFILE%\\.pypirc - echo repository=https://pypi.python.org/pypi >> %USERPROFILE%\\.pypirc - echo username=zope.wheelbuilder >> %USERPROFILE%\\.pypirc - echo password=%password% >> %USERPROFILE%\\.pypirc - set HOME=%USERPROFILE% - pip install wheel twine - ps: if($env:APPVEYOR_REPO_TAG -eq $TRUE) { python -W ignore setup.py bdist_wheel bdist_egg; twine upload dist/* } deploy : on BTrees-4.3.1/PKG-INFO0000664000175000017500000002143312716353165015162 0ustar tseavertseaver00000000000000Metadata-Version: 1.1 Name: BTrees Version: 4.3.1 Summary: Scalable persistent object containers Home-page: http://packages.python.org/BTrees Author: Zope Foundation Author-email: zodb-dev@zope.org License: ZPL 2.1 Description: ``BTrees``: scalable persistent components =========================================== .. image:: https://travis-ci.org/zopefoundation/BTrees.svg?branch=master :target: https://travis-ci.org/zopefoundation/BTrees .. image:: https://ci.appveyor.com/api/projects/status/github/zopefoundation/BTrees?branch=master&svg=true :target: https://ci.appveyor.com/project/mgedmin/BTrees This package contains a set of persistent object containers built around a modified BTree data structure. The trees are optimized for use inside ZODB's "optimistic concurrency" paradigm, and include explicit resolution of conflicts detected by that mechannism. Please see the Sphinx documentation (``docs/index.rst``) for further information. ``BTrees`` Changelog ==================== 4.3.1 (2016-05-16) ------------------ - Packaging: fix password used to automate wheel creation on Travis. 4.3.0 (2016-05-10) ------------------ - Fix unexpected ``OverflowError`` when passing 64bit values to long keys / values on Win64. See: https://github.com/zopefoundation/BTrees/issues/32 - When testing ``PURE_PYTHON`` environments under ``tox``, avoid poisoning the user's global wheel cache. - Ensure that he pure-Python implementation, used on PyPy and when a C compiler isn't available for CPython, pickles identically to the C version. Unpickling will choose the best available implementation. This change prevents interoperability problems and database corruption if both implementations are in use. While it is no longer possible to pickle a Python implementation and have it unpickle to the Python implementation if the C implementation is available, existing Python pickles will still unpickle to the Python implementation (until pickled again). See: https://github.com/zopefoundation/BTrees/issues/19 - Avoid creating invalid objects when unpickling empty BTrees in a pure-Python environment. - Drop support for Python 2.6 and 3.2. 4.2.0 (2015-11-13) ------------------ - Add support for Python 3.5. 4.1.4 (2015-06-02) ------------------ - Ensure that pure-Python Bucket and Set objects have a human readable ``__repr__`` like the C versions. 4.1.3 (2015-05-19) ------------------ - Fix ``_p_changed`` when removing items from small pure-Python BTrees/TreeSets and when adding items to small pure-Python Sets. See: https://github.com/zopefoundation/BTrees/issues/13 4.1.2 (2015-04-07) ------------------ - Suppress testing 64-bit values in OLBTrees on 32 bit machines. See: https://github.com/zopefoundation/BTrees/issues/9 - Fix ``_p_changed`` when adding items to small pure-Python BTrees/TreeSets. See: https://github.com/zopefoundation/BTrees/issues/11 4.1.1 (2014-12-27) ------------------ - Accomodate long values in pure-Python OLBTrees. 4.1.0 (2014-12-26) ------------------ - Add support for PyPy and PyPy3. - Add support for Python 3.4. - BTree subclasses can define ``max_leaf_size`` or ``max_internal_size`` to control maximum sizes for Bucket/Set and BTree/TreeSet nodes. - Detect integer overflow on 32-bit machines correctly under Python 3. - Update pure-Python and C trees / sets to accept explicit None to indicate max / min value for ``minKey``, ``maxKey``. (PR #3) - Update pure-Python trees / sets to accept explicit None to indicate open ranges for ``keys``, ``values``, ``items``. (PR #3) 4.0.8 (2013-05-25) ------------------ - Fix value-based comparison for objects under Py3k: addresses invalid merges of ``[OLI]OBTrees/OBuckets``. - Ensure that pure-Python implementation of ``OOBTree.byValue`` matches semantics (reversed-sort) of C implementation. 4.0.7 (2013-05-22) ------------------ - Issue #2: compilation error on 32-bit mode of OS/X. - Test ``PURE_PYTHON`` environment variable support: if set, the C extensions will not be built, imported, or tested. 4.0.6 (2013-05-14) ------------------ - Changed the ``ZODB`` extra to require only the real ``ZODB`` package, rather than the ``ZODB3`` metapackage: depending on the version used, the metapackage could pull in stale versions of **this** package and ``persistent``. - Fixed Python version check in ``setup.py``. 4.0.5 (2013-01-15) ------------------ - Fit the ``repr`` of bucket objects, which could contain garbage characters. 4.0.4 (2013-01-12) ------------------ - Emulate the (private) iterators used by the C extension modules from pure Python. This change is "cosmetic" only: it prevents the ZCML ``zope.app.security:permission.zcml`` from failing. The emulated classes are **not** functional, and should be considered implementation details. - Accomodate buildout to the fact that we no longer bundle a copy of 'persistent.h'. - Fix test failures on Windows: no longer rely on overflows from ``sys.maxint``. 4.0.3 (2013-01-04) ------------------ - Added ``setup_requires==['persistent']``. 4.0.2 (2013-01-03) ------------------ - Updated Trove classifiers. - Added explicit support for Python 3.2, Python 3.3, and PyPy. Note that the C extensions are not (yet) available on PyPy. - Python reference implementations now tested separately from the C verions on all platforms. - 100% unit test coverage. 4.0.1 (2012-10-21) ------------------ - Provide local fallback for persistent C header inclusion if the persistent distribution isn't installed. This makes the winbot happy. 4.0.0 (2012-10-20) ------------------ Platform Changes ################ - Dropped support for Python < 2.6. - Factored ``BTrees`` as a separate distribution. Testing Changes ############### - All covered platforms tested under ``tox``. - Added support for continuous integration using ``tox`` and ``jenkins``. - Added ``setup.py dev`` alias (installs ``nose`` and ``coverage``). - Dropped dependency on ``zope.testing`` / ``zope.testrunner``: tests now run with ``setup.py test``. Documentation Changes ##################### - Added API reference, generated via Spinx' autodoc. - Added Sphinx documentation based on ZODB Guide (snippets are exercised via 'tox'). - Added ``setup.py docs`` alias (installs ``Sphinx`` and ``repoze.sphinx.autointerface``). Platform: any Classifier: Development Status :: 6 - Mature Classifier: License :: OSI Approved :: Zope Public License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Framework :: ZODB Classifier: Topic :: Database Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: Unix BTrees-4.3.1/tox.ini0000664000175000017500000000244712703730537015402 0ustar tseavertseaver00000000000000[tox] envlist = # Jython support pending 2.7 support, due 2012-07-15 or so. See: # http://fwierzbicki.blogspot.com/2012/03/adconion-to-fund-jython-27.html # py27,jython,pypy,coverage,docs py27,py27-pure,pypy,py33,py34,py35,pypy3,w_zodb,coverage,docs [testenv] deps = zope.interface persistent transaction commands = python setup.py -q test -q [testenv:py27-pure] basepython = python2.7 setenv = PURE_PYTHON = 1 PIP_CACHE_DIR = {envdir}/.cache deps = {[testenv]deps} commands = python setup.py -q test -q #[testenv:jython] #commands = # jython setup.py test -q [testenv:w_zodb] basepython = python2.7 commands = python setup.py -q test -q deps = zope.interface persistent transaction ZODB nose coverage nosexcover [testenv:coverage] basepython = python2.7 commands = nosetests --with-xunit --with-xcoverage --cover-package=BTrees deps = zope.interface persistent transaction nose coverage nosexcover [testenv:docs] basepython = python2.7 commands = sphinx-build -b html -d docs/_build/doctrees docs docs/_build/html sphinx-build -b doctest -d docs/_build/doctrees docs docs/_build/doctest deps = zope.interface persistent transaction Sphinx repoze.sphinx.autointerface BTrees-4.3.1/.gitignore0000664000175000017500000000025212510776217016050 0ustar tseavertseaver00000000000000bin eggs develop-eggs parts .installed.cfg build docs/_build __pycache__ *.pyc *.so .tox .coverage nosetests.xml coverage.xml *.egg-info *.egg dist .eggs/ .dir-locals.el BTrees-4.3.1/setup.py0000664000175000017500000001233412716353114015571 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## __version__ = '4.3.1' import os import platform import sys from setuptools import Extension from setuptools import find_packages from setuptools import setup here = os.path.abspath(os.path.dirname(__file__)) README = (open(os.path.join(here, 'README.rst')).read() + '\n\n' + open(os.path.join(here, 'CHANGES.rst')).read()) # Include directories for C extensions # Sniff the location of the headers in 'persistent' or fall back # to local headers in the include sub-directory class ModuleHeaderDir(object): def __init__(self, require_spec, where='..'): # By default, assume top-level pkg has the same name as the dist. # Also assume that headers are located in the package dir, and # are meant to be included as follows: # #include "module/header_name.h" self._require_spec = require_spec self._where = where def __str__(self): from pkg_resources import require from pkg_resources import resource_filename require(self._require_spec) path = resource_filename(self._require_spec, self._where) return os.path.abspath(path) include = [ModuleHeaderDir('persistent')] # Set up dependencies for the BTrees package base_btrees_depends = [ "BTrees/BTreeItemsTemplate.c", "BTrees/BTreeModuleTemplate.c", "BTrees/BTreeTemplate.c", "BTrees/BucketTemplate.c", "BTrees/MergeTemplate.c", "BTrees/SetOpTemplate.c", "BTrees/SetTemplate.c", "BTrees/TreeSetTemplate.c", "BTrees/sorters.c", ] FLAVORS = {"O": "object", "I": "int", "F": "float", 'L': 'int'} #XXX should 'fs' be in ZODB instead? FAMILIES = ("OO", "IO", "OI", "II", "IF", "fs", "LO", "OL", "LL", "LF") KEY_H = "BTrees/%skeymacros.h" VALUE_H = "BTrees/%svaluemacros.h" def BTreeExtension(family): key = family[0] value = family[1] name = "BTrees._%sBTree" % family sources = ["BTrees/_%sBTree.c" % family] kwargs = {"include_dirs": include} if family != "fs": kwargs["depends"] = (base_btrees_depends + [KEY_H % FLAVORS[key], VALUE_H % FLAVORS[value]]) else: kwargs["depends"] = base_btrees_depends if key != "O": kwargs["define_macros"] = [('EXCLUDE_INTSET_SUPPORT', None)] return Extension(name, sources, **kwargs) py_impl = getattr(platform, 'python_implementation', lambda: None) pure_python = os.environ.get('PURE_PYTHON', False) is_pypy = py_impl() == 'PyPy' is_jython = 'java' in sys.platform # Jython cannot build the C optimizations, while on PyPy they are # anti-optimizations (the C extension compatibility layer is known-slow, # and defeats JIT opportunities). if pure_python or is_pypy or is_jython: ext_modules = [] else: ext_modules = [BTreeExtension(family) for family in FAMILIES] if sys.version_info[0] >= 3: REQUIRES = [ 'persistent>=4.0.4', 'zope.interface', ] else: REQUIRES = [ 'persistent', 'zope.interface', ] TESTS_REQUIRE = REQUIRES + ['transaction'] setup(name='BTrees', version=__version__, description='Scalable persistent object containers', long_description=README, classifiers=[ "Development Status :: 6 - Mature", "License :: OSI Approved :: Zope Public License", "Programming Language :: Python", 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", "Framework :: ZODB", "Topic :: Database", "Topic :: Software Development :: Libraries :: Python Modules", "Operating System :: Microsoft :: Windows", "Operating System :: Unix", ], author="Zope Foundation", author_email="zodb-dev@zope.org", url="http://packages.python.org/BTrees", license="ZPL 2.1", platforms=["any"], packages=find_packages(), include_package_data=True, zip_safe=False, ext_modules = ext_modules, setup_requires=['persistent'], extras_require = { 'test': TESTS_REQUIRE, 'ZODB': ['ZODB'], 'testing': TESTS_REQUIRE + ['nose', 'coverage'], 'docs': ['Sphinx', 'repoze.sphinx.autointerface'], }, test_suite="BTrees.tests", tests_require=TESTS_REQUIRE, install_requires=REQUIRES, entry_points = """\ """ ) BTrees-4.3.1/bootstrap.py0000664000175000017500000001644212714130453016447 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2006 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Bootstrap a buildout-based project Simply run this script in a directory containing a buildout.cfg. The script accepts buildout command-line options, so you can use the -c option to specify an alternate configuration file. """ import os import shutil import sys import tempfile from optparse import OptionParser __version__ = '2015-07-01' # See zc.buildout's changelog if this version is up to date. tmpeggs = tempfile.mkdtemp(prefix='bootstrap-') usage = '''\ [DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options] Bootstraps a buildout-based project. Simply run this script in a directory containing a buildout.cfg, using the Python that you want bin/buildout to use. Note that by using --find-links to point to local resources, you can keep this script from going over the network. ''' parser = OptionParser(usage=usage) parser.add_option("--version", action="store_true", default=False, help=("Return bootstrap.py version.")) parser.add_option("-t", "--accept-buildout-test-releases", dest='accept_buildout_test_releases', action="store_true", default=False, help=("Normally, if you do not specify a --version, the " "bootstrap script and buildout gets the newest " "*final* versions of zc.buildout and its recipes and " "extensions for you. If you use this flag, " "bootstrap and buildout will get the newest releases " "even if they are alphas or betas.")) parser.add_option("-c", "--config-file", help=("Specify the path to the buildout configuration " "file to be used.")) parser.add_option("-f", "--find-links", help=("Specify a URL to search for buildout releases")) parser.add_option("--allow-site-packages", action="store_true", default=False, help=("Let bootstrap.py use existing site packages")) parser.add_option("--buildout-version", help="Use a specific zc.buildout version") parser.add_option("--setuptools-version", help="Use a specific setuptools version") parser.add_option("--setuptools-to-dir", help=("Allow for re-use of existing directory of " "setuptools versions")) options, args = parser.parse_args() if options.version: print("bootstrap.py version %s" % __version__) sys.exit(0) ###################################################################### # load/install setuptools try: from urllib.request import urlopen except ImportError: from urllib2 import urlopen ez = {} if os.path.exists('ez_setup.py'): exec(open('ez_setup.py').read(), ez) else: exec(urlopen('https://bootstrap.pypa.io/ez_setup.py').read(), ez) if not options.allow_site_packages: # ez_setup imports site, which adds site packages # this will remove them from the path to ensure that incompatible versions # of setuptools are not in the path import site # inside a virtualenv, there is no 'getsitepackages'. # We can't remove these reliably if hasattr(site, 'getsitepackages'): for sitepackage_path in site.getsitepackages(): # Strip all site-packages directories from sys.path that # are not sys.prefix; this is because on Windows # sys.prefix is a site-package directory. if sitepackage_path != sys.prefix: sys.path[:] = [x for x in sys.path if sitepackage_path not in x] setup_args = dict(to_dir=tmpeggs, download_delay=0) if options.setuptools_version is not None: setup_args['version'] = options.setuptools_version if options.setuptools_to_dir is not None: setup_args['to_dir'] = options.setuptools_to_dir ez['use_setuptools'](**setup_args) import setuptools import pkg_resources # This does not (always?) update the default working set. We will # do it. for path in sys.path: if path not in pkg_resources.working_set.entries: pkg_resources.working_set.add_entry(path) ###################################################################### # Install buildout ws = pkg_resources.working_set setuptools_path = ws.find( pkg_resources.Requirement.parse('setuptools')).location # Fix sys.path here as easy_install.pth added before PYTHONPATH cmd = [sys.executable, '-c', 'import sys; sys.path[0:0] = [%r]; ' % setuptools_path + 'from setuptools.command.easy_install import main; main()', '-mZqNxd', tmpeggs] find_links = os.environ.get( 'bootstrap-testing-find-links', options.find_links or ('http://downloads.buildout.org/' if options.accept_buildout_test_releases else None) ) if find_links: cmd.extend(['-f', find_links]) requirement = 'zc.buildout' version = options.buildout_version if version is None and not options.accept_buildout_test_releases: # Figure out the most recent final version of zc.buildout. import setuptools.package_index _final_parts = '*final-', '*final' def _final_version(parsed_version): try: return not parsed_version.is_prerelease except AttributeError: # Older setuptools for part in parsed_version: if (part[:1] == '*') and (part not in _final_parts): return False return True index = setuptools.package_index.PackageIndex( search_path=[setuptools_path]) if find_links: index.add_find_links((find_links,)) req = pkg_resources.Requirement.parse(requirement) if index.obtain(req) is not None: best = [] bestv = None for dist in index[req.project_name]: distv = dist.parsed_version if _final_version(distv): if bestv is None or distv > bestv: best = [dist] bestv = distv elif distv == bestv: best.append(dist) if best: best.sort() version = best[-1].version if version: requirement = '=='.join((requirement, version)) cmd.append(requirement) import subprocess if subprocess.call(cmd) != 0: raise Exception( "Failed to execute command:\n%s" % repr(cmd)[1:-1]) ###################################################################### # Import and run buildout ws.add_entry(tmpeggs) ws.require(requirement) import zc.buildout.buildout if not [a for a in args if '=' not in a]: args.append('bootstrap') # if -c was provided, we push it back into args for buildout' main function if options.config_file is not None: args[0:0] = ['-c', options.config_file] zc.buildout.buildout.main(args) shutil.rmtree(tmpeggs) BTrees-4.3.1/.travis.yml0000664000175000017500000000351612716352723016177 0ustar tseavertseaver00000000000000language: python sudo: false matrix: include: - os: linux python: 2.7 - os: linux python: 3.3 - os: linux python: 3.4 - os: linux python: 3.5 - os: linux python: pypy - os: linux python: pypy3 - os: osx language: generic env: TERRYFY_PYTHON='homebrew 2' - os: osx language: generic env: TERRYFY_PYTHON='homebrew 3' before_install: - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then git clone https://github.com/MacPython/terryfy; fi - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then source terryfy/travis_tools.sh; fi - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then get_python_environment $TERRYFY_PYTHON venv; fi - if [[ "$TERRYFY_PYTHON" == "homebrew 3" ]]; then alias pip=`which pip3` ; fi install: - pip install -e .[ZODB] script: - python setup.py -q test -q notifications: email: false after_success: - echo [distutils] > ~/.pypirc - echo index-servers = pypi >> ~/.pypirc - echo [pypi] >> ~/.pypirc - echo repository=https://pypi.python.org/pypi >> ~/.pypirc - echo username=zope.wheelbuilder >> ~/.pypirc - echo password=$PYPIPASSWORD >> ~/.pypirc - if [[ $TRAVIS_TAG && "$TRAVIS_OS_NAME" == "osx" ]]; then pip install twine; fi - if [[ $TRAVIS_TAG && "$TRAVIS_OS_NAME" == "osx" ]]; then python setup.py bdist_wheel; fi - if [[ $TRAVIS_TAG && "$TRAVIS_OS_NAME" == "osx" ]]; then twine upload dist/*; fi env: global: secure: "CpYDbVb6YOyIAnCcvBEiDLeYmNhfDDqJosIV0NussN9dNe1hsfHQK98evAEe9F4kXuWo6SmmtT0uX2RXOojB5If/z7sAcXSDaGWCMJK0FY98sWL2DCRtb1/O1puo/uvn3RJlpe4bkxQoyFCKjpWaTJOM+NSEt7YbTTTdCpG7/zU=" BTrees-4.3.1/BTrees/0000775000175000017500000000000012716353165015246 5ustar tseavertseaver00000000000000BTrees-4.3.1/BTrees/_IOBTree.c0000664000175000017500000000216512510775554017010 0ustar tseavertseaver00000000000000/*############################################################################ # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id$\n" /* IOBTree - int key, object value BTree Implements a collection using int type keys and object type values */ #define PERSISTENT #define MOD_NAME_PREFIX "IO" #define DEFAULT_MAX_BUCKET_SIZE 60 #define DEFAULT_MAX_BTREE_SIZE 500 #include "_compat.h" #include "intkeymacros.h" #include "objectvaluemacros.h" #ifdef PY3K #define INITMODULE PyInit__IOBTree #else #define INITMODULE init_IOBTree #endif #include "BTreeModuleTemplate.c" BTrees-4.3.1/BTrees/IOBTree.py0000664000175000017500000000656412665316015017060 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __all__ = ('Bucket', 'Set', 'BTree', 'TreeSet', 'IOBucket', 'IOSet', 'IOBTree', 'IOTreeSet', 'union', 'intersection', 'difference', 'multiunion', ) from zope.interface import moduleProvides from .Interfaces import IIntegerObjectBTreeModule from ._base import Bucket from ._base import MERGE_WEIGHT_default from ._base import Set from ._base import Tree as BTree from ._base import TreeSet from ._base import _TreeIterator from ._base import difference as _difference from ._base import intersection as _intersection from ._base import multiunion as _multiunion from ._base import set_operation as _set_operation from ._base import to_int as _to_key from ._base import to_ob as _to_value from ._base import union as _union from ._base import _fix_pickle _BUCKET_SIZE = 60 _TREE_SIZE = 500 using64bits = False class IOBucketPy(Bucket): _to_key = _to_key _to_value = _to_value MERGE_WEIGHT = MERGE_WEIGHT_default class IOSetPy(Set): _to_key = _to_key class IOBTreePy(BTree): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key _to_value = _to_value MERGE_WEIGHT = MERGE_WEIGHT_default class IOTreeSetPy(TreeSet): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key class IOTreeIteratorPy(_TreeIterator): pass # Can't declare forward refs, so fix up afterwards: IOBucketPy._mapping_type = IOBucketPy._bucket_type = IOBucketPy IOBucketPy._set_type = IOSetPy IOSetPy._mapping_type = IOBucketPy IOSetPy._set_type = IOSetPy._bucket_type = IOSetPy IOBTreePy._mapping_type = IOBTreePy._bucket_type = IOBucketPy IOBTreePy._set_type = IOSetPy IOTreeSetPy._mapping_type = IOBucketPy IOTreeSetPy._set_type = IOTreeSetPy._bucket_type = IOSetPy differencePy = _set_operation(_difference, IOSetPy) unionPy = _set_operation(_union, IOSetPy) intersectionPy = _set_operation(_intersection, IOSetPy) multiunionPy = _set_operation(_multiunion, IOSetPy) try: from ._IOBTree import IOBucket except ImportError: #pragma NO COVER w/ C extensions IOBucket = IOBucketPy IOSet = IOSetPy IOBTree = IOBTreePy IOTreeSet = IOTreeSetPy IOTreeIterator = IOTreeIteratorPy difference = differencePy union = unionPy intersection = intersectionPy multiunion = multiunionPy else: #pragma NO COVER w/o C extensions from ._IOBTree import IOSet from ._IOBTree import IOBTree from ._IOBTree import IOTreeSet from ._IOBTree import IOTreeIterator from ._IOBTree import difference from ._IOBTree import union from ._IOBTree import intersection from ._IOBTree import multiunion Bucket = IOBucket Set = IOSet BTree = IOBTree TreeSet = IOTreeSet _fix_pickle(globals(), __name__) moduleProvides(IIntegerObjectBTreeModule) BTrees-4.3.1/BTrees/_LFBTree.c0000664000175000017500000000227012510775554016777 0ustar tseavertseaver00000000000000/*############################################################################ # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id: _IFBTree.c 67074 2006-04-17 19:13:39Z fdrake $\n" /* IFBTree - int key, float value BTree Implements a collection using int type keys and float type values */ /* Setup template macros */ #define PERSISTENT #define MOD_NAME_PREFIX "LF" #define DEFAULT_MAX_BUCKET_SIZE 120 #define DEFAULT_MAX_BTREE_SIZE 500 #define ZODB_64BIT_INTS #include "_compat.h" #include "intkeymacros.h" #include "floatvaluemacros.h" #ifdef PY3K #define INITMODULE PyInit__LFBTree #else #define INITMODULE init_LFBTree #endif #include "BTreeModuleTemplate.c" BTrees-4.3.1/BTrees/_IIBTree.c0000664000175000017500000000215212510775554016776 0ustar tseavertseaver00000000000000/*############################################################################ # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id$\n" /* IIBTree - int key, int value BTree Implements a collection using int type keys and int type values */ /* Setup template macros */ #define PERSISTENT #define MOD_NAME_PREFIX "II" #define DEFAULT_MAX_BUCKET_SIZE 120 #define DEFAULT_MAX_BTREE_SIZE 500 #include "_compat.h" #include "intkeymacros.h" #include "intvaluemacros.h" #ifdef PY3K #define INITMODULE PyInit__IIBTree #else #define INITMODULE init_IIBTree #endif #include "BTreeModuleTemplate.c" BTrees-4.3.1/BTrees/SetOpTemplate.c0000664000175000017500000003632312510775554020151 0ustar tseavertseaver00000000000000/***************************************************************************** Copyright (c) 2001, 2002 Zope Foundation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ /**************************************************************************** Set operations ****************************************************************************/ #define SETOPTEMPLATE_C "$Id$\n" #ifdef KEY_CHECK static int nextKeyAsSet(SetIteration *i) { if (i->position >= 0) { if (i->position) { DECREF_KEY(i->key); i->position = -1; } else i->position = 1; } return 0; } #endif /* initSetIteration * * Start the set iteration protocol. See the comments at struct SetIteration. * * Arguments * i The address of a SetIteration control struct. * s The address of the set, bucket, BTree, ..., to be iterated. * useValues Boolean; if true, and s has values (is a mapping), copy * them into i->value each time i->next() is called; else * ignore s's values even if s is a mapping. * * Return * 0 on success; -1 and an exception set if error. * i.usesValue is set to 1 (true) if s has values and useValues was * true; else usesValue is set to 0 (false). * i.set gets a new reference to s, or to some other object used to * iterate over s. * i.position is set to 0. * i.next is set to an appropriate iteration function. * i.key and i.value are left alone. * * Internal * i.position < 0 means iteration terminated. * i.position = 0 means iteration hasn't yet begun (next() hasn't * been called yet). * In all other cases, i.key, and possibly i.value, own references. * These must be cleaned up, either by next() routines, or by * finiSetIteration. * next() routines must ensure the above. They should return without * doing anything when i.position < 0. * It's the responsibility of {init, fini}setIteration to clean up * the reference in i.set, and to ensure that no stale references * live in i.key or i.value if iteration terminates abnormally. * A SetIteration struct has been cleaned up iff i.set is NULL. */ static int initSetIteration(SetIteration *i, PyObject *s, int useValues) { i->set = NULL; i->position = -1; /* set to 0 only on normal return */ i->usesValue = 0; /* assume it's a set or that values aren't iterated */ if (PyObject_IsInstance(s, (PyObject *)&BucketType)) { i->set = s; Py_INCREF(s); if (useValues) { i->usesValue = 1; i->next = nextBucket; } else i->next = nextSet; } else if (PyObject_IsInstance(s, (PyObject *)&SetType)) { i->set = s; Py_INCREF(s); i->next = nextSet; } else if (PyObject_IsInstance(s, (PyObject *)&BTreeType)) { i->set = BTree_rangeSearch(BTREE(s), NULL, NULL, 'i'); UNLESS(i->set) return -1; if (useValues) { i->usesValue = 1; i->next = nextBTreeItems; } else i->next = nextTreeSetItems; } else if (PyObject_IsInstance(s, (PyObject *)&TreeSetType)) { i->set = BTree_rangeSearch(BTREE(s), NULL, NULL, 'k'); UNLESS(i->set) return -1; i->next = nextTreeSetItems; } #ifdef KEY_CHECK else if (KEY_CHECK(s)) { int copied = 1; COPY_KEY_FROM_ARG(i->key, s, copied); UNLESS (copied) return -1; INCREF_KEY(i->key); i->set = s; Py_INCREF(s); i->next = nextKeyAsSet; } #endif else { PyErr_SetString(PyExc_TypeError, "invalid argument"); return -1; } i->position = 0; return 0; } #ifndef MERGE_WEIGHT #define MERGE_WEIGHT(O, w) (O) #endif static int copyRemaining(Bucket *r, SetIteration *i, int merge, /* See comment # 42 */ #ifdef MERGE VALUE_TYPE w) #else int w) #endif { while (i->position >= 0) { if(r->len >= r->size && Bucket_grow(r, -1, ! merge) < 0) return -1; COPY_KEY(r->keys[r->len], i->key); INCREF_KEY(r->keys[r->len]); if (merge) { COPY_VALUE(r->values[r->len], MERGE_WEIGHT(i->value, w)); INCREF_VALUE(r->values[r->len]); } r->len++; if (i->next(i) < 0) return -1; } return 0; } /* This is the workhorse for all set merge operations: the weighted and * unweighted flavors of union and intersection, and set difference. The * algorithm is conceptually simple but the code is complicated due to all * the options. * * s1, s2 * The input collections to be merged. * * usevalues1, usevalues2 * Booleans. In the output, should values from s1 (or s2) be used? This * only makes sense when an operation intends to support mapping outputs; * these should both be false for operations that want pure set outputs. * * w1, w2 * If usevalues1(2) are true, these are the weights to apply to the * input values. * * c1 * Boolean. Should keys that appear in c1 but not c2 appear in the output? * c12 * Boolean. Should keys that appear in both inputs appear in the output? * c2 * Boolean. Should keys that appear in c2 but not c1 appear in the output? * * Returns NULL if error, else a Set or Bucket, depending on whether a set or * mapping was requested. */ static PyObject * set_operation(PyObject *s1, PyObject *s2, int usevalues1, int usevalues2, /* Comment # 42 The following ifdef works around a template/type problem Weights are passed as integers. In particular, the weight passed by difference is one. This works fine in the int value and float value cases but makes no sense in the object value case. In the object value case, we don't do merging, so we don't use the weights, so it doesn't matter what they are. */ #ifdef MERGE VALUE_TYPE w1, VALUE_TYPE w2, #else int w1, int w2, #endif int c1, int c12, int c2) { Bucket *r=0; SetIteration i1 = {0,0,0}, i2 = {0,0,0}; int cmp, merge; if (initSetIteration(&i1, s1, usevalues1) < 0) goto err; if (initSetIteration(&i2, s2, usevalues2) < 0) goto err; merge = i1.usesValue | i2.usesValue; if (merge) { #ifndef MERGE if (c12 && i1.usesValue && i2.usesValue) goto invalid_set_operation; #endif if (! i1.usesValue&& i2.usesValue) { SetIteration t; int i; /* See comment # 42 above */ #ifdef MERGE VALUE_TYPE v; #else int v; #endif t=i1; i1=i2; i2=t; i=c1; c1=c2; c2=i; v=w1; w1=w2; w2=v; } #ifdef MERGE_DEFAULT i1.value=MERGE_DEFAULT; i2.value=MERGE_DEFAULT; #else if (i1.usesValue) { if (! i2.usesValue && c2) goto invalid_set_operation; } else { if (c1 || c12) goto invalid_set_operation; } #endif UNLESS(r=BUCKET(PyObject_CallObject(OBJECT(&BucketType), NULL))) goto err; } else { UNLESS(r=BUCKET(PyObject_CallObject(OBJECT(&SetType), NULL))) goto err; } if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; while (i1.position >= 0 && i2.position >= 0) { TEST_KEY_SET_OR(cmp, i1.key, i2.key) goto err; if(cmp < 0) { if(c1) { if(r->len >= r->size && Bucket_grow(r, -1, ! merge) < 0) goto err; COPY_KEY(r->keys[r->len], i1.key); INCREF_KEY(r->keys[r->len]); if (merge) { COPY_VALUE(r->values[r->len], MERGE_WEIGHT(i1.value, w1)); INCREF_VALUE(r->values[r->len]); } r->len++; } if (i1.next(&i1) < 0) goto err; } else if(cmp==0) { if(c12) { if(r->len >= r->size && Bucket_grow(r, -1, ! merge) < 0) goto err; COPY_KEY(r->keys[r->len], i1.key); INCREF_KEY(r->keys[r->len]); if (merge) { #ifdef MERGE r->values[r->len] = MERGE(i1.value, w1, i2.value, w2); #else COPY_VALUE(r->values[r->len], i1.value); INCREF_VALUE(r->values[r->len]); #endif } r->len++; } if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; } else { if(c2) { if(r->len >= r->size && Bucket_grow(r, -1, ! merge) < 0) goto err; COPY_KEY(r->keys[r->len], i2.key); INCREF_KEY(r->keys[r->len]); if (merge) { COPY_VALUE(r->values[r->len], MERGE_WEIGHT(i2.value, w2)); INCREF_VALUE(r->values[r->len]); } r->len++; } if (i2.next(&i2) < 0) goto err; } } if(c1 && copyRemaining(r, &i1, merge, w1) < 0) goto err; if(c2 && copyRemaining(r, &i2, merge, w2) < 0) goto err; finiSetIteration(&i1); finiSetIteration(&i2); return OBJECT(r); #ifndef MERGE_DEFAULT invalid_set_operation: PyErr_SetString(PyExc_TypeError, "invalid set operation"); #endif err: finiSetIteration(&i1); finiSetIteration(&i2); Py_XDECREF(r); return NULL; } static PyObject * difference_m(PyObject *ignored, PyObject *args) { PyObject *o1, *o2; UNLESS(PyArg_ParseTuple(args, "OO", &o1, &o2)) return NULL; if (o1 == Py_None || o2 == Py_None) { /* difference(None, X) -> None; difference(X, None) -> X */ Py_INCREF(o1); return o1; } return set_operation(o1, o2, 1, 0, /* preserve values from o1, ignore o2's */ 1, 0, /* o1's values multiplied by 1 */ 1, 0, 0); /* take only keys unique to o1 */ } static PyObject * union_m(PyObject *ignored, PyObject *args) { PyObject *o1, *o2; UNLESS(PyArg_ParseTuple(args, "OO", &o1, &o2)) return NULL; if (o1 == Py_None) { Py_INCREF(o2); return o2; } else if (o2 == Py_None) { Py_INCREF(o1); return o1; } return set_operation(o1, o2, 0, 0, /* ignore values in both */ 1, 1, /* the weights are irrelevant */ 1, 1, 1); /* take all keys */ } static PyObject * intersection_m(PyObject *ignored, PyObject *args) { PyObject *o1, *o2; UNLESS(PyArg_ParseTuple(args, "OO", &o1, &o2)) return NULL; if (o1 == Py_None) { Py_INCREF(o2); return o2; } else if (o2 == Py_None) { Py_INCREF(o1); return o1; } return set_operation(o1, o2, 0, 0, /* ignore values in both */ 1, 1, /* the weights are irrelevant */ 0, 1, 0); /* take only keys common to both */ } #ifdef MERGE static PyObject * wunion_m(PyObject *ignored, PyObject *args) { PyObject *o1, *o2; VALUE_TYPE w1 = 1, w2 = 1; UNLESS(PyArg_ParseTuple(args, "OO|" VALUE_PARSE VALUE_PARSE, &o1, &o2, &w1, &w2) ) return NULL; if (o1 == Py_None) return Py_BuildValue(VALUE_PARSE "O", (o2 == Py_None ? 0 : w2), o2); else if (o2 == Py_None) return Py_BuildValue(VALUE_PARSE "O", w1, o1); o1 = set_operation(o1, o2, 1, 1, w1, w2, 1, 1, 1); if (o1) ASSIGN(o1, Py_BuildValue(VALUE_PARSE "O", (VALUE_TYPE)1, o1)); return o1; } static PyObject * wintersection_m(PyObject *ignored, PyObject *args) { PyObject *o1, *o2; VALUE_TYPE w1 = 1, w2 = 1; UNLESS(PyArg_ParseTuple(args, "OO|" VALUE_PARSE VALUE_PARSE, &o1, &o2, &w1, &w2) ) return NULL; if (o1 == Py_None) return Py_BuildValue(VALUE_PARSE "O", (o2 == Py_None ? 0 : w2), o2); else if (o2 == Py_None) return Py_BuildValue(VALUE_PARSE "O", w1, o1); o1 = set_operation(o1, o2, 1, 1, w1, w2, 0, 1, 0); if (o1) ASSIGN(o1, Py_BuildValue(VALUE_PARSE "O", ((o1->ob_type == (PyTypeObject*)(&SetType)) ? w2+w1 : 1), o1)); return o1; } #endif #ifdef MULTI_INT_UNION #include "sorters.c" /* Input is a sequence of integer sets (or convertible to sets by the set iteration protocol). Output is the union of the sets. The point is to run much faster than doing pairs of unions. */ static PyObject * multiunion_m(PyObject *ignored, PyObject *args) { PyObject *seq; /* input sequence */ int n; /* length of input sequence */ PyObject *set = NULL; /* an element of the input sequence */ Bucket *result; /* result set */ SetIteration setiter = {0}; int i; UNLESS(PyArg_ParseTuple(args, "O", &seq)) return NULL; n = PyObject_Length(seq); if (n < 0) return NULL; /* Construct an empty result set. */ result = BUCKET(PyObject_CallObject(OBJECT(&SetType), NULL)); if (result == NULL) return NULL; /* For each set in the input sequence, append its elements to the result set. At this point, we ignore the possibility of duplicates. */ for (i = 0; i < n; ++i) { set = PySequence_GetItem(seq, i); if (set == NULL) goto Error; /* If set is a bucket, do a straight resize + memcpy. */ if (set->ob_type == (PyTypeObject*)&SetType || set->ob_type == (PyTypeObject*)&BucketType) { Bucket *b = BUCKET(set); int status = 0; UNLESS (PER_USE(b)) goto Error; if (b->len) status = bucket_append(result, b, 0, b->len, 0, i < n-1); PER_UNUSE(b); if (status < 0) goto Error; } else { /* No cheap way: iterate over set's elements one at a time. */ if (initSetIteration(&setiter, set, 0) < 0) goto Error; if (setiter.next(&setiter) < 0) goto Error; while (setiter.position >= 0) { if (result->len >= result->size && Bucket_grow(result, -1, 1) < 0) goto Error; COPY_KEY(result->keys[result->len], setiter.key); ++result->len; /* We know the key is an int, so no need to incref it. */ if (setiter.next(&setiter) < 0) goto Error; } finiSetIteration(&setiter); } Py_DECREF(set); set = NULL; } /* Combine, sort, remove duplicates, and reset the result's len. If the set shrinks (which happens if and only if there are duplicates), no point to realloc'ing the set smaller, as we expect the result set to be short-lived. */ if (result->len > 0) { size_t newlen; /* number of elements in final result set */ newlen = sort_int_nodups(result->keys, (size_t)result->len); result->len = (int)newlen; } return (PyObject *)result; Error: Py_DECREF(result); Py_XDECREF(set); finiSetIteration(&setiter); return NULL; } #endif BTrees-4.3.1/BTrees/_compat.py0000664000175000017500000000256312510775554017252 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import sys if sys.version_info[0] < 3: #pragma NO COVER Python2 PY2 = True PY3 = False from StringIO import StringIO BytesIO = StringIO int_types = int, long xrange = xrange cmp = cmp _bytes = str def _ascii(x): return bytes(x) def _u(s, encoding='unicode_escape'): return unicode(s, encoding) else: #pragma NO COVER Python3 PY2 = False PY3 = True from io import StringIO from io import BytesIO int_types = int, xrange = range def cmp(x, y): return (x > y) - (y > x) _bytes = bytes def _ascii(x): return bytes(x, 'ascii') def _u(s, encoding=None): if encoding is None: return s return str(s, encoding) BTrees-4.3.1/BTrees/intvaluemacros.h0000664000175000017500000000325012704044565020451 0ustar tseavertseaver00000000000000 #define VALUEMACROS_H "$Id$\n" #ifdef ZODB_64BIT_INTS #define NEED_LONG_LONG_SUPPORT #define VALUE_TYPE PY_LONG_LONG #define VALUE_PARSE "L" #define COPY_VALUE_TO_OBJECT(O, K) O=longlong_as_object(K) #define COPY_VALUE_FROM_ARG(TARGET, ARG, STATUS) \ if (!longlong_convert((ARG), &TARGET)) \ { \ (STATUS)=0; (TARGET)=0; \ } #else #define VALUE_TYPE int #define VALUE_PARSE "i" #define COPY_VALUE_TO_OBJECT(O, K) O=INT_FROM_LONG(K) #define COPY_VALUE_FROM_ARG(TARGET, ARG, STATUS) \ if (INT_CHECK(ARG)) { \ long vcopy = INT_AS_LONG(ARG); \ if (PyErr_Occurred()) { (STATUS)=0; (TARGET)=0; } \ else if ((int)vcopy != vcopy) { \ PyErr_SetString(PyExc_TypeError, "integer out of range"); \ (STATUS)=0; (TARGET)=0; \ } \ else TARGET = vcopy; \ } else { \ PyErr_SetString(PyExc_TypeError, "expected integer key"); \ (STATUS)=0; (TARGET)=0; } #endif #undef VALUE_TYPE_IS_PYOBJECT #define TEST_VALUE(K, T) (((K) < (T)) ? -1 : (((K) > (T)) ? 1: 0)) #define VALUE_SAME(VALUE, TARGET) ( (VALUE) == (TARGET) ) #define DECLARE_VALUE(NAME) VALUE_TYPE NAME #define DECREF_VALUE(k) #define INCREF_VALUE(k) #define COPY_VALUE(V, E) (V=(E)) #define NORMALIZE_VALUE(V, MIN) ((MIN) > 0) ? ((V)/=(MIN)) : 0 #define MERGE_DEFAULT 1 #define MERGE(O1, w1, O2, w2) ((O1)*(w1)+(O2)*(w2)) #define MERGE_WEIGHT(O, w) ((O)*(w)) BTrees-4.3.1/BTrees/objectkeymacros.h0000664000175000017500000000230012576072533020577 0ustar tseavertseaver00000000000000#define KEYMACROS_H "$Id$\n" #define KEY_TYPE PyObject * #define KEY_TYPE_IS_PYOBJECT #include "Python.h" #include "_compat.h" static PyObject *object_; /* initialized in BTreeModuleTemplate init */ static int check_argument_cmp(PyObject *arg) { /* printf("check cmp %p %p %p %p\n", */ /* arg->ob_type->tp_richcompare, */ /* ((PyTypeObject *)object_)->ob_type->tp_richcompare, */ /* arg->ob_type->tp_compare, */ /* ((PyTypeObject *)object_)->ob_type->tp_compare); */ #ifdef PY3K if (Py_TYPE(arg)->tp_richcompare == Py_TYPE(object_)->tp_richcompare) #else if (Py_TYPE(arg)->tp_richcompare == NULL && Py_TYPE(arg)->tp_compare == Py_TYPE(object_)->tp_compare) #endif { PyErr_SetString(PyExc_TypeError, "Object has default comparison"); return 0; } return 1; } #define TEST_KEY_SET_OR(V, KEY, TARGET) \ if ( ( (V) = COMPARE((KEY),(TARGET)) ), PyErr_Occurred() ) #define INCREF_KEY(k) Py_INCREF(k) #define DECREF_KEY(KEY) Py_DECREF(KEY) #define COPY_KEY(KEY, E) KEY=(E) #define COPY_KEY_TO_OBJECT(O, K) O=(K); Py_INCREF(O) #define COPY_KEY_FROM_ARG(TARGET, ARG, S) \ TARGET=(ARG); \ (S) = check_argument_cmp(ARG); BTrees-4.3.1/BTrees/__init__.py0000664000175000017500000000351712510775554017367 0ustar tseavertseaver00000000000000############################################################################# # # Copyright (c) 2007 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################# import zope.interface import BTrees.Interfaces @zope.interface.implementer(BTrees.Interfaces.IBTreeFamily) class _Family(object): from BTrees import OOBTree as OO class _Family32(_Family): from BTrees import OIBTree as OI from BTrees import IIBTree as II from BTrees import IOBTree as IO from BTrees import IFBTree as IF maxint = int(2**31-1) minint = -maxint - 1 def __reduce__(self): return _family32, () class _Family64(_Family): from BTrees import OLBTree as OI from BTrees import LLBTree as II from BTrees import LOBTree as IO from BTrees import LFBTree as IF maxint = 2**63-1 minint = -maxint - 1 def __reduce__(self): return _family64, () def _family32(): return family32 _family32.__safe_for_unpickling__ = True def _family64(): return family64 _family64.__safe_for_unpickling__ = True family32 = _Family32() family64 = _Family64() BTrees.family64.IO.family = family64 BTrees.family64.OI.family = family64 BTrees.family64.IF.family = family64 BTrees.family64.II.family = family64 BTrees.family32.IO.family = family32 BTrees.family32.OI.family = family32 BTrees.family32.IF.family = family32 BTrees.family32.II.family = family32 BTrees-4.3.1/BTrees/_IFBTree.c0000664000175000017500000000216012510775554016772 0ustar tseavertseaver00000000000000/*############################################################################ # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id$\n" /* IFBTree - int key, float value BTree Implements a collection using int type keys and float type values */ /* Setup template macros */ #define PERSISTENT #define MOD_NAME_PREFIX "IF" #define DEFAULT_MAX_BUCKET_SIZE 120 #define DEFAULT_MAX_BTREE_SIZE 500 #include "_compat.h" #include "intkeymacros.h" #include "floatvaluemacros.h" #ifdef PY3K #define INITMODULE PyInit__IFBTree #else #define INITMODULE init_IFBTree #endif #include "BTreeModuleTemplate.c" BTrees-4.3.1/BTrees/_LOBTree.c0000664000175000017500000000223212510775554017006 0ustar tseavertseaver00000000000000/*############################################################################ # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id: _IOBTree.c 25186 2004-06-02 15:07:33Z jim $\n" /* IOBTree - int key, object value BTree Implements a collection using int type keys and object type values */ #define PERSISTENT #define MOD_NAME_PREFIX "LO" #define DEFAULT_MAX_BUCKET_SIZE 60 #define DEFAULT_MAX_BTREE_SIZE 500 #define ZODB_64BIT_INTS #include "_compat.h" #include "intkeymacros.h" #include "objectvaluemacros.h" #ifdef PY3K #define INITMODULE PyInit__LOBTree #else #define INITMODULE init_LOBTree #endif #include "BTreeModuleTemplate.c" BTrees-4.3.1/BTrees/objectvaluemacros.h0000664000175000017500000000071412510775554021133 0ustar tseavertseaver00000000000000#define VALUEMACROS_H "$Id$\n" #define VALUE_TYPE PyObject * #define VALUE_TYPE_IS_PYOBJECT #define TEST_VALUE(VALUE, TARGET) (COMPARE((VALUE),(TARGET))) #define DECLARE_VALUE(NAME) VALUE_TYPE NAME #define INCREF_VALUE(k) Py_INCREF(k) #define DECREF_VALUE(k) Py_DECREF(k) #define COPY_VALUE(k,e) k=(e) #define COPY_VALUE_TO_OBJECT(O, K) O=(K); Py_INCREF(O) #define COPY_VALUE_FROM_ARG(TARGET, ARG, S) TARGET=(ARG) #define NORMALIZE_VALUE(V, MIN) Py_INCREF(V) BTrees-4.3.1/BTrees/BucketTemplate.c0000664000175000017500000014730312576065015020331 0ustar tseavertseaver00000000000000/***************************************************************************** Copyright (c) 2001, 2002 Zope Foundation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #define BUCKETTEMPLATE_C "$Id$\n" /* Use BUCKET_SEARCH to find the index at which a key belongs. * INDEX An int lvalue to hold the index i such that KEY belongs at * SELF->keys[i]. Note that this will equal SELF->len if KEY * is larger than the bucket's largest key. Else it's the * smallest i such that SELF->keys[i] >= KEY. * ABSENT An int lvalue to hold a Boolean result, true (!= 0) if the * key is absent, false (== 0) if the key is at INDEX. * SELF A pointer to a Bucket node. * KEY The key you're looking for, of type KEY_TYPE. * ONERROR What to do if key comparison raises an exception; for example, * perhaps 'return NULL'. * * See Maintainer.txt for discussion: this is optimized in subtle ways. * It's recommended that you call this at the start of a routine, waiting * to check for self->len == 0 after (if an empty bucket is special in * context; INDEX becomes 0 and ABSENT becomes true if this macro is run * with an empty SELF, and that may be all the invoker needs to know). */ #define BUCKET_SEARCH(INDEX, ABSENT, SELF, KEY, ONERROR) { \ int _lo = 0; \ int _hi = (SELF)->len; \ int _i; \ int _cmp = 1; \ for (_i = _hi >> 1; _lo < _hi; _i = (_lo + _hi) >> 1) { \ TEST_KEY_SET_OR(_cmp, (SELF)->keys[_i], (KEY)) \ ONERROR; \ if (_cmp < 0) _lo = _i + 1; \ else if (_cmp == 0) break; \ else _hi = _i; \ } \ (INDEX) = _i; \ (ABSENT) = _cmp; \ } /* ** _bucket_get ** ** Search a bucket for a given key. ** ** Arguments ** self The bucket ** keyarg The key to look for ** has_key Boolean; if true, return a true/false result; else return ** the value associated with the key. ** ** Return ** If has_key: ** Returns the Python int 0 if the key is absent, else returns ** has_key itself as a Python int. A BTree caller generally passes ** the depth of the bucket for has_key, so a true result returns ** the bucket depth then. ** Note that has_key should be true when searching set buckets. ** If not has_key: ** If the key is present, returns the associated value, and the ** caller owns the reference. Else returns NULL and sets KeyError. ** Whether or not has_key: ** If a comparison sets an exception, returns NULL. */ static PyObject * _bucket_get(Bucket *self, PyObject *keyarg, int has_key) { int i, cmp; KEY_TYPE key; PyObject *r = NULL; int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); UNLESS (copied) return NULL; UNLESS (PER_USE(self)) return NULL; BUCKET_SEARCH(i, cmp, self, key, goto Done); if (has_key) r = INT_FROM_LONG(cmp ? 0 : has_key); else { if (cmp == 0) { COPY_VALUE_TO_OBJECT(r, self->values[i]); } else PyErr_SetObject(PyExc_KeyError, keyarg); } Done: PER_UNUSE(self); return r; } static PyObject * bucket_getitem(Bucket *self, PyObject *key) { return _bucket_get(self, key, 0); } /* ** Bucket_grow ** ** Resize a bucket. ** ** Arguments: self The bucket. ** newsize The new maximum capacity. If < 0, double the ** current size unless the bucket is currently empty, ** in which case use MIN_BUCKET_ALLOC. ** noval Boolean; if true, allocate only key space and not ** value space ** ** Returns: -1 on error, and MemoryError exception is set ** 0 on success */ static int Bucket_grow(Bucket *self, int newsize, int noval) { KEY_TYPE *keys; VALUE_TYPE *values; if (self->size) { if (newsize < 0) newsize = self->size * 2; if (newsize < 0) /* int overflow */ goto Overflow; UNLESS (keys = BTree_Realloc(self->keys, sizeof(KEY_TYPE) * newsize)) return -1; UNLESS (noval) { values = BTree_Realloc(self->values, sizeof(VALUE_TYPE) * newsize); if (values == NULL) { free(keys); return -1; } self->values = values; } self->keys = keys; } else { if (newsize < 0) newsize = MIN_BUCKET_ALLOC; UNLESS (self->keys = BTree_Malloc(sizeof(KEY_TYPE) * newsize)) return -1; UNLESS (noval) { self->values = BTree_Malloc(sizeof(VALUE_TYPE) * newsize); if (self->values == NULL) { free(self->keys); self->keys = NULL; return -1; } } } self->size = newsize; return 0; Overflow: PyErr_NoMemory(); return -1; } /* So far, bucket_append is called only by multiunion_m(), so is called * only when MULTI_INT_UNION is defined. Flavors of BTree/Bucket that * don't support MULTI_INT_UNION don't call bucket_append (yet), and * gcc complains if bucket_append is compiled in those cases. So only * compile bucket_append if it's going to be used. */ #ifdef MULTI_INT_UNION /* * Append a slice of the "from" bucket to self. * * self Append (at least keys) to this bucket. self must be activated * upon entry, and remains activated at exit. If copyValues * is true, self must be empty or already have a non-NULL values * pointer. self's access and modification times aren't updated. * from The bucket from which to take keys, and possibly values. from * must be activated upon entry, and remains activated at exit. * If copyValues is true, from must have a non-NULL values * pointer. self and from must not be the same. from's access * time isn't updated. * i, n The slice from[i : i+n] is appended to self. Must have * i >= 0, n > 0 and i+n <= from->len. * copyValues Boolean. If true, copy values from the slice as well as keys. * In this case, from must have a non-NULL values pointer, and * self must too (unless self is empty, in which case a values * vector will be allocated for it). * overallocate Boolean. If self doesn't have enough room upon entry to hold * all the appended stuff, then if overallocate is false exactly * enough room will be allocated to hold the new stuff, else if * overallocate is true an excess will be allocated. overallocate * may be a good idea if you expect to append more stuff to self * later; else overallocate should be false. * * CAUTION: If self is empty upon entry (self->size == 0), and copyValues is * false, then no space for values will get allocated. This can be a trap if * the caller intends to copy values itself. * * Return * -1 Error. * 0 OK. */ static int bucket_append(Bucket *self, Bucket *from, int i, int n, int copyValues, int overallocate) { int newlen; assert(self && from && self != from); assert(i >= 0); assert(n > 0); assert(i+n <= from->len); /* Make room. */ newlen = self->len + n; if (newlen > self->size) { int newsize = newlen; if (overallocate) /* boost by 25% -- pretty arbitrary */ newsize += newsize >> 2; if (Bucket_grow(self, newsize, ! copyValues) < 0) return -1; } assert(newlen <= self->size); /* Copy stuff. */ memcpy(self->keys + self->len, from->keys + i, n * sizeof(KEY_TYPE)); if (copyValues) { assert(self->values); assert(from->values); memcpy(self->values + self->len, from->values + i, n * sizeof(VALUE_TYPE)); } self->len = newlen; /* Bump refcounts. */ #ifdef KEY_TYPE_IS_PYOBJECT { int j; PyObject **p = from->keys + i; for (j = 0; j < n; ++j, ++p) { Py_INCREF(*p); } } #endif #ifdef VALUE_TYPE_IS_PYOBJECT if (copyValues) { int j; PyObject **p = from->values + i; for (j = 0; j < n; ++j, ++p) { Py_INCREF(*p); } } #endif return 0; } #endif /* MULTI_INT_UNION */ /* ** _bucket_set: Assign a value to a key in a bucket, delete a key+value ** pair, or just insert a key. ** ** Arguments ** self The bucket ** keyarg The key to look for ** v The value to associate with key; NULL means delete the key. ** If NULL, it's an error (KeyError) if the key isn't present. ** Note that if this is a set bucket, and you want to insert ** a new set element, v must be non-NULL although its exact ** value will be ignored. Passing Py_None is good for this. ** unique Boolean; when true, don't replace the value if the key is ** already present. ** noval Boolean; when true, operate on keys only (ignore values) ** changed ignored on input ** ** Return ** -1 on error ** 0 on success and the # of bucket entries didn't change ** 1 on success and the # of bucket entries did change ** *changed If non-NULL, set to 1 on any mutation of the bucket. */ static int _bucket_set(Bucket *self, PyObject *keyarg, PyObject *v, int unique, int noval, int *changed) { int i, cmp; KEY_TYPE key; /* Subtle: there may or may not be a value. If there is, we need to * check its type early, so that in case of error we can get out before * mutating the bucket. But because value isn't used on all paths, if * we don't initialize value then gcc gives a nuisance complaint that * value may be used initialized (it can't be, but gcc doesn't know * that). So we initialize it. However, VALUE_TYPE can be various types, * including int, PyObject*, and char[6], so it's a puzzle to spell * initialization. It so happens that {0} is a valid initializer for all * these types. */ VALUE_TYPE value = {0}; /* squash nuisance warning */ int result = -1; /* until proven innocent */ int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); UNLESS(copied) return -1; /* Copy the value early (if needed), so that in case of error a * pile of bucket mutations don't need to be undone. */ if (v && !noval) { COPY_VALUE_FROM_ARG(value, v, copied); UNLESS(copied) return -1; } UNLESS (PER_USE(self)) return -1; BUCKET_SEARCH(i, cmp, self, key, goto Done); if (cmp == 0) { /* The key exists, at index i. */ if (v) { /* The key exists at index i, and there's a new value. * If unique, we're not supposed to replace it. If noval, or this * is a set bucket (self->values is NULL), there's nothing to do. */ if (unique || noval || self->values == NULL) { result = 0; goto Done; } /* The key exists at index i, and we need to replace the value. */ #ifdef VALUE_SAME /* short-circuit if no change */ if (VALUE_SAME(self->values[i], value)) { result = 0; goto Done; } #endif if (changed) *changed = 1; DECREF_VALUE(self->values[i]); COPY_VALUE(self->values[i], value); INCREF_VALUE(self->values[i]); if (PER_CHANGED(self) >= 0) result = 0; goto Done; } /* The key exists at index i, and should be deleted. */ DECREF_KEY(self->keys[i]); self->len--; if (i < self->len) memmove(self->keys + i, self->keys + i+1, sizeof(KEY_TYPE)*(self->len - i)); if (self->values) { DECREF_VALUE(self->values[i]); if (i < self->len) memmove(self->values + i, self->values + i+1, sizeof(VALUE_TYPE)*(self->len - i)); } if (! self->len) { self->size = 0; free(self->keys); self->keys = NULL; if (self->values) { free(self->values); self->values = NULL; } } if (changed) *changed = 1; if (PER_CHANGED(self) >= 0) result = 1; goto Done; } /* The key doesn't exist, and belongs at index i. */ if (!v) { /* Can't delete a non-existent key. */ PyErr_SetObject(PyExc_KeyError, keyarg); goto Done; } /* The key doesn't exist and should be inserted at index i. */ if (self->len == self->size && Bucket_grow(self, -1, noval) < 0) goto Done; if (self->len > i) { memmove(self->keys + i + 1, self->keys + i, sizeof(KEY_TYPE) * (self->len - i)); if (self->values) { memmove(self->values + i + 1, self->values + i, sizeof(VALUE_TYPE) * (self->len - i)); } } COPY_KEY(self->keys[i], key); INCREF_KEY(self->keys[i]); if (! noval) { COPY_VALUE(self->values[i], value); INCREF_VALUE(self->values[i]); } self->len++; if (changed) *changed = 1; if (PER_CHANGED(self) >= 0) result = 1; Done: PER_UNUSE(self); return result; } /* ** bucket_setitem ** ** wrapper for _bucket_setitem (eliminates +1 return code) ** ** Arguments: self The bucket ** key The key to insert under ** v The value to insert ** ** Returns 0 on success ** -1 on failure */ static int bucket_setitem(Bucket *self, PyObject *key, PyObject *v) { if (_bucket_set(self, key, v, 0, 0, 0) < 0) return -1; return 0; } /** ** Accepts a sequence of 2-tuples, or any object with an items() ** method that returns an iterable object producing 2-tuples. */ static int update_from_seq(PyObject *map, PyObject *seq) { PyObject *iter, *o, *k, *v; int err = -1; /* One path creates a new seq object. The other path has an INCREF of the seq argument. So seq must always be DECREFed on the way out. */ /* Use items() if it's not a sequence. Alas, PySequence_Check() * returns true for a PeristentMapping or PersistentDict, and we * want to use items() in those cases too. */ #ifdef PY3K #define ITERITEMS "items" #else #define ITERITEMS "iteritems" #endif if (!PySequence_Check(seq) || /* or it "looks like a dict" */ PyObject_HasAttrString(seq, ITERITEMS)) #undef ITERITEMS { PyObject *items; items = PyObject_GetAttrString(seq, "items"); if (items == NULL) return -1; seq = PyObject_CallObject(items, NULL); Py_DECREF(items); if (seq == NULL) return -1; } else Py_INCREF(seq); iter = PyObject_GetIter(seq); if (iter == NULL) goto err; while (1) { o = PyIter_Next(iter); if (o == NULL) { if (PyErr_Occurred()) goto err; else break; } if (!PyTuple_Check(o) || PyTuple_GET_SIZE(o) != 2) { Py_DECREF(o); PyErr_SetString(PyExc_TypeError, "Sequence must contain 2-item tuples"); goto err; } k = PyTuple_GET_ITEM(o, 0); v = PyTuple_GET_ITEM(o, 1); if (PyObject_SetItem(map, k, v) < 0) { Py_DECREF(o); goto err; } Py_DECREF(o); } err = 0; err: Py_DECREF(iter); Py_DECREF(seq); return err; } static PyObject * Mapping_update(PyObject *self, PyObject *seq) { if (update_from_seq(self, seq) < 0) return NULL; Py_INCREF(Py_None); return Py_None; } /* ** bucket_split ** ** Splits one bucket into two ** ** Arguments: self The bucket ** index the index of the key to split at (O.O.B use midpoint) ** next the new bucket to split into ** ** Returns: 0 on success ** -1 on failure */ static int bucket_split(Bucket *self, int index, Bucket *next) { int next_size; ASSERT(self->len > 1, "split of empty bucket", -1); if (index < 0 || index >= self->len) index = self->len / 2; next_size = self->len - index; next->keys = BTree_Malloc(sizeof(KEY_TYPE) * next_size); if (!next->keys) return -1; memcpy(next->keys, self->keys + index, sizeof(KEY_TYPE) * next_size); if (self->values) { next->values = BTree_Malloc(sizeof(VALUE_TYPE) * next_size); if (!next->values) { free(next->keys); next->keys = NULL; return -1; } memcpy(next->values, self->values + index, sizeof(VALUE_TYPE) * next_size); } next->size = next_size; next->len = next_size; self->len = index; next->next = self->next; Py_INCREF(next); self->next = next; if (PER_CHANGED(self) < 0) return -1; return 0; } /* Set self->next to self->next->next, i.e. unlink self's successor from * the chain. * * Return: * -1 error * 0 OK */ static int Bucket_deleteNextBucket(Bucket *self) { int result = -1; /* until proven innocent */ Bucket *successor; PER_USE_OR_RETURN(self, -1); successor = self->next; if (successor) { Bucket *next; /* Before: self -> successor -> next * After: self --------------> next */ UNLESS (PER_USE(successor)) goto Done; next = successor->next; PER_UNUSE(successor); Py_XINCREF(next); /* it may be NULL, of course */ self->next = next; Py_DECREF(successor); if (PER_CHANGED(self) < 0) goto Done; } result = 0; Done: PER_UNUSE(self); return result; } /* Bucket_findRangeEnd -- Find the index of a range endpoint (possibly) contained in a bucket. Arguments: self The bucket keyarg The key to match against low Boolean; true for low end of range, false for high exclude_equal Boolean; if true, don't accept an exact match, and if there is one then move right if low and left if !low. offset The output offset If low true, *offset <- index of the smallest item >= key, if low false the index of the largest item <= key. In either case, if there is no such index, *offset is left alone and 0 is returned. Return: 0 No suitable index exists; *offset has not been changed 1 The correct index was stored into *offset -1 Error Example: Suppose the keys are [2, 4], and exclude_equal is false. Searching for 2 sets *offset to 0 and returns 1 regardless of low. Searching for 4 sets *offset to 1 and returns 1 regardless of low. Searching for 1: If low true, sets *offset to 0, returns 1. If low false, returns 0. Searching for 3: If low true, sets *offset to 1, returns 1. If low false, sets *offset to 0, returns 1. Searching for 5: If low true, returns 0. If low false, sets *offset to 1, returns 1. The 1, 3 and 5 examples are the same when exclude_equal is true. */ static int Bucket_findRangeEnd(Bucket *self, PyObject *keyarg, int low, int exclude_equal, int *offset) { int i, cmp; int result = -1; /* until proven innocent */ KEY_TYPE key; int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); UNLESS (copied) return -1; UNLESS (PER_USE(self)) return -1; BUCKET_SEARCH(i, cmp, self, key, goto Done); if (cmp == 0) { /* exact match at index i */ if (exclude_equal) { /* but we don't want an exact match */ if (low) ++i; else --i; } } /* Else keys[i-1] < key < keys[i], picturing infinities at OOB indices, * and i has the smallest item > key, which is correct for low. */ else if (! low) /* i-1 has the largest item < key (unless i-1 is 0OB) */ --i; result = 0 <= i && i < self->len; if (result) *offset = i; Done: PER_UNUSE(self); return result; } static PyObject * Bucket_maxminKey(Bucket *self, PyObject *args, int min) { PyObject *key=0; int rc, offset = 0; int empty_bucket = 1; if (args && ! PyArg_ParseTuple(args, "|O", &key)) return NULL; PER_USE_OR_RETURN(self, NULL); UNLESS (self->len) goto empty; /* Find the low range */ if (key && key != Py_None) { if ((rc = Bucket_findRangeEnd(self, key, min, 0, &offset)) <= 0) { if (rc < 0) return NULL; empty_bucket = 0; goto empty; } } else if (min) offset = 0; else offset = self->len -1; COPY_KEY_TO_OBJECT(key, self->keys[offset]); PER_UNUSE(self); return key; empty: PyErr_SetString(PyExc_ValueError, empty_bucket ? "empty bucket" : "no key satisfies the conditions"); PER_UNUSE(self); return NULL; } static PyObject * Bucket_minKey(Bucket *self, PyObject *args) { return Bucket_maxminKey(self, args, 1); } static PyObject * Bucket_maxKey(Bucket *self, PyObject *args) { return Bucket_maxminKey(self, args, 0); } static int Bucket_rangeSearch(Bucket *self, PyObject *args, PyObject *kw, int *low, int *high) { PyObject *min = Py_None; PyObject *max = Py_None; int excludemin = 0; int excludemax = 0; int rc; if (args) { if (! PyArg_ParseTupleAndKeywords(args, kw, "|OOii", search_keywords, &min, &max, &excludemin, &excludemax)) return -1; } UNLESS (self->len) goto empty; /* Find the low range */ if (min != Py_None) { rc = Bucket_findRangeEnd(self, min, 1, excludemin, low); if (rc < 0) return -1; if (rc == 0) goto empty; } else { *low = 0; if (excludemin) { if (self->len < 2) goto empty; ++*low; } } /* Find the high range */ if (max != Py_None) { rc = Bucket_findRangeEnd(self, max, 0, excludemax, high); if (rc < 0) return -1; if (rc == 0) goto empty; } else { *high = self->len - 1; if (excludemax) { if (self->len < 2) goto empty; --*high; } } /* If min < max to begin with, it's quite possible that low > high now. */ if (*low <= *high) return 0; empty: *low = 0; *high = -1; return 0; } /* ** bucket_keys ** ** Generate a list of all keys in the bucket ** ** Arguments: self The Bucket ** args (unused) ** ** Returns: list of bucket keys */ static PyObject * bucket_keys(Bucket *self, PyObject *args, PyObject *kw) { PyObject *r = NULL, *key; int i, low, high; PER_USE_OR_RETURN(self, NULL); if (Bucket_rangeSearch(self, args, kw, &low, &high) < 0) goto err; r = PyList_New(high-low+1); if (r == NULL) goto err; for (i=low; i <= high; i++) { COPY_KEY_TO_OBJECT(key, self->keys[i]); if (PyList_SetItem(r, i-low , key) < 0) goto err; } PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); return NULL; } /* ** bucket_values ** ** Generate a list of all values in the bucket ** ** Arguments: self The Bucket ** args (unused) ** ** Returns list of values */ static PyObject * bucket_values(Bucket *self, PyObject *args, PyObject *kw) { PyObject *r=0, *v; int i, low, high; PER_USE_OR_RETURN(self, NULL); if (Bucket_rangeSearch(self, args, kw, &low, &high) < 0) goto err; UNLESS (r=PyList_New(high-low+1)) goto err; for (i=low; i <= high; i++) { COPY_VALUE_TO_OBJECT(v, self->values[i]); UNLESS (v) goto err; if (PyList_SetItem(r, i-low, v) < 0) goto err; } PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); return NULL; } /* ** bucket_items ** ** Returns a list of all items in a bucket ** ** Arguments: self The Bucket ** args (unused) ** ** Returns: list of all items in the bucket */ static PyObject * bucket_items(Bucket *self, PyObject *args, PyObject *kw) { PyObject *r=0, *o=0, *item=0; int i, low, high; PER_USE_OR_RETURN(self, NULL); if (Bucket_rangeSearch(self, args, kw, &low, &high) < 0) goto err; UNLESS (r=PyList_New(high-low+1)) goto err; for (i=low; i <= high; i++) { UNLESS (item = PyTuple_New(2)) goto err; COPY_KEY_TO_OBJECT(o, self->keys[i]); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 0, o); COPY_VALUE_TO_OBJECT(o, self->values[i]); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 1, o); if (PyList_SetItem(r, i-low, item) < 0) goto err; item = 0; } PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); Py_XDECREF(item); return NULL; } static PyObject * bucket_byValue(Bucket *self, PyObject *omin) { PyObject *r=0, *o=0, *item=0; VALUE_TYPE min; VALUE_TYPE v; int i, l, copied=1; PER_USE_OR_RETURN(self, NULL); COPY_VALUE_FROM_ARG(min, omin, copied); UNLESS(copied) return NULL; for (i=0, l=0; i < self->len; i++) if (TEST_VALUE(self->values[i], min) >= 0) l++; UNLESS (r=PyList_New(l)) goto err; for (i=0, l=0; i < self->len; i++) { if (TEST_VALUE(self->values[i], min) < 0) continue; UNLESS (item = PyTuple_New(2)) goto err; COPY_KEY_TO_OBJECT(o, self->keys[i]); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 1, o); COPY_VALUE(v, self->values[i]); NORMALIZE_VALUE(v, min); COPY_VALUE_TO_OBJECT(o, v); DECREF_VALUE(v); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 0, o); if (PyList_SetItem(r, l, item) < 0) goto err; l++; item = 0; } item=PyObject_GetAttr(r,sort_str); UNLESS (item) goto err; ASSIGN(item, PyObject_CallObject(item, NULL)); UNLESS (item) goto err; ASSIGN(item, PyObject_GetAttr(r, reverse_str)); UNLESS (item) goto err; ASSIGN(item, PyObject_CallObject(item, NULL)); UNLESS (item) goto err; Py_DECREF(item); PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); Py_XDECREF(item); return NULL; } static int _bucket_clear(Bucket *self) { const int len = self->len; /* Don't declare i at this level. If neither keys nor values are * PyObject*, i won't be referenced, and you'll get a nuisance compiler * wng for declaring it here. */ self->len = self->size = 0; if (self->next) { Py_DECREF(self->next); self->next = NULL; } /* Silence compiler warning about unused variable len for the case when neither key nor value is an object, i.e. II. */ (void)len; if (self->keys) { #ifdef KEY_TYPE_IS_PYOBJECT int i; for (i = 0; i < len; ++i) DECREF_KEY(self->keys[i]); #endif free(self->keys); self->keys = NULL; } if (self->values) { #ifdef VALUE_TYPE_IS_PYOBJECT int i; for (i = 0; i < len; ++i) DECREF_VALUE(self->values[i]); #endif free(self->values); self->values = NULL; } return 0; } #ifdef PERSISTENT static PyObject * bucket__p_deactivate(Bucket *self, PyObject *args, PyObject *keywords) { int ghostify = 1; PyObject *force = NULL; if (args && PyTuple_GET_SIZE(args) > 0) { PyErr_SetString(PyExc_TypeError, "_p_deactivate takes no positional arguments"); return NULL; } if (keywords) { int size = PyDict_Size(keywords); force = PyDict_GetItemString(keywords, "force"); if (force) size--; if (size) { PyErr_SetString(PyExc_TypeError, "_p_deactivate only accepts keyword arg force"); return NULL; } } if (self->jar && self->oid) { ghostify = self->state == cPersistent_UPTODATE_STATE; if (!ghostify && force) { if (PyObject_IsTrue(force)) ghostify = 1; if (PyErr_Occurred()) return NULL; } if (ghostify) { if (_bucket_clear(self) < 0) return NULL; PER_GHOSTIFY(self); } } Py_INCREF(Py_None); return Py_None; } #endif static PyObject * bucket_clear(Bucket *self, PyObject *args) { PER_USE_OR_RETURN(self, NULL); if (self->len) { if (_bucket_clear(self) < 0) return NULL; if (PER_CHANGED(self) < 0) goto err; } PER_UNUSE(self); Py_INCREF(Py_None); return Py_None; err: PER_UNUSE(self); return NULL; } /* * Return: * * For a set bucket (self->values is NULL), a one-tuple or two-tuple. The * first element is a tuple of keys, of length self->len. The second element * is the next bucket, present if and only if next is non-NULL: * * ( * (keys[0], keys[1], ..., keys[len-1]), * next iff non-NULL> * ) * * For a mapping bucket (self->values is not NULL), a one-tuple or two-tuple. * The first element is a tuple interleaving keys and values, of length * 2 * self->len. The second element is the next bucket, present iff next is * non-NULL: * * ( * (keys[0], values[0], keys[1], values[1], ..., * keys[len-1], values[len-1]), * next iff non-NULL> * ) */ static PyObject * bucket_getstate(Bucket *self) { PyObject *o = NULL, *items = NULL, *state; int i, len, l; PER_USE_OR_RETURN(self, NULL); len = self->len; if (self->values) /* Bucket */ { items = PyTuple_New(len * 2); if (items == NULL) goto err; for (i = 0, l = 0; i < len; i++) { COPY_KEY_TO_OBJECT(o, self->keys[i]); if (o == NULL) goto err; PyTuple_SET_ITEM(items, l, o); l++; COPY_VALUE_TO_OBJECT(o, self->values[i]); if (o == NULL) goto err; PyTuple_SET_ITEM(items, l, o); l++; } } else /* Set */ { items = PyTuple_New(len); if (items == NULL) goto err; for (i = 0; i < len; i++) { COPY_KEY_TO_OBJECT(o, self->keys[i]); if (o == NULL) goto err; PyTuple_SET_ITEM(items, i, o); } } if (self->next) state = Py_BuildValue("OO", items, self->next); else state = Py_BuildValue("(O)", items); Py_DECREF(items); PER_UNUSE(self); return state; err: PER_UNUSE(self); Py_XDECREF(items); return NULL; } static int _bucket_setstate(Bucket *self, PyObject *state) { PyObject *k, *v, *items; Bucket *next = NULL; int i, l, len, copied=1; KEY_TYPE *keys; VALUE_TYPE *values; if (!PyArg_ParseTuple(state, "O|O:__setstate__", &items, &next)) return -1; if (!PyTuple_Check(items)) { PyErr_SetString(PyExc_TypeError, "tuple required for first state element"); return -1; } len = PyTuple_Size(items); if (len < 0) return -1; len /= 2; for (i = self->len; --i >= 0; ) { DECREF_KEY(self->keys[i]); DECREF_VALUE(self->values[i]); } self->len = 0; if (self->next) { Py_DECREF(self->next); self->next = NULL; } if (len > self->size) { keys = BTree_Realloc(self->keys, sizeof(KEY_TYPE)*len); if (keys == NULL) return -1; values = BTree_Realloc(self->values, sizeof(VALUE_TYPE)*len); if (values == NULL) return -1; self->keys = keys; self->values = values; self->size = len; } for (i=0, l=0; i < len; i++) { k = PyTuple_GET_ITEM(items, l); l++; v = PyTuple_GET_ITEM(items, l); l++; COPY_KEY_FROM_ARG(self->keys[i], k, copied); if (!copied) return -1; COPY_VALUE_FROM_ARG(self->values[i], v, copied); if (!copied) return -1; INCREF_KEY(self->keys[i]); INCREF_VALUE(self->values[i]); } self->len = len; if (next) { self->next = next; Py_INCREF(next); } return 0; } static PyObject * bucket_setstate(Bucket *self, PyObject *state) { int r; PER_PREVENT_DEACTIVATION(self); r = _bucket_setstate(self, state); PER_UNUSE(self); if (r < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static PyObject * bucket_has_key(Bucket *self, PyObject *key) { return _bucket_get(self, key, 1); } static PyObject * bucket_setdefault(Bucket *self, PyObject *args) { PyObject *key; PyObject *failobj; /* default */ PyObject *value; /* return value */ int dummy_changed; /* in order to call _bucket_set */ if (! PyArg_UnpackTuple(args, "setdefault", 2, 2, &key, &failobj)) return NULL; value = _bucket_get(self, key, 0); if (value != NULL) return value; /* The key isn't in the bucket. If that's not due to a KeyError exception, * pass back the unexpected exception. */ if (! PyErr_ExceptionMatches(PyExc_KeyError)) return NULL; PyErr_Clear(); /* Associate `key` with `failobj` in the bucket, and return `failobj`. */ value = failobj; if (_bucket_set(self, key, failobj, 0, 0, &dummy_changed) < 0) value = NULL; Py_XINCREF(value); return value; } /* forward declaration */ static int Bucket_length(Bucket *self); static PyObject * bucket_pop(Bucket *self, PyObject *args) { PyObject *key; PyObject *failobj = NULL; /* default */ PyObject *value; /* return value */ int dummy_changed; /* in order to call _bucket_set */ if (! PyArg_UnpackTuple(args, "pop", 1, 2, &key, &failobj)) return NULL; value = _bucket_get(self, key, 0); if (value != NULL) { /* Delete key and associated value. */ if (_bucket_set(self, key, NULL, 0, 0, &dummy_changed) < 0) { Py_DECREF(value); return NULL; } return value; } /* The key isn't in the bucket. If that's not due to a KeyError exception, * pass back the unexpected exception. */ if (! PyErr_ExceptionMatches(PyExc_KeyError)) return NULL; if (failobj != NULL) { /* Clear the KeyError and return the explicit default. */ PyErr_Clear(); Py_INCREF(failobj); return failobj; } /* No default given. The only difference in this case is the error * message, which depends on whether the bucket is empty. */ if (Bucket_length(self) == 0) PyErr_SetString(PyExc_KeyError, "pop(): Bucket is empty"); return NULL; } /* Search bucket self for key. This is the sq_contains slot of the * PySequenceMethods. * * Return: * -1 error * 0 not found * 1 found */ static int bucket_contains(Bucket *self, PyObject *key) { PyObject *asobj = _bucket_get(self, key, 1); int result = -1; if (asobj != NULL) { result = INT_AS_LONG(asobj) ? 1 : 0; Py_DECREF(asobj); } return result; } /* ** bucket_getm ** */ static PyObject * bucket_getm(Bucket *self, PyObject *args) { PyObject *key, *d=Py_None, *r; if (!PyArg_ParseTuple(args, "O|O:get", &key, &d)) return NULL; r = _bucket_get(self, key, 0); if (r) return r; if (!PyErr_ExceptionMatches(PyExc_KeyError)) return NULL; PyErr_Clear(); Py_INCREF(d); return d; } /**************************************************************************/ /* Iterator support. */ /* A helper to build all the iterators for Buckets and Sets. * If args is NULL, the iterator spans the entire structure. Else it's an * argument tuple, with optional low and high arguments. * kind is 'k', 'v' or 'i'. * Returns a BTreeIter object, or NULL if error. */ static PyObject * buildBucketIter(Bucket *self, PyObject *args, PyObject *kw, char kind) { BTreeItems *items; int lowoffset, highoffset; BTreeIter *result = NULL; PER_USE_OR_RETURN(self, NULL); if (Bucket_rangeSearch(self, args, kw, &lowoffset, &highoffset) < 0) goto Done; items = (BTreeItems *)newBTreeItems(kind, self, lowoffset, self, highoffset); if (items == NULL) goto Done; result = BTreeIter_new(items); /* win or lose, we're done */ Py_DECREF(items); Done: PER_UNUSE(self); return (PyObject *)result; } /* The implementation of iter(Bucket_or_Set); the Bucket tp_iter slot. */ static PyObject * Bucket_getiter(Bucket *self) { return buildBucketIter(self, NULL, NULL, 'k'); } /* The implementation of Bucket.iterkeys(). */ static PyObject * Bucket_iterkeys(Bucket *self, PyObject *args, PyObject *kw) { return buildBucketIter(self, args, kw, 'k'); } /* The implementation of Bucket.itervalues(). */ static PyObject * Bucket_itervalues(Bucket *self, PyObject *args, PyObject *kw) { return buildBucketIter(self, args, kw, 'v'); } /* The implementation of Bucket.iteritems(). */ static PyObject * Bucket_iteritems(Bucket *self, PyObject *args, PyObject *kw) { return buildBucketIter(self, args, kw, 'i'); } /* End of iterator support. */ #ifdef PERSISTENT static PyObject *merge_error(int p1, int p2, int p3, int reason); static PyObject *bucket_merge(Bucket *s1, Bucket *s2, Bucket *s3); static PyObject * _bucket__p_resolveConflict(PyObject *ob_type, PyObject *s[3]) { PyObject *result = NULL; /* guilty until proved innocent */ Bucket *b[3] = {NULL, NULL, NULL}; PyObject *meth = NULL; PyObject *a = NULL; int i; for (i = 0; i < 3; i++) { PyObject *r; b[i] = (Bucket*)PyObject_CallObject((PyObject *)ob_type, NULL); if (b[i] == NULL) goto Done; if (s[i] == Py_None) /* None is equivalent to empty, for BTrees */ continue; meth = PyObject_GetAttr((PyObject *)b[i], __setstate___str); if (meth == NULL) goto Done; a = PyTuple_New(1); if (a == NULL) goto Done; PyTuple_SET_ITEM(a, 0, s[i]); Py_INCREF(s[i]); r = PyObject_CallObject(meth, a); /* b[i].__setstate__(s[i]) */ if (r == NULL) goto Done; Py_DECREF(r); Py_DECREF(a); Py_DECREF(meth); a = meth = NULL; } if (b[0]->next != b[1]->next || b[0]->next != b[2]->next) merge_error(-1, -1, -1, 0); else result = bucket_merge(b[0], b[1], b[2]); Done: Py_XDECREF(meth); Py_XDECREF(a); Py_XDECREF(b[0]); Py_XDECREF(b[1]); Py_XDECREF(b[2]); return result; } static PyObject * bucket__p_resolveConflict(Bucket *self, PyObject *args) { PyObject *s[3]; if (!PyArg_ParseTuple(args, "OOO", &s[0], &s[1], &s[2])) return NULL; return _bucket__p_resolveConflict((PyObject *)Py_TYPE(self), s); } #endif /* Caution: Even though the _next attribute is read-only, a program could do arbitrary damage to the btree internals. For example, it could call clear() on a bucket inside a BTree. We need to decide if the convenience for inspecting BTrees is worth the risk. */ static struct PyMemberDef Bucket_members[] = { {"_next", T_OBJECT, offsetof(Bucket, next)}, {NULL} }; static struct PyMethodDef Bucket_methods[] = { {"__getstate__", (PyCFunction) bucket_getstate, METH_NOARGS, "__getstate__() -- Return the picklable state of the object"}, {"__setstate__", (PyCFunction) bucket_setstate, METH_O, "__setstate__() -- Set the state of the object"}, {"keys", (PyCFunction) bucket_keys, METH_VARARGS | METH_KEYWORDS, "keys([min, max]) -- Return the keys"}, {"has_key", (PyCFunction) bucket_has_key, METH_O, "has_key(key) -- Test whether the bucket contains the given key"}, {"clear", (PyCFunction) bucket_clear, METH_VARARGS, "clear() -- Remove all of the items from the bucket"}, {"update", (PyCFunction) Mapping_update, METH_O, "update(collection) -- Add the items from the given collection"}, {"maxKey", (PyCFunction) Bucket_maxKey, METH_VARARGS, "maxKey([key]) -- Find the maximum key\n\n" "If an argument is given, find the maximum <= the argument"}, {"minKey", (PyCFunction) Bucket_minKey, METH_VARARGS, "minKey([key]) -- Find the minimum key\n\n" "If an argument is given, find the minimum >= the argument"}, {"values", (PyCFunction) bucket_values, METH_VARARGS | METH_KEYWORDS, "values([min, max]) -- Return the values"}, {"items", (PyCFunction) bucket_items, METH_VARARGS | METH_KEYWORDS, "items([min, max])) -- Return the items"}, {"byValue", (PyCFunction) bucket_byValue, METH_O, "byValue(min) -- " "Return value-keys with values >= min and reverse sorted by values"}, {"get", (PyCFunction) bucket_getm, METH_VARARGS, "get(key[,default]) -- Look up a value\n\n" "Return the default (or None) if the key is not found."}, {"setdefault", (PyCFunction) bucket_setdefault, METH_VARARGS, "D.setdefault(k, d) -> D.get(k, d), also set D[k]=d if k not in D.\n\n" "Return the value like get() except that if key is missing, d is both\n" "returned and inserted into the bucket as the value of k."}, {"pop", (PyCFunction) bucket_pop, METH_VARARGS, "D.pop(k[, d]) -> v, remove key and return the corresponding value.\n\n" "If key is not found, d is returned if given, otherwise KeyError\n" "is raised."}, {"iterkeys", (PyCFunction) Bucket_iterkeys, METH_VARARGS | METH_KEYWORDS, "B.iterkeys([min[,max]]) -> an iterator over the keys of B"}, {"itervalues", (PyCFunction) Bucket_itervalues, METH_VARARGS | METH_KEYWORDS, "B.itervalues([min[,max]]) -> an iterator over the values of B"}, {"iteritems", (PyCFunction) Bucket_iteritems, METH_VARARGS | METH_KEYWORDS, "B.iteritems([min[,max]]) -> an iterator over the (key, value) " "items of B"}, #ifdef EXTRA_BUCKET_METHODS EXTRA_BUCKET_METHODS #endif #ifdef PERSISTENT {"_p_resolveConflict", (PyCFunction) bucket__p_resolveConflict, METH_VARARGS, "_p_resolveConflict() -- Reinitialize from a newly created copy"}, {"_p_deactivate", (PyCFunction) bucket__p_deactivate, METH_VARARGS | METH_KEYWORDS, "_p_deactivate() -- Reinitialize from a newly created copy"}, #endif {NULL, NULL} }; static int Bucket_init(PyObject *self, PyObject *args, PyObject *kwds) { PyObject *v = NULL; if (!PyArg_ParseTuple(args, "|O:" MOD_NAME_PREFIX "Bucket", &v)) return -1; if (v) return update_from_seq(self, v); else return 0; } static void bucket_dealloc(Bucket *self) { if (self->state != cPersistent_GHOST_STATE) _bucket_clear(self); cPersistenceCAPI->pertype->tp_dealloc((PyObject *)self); } static int bucket_traverse(Bucket *self, visitproc visit, void *arg) { int err = 0; int i, len; #define VISIT(SLOT) \ if (SLOT) { \ err = visit((PyObject *)(SLOT), arg); \ if (err) \ goto Done; \ } /* Call our base type's traverse function. Because buckets are * subclasses of Peristent, there must be one. */ err = cPersistenceCAPI->pertype->tp_traverse((PyObject *)self, visit, arg); if (err) goto Done; /* If this is registered with the persistence system, cleaning up cycles * is the database's problem. It would be horrid to unghostify buckets * here just to chase pointers every time gc runs. */ if (self->state == cPersistent_GHOST_STATE) goto Done; len = self->len; /* if neither keys nor values are PyObject*, "i" is otherwise unreferenced and we get a nuisance compiler wng */ (void)i; (void)len; #ifdef KEY_TYPE_IS_PYOBJECT /* Keys are Python objects so need to be traversed. */ for (i = 0; i < len; i++) VISIT(self->keys[i]); #endif #ifdef VALUE_TYPE_IS_PYOBJECT if (self->values != NULL) { /* self->values exists (this is a mapping bucket, not a set bucket), * and are Python objects, so need to be traversed. */ for (i = 0; i < len; i++) VISIT(self->values[i]); } #endif VISIT(self->next); Done: return err; #undef VISIT } static int bucket_tp_clear(Bucket *self) { if (self->state != cPersistent_GHOST_STATE) _bucket_clear(self); return 0; } /* Code to access Bucket objects as mappings */ static int Bucket_length( Bucket *self) { int r; UNLESS (PER_USE(self)) return -1; r = self->len; PER_UNUSE(self); return r; } static PyMappingMethods Bucket_as_mapping = { (lenfunc)Bucket_length, /*mp_length*/ (binaryfunc)bucket_getitem, /*mp_subscript*/ (objobjargproc)bucket_setitem, /*mp_ass_subscript*/ }; static PySequenceMethods Bucket_as_sequence = { (lenfunc)0, /* sq_length */ (binaryfunc)0, /* sq_concat */ (ssizeargfunc)0, /* sq_repeat */ (ssizeargfunc)0, /* sq_item */ (ssizessizeargfunc)0, /* sq_slice */ (ssizeobjargproc)0, /* sq_ass_item */ (ssizessizeobjargproc)0, /* sq_ass_slice */ (objobjproc)bucket_contains, /* sq_contains */ 0, /* sq_inplace_concat */ 0, /* sq_inplace_repeat */ }; static PyObject * bucket_repr(Bucket *self) { PyObject *i, *r; #ifdef PY3K PyObject *rb; #endif char repr[10000]; int rv; i = bucket_items(self, NULL, NULL); if (!i) { return NULL; } r = PyObject_Repr(i); Py_DECREF(i); if (!r) { return NULL; } #ifdef PY3K rb = PyUnicode_AsLatin1String(r); rv = PyOS_snprintf(repr, sizeof(repr), "%s(%s)", Py_TYPE(self)->tp_name, PyBytes_AsString(rb)); Py_DECREF(rb); #else rv = PyOS_snprintf(repr, sizeof(repr), "%s(%s)", Py_TYPE(self)->tp_name, PyBytes_AS_STRING(r)); #endif if (rv > 0 && (size_t)rv < sizeof(repr)) { Py_DECREF(r); #ifdef PY3K return PyUnicode_DecodeLatin1(repr, strlen(repr), "surrogateescape"); #else return PyBytes_FromStringAndSize(repr, strlen(repr)); #endif } else { /* The static buffer wasn't big enough */ int size; PyObject *s; #ifdef PY3K PyObject *result; #endif /* 3 for the parens and the null byte */ size = strlen(Py_TYPE(self)->tp_name) + PyBytes_GET_SIZE(r) + 3; s = PyBytes_FromStringAndSize(NULL, size); if (!s) { Py_DECREF(r); return r; } PyOS_snprintf(PyBytes_AS_STRING(s), size, "%s(%s)", Py_TYPE(self)->tp_name, PyBytes_AS_STRING(r)); Py_DECREF(r); #ifdef PY3K result = PyUnicode_FromEncodedObject(s, "latin1", "surrogateescape"); Py_DECREF(s); return result; #else return s; #endif } } static PyTypeObject BucketType = { PyVarObject_HEAD_INIT(NULL, 0) MODULE_NAME MOD_NAME_PREFIX "Bucket", /* tp_name */ sizeof(Bucket), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)bucket_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ (reprfunc)bucket_repr, /* tp_repr */ 0, /* tp_as_number */ &Bucket_as_sequence, /* tp_as_sequence */ &Bucket_as_mapping, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ (traverseproc)bucket_traverse, /* tp_traverse */ (inquiry)bucket_tp_clear, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ (getiterfunc)Bucket_getiter, /* tp_iter */ 0, /* tp_iternext */ Bucket_methods, /* tp_methods */ Bucket_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ Bucket_init, /* tp_init */ 0, /* tp_alloc */ 0, /*PyType_GenericNew,*/ /* tp_new */ }; static int nextBucket(SetIteration *i) { if (i->position >= 0) { UNLESS(PER_USE(BUCKET(i->set))) return -1; if (i->position) { DECREF_KEY(i->key); DECREF_VALUE(i->value); } if (i->position < BUCKET(i->set)->len) { COPY_KEY(i->key, BUCKET(i->set)->keys[i->position]); INCREF_KEY(i->key); COPY_VALUE(i->value, BUCKET(i->set)->values[i->position]); INCREF_VALUE(i->value); i->position ++; } else { i->position = -1; PER_ACCESSED(BUCKET(i->set)); } PER_ALLOW_DEACTIVATION(BUCKET(i->set)); } return 0; } BTrees-4.3.1/BTrees/Interfaces.py0000664000175000017500000004264712510775554017722 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001, 2002 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from zope.interface import Interface, Attribute class ICollection(Interface): def clear(): """Remove all of the items from the collection.""" def __nonzero__(): """Check if the collection is non-empty. Return a true value if the collection is non-empty and a false value otherwise. """ class IReadSequence(Interface): def __getitem__(index): """Return the value at the given index. An IndexError is raised if the index cannot be found. """ def __getslice__(index1, index2): """Return a subsequence from the original sequence. The subsequence includes the items from index1 up to, but not including, index2. """ class IKeyed(ICollection): def has_key(key): """Check whether the object has an item with the given key. Return a true value if the key is present, else a false value. """ def keys(min=None, max=None, excludemin=False, excludemax=False): """Return an IReadSequence containing the keys in the collection. The type of the IReadSequence is not specified. It could be a list or a tuple or some other type. All arguments are optional, and may be specified as keyword arguments, or by position. If a min is specified, then output is constrained to keys greater than or equal to the given min, and, if excludemin is specified and true, is further constrained to keys strictly greater than min. A min value of None is ignored. If min is None or not specified, and excludemin is true, the smallest key is excluded. If a max is specified, then output is constrained to keys less than or equal to the given max, and, if excludemax is specified and true, is further constrained to keys strictly less than max. A max value of None is ignored. If max is None or not specified, and excludemax is true, the largest key is excluded. """ def maxKey(key=None): """Return the maximum key. If a key argument if provided and not None, return the largest key that is less than or equal to the argument. Raise an exception if no such key exists. """ def minKey(key=None): """Return the minimum key. If a key argument if provided and not None, return the smallest key that is greater than or equal to the argument. Raise an exception if no such key exists. """ class ISetMutable(IKeyed): def insert(key): """Add the key (value) to the set. If the key was already in the set, return 0, otherwise return 1. """ def remove(key): """Remove the key from the set. Raises KeyError if key is not in the set. """ def update(seq): """Add the items from the given sequence to the set.""" class ISized(Interface): """An object that supports __len__.""" def __len__(): """Return the number of items in the container.""" class IKeySequence(IKeyed, ISized): def __getitem__(index): """Return the key in the given index position. This allows iteration with for loops and use in functions, like map and list, that read sequences. """ class ISet(IKeySequence, ISetMutable): pass class ITreeSet(IKeyed, ISetMutable): pass class IMinimalDictionary(ISized, IKeyed): def get(key, default): """Get the value associated with the given key. Return the default if has_key(key) is false. """ def __getitem__(key): """Get the value associated with the given key. Raise KeyError if has_key(key) is false. """ def __setitem__(key, value): """Set the value associated with the given key.""" def __delitem__(key): """Delete the value associated with the given key. Raise KeyError if has_key(key) is false. """ def values(min=None, max=None, excludemin=False, excludemax=False): """Return an IReadSequence containing the values in the collection. The type of the IReadSequence is not specified. It could be a list or a tuple or some other type. All arguments are optional, and may be specified as keyword arguments, or by position. If a min is specified, then output is constrained to values whose keys are greater than or equal to the given min, and, if excludemin is specified and true, is further constrained to values whose keys are strictly greater than min. A min value of None is ignored. If min is None or not specified, and excludemin is true, the value corresponding to the smallest key is excluded. If a max is specified, then output is constrained to values whose keys are less than or equal to the given max, and, if excludemax is specified and true, is further constrained to values whose keys are strictly less than max. A max value of None is ignored. If max is None or not specified, and excludemax is true, the value corresponding to the largest key is excluded. """ def items(min=None, max=None, excludemin=False, excludemax=False): """Return an IReadSequence containing the items in the collection. An item is a 2-tuple, a (key, value) pair. The type of the IReadSequence is not specified. It could be a list or a tuple or some other type. All arguments are optional, and may be specified as keyword arguments, or by position. If a min is specified, then output is constrained to items whose keys are greater than or equal to the given min, and, if excludemin is specified and true, is further constrained to items whose keys are strictly greater than min. A min value of None is ignored. If min is None or not specified, and excludemin is true, the item with the smallest key is excluded. If a max is specified, then output is constrained to items whose keys are less than or equal to the given max, and, if excludemax is specified and true, is further constrained to items whose keys are strictly less than max. A max value of None is ignored. If max is None or not specified, and excludemax is true, the item with the largest key is excluded. """ class IDictionaryIsh(IMinimalDictionary): def update(collection): """Add the items from the given collection object to the collection. The input collection must be a sequence of (key, value) 2-tuples, or an object with an 'items' method that returns a sequence of (key, value) pairs. """ def byValue(minValue): """Return a sequence of (value, key) pairs, sorted by value. Values < minValue are omitted and other values are "normalized" by the minimum value. This normalization may be a noop, but, for integer values, the normalization is division. """ def setdefault(key, d): """D.setdefault(k, d) -> D.get(k, d), also set D[k]=d if k not in D. Return the value like get() except that if key is missing, d is both returned and inserted into the dictionary as the value of k. Note that, unlike as for Python's dict.setdefault(), d is not optional. Python defaults d to None, but that doesn't make sense for mappings that can't have None as a value (for example, an IIBTree can have only integers as values). """ def pop(key, d): """D.pop(k[, d]) -> v, remove key and return the corresponding value. If key is not found, d is returned if given, otherwise KeyError is raised. """ class IBTree(IDictionaryIsh): def insert(key, value): """Insert a key and value into the collection. If the key was already in the collection, then there is no change and 0 is returned. If the key was not already in the collection, then the item is added and 1 is returned. This method is here to allow one to generate random keys and to insert and test whether the key was there in one operation. A standard idiom for generating new keys will be:: key = generate_key() while not t.insert(key, value): key=generate_key() """ class IMerge(Interface): """Object with methods for merging sets, buckets, and trees. These methods are supplied in modules that define collection classes with particular key and value types. The operations apply only to collections from the same module. For example, the IIBTree.union can only be used with IIBTree.IIBTree, IIBTree.IIBucket, IIBTree.IISet, and IIBTree.IITreeSet. The implementing module has a value type. The IOBTree and OOBTree modules have object value type. The IIBTree and OIBTree modules have integer value types. Other modules may be defined in the future that have other value types. The individual types are classified into set (Set and TreeSet) and mapping (Bucket and BTree) types. """ def difference(c1, c2): """Return the keys or items in c1 for which there is no key in c2. If c1 is None, then None is returned. If c2 is None, then c1 is returned. If neither c1 nor c2 is None, the output is a Set if c1 is a Set or TreeSet, and is a Bucket if c1 is a Bucket or BTree. """ def union(c1, c2): """Compute the Union of c1 and c2. If c1 is None, then c2 is returned, otherwise, if c2 is None, then c1 is returned. The output is a Set containing keys from the input collections. """ def intersection(c1, c2): """Compute the intersection of c1 and c2. If c1 is None, then c2 is returned, otherwise, if c2 is None, then c1 is returned. The output is a Set containing matching keys from the input collections. """ class IBTreeModule(Interface): """These are available in all modules (IOBTree, OIBTree, OOBTree, IIBTree, IFBTree, LFBTree, LOBTree, OLBTree, and LLBTree). """ BTree = Attribute( """The IBTree for this module. Also available as [prefix]BTree, as in IOBTree.""") Bucket = Attribute( """The leaf-node data buckets used by the BTree. (IBucket is not currently defined in this file, but is essentially IDictionaryIsh, with the exception of __nonzero__, as of this writing.) Also available as [prefix]Bucket, as in IOBucket.""") TreeSet = Attribute( """The ITreeSet for this module. Also available as [prefix]TreeSet, as in IOTreeSet.""") Set = Attribute( """The ISet for this module: the leaf-node data buckets used by the TreeSet. Also available as [prefix]BTree, as in IOSet.""") class IIMerge(IMerge): """Merge collections with integer value type. A primary intent is to support operations with no or integer values, which are used as "scores" to rate indiviual keys. That is, in this context, a BTree or Bucket is viewed as a set with scored keys, using integer scores. """ def weightedUnion(c1, c2, weight1=1, weight2=1): """Compute the weighted union of c1 and c2. If c1 and c2 are None, the output is (0, None). If c1 is None and c2 is not None, the output is (weight2, c2). If c1 is not None and c2 is None, the output is (weight1, c1). Else, and hereafter, c1 is not None and c2 is not None. If c1 and c2 are both sets, the output is 1 and the (unweighted) union of the sets. Else the output is 1 and a Bucket whose keys are the union of c1 and c2's keys, and whose values are:: v1*weight1 + v2*weight2 where: v1 is 0 if the key is not in c1 1 if the key is in c1 and c1 is a set c1[key] if the key is in c1 and c1 is a mapping v2 is 0 if the key is not in c2 1 if the key is in c2 and c2 is a set c2[key] if the key is in c2 and c2 is a mapping Note that c1 and c2 must be collections. """ def weightedIntersection(c1, c2, weight1=1, weight2=1): """Compute the weighted intersection of c1 and c2. If c1 and c2 are None, the output is (0, None). If c1 is None and c2 is not None, the output is (weight2, c2). If c1 is not None and c2 is None, the output is (weight1, c1). Else, and hereafter, c1 is not None and c2 is not None. If c1 and c2 are both sets, the output is the sum of the weights and the (unweighted) intersection of the sets. Else the output is 1 and a Bucket whose keys are the intersection of c1 and c2's keys, and whose values are:: v1*weight1 + v2*weight2 where: v1 is 1 if c1 is a set c1[key] if c1 is a mapping v2 is 1 if c2 is a set c2[key] if c2 is a mapping Note that c1 and c2 must be collections. """ class IMergeIntegerKey(IMerge): """IMerge-able objects with integer keys. Concretely, this means the types in IOBTree and IIBTree. """ def multiunion(seq): """Return union of (zero or more) integer sets, as an integer set. seq is a sequence of objects each convertible to an integer set. These objects are convertible to an integer set: + An integer, which is added to the union. + A Set or TreeSet from the same module (for example, an IIBTree.TreeSet for IIBTree.multiunion()). The elements of the set are added to the union. + A Bucket or BTree from the same module (for example, an IOBTree.IOBTree for IOBTree.multiunion()). The keys of the mapping are added to the union. The union is returned as a Set from the same module (for example, IIBTree.multiunion() returns an IIBTree.IISet). The point to this method is that it can run much faster than doing a sequence of two-input union() calls. Under the covers, all the integers in all the inputs are sorted via a single linear-time radix sort, then duplicates are removed in a second linear-time pass. """ class IBTreeFamily(Interface): """the 64-bit or 32-bit family""" IO = Attribute('The IIntegerObjectBTreeModule for this family') OI = Attribute('The IObjectIntegerBTreeModule for this family') II = Attribute('The IIntegerIntegerBTreeModule for this family') IF = Attribute('The IIntegerFloatBTreeModule for this family') OO = Attribute('The IObjectObjectBTreeModule for this family') maxint = Attribute('The maximum integer storable in this family') minint = Attribute('The minimum integer storable in this family') class IIntegerObjectBTreeModule(IBTreeModule, IMerge): """keys, or set values, are integers; values are objects. describes IOBTree and LOBTree""" family = Attribute('The IBTreeFamily of this module') class IObjectIntegerBTreeModule(IBTreeModule, IIMerge): """keys, or set values, are objects; values are integers. Object keys (and set values) must sort reliably (for instance, *not* on object id)! Homogenous key types recommended. describes OIBTree and LOBTree""" family = Attribute('The IBTreeFamily of this module') class IIntegerIntegerBTreeModule(IBTreeModule, IIMerge, IMergeIntegerKey): """keys, or set values, are integers; values are also integers. describes IIBTree and LLBTree""" family = Attribute('The IBTreeFamily of this module') class IObjectObjectBTreeModule(IBTreeModule, IMerge): """keys, or set values, are objects; values are also objects. Object keys (and set values) must sort reliably (for instance, *not* on object id)! Homogenous key types recommended. describes OOBTree""" # Note that there's no ``family`` attribute; all families include # the OO flavor of BTrees. class IIntegerFloatBTreeModule(IBTreeModule, IMerge): """keys, or set values, are integers; values are floats. describes IFBTree and LFBTree""" family = Attribute('The IBTreeFamily of this module') try: from ZODB.POSException import BTreesConflictError except ImportError: class BTreesConflictError(ValueError): @property def reason(self): return self.args[-1] ############################################################### # IMPORTANT NOTE # # Getting the length of a BTree, TreeSet, or output of keys, # values, or items of same is expensive. If you need to get the # length, you need to maintain this separately. # # Eventually, I need to express this through the interfaces. # ################################################################ BTrees-4.3.1/BTrees/OLBTree.py0000664000175000017500000001001012665316015017040 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __all__ = ('Bucket', 'Set', 'BTree', 'TreeSet', 'OLBucket', 'OLSet', 'OLBTree', 'OLTreeSet', 'union', 'intersection', 'difference', 'weightedUnion', 'weightedIntersection', ) from zope.interface import moduleProvides from .Interfaces import IObjectIntegerBTreeModule from ._base import Bucket from ._base import MERGE from ._base import MERGE_WEIGHT_numeric from ._base import MERGE_DEFAULT_int from ._base import Set from ._base import Tree as BTree from ._base import TreeSet from ._base import _TreeIterator from ._base import difference as _difference from ._base import intersection as _intersection from ._base import set_operation as _set_operation from ._base import to_ob as _to_key from ._base import to_long as _to_value from ._base import union as _union from ._base import weightedIntersection as _weightedIntersection from ._base import weightedUnion as _weightedUnion from ._base import _fix_pickle _BUCKET_SIZE = 60 _TREE_SIZE = 250 using64bits = True class OLBucketPy(Bucket): _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class OLSetPy(Set): _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class OLBTreePy(BTree): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class OLTreeSetPy(TreeSet): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class OLTreeIteratorPy(_TreeIterator): pass # Can't declare forward refs, so fix up afterwards: OLBucketPy._mapping_type = OLBucketPy._bucket_type = OLBucketPy OLBucketPy._set_type = OLSetPy OLSetPy._mapping_type = OLBucketPy OLSetPy._set_type = OLSetPy._bucket_type = OLSetPy OLBTreePy._mapping_type = OLBTreePy._bucket_type = OLBucketPy OLBTreePy._set_type = OLSetPy OLTreeSetPy._mapping_type = OLBucketPy OLTreeSetPy._set_type = OLTreeSetPy._bucket_type = OLSetPy differencePy = _set_operation(_difference, OLSetPy) unionPy = _set_operation(_union, OLSetPy) intersectionPy = _set_operation(_intersection, OLSetPy) weightedUnionPy = _set_operation(_weightedUnion, OLSetPy) weightedIntersectionPy = _set_operation(_weightedIntersection, OLSetPy) try: from ._OLBTree import OLBucket except ImportError: #pragma NO COVER w/ C extensions OLBucket = OLBucketPy OLSet = OLSetPy OLBTree = OLBTreePy OLTreeSet = OLTreeSetPy OLTreeIterator = OLTreeIteratorPy difference = differencePy union = unionPy intersection = intersectionPy weightedUnion = weightedUnionPy weightedIntersection = weightedIntersectionPy else: #pragma NO COVER w/o C extensions from ._OLBTree import OLSet from ._OLBTree import OLBTree from ._OLBTree import OLTreeSet from ._OLBTree import OLTreeIterator from ._OLBTree import difference from ._OLBTree import union from ._OLBTree import intersection from ._OLBTree import weightedUnion from ._OLBTree import weightedIntersection Bucket = OLBucket Set = OLSet BTree = OLBTree TreeSet = OLTreeSet _fix_pickle(globals(), __name__) moduleProvides(IObjectIntegerBTreeModule) BTrees-4.3.1/BTrees/_base.py0000664000175000017500000014037412702742621016674 0ustar tseavertseaver00000000000000############################################################################## # # Copyright 2011 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Python BTree implementation """ from struct import pack from struct import unpack from struct import error as struct_error from persistent import Persistent from .Interfaces import BTreesConflictError from ._compat import PY3 from ._compat import cmp from ._compat import int_types from ._compat import xrange _marker = object() class _Base(Persistent): __slots__ = () _key_type = list def __init__(self, items=None): self.clear() if items: self.update(items) try: # Detect the presence of the C extensions. # If they're NOT around, we don't need to do any of the # special pickle support to make Python versions look like # C---we just rename the classes. By not defining these methods, # we can (theoretically) avoid a bit of a slowdown. # If the C extensions are around, we do need these methods, but # these classes are unlikely to be used in production anyway. __import__('BTrees._OOBTree') except ImportError: # pragma: no cover pass else: def __reduce__(self): # Swap out the type constructor for the C version, if present. func, typ_gna, state = Persistent.__reduce__(self) # We ignore the returned type altogether in favor of # our calculated class (which allows subclasses but replaces our exact # type with the C equivalent) typ = self.__class__ gna = typ_gna[1:] return (func, (typ,) + gna, state) @property def __class__(self): type_self = type(self) return type_self._BTree_reduce_as if type_self._BTree_reduce_up_bound is type_self else type_self @property def _BTree_reduce_as(self): # Return the pickle replacement class for this object. # If the C extensions are available, this will be the # C type (setup by _fix_pickle), otherwise it will be the real # type of this object. # This implementation is replaced by _fix_pickle and exists for # testing purposes. return type(self) # pragma: no cover _BTree_reduce_up_bound = _BTree_reduce_as class _BucketBase(_Base): __slots__ = ('_keys', '_next', '_to_key', ) def clear(self): self._keys = self._key_type() self._next = None def __len__(self): return len(self._keys) @property def size(self): return len(self._keys) def _deleteNextBucket(self): next = self._next if next is not None: self._next = next._next def _search(self, key): # Return non-negative index on success # return -(insertion_index + 1) on fail low = 0 keys = self._keys high = len(keys) while low < high: i = (low + high) // 2 k = keys[i] if k == key: return i if k < key: low = i + 1 else: high = i return -1 - low def minKey(self, key=_marker): if key is _marker or key is None: return self._keys[0] key = self._to_key(key) index = self._search(key) if index >= 0: return key index = -index - 1 if index < len(self._keys): return self._keys[index] else: raise ValueError("no key satisfies the conditions") def maxKey(self, key=_marker): if key is _marker or key is None: return self._keys[-1] key = self._to_key(key) index = self._search(key) if index >= 0: return key else: index = -index-1 if index: return self._keys[index-1] else: raise ValueError("no key satisfies the conditions") def _range(self, min=_marker, max=_marker, excludemin=False, excludemax=False): if min is _marker or min is None: start = 0 if excludemin: start = 1 else: min = self._to_key(min) start = self._search(min) if start >= 0: if excludemin: start += 1 else: start = -start - 1 if max is _marker or max is None: end = len(self._keys) if excludemax: end -= 1 else: max = self._to_key(max) end = self._search(max) if end >= 0: if not excludemax: end += 1 else: end = -end - 1 return start, end def keys(self, *args, **kw): start, end = self._range(*args, **kw) return self._keys[start:end] def iterkeys(self, *args, **kw): if not (args or kw): return iter(self._keys) keys = self._keys return (keys[i] for i in xrange(*self._range(*args, **kw))) def __iter__(self): return iter(self._keys) def __contains__(self, key): return (self._search(self._to_key(key)) >= 0) has_key = __contains__ def _repr_helper(self, items): type_self = type(self) mod = type_self.__module__ name = type_self.__name__ name = name[:-2] if name.endswith("Py") else name return "%s.%s(%r)" % (mod, name, items) class _SetIteration(object): __slots__ = ('to_iterate', 'useValues', '_iter', 'active', 'position', 'key', 'value', ) def __init__(self, to_iterate, useValues=False, default=None): if to_iterate is None: to_iterate = () self.to_iterate = to_iterate if useValues: try: itmeth = to_iterate.iteritems except AttributeError: if PY3 and isinstance(to_iterate, dict): #pragma no cover Py3k itmeth = to_iterate.items().__iter__ else: itmeth = to_iterate.__iter__ useValues = False else: self.value = None else: itmeth = to_iterate.__iter__ self.useValues = useValues self._iter = itmeth() self.active = True self.position = 0 self.key = _marker self.value = default self.advance() def advance(self): try: if self.useValues: self.key, self.value = next(self._iter) else: self.key = next(self._iter) self.position += 1 except StopIteration: self.active = False self.position = -1 return self _object_lt = getattr(object, '__lt__', _marker) def _no_default_comparison(key): # Enforce test that key has non-default comparison. if key is None: raise TypeError("Can't use None as a key") if type(key) is object: raise TypeError("Can't use object() as keys") lt = getattr(key, '__lt__', None) if lt is not None: # CPython 3.x follows PEP 252, defining '__objclass__' if getattr(lt, '__objclass__', None) is object: lt = None # pragma: no cover Py3k # PyPy3 doesn't follow PEP 252, but defines '__func__' elif getattr(lt, '__func__', None) is _object_lt: lt = None # pragma: no cover PyPy3 if (lt is None and getattr(key, '__cmp__', None) is None): raise TypeError("Can't use default __cmp__") class Bucket(_BucketBase): __slots__ = () _value_type = list _to_value = lambda self, x: x VALUE_SAME_CHECK = False def setdefault(self, key, value): key, value = self._to_key(key), self._to_value(value) status, value = self._set(key, value, True) return value def pop(self, key, default=_marker): try: status, value = self._del(self._to_key(key)) except KeyError: if default is _marker: raise return default else: return value def update(self, items): if hasattr(items, 'iteritems'): items = items.iteritems() elif hasattr(items, 'items'): items = items.items() _si = self.__setitem__ try: for key, value in items: _si(key, value) except ValueError: raise TypeError('items must be a sequence of 2-tuples') def __setitem__(self, key, value): _no_default_comparison(key) self._set(self._to_key(key), self._to_value(value)) def __delitem__(self, key): self._del(self._to_key(key)) def clear(self): _BucketBase.clear(self) self._values = self._value_type() def get(self, key, default=None): index = self._search(self._to_key(key)) if index < 0: return default return self._values[index] def __getitem__(self, key): index = self._search(self._to_key(key)) if index < 0: raise KeyError(key) return self._values[index] def _set(self, key, value, ifunset=False): """Set a value Return: status, value Status is: None if no change 0 if change, but not size change 1 if change and size change """ index = self._search(key) if index >= 0: if (ifunset or self.VALUE_SAME_CHECK and value == self._values[index] ): return None, self._values[index] self._p_changed = True self._values[index] = value return 0, value else: self._p_changed = True index = -index - 1 self._keys.insert(index, key) self._values.insert(index, value) return 1, value def _del(self, key): index = self._search(key) if index >= 0: self._p_changed = True del self._keys[index] return 0, self._values.pop(index) raise KeyError(key) def _split(self, index=-1): if index < 0 or index >= len(self._keys): index = len(self._keys) // 2 new_instance = type(self)() new_instance._keys = self._keys[index:] new_instance._values = self._values[index:] del self._keys[index:] del self._values[index:] new_instance._next = self._next self._next = new_instance return new_instance def values(self, *args, **kw): start, end = self._range(*args, **kw) return self._values[start:end] def itervalues(self, *args, **kw): values = self._values return (values[i] for i in xrange(*self._range(*args, **kw))) def items(self, *args, **kw): keys = self._keys values = self._values return [(keys[i], values[i]) for i in xrange(*self._range(*args, **kw))] def iteritems(self, *args, **kw): keys = self._keys values = self._values return ((keys[i], values[i]) for i in xrange(*self._range(*args, **kw))) def __getstate__(self): keys = self._keys values = self._values data = [] for i in range(len(keys)): data.append(keys[i]) data.append(values[i]) data = tuple(data) if self._next is not None: return data, self._next return (data, ) def __setstate__(self, state): if not isinstance(state[0], tuple): raise TypeError("tuple required for first state element") self.clear() if len(state) == 2: state, self._next = state else: self._next = None state = state[0] keys = self._keys values = self._values for i in range(0, len(state), 2): keys.append(state[i]) values.append(state[i+1]) def _p_resolveConflict(self, s_old, s_com, s_new): b_old = type(self)() if s_old is not None: b_old.__setstate__(s_old) b_com = type(self)() if s_com is not None: b_com.__setstate__(s_com) b_new = type(self)() if s_new is not None: b_new.__setstate__(s_new) if (b_com._next != b_old._next or b_new._next != b_old._next): raise BTreesConflictError(-1, -1, -1, 0) if not b_com or not b_new: raise BTreesConflictError(-1, -1, -1, 12) i_old = _SetIteration(b_old, True) i_com = _SetIteration(b_com, True) i_new = _SetIteration(b_new, True) def merge_error(reason): return BTreesConflictError( i_old.position, i_com.position, i_new.position, reason) result = type(self)() def merge_output(it): result._keys.append(it.key) result._values.append(it.value) it.advance() while i_old.active and i_com.active and i_new.active: cmpOC = cmp(i_old.key, i_com.key) cmpON = cmp(i_old.key, i_new.key) if cmpOC == 0: if cmpON == 0: if i_com.value == i_old.value: result[i_old.key] = i_new.value elif i_new.value == i_old.value: result[i_old.key] = i_com.value else: raise merge_error(1) i_old.advance() i_com.advance() i_new.advance() elif (cmpON > 0): # insert in new merge_output(i_new) elif i_old.value == i_com.value: # deleted new if i_new.position == 1: # Deleted the first item. This will modify the # parent node, so we don't know if merging will be # safe raise merge_error(13) i_old.advance() i_com.advance() else: raise merge_error(2) elif cmpON == 0: if cmpOC > 0: # insert committed merge_output(i_com) elif i_old.value == i_new.value: # delete committed if i_com.position == 1: # Deleted the first item. This will modify the # parent node, so we don't know if merging will be # safe raise merge_error(13) i_old.advance() i_new.advance() else: raise merge_error(3) else: # both keys changed cmpCN = cmp(i_com.key, i_new.key) if cmpCN == 0: # dueling insert raise merge_error(4) if cmpOC > 0: # insert committed if cmpCN > 0: # insert i_new first merge_output(i_new) else: merge_output(i_com) elif cmpON > 0: # insert i_new merge_output(i_new) else: raise merge_error(5) # both deleted same key while i_com.active and i_new.active: # new inserts cmpCN = cmp(i_com.key, i_new.key) if cmpCN == 0: raise merge_error(6) # dueling insert if cmpCN > 0: # insert new merge_output(i_new) else: # insert committed merge_output(i_com) while i_old.active and i_com.active: # new deletes rest of original cmpOC = cmp(i_old.key, i_com.key) if cmpOC > 0: # insert committed merge_output(i_com) elif cmpOC == 0 and (i_old.value == i_com.value): # del in new i_old.advance() i_com.advance() else: # dueling deletes or delete and change raise merge_error(7) while i_old.active and i_new.active: # committed deletes rest of original cmpON = cmp(i_old.key, i_new.key) if cmpON > 0: # insert new merge_output(i_new) elif cmpON == 0 and (i_old.value == i_new.value): # deleted in committed i_old.advance() i_new.advance() else: # dueling deletes or delete and change raise merge_error(8) if i_old.active: # dueling deletes raise merge_error(9) while i_com.active: merge_output(i_com) while i_new.active: merge_output(i_new) if len(result._keys) == 0: #pragma: no cover # If the output bucket is empty, conflict resolution doesn't have # enough info to unlink it from its containing BTree correctly. # # XXX TS, 2012-11-16: I don't think this is possible # raise merge_error(10) result._next = b_old._next return result.__getstate__() def __repr__(self): return self._repr_helper(self.items()) class Set(_BucketBase): __slots__ = () def add(self, key): return self._set(self._to_key(key))[0] insert = add def remove(self, key): self._del(self._to_key(key)) def update(self, items): add = self.add for i in items: add(i) def __getstate__(self): data = tuple(self._keys) if self._next is not None: return data, self._next return (data, ) def __setstate__(self, state): if not isinstance(state[0], tuple): raise TypeError('tuple required for first state element') self.clear() if len(state) == 2: state, self._next = state else: self._next = None state = state[0] self._keys.extend(state) def _set(self, key, value=None, ifunset=False): index = self._search(key) if index < 0: index = -index - 1 self._p_changed = True self._keys.insert(index, key) return True, None return False, None def _del(self, key): index = self._search(key) if index >= 0: self._p_changed = True del self._keys[index] return 0, 0 raise KeyError(key) def __getitem__(self, i): return self._keys[i] def _split(self, index=-1): if index < 0 or index >= len(self._keys): index = len(self._keys) // 2 new_instance = type(self)() new_instance._keys = self._keys[index:] del self._keys[index:] new_instance._next = self._next self._next = new_instance return new_instance def _p_resolveConflict(self, s_old, s_com, s_new): b_old = type(self)() if s_old is not None: b_old.__setstate__(s_old) b_com = type(self)() if s_com is not None: b_com.__setstate__(s_com) b_new = type(self)() if s_new is not None: b_new.__setstate__(s_new) if (b_com._next != b_old._next or b_new._next != b_old._next): # conflict: com or new changed _next raise BTreesConflictError(-1, -1, -1, 0) if not b_com or not b_new: # conflict: com or new empty raise BTreesConflictError(-1, -1, -1, 12) i_old = _SetIteration(b_old, True) i_com = _SetIteration(b_com, True) i_new = _SetIteration(b_new, True) def merge_error(reason): return BTreesConflictError( i_old.position, i_com.position, i_new.position, reason) result = type(self)() def merge_output(it): result._keys.append(it.key) it.advance() while i_old.active and i_com.active and i_new.active: cmpOC = cmp(i_old.key, i_com.key) cmpON = cmp(i_old.key, i_new.key) if cmpOC == 0: if cmpON == 0: # all match merge_output(i_old) i_com.advance() i_new.advance() elif cmpON > 0: # insert in new merge_output(i_new) else: # deleted new if i_new.position == 1: # Deleted the first item. This will modify the # parent node, so we don't know if merging will be # safe raise merge_error(13) i_old.advance() i_com.advance() elif cmpON == 0: if cmpOC > 0: # insert committed merge_output(i_com) else: # delete committed if i_com.position == 1: # Deleted the first item. This will modify the # parent node, so we don't know if merging will be # safe raise merge_error(13) i_old.advance() i_new.advance() else: # both com and new keys changed cmpCN = cmp(i_com.key, i_new.key) if cmpCN == 0: # both inserted same key raise merge_error(4) if cmpOC > 0: # insert committed if cmpCN > 0: # insert i_new first merge_output(i_new) else: merge_output(i_com) elif cmpON > 0: # insert i_new merge_output(i_new) else: # both com and new deleted same key raise merge_error(5) while i_com.active and i_new.active: # new inserts cmpCN = cmp(i_com.key, i_new.key) if cmpCN == 0: # dueling insert raise merge_error(6) if cmpCN > 0: # insert new merge_output(i_new) else: # insert committed merge_output(i_com) while i_old.active and i_com.active: # new deletes rest of original cmpOC = cmp(i_old.key, i_com.key) if cmpOC > 0: # insert committed merge_output(i_com) elif cmpOC == 0: # del in new i_old.advance() i_com.advance() else: # dueling deletes or delete and change raise merge_error(7) while i_old.active and i_new.active: # committed deletes rest of original cmpON = cmp(i_old.key, i_new.key) if cmpON > 0: # insert new merge_output(i_new) elif cmpON == 0: # deleted in committed i_old.advance() i_new.advance() else: # dueling deletes or delete and change raise merge_error(8) if i_old.active: # dueling deletes raise merge_error(9) while i_com.active: merge_output(i_com) while i_new.active: merge_output(i_new) if len(result._keys) == 0: #pragma: no cover # If the output bucket is empty, conflict resolution doesn't have # enough info to unlink it from its containing BTree correctly. # # XXX TS, 2012-11-16: I don't think this is possible # raise merge_error(10) result._next = b_old._next return result.__getstate__() def __repr__(self): return self._repr_helper(self._keys) class _TreeItem(object): __slots__ = ('key', 'child', ) def __init__(self, key, child): self.key = key self.child = child class _Tree(_Base): __slots__ = ('_data', '_firstbucket', ) def __new__(cls, *args): value = _Base.__new__(cls, *args) # Empty trees don't get their __setstate__ called upon # unpickling (or __init__, obviously), so clear() is never called # and _data and _firstbucket are never defined, unless we do it here. value._data = [] value._firstbucket = None return value def setdefault(self, key, value): return self._set(self._to_key(key), self._to_value(value), True)[1] def pop(self, key, default=_marker): try: return self._del(self._to_key(key))[1] except KeyError: if default is _marker: raise return default def update(self, items): if hasattr(items, 'iteritems'): items = items.iteritems() elif hasattr(items, 'items'): items = items.items() set = self.__setitem__ for i in items: set(*i) def __setitem__(self, key, value): _no_default_comparison(key) self._set(self._to_key(key), self._to_value(value)) def __delitem__(self, key): self._del(self._to_key(key)) def clear(self): if self._data: # In the case of __init__, this was already set by __new__ self._data = [] self._firstbucket = None def __nonzero__(self): return bool(self._data) __bool__ = __nonzero__ #Py3k rename def __len__(self): l = 0 bucket = self._firstbucket while bucket is not None: l += len(bucket._keys) bucket = bucket._next return l @property def size(self): return len(self._data) def _search(self, key): data = self._data if data: lo = 0 hi = len(data) i = hi // 2 while i > lo: cmp_ = cmp(data[i].key, key) if cmp_ < 0: lo = i elif cmp_ > 0: hi = i else: break i = (lo + hi) // 2 return i return -1 def _findbucket(self, key): index = self._search(key) if index >= 0: child = self._data[index].child if isinstance(child, self._bucket_type): return child return child._findbucket(key) def __contains__(self, key): return key in (self._findbucket(self._to_key(key)) or ()) def has_key(self, key): index = self._search(key) if index < 0: return False r = self._data[index].child.has_key(key) return r and r + 1 def keys(self, min=_marker, max=_marker, excludemin=False, excludemax=False, itertype='iterkeys'): if not self._data: return () if min is not _marker and min is not None: min = self._to_key(min) bucket = self._findbucket(min) else: bucket = self._firstbucket iterargs = min, max, excludemin, excludemax return _TreeItems(bucket, itertype, iterargs) def iterkeys(self, min=_marker, max=_marker, excludemin=False, excludemax=False): return iter(self.keys(min, max, excludemin, excludemax)) def __iter__(self): return iter(self.keys()) def minKey(self, min=_marker): if min is _marker or min is None: bucket = self._firstbucket else: min = self._to_key(min) bucket = self._findbucket(min) if bucket is not None: return bucket.minKey(min) raise ValueError('empty tree') def maxKey(self, max=_marker): data = self._data if not data: raise ValueError('empty tree') if max is _marker or max is None: return data[-1].child.maxKey() max = self._to_key(max) index = self._search(max) if index and data[index].child.minKey() > max: index -= 1 #pragma: no cover no idea how to provoke this return data[index].child.maxKey(max) def _set(self, key, value=None, ifunset=False): if (self._p_jar is not None and self._p_oid is not None and self._p_serial is not None): self._p_jar.readCurrent(self) data = self._data if data: index = self._search(key) child = data[index].child else: index = 0 child = self._bucket_type() self._firstbucket = child data.append(_TreeItem(None, child)) result = child._set(key, value, ifunset) grew = result[0] if grew: if type(child) is type(self): max_size = self.max_internal_size else: max_size = self.max_leaf_size if child.size > max_size: self._grow(child, index) # If a BTree contains only a single bucket, BTree.__getstate__() # includes the bucket's entire state, and the bucket doesn't get # an oid of its own. So if we have a single oid-less bucket that # changed, it's *our* oid that should be marked as changed -- the # bucket doesn't have one. if (grew is not None and type(child) is self._bucket_type and len(data) == 1 and child._p_oid is None): self._p_changed = 1 return result def _grow(self, child, index): self._p_changed = True new_child = child._split() self._data.insert(index+1, _TreeItem(new_child.minKey(), new_child)) if len(self._data) >= self.max_internal_size * 2: self._split_root() def _split_root(self): child = type(self)() child._data = self._data child._firstbucket = self._firstbucket self._data = [_TreeItem(None, child)] self._grow(child, 0) def _split(self, index=None): data = self._data if index is None: index = len(data) // 2 next = type(self)() next._data = data[index:] first = data[index] del data[index:] if len(data) == 0: self._firstbucket = None # lost our bucket, can't buy no beer if isinstance(first.child, type(self)): next._firstbucket = first.child._firstbucket else: next._firstbucket = first.child; return next def _del(self, key): if (self._p_jar is not None and self._p_oid is not None and self._p_serial is not None): self._p_jar.readCurrent(self) data = self._data if not data: raise KeyError(key) index = self._search(key) child = data[index].child removed_first_bucket, value = child._del(key) # See comment in _set about small trees if (len(data) == 1 and type(child) is self._bucket_type and child._p_oid is None): self._p_changed = True # fix up the node key, but not for the 0'th one. if index > 0 and child.size and key == data[index].key: self._p_changed = True data[index].key = child.minKey() if removed_first_bucket: if index: data[index-1].child._deleteNextBucket() removed_first_bucket = False # clear flag else: self._firstbucket = child._firstbucket if not child.size: if type(child) is self._bucket_type: if index: data[index-1].child._deleteNextBucket() else: self._firstbucket = child._next removed_first_bucket = True del data[index] return removed_first_bucket, value def _deleteNextBucket(self): self._data[-1].child._deleteNextBucket() def __getstate__(self): data = self._data if not data: # Note: returning None here causes our __setstate__ # to not be called on unpickling return None if (len(data) == 1 and type(data[0].child) is not type(self) and data[0].child._p_oid is None ): return ((data[0].child.__getstate__(), ), ) sdata = [] for item in data: if sdata: sdata.append(item.key) sdata.append(item.child) else: sdata.append(item.child) return tuple(sdata), self._firstbucket def __setstate__(self, state): if state and not isinstance(state[0], tuple): raise TypeError('tuple required for first state element') self.clear() if state is None: return if len(state) == 1: bucket = self._bucket_type() bucket.__setstate__(state[0][0]) state = [bucket], bucket data, self._firstbucket = state data = list(reversed(data)) self._data.append(_TreeItem(None, data.pop())) while data: key = data.pop() child = data.pop() self._data.append(_TreeItem(key, child)) def _assert(self, condition, message): if not condition: raise AssertionError(message) def _check(self, nextbucket=None): data = self._data assert_ = self._assert if not data: assert_(self._firstbucket is None, "Empty BTree has non-NULL firstbucket") return assert_(self._firstbucket is not None, "Non-empty BTree has NULL firstbucket") child_class = type(data[0].child) for i in data: assert_(i.child is not None, "BTree has NULL child") assert_(type(i.child) is child_class, "BTree children have different types") assert_(i.child.size, "Bucket length < 1") if child_class is type(self): assert_(self._firstbucket is data[0].child._firstbucket, "BTree has firstbucket different than " "its first child's firstbucket") for i in range(len(data)-1): data[i].child._check(data[i+1].child._firstbucket) data[-1].child._check(nextbucket) elif child_class is self._bucket_type: assert_(self._firstbucket is data[0].child, "Bottom-level BTree node has inconsistent firstbucket " "belief") for i in range(len(data)-1): assert_(data[i].child._next is data[i+1].child, "Bucket next pointer is damaged") assert_(data[-1].child._next is nextbucket, "Bucket next pointer is damaged") else: assert_(False, "Incorrect child type") def _p_resolveConflict(self, old, com, new): s_old = _get_simple_btree_bucket_state(old) s_com = _get_simple_btree_bucket_state(com) s_new = _get_simple_btree_bucket_state(new) return (( self._bucket_type()._p_resolveConflict(s_old, s_com, s_new), ), ) def __repr__(self): r = super(_Tree, self).__repr__() r = r.replace('Py', '') return r def _get_simple_btree_bucket_state(state): if state is None: return state if not isinstance(state, tuple): raise TypeError("_p_resolveConflict: expected tuple or None for state") if len(state) == 2: # non-degenerate BTree, can't resolve raise BTreesConflictError(-1, -1, -1, 11) # Peel away wrapper to get to only-bucket state. if len(state) != 1: raise TypeError("_p_resolveConflict: expected 1- or 2-tuple for state") state = state[0] if not isinstance(state, tuple) or len(state) != 1: raise TypeError("_p_resolveConflict: expected 1-tuple containing " "bucket state") state = state[0] if not isinstance(state, tuple): raise TypeError("_p_resolveConflict: expected tuple for bucket state") return state class _TreeItems(object): __slots__ = ('firstbucket', 'itertype', 'iterargs', 'index', 'it', 'v', '_len', ) def __init__(self, firstbucket, itertype, iterargs): self.firstbucket = firstbucket self.itertype = itertype self.iterargs = iterargs self.index = -1 self.it = iter(self) self.v = None self._len = None def __getitem__(self, i): if isinstance(i, slice): return list(self)[i] if i < 0: i = len(self) + i if i < 0: raise IndexError(i) if i < self.index: self.index = -1 self.it = iter(self) while i > self.index: try: self.v = next(self.it) except StopIteration: raise IndexError(i) else: self.index += 1 return self.v def __len__(self): if self._len is None: i = 0 for _ in self: i += 1 self._len = i return self._len def __iter__(self): bucket = self.firstbucket itertype = self.itertype iterargs = self.iterargs done = 0 # Note that we don't mind if the first bucket yields no # results due to an idiosyncrasy in how range searches are done. while bucket is not None: for k in getattr(bucket, itertype)(*iterargs): yield k done = 0 if done: return bucket = bucket._next done = 1 class _TreeIterator(object): """ Faux implementation for BBB only. """ def __init__(self, items): #pragma: no cover raise TypeError( "TreeIterators are private implementation details " "of the C-based BTrees.\n\n" "Please use 'iter(tree)', rather than instantiating " "one directly." ) class Tree(_Tree): __slots__ = () def get(self, key, default=None): bucket = self._findbucket(key) if bucket: return bucket.get(key, default) return default def __getitem__(self, key): bucket = self._findbucket(key) if bucket: return bucket[key] raise KeyError(key) def values(self, min=_marker, max=_marker, excludemin=False, excludemax=False): return self.keys(min, max, excludemin, excludemax, 'itervalues') def itervalues(self, min=_marker, max=_marker, excludemin=False, excludemax=False): return iter(self.values(min, max, excludemin, excludemax)) def items(self, min=_marker, max=_marker, excludemin=False, excludemax=False): return self.keys(min, max, excludemin, excludemax, 'iteritems') def iteritems(self, min=_marker, max=_marker, excludemin=False, excludemax=False): return iter(self.items(min, max, excludemin, excludemax)) def byValue(self, min): return reversed( sorted((v, k) for (k, v) in self.iteritems() if v >= min)) def insert(self, key, value): return bool(self._set(key, value, True)[0]) class TreeSet(_Tree): __slots__ = () def add(self, key): return self._set(self._to_key(key))[0] insert = add def remove(self, key): self._del(self._to_key(key)) def update(self, items): add = self.add for i in items: add(i) _p_resolveConflict = _Tree._p_resolveConflict class set_operation(object): __slots__ = ('func', 'set_type', ) def __init__(self, func, set_type): self.func = func self.set_type = set_type def __call__(self, *a, **k): return self.func(self.set_type, *a, **k) def difference(set_type, o1, o2): if o1 is None or o2 is None: return o1 i1 = _SetIteration(o1, True, 0) i2 = _SetIteration(o2, False, 0) if i1.useValues: result = o1._mapping_type() def copy(i): result._keys.append(i.key) result._values.append(i.value) else: result = o1._set_type() def copy(i): result._keys.append(i.key) while i1.active and i2.active: cmp_ = cmp(i1.key, i2.key) if cmp_ < 0: copy(i1) i1.advance() elif cmp_ == 0: i1.advance() i2.advance() else: i2.advance() while i1.active: copy(i1) i1.advance() return result def union(set_type, o1, o2): if o1 is None: return o2 if o2 is None: return o1 i1 = _SetIteration(o1, False, 0) i2 = _SetIteration(o2, False, 0) result = o1._set_type() def copy(i): result._keys.append(i.key) while i1.active and i2.active: cmp_ = cmp(i1.key, i2.key) if cmp_ < 0: copy(i1) i1.advance() elif cmp_ == 0: copy(i1) i1.advance() i2.advance() else: copy(i2) i2.advance() while i1.active: copy(i1) i1.advance() while i2.active: copy(i2) i2.advance() return result def intersection(set_type, o1, o2): if o1 is None: return o2 if o2 is None: return o1 i1 = _SetIteration(o1, False, 0) i2 = _SetIteration(o2, False, 0) result = o1._set_type() def copy(i): result._keys.append(i.key) while i1.active and i2.active: cmp_ = cmp(i1.key, i2.key) if cmp_ < 0: i1.advance() elif cmp_ == 0: copy(i1) i1.advance() i2.advance() else: i2.advance() return result def _prepMergeIterators(o1, o2): MERGE_DEFAULT = getattr(o1, 'MERGE_DEFAULT', None) if MERGE_DEFAULT is None: raise TypeError("invalid set operation") i1 = _SetIteration(o1, True, MERGE_DEFAULT) i2 = _SetIteration(o2, True, MERGE_DEFAULT) return i1, i2 def weightedUnion(set_type, o1, o2, w1=1, w2=1): if o1 is None: if o2 is None: return 0, None return w2, o2 if o2 is None: return w1, o1 i1, i2 = _prepMergeIterators(o1, o2) MERGE = getattr(o1, 'MERGE', None) if MERGE is None and i1.useValues and i2.useValues: raise TypeError("invalid set operation") MERGE_WEIGHT = getattr(o1, 'MERGE_WEIGHT') if (not i1.useValues) and i2.useValues: i1, i2 = i2, i1 w1, w2 = w2, w1 _merging = i1.useValues or i2.useValues if _merging: result = o1._mapping_type() def copy(i, w): result._keys.append(i.key) result._values.append(MERGE_WEIGHT(i.value, w)) else: result = o1._set_type() def copy(i, w): result._keys.append(i.key) while i1.active and i2.active: cmp_ = cmp(i1.key, i2.key) if cmp_ < 0: copy(i1, w1) i1.advance() elif cmp_ == 0: result._keys.append(i1.key) if _merging: result._values.append(MERGE(i1.value, w1, i2.value, w2)) i1.advance() i2.advance() else: copy(i2, w2) i2.advance() while i1.active: copy(i1, w1) i1.advance() while i2.active: copy(i2, w2) i2.advance() return 1, result def weightedIntersection(set_type, o1, o2, w1=1, w2=1): if o1 is None: if o2 is None: return 0, None return w2, o2 if o2 is None: return w1, o1 i1, i2 = _prepMergeIterators(o1, o2) MERGE = getattr(o1, 'MERGE', None) if MERGE is None and i1.useValues and i2.useValues: raise TypeError("invalid set operation") if (not i1.useValues) and i2.useValues: i1, i2 = i2, i1 w1, w2 = w2, w1 _merging = i1.useValues or i2.useValues if _merging: result = o1._mapping_type() else: result = o1._set_type() while i1.active and i2.active: cmp_ = cmp(i1.key, i2.key) if cmp_ < 0: i1.advance() elif cmp_ == 0: result._keys.append(i1.key) if _merging: result._values.append(MERGE(i1.value, w1, i2.value, w2)) i1.advance() i2.advance() else: i2.advance() if isinstance(result, (Set, TreeSet)): return w1 + w2, result return 1, result def multiunion(set_type, seqs): # XXX simple/slow implementation. Goal is just to get tests to pass. result = set_type() for s in seqs: try: iter(s) except TypeError: s = set_type((s, )) result.update(s) return result def to_ob(self, v): return v def to_int(self, v): try: # XXX Python 2.6 doesn't truncate, it spews a warning. if not unpack("i", pack("i", v))[0] == v: #pragma: no cover raise TypeError('32-bit integer expected') except (struct_error, OverflowError, #PyPy ): raise TypeError('32-bit integer expected') return int(v) def to_float(self, v): try: pack("f", v) except struct_error: raise TypeError('float expected') return float(v) def to_long(self, v): try: # XXX Python 2.6 doesn't truncate, it spews a warning. if not unpack("q", pack("q", v))[0] == v: #pragma: no cover if isinstance(v, int_types): raise ValueError("Value out of range", v) raise TypeError('64-bit integer expected') except (struct_error, OverflowError, #PyPy ): if isinstance(v, int_types): raise ValueError("Value out of range", v) raise TypeError('64-bit integer expected') return int(v) def to_bytes(l): def to(self, v): if not (isinstance(v, bytes) and len(v) == l): raise TypeError("%s-byte array expected" % l) return v return to tos = dict(I=to_int, L=to_long, F=to_float, O=to_ob) MERGE_DEFAULT_int = 1 MERGE_DEFAULT_float = 1.0 def MERGE(self, value1, weight1, value2, weight2): return (value1 * weight1) + (value2 * weight2) def MERGE_WEIGHT_default(self, value, weight): return value def MERGE_WEIGHT_numeric(self, value, weight): return value * weight def _fix_pickle(mod_dict, mod_name): # Make the pure-Python objects pickle with the same # class names and types as the C extensions by setting the appropriate # _BTree_reduce_as attribute. # If the C extensions are not available, we also change the # __name__ attribute of the type to match the C name (otherwise # we wind up with *Py in the pickles) # Each module must call this as `_fix_pickle(globals(), __name__)` # at the bottom. mod_prefix = mod_name.split('.')[-1][:2] # BTrees.OOBTree -> 'OO' bucket_name = mod_prefix + 'Bucket' py_bucket_name = bucket_name + 'Py' have_c_extensions = mod_dict[bucket_name] is not mod_dict[py_bucket_name] for name in 'Bucket', 'Set', 'BTree', 'TreeSet', 'TreeIterator': raw_name = mod_prefix + name py_name = raw_name + 'Py' try: py_type = mod_dict[py_name] except KeyError: if name == 'TreeIterator': # Optional continue raise # pragma: no cover raw_type = mod_dict[raw_name] # Could be C or Python py_type._BTree_reduce_as = raw_type py_type._BTree_reduce_up_bound = py_type if not have_c_extensions: # pragma: no cover # Set FooPy to have the __name__ of simply Foo. # We can't do this if the C extension is available, # because then mod_dict[FooPy.__name__] is not FooPy # and pickle refuses to save something like that. # On the other hand (no C extension) this makes our # Python pickle match the C version by default py_type.__name__ = raw_name py_type.__qualname__ = raw_name # Py 3.3+ BTrees-4.3.1/BTrees/SetTemplate.c0000664000175000017500000002360712510775554017653 0ustar tseavertseaver00000000000000/***************************************************************************** Copyright (c) 2001, 2002 Zope Foundation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #include "_compat.h" #define SETTEMPLATE_C "$Id$\n" static PyObject * Set_insert(Bucket *self, PyObject *args) { PyObject *key; int i; UNLESS (PyArg_ParseTuple(args, "O", &key)) return NULL; if ( (i=_bucket_set(self, key, Py_None, 1, 1, 0)) < 0) return NULL; return INT_FROM_LONG(i); } /* _Set_update and _TreeSet_update are identical except for the function they call to add the element to the set. */ static int _Set_update(Bucket *self, PyObject *seq) { int n=0, ind=0; PyObject *iter, *v; iter = PyObject_GetIter(seq); if (iter == NULL) return -1; while (1) { v = PyIter_Next(iter); if (v == NULL) { if (PyErr_Occurred()) goto err; else break; } ind = _bucket_set(self, v, Py_None, 1, 1, 0); Py_DECREF(v); if (ind < 0) goto err; else n += ind; } err: Py_DECREF(iter); if (ind < 0) return -1; return n; } static PyObject * Set_update(Bucket *self, PyObject *args) { PyObject *seq = NULL; int n = 0; if (!PyArg_ParseTuple(args, "|O:update", &seq)) return NULL; if (seq) { n = _Set_update(self, seq); if (n < 0) return NULL; } return INT_FROM_LONG(n); } static PyObject * Set_remove(Bucket *self, PyObject *args) { PyObject *key; UNLESS (PyArg_ParseTuple(args, "O", &key)) return NULL; if (_bucket_set(self, key, NULL, 0, 1, 0) < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static int _set_setstate(Bucket *self, PyObject *args) { PyObject *k, *items; Bucket *next=0; int i, l, copied=1; KEY_TYPE *keys; UNLESS (PyArg_ParseTuple(args, "O|O", &items, &next)) return -1; if (!PyTuple_Check(items)) { PyErr_SetString(PyExc_TypeError, "tuple required for first state element"); return -1; } if ((l=PyTuple_Size(items)) < 0) return -1; for (i=self->len; --i >= 0; ) { DECREF_KEY(self->keys[i]); } self->len=0; if (self->next) { Py_DECREF(self->next); self->next=0; } if (l > self->size) { UNLESS (keys=BTree_Realloc(self->keys, sizeof(KEY_TYPE)*l)) return -1; self->keys=keys; self->size=l; } for (i=0; ikeys[i], k, copied); UNLESS (copied) return -1; INCREF_KEY(self->keys[i]); } self->len=l; if (next) { self->next=next; Py_INCREF(next); } return 0; } static PyObject * set_setstate(Bucket *self, PyObject *args) { int r; UNLESS (PyArg_ParseTuple(args, "O", &args)) return NULL; PER_PREVENT_DEACTIVATION(self); r=_set_setstate(self, args); PER_UNUSE(self); if (r < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static struct PyMethodDef Set_methods[] = { {"__getstate__", (PyCFunction) bucket_getstate, METH_VARARGS, "__getstate__() -- Return the picklable state of the object"}, {"__setstate__", (PyCFunction) set_setstate, METH_VARARGS, "__setstate__() -- Set the state of the object"}, {"keys", (PyCFunction) bucket_keys, METH_VARARGS | METH_KEYWORDS, "keys() -- Return the keys"}, {"has_key", (PyCFunction) bucket_has_key, METH_O, "has_key(key) -- Test whether the bucket contains the given key"}, {"clear", (PyCFunction) bucket_clear, METH_VARARGS, "clear() -- Remove all of the items from the bucket"}, {"maxKey", (PyCFunction) Bucket_maxKey, METH_VARARGS, "maxKey([key]) -- Find the maximum key\n\n" "If an argument is given, find the maximum <= the argument"}, {"minKey", (PyCFunction) Bucket_minKey, METH_VARARGS, "minKey([key]) -- Find the minimum key\n\n" "If an argument is given, find the minimum >= the argument"}, #ifdef PERSISTENT {"_p_resolveConflict", (PyCFunction) bucket__p_resolveConflict, METH_VARARGS, "_p_resolveConflict() -- Reinitialize from a newly created copy"}, {"_p_deactivate", (PyCFunction) bucket__p_deactivate, METH_VARARGS | METH_KEYWORDS, "_p_deactivate() -- Reinitialize from a newly created copy"}, #endif {"add", (PyCFunction)Set_insert, METH_VARARGS, "add(id) -- Add a key to the set"}, {"insert", (PyCFunction)Set_insert, METH_VARARGS, "insert(id) -- Add a key to the set"}, {"update", (PyCFunction)Set_update, METH_VARARGS, "update(seq) -- Add the items from the given sequence to the set"}, {"remove", (PyCFunction)Set_remove, METH_VARARGS, "remove(id) -- Remove an id from the set"}, {NULL, NULL} /* sentinel */ }; static int Set_init(PyObject *self, PyObject *args, PyObject *kwds) { PyObject *v = NULL; if (!PyArg_ParseTuple(args, "|O:" MOD_NAME_PREFIX "Set", &v)) return -1; if (v) return _Set_update((Bucket *)self, v); else return 0; } static PyObject * set_repr(Bucket *self) { static PyObject *format; PyObject *r, *t; if (!format) format = TEXT_FROM_STRING(MOD_NAME_PREFIX "Set(%s)"); UNLESS (t = PyTuple_New(1)) return NULL; UNLESS (r = bucket_keys(self, NULL, NULL)) goto err; PyTuple_SET_ITEM(t, 0, r); r = t; ASSIGN(r, TEXT_FORMAT(format, r)); return r; err: Py_DECREF(t); return NULL; } static Py_ssize_t set_length(Bucket *self) { int r; PER_USE_OR_RETURN(self, -1); r = self->len; PER_UNUSE(self); return r; } static PyObject * set_item(Bucket *self, Py_ssize_t index) { PyObject *r=0; PER_USE_OR_RETURN(self, NULL); if (index >= 0 && index < self->len) { COPY_KEY_TO_OBJECT(r, self->keys[index]); } else IndexError(index); PER_UNUSE(self); return r; } static PySequenceMethods set_as_sequence = { (lenfunc)set_length, /* sq_length */ (binaryfunc)0, /* sq_concat */ (ssizeargfunc)0, /* sq_repeat */ (ssizeargfunc)set_item, /* sq_item */ (ssizessizeargfunc)0, /* sq_slice */ (ssizeobjargproc)0, /* sq_ass_item */ (ssizessizeobjargproc)0, /* sq_ass_slice */ (objobjproc)bucket_contains, /* sq_contains */ 0, /* sq_inplace_concat */ 0, /* sq_inplace_repeat */ }; static PyTypeObject SetType = { PyVarObject_HEAD_INIT(NULL, 0) /* PyPersist_Type */ MODULE_NAME MOD_NAME_PREFIX "Set", /* tp_name */ sizeof(Bucket), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)bucket_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ (reprfunc)set_repr, /* tp_repr */ 0, /* tp_as_number */ &set_as_sequence, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ (traverseproc)bucket_traverse, /* tp_traverse */ (inquiry)bucket_tp_clear, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ (getiterfunc)Bucket_getiter, /* tp_iter */ 0, /* tp_iternext */ Set_methods, /* tp_methods */ Bucket_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ Set_init, /* tp_init */ 0, /* tp_alloc */ 0, /*PyType_GenericNew,*/ /* tp_new */ }; static int nextSet(SetIteration *i) { if (i->position >= 0) { UNLESS(PER_USE(BUCKET(i->set))) return -1; if (i->position) { DECREF_KEY(i->key); } if (i->position < BUCKET(i->set)->len) { COPY_KEY(i->key, BUCKET(i->set)->keys[i->position]); INCREF_KEY(i->key); i->position ++; } else { i->position = -1; PER_ACCESSED(BUCKET(i->set)); } PER_ALLOW_DEACTIVATION(BUCKET(i->set)); } return 0; } BTrees-4.3.1/BTrees/_fsBTree.c0000664000175000017500000001130512510775554017105 0ustar tseavertseaver00000000000000/*############################################################################ # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id$\n" /* fsBTree - FileStorage index BTree This BTree implements a mapping from 2-character strings to six-character strings. This allows us to efficiently store a FileStorage index as a nested mapping of 6-character oid prefix to mapping of 2-character oid suffix to 6-character (byte) file positions. */ typedef unsigned char char2[2]; typedef unsigned char char6[6]; /* Setup template macros */ #define PERSISTENT #define MOD_NAME_PREFIX "fs" #define DEFAULT_MAX_BUCKET_SIZE 500 #define DEFAULT_MAX_BTREE_SIZE 500 #include "_compat.h" /*#include "intkeymacros.h"*/ #define KEYMACROS_H "$Id$\n" #define KEY_TYPE char2 #undef KEY_TYPE_IS_PYOBJECT #define KEY_CHECK(K) (PyBytes_Check(K) && PyBytes_GET_SIZE(K)==2) #define TEST_KEY_SET_OR(V, K, T) if ( ( (V) = ((*(K) < *(T) || (*(K) == *(T) && (K)[1] < (T)[1])) ? -1 : ((*(K) == *(T) && (K)[1] == (T)[1]) ? 0 : 1)) ), 0 ) #define DECREF_KEY(KEY) #define INCREF_KEY(k) #define COPY_KEY(KEY, E) (*(KEY)=*(E), (KEY)[1]=(E)[1]) #define COPY_KEY_TO_OBJECT(O, K) O=PyBytes_FromStringAndSize((const char*)K,2) #define COPY_KEY_FROM_ARG(TARGET, ARG, STATUS) \ if (KEY_CHECK(ARG)) memcpy(TARGET, PyBytes_AS_STRING(ARG), 2); else { \ PyErr_SetString(PyExc_TypeError, "expected two-character string key"); \ (STATUS)=0; } /*#include "intvaluemacros.h"*/ #define VALUEMACROS_H "$Id$\n" #define VALUE_TYPE char6 #undef VALUE_TYPE_IS_PYOBJECT #define TEST_VALUE(K, T) memcmp(K,T,6) #define DECREF_VALUE(k) #define INCREF_VALUE(k) #define COPY_VALUE(V, E) (memcpy(V, E, 6)) #define COPY_VALUE_TO_OBJECT(O, K) O=PyBytes_FromStringAndSize((const char*)K,6) #define COPY_VALUE_FROM_ARG(TARGET, ARG, STATUS) \ if ((PyBytes_Check(ARG) && PyBytes_GET_SIZE(ARG)==6)) \ memcpy(TARGET, PyBytes_AS_STRING(ARG), 6); else { \ PyErr_SetString(PyExc_TypeError, "expected six-character string key"); \ (STATUS)=0; } #define NORMALIZE_VALUE(V, MIN) #include "Python.h" static PyObject *bucket_toBytes(PyObject *self); static PyObject *bucket_fromBytes(PyObject *self, PyObject *state); #define EXTRA_BUCKET_METHODS \ {"toBytes", (PyCFunction) bucket_toBytes, METH_NOARGS, \ "toBytes() -- Return the state as a bytes array"}, \ {"fromBytes", (PyCFunction) bucket_fromBytes, METH_O, \ "fromSBytes(s) -- Set the state of the object from a bytes array"}, \ {"toString", (PyCFunction) bucket_toBytes, METH_NOARGS, \ "toString() -- Deprecated alias for 'toBytes'"}, \ {"fromString", (PyCFunction) bucket_fromBytes, METH_O, \ "fromString(s) -- Deprecated alias for 'fromBytes'"}, \ #ifdef PY3K #define INITMODULE PyInit__fsBTree #else #define INITMODULE init_fsBTree #endif #include "BTreeModuleTemplate.c" static PyObject * bucket_toBytes(PyObject *oself) { Bucket *self = (Bucket *)oself; PyObject *items = NULL; int len; PER_USE_OR_RETURN(self, NULL); len = self->len; items = PyBytes_FromStringAndSize(NULL, len*8); if (items == NULL) goto err; memcpy(PyBytes_AS_STRING(items), self->keys, len*2); memcpy(PyBytes_AS_STRING(items)+len*2, self->values, len*6); PER_UNUSE(self); return items; err: PER_UNUSE(self); Py_XDECREF(items); return NULL; } static PyObject * bucket_fromBytes(PyObject *oself, PyObject *state) { Bucket *self = (Bucket *)oself; int len; KEY_TYPE *keys; VALUE_TYPE *values; len = PyBytes_Size(state); if (len < 0) return NULL; if (len%8) { PyErr_SetString(PyExc_ValueError, "state string of wrong size"); return NULL; } len /= 8; if (self->next) { Py_DECREF(self->next); self->next = NULL; } if (len > self->size) { keys = BTree_Realloc(self->keys, sizeof(KEY_TYPE)*len); if (keys == NULL) return NULL; values = BTree_Realloc(self->values, sizeof(VALUE_TYPE)*len); if (values == NULL) return NULL; self->keys = keys; self->values = values; self->size = len; } memcpy(self->keys, PyBytes_AS_STRING(state), len*2); memcpy(self->values, PyBytes_AS_STRING(state)+len*2, len*6); self->len = len; Py_INCREF(self); return (PyObject *)self; } BTrees-4.3.1/BTrees/intkeymacros.h0000664000175000017500000000301312704024032020105 0ustar tseavertseaver00000000000000 #define KEYMACROS_H "$Id$\n" #ifdef ZODB_64BIT_INTS /* PY_LONG_LONG as key */ #define NEED_LONG_LONG_SUPPORT #define NEED_LONG_LONG_KEYS #define KEY_TYPE PY_LONG_LONG #define KEY_CHECK longlong_check #define COPY_KEY_TO_OBJECT(O, K) O=longlong_as_object(K) #define COPY_KEY_FROM_ARG(TARGET, ARG, STATUS) \ if (!longlong_convert((ARG), &TARGET)) \ { \ (STATUS)=0; (TARGET)=0; \ } #else /* C int as key */ #define KEY_TYPE int #define KEY_CHECK INT_CHECK #define COPY_KEY_TO_OBJECT(O, K) O=INT_FROM_LONG(K) #define COPY_KEY_FROM_ARG(TARGET, ARG, STATUS) \ if (INT_CHECK(ARG)) { \ long vcopy = INT_AS_LONG(ARG); \ if (PyErr_Occurred()) { (STATUS)=0; (TARGET)=0; } \ else if ((int)vcopy != vcopy) { \ PyErr_SetString(PyExc_TypeError, "integer out of range"); \ (STATUS)=0; (TARGET)=0; \ } \ else TARGET = vcopy; \ } else { \ PyErr_SetString(PyExc_TypeError, "expected integer key"); \ (STATUS)=0; (TARGET)=0; } #endif #undef KEY_TYPE_IS_PYOBJECT #define TEST_KEY_SET_OR(V, K, T) if ( ( (V) = (((K) < (T)) ? -1 : (((K) > (T)) ? 1: 0)) ) , 0 ) #define DECREF_KEY(KEY) #define INCREF_KEY(k) #define COPY_KEY(KEY, E) (KEY=(E)) #define MULTI_INT_UNION 1 BTrees-4.3.1/BTrees/_LLBTree.c0000664000175000017500000000225712510775554017012 0ustar tseavertseaver00000000000000/*############################################################################ # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id: _IIBTree.c 25186 2004-06-02 15:07:33Z jim $\n" /* IIBTree - int key, int value BTree Implements a collection using int type keys and int type values */ /* Setup template macros */ #define PERSISTENT #define MOD_NAME_PREFIX "LL" #define DEFAULT_MAX_BUCKET_SIZE 120 #define DEFAULT_MAX_BTREE_SIZE 500 #define ZODB_64BIT_INTS #include "_compat.h" #include "intkeymacros.h" #include "intvaluemacros.h" #ifdef PY3K #define INITMODULE PyInit__LLBTree #else #define INITMODULE init_LLBTree #endif #include "BTreeModuleTemplate.c" BTrees-4.3.1/BTrees/LFBTree.py0000664000175000017500000001030612665316015017037 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __all__ = ('Bucket', 'Set', 'BTree', 'TreeSet', 'LFBucket', 'LFSet', 'LFBTree', 'LFTreeSet', 'union', 'intersection', 'difference', 'weightedUnion', 'weightedIntersection', 'multiunion', ) from zope.interface import moduleProvides from .Interfaces import IIntegerFloatBTreeModule from ._base import Bucket from ._base import MERGE from ._base import MERGE_WEIGHT_numeric from ._base import MERGE_DEFAULT_float from ._base import Set from ._base import Tree as BTree from ._base import TreeSet from ._base import _TreeIterator from ._base import difference as _difference from ._base import intersection as _intersection from ._base import multiunion as _multiunion from ._base import set_operation as _set_operation from ._base import to_long as _to_key from ._base import to_float as _to_value from ._base import union as _union from ._base import weightedIntersection as _weightedIntersection from ._base import weightedUnion as _weightedUnion from ._base import _fix_pickle _BUCKET_SIZE = 120 _TREE_SIZE = 500 using64bits = True class LFBucketPy(Bucket): _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class LFSetPy(Set): _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class LFBTreePy(BTree): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class LFTreeSetPy(TreeSet): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class LFTreeIteratorPy(_TreeIterator): pass # Can't declare forward refs, so fix up afterwards: LFBucketPy._mapping_type = LFBucketPy._bucket_type = LFBucketPy LFBucketPy._set_type = LFSetPy LFSetPy._mapping_type = LFBucketPy LFSetPy._set_type = LFSetPy._bucket_type = LFSetPy LFBTreePy._mapping_type = LFBTreePy._bucket_type = LFBucketPy LFBTreePy._set_type = LFSetPy LFTreeSetPy._mapping_type = LFBucketPy LFTreeSetPy._set_type = LFTreeSetPy._bucket_type = LFSetPy differencePy = _set_operation(_difference, LFSetPy) unionPy = _set_operation(_union, LFSetPy) intersectionPy = _set_operation(_intersection, LFSetPy) multiunionPy = _set_operation(_multiunion, LFSetPy) weightedUnionPy = _set_operation(_weightedUnion, LFSetPy) weightedIntersectionPy = _set_operation(_weightedIntersection, LFSetPy) try: from ._LFBTree import LFBucket except ImportError: #pragma NO COVER w/ C extensions LFBucket = LFBucketPy LFSet = LFSetPy LFBTree = LFBTreePy LFTreeSet = LFTreeSetPy LFTreeIterator = LFTreeIteratorPy difference = differencePy union = unionPy intersection = intersectionPy multiunion = multiunionPy weightedUnion = weightedUnionPy weightedIntersection = weightedIntersectionPy else: #pragma NO COVER w/o C extensions from ._LFBTree import LFSet from ._LFBTree import LFBTree from ._LFBTree import LFTreeSet from ._LFBTree import LFTreeIterator from ._LFBTree import difference from ._LFBTree import union from ._LFBTree import intersection from ._LFBTree import multiunion from ._LFBTree import weightedUnion from ._LFBTree import weightedIntersection Bucket = LFBucket Set = LFSet BTree = LFBTree TreeSet = LFTreeSet _fix_pickle(globals(), __name__) moduleProvides(IIntegerFloatBTreeModule) BTrees-4.3.1/BTrees/IFBTree.py0000664000175000017500000001030512665316015017033 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __all__ = ('Bucket', 'Set', 'BTree', 'TreeSet', 'IFBucket', 'IFSet', 'IFBTree', 'IFTreeSet', 'union', 'intersection', 'difference', 'weightedUnion', 'weightedIntersection', 'multiunion', ) from zope.interface import moduleProvides from .Interfaces import IIntegerFloatBTreeModule from ._base import Bucket from ._base import MERGE from ._base import MERGE_WEIGHT_numeric from ._base import MERGE_DEFAULT_float from ._base import Set from ._base import Tree as BTree from ._base import TreeSet from ._base import _TreeIterator from ._base import difference as _difference from ._base import intersection as _intersection from ._base import multiunion as _multiunion from ._base import set_operation as _set_operation from ._base import to_int as _to_key from ._base import to_float as _to_value from ._base import union as _union from ._base import weightedIntersection as _weightedIntersection from ._base import weightedUnion as _weightedUnion from ._base import _fix_pickle _BUCKET_SIZE = 120 _TREE_SIZE = 500 using64bits = False class IFBucketPy(Bucket): _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class IFSetPy(Set): _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class IFBTreePy(BTree): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class IFTreeSetPy(TreeSet): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class IFTreeIteratorPy(_TreeIterator): pass # Can't declare forward refs, so fix up afterwards: IFBucketPy._mapping_type = IFBucketPy._bucket_type = IFBucketPy IFBucketPy._set_type = IFSetPy IFSetPy._mapping_type = IFBucketPy IFSetPy._set_type = IFSetPy._bucket_type = IFSetPy IFBTreePy._mapping_type = IFBTreePy._bucket_type = IFBucketPy IFBTreePy._set_type = IFSetPy IFTreeSetPy._mapping_type = IFBucketPy IFTreeSetPy._set_type = IFTreeSetPy._bucket_type = IFSetPy differencePy = _set_operation(_difference, IFSetPy) unionPy = _set_operation(_union, IFSetPy) intersectionPy = _set_operation(_intersection, IFSetPy) multiunionPy = _set_operation(_multiunion, IFSetPy) weightedUnionPy = _set_operation(_weightedUnion, IFSetPy) weightedIntersectionPy = _set_operation(_weightedIntersection, IFSetPy) try: from ._IFBTree import IFBucket except ImportError: #pragma NO COVER w/ C extensions IFBucket = IFBucketPy IFSet = IFSetPy IFBTree = IFBTreePy IFTreeSet = IFTreeSetPy IFTreeIterator = IFTreeIteratorPy difference = differencePy union = unionPy intersection = intersectionPy multiunion = multiunionPy weightedUnion = weightedUnionPy weightedIntersection = weightedIntersectionPy else: #pragma NO COVER w/o C extensions from ._IFBTree import IFSet from ._IFBTree import IFBTree from ._IFBTree import IFTreeSet from ._IFBTree import IFTreeIterator from ._IFBTree import difference from ._IFBTree import union from ._IFBTree import intersection from ._IFBTree import multiunion from ._IFBTree import weightedUnion from ._IFBTree import weightedIntersection Bucket = IFBucket Set = IFSet BTree = IFBTree TreeSet = IFTreeSet _fix_pickle(globals(), __name__) moduleProvides(IIntegerFloatBTreeModule) BTrees-4.3.1/BTrees/Development.txt0000664000175000017500000004074312510775554020303 0ustar tseavertseaver00000000000000===================== Developer Information ===================== This document provides information for developers who maintain or extend `BTrees`. Macros ====== `BTrees` are defined using a "template", roughly akin to a C++ template. To create a new family of `BTrees`, create a source file that defines macros used to handle differences in key and value types: Configuration Macros -------------------- ``MASTER_ID`` A string to hold an RCS/CVS Id key to be included in compiled binaries. ``MOD_NAME_PREFIX`` A string (like "IO" or "OO") that provides the prefix used for the module. This gets used to generate type names and the internal module name string. ``DEFAULT_MAX_BUCKET_SIZE`` An int giving the maximum bucket size (number of key/value pairs). When a bucket gets larger than this due to an insertion *into a BTREE*, it splits. Inserting into a bucket directly doesn't split, and functions that produce a bucket output (e.g., ``union()``) also have no bound on how large a bucket may get. Someday this will be tunable on `BTree`. instances. ``DEFAULT_MAX_BTREE_SIZE`` An ``int`` giving the maximum size (number of children) of an internal btree node. Someday this will be tunable on ``BTree`` instances. Macros for Keys --------------- ``KEY_TYPE`` The C type declaration for keys (e.g., ``int`` or ``PyObject*``). ``KEY_TYPE_IS_PYOBJECT`` Define if ``KEY_TYPE`` is a ``PyObject*`, else ``undef``. ``KEY_CHECK(K)`` Tests whether the ``PyObject* K`` can be converted to the (``C``) key type (``KEY_TYPE``). The macro should return a boolean (zero for false, non-zero for true). When it returns false, its caller should probably set a ``TypeError`` exception. ``TEST_KEY_SET_OR(V, K, T)`` Like Python's ``cmp()``. Compares K(ey) to T(arget), where ``K`` and ``T`` are ``C`` values of type `KEY_TYPE`. ``V`` is assigned an `int` value depending on the outcome:: < 0 if K < T == 0 if K == T > 0 if K > T This macro acts like an ``if``, where the following statement is executed only if a Python exception has been raised because the values could not be compared. ``DECREF_KEY(K)`` ``K`` is a value of ``KEY_TYPE``. If ``KEY_TYPE`` is a flavor of ``PyObject*``, write this to do ``Py_DECREF(K)``. Else (e.g., ``KEY_TYPE`` is ``int``) make it a nop. ``INCREF_KEY(K)`` ``K`` is a value of `KEY_TYPE`. If `KEY_TYPE` is a flavor of ``PyObject*``, write this to do ``Py_INCREF(K)``. Else (e.g., `KEY_TYPE` is ``int``) make it a nop. ``COPY_KEY(K, E)`` Like ``K=E``. Copy a key from ``E`` to ``K``, both of ``KEY_TYPE``. Note that this doesn't ``decref K`` or ``incref E`` when ``KEY_TYPE`` is a ``PyObject*``; the caller is responsible for keeping refcounts straight. ``COPY_KEY_TO_OBJECT(O, K)`` Roughly like ``O=K``. ``O`` is a ``PyObject*``, and the macro must build a Python object form of ``K``, assign it to ``O``, and ensure that ``O`` owns the reference to its new value. It may do this by creating a new Python object based on ``K`` (e.g., ``PyInt_FromLong(K)`` when ``KEY_TYPE`` is ``int``), or simply by doing ``Py_INCREF(K)`` if ``KEY_TYPE`` is a ``PyObject*``. ``COPY_KEY_FROM_ARG(TARGET, ARG, STATUS)`` Copy an argument to the target without creating a new reference to ``ARG``. ``ARG`` is a ``PyObject*``, and ``TARGET`` is of type ``KEY_TYPE``. If this can't be done (for example, ``KEY_CHECK(ARG)`` returns false), set a Python error and set status to ``0``. If there is no error, leave status alone. Macros for Values ----------------- ``VALUE_TYPE`` The C type declaration for values (e.g., ``int`` or ``PyObject*``). ``VALUE_TYPE_IS_PYOBJECT`` Define if ``VALUE_TYPE`` is a ``PyObject*``, else ``undef``. ``TEST_VALUE(X, Y)`` Like Python's ``cmp()``. Compares ``X`` to ``Y``, where ``X`` & ``Y`` are ``C`` values of type ``VALUE_TYPE``. The macro returns an ``int``, with value:: < 0 if X < Y == 0 if X == Y > 0 if X > Y Bug: There is no provision for determining whether the comparison attempt failed (set a Python exception). ``DECREF_VALUE(K)`` Like ``DECREF_KEY``, except applied to values of ``VALUE_TYPE``. ``INCREF_VALUE(K)`` Like ``INCREF_KEY``, except applied to values of ``VALUE_TYPE``. ``COPY_VALUE(K, E)`` Like ``COPY_KEY``, except applied to values of ``VALUE_TYPE``. ``COPY_VALUE_TO_OBJECT(O, K)`` Like ``COPY_KEY_TO_OBJECT``, except applied to values of ``VALUE_TYPE``. ``COPY_VALUE_FROM_ARG(TARGET, ARG, STATUS)`` Like ``COPY_KEY_FROM_ARG``, except applied to values of ``VALUE_TYPE``. ``NORMALIZE_VALUE(V, MIN)`` Normalize the value, ``V``, using the parameter ``MIN``. This is almost certainly a YAGNI. It is a no-op for most types. For integers, ``V`` is replaced by ``V/MIN`` only if ``MIN > 0``. Macros for Set Operations ------------------------- ``MERGE_DEFAULT`` A value of ``VALUE_TYPE`` specifying the value to associate with set elements when sets are merged with mappings via weighed union or weighted intersection. ``MERGE(O1, w1, O2, w2)`` Performs a weighted merge of two values, ``O1`` and ``O2``, using weights ``w1`` and ``w2``. The result must be of ``VALUE_TYPE``. Note that weighted unions and weighted intersections are not enabled if this macro is left undefined. ``MERGE_WEIGHT(O, w)`` Computes a weighted value for ``O``. The result must be of ``VALUE_TYPE``. This is used for "filling out" weighted unions, i.e. to compute a weighted value for keys that appear in only one of the input mappings. If left undefined, ``MERGE_WEIGHT`` defaults to:: #define MERGE_WEIGHT(O, w) (O) ``MULTI_INT_UNION`` The value doesn't matter. If defined, `SetOpTemplate.c` compiles code for a ``multiunion()`` function (compute a union of many input sets at high speed). This currently makes sense only for structures with integer keys. BTree Clues =========== More or less random bits of helpful info. + In papers and textbooks, this flavor of BTree is usually called a B+-Tree, where "+" is a superscript. + All keys and all values live in the bucket leaf nodes. Keys in interior (BTree) nodes merely serve to guide a search efficiently toward the correct leaf. + When a key is deleted, it's physically removed from the bucket it's in, but this doesn't propagate back up the tree: since keys in interior nodes only serve to guide searches, it's OK-- and saves time --to leave "stale" keys in interior nodes. + No attempt is made to rebalance the tree after a deletion, unless a bucket thereby becomes entirely empty. "Classic BTrees" do rebalance, keeping all buckets at least half full (provided there are enough keys in the entire tree to fill half a bucket). The tradeoffs are murky. Pathological cases in the presence of deletion do exist. Pathologies include trees tending toward only one key per bucket, and buckets at differing depths (all buckets are at the same depth in a classic BTree). + ``DEFAULT_MAX_BUCKET_SIZE`` and ``DEFAULT_MAX_BTREE_SIZE`` are chosen mostly to "even out" pickle sizes in storage. That's why, e.g., an `IIBTree` has larger values than an `OOBTree`: pickles store ints more efficiently than they can store arbitrary Python objects. + In a non-empty BTree, every bucket node contains at least one key, and every BTree node contains at least one child and a non-NULL firstbucket pointer. However, a BTree node may not contain any keys. + An empty BTree consists solely of a BTree node with ``len==0`` and ``firstbucket==NULL``. + Although a BTree can become unbalanced under a mix of inserts and deletes (meaning both that there's nothing stronger that can be said about buckets than that they're not empty, and that buckets can appear at different depths), a BTree node always has children of the same kind: they're all buckets, or they're all BTree nodes. The ``BTREE_SEARCH`` Macro ========================== For notational ease, consider a fixed BTree node ``x``, and let :: K(i) mean x->data.key[i] C(i) mean all the keys reachable from x->data.child[i] For each ``i`` in ``0`` to ``x->len-1`` inclusive, :: K(i) <= C(i) < K(i+1) is a BTree node invariant, where we pretend that ``K(0)`` holds a key smaller than any possible key, and ``K(x->len)`` holds a key larger than any possible key. (Note that ``K(x->len)`` doesn't actually exist, and ``K(0)`` is never used although space for it exists in non-empty BTree nodes.) When searching for a key ``k``, then, the child pointer we want to follow is the one at index ``i`` such that ``K(i) <= k < K(i+1)``. There can be at most one such ``i``, since the ``K(i)`` are strictly increasing. And there is at least one such ``i`` provided the tree isn't empty (so that ``0 < len``). For the moment, assume the tree isn't empty (we'll get back to that later). The macro's chief loop invariant is :: K(lo) < k < K(hi) This holds trivially at the start, since ``lo`` is set to ``0``, and ``hi`` to ``x->len``, and we pretend ``K(0)`` is minus infinity and ``K(len)`` is plus infinity. Inside the loop, if ``K(i) < k`` we set ``lo`` to ``i``, and if ``K(i) > k`` we set ``hi`` to ``i``. These obviously preserve the invariant. If ``K(i) == k``, the loop breaks and sets the result to ``i``, and since ``K(i) == k`` in that case ``i`` is obviously the correct result. Other cases depend on how ``i = floor((lo + hi)/2)`` works, exactly. Suppose ``lo + d = hi`` for some ``d >= 0``. Then ``i = floor((lo + lo + d)/2) = floor(lo + d/2) = lo + floor(d/2)``. So: a. ``[d == 0] (lo == i == hi)`` if and only if ``(lo == hi)``. b. ``[d == 1] (lo == i < hi)`` if and only if ``(lo+1 == hi)``. c. ``[d > 1] (lo < i < hi)`` if and only if ``(lo+1 < hi)``. If the node is empty ``(x->len == 0)``, then ``lo==i==hi==0`` at the start, and the loop exits immediately (the first ``i > lo`` test fails), without entering the body. Else ``lo < hi`` at the start, and the invariant ``K(lo) < k < K(hi)`` holds. If ``lo+1 < hi``, we're in case (c): ``i`` is strictly between ``lo`` and ``hi``, so the loop body is entered, and regardless of whether the body sets the new ``lo`` or the new ``hi`` to ``i``, the new ``lo`` is strictly less than the new ``hi``, and the difference between the new ``lo`` and new ``hi`` is strictly less than the difference between the old ``lo`` and old ``hi``. So long as the new ``lo + 1`` remains < the new ``hi``, we stay in this case. We can't stay in this case forever, though: because ``hi-lo`` decreases on each trip but remains > ``0``, ``lo+1 == hi`` must eventually become true. (In fact, it becomes true quickly, in about ``log2(x->len)`` trips; the point is more that ``lo`` doesn't equal ``hi`` when the loop ends, it has to end with ``lo+1==hi`` and ``i==lo``). Then we're in case (b): ``i==lo==hi-1`` then, and the loop exits. The invariant still holds, with ``lo==i`` and ``hi==lo+1==i+1``:: K(i) < k < K(i+1) so ``i`` is again the correct answer. Optimization points: -------------------- + Division by 2 is done via shift rather via "/2". These are signed ints, and almost all C compilers treat signed int division as truncating, and shifting is not the same as truncation for signed int division. The compiler has no way to know these values aren't negative, so has to generate longer-winded code for "/2". But we know these values aren't negative, and exploit it. + The order of _cmp comparisons matters. We're in an interior BTree node, and are looking at only a tiny fraction of all the keys that exist. So finding the key exactly in this node is unlikely, and checking ``_cmp == 0`` is a waste of time to the same extent. It doesn't matter whether we check for ``_cmp < 0`` or ``_cmp > 0`` first, so long as we do both before worrying about equality. + At the start of a routine, it's better to run this macro even if ``x->len`` is ``0`` (check for that afterwards). We just called a function and so probably drained the pipeline. If the first thing we do then is read up ``self->len`` and check it against ``0``, we just sit there waiting for the data to get read up, and then another immediate test-and-branch, and for a very unlikely case (BTree nodes are rarely empty). It's better to get into the loop right away so the normal case makes progress ASAP. The ``BUCKET_SEARCH`` Macro =========================== This has a different job than ``BTREE_SEARCH``: the key ``0`` slot is legitimate in a bucket, and we want to find the index at which the key belongs. If the key is larger than the bucket's largest key, a new slot at index len is where it belongs, else it belongs at the smallest ``i`` with ``keys[i]`` >= the key we're looking for. We also need to know whether or not the key is present (``BTREE_SEARCH`` didn't care; it only wanted to find the next node to search). The mechanics of the search are quite similar, though. The primary loop invariant changes to (say we're searching for key ``k``):: K(lo-1) < k < K(hi) where ``K(i)`` means ``keys[i]``, and we pretend ``K(-1)`` is minus infinity and ``K(len)`` is plus infinity. If the bucket is empty, ``lo=hi=i=0`` at the start, the loop body is never entered, and the macro sets ``INDEX`` to 0 and ``ABSENT`` to true. That's why ``_cmp`` is initialized to 1 (``_cmp`` becomes ``ABSENT``). Else the bucket is not empty, lok``, ``hi`` is set to ``i``, preserving that ``K[hi] = K[i] > k``. If the loop exits after either of those, ``_cmp != 0``, so ``ABSENT`` becomes true. If ``K[i]=k``, the loop breaks, so that ``INDEX`` becomes ``i``, and ``ABSENT`` becomes false (``_cmp=0`` in this case). The same case analysis for ``BTREE_SEARCH`` on ``lo`` and ``hi`` holds here: a. ``(lo == i == hi)`` if and only if ``(lo == hi)``. b. ``(lo == i < hi)`` if and only if ``(lo+1 == hi)``. c. ``(lo < i < hi)`` if and only if ``(lo+1 < hi)``. So long as ``lo+1 < hi``, we're in case (c), and either break with equality (in which case the right results are obviously computed) or narrow the range. If equality doesn't obtain, the range eventually narrows to cases (a) or (b). To go from (c) to (a), we must have ``lo+2==hi`` at the start, and ``K[i]=K[lo+1] key``), because when it pays it narrows the range more (we get a little boost from setting ``lo=i+1`` in this case; the other case sets ``hi=i``, which isn't as much of a narrowing). BTrees-4.3.1/BTrees/LOBTree.py0000664000175000017500000000656512665316015017064 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __all__ = ('Bucket', 'Set', 'BTree', 'TreeSet', 'LOBucket', 'LOSet', 'LOBTree', 'LOTreeSet', 'union', 'intersection', 'difference', 'multiunion', ) from zope.interface import moduleProvides from .Interfaces import IIntegerObjectBTreeModule from ._base import Bucket from ._base import MERGE_WEIGHT_default from ._base import Set from ._base import Tree as BTree from ._base import TreeSet from ._base import _TreeIterator from ._base import difference as _difference from ._base import intersection as _intersection from ._base import multiunion as _multiunion from ._base import set_operation as _set_operation from ._base import to_long as _to_key from ._base import to_ob as _to_value from ._base import union as _union from ._base import _fix_pickle _BUCKET_SIZE = 60 _TREE_SIZE = 500 using64bits = True class LOBucketPy(Bucket): _to_key = _to_key _to_value = _to_value MERGE_WEIGHT = MERGE_WEIGHT_default class LOSetPy(Set): _to_key = _to_key class LOBTreePy(BTree): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key _to_value = _to_value MERGE_WEIGHT = MERGE_WEIGHT_default class LOTreeSetPy(TreeSet): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key class LOTreeIteratorPy(_TreeIterator): pass # Can't declare forward refs, so fix up afterwards: LOBucketPy._mapping_type = LOBucketPy._bucket_type = LOBucketPy LOBucketPy._set_type = LOSetPy LOSetPy._mapping_type = LOBucketPy LOSetPy._set_type = LOSetPy._bucket_type = LOSetPy LOBTreePy._mapping_type = LOBTreePy._bucket_type = LOBucketPy LOBTreePy._set_type = LOSetPy LOTreeSetPy._mapping_type = LOBucketPy LOTreeSetPy._set_type = LOTreeSetPy._bucket_type = LOSetPy differencePy = _set_operation(_difference, LOSetPy) unionPy = _set_operation(_union, LOSetPy) intersectionPy = _set_operation(_intersection, LOSetPy) multiunionPy = _set_operation(_multiunion, LOSetPy) try: from ._LOBTree import LOBucket except ImportError: #pragma NO COVER w/ C extensions LOBucket = LOBucketPy LOSet = LOSetPy LOBTree = LOBTreePy LOTreeSet = LOTreeSetPy LOTreeIterator = LOTreeIteratorPy difference = differencePy union = unionPy intersection = intersectionPy multiunion = multiunionPy else: #pragma NO COVER w/o C extensions from ._LOBTree import LOSet from ._LOBTree import LOBTree from ._LOBTree import LOTreeSet from ._LOBTree import LOTreeIterator from ._LOBTree import difference from ._LOBTree import union from ._LOBTree import intersection from ._LOBTree import multiunion Bucket = LOBucket Set = LOSet BTree = LOBTree TreeSet = LOTreeSet _fix_pickle(globals(), __name__) moduleProvides(IIntegerObjectBTreeModule) BTrees-4.3.1/BTrees/_OLBTree.c0000664000175000017500000000223212510775554017006 0ustar tseavertseaver00000000000000/*############################################################################ # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id: _OIBTree.c 25186 2004-06-02 15:07:33Z jim $\n" /* OIBTree - object key, int value BTree Implements a collection using object type keys and int type values */ #define PERSISTENT #define MOD_NAME_PREFIX "OL" #define DEFAULT_MAX_BUCKET_SIZE 60 #define DEFAULT_MAX_BTREE_SIZE 250 #define ZODB_64BIT_INTS #include "_compat.h" #include "objectkeymacros.h" #include "intvaluemacros.h" #ifdef PY3K #define INITMODULE PyInit__OLBTree #else #define INITMODULE init_OLBTree #endif #include "BTreeModuleTemplate.c" BTrees-4.3.1/BTrees/_compat.h0000664000175000017500000000231612634016266017040 0ustar tseavertseaver00000000000000/* Straddle Python 2 / 3 */ #ifndef BTREES__COMPAT_H #define BTREES__COMPAT_H #include "Python.h" #ifdef INTERN #undef INTERN #endif #ifdef INT_FROM_LONG #undef INT_FROM_LONG #endif #ifdef INT_CHECK #undef INT_CHECK #endif #if PY_MAJOR_VERSION >= 3 #define PY3K #define INTERN PyUnicode_InternFromString #define INT_FROM_LONG(x) PyLong_FromLong(x) #define INT_CHECK(x) PyLong_Check(x) #define INT_AS_LONG(x) PyLong_AS_LONG(x) #define TEXT_FROM_STRING PyUnicode_FromString #define TEXT_FORMAT PyUnicode_Format /* Note that the second comparison is skipped if the first comparison returns: 1 -> There was no error and the answer is -1 -1 -> There was an error, which the caller will detect with PyError_Occurred. */ #define COMPARE(lhs, rhs) \ PyObject_RichCompareBool((lhs), (rhs), Py_LT) != 0 ? -1 : \ (PyObject_RichCompareBool((lhs), (rhs), Py_EQ) > 0 ? 0 : 1) #else #define INTERN PyString_InternFromString #define INT_FROM_LONG(x) PyInt_FromLong(x) #define INT_CHECK(x) PyInt_Check(x) #define INT_AS_LONG(x) PyInt_AS_LONG(x) #define TEXT_FROM_STRING PyString_FromString #define TEXT_FORMAT PyString_Format #define COMPARE(lhs, rhs) PyObject_Compare((lhs), (rhs)) #endif #endif /* BTREES__COMPAT_H */ BTrees-4.3.1/BTrees/tests/0000775000175000017500000000000012716353165016410 5ustar tseavertseaver00000000000000BTrees-4.3.1/BTrees/tests/test__base.py0000664000175000017500000033731112510775554021104 0ustar tseavertseaver00000000000000############################################################################## # # Copyright 2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest def _assertRaises(self, e_type, checked, *args, **kw): try: checked(*args, **kw) except e_type as e: return e self.fail("Didn't raise: %s" % e_type.__name__) class Test_Base(unittest.TestCase): def _getTargetClass(self): from .._base import _Base return _Base def _makeOne(self, items=None): class _Test(self._getTargetClass()): max_leaf_size = 10 max_internal_size = 15 def clear(self): self._data = {} def update(self, d): self._data.update(d) return _Test(items) def test_ctor_wo_items(self): base = self._makeOne() self.assertEqual(base._data, {}) def test_ctor_w_items(self): base = self._makeOne({'a': 'b'}) self.assertEqual(base._data, {'a': 'b'}) class Test_BucketBase(unittest.TestCase): def _getTargetClass(self): from .._base import _BucketBase return _BucketBase def _makeOne(self, *args, **kw): return self._getTargetClass()(*args, **kw) def test_ctor_defaults(self): bucket = self._makeOne() self.assertEqual(bucket._keys, []) self.assertEqual(bucket._next, None) self.assertEqual(len(bucket), 0) self.assertEqual(bucket.size, 0) def test__deleteNextBucket_none(self): bucket = self._makeOne() bucket._deleteNextBucket() # no raise self.assertTrue(bucket._next is None) def test__deleteNextBucket_one(self): bucket1 = self._makeOne() bucket2 = bucket1._next = self._makeOne() bucket1._deleteNextBucket() # no raise self.assertTrue(bucket1._next is None) def test__deleteNextBucket_two(self): bucket1 = self._makeOne() bucket2 = bucket1._next = self._makeOne() bucket3 = bucket2._next = self._makeOne() bucket1._deleteNextBucket() # no raise self.assertTrue(bucket1._next is bucket3) def test__search_empty(self): bucket = self._makeOne() self.assertEqual(bucket._search('nonesuch'), -1) def test__search_nonempty_miss(self): bucket = self._makeOne() bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket._search('candy'), -3) def test__search_nonempty_hit(self): bucket = self._makeOne() bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket._search('charlie'), 2) def test_minKey_empty(self): bucket = self._makeOne() self.assertRaises(IndexError, bucket.minKey) def test_minKey_no_bound(self): bucket = self._makeOne() bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.minKey(), 'alpha') def test_minKey_w_bound_hit(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.minKey('bravo'), 'bravo') def test_minKey_w_bound_miss(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.minKey('candy'), 'charlie') def test_minKey_w_bound_fail(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertRaises(ValueError, bucket.minKey, 'foxtrot') def test_maxKey_empty(self): bucket = self._makeOne() self.assertRaises(IndexError, bucket.maxKey) def test_maxKey_no_bound(self): bucket = self._makeOne() bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.maxKey(), 'echo') def test_maxKey_w_bound_hit(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.maxKey('bravo'), 'bravo') def test_maxKey_w_bound_miss(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.maxKey('candy'), 'bravo') def test_maxKey_w_bound_fail(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertRaises(ValueError, bucket.maxKey, 'abacus') def test__range_defaults_empty(self): bucket = self._makeOne() self.assertEqual(bucket._range(), (0, 0)) def test__range_defaults_filled(self): bucket = self._makeOne() bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket._range(), (0, 5)) def test__range_defaults_exclude_min(self): bucket = self._makeOne() bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket._range(excludemin=True), (1, 5)) def test__range_defaults_exclude_max(self): bucket = self._makeOne() bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket._range(excludemax=True), (0, 4)) def test__range_w_min_hit(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket._range(min='bravo'), (1, 5)) def test__range_w_min_miss(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket._range(min='candy'), (2, 5)) def test__range_w_min_hit_w_exclude_min(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket._range(min='bravo', excludemin=True), (2, 5)) def test__range_w_min_miss_w_exclude_min(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] # 'excludemin' doesn't fire on miss self.assertEqual(bucket._range(min='candy', excludemin=True), (2, 5)) def test__range_w_max_hit(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket._range(max='delta'), (0, 4)) def test__range_w_max_miss(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket._range(max='dandy'), (0, 3)) def test__range_w_max_hit_w_exclude_max(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket._range(max='delta', excludemax=True), (0, 3)) def test__range_w_max_miss_w_exclude_max(self): bucket = self._makeOne() bucket._to_key = lambda x: x bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] # 'excludemax' doesn't fire on miss self.assertEqual(bucket._range(max='dandy', excludemax=True), (0, 3)) def test_keys_defaults_empty(self): bucket = self._makeOne() self.assertEqual(bucket.keys(), []) def test_keys_defaults_filled(self): bucket = self._makeOne() KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.keys(), KEYS[0: 5]) def test_keys_defaults_exclude_min(self): bucket = self._makeOne() KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.keys(excludemin=True), KEYS[1: 5]) def test_keys_defaults_exclude_max(self): bucket = self._makeOne() KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.keys(excludemax=True), KEYS[0: 4]) def test_keys_w_min_hit(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.keys(min='bravo'), KEYS[1: 5]) def test_keys_w_min_miss(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.keys(min='candy'), KEYS[2: 5]) def test_keys_w_min_hit_w_exclude_min(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.keys(min='bravo', excludemin=True), KEYS[2: 5]) def test_keys_w_min_miss_w_exclude_min(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] # 'excludemin' doesn't fire on miss self.assertEqual(bucket.keys(min='candy', excludemin=True), KEYS[2: 5]) def test_keys_w_max_hit(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.keys(max='delta'), KEYS[0: 4]) def test_keys_w_max_miss(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.keys(max='dandy'), KEYS[0: 3]) def test_keys_w_max_hit_w_exclude_max(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(bucket.keys(max='delta', excludemax=True), KEYS[0: 3]) def test_keys_w_max_miss_w_exclude_max(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] # 'excludemax' doesn't fire on miss self.assertEqual(bucket.keys(max='dandy', excludemax=True), KEYS[0: 3]) def test_iterkeys_defaults_empty(self): bucket = self._makeOne() self.assertEqual(list(bucket.iterkeys()), []) def test_iterkeys_defaults_filled(self): bucket = self._makeOne() KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(list(bucket.iterkeys()), KEYS[0: 5]) def test_iterkeys_defaults_exclude_min(self): bucket = self._makeOne() KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(list(bucket.iterkeys(excludemin=True)), KEYS[1: 5]) def test_iterkeys_defaults_exclude_max(self): bucket = self._makeOne() KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(list(bucket.iterkeys(excludemax=True)), KEYS[0: 4]) def test_iterkeys_w_min_hit(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(list(bucket.iterkeys(min='bravo')), KEYS[1: 5]) def test_iterkeys_w_min_miss(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(list(bucket.iterkeys(min='candy')), KEYS[2: 5]) def test_iterkeys_w_min_hit_w_exclude_min(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(list(bucket.iterkeys(min='bravo', excludemin=True)), KEYS[2: 5]) def test_iterkeys_w_min_miss_w_exclude_min(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] # 'excludemin' doesn't fire on miss self.assertEqual(list(bucket.iterkeys(min='candy', excludemin=True)), KEYS[2: 5]) def test_iterkeys_w_max_hit(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(list(bucket.iterkeys(max='delta')), KEYS[0: 4]) def test_iterkeys_w_max_miss(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(list(bucket.iterkeys(max='dandy')), KEYS[0: 3]) def test_iterkeys_w_max_hit_w_exclude_max(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual(list(bucket.keys(max='delta', excludemax=True)), KEYS[0: 3]) def test_iterkeys_w_max_miss_w_exclude_max(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] # 'excludemax' doesn't fire on miss self.assertEqual(list(bucket.iterkeys(max='dandy', excludemax=True)), KEYS[0: 3]) def test___iter___empty(self): bucket = self._makeOne() self.assertEqual([x for x in bucket], []) def test___iter___filled(self): bucket = self._makeOne() KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertEqual([x for x in bucket], KEYS[0: 5]) def test___contains___empty(self): bucket = self._makeOne() bucket._to_key = lambda x: x self.assertFalse('nonesuch' in bucket) def test___contains___filled_miss(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] self.assertFalse('nonesuch' in bucket) def test___contains___filled_hit(self): bucket = self._makeOne() bucket._to_key = lambda x: x KEYS = bucket._keys = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: self.assertTrue(key in bucket) class Test_SetIteration(unittest.TestCase): assertRaises = _assertRaises def _getTargetClass(self): from .._base import _SetIteration return _SetIteration def _makeOne(self, to_iterate, useValues=False, default=None): return self._getTargetClass()(to_iterate, useValues, default) def test_ctor_w_None(self): from .._base import _marker si = self._makeOne(None) self.assertEqual(si.useValues, False) self.assertTrue(si.key is _marker) self.assertEqual(si.value, None) self.assertEqual(si.active, False) self.assertEqual(si.position, -1) def test_ctor_w_non_empty_list(self): si = self._makeOne(['a', 'b', 'c']) self.assertEqual(si.useValues, False) self.assertEqual(si.key, 'a') self.assertEqual(si.value, None) self.assertEqual(si.active, True) self.assertEqual(si.position, 1) class BucketTests(unittest.TestCase): assertRaises = _assertRaises def _getTargetClass(self): from .._base import Bucket return Bucket def _makeOne(self): class _Bucket(self._getTargetClass()): def _to_key(self, x): return x def _to_value(self, x): return x return _Bucket() def test_ctor_defaults(self): bucket = self._makeOne() self.assertEqual(bucket._keys, []) self.assertEqual(bucket._values, []) def test_setdefault_miss(self): bucket = self._makeOne() self.assertEqual(bucket.setdefault('a', 'b'), 'b') self.assertEqual(bucket._keys, ['a']) self.assertEqual(bucket._values, ['b']) def test_setdefault_hit(self): bucket = self._makeOne() bucket._keys.append('a') bucket._values.append('b') self.assertEqual(bucket.setdefault('a', 'b'), 'b') self.assertEqual(bucket._keys, ['a']) self.assertEqual(bucket._values, ['b']) def test_pop_miss_no_default(self): bucket = self._makeOne() self.assertRaises(KeyError, bucket.pop, 'nonesuch') def test_pop_miss_w_default(self): bucket = self._makeOne() self.assertEqual(bucket.pop('nonesuch', 'b'), 'b') def test_pop_hit(self): bucket = self._makeOne() bucket._keys.append('a') bucket._values.append('b') self.assertEqual(bucket.pop('a'), 'b') self.assertEqual(bucket._keys, []) self.assertEqual(bucket._values, []) def test_update_value_w_iteritems(self): bucket = self._makeOne() bucket.update({'a': 'b'}) self.assertEqual(bucket._keys, ['a']) self.assertEqual(bucket._values, ['b']) def test_update_value_w_items(self): bucket = self._makeOne() class Foo(object): def items(self): return [('a', 'b')] bucket.update(Foo()) self.assertEqual(bucket._keys, ['a']) self.assertEqual(bucket._values, ['b']) def test_update_value_w_invalid_items(self): bucket = self._makeOne() class Foo(object): def items(self): return ('a', 'b', 'c') self.assertRaises(TypeError, bucket.update, Foo()) def test_update_sequence(self): bucket = self._makeOne() bucket.update([('a', 'b')]) self.assertEqual(bucket._keys, ['a']) self.assertEqual(bucket._values, ['b']) def test_update_replacing(self): bucket = self._makeOne() bucket['a'] = 'b' bucket.update([('a', 'c')]) self.assertEqual(bucket['a'], 'c') def test___setitem___incomparable(self): bucket = self._makeOne() def _should_error(): bucket[object()] = 'b' self.assertRaises(TypeError, _should_error) def test___setitem___comparable(self): bucket = self._makeOne() bucket['a'] = 'b' self.assertEqual(bucket['a'], 'b') def test___setitem___replace(self): bucket = self._makeOne() bucket['a'] = 'b' bucket['a'] = 'c' self.assertEqual(bucket['a'], 'c') def test___delitem___miss(self): bucket = self._makeOne() def _should_error(): del bucket['nonesuch'] self.assertRaises(KeyError, _should_error) def test___delitem___hit(self): bucket = self._makeOne() bucket._keys.append('a') bucket._values.append('b') del bucket['a'] self.assertEqual(bucket._keys, []) self.assertEqual(bucket._values, []) def test_clear_filled(self): bucket = self._makeOne() bucket['a'] = 'b' bucket['c'] = 'd' bucket.clear() self.assertEqual(len(bucket._keys), 0) self.assertEqual(len(bucket._values), 0) def test_clear_empty(self): bucket = self._makeOne() bucket.clear() self.assertEqual(len(bucket._keys), 0) self.assertEqual(len(bucket._values), 0) def test_get_miss_no_default(self): bucket = self._makeOne() self.assertEqual(bucket.get('nonesuch'), None) def test_get_miss_w_default(self): bucket = self._makeOne() self.assertEqual(bucket.get('nonesuch', 'b'), 'b') def test_get_hit(self): bucket = self._makeOne() bucket._keys.append('a') bucket._values.append('b') self.assertEqual(bucket.get('a'), 'b') def test___getitem___miss(self): bucket = self._makeOne() def _should_error(): return bucket['nonesuch'] self.assertRaises(KeyError, _should_error) def test___getitem___hit(self): bucket = self._makeOne() bucket._keys.append('a') bucket._values.append('b') self.assertEqual(bucket['a'], 'b') def test__split_empty(self): bucket = self._makeOne() next_b = bucket._next = self._makeOne() new_b = bucket._split() self.assertEqual(len(bucket._keys), 0) self.assertEqual(len(bucket._values), 0) self.assertEqual(len(new_b._keys), 0) self.assertEqual(len(new_b._values), 0) self.assertTrue(bucket._next is new_b) self.assertTrue(new_b._next is next_b) def test__split_filled_default_index(self): bucket = self._makeOne() next_b = bucket._next = self._makeOne() for i, c in enumerate('abcdef'): bucket[c] = i new_b = bucket._split() self.assertEqual(list(bucket._keys), ['a', 'b', 'c']) self.assertEqual(list(bucket._values), [0, 1, 2]) self.assertEqual(list(new_b._keys), ['d', 'e', 'f']) self.assertEqual(list(new_b._values), [3, 4, 5]) self.assertTrue(bucket._next is new_b) self.assertTrue(new_b._next is next_b) def test__split_filled_explicit_index(self): bucket = self._makeOne() next_b = bucket._next = self._makeOne() for i, c in enumerate('abcdef'): bucket[c] = i new_b = bucket._split(2) self.assertEqual(list(bucket._keys), ['a', 'b']) self.assertEqual(list(bucket._values), [0, 1]) self.assertEqual(list(new_b._keys), ['c', 'd', 'e', 'f']) self.assertEqual(list(new_b._values), [2, 3, 4, 5]) self.assertTrue(bucket._next is new_b) self.assertTrue(new_b._next is next_b) def test_keys_empty_no_args(self): bucket = self._makeOne() self.assertEqual(bucket.keys(), []) def test_keys_filled_no_args(self): bucket = self._makeOne() for i, c in enumerate('abcdef'): bucket[c] = i self.assertEqual(bucket.keys(), ['a', 'b', 'c', 'd', 'e', 'f']) def test_keys_filled_w_args(self): bucket = self._makeOne() for i, c in enumerate('abcdef'): bucket[c] = i self.assertEqual(bucket.keys(min='b', excludemin=True, max='f', excludemax=True), ['c', 'd', 'e']) def test_iterkeys_empty_no_args(self): bucket = self._makeOne() self.assertEqual(list(bucket.iterkeys()), []) def test_iterkeys_filled_no_args(self): bucket = self._makeOne() for i, c in enumerate('abcdef'): bucket[c] = i self.assertEqual(list(bucket.iterkeys()), ['a', 'b', 'c', 'd', 'e', 'f']) def test_iterkeys_filled_w_args(self): bucket = self._makeOne() for i, c in enumerate('abcdef'): bucket[c] = i self.assertEqual(list(bucket.iterkeys( min='b', excludemin=True, max='f', excludemax=True)), ['c', 'd', 'e']) def test_values_empty_no_args(self): bucket = self._makeOne() self.assertEqual(bucket.values(), []) def test_values_filled_no_args(self): bucket = self._makeOne() for i, c in enumerate('abcdef'): bucket[c] = i self.assertEqual(bucket.values(), list(range(6))) def test_values_filled_w_args(self): bucket = self._makeOne() for i, c in enumerate('abcdef'): bucket[c] = i self.assertEqual(bucket.values(min='b', excludemin=True, max='f', excludemax=True), [2, 3, 4]) def test_itervalues_empty_no_args(self): bucket = self._makeOne() self.assertEqual(list(bucket.itervalues()), []) def test_itervalues_filled_no_args(self): bucket = self._makeOne() for i, c in enumerate('abcdef'): bucket[c] = i self.assertEqual(list(bucket.itervalues()), list(range(6))) def test_itervalues_filled_w_args(self): bucket = self._makeOne() for i, c in enumerate('abcdef'): bucket[c] = i self.assertEqual(list(bucket.itervalues( min='b', excludemin=True, max='f', excludemax=True)), [2, 3, 4]) def test_items_empty_no_args(self): bucket = self._makeOne() self.assertEqual(bucket.items(), []) def test_items_filled_no_args(self): bucket = self._makeOne() EXPECTED = [] for i, c in enumerate('abcdef'): bucket[c] = i EXPECTED.append((c, i)) self.assertEqual(bucket.items(), EXPECTED) def test_items_filled_w_args(self): bucket = self._makeOne() EXPECTED = [] for i, c in enumerate('abcdef'): bucket[c] = i EXPECTED.append((c, i)) self.assertEqual(bucket.items(min='b', excludemin=True, max='f', excludemax=True), EXPECTED[2:5]) def test_iteritems_empty_no_args(self): bucket = self._makeOne() self.assertEqual(list(bucket.iteritems()), []) def test_iteritems_filled_no_args(self): bucket = self._makeOne() EXPECTED = [] for i, c in enumerate('abcdef'): bucket[c] = i EXPECTED.append((c, i)) self.assertEqual(list(bucket.iteritems()), EXPECTED) def test_iteritems_filled_w_args(self): bucket = self._makeOne() EXPECTED = [] for i, c in enumerate('abcdef'): bucket[c] = i EXPECTED.append((c, i)) self.assertEqual(list(bucket.iteritems(min='b', excludemin=True, max='f', excludemax=True)), EXPECTED[2:5]) def test___getstate___empty_no_next(self): bucket = self._makeOne() self.assertEqual(bucket.__getstate__(), ((),)) def test___getstate___empty_w_next(self): bucket = self._makeOne() bucket._next = next_b = self._makeOne() self.assertEqual(bucket.__getstate__(), ((), next_b)) def test___getstate___non_empty_no_next(self): bucket = self._makeOne() EXPECTED = () for i, c in enumerate('abcdef'): bucket[c] = i EXPECTED += (c, i) self.assertEqual(bucket.__getstate__(), (EXPECTED,)) def test___getstate___non_empty_w_next(self): bucket = self._makeOne() bucket._next = next_b = self._makeOne() EXPECTED = () for i, c in enumerate('abcdef'): bucket[c] = i EXPECTED += (c, i) self.assertEqual(bucket.__getstate__(), (EXPECTED, next_b)) def test___setstate___w_non_tuple(self): bucket = self._makeOne() self.assertRaises(TypeError, bucket.__setstate__, (None,)) def test___setstate___w_empty_no_next(self): bucket = self._makeOne() bucket._next = next_b = self._makeOne() for i, c in enumerate('abcdef'): bucket[c] = i bucket.__setstate__(((),)) self.assertEqual(len(bucket.keys()), 0) self.assertTrue(bucket._next is None) def test___setstate___w_non_empty_w_next(self): bucket = self._makeOne() next_b = self._makeOne() ITEMS = () EXPECTED = [] for i, c in enumerate('abcdef'): ITEMS += (c, i) EXPECTED.append((c, i)) bucket.__setstate__((ITEMS, next_b)) self.assertEqual(bucket.items(), EXPECTED) self.assertTrue(bucket._next is next_b) def test__p_resolveConflict_x_on_com_next_old_new_None(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() N_NEW = object() s_old = None s_com = ((), N_NEW) s_new = None e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 0) def test__p_resolveConflict_x_on_com_next(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() N_NEW = object() s_old = ((), None) s_com = ((), N_NEW) s_new = ((), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 0) def test__p_resolveConflict_x_on_new_next_old_com_None(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() N_NEW = object() s_old = None s_com = None s_new = ((), N_NEW) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 0) def test__p_resolveConflict_x_on_new_next(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() N_NEW = object() s_old = ((), None) s_com = ((), None) s_new = ((), N_NEW) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 0) def test__p_resolveConflict_x_on_com_empty(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 'b', 'c', 'd'), None) s_com = ((), None) s_new = (('a', 'b'), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 12) def test__p_resolveConflict_x_on_new_empty(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0, 'b', 1), None) s_com = (('a', 0), None) s_new = ((), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 12) def test__p_resolveConflict_x_both_update_same_key(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0), None) s_com = (('a', 5, 'b', 1, 'c', 2), None) s_new = (('a', 6, 'd', 3), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 1) def test__p_resolveConflict_x_on_del_first_com_x(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0, 'b', 1, 'c', 2), None) s_com = (('b', 1), None) s_new = (('a', 0, 'b', 1), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 13) def test__p_resolveConflict_x_on_del_first_new_x(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0, 'b', 1, 'c', 2), None) s_com = (('a', 0, 'b', 1), None) s_new = (('b', 1), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 13) def test__p_resolveConflict_x_on_del_first_new(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0, 'b', 1), None) s_com = (('a', 1, 'b', 2, 'c', 3), None) s_new = (('b', 4), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 2) def test__p_resolveConflict_x_on_del_first_com(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0, 'b', 1), None) s_com = (('b', 4), None) s_new = (('a', 1, 'b', 2, 'c', 3), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 3) def test__p_resolveConflict_x_on_ins_same_after_del(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0, 'b', 1), None) s_com = (('a', 0, 'c', 2), None) s_new = (('a', 0, 'c', 2, 'd', 3), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 4) def test__p_resolveConflict_x_on_del_same(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0, 'b', 1, 'c', 2), None) s_com = (('a', 0, 'c', 2), None) s_new = (('a', 0, 'd', 3, 'e', 4), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 5) def test__p_resolveConflict_x_on_append_same(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0, ), None) s_com = (('a', 0, 'b', 1), None) s_new = (('a', 0, 'b', 1, 'c', 2), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 6) def test__p_resolveConflict_x_on_new_deletes_all_com_adds(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0, 'b', 1, 'c', 2), None) s_com = (('a', 0, 'd', 3, 'e', 4, 'f', 5), None) s_new = (('a', 0, ), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 7) def test__p_resolveConflict_x_on_com_deletes_all_new_adds(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0, 'b', 1, 'c', 2), None) s_com = (('a', 0, ), None) s_new = (('a', 0, 'd', 3, 'e', 4, 'f', 5), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 8) def test__p_resolveConflict_x_on_com_deletes_all_new_deletes(self): from ..Interfaces import BTreesConflictError bucket = self._makeOne() s_old = (('a', 0, 'b', 1, 'c', 2), None) s_com = (('a', 0, ), None) s_new = (('a', 0, 'b', 1), None) e = self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 9) def test__p_resolveConflict_ok_both_add_new_max(self): bucket = self._makeOne() s_old = (('a', 0), None) s_com = (('a', 0, 'b', 1), None) s_new = (('a', 0, 'c', 2), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 0, 'b', 1, 'c', 2),)) def test__p_resolveConflict_ok_com_updates(self): bucket = self._makeOne() s_old = (('a', 0), None) s_com = (('a', 5), None) s_new = (('a', 0, 'd', 3), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 5, 'd', 3),)) def test__p_resolveConflict_ok_new_updates(self): bucket = self._makeOne() s_old = (('a', 0), None) s_com = (('a', 0, 'd', 3), None) s_new = (('a', 5), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 5, 'd', 3),)) def test__p_resolveConflict_ok_com_inserts_new_adds(self): bucket = self._makeOne() s_old = (('a', 0, 'c', 2), None) s_com = (('a', 0, 'b', 1, 'c', 2), None) s_new = (('a', 0, 'c', 2, 'd', 3), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 0, 'b', 1, 'c', 2, 'd', 3),)) def test__p_resolveConflict_ok_com_adds_new_inserts(self): bucket = self._makeOne() s_old = (('a', 0, 'c', 2), None) s_com = (('a', 0, 'c', 2, 'd', 3), None) s_new = (('a', 0, 'b', 1, 'c', 2), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 0, 'b', 1, 'c', 2, 'd', 3),)) def test__p_resolveConflict_ok_com_adds_new_deletes(self): bucket = self._makeOne() s_old = (('a', 0, 'b', 1, 'c', 2), None) s_com = (('a', 0, 'b', 1, 'c', 2, 'd', 3), None) s_new = (('a', 0, 'e', 4), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 0, 'd', 3, 'e', 4),)) def test__p_resolveConflict_ok_com_deletes_new_adds(self): bucket = self._makeOne() s_old = (('a', 0, 'b', 1, 'c', 2), None) s_com = (('a', 0, 'e', 4), None) s_new = (('a', 0, 'b', 1, 'c', 2, 'd', 3), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 0, 'd', 3, 'e', 4),)) def test__p_resolveConflict_ok_both_insert_new_lt_com(self): bucket = self._makeOne() s_old = (('a', 0, 'd', 3), None) s_com = (('a', 0, 'c', 2, 'd', 3), None) s_new = (('a', 0, 'b', 1, 'd', 3), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 0, 'b', 1, 'c', 2, 'd', 3),)) def test__p_resolveConflict_ok_both_insert_new_gt_com(self): bucket = self._makeOne() s_old = (('a', 0, 'd', 3), None) s_com = (('a', 0, 'b', 1, 'd', 3), None) s_new = (('a', 0, 'c', 2, 'd', 3), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 0, 'b', 1, 'c', 2, 'd', 3),)) def test__p_resolveConflict_ok_new_insert_then_com_append(self): bucket = self._makeOne() s_old = (('a', 0, 'd', 3), None) s_com = (('a', 0, 'e', 4), None) s_new = (('a', 0, 'b', 1, 'd', 3), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 0, 'b', 1, 'e', 4),)) def test__p_resolveConflict_ok_com_insert_then_new_append(self): bucket = self._makeOne() s_old = (('a', 0, 'd', 3), None) s_com = (('a', 0, 'b', 1, 'd', 3), None) s_new = (('a', 0, 'e', 4), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 0, 'b', 1, 'e', 4),)) def test__p_resolveConflict_ok_new_deletes_tail_com_inserts(self): bucket = self._makeOne() s_old = (('a', 0, 'b', 1, 'd', 3), None) s_com = (('a', 0, 'b', 1, 'c', 2, 'd', 3), None) s_new = (('a', 0), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 0, 'c', 2),)) def test__p_resolveConflict_ok_com_deletes_tail_new_inserts(self): bucket = self._makeOne() s_old = (('a', 0, 'b', 1, 'd', 3), None) s_com = (('a', 0), None) s_new = (('a', 0, 'b', 1, 'c', 2, 'd', 3), None) result = bucket._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 0, 'c', 2),)) class SetTests(unittest.TestCase): assertRaises = _assertRaises def _getTargetClass(self): from .._base import Set return Set def _makeOne(self): class _Set(self._getTargetClass()): def _to_key(self, x): return x return _Set() def test_add_not_extant(self): _set = self._makeOne() _set.add('not_extant') self.assertEqual(list(_set), ['not_extant']) def test_add_extant(self): _set = self._makeOne() _set.add('extant') _set.add('extant') self.assertEqual(list(_set), ['extant']) def test_insert(self): _set = self._makeOne() _set.insert('inserted') self.assertEqual(list(_set), ['inserted']) def test_remove_miss(self): _set = self._makeOne() self.assertRaises(KeyError, _set.remove, 'not_extant') def test_remove_extant(self): _set = self._makeOne() _set.add('one') _set.add('another') _set.remove('one') self.assertEqual(list(_set), ['another']) def test_update(self): _set = self._makeOne() _set.update(['one', 'after', 'another']) self.assertEqual(sorted(_set), ['after', 'another', 'one']) def test___getstate___empty_no_next(self): _set = self._makeOne() self.assertEqual(_set.__getstate__(), ((),)) def test___getstate___empty_w_next(self): _set = self._makeOne() _set._next = next_s = self._makeOne() self.assertEqual(_set.__getstate__(), ((), next_s)) def test___getstate___non_empty_no_next(self): _set = self._makeOne() EXPECTED = () for c in 'abcdef': _set.add(c) EXPECTED += (c,) self.assertEqual(_set.__getstate__(), (EXPECTED,)) def test___getstate___non_empty_w_next(self): _set = self._makeOne() _set._next = next_s = self._makeOne() EXPECTED = () for c in 'abcdef': _set.add(c) EXPECTED += (c,) self.assertEqual(_set.__getstate__(), (EXPECTED, next_s)) def test___setstate___w_non_tuple(self): _set = self._makeOne() self.assertRaises(TypeError, _set.__setstate__, (None,)) def test___setstate___w_empty_no_next(self): _set = self._makeOne() _set._next = next_s = self._makeOne() for c in 'abcdef': _set.add(c) _set.__setstate__(((),)) self.assertEqual(len(_set), 0) self.assertTrue(_set._next is None) def test___setstate___w_non_empty_w_next(self): _set = self._makeOne() next_s = self._makeOne() ITEMS = () EXPECTED = [] for c in 'abcdef': ITEMS += (c,) EXPECTED.append(c) _set.__setstate__((ITEMS, next_s)) self.assertEqual(sorted(_set), EXPECTED) self.assertTrue(_set._next is next_s) def test___getitem___out_of_bounds(self): _set = self._makeOne() self.assertRaises(IndexError, _set.__getitem__, 1) def test___getitem___hit_bounds(self): _set = self._makeOne() _set.add('b') _set.add('a') _set.add('c') self.assertEqual(_set[0], 'a') self.assertEqual(_set[1], 'b') self.assertEqual(_set[2], 'c') def test__split_empty(self): _set = self._makeOne() next_b = _set._next = self._makeOne() new_b = _set._split() self.assertEqual(len(_set._keys), 0) self.assertEqual(len(new_b._keys), 0) self.assertTrue(_set._next is new_b) self.assertTrue(new_b._next is next_b) def test__split_filled_default_index(self): _set = self._makeOne() next_b = _set._next = self._makeOne() for c in 'abcdef': _set.add(c) new_b = _set._split() self.assertEqual(list(_set._keys), ['a', 'b', 'c']) self.assertEqual(list(new_b._keys), ['d', 'e', 'f']) self.assertTrue(_set._next is new_b) self.assertTrue(new_b._next is next_b) def test__split_filled_explicit_index(self): _set = self._makeOne() next_b = _set._next = self._makeOne() for c in 'abcdef': _set.add(c) new_b = _set._split(2) self.assertEqual(list(_set._keys), ['a', 'b']) self.assertEqual(list(new_b._keys), ['c', 'd', 'e', 'f']) self.assertTrue(_set._next is new_b) self.assertTrue(new_b._next is next_b) def test__p_resolveConflict_x_on_com_next_old_new_None(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() N_NEW = object() s_old = None s_com = ((), N_NEW) s_new = None e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 0) def test__p_resolveConflict_x_on_com_next(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() N_NEW = object() s_old = ((), None) s_com = ((), N_NEW) s_new = ((), None) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 0) def test__p_resolveConflict_x_on_new_next_old_com_None(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() N_NEW = object() s_old = None s_com = None s_new = ((), N_NEW) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 0) def test__p_resolveConflict_x_on_new_next(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() N_NEW = object() s_old = ((), None) s_com = ((), None) s_new = ((), N_NEW) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 0) def test__p_resolveConflict_x_on_com_empty(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() s_old = (('a', 'b'), None) s_com = ((), None) s_new = (('a',), None) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 12) def test__p_resolveConflict_x_on_new_empty(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() s_old = (('a', 'b'), None) s_com = (('a',), None) s_new = ((), None) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 12) def test__p_resolveConflict_x_on_del_first_com(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() s_old = (('a','b'), None) s_com = (('b',), None) s_new = (('a', 'b', 'c'), None) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 13) def test__p_resolveConflict_x_on_del_first_new(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() s_old = (('a', 'b'), None) s_com = (('a', 'b', 'c'), None) s_new = (('b',), None) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 13) def test__p_resolveConflict_x_on_ins_same_after_del(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() s_old = (('a', 'b'), None) s_com = (('a', 'c'), None) s_new = (('a', 'c', 'd'), None) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 4) def test__p_resolveConflict_x_on_del_same(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() s_old = (('a', 'b', 'c'), None) s_com = (('a', 'c'), None) s_new = (('a', 'd', 'e'), None) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 5) def test__p_resolveConflict_x_on_append_same(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() s_old = (('a',), None) s_com = (('a', 'b'), None) s_new = (('a', 'b', 'c'), None) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 6) def test__p_resolveConflict_x_on_new_deletes_all_com_adds(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() s_old = (('a', 'b', 'c'), None) s_com = (('a', 'd', 'e', 'f'), None) s_new = (('a',), None) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 7) def test__p_resolveConflict_x_on_com_deletes_all_new_adds(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() s_old = (('a', 'b', 'c'), None) s_com = (('a',), None) s_new = (('a', 'd', 'e', 'f'), None) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 8) def test__p_resolveConflict_x_on_com_deletes_all_new_deletes(self): from ..Interfaces import BTreesConflictError _set = self._makeOne() s_old = (('a', 'b', 'c'), None) s_com = (('a',), None) s_new = (('a', 'b'), None) e = self.assertRaises(BTreesConflictError, _set._p_resolveConflict, s_old, s_com, s_new) self.assertEqual(e.reason, 9) def test__p_resolveConflict_ok_insert_in_new_add_in_com(self): _set = self._makeOne() s_old = (('a', 'c'), None) s_com = (('a', 'c', 'd'), None) s_new = (('a', 'b', 'c'), None) result = _set._p_resolveConflict(s_old, s_com, s_new) # Note that _SetBase uses default __getstate__ self.assertEqual(result, (('a', 'b', 'c', 'd'),)) def test__p_resolveConflict_ok_insert_in_com_add_in_new(self): _set = self._makeOne() s_old = (('a', 'c'), None) s_com = (('a', 'b', 'c'), None) s_new = (('a', 'c', 'd'), None) result = _set._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 'b', 'c', 'd'),)) def test__p_resolveConflict_ok_delete_in_new_add_in_com(self): _set = self._makeOne() s_old = (('a', 'b', 'c'), None) s_com = (('a', 'b', 'c', 'd'), None) s_new = (('a', 'c'), None) result = _set._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 'c', 'd'),)) def test__p_resolveConflict_ok_delete_in_com_add_in_new(self): _set = self._makeOne() s_old = (('a', 'b', 'c'), None) s_com = (('a', 'c'), None) s_new = (('a', 'b', 'c', 'd'), None) result = _set._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 'c', 'd'),)) def test__p_resolveConflict_ok_add_new_lt_add_com(self): _set = self._makeOne() s_old = (('a',), None) s_com = (('a', 'd'), None) s_new = (('a', 'b', 'c'), None) result = _set._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 'b', 'c', 'd'),)) def test__p_resolveConflict_ok_add_com_lt_add_new(self): _set = self._makeOne() s_old = (('a',), None) s_com = (('a', 'b', 'c'), None) s_new = (('a', 'd'), None) result = _set._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 'b', 'c', 'd'),)) def test__p_resolveConflict_ok_ins_in_com_del_add_in_new(self): _set = self._makeOne() s_old = (('a', 'c'), None) s_com = (('a', 'b', 'c'), None) s_new = (('a', 'd', 'e'), None) result = _set._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 'b', 'd', 'e'),)) def test__p_resolveConflict_ok_ins_in_new_del_add_in_com(self): _set = self._makeOne() s_old = (('a', 'c'), None) s_com = (('a', 'd', 'e'), None) s_new = (('a', 'b', 'c'), None) result = _set._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 'b', 'd', 'e'),)) def test__p_resolveConflict_ok_ins_both_new_lt_com(self): _set = self._makeOne() s_old = (('a', 'e'), None) s_com = (('a', 'c', 'd', 'e'), None) s_new = (('a', 'b', 'e'), None) result = _set._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 'b', 'c', 'd', 'e'),)) def test__p_resolveConflict_ok_del_new_add_com(self): _set = self._makeOne() s_old = (('a', 'e'), None) s_com = (('a', 'c', 'd', 'e'), None) s_new = (('a',), None) result = _set._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 'c', 'd'),)) def test__p_resolveConflict_ok_del_com_add_new(self): _set = self._makeOne() s_old = (('a', 'e'), None) s_com = (('a',), None) s_new = (('a', 'c', 'd', 'e'), None) result = _set._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 'c', 'd'),)) def test__p_resolveConflict_add_new_gt_old_com_lt_old(self): _set = self._makeOne() s_old = (('a', 'b', 'c'), None) s_com = (('a', 'b', 'bb', 'c'), None) s_new = (('a', 'b', 'c', 'd'), None) result = _set._p_resolveConflict(s_old, s_com, s_new) self.assertEqual(result, (('a', 'b', 'bb', 'c', 'd'),)) class Test_TreeItem(unittest.TestCase): def _getTargetClass(self): from .._base import _TreeItem return _TreeItem def _makeOne(self, key, child): return self._getTargetClass()(key, child) def test_ctor(self): child = object() item = self._makeOne('key', child) self.assertEqual(item.key, 'key') self.assertTrue(item.child is child) class Test_Tree(unittest.TestCase): assertRaises = _assertRaises def _getTargetClass(self): from .._base import _Tree return _Tree def _makeOne(self, items=None): from .._base import Bucket class _Bucket(Bucket): def _to_key(self, k): return k class _Test(self._getTargetClass()): _to_key = _to_value = lambda self, x: x _bucket_type = _Bucket max_leaf_size = 10 max_internal_size = 15 return _Test(items) def test_setdefault_miss(self): tree = self._makeOne() value = object() self.assertTrue(tree.setdefault('non_extant', value) is value) self.assertTrue('non_extant' in tree) self.assertTrue(tree._findbucket('non_extant')['non_extant'] is value) def test_setdefault_hit(self): tree = self._makeOne() value1 = object() value2 = object() tree['extant'] = value1 self.assertTrue(tree.setdefault('extant', value2) is value1) self.assertTrue('extant' in tree) self.assertTrue(tree._findbucket('extant')['extant'] is value1) def test_pop_miss_no_default(self): tree = self._makeOne() self.assertRaises(KeyError, tree.pop, 'nonesuch') def test_pop_miss_w_default(self): default = object() tree = self._makeOne() self.assertTrue(tree.pop('nonesuch', default) is default) def test_pop_hit(self): tree = self._makeOne() value = object() tree['extant'] = value self.assertTrue(tree.pop('extant', value) is value) self.assertFalse('extant' in tree) def test_update_value_w_iteritems(self): tree = self._makeOne() tree.update({'a': 'b'}) self.assertEqual(tree._findbucket('a')['a'], 'b') def test_update_value_w_items(self): tree = self._makeOne() class Foo(object): def items(self): return [('a', 'b')] tree.update(Foo()) self.assertEqual(tree._findbucket('a')['a'], 'b') def test_update_value_w_invalid_items(self): tree = self._makeOne() class Foo(object): def items(self): return ('a', 'b', 'c') self.assertRaises(TypeError, tree.update, Foo()) def test_update_sequence(self): tree = self._makeOne() tree.update([('a', 'b')]) self.assertEqual(tree._findbucket('a')['a'], 'b') def test_update_replacing(self): tree = self._makeOne() tree['a'] = 'b' tree.update([('a', 'c')]) self.assertEqual(tree._findbucket('a')['a'], 'c') def test___setitem___incomparable(self): tree = self._makeOne() def _should_error(): tree[object()] = 'b' self.assertRaises(TypeError, _should_error) def test___delitem___miss(self): tree = self._makeOne() def _should_error(): del tree['a'] self.assertRaises(KeyError, _should_error) def test___delitem___hit(self): tree = self._makeOne() tree['a'] = 'b' del tree['a'] self.assertFalse('a' in tree) def test_clear(self): tree = self._makeOne() tree['a'] = 'b' tree.clear() self.assertFalse('a' in tree) self.assertEqual(tree._firstbucket, None) def test___nonzero___empty(self): tree = self._makeOne() self.assertFalse(tree) def test___nonzero___nonempty(self): tree = self._makeOne() tree['a'] = 'b' self.assertTrue(tree) def test___len__empty(self): tree = self._makeOne() self.assertEqual(len(tree), 0) def test___len__nonempty(self): tree = self._makeOne() tree['a'] = 'b' self.assertEqual(len(tree), 1) def test___len__nonempty_multiple_buckets(self): tree = self._makeOne() for i in range(100): tree[str(i)] = i self.assertEqual(len(tree), 100) b_count = 0 bucket = tree._firstbucket def test_size_empty(self): tree = self._makeOne() self.assertEqual(tree.size, 0) def test_size_nonempty(self): tree = self._makeOne() tree['a'] = 'b' self.assertEqual(tree.size, 1) def test_size_nonempty_multiple_buckets(self): tree = self._makeOne() for i in range(100): tree[str(i)] = i b_count = 0 bucket = tree._firstbucket while bucket is not None: b_count += 1 bucket = bucket._next self.assertEqual(tree.size, b_count) def test__search_empty(self): tree = self._makeOne() self.assertEqual(tree._search('nonesuch'), -1) def test__search_miss_high(self): tree = self._makeOne() for i in range(100): tree[float(i)] = i b_count = 0 bucket = tree._firstbucket while bucket is not None: b_count += 1 bucket = bucket._next self.assertEqual(tree.size, b_count) self.assertEqual(tree._search(99.5), b_count - 1) def test__search_miss_low(self): tree = self._makeOne() for i in range(100): tree[float(i)] = i self.assertEqual(tree._search(0.1), 0) def test__search_miss_between(self): tree = self._makeOne() for i in range(100): tree[float(i)] = i self.assertEqual(tree._search(1.5), 0) def test__search_hit(self): tree = self._makeOne() for i in range(100): tree[float(i)] = i key = tree._data[1].key self.assertEqual(tree._search(key), 1) def test__find_bucket_low(self): tree = self._makeOne() for i in range(1000): tree[float(i)] = i self.assertTrue(tree._findbucket(0.1) is tree._firstbucket) def test__find_bucket_high(self): tree = self._makeOne() for i in range(1000): tree[float(i)] = i bucket = tree._firstbucket while bucket._next is not None: bucket = bucket._next self.assertTrue(tree._findbucket(999.5) is bucket) def test___contains___empty(self): tree = self._makeOne() self.assertFalse('nonesuch' in tree) def test___contains___miss(self): tree = self._makeOne() for i in range(1000): tree[float(i)] = i self.assertFalse(1000.0 in tree) def test___contains___hit(self): tree = self._makeOne() keys = [] for i in range(1000): key = float(i) tree[key] = i keys.append(key) for key in keys: self.assertTrue(key in tree) def test_has_key_empty(self): tree = self._makeOne() self.assertFalse(tree.has_key('nonesuch')) def test_has_key_miss(self): tree = self._makeOne() for i in range(1000): tree[float(i)] = i self.assertFalse(tree.has_key(1000.0)) def test_has_key_hit(self): tree = self._makeOne() KEYS = [] for i in range(1000): key = float(i) tree[key] = i KEYS.append(key) for key in KEYS: # XXX should we be testing for the 'depth' value? self.assertTrue(tree.has_key(key)) def test_keys_defaults_empty(self): tree = self._makeOne() self.assertEqual(list(tree.keys()), []) def test_keys_defaults_filled(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(list(tree.keys()), KEYS[:]) def test_keys_defaults_exclude_min(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(list(tree.keys(excludemin=True)), KEYS[1: 5]) def test_keys_defaults_exclude_max(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(list(tree.keys(excludemax=True)), KEYS[0: 4]) def test_keys_w_min_hit(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(list(tree.keys(min='bravo')), KEYS[1: 5]) def test_keys_w_min_miss(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(list(tree.keys(min='candy')), KEYS[2: 5]) def test_keys_w_min_hit_w_exclude_min(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(list(tree.keys(min='bravo', excludemin=True)), KEYS[2: 5]) def test_keys_w_min_miss_w_exclude_min(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() # 'excludemin' doesn't fire on miss self.assertEqual(list(tree.keys(min='candy', excludemin=True)), KEYS[2: 5]) def test_keys_w_max_hit(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(list(tree.keys(max='delta')), KEYS[0: 4]) def test_keys_w_max_miss(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(list(tree.keys(max='dandy')), KEYS[0: 3]) def test_keys_w_max_hit_w_exclude_max(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(list(tree.keys(max='delta', excludemax=True)), KEYS[0: 3]) def test_keys_w_max_miss_w_exclude_max(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() # 'excludemax' doesn't fire on miss self.assertEqual(list(tree.keys(max='dandy', excludemax=True)), KEYS[0: 3]) def test_iterkeys(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(list(tree.iterkeys()), KEYS) def test___iter__(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(list(tree), KEYS) def test_minKey_empty(self): tree = self._makeOne() self.assertRaises(ValueError, tree.minKey) def test_minKey_filled_default(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(tree.minKey(), KEYS[0]) def test_minKey_filled_explicit_hit(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(tree.minKey(min='bravo'), 'bravo') def test_minKey_filled_explicit_miss(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(tree.minKey(min='basso'), 'bravo') def test_maxKey_empty(self): tree = self._makeOne() self.assertRaises(ValueError, tree.maxKey) def test_maxKey_filled_default(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(tree.maxKey(), 'echo') def test_maxKey_filled_explicit_hit(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(tree.maxKey('bravo'), 'bravo') def test_maxKey_filled_explicit_miss(self): tree = self._makeOne() KEYS = ['alpha', 'bravo', 'charlie', 'delta', 'echo'] for key in KEYS: tree[key] = key.upper() self.assertEqual(tree.maxKey('candy'), 'bravo') def test__set_calls_readCurrent_on_jar(self): tree = self._makeOne() tree._p_oid = b'OID' tree._p_serial = b'01234567' tree._p_jar = jar = _Jar() tree._set('a', 'b') self.assertTrue(tree in jar._current) def test__split_empty(self): tree = self._makeOne() self.assertRaises(IndexError, tree._split) def test__split_filled_empties_original(self): tree = self._makeOne() next_t = tree._next = self._makeOne() for i, c in enumerate('abcdef'): tree[c] = i fb = tree._firstbucket new_t = tree._split() self.assertEqual(list(tree), []) self.assertTrue(tree._firstbucket is None) self.assertEqual(list(new_t), ['a', 'b', 'c', 'd', 'e', 'f']) self.assertTrue(new_t._firstbucket is fb) def test__split_filled_divides_original(self): tree = self._makeOne() next_t = tree._next = self._makeOne() LETTERS = 'abcdefghijklmnopqrstuvwxyz' for i, c in enumerate(LETTERS): tree[c] = i fb = tree._firstbucket new_t = tree._split() # Note that original tree still links to split buckets self.assertEqual(''.join(list(tree)), LETTERS) self.assertTrue(tree._firstbucket is fb) self.assertEqual(''.join(list(new_t)), LETTERS[10:]) self.assertFalse(new_t._firstbucket is fb) def test__split_filled_divides_deeper(self): tree = self._makeOne() next_t = tree._next = self._makeOne() KEYS = [] FMT = '%05d' for i in range(1000): key = FMT % i tree[key] = i KEYS.append(key) fb = tree._firstbucket new_t = tree._split(tree.max_internal_size - 2) # Note that original tree still links to split buckets self.assertEqual(list(tree), KEYS) self.assertTrue(tree._firstbucket is fb) new_min = new_t.minKey() self.assertEqual(list(new_t), KEYS[int(new_min):]) self.assertFalse(new_t._firstbucket is fb) def test__del_calls_readCurrent_on_jar(self): tree = self._makeOne({'a': 'b'}) tree._p_oid = b'OID' tree._p_serial = b'01234567' tree._p_jar = jar = _Jar() tree._del('a') self.assertTrue(tree in jar._current) def test__del_miss(self): tree = self._makeOne({'a': 'b'}) self.assertRaises(KeyError, tree._del, 'nonesuch') def test__del_fixes_up_node_key(self): SOURCE = dict([('%05d' % i, i) for i in range(1000)]) tree = self._makeOne(SOURCE) before = tree._data[1].key del tree[before] after = tree._data[1].key self.assertTrue(after > before) def test__del_empties_first_bucket_not_zeroth_item(self): SOURCE = dict([('%05d' % i, i) for i in range(1000)]) tree = self._makeOne(SOURCE) bucket = tree._data[1].child._firstbucket next_b = bucket._next for key in list(bucket): # don't del while iterting del tree[key] self.assertTrue(tree._data[1].child._firstbucket is next_b) def test__del_empties_first_bucket_zeroth_item(self): SOURCE = dict([('%05d' % i, i) for i in range(1000)]) tree = self._makeOne(SOURCE) bucket = tree._data[0].child._firstbucket next_b = bucket._next for key in list(bucket): # don't del while iterting del tree[key] self.assertTrue(tree._data[0].child._firstbucket is next_b) self.assertTrue(tree._firstbucket is next_b) def test__del_empties_other_bucket_not_zeroth_item(self): SOURCE = dict([('%05d' % i, i) for i in range(1000)]) tree = self._makeOne(SOURCE) bucket = tree._data[1].child._firstbucket._next next_b = bucket._next for key in list(bucket): # don't del while iterting del tree[key] self.assertTrue(tree._data[1].child._firstbucket._next is next_b) def test___getstate___empty(self): tree = self._makeOne() self.assertEqual(tree.__getstate__(), None) def test___getstate___single_bucket_wo_oid(self): tree = self._makeOne({'a': 'b'}) self.assertEqual(tree.__getstate__(), (((('a', 'b'),),),)) def test___getstate___single_bucket_w_oid(self): tree = self._makeOne({'a': 'b'}) bucket = tree._firstbucket jar = _Jar() bucket._p_jar = jar bucket._p_oid = b'OID' self.assertEqual(tree.__getstate__(), ((bucket,), bucket)) def test___getstate___multiple_buckets(self): tree = self._makeOne() FMT = '%05d' for i in range(1000): key = FMT % i tree[key] = i bucket = tree._firstbucket EXPECTED = (tree._data[0].child,) for item in tree._data[1:]: EXPECTED += (item.key, item.child) self.assertEqual(tree.__getstate__(), (EXPECTED, bucket)) def test___setstate___invalid(self): tree = self._makeOne() self.assertRaises(TypeError, tree.__setstate__, ('a', 'b')) def test___setstate___to_empty(self): tree = self._makeOne({'a': 'b'}) tree.__setstate__(None) self.assertEqual(len(tree), 0) def test___setstate___to_single_bucket_wo_oid(self): tree = self._makeOne() tree.__setstate__((((('a', 'b'),),),)) self.assertEqual(list(tree.keys()), ['a']) self.assertEqual(tree._findbucket('a')['a'], 'b') self.assertTrue(len(tree._data), 1) self.assertTrue(tree._data[0].child is tree._firstbucket) self.assertTrue(tree._firstbucket._p_oid is None) def test___setstate___to_multiple_buckets(self): from .._base import Bucket class _Bucket(Bucket): def _to_key(self, x): return x tree = self._makeOne() b1 = _Bucket({'a': 0, 'b': 1}) b2 = _Bucket({'c': 2, 'd': 3}) b1._next = b2 tree.__setstate__(((b1, 'c', b2), b1)) self.assertEqual(list(tree.keys()), ['a', 'b', 'c', 'd']) self.assertTrue(len(tree._data), 2) self.assertEqual(tree._data[0].key, None) self.assertEqual(tree._data[0].child, b1) self.assertEqual(tree._data[1].key, 'c') self.assertEqual(tree._data[1].child, b2) self.assertTrue(tree._firstbucket is b1) def test__check_empty_wo_firstbucket(self): tree = self._makeOne() tree._check() # no raise def test__check_empty_w_firstbucket(self): tree = self._makeOne() tree._firstbucket = object() e = self.assertRaises(AssertionError, tree._check) self.assertEqual(str(e), "Empty BTree has non-NULL firstbucket") def test__check_nonempty_wo_firstbucket(self): tree = self._makeOne({'a': 'b'}) tree._firstbucket = None e = self.assertRaises(AssertionError, tree._check) self.assertEqual(str(e), "Non-empty BTree has NULL firstbucket") def test__check_nonempty_w_null_child(self): tree = self._makeOne({'a': 'b'}) tree._data.append(tree._data[0].__class__('c', None)) e = self.assertRaises(AssertionError, tree._check) self.assertEqual(str(e), "BTree has NULL child") def test__check_nonempty_w_heterogenous_child(self): class Other(object): pass tree = self._makeOne({'a': 'b'}) tree._data.append(tree._data[0].__class__('c', Other())) e = self.assertRaises(AssertionError, tree._check) self.assertEqual(str(e), "BTree children have different types") def test__check_nonempty_w_empty_child(self): tree = self._makeOne({'a': 'b'}) first = tree._data[0] tree._data.append(first.__class__('c', first.child.__class__())) e = self.assertRaises(AssertionError, tree._check) self.assertEqual(str(e), "Bucket length < 1") def test__check_branch_w_mismatched_firstbucket(self): tree = self._makeOne() c_tree = tree.__class__({'a': 'b'}) c_first = c_tree._data[0] tree._data.append(c_first.__class__('a', c_tree)) tree._firstbucket = object() e = self.assertRaises(AssertionError, tree._check) self.assertEqual(str(e), "BTree has firstbucket different than " "its first child's firstbucket") def test__check_nonempty_w_invalid_child(self): class Invalid(object): size = 2 tree = self._makeOne({'a': 'b'}) tree._data[0].child = Invalid() e = self.assertRaises(AssertionError, tree._check) self.assertEqual(str(e), "Incorrect child type") def test__check_branch_traverse_bucket_pointers(self): tree = self._makeOne() t_first = tree.__class__({'a': 'b'}) c_first = t_first._data[0] b_first = c_first.child t_second = tree.__class__({'c': 'd'}) b_first._next = t_second._firstbucket tree._data.append(c_first.__class__('a', t_first)) tree._data.append(c_first.__class__('c', t_second)) tree._firstbucket = t_first._firstbucket tree._check() #no raise def test__check_nonempty_leaf_traverse_bucket_pointers(self): tree = self._makeOne({'a': 'b'}) first = tree._data[0] first.child._next = b2 = first.child.__class__({'c': 'd'}) tree._data.append(first.__class__('c', b2)) tree._check() #no raise def test__p_resolveConflict_invalid_state_non_tuple(self): tree = self._makeOne() INVALID = [] EMPTY = None DEGEN = (((('a', 'b'),),),) self.assertRaises(TypeError, tree._p_resolveConflict, INVALID, EMPTY, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, INVALID, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, DEGEN, INVALID) def test__p_resolveConflict_non_degenerate_state(self): from ..Interfaces import BTreesConflictError tree = self._makeOne() FIRST = object() NON_DEGEN = ((FIRST, 'a', object(), 'b', object()), FIRST) EMPTY = None DEGEN = (((('a', 'b'),),),) e = self.assertRaises(BTreesConflictError, tree._p_resolveConflict, NON_DEGEN, EMPTY, DEGEN) self.assertEqual(e.reason, 11) e = self.assertRaises(BTreesConflictError, tree._p_resolveConflict, EMPTY, NON_DEGEN, DEGEN) self.assertEqual(e.reason, 11) e = self.assertRaises(BTreesConflictError, tree._p_resolveConflict, EMPTY, DEGEN, NON_DEGEN) self.assertEqual(e.reason, 11) def test__p_resolveConflict_invalid_state_non_1_tuple(self): tree = self._makeOne() INVALID = ('a', 'b', 'c') EMPTY = None DEGEN = (((('a', 'b'),),),) self.assertRaises(TypeError, tree._p_resolveConflict, INVALID, EMPTY, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, INVALID, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, DEGEN, INVALID) def test__p_resolveConflict_invalid_state_nested_non_tuple(self): tree = self._makeOne() INVALID = ([],) EMPTY = None DEGEN = (((('a', 'b'),),),) self.assertRaises(TypeError, tree._p_resolveConflict, INVALID, EMPTY, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, INVALID, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, DEGEN, INVALID) def test__p_resolveConflict_invalid_state_nested_non_1_tuple(self): tree = self._makeOne() INVALID = (('a', 'b', 'c'),) EMPTY = None DEGEN = (((('a', 'b'),),),) self.assertRaises(TypeError, tree._p_resolveConflict, INVALID, EMPTY, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, INVALID, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, DEGEN, INVALID) def test__p_resolveConflict_invalid_state_nested2_non_tuple(self): tree = self._makeOne() INVALID = (([],),) EMPTY = None DEGEN = (((('a', 'b'),),),) self.assertRaises(TypeError, tree._p_resolveConflict, INVALID, EMPTY, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, INVALID, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, DEGEN, INVALID) def test__p_resolveConflict_invalid_state_nested2_non_1_tuple(self): tree = self._makeOne() INVALID = ((('a', 'b', 'c'),)) EMPTY = None DEGEN = (((('a', 'b'),),),) self.assertRaises(TypeError, tree._p_resolveConflict, INVALID, EMPTY, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, INVALID, DEGEN) self.assertRaises(TypeError, tree._p_resolveConflict, EMPTY, DEGEN, INVALID) def test__p_resolveConflict_w_degenerate_state(self): tree = self._makeOne() OLD = (((('a', 'b', 'c', 'd'),),),) COM = (((('a', 'b', 'c', 'd', 'e', 'f'),),),) NEW = (((('a', 'b'),),),) resolved = tree._p_resolveConflict(OLD, COM, NEW) self.assertEqual(resolved, (((('a', 'b', 'e', 'f'),),),)) class Test_TreeItems(unittest.TestCase): assertRaises = _assertRaises def _getTargetClass(self): from .._base import _TreeItems return _TreeItems def _makeOne(self, firstbucket, itertype, iterargs): return self._getTargetClass()(firstbucket, itertype, iterargs) def _makeBucket(self, items=None): from .._base import Bucket class _Bucket(Bucket): def _to_key(self, k): return k return _Bucket(items) def test___getitem___w_slice(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] bucket = self._makeBucket(ITEMS) ti = self._makeOne(bucket, 'iterkeys', ()) self.assertEqual(list(ti[0:3]), ['a', 'b', 'c']) def test___getitem___w_negative_index_le_minus_length(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] bucket = self._makeBucket(ITEMS) ti = self._makeOne(bucket, 'iterkeys', ()) def _should_error(): return ti[-27] self.assertRaises(IndexError, _should_error) def test___getitem___w_index_gt_length(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] bucket = self._makeBucket(ITEMS) ti = self._makeOne(bucket, 'iterkeys', ()) def _should_error(): return ti[27] self.assertRaises(IndexError, _should_error) def test___getitem___w_index_smaller_than_cursor(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] bucket = self._makeBucket(ITEMS) ti = self._makeOne(bucket, 'iterkeys', ()) ti[12] self.assertEqual(ti[1], 'b') def test___len__(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] bucket = self._makeBucket(ITEMS) ti = self._makeOne(bucket, 'iterkeys', ()) self.assertEqual(len(ti), 26) # short-circuit on second pass self.assertEqual(len(ti), 26) def test___iter___w_iterkeys(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] bucket = self._makeBucket(ITEMS) ti = self._makeOne(bucket, 'iterkeys', ()) self.assertEqual(list(ti), [x[0] for x in ITEMS]) def test___iter___w_iteritems(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] bucket = self._makeBucket(ITEMS) ti = self._makeOne(bucket, 'iteritems', ()) self.assertEqual(list(ti), ITEMS) def test___iter___w_itervalues(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] bucket = self._makeBucket(ITEMS) ti = self._makeOne(bucket, 'itervalues', ()) self.assertEqual(list(ti), [x[1] for x in ITEMS]) def test___iter___w_empty_last_bucket(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] bucket1 = self._makeBucket(ITEMS) bucket2 = bucket1._next = self._makeBucket() ti = self._makeOne(bucket1, 'iterkeys', ()) self.assertEqual(list(ti), [x[0] for x in ITEMS]) class TreeTests(unittest.TestCase): assertRaises = _assertRaises def _getTargetClass(self): from .._base import Tree return Tree def _makeOne(self, items=None): from .._base import Bucket class _Bucket(Bucket): def _to_key(self, k): return k class _Test(self._getTargetClass()): _to_key = _to_value = lambda self, x: x _bucket_type = _Bucket max_leaf_size = 10 max_internal_size = 15 return _Test(items) def test_get_empty_miss(self): tree = self._makeOne() self.assertEqual(tree.get('nonesuch'), None) def test_get_empty_miss_w_default(self): DEFAULT = object() tree = self._makeOne() self.assertTrue(tree.get('nonesuch', DEFAULT) is DEFAULT) def test_get_filled_miss(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(tree.get('nonesuch'), None) def test_get_filled_miss_w_default(self): DEFAULT = object() ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertTrue(tree.get('nonesuch', DEFAULT) is DEFAULT) def test_get_filled_hit(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(tree.get('a'), 0) def test___getitem___empty_miss(self): tree = self._makeOne() def _should_error(): return tree['nonesuch'] self.assertRaises(KeyError, _should_error) def test___getitem___filled_miss(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) def _should_error(): return tree['nonesuch'] self.assertRaises(KeyError, _should_error) def test___getitem___filled_hit(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(tree['a'], 0) def test_values_empty_no_args(self): tree = self._makeOne() self.assertEqual(list(tree.values()), []) def test_values_filled_no_args(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(list(tree.values()), list(range(26))) def test_values_filled_w_args(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(list(tree.values(min='b', excludemin=True, max='f', excludemax=True)), [2, 3, 4]) def test_itervalues_empty_no_args(self): tree = self._makeOne() self.assertEqual(list(tree.itervalues()), []) def test_itervalues_filled_no_args(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(list(tree.itervalues()), list(range(26))) def test_itervalues_filled_w_args(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(list(tree.itervalues(min='b', excludemin=True, max='f', excludemax=True)), [2, 3, 4]) def test_items_empty_no_args(self): tree = self._makeOne() self.assertEqual(list(tree.items()), []) def test_items_filled_no_args(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(list(tree.items()), ITEMS) def test_items_filled_w_args(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(list(tree.items(min='b', excludemin=True, max='f', excludemax=True)), ITEMS[2:5]) def test_iteritems_empty_no_args(self): tree = self._makeOne() self.assertEqual(list(tree.iteritems()), []) def test_iteritems_filled_no_args(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(list(tree.iteritems()), ITEMS) def test_iteritems_filled_w_args(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(list(tree.iteritems(min='b', excludemin=True, max='f', excludemax=True)), ITEMS[2:5]) def test_byValue(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(list(tree.byValue(min=22)), [(y, x) for x, y in reversed(ITEMS[22:])]) def test_insert_new_key(self): tree = self._makeOne() self.assertTrue(tree.insert('a', 0)) self.assertEqual(tree['a'], 0) def test_insert_would_change_key(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertFalse(tree.insert('a', 1)) self.assertEqual(tree['a'], 0) class TreeSetTests(unittest.TestCase): assertRaises = _assertRaises def _getTargetClass(self): from .._base import TreeSet return TreeSet def _makeOne(self, items=None): from .._base import Bucket class _Bucket(Bucket): def _to_key(self, k): return k class _Test(self._getTargetClass()): _to_key = _to_value = lambda self, x: x _bucket_type = _Bucket max_leaf_size = 10 max_internal_size = 15 return _Test(items) def test_add_new_key(self): _set = self._makeOne() self.assertTrue(_set.add('a')) self.assertTrue('a' in _set) def test_add_existing_key(self): _set = self._makeOne() _set.add('a') self.assertFalse(_set.add('a')) def test_remove_miss(self): _set = self._makeOne() self.assertRaises(KeyError, _set.remove, 'a') def test_remove_hit(self): _set = self._makeOne() _set.add('a') self.assertEqual(_set.remove('a'), None) self.assertFalse('a' in _set) def test_update_empty_sequence(self): _set = self._makeOne() _set.update(()) self.assertEqual(len(_set), 0) def test_update_simple_sequence(self): _set = self._makeOne() LETTERS = 'abcdefghijklmnopqrstuvwxyz' _set.update(LETTERS) self.assertEqual(len(_set), len(LETTERS)) for letter in LETTERS: self.assertTrue(letter in _set) def test_update_mppaing(self): _set = self._makeOne() LETTERS = 'abcdefghijklmnopqrstuvwxyz' a_dict = dict([(y, x) for x, y in enumerate(LETTERS)]) _set.update(a_dict) self.assertEqual(len(_set), len(LETTERS)) for letter in LETTERS: self.assertTrue(letter in _set) class Test_set_operation(unittest.TestCase): assertRaises = _assertRaises def _getTargetClass(self): from .._base import set_operation return set_operation def _makeOne(self, func, set_type): return self._getTargetClass()(func, set_type) def test_it(self): class _SetType(object): pass _called_with = [] def _func(*args, **kw): _called_with.append((args, kw)) set_op = self._makeOne(_func, _SetType) set_op('a', b=1) self.assertEqual(_called_with, [((_SetType, 'a',), {'b': 1})]) class _SetObBase(object): def _makeSet(self, *args): return _Set(*args) def _makeMapping(self, *args, **kw): return _Mapping(*args, **kw) class Test_difference(unittest.TestCase, _SetObBase): def _callFUT(self, *args, **kw): from .._base import difference return difference(*args, **kw) def test_lhs_none(self): rhs = self._makeSet('a', 'b', 'c') self.assertEqual(self._callFUT(rhs.__class__, None, rhs), None) def test_rhs_none(self): lhs = self._makeSet('a', 'b', 'c') self.assertEqual(self._callFUT(lhs.__class__, lhs, None), lhs) def test_both_sets_rhs_empty(self): lhs = self._makeSet('a', 'b', 'c') rhs = self._makeSet() result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), list(lhs)) def test_both_sets_lhs_empty(self): lhs = self._makeSet() rhs = self._makeSet('a', 'b', 'c') result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), list(lhs)) def test_lhs_set_rhs_mapping(self): lhs = self._makeSet('a', 'b', 'c') rhs = self._makeMapping({'a': 13, 'b': 12}) result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), ['c']) def test_lhs_mapping_rhs_set(self): lhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) rhs = self._makeSet('a', 'b') result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), ['c']) self.assertEqual(result['c'], 11) def test_both_mappings_rhs_empty(self): lhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) rhs = self._makeMapping({}) result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), ['a', 'b', 'c']) self.assertEqual(result['a'], 13) self.assertEqual(result['b'], 12) self.assertEqual(result['c'], 11) def test_both_mappings_rhs_non_empty(self): lhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11, 'f': 10}) rhs = self._makeMapping({'b': 22, 'e': 37}) result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), ['a', 'c', 'f']) self.assertEqual(result['a'], 13) self.assertEqual(result['c'], 11) self.assertEqual(result['f'], 10) class Test_union(unittest.TestCase, _SetObBase): def _callFUT(self, *args, **kw): from .._base import union return union(*args, **kw) def test_lhs_none(self): rhs = self._makeSet('a', 'b', 'c') self.assertEqual(self._callFUT(rhs.__class__, None, rhs), rhs) def test_rhs_none(self): lhs = self._makeSet('a', 'b', 'c') self.assertEqual(self._callFUT(lhs.__class__, lhs, None), lhs) def test_both_sets_rhs_empty(self): lhs = self._makeSet('a', 'b', 'c') rhs = self._makeSet() result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), list(lhs)) def test_both_sets_lhs_empty(self): lhs = self._makeSet() rhs = self._makeSet('a', 'b', 'c') result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), list(rhs)) def test_lhs_set_rhs_mapping(self): lhs = self._makeSet('a', 'b', 'c') rhs = self._makeMapping({'a': 13, 'd': 12}) result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), ['a', 'b', 'c', 'd']) def test_lhs_mapping_rhs_set(self): lhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) rhs = self._makeSet('a', 'd') result = self._callFUT(lhs.__class__, lhs, rhs) self.assertTrue(isinstance(result, _Set)) self.assertEqual(list(result), ['a', 'b', 'c', 'd']) def test_both_mappings_rhs_empty(self): lhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) rhs = self._makeMapping({}) result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), ['a', 'b', 'c']) def test_both_mappings_rhs_non_empty(self): lhs = self._makeMapping({'a': 13, 'c': 12, 'e': 11}) rhs = self._makeMapping({'b': 22, 'd': 33}) result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), ['a', 'b', 'c', 'd', 'e']) class Test_intersection(unittest.TestCase, _SetObBase): def _callFUT(self, *args, **kw): from .._base import intersection return intersection(*args, **kw) def test_lhs_none(self): rhs = self._makeSet(('a', 'b', 'c')) self.assertEqual(self._callFUT(rhs.__class__, None, rhs), rhs) def test_rhs_none(self): lhs = self._makeSet(('a', 'b', 'c')) self.assertEqual(self._callFUT(lhs.__class__, lhs, None), lhs) def test_both_sets_rhs_empty(self): lhs = self._makeSet('a', 'b', 'c') rhs = self._makeSet() result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), []) def test_both_sets_lhs_empty(self): lhs = self._makeSet() rhs = self._makeSet('a', 'b', 'c') result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), []) def test_lhs_set_rhs_mapping(self): lhs = self._makeSet('a', 'b', 'c') rhs = self._makeMapping({'a': 13, 'd': 12}) result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), ['a']) def test_lhs_mapping_rhs_set(self): lhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) rhs = self._makeSet('a', 'd') result = self._callFUT(lhs.__class__, lhs, rhs) self.assertTrue(isinstance(result, _Set)) self.assertEqual(list(result), ['a']) def test_both_mappings_rhs_empty(self): lhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) rhs = self._makeMapping({}) result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), []) def test_both_mappings_rhs_non_empty(self): lhs = self._makeMapping({'a': 13, 'c': 12, 'e': 11}) rhs = self._makeMapping({'b': 22, 'c': 44, 'd': 33}) result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(list(result), ['c']) class Test_weightedUnion(unittest.TestCase, _SetObBase): def _callFUT(self, *args, **kw): from .._base import weightedUnion return weightedUnion(*args, **kw) def test_both_none(self): self.assertEqual(self._callFUT(_Mapping, None, None), (0, None)) def test_lhs_none(self): rhs = self._makeMapping({'a': 13, 'c': 12, 'e': 11}) self.assertEqual(self._callFUT(rhs.__class__, None, rhs), (1, rhs)) def test_rhs_none(self): lhs = self._makeMapping({'a': 13, 'c': 12, 'e': 11}) self.assertEqual(self._callFUT(lhs.__class__, lhs, None), (1, lhs)) def test_both_mappings_but_no_merge(self): lhs = {'a': 13, 'b': 12, 'c': 11} rhs = {'b': 22, 'd': 14} self.assertRaises(TypeError, self._callFUT, lhs.__class__, lhs, rhs) def test_lhs_set_wo_MERGE_DEFAULT_rhs_set(self): lhs = self._makeSet('a', 'd') lhs.MERGE = lambda v1, w1, v2, w2: (v1 * w1) + (v2 * w2) lhs.MERGE_WEIGHT = lambda v, w: v lhs._mapping_type = _Mapping rhs = self._makeSet('a', 'b', 'c') self.assertRaises(TypeError, self._callFUT, lhs.__class__, lhs, rhs) def test_lhs_mapping_wo_MERGE_DEFAULT_rhs_set(self): class _MappingWoDefault(dict): def MERGE(self, v1, w1, v2, w2): return (v1 * w1) + (v2 * w2) def MERGE_WEIGHT(self, v, w): return v lhs = _MappingWoDefault({'a': 13, 'b': 12, 'c': 11}) lhs._mapping_type = _MappingWoDefault rhs = self._makeSet('a', 'b', 'c') self.assertRaises(TypeError, self._callFUT, lhs.__class__, lhs, rhs) def test_lhs_mapping_wo_MERGE_rhs_mapping(self): class _MappingWoMerge(dict): def MERGE_DEFAULT(self): return 1 def MERGE_WEIGHT(self, v, w): return v lhs = _MappingWoMerge({'a': 13, 'b': 12, 'c': 11}) lhs._mapping_type = _MappingWoMerge rhs = self._makeMapping({'a': 1, 'b': 2, 'c': 3}) self.assertRaises(TypeError, self._callFUT, lhs.__class__, lhs, rhs) def test_lhs_set_wo_MERGE_DEFAULT_rhs_mapping(self): lhs = self._makeSet('a', 'd') lhs.MERGE = lambda v1, w1, v2, w2: (v1 * w1) + (v2 * w2) lhs.MERGE_WEIGHT = lambda v, w: v lhs._mapping_type = _Mapping rhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) self.assertRaises(TypeError, self._callFUT, lhs.__class__, lhs, rhs) def test_lhs_mergeable_set_rhs_mapping(self): lhs = self._makeSet('a', 'd') lhs.MERGE = lambda v1, w1, v2, w2: (v1 * w1) + (v2 * w2) lhs.MERGE_WEIGHT = lambda v, w: v lhs.MERGE_DEFAULT = 1 lhs._mapping_type = _Mapping rhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) weight, result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(weight, 1) self.assertTrue(isinstance(result, _Mapping)) self.assertEqual(list(result), ['a', 'b', 'c', 'd']) self.assertEqual(result['a'], 14) self.assertEqual(result['b'], 12) self.assertEqual(result['c'], 11) self.assertEqual(result['d'], 1) def test_lhs_mapping_rhs_set(self): lhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) rhs = self._makeSet('a', 'd') weight, result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(weight, 1) self.assertTrue(isinstance(result, _Mapping)) self.assertEqual(list(result), ['a', 'b', 'c', 'd']) self.assertEqual(result['a'], 55) self.assertEqual(result['b'], 12) self.assertEqual(result['c'], 11) self.assertEqual(result['d'], 42) def test_both_mappings_rhs_empty(self): lhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) rhs = self._makeMapping({}) weight, result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(weight, 1) self.assertEqual(list(result), ['a', 'b', 'c']) self.assertEqual(result['a'], 13) self.assertEqual(result['b'], 12) self.assertEqual(result['c'], 11) def test_both_mappings_rhs_non_empty(self): lhs = self._makeMapping({'a': 13, 'c': 12, 'e': 11}) rhs = self._makeMapping({'a': 10, 'b': 22, 'd': 33}) weight, result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(weight, 1) self.assertEqual(list(result), ['a', 'b', 'c', 'd', 'e']) self.assertEqual(result['a'], 23) self.assertEqual(result['b'], 22) self.assertEqual(result['c'], 12) self.assertEqual(result['d'], 33) self.assertEqual(result['e'], 11) def test_w_lhs_Set_rhs_Set(self): from BTrees.IIBTree import IISetPy lhs = IISetPy([1, 2, 3]) rhs = IISetPy([1, 4]) weight, result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(weight, 1) self.assertEqual(list(result), [1, 2, 3, 4]) #TODO: test non-default weights class Test_weightedIntersection(unittest.TestCase, _SetObBase): def _callFUT(self, *args, **kw): from .._base import weightedIntersection return weightedIntersection(*args, **kw) def test_both_none(self): self.assertEqual(self._callFUT(_Mapping, None, None), (0, None)) def test_lhs_none(self): rhs = self._makeMapping({'a': 13, 'c': 12, 'e': 11}) self.assertEqual(self._callFUT(rhs.__class__, None, rhs), (1, rhs)) def test_rhs_none(self): lhs = self._makeMapping({'a': 13, 'c': 12, 'e': 11}) self.assertEqual(self._callFUT(lhs.__class__, lhs, None), (1, lhs)) def test_both_mappings_but_no_merge(self): lhs = {'a': 13, 'b': 12, 'c': 11} rhs = {'b': 22, 'd': 14} self.assertRaises(TypeError, self._callFUT, lhs.__class__, lhs, rhs) def test_lhs_mapping_wo_MERGE_rhs_mapping(self): class _MappingWoMerge(dict): def MERGE_DEFAULT(self): return 1 def MERGE_WEIGHT(self, v, w): return v lhs = _MappingWoMerge({'a': 13, 'b': 12, 'c': 11}) lhs._mapping_type = _MappingWoMerge rhs = self._makeMapping({'a': 1, 'b': 2, 'c': 3}) self.assertRaises(TypeError, self._callFUT, lhs.__class__, lhs, rhs) def test_lhs_set_wo_MERGE_DEFAULT_rhs_set(self): lhs = self._makeSet('a', 'd') lhs.MERGE = lambda v1, w1, v2, w2: (v1 * w1) + (v2 * w2) lhs.MERGE_WEIGHT = lambda v, w: v lhs._mapping_type = _Mapping rhs = self._makeSet('a', 'b', 'c') self.assertRaises(TypeError, self._callFUT, lhs.__class__, lhs, rhs) def test_lhs_set_wo_MERGE_DEFAULT_rhs_mapping(self): lhs = self._makeSet('a', 'd') lhs.MERGE = lambda v1, w1, v2, w2: (v1 * w1) + (v2 * w2) lhs.MERGE_WEIGHT = lambda v, w: v lhs._mapping_type = _Mapping rhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) self.assertRaises(TypeError, self._callFUT, lhs.__class__, lhs, rhs) def test_lhs_mapping_rhs_set(self): lhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) rhs = self._makeSet('a', 'd') weight, result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(weight, 1) self.assertTrue(isinstance(result, _Mapping)) self.assertEqual(list(result), ['a']) self.assertEqual(result['a'], 55) def test_both_mappings_rhs_empty(self): lhs = self._makeMapping({'a': 13, 'b': 12, 'c': 11}) rhs = self._makeMapping({}) weight, result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(weight, 1) self.assertEqual(list(result), []) def test_both_mappings_rhs_non_empty(self): lhs = self._makeMapping({'a': 13, 'c': 12, 'e': 11}) rhs = self._makeMapping({'a': 10, 'b': 22, 'd': 33}) weight, result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(weight, 1) self.assertEqual(list(result), ['a']) self.assertEqual(result['a'], 23) def test_w_lhs_Set_rhs_Set(self): from BTrees.IIBTree import IISetPy lhs = IISetPy([1, 2, 3]) rhs = IISetPy([1, 4]) weight, result = self._callFUT(lhs.__class__, lhs, rhs) self.assertEqual(weight, 2) self.assertEqual(list(result), [1]) #TODO: test non-default weights class Test_multiunion(unittest.TestCase, _SetObBase): def _callFUT(self, *args, **kw): from .._base import multiunion return multiunion(*args, **kw) def test_no_seqs(self): result = self._callFUT(_Set, ()) self.assertEqual(list(result), []) def test_w_non_iterable_seq(self): result = self._callFUT(_Set, (1, 2)) self.assertEqual(list(result), [1, 2]) def test_w_iterable_seqs(self): result = self._callFUT(_Set, [(1,), (2,)]) self.assertEqual(list(result), [1, 2]) def test_w_mix(self): result = self._callFUT(_Set, [1, (2,)]) self.assertEqual(list(result), [1, 2]) class Test_helpers(unittest.TestCase): def test_to_ob(self): from BTrees._base import to_ob faux_self = object() for thing in "abc", 0, 1.3, (), frozenset((1, 2)), object(): self.assertTrue(to_ob(faux_self, thing) is thing) def test_to_int_w_int(self): from BTrees._base import to_int faux_self = object() self.assertEqual(to_int(faux_self, 3), 3) def test_to_int_w_long_in_range(self): from BTrees._base import to_int faux_self = object() try: self.assertEqual(to_int(faux_self, long(3)), 3) except NameError: #Python3 pass def test_to_int_w_overflow(self): from BTrees._base import to_int faux_self = object() self.assertRaises(TypeError, to_int, faux_self, 2**64) def test_to_int_w_invalid(self): from BTrees._base import to_int faux_self = object() self.assertRaises(TypeError, to_int, faux_self, ()) def test_to_float_w_float(self): from BTrees._base import to_float faux_self = object() self.assertEqual(to_float(faux_self, 3.14159), 3.14159) def test_to_float_w_int(self): from BTrees._base import to_float faux_self = object() self.assertEqual(to_float(faux_self, 3), 3.0) def test_to_float_w_invalid(self): from BTrees._base import to_float faux_self = object() self.assertRaises(TypeError, to_float, faux_self, ()) def test_to_long_w_int(self): from BTrees._base import to_long faux_self = object() self.assertEqual(to_long(faux_self, 3), 3) def test_to_long_w_long_in_range(self): from BTrees._base import to_long faux_self = object() try: self.assertEqual(to_long(faux_self, long(3)), 3) except NameError: #Python3 pass def test_to_long_w_overflow(self): from BTrees._base import to_long faux_self = object() self.assertRaises(ValueError, to_long, faux_self, 2**64) def test_to_long_w_invalid(self): from BTrees._base import to_long faux_self = object() self.assertRaises(TypeError, to_long, faux_self, ()) def test_to_bytes_w_ok(self): from BTrees._base import to_bytes faux_self = object() conv = to_bytes(3) self.assertEqual(conv(faux_self, b'abc'), b'abc') def test_to_bytes_w_invalid_length(self): from BTrees._base import to_bytes faux_self = object() conv = to_bytes(3) self.assertRaises(TypeError, conv, faux_self, b'ab') self.assertRaises(TypeError, conv, faux_self, b'abcd') def test_MERGE(self): from BTrees._base import MERGE faux_self = object() self.assertEqual(MERGE(faux_self, 1, 1, 1, 1), 2) self.assertEqual(MERGE(faux_self, 1, 2, 1, 3), 5) def test_MERGE_WEIGHT_default(self): from BTrees._base import MERGE_WEIGHT_default faux_self = object() self.assertEqual(MERGE_WEIGHT_default(faux_self, 1, 17), 1) self.assertEqual(MERGE_WEIGHT_default(faux_self, 7, 1), 7) def test_MERGE_WEIGHT_numeric(self): from BTrees._base import MERGE_WEIGHT_numeric faux_self = object() self.assertEqual(MERGE_WEIGHT_numeric(faux_self, 1, 17), 17) self.assertEqual(MERGE_WEIGHT_numeric(faux_self, 7, 1), 7) class _Cache(object): def __init__(self): self._mru = [] def mru(self, oid): self._mru.append(oid) class _Jar(object): def __init__(self): self._current = set() self._cache = _Cache() def readCurrent(self, obj): self._current.add(obj) def register(self, obj): pass class _Set(object): def __init__(self, *args, **kw): if len(args) == 1 and isinstance(args[0], tuple): keys = args[0] else: keys = set(args) self._keys = sorted(keys) def keys(self): return self._keys def __iter__(self): return iter(self._keys) def update(self, items): self._keys = sorted(self._keys + list(items)) _Set._set_type = _Set class _Mapping(dict): def __init__(self, source=None): if source is None: source = {} self._keys = [] self._values = [] for k, v in sorted(source.items()): self._keys.append(k) self._values.append(v) MERGE_DEFAULT = 42 def MERGE_WEIGHT(self, v, w): return v def MERGE(self, v1, w1, v2, w2): return v1 * w1 + v2 * w2 def iteritems(self): for k, v in zip(self._keys, self._values): yield k,v def __iter__(self): return iter(self._keys) def __getitem__(self, key): search = dict(zip(self._keys, self._values)) return search[key] def __repr__(self): return repr(dict(zip(self._keys, self._values))) _Mapping._set_type = _Set _Mapping._mapping_type = _Mapping def test_suite(): return unittest.TestSuite(( unittest.makeSuite(Test_Base), unittest.makeSuite(Test_BucketBase), unittest.makeSuite(Test_SetIteration), unittest.makeSuite(BucketTests), unittest.makeSuite(SetTests), unittest.makeSuite(Test_TreeItem), unittest.makeSuite(Test_Tree), unittest.makeSuite(Test_TreeItems), unittest.makeSuite(TreeTests), unittest.makeSuite(TreeSetTests), unittest.makeSuite(Test_set_operation), unittest.makeSuite(Test_difference), unittest.makeSuite(Test_union), unittest.makeSuite(Test_intersection), unittest.makeSuite(Test_weightedUnion), unittest.makeSuite(Test_weightedIntersection), unittest.makeSuite(Test_multiunion), unittest.makeSuite(Test_helpers), )) BTrees-4.3.1/BTrees/tests/test_Length.py0000664000175000017500000000665612510775554021261 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2008 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest _marker = object() class LengthTestCase(unittest.TestCase): def _getTargetClass(self): from BTrees.Length import Length return Length def _makeOne(self, value=_marker): if value is _marker: return self._getTargetClass()() return self._getTargetClass()(value) def test_ctor_default(self): length = self._makeOne() self.assertEqual(length.value, 0) def test_ctor_explict(self): length = self._makeOne(42) self.assertEqual(length.value, 42) def test___getstate___(self): length = self._makeOne(42) self.assertEqual(length.__getstate__(), 42) def test___setstate__(self): length = self._makeOne() length.__setstate__(42) self.assertEqual(length.value, 42) def test_set(self): length = self._makeOne() length.set(42) self.assertEqual(length.value, 42) def test__p_resolveConflict(self): length = self._makeOne() self.assertEqual(length._p_resolveConflict(5, 7, 9), 11) def test_change_w_positive_delta(self): length = self._makeOne() length.change(3) self.assertEqual(length.value, 3) def test_change_w_negative_delta(self): length = self._makeOne() length.change(-3) self.assertEqual(length.value, -3) def test_change_overflows_to_long(self): import sys try: length = self._makeOne(sys.maxint) except AttributeError: #pragma NO COVER Py3k return else: #pragma NO COVER Py2 self.assertEqual(length(), sys.maxint) self.assertTrue(type(length()) is int) length.change(+1) self.assertEqual(length(), sys.maxint + 1) self.assertTrue(type(length()) is long) def test_change_underflows_to_long(self): import sys try: minint = (-sys.maxint) - 1 except AttributeError: #pragma NO COVER Py3k return else: #pragma NO COVER Py2 length = self._makeOne(minint) self.assertEqual(length(), minint) self.assertTrue(type(length()) is int) length.change(-1) self.assertEqual(length(), minint - 1) self.assertTrue(type(length()) is long) def test___call___no_args(self): length = self._makeOne(42) self.assertEqual(length(), 42) def test___call___w_args(self): length = self._makeOne(42) self.assertEqual(length(0, None, (), [], {}), 42) def test_lp_516653(self): # Test for https://bugs.launchpad.net/zodb/+bug/516653 import copy length = self._makeOne() other = copy.copy(length) self.assertEqual(other(), 0) def test_suite(): return unittest.makeSuite(LengthTestCase) BTrees-4.3.1/BTrees/tests/test_IOBTree.py0000664000175000017500000002566312510775554021270 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from .common import BTreeTests from .common import ExtendedSetTests from .common import I_SetsBase from .common import InternalKeysMappingTest from .common import InternalKeysSetTest from .common import MappingBase from .common import MappingConflictTestBase from .common import ModuleTest from .common import MultiUnion from .common import NormalSetTests from .common import SetConflictTestBase from .common import SetResult from .common import TypeTest from .common import TestLongIntKeys from .common import makeBuilder from BTrees.IIBTree import using64bits #XXX Ugly, but unavoidable class IOBTreeInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOBTree return IOBTree class IOBTreePyInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOBTreePy return IOBTreePy class IOTreeSetInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOTreeSet return IOTreeSet class IOTreeSetPyInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOTreeSetPy return IOTreeSetPy class IOBucketTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOBucket return IOBucket class IOBucketPyTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOBucketPy return IOBucketPy class IOTreeSetTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOTreeSet return IOTreeSet class IOTreeSetPyTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOTreeSetPy return IOTreeSetPy class IOSetTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOSet return IOSet class IOSetPyTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOSetPy return IOSetPy class IOBTreeTest(BTreeTests, unittest.TestCase): def _makeOne(self): from BTrees.IOBTree import IOBTree return IOBTree() class IOBTreePyTest(BTreeTests, unittest.TestCase): def _makeOne(self): from BTrees.IOBTree import IOBTreePy return IOBTreePy() if using64bits: class IOBTreeTest(BTreeTests, TestLongIntKeys, unittest.TestCase): def _makeOne(self): from BTrees.IOBTree import IOBTree return IOBTree() class IOBTreePyTest(BTreeTests, TestLongIntKeys, unittest.TestCase): def _makeOne(self): from BTrees.IOBTree import IOBTreePy return IOBTreePy() class _TestIOBTreesBase(TypeTest): def _stringraises(self): self._makeOne()['c'] = 1 def _floatraises(self): self._makeOne()[2.5] = 1 def _noneraises(self): self._makeOne()[None] = 1 class TestIOBTrees(_TestIOBTreesBase, unittest.TestCase): def _makeOne(self): from BTrees.IOBTree import IOBTree return IOBTree() class TestIOBTreesPy(_TestIOBTreesBase, unittest.TestCase): def _makeOne(self): from BTrees.IOBTree import IOBTreePy return IOBTreePy() class TestIOSets(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.IOBTree import IOSet return IOSet() class TestIOSetsPy(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.IOBTree import IOSetPy return IOSetPy() class TestIOTreeSets(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.IOBTree import IOTreeSet return IOTreeSet() class TestIOTreeSetsPy(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.IOBTree import IOTreeSetPy return IOTreeSetPy() class PureIO(SetResult, unittest.TestCase): def union(self, *args): from BTrees.IOBTree import union return union(*args) def intersection(self, *args): from BTrees.IOBTree import intersection return intersection(*args) def difference(self, *args): from BTrees.IOBTree import difference return difference(*args) def builders(self): from BTrees.IOBTree import IOBTree from BTrees.IOBTree import IOBucket from BTrees.IOBTree import IOTreeSet from BTrees.IOBTree import IOSet return IOSet, IOTreeSet, makeBuilder(IOBTree), makeBuilder(IOBucket) class PureIOPy(SetResult, unittest.TestCase): def union(self, *args): from BTrees.IOBTree import unionPy return unionPy(*args) def intersection(self, *args): from BTrees.IOBTree import intersectionPy return intersectionPy(*args) def difference(self, *args): from BTrees.IOBTree import differencePy return differencePy(*args) def builders(self): from BTrees.IOBTree import IOBTreePy from BTrees.IOBTree import IOBucketPy from BTrees.IOBTree import IOTreeSetPy from BTrees.IOBTree import IOSetPy return (IOSetPy, IOTreeSetPy, makeBuilder(IOBTreePy), makeBuilder(IOBucketPy)) class TestIOMultiUnion(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.IOBTree import multiunion return multiunion(*args) def union(self, *args): from BTrees.IOBTree import union return union(*args) def mkset(self, *args): from BTrees.IOBTree import IOSet as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.IOBTree import IOTreeSet as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.IOBTree import IOBucket as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.IOBTree import IOBTree as mkbtree return mkbtree(*args) class TestIOMultiUnionPy(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.IOBTree import multiunionPy return multiunionPy(*args) def union(self, *args): from BTrees.IOBTree import unionPy return unionPy(*args) def mkset(self, *args): from BTrees.IOBTree import IOSetPy as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.IOBTree import IOTreeSetPy as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.IOBTree import IOBucketPy as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.IOBTree import IOBTreePy as mkbtree return mkbtree(*args) class IOBTreeConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOBTree return IOBTree class IOBTreeConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOBTreePy return IOBTreePy class IOBucketConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOBucket return IOBucket class IOBucketConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOBucketPy return IOBucketPy class IOTreeSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOTreeSet return IOTreeSet class IOTreeSetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOTreeSetPy return IOTreeSetPy class IOSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOSet return IOSet class IOSetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IOBTree import IOSetPy return IOSetPy class IOModuleTest(ModuleTest, unittest.TestCase): prefix = 'IO' def _getModule(self): import BTrees return BTrees.IOBTree def _getInterface(self): import BTrees.Interfaces return BTrees.Interfaces.IIntegerObjectBTreeModule def test_weightedUnion_not_present(self): try: from BTrees.IOBTree import weightedUnion except ImportError: pass else: self.fail("IOBTree shouldn't have weightedUnion") def test_weightedIntersection_not_present(self): try: from BTrees.IOBTree import weightedIntersection except ImportError: pass else: self.fail("IOBTree shouldn't have weightedIntersection") def test_suite(): return unittest.TestSuite(( unittest.makeSuite(IOBTreeInternalKeyTest), unittest.makeSuite(IOBTreePyInternalKeyTest), unittest.makeSuite(IOTreeSetInternalKeyTest), unittest.makeSuite(IOTreeSetPyInternalKeyTest), unittest.makeSuite(IOBucketTest), unittest.makeSuite(IOBucketPyTest), unittest.makeSuite(IOTreeSetTest), unittest.makeSuite(IOTreeSetPyTest), unittest.makeSuite(IOSetTest), unittest.makeSuite(IOSetPyTest), unittest.makeSuite(IOBTreeTest), unittest.makeSuite(IOBTreePyTest), unittest.makeSuite(TestIOBTrees), unittest.makeSuite(TestIOBTreesPy), unittest.makeSuite(TestIOSets), unittest.makeSuite(TestIOSetsPy), unittest.makeSuite(TestIOTreeSets), unittest.makeSuite(TestIOTreeSetsPy), unittest.makeSuite(TestIOMultiUnion), unittest.makeSuite(TestIOMultiUnionPy), unittest.makeSuite(PureIO), unittest.makeSuite(PureIOPy), unittest.makeSuite(IOBTreeConflictTests), unittest.makeSuite(IOBTreeConflictTestsPy), unittest.makeSuite(IOBucketConflictTests), unittest.makeSuite(IOBucketConflictTestsPy), unittest.makeSuite(IOTreeSetConflictTests), unittest.makeSuite(IOTreeSetConflictTestsPy), unittest.makeSuite(IOSetConflictTests), unittest.makeSuite(IOSetConflictTestsPy), unittest.makeSuite(IOModuleTest), )) BTrees-4.3.1/BTrees/tests/test_fsBTree.py0000664000175000017500000000413012510775554021353 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2010 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest class fsBucketBase(object): def _makeOne(self, *args, **kw): return self._getTargetClass()(*args, **kw) def _makeBytesItems(self): from .._compat import _ascii return[(_ascii(c*2), _ascii(c*6)) for c in 'abcdef'] def test_toString(self): bucket = self._makeOne(self._makeBytesItems()) self.assertEqual(bucket.toString(), b'aabbccddeeffaaaaaabbbbbbccccccddddddeeeeeeffffff') def test_fromString(self): before = self._makeOne(self._makeBytesItems()) after = before.fromString(before.toString()) self.assertEqual(before.__getstate__(), after.__getstate__()) def test_fromString_empty(self): before = self._makeOne(self._makeBytesItems()) after = before.fromString(b'') self.assertEqual(after.__getstate__(), ((),)) def test_fromString_invalid_length(self): bucket = self._makeOne(self._makeBytesItems()) self.assertRaises(ValueError, bucket.fromString, b'xxx') class fsBucketTests(unittest.TestCase, fsBucketBase): def _getTargetClass(self): from BTrees.fsBTree import fsBucket return fsBucket class fsBucketPyTests(unittest.TestCase, fsBucketBase): def _getTargetClass(self): from BTrees.fsBTree import fsBucketPy return fsBucketPy def test_suite(): return unittest.TestSuite(( unittest.makeSuite(fsBucketTests), unittest.makeSuite(fsBucketPyTests), )) BTrees-4.3.1/BTrees/tests/__init__.py0000664000175000017500000000006412510775554020523 0ustar tseavertseaver00000000000000# If tests is a package, debugging is a bit easier. BTrees-4.3.1/BTrees/tests/test_OLBTree.py0000664000175000017500000002406512510775554021266 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from .common import BTreeTests from .common import ExtendedSetTests from .common import InternalKeysMappingTest from .common import InternalKeysSetTest from .common import MappingBase from .common import MappingConflictTestBase from .common import ModuleTest from .common import NormalSetTests from .common import SetConflictTestBase from .common import SetResult from .common import TestLongIntValues from .common import Weighted from .common import itemsToSet from .common import makeBuilder from .common import _skip_on_32_bits class OLBTreeInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLBTree return OLBTree class OLBTreePyInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLBTreePy return OLBTreePy class OLTreeSetInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLTreeSet return OLTreeSet class OLTreeSetPyInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLTreeSetPy return OLTreeSetPy class OLBucketTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLBucket return OLBucket class OLBucketPyTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLBucketPy return OLBucketPy class OLTreeSetTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLTreeSet return OLTreeSet class OLTreeSetPyTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLTreeSetPy return OLTreeSetPy class OLSetTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLSet return OLSet class OLSetPyTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLSetPy return OLSetPy class OLBTreeTest(BTreeTests, TestLongIntValues, unittest.TestCase): def _makeOne(self): from BTrees.OLBTree import OLBTree return OLBTree() def getTwoKeys(self): return "abc", "def" @_skip_on_32_bits def test_extremes(self): from BTrees.tests.common import SMALLEST_64_BITS from BTrees.tests.common import SMALLEST_POSITIVE_65_BITS from BTrees.tests.common import LARGEST_64_BITS from BTrees.tests.common import LARGEST_NEGATIVE_65_BITS btree = self._makeOne() btree['ZERO'] = 0 btree['SMALLEST_64_BITS'] = SMALLEST_64_BITS btree['LARGEST_64_BITS'] = LARGEST_64_BITS self.assertRaises((ValueError, OverflowError), btree.__setitem__, 'SMALLEST_POSITIVE_65_BITS', SMALLEST_POSITIVE_65_BITS) self.assertRaises((ValueError, OverflowError), btree.__setitem__, 'LARGEST_NEGATIVE_65_BITS', LARGEST_NEGATIVE_65_BITS) class OLBTreePyTest(BTreeTests, TestLongIntValues, unittest.TestCase): def _makeOne(self): from BTrees.OLBTree import OLBTreePy return OLBTreePy() def getTwoKeys(self): return "abc", "def" class PureOL(SetResult, unittest.TestCase): def union(self, *args): from BTrees.OLBTree import union return union(*args) def intersection(self, *args): from BTrees.OLBTree import intersection return intersection(*args) def difference(self, *args): from BTrees.OLBTree import difference return difference(*args) def builders(self): from BTrees.OLBTree import OLBTree from BTrees.OLBTree import OLBucket from BTrees.OLBTree import OLTreeSet from BTrees.OLBTree import OLSet return OLSet, OLTreeSet, makeBuilder(OLBTree), makeBuilder(OLBucket) class PureOLPy(SetResult, unittest.TestCase): def union(self, *args): from BTrees.OLBTree import unionPy return unionPy(*args) def intersection(self, *args): from BTrees.OLBTree import intersectionPy return intersectionPy(*args) def difference(self, *args): from BTrees.OLBTree import differencePy return differencePy(*args) def builders(self): from BTrees.OLBTree import OLBTreePy from BTrees.OLBTree import OLBucketPy from BTrees.OLBTree import OLTreeSetPy from BTrees.OLBTree import OLSetPy return (OLSetPy, OLTreeSetPy, makeBuilder(OLBTreePy), makeBuilder(OLBucketPy)) class TestWeightedOL(Weighted, unittest.TestCase): def weightedUnion(self): from BTrees.OLBTree import weightedUnion return weightedUnion def weightedIntersection(self): from BTrees.OLBTree import weightedIntersection return weightedIntersection def union(self): from BTrees.OLBTree import union return union def intersection(self): from BTrees.OLBTree import intersection return intersection def mkbucket(self, *args): from BTrees.OLBTree import OLBucket as mkbucket return mkbucket(*args) def builders(self): from BTrees.OLBTree import OLBTree from BTrees.OLBTree import OLBucket from BTrees.OLBTree import OLTreeSet from BTrees.OLBTree import OLSet return OLBucket, OLBTree, itemsToSet(OLSet), itemsToSet(OLTreeSet) class TestWeightedOLPy(Weighted, unittest.TestCase): def weightedUnion(self): from BTrees.OLBTree import weightedUnionPy return weightedUnionPy def weightedIntersection(self): from BTrees.OLBTree import weightedIntersectionPy return weightedIntersectionPy def union(self): from BTrees.OLBTree import unionPy return unionPy def intersection(self): from BTrees.OLBTree import intersectionPy return intersectionPy def mkbucket(self, *args): from BTrees.OLBTree import OLBucketPy as mkbucket return mkbucket(*args) def builders(self): from BTrees.OLBTree import OLBTreePy from BTrees.OLBTree import OLBucketPy from BTrees.OLBTree import OLTreeSetPy from BTrees.OLBTree import OLSetPy return (OLBucketPy, OLBTreePy, itemsToSet(OLSetPy), itemsToSet(OLTreeSetPy)) class OLBucketConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLBucket return OLBucket class OLBucketPyConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLBucketPy return OLBucketPy class OLSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLSet return OLSet class OLSetPyConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLSetPy return OLSetPy class OLBTreeConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLBTree return OLBTree class OLBTreePyConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLBTreePy return OLBTreePy class OLTreeSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLTreeSet return OLTreeSet class OLTreeSetPyConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OLBTree import OLTreeSetPy return OLTreeSetPy class OLModuleTest(ModuleTest, unittest.TestCase): prefix = 'OL' def _getModule(self): import BTrees return BTrees.OLBTree def _getInterface(self): import BTrees.Interfaces return BTrees.Interfaces.IObjectIntegerBTreeModule def test_multiunion_not_present(self): try: from BTrees.OLBTree import multiunion except ImportError: pass else: self.fail("OLBTree shouldn't have multiunion") def test_suite(): return unittest.TestSuite(( unittest.makeSuite(OLBTreeInternalKeyTest), unittest.makeSuite(OLBTreePyInternalKeyTest), unittest.makeSuite(OLTreeSetInternalKeyTest), unittest.makeSuite(OLTreeSetPyInternalKeyTest), unittest.makeSuite(OLBucketTest), unittest.makeSuite(OLBucketPyTest), unittest.makeSuite(OLTreeSetTest), unittest.makeSuite(OLTreeSetPyTest), unittest.makeSuite(OLSetTest), unittest.makeSuite(OLSetPyTest), unittest.makeSuite(OLBTreeTest), unittest.makeSuite(OLBTreePyTest), unittest.makeSuite(PureOL), unittest.makeSuite(PureOLPy), unittest.makeSuite(TestWeightedOL), unittest.makeSuite(TestWeightedOLPy), unittest.makeSuite(OLBucketConflictTests), unittest.makeSuite(OLBucketPyConflictTests), unittest.makeSuite(OLSetConflictTests), unittest.makeSuite(OLSetPyConflictTests), unittest.makeSuite(OLBTreeConflictTests), unittest.makeSuite(OLBTreePyConflictTests), unittest.makeSuite(OLTreeSetConflictTests), unittest.makeSuite(OLTreeSetPyConflictTests), unittest.makeSuite(OLModuleTest), )) BTrees-4.3.1/BTrees/tests/testBTreesUnicode.py0000664000175000017500000000511212510775554022356 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001, 2002 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from .common import _skip_under_Py3k # When an OOBtree contains unicode strings as keys, # it is neccessary accessing non-unicode strings are # either ascii strings or encoded as unicoded using the # corresponding encoding encoding = 'ISO-8859-1' class TestBTreesUnicode(unittest.TestCase): """ test unicode""" def setUp(self): #setup an OOBTree with some unicode strings from BTrees.OOBTree import OOBTree from BTrees._compat import _bytes from BTrees._compat import _u self.s = _u(b'dreit\xe4gigen', 'latin1') self.data = [(b'alien', 1), (b'k\xf6nnten', 2), (b'fox', 3), (b'future', 4), (b'quick', 5), (b'zerst\xf6rt', 6), (_u(b'dreit\xe4gigen','latin1'), 7), ] self.tree = OOBTree() for k, v in self.data: if isinstance(k, _bytes): k = _u(k, 'latin1') self.tree[k] = v @_skip_under_Py3k def testAllKeys(self): # check every item of the tree from BTrees._compat import _u from BTrees._compat import _bytes for k, v in self.data: if isinstance(k, _bytes): k = _u(k, encoding) self.assertTrue(k in self.tree) self.assertEqual(self.tree[k], v) @_skip_under_Py3k def testUnicodeKeys(self): # try to access unicode keys in tree k, v = self.data[-1] self.assertEqual(k, self.s) self.assertEqual(self.tree[k], v) self.assertEqual(self.tree[self.s], v) @_skip_under_Py3k def testAsciiKeys(self): # try to access some "plain ASCII" keys in the tree for k, v in self.data[0], self.data[2]: self.assertTrue(isinstance(k, str)) self.assertEqual(self.tree[k], v) def test_suite(): return unittest.makeSuite(TestBTreesUnicode) BTrees-4.3.1/BTrees/tests/common.py0000664000175000017500000024452012714130453020250 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import platform from unittest import skip def _skip_wo_ZODB(test_method): # pragma: no COVER try: import ZODB # noqa except ImportError: # skip this test if ZODB is not available return skip("ZODB not available")(test_method) else: return test_method def _skip_under_Py3k(test_method): # pragma: no COVER try: unicode except NameError: # skip this test return skip("Python 3")(test_method) else: return test_method def _skip_on_32_bits(test_method): # pragma: no COVER if platform.architecture()[0] == '32bit': return skip("32-bit platform")(test_method) return test_method class Base(object): # Tests common to all types: sets, buckets, and BTrees db = None def _getTargetClass(self): raise NotImplementedError("subclass should return the target type") def _makeOne(self): return self._getTargetClass()() def tearDown(self): if self.db is not None: self.db.close() def _getRoot(self): from ZODB import DB from ZODB.MappingStorage import MappingStorage if self.db is None: # Unclear: On the next line, the ZODB4 flavor of this routine # [asses a cache_size argument: # self.db = DB(MappingStorage(), cache_size=1) # If that's done here, though, testLoadAndStore() and # testGhostUnghost() both nail the CPU and seemingly # never finish. self.db = DB(MappingStorage()) return self.db.open().root() def _closeRoot(self, root): import transaction # If we don't commit/abort the transaction, then # closing the Connection tends to fail with # "Cannot close connection joined to transaction" transaction.abort() root._p_jar.close() @_skip_wo_ZODB def testLoadAndStore(self): import transaction for i in 0, 10, 1000: t = self._makeOne() self._populate(t, i) root = None root = self._getRoot() root[i] = t transaction.commit() root2 = self._getRoot() if hasattr(t, 'items'): self.assertEqual(list(root2[i].items()) , list(t.items())) else: self.assertEqual(list(root2[i].keys()) , list(t.keys())) self._closeRoot(root) self._closeRoot(root2) def testSetstateArgumentChecking(self): try: self._makeOne().__setstate__(('',)) except TypeError as v: self.assertEqual(str(v), 'tuple required for first state element') else: raise AssertionError("Expected exception") @_skip_wo_ZODB def testGhostUnghost(self): import transaction for i in 0, 10, 1000: t = self._makeOne() self._populate(t, i) root = self._getRoot() root[i] = t transaction.commit() root2 = self._getRoot() root2[i]._p_deactivate() transaction.commit() if hasattr(t, 'items'): self.assertEqual(list(root2[i].items()) , list(t.items())) else: self.assertEqual(list(root2[i].keys()) , list(t.keys())) self._closeRoot(root) self._closeRoot(root2) def testSimpleExclusiveKeyRange(self): t = self._makeOne() self.assertEqual(list(t.keys()), []) self.assertEqual(list(t.keys(excludemin=True)), []) self.assertEqual(list(t.keys(excludemax=True)), []) self.assertEqual(list(t.keys(excludemin=True, excludemax=True)), []) self._populate(t, 1) self.assertEqual(list(t.keys()), [0]) self.assertEqual(list(t.keys(excludemin=True)), []) self.assertEqual(list(t.keys(excludemax=True)), []) self.assertEqual(list(t.keys(excludemin=True, excludemax=True)), []) t.clear() self._populate(t, 2) self.assertEqual(list(t.keys()), [0, 1]) self.assertEqual(list(t.keys(excludemin=True)), [1]) self.assertEqual(list(t.keys(excludemax=True)), [0]) self.assertEqual(list(t.keys(excludemin=True, excludemax=True)), []) t.clear() self._populate(t, 3) self.assertEqual(list(t.keys()), [0, 1, 2]) self.assertEqual(list(t.keys(excludemin=True)), [1, 2]) self.assertEqual(list(t.keys(excludemax=True)), [0, 1]) self.assertEqual(list(t.keys(excludemin=True, excludemax=True)), [1]) self.assertEqual(list(t.keys(-1, 3, excludemin=True, excludemax=True)), [0, 1, 2]) self.assertEqual(list(t.keys(0, 3, excludemin=True, excludemax=True)), [1, 2]) self.assertEqual(list(t.keys(-1, 2, excludemin=True, excludemax=True)), [0, 1]) self.assertEqual(list(t.keys(0, 2, excludemin=True, excludemax=True)), [1]) @_skip_wo_ZODB def test_UpdatesDoReadChecksOnInternalNodes(self): import transaction from ZODB import DB from ZODB.MappingStorage import MappingStorage t = self._makeOne() if not hasattr(t, '_firstbucket'): return self._populate(t, 1000) store = MappingStorage() db = DB(store) conn = db.open() conn.root.t = t transaction.commit() read = [] def readCurrent(ob): read.append(ob) conn.__class__.readCurrent(conn, ob) return 1 conn.readCurrent = readCurrent try: add = t.add remove = t.remove except AttributeError: def add(i): t[i] = i def remove(i): del t[i] # Modifying a thing remove(100) self.assertTrue(t in read) del read[:] add(100) self.assertTrue(t in read) del read[:] transaction.abort() conn.cacheMinimize() list(t) self.assertTrue(100 in t) self.assertTrue(not read) def test_impl_pickle(self): # Issue #2 # Nothing we pickle should include the 'Py' suffix of # implementation classes, and unpickling should give us # back the best available type import pickle made_one = self._makeOne() for proto in range(1, pickle.HIGHEST_PROTOCOL + 1): dumped_str = pickle.dumps(made_one, proto) self.assertTrue(b'Py' not in dumped_str, repr(dumped_str)) loaded_one = pickle.loads(dumped_str) # If we're testing the pure-Python version, but we have the # C extension available, then the loaded type will be the C # extension but the made type will be the Python version. # Otherwise, they match. (Note that if we don't have C extensions # available, the __name__ will be altered to not have Py in it. See _fix_pickle) if 'Py' in type(made_one).__name__: self.assertTrue(type(loaded_one) is not type(made_one)) else: self.assertTrue(type(loaded_one) is type(made_one) is self._getTargetClass(), (type(loaded_one), type(made_one), self._getTargetClass(), repr(dumped_str))) dumped_str2 = pickle.dumps(loaded_one, proto) self.assertEqual(dumped_str, dumped_str2) def test_pickle_empty(self): # Issue #2 # Pickling an empty object and unpickling it should result # in an object that can be pickled, yielding an identical # pickle (and not an AttributeError) import pickle t = self._makeOne() s = pickle.dumps(t) t2 = pickle.loads(s) s2 = pickle.dumps(t2) self.assertEqual(s, s2) if hasattr(t2, '__len__'): # checks for _firstbucket self.assertEqual(0, len(t2)) # This doesn't hold for things like Bucket and Set, sadly # self.assertEqual(t, t2) def test_pickle_subclass(self): # Issue #2: Make sure our class swizzling doesn't break # pickling subclasses # We need a globally named subclass for pickle, but it needs # to be unique in case tests run in parallel base_class = type(self._makeOne()) class_name = 'PickleSubclassOf' + base_class.__name__ PickleSubclass = type(class_name, (base_class,), {}) globals()[class_name] = PickleSubclass import pickle loaded = pickle.loads(pickle.dumps(PickleSubclass())) self.assertTrue(type(loaded) is PickleSubclass, type(loaded)) self.assertTrue(PickleSubclass().__class__ is PickleSubclass) def test_isinstance_subclass(self): # Issue #2: # In some cases we define a __class__ attribute that gets # invoked for isinstance and *lies*. Check that isinstance still # works (almost) as expected. t = self._makeOne() # It's a little bit weird, but in the fibbing case, # we're an instance of two unrelated classes self.assertTrue(isinstance(t, type(t)), (t, type(t))) self.assertTrue(isinstance(t, t.__class__)) class Sub(type(t)): pass self.assertTrue(issubclass(Sub, type(t))) if type(t) is not t.__class__: # We're fibbing; this breaks issubclass of itself, # contrary to the usual mechanism self.assertFalse(issubclass(t.__class__, type(t))) class NonSub(object): pass self.assertFalse(issubclass(NonSub, type(t))) self.assertFalse(isinstance(NonSub(), type(t))) class MappingBase(Base): # Tests common to mappings (buckets, btrees) def _populate(self, t, l): # Make some data for i in range(l): t[i]=i def testShortRepr(self): # test the repr because buckets have a complex repr implementation # internally the cutoff from a stack allocated buffer to a heap # allocated buffer is 10000. t = self._makeOne() for i in range(5): t[i] = i r = repr(t) # Make sure the repr is **not* 10000 bytes long for a shrort bucket. # (the buffer must be terminated when copied). self.assertTrue(len(r) < 10000) # Make sure the repr is human readable if it's a bucket if 'Bucket' in r: self.assertTrue(r.startswith("BTrees")) self.assertTrue(r.endswith(repr(t.items()) + ')'), r) else: self.assertEqual(r[:8], ' 10000) def testGetItemFails(self): self.assertRaises(KeyError, self._getitemfail) def _getitemfail(self): return self._makeOne()[1] def testGetReturnsDefault(self): self.assertEqual(self._makeOne().get(1) , None) self.assertEqual(self._makeOne().get(1, 'foo') , 'foo') def testSetItemGetItemWorks(self): t = self._makeOne() t[1] = 1 a = t[1] self.assertEqual(a , 1, repr(a)) def testReplaceWorks(self): t = self._makeOne() t[1] = 1 self.assertEqual(t[1] , 1, t[1]) t[1] = 2 self.assertEqual(t[1] , 2, t[1]) def testLen(self): import random t = self._makeOne() added = {} r = list(range(1000)) for x in r: k = random.choice(r) t[k] = x added[k] = x addl = added.keys() self.assertEqual(len(t) , len(addl), len(t)) def testHasKeyWorks(self): from .._compat import PY2 t = self._makeOne() t[1] = 1 if PY2: self.assertTrue(t.has_key(1)) self.assertTrue(1 in t) self.assertTrue(0 not in t) self.assertTrue(2 not in t) def testValuesWorks(self): t = self._makeOne() for x in range(100): t[x] = x*x v = t.values() for i in range(100): self.assertEqual(v[i], i*i) self.assertRaises(IndexError, lambda: v[i+1]) i = 0 for value in t.itervalues(): self.assertEqual(value, i*i) i += 1 def testValuesWorks1(self): t = self._makeOne() for x in range(100): t[99-x] = x for x in range(40): lst = sorted(t.values(0+x,99-x)) self.assertEqual(lst, list(range(0+x,99-x+1))) lst = sorted(t.values(max=99-x, min=0+x)) self.assertEqual(lst, list(range(0+x,99-x+1))) def testValuesNegativeIndex(self): t = self._makeOne() L = [-3, 6, -11, 4] for i in L: t[i] = i L = sorted(L) vals = t.values() for i in range(-1, -5, -1): self.assertEqual(vals[i], L[i]) self.assertRaises(IndexError, lambda: vals[-5]) def testKeysWorks(self): t = self._makeOne() for x in range(100): t[x] = x v = t.keys() i = 0 for x in v: self.assertEqual(x,i) i = i + 1 self.assertRaises(IndexError, lambda: v[i]) for x in range(40): lst = t.keys(0+x,99-x) self.assertEqual(list(lst), list(range(0+x, 99-x+1))) lst = t.keys(max=99-x, min=0+x) self.assertEqual(list(lst), list(range(0+x, 99-x+1))) self.assertEqual(len(v), 100) def testKeysNegativeIndex(self): t = self._makeOne() L = [-3, 6, -11, 4] for i in L: t[i] = i L = sorted(L) keys = t.keys() for i in range(-1, -5, -1): self.assertEqual(keys[i], L[i]) self.assertRaises(IndexError, lambda: keys[-5]) def testItemsWorks(self): t = self._makeOne() for x in range(100): t[x] = 2*x v = t.items() i = 0 for x in v: self.assertEqual(x[0], i) self.assertEqual(x[1], 2*i) i += 1 self.assertRaises(IndexError, lambda: v[i+1]) i = 0 for x in t.iteritems(): self.assertEqual(x, (i, 2*i)) i += 1 items = list(t.items(min=12, max=20)) self.assertEqual(items, list(zip(range(12, 21), range(24, 43, 2)))) items = list(t.iteritems(min=12, max=20)) self.assertEqual(items, list(zip(range(12, 21), range(24, 43, 2)))) def testItemsNegativeIndex(self): t = self._makeOne() L = [-3, 6, -11, 4] for i in L: t[i] = i L = sorted(L) items = t.items() for i in range(-1, -5, -1): self.assertEqual(items[i], (L[i], L[i])) self.assertRaises(IndexError, lambda: items[-5]) def testDeleteInvalidKeyRaisesKeyError(self): self.assertRaises(KeyError, self._deletefail) def _deletefail(self): t = self._makeOne() del t[1] def testMaxKeyMinKey(self): t = self._makeOne() t[7] = 6 t[3] = 10 t[8] = 12 t[1] = 100 t[5] = 200 t[10] = 500 t[6] = 99 t[4] = 150 del t[7] self.assertEqual(t.maxKey(), 10) self.assertEqual(t.maxKey(None), 10) self.assertEqual(t.maxKey(6), 6) self.assertEqual(t.maxKey(9), 8) self.assertEqual(t.minKey(), 1) self.assertEqual(t.minKey(None), 1) self.assertEqual(t.minKey(3), 3) self.assertEqual(t.minKey(9), 10) try: t.maxKey(t.minKey() - 1) except ValueError as err: self.assertEqual(str(err), "no key satisfies the conditions") else: self.fail("expected ValueError") try: t.minKey(t.maxKey() + 1) except ValueError as err: self.assertEqual(str(err), "no key satisfies the conditions") else: self.fail("expected ValueError") def testClear(self): import random t = self._makeOne() r = list(range(100)) for x in r: rnd = random.choice(r) t[rnd] = 0 t.clear() diff = lsubtract(list(t.keys()), []) self.assertEqual(diff, []) def testUpdate(self): import random t = self._makeOne() d={} l=[] for i in range(10000): k=random.randrange(-2000, 2001) d[k]=i l.append((k, i)) items= sorted(d.items()) t.update(d) self.assertEqual(list(t.items()), items) t.clear() self.assertEqual(list(t.items()), []) t.update(l) self.assertEqual(list(t.items()), items) # Before ZODB 3.4.2, update/construction from PersistentMapping failed. def testUpdateFromPersistentMapping(self): from persistent.mapping import PersistentMapping t = self._makeOne() pm = PersistentMapping({1: 2}) t.update(pm) self.assertEqual(list(t.items()), [(1, 2)]) # Construction goes thru the same internals as .update(). t = t.__class__(pm) self.assertEqual(list(t.items()), [(1, 2)]) def testEmptyRangeSearches(self): t = self._makeOne() t.update([(1,1), (5,5), (9,9)]) self.assertEqual(list(t.keys(-6,-4)), [], list(t.keys(-6,-4))) self.assertEqual(list(t.keys(2,4)), [], list(t.keys(2,4))) self.assertEqual(list(t.keys(6,8)), [], list(t.keys(6,8))) self.assertEqual(list(t.keys(10,12)), [], list(t.keys(10,12))) self.assertEqual(list(t.keys(9, 1)), [], list(t.keys(9, 1))) # For IITreeSets, this one was returning 31 for len(keys), and # list(keys) produced a list with 100 elements. t.clear() t.update(list(zip(range(300), range(300)))) keys = t.keys(200, 50) self.assertEqual(len(keys), 0) self.assertEqual(list(keys), []) self.assertEqual(list(t.iterkeys(200, 50)), []) keys = t.keys(max=50, min=200) self.assertEqual(len(keys), 0) self.assertEqual(list(keys), []) self.assertEqual(list(t.iterkeys(max=50, min=200)), []) def testSlicing(self): # Test that slicing of .keys()/.values()/.items() works exactly the # same way as slicing a Python list with the same contents. # This tests fixes to several bugs in this area, starting with # http://collector.zope.org/Zope/419, # "BTreeItems slice contains 1 too many elements". from .._compat import xrange t = self._makeOne() for n in range(10): t.clear() self.assertEqual(len(t), 0) keys = [] values = [] items = [] for key in range(n): value = -2 * key t[key] = value keys.append(key) values.append(value) items.append((key, value)) self.assertEqual(len(t), n) kslice = t.keys() vslice = t.values() islice = t.items() self.assertEqual(len(kslice), n) self.assertEqual(len(vslice), n) self.assertEqual(len(islice), n) # Test whole-structure slices. x = kslice[:] self.assertEqual(list(x), keys[:]) x = vslice[:] self.assertEqual(list(x), values[:]) x = islice[:] self.assertEqual(list(x), items[:]) for lo in range(-2*n, 2*n+1): # Test one-sided slices. x = kslice[:lo] self.assertEqual(list(x), keys[:lo]) x = kslice[lo:] self.assertEqual(list(x), keys[lo:]) x = vslice[:lo] self.assertEqual(list(x), values[:lo]) x = vslice[lo:] self.assertEqual(list(x), values[lo:]) x = islice[:lo] self.assertEqual(list(x), items[:lo]) x = islice[lo:] self.assertEqual(list(x), items[lo:]) for hi in range(-2*n, 2*n+1): # Test two-sided slices. x = kslice[lo:hi] self.assertEqual(list(x), keys[lo:hi]) x = vslice[lo:hi] self.assertEqual(list(x), values[lo:hi]) x = islice[lo:hi] self.assertEqual(list(x), items[lo:hi]) # The specific test case from Zope collector 419. t.clear() for i in xrange(100): t[i] = 1 tslice = t.items()[20:80] self.assertEqual(len(tslice), 60) self.assertEqual(list(tslice), list(zip(range(20, 80), [1]*60))) def testIterators(self): t = self._makeOne() for keys in [], [-2], [1, 4], list(range(-170, 2000, 6)): t.clear() for k in keys: t[k] = -3 * k self.assertEqual(list(t), keys) x = [] for k in t: x.append(k) self.assertEqual(x, keys) it = iter(t) self.assertTrue(it is iter(it)) x = [] try: while 1: x.append(next(it)) except StopIteration: pass self.assertEqual(x, keys) self.assertEqual(list(t.iterkeys()), keys) self.assertEqual(list(t.itervalues()), list(t.values())) self.assertEqual(list(t.iteritems()), list(t.items())) def testRangedIterators(self): t = self._makeOne() for keys in [], [-2], [1, 4], list(range(-170, 2000, 13)): t.clear() values = [] for k in keys: value = -3 * k t[k] = value values.append(value) items = list(zip(keys, values)) self.assertEqual(list(t.iterkeys()), keys) self.assertEqual(list(t.itervalues()), values) self.assertEqual(list(t.iteritems()), items) if not keys: continue min_mid_max = (keys[0], keys[len(keys) >> 1], keys[-1]) for key1 in min_mid_max: for lo in range(key1 - 1, key1 + 2): # Test one-sided range iterators. goodkeys = [k for k in keys if lo <= k] got = t.iterkeys(lo) self.assertEqual(goodkeys, list(got)) goodvalues = [t[k] for k in goodkeys] got = t.itervalues(lo) self.assertEqual(goodvalues, list(got)) gooditems = list(zip(goodkeys, goodvalues)) got = t.iteritems(lo) self.assertEqual(gooditems, list(got)) for key2 in min_mid_max: for hi in range(key2 - 1, key2 + 2): goodkeys = [k for k in keys if lo <= k <= hi] got = t.iterkeys(min=lo, max=hi) self.assertEqual(goodkeys, list(got)) goodvalues = [t[k] for k in goodkeys] got = t.itervalues(lo, max=hi) self.assertEqual(goodvalues, list(got)) gooditems = list(zip(goodkeys, goodvalues)) got = t.iteritems(max=hi, min=lo) self.assertEqual(gooditems, list(got)) def testBadUpdateTupleSize(self): # This one silently ignored the excess in Zope3. t = self._makeOne() self.assertRaises(TypeError, t.update, [(1, 2, 3)]) # This one dumped core in Zope3. self.assertRaises(TypeError, t.update, [(1,)]) # This one should simply succeed. t.update([(1, 2)]) self.assertEqual(list(t.items()), [(1, 2)]) def testSimpleExclusivRanges(self): def identity(x): return x def dup(x): return [(y, y) for y in x] for methodname, f in (("keys", identity), ("values", identity), ("items", dup), ("iterkeys", identity), ("itervalues", identity), ("iteritems", dup)): t = self._makeOne() meth = getattr(t, methodname, None) if meth is None: continue self.assertEqual(list(meth()), []) self.assertEqual(list(meth(excludemin=True)), []) self.assertEqual(list(meth(excludemax=True)), []) self.assertEqual(list(meth(excludemin=True, excludemax=True)), []) self._populate(t, 1) self.assertEqual(list(meth()), f([0])) self.assertEqual(list(meth(excludemin=True)), []) self.assertEqual(list(meth(excludemax=True)), []) self.assertEqual(list(meth(excludemin=True, excludemax=True)), []) t.clear() self._populate(t, 2) self.assertEqual(list(meth()), f([0, 1])) self.assertEqual(list(meth(excludemin=True)), f([1])) self.assertEqual(list(meth(excludemax=True)), f([0])) self.assertEqual(list(meth(excludemin=True, excludemax=True)), []) t.clear() self._populate(t, 3) self.assertEqual(list(meth()), f([0, 1, 2])) self.assertEqual(list(meth(excludemin=True)), f([1, 2])) self.assertEqual(list(meth(excludemax=True)), f([0, 1])) self.assertEqual(list(meth(excludemin=True, excludemax=True)), f([1])) self.assertEqual(list(meth(-1, 3, excludemin=True, excludemax=True)), f([0, 1, 2])) self.assertEqual(list(meth(0, 3, excludemin=True, excludemax=True)), f([1, 2])) self.assertEqual(list(meth(-1, 2, excludemin=True, excludemax=True)), f([0, 1])) self.assertEqual(list(meth(0, 2, excludemin=True, excludemax=True)), f([1])) def testSetdefault(self): t = self._makeOne() self.assertEqual(t.setdefault(1, 2), 2) # That should also have associated 1 with 2 in the tree. self.assertTrue(1 in t) self.assertEqual(t[1], 2) # And trying to change it again should have no effect. self.assertEqual(t.setdefault(1, 666), 2) self.assertEqual(t[1], 2) # Not enough arguments. self.assertRaises(TypeError, t.setdefault) self.assertRaises(TypeError, t.setdefault, 1) # Too many arguments. self.assertRaises(TypeError, t.setdefault, 1, 2, 3) def testPop(self): t = self._makeOne() # Empty container. # If no default given, raises KeyError. self.assertRaises(KeyError, t.pop, 1) # But if default given, returns that instead. self.assertEqual(t.pop(1, 42), 42) t[1] = 3 # KeyError when key is not in container and default is not passed. self.assertRaises(KeyError, t.pop, 5) self.assertEqual(list(t.items()), [(1, 3)]) # If key is in container, returns the value and deletes the key. self.assertEqual(t.pop(1), 3) self.assertEqual(len(t), 0) # If key is present, return value bypassing default. t[1] = 3 self.assertEqual(t.pop(1, 7), 3) self.assertEqual(len(t), 0) # Pop only one item. t[1] = 3 t[2] = 4 self.assertEqual(len(t), 2) self.assertEqual(t.pop(1), 3) self.assertEqual(len(t), 1) self.assertEqual(t[2], 4) self.assertEqual(t.pop(1, 3), 3) # Too few arguments. self.assertRaises(TypeError, t.pop) # Too many arguments. self.assertRaises(TypeError, t.pop, 1, 2, 3) class BTreeTests(MappingBase): # Tests common to all BTrees def _getTargetClass(self): # Most of the subclasses override _makeOne and not # _getTargetClass, so we can get the type that way. # TODO: This could change for less repetition in the subclasses, # using the name of the class to import the module and find # the type. if type(self)._makeOne is not BTreeTests._makeOne: return type(self._makeOne()) raise NotImplementedError() def _makeOne(self, *args): return self._getTargetClass()(*args) def _checkIt(self, t): from BTrees.check import check t._check() check(t) def testDeleteNoChildrenWorks(self): t = self._makeOne() t[5] = 6 t[2] = 10 t[6] = 12 t[1] = 100 t[3] = 200 t[10] = 500 t[4] = 99 del t[4] diff = lsubtract(t.keys(), [1,2,3,5,6,10]) self.assertEqual(diff , [], diff) self._checkIt(t) def testDeleteOneChildWorks(self): t = self._makeOne() t[5] = 6 t[2] = 10 t[6] = 12 t[1] = 100 t[3] = 200 t[10] = 500 t[4] = 99 del t[3] diff = lsubtract(t.keys(), [1,2,4,5,6,10]) self.assertEqual(diff , [], diff) self._checkIt(t) def testDeleteTwoChildrenNoInorderSuccessorWorks(self): t = self._makeOne() t[5] = 6 t[2] = 10 t[6] = 12 t[1] = 100 t[3] = 200 t[10] = 500 t[4] = 99 del t[2] diff = lsubtract(t.keys(), [1,3,4,5,6,10]) self.assertEqual(diff , [], diff) self._checkIt(t) def testDeleteTwoChildrenInorderSuccessorWorks(self): # 7, 3, 8, 1, 5, 10, 6, 4 -- del 3 t = self._makeOne() t[7] = 6 t[3] = 10 t[8] = 12 t[1] = 100 t[5] = 200 t[10] = 500 t[6] = 99 t[4] = 150 del t[3] diff = lsubtract(t.keys(), [1,4,5,6,7,8,10]) self.assertEqual(diff , [], diff) self._checkIt(t) def testDeleteRootWorks(self): # 7, 3, 8, 1, 5, 10, 6, 4 -- del 7 t = self._makeOne() t[7] = 6 t[3] = 10 t[8] = 12 t[1] = 100 t[5] = 200 t[10] = 500 t[6] = 99 t[4] = 150 del t[7] diff = lsubtract(t.keys(), [1,3,4,5,6,8,10]) self.assertEqual(diff , [], diff) self._checkIt(t) def testRandomNonOverlappingInserts(self): import random t = self._makeOne() added = {} r = list(range(100)) for x in r: k = random.choice(r) if k not in added: t[k] = x added[k] = 1 addl = sorted(added.keys()) diff = lsubtract(list(t.keys()), addl) self.assertEqual(diff , [], (diff, addl, list(t.keys()))) self._checkIt(t) def testRandomOverlappingInserts(self): import random t = self._makeOne() added = {} r = list(range(100)) for x in r: k = random.choice(r) t[k] = x added[k] = 1 addl = sorted(added.keys()) diff = lsubtract(t.keys(), addl) self.assertEqual(diff , [], diff) self._checkIt(t) def testRandomDeletes(self): import random t = self._makeOne() r = list(range(1000)) added = [] for x in r: k = random.choice(r) t[k] = x added.append(k) deleted = [] for x in r: k = random.choice(r) if k in t: self.assertTrue(k in t) del t[k] deleted.append(k) if k in t: self.fail( "had problems deleting %s" % k ) badones = [] for x in deleted: if x in t: badones.append(x) self.assertEqual(badones , [], (badones, added, deleted)) self._checkIt(t) def testTargetedDeletes(self): import random t = self._makeOne() r = list(range(1000)) for x in r: k = random.choice(r) t[k] = x for x in r: try: del t[x] except KeyError: pass self.assertEqual(realseq(t.keys()) , [], realseq(t.keys())) self._checkIt(t) def testPathologicalRightBranching(self): t = self._makeOne() r = list(range(1000)) for x in r: t[x] = 1 self.assertEqual(realseq(t.keys()) , r, realseq(t.keys())) for x in r: del t[x] self.assertEqual(realseq(t.keys()) , [], realseq(t.keys())) self._checkIt(t) def testPathologicalLeftBranching(self): t = self._makeOne() r = list(range(1000)) revr = list(reversed(r[:])) for x in revr: t[x] = 1 self.assertEqual(realseq(t.keys()) , r, realseq(t.keys())) for x in revr: del t[x] self.assertEqual(realseq(t.keys()) , [], realseq(t.keys())) self._checkIt(t) def testSuccessorChildParentRewriteExerciseCase(self): t = self._makeOne() add_order = [ 85, 73, 165, 273, 215, 142, 233, 67, 86, 166, 235, 225, 255, 73, 175, 171, 285, 162, 108, 28, 283, 258, 232, 199, 260, 298, 275, 44, 261, 291, 4, 181, 285, 289, 216, 212, 129, 243, 97, 48, 48, 159, 22, 285, 92, 110, 27, 55, 202, 294, 113, 251, 193, 290, 55, 58, 239, 71, 4, 75, 129, 91, 111, 271, 101, 289, 194, 218, 77, 142, 94, 100, 115, 101, 226, 17, 94, 56, 18, 163, 93, 199, 286, 213, 126, 240, 245, 190, 195, 204, 100, 199, 161, 292, 202, 48, 165, 6, 173, 40, 218, 271, 228, 7, 166, 173, 138, 93, 22, 140, 41, 234, 17, 249, 215, 12, 292, 246, 272, 260, 140, 58, 2, 91, 246, 189, 116, 72, 259, 34, 120, 263, 168, 298, 118, 18, 28, 299, 192, 252, 112, 60, 277, 273, 286, 15, 263, 141, 241, 172, 255, 52, 89, 127, 119, 255, 184, 213, 44, 116, 231, 173, 298, 178, 196, 89, 184, 289, 98, 216, 115, 35, 132, 278, 238, 20, 241, 128, 179, 159, 107, 206, 194, 31, 260, 122, 56, 144, 118, 283, 183, 215, 214, 87, 33, 205, 183, 212, 221, 216, 296, 40, 108, 45, 188, 139, 38, 256, 276, 114, 270, 112, 214, 191, 147, 111, 299, 107, 101, 43, 84, 127, 67, 205, 251, 38, 91, 297, 26, 165, 187, 19, 6, 73, 4, 176, 195, 90, 71, 30, 82, 139, 210, 8, 41, 253, 127, 190, 102, 280, 26, 233, 32, 257, 194, 263, 203, 190, 111, 218, 199, 29, 81, 207, 18, 180, 157, 172, 192, 135, 163, 275, 74, 296, 298, 265, 105, 191, 282, 277, 83, 188, 144, 259, 6, 173, 81, 107, 292, 231, 129, 65, 161, 113, 103, 136, 255, 285, 289, 1 ] delete_order = [ 276, 273, 12, 275, 2, 286, 127, 83, 92, 33, 101, 195, 299, 191, 22, 232, 291, 226, 110, 94, 257, 233, 215, 184, 35, 178, 18, 74, 296, 210, 298, 81, 265, 175, 116, 261, 212, 277, 260, 234, 6, 129, 31, 4, 235, 249, 34, 289, 105, 259, 91, 93, 119, 7, 183, 240, 41, 253, 290, 136, 75, 292, 67, 112, 111, 256, 163, 38, 126, 139, 98, 56, 282, 60, 26, 55, 245, 225, 32, 52, 40, 271, 29, 252, 239, 89, 87, 205, 213, 180, 97, 108, 120, 218, 44, 187, 196, 251, 202, 203, 172, 28, 188, 77, 90, 199, 297, 282, 141, 100, 161, 216, 73, 19, 17, 189, 30, 258 ] for x in add_order: t[x] = 1 for x in delete_order: try: del t[x] except KeyError: if x in t: self.assertEqual(1,2,"failed to delete %s" % x) self._checkIt(t) def testRangeSearchAfterSequentialInsert(self): t = self._makeOne() r = range(100) for x in r: t[x] = 0 diff = lsubtract(list(t.keys(0, 100)), r) self.assertEqual(diff , [], diff) # The same thing with no bounds diff = lsubtract(list(t.keys(None, None)), r) self.assertEqual(diff , [], diff) # The same thing with each bound set and the other # explicitly None diff = lsubtract(list(t.keys(0, None)), r) self.assertEqual(diff , [], diff) diff = lsubtract(list(t.keys(None,100)), r) self.assertEqual(diff , [], diff) self._checkIt(t) def testRangeSearchAfterRandomInsert(self): import random t = self._makeOne() r = range(100) a = {} for x in r: rnd = random.choice(r) t[rnd] = 0 a[rnd] = 0 diff = lsubtract(list(t.keys(0, 100)), a.keys()) self.assertEqual(diff, [], diff) self._checkIt(t) def testPathologicalRangeSearch(self): t = self._makeOne() # Build a 2-level tree with at least two buckets. for i in range(200): t[i] = i items, dummy = t.__getstate__() self.assertTrue(len(items) > 2) # at least two buckets and a key # All values in the first bucket are < firstkey. All in the # second bucket are >= firstkey, and firstkey is the first key in # the second bucket. firstkey = items[1] therange = t.keys(-1, firstkey) self.assertEqual(len(therange), firstkey + 1) self.assertEqual(list(therange), list(range(firstkey + 1))) # Now for the tricky part. If we delete firstkey, the second bucket # loses its smallest key, but firstkey remains in the BTree node. # If we then do a high-end range search on firstkey, the BTree node # directs us to look in the second bucket, but there's no longer any # key <= firstkey in that bucket. The correct answer points to the # end of the *first* bucket. The algorithm has to be smart enough # to "go backwards" in the BTree then; if it doesn't, it will # erroneously claim that the range is empty. del t[firstkey] therange = t.keys(min=-1, max=firstkey) self.assertEqual(len(therange), firstkey) self.assertEqual(list(therange), list(range(firstkey))) self._checkIt(t) def testInsertMethod(self): t = self._makeOne() t[0] = 1 self.assertEqual(t.insert(0, 1) , 0) self.assertEqual(t.insert(1, 1) , 1) self.assertEqual(lsubtract(list(t.keys()), [0,1]) , []) self._checkIt(t) def testDamagedIterator(self): # A cute one from Steve Alexander. This caused the BTreeItems # object to go insane, accessing memory beyond the allocated part # of the bucket. If it fails, the symptom is either a C-level # assertion error (if the BTree code was compiled without NDEBUG), # or most likely a segfault (if the BTree code was compiled with # NDEBUG). t = self._makeOne() self._populate(t, 10) # In order for this to fail, it's important that k be a "lazy" # iterator, referring to the BTree by indirect position (index) # instead of a fully materialized list. Then the position can # end up pointing into trash memory, if the bucket pointed to # shrinks. k = t.keys() for dummy in range(20): try: del t[k[0]] except RuntimeError as detail: self.assertEqual(str(detail), "the bucket being iterated " "changed size") break except KeyError as v: # The Python implementation behaves very differently and # gives a key error in this situation. It can't mess up # memory and can't readily detect changes to underlying buckets # in any sane way. self.assertEqual(str(v), str(k[0])) self._checkIt(t) def testAddTwoSetsChanged(self): # A bug in the BTree Python implementation once # caused adding a second item to a tree to fail # to set _p_changed (adding the first item sets it because # the _firstbucket gets set, but the second item only grew the # existing bucket) t = self._makeOne() # Note that for the property to actually hold, we have to fake a # _p_jar and _p_oid t._p_oid = b'\0\0\0\0\0' class Jar(object): def __init__(self): self._cache = self self.registered = None def mru(self, arg): pass def readCurrent(self, arg): pass def register(self, arg): self.registered = arg t._p_jar = Jar() t[1] = 3 # reset these, setting _firstbucket triggered a change t._p_changed = False t._p_jar.registered = None t[2] = 4 self.assertTrue(t._p_changed) self.assertEqual(t, t._p_jar.registered) # Setting the same key to a different value also triggers a change t._p_changed = False t._p_jar.registered = None t[2] = 5 self.assertTrue(t._p_changed) self.assertEqual(t, t._p_jar.registered) # Likewise with only a single value t = self._makeOne() t._p_oid = b'\0\0\0\0\0' t._p_jar = Jar() t[1] = 3 # reset these, setting _firstbucket triggered a change t._p_changed = False t._p_jar.registered = None t[1] = 6 self.assertTrue(t._p_changed) self.assertEqual(t, t._p_jar.registered) def testRemoveInSmallMapSetsChanged(self): # A bug in the BTree Python implementation once caused # deleting from a small btree to set _p_changed. # There must be at least two objects so that _firstbucket doesn't # get set t = self._makeOne() # Note that for the property to actually hold, we have to fake a # _p_jar and _p_oid t._p_oid = b'\0\0\0\0\0' class Jar(object): def __init__(self): self._cache = self self.registered = None def mru(self, arg): pass def readCurrent(self, arg): pass def register(self, arg): self.registered = arg t._p_jar = Jar() t[0] = 1 t[1] = 2 # reset these, setting _firstbucket triggered a change t._p_changed = False t._p_jar.registered = None # now remove the second value del t[1] self.assertTrue(t._p_changed) self.assertEqual(t, t._p_jar.registered) def test_legacy_py_pickle(self): # Issue #2 # If we have a pickle that includes the 'Py' suffix, # it (unfortunately) unpickles to the python type. But # new pickles never produce that. import pickle made_one = self._makeOne() for proto in (1, 2): s = pickle.dumps(made_one, proto) # It's not legacy assert b'TreePy\n' not in s, repr(s) # \np for protocol 1, \nq for proto 2, assert b'Tree\np' in s or b'Tree\nq' in s, repr(s) # Now make it pseudo-legacy legacys = s.replace(b'Tree\np', b'TreePy\np').replace(b'Tree\nq', b'TreePy\nq') # It loads up as the specified class loaded_one = pickle.loads(legacys) # It still functions and can be dumped again, as the original class s2 = pickle.dumps(loaded_one, proto) self.assertTrue(b'Py' not in s2) self.assertEqual(s2, s) class NormalSetTests(Base): # Test common to all set types def _populate(self, t, l): # Make some data t.update(range(l)) def testShortRepr(self): t = self._makeOne() for i in range(5): t.add(i) r = repr(t) # Make sure the repr is **not* 10000 bytes long for a shrort bucket. # (the buffer must be terminated when copied). self.assertTrue(len(r) < 10000) # Make sure the repr is human readable, unless it's a tree if 'TreeSet' not in r: self.assertTrue(r.endswith("Set(%r)" % t.keys())) else: self.assertEqual(r[:7], '= 3: break transaction.commit() # Now, delete the internal key and make sure it's really gone key = data[1] del tree[key] data = tree.__getstate__()[0] self.assertTrue(data[1] != key) # The tree should have changed: self.assertTrue(tree._p_changed) # Grow the btree until we have multiple levels while 1: i += 1 self.add_key(tree, i) data = tree.__getstate__()[0] if data[0].__class__ == tree.__class__: assert len(data[2].__getstate__()[0]) >= 3 break # Now, delete the internal key and make sure it's really gone key = data[1] del tree[key] data = tree.__getstate__()[0] self.assertTrue(data[1] != key) transaction.abort() db.close() class InternalKeysSetTest(object): # There must not be any internal keys not in the TreeSet def add_key(self, tree, key): tree.add(key) class ModuleTest(object): # test for presence of generic names in module prefix = None def _getModule(self): pass def testNames(self): names = ['Bucket', 'BTree', 'Set', 'TreeSet'] for name in names: klass = getattr(self._getModule(), name) self.assertEqual(klass.__module__, self._getModule().__name__) self.assertTrue(klass is getattr(self._getModule(), self.prefix + name)) # BBB for zope.app.security ZCML :( pfx_iter = self.prefix + 'TreeIterator' klass = getattr(self._getModule(), pfx_iter) self.assertEqual(klass.__module__, self._getModule().__name__) def testModuleProvides(self): from zope.interface.verify import verifyObject verifyObject(self._getInterface(), self._getModule()) def testFamily(self): import BTrees if self.prefix == 'OO': self.assertTrue( getattr(self._getModule(), 'family', self) is self) elif 'L' in self.prefix: self.assertTrue(self._getModule().family is BTrees.family64) elif 'I' in self.prefix: self.assertTrue(self._getModule().family is BTrees.family32) class TypeTest(object): # tests of various type errors def testBadTypeRaises(self): self.assertRaises(TypeError, self._stringraises) self.assertRaises(TypeError, self._floatraises) self.assertRaises(TypeError, self._noneraises) class I_SetsBase(object): def testBadBadKeyAfterFirst(self): t = self._makeOne() self.assertRaises(TypeError, t.__class__, [1, '']) self.assertRaises(TypeError, t.update, [1, '']) def testNonIntegerInsertRaises(self): self.assertRaises(TypeError,self._insertstringraises) self.assertRaises(TypeError,self._insertfloatraises) self.assertRaises(TypeError,self._insertnoneraises) def _insertstringraises(self): self._makeOne().insert('a') def _insertfloatraises(self): self._makeOne().insert(1.4) def _insertnoneraises(self): self._makeOne().insert(None) LARGEST_32_BITS = 2147483647 SMALLEST_32_BITS = -LARGEST_32_BITS - 1 SMALLEST_POSITIVE_33_BITS = LARGEST_32_BITS + 1 LARGEST_NEGATIVE_33_BITS = SMALLEST_32_BITS - 1 LARGEST_64_BITS = 0x7fffffffffffffff SMALLEST_64_BITS = -LARGEST_64_BITS - 1 SMALLEST_POSITIVE_65_BITS = LARGEST_64_BITS + 1 LARGEST_NEGATIVE_65_BITS = SMALLEST_64_BITS - 1 class TestLongIntSupport: def getTwoValues(self): # Return two distinct values; these must compare as un-equal. # # These values must be usable as values. return object(), object() def getTwoKeys(self): # Return two distinct values, these must compare as un-equal. # #These values must be usable as keys. return 0, 1 def _set_value(self, key, value): t = self._makeOne() t[key] = value class TestLongIntKeys(TestLongIntSupport): def _makeLong(self, v): try: return long(v) except NameError: #pragma NO COVER Py3k return int(v) def testLongIntKeysWork(self): from BTrees.IIBTree import using64bits if not using64bits: return t = self._makeOne() o1, o2 = self.getTwoValues() assert o1 != o2 # Test some small key values first: zero_long = self._makeLong(0) t[zero_long] = o1 self.assertEqual(t[0], o1) t[0] = o2 self.assertEqual(t[zero_long], o2) self.assertEqual(list(t.keys()), [0]) self.assertEqual(list(t.keys(None,None)), [0]) # Test some large key values too: k1 = SMALLEST_POSITIVE_33_BITS k2 = LARGEST_64_BITS k3 = SMALLEST_64_BITS t[k1] = o1 t[k2] = o2 t[k3] = o1 self.assertEqual(t[k1], o1) self.assertEqual(t[k2], o2) self.assertEqual(t[k3], o1) self.assertEqual(list(t.keys()), [k3, 0, k1, k2]) self.assertEqual(list(t.keys(k3,None)), [k3, 0, k1, k2]) self.assertEqual(list(t.keys(None,k2)), [k3, 0, k1, k2]) def testLongIntKeysOutOfRange(self): from BTrees.IIBTree import using64bits if not using64bits: return o1, o2 = self.getTwoValues() self.assertRaises( ValueError, self._set_value, SMALLEST_POSITIVE_65_BITS, o1) self.assertRaises( ValueError, self._set_value, LARGEST_NEGATIVE_65_BITS, o1) class TestLongIntValues(TestLongIntSupport): def testLongIntValuesWork(self): from BTrees.IIBTree import using64bits if not using64bits: return t = self._makeOne() keys = sorted(self.getTwoKeys()) k1, k2 = keys assert k1 != k2 # This is the smallest positive integer that requires 33 bits: v1 = SMALLEST_POSITIVE_33_BITS v2 = v1 + 1 t[k1] = v1 t[k2] = v2 self.assertEqual(t[k1], v1) self.assertEqual(t[k2], v2) self.assertEqual(list(t.values()), [v1, v2]) self.assertEqual(list(t.values(None,None)), [v1, v2]) def testLongIntValuesOutOfRange(self): from BTrees.IIBTree import using64bits if not using64bits: return k1, k2 = self.getTwoKeys() self.assertRaises( ValueError, self._set_value, k1, SMALLEST_POSITIVE_65_BITS) self.assertRaises( ValueError, self._set_value, k1, LARGEST_NEGATIVE_65_BITS) # Given a mapping builder (IIBTree, OOBucket, etc), return a function # that builds an object of that type given only a list of keys. def makeBuilder(mapbuilder): def result(keys=[], mapbuilder=mapbuilder): return mapbuilder(list(zip(keys, keys))) return result # Subclasses have to set up: # builders() - function returning functions to build inputs, # each returned callable tkes an optional keys arg # intersection, union, difference - set to the type-correct versions class SetResult(object): def setUp(self): self.Akeys = [1, 3, 5, 6 ] self.Bkeys = [ 2, 3, 4, 6, 7] self.As = [makeset(self.Akeys) for makeset in self.builders()] self.Bs = [makeset(self.Bkeys) for makeset in self.builders()] self.emptys = [makeset() for makeset in self.builders()] # Slow but obviously correct Python implementations of basic ops. def _union(self, x, y): result = list(x) for e in y: if e not in result: result.append(e) return sorted(result) def _intersection(self, x, y): result = [] for e in x: if e in y: result.append(e) return result def _difference(self, x, y): result = list(x) for e in y: if e in result: result.remove(e) # Difference preserves LHS values. if hasattr(x, "values"): result = [(k, x[k]) for k in result] return result def testNone(self): for op in self.union, self.intersection, self.difference: C = op(None, None) self.assertTrue(C is None) for op in self.union, self.intersection, self.difference: for A in self.As: C = op(A, None) self.assertTrue(C is A) C = op(None, A) if op == self.difference: self.assertTrue(C is None) else: self.assertTrue(C is A) def testEmptyUnion(self): for A in self.As: for E in self.emptys: C = self.union(A, E) self.assertTrue(not hasattr(C, "values")) self.assertEqual(list(C), self.Akeys) C = self.union(E, A) self.assertTrue(not hasattr(C, "values")) self.assertEqual(list(C), self.Akeys) def testEmptyIntersection(self): for A in self.As: for E in self.emptys: C = self.intersection(A, E) self.assertTrue(not hasattr(C, "values")) self.assertEqual(list(C), []) C = self.intersection(E, A) self.assertTrue(not hasattr(C, "values")) self.assertEqual(list(C), []) def testEmptyDifference(self): for A in self.As: for E in self.emptys: C = self.difference(A, E) # Difference preserves LHS values. self.assertEqual(hasattr(C, "values"), hasattr(A, "values")) if hasattr(A, "values"): self.assertEqual(list(C.items()), list(A.items())) else: self.assertEqual(list(C), self.Akeys) C = self.difference(E, A) self.assertEqual(hasattr(C, "values"), hasattr(E, "values")) self.assertEqual(list(C), []) def testUnion(self): inputs = self.As + self.Bs for A in inputs: for B in inputs: C = self.union(A, B) self.assertTrue(not hasattr(C, "values")) self.assertEqual(list(C), self._union(A, B)) def testIntersection(self): inputs = self.As + self.Bs for A in inputs: for B in inputs: C = self.intersection(A, B) self.assertTrue(not hasattr(C, "values")) self.assertEqual(list(C), self._intersection(A, B)) def testDifference(self): inputs = self.As + self.Bs for A in inputs: for B in inputs: C = self.difference(A, B) # Difference preserves LHS values. self.assertEqual(hasattr(C, "values"), hasattr(A, "values")) want = self._difference(A, B) if hasattr(A, "values"): self.assertEqual(list(C.items()), want) else: self.assertEqual(list(C), want) def testLargerInputs(self): from BTrees.IIBTree import IISet from random import randint MAXSIZE = 200 MAXVAL = 400 for i in range(3): n = randint(0, MAXSIZE) Akeys = [randint(1, MAXVAL) for j in range(n)] As = [makeset(Akeys) for makeset in self.builders()] Akeys = IISet(Akeys) n = randint(0, MAXSIZE) Bkeys = [randint(1, MAXVAL) for j in range(n)] Bs = [makeset(Bkeys) for makeset in self.builders()] Bkeys = IISet(Bkeys) for op, simulator in ((self.union, self._union), (self.intersection, self._intersection), (self.difference, self._difference)): for A in As: for B in Bs: got = op(A, B) want = simulator(Akeys, Bkeys) self.assertEqual(list(got), want, (A, B, Akeys, Bkeys, list(got), want)) # Subclasses must set up (as class variables): # weightedUnion, weightedIntersection # builders -- sequence of constructors, taking items # union, intersection -- the module routines of those names # mkbucket -- the module bucket builder class Weighted(object): def setUp(self): self.Aitems = [(1, 10), (3, 30), (5, 50), (6, 60)] self.Bitems = [(2, 21), (3, 31), (4, 41), (6, 61), (7, 71)] self.As = [make(self.Aitems) for make in self.builders()] self.Bs = [make(self.Bitems) for make in self.builders()] self.emptys = [make([]) for make in self.builders()] weights = [] for w1 in -3, -1, 0, 1, 7: for w2 in -3, -1, 0, 1, 7: weights.append((w1, w2)) self.weights = weights def testBothNone(self): for op in self.weightedUnion(), self.weightedIntersection(): w, C = op(None, None) self.assertTrue(C is None) self.assertEqual(w, 0) w, C = op(None, None, 42, 666) self.assertTrue(C is None) self.assertEqual(w, 0) def testLeftNone(self): for op in self.weightedUnion(), self.weightedIntersection(): for A in self.As + self.emptys: w, C = op(None, A) self.assertTrue(C is A) self.assertEqual(w, 1) w, C = op(None, A, 42, 666) self.assertTrue(C is A) self.assertEqual(w, 666) def testRightNone(self): for op in self.weightedUnion(), self.weightedIntersection(): for A in self.As + self.emptys: w, C = op(A, None) self.assertTrue(C is A) self.assertEqual(w, 1) w, C = op(A, None, 42, 666) self.assertTrue(C is A) self.assertEqual(w, 42) # If obj is a set, return a bucket with values all 1; else return obj. def _normalize(self, obj): if isaset(obj): obj = self.mkbucket(list(zip(obj, [1] * len(obj)))) return obj # Python simulation of weightedUnion. def _wunion(self, A, B, w1=1, w2=1): if isaset(A) and isaset(B): return 1, self.union()(A, B).keys() A = self._normalize(A) B = self._normalize(B) result = [] for key in self.union()(A, B): v1 = A.get(key, 0) v2 = B.get(key, 0) result.append((key, v1*w1 + v2*w2)) return 1, result def testUnion(self): inputs = self.As + self.Bs + self.emptys for A in inputs: for B in inputs: want_w, want_s = self._wunion(A, B) got_w, got_s = self.weightedUnion()(A, B) self.assertEqual(got_w, want_w) if isaset(got_s): self.assertEqual(got_s.keys(), want_s) else: self.assertEqual(got_s.items(), want_s) for w1, w2 in self.weights: want_w, want_s = self._wunion(A, B, w1, w2) got_w, got_s = self.weightedUnion()(A, B, w1, w2) self.assertEqual(got_w, want_w) if isaset(got_s): self.assertEqual(got_s.keys(), want_s) else: self.assertEqual(got_s.items(), want_s) # Python simulation weightedIntersection. def _wintersection(self, A, B, w1=1, w2=1): if isaset(A) and isaset(B): return w1 + w2, self.intersection()(A, B).keys() A = self._normalize(A) B = self._normalize(B) result = [] for key in self.intersection()(A, B): result.append((key, A[key]*w1 + B[key]*w2)) return 1, result def testIntersection(self): inputs = self.As + self.Bs + self.emptys for A in inputs: for B in inputs: want_w, want_s = self._wintersection(A, B) got_w, got_s = self.weightedIntersection()(A, B) self.assertEqual(got_w, want_w) if isaset(got_s): self.assertEqual(got_s.keys(), want_s) else: self.assertEqual(got_s.items(), want_s) for w1, w2 in self.weights: want_w, want_s = self._wintersection(A, B, w1, w2) got_w, got_s = self.weightedIntersection()(A, B, w1, w2) self.assertEqual(got_w, want_w) if isaset(got_s): self.assertEqual(got_s.keys(), want_s) else: self.assertEqual(got_s.items(), want_s) # Given a set builder (like OITreeSet or OISet), return a function that # takes a list of (key, value) pairs and builds a set out of the keys. def itemsToSet(setbuilder): def result(items, setbuilder=setbuilder): return setbuilder([key for key, value in items]) return result # 'thing' is a bucket, btree, set or treeset. Return true iff it's one of the # latter two. def isaset(thing): return not hasattr(thing, 'values') # Subclasses must set up (as class variables): # multiunion, union # mkset, mktreeset # mkbucket, mkbtree class MultiUnion(object): def testEmpty(self): self.assertEqual(len(self.multiunion([])), 0) def testOne(self): for sequence in ([3], list(range(20)), list(range(-10, 0, 2)) + list(range(1, 10, 2)), ): seq1 = sequence[:] seq2 = list(reversed(sequence[:])) seqsorted = sorted(sequence[:]) for seq in seq1, seq2, seqsorted: for builder in self.mkset, self.mktreeset: input = builder(seq) output = self.multiunion([input]) self.assertEqual(len(seq), len(output)) self.assertEqual(seqsorted, list(output)) def testValuesIgnored(self): for builder in self.mkbucket, self.mkbtree: input = builder([(1, 2), (3, 4), (5, 6)]) output = self.multiunion([input]) self.assertEqual([1, 3, 5], list(output)) def testBigInput(self): N = 100000 input = self.mkset(list(range(N))) output = self.multiunion([input] * 10) self.assertEqual(len(output), N) self.assertEqual(output.minKey(), 0) self.assertEqual(output.maxKey(), N-1) self.assertEqual(list(output), list(range(N))) def testLotsOfLittleOnes(self): from random import shuffle N = 5000 inputs = [] mkset, mktreeset = self.mkset, self.mktreeset for i in range(N): base = i * 4 - N inputs.append(mkset([base, base+1])) inputs.append(mktreeset([base+2, base+3])) shuffle(inputs) output = self.multiunion(inputs) self.assertEqual(len(output), N*4) self.assertEqual(list(output), list(range(-N, 3*N))) def testFunkyKeyIteration(self): # The internal set iteration protocol allows "iterating over" a # a single key as if it were a set. N = 100 union, mkset = self.union, self.mkset slow = mkset() for i in range(N): slow = union(slow, mkset([i])) fast = self.multiunion(list(range(N))) # like N distinct singleton sets self.assertEqual(len(slow), N) self.assertEqual(len(fast), N) self.assertEqual(list(slow), list(fast)) self.assertEqual(list(fast), list(range(N))) class ConflictTestBase(object): # Tests common to all types: sets, buckets, and BTrees storage = None def tearDown(self): import transaction transaction.abort() if self.storage is not None: self.storage.close() self.storage.cleanup() def _makeOne(self): return self._getTargetClass()() def openDB(self): import os from ZODB.FileStorage import FileStorage from ZODB.DB import DB n = 'fs_tmp__%s' % os.getpid() self.storage = FileStorage(n) self.db = DB(self.storage) return self.db def _test_merge(o1, o2, o3, expect, message='failed to merge', should_fail=0): from BTrees.Interfaces import BTreesConflictError s1 = o1.__getstate__() s2 = o2.__getstate__() s3 = o3.__getstate__() expected = expect.__getstate__() if expected is None: expected = ((((),),),) if should_fail: try: merged = o1._p_resolveConflict(s1, s2, s3) except BTreesConflictError as err: pass else: assert 0, message else: merged = o1._p_resolveConflict(s1, s2, s3) assert merged == expected, message class MappingConflictTestBase(ConflictTestBase): # Tests common to mappings (buckets, btrees). def _deletefail(self): t = self._makeOne() del t[1] def _setupConflict(self): l=[ -5124, -7377, 2274, 8801, -9901, 7327, 1565, 17, -679, 3686, -3607, 14, 6419, -5637, 6040, -4556, -8622, 3847, 7191, -4067] e1=[(-1704, 0), (5420, 1), (-239, 2), (4024, 3), (-6984, 4)] e2=[(7745, 0), (4868, 1), (-2548, 2), (-2711, 3), (-3154, 4)] base = self._makeOne() base.update([(i, i*i) for i in l[:20]]) b1 = type(base)(base) b2 = type(base)(base) bm = type(base)(base) items=base.items() return base, b1, b2, bm, e1, e2, items def testMergeDelete(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() del b1[items[1][0]] del b2[items[5][0]] del b1[items[-1][0]] del b2[items[-2][0]] del bm[items[1][0]] del bm[items[5][0]] del bm[items[-1][0]] del bm[items[-2][0]] _test_merge(base, b1, b2, bm, 'merge delete') def testMergeDeleteAndUpdate(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() del b1[items[1][0]] b2[items[5][0]]=1 del b1[items[-1][0]] b2[items[-2][0]]=2 del bm[items[1][0]] bm[items[5][0]]=1 del bm[items[-1][0]] bm[items[-2][0]]=2 _test_merge(base, b1, b2, bm, 'merge update and delete') def testMergeUpdate(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1[items[0][0]]=1 b2[items[5][0]]=2 b1[items[-1][0]]=3 b2[items[-2][0]]=4 bm[items[0][0]]=1 bm[items[5][0]]=2 bm[items[-1][0]]=3 bm[items[-2][0]]=4 _test_merge(base, b1, b2, bm, 'merge update') def testFailMergeDelete(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() del b1[items[0][0]] del b2[items[0][0]] _test_merge(base, b1, b2, bm, 'merge conflicting delete', should_fail=1) def testFailMergeUpdate(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1[items[0][0]]=1 b2[items[0][0]]=2 _test_merge(base, b1, b2, bm, 'merge conflicting update', should_fail=1) def testFailMergeDeleteAndUpdate(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() del b1[items[0][0]] b2[items[0][0]]=-9 _test_merge(base, b1, b2, bm, 'merge conflicting update and delete', should_fail=1) def testMergeInserts(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1[-99999]=-99999 b1[e1[0][0]]=e1[0][1] b2[99999]=99999 b2[e1[2][0]]=e1[2][1] bm[-99999]=-99999 bm[e1[0][0]]=e1[0][1] bm[99999]=99999 bm[e1[2][0]]=e1[2][1] _test_merge(base, b1, b2, bm, 'merge insert') def testMergeInsertsFromEmpty(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() base.clear() b1.clear() b2.clear() bm.clear() b1.update(e1) bm.update(e1) b2.update(e2) bm.update(e2) _test_merge(base, b1, b2, bm, 'merge insert from empty') def testFailMergeEmptyAndFill(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.clear() bm.clear() b2.update(e2) bm.update(e2) _test_merge(base, b1, b2, bm, 'merge insert from empty', should_fail=1) def testMergeEmpty(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.clear() bm.clear() _test_merge(base, b1, b2, bm, 'empty one and not other', should_fail=1) def testFailMergeInsert(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1[-99999]=-99999 b1[e1[0][0]]=e1[0][1] b2[99999]=99999 b2[e1[0][0]]=e1[0][1] _test_merge(base, b1, b2, bm, 'merge conflicting inserts', should_fail=1) class SetConflictTestBase(ConflictTestBase): "Set (as opposed to TreeSet) specific tests." def _setupConflict(self): l=[ -5124, -7377, 2274, 8801, -9901, 7327, 1565, 17, -679, 3686, -3607, 14, 6419, -5637, 6040, -4556, -8622, 3847, 7191, -4067] e1=[-1704, 5420, -239, 4024, -6984] e2=[7745, 4868, -2548, -2711, -3154] base = self._makeOne() base.update(l) b1=base.__class__(base) b2=base.__class__(base) bm=base.__class__(base) items=base.keys() return base, b1, b2, bm, e1, e2, items def testMergeDelete(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.remove(items[1]) b2.remove(items[5]) b1.remove(items[-1]) b2.remove(items[-2]) bm.remove(items[1]) bm.remove(items[5]) bm.remove(items[-1]) bm.remove(items[-2]) _test_merge(base, b1, b2, bm, 'merge delete') def testFailMergeDelete(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.remove(items[0]) b2.remove(items[0]) _test_merge(base, b1, b2, bm, 'merge conflicting delete', should_fail=1) def testMergeInserts(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.insert(-99999) b1.insert(e1[0]) b2.insert(99999) b2.insert(e1[2]) bm.insert(-99999) bm.insert(e1[0]) bm.insert(99999) bm.insert(e1[2]) _test_merge(base, b1, b2, bm, 'merge insert') def testMergeInsertsFromEmpty(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() base.clear() b1.clear() b2.clear() bm.clear() b1.update(e1) bm.update(e1) b2.update(e2) bm.update(e2) _test_merge(base, b1, b2, bm, 'merge insert from empty') def testFailMergeEmptyAndFill(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.clear() bm.clear() b2.update(e2) bm.update(e2) _test_merge(base, b1, b2, bm, 'merge insert from empty', should_fail=1) def testMergeEmpty(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.clear() bm.clear() _test_merge(base, b1, b2, bm, 'empty one and not other', should_fail=1) def testFailMergeInsert(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.insert(-99999) b1.insert(e1[0]) b2.insert(99999) b2.insert(e1[0]) _test_merge(base, b1, b2, bm, 'merge conflicting inserts', should_fail=1) ## utility functions def lsubtract(l1, l2): l1 = list(l1) l2 = list(l2) return (list(filter(lambda x, l1=l1: x not in l1, l2)) + list(filter(lambda x, l2=l2: x not in l2, l1))) def realseq(itemsob): return [x for x in itemsob] def permutations(x): # Return a list of all permutations of list x. n = len(x) if n <= 1: return [x] result = [] x0 = x[0] for i in range(n): # Build the (n-1)! permutations with x[i] in the first position. xcopy = x[:] first, xcopy[i] = xcopy[i], x0 result.extend([[first] + p for p in permutations(xcopy[1:])]) return result BTrees-4.3.1/BTrees/tests/test_check.py0000664000175000017500000003122412510775554021102 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest def _assertRaises(self, e_type, checked, *args, **kw): try: checked(*args, **kw) except e_type as e: return e self.fail("Didn't raise: %s" % e_type.__name__) class Test_classify(unittest.TestCase): def _callFUT(self, obj): from BTrees.check import classify return classify(obj) def test_classify_w_unknown(self): class NotClassified(object): pass self.assertRaises(KeyError, self._callFUT, NotClassified()) def test_classify_w_bucket(self): from BTrees.OOBTree import OOBucketPy from BTrees.check import TYPE_BUCKET kind, is_mapping = self._callFUT(OOBucketPy()) self.assertEqual(kind, TYPE_BUCKET) self.assertTrue(is_mapping) def test_classify_w_set(self): from BTrees.OOBTree import OOSetPy from BTrees.check import TYPE_BUCKET kind, is_mapping = self._callFUT(OOSetPy()) self.assertEqual(kind, TYPE_BUCKET) self.assertFalse(is_mapping) def test_classify_w_tree(self): from BTrees.OOBTree import OOBTreePy from BTrees.check import TYPE_BTREE kind, is_mapping = self._callFUT(OOBTreePy()) self.assertEqual(kind, TYPE_BTREE) self.assertTrue(is_mapping) def test_classify_w_treeset(self): from BTrees.OOBTree import OOTreeSetPy from BTrees.check import TYPE_BTREE kind, is_mapping = self._callFUT(OOTreeSetPy()) self.assertEqual(kind, TYPE_BTREE) self.assertFalse(is_mapping) class Test_crack_btree(unittest.TestCase): def _callFUT(self, obj, is_mapping): from BTrees.check import crack_btree return crack_btree(obj, is_mapping) def test_w_empty_tree(self): from BTrees.check import BTREE_EMPTY class Empty(object): def __getstate__(self): return None kind, keys, kids = self._callFUT(Empty(), True) self.assertEqual(kind, BTREE_EMPTY) self.assertEqual(keys, []) self.assertEqual(kids, []) def test_w_degenerate_tree(self): from BTrees.check import BTREE_ONE class Degenerate(object): def __getstate__(self): return ((('a', 1, 'b', 2),),) kind, keys, kids = self._callFUT(Degenerate(), True) self.assertEqual(kind, BTREE_ONE) self.assertEqual(keys, ('a', 1, 'b', 2)) self.assertEqual(kids, None) def test_w_normal_tree(self): from BTrees.check import BTREE_NORMAL first_bucket = [object()] * 8 second_bucket = [object()] * 8 class Normal(object): def __getstate__(self): return ((first_bucket, 'b', second_bucket), first_bucket) kind, keys, kids = self._callFUT(Normal(), True) self.assertEqual(kind, BTREE_NORMAL) self.assertEqual(keys, ['b']) self.assertEqual(kids, [first_bucket, second_bucket]) class Test_crack_bucket(unittest.TestCase): def _callFUT(self, obj, is_mapping): from BTrees.check import crack_bucket return crack_bucket(obj, is_mapping) def test_w_empty_set(self): class EmptySet(object): def __getstate__(self): return ([],) keys, values = self._callFUT(EmptySet(), False) self.assertEqual(keys, []) self.assertEqual(values, []) def test_w_non_empty_set(self): class NonEmptySet(object): def __getstate__(self): return (['a', 'b', 'c'],) keys, values = self._callFUT(NonEmptySet(), False) self.assertEqual(keys, ['a', 'b', 'c']) self.assertEqual(values, []) def test_w_empty_mapping(self): class EmptyMapping(object): def __getstate__(self): return ([], object()) keys, values = self._callFUT(EmptyMapping(), True) self.assertEqual(keys, []) self.assertEqual(values, []) def test_w_non_empty_mapping(self): class NonEmptyMapping(object): def __getstate__(self): return (['a', 1, 'b', 2, 'c', 3], object()) keys, values = self._callFUT(NonEmptyMapping(), True) self.assertEqual(keys, ['a', 'b', 'c']) self.assertEqual(values, [1, 2, 3]) class Test_type_and_adr(unittest.TestCase): def _callFUT(self, obj): from BTrees.check import type_and_adr return type_and_adr(obj) def test_type_and_adr_w_oid(self): from BTrees.utils import oid_repr class WithOid(object): _p_oid = b'DEADBEEF' t_and_a = self._callFUT(WithOid()) self.assertTrue(t_and_a.startswith('WithOid (0x')) self.assertTrue(t_and_a.endswith('oid=%s)' % oid_repr(b'DEADBEEF'))) def test_type_and_adr_wo_oid(self): class WithoutOid(object): pass t_and_a = self._callFUT(WithoutOid()) self.assertTrue(t_and_a.startswith('WithoutOid (0x')) self.assertTrue(t_and_a.endswith('oid=None)')) class WalkerTests(unittest.TestCase): def _getTargetClass(self): from BTrees.check import Walker return Walker def _makeOne(self, obj): return self._getTargetClass()(obj) def test_visit_btree_abstract(self): walker = self._makeOne(object()) obj = object() path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None self.assertRaises(NotImplementedError, walker.visit_btree, obj, path, parent, is_mapping, keys, kids, lo, hi) def test_visit_bucket_abstract(self): walker = self._makeOne(object()) obj = object() path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None self.assertRaises(NotImplementedError, walker.visit_bucket, obj, path, parent, is_mapping, keys, kids, lo, hi) def test_walk_w_empty_bucket(self): from BTrees.OOBTree import OOBucket obj = OOBucket() walker = self._makeOne(obj) path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None self.assertRaises(NotImplementedError, walker.walk) def test_walk_w_empty_btree(self): from BTrees.OOBTree import OOBTree obj = OOBTree() walker = self._makeOne(obj) path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None self.assertRaises(NotImplementedError, walker.walk) def test_walk_w_degenerate_btree(self): from BTrees.OOBTree import OOBTree obj = OOBTree() obj['a'] = 1 walker = self._makeOne(obj) path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None self.assertRaises(NotImplementedError, walker.walk) def test_walk_w_normal_btree(self): from BTrees.IIBTree import IIBTree obj = IIBTree() for i in range(1000): obj[i] = i walker = self._makeOne(obj) path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None self.assertRaises(NotImplementedError, walker.walk) class CheckerTests(unittest.TestCase): assertRaises = _assertRaises def _getTargetClass(self): from BTrees.check import Checker return Checker def _makeOne(self, obj): return self._getTargetClass()(obj) def test_walk_w_empty_bucket(self): from BTrees.OOBTree import OOBucket obj = OOBucket() checker = self._makeOne(obj) path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None checker.check() #noraise def test_walk_w_empty_btree(self): obj = _makeTree(False) checker = self._makeOne(obj) path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None checker.check() #noraise def test_walk_w_degenerate_btree(self): obj = _makeTree(False) obj['a'] = 1 checker = self._makeOne(obj) path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None checker.check() #noraise def test_walk_w_normal_btree(self): obj = _makeTree(False) checker = self._makeOne(obj) path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None checker.check() #noraise def test_walk_w_key_too_large(self): obj = _makeTree(True) state = obj.__getstate__() # Damage an invariant by dropping the BTree key to 14. new_state = (state[0][0], 14, state[0][2]), state[1] obj.__setstate__(new_state) checker = self._makeOne(obj) path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None e = self.assertRaises(AssertionError, checker.check) self.assertTrue(">= upper bound" in str(e)) def test_walk_w_key_too_small(self): obj = _makeTree(True) state = obj.__getstate__() # Damage an invariant by bumping the BTree key to 16. new_state = (state[0][0], 16, state[0][2]), state[1] obj.__setstate__(new_state) checker = self._makeOne(obj) path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None e = self.assertRaises(AssertionError, checker.check) self.assertTrue("< lower bound" in str(e)) def test_walk_w_keys_swapped(self): obj = _makeTree(True) state = obj.__getstate__() # Damage an invariant by bumping the BTree key to 16. (b0, num, b1), firstbucket = state self.assertEqual(b0[4], 8) self.assertEqual(b0[5], 10) b0state = b0.__getstate__() self.assertEqual(len(b0state), 2) # b0state looks like # ((k0, v0, k1, v1, ...), nextbucket) pairs, nextbucket = b0state self.assertEqual(pairs[8], 4) self.assertEqual(pairs[9], 8) self.assertEqual(pairs[10], 5) self.assertEqual(pairs[11], 10) newpairs = pairs[:8] + (5, 10, 4, 8) + pairs[12:] b0.__setstate__((newpairs, nextbucket)) checker = self._makeOne(obj) path = '/' parent = object() is_mapping = True keys = [] kids = [] lo = 0 hi = None e = self.assertRaises(AssertionError, checker.check) self.assertTrue("key 5 at index 4 >= key 4 at index 5" in str(e)) class Test_check(unittest.TestCase): def _callFUT(self, tree): from BTrees.check import check return check(tree) def _makeOne(self): from BTrees.OOBTree import OOBTree tree = OOBTree() for i in range(31): tree[i] = 2*i return tree def test_normal(self): from BTrees.OOBTree import OOBTree tree = OOBTree() for i in range(31): tree[i] = 2*i state = tree.__getstate__() self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 3) self.assertEqual(state[0][1], 15) self._callFUT(tree) #noraise def _makeTree(fill): from BTrees.OOBTree import OOBTree from BTrees.OOBTree import _BUCKET_SIZE tree = OOBTree() if fill: for i in range(_BUCKET_SIZE + 1): tree[i] = 2*i return tree def test_suite(): return unittest.TestSuite(( unittest.makeSuite(Test_classify), unittest.makeSuite(Test_crack_btree), unittest.makeSuite(Test_crack_bucket), unittest.makeSuite(Test_type_and_adr), unittest.makeSuite(WalkerTests), unittest.makeSuite(CheckerTests), unittest.makeSuite(Test_check), )) BTrees-4.3.1/BTrees/tests/test_OOBTree.py0000664000175000017500000002214212510775554021263 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from .common import BTreeTests from .common import ExtendedSetTests from .common import InternalKeysMappingTest from .common import InternalKeysSetTest from .common import MappingBase from .common import MappingConflictTestBase from .common import ModuleTest from .common import NormalSetTests from .common import SetResult from .common import SetConflictTestBase from .common import makeBuilder class OOBTreeInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOBTreePy return OOBTreePy class OOBTreePyInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOBTree return OOBTree class OOTreeSetInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOTreeSet return OOTreeSet class OOTreeSetPyInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOTreeSetPy return OOTreeSetPy class OOBucketTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOBucket return OOBucket class OOBucketPyTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOBucketPy return OOBucketPy class OOTreeSetTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOTreeSet return OOTreeSet class OOTreeSetPyTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOTreeSetPy return OOTreeSetPy class OOSetTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOSet return OOSet class OOSetPyTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOSetPy return OOSetPy class OOBTreeTest(BTreeTests, unittest.TestCase): def _makeOne(self, *args): from BTrees.OOBTree import OOBTree return OOBTree(*args) def test_byValue(self): ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] tree = self._makeOne(ITEMS) self.assertEqual(list(tree.byValue(22)), [(y, x) for x, y in reversed(ITEMS[22:])]) def testRejectDefaultComparison(self): # Check that passing int keys w default comparison fails. # Only applies to new-style class instances. Old-style # instances are too hard to introspect. # This is white box because we know that the check is being # used in a function that's used in lots of places. # Otherwise, there are many permutations that would have to be # checked. from .._compat import PY2 t = self._makeOne() class C(object): pass self.assertRaises(TypeError, lambda : t.__setitem__(C(), 1)) if PY2: # we only check for __cmp__ on Python2 class With___cmp__(object): def __cmp__(*args): return 1 c = With___cmp__() t[c] = 1 t.clear() class With___lt__(object): def __lt__(*args): return 1 c = With___lt__() t[c] = 1 t.clear() class OOBTreePyTest(OOBTreeTest): # # Right now, we can't match the C extension's test / prohibition of the # default 'object' comparison semantics. #class OOBTreePyTest(BTreeTests, unittest.TestCase): def _makeOne(self, *args): from BTrees.OOBTree import OOBTreePy return OOBTreePy(*args) class PureOO(SetResult, unittest.TestCase): def union(self, *args): from BTrees.OOBTree import union return union(*args) def intersection(self, *args): from BTrees.OOBTree import intersection return intersection(*args) def difference(self, *args): from BTrees.OOBTree import difference return difference(*args) def builders(self): from BTrees.OOBTree import OOBTree from BTrees.OOBTree import OOBucket from BTrees.OOBTree import OOTreeSet from BTrees.OOBTree import OOSet return OOSet, OOTreeSet, makeBuilder(OOBTree), makeBuilder(OOBucket) class PureOOPy(SetResult, unittest.TestCase): def union(self, *args): from BTrees.OOBTree import unionPy return unionPy(*args) def intersection(self, *args): from BTrees.OOBTree import intersectionPy return intersectionPy(*args) def difference(self, *args): from BTrees.OOBTree import differencePy return differencePy(*args) def builders(self): from BTrees.OOBTree import OOBTreePy from BTrees.OOBTree import OOBucketPy from BTrees.OOBTree import OOTreeSetPy from BTrees.OOBTree import OOSetPy return (OOSetPy, OOTreeSetPy, makeBuilder(OOBTreePy), makeBuilder(OOBucketPy)) class OOBucketConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOBucket return OOBucket class OOBucketPyConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOBucketPy return OOBucketPy class OOSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOSet return OOSet class OOSetPyConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOSetPy return OOSetPy class OOBTreeConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOBTree return OOBTree class OOBTreePyConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOBTreePy return OOBTreePy class OOTreeSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOTreeSet return OOTreeSet class OOTreeSetPyConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OOBTree import OOTreeSetPy return OOTreeSetPy class OOModuleTest(ModuleTest, unittest.TestCase): prefix = 'OO' def _getModule(self): import BTrees return BTrees.OOBTree def _getInterface(self): import BTrees.Interfaces return BTrees.Interfaces.IObjectObjectBTreeModule def test_weightedUnion_not_present(self): try: from BTrees.OOBTree import weightedUnion except ImportError: pass else: self.fail("OOBTree shouldn't have weightedUnion") def test_weightedIntersection_not_present(self): try: from BTrees.OOBTree import weightedIntersection except ImportError: pass else: self.fail("OOBTree shouldn't have weightedIntersection") def test_multiunion_not_present(self): try: from BTrees.OOBTree import multiunion except ImportError: pass else: self.fail("OOBTree shouldn't have multiunion") def test_suite(): return unittest.TestSuite(( unittest.makeSuite(OOBTreeInternalKeyTest), unittest.makeSuite(OOBTreePyInternalKeyTest), unittest.makeSuite(OOTreeSetInternalKeyTest), unittest.makeSuite(OOTreeSetPyInternalKeyTest), unittest.makeSuite(OOBucketTest), unittest.makeSuite(OOBucketPyTest), unittest.makeSuite(OOTreeSetTest), unittest.makeSuite(OOTreeSetPyTest), unittest.makeSuite(OOSetTest), unittest.makeSuite(OOSetPyTest), unittest.makeSuite(OOBTreeTest), unittest.makeSuite(OOBTreePyTest), unittest.makeSuite(PureOO), unittest.makeSuite(PureOOPy), unittest.makeSuite(OOBucketConflictTests), unittest.makeSuite(OOBucketPyConflictTests), unittest.makeSuite(OOSetConflictTests), unittest.makeSuite(OOSetPyConflictTests), unittest.makeSuite(OOBTreeConflictTests), unittest.makeSuite(OOBTreePyConflictTests), unittest.makeSuite(OOTreeSetConflictTests), unittest.makeSuite(OOTreeSetPyConflictTests), unittest.makeSuite(OOModuleTest), )) BTrees-4.3.1/BTrees/tests/test_LLBTree.py0000664000175000017500000002676012510775554021267 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from .common import BTreeTests from .common import ExtendedSetTests from .common import I_SetsBase from .common import InternalKeysMappingTest from .common import InternalKeysSetTest from .common import MappingBase from .common import MappingConflictTestBase from .common import ModuleTest from .common import MultiUnion from .common import NormalSetTests from .common import SetConflictTestBase from .common import SetResult from .common import TestLongIntKeys from .common import TestLongIntValues from .common import Weighted from .common import itemsToSet from .common import makeBuilder class LLBTreeInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLBTree return LLBTree class LLBTreePyInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLBTreePy return LLBTreePy class LLTreeSetInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLTreeSet return LLTreeSet class LLTreeSetPyInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLTreeSetPy return LLTreeSetPy class LLBucketTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLBucket return LLBucket class LLBucketTestPy(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLBucketPy return LLBucketPy class LLTreeSetTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLTreeSet return LLTreeSet class LLTreeSetTestPy(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLTreeSetPy return LLTreeSetPy class LLSetTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLSet return LLSet class LLSetTestPy(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLSetPy return LLSetPy class LLBTreeTest(BTreeTests, TestLongIntKeys, TestLongIntValues, unittest.TestCase): def _makeOne(self): from BTrees.LLBTree import LLBTree return LLBTree() def getTwoValues(self): return 1, 2 class LLBTreeTestPy(BTreeTests, TestLongIntKeys, TestLongIntValues, unittest.TestCase): def _makeOne(self): from BTrees.LLBTree import LLBTreePy return LLBTreePy() def getTwoValues(self): return 1, 2 class TestLLSets(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.LLBTree import LLSet return LLSet() class TestLLSetsPy(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.LLBTree import LLSetPy return LLSetPy() class TestLLTreeSets(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.LLBTree import LLTreeSet return LLTreeSet() class TestLLTreeSetsPy(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.LLBTree import LLTreeSetPy return LLTreeSetPy() class PureLL(SetResult, unittest.TestCase): def union(self, *args): from BTrees.LLBTree import union return union(*args) def intersection(self, *args): from BTrees.LLBTree import intersection return intersection(*args) def difference(self, *args): from BTrees.LLBTree import difference return difference(*args) def builders(self): from BTrees.LLBTree import LLBTree from BTrees.LLBTree import LLBucket from BTrees.LLBTree import LLTreeSet from BTrees.LLBTree import LLSet return LLSet, LLTreeSet, makeBuilder(LLBTree), makeBuilder(LLBucket) class PureLLPy(SetResult, unittest.TestCase): def union(self, *args): from BTrees.LLBTree import unionPy return unionPy(*args) def intersection(self, *args): from BTrees.LLBTree import intersectionPy return intersectionPy(*args) def difference(self, *args): from BTrees.LLBTree import differencePy return differencePy(*args) def builders(self): from BTrees.LLBTree import LLBTreePy from BTrees.LLBTree import LLBucketPy from BTrees.LLBTree import LLTreeSetPy from BTrees.LLBTree import LLSetPy return (LLSetPy, LLTreeSetPy, makeBuilder(LLBTreePy), makeBuilder(LLBucketPy)) class TestLLMultiUnion(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.LLBTree import multiunion return multiunion(*args) def union(self, *args): from BTrees.LLBTree import union return union(*args) def mkset(self, *args): from BTrees.LLBTree import LLSet as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.LLBTree import LLTreeSet as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.LLBTree import LLBucket as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.LLBTree import LLBTree as mkbtree return mkbtree(*args) class TestLLMultiUnionPy(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.LLBTree import multiunionPy return multiunionPy(*args) def union(self, *args): from BTrees.LLBTree import unionPy return unionPy(*args) def mkset(self, *args): from BTrees.LLBTree import LLSetPy as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.LLBTree import LLTreeSetPy as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.LLBTree import LLBucketPy as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.LLBTree import LLBTreePy as mkbtree return mkbtree(*args) class TestWeightedLL(Weighted, unittest.TestCase): def weightedUnion(self): from BTrees.LLBTree import weightedUnion return weightedUnion def weightedIntersection(self): from BTrees.LLBTree import weightedIntersection return weightedIntersection def union(self): from BTrees.LLBTree import union return union def intersection(self): from BTrees.LLBTree import intersection return intersection def mkbucket(self, *args): from BTrees.LLBTree import LLBucket as mkbucket return mkbucket(*args) def builders(self): from BTrees.LLBTree import LLBTree from BTrees.LLBTree import LLBucket from BTrees.LLBTree import LLTreeSet from BTrees.LLBTree import LLSet return LLBucket, LLBTree, itemsToSet(LLSet), itemsToSet(LLTreeSet) class TestWeightedLLPy(Weighted, unittest.TestCase): def weightedUnion(self): from BTrees.LLBTree import weightedUnionPy return weightedUnionPy def weightedIntersection(self): from BTrees.LLBTree import weightedIntersectionPy return weightedIntersectionPy def union(self): from BTrees.LLBTree import unionPy return unionPy def intersection(self): from BTrees.LLBTree import intersectionPy return intersectionPy def mkbucket(self, *args): from BTrees.LLBTree import LLBucketPy as mkbucket return mkbucket(*args) def builders(self): from BTrees.LLBTree import LLBTreePy from BTrees.LLBTree import LLBucketPy from BTrees.LLBTree import LLTreeSetPy from BTrees.LLBTree import LLSetPy return (LLBucketPy, LLBTreePy, itemsToSet(LLSetPy), itemsToSet(LLTreeSetPy)) class LLBTreeConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLBTree return LLBTree class LLBTreeConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLBTreePy return LLBTreePy class LLBucketConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLBucket return LLBucket class LLBucketConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLBucketPy return LLBucketPy class LLTreeSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLTreeSet return LLTreeSet class LLTreeSetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLTreeSetPy return LLTreeSetPy class LLSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLSet return LLSet class LLSetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LLBTree import LLSetPy return LLSetPy class LLModuleTest(ModuleTest, unittest.TestCase): prefix = 'LL' def _getModule(self): import BTrees return BTrees.LLBTree def _getInterface(self): import BTrees.Interfaces return BTrees.Interfaces.IIntegerIntegerBTreeModule def test_suite(): return unittest.TestSuite(( unittest.makeSuite(LLBTreeInternalKeyTest), unittest.makeSuite(LLBTreeInternalKeyTest), unittest.makeSuite(LLTreeSetInternalKeyTest), unittest.makeSuite(LLTreeSetInternalKeyTest), unittest.makeSuite(LLBucketTest), unittest.makeSuite(LLBucketTest), unittest.makeSuite(LLTreeSetTest), unittest.makeSuite(LLTreeSetTest), unittest.makeSuite(LLSetTest), unittest.makeSuite(LLSetTest), unittest.makeSuite(LLBTreeTest), unittest.makeSuite(LLBTreeTest), unittest.makeSuite(TestLLSets), unittest.makeSuite(TestLLSets), unittest.makeSuite(TestLLTreeSets), unittest.makeSuite(TestLLTreeSets), unittest.makeSuite(TestLLMultiUnion), unittest.makeSuite(TestLLMultiUnion), unittest.makeSuite(PureLL), unittest.makeSuite(PureLL), unittest.makeSuite(TestWeightedLL), unittest.makeSuite(TestWeightedLL), unittest.makeSuite(LLBTreeConflictTests), unittest.makeSuite(LLBTreeConflictTests), unittest.makeSuite(LLBucketConflictTests), unittest.makeSuite(LLBucketConflictTests), unittest.makeSuite(LLTreeSetConflictTests), unittest.makeSuite(LLTreeSetConflictTests), unittest.makeSuite(LLSetConflictTests), unittest.makeSuite(LLSetConflictTests), unittest.makeSuite(LLModuleTest), )) BTrees-4.3.1/BTrees/tests/test_LOBTree.py0000664000175000017500000002361412510775554021265 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from .common import BTreeTests from .common import ExtendedSetTests from .common import I_SetsBase from .common import InternalKeysMappingTest from .common import InternalKeysSetTest from .common import MappingBase from .common import MappingConflictTestBase from .common import ModuleTest from .common import MultiUnion from .common import NormalSetTests from .common import SetConflictTestBase from .common import SetResult from .common import TestLongIntKeys from .common import makeBuilder class LOBTreeInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOBTree return LOBTree class LOBTreePyInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOBTreePy return LOBTreePy class LOTreeSetInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOTreeSet return LOTreeSet class LOTreeSetPyInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOTreeSetPy return LOTreeSetPy class LOBucketTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOBucket return LOBucket class LOBucketPyTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOBucketPy return LOBucketPy class LOTreeSetTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOTreeSet return LOTreeSet class LOTreeSetPyTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOTreeSetPy return LOTreeSetPy class LOSetTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOSet return LOSet class LOSetPyTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOSetPy return LOSetPy class LOBTreeTest(BTreeTests, TestLongIntKeys, unittest.TestCase): def _makeOne(self): from BTrees.LOBTree import LOBTree return LOBTree() class LOBTreePyTest(BTreeTests, TestLongIntKeys, unittest.TestCase): def _makeOne(self): from BTrees.LOBTree import LOBTreePy return LOBTreePy() class TestLOSets(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.LOBTree import LOSet return LOSet() class TestLOSetsPy(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.LOBTree import LOSetPy return LOSetPy() class TestLOTreeSets(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.LOBTree import LOTreeSet return LOTreeSet() class TestLOTreeSetsPy(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.LOBTree import LOTreeSetPy return LOTreeSetPy() class TestLOMultiUnion(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.LOBTree import multiunion return multiunion(*args) def union(self, *args): from BTrees.LOBTree import union return union(*args) def mkset(self, *args): from BTrees.LOBTree import LOSet as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.LOBTree import LOTreeSet as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.LOBTree import LOBucket as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.LOBTree import LOBTree as mkbtree return mkbtree(*args) class TestLOMultiUnionPy(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.LOBTree import multiunionPy return multiunionPy(*args) def union(self, *args): from BTrees.LOBTree import unionPy return unionPy(*args) def mkset(self, *args): from BTrees.LOBTree import LOSetPy as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.LOBTree import LOTreeSetPy as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.LOBTree import LOBucketPy as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.LOBTree import LOBTreePy as mkbtree return mkbtree(*args) class PureLO(SetResult, unittest.TestCase): def union(self, *args): from BTrees.LOBTree import union return union(*args) def intersection(self, *args): from BTrees.LOBTree import intersection return intersection(*args) def difference(self, *args): from BTrees.LOBTree import difference return difference(*args) def builders(self): from BTrees.LOBTree import LOBTree from BTrees.LOBTree import LOBucket from BTrees.LOBTree import LOTreeSet from BTrees.LOBTree import LOSet return LOSet, LOTreeSet, makeBuilder(LOBTree), makeBuilder(LOBucket) class PureLOPy(SetResult, unittest.TestCase): def union(self, *args): from BTrees.LOBTree import unionPy return unionPy(*args) def intersection(self, *args): from BTrees.LOBTree import intersectionPy return intersectionPy(*args) def difference(self, *args): from BTrees.LOBTree import differencePy return differencePy(*args) def builders(self): from BTrees.LOBTree import LOBTreePy from BTrees.LOBTree import LOBucketPy from BTrees.LOBTree import LOTreeSetPy from BTrees.LOBTree import LOSetPy return (LOSetPy, LOTreeSetPy, makeBuilder(LOBTreePy), makeBuilder(LOBucketPy)) class LOBTreeConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOBTree return LOBTree class LOBTreeConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOBTreePy return LOBTreePy class LOBucketConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOBucket return LOBucket class LOBucketConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOBucketPy return LOBucketPy class LOTreeSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOTreeSet return LOTreeSet class LOTreeSetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOTreeSetPy return LOTreeSetPy class LOSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOSet return LOSet class LOSetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LOBTree import LOSetPy return LOSetPy class LOModuleTest(ModuleTest, unittest.TestCase): prefix = 'LO' def _getModule(self): import BTrees return BTrees.LOBTree def _getInterface(self): import BTrees.Interfaces return BTrees.Interfaces.IIntegerObjectBTreeModule def test_weightedUnion_not_present(self): try: from BTrees.LOBTree import weightedUnion except ImportError: pass else: self.fail("LOBTree shouldn't have weightedUnion") def test_weightedIntersection_not_present(self): try: from BTrees.LOBTree import weightedIntersection except ImportError: pass else: self.fail("LOBTree shouldn't have weightedIntersection") def test_suite(): return unittest.TestSuite(( unittest.makeSuite(LOBTreeInternalKeyTest), unittest.makeSuite(LOBTreePyInternalKeyTest), unittest.makeSuite(LOTreeSetInternalKeyTest), unittest.makeSuite(LOTreeSetPyInternalKeyTest), unittest.makeSuite(LOBucketTest), unittest.makeSuite(LOBucketPyTest), unittest.makeSuite(LOTreeSetTest), unittest.makeSuite(LOTreeSetPyTest), unittest.makeSuite(LOSetTest), unittest.makeSuite(LOSetPyTest), unittest.makeSuite(LOBTreeTest), unittest.makeSuite(LOBTreePyTest), unittest.makeSuite(TestLOSets), unittest.makeSuite(TestLOSetsPy), unittest.makeSuite(TestLOTreeSets), unittest.makeSuite(TestLOTreeSetsPy), unittest.makeSuite(TestLOMultiUnion), unittest.makeSuite(TestLOMultiUnionPy), unittest.makeSuite(PureLO), unittest.makeSuite(PureLOPy), unittest.makeSuite(LOBTreeConflictTests), unittest.makeSuite(LOBTreeConflictTestsPy), unittest.makeSuite(LOBucketConflictTests), unittest.makeSuite(LOBucketConflictTestsPy), unittest.makeSuite(LOTreeSetConflictTests), unittest.makeSuite(LOTreeSetConflictTestsPy), unittest.makeSuite(LOSetConflictTests), unittest.makeSuite(LOSetConflictTestsPy), unittest.makeSuite(LOModuleTest), )) BTrees-4.3.1/BTrees/tests/test_utils.py0000664000175000017500000000521712510775554021170 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest class Test_non_negative(unittest.TestCase): def _callFUT(self, int_val): from BTrees.utils import non_negative return non_negative(int_val) def test_w_big_negative(self): self.assertEqual(self._callFUT(-(2**63 - 1)), 1) def test_w_negative(self): self.assertEqual(self._callFUT(-1), 2**63 - 1) def test_w_zero(self): self.assertEqual(self._callFUT(0), 0) def test_w_positive(self): self.assertEqual(self._callFUT(1), 1) def test_w_big_positive(self): import sys try: self.assertEqual(self._callFUT(sys.maxint), sys.maxint) except AttributeError: #pragma NO COVER Py3k pass class Test_oid_repr(unittest.TestCase): def _callFUT(self, oid): from BTrees.utils import oid_repr return oid_repr(oid) def test_w_non_strings(self): self.assertEqual(self._callFUT(None), repr(None)) self.assertEqual(self._callFUT(()), repr(())) self.assertEqual(self._callFUT([]), repr([])) self.assertEqual(self._callFUT({}), repr({})) self.assertEqual(self._callFUT(0), repr(0)) def test_w_short_strings(self): for length in range(8): faux = 'x' * length self.assertEqual(self._callFUT(faux), repr(faux)) def test_w_long_strings(self): for length in range(9, 1024): faux = 'x' * length self.assertEqual(self._callFUT(faux), repr(faux)) def test_w_zero(self): self.assertEqual(self._callFUT(b'\0\0\0\0\0\0\0\0'), b'0x00') def test_w_one(self): self.assertEqual(self._callFUT(b'\0\0\0\0\0\0\0\1'), b'0x01') def test_w_even_length(self): self.assertEqual(self._callFUT(b'\0\0\0\0\0\0\xAB\xC4'), b'0xabc4') def test_w_odd_length(self): self.assertEqual(self._callFUT(b'\0\0\0\0\0\0\x0D\xEF'), b'0x0def') def test_suite(): return unittest.TestSuite(( unittest.makeSuite(Test_non_negative), unittest.makeSuite(Test_oid_repr), )) BTrees-4.3.1/BTrees/tests/test_IIBTree.py0000664000175000017500000003356512510775554021262 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from .common import BTreeTests from .common import ExtendedSetTests from .common import I_SetsBase from .common import InternalKeysMappingTest from .common import InternalKeysSetTest from .common import MappingBase from .common import MappingConflictTestBase from .common import ModuleTest from .common import MultiUnion from .common import NormalSetTests from .common import SetConflictTestBase from .common import SetResult from .common import TestLongIntKeys from .common import TestLongIntValues from .common import Weighted from .common import itemsToSet from .common import makeBuilder from BTrees.IIBTree import using64bits #XXX Ugly, but unavoidable class IIBTreeInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IIBTree return IIBTree class IIBTreePyInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IIBTreePy return IIBTreePy class IITreeSetInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IITreeSet return IITreeSet class IITreeSetPyInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IITreeSetPy return IITreeSetPy class IIBucketTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IIBucket return IIBucket class IIBucketPyTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IIBucketPy return IIBucketPy class IITreeSetTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IITreeSet return IITreeSet class IITreeSetPyTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IITreeSetPy return IITreeSetPy class IISetTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IISet return IISet class IISetPyTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IISetPy return IISetPy class _IIBTreeTestBase(BTreeTests): def testIIBTreeOverflow(self): good = set() b = self._makeOne() def trial(i): i = int(i) try: b[i] = 0 except OverflowError: self.assertRaises(OverflowError, b.__setitem__, 0, i) except TypeError: self.assertRaises(TypeError, b.__setitem__, 0, i) else: good.add(i) b[0] = i self.assertEqual(b[0], i) for i in range((1<<31) - 3, (1<<31) + 3): trial(i) trial(-i) del b[0] self.assertEqual(sorted(good), sorted(b)) class IIBTreeTest(_IIBTreeTestBase, unittest.TestCase): def _makeOne(self): from BTrees.IIBTree import IIBTree return IIBTree() class IIBTreeTestPy(_IIBTreeTestBase, unittest.TestCase): def _makeOne(self): from BTrees.IIBTree import IIBTreePy return IIBTreePy() if using64bits: class IIBTreeTest(BTreeTests, TestLongIntKeys, TestLongIntValues, unittest.TestCase): def _makeOne(self): from BTrees.IIBTree import IIBTree return IIBTree() def getTwoValues(self): return 1, 2 class IIBTreeTest(BTreeTests, TestLongIntKeys, TestLongIntValues, unittest.TestCase): def _makeOne(self): from BTrees.IIBTree import IIBTreePy return IIBTreePy() def getTwoValues(self): return 1, 2 class _TestIIBTreesBase(object): def testNonIntegerKeyRaises(self): self.assertRaises(TypeError, self._stringraiseskey) self.assertRaises(TypeError, self._floatraiseskey) self.assertRaises(TypeError, self._noneraiseskey) def testNonIntegerValueRaises(self): self.assertRaises(TypeError, self._stringraisesvalue) self.assertRaises(TypeError, self._floatraisesvalue) self.assertRaises(TypeError, self._noneraisesvalue) def _stringraiseskey(self): self._makeOne()['c'] = 1 def _floatraiseskey(self): self._makeOne()[2.5] = 1 def _noneraiseskey(self): self._makeOne()[None] = 1 def _stringraisesvalue(self): self._makeOne()[1] = 'c' def _floatraisesvalue(self): self._makeOne()[1] = 1.4 def _noneraisesvalue(self): self._makeOne()[1] = None class TestIIBTrees(_TestIIBTreesBase, unittest.TestCase): def _makeOne(self): from BTrees.IIBTree import IIBTree return IIBTree() class TestIIBTreesPy(_TestIIBTreesBase, unittest.TestCase): def _makeOne(self): from BTrees.IIBTree import IIBTreePy return IIBTreePy() class TestIISets(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.IIBTree import IISet return IISet() class TestIISetsPy(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.IIBTree import IISetPy return IISetPy() class TestIITreeSets(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.IIBTree import IITreeSet return IITreeSet() class TestIITreeSetsPy(I_SetsBase, unittest.TestCase): def _makeOne(self): from BTrees.IIBTree import IITreeSetPy return IITreeSetPy() class PureII(SetResult, unittest.TestCase): def union(self, *args): from BTrees.IIBTree import union return union(*args) def intersection(self, *args): from BTrees.IIBTree import intersection return intersection(*args) def difference(self, *args): from BTrees.IIBTree import difference return difference(*args) def builders(self): from BTrees.IIBTree import IIBTree from BTrees.IIBTree import IIBucket from BTrees.IIBTree import IITreeSet from BTrees.IIBTree import IISet return IISet, IITreeSet, makeBuilder(IIBTree), makeBuilder(IIBucket) class PureIIPy(SetResult, unittest.TestCase): def union(self, *args): from BTrees.IIBTree import unionPy return unionPy(*args) def intersection(self, *args): from BTrees.IIBTree import intersectionPy return intersectionPy(*args) def difference(self, *args): from BTrees.IIBTree import differencePy return differencePy(*args) def builders(self): from BTrees.IIBTree import IIBTreePy from BTrees.IIBTree import IIBucketPy from BTrees.IIBTree import IITreeSetPy from BTrees.IIBTree import IISetPy return (IISetPy, IITreeSetPy, makeBuilder(IIBTreePy), makeBuilder(IIBucketPy)) class TestIIMultiUnion(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.IIBTree import multiunion return multiunion(*args) def union(self, *args): from BTrees.IIBTree import union return union(*args) def mkset(self, *args): from BTrees.IIBTree import IISet as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.IIBTree import IITreeSet as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.IIBTree import IIBucket as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.IIBTree import IIBTree as mkbtree return mkbtree(*args) class TestIIMultiUnionPy(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.IIBTree import multiunionPy return multiunionPy(*args) def union(self, *args): from BTrees.IIBTree import unionPy return unionPy(*args) def mkset(self, *args): from BTrees.IIBTree import IISetPy as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.IIBTree import IITreeSetPy as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.IIBTree import IIBucketPy as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.IIBTree import IIBTreePy as mkbtree return mkbtree(*args) class TestWeightedII(Weighted, unittest.TestCase): def weightedUnion(self): from BTrees.IIBTree import weightedUnion return weightedUnion def weightedIntersection(self): from BTrees.IIBTree import weightedIntersection return weightedIntersection def union(self): from BTrees.IIBTree import union return union def intersection(self): from BTrees.IIBTree import intersection return intersection def mkbucket(self, *args): from BTrees.IIBTree import IIBucket as mkbucket return mkbucket(*args) def builders(self): from BTrees.IIBTree import IIBTree from BTrees.IIBTree import IIBucket from BTrees.IIBTree import IITreeSet from BTrees.IIBTree import IISet return IIBucket, IIBTree, itemsToSet(IISet), itemsToSet(IITreeSet) class TestWeightedIIPy(Weighted, unittest.TestCase): def weightedUnion(self): from BTrees.IIBTree import weightedUnionPy return weightedUnionPy def weightedIntersection(self): from BTrees.IIBTree import weightedIntersectionPy return weightedIntersectionPy def union(self): from BTrees.IIBTree import unionPy return unionPy def intersection(self): from BTrees.IIBTree import intersectionPy return intersectionPy def mkbucket(self, *args): from BTrees.IIBTree import IIBucketPy as mkbucket return mkbucket(*args) def builders(self): from BTrees.IIBTree import IIBTreePy from BTrees.IIBTree import IIBucketPy from BTrees.IIBTree import IITreeSetPy from BTrees.IIBTree import IISetPy return (IIBucketPy, IIBTreePy, itemsToSet(IISetPy), itemsToSet(IITreeSetPy)) class IIBTreeConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IIBTree return IIBTree class IIBTreeConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IIBTreePy return IIBTreePy class IIBucketConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IIBucket return IIBucket class IIBucketConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IIBucketPy return IIBucketPy class IITreeSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IITreeSet return IITreeSet class IITreeSetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IITreeSetPy return IITreeSetPy class IISetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IISet return IISet class IISetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IIBTree import IISetPy return IISetPy class IIModuleTest(ModuleTest, unittest.TestCase): prefix = 'II' def _getModule(self): import BTrees return BTrees.IIBTree def _getInterface(self): import BTrees.Interfaces return BTrees.Interfaces.IIntegerIntegerBTreeModule def test_suite(): return unittest.TestSuite(( unittest.makeSuite(IIBTreeInternalKeyTest), unittest.makeSuite(IIBTreePyInternalKeyTest), unittest.makeSuite(IITreeSetInternalKeyTest), unittest.makeSuite(IITreeSetPyInternalKeyTest), unittest.makeSuite(IIBucketTest), unittest.makeSuite(IIBucketPyTest), unittest.makeSuite(IITreeSetTest), unittest.makeSuite(IITreeSetPyTest), unittest.makeSuite(IISetTest), unittest.makeSuite(IISetPyTest), unittest.makeSuite(IIBTreeTest), unittest.makeSuite(IIBTreeTestPy), unittest.makeSuite(TestIIBTrees), unittest.makeSuite(TestIIBTreesPy), unittest.makeSuite(TestIISets), unittest.makeSuite(TestIISetsPy), unittest.makeSuite(TestIITreeSets), unittest.makeSuite(TestIITreeSetsPy), unittest.makeSuite(TestIIMultiUnion), unittest.makeSuite(TestIIMultiUnionPy), unittest.makeSuite(PureII), unittest.makeSuite(PureIIPy), unittest.makeSuite(TestWeightedII), unittest.makeSuite(TestWeightedIIPy), unittest.makeSuite(IIBTreeConflictTests), unittest.makeSuite(IIBTreeConflictTestsPy), unittest.makeSuite(IIBucketConflictTests), unittest.makeSuite(IIBucketConflictTestsPy), unittest.makeSuite(IITreeSetConflictTests), unittest.makeSuite(IITreeSetConflictTestsPy), unittest.makeSuite(IISetConflictTests), unittest.makeSuite(IISetConflictTestsPy), unittest.makeSuite(IIModuleTest), )) BTrees-4.3.1/BTrees/tests/testConflict.py0000664000175000017500000005151312526654227021432 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001, 2002 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from .common import _skip_wo_ZODB from .common import ConflictTestBase class NastyConfictFunctionalTests(ConflictTestBase, unittest.TestCase): # FUNCTESTS: Provoke various conflict scenarios using ZODB + transaction def _getTargetClass(self): from BTrees.OOBTree import OOBTree return OOBTree def openDB(self): # The conflict tests tend to open two or more connections # and then try to commit them. A standard FileStorage # is not MVCC aware, and so each connection would have the same # instance of the storage, leading to the error # "Duplicate tpc_begin calls for same transaction" on commit; # thus we use a MVCCMappingStorage for these tests, ensuring each # connection has its own storage. # Unfortunately, it wants to acquire the identically same # non-recursive lock in each of its *its* tpc_* methods, which deadlocks. # The solution is to give each instance its own lock, and trust in the # serialization (ordering) of the datamanager, and the fact that these tests are # single-threaded. import threading from ZODB.tests.MVCCMappingStorage import MVCCMappingStorage class _MVCCMappingStorage(MVCCMappingStorage): def new_instance(self): inst = MVCCMappingStorage.new_instance(self) inst._commit_lock = threading.Lock() return inst from ZODB.DB import DB self.storage = _MVCCMappingStorage() self.db = DB(self.storage) return self.db @_skip_wo_ZODB def testSimpleConflict(self): # Invoke conflict resolution by committing a transaction and # catching a conflict in the storage. import transaction self.openDB() r1 = self.db.open().root() r1["t"] = t = self._makeOne() transaction.commit() r2 = self.db.open().root() copy = r2["t"] list(copy) # unghostify self.assertEqual(t._p_serial, copy._p_serial) t.update({1:2, 2:3}) transaction.commit() copy.update({3:4}) transaction.commit() # This tests a problem that cropped up while trying to write # testBucketSplitConflict (below): conflict resolution wasn't # working at all in non-trivial cases. Symptoms varied from # strange complaints about pickling (despite that the test isn't # doing any *directly*), thru SystemErrors from Python and # AssertionErrors inside the BTree code. @_skip_wo_ZODB def testResolutionBlowsUp(self): import transaction b = self._makeOne() for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Invoke conflict resolution by committing a transaction. self.openDB() r1 = self.db.open().root() r1["t"] = b transaction.commit() r2 = self.db.open().root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(b._p_serial, copy._p_serial) b.update({1:2, 2:3}) transaction.commit() copy.update({3:4}) transaction.commit() # if this doesn't blow up list(copy.values()) # and this doesn't either, then fine @_skip_wo_ZODB def testBucketSplitConflict(self): # Tests that a bucket split is viewed as a conflict. # It's (almost necessarily) a white-box test, and sensitive to # implementation details. import transaction from ZODB.POSException import ConflictError b = orig = self._makeOne() for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Invoke conflict resolution by committing a transaction. self.openDB() tm1 = transaction.TransactionManager() r1 = self.db.open(transaction_manager=tm1).root() r1["t"] = b tm1.commit() tm2 = transaction.TransactionManager() r2 = self.db.open(transaction_manager=tm2).root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(orig._p_serial, copy._p_serial) # In one transaction, add 16 new keys to bucket1, to force a bucket # split. b = orig numtoadd = 16 candidate = 60 while numtoadd: if candidate not in b: b[candidate] = candidate numtoadd -= 1 candidate += 1 # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 61 .. 74 # bucket 2 has 16 values: [75, 76 .. 81] + [84, 88 ..116] # bucket 3 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((b0, 60, b1, 75, b2, 120, b3), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state) , 2) self.assertEqual(len(state[0]), 7) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 75) self.assertEqual(state[0][5], 120) tm1.commit() # In the other transaction, add 3 values near the tail end of bucket1. # This doesn't cause a split. b = copy for i in range(112, 116): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 18 values: 60, 64 .. 112, 113, 114, 115, 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) self.assertRaises(ConflictError, tm2.commit) @_skip_wo_ZODB def testEmptyBucketConflict(self): # Tests that an emptied bucket *created by* conflict resolution is # viewed as a conflict: conflict resolution doesn't have enough # info to unlink the empty bucket from the BTree correctly. import transaction from ZODB.POSException import ConflictError b = orig = self._makeOne() for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Invoke conflict resolution by committing a transaction. self.openDB() tm1 = transaction.TransactionManager() r1 = self.db.open(transaction_manager=tm1).root() r1["t"] = b tm1.commit() tm2 = transaction.TransactionManager() r2 = self.db.open(transaction_manager=tm2).root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(orig._p_serial, copy._p_serial) # In one transaction, delete half of bucket 1. b = orig for k in 60, 64, 68, 72, 76, 80, 84, 88: del b[k] # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 7 values: 92, 96, 100, 104, 108, 112, 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state) , 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 92) self.assertEqual(state[0][3], 120) tm1.commit() # In the other transaction, delete the other half of bucket 1. b = copy for k in 92, 96, 100, 104, 108, 112, 116: del b[k] # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 8 values: 60, 64, 68, 72, 76, 80, 84, 88 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Conflict resolution empties bucket1 entirely. This used to # create an "insane" BTree (a legit BTree cannot contain an empty # bucket -- it contains NULL pointers the BTree code doesn't # expect, and segfaults result). self.assertRaises(ConflictError, tm2.commit) @_skip_wo_ZODB def testEmptyBucketNoConflict(self): # Tests that a plain empty bucket (on input) is not viewed as a # conflict. import transaction b = orig = self._makeOne() for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Invoke conflict resolution by committing a transaction. self.openDB() r1 = self.db.open().root() r1["t"] = orig transaction.commit() r2 = self.db.open().root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(orig._p_serial, copy._p_serial) # In one transaction, just add a key. b = orig b[1] = 1 # bucket 0 has 16 values: [0, 1] + [4, 8 .. 56] # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) transaction.commit() # In the other transaction, delete bucket 2. b = copy for k in range(120, 200, 4): del b[k] # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 3) self.assertEqual(state[0][1], 60) # This shouldn't create a ConflictError. transaction.commit() # And the resulting BTree shouldn't have internal damage. b._check() # The snaky control flow in _bucket__p_resolveConflict ended up trying # to decref a NULL pointer if conflict resolution was fed 3 empty # buckets. http://collector.zope.org/Zope/553 def testThreeEmptyBucketsNoSegfault(self): # Note that the conflict is raised by our C extension, rather than # indirectly via the storage, and hence is a more specialized type. # This test therefore does not require ZODB. from BTrees.Interfaces import BTreesConflictError t = self._makeOne() t[1] = 1 bucket = t._firstbucket del t[1] state1 = bucket.__getstate__() state2 = bucket.__getstate__() state3 = bucket.__getstate__() self.assertTrue(state2 is not state1 and state2 is not state3 and state3 is not state1) self.assertTrue(state2 == state1 and state3 == state1) self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, state1, state2, state3) # When an empty BTree resolves conflicts, it computes the # bucket state as None, so... self.assertRaises(BTreesConflictError, bucket._p_resolveConflict, None, None, None) @_skip_wo_ZODB def testCantResolveBTreeConflict(self): # Test that a conflict involving two different changes to # an internal BTree node is unresolvable. An internal node # only changes when there are enough additions or deletions # to a child bucket that the bucket is split or removed. # It's (almost necessarily) a white-box test, and sensitive to # implementation details. import transaction from ZODB.POSException import ConflictError b = orig = self._makeOne() for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Set up database connections to provoke conflict. self.openDB() tm1 = transaction.TransactionManager() r1 = self.db.open(transaction_manager=tm1).root() r1["t"] = orig tm1.commit() tm2 = transaction.TransactionManager() r2 = self.db.open(transaction_manager=tm2).root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(orig._p_serial, copy._p_serial) # Now one transaction should add enough keys to cause a split, # and another should remove all the keys in one bucket. for k in range(200, 300, 4): orig[k] = k tm1.commit() for k in range(0, 60, 4): del copy[k] self.assertRaises(ConflictError, tm2.commit) @_skip_wo_ZODB def testConflictWithOneEmptyBucket(self): # If one transaction empties a bucket, while another adds an item # to the bucket, all the changes "look resolvable": bucket conflict # resolution returns a bucket containing (only) the item added by # the latter transaction, but changes from the former transaction # removing the bucket are uncontested: the bucket is removed from # the BTree despite that resolution thinks it's non-empty! This # was first reported by Dieter Maurer, to zodb-dev on 22 Mar 2005. import transaction from ZODB.POSException import ConflictError b = orig = self._makeOne() for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Set up database connections to provoke conflict. self.openDB() tm1 = transaction.TransactionManager() r1 = self.db.open(transaction_manager=tm1).root() r1["t"] = orig tm1.commit() tm2 = transaction.TransactionManager() r2 = self.db.open(transaction_manager=tm2).root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(orig._p_serial, copy._p_serial) # Now one transaction empties the first bucket, and another adds a # key to the first bucket. for k in range(0, 60, 4): del orig[k] tm1.commit() copy[1] = 1 self.assertRaises(ConflictError, tm2.commit) # Same thing, except commit the transactions in the opposite order. b = self._makeOne() for i in range(0, 200, 4): b[i] = i tm1 = transaction.TransactionManager() r1 = self.db.open(transaction_manager=tm1).root() r1["t"] = b tm1.commit() tm2 = transaction.TransactionManager() r2 = self.db.open(transaction_manager=tm2).root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(b._p_serial, copy._p_serial) # Now one transaction empties the first bucket, and another adds a # key to the first bucket. b[1] = 1 tm1.commit() for k in range(0, 60, 4): del copy[k] self.assertRaises(ConflictError, tm2.commit) @_skip_wo_ZODB def testConflictOfInsertAndDeleteOfFirstBucketItem(self): # Recently, BTrees became careful about removing internal keys # (keys in internal aka BTree nodes) when they were deleted from # buckets. This poses a problem for conflict resolution. # We want to guard against a case in which the first key in a # bucket is removed in one transaction while a key is added # after that key but before the next key in another transaction # with the result that the added key is unreachable. # original: # Bucket(...), k1, Bucket((k1, v1), (k3, v3), ...) # tran1 # Bucket(...), k3, Bucket(k3, v3), ...) # tran2 # Bucket(...), k1, Bucket((k1, v1), (k2, v2), (k3, v3), ...) # where k1 < k2 < k3 # We don't want: # Bucket(...), k3, Bucket((k2, v2), (k3, v3), ...) # as k2 would be unfindable, so we want a conflict. import transaction from ZODB.POSException import ConflictError mytype = self._getTargetClass() db = self.openDB() tm1 = transaction.TransactionManager() conn1 = db.open(tm1) conn1.root.t = t = mytype() for i in range(0, 200, 2): t[i] = i tm1.commit() k = t.__getstate__()[0][1] assert t.__getstate__()[0][2].keys()[0] == k tm2 = transaction.TransactionManager() conn2 = db.open(tm2) t[k+1] = k+1 del conn2.root.t[k] for i in range(200,300): conn2.root.t[i] = i tm1.commit() self.assertRaises(ConflictError, tm2.commit) tm2.abort() k = t.__getstate__()[0][1] t[k+1] = k+1 del conn2.root.t[k] tm2.commit() self.assertRaises(ConflictError, tm1.commit) tm1.abort() def test_suite(): return unittest.TestSuite(( unittest.makeSuite(NastyConfictFunctionalTests), )) BTrees-4.3.1/BTrees/tests/test_IFBTree.py0000664000175000017500000002462112510775554021250 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from .common import BTreeTests from .common import ExtendedSetTests from .common import InternalKeysMappingTest from .common import InternalKeysSetTest from .common import MappingBase from .common import MappingConflictTestBase from .common import ModuleTest from .common import MultiUnion from .common import NormalSetTests from .common import SetConflictTestBase from .common import SetResult from .common import TestLongIntKeys from .common import makeBuilder from BTrees.IIBTree import using64bits #XXX Ugly, but unavoidable class IFBTreeInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFBTree return IFBTree class IFBTreePyInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFBTreePy return IFBTreePy class IFTreeSetInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFTreeSet return IFTreeSet class IFTreeSetPyInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFTreeSetPy return IFTreeSetPy class IFBucketTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFBucket return IFBucket class IFBucketPyTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFBucketPy return IFBucketPy class IFTreeSetTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFTreeSet return IFTreeSet class IFTreeSetPyTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFTreeSetPy return IFTreeSetPy class IFSetTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFSet return IFSet class IFSetPyTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFSetPy return IFSetPy class IFBTreeTest(BTreeTests, unittest.TestCase): def _makeOne(self): from BTrees.IFBTree import IFBTree return IFBTree() class IFBTreePyTest(BTreeTests, unittest.TestCase): def _makeOne(self): from BTrees.IFBTree import IFBTreePy return IFBTreePy() if using64bits: class IFBTreeTest(BTreeTests, TestLongIntKeys, unittest.TestCase): def _makeOne(self): from BTrees.IFBTree import IFBTree return IFBTree() def getTwoValues(self): return 0.5, 1.5 class IFBTreePyTest(BTreeTests, TestLongIntKeys, unittest.TestCase): def _makeOne(self): from BTrees.IFBTree import IFBTreePy return IFBTreePy() def getTwoValues(self): return 0.5, 1.5 class _TestIFBTreesBase(object): def testNonIntegerKeyRaises(self): self.assertRaises(TypeError, self._stringraiseskey) self.assertRaises(TypeError, self._floatraiseskey) self.assertRaises(TypeError, self._noneraiseskey) def testNonNumericValueRaises(self): self.assertRaises(TypeError, self._stringraisesvalue) self.assertRaises(TypeError, self._noneraisesvalue) self._makeOne()[1] = 1 self._makeOne()[1] = 1.0 def _stringraiseskey(self): self._makeOne()['c'] = 1 def _floatraiseskey(self): self._makeOne()[2.5] = 1 def _noneraiseskey(self): self._makeOne()[None] = 1 def _stringraisesvalue(self): self._makeOne()[1] = 'c' def _floatraisesvalue(self): self._makeOne()[1] = 1.4 def _noneraisesvalue(self): self._makeOne()[1] = None class TestIFBTrees(_TestIFBTreesBase, unittest.TestCase): def _makeOne(self): from BTrees.IFBTree import IFBTree return IFBTree() class TestIFBTreesPy(_TestIFBTreesBase, unittest.TestCase): def _makeOne(self): from BTrees.IFBTree import IFBTreePy return IFBTreePy() class TestIFMultiUnion(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.IFBTree import multiunion return multiunion(*args) def union(self, *args): from BTrees.IFBTree import union return union(*args) def mkset(self, *args): from BTrees.IFBTree import IFSet as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.IFBTree import IFTreeSet as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.IFBTree import IFBucket as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.IFBTree import IFBTree as mkbtree return mkbtree(*args) class TestIFMultiUnionPy(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.IFBTree import multiunionPy return multiunionPy(*args) def union(self, *args): from BTrees.IFBTree import unionPy return unionPy(*args) def mkset(self, *args): from BTrees.IFBTree import IFSetPy as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.IFBTree import IFTreeSetPy as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.IFBTree import IFBucketPy as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.IFBTree import IFBTreePy as mkbtree return mkbtree(*args) class PureIF(SetResult, unittest.TestCase): def union(self, *args): from BTrees.IFBTree import union return union(*args) def intersection(self, *args): from BTrees.IFBTree import intersection return intersection(*args) def difference(self, *args): from BTrees.IFBTree import difference return difference(*args) def builders(self): from BTrees.IFBTree import IFBTree from BTrees.IFBTree import IFBucket from BTrees.IFBTree import IFTreeSet from BTrees.IFBTree import IFSet return IFSet, IFTreeSet, makeBuilder(IFBTree), makeBuilder(IFBucket) class PureIFPy(SetResult, unittest.TestCase): def union(self, *args): from BTrees.IFBTree import unionPy return unionPy(*args) def intersection(self, *args): from BTrees.IFBTree import intersectionPy return intersectionPy(*args) def difference(self, *args): from BTrees.IFBTree import differencePy return differencePy(*args) def builders(self): from BTrees.IFBTree import IFBTreePy from BTrees.IFBTree import IFBucketPy from BTrees.IFBTree import IFTreeSetPy from BTrees.IFBTree import IFSetPy return (IFSetPy, IFTreeSetPy, makeBuilder(IFBTreePy), makeBuilder(IFBucketPy)) class IFBTreeConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFBTree return IFBTree class IFBTreePyConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFBTreePy return IFBTreePy class IFBucketConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFBucket return IFBucket class IFBucketPyConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFBucketPy return IFBucketPy class IFTreeSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFTreeSet return IFTreeSet class IFTreeSetPyConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFTreeSetPy return IFTreeSetPy class IFSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFSet return IFSet class IFSetPyConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.IFBTree import IFSetPy return IFSetPy class IFModuleTest(ModuleTest, unittest.TestCase): prefix = 'IF' def _getModule(self): import BTrees return BTrees.IFBTree def _getInterface(self): import BTrees.Interfaces return BTrees.Interfaces.IIntegerFloatBTreeModule def test_suite(): return unittest.TestSuite(( unittest.makeSuite(IFBTreeInternalKeyTest), unittest.makeSuite(IFBTreePyInternalKeyTest), unittest.makeSuite(IFTreeSetInternalKeyTest), unittest.makeSuite(IFTreeSetPyInternalKeyTest), unittest.makeSuite(IFBucketTest), unittest.makeSuite(IFBucketPyTest), unittest.makeSuite(IFTreeSetTest), unittest.makeSuite(IFTreeSetPyTest), unittest.makeSuite(IFSetTest), unittest.makeSuite(IFSetPyTest), unittest.makeSuite(IFBTreeTest), unittest.makeSuite(IFBTreePyTest), unittest.makeSuite(TestIFBTrees), unittest.makeSuite(TestIFBTreesPy), unittest.makeSuite(TestIFMultiUnion), unittest.makeSuite(TestIFMultiUnionPy), unittest.makeSuite(PureIF), unittest.makeSuite(PureIFPy), unittest.makeSuite(IFBTreeConflictTests), unittest.makeSuite(IFBTreePyConflictTests), unittest.makeSuite(IFBucketConflictTests), unittest.makeSuite(IFBucketPyConflictTests), unittest.makeSuite(IFTreeSetConflictTests), unittest.makeSuite(IFTreeSetPyConflictTests), unittest.makeSuite(IFSetConflictTests), unittest.makeSuite(IFSetPyConflictTests), unittest.makeSuite(IFModuleTest), )) BTrees-4.3.1/BTrees/tests/test_btreesubclass.py0000664000175000017500000000315612510775554022671 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001, 2002 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from BTrees.OOBTree import OOBTree, OOBucket class B(OOBucket): pass class T(OOBTree): _bucket_type = B max_leaf_size = 2 max_internal_size = 3 class S(T): pass import unittest class SubclassTest(unittest.TestCase): def testSubclass(self): # test that a subclass that defines _bucket_type gets buckets # of that type t = T() t[0] = 0 self.assertTrue(t._firstbucket.__class__ is B) def testCustomNodeSizes(self): # We override btree and bucket split sizes in BTree subclasses. t = S() for i in range(8): t[i] = i state = t.__getstate__()[0] self.assertEqual(len(state), 5) sub = state[0] self.assertEqual(sub.__class__, S) sub = sub.__getstate__()[0] self.assertEqual(len(sub), 5) sub = sub[0] self.assertEqual(sub.__class__, B) self.assertEqual(len(sub), 1) def test_suite(): return unittest.makeSuite(SubclassTest) BTrees-4.3.1/BTrees/tests/test_LFBTree.py0000664000175000017500000002137512510775554021256 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from .common import BTreeTests from .common import ExtendedSetTests from .common import InternalKeysMappingTest from .common import InternalKeysSetTest from .common import MappingBase from .common import MappingConflictTestBase from .common import ModuleTest from .common import MultiUnion from .common import NormalSetTests from .common import SetConflictTestBase from .common import SetResult from .common import TestLongIntKeys from .common import makeBuilder class LFBTreeInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFBTree return LFBTree class LFBTreePyInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFBTreePy return LFBTreePy class LFTreeSetInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFTreeSet return LFTreeSet class LFTreeSetPyInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFTreeSetPy return LFTreeSetPy class LFBucketTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFBucket return LFBucket class LFBucketPyTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFBucketPy return LFBucketPy class LFTreeSetTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFTreeSet return LFTreeSet class LFTreeSetPyTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFTreeSetPy return LFTreeSetPy class LFSetTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFSet return LFSet class LFSetPyTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFSetPy return LFSetPy class LFBTreeTest(BTreeTests, TestLongIntKeys, unittest.TestCase): def _makeOne(self): from BTrees.LFBTree import LFBTree return LFBTree() def getTwoValues(self): return 0.5, 1.5 class LFBTreePyTest(BTreeTests, TestLongIntKeys, unittest.TestCase): def _makeOne(self): from BTrees.LFBTree import LFBTreePy return LFBTreePy() def getTwoValues(self): return 0.5, 1.5 class TestLFMultiUnion(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.LFBTree import multiunionPy return multiunionPy(*args) def union(self, *args): from BTrees.LFBTree import unionPy return unionPy(*args) def mkset(self, *args): from BTrees.LFBTree import LFSetPy as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.LFBTree import LFTreeSetPy as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.LFBTree import LFBucketPy as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.LFBTree import LFBTreePy as mkbtree return mkbtree(*args) class TestLFMultiUnionPy(MultiUnion, unittest.TestCase): def multiunion(self, *args): from BTrees.LFBTree import multiunionPy return multiunionPy(*args) def union(self, *args): from BTrees.LFBTree import unionPy return unionPy(*args) def mkset(self, *args): from BTrees.LFBTree import LFSetPy as mkset return mkset(*args) def mktreeset(self, *args): from BTrees.LFBTree import LFTreeSetPy as mktreeset return mktreeset(*args) def mkbucket(self, *args): from BTrees.LFBTree import LFBucketPy as mkbucket return mkbucket(*args) def mkbtree(self, *args): from BTrees.LFBTree import LFBTreePy as mkbtree return mkbtree(*args) class PureLF(SetResult, unittest.TestCase): def union(self, *args): from BTrees.LFBTree import union return union(*args) def intersection(self, *args): from BTrees.LFBTree import intersection return intersection(*args) def difference(self, *args): from BTrees.LFBTree import difference return difference(*args) def builders(self): from BTrees.LFBTree import LFBTree from BTrees.LFBTree import LFBucket from BTrees.LFBTree import LFTreeSet from BTrees.LFBTree import LFSet return LFSet, LFTreeSet, makeBuilder(LFBTree), makeBuilder(LFBucket) class PureLFPy(SetResult, unittest.TestCase): def union(self, *args): from BTrees.LFBTree import unionPy return unionPy(*args) def intersection(self, *args): from BTrees.LFBTree import intersectionPy return intersectionPy(*args) def difference(self, *args): from BTrees.LFBTree import differencePy return differencePy(*args) def builders(self): from BTrees.LFBTree import LFBTreePy from BTrees.LFBTree import LFBucketPy from BTrees.LFBTree import LFTreeSetPy from BTrees.LFBTree import LFSetPy return (LFSetPy, LFTreeSetPy, makeBuilder(LFBTreePy), makeBuilder(LFBucketPy)) class LFBTreeConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFBTree return LFBTree class LFBTreeConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFBTreePy return LFBTreePy class LFBucketConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFBucket return LFBucket class LFBucketConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFBucketPy return LFBucketPy class LFTreeSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFTreeSet return LFTreeSet class LFTreeSetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFTreeSetPy return LFTreeSetPy class LFSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFSet return LFSet class LFSetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.LFBTree import LFSetPy return LFSetPy class LFModuleTest(ModuleTest, unittest.TestCase): prefix = 'LF' def _getModule(self): import BTrees return BTrees.LFBTree def _getInterface(self): import BTrees.Interfaces return BTrees.Interfaces.IIntegerFloatBTreeModule def test_suite(): return unittest.TestSuite(( unittest.makeSuite(LFBTreeInternalKeyTest), unittest.makeSuite(LFBTreePyInternalKeyTest), unittest.makeSuite(LFTreeSetInternalKeyTest), unittest.makeSuite(LFTreeSetPyInternalKeyTest), unittest.makeSuite(LFBucketTest), unittest.makeSuite(LFBucketPyTest), unittest.makeSuite(LFTreeSetTest), unittest.makeSuite(LFTreeSetPyTest), unittest.makeSuite(LFSetTest), unittest.makeSuite(LFSetPyTest), unittest.makeSuite(LFBTreeTest), unittest.makeSuite(LFBTreePyTest), unittest.makeSuite(TestLFMultiUnion), unittest.makeSuite(TestLFMultiUnionPy), unittest.makeSuite(PureLF), unittest.makeSuite(PureLFPy), unittest.makeSuite(LFBTreeConflictTests), unittest.makeSuite(LFBTreeConflictTestsPy), unittest.makeSuite(LFBucketConflictTests), unittest.makeSuite(LFBucketConflictTestsPy), unittest.makeSuite(LFTreeSetConflictTests), unittest.makeSuite(LFTreeSetConflictTestsPy), unittest.makeSuite(LFSetConflictTests), unittest.makeSuite(LFSetConflictTestsPy), unittest.makeSuite(LFModuleTest), )) BTrees-4.3.1/BTrees/tests/testBTrees.py0000664000175000017500000004215612533366331021052 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001, 2002 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import sys import unittest python3 = sys.version_info >= (3, ) from BTrees.tests.common import permutations class DegenerateBTree(unittest.TestCase): # Build a degenerate tree (set). Boxes are BTree nodes. There are # 5 leaf buckets, each containing a single int. Keys in the BTree # nodes don't appear in the buckets. Seven BTree nodes are purely # indirection nodes (no keys). Buckets aren't all at the same depth: # # +------------------------+ # | 4 | # +------------------------+ # | | # | v # | +-+ # | | | # | +-+ # | | # v v # +-------+ +-------------+ # | 2 | | 6 10 | # +-------+ +-------------+ # | | | | | # v v v v v # +-+ +-+ +-+ +-+ +-+ # | | | | | | | | | | # +-+ +-+ +-+ +-+ +-+ # | | | | | # v v v v v # 1 3 +-+ 7 11 # | | # +-+ # | # v # 5 # # This is nasty for many algorithms. Consider a high-end range search # for 4. The BTree nodes direct it to the 5 bucket, but the correct # answer is the 3 bucket, which requires going in a different direction # at the very top node already. Consider a low-end range search for # 9. The BTree nodes direct it to the 7 bucket, but the correct answer # is the 11 bucket. This is also a nasty-case tree for deletions. def _build_degenerate_tree(self): # Build the buckets and chain them together. from BTrees.IIBTree import IISet from BTrees.IIBTree import IITreeSet from BTrees.check import check bucket11 = IISet([11]) bucket7 = IISet() bucket7.__setstate__(((7,), bucket11)) bucket5 = IISet() bucket5.__setstate__(((5,), bucket7)) bucket3 = IISet() bucket3.__setstate__(((3,), bucket5)) bucket1 = IISet() bucket1.__setstate__(((1,), bucket3)) # Build the deepest layers of indirection nodes. ts = IITreeSet tree1 = ts() tree1.__setstate__(((bucket1,), bucket1)) tree3 = ts() tree3.__setstate__(((bucket3,), bucket3)) tree5lower = ts() tree5lower.__setstate__(((bucket5,), bucket5)) tree5 = ts() tree5.__setstate__(((tree5lower,), bucket5)) tree7 = ts() tree7.__setstate__(((bucket7,), bucket7)) tree11 = ts() tree11.__setstate__(((bucket11,), bucket11)) # Paste together the middle layers. tree13 = ts() tree13.__setstate__(((tree1, 2, tree3), bucket1)) tree5711lower = ts() tree5711lower.__setstate__(((tree5, 6, tree7, 10, tree11), bucket5)) tree5711 = ts() tree5711.__setstate__(((tree5711lower,), bucket5)) # One more. t = ts() t.__setstate__(((tree13, 4, tree5711), bucket1)) t._check() check(t) return t, [1, 3, 5, 7, 11] def testBasicOps(self): t, keys = self._build_degenerate_tree() self.assertEqual(len(t), len(keys)) self.assertEqual(list(t.keys()), keys) # has_key actually returns the depth of a bucket. self.assertEqual(t.has_key(1), 4) self.assertEqual(t.has_key(3), 4) self.assertEqual(t.has_key(5), 6) self.assertEqual(t.has_key(7), 5) self.assertEqual(t.has_key(11), 5) for i in 0, 2, 4, 6, 8, 9, 10, 12: self.assertTrue(i not in t) def _checkRanges(self, tree, keys): self.assertEqual(len(tree), len(keys)) sorted_keys = keys[:] sorted_keys.sort() self.assertEqual(list(tree.keys()), sorted_keys) for k in keys: self.assertTrue(k in tree) if keys: lokey = sorted_keys[0] hikey = sorted_keys[-1] self.assertEqual(lokey, tree.minKey()) self.assertEqual(hikey, tree.maxKey()) else: lokey = hikey = 42 # Try all range searches. for lo in range(lokey - 1, hikey + 2): for hi in range(lo - 1, hikey + 2): for skipmin in False, True: for skipmax in False, True: wantlo, wanthi = lo, hi if skipmin: wantlo += 1 if skipmax: wanthi -= 1 want = [k for k in keys if wantlo <= k <= wanthi] got = list(tree.keys(lo, hi, skipmin, skipmax)) self.assertEqual(want, got) def testRanges(self): t, keys = self._build_degenerate_tree() self._checkRanges(t, keys) def testDeletes(self): # Delete keys in all possible orders, checking each tree along # the way. # This is a tough test. Previous failure modes included: # 1. A variety of assertion failures in _checkRanges. # 2. Assorted "Invalid firstbucket pointer" failures at # seemingly random times, coming out of the BTree destructor. # 3. Under Python 2.3 CVS, some baffling # RuntimeWarning: tp_compare didn't return -1 or -2 for exception # warnings, possibly due to memory corruption after a BTree # goes insane. from BTrees.check import check t, keys = self._build_degenerate_tree() for oneperm in permutations(keys): t, keys = self._build_degenerate_tree() for key in oneperm: t.remove(key) keys.remove(key) t._check() check(t) self._checkRanges(t, keys) # We removed all the keys, so the tree should be empty now. self.assertEqual(t.__getstate__(), None) # A damaged tree may trigger an "invalid firstbucket pointer" # failure at the time its destructor is invoked. Try to force # that to happen now, so it doesn't look like a baffling failure # at some unrelated line. del t # trigger destructor LP294788_ids = {} class ToBeDeleted(object): def __init__(self, id): assert type(id) is int #we don't want to store any object ref here self.id = id global LP294788_ids LP294788_ids[id] = 1 def __del__(self): global LP294788_ids LP294788_ids.pop(self.id, None) def __cmp__(self, other): return cmp(self.id, other.id) def __le__(self, other): return self.id <= other.id def __lt__(self, other): return self.id < other.id def __eq__(self, other): return self.id == other.id def __ne__(self, other): return self.id != other.id def __gt__(self, other): return self.id > other.id def __ge__(self, other): return self.id >= other.id def __hash__(self): return hash(self.id) class BugFixes(unittest.TestCase): # Collector 1843. Error returns were effectively ignored in # Bucket_rangeSearch(), leading to "delayed" errors, or worse. def testFixed1843(self): from BTrees.IIBTree import IISet t = IISet() t.insert(1) # This one used to fail to raise the TypeError when it occurred. self.assertRaises(TypeError, t.keys, "") # This one used to segfault. self.assertRaises(TypeError, t.keys, 0, "") def test_LP294788(self): # https://bugs.launchpad.net/bugs/294788 # BTree keeps some deleted objects referenced # The logic here together with the ToBeDeleted class is that # a separate reference dict is populated on object creation # and removed in __del__ # That means what's left in the reference dict is never GC'ed # therefore referenced somewhere # To simulate real life, some random data is used to exercise the tree import gc import random from BTrees.OOBTree import OOBTree from .._compat import _u from .._compat import xrange t = OOBTree() trandom = random.Random('OOBTree') global LP294788_ids # /// BTree keys are integers, value is an object LP294788_ids = {} ids = {} for i in xrange(1024): if trandom.random() > 0.1 or len(ids) == 0: #add id = None while id is None or id in ids: id = trandom.randint(0,1000000) ids[id] = 1 t[id] = ToBeDeleted(id) else: #del keys = list(ids.keys()) if keys: id = trandom.choice(list(ids.keys())) del t[id] del ids[id] ids = ids.keys() trandom.shuffle(list(ids)) for id in ids: del t[id] ids = None #to be on the safe side run a full GC gc.collect() #print LP294788_ids self.assertEqual(len(t), 0) self.assertEqual(len(LP294788_ids), 0) # \\\ # /// BTree keys are integers, value is a tuple having an object LP294788_ids = {} ids = {} for i in xrange(1024): if trandom.random() > 0.1 or len(ids) == 0: #add id = None while id is None or id in ids: id = trandom.randint(0,1000000) ids[id] = 1 t[id] = (id, ToBeDeleted(id), _u('somename')) else: #del keys = list(ids.keys()) if keys: id = trandom.choice(keys) del t[id] del ids[id] ids = ids.keys() trandom.shuffle(list(ids)) for id in ids: del t[id] ids = None #to be on the safe side run a full GC gc.collect() #print LP294788_ids self.assertEqual(len(t), 0) self.assertEqual(len(LP294788_ids), 0) # \\\ # /// BTree keys are objects, value is an int t = OOBTree() LP294788_ids = {} ids = {} for i in xrange(1024): if trandom.random() > 0.1 or len(ids) == 0: #add id = None while id is None or id in ids: id = ToBeDeleted(trandom.randint(0,1000000)) ids[id] = 1 t[id] = 1 else: #del id = trandom.choice(list(ids.keys())) del ids[id] del t[id] ids = ids.keys() trandom.shuffle(list(ids)) for id in ids: del t[id] #release all refs ids = obj = id = None #to be on the safe side run a full GC gc.collect() #print LP294788_ids self.assertEqual(len(t), 0) self.assertEqual(len(LP294788_ids), 0) # /// BTree keys are tuples having objects, value is an int t = OOBTree() LP294788_ids = {} ids = {} for i in xrange(1024): if trandom.random() > 0.1 or len(ids) == 0: #add id = None while id is None or id in ids: id = trandom.randint(0,1000000) id = (id, ToBeDeleted(id), _u('somename')) ids[id] = 1 t[id] = 1 else: #del id = trandom.choice(list(ids.keys())) del ids[id] del t[id] ids = ids.keys() trandom.shuffle(list(ids)) for id in ids: del t[id] #release all refs ids = id = obj = key = None #to be on the safe side run a full GC gc.collect() #print LP294788_ids self.assertEqual(len(t), 0) self.assertEqual(len(LP294788_ids), 0) # cmp error propagation tests class DoesntLikeBeingCompared: def __cmp__(self,other): raise ValueError('incomparable') __lt__ = __le__ = __eq__ = __ne__ = __ge__ = __gt__ = __cmp__ class TestCmpError(unittest.TestCase): def testFoo(self): from BTrees.OOBTree import OOBTree t = OOBTree() t['hello world'] = None try: t[DoesntLikeBeingCompared()] = None except ValueError as e: self.assertEqual(str(e), 'incomparable') else: self.fail('incomarable objects should not be allowed into ' 'the tree') class FamilyTest(unittest.TestCase): def test32(self): from zope.interface.verify import verifyObject import BTrees from BTrees.IOBTree import IOTreeSet verifyObject(BTrees.Interfaces.IBTreeFamily, BTrees.family32) self.assertEqual( BTrees.family32.IO, BTrees.IOBTree) self.assertEqual( BTrees.family32.OI, BTrees.OIBTree) self.assertEqual( BTrees.family32.II, BTrees.IIBTree) self.assertEqual( BTrees.family32.IF, BTrees.IFBTree) self.assertEqual( BTrees.family32.OO, BTrees.OOBTree) s = IOTreeSet() s.insert(BTrees.family32.maxint) self.assertTrue(BTrees.family32.maxint in s) s = IOTreeSet() s.insert(BTrees.family32.minint) self.assertTrue(BTrees.family32.minint in s) s = IOTreeSet() # this next bit illustrates an, um, "interesting feature". If # the characteristics change to match the 64 bit version, please # feel free to change. try: s.insert(BTrees.family32.maxint + 1) except (TypeError, OverflowError): pass else: self.assert_(False) try: s.insert(BTrees.family32.minint - 1) except (TypeError, OverflowError): pass else: self.assert_(False) self.check_pickling(BTrees.family32) def test64(self): from zope.interface.verify import verifyObject import BTrees from BTrees.LOBTree import LOTreeSet verifyObject(BTrees.Interfaces.IBTreeFamily, BTrees.family64) self.assertEqual( BTrees.family64.IO, BTrees.LOBTree) self.assertEqual( BTrees.family64.OI, BTrees.OLBTree) self.assertEqual( BTrees.family64.II, BTrees.LLBTree) self.assertEqual( BTrees.family64.IF, BTrees.LFBTree) self.assertEqual( BTrees.family64.OO, BTrees.OOBTree) s = LOTreeSet() s.insert(BTrees.family64.maxint) self.assertTrue(BTrees.family64.maxint in s) s = LOTreeSet() s.insert(BTrees.family64.minint) self.assertTrue(BTrees.family64.minint in s) s = LOTreeSet() # XXX why oh why do we expect ValueError here, but TypeError in test32? self.assertRaises(ValueError, s.insert, BTrees.family64.maxint + 1) self.assertRaises(ValueError, s.insert, BTrees.family64.minint - 1) self.check_pickling(BTrees.family64) def check_pickling(self, family): # The "family" objects are singletons; they can be pickled and # unpickled, and the same instances will always be returned on # unpickling, whether from the same unpickler or different # unpicklers. import pickle from .._compat import BytesIO s = pickle.dumps((family, family)) (f1, f2) = pickle.loads(s) self.assertTrue(f1 is family) self.assertTrue(f2 is family) # Using a single memo across multiple pickles: sio = BytesIO() p = pickle.Pickler(sio) p.dump(family) p.dump([family]) u = pickle.Unpickler(BytesIO(sio.getvalue())) f1 = u.load() f2, = u.load() self.assertTrue(f1 is family) self.assertTrue(f2 is family) # Using separate memos for each pickle: sio = BytesIO() p = pickle.Pickler(sio) p.dump(family) p.clear_memo() p.dump([family]) u = pickle.Unpickler(BytesIO(sio.getvalue())) f1 = u.load() f2, = u.load() self.assertTrue(f1 is family) self.assertTrue(f2 is family) def test_suite(): return unittest.TestSuite(( unittest.makeSuite(DegenerateBTree), unittest.makeSuite(BugFixes), unittest.makeSuite(TestCmpError), unittest.makeSuite(FamilyTest), )) BTrees-4.3.1/BTrees/tests/test_OIBTree.py0000664000175000017500000002530312510775554021257 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from .common import BTreeTests from .common import ExtendedSetTests from .common import InternalKeysMappingTest from .common import InternalKeysSetTest from .common import MappingBase from .common import MappingConflictTestBase from .common import ModuleTest from .common import NormalSetTests from .common import SetConflictTestBase from .common import SetResult from .common import TestLongIntValues from .common import TypeTest from .common import Weighted from .common import itemsToSet from .common import makeBuilder from BTrees.IIBTree import using64bits #XXX Ugly, but necessary class OIBTreeInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OIBTree return OIBTree class OIBTreePyInternalKeyTest(InternalKeysMappingTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OIBTreePy return OIBTreePy class OITreeSetInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OITreeSet return OITreeSet class OITreeSetPyInternalKeyTest(InternalKeysSetTest, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OITreeSetPy return OITreeSetPy class OIBucketTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OIBucket return OIBucket class OIBucketPyTest(MappingBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OIBucketPy return OIBucketPy class OITreeSetTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OITreeSet return OITreeSet class OITreeSetPyTest(NormalSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OITreeSetPy return OITreeSetPy class OISetTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OISet return OISet class OISetPyTest(ExtendedSetTests, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OISetPy return OISetPy class OIBTreeTest(BTreeTests, unittest.TestCase): def _makeOne(self): from BTrees.OIBTree import OIBTree return OIBTree() class OIBTreePyTest(BTreeTests, unittest.TestCase): def _makeOne(self): from BTrees.OIBTree import OIBTreePy return OIBTreePy() if using64bits: class OIBTreeTest(BTreeTests, TestLongIntValues, unittest.TestCase): def _makeOne(self): from BTrees.OIBTree import OIBTree return OIBTree() def getTwoKeys(self): return object(), object() class OIBTreePyTest(BTreeTests, TestLongIntValues, unittest.TestCase): def _makeOne(self): from BTrees.OIBTree import OIBTreePy return OIBTreePy() def getTwoKeys(self): return object(), object() class _TestOIBTreesBase(TypeTest): def _stringraises(self): self._makeOne()[1] = 'c' def _floatraises(self): self._makeOne()[1] = 1.4 def _noneraises(self): self._makeOne()[1] = None def testEmptyFirstBucketReportedByGuido(self): from .._compat import xrange b = self._makeOne() for i in xrange(29972): # reduce to 29971 and it works b[i] = i for i in xrange(30): # reduce to 29 and it works del b[i] b[i+40000] = i self.assertEqual(b.keys()[0], 30) class TestOIBTrees(_TestOIBTreesBase, unittest.TestCase): def _makeOne(self): from BTrees.OIBTree import OIBTree return OIBTree() class TestOIBTreesPy(_TestOIBTreesBase, unittest.TestCase): def _makeOne(self): from BTrees.OIBTree import OIBTreePy return OIBTreePy() class PureOI(SetResult, unittest.TestCase): def union(self, *args): from BTrees.OIBTree import union return union(*args) def intersection(self, *args): from BTrees.OIBTree import intersection return intersection(*args) def difference(self, *args): from BTrees.OIBTree import difference return difference(*args) def builders(self): from BTrees.OIBTree import OIBTree from BTrees.OIBTree import OIBucket from BTrees.OIBTree import OITreeSet from BTrees.OIBTree import OISet return OISet, OITreeSet, makeBuilder(OIBTree), makeBuilder(OIBucket) class PureOIPy(SetResult, unittest.TestCase): def union(self, *args): from BTrees.OIBTree import unionPy return unionPy(*args) def intersection(self, *args): from BTrees.OIBTree import intersectionPy return intersectionPy(*args) def difference(self, *args): from BTrees.OIBTree import differencePy return differencePy(*args) def builders(self): from BTrees.OIBTree import OIBTreePy from BTrees.OIBTree import OIBucketPy from BTrees.OIBTree import OITreeSetPy from BTrees.OIBTree import OISetPy return (OISetPy, OITreeSetPy, makeBuilder(OIBTreePy), makeBuilder(OIBucketPy)) class TestWeightedOI(Weighted, unittest.TestCase): def weightedUnion(self): from BTrees.OIBTree import weightedUnion return weightedUnion def weightedIntersection(self): from BTrees.OIBTree import weightedIntersection return weightedIntersection def union(self): from BTrees.OIBTree import union return union def intersection(self): from BTrees.OIBTree import intersection return intersection def mkbucket(self, *args): from BTrees.OIBTree import OIBucket as mkbucket return mkbucket(*args) def builders(self): from BTrees.OIBTree import OIBTree from BTrees.OIBTree import OIBucket from BTrees.OIBTree import OITreeSet from BTrees.OIBTree import OISet return OIBucket, OIBTree, itemsToSet(OISet), itemsToSet(OITreeSet) class TestWeightedOIPy(Weighted, unittest.TestCase): def weightedUnion(self): from BTrees.OIBTree import weightedUnionPy return weightedUnionPy def weightedIntersection(self): from BTrees.OIBTree import weightedIntersectionPy return weightedIntersectionPy def union(self): from BTrees.OIBTree import unionPy return unionPy def intersection(self): from BTrees.OIBTree import intersectionPy return intersectionPy def mkbucket(self, *args): from BTrees.OIBTree import OIBucketPy as mkbucket return mkbucket(*args) def builders(self): from BTrees.OIBTree import OIBTreePy from BTrees.OIBTree import OIBucketPy from BTrees.OIBTree import OITreeSetPy from BTrees.OIBTree import OISetPy return (OIBucketPy, OIBTreePy, itemsToSet(OISetPy), itemsToSet(OITreeSetPy)) class OIBucketConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OIBucket return OIBucket class OIBucketConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OIBucketPy return OIBucketPy class OISetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OISet return OISet class OISetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OISetPy return OISetPy class OIBTreeConflictTests(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OIBTree return OIBTree class OIBTreeConflictTestsPy(MappingConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OIBTreePy return OIBTreePy class OITreeSetConflictTests(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OITreeSet return OITreeSet class OITreeSetConflictTestsPy(SetConflictTestBase, unittest.TestCase): def _getTargetClass(self): from BTrees.OIBTree import OITreeSetPy return OITreeSetPy class OIModuleTest(ModuleTest, unittest.TestCase): prefix = 'OI' def _getModule(self): import BTrees return BTrees.OIBTree def _getInterface(self): import BTrees.Interfaces return BTrees.Interfaces.IObjectIntegerBTreeModule def test_multiunion_not_present(self): try: from BTrees.OIBTree import multiunion except ImportError: pass else: self.fail("OIBTree shouldn't have multiunion") def test_suite(): return unittest.TestSuite(( unittest.makeSuite(OIBTreeInternalKeyTest), unittest.makeSuite(OIBTreePyInternalKeyTest), unittest.makeSuite(OITreeSetInternalKeyTest), unittest.makeSuite(OITreeSetPyInternalKeyTest), unittest.makeSuite(OIBucketTest), unittest.makeSuite(OIBucketPyTest), unittest.makeSuite(OITreeSetTest), unittest.makeSuite(OITreeSetPyTest), unittest.makeSuite(OISetTest), unittest.makeSuite(OISetPyTest), unittest.makeSuite(OIBTreeTest), unittest.makeSuite(OIBTreePyTest), unittest.makeSuite(TestOIBTrees), unittest.makeSuite(TestOIBTreesPy), unittest.makeSuite(PureOI), unittest.makeSuite(PureOIPy), unittest.makeSuite(TestWeightedOI), unittest.makeSuite(TestWeightedOIPy), unittest.makeSuite(OIBucketConflictTests), unittest.makeSuite(OIBucketConflictTestsPy), unittest.makeSuite(OISetConflictTests), unittest.makeSuite(OISetConflictTestsPy), unittest.makeSuite(OIBTreeConflictTests), unittest.makeSuite(OIBTreeConflictTestsPy), unittest.makeSuite(OITreeSetConflictTests), unittest.makeSuite(OITreeSetConflictTestsPy), unittest.makeSuite(OIModuleTest), )) BTrees-4.3.1/BTrees/BTreeItemsTemplate.c0000664000175000017500000006037112510775554021122 0ustar tseavertseaver00000000000000/***************************************************************************** Copyright (c) 2001, 2002 Zope Foundation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #define BTREEITEMSTEMPLATE_C "$Id$\n" /* A BTreeItems struct is returned from calling .items(), .keys() or * .values() on a BTree-based data structure, and is also the result of * taking slices of those. It represents a contiguous slice of a BTree. * * The start of the slice is in firstbucket, at offset first. The end of * the slice is in lastbucket, at offset last. Both endpoints are inclusive. * It must possible to get from firstbucket to lastbucket via following * bucket 'next' pointers zero or more times. firstbucket, first, lastbucket, * and last are readonly after initialization. An empty slice is represented * by firstbucket == lastbucket == currentbucket == NULL. * * 'kind' determines whether this slice represents 'k'eys alone, 'v'alues * alone, or 'i'items (key+value pairs). 'kind' is also readonly after * initialization. * * The combination of currentbucket, currentoffset and pseudoindex acts as * a search finger. Offset currentoffset in bucket currentbucket is at index * pseudoindex, where pseudoindex==0 corresponds to offset first in bucket * firstbucket, and pseudoindex==-1 corresponds to offset last in bucket * lastbucket. The function BTreeItems_seek() can be used to set this combo * correctly for any in-bounds index, and uses this combo on input to avoid * needing to search from the start (or end) on each call. Calling * BTreeItems_seek() with consecutive larger positions is very efficent. * Calling it with consecutive smaller positions is more efficient than if * a search finger weren't being used at all, but is still quadratic time * in the number of buckets in the slice. */ typedef struct { PyObject_HEAD Bucket *firstbucket; /* First bucket */ Bucket *currentbucket; /* Current bucket (search finger) */ Bucket *lastbucket; /* Last bucket */ int currentoffset; /* Offset in currentbucket */ int pseudoindex; /* search finger index */ int first; /* Start offset in firstbucket */ int last; /* End offset in lastbucket */ char kind; /* 'k', 'v', 'i' */ } BTreeItems; #define ITEMS(O)((BTreeItems*)(O)) static PyObject * newBTreeItems(char kind, Bucket *lowbucket, int lowoffset, Bucket *highbucket, int highoffset); static void BTreeItems_dealloc(BTreeItems *self) { Py_XDECREF(self->firstbucket); Py_XDECREF(self->lastbucket); Py_XDECREF(self->currentbucket); PyObject_DEL(self); } static Py_ssize_t BTreeItems_length_or_nonzero(BTreeItems *self, int nonzero) { Py_ssize_t r; Bucket *b, *next; b = self->firstbucket; if (b == NULL) return 0; r = self->last + 1 - self->first; if (nonzero && r > 0) /* Short-circuit if all we care about is nonempty */ return 1; if (b == self->lastbucket) return r; Py_INCREF(b); PER_USE_OR_RETURN(b, -1); while ((next = b->next)) { r += b->len; if (nonzero && r > 0) /* Short-circuit if all we care about is nonempty */ break; if (next == self->lastbucket) break; /* we already counted the last bucket */ Py_INCREF(next); PER_UNUSE(b); Py_DECREF(b); b = next; PER_USE_OR_RETURN(b, -1); } PER_UNUSE(b); Py_DECREF(b); return r >= 0 ? r : 0; } static Py_ssize_t BTreeItems_length(BTreeItems *self) { return BTreeItems_length_or_nonzero(self, 0); } /* ** BTreeItems_seek ** ** Find the ith position in the BTreeItems. ** ** Arguments: self The BTree ** i the index to seek to, in 0 .. len(self)-1, or in ** -len(self) .. -1, as for indexing a Python sequence. ** ** ** Returns 0 if successful, -1 on failure to seek (like out-of-bounds). ** Upon successful return, index i is at offset self->currentoffset in bucket ** self->currentbucket. */ static int BTreeItems_seek(BTreeItems *self, Py_ssize_t i) { int delta, pseudoindex, currentoffset; Bucket *b, *currentbucket; int error; pseudoindex = self->pseudoindex; currentoffset = self->currentoffset; currentbucket = self->currentbucket; if (currentbucket == NULL) goto no_match; delta = i - pseudoindex; while (delta > 0) /* move right */ { int max; /* Want to move right delta positions; the most we can move right in * this bucket is currentbucket->len - currentoffset - 1 positions. */ PER_USE_OR_RETURN(currentbucket, -1); max = currentbucket->len - currentoffset - 1; b = currentbucket->next; PER_UNUSE(currentbucket); if (delta <= max) { currentoffset += delta; pseudoindex += delta; if (currentbucket == self->lastbucket && currentoffset > self->last) goto no_match; break; } /* Move to start of next bucket. */ if (currentbucket == self->lastbucket || b == NULL) goto no_match; currentbucket = b; pseudoindex += max + 1; delta -= max + 1; currentoffset = 0; } while (delta < 0) /* move left */ { int status; /* Want to move left -delta positions; the most we can move left in * this bucket is currentoffset positions. */ if ((-delta) <= currentoffset) { currentoffset += delta; pseudoindex += delta; if (currentbucket == self->firstbucket && currentoffset < self->first) goto no_match; break; } /* Move to end of previous bucket. */ if (currentbucket == self->firstbucket) goto no_match; status = PreviousBucket(¤tbucket, self->firstbucket); if (status == 0) goto no_match; else if (status < 0) return -1; pseudoindex -= currentoffset + 1; delta += currentoffset + 1; PER_USE_OR_RETURN(currentbucket, -1); currentoffset = currentbucket->len - 1; PER_UNUSE(currentbucket); } assert(pseudoindex == i); /* Alas, the user may have mutated the bucket since the last time we * were called, and if they deleted stuff, we may be pointing into * trash memory now. */ PER_USE_OR_RETURN(currentbucket, -1); error = currentoffset < 0 || currentoffset >= currentbucket->len; PER_UNUSE(currentbucket); if (error) { PyErr_SetString(PyExc_RuntimeError, "the bucket being iterated changed size"); return -1; } Py_INCREF(currentbucket); Py_DECREF(self->currentbucket); self->currentbucket = currentbucket; self->currentoffset = currentoffset; self->pseudoindex = pseudoindex; return 0; no_match: IndexError(i); return -1; } /* Return the right kind ('k','v','i') of entry from bucket b at offset i. * b must be activated. Returns NULL on error. */ static PyObject * getBucketEntry(Bucket *b, int i, char kind) { PyObject *result = NULL; assert(b); assert(0 <= i && i < b->len); switch (kind) { case 'k': COPY_KEY_TO_OBJECT(result, b->keys[i]); break; case 'v': COPY_VALUE_TO_OBJECT(result, b->values[i]); break; case 'i': { PyObject *key; PyObject *value;; COPY_KEY_TO_OBJECT(key, b->keys[i]); if (!key) break; COPY_VALUE_TO_OBJECT(value, b->values[i]); if (!value) { Py_DECREF(key); break; } result = PyTuple_New(2); if (result) { PyTuple_SET_ITEM(result, 0, key); PyTuple_SET_ITEM(result, 1, value); } else { Py_DECREF(key); Py_DECREF(value); } break; } default: PyErr_SetString(PyExc_AssertionError, "getBucketEntry: unknown kind"); break; } return result; } /* ** BTreeItems_item ** ** Arguments: self a BTreeItems structure ** i Which item to inspect ** ** Returns: the BTreeItems_item_BTree of self->kind, i ** (ie pulls the ith item out) */ static PyObject * BTreeItems_item(BTreeItems *self, Py_ssize_t i) { PyObject *result; if (BTreeItems_seek(self, i) < 0) return NULL; PER_USE_OR_RETURN(self->currentbucket, NULL); result = getBucketEntry(self->currentbucket, self->currentoffset, self->kind); PER_UNUSE(self->currentbucket); return result; } /* ** BTreeItems_slice ** ** Creates a new BTreeItems structure representing the slice ** between the low and high range ** ** Arguments: self The old BTreeItems structure ** ilow The start index ** ihigh The end index ** ** Returns: BTreeItems item */ static PyObject * BTreeItems_slice(BTreeItems *self, Py_ssize_t ilow, Py_ssize_t ihigh) { Bucket *lowbucket; Bucket *highbucket; int lowoffset; int highoffset; Py_ssize_t length = -1; /* len(self), but computed only if needed */ /* Complications: * A Python slice never raises IndexError, but BTreeItems_seek does. * Python did only part of index normalization before calling this: * ilow may be < 0 now, and ihigh may be arbitrarily large. It's * our responsibility to clip them. * A Python slice is exclusive of the high index, but a BTreeItems * struct is inclusive on both ends. */ /* First adjust ilow and ihigh to be legit endpoints in the Python * sense (ilow inclusive, ihigh exclusive). This block duplicates the * logic from Python's list_slice function (slicing for builtin lists). */ if (ilow < 0) ilow = 0; else { if (length < 0) length = BTreeItems_length(self); if (ilow > length) ilow = length; } if (ihigh < ilow) ihigh = ilow; else { if (length < 0) length = BTreeItems_length(self); if (ihigh > length) ihigh = length; } assert(0 <= ilow && ilow <= ihigh); assert(length < 0 || ihigh <= length); /* Now adjust for that our struct is inclusive on both ends. This is * easy *except* when the slice is empty: there's no good way to spell * that in an inclusive-on-both-ends scheme. For example, if the * slice is btree.items([:0]), ilow == ihigh == 0 at this point, and if * we were to subtract 1 from ihigh that would get interpreted by * BTreeItems_seek as meaning the *entire* set of items. Setting ilow==1 * and ihigh==0 doesn't work either, as BTreeItems_seek raises IndexError * if we attempt to seek to ilow==1 when the underlying sequence is empty. * It seems simplest to deal with empty slices as a special case here. */ if (ilow == ihigh) /* empty slice */ { lowbucket = highbucket = NULL; lowoffset = 1; highoffset = 0; } else { assert(ilow < ihigh); --ihigh; /* exclusive -> inclusive */ if (BTreeItems_seek(self, ilow) < 0) return NULL; lowbucket = self->currentbucket; lowoffset = self->currentoffset; if (BTreeItems_seek(self, ihigh) < 0) return NULL; highbucket = self->currentbucket; highoffset = self->currentoffset; } return newBTreeItems(self->kind, lowbucket, lowoffset, highbucket, highoffset); } static PyObject * BTreeItems_subscript(BTreeItems *self, PyObject* subscript) { Py_ssize_t len = BTreeItems_length_or_nonzero(self, 0); if (PyIndex_Check(subscript)) { Py_ssize_t i = PyNumber_AsSsize_t(subscript, PyExc_IndexError); if (i == -1 && PyErr_Occurred()) return NULL; if (i < 0) i += len; return BTreeItems_item(self, i); } if (PySlice_Check(subscript)) { Py_ssize_t start, stop, step, slicelength; #ifdef PY3K #define SLICEOBJ(x) (x) #else #define SLICEOBJ(x) (PySliceObject*)(x) #endif if (PySlice_GetIndicesEx(SLICEOBJ(subscript), len, &start, &stop, &step, &slicelength) < 0) { return NULL; } if (step != 1) { PyErr_SetString(PyExc_RuntimeError, "slices must have step size of 1"); return NULL; } return BTreeItems_slice(self, start, stop); } PyErr_SetString(PyExc_RuntimeError, "Unknown index type: must be int or slice"); return NULL; } /* Py3K doesn't honor sequence slicing, so implement via mapping */ static PyMappingMethods BTreeItems_as_mapping = { (lenfunc)BTreeItems_length, /* mp_length */ (binaryfunc)BTreeItems_subscript, /* mp_subscript */ }; static PySequenceMethods BTreeItems_as_sequence = { (lenfunc) BTreeItems_length, /* sq_length */ (binaryfunc)0, /* sq_concat */ (ssizeargfunc)0, /* sq_repeat */ (ssizeargfunc) BTreeItems_item, /* sq_item */ #ifndef PY3K /* Py3K doesn't honor this slot */ (ssizessizeargfunc) BTreeItems_slice, /* sq_slice */ #endif }; /* Number Method items (just for nb_nonzero!) */ static int BTreeItems_nonzero(BTreeItems *self) { return BTreeItems_length_or_nonzero(self, 1); } static PyNumberMethods BTreeItems_as_number_for_nonzero = { 0, /* nb_add */ 0, /* nb_subtract */ 0, /* nb_multiply */ #ifndef PY3K 0, /* nb_divide */ #endif 0, /* nb_remainder */ 0, /* nb_divmod */ 0, /* nb_power */ 0, /* nb_negative */ 0, /* nb_positive */ 0, /* nb_absolute */ (inquiry)BTreeItems_nonzero /* nb_nonzero */ }; static PyTypeObject BTreeItemsType = { PyVarObject_HEAD_INIT(NULL, 0) MOD_NAME_PREFIX "BTreeItems", /* tp_name */ sizeof(BTreeItems), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ (destructor) BTreeItems_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* obsolete tp_getattr */ 0, /* obsolete tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ &BTreeItems_as_number_for_nonzero, /* tp_as_number */ &BTreeItems_as_sequence, /* tp_as_sequence */ &BTreeItems_as_mapping, /* tp_as_mapping */ (hashfunc)0, /* tp_hash */ (ternaryfunc)0, /* tp_call */ (reprfunc)0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ /* Space for future expansion */ 0L,0L, "Sequence type used to iterate over BTree items." /* Documentation string */ }; /* Returns a new BTreeItems object representing the contiguous slice from * offset lowoffset in bucket lowbucket through offset highoffset in bucket * highbucket, inclusive. Pass lowbucket == NULL for an empty slice. * The currentbucket is set to lowbucket, currentoffset ot lowoffset, and * pseudoindex to 0. kind is 'k', 'v' or 'i' (see BTreeItems struct docs). */ static PyObject * newBTreeItems(char kind, Bucket *lowbucket, int lowoffset, Bucket *highbucket, int highoffset) { BTreeItems *self; UNLESS (self = PyObject_NEW(BTreeItems, &BTreeItemsType)) return NULL; self->kind=kind; self->first=lowoffset; self->last=highoffset; if (! lowbucket || ! highbucket || (lowbucket == highbucket && lowoffset > highoffset)) { self->firstbucket = 0; self->lastbucket = 0; self->currentbucket = 0; } else { Py_INCREF(lowbucket); self->firstbucket = lowbucket; Py_INCREF(highbucket); self->lastbucket = highbucket; Py_INCREF(lowbucket); self->currentbucket = lowbucket; } self->currentoffset = lowoffset; self->pseudoindex = 0; return OBJECT(self); } static int nextBTreeItems(SetIteration *i) { if (i->position >= 0) { if (i->position) { DECREF_KEY(i->key); DECREF_VALUE(i->value); } if (BTreeItems_seek(ITEMS(i->set), i->position) >= 0) { Bucket *currentbucket; currentbucket = BUCKET(ITEMS(i->set)->currentbucket); UNLESS(PER_USE(currentbucket)) { /* Mark iteration terminated, so that finiSetIteration doesn't * try to redundantly decref the key and value */ i->position = -1; return -1; } COPY_KEY(i->key, currentbucket->keys[ITEMS(i->set)->currentoffset]); INCREF_KEY(i->key); COPY_VALUE(i->value, currentbucket->values[ITEMS(i->set)->currentoffset]); INCREF_VALUE(i->value); i->position ++; PER_UNUSE(currentbucket); } else { i->position = -1; PyErr_Clear(); } } return 0; } static int nextTreeSetItems(SetIteration *i) { if (i->position >= 0) { if (i->position) { DECREF_KEY(i->key); } if (BTreeItems_seek(ITEMS(i->set), i->position) >= 0) { Bucket *currentbucket; currentbucket = BUCKET(ITEMS(i->set)->currentbucket); UNLESS(PER_USE(currentbucket)) { /* Mark iteration terminated, so that finiSetIteration doesn't * try to redundantly decref the key and value */ i->position = -1; return -1; } COPY_KEY(i->key, currentbucket->keys[ITEMS(i->set)->currentoffset]); INCREF_KEY(i->key); i->position ++; PER_UNUSE(currentbucket); } else { i->position = -1; PyErr_Clear(); } } return 0; } /* Support for the iteration protocol new in Python 2.2. */ static PyTypeObject BTreeIter_Type; /* The type of iterator objects, returned by e.g. iter(IIBTree()). */ typedef struct { PyObject_HEAD /* We use a BTreeItems object because it's convenient and flexible. * We abuse it two ways: * 1. We set currentbucket to NULL when the iteration is finished. * 2. We don't bother keeping pseudoindex in synch. */ BTreeItems *pitems; } BTreeIter; /* Return a new iterator object, to traverse the keys and/or values * represented by pitems. pitems must not be NULL. Returns NULL if error. */ static BTreeIter * BTreeIter_new(BTreeItems *pitems) { BTreeIter *result; assert(pitems != NULL); result = PyObject_New(BTreeIter, &BTreeIter_Type); if (result) { Py_INCREF(pitems); result->pitems = pitems; } return result; } /* The iterator's tp_dealloc slot. */ static void BTreeIter_dealloc(BTreeIter *bi) { Py_DECREF(bi->pitems); PyObject_Del(bi); } /* The implementation of the iterator's tp_iternext slot. Returns "the next" * item; returns NULL if error; returns NULL without setting an error if the * iteration is exhausted (that's the way to terminate the iteration protocol). */ static PyObject * BTreeIter_next(BTreeIter *bi, PyObject *args) { PyObject *result = NULL; /* until proven innocent */ BTreeItems *items = bi->pitems; int i = items->currentoffset; Bucket *bucket = items->currentbucket; if (bucket == NULL) /* iteration termination is sticky */ return NULL; PER_USE_OR_RETURN(bucket, NULL); if (i >= bucket->len) { /* We never leave this routine normally with i >= len: somebody * else mutated the current bucket. */ PyErr_SetString(PyExc_RuntimeError, "the bucket being iterated changed size"); /* Arrange for that this error is sticky too. */ items->currentoffset = INT_MAX; goto Done; } /* Build the result object, from bucket at offset i. */ result = getBucketEntry(bucket, i, items->kind); /* Advance position for next call. */ if (bucket == items->lastbucket && i >= items->last) { /* Next call should terminate the iteration. */ Py_DECREF(items->currentbucket); items->currentbucket = NULL; } else { ++i; if (i >= bucket->len) { Py_XINCREF(bucket->next); items->currentbucket = bucket->next; Py_DECREF(bucket); i = 0; } items->currentoffset = i; } Done: PER_UNUSE(bucket); return result; } static PyObject * BTreeIter_getiter(PyObject *it) { Py_INCREF(it); return it; } static PyTypeObject BTreeIter_Type = { PyVarObject_HEAD_INIT(NULL, 0) MODULE_NAME MOD_NAME_PREFIX "TreeIterator", /* tp_name */ sizeof(BTreeIter), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ (destructor)BTreeIter_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /*PyObject_GenericGetAttr,*/ /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT, /* tp_flags */ 0, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ (getiterfunc)BTreeIter_getiter, /* tp_iter */ (iternextfunc)BTreeIter_next, /* tp_iternext */ 0, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ }; BTrees-4.3.1/BTrees/utils.py0000664000175000017500000000264212510775554016766 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # Copied from ZODB/utils.py from binascii import hexlify from ._compat import _bytes def non_negative(int_val): if int_val < 0: # Coerce to non-negative. int_val &= 0x7FFFFFFFFFFFFFFF return int_val def positive_id(obj): #pragma NO COVER """Return id(obj) as a non-negative integer.""" return non_negative(id(obj)) def oid_repr(oid): if isinstance(oid, _bytes) and len(oid) == 8: # Convert to hex and strip leading zeroes. as_hex = hexlify(oid).lstrip(b'0') # Ensure two characters per input byte. chunks = [b'0x'] if len(as_hex) & 1: chunks.append(b'0') elif as_hex == b'': as_hex = b'00' chunks.append(as_hex) return b''.join(chunks) else: return repr(oid) BTrees-4.3.1/BTrees/OOBTree.py0000664000175000017500000000611612665316015017057 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __all__ = ('Bucket', 'Set', 'BTree', 'TreeSet', 'OOBucket', 'OOSet', 'OOBTree', 'OOTreeSet', 'union', 'intersection','difference', ) from zope.interface import moduleProvides from .Interfaces import IObjectObjectBTreeModule from ._base import Bucket from ._base import Set from ._base import Tree as BTree from ._base import TreeSet from ._base import _TreeIterator from ._base import difference as _difference from ._base import intersection as _intersection from ._base import set_operation as _set_operation from ._base import to_ob as _to_key from ._base import to_ob as _to_value from ._base import union as _union from ._base import _fix_pickle _BUCKET_SIZE = 30 _TREE_SIZE = 250 using64bits = False class OOBucketPy(Bucket): _to_key = _to_key _to_value = _to_value class OOSetPy(Set): _to_key = _to_key class OOBTreePy(BTree): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key _to_value = _to_value class OOTreeSetPy(TreeSet): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key class OOTreeIteratorPy(_TreeIterator): pass # Can't declare forward refs, so fix up afterwards: OOBucketPy._mapping_type = OOBucketPy._bucket_type = OOBucketPy OOBucketPy._set_type = OOSetPy OOSetPy._mapping_type = OOBucketPy OOSetPy._set_type = OOSetPy._bucket_type = OOSetPy OOBTreePy._mapping_type = OOBTreePy._bucket_type = OOBucketPy OOBTreePy._set_type = OOSetPy OOTreeSetPy._mapping_type = OOBucketPy OOTreeSetPy._set_type = OOTreeSetPy._bucket_type = OOSetPy differencePy = _set_operation(_difference, OOSetPy) unionPy = _set_operation(_union, OOSetPy) intersectionPy = _set_operation(_intersection, OOSetPy) try: from ._OOBTree import OOBucket except ImportError as e: #pragma NO COVER w/ C extensions OOBucket = OOBucketPy OOSet = OOSetPy OOBTree = OOBTreePy OOTreeSet = OOTreeSetPy OOTreeIterator = OOTreeIteratorPy difference = differencePy union = unionPy intersection = intersectionPy else: #pragma NO COVER w/o C extensions from ._OOBTree import OOSet from ._OOBTree import OOBTree from ._OOBTree import OOTreeSet from ._OOBTree import OOTreeIterator from ._OOBTree import difference from ._OOBTree import union from ._OOBTree import intersection Bucket = OOBucket Set = OOSet BTree = OOBTree TreeSet = OOTreeSet _fix_pickle(globals(), __name__) moduleProvides(IObjectObjectBTreeModule) BTrees-4.3.1/BTrees/OIBTree.py0000664000175000017500000001002112665316015017037 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __all__ = ('Bucket', 'Set', 'BTree', 'TreeSet', 'OIBucket', 'OISet', 'OIBTree', 'OITreeSet', 'union', 'intersection', 'difference', 'weightedUnion', 'weightedIntersection', ) from zope.interface import moduleProvides from .Interfaces import IObjectIntegerBTreeModule from ._base import Bucket from ._base import MERGE from ._base import MERGE_WEIGHT_numeric from ._base import MERGE_DEFAULT_float from ._base import Set from ._base import Tree as BTree from ._base import TreeSet from ._base import _TreeIterator from ._base import difference as _difference from ._base import intersection as _intersection from ._base import set_operation as _set_operation from ._base import to_ob as _to_key from ._base import to_int as _to_value from ._base import union as _union from ._base import weightedIntersection as _weightedIntersection from ._base import weightedUnion as _weightedUnion from ._base import _fix_pickle _BUCKET_SIZE = 60 _TREE_SIZE = 250 using64bits = True class OIBucketPy(Bucket): _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class OISetPy(Set): _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class OIBTreePy(BTree): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class OITreeSetPy(TreeSet): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_float class OITreeIteratorPy(_TreeIterator): pass # Can't declare forward refs, so fix up afterwards: OIBucketPy._mapping_type = OIBucketPy._bucket_type = OIBucketPy OIBucketPy._set_type = OISetPy OISetPy._mapping_type = OIBucketPy OISetPy._set_type = OISetPy._bucket_type = OISetPy OIBTreePy._mapping_type = OIBTreePy._bucket_type = OIBucketPy OIBTreePy._set_type = OISetPy OITreeSetPy._mapping_type = OIBucketPy OITreeSetPy._set_type = OITreeSetPy._bucket_type = OISetPy differencePy = _set_operation(_difference, OISetPy) unionPy = _set_operation(_union, OISetPy) intersectionPy = _set_operation(_intersection, OISetPy) weightedUnionPy = _set_operation(_weightedUnion, OISetPy) weightedIntersectionPy = _set_operation(_weightedIntersection, OISetPy) try: from ._OIBTree import OIBucket except ImportError: #pragma NO COVER w/ C extensions OIBucket = OIBucketPy OISet = OISetPy OIBTree = OIBTreePy OITreeSet = OITreeSetPy OITreeIterator = OITreeIteratorPy difference = differencePy union = unionPy intersection = intersectionPy weightedUnion = weightedUnionPy weightedIntersection = weightedIntersectionPy else: #pragma NO COVER w/o C extensions from ._OIBTree import OISet from ._OIBTree import OIBTree from ._OIBTree import OITreeSet from ._OIBTree import OITreeIterator from ._OIBTree import difference from ._OIBTree import union from ._OIBTree import intersection from ._OIBTree import weightedUnion from ._OIBTree import weightedIntersection Bucket = OIBucket Set = OISet BTree = OIBTree TreeSet = OITreeSet _fix_pickle(globals(), __name__) moduleProvides(IObjectIntegerBTreeModule) BTrees-4.3.1/BTrees/floatvaluemacros.h0000664000175000017500000000160312510775554020770 0ustar tseavertseaver00000000000000 #define VALUEMACROS_H "$Id$\n" #define VALUE_TYPE float #undef VALUE_TYPE_IS_PYOBJECT #define TEST_VALUE(K, T) (((K) < (T)) ? -1 : (((K) > (T)) ? 1: 0)) #define VALUE_SAME(VALUE, TARGET) ( (VALUE) == (TARGET) ) #define DECLARE_VALUE(NAME) VALUE_TYPE NAME #define VALUE_PARSE "f" #define DECREF_VALUE(k) #define INCREF_VALUE(k) #define COPY_VALUE(V, E) (V=(E)) #define COPY_VALUE_TO_OBJECT(O, K) O=PyFloat_FromDouble(K) #define COPY_VALUE_FROM_ARG(TARGET, ARG, STATUS) \ if (PyFloat_Check(ARG)) TARGET = (float)PyFloat_AsDouble(ARG); \ else if (INT_CHECK(ARG)) TARGET = (float)INT_AS_LONG(ARG); \ else { \ PyErr_SetString(PyExc_TypeError, "expected float or int value"); \ (STATUS)=0; (TARGET)=0; } #define NORMALIZE_VALUE(V, MIN) ((MIN) > 0) ? ((V)/=(MIN)) : 0 #define MERGE_DEFAULT 1.0f #define MERGE(O1, w1, O2, w2) ((O1)*(w1)+(O2)*(w2)) #define MERGE_WEIGHT(O, w) ((O)*(w)) BTrees-4.3.1/BTrees/_OIBTree.c0000664000175000017500000000216512510775554017010 0ustar tseavertseaver00000000000000/*############################################################################ # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id$\n" /* OIBTree - object key, int value BTree Implements a collection using object type keys and int type values */ #define PERSISTENT #define MOD_NAME_PREFIX "OI" #define DEFAULT_MAX_BUCKET_SIZE 60 #define DEFAULT_MAX_BTREE_SIZE 250 #include "_compat.h" #include "objectkeymacros.h" #include "intvaluemacros.h" #ifdef PY3K #define INITMODULE PyInit__OIBTree #else #define INITMODULE init_OIBTree #endif #include "BTreeModuleTemplate.c" BTrees-4.3.1/BTrees/sorters.c0000664000175000017500000003556212576064726017134 0ustar tseavertseaver00000000000000/***************************************************************************** Copyright (c) 2002 Zope Foundation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ /* Revision information: $Id$ */ /* The only routine here intended to be used outside the file is size_t sort_int_nodups(int *p, size_t n) Sort the array of n ints pointed at by p, in place, and also remove duplicates. Return the number of unique elements remaining, which occupy a contiguous and monotonically increasing slice of the array starting at p. Example: If the input array is [3, 1, 2, 3, 1, 5, 2], sort_int_nodups returns 4, and the first 4 elements of the array are changed to [1, 2, 3, 5]. The content of the remaining array positions is not defined. Notes: + This is specific to n-byte signed ints, with endianness natural to the platform. `n` is determined based on ZODB_64BIT_INTS. + 4*n bytes of available heap memory are required for best speed (8*n when ZODB_64BIT_INTS is defined). */ #include #include #include #include #include /* The type of array elements to be sorted. Most of the routines don't care about the type, and will work fine for any scalar C type (provided they're recompiled with element_type appropriately redefined). However, the radix sort has to know everything about the type's internal representation. */ typedef KEY_TYPE element_type; /* The radixsort is faster than the quicksort for large arrays, but radixsort has high fixed overhead, making it a poor choice for small arrays. The crossover point isn't critical, and is sensitive to things like compiler and machine cache structure, so don't worry much about this. */ #define QUICKSORT_BEATS_RADIXSORT 800U /* In turn, the quicksort backs off to an insertion sort for very small slices. MAX_INSERTION is the largest slice quicksort leaves entirely to insertion. Because this version of quicksort uses a median-of-3 rule for selecting a pivot, MAX_INSERTION must be at least 2 (so that quicksort has at least 3 values to look at in a slice). Again, the exact value here isn't critical. */ #define MAX_INSERTION 25U #if MAX_INSERTION < 2U # error "MAX_INSERTION must be >= 2" #endif /* LSB-first radix sort of the n elements in 'in'. 'work' is work storage at least as large as 'in'. Depending on how many swaps are done internally, the final result may come back in 'in' or 'work'; and that pointer is returned. radixsort_int is specific to signed n-byte ints, with natural machine endianness. `n` is determined based on ZODB_64BIT_INTS. */ static element_type* radixsort_int(element_type *in, element_type *work, size_t n) { /* count[i][j] is the number of input elements that have byte value j in byte position i, where byte position 0 is the LSB. Note that holding i fixed, the sum of count[i][j] over all j in range(256) is n. */ #ifdef ZODB_64BIT_INTS size_t count[8][256]; #else size_t count[4][256]; #endif size_t i; int offset, offsetinc; /* Which byte position are we working on now? 0=LSB, 1, 2, ... */ size_t bytenum; #ifdef ZODB_64BIT_INTS assert(sizeof(element_type) == 8); #else assert(sizeof(element_type) == 4); #endif assert(in); assert(work); /* Compute all of count in one pass. */ memset(count, 0, sizeof(count)); for (i = 0; i < n; ++i) { element_type const x = in[i]; ++count[0][(x ) & 0xff]; ++count[1][(x >> 8) & 0xff]; ++count[2][(x >> 16) & 0xff]; ++count[3][(x >> 24) & 0xff]; #ifdef ZODB_64BIT_INTS ++count[4][(x >> 32) & 0xff]; ++count[5][(x >> 40) & 0xff]; ++count[6][(x >> 48) & 0xff]; ++count[7][(x >> 56) & 0xff]; #endif } /* For p an element_type* cast to char*, offset is how much farther we have to go to get to the LSB of the element; this is 0 for little- endian boxes and sizeof(element_type)-1 for big-endian. offsetinc is 1 or -1, respectively, telling us which direction to go from p+offset to get to the element's more-significant bytes. */ { element_type one = 1; if (*(char*)&one) { /* Little endian. */ offset = 0; offsetinc = 1; } else { /* Big endian. */ offset = sizeof(element_type) - 1; offsetinc = -1; } } /* The radix sort. */ for (bytenum = 0; bytenum < sizeof(element_type); ++bytenum, offset += offsetinc) { /* Do a stable distribution sort on byte position bytenum, from in to work. index[i] tells us the work index at which to store the next in element with byte value i. pinbyte points to the correct byte in the input array. */ size_t index[256]; unsigned char* pinbyte; size_t total = 0; size_t *pcount = count[bytenum]; /* Compute the correct output starting index for each possible byte value. */ if (bytenum < sizeof(element_type) - 1) { for (i = 0; i < 256; ++i) { const size_t icount = pcount[i]; index[i] = total; total += icount; if (icount == n) break; } if (i < 256) { /* All bytes in the current position have value i, so there's nothing to do on this pass. */ continue; } } else { /* The MSB of signed ints needs to be distributed differently than the other bytes, in order 0x80, 0x81, ... 0xff, 0x00, 0x01, ... 0x7f */ for (i = 128; i < 256; ++i) { const size_t icount = pcount[i]; index[i] = total; total += icount; if (icount == n) break; } if (i < 256) continue; for (i = 0; i < 128; ++i) { const size_t icount = pcount[i]; index[i] = total; total += icount; if (icount == n) break; } if (i < 128) continue; } assert(total == n); /* Distribute the elements according to byte value. Note that this is where most of the time is spent. Note: The loop is unrolled 4x by hand, for speed. This may be a pessimization someday, but was a significant win on my MSVC 6.0 timing tests. */ pinbyte = (unsigned char *)in + offset; i = 0; /* Reduce number of elements to copy to a multiple of 4. */ while ((n - i) & 0x3) { unsigned char byte = *pinbyte; work[index[byte]++] = in[i]; ++i; pinbyte += sizeof(element_type); } for (; i < n; i += 4, pinbyte += 4 * sizeof(element_type)) { unsigned char byte1 = *(pinbyte ); unsigned char byte2 = *(pinbyte + sizeof(element_type)); unsigned char byte3 = *(pinbyte + 2 * sizeof(element_type)); unsigned char byte4 = *(pinbyte + 3 * sizeof(element_type)); element_type in1 = in[i ]; element_type in2 = in[i+1]; element_type in3 = in[i+2]; element_type in4 = in[i+3]; work[index[byte1]++] = in1; work[index[byte2]++] = in2; work[index[byte3]++] = in3; work[index[byte4]++] = in4; } /* Swap in and work (just a pointer swap). */ { element_type *temp = in; in = work; work = temp; } } return in; } /* Remove duplicates from sorted array in, storing exactly one of each distinct element value into sorted array out. It's OK (and expected!) for in == out, but otherwise the n elements beginning at in must not overlap with the n beginning at out. Return the number of elements in out. */ static size_t uniq(element_type *out, element_type *in, size_t n) { size_t i; element_type lastelt; element_type *pout; assert(out); assert(in); if (n == 0) return 0; /* i <- first index in 'in' that contains a duplicate. in[0], in[1], ... in[i-1] are unique, but in[i-1] == in[i]. Set i to n if everything is unique. */ for (i = 1; i < n; ++i) { if (in[i-1] == in[i]) break; } /* in[:i] is unique; copy to out[:i] if needed. */ assert(i > 0); if (in != out) memcpy(out, in, i * sizeof(element_type)); pout = out + i; lastelt = in[i-1]; /* safe even when i == n */ for (++i; i < n; ++i) { element_type elt = in[i]; if (elt != lastelt) *pout++ = lastelt = elt; } return pout - out; } #if 0 /* insertionsort is no longer referenced directly, but I'd like to keep * the code here just in case. */ /* Straight insertion sort of the n elements starting at 'in'. */ static void insertionsort(element_type *in, size_t n) { element_type *p, *q; element_type minimum; /* smallest seen so far */ element_type *plimit = in + n; assert(in); if (n < 2) return; minimum = *in; for (p = in+1; p < plimit; ++p) { /* *in <= *(in+1) <= ... <= *(p-1). Slide *p into place. */ element_type thiselt = *p; if (thiselt < minimum) { /* This is a new minimum. This saves p-in compares when it happens, but should happen so rarely that it's not worth checking for its own sake: the point is that the far more popular 'else' branch can exploit that thiselt is *not* the smallest so far. */ memmove(in+1, in, (p - in) * sizeof(*in)); *in = minimum = thiselt; } else { /* thiselt >= minimum, so the loop will find a q with *q <= thiselt. This saves testing q >= in on each trip. It's such a simple loop that saving a per-trip test is a major speed win. */ for (q = p-1; *q > thiselt; --q) *(q+1) = *q; *(q+1) = thiselt; } } } #endif /* The maximum number of elements in the pending-work stack quicksort maintains. The maximum stack depth is approximately log2(n), so arrays of size up to approximately MAX_INSERTION * 2**STACKSIZE can be sorted. The memory burden for the stack is small, so better safe than sorry. */ #define STACKSIZE 60 /* A _stacknode remembers a contiguous slice of an array that needs to sorted. lo must be <= hi, and, unlike Python array slices, this includes both ends. */ struct _stacknode { element_type *lo; element_type *hi; }; static void quicksort(element_type *plo, size_t n) { element_type *phi; /* Swap two array elements. */ element_type _temp; #define SWAP(P, Q) (_temp = *(P), *(P) = *(Q), *(Q) = _temp) /* Stack of pending array slices to be sorted. */ struct _stacknode stack[STACKSIZE]; struct _stacknode *stackfree = stack; /* available stack slot */ /* Push an array slice on the pending-work stack. */ #define PUSH(PLO, PHI) \ do { \ assert(stackfree - stack < STACKSIZE); \ assert((PLO) <= (PHI)); \ stackfree->lo = (PLO); \ stackfree->hi = (PHI); \ ++stackfree; \ } while(0) assert(plo); phi = plo + n - 1; for (;;) { element_type pivot; element_type *pi, *pj; assert(plo <= phi); n = phi - plo + 1; if (n <= MAX_INSERTION) { /* Do a small insertion sort. Contra Knuth, we do this now instead of waiting until the end, because this little slice is likely still in cache now. */ element_type *p, *q; element_type minimum = *plo; for (p = plo+1; p <= phi; ++p) { /* *plo <= *(plo+1) <= ... <= *(p-1). Slide *p into place. */ element_type thiselt = *p; if (thiselt < minimum) { /* New minimum. */ memmove(plo+1, plo, (p - plo) * sizeof(*p)); *plo = minimum = thiselt; } else { /* thiselt >= minimum, so the loop will find a q with *q <= thiselt. */ for (q = p-1; *q > thiselt; --q) *(q+1) = *q; *(q+1) = thiselt; } } /* Pop another slice off the stack. */ if (stack == stackfree) break; /* no more slices -- we're done */ --stackfree; plo = stackfree->lo; phi = stackfree->hi; continue; } /* Parition the slice. For pivot, take the median of the leftmost, rightmost, and middle elements. First sort those three; then the median is the middle one. For technical reasons, the middle element is swapped to plo+1 first (see Knuth Vol 3 Ed 2 section 5.2.2 exercise 55 -- reverse-sorted arrays can take quadratic time otherwise!). */ { element_type *plop1 = plo + 1; element_type *pmid = plo + (n >> 1); assert(plo < pmid && pmid < phi); SWAP(plop1, pmid); /* Sort plo, plop1, phi. */ /* Smaller of rightmost two -> middle. */ if (*plop1 > *phi) SWAP(plop1, phi); /* Smallest of all -> left; if plo is already the smallest, the sort is complete. */ if (*plo > *plop1) { SWAP(plo, plop1); /* Largest of all -> right. */ if (*plop1 > *phi) SWAP(plop1, phi); } pivot = *plop1; pi = plop1; } assert(*plo <= pivot); assert(*pi == pivot); assert(*phi >= pivot); pj = phi; /* Partition wrt pivot. This is the time-critical part, and nearly every decision in the routine aims at making this loop as fast as possible -- even small points like arranging that all loop tests can be done correctly at the bottoms of loops instead of the tops, and that pointers can be derefenced directly as-is (without fiddly +1 or -1). The aim is to make the C here so simple that a compiler has a good shot at doing as well as hand-crafted assembler. */ for (;;) { /* Invariants: 1. pi < pj. 2. All elements at plo, plo+1 .. pi are <= pivot. 3. All elements at pj, pj+1 .. phi are >= pivot. 4. There is an element >= pivot to the right of pi. 5. There is an element <= pivot to the left of pj. Note that #4 and #5 save us from needing to check that the pointers stay in bounds. */ assert(pi < pj); do { ++pi; } while (*pi < pivot); assert(pi <= pj); do { --pj; } while (*pj > pivot); assert(pj >= pi - 1); if (pi < pj) SWAP(pi, pj); else break; } assert(plo+1 < pi && pi <= phi); assert(plo < pj && pj < phi); assert(*pi >= pivot); assert( (pi == pj && *pj == pivot) || (pj + 1 == pi && *pj <= pivot) ); /* Swap pivot into its final position, pj. */ assert(plo[1] == pivot); plo[1] = *pj; *pj = pivot; /* Subfiles are from plo to pj-1 inclusive, and pj+1 to phi inclusive. Push the larger one, and loop back to do the smaller one directly. */ if (pj - plo >= phi - pj) { PUSH(plo, pj-1); plo = pj+1; } else { PUSH(pj+1, phi); phi = pj-1; } } #undef PUSH #undef SWAP } /* Sort p and remove duplicates, as fast as we can. */ static size_t sort_int_nodups(KEY_TYPE *p, size_t n) { size_t nunique; element_type *work; assert(sizeof(KEY_TYPE) == sizeof(element_type)); assert(p); /* Use quicksort if the array is small, OR if malloc can't find enough temp memory for radixsort. */ work = NULL; if (n > QUICKSORT_BEATS_RADIXSORT) work = (element_type *)malloc(n * sizeof(element_type)); if (work) { element_type *out = radixsort_int(p, work, n); nunique = uniq(p, out, n); free(work); } else { quicksort(p, n); nunique = uniq(p, p, n); } return nunique; } BTrees-4.3.1/BTrees/MergeTemplate.c0000664000175000017500000002744612510775554020164 0ustar tseavertseaver00000000000000/***************************************************************************** Copyright (c) 2001, 2002 Zope Foundation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #define MERGETEMPLATE_C "$Id$\n" /**************************************************************************** Set operations ****************************************************************************/ static int merge_output(Bucket *r, SetIteration *i, int mapping) { if (r->len >= r->size && Bucket_grow(r, -1, !mapping) < 0) return -1; COPY_KEY(r->keys[r->len], i->key); INCREF_KEY(r->keys[r->len]); if (mapping) { COPY_VALUE(r->values[r->len], i->value); INCREF_VALUE(r->values[r->len]); } r->len++; return 0; } /* The "reason" argument is a little integer giving "a reason" for the * error. In the Zope3 codebase, these are mapped to explanatory strings * via zodb/btrees/interfaces.py. */ static PyObject * merge_error(int p1, int p2, int p3, int reason) { PyObject *r; UNLESS (r=Py_BuildValue("iiii", p1, p2, p3, reason)) r=Py_None; if (ConflictError == NULL) { ConflictError = PyExc_ValueError; Py_INCREF(ConflictError); } PyErr_SetObject(ConflictError, r); if (r != Py_None) { Py_DECREF(r); } return NULL; } /* It's hard to explain "the rules" for bucket_merge, in large part because * any automatic conflict-resolution scheme is going to be incorrect for * some endcases of *some* app. The scheme here is pretty conservative, * and should be OK for most apps. It's easier to explain what the code * allows than what it forbids: * * Leaving things alone: it's OK if both s2 and s3 leave a piece of s1 * alone (don't delete the key, and don't change the value). * * Key deletion: a transaction (s2 or s3) can delete a key (from s1), but * only if the other transaction (of s2 and s3) doesn't delete the same key. * However, it's not OK for s2 and s3 to, between them, end up deleting all * the keys. This is a higher-level constraint, due to that the caller of * bucket_merge() doesn't have enough info to unlink the resulting empty * bucket from its BTree correctly. It's also not OK if s2 or s3 are empty, * because the transaction that emptied the bucket unlinked the bucket from * the tree, and nothing we do here can get it linked back in again. * * Key insertion: s2 or s3 can add a new key, provided the other transaction * doesn't insert the same key. It's not OK even if they insert the same * pair. * * Mapping value modification: s2 or s3 can modify the value associated * with a key in s1, provided the other transaction doesn't make a * modification of the same key to a different value. It's OK if s2 and s3 * both give the same new value to the key while it's hard to be precise about * why, this doesn't seem consistent with that it's *not* OK for both to add * a new key mapping to the same value). */ static PyObject * bucket_merge(Bucket *s1, Bucket *s2, Bucket *s3) { Bucket *r=0; PyObject *s; SetIteration i1 = {0,0,0}, i2 = {0,0,0}, i3 = {0,0,0}; int cmp12, cmp13, cmp23, mapping, set; /* If either "after" bucket is empty, punt. */ if (s2->len == 0 || s3->len == 0) { merge_error(-1, -1, -1, 12); goto err; } if (initSetIteration(&i1, OBJECT(s1), 1) < 0) goto err; if (initSetIteration(&i2, OBJECT(s2), 1) < 0) goto err; if (initSetIteration(&i3, OBJECT(s3), 1) < 0) goto err; mapping = i1.usesValue | i2.usesValue | i3.usesValue; set = !mapping; if (mapping) r = (Bucket *)PyObject_CallObject((PyObject *)&BucketType, NULL); else r = (Bucket *)PyObject_CallObject((PyObject *)&SetType, NULL); if (r == NULL) goto err; if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; if (i3.next(&i3) < 0) goto err; /* Consult zodb/btrees/interfaces.py for the meaning of the last * argument passed to merge_error(). */ /* TODO: This isn't passing on errors raised by value comparisons. */ while (i1.position >= 0 && i2.position >= 0 && i3.position >= 0) { TEST_KEY_SET_OR(cmp12, i1.key, i2.key) goto err; TEST_KEY_SET_OR(cmp13, i1.key, i3.key) goto err; if (cmp12==0) { if (cmp13==0) { if (set || (TEST_VALUE(i1.value, i2.value) == 0)) { /* change in i3 value or all same */ if (merge_output(r, &i3, mapping) < 0) goto err; } else if (set || (TEST_VALUE(i1.value, i3.value) == 0)) { /* change in i2 value */ if (merge_output(r, &i2, mapping) < 0) goto err; } else { /* conflicting value changes in i2 and i3 */ merge_error(i1.position, i2.position, i3.position, 1); goto err; } if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else if (cmp13 > 0) { /* insert i3 */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else if (set || (TEST_VALUE(i1.value, i2.value) == 0)) { /* deleted in i3 */ if (i3.position == 1) { /* Deleted the first item. This will modify the parent node, so we don't know if merging will be safe */ merge_error(i1.position, i2.position, i3.position, 13); goto err; } if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; } else { /* conflicting del in i3 and change in i2 */ merge_error(i1.position, i2.position, i3.position, 2); goto err; } } else if (cmp13 == 0) { if (cmp12 > 0) { /* insert i2 */ if (merge_output(r, &i2, mapping) < 0) goto err; if (i2.next(&i2) < 0) goto err; } else if (set || (TEST_VALUE(i1.value, i3.value) == 0)) { /* deleted in i2 */ if (i2.position == 1) { /* Deleted the first item. This will modify the parent node, so we don't know if merging will be safe */ merge_error(i1.position, i2.position, i3.position, 13); goto err; } if (i1.next(&i1) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else { /* conflicting del in i2 and change in i3 */ merge_error(i1.position, i2.position, i3.position, 3); goto err; } } else { /* Both keys changed */ TEST_KEY_SET_OR(cmp23, i2.key, i3.key) goto err; if (cmp23==0) { /* dueling inserts or deletes */ merge_error(i1.position, i2.position, i3.position, 4); goto err; } if (cmp12 > 0) { /* insert i2 */ if (cmp23 > 0) { /* insert i3 first */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else { /* insert i2 first */ if (merge_output(r, &i2, mapping) < 0) goto err; if (i2.next(&i2) < 0) goto err; } } else if (cmp13 > 0) { /* Insert i3 */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else { /* 1<2 and 1<3: both deleted 1.key */ merge_error(i1.position, i2.position, i3.position, 5); goto err; } } } while (i2.position >= 0 && i3.position >= 0) { /* New inserts */ TEST_KEY_SET_OR(cmp23, i2.key, i3.key) goto err; if (cmp23==0) { /* dueling inserts */ merge_error(i1.position, i2.position, i3.position, 6); goto err; } if (cmp23 > 0) { /* insert i3 */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else { /* insert i2 */ if (merge_output(r, &i2, mapping) < 0) goto err; if (i2.next(&i2) < 0) goto err; } } while (i1.position >= 0 && i2.position >= 0) { /* remainder of i1 deleted in i3 */ TEST_KEY_SET_OR(cmp12, i1.key, i2.key) goto err; if (cmp12 > 0) { /* insert i2 */ if (merge_output(r, &i2, mapping) < 0) goto err; if (i2.next(&i2) < 0) goto err; } else if (cmp12==0 && (set || (TEST_VALUE(i1.value, i2.value) == 0))) { /* delete i3 */ if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; } else { /* Dueling deletes or delete and change */ merge_error(i1.position, i2.position, i3.position, 7); goto err; } } while (i1.position >= 0 && i3.position >= 0) { /* remainder of i1 deleted in i2 */ TEST_KEY_SET_OR(cmp13, i1.key, i3.key) goto err; if (cmp13 > 0) { /* insert i3 */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else if (cmp13==0 && (set || (TEST_VALUE(i1.value, i3.value) == 0))) { /* delete i2 */ if (i1.next(&i1) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else { /* Dueling deletes or delete and change */ merge_error(i1.position, i2.position, i3.position, 8); goto err; } } if (i1.position >= 0) { /* Dueling deletes */ merge_error(i1.position, i2.position, i3.position, 9); goto err; } while (i2.position >= 0) { /* Inserting i2 at end */ if (merge_output(r, &i2, mapping) < 0) goto err; if (i2.next(&i2) < 0) goto err; } while (i3.position >= 0) { /* Inserting i3 at end */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } /* If the output bucket is empty, conflict resolution doesn't have * enough info to unlink it from its containing BTree correctly. */ if (r->len == 0) { merge_error(-1, -1, -1, 10); goto err; } finiSetIteration(&i1); finiSetIteration(&i2); finiSetIteration(&i3); if (s1->next) { Py_INCREF(s1->next); r->next = s1->next; } s = bucket_getstate(r); Py_DECREF(r); return s; err: finiSetIteration(&i1); finiSetIteration(&i2); finiSetIteration(&i3); Py_XDECREF(r); return NULL; } BTrees-4.3.1/BTrees/check.py0000664000175000017500000003572112510775554016707 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2003 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Utilities for working with BTrees (TreeSets, Buckets, and Sets) at a low level. The primary function is check(btree), which performs value-based consistency checks of a kind btree._check() does not perform. See the function docstring for details. display(btree) displays the internal structure of a BTree (TreeSet, etc) to stdout. CAUTION: When a BTree node has only a single bucket child, it can be impossible to get at the bucket from Python code (__getstate__() may squash the bucket object out of existence, as a pickling storage optimization). In such a case, the code here synthesizes a temporary bucket with the same keys (and values, if the bucket is of a mapping type). This has no first-order consequences, but can mislead if you pay close attention to reported object addresses and/or object identity (the synthesized bucket has an address that doesn't exist in the actual BTree). """ from BTrees.IFBTree import IFBTree, IFBucket, IFSet, IFTreeSet from BTrees.IFBTree import IFBTreePy, IFBucketPy, IFSetPy, IFTreeSetPy from BTrees.IIBTree import IIBTree, IIBucket, IISet, IITreeSet from BTrees.IIBTree import IIBTreePy, IIBucketPy, IISetPy, IITreeSetPy from BTrees.IOBTree import IOBTree, IOBucket, IOSet, IOTreeSet from BTrees.IOBTree import IOBTreePy, IOBucketPy, IOSetPy, IOTreeSetPy from BTrees.LFBTree import LFBTree, LFBucket, LFSet, LFTreeSet from BTrees.LFBTree import LFBTreePy, LFBucketPy, LFSetPy, LFTreeSetPy from BTrees.LLBTree import LLBTree, LLBucket, LLSet, LLTreeSet from BTrees.LLBTree import LLBTreePy, LLBucketPy, LLSetPy, LLTreeSetPy from BTrees.LOBTree import LOBTree, LOBucket, LOSet, LOTreeSet from BTrees.LOBTree import LOBTreePy, LOBucketPy, LOSetPy, LOTreeSetPy from BTrees.OIBTree import OIBTree, OIBucket, OISet, OITreeSet from BTrees.OIBTree import OIBTreePy, OIBucketPy, OISetPy, OITreeSetPy from BTrees.OLBTree import OLBTree, OLBucket, OLSet, OLTreeSet from BTrees.OLBTree import OLBTreePy, OLBucketPy, OLSetPy, OLTreeSetPy from BTrees.OOBTree import OOBTree, OOBucket, OOSet, OOTreeSet from BTrees.OOBTree import OOBTreePy, OOBucketPy, OOSetPy, OOTreeSetPy from BTrees.utils import positive_id from BTrees.utils import oid_repr TYPE_UNKNOWN, TYPE_BTREE, TYPE_BUCKET = range(3) _type2kind = {} for kv in ('OO', 'II', 'IO', 'OI', 'IF', 'LL', 'LO', 'OL', 'LF', ): for name, kind in ( ('BTree', (TYPE_BTREE, True)), ('Bucket', (TYPE_BUCKET, True)), ('TreeSet', (TYPE_BTREE, False)), ('Set', (TYPE_BUCKET, False)), ): _type2kind[globals()[kv+name]] = kind py = kv + name + 'Py' _type2kind[globals()[py]] = kind # Return pair # # TYPE_BTREE or TYPE_BUCKET, is_mapping def classify(obj): return _type2kind[type(obj)] BTREE_EMPTY, BTREE_ONE, BTREE_NORMAL = range(3) # If the BTree is empty, returns # # BTREE_EMPTY, [], [] # # If the BTree has only one bucket, sometimes returns # # BTREE_ONE, bucket_state, None # # Else returns # # BTREE_NORMAL, list of keys, list of kids # # and the list of kids has one more entry than the list of keys. # # BTree.__getstate__() docs: # # For an empty BTree (self->len == 0), None. # # For a BTree with one child (self->len == 1), and that child is a bucket, # and that bucket has a NULL oid, a one-tuple containing a one-tuple # containing the bucket's state: # # ( # ( # child[0].__getstate__(), # ), # ) # # Else a two-tuple. The first element is a tuple interleaving the BTree's # keys and direct children, of size 2*self->len - 1 (key[0] is unused and # is not saved). The second element is the firstbucket: # # ( # (child[0], key[1], child[1], key[2], child[2], ..., # key[len-1], child[len-1]), # self->firstbucket # ) _btree2bucket = {} for kv in ('OO', 'II', 'IO', 'OI', 'IF', 'LL', 'LO', 'OL', 'LF', ): _btree2bucket[globals()[kv+'BTree']] = globals()[kv+'Bucket'] py = kv + 'BTreePy' _btree2bucket[globals()[py]] = globals()[kv+'BucketPy'] _btree2bucket[globals()[kv+'TreeSet']] = globals()[kv+'Set'] py = kv + 'TreeSetPy' _btree2bucket[globals()[kv+'TreeSetPy']] = globals()[kv+'SetPy'] def crack_btree(t, is_mapping): state = t.__getstate__() if state is None: return BTREE_EMPTY, [], [] assert isinstance(state, tuple) if len(state) == 1: state = state[0] assert isinstance(state, tuple) and len(state) == 1 state = state[0] return BTREE_ONE, state, None assert len(state) == 2 data, firstbucket = state n = len(data) assert n & 1 kids = [] keys = [] i = 0 for x in data: if i & 1: keys.append(x) else: kids.append(x) i += 1 return BTREE_NORMAL, keys, kids # Returns # # keys, values # for a mapping; len(keys) == len(values) in this case # or # keys, [] # for a set # # bucket.__getstate__() docs: # # For a set bucket (self->values is NULL), a one-tuple or two-tuple. The # first element is a tuple of keys, of length self->len. The second element # is the next bucket, present if and only if next is non-NULL: # # ( # (keys[0], keys[1], ..., keys[len-1]), # next iff non-NULL> # ) # # For a mapping bucket (self->values is not NULL), a one-tuple or two-tuple. # The first element is a tuple interleaving keys and values, of length # 2 * self->len. The second element is the next bucket, present iff next is # non-NULL: # # ( # (keys[0], values[0], keys[1], values[1], ..., # keys[len-1], values[len-1]), # next iff non-NULL> # ) def crack_bucket(b, is_mapping): state = b.__getstate__() assert isinstance(state, tuple) assert 1 <= len(state) <= 2 data = state[0] if not is_mapping: return data, [] keys = [] values = [] n = len(data) assert n & 1 == 0 i = 0 for x in data: if i & 1: values.append(x) else: keys.append(x) i += 1 return keys, values def type_and_adr(obj): if hasattr(obj, '_p_oid'): oid = oid_repr(obj._p_oid) else: oid = 'None' return "%s (0x%x oid=%s)" % (type(obj).__name__, positive_id(obj), oid) # Walker implements a depth-first search of a BTree (or TreeSet or Set or # Bucket). Subclasses must implement the visit_btree() and visit_bucket() # methods, and arrange to call the walk() method. walk() calls the # visit_XYZ() methods once for each node in the tree, in depth-first # left-to-right order. class Walker: def __init__(self, obj): self.obj = obj # obj is the BTree (BTree or TreeSet). # path is a list of indices, from the root. For example, if a BTree node # is child[5] of child[3] of the root BTree, [3, 5]. # parent is the parent BTree object, or None if this is the root BTree. # is_mapping is True for a BTree and False for a TreeSet. # keys is a list of the BTree's internal keys. # kids is a list of the BTree's children. # If the BTree is an empty root node, keys == kids == []. # Else len(kids) == len(keys) + 1. # lo and hi are slice bounds on the values the elements of keys *should* # lie in (lo inclusive, hi exclusive). lo is None if there is no lower # bound known, and hi is None if no upper bound is known. def visit_btree(self, obj, path, parent, is_mapping, keys, kids, lo, hi): raise NotImplementedError # obj is the bucket (Bucket or Set). # path is a list of indices, from the root. For example, if a bucket # node is child[5] of child[3] of the root BTree, [3, 5]. # parent is the parent BTree object. # is_mapping is True for a Bucket and False for a Set. # keys is a list of the bucket's keys. # values is a list of the bucket's values. # If is_mapping is false, values == []. Else len(keys) == len(values). # lo and hi are slice bounds on the values the elements of keys *should* # lie in (lo inclusive, hi exclusive). lo is None if there is no lower # bound known, and hi is None if no upper bound is known. def visit_bucket(self, obj, path, parent, is_mapping, keys, values, lo, hi): raise NotImplementedError def walk(self): obj = self.obj path = [] stack = [(obj, path, None, None, None)] while stack: obj, path, parent, lo, hi = stack.pop() kind, is_mapping = classify(obj) if kind is TYPE_BTREE: bkind, keys, kids = crack_btree(obj, is_mapping) if bkind is BTREE_NORMAL: # push the kids, in reverse order (so they're popped off # the stack in forward order) n = len(kids) for i in range(len(kids)-1, -1, -1): newlo, newhi = lo, hi if i < n-1: newhi = keys[i] if i > 0: newlo = keys[i-1] stack.append((kids[i], path + [i], obj, newlo, newhi)) elif bkind is BTREE_EMPTY: pass else: assert bkind is BTREE_ONE # Yuck. There isn't a bucket object to pass on, as # the bucket state is embedded directly in the BTree # state. Synthesize a bucket. assert kids is None # and "keys" is really the bucket # state bucket = _btree2bucket[type(obj)]() bucket.__setstate__(keys) stack.append((bucket, path + [0], obj, lo, hi)) keys = [] kids = [bucket] self.visit_btree(obj, path, parent, is_mapping, keys, kids, lo, hi) else: assert kind is TYPE_BUCKET keys, values = crack_bucket(obj, is_mapping) self.visit_bucket(obj, path, parent, is_mapping, keys, values, lo, hi) class Checker(Walker): def __init__(self, obj): Walker.__init__(self, obj) self.errors = [] def check(self): self.walk() if self.errors: s = "Errors found in %s:" % type_and_adr(self.obj) self.errors.insert(0, s) s = "\n".join(self.errors) raise AssertionError(s) def visit_btree(self, obj, path, parent, is_mapping, keys, kids, lo, hi): self.check_sorted(obj, path, keys, lo, hi) def visit_bucket(self, obj, path, parent, is_mapping, keys, values, lo, hi): self.check_sorted(obj, path, keys, lo, hi) def check_sorted(self, obj, path, keys, lo, hi): i, n = 0, len(keys) for x in keys: if lo is not None and not lo <= x: s = "key %r < lower bound %r at index %d" % (x, lo, i) self.complain(s, obj, path) if hi is not None and not x < hi: s = "key %r >= upper bound %r at index %d" % (x, hi, i) self.complain(s, obj, path) if i < n-1 and not x < keys[i+1]: s = "key %r at index %d >= key %r at index %d" % ( x, i, keys[i+1], i+1) self.complain(s, obj, path) i += 1 def complain(self, msg, obj, path): s = "%s, in %s, path from root %s" % ( msg, type_and_adr(obj), ".".join(map(str, path))) self.errors.append(s) class Printer(Walker): #pragma NO COVER def __init__(self, obj): Walker.__init__(self, obj) def display(self): self.walk() def visit_btree(self, obj, path, parent, is_mapping, keys, kids, lo, hi): indent = " " * len(path) print("%s%s %s with %d children" % ( indent, ".".join(map(str, path)), type_and_adr(obj), len(kids))) indent += " " n = len(keys) for i in range(n): print("%skey %d: %r" % (indent, i, keys[i])) def visit_bucket(self, obj, path, parent, is_mapping, keys, values, lo, hi): indent = " " * len(path) print("%s%s %s with %d keys" % ( indent, ".".join(map(str, path)), type_and_adr(obj), len(keys))) indent += " " n = len(keys) for i in range(n): print("%skey %d: %r" % (indent, i, keys[i]),) if is_mapping: print("value %r" % (values[i],)) def check(btree): """Check internal value-based invariants in a BTree or TreeSet. The btree._check() method checks internal C-level pointer consistency. The check() function here checks value-based invariants: whether the keys in leaf bucket and internal nodes are in strictly increasing order, and whether they all lie in their expected range. The latter is a subtle invariant that can't be checked locally -- it requires propagating range info down from the root of the tree, and modifying it at each level for each child. Raises AssertionError if anything is wrong, with a string detail explaining the problems. The entire tree is checked before AssertionError is raised, and the string detail may be large (depending on how much went wrong). """ Checker(btree).check() def display(btree): #pragma NO COVER "Display the internal structure of a BTree, Bucket, TreeSet or Set." Printer(btree).display() BTrees-4.3.1/BTrees/BTreeModuleTemplate.c0000664000175000017500000004717512704044565021271 0ustar tseavertseaver00000000000000/***************************************************************************** Copyright (c) 2001, 2002 Zope Foundation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #include "Python.h" /* include structmember.h for offsetof */ #include "structmember.h" #include "bytesobject.h" #ifdef PERSISTENT #include "persistent/cPersistence.h" #else #define PER_USE_OR_RETURN(self, NULL) #define PER_ALLOW_DEACTIVATION(self) #define PER_PREVENT_DEACTIVATION(self) #define PER_DEL(self) #define PER_USE(O) 1 #define PER_ACCESSED(O) 1 #endif #include "_compat.h" /* So sue me. This pair gets used all over the place, so much so that it * interferes with understanding non-persistence parts of algorithms. * PER_UNUSE can be used after a successul PER_USE or PER_USE_OR_RETURN. * It allows the object to become ghostified, and tells the persistence * machinery that the object's fields were used recently. */ #define PER_UNUSE(OBJ) do { \ PER_ALLOW_DEACTIVATION(OBJ); \ PER_ACCESSED(OBJ); \ } while (0) /* The tp_name slots of the various BTree types contain the fully * qualified names of the types, e.g. zodb.btrees.OOBTree.OOBTree. * The full name is usd to support pickling and because it is not * possible to modify the __module__ slot of a type dynamically. (This * may be a bug in Python 2.2). * * The MODULE_NAME here used to be "BTrees._". We actually want the module * name to point to the Python module rather than the C, so the underline * is now removed. */ #define MODULE_NAME "BTrees." MOD_NAME_PREFIX "BTree." static PyObject *sort_str, *reverse_str, *__setstate___str; static PyObject *_bucket_type_str, *max_internal_size_str, *max_leaf_size_str; static PyObject *ConflictError = NULL; static void PyVar_Assign(PyObject **v, PyObject *e) { Py_XDECREF(*v); *v=e;} #define ASSIGN(V,E) PyVar_Assign(&(V),(E)) #define UNLESS(E) if (!(E)) #define OBJECT(O) ((PyObject*)(O)) #define MIN_BUCKET_ALLOC 16 #define SameType_Check(O1, O2) (Py_TYPE((O1))==Py_TYPE((O2))) #define ASSERT(C, S, R) if (! (C)) { \ PyErr_SetString(PyExc_AssertionError, (S)); return (R); } #ifdef NEED_LONG_LONG_SUPPORT /* Helper code used to support long long instead of int. */ #ifndef PY_LONG_LONG #error "PY_LONG_LONG required but not defined" #endif #ifdef NEED_LONG_LONG_KEYS static int longlong_check(PyObject *ob) { if (INT_CHECK(ob)) return 1; if (PyLong_Check(ob)) { int overflow; (void)PyLong_AsLongLongAndOverflow(ob, &overflow); if (overflow) goto overflow; return 1; } return 0; overflow: PyErr_SetString(PyExc_ValueError, "longlong_check: long integer out of range"); return 0; } #endif static PyObject * longlong_as_object(PY_LONG_LONG val) { if ((val > LONG_MAX) || (val < LONG_MIN)) return PyLong_FromLongLong(val); return INT_FROM_LONG((long)val); } static int longlong_convert(PyObject *ob, PY_LONG_LONG *value) { #ifndef PY3K if (PyInt_Check(ob)) { (*value) = (PY_LONG_LONG)PyInt_AS_LONG(ob); return 1; } #endif if (!PyLong_Check(ob)) { PyErr_SetString(PyExc_TypeError, "expected integer key"); return 0; } else { PY_LONG_LONG val; int overflow; val = PyLong_AsLongLongAndOverflow(ob, &overflow); if (overflow) goto overflow; (*value) = val; return 1; } overflow: PyErr_SetString(PyExc_ValueError, "long integer out of range"); return 0; } #endif /* NEED_LONG_LONG_SUPPORT */ /* Various kinds of BTree and Bucket structs are instances of * "sized containers", and have a common initial layout: * The stuff needed for all Python objects, or all Persistent objects. * int size: The maximum number of things that could be contained * without growing the container. * int len: The number of things currently contained. * * Invariant: 0 <= len <= size. * * A sized container typically goes on to declare one or more pointers * to contiguous arrays with 'size' elements each, the initial 'len' of * which are currently in use. */ #ifdef PERSISTENT #define sizedcontainer_HEAD \ cPersistent_HEAD \ int size; \ int len; #else #define sizedcontainer_HEAD \ PyObject_HEAD \ int size; \ int len; #endif /* Nothing is actually of type Sized, but (pointers to) BTree nodes and * Buckets can be cast to Sized* in contexts that only need to examine * the members common to all sized containers. */ typedef struct Sized_s { sizedcontainer_HEAD } Sized; #define SIZED(O) ((Sized*)(O)) /* A Bucket wraps contiguous vectors of keys and values. Keys are unique, * and stored in sorted order. The 'values' pointer may be NULL if the * Bucket is used to implement a set. Buckets serving as leafs of BTrees * are chained together via 'next', so that the entire BTree contents * can be traversed in sorted order quickly and easily. */ typedef struct Bucket_s { sizedcontainer_HEAD struct Bucket_s *next; /* the bucket with the next-larger keys */ KEY_TYPE *keys; /* 'len' keys, in increasing order */ VALUE_TYPE *values; /* 'len' corresponding values; NULL if a set */ } Bucket; #define BUCKET(O) ((Bucket*)(O)) /* A BTree is complicated. See Maintainer.txt. */ typedef struct BTreeItem_s { KEY_TYPE key; Sized *child; /* points to another BTree, or to a Bucket of some sort */ } BTreeItem; typedef struct BTree_s { sizedcontainer_HEAD /* firstbucket points to the bucket containing the smallest key in * the BTree. This is found by traversing leftmost child pointers * (data[0].child) until reaching a Bucket. */ Bucket *firstbucket; /* The BTree points to 'len' children, via the "child" fields of the data * array. There are len-1 keys in the 'key' fields, stored in increasing * order. data[0].key is unused. For i in 0 .. len-1, all keys reachable * from data[i].child are >= data[i].key and < data[i+1].key, at the * endpoints pretending that data[0].key is minus infinity and * data[len].key is positive infinity. */ BTreeItem *data; long max_internal_size; long max_leaf_size; } BTree; static PyTypeObject BTreeType; static PyTypeObject BucketType; #define BTREE(O) ((BTree*)(O)) /* Use BTREE_SEARCH to find which child pointer to follow. * RESULT An int lvalue to hold the index i such that SELF->data[i].child * is the correct node to search next. * SELF A pointer to a BTree node. * KEY The key you're looking for, of type KEY_TYPE. * ONERROR What to do if key comparison raises an exception; for example, * perhaps 'return NULL'. * * See Maintainer.txt for discussion: this is optimized in subtle ways. * It's recommended that you call this at the start of a routine, waiting * to check for self->len == 0 after. */ #define BTREE_SEARCH(RESULT, SELF, KEY, ONERROR) { \ int _lo = 0; \ int _hi = (SELF)->len; \ int _i, _cmp; \ for (_i = _hi >> 1; _i > _lo; _i = (_lo + _hi) >> 1) { \ TEST_KEY_SET_OR(_cmp, (SELF)->data[_i].key, (KEY)) \ ONERROR; \ if (_cmp < 0) _lo = _i; \ else if (_cmp > 0) _hi = _i; \ else /* equal */ break; \ } \ (RESULT) = _i; \ } /* SetIteration structs are used in the internal set iteration protocol. * When you want to iterate over a set or bucket or BTree (even an * individual key!), * 1. Declare a new iterator: * SetIteration si = {0,0,0}; * Using "{0,0,0}" or "{0,0}" appear most common. Only one {0} is * necssary. At least one must be given so that finiSetIteration() works * correctly even if you don't get around to calling initSetIteration(). * 2. Initialize it via * initSetIteration(&si, PyObject *s, useValues) * It's an error if that returns an int < 0. In case of error on the * init call, calling finiSetIteration(&si) is optional. But if the * init call succeeds, you must eventually call finiSetIteration(), * and whether or not subsequent calls to si.next() fail. * 3. Get the first element: * if (si.next(&si) < 0) { there was an error } * If the set isn't empty, this sets si.position to an int >= 0, * si.key to the element's key (of type KEY_TYPE), and maybe si.value to * the element's value (of type VALUE_TYPE). si.value is defined * iff si.usesValue is true. * 4. Process all the elements: * while (si.position >= 0) { * do something with si.key and/or si.value; * if (si.next(&si) < 0) { there was an error; } * } * 5. Finalize the SetIterator: * finiSetIteration(&si); * This is mandatory! si may contain references to iterator objects, * keys and values, and they must be cleaned up else they'll leak. If * this were C++ we'd hide that in the destructor, but in C you have to * do it by hand. */ typedef struct SetIteration_s { PyObject *set; /* the set, bucket, BTree, ..., being iterated */ int position; /* initialized to 0; set to -1 by next() when done */ int usesValue; /* true iff 'set' has values & we iterate them */ KEY_TYPE key; /* next() sets to next key */ VALUE_TYPE value; /* next() may set to next value */ int (*next)(struct SetIteration_s*); /* function to get next key+value */ } SetIteration; /* Finish the set iteration protocol. This MUST be called by everyone * who starts a set iteration, unless the initial call to initSetIteration * failed; in that case, and only that case, calling finiSetIteration is * optional. */ static void finiSetIteration(SetIteration *i) { assert(i != NULL); if (i->set == NULL) return; Py_DECREF(i->set); i->set = NULL; /* so it doesn't hurt to call this again */ if (i->position > 0) { /* next() was called at least once, but didn't finish iterating * (else position would be negative). So the cached key and * value need to be cleaned up. */ DECREF_KEY(i->key); if (i->usesValue) { DECREF_VALUE(i->value); } } i->position = -1; /* stop any stray next calls from doing harm */ } static PyObject * IndexError(int i) { PyObject *v; v = INT_FROM_LONG(i); if (!v) { v = Py_None; Py_INCREF(v); } PyErr_SetObject(PyExc_IndexError, v); Py_DECREF(v); return NULL; } /* Search for the bucket immediately preceding *current, in the bucket chain * starting at first. current, *current and first must not be NULL. * * Return: * 1 *current holds the correct bucket; this is a borrowed reference * 0 no such bucket exists; *current unaltered * -1 error; *current unaltered */ static int PreviousBucket(Bucket **current, Bucket *first) { Bucket *trailing = NULL; /* first travels; trailing follows it */ int result = 0; assert(current && *current && first); if (first == *current) return 0; do { trailing = first; PER_USE_OR_RETURN(first, -1); first = first->next; ((trailing)->state==cPersistent_STICKY_STATE && ((trailing)->state=cPersistent_UPTODATE_STATE)); PER_ACCESSED(trailing); if (first == *current) { *current = trailing; result = 1; break; } } while (first); return result; } static void * BTree_Malloc(size_t sz) { void *r; ASSERT(sz > 0, "non-positive size malloc", NULL); r = malloc(sz); if (r) return r; PyErr_NoMemory(); return NULL; } static void * BTree_Realloc(void *p, size_t sz) { void *r; ASSERT(sz > 0, "non-positive size realloc", NULL); if (p) r = realloc(p, sz); else r = malloc(sz); UNLESS (r) PyErr_NoMemory(); return r; } /* Shared keyword-argument list for BTree/Bucket * (iter)?(keys|values|items) */ static char *search_keywords[] = {"min", "max", "excludemin", "excludemax", 0}; #include "BTreeItemsTemplate.c" #include "BucketTemplate.c" #include "SetTemplate.c" #include "BTreeTemplate.c" #include "TreeSetTemplate.c" #include "SetOpTemplate.c" #include "MergeTemplate.c" static struct PyMethodDef module_methods[] = { {"difference", (PyCFunction) difference_m, METH_VARARGS, "difference(o1, o2) -- " "compute the difference between o1 and o2" }, {"union", (PyCFunction) union_m, METH_VARARGS, "union(o1, o2) -- compute the union of o1 and o2\n" }, {"intersection", (PyCFunction) intersection_m, METH_VARARGS, "intersection(o1, o2) -- " "compute the intersection of o1 and o2" }, #ifdef MERGE {"weightedUnion", (PyCFunction) wunion_m, METH_VARARGS, "weightedUnion(o1, o2 [, w1, w2]) -- compute the union of o1 and o2\n" "\nw1 and w2 are weights." }, {"weightedIntersection", (PyCFunction) wintersection_m, METH_VARARGS, "weightedIntersection(o1, o2 [, w1, w2]) -- " "compute the intersection of o1 and o2\n" "\nw1 and w2 are weights." }, #endif #ifdef MULTI_INT_UNION {"multiunion", (PyCFunction) multiunion_m, METH_VARARGS, "multiunion(seq) -- compute union of a sequence of integer sets.\n" "\n" "Each element of seq must be an integer set, or convertible to one\n" "via the set iteration protocol. The union returned is an IISet." }, #endif {NULL, NULL} /* sentinel */ }; static char BTree_module_documentation[] = "\n" MASTER_ID BTREEITEMSTEMPLATE_C "$Id$\n" BTREETEMPLATE_C BUCKETTEMPLATE_C KEYMACROS_H MERGETEMPLATE_C SETOPTEMPLATE_C SETTEMPLATE_C TREESETTEMPLATE_C VALUEMACROS_H BTREEITEMSTEMPLATE_C ; int init_persist_type(PyTypeObject *type) { #ifdef PY3K ((PyObject*)type)->ob_type = &PyType_Type; #else type->ob_type = &PyType_Type; #endif type->tp_base = cPersistenceCAPI->pertype; if (PyType_Ready(type) < 0) return 0; return 1; } #ifdef PY3K static struct PyModuleDef moduledef = { PyModuleDef_HEAD_INIT, "_" MOD_NAME_PREFIX "BTree", /* m_name */ BTree_module_documentation, /* m_doc */ -1, /* m_size */ module_methods, /* m_methods */ NULL, /* m_reload */ NULL, /* m_traverse */ NULL, /* m_clear */ NULL, /* m_free */ }; #endif static PyObject* module_init(void) { PyObject *module, *mod_dict, *interfaces, *conflicterr; #ifdef KEY_TYPE_IS_PYOBJECT object_ = PyTuple_GetItem(Py_TYPE(Py_None)->tp_bases, 0); if (object_ == NULL) return NULL; #endif sort_str = INTERN("sort"); if (!sort_str) return NULL; reverse_str = INTERN("reverse"); if (!reverse_str) return NULL; __setstate___str = INTERN("__setstate__"); if (!__setstate___str) return NULL; _bucket_type_str = INTERN("_bucket_type"); if (!_bucket_type_str) return NULL; max_internal_size_str = INTERN("max_internal_size"); if (! max_internal_size_str) return NULL; max_leaf_size_str = INTERN("max_leaf_size"); if (! max_leaf_size_str) return NULL; /* Grab the ConflictError class */ interfaces = PyImport_ImportModule("BTrees.Interfaces"); if (interfaces != NULL) { conflicterr = PyObject_GetAttrString(interfaces, "BTreesConflictError"); if (conflicterr != NULL) ConflictError = conflicterr; Py_DECREF(interfaces); } if (ConflictError == NULL) { Py_INCREF(PyExc_ValueError); ConflictError=PyExc_ValueError; } /* Initialize the PyPersist_C_API and the type objects. */ #ifdef PY3K cPersistenceCAPI = (cPersistenceCAPIstruct *)PyCapsule_Import( "persistent.cPersistence.CAPI", 0); #else cPersistenceCAPI = (cPersistenceCAPIstruct *)PyCObject_Import( "persistent.cPersistence", "CAPI"); #endif if (cPersistenceCAPI == NULL) return NULL; #ifdef PY3K #define _SET_TYPE(typ) ((PyObject*)(&typ))->ob_type = &PyType_Type #else #define _SET_TYPE(typ) (typ).ob_type = &PyType_Type #endif _SET_TYPE(BTreeItemsType); _SET_TYPE(BTreeIter_Type); BTreeIter_Type.tp_getattro = PyObject_GenericGetAttr; BucketType.tp_new = PyType_GenericNew; SetType.tp_new = PyType_GenericNew; BTreeType.tp_new = PyType_GenericNew; TreeSetType.tp_new = PyType_GenericNew; if (!init_persist_type(&BucketType)) return NULL; if (!init_persist_type(&BTreeType)) return NULL; if (!init_persist_type(&SetType)) return NULL; if (!init_persist_type(&TreeSetType)) return NULL; if (PyDict_SetItem(BTreeType.tp_dict, _bucket_type_str, (PyObject *)&BucketType) < 0) { fprintf(stderr, "btree failed\n"); return NULL; } if (PyDict_SetItem(TreeSetType.tp_dict, _bucket_type_str, (PyObject *)&SetType) < 0) { fprintf(stderr, "bucket failed\n"); return NULL; } /* Create the module and add the functions */ #ifdef PY3K module = PyModule_Create(&moduledef); #else module = Py_InitModule4("_" MOD_NAME_PREFIX "BTree", module_methods, BTree_module_documentation, (PyObject *)NULL, PYTHON_API_VERSION); #endif /* Add some symbolic constants to the module */ mod_dict = PyModule_GetDict(module); if (PyDict_SetItemString(mod_dict, MOD_NAME_PREFIX "Bucket", (PyObject *)&BucketType) < 0) return NULL; if (PyDict_SetItemString(mod_dict, MOD_NAME_PREFIX "BTree", (PyObject *)&BTreeType) < 0) return NULL; if (PyDict_SetItemString(mod_dict, MOD_NAME_PREFIX "Set", (PyObject *)&SetType) < 0) return NULL; if (PyDict_SetItemString(mod_dict, MOD_NAME_PREFIX "TreeSet", (PyObject *)&TreeSetType) < 0) return NULL; if (PyDict_SetItemString(mod_dict, MOD_NAME_PREFIX "TreeIterator", (PyObject *)&BTreeIter_Type) < 0) return NULL; /* We also want to be able to access these constants without the prefix * so that code can more easily exchange modules (particularly the integer * and long modules, but also others). The TreeIterator is only internal, * so we don't bother to expose that. */ if (PyDict_SetItemString(mod_dict, "Bucket", (PyObject *)&BucketType) < 0) return NULL; if (PyDict_SetItemString(mod_dict, "BTree", (PyObject *)&BTreeType) < 0) return NULL; if (PyDict_SetItemString(mod_dict, "Set", (PyObject *)&SetType) < 0) return NULL; if (PyDict_SetItemString(mod_dict, "TreeSet", (PyObject *)&TreeSetType) < 0) return NULL; #if defined(ZODB_64BIT_INTS) && defined(NEED_LONG_LONG_SUPPORT) if (PyDict_SetItemString(mod_dict, "using64bits", Py_True) < 0) return NULL; #else if (PyDict_SetItemString(mod_dict, "using64bits", Py_False) < 0) return NULL; #endif return module; } #ifdef PY3K PyMODINIT_FUNC INITMODULE(void) { return module_init(); } #else PyMODINIT_FUNC INITMODULE(void) { module_init(); } #endif BTrees-4.3.1/BTrees/IIBTree.py0000664000175000017500000001027612665316015017045 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __all__ = ('Bucket', 'Set', 'BTree', 'TreeSet', 'IIBucket', 'IISet', 'IIBTree', 'IITreeSet', 'union', 'intersection', 'difference', 'weightedUnion', 'weightedIntersection', 'multiunion', ) from zope.interface import moduleProvides from .Interfaces import IIntegerIntegerBTreeModule from ._base import Bucket from ._base import MERGE from ._base import MERGE_WEIGHT_numeric from ._base import MERGE_DEFAULT_int from ._base import Set from ._base import Tree as BTree from ._base import TreeSet from ._base import _TreeIterator from ._base import difference as _difference from ._base import intersection as _intersection from ._base import multiunion as _multiunion from ._base import set_operation as _set_operation from ._base import to_int as _to_key from ._base import to_int as _to_value from ._base import union as _union from ._base import weightedIntersection as _weightedIntersection from ._base import weightedUnion as _weightedUnion from ._base import _fix_pickle _BUCKET_SIZE = 120 _TREE_SIZE = 500 using64bits = False class IIBucketPy(Bucket): _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class IISetPy(Set): _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class IIBTreePy(BTree): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class IITreeSetPy(TreeSet): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class IITreeIteratorPy(_TreeIterator): pass # Can't declare forward refs, so fix up afterwards: IIBucketPy._mapping_type = IIBucketPy._bucket_type = IIBucketPy IIBucketPy._set_type = IISetPy IISetPy._mapping_type = IIBucketPy IISetPy._set_type = IISetPy._bucket_type = IISetPy IIBTreePy._mapping_type = IIBTreePy._bucket_type = IIBucketPy IIBTreePy._set_type = IISetPy IITreeSetPy._mapping_type = IIBucketPy IITreeSetPy._set_type = IITreeSetPy._bucket_type = IISetPy differencePy = _set_operation(_difference, IISetPy) unionPy = _set_operation(_union, IISetPy) intersectionPy = _set_operation(_intersection, IISetPy) multiunionPy = _set_operation(_multiunion, IISetPy) weightedUnionPy = _set_operation(_weightedUnion, IISetPy) weightedIntersectionPy = _set_operation(_weightedIntersection, IISetPy) try: from ._IIBTree import IIBucket except ImportError: #pragma NO COVER w/ C extensions IIBucket = IIBucketPy IISet = IISetPy IIBTree = IIBTreePy IITreeSet = IITreeSetPy IITreeIterator = IITreeIteratorPy difference = differencePy union = unionPy intersection = intersectionPy multiunion = multiunionPy weightedUnion = weightedUnionPy weightedIntersection = weightedIntersectionPy else: #pragma NO COVER w/o C extensions from ._IIBTree import IISet from ._IIBTree import IIBTree from ._IIBTree import IITreeSet from ._IIBTree import IITreeIterator from ._IIBTree import difference from ._IIBTree import union from ._IIBTree import intersection from ._IIBTree import multiunion from ._IIBTree import weightedUnion from ._IIBTree import weightedIntersection Bucket = IIBucket Set = IISet BTree = IIBTree TreeSet = IITreeSet _fix_pickle(globals(), __name__) moduleProvides(IIntegerIntegerBTreeModule) BTrees-4.3.1/BTrees/BTreeTemplate.c0000664000175000017500000021127012576063313020107 0ustar tseavertseaver00000000000000/***************************************************************************** Copyright (c) 2001, 2002 Zope Foundation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #include "_compat.h" #define BTREETEMPLATE_C "$Id$\n" static long _get_max_size(BTree *self, PyObject *name, long default_max) { PyObject *size; long isize; size = PyObject_GetAttr(OBJECT(OBJECT(self)->ob_type), name); if (size == NULL) { PyErr_Clear(); return default_max; } #ifdef PY3K isize = PyLong_AsLong(size); #else isize = PyInt_AsLong(size); #endif Py_DECREF(size); if (isize <= 0 && ! PyErr_Occurred()) { PyErr_SetString(PyExc_ValueError, "non-positive max size in BTree subclass"); return -1; } return isize; } static int _max_internal_size(BTree *self) { long isize; if (self->max_internal_size > 0) return self->max_internal_size; isize = _get_max_size(self, max_internal_size_str, DEFAULT_MAX_BTREE_SIZE); self->max_internal_size = isize; return isize; } static int _max_leaf_size(BTree *self) { long isize; if (self->max_leaf_size > 0) return self->max_leaf_size; isize = _get_max_size(self, max_leaf_size_str, DEFAULT_MAX_BUCKET_SIZE); self->max_leaf_size = isize; return isize; } /* Sanity-check a BTree. This is a private helper for BTree_check. Return: * -1 Error. If it's an internal inconsistency in the BTree, * AssertionError is set. * 0 No problem found. * * nextbucket is the bucket "one beyond the end" of the BTree; the last bucket * directly reachable from following right child pointers *should* be linked * to nextbucket (and this is checked). */ static int BTree_check_inner(BTree *self, Bucket *nextbucket) { int i; Bucket *bucketafter; Sized *child; char *errormsg = "internal error"; /* someone should have overriden */ Sized *activated_child = NULL; int result = -1; /* until proved innocent */ #define CHECK(CONDITION, ERRORMSG) \ if (!(CONDITION)) { \ errormsg = (ERRORMSG); \ goto Error; \ } PER_USE_OR_RETURN(self, -1); CHECK(self->len >= 0, "BTree len < 0"); CHECK(self->len <= self->size, "BTree len > size"); if (self->len == 0) /* Empty BTree. */ { CHECK(self->firstbucket == NULL, "Empty BTree has non-NULL firstbucket"); result = 0; goto Done; } /* Non-empty BTree. */ CHECK(self->firstbucket != NULL, "Non-empty BTree has NULL firstbucket"); /* Obscure: The first bucket is pointed to at least by self->firstbucket * and data[0].child of whichever BTree node it's a child of. However, * if persistence is enabled then the latter BTree node may be a ghost * at this point, and so its pointers "don't count": we can only rely * on self's pointers being intact. */ #ifdef PERSISTENT CHECK(Py_REFCNT(self->firstbucket) >= 1, "Non-empty BTree firstbucket has refcount < 1"); #else CHECK(Py_REFCNT(self->firstbucket) >= 2, "Non-empty BTree firstbucket has refcount < 2"); #endif for (i = 0; i < self->len; ++i) { CHECK(self->data[i].child != NULL, "BTree has NULL child"); } if (SameType_Check(self, self->data[0].child)) { /* Our children are also BTrees. */ child = self->data[0].child; UNLESS (PER_USE(child)) goto Done; activated_child = child; CHECK(self->firstbucket == BTREE(child)->firstbucket, "BTree has firstbucket different than " "its first child's firstbucket"); PER_ALLOW_DEACTIVATION(child); activated_child = NULL; for (i = 0; i < self->len; ++i) { child = self->data[i].child; CHECK(SameType_Check(self, child), "BTree children have different types"); if (i == self->len - 1) bucketafter = nextbucket; else { BTree *child2 = BTREE(self->data[i+1].child); UNLESS (PER_USE(child2)) goto Done; bucketafter = child2->firstbucket; PER_ALLOW_DEACTIVATION(child2); } if (BTree_check_inner(BTREE(child), bucketafter) < 0) goto Done; } } else /* Our children are buckets. */ { CHECK(self->firstbucket == BUCKET(self->data[0].child), "Bottom-level BTree node has inconsistent firstbucket belief"); for (i = 0; i < self->len; ++i) { child = self->data[i].child; UNLESS (PER_USE(child)) goto Done; activated_child = child; CHECK(!SameType_Check(self, child), "BTree children have different types"); CHECK(child->len >= 1, "Bucket length < 1");/* no empty buckets! */ CHECK(child->len <= child->size, "Bucket len > size"); #ifdef PERSISTENT CHECK(Py_REFCNT(child) >= 1, "Bucket has refcount < 1"); #else CHECK(Py_REFCNT(child) >= 2, "Bucket has refcount < 2"); #endif if (i == self->len - 1) bucketafter = nextbucket; else bucketafter = BUCKET(self->data[i+1].child); CHECK(BUCKET(child)->next == bucketafter, "Bucket next pointer is damaged"); PER_ALLOW_DEACTIVATION(child); activated_child = NULL; } } result = 0; goto Done; Error: PyErr_SetString(PyExc_AssertionError, errormsg); result = -1; Done: /* No point updating access time -- this isn't a "real" use. */ PER_ALLOW_DEACTIVATION(self); if (activated_child) { PER_ALLOW_DEACTIVATION(activated_child); } return result; #undef CHECK } /* Sanity-check a BTree. This is the ._check() method. Return: * NULL Error. If it's an internal inconsistency in the BTree, * AssertionError is set. * Py_None No problem found. */ static PyObject* BTree_check(BTree *self) { PyObject *result = NULL; int i = BTree_check_inner(self, NULL); if (i >= 0) { result = Py_None; Py_INCREF(result); } return result; } /* ** _BTree_get ** ** Search a BTree. ** ** Arguments ** self a pointer to a BTree ** keyarg the key to search for, as a Python object ** has_key true/false; when false, try to return the associated ** value; when true, return a boolean ** Return ** When has_key false: ** If key exists, its associated value. ** If key doesn't exist, NULL and KeyError is set. ** When has_key true: ** A Python int is returned in any case. ** If key exists, the depth of the bucket in which it was found. ** If key doesn't exist, 0. */ static PyObject * _BTree_get(BTree *self, PyObject *keyarg, int has_key) { KEY_TYPE key; PyObject *result = NULL; /* guilty until proved innocent */ int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); UNLESS (copied) return NULL; PER_USE_OR_RETURN(self, NULL); if (self->len == 0) { /* empty BTree */ if (has_key) result = INT_FROM_LONG(0); else PyErr_SetObject(PyExc_KeyError, keyarg); } else { for (;;) { int i; Sized *child; BTREE_SEARCH(i, self, key, goto Done); child = self->data[i].child; has_key += has_key != 0; /* bump depth counter, maybe */ if (SameType_Check(self, child)) { PER_UNUSE(self); self = BTREE(child); PER_USE_OR_RETURN(self, NULL); } else { result = _bucket_get(BUCKET(child), keyarg, has_key); break; } } } Done: PER_UNUSE(self); return result; } static PyObject * BTree_get(BTree *self, PyObject *key) { return _BTree_get(self, key, 0); } /* Create a new bucket for the BTree or TreeSet using the class attribute _bucket_type, which is normally initialized to BucketType or SetType as appropriate. */ static Sized * BTree_newBucket(BTree *self) { PyObject *factory; Sized *result; /* _bucket_type_str defined in BTreeModuleTemplate.c */ factory = PyObject_GetAttr((PyObject *)Py_TYPE(self), _bucket_type_str); if (factory == NULL) return NULL; /* TODO: Should we check that the factory actually returns something of the appropriate type? How? The C code here is going to depend on any custom bucket type having the same layout at the C level. */ result = SIZED(PyObject_CallObject(factory, NULL)); Py_DECREF(factory); return result; } /* * Move data from the current BTree, from index onward, to the newly created * BTree 'next'. self and next must both be activated. If index is OOB (< 0 * or >= self->len), use self->len / 2 as the index (i.e., split at the * midpoint). self must have at least 2 children on entry, and index must * be such that self and next each have at least one child at exit. self's * accessed time is updated. * * Return: * -1 error * 0 OK */ static int BTree_split(BTree *self, int index, BTree *next) { int next_size; Sized *child; if (index < 0 || index >= self->len) index = self->len / 2; next_size = self->len - index; ASSERT(index > 0, "split creates empty tree", -1); ASSERT(next_size > 0, "split creates empty tree", -1); next->data = BTree_Malloc(sizeof(BTreeItem) * next_size); if (!next->data) return -1; memcpy(next->data, self->data + index, sizeof(BTreeItem) * next_size); next->size = next_size; /* but don't set len until we succeed */ /* Set next's firstbucket. self->firstbucket is still correct. */ child = next->data[0].child; if (SameType_Check(self, child)) { PER_USE_OR_RETURN(child, -1); next->firstbucket = BTREE(child)->firstbucket; PER_UNUSE(child); } else next->firstbucket = BUCKET(child); Py_INCREF(next->firstbucket); next->len = next_size; self->len = index; return PER_CHANGED(self) >= 0 ? 0 : -1; } /* Fwd decl -- BTree_grow and BTree_split_root reference each other. */ static int BTree_grow(BTree *self, int index, int noval); /* Split the root. This is a little special because the root isn't a child * of anything else, and the root needs to retain its object identity. So * this routine moves the root's data into a new child, and splits the * latter. This leaves the root with two children. * * Return: * 0 OK * -1 error * * CAUTION: The caller must call PER_CHANGED on self. */ static int BTree_split_root(BTree *self, int noval) { BTree *child; BTreeItem *d; /* Create a child BTree, and a new data vector for self. */ child = BTREE(PyObject_CallObject(OBJECT(Py_TYPE(self)), NULL)); if (!child) return -1; d = BTree_Malloc(sizeof(BTreeItem) * 2); if (!d) { Py_DECREF(child); return -1; } /* Move our data to new BTree. */ child->size = self->size; child->len = self->len; child->data = self->data; child->firstbucket = self->firstbucket; Py_INCREF(child->firstbucket); /* Point self to child and split the child. */ self->data = d; self->len = 1; self->size = 2; self->data[0].child = SIZED(child); /* transfers reference ownership */ return BTree_grow(self, 0, noval); } /* ** BTree_grow ** ** Grow a BTree ** ** Arguments: self The BTree ** index self->data[index].child needs to be split. index ** must be 0 if self is empty (len == 0), and a new ** empty bucket is created then. ** noval Boolean; is this a set (true) or mapping (false)? ** ** Returns: 0 on success ** -1 on failure ** ** CAUTION: If self is empty on entry, this routine adds an empty bucket. ** That isn't a legitimate BTree; if the caller doesn't put something in ** in the bucket (say, because of a later error), the BTree must be cleared ** to get rid of the empty bucket. */ static int BTree_grow(BTree *self, int index, int noval) { int i; Sized *v, *e = 0; BTreeItem *d; if (self->len == self->size) { if (self->size) { d = BTree_Realloc(self->data, sizeof(BTreeItem) * self->size * 2); if (d == NULL) return -1; self->data = d; self->size *= 2; } else { d = BTree_Malloc(sizeof(BTreeItem) * 2); if (d == NULL) return -1; self->data = d; self->size = 2; } } if (self->len) { long max_size = _max_internal_size(self); if (max_size < 0) return -1; d = self->data + index; v = d->child; /* Create a new object of the same type as the target value */ e = (Sized *)PyObject_CallObject((PyObject *)Py_TYPE(v), NULL); if (e == NULL) return -1; UNLESS(PER_USE(v)) { Py_DECREF(e); return -1; } /* Now split between the original (v) and the new (e) at the midpoint*/ if (SameType_Check(self, v)) i = BTree_split((BTree *)v, -1, (BTree *)e); else i = bucket_split((Bucket *)v, -1, (Bucket *)e); PER_ALLOW_DEACTIVATION(v); if (i < 0) { Py_DECREF(e); assert(PyErr_Occurred()); return -1; } index++; d++; if (self->len > index) /* Shift up the old values one array slot */ memmove(d+1, d, sizeof(BTreeItem)*(self->len-index)); if (SameType_Check(self, v)) { COPY_KEY(d->key, BTREE(e)->data->key); /* We take the unused reference from e, so there's no reason to INCREF! */ /* INCREF_KEY(self->data[1].key); */ } else { COPY_KEY(d->key, BUCKET(e)->keys[0]); INCREF_KEY(d->key); } d->child = e; self->len++; if (self->len >= max_size * 2) /* the root is huge */ return BTree_split_root(self, noval); } else { /* The BTree is empty. Create an empty bucket. See CAUTION in * the comments preceding. */ assert(index == 0); d = self->data; d->child = BTree_newBucket(self); if (d->child == NULL) return -1; self->len = 1; Py_INCREF(d->child); self->firstbucket = (Bucket *)d->child; } return 0; } /* Return the rightmost bucket reachable from following child pointers * from self. The caller gets a new reference to this bucket. Note that * bucket 'next' pointers are not followed: if self is an interior node * of a BTree, this returns the rightmost bucket in that node's subtree. * In case of error, returns NULL. * * self must not be a ghost; this isn't checked. The result may be a ghost. * * Pragmatics: Note that the rightmost bucket's last key is the largest * key in self's subtree. */ static Bucket * BTree_lastBucket(BTree *self) { Sized *pchild; Bucket *result; UNLESS (self->data && self->len) { IndexError(-1); /* is this the best action to take? */ return NULL; } pchild = self->data[self->len - 1].child; if (SameType_Check(self, pchild)) { self = BTREE(pchild); PER_USE_OR_RETURN(self, NULL); result = BTree_lastBucket(self); PER_UNUSE(self); } else { Py_INCREF(pchild); result = BUCKET(pchild); } return result; } static int BTree_deleteNextBucket(BTree *self) { Bucket *b; UNLESS (PER_USE(self)) return -1; b = BTree_lastBucket(self); if (b == NULL) goto err; if (Bucket_deleteNextBucket(b) < 0) goto err; Py_DECREF(b); PER_UNUSE(self); return 0; err: Py_XDECREF(b); PER_ALLOW_DEACTIVATION(self); return -1; } /* ** _BTree_clear ** ** Clears out all of the values in the BTree (firstbucket, keys, and children); ** leaving self an empty BTree. ** ** Arguments: self The BTree ** ** Returns: 0 on success ** -1 on failure ** ** Internal: Deallocation order is important. The danger is that a long ** list of buckets may get freed "at once" via decref'ing the first bucket, ** in which case a chain of consequenct Py_DECREF calls may blow the stack. ** Luckily, every bucket has a refcount of at least two, one due to being a ** BTree node's child, and another either because it's not the first bucket in ** the chain (so the preceding bucket points to it), or because firstbucket ** points to it. By clearing in the natural depth-first, left-to-right ** order, the BTree->bucket child pointers prevent Py_DECREF(bucket->next) ** calls from freeing bucket->next, and the maximum stack depth is equal ** to the height of the tree. **/ static int _BTree_clear(BTree *self) { const int len = self->len; if (self->firstbucket) { /* Obscure: The first bucket is pointed to at least by * self->firstbucket and data[0].child of whichever BTree node it's * a child of. However, if persistence is enabled then the latter * BTree node may be a ghost at this point, and so its pointers "don't * count": we can only rely on self's pointers being intact. */ #ifdef PERSISTENT ASSERT(Py_REFCNT(self->firstbucket) > 0, "Invalid firstbucket pointer", -1); #else ASSERT(Py_REFCNT(self->firstbucket) > 1, "Invalid firstbucket pointer", -1); #endif Py_DECREF(self->firstbucket); self->firstbucket = NULL; } if (self->data) { int i; if (len > 0) /* 0 is special because key 0 is trash */ { Py_DECREF(self->data[0].child); } for (i = 1; i < len; i++) { #ifdef KEY_TYPE_IS_PYOBJECT DECREF_KEY(self->data[i].key); #endif Py_DECREF(self->data[i].child); } free(self->data); self->data = NULL; } self->len = self->size = 0; return 0; } /* Set (value != 0) or delete (value=0) a tree item. If unique is non-zero, then only change if the key is new. If noval is non-zero, then don't set a value (the tree is a set). Return: -1 error 0 successful, and number of entries didn't change >0 successful, and number of entries did change Internal There are two distinct return values > 0: 1 Successful, number of entries changed, but firstbucket did not go away. 2 Successful, number of entries changed, firstbucket did go away. This can only happen on a delete (value == NULL). The caller may need to change its own firstbucket pointer, and in any case *someone* needs to adjust the 'next' pointer of the bucket immediately preceding the bucket that went away (it needs to point to the bucket immediately following the bucket that went away). */ static int _BTree_set(BTree *self, PyObject *keyarg, PyObject *value, int unique, int noval) { int changed = 0; /* did I mutate? */ int min; /* index of child I searched */ BTreeItem *d; /* self->data[min] */ int childlength; /* len(self->data[min].child) */ int status; /* our return value; and return value from callee */ int self_was_empty; /* was self empty at entry? */ KEY_TYPE key; int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); if (!copied) return -1; PER_USE_OR_RETURN(self, -1); self_was_empty = self->len == 0; if (self_was_empty) { /* We're empty. Make room. */ if (value) { if (BTree_grow(self, 0, noval) < 0) goto Error; } else { /* Can't delete a key from an empty BTree. */ PyErr_SetObject(PyExc_KeyError, keyarg); goto Error; } } /* Find the right child to search, and hand the work off to it. */ BTREE_SEARCH(min, self, key, goto Error); d = self->data + min; #ifdef PERSISTENT PER_READCURRENT(self, goto Error); #endif if (SameType_Check(self, d->child)) status = _BTree_set(BTREE(d->child), keyarg, value, unique, noval); else { int bucket_changed = 0; status = _bucket_set(BUCKET(d->child), keyarg, value, unique, noval, &bucket_changed); #ifdef PERSISTENT /* If a BTree contains only a single bucket, BTree.__getstate__() * includes the bucket's entire state, and the bucket doesn't get * an oid of its own. So if we have a single oid-less bucket that * changed, it's *our* oid that should be marked as changed -- the * bucket doesn't have one. */ if (bucket_changed && self->len == 1 && self->data[0].child->oid == NULL) { changed = 1; } #endif } if (status == 0) goto Done; if (status < 0) goto Error; assert(status == 1 || status == 2); /* The child changed size. Get its new size. Note that since the tree * rooted at the child changed size, so did the tree rooted at self: * our status must be >= 1 too. */ UNLESS(PER_USE(d->child)) goto Error; childlength = d->child->len; PER_UNUSE(d->child); if (value) { /* A bucket got bigger -- if it's "too big", split it. */ int toobig; assert(status == 1); /* can be 2 only on deletes */ if (SameType_Check(self, d->child)) { long max_size = _max_internal_size(self); if (max_size < 0) return -1; toobig = childlength > max_size; } else { long max_size = _max_leaf_size(self); if (max_size < 0) return -1; toobig = childlength > max_size; } if (toobig) { if (BTree_grow(self, min, noval) < 0) goto Error; changed = 1; /* BTree_grow mutated self */ } goto Done; /* and status still == 1 */ } /* A bucket got smaller. This is much harder, and despite that we * don't try to rebalance the tree. */ if (min && childlength) { /* We removed a key. but the node child is non-empty. If the deleted key is the node key, then update the node key using the smallest key of the node child. This doesn't apply to the 0th node, whos key is unused. */ int _cmp = 1; TEST_KEY_SET_OR(_cmp, key, d->key) goto Error; if (_cmp == 0) /* Need to replace key with first key from child */ { Bucket *bucket; if (SameType_Check(self, d->child)) { UNLESS(PER_USE(d->child)) goto Error; bucket = BTREE(d->child)->firstbucket; PER_UNUSE(d->child); } else bucket = BUCKET(d->child); UNLESS(PER_USE(bucket)) goto Error; DECREF_KEY(d->key); COPY_KEY(d->key, bucket->keys[0]); INCREF_KEY(d->key); PER_UNUSE(bucket); if (PER_CHANGED(self) < 0) goto Error; } } if (status == 2) { /* The child must be a BTree because bucket.set never returns 2 */ /* Two problems to solve: May have to adjust our own firstbucket, * and the bucket that went away needs to get unlinked. */ if (min) { /* This wasn't our firstbucket, so no need to adjust ours (note * that it can't be the firstbucket of any node above us either). * Tell "the tree to the left" to do the unlinking. */ if (BTree_deleteNextBucket(BTREE(d[-1].child)) < 0) goto Error; status = 1; /* we solved the child's firstbucket problem */ } else { /* This was our firstbucket. Update to new firstbucket value. */ Bucket *nextbucket; UNLESS(PER_USE(d->child)) goto Error; nextbucket = BTREE(d->child)->firstbucket; PER_UNUSE(d->child); Py_XINCREF(nextbucket); Py_DECREF(self->firstbucket); self->firstbucket = nextbucket; changed = 1; /* The caller has to do the unlinking -- we can't. Also, since * it was our firstbucket, it may also be theirs. */ assert(status == 2); } } /* If the child isn't empty, we're done! We did all that was possible for * us to do with the firstbucket problems the child gave us, and since the * child isn't empty don't create any new firstbucket problems of our own. */ if (childlength) goto Done; /* The child became empty: we need to remove it from self->data. * But first, if we're a bottom-level node, we've got more bucket-fiddling * to set up. */ if (! SameType_Check(self, d->child)) { /* We're about to delete a bucket, so need to adjust bucket pointers. */ if (min) { /* It's not our first bucket, so we can tell the previous * bucket to adjust its reference to it. It can't be anyone * else's first bucket either, so the caller needn't do anything. */ if (Bucket_deleteNextBucket(BUCKET(d[-1].child)) < 0) goto Error; /* status should be 1, and already is: if it were 2, the * block above would have set it to 1 in its min != 0 branch. */ assert(status == 1); } else { Bucket *nextbucket; /* It's our first bucket. We can't unlink it directly. */ /* 'changed' will be set true by the deletion code following. */ UNLESS(PER_USE(d->child)) goto Error; nextbucket = BUCKET(d->child)->next; PER_UNUSE(d->child); Py_XINCREF(nextbucket); Py_DECREF(self->firstbucket); self->firstbucket = nextbucket; status = 2; /* we're giving our caller a new firstbucket problem */ } } /* Remove the child from self->data. */ Py_DECREF(d->child); #ifdef KEY_TYPE_IS_PYOBJECT if (min) { DECREF_KEY(d->key); } else if (self->len > 1) { /* We're deleting the first child of a BTree with more than one * child. The key at d+1 is about to be shifted into slot 0, * and hence never to be referenced again (the key in slot 0 is * trash). */ DECREF_KEY((d+1)->key); } /* Else min==0 and len==1: we're emptying the BTree entirely, and * there is no key in need of decrefing. */ #endif --self->len; if (min < self->len) memmove(d, d+1, (self->len - min) * sizeof(BTreeItem)); changed = 1; Done: #ifdef PERSISTENT if (changed) { if (PER_CHANGED(self) < 0) goto Error; } #endif PER_UNUSE(self); return status; Error: assert(PyErr_Occurred()); if (self_was_empty) { /* BTree_grow may have left the BTree in an invalid state. Make * sure the tree is a legitimate empty tree. */ _BTree_clear(self); } PER_UNUSE(self); return -1; } /* ** BTree_setitem ** ** wrapper for _BTree_set ** ** Arguments: self The BTree ** key The key to insert ** v The value to insert ** ** Returns -1 on failure ** 0 on success */ static int BTree_setitem(BTree *self, PyObject *key, PyObject *v) { if (_BTree_set(self, key, v, 0, 0) < 0) return -1; return 0; } #ifdef PERSISTENT static PyObject * BTree__p_deactivate(BTree *self, PyObject *args, PyObject *keywords) { int ghostify = 1; PyObject *force = NULL; if (args && PyTuple_GET_SIZE(args) > 0) { PyErr_SetString(PyExc_TypeError, "_p_deactivate takes not positional arguments"); return NULL; } if (keywords) { int size = PyDict_Size(keywords); force = PyDict_GetItemString(keywords, "force"); if (force) size--; if (size) { PyErr_SetString(PyExc_TypeError, "_p_deactivate only accepts keyword arg force"); return NULL; } } if (self->jar && self->oid) { ghostify = self->state == cPersistent_UPTODATE_STATE; if (!ghostify && force) { if (PyObject_IsTrue(force)) ghostify = 1; if (PyErr_Occurred()) return NULL; } if (ghostify) { if (_BTree_clear(self) < 0) return NULL; PER_GHOSTIFY(self); } } Py_INCREF(Py_None); return Py_None; } #endif static PyObject * BTree_clear(BTree *self) { UNLESS (PER_USE(self)) return NULL; if (self->len) { if (_BTree_clear(self) < 0) goto err; if (PER_CHANGED(self) < 0) goto err; } PER_UNUSE(self); Py_INCREF(Py_None); return Py_None; err: PER_UNUSE(self); return NULL; } /* * Return: * * For an empty BTree (self->len == 0), None. * * For a BTree with one child (self->len == 1), and that child is a bucket, * and that bucket has a NULL oid, a one-tuple containing a one-tuple * containing the bucket's state: * * ( * ( * child[0].__getstate__(), * ), * ) * * Else a two-tuple. The first element is a tuple interleaving the BTree's * keys and direct children, of size 2*self->len - 1 (key[0] is unused and * is not saved). The second element is the firstbucket: * * ( * (child[0], key[1], child[1], key[2], child[2], ..., * key[len-1], child[len-1]), * self->firstbucket * ) * * In the above, key[i] means self->data[i].key, and similarly for child[i]. */ static PyObject * BTree_getstate(BTree *self) { PyObject *r = NULL; PyObject *o; int i, l; UNLESS (PER_USE(self)) return NULL; if (self->len) { r = PyTuple_New(self->len * 2 - 1); if (r == NULL) goto err; if (self->len == 1 && Py_TYPE(self->data->child) != Py_TYPE(self) #ifdef PERSISTENT && BUCKET(self->data->child)->oid == NULL #endif ) { /* We have just one bucket. Save its data directly. */ o = bucket_getstate((Bucket *)self->data->child); if (o == NULL) goto err; PyTuple_SET_ITEM(r, 0, o); ASSIGN(r, Py_BuildValue("(O)", r)); } else { for (i=0, l=0; i < self->len; i++) { if (i) { COPY_KEY_TO_OBJECT(o, self->data[i].key); PyTuple_SET_ITEM(r, l, o); l++; } o = (PyObject *)self->data[i].child; Py_INCREF(o); PyTuple_SET_ITEM(r,l,o); l++; } ASSIGN(r, Py_BuildValue("OO", r, self->firstbucket)); } } else { r = Py_None; Py_INCREF(r); } PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); return NULL; } static int _BTree_setstate(BTree *self, PyObject *state, int noval) { PyObject *items, *firstbucket = NULL; BTreeItem *d; int len, l, i, copied=1; if (_BTree_clear(self) < 0) return -1; /* The state of a BTree can be one of the following: None -- an empty BTree A one-tuple -- a single bucket btree A two-tuple -- a BTree with more than one bucket See comments for BTree_getstate() for the details. */ if (state == Py_None) return 0; if (!PyArg_ParseTuple(state, "O|O:__setstate__", &items, &firstbucket)) return -1; if (!PyTuple_Check(items)) { PyErr_SetString(PyExc_TypeError, "tuple required for first state element"); return -1; } len = PyTuple_Size(items); if (len < 0) return -1; len = (len + 1) / 2; assert(len > 0); /* If the BTree is empty, it's state is None. */ assert(self->size == 0); /* We called _BTree_clear(). */ self->data = BTree_Malloc(sizeof(BTreeItem) * len); if (self->data == NULL) return -1; self->size = len; for (i = 0, d = self->data, l = 0; i < len; i++, d++) { PyObject *v; if (i) { /* skip the first key slot */ COPY_KEY_FROM_ARG(d->key, PyTuple_GET_ITEM(items, l), copied); l++; if (!copied) return -1; INCREF_KEY(d->key); } v = PyTuple_GET_ITEM(items, l); if (PyTuple_Check(v)) { /* Handle the special case in __getstate__() for a BTree with a single bucket. */ d->child = BTree_newBucket(self); if (!d->child) return -1; if (noval) { if (_set_setstate(BUCKET(d->child), v) < 0) return -1; } else { if (_bucket_setstate(BUCKET(d->child), v) < 0) return -1; } } else { d->child = (Sized *)v; Py_INCREF(v); } l++; } if (!firstbucket) firstbucket = (PyObject *)self->data->child; if (!PyObject_IsInstance(firstbucket, (PyObject *) (noval ? &SetType : &BucketType))) { PyErr_SetString(PyExc_TypeError, "No firstbucket in non-empty BTree"); return -1; } self->firstbucket = BUCKET(firstbucket); Py_INCREF(firstbucket); #ifndef PERSISTENT /* firstbucket is also the child of some BTree node, but that node may * be a ghost if persistence is enabled. */ assert(Py_REFCNT(self->firstbucket) > 1); #endif self->len = len; return 0; } static PyObject * BTree_setstate(BTree *self, PyObject *arg) { int r; PER_PREVENT_DEACTIVATION(self); r = _BTree_setstate(self, arg, 0); PER_UNUSE(self); if (r < 0) return NULL; Py_INCREF(Py_None); return Py_None; } #ifdef PERSISTENT /* Recognize the special cases of a BTree that's empty or contains a single * bucket. In the former case, return a borrowed reference to Py_None. * In this single-bucket case, the bucket state is embedded directly in the * BTree state, like so: * * ( * ( * thebucket.__getstate__(), * ), * ) * * When this obtains, return a borrowed reference to thebucket.__getstate__(). * Else return NULL with an exception set. The exception should always be * ConflictError then, but may be TypeError if the state makes no sense at all * for a BTree (corrupted or hostile state). */ PyObject * get_bucket_state(PyObject *t) { if (t == Py_None) return Py_None; /* an empty BTree */ if (! PyTuple_Check(t)) { PyErr_SetString(PyExc_TypeError, "_p_resolveConflict: expected tuple or None for state"); return NULL; } if (PyTuple_GET_SIZE(t) == 2) { /* A non-degenerate BTree. */ return merge_error(-1, -1, -1, 11); } /* We're in the one-bucket case. */ if (PyTuple_GET_SIZE(t) != 1) { PyErr_SetString(PyExc_TypeError, "_p_resolveConflict: expected 1- or 2-tuple for state"); return NULL; } t = PyTuple_GET_ITEM(t, 0); if (! PyTuple_Check(t) || PyTuple_GET_SIZE(t) != 1) { PyErr_SetString(PyExc_TypeError, "_p_resolveConflict: expected 1-tuple containing " "bucket state"); return NULL; } t = PyTuple_GET_ITEM(t, 0); if (! PyTuple_Check(t)) { PyErr_SetString(PyExc_TypeError, "_p_resolveConflict: expected tuple for bucket state"); return NULL; } return t; } /* Tricky. The only kind of BTree conflict we can actually potentially * resolve is the special case of a BTree containing a single bucket, * in which case this becomes a fancy way of calling the bucket conflict * resolution code. */ static PyObject * BTree__p_resolveConflict(BTree *self, PyObject *args) { PyObject *s[3]; PyObject *x, *y, *z; if (!PyArg_ParseTuple(args, "OOO", &x, &y, &z)) return NULL; s[0] = get_bucket_state(x); if (s[0] == NULL) return NULL; s[1] = get_bucket_state(y); if (s[1] == NULL) return NULL; s[2] = get_bucket_state(z); if (s[2] == NULL) return NULL; if (PyObject_IsInstance((PyObject *)self, (PyObject *)&BTreeType)) x = _bucket__p_resolveConflict(OBJECT(&BucketType), s); else x = _bucket__p_resolveConflict(OBJECT(&SetType), s); if (x == NULL) return NULL; return Py_BuildValue("((N))", x); } #endif /* BTree_findRangeEnd -- Find one end, expressed as a bucket and position, for a range search. If low, return bucket and index of the smallest item >= key, otherwise return bucket and index of the largest item <= key. If exclude_equal, exact matches aren't acceptable; if one is found, move right if low, or left if !low (this is for range searches exclusive of an endpoint). Return: -1 Error; offset and bucket unchanged 0 Not found; offset and bucket unchanged 1 Correct bucket and offset stored; the caller owns a new reference to the bucket. Internal: We do binary searches in BTree nodes downward, at each step following C(i) where K(i) <= key < K(i+1). As always, K(i) <= C(i) < K(i+1) too. (See Maintainer.txt for the meaning of that notation.) That eventually leads to a bucket where we do Bucket_findRangeEnd. That usually works, but there are two cases where it can fail to find the correct answer: 1. On a low search, we find a bucket with keys >= K(i), but that doesn't imply there are keys in the bucket >= key. For example, suppose a bucket has keys in 1..100, its successor's keys are in 200..300, and we're doing a low search on 150. We'll end up in the first bucket, but there are no keys >= 150 in it. K(i+1) > key, though, and all the keys in C(i+1) >= K(i+1) > key, so the first key in the next bucket (if any) is the correct result. This is easy to find by following the bucket 'next' pointer. 2. On a high search, again that the keys in the bucket are >= K(i) doesn't imply that any key in the bucket is <= key, but it's harder for this to fail (and an earlier version of this routine didn't catch it): if K(i) itself is in the bucket, it works (then K(i) <= key is *a* key in the bucket that's in the desired range). But when keys get deleted from buckets, they aren't also deleted from BTree nodes, so there's no guarantee that K(i) is in the bucket. For example, delete the smallest key S from some bucket, and S remains in the interior BTree nodes. Do a high search for S, and the BTree nodes direct the search to the bucket S used to be in, but all keys remaining in that bucket are > S. The largest key in the *preceding* bucket (if any) is < K(i), though, and K(i) <= key, so the largest key in the preceding bucket is < key and so is the proper result. This is harder to get at efficiently, as buckets are linked only in the increasing direction. While we're searching downward, deepest_smaller is set to the node deepest in the tree where we *could* have gone to the left of C(i). The rightmost bucket in deepest_smaller's subtree is the bucket preceding the bucket we find at first. This is clumsy to get at, but efficient. */ static int BTree_findRangeEnd(BTree *self, PyObject *keyarg, int low, int exclude_equal, Bucket **bucket, int *offset) { Sized *deepest_smaller = NULL; /* last possibility to move left */ int deepest_smaller_is_btree = 0; /* Boolean; if false, it's a bucket */ Bucket *pbucket; int self_got_rebound = 0; /* Boolean; when true, deactivate self */ int result = -1; /* Until proven innocent */ int i; KEY_TYPE key; int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); UNLESS (copied) return -1; /* We don't need to: PER_USE_OR_RETURN(self, -1); because the caller does. */ UNLESS (self->data && self->len) return 0; /* Search downward until hitting a bucket, stored in pbucket. */ for (;;) { Sized *pchild; int pchild_is_btree; BTREE_SEARCH(i, self, key, goto Done); pchild = self->data[i].child; pchild_is_btree = SameType_Check(self, pchild); if (i) { deepest_smaller = self->data[i-1].child; deepest_smaller_is_btree = pchild_is_btree; } if (pchild_is_btree) { if (self_got_rebound) { PER_UNUSE(self); } self = BTREE(pchild); self_got_rebound = 1; PER_USE_OR_RETURN(self, -1); } else { pbucket = BUCKET(pchild); break; } } /* Search the bucket for a suitable key. */ i = Bucket_findRangeEnd(pbucket, keyarg, low, exclude_equal, offset); if (i < 0) goto Done; if (i > 0) { Py_INCREF(pbucket); *bucket = pbucket; result = 1; goto Done; } /* This may be one of the two difficult cases detailed in the comments. */ if (low) { Bucket *next; UNLESS(PER_USE(pbucket)) goto Done; next = pbucket->next; if (next) { result = 1; Py_INCREF(next); *bucket = next; *offset = 0; } else result = 0; PER_UNUSE(pbucket); } /* High-end search: if it's possible to go left, do so. */ else if (deepest_smaller) { if (deepest_smaller_is_btree) { UNLESS(PER_USE(deepest_smaller)) goto Done; /* We own the reference this returns. */ pbucket = BTree_lastBucket(BTREE(deepest_smaller)); PER_UNUSE(deepest_smaller); if (pbucket == NULL) goto Done; /* error */ } else { pbucket = BUCKET(deepest_smaller); Py_INCREF(pbucket); } UNLESS(PER_USE(pbucket)) goto Done; result = 1; *bucket = pbucket; /* transfer ownership to caller */ *offset = pbucket->len - 1; PER_UNUSE(pbucket); } else result = 0; /* simply not found */ Done: if (self_got_rebound) { PER_UNUSE(self); } return result; } static PyObject * BTree_maxminKey(BTree *self, PyObject *args, int min) { PyObject *key=0; Bucket *bucket = NULL; int offset, rc; int empty_tree = 1; UNLESS (PyArg_ParseTuple(args, "|O", &key)) return NULL; UNLESS (PER_USE(self)) return NULL; UNLESS (self->data && self->len) goto empty; /* Find the range */ if (key && key != Py_None) { if ((rc = BTree_findRangeEnd(self, key, min, 0, &bucket, &offset)) <= 0) { if (rc < 0) goto err; empty_tree = 0; goto empty; } PER_UNUSE(self); UNLESS (PER_USE(bucket)) { Py_DECREF(bucket); return NULL; } } else if (min) { bucket = self->firstbucket; PER_UNUSE(self); PER_USE_OR_RETURN(bucket, NULL); Py_INCREF(bucket); offset = 0; } else { bucket = BTree_lastBucket(self); PER_UNUSE(self); UNLESS (PER_USE(bucket)) { Py_DECREF(bucket); return NULL; } assert(bucket->len); offset = bucket->len - 1; } COPY_KEY_TO_OBJECT(key, bucket->keys[offset]); PER_UNUSE(bucket); Py_DECREF(bucket); return key; empty: PyErr_SetString(PyExc_ValueError, empty_tree ? "empty tree" : "no key satisfies the conditions"); err: PER_UNUSE(self); if (bucket) { PER_UNUSE(bucket); Py_DECREF(bucket); } return NULL; } static PyObject * BTree_minKey(BTree *self, PyObject *args) { return BTree_maxminKey(self, args, 1); } static PyObject * BTree_maxKey(BTree *self, PyObject *args) { return BTree_maxminKey(self, args, 0); } /* ** BTree_rangeSearch ** ** Generates a BTreeItems object based on the two indexes passed in, ** being the range between them. ** */ static PyObject * BTree_rangeSearch(BTree *self, PyObject *args, PyObject *kw, char type) { PyObject *min = Py_None; PyObject *max = Py_None; int excludemin = 0; int excludemax = 0; int rc; Bucket *lowbucket = NULL; Bucket *highbucket = NULL; int lowoffset; int highoffset; PyObject *result; if (args) { if (! PyArg_ParseTupleAndKeywords(args, kw, "|OOii", search_keywords, &min, &max, &excludemin, &excludemax)) return NULL; } UNLESS (PER_USE(self)) return NULL; UNLESS (self->data && self->len) goto empty; /* Find the low range */ if (min != Py_None) { if ((rc = BTree_findRangeEnd(self, min, 1, excludemin, &lowbucket, &lowoffset)) <= 0) { if (rc < 0) goto err; goto empty; } } else { lowbucket = self->firstbucket; lowoffset = 0; if (excludemin) { int bucketlen; UNLESS (PER_USE(lowbucket)) goto err; bucketlen = lowbucket->len; PER_UNUSE(lowbucket); if (bucketlen > 1) lowoffset = 1; else if (self->len < 2) goto empty; else { /* move to first item in next bucket */ Bucket *next; UNLESS (PER_USE(lowbucket)) goto err; next = lowbucket->next; PER_UNUSE(lowbucket); assert(next != NULL); lowbucket = next; /* and lowoffset is still 0 */ assert(lowoffset == 0); } } Py_INCREF(lowbucket); } /* Find the high range */ if (max != Py_None) { if ((rc = BTree_findRangeEnd(self, max, 0, excludemax, &highbucket, &highoffset)) <= 0) { Py_DECREF(lowbucket); if (rc < 0) goto err; goto empty; } } else { int bucketlen; highbucket = BTree_lastBucket(self); assert(highbucket != NULL); /* we know self isn't empty */ UNLESS (PER_USE(highbucket)) goto err_and_decref_buckets; bucketlen = highbucket->len; PER_UNUSE(highbucket); highoffset = bucketlen - 1; if (excludemax) { if (highoffset > 0) --highoffset; else if (self->len < 2) goto empty_and_decref_buckets; else /* move to last item of preceding bucket */ { int status; assert(highbucket != self->firstbucket); Py_DECREF(highbucket); status = PreviousBucket(&highbucket, self->firstbucket); if (status < 0) { Py_DECREF(lowbucket); goto err; } assert(status > 0); Py_INCREF(highbucket); UNLESS (PER_USE(highbucket)) goto err_and_decref_buckets; highoffset = highbucket->len - 1; PER_UNUSE(highbucket); } } assert(highoffset >= 0); } /* It's still possible that the range is empty, even if min < max. For * example, if min=3 and max=4, and 3 and 4 aren't in the BTree, but 2 and * 5 are, then the low position points to the 5 now and the high position * points to the 2 now. They're not necessarily even in the same bucket, * so there's no trick we can play with pointer compares to get out * cheap in general. */ if (lowbucket == highbucket && lowoffset > highoffset) goto empty_and_decref_buckets; /* definitely empty */ /* The buckets differ, or they're the same and the offsets show a non- * empty range. */ if (min != Py_None && max != Py_None && /* both args user-supplied */ lowbucket != highbucket) /* and different buckets */ { KEY_TYPE first; KEY_TYPE last; int cmp; /* Have to check the hard way: see how the endpoints compare. */ UNLESS (PER_USE(lowbucket)) goto err_and_decref_buckets; COPY_KEY(first, lowbucket->keys[lowoffset]); PER_UNUSE(lowbucket); UNLESS (PER_USE(highbucket)) goto err_and_decref_buckets; COPY_KEY(last, highbucket->keys[highoffset]); PER_UNUSE(highbucket); TEST_KEY_SET_OR(cmp, first, last) goto err_and_decref_buckets; if (cmp > 0) goto empty_and_decref_buckets; } PER_UNUSE(self); result = newBTreeItems(type, lowbucket, lowoffset, highbucket, highoffset); Py_DECREF(lowbucket); Py_DECREF(highbucket); return result; err_and_decref_buckets: Py_DECREF(lowbucket); Py_DECREF(highbucket); err: PER_UNUSE(self); return NULL; empty_and_decref_buckets: Py_DECREF(lowbucket); Py_DECREF(highbucket); empty: PER_UNUSE(self); return newBTreeItems(type, 0, 0, 0, 0); } /* ** BTree_keys */ static PyObject * BTree_keys(BTree *self, PyObject *args, PyObject *kw) { return BTree_rangeSearch(self, args, kw, 'k'); } /* ** BTree_values */ static PyObject * BTree_values(BTree *self, PyObject *args, PyObject *kw) { return BTree_rangeSearch(self, args, kw, 'v'); } /* ** BTree_items */ static PyObject * BTree_items(BTree *self, PyObject *args, PyObject *kw) { return BTree_rangeSearch(self, args, kw, 'i'); } static PyObject * BTree_byValue(BTree *self, PyObject *omin) { PyObject *r=0, *o=0, *item=0; VALUE_TYPE min; VALUE_TYPE v; int copied=1; SetIteration it = {0, 0, 1}; UNLESS (PER_USE(self)) return NULL; COPY_VALUE_FROM_ARG(min, omin, copied); UNLESS(copied) return NULL; UNLESS (r=PyList_New(0)) goto err; it.set=BTree_rangeSearch(self, NULL, NULL, 'i'); UNLESS(it.set) goto err; if (nextBTreeItems(&it) < 0) goto err; while (it.position >= 0) { if (TEST_VALUE(it.value, min) >= 0) { UNLESS (item = PyTuple_New(2)) goto err; COPY_KEY_TO_OBJECT(o, it.key); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 1, o); COPY_VALUE(v, it.value); NORMALIZE_VALUE(v, min); COPY_VALUE_TO_OBJECT(o, v); DECREF_VALUE(v); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 0, o); if (PyList_Append(r, item) < 0) goto err; Py_DECREF(item); item = 0; } if (nextBTreeItems(&it) < 0) goto err; } item=PyObject_GetAttr(r,sort_str); UNLESS (item) goto err; ASSIGN(item, PyObject_CallObject(item, NULL)); UNLESS (item) goto err; ASSIGN(item, PyObject_GetAttr(r, reverse_str)); UNLESS (item) goto err; ASSIGN(item, PyObject_CallObject(item, NULL)); UNLESS (item) goto err; Py_DECREF(item); finiSetIteration(&it); PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); finiSetIteration(&it); Py_XDECREF(item); return NULL; } /* ** BTree_getm */ static PyObject * BTree_getm(BTree *self, PyObject *args) { PyObject *key, *d=Py_None, *r; UNLESS (PyArg_ParseTuple(args, "O|O", &key, &d)) return NULL; if ((r=_BTree_get(self, key, 0))) return r; UNLESS (PyErr_ExceptionMatches(PyExc_KeyError)) return NULL; PyErr_Clear(); Py_INCREF(d); return d; } static PyObject * BTree_has_key(BTree *self, PyObject *key) { return _BTree_get(self, key, 1); } static PyObject * BTree_setdefault(BTree *self, PyObject *args) { PyObject *key; PyObject *failobj; /* default */ PyObject *value; /* return value */ if (! PyArg_UnpackTuple(args, "setdefault", 2, 2, &key, &failobj)) return NULL; value = _BTree_get(self, key, 0); if (value != NULL) return value; /* The key isn't in the tree. If that's not due to a KeyError exception, * pass back the unexpected exception. */ if (! PyErr_ExceptionMatches(PyExc_KeyError)) return NULL; PyErr_Clear(); /* Associate `key` with `failobj` in the tree, and return `failobj`. */ value = failobj; if (_BTree_set(self, key, failobj, 0, 0) < 0) value = NULL; Py_XINCREF(value); return value; } /* forward declaration */ static Py_ssize_t BTree_length_or_nonzero(BTree *self, int nonzero); static PyObject * BTree_pop(BTree *self, PyObject *args) { PyObject *key; PyObject *failobj = NULL; /* default */ PyObject *value; /* return value */ if (! PyArg_UnpackTuple(args, "pop", 1, 2, &key, &failobj)) return NULL; value = _BTree_get(self, key, 0); if (value != NULL) { /* Delete key and associated value. */ if (_BTree_set(self, key, NULL, 0, 0) < 0) { Py_DECREF(value); return NULL;; } return value; } /* The key isn't in the tree. If that's not due to a KeyError exception, * pass back the unexpected exception. */ if (! PyErr_ExceptionMatches(PyExc_KeyError)) return NULL; if (failobj != NULL) { /* Clear the KeyError and return the explicit default. */ PyErr_Clear(); Py_INCREF(failobj); return failobj; } /* No default given. The only difference in this case is the error * message, which depends on whether the tree is empty. */ if (BTree_length_or_nonzero(self, 1) == 0) /* tree is empty */ PyErr_SetString(PyExc_KeyError, "pop(): BTree is empty"); return NULL; } /* Search BTree self for key. This is the sq_contains slot of the * PySequenceMethods. * * Return: * -1 error * 0 not found * 1 found */ static int BTree_contains(BTree *self, PyObject *key) { PyObject *asobj = _BTree_get(self, key, 1); int result = -1; if (asobj != NULL) { result = INT_AS_LONG(asobj) ? 1 : 0; Py_DECREF(asobj); } return result; } static PyObject * BTree_addUnique(BTree *self, PyObject *args) { int grew; PyObject *key, *v; UNLESS (PyArg_ParseTuple(args, "OO", &key, &v)) return NULL; if ((grew=_BTree_set(self, key, v, 1, 0)) < 0) return NULL; return INT_FROM_LONG(grew); } /**************************************************************************/ /* Iterator support. */ /* A helper to build all the iterators for BTrees and TreeSets. * If args is NULL, the iterator spans the entire structure. Else it's an * argument tuple, with optional low and high arguments. * kind is 'k', 'v' or 'i'. * Returns a BTreeIter object, or NULL if error. */ static PyObject * buildBTreeIter(BTree *self, PyObject *args, PyObject *kw, char kind) { BTreeIter *result = NULL; BTreeItems *items = (BTreeItems *)BTree_rangeSearch(self, args, kw, kind); if (items) { result = BTreeIter_new(items); Py_DECREF(items); } return (PyObject *)result; } /* The implementation of iter(BTree_or_TreeSet); the BTree tp_iter slot. */ static PyObject * BTree_getiter(BTree *self) { return buildBTreeIter(self, NULL, NULL, 'k'); } /* The implementation of BTree.iterkeys(). */ static PyObject * BTree_iterkeys(BTree *self, PyObject *args, PyObject *kw) { return buildBTreeIter(self, args, kw, 'k'); } /* The implementation of BTree.itervalues(). */ static PyObject * BTree_itervalues(BTree *self, PyObject *args, PyObject *kw) { return buildBTreeIter(self, args, kw, 'v'); } /* The implementation of BTree.iteritems(). */ static PyObject * BTree_iteritems(BTree *self, PyObject *args, PyObject *kw) { return buildBTreeIter(self, args, kw, 'i'); } /* End of iterator support. */ /* Caution: Even though the _firstbucket attribute is read-only, a program could do arbitrary damage to the btree internals. For example, it could call clear() on a bucket inside a BTree. We need to decide if the convenience for inspecting BTrees is worth the risk. */ static struct PyMemberDef BTree_members[] = { {"_firstbucket", T_OBJECT, offsetof(BTree, firstbucket), READONLY}, {NULL} }; static struct PyMethodDef BTree_methods[] = { {"__getstate__", (PyCFunction) BTree_getstate, METH_NOARGS, "__getstate__() -> state\n\n" "Return the picklable state of the BTree."}, {"__setstate__", (PyCFunction) BTree_setstate, METH_O, "__setstate__(state)\n\n" "Set the state of the BTree."}, {"has_key", (PyCFunction) BTree_has_key, METH_O, "has_key(key)\n\n" "Return true if the BTree contains the given key."}, {"keys", (PyCFunction) BTree_keys, METH_VARARGS | METH_KEYWORDS, "keys([min, max]) -> list of keys\n\n" "Returns the keys of the BTree. If min and max are supplied, only\n" "keys greater than min and less than max are returned."}, {"values", (PyCFunction) BTree_values, METH_VARARGS | METH_KEYWORDS, "values([min, max]) -> list of values\n\n" "Returns the values of the BTree. If min and max are supplied, only\n" "values corresponding to keys greater than min and less than max are\n" "returned."}, {"items", (PyCFunction) BTree_items, METH_VARARGS | METH_KEYWORDS, "items([min, max]) -> -- list of key, value pairs\n\n" "Returns the items of the BTree. If min and max are supplied, only\n" "items with keys greater than min and less than max are returned."}, {"byValue", (PyCFunction) BTree_byValue, METH_O, "byValue(min) -> list of value, key pairs\n\n" "Returns list of value, key pairs where the value is >= min. The\n" "list is sorted by value. Note that items() returns keys in the\n" "opposite order."}, {"get", (PyCFunction) BTree_getm, METH_VARARGS, "get(key[, default=None]) -> Value for key or default\n\n" "Return the value or the default if the key is not found."}, {"setdefault", (PyCFunction) BTree_setdefault, METH_VARARGS, "D.setdefault(k, d) -> D.get(k, d), also set D[k]=d if k not in D.\n\n" "Return the value like get() except that if key is missing, d is both\n" "returned and inserted into the BTree as the value of k."}, {"pop", (PyCFunction) BTree_pop, METH_VARARGS, "D.pop(k[, d]) -> v, remove key and return the corresponding value.\n\n" "If key is not found, d is returned if given, otherwise KeyError\n" "is raised."}, {"maxKey", (PyCFunction) BTree_maxKey, METH_VARARGS, "maxKey([max]) -> key\n\n" "Return the largest key in the BTree. If max is specified, return\n" "the largest key <= max."}, {"minKey", (PyCFunction) BTree_minKey, METH_VARARGS, "minKey([mi]) -> key\n\n" "Return the smallest key in the BTree. If min is specified, return\n" "the smallest key >= min."}, {"clear", (PyCFunction) BTree_clear, METH_NOARGS, "clear()\n\nRemove all of the items from the BTree."}, {"insert", (PyCFunction)BTree_addUnique, METH_VARARGS, "insert(key, value) -> 0 or 1\n\n" "Add an item if the key is not already used. Return 1 if the item was\n" "added, or 0 otherwise."}, {"update", (PyCFunction) Mapping_update, METH_O, "update(collection)\n\n Add the items from the given collection."}, {"iterkeys", (PyCFunction) BTree_iterkeys, METH_VARARGS | METH_KEYWORDS, "B.iterkeys([min[,max]]) -> an iterator over the keys of B"}, {"itervalues", (PyCFunction) BTree_itervalues, METH_VARARGS | METH_KEYWORDS, "B.itervalues([min[,max]]) -> an iterator over the values of B"}, {"iteritems", (PyCFunction) BTree_iteritems, METH_VARARGS | METH_KEYWORDS, "B.iteritems([min[,max]]) -> an iterator over the (key, value) " "items of B"}, {"_check", (PyCFunction) BTree_check, METH_NOARGS, "Perform sanity check on BTree, and raise exception if flawed."}, #ifdef PERSISTENT {"_p_resolveConflict", (PyCFunction) BTree__p_resolveConflict, METH_VARARGS, "_p_resolveConflict() -- Reinitialize from a newly created copy"}, {"_p_deactivate", (PyCFunction) BTree__p_deactivate, METH_VARARGS | METH_KEYWORDS, "_p_deactivate()\n\nReinitialize from a newly created copy."}, #endif {NULL, NULL} }; static int BTree_init(PyObject *self, PyObject *args, PyObject *kwds) { PyObject *v = NULL; BTREE(self)->max_leaf_size = 0; BTREE(self)->max_internal_size = 0; if (!PyArg_ParseTuple(args, "|O:" MOD_NAME_PREFIX "BTree", &v)) return -1; if (v) return update_from_seq(self, v); else return 0; } static void BTree_dealloc(BTree *self) { if (self->state != cPersistent_GHOST_STATE) _BTree_clear(self); cPersistenceCAPI->pertype->tp_dealloc((PyObject *)self); } static int BTree_traverse(BTree *self, visitproc visit, void *arg) { int err = 0; int i, len; #define VISIT(SLOT) \ if (SLOT) { \ err = visit((PyObject *)(SLOT), arg); \ if (err) \ goto Done; \ } if (Py_TYPE(self) == &BTreeType) assert(Py_TYPE(self)->tp_dictoffset == 0); /* Call our base type's traverse function. Because BTrees are * subclasses of Peristent, there must be one. */ err = cPersistenceCAPI->pertype->tp_traverse((PyObject *)self, visit, arg); if (err) goto Done; /* If this is registered with the persistence system, cleaning up cycles * is the database's problem. It would be horrid to unghostify BTree * nodes here just to chase pointers every time gc runs. */ if (self->state == cPersistent_GHOST_STATE) goto Done; len = self->len; #ifdef KEY_TYPE_IS_PYOBJECT /* Keys are Python objects so need to be traversed. Note that the * key 0 slot is unused and should not be traversed. */ for (i = 1; i < len; i++) VISIT(self->data[i].key); #endif /* Children are always pointers, and child 0 is legit. */ for (i = 0; i < len; i++) VISIT(self->data[i].child); VISIT(self->firstbucket); Done: return err; #undef VISIT } static int BTree_tp_clear(BTree *self) { if (self->state != cPersistent_GHOST_STATE) _BTree_clear(self); return 0; } /* * Return the number of elements in a BTree. nonzero is a Boolean, and * when true requests just a non-empty/empty result. Testing for emptiness * is efficient (constant-time). Getting the true length takes time * proportional to the number of leaves (buckets). * * Return: * When nonzero true: * -1 error * 0 empty * 1 not empty * When nonzero false (possibly expensive!): * -1 error * >= 0 number of elements. */ static Py_ssize_t BTree_length_or_nonzero(BTree *self, int nonzero) { int result; Bucket *b; Bucket *next; PER_USE_OR_RETURN(self, -1); b = self->firstbucket; PER_UNUSE(self); if (nonzero) return b != NULL; result = 0; while (b) { PER_USE_OR_RETURN(b, -1); result += b->len; next = b->next; PER_UNUSE(b); b = next; } return result; } static Py_ssize_t BTree_length(BTree *self) { return BTree_length_or_nonzero(self, 0); } static PyMappingMethods BTree_as_mapping = { (lenfunc)BTree_length, /* mp_length */ (binaryfunc)BTree_get, /* mp_subscript */ (objobjargproc)BTree_setitem, /* mp_ass_subscript */ }; static PySequenceMethods BTree_as_sequence = { (lenfunc)0, /* sq_length */ (binaryfunc)0, /* sq_concat */ (ssizeargfunc)0, /* sq_repeat */ (ssizeargfunc)0, /* sq_item */ (ssizessizeargfunc)0, /* sq_slice */ (ssizeobjargproc)0, /* sq_ass_item */ (ssizessizeobjargproc)0, /* sq_ass_slice */ (objobjproc)BTree_contains, /* sq_contains */ 0, /* sq_inplace_concat */ 0, /* sq_inplace_repeat */ }; static Py_ssize_t BTree_nonzero(BTree *self) { return BTree_length_or_nonzero(self, 1); } static PyNumberMethods BTree_as_number_for_nonzero = { 0, /* nb_add */ 0, /* nb_subtract */ 0, /* nb_multiply */ #ifndef PY3K 0, /* nb_divide */ #endif 0, /* nb_remainder */ 0, /* nb_divmod */ 0, /* nb_power */ 0, /* nb_negative */ 0, /* nb_positive */ 0, /* nb_absolute */ (inquiry)BTree_nonzero /* nb_nonzero */ }; static PyTypeObject BTreeType = { PyVarObject_HEAD_INIT(NULL, 0) MODULE_NAME MOD_NAME_PREFIX "BTree", /* tp_name */ sizeof(BTree), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)BTree_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ &BTree_as_number_for_nonzero, /* tp_as_number */ &BTree_as_sequence, /* tp_as_sequence */ &BTree_as_mapping, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ (traverseproc)BTree_traverse, /* tp_traverse */ (inquiry)BTree_tp_clear, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ (getiterfunc)BTree_getiter, /* tp_iter */ 0, /* tp_iternext */ BTree_methods, /* tp_methods */ BTree_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ BTree_init, /* tp_init */ 0, /* tp_alloc */ 0, /*PyType_GenericNew,*/ /* tp_new */ }; BTrees-4.3.1/BTrees/LLBTree.py0000664000175000017500000001027712665316015017054 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __all__ = ('Bucket', 'Set', 'BTree', 'TreeSet', 'LLBucket', 'LLSet', 'LLBTree', 'LLTreeSet', 'union', 'intersection', 'difference', 'weightedUnion', 'weightedIntersection', 'multiunion', ) from zope.interface import moduleProvides from .Interfaces import IIntegerIntegerBTreeModule from ._base import Bucket from ._base import MERGE from ._base import MERGE_WEIGHT_numeric from ._base import MERGE_DEFAULT_int from ._base import Set from ._base import Tree as BTree from ._base import TreeSet from ._base import _TreeIterator from ._base import difference as _difference from ._base import intersection as _intersection from ._base import multiunion as _multiunion from ._base import set_operation as _set_operation from ._base import to_long as _to_key from ._base import to_long as _to_value from ._base import union as _union from ._base import weightedIntersection as _weightedIntersection from ._base import weightedUnion as _weightedUnion from ._base import _fix_pickle _BUCKET_SIZE = 120 _TREE_SIZE = 500 using64bits = True class LLBucketPy(Bucket): _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class LLSetPy(Set): _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class LLBTreePy(BTree): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key _to_value = _to_value MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class LLTreeSetPy(TreeSet): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key MERGE = MERGE MERGE_WEIGHT = MERGE_WEIGHT_numeric MERGE_DEFAULT = MERGE_DEFAULT_int class LLTreeIteratorPy(_TreeIterator): pass # Can't declare forward refs, so fix up afterwards: LLBucketPy._mapping_type = LLBucketPy._bucket_type = LLBucketPy LLBucketPy._set_type = LLSetPy LLSetPy._mapping_type = LLBucketPy LLSetPy._set_type = LLSetPy._bucket_type = LLSetPy LLBTreePy._mapping_type = LLBTreePy._bucket_type = LLBucketPy LLBTreePy._set_type = LLSetPy LLTreeSetPy._mapping_type = LLBucketPy LLTreeSetPy._set_type = LLTreeSetPy._bucket_type = LLSetPy differencePy = _set_operation(_difference, LLSetPy) unionPy = _set_operation(_union, LLSetPy) intersectionPy = _set_operation(_intersection, LLSetPy) multiunionPy = _set_operation(_multiunion, LLSetPy) weightedUnionPy = _set_operation(_weightedUnion, LLSetPy) weightedIntersectionPy = _set_operation(_weightedIntersection, LLSetPy) try: from ._LLBTree import LLBucket except ImportError: #pragma NO COVER w/ C extensions LLBucket = LLBucketPy LLSet = LLSetPy LLBTree = LLBTreePy LLTreeSet = LLTreeSetPy LLTreeIterator = LLTreeIteratorPy difference = differencePy union = unionPy intersection = intersectionPy multiunion = multiunionPy weightedUnion = weightedUnionPy weightedIntersection = weightedIntersectionPy else: #pragma NO COVER w/o C extensions from ._LLBTree import LLSet from ._LLBTree import LLBTree from ._LLBTree import LLTreeSet from ._LLBTree import LLTreeIterator from ._LLBTree import difference from ._LLBTree import union from ._LLBTree import intersection from ._LLBTree import multiunion from ._LLBTree import weightedUnion from ._LLBTree import weightedIntersection Bucket = LLBucket Set = LLSet BTree = LLBTree TreeSet = LLTreeSet _fix_pickle(globals(), __name__) moduleProvides(IIntegerIntegerBTreeModule) BTrees-4.3.1/BTrees/_OOBTree.c0000664000175000017500000000217612510775554017020 0ustar tseavertseaver00000000000000/*############################################################################ # # Copyright (c) 2004 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id$\n" /* OOBTree - object key, object value BTree Implements a collection using object type keys and object type values */ #define PERSISTENT #define MOD_NAME_PREFIX "OO" #define DEFAULT_MAX_BUCKET_SIZE 30 #define DEFAULT_MAX_BTREE_SIZE 250 #include "_compat.h" #include "objectkeymacros.h" #include "objectvaluemacros.h" #ifdef PY3K #define INITMODULE PyInit__OOBTree #else #define INITMODULE init_OOBTree #endif #include "BTreeModuleTemplate.c" BTrees-4.3.1/BTrees/TreeSetTemplate.c0000664000175000017500000002040712510775554020466 0ustar tseavertseaver00000000000000/***************************************************************************** Copyright (c) 2001, 2002 Zope Foundation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #include "_compat.h" #define TREESETTEMPLATE_C "$Id$\n" static PyObject * TreeSet_insert(BTree *self, PyObject *args) { PyObject *key; int i; if (!PyArg_ParseTuple(args, "O:insert", &key)) return NULL; i = _BTree_set(self, key, Py_None, 1, 1); if (i < 0) return NULL; return INT_FROM_LONG(i); } /* _Set_update and _TreeSet_update are identical except for the function they call to add the element to the set. */ static int _TreeSet_update(BTree *self, PyObject *seq) { int n=0, ind=0; PyObject *iter, *v; iter = PyObject_GetIter(seq); if (iter == NULL) return -1; while (1) { v = PyIter_Next(iter); if (v == NULL) { if (PyErr_Occurred()) goto err; else break; } ind = _BTree_set(self, v, Py_None, 1, 1); Py_DECREF(v); if (ind < 0) goto err; else n += ind; } err: Py_DECREF(iter); if (ind < 0) return -1; return n; } static PyObject * TreeSet_update(BTree *self, PyObject *args) { PyObject *seq = NULL; int n = 0; if (!PyArg_ParseTuple(args, "|O:update", &seq)) return NULL; if (seq) { n = _TreeSet_update(self, seq); if (n < 0) return NULL; } return INT_FROM_LONG(n); } static PyObject * TreeSet_remove(BTree *self, PyObject *args) { PyObject *key; UNLESS (PyArg_ParseTuple(args, "O", &key)) return NULL; if (_BTree_set(self, key, NULL, 0, 1) < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static PyObject * TreeSet_setstate(BTree *self, PyObject *args) { int r; if (!PyArg_ParseTuple(args,"O",&args)) return NULL; PER_PREVENT_DEACTIVATION(self); r=_BTree_setstate(self, args, 1); PER_UNUSE(self); if (r < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static struct PyMethodDef TreeSet_methods[] = { {"__getstate__", (PyCFunction) BTree_getstate, METH_NOARGS, "__getstate__() -> state\n\n" "Return the picklable state of the TreeSet."}, {"__setstate__", (PyCFunction) TreeSet_setstate, METH_VARARGS, "__setstate__(state)\n\n" "Set the state of the TreeSet."}, {"has_key", (PyCFunction) BTree_has_key, METH_O, "has_key(key)\n\n" "Return true if the TreeSet contains the given key."}, {"keys", (PyCFunction) BTree_keys, METH_VARARGS | METH_KEYWORDS, "keys([min, max]) -> list of keys\n\n" "Returns the keys of the TreeSet. If min and max are supplied, only\n" "keys greater than min and less than max are returned."}, {"maxKey", (PyCFunction) BTree_maxKey, METH_VARARGS, "maxKey([max]) -> key\n\n" "Return the largest key in the BTree. If max is specified, return\n" "the largest key <= max."}, {"minKey", (PyCFunction) BTree_minKey, METH_VARARGS, "minKey([mi]) -> key\n\n" "Return the smallest key in the BTree. If min is specified, return\n" "the smallest key >= min."}, {"clear", (PyCFunction) BTree_clear, METH_NOARGS, "clear()\n\nRemove all of the items from the BTree."}, {"add", (PyCFunction)TreeSet_insert, METH_VARARGS, "add(id) -- Add an item to the set"}, {"insert", (PyCFunction)TreeSet_insert, METH_VARARGS, "insert(id) -- Add an item to the set"}, {"update", (PyCFunction)TreeSet_update, METH_VARARGS, "update(collection)\n\n Add the items from the given collection."}, {"remove", (PyCFunction)TreeSet_remove, METH_VARARGS, "remove(id) -- Remove a key from the set"}, {"_check", (PyCFunction) BTree_check, METH_NOARGS, "Perform sanity check on TreeSet, and raise exception if flawed."}, #ifdef PERSISTENT {"_p_resolveConflict", (PyCFunction) BTree__p_resolveConflict, METH_VARARGS, "_p_resolveConflict() -- Reinitialize from a newly created copy"}, {"_p_deactivate", (PyCFunction) BTree__p_deactivate, METH_VARARGS | METH_KEYWORDS, "_p_deactivate()\n\nReinitialize from a newly created copy."}, #endif {NULL, NULL} /* sentinel */ }; static PyMappingMethods TreeSet_as_mapping = { (lenfunc)BTree_length, /*mp_length*/ }; static PySequenceMethods TreeSet_as_sequence = { (lenfunc)0, /* sq_length */ (binaryfunc)0, /* sq_concat */ (ssizeargfunc)0, /* sq_repeat */ (ssizeargfunc)0, /* sq_item */ (ssizessizeargfunc)0, /* sq_slice */ (ssizeobjargproc)0, /* sq_ass_item */ (ssizessizeobjargproc)0, /* sq_ass_slice */ (objobjproc)BTree_contains, /* sq_contains */ 0, /* sq_inplace_concat */ 0, /* sq_inplace_repeat */ }; static int TreeSet_init(PyObject *self, PyObject *args, PyObject *kwds) { PyObject *v = NULL; if (!PyArg_ParseTuple(args, "|O:" MOD_NAME_PREFIX "TreeSet", &v)) return -1; if (v) return _TreeSet_update((BTree *)self, v); else return 0; } static PyTypeObject TreeSetType = { PyVarObject_HEAD_INIT(NULL, 0) MODULE_NAME MOD_NAME_PREFIX "TreeSet", /* tp_name */ sizeof(BTree), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)BTree_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ &BTree_as_number_for_nonzero, /* tp_as_number */ &TreeSet_as_sequence, /* tp_as_sequence */ &TreeSet_as_mapping, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ (traverseproc)BTree_traverse, /* tp_traverse */ (inquiry)BTree_tp_clear, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ (getiterfunc)BTree_getiter, /* tp_iter */ 0, /* tp_iternext */ TreeSet_methods, /* tp_methods */ BTree_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ TreeSet_init, /* tp_init */ 0, /* tp_alloc */ 0, /*PyType_GenericNew,*/ /* tp_new */ }; BTrees-4.3.1/BTrees/nosetests.xml0000664000175000017500000142636512307662071020033 0ustar tseavertseaver00000000000000BTrees-4.3.1/BTrees/fsBTree.py0000664000175000017500000000717212665316015017155 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001-2012 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # fsBTrees are data structures used for ZODB FileStorage. They are not # expected to be "public" excpect to FileStorage. # Each item in an fsBTree maps a two-byte key to a six-byte value. __all__ = ('Bucket', 'Set', 'BTree', 'TreeSet', 'fsBucket', 'fsSet', 'fsBTree', 'fsTreeSet', 'union', 'intersection', 'difference', ) from zope.interface import moduleProvides from .Interfaces import IIntegerObjectBTreeModule from ._base import Bucket from ._base import Set from ._base import Tree as BTree from ._base import TreeSet from ._base import difference as _difference from ._base import intersection as _intersection from ._base import set_operation as _set_operation from ._base import to_bytes as _to_bytes from ._base import union as _union from ._base import _fix_pickle _BUCKET_SIZE = 500 _TREE_SIZE = 500 using64bits = False _to_key = _to_bytes(2) _to_value = _to_bytes(6) class fsBucketPy(Bucket): _to_key = _to_key _to_value = _to_value def toString(self): return b''.join(self._keys) + b''.join(self._values) def fromString(self, v): length = len(v) if length % 8 != 0: raise ValueError() count = length // 8 keys, values = v[:count*2], v[count*2:] self.clear() while keys and values: key, keys = keys[:2], keys[2:] value, values = values[:6], values[6:] self._keys.append(key) self._values.append(value) return self class fsSetPy(Set): _to_key = _to_key class fsBTreePy(BTree): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key _to_value = _to_value class fsTreeSetPy(TreeSet): max_leaf_size = _BUCKET_SIZE max_internal_size = _TREE_SIZE _to_key = _to_key # Can't declare forward refs, so fix up afterwards: fsBucketPy._mapping_type = fsBucketPy._bucket_type = fsBucketPy fsBucketPy._set_type = fsSetPy fsSetPy._mapping_type = fsBucketPy fsSetPy._set_type = fsSetPy._bucket_type = fsSetPy fsBTreePy._mapping_type = fsBTreePy._bucket_type = fsBucketPy fsBTreePy._set_type = fsSetPy fsTreeSetPy._mapping_type = fsBucketPy fsTreeSetPy._set_type = fsTreeSetPy._bucket_type = fsSetPy differencePy = _set_operation(_difference, fsSetPy) unionPy = _set_operation(_union, fsSetPy) intersectionPy = _set_operation(_intersection, fsSetPy) try: from ._fsBTree import fsBucket except ImportError: #pragma NO COVER w/ C extensions fsBucket = fsBucketPy fsSet = fsSetPy fsBTree = fsBTreePy fsTreeSet = fsTreeSetPy difference = differencePy union = unionPy intersection = intersectionPy else: #pragma NO COVER w/o C extensions from ._fsBTree import fsSet from ._fsBTree import fsBTree from ._fsBTree import fsTreeSet from ._fsBTree import difference from ._fsBTree import union from ._fsBTree import intersection Bucket = fsBucket Set = fsSet BTree = fsBTree TreeSet = fsTreeSet _fix_pickle(globals(), __name__) moduleProvides(IIntegerObjectBTreeModule) BTrees-4.3.1/BTrees/Length.py0000664000175000017500000000343012510775554017043 0ustar tseavertseaver00000000000000############################################################################## # # Copyright (c) 2001, 2002 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import persistent class Length(persistent.Persistent): """BTree lengths are often too expensive to compute. Objects that use BTrees need to keep track of lengths themselves. This class provides an object for doing this. As a bonus, the object support application-level conflict resolution. It is tempting to to assign length objects to __len__ attributes to provide instance-specific __len__ methods. However, this no longer works as expected, because new-style classes cache class-defined slot methods (like __len__) in C type slots. Thus, instance-defined slot fillers are ignored. """ # class-level default required to keep copy.deepcopy happy -- see # https://bugs.launchpad.net/zodb/+bug/516653 value = 0 def __init__(self, v=0): self.value = v def __getstate__(self): return self.value def __setstate__(self, v): self.value = v def set(self, v): self.value = v def _p_resolveConflict(self, old, s1, s2): return s1 + s2 - old def change(self, delta): self.value += delta def __call__(self, *args): return self.value BTrees-4.3.1/.coveragerc0000664000175000017500000000013512510775554016204 0ustar tseavertseaver00000000000000[report] exclude_lines = # pragma: no cover class I[A-Z]\w+\((Interface|I[A-Z].*)\): BTrees-4.3.1/BTrees.egg-info/0000775000175000017500000000000012716353165016740 5ustar tseavertseaver00000000000000BTrees-4.3.1/BTrees.egg-info/top_level.txt0000664000175000017500000000000712716353161021463 0ustar tseavertseaver00000000000000BTrees BTrees-4.3.1/BTrees.egg-info/SOURCES.txt0000664000175000017500000000364312716353165020632 0ustar tseavertseaver00000000000000.coveragerc .gitignore .travis.yml CHANGES.rst COPYRIGHT.txt LICENSE.txt MANIFEST.in README.rst appveyor.yml bootstrap.py buildout.cfg setup.cfg setup.py tox.ini BTrees/BTreeItemsTemplate.c BTrees/BTreeModuleTemplate.c BTrees/BTreeTemplate.c BTrees/BucketTemplate.c BTrees/Development.txt BTrees/IFBTree.py BTrees/IIBTree.py BTrees/IOBTree.py BTrees/Interfaces.py BTrees/LFBTree.py BTrees/LLBTree.py BTrees/LOBTree.py BTrees/Length.py BTrees/MergeTemplate.c BTrees/OIBTree.py BTrees/OLBTree.py BTrees/OOBTree.py BTrees/SetOpTemplate.c BTrees/SetTemplate.c BTrees/TreeSetTemplate.c BTrees/_IFBTree.c BTrees/_IIBTree.c BTrees/_IOBTree.c BTrees/_LFBTree.c BTrees/_LLBTree.c BTrees/_LOBTree.c BTrees/_OIBTree.c BTrees/_OLBTree.c BTrees/_OOBTree.c BTrees/__init__.py BTrees/_base.py BTrees/_compat.h BTrees/_compat.py BTrees/_fsBTree.c BTrees/check.py BTrees/floatvaluemacros.h BTrees/fsBTree.py BTrees/intkeymacros.h BTrees/intvaluemacros.h BTrees/nosetests.xml BTrees/objectkeymacros.h BTrees/objectvaluemacros.h BTrees/sorters.c BTrees/utils.py BTrees.egg-info/PKG-INFO BTrees.egg-info/SOURCES.txt BTrees.egg-info/dependency_links.txt BTrees.egg-info/entry_points.txt BTrees.egg-info/not-zip-safe BTrees.egg-info/requires.txt BTrees.egg-info/top_level.txt BTrees/tests/__init__.py BTrees/tests/common.py BTrees/tests/testBTrees.py BTrees/tests/testBTreesUnicode.py BTrees/tests/testConflict.py BTrees/tests/test_IFBTree.py BTrees/tests/test_IIBTree.py BTrees/tests/test_IOBTree.py BTrees/tests/test_LFBTree.py BTrees/tests/test_LLBTree.py BTrees/tests/test_LOBTree.py BTrees/tests/test_Length.py BTrees/tests/test_OIBTree.py BTrees/tests/test_OLBTree.py BTrees/tests/test_OOBTree.py BTrees/tests/test__base.py BTrees/tests/test_btreesubclass.py BTrees/tests/test_check.py BTrees/tests/test_fsBTree.py BTrees/tests/test_utils.py docs/Makefile docs/api.rst docs/conf.py docs/index.rst docs/make.bat docs/_static/placeholder.txt docs/_templates/placeholder.txtBTrees-4.3.1/BTrees.egg-info/PKG-INFO0000664000175000017500000002143312716353161020034 0ustar tseavertseaver00000000000000Metadata-Version: 1.1 Name: BTrees Version: 4.3.1 Summary: Scalable persistent object containers Home-page: http://packages.python.org/BTrees Author: Zope Foundation Author-email: zodb-dev@zope.org License: ZPL 2.1 Description: ``BTrees``: scalable persistent components =========================================== .. image:: https://travis-ci.org/zopefoundation/BTrees.svg?branch=master :target: https://travis-ci.org/zopefoundation/BTrees .. image:: https://ci.appveyor.com/api/projects/status/github/zopefoundation/BTrees?branch=master&svg=true :target: https://ci.appveyor.com/project/mgedmin/BTrees This package contains a set of persistent object containers built around a modified BTree data structure. The trees are optimized for use inside ZODB's "optimistic concurrency" paradigm, and include explicit resolution of conflicts detected by that mechannism. Please see the Sphinx documentation (``docs/index.rst``) for further information. ``BTrees`` Changelog ==================== 4.3.1 (2016-05-16) ------------------ - Packaging: fix password used to automate wheel creation on Travis. 4.3.0 (2016-05-10) ------------------ - Fix unexpected ``OverflowError`` when passing 64bit values to long keys / values on Win64. See: https://github.com/zopefoundation/BTrees/issues/32 - When testing ``PURE_PYTHON`` environments under ``tox``, avoid poisoning the user's global wheel cache. - Ensure that he pure-Python implementation, used on PyPy and when a C compiler isn't available for CPython, pickles identically to the C version. Unpickling will choose the best available implementation. This change prevents interoperability problems and database corruption if both implementations are in use. While it is no longer possible to pickle a Python implementation and have it unpickle to the Python implementation if the C implementation is available, existing Python pickles will still unpickle to the Python implementation (until pickled again). See: https://github.com/zopefoundation/BTrees/issues/19 - Avoid creating invalid objects when unpickling empty BTrees in a pure-Python environment. - Drop support for Python 2.6 and 3.2. 4.2.0 (2015-11-13) ------------------ - Add support for Python 3.5. 4.1.4 (2015-06-02) ------------------ - Ensure that pure-Python Bucket and Set objects have a human readable ``__repr__`` like the C versions. 4.1.3 (2015-05-19) ------------------ - Fix ``_p_changed`` when removing items from small pure-Python BTrees/TreeSets and when adding items to small pure-Python Sets. See: https://github.com/zopefoundation/BTrees/issues/13 4.1.2 (2015-04-07) ------------------ - Suppress testing 64-bit values in OLBTrees on 32 bit machines. See: https://github.com/zopefoundation/BTrees/issues/9 - Fix ``_p_changed`` when adding items to small pure-Python BTrees/TreeSets. See: https://github.com/zopefoundation/BTrees/issues/11 4.1.1 (2014-12-27) ------------------ - Accomodate long values in pure-Python OLBTrees. 4.1.0 (2014-12-26) ------------------ - Add support for PyPy and PyPy3. - Add support for Python 3.4. - BTree subclasses can define ``max_leaf_size`` or ``max_internal_size`` to control maximum sizes for Bucket/Set and BTree/TreeSet nodes. - Detect integer overflow on 32-bit machines correctly under Python 3. - Update pure-Python and C trees / sets to accept explicit None to indicate max / min value for ``minKey``, ``maxKey``. (PR #3) - Update pure-Python trees / sets to accept explicit None to indicate open ranges for ``keys``, ``values``, ``items``. (PR #3) 4.0.8 (2013-05-25) ------------------ - Fix value-based comparison for objects under Py3k: addresses invalid merges of ``[OLI]OBTrees/OBuckets``. - Ensure that pure-Python implementation of ``OOBTree.byValue`` matches semantics (reversed-sort) of C implementation. 4.0.7 (2013-05-22) ------------------ - Issue #2: compilation error on 32-bit mode of OS/X. - Test ``PURE_PYTHON`` environment variable support: if set, the C extensions will not be built, imported, or tested. 4.0.6 (2013-05-14) ------------------ - Changed the ``ZODB`` extra to require only the real ``ZODB`` package, rather than the ``ZODB3`` metapackage: depending on the version used, the metapackage could pull in stale versions of **this** package and ``persistent``. - Fixed Python version check in ``setup.py``. 4.0.5 (2013-01-15) ------------------ - Fit the ``repr`` of bucket objects, which could contain garbage characters. 4.0.4 (2013-01-12) ------------------ - Emulate the (private) iterators used by the C extension modules from pure Python. This change is "cosmetic" only: it prevents the ZCML ``zope.app.security:permission.zcml`` from failing. The emulated classes are **not** functional, and should be considered implementation details. - Accomodate buildout to the fact that we no longer bundle a copy of 'persistent.h'. - Fix test failures on Windows: no longer rely on overflows from ``sys.maxint``. 4.0.3 (2013-01-04) ------------------ - Added ``setup_requires==['persistent']``. 4.0.2 (2013-01-03) ------------------ - Updated Trove classifiers. - Added explicit support for Python 3.2, Python 3.3, and PyPy. Note that the C extensions are not (yet) available on PyPy. - Python reference implementations now tested separately from the C verions on all platforms. - 100% unit test coverage. 4.0.1 (2012-10-21) ------------------ - Provide local fallback for persistent C header inclusion if the persistent distribution isn't installed. This makes the winbot happy. 4.0.0 (2012-10-20) ------------------ Platform Changes ################ - Dropped support for Python < 2.6. - Factored ``BTrees`` as a separate distribution. Testing Changes ############### - All covered platforms tested under ``tox``. - Added support for continuous integration using ``tox`` and ``jenkins``. - Added ``setup.py dev`` alias (installs ``nose`` and ``coverage``). - Dropped dependency on ``zope.testing`` / ``zope.testrunner``: tests now run with ``setup.py test``. Documentation Changes ##################### - Added API reference, generated via Spinx' autodoc. - Added Sphinx documentation based on ZODB Guide (snippets are exercised via 'tox'). - Added ``setup.py docs`` alias (installs ``Sphinx`` and ``repoze.sphinx.autointerface``). Platform: any Classifier: Development Status :: 6 - Mature Classifier: License :: OSI Approved :: Zope Public License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Framework :: ZODB Classifier: Topic :: Database Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: Unix BTrees-4.3.1/BTrees.egg-info/dependency_links.txt0000664000175000017500000000000112716353161023002 0ustar tseavertseaver00000000000000 BTrees-4.3.1/BTrees.egg-info/not-zip-safe0000664000175000017500000000000112125365116021157 0ustar tseavertseaver00000000000000 BTrees-4.3.1/BTrees.egg-info/requires.txt0000664000175000017500000000027712716353161021342 0ustar tseavertseaver00000000000000persistent zope.interface [ZODB] ZODB [docs] Sphinx repoze.sphinx.autointerface [test] persistent zope.interface transaction [testing] persistent zope.interface transaction nose coverage BTrees-4.3.1/BTrees.egg-info/entry_points.txt0000664000175000017500000000000612716353161022226 0ustar tseavertseaver00000000000000 BTrees-4.3.1/COPYRIGHT.txt0000664000175000017500000000004012510775554016167 0ustar tseavertseaver00000000000000Zope Foundation and ContributorsBTrees-4.3.1/CHANGES.rst0000664000175000017500000001251212716353101015653 0ustar tseavertseaver00000000000000``BTrees`` Changelog ==================== 4.3.1 (2016-05-16) ------------------ - Packaging: fix password used to automate wheel creation on Travis. 4.3.0 (2016-05-10) ------------------ - Fix unexpected ``OverflowError`` when passing 64bit values to long keys / values on Win64. See: https://github.com/zopefoundation/BTrees/issues/32 - When testing ``PURE_PYTHON`` environments under ``tox``, avoid poisoning the user's global wheel cache. - Ensure that he pure-Python implementation, used on PyPy and when a C compiler isn't available for CPython, pickles identically to the C version. Unpickling will choose the best available implementation. This change prevents interoperability problems and database corruption if both implementations are in use. While it is no longer possible to pickle a Python implementation and have it unpickle to the Python implementation if the C implementation is available, existing Python pickles will still unpickle to the Python implementation (until pickled again). See: https://github.com/zopefoundation/BTrees/issues/19 - Avoid creating invalid objects when unpickling empty BTrees in a pure-Python environment. - Drop support for Python 2.6 and 3.2. 4.2.0 (2015-11-13) ------------------ - Add support for Python 3.5. 4.1.4 (2015-06-02) ------------------ - Ensure that pure-Python Bucket and Set objects have a human readable ``__repr__`` like the C versions. 4.1.3 (2015-05-19) ------------------ - Fix ``_p_changed`` when removing items from small pure-Python BTrees/TreeSets and when adding items to small pure-Python Sets. See: https://github.com/zopefoundation/BTrees/issues/13 4.1.2 (2015-04-07) ------------------ - Suppress testing 64-bit values in OLBTrees on 32 bit machines. See: https://github.com/zopefoundation/BTrees/issues/9 - Fix ``_p_changed`` when adding items to small pure-Python BTrees/TreeSets. See: https://github.com/zopefoundation/BTrees/issues/11 4.1.1 (2014-12-27) ------------------ - Accomodate long values in pure-Python OLBTrees. 4.1.0 (2014-12-26) ------------------ - Add support for PyPy and PyPy3. - Add support for Python 3.4. - BTree subclasses can define ``max_leaf_size`` or ``max_internal_size`` to control maximum sizes for Bucket/Set and BTree/TreeSet nodes. - Detect integer overflow on 32-bit machines correctly under Python 3. - Update pure-Python and C trees / sets to accept explicit None to indicate max / min value for ``minKey``, ``maxKey``. (PR #3) - Update pure-Python trees / sets to accept explicit None to indicate open ranges for ``keys``, ``values``, ``items``. (PR #3) 4.0.8 (2013-05-25) ------------------ - Fix value-based comparison for objects under Py3k: addresses invalid merges of ``[OLI]OBTrees/OBuckets``. - Ensure that pure-Python implementation of ``OOBTree.byValue`` matches semantics (reversed-sort) of C implementation. 4.0.7 (2013-05-22) ------------------ - Issue #2: compilation error on 32-bit mode of OS/X. - Test ``PURE_PYTHON`` environment variable support: if set, the C extensions will not be built, imported, or tested. 4.0.6 (2013-05-14) ------------------ - Changed the ``ZODB`` extra to require only the real ``ZODB`` package, rather than the ``ZODB3`` metapackage: depending on the version used, the metapackage could pull in stale versions of **this** package and ``persistent``. - Fixed Python version check in ``setup.py``. 4.0.5 (2013-01-15) ------------------ - Fit the ``repr`` of bucket objects, which could contain garbage characters. 4.0.4 (2013-01-12) ------------------ - Emulate the (private) iterators used by the C extension modules from pure Python. This change is "cosmetic" only: it prevents the ZCML ``zope.app.security:permission.zcml`` from failing. The emulated classes are **not** functional, and should be considered implementation details. - Accomodate buildout to the fact that we no longer bundle a copy of 'persistent.h'. - Fix test failures on Windows: no longer rely on overflows from ``sys.maxint``. 4.0.3 (2013-01-04) ------------------ - Added ``setup_requires==['persistent']``. 4.0.2 (2013-01-03) ------------------ - Updated Trove classifiers. - Added explicit support for Python 3.2, Python 3.3, and PyPy. Note that the C extensions are not (yet) available on PyPy. - Python reference implementations now tested separately from the C verions on all platforms. - 100% unit test coverage. 4.0.1 (2012-10-21) ------------------ - Provide local fallback for persistent C header inclusion if the persistent distribution isn't installed. This makes the winbot happy. 4.0.0 (2012-10-20) ------------------ Platform Changes ################ - Dropped support for Python < 2.6. - Factored ``BTrees`` as a separate distribution. Testing Changes ############### - All covered platforms tested under ``tox``. - Added support for continuous integration using ``tox`` and ``jenkins``. - Added ``setup.py dev`` alias (installs ``nose`` and ``coverage``). - Dropped dependency on ``zope.testing`` / ``zope.testrunner``: tests now run with ``setup.py test``. Documentation Changes ##################### - Added API reference, generated via Spinx' autodoc. - Added Sphinx documentation based on ZODB Guide (snippets are exercised via 'tox'). - Added ``setup.py docs`` alias (installs ``Sphinx`` and ``repoze.sphinx.autointerface``). BTrees-4.3.1/LICENSE.txt0000664000175000017500000000402612510775554015711 0ustar tseavertseaver00000000000000Zope Public License (ZPL) Version 2.1 A copyright notice accompanies this license document that identifies the copyright holders. This license has been certified as open source. It has also been designated as GPL compatible by the Free Software Foundation (FSF). Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions in source code must retain the accompanying copyright notice, this list of conditions, and the following disclaimer. 2. Redistributions in binary form must reproduce the accompanying copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Names of the copyright holders must not be used to endorse or promote products derived from this software without prior written permission from the copyright holders. 4. The right to distribute this software or to use it for any purpose does not give you the right to use Servicemarks (sm) or Trademarks (tm) of the copyright holders. Use of them is covered by separate agreement with the copyright holders. 5. If any files are modified, you must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. Disclaimer THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. BTrees-4.3.1/README.rst0000664000175000017500000000133512714130453015542 0ustar tseavertseaver00000000000000``BTrees``: scalable persistent components =========================================== .. image:: https://travis-ci.org/zopefoundation/BTrees.svg?branch=master :target: https://travis-ci.org/zopefoundation/BTrees .. image:: https://ci.appveyor.com/api/projects/status/github/zopefoundation/BTrees?branch=master&svg=true :target: https://ci.appveyor.com/project/mgedmin/BTrees This package contains a set of persistent object containers built around a modified BTree data structure. The trees are optimized for use inside ZODB's "optimistic concurrency" paradigm, and include explicit resolution of conflicts detected by that mechannism. Please see the Sphinx documentation (``docs/index.rst``) for further information. BTrees-4.3.1/docs/0000775000175000017500000000000012716353165015012 5ustar tseavertseaver00000000000000BTrees-4.3.1/docs/api.rst0000664000175000017500000000353412510775554016324 0ustar tseavertseaver00000000000000:mod:`BTrees` API Reference =========================== Protocol APIs ------------- .. automodule:: BTrees.Interfaces .. autointerface:: ICollection :members: :member-order: bysource .. autointerface:: IReadSequence :members: :member-order: bysource .. autointerface:: IKeyed :members: :member-order: bysource .. autointerface:: ISetMutable :members: :member-order: bysource .. autointerface:: ISized :members: :member-order: bysource .. autointerface:: IKeySequence :members: :member-order: bysource .. autointerface:: IMinimalDictionary :members: :member-order: bysource .. autointerface:: IDictionaryIsh :members: :member-order: bysource .. autointerface:: IMerge :members: :member-order: bysource .. autointerface:: IIMerge :members: :member-order: bysource .. autointerface:: IMergeIntegerKey :members: :member-order: bysource BTree Family APIs ----------------- .. autointerface:: ISet :members: :member-order: bysource .. autointerface:: ITreeSet :members: :member-order: bysource .. autointerface:: IBTree :members: :member-order: bysource .. autointerface:: IBTreeFamily :members: :member-order: bysource Module APIs ----------- .. autointerface:: IBTreeModule :members: :member-order: bysource .. autointerface:: IObjectObjectBTreeModule :members: :member-order: bysource .. autointerface:: IIntegerObjectBTreeModule :members: :member-order: bysource .. autointerface:: IObjectIntegerBTreeModule :members: :member-order: bysource .. autointerface:: IIntegerIntegerBTreeModule :members: :member-order: bysource .. autointerface:: IIntegerFloatBTreeModule :members: :member-order: bysource BTrees-4.3.1/docs/make.bat0000664000175000017500000001175012510775554016425 0ustar tseavertseaver00000000000000@ECHO OFF REM Command file for Sphinx documentation if "%SPHINXBUILD%" == "" ( set SPHINXBUILD=sphinx-build ) set BUILDDIR=_build set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . set I18NSPHINXOPTS=%SPHINXOPTS% . if NOT "%PAPER%" == "" ( set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% ) if "%1" == "" goto help if "%1" == "help" ( :help echo.Please use `make ^` where ^ is one of echo. html to make standalone HTML files echo. dirhtml to make HTML files named index.html in directories echo. singlehtml to make a single large HTML file echo. pickle to make pickle files echo. json to make JSON files echo. htmlhelp to make HTML files and a HTML help project echo. qthelp to make HTML files and a qthelp project echo. devhelp to make HTML files and a Devhelp project echo. epub to make an epub echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter echo. text to make text files echo. man to make manual pages echo. texinfo to make Texinfo files echo. gettext to make PO message catalogs echo. changes to make an overview over all changed/added/deprecated items echo. linkcheck to check all external links for integrity echo. doctest to run all doctests embedded in the documentation if enabled goto end ) if "%1" == "clean" ( for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i del /q /s %BUILDDIR%\* goto end ) if "%1" == "html" ( %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html if errorlevel 1 exit /b 1 echo. echo.Build finished. The HTML pages are in %BUILDDIR%/html. goto end ) if "%1" == "dirhtml" ( %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml if errorlevel 1 exit /b 1 echo. echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. goto end ) if "%1" == "singlehtml" ( %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml if errorlevel 1 exit /b 1 echo. echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. goto end ) if "%1" == "pickle" ( %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can process the pickle files. goto end ) if "%1" == "json" ( %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can process the JSON files. goto end ) if "%1" == "htmlhelp" ( %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can run HTML Help Workshop with the ^ .hhp project file in %BUILDDIR%/htmlhelp. goto end ) if "%1" == "qthelp" ( %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can run "qcollectiongenerator" with the ^ .qhcp project file in %BUILDDIR%/qthelp, like this: echo.^> qcollectiongenerator %BUILDDIR%\qthelp\BTrees.qhcp echo.To view the help file: echo.^> assistant -collectionFile %BUILDDIR%\qthelp\BTrees.ghc goto end ) if "%1" == "devhelp" ( %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp if errorlevel 1 exit /b 1 echo. echo.Build finished. goto end ) if "%1" == "epub" ( %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub if errorlevel 1 exit /b 1 echo. echo.Build finished. The epub file is in %BUILDDIR%/epub. goto end ) if "%1" == "latex" ( %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex if errorlevel 1 exit /b 1 echo. echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. goto end ) if "%1" == "text" ( %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text if errorlevel 1 exit /b 1 echo. echo.Build finished. The text files are in %BUILDDIR%/text. goto end ) if "%1" == "man" ( %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man if errorlevel 1 exit /b 1 echo. echo.Build finished. The manual pages are in %BUILDDIR%/man. goto end ) if "%1" == "texinfo" ( %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo if errorlevel 1 exit /b 1 echo. echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. goto end ) if "%1" == "gettext" ( %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale if errorlevel 1 exit /b 1 echo. echo.Build finished. The message catalogs are in %BUILDDIR%/locale. goto end ) if "%1" == "changes" ( %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes if errorlevel 1 exit /b 1 echo. echo.The overview file is in %BUILDDIR%/changes. goto end ) if "%1" == "linkcheck" ( %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck if errorlevel 1 exit /b 1 echo. echo.Link check complete; look for any errors in the above output ^ or in %BUILDDIR%/linkcheck/output.txt. goto end ) if "%1" == "doctest" ( %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest if errorlevel 1 exit /b 1 echo. echo.Testing of doctests in the sources finished, look at the ^ results in %BUILDDIR%/doctest/output.txt. goto end ) :end BTrees-4.3.1/docs/_templates/0000775000175000017500000000000012716353165017147 5ustar tseavertseaver00000000000000BTrees-4.3.1/docs/_templates/placeholder.txt0000664000175000017500000000000012510775554022162 0ustar tseavertseaver00000000000000BTrees-4.3.1/docs/Makefile0000664000175000017500000001267412510775554016466 0ustar tseavertseaver00000000000000# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build PAPER = BUILDDIR = _build # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " text to make text files" @echo " man to make manual pages" @echo " texinfo to make Texinfo files" @echo " info to make Texinfo files and run them through makeinfo" @echo " gettext to make PO message catalogs" @echo " changes to make an overview of all changed/added/deprecated items" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: -rm -rf $(BUILDDIR)/* html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/BTrees.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/BTrees.qhc" devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/BTrees" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/BTrees" @echo "# devhelp" epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." @echo "Run \`make' in that directory to run these through makeinfo" \ "(use \`make info' here to do that automatically)." info: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo "Running Texinfo files through makeinfo..." make -C $(BUILDDIR)/texinfo info @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." gettext: $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale @echo @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." BTrees-4.3.1/docs/_static/0000775000175000017500000000000012716353165016440 5ustar tseavertseaver00000000000000BTrees-4.3.1/docs/_static/placeholder.txt0000664000175000017500000000000012510775554021453 0ustar tseavertseaver00000000000000BTrees-4.3.1/docs/conf.py0000664000175000017500000001762712510775554016330 0ustar tseavertseaver00000000000000# -*- coding: utf-8 -*- # # BTrees documentation build configuration file, created by # sphinx-quickstart on Thu Oct 18 00:16:24 2012. # # This file is execfile()d with the current directory set to its containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. #import sys, os # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) # -- General configuration ----------------------------------------------------- # If your documentation needs a minimal Sphinx version, state it here. #needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx.ext.viewcode', 'repoze.sphinx.autointerface', ] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = u'BTrees' copyright = u'2012, Zope Foundation Contributors' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. version = '4.0' # The full version, including alpha/beta/rc tags. release = '4.0.2dev' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. #language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = ['_build'] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'default' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. #html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_domain_indices = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. #html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. #html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = None # Output file base name for HTML help builder. htmlhelp_basename = 'BTreesdoc' # -- Options for LaTeX output -------------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). #'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). #'pointsize': '10pt', # Additional stuff for the LaTeX preamble. #'preamble': '', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ ('index', 'BTrees.tex', u'BTrees Documentation', u'Zope Foundation Contributors', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # If true, show page references after internal links. #latex_show_pagerefs = False # If true, show URL addresses after external links. #latex_show_urls = False # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_domain_indices = True # -- Options for manual page output -------------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('index', 'btrees', u'BTrees Documentation', [u'Zope Foundation Contributors'], 1) ] # If true, show URL addresses after external links. #man_show_urls = False # -- Options for Texinfo output ------------------------------------------------ # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ('index', 'BTrees', u'BTrees Documentation', u'Zope Foundation Contributors', 'BTrees', 'One line description of project.', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. #texinfo_appendices = [] # If false, no module index is generated. #texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. #texinfo_show_urls = 'footnote' # Example configuration for intersphinx: refer to the Python standard library. intersphinx_mapping = {'http://docs.python.org/': None} BTrees-4.3.1/docs/index.rst0000664000175000017500000004725712510775554016674 0ustar tseavertseaver00000000000000:mod:`BTrees` Documentation =========================== This package contains a set of persistent object containers built around a modified BTree data structure. The trees are optimized for use inside ZODB's "optimistic concurrency" paradigm, and include explicit resolution of conflicts detected by that mechannism. Contents: .. toctree:: :maxdepth: 2 api Overview -------- When programming with the ZODB, Python dictionaries aren't always what you need. The most important case is where you want to store a very large mapping. When a Python dictionary is accessed in a ZODB, the whole dictionary has to be unpickled and brought into memory. If you're storing something very large, such as a 100,000-entry user database, unpickling such a large object will be slow. BTrees are a balanced tree data structure that behave like a mapping but distribute keys throughout a number of tree nodes. The nodes are stored in sorted order (this has important consequences -- see below). Nodes are then only unpickled and brought into memory as they're accessed, so the entire tree doesn't have to occupy memory (unless you really are touching every single key). The BTrees package provides a large collection of related data structures. There are variants of the data structures specialized to integers, which are faster and use less memory. There are five modules that handle the different variants. The first two letters of the module name specify the types of the keys and values in mappings -- O for any object, I for 32-bit signed integer, and (new in ZODB 3.4) F for 32-bit C float. For example, the :mod:`BTrees.IOBTree` module provides a mapping with integer keys and arbitrary objects as values. The four data structures provide by each module are a BTree, a Bucket, a TreeSet, and a Set. The BTree and Bucket types are mappings and support all the usual mapping methods, e.g. :func:`update` and :func:`keys`. The TreeSet and Set types are similar to mappings but they have no values; they support the methods that make sense for a mapping with no keys, e.g. :func:`keys` but not :func:`items`. The Bucket and Set types are the individual building blocks for BTrees and TreeSets, respectively. A Bucket or Set can be used when you are sure that it will have few elements. If the data structure will grow large, you should use a BTree or TreeSet. Like Python lists, Buckets and Sets are allocated in one contiguous piece, and insertions and deletions can take time proportional to the number of existing elements. Also like Python lists, a Bucket or Set is a single object, and is pickled and unpickled in its entirety. BTrees and TreeSets are multi-level tree structures with much better (logarithmic) worst- case time bounds, and the tree structure is built out of multiple objects, which ZODB can load individually as needed. The five modules are named :mod:`OOBTree`, :mod:`IOBTree`, :mod:`OIBTree`, :mod:`IIBTree`, and (new in ZODB 3.4) :mod:`IFBTree`. The two letter prefixes are repeated in the data types names. The :mod:`BTrees.OOBTree` module defines the following types: :class:`OOBTree`, :class:`OOBucket`, :class:`OOSet`, and :class:`OOTreeSet`. Similarly, the other four modules each define their own variants of those four types. The :func:`keys`, :func:`values`, and :func:`items` methods on BTree and TreeSet types do not materialize a list with all of the data. Instead, they return lazy sequences that fetch data from the BTree as needed. They also support optional arguments to specify the minimum and maximum values to return, often called "range searching". Because all these types are stored in sorted order, range searching is very efficient. The :func:`keys`, :func:`values`, and :func:`items` methods on Bucket and Set types do return lists with all the data. Starting in ZODB 3.3, there are also :func:`iterkeys`, :func:`itervalues`, and :func:`iteritems` methods that return iterators (in the Python 2.2 sense). Those methods also apply to BTree and TreeSet objects. A BTree object supports all the methods you would expect of a mapping, with a few extensions that exploit the fact that the keys are sorted. The example below demonstrates how some of the methods work. The extra methods are :func:`minKey` and :func:`maxKey`, which find the minimum and maximum key value subject to an optional bound argument, and :func:`byValue`, which should probably be ignored (it's hard to explain exactly what it does, and as a result it's almost never used -- best to consider it deprecated). The various methods for enumerating keys, values and items also accept minimum and maximum key arguments ("range search"), and (new in ZODB 3.3) optional Boolean arguments to control whether a range search is inclusive or exclusive of the range's endpoints. .. doctest:: >>> from BTrees.OOBTree import OOBTree >>> t = OOBTree() >>> t.update({1: "red", 2: "green", 3: "blue", 4: "spades"}) >>> len(t) 4 >>> t[2] 'green' >>> s = t.keys() # this is a "lazy" sequence object >>> s >>> len(s) # it acts like a Python list 4 >>> s[-2] 3 >>> list(s) # materialize the full list [1, 2, 3, 4] >>> list(t.values()) ['red', 'green', 'blue', 'spades'] >>> list(t.values(1, 2)) # values at keys in 1 to 2 inclusive ['red', 'green'] >>> list(t.values(2)) # values at keys >= 2 ['green', 'blue', 'spades'] >>> list(t.values(min=1, max=4)) # keyword args new in ZODB 3.3 ['red', 'green', 'blue', 'spades'] >>> list(t.values(min=1, max=4, excludemin=True, excludemax=True)) ['green', 'blue'] >>> t.minKey() # smallest key 1 >>> t.minKey(1.5) # smallest key >= 1.5 2 >>> for k in t.keys(): ... print k, 1 2 3 4 >>> for k in t: # new in ZODB 3.3 ... print k, 1 2 3 4 >>> for pair in t.iteritems(): # new in ZODB 3.3 ... print pair, ... (1, 'red') (2, 'green') (3, 'blue') (4, 'spades') >>> t.has_key(4) # returns a true value, but exactly what undefined 2 >>> t.has_key(5) 0 >>> 4 in t # new in ZODB 3.3 True >>> 5 in t # new in ZODB 3.3 False >>> Each of the modules also defines some functions that operate on BTrees -- :func:`difference`, :func:`union`, and :func:`intersection`. The :func:`difference` function returns a Bucket, while the other two methods return a Set. If the keys are integers, then the module also defines :func:`multiunion`. If the values are integers or floats, then the module also defines :func:`weightedIntersection` and :func:`weightedUnion`. The function doc strings describe each function briefly. .. % XXX I'm not sure all of the following is actually correct. The .. % XXX set functions have complicated behavior. ``BTrees/Interfaces.py`` defines the operations, and is the official documentation. Note that the interfaces don't define the concrete types returned by most operations, and you shouldn't rely on the concrete types that happen to be returned: stick to operations guaranteed by the interface. In particular, note that the interfaces don't specify anything about comparison behavior, and so nothing about it is guaranteed. In ZODB 3.3, for example, two BTrees happen to use Python's default object comparison, which amounts to comparing the (arbitrary but fixed) memory addresses of the BTrees. This may or may not be true in future releases. If the interfaces don't specify a behavior, then whether that behavior appears to work, and exactly happens if it does appear to work, are undefined and should not be relied on. Total Ordering and Persistence ------------------------------ The BTree-based data structures differ from Python dicts in several fundamental ways. One of the most important is that while dicts require that keys support hash codes and equality comparison, the BTree-based structures don't use hash codes and require a total ordering on keys. Total ordering means three things: #. Reflexive. For each *x*, ``x == x`` is true. #. Trichotomy. For each *x* and *y*, exactly one of ``x < y``, ``x == y``, and ``x > y`` is true. #. Transitivity. Whenever ``x <= y`` and ``y <= z``, it's also true that ``x <= z``. The default comparison functions for most objects that come with Python satisfy these rules, with some crucial cautions explained later. Complex numbers are an example of an object whose default comparison function does not satisfy these rules: complex numbers only support ``==`` and ``!=`` comparisons, and raise an exception if you try to compare them in any other way. They don't satisfy the trichotomy rule, and must not be used as keys in BTree-based data structures (although note that complex numbers can be used as keys in Python dicts, which do not require a total ordering). Examples of objects that are wholly safe to use as keys in BTree-based structures include ints, longs, floats, 8-bit strings, Unicode strings, and tuples composed (possibly recursively) of objects of wholly safe types. It's important to realize that even if two types satisfy the rules on their own, mixing objects of those types may not. For example, 8-bit strings and Unicode strings both supply total orderings, but mixing the two loses trichotomy; e.g., ``'x' < chr(255)`` and ``u'x' == 'x'``, but trying to compare ``chr(255)`` to ``u'x'`` raises an exception. Partly for this reason (another is given later), it can be dangerous to use keys with multiple types in a single BTree-based structure. Don't try to do that, and you don't have to worry about it. Another potential problem is mutability: when a key is inserted in a BTree- based structure, it must retain the same order relative to the other keys over time. This is easy to run afoul of if you use mutable objects as keys. For example, lists supply a total ordering, and then .. doctest:: >>> L1, L2, L3 = [1], [2], [3] >>> from BTrees.OOBTree import OOSet >>> s = OOSet((L2, L3, L1)) # this is fine, so far >>> list(s.keys()) # note that the lists are in sorted order [[1], [2], [3]] >>> s.has_key([3]) # and [3] is in the set 1 >>> L2[0] = 5 # horrible -- the set is insane now >>> s.has_key([3]) # for example, it's insane this way 0 >>> s OOSet([[1], [5], [3]]) >>> Key lookup relies on that the keys remain in sorted order (an efficient form of binary search is used). By mutating key L2 after inserting it, we destroyed the invariant that the OOSet is sorted. As a result, all future operations on this set are unpredictable. A subtler variant of this problem arises due to persistence: by default, Python does several kinds of comparison by comparing the memory addresses of two objects. Because Python never moves an object in memory, this does supply a usable (albeit arbitrary) total ordering across the life of a program run (an object's memory address doesn't change). But if objects compared in this way are used as keys of a BTree-based structure that's stored in a database, when the objects are loaded from the database again they will almost certainly wind up at different memory addresses. There's no guarantee then that if key K1 had a memory address smaller than the memory address of key K2 at the time K1 and K2 were inserted in a BTree, K1's address will also be smaller than K2's when that BTree is loaded from a database later. The result will be an insane BTree, where various operations do and don't work as expected, seemingly at random. Now each of the types identified above as "wholly safe to use" never compares two instances of that type by memory address, so there's nothing to worry about here if you use keys of those types. The most common mistake is to use keys that are instances of a user-defined class that doesn't supply its own :meth:`__cmp__` method. Python compares such instances by memory address. This is fine if such instances are used as keys in temporary BTree-based structures used only in a single program run. It can be disastrous if that BTree-based structure is stored to a database, though. .. doctest:: :options: +SKIP >>> class C: ... pass ... >>> a, b = C(), C() >>> print a < b # this may print 0 if you try it True >>> del a, b >>> a, b = C(), C() >>> print a < b # and this may print 0 or 1 False >>> That example illustrates that comparison of instances of classes that don't define :meth:`__cmp__` yields arbitrary results (but consistent results within a single program run). Another problem occurs with instances of classes that do define :meth:`__cmp__`, but define it incorrectly. It's possible but rare for a custom :meth:`__cmp__` implementation to violate one of the three required formal properties directly. It's more common for it to "fall back" to address-based comparison by mistake. For example, .. doctest:: class Mine: def __cmp__(self, other): if other.__class__ is Mine: return cmp(self.data, other.data) else: return cmp(self.data, other) It's quite possible there that the :keyword:`else` clause allows a result to be computed based on memory address. The bug won't show up until a BTree-based structure uses objects of class :class:`Mine` as keys, and also objects of other types as keys, and the structure is loaded from a database, and a sequence of comparisons happens to execute the :keyword:`else` clause in a case where the relative order of object memory addresses happened to change. This is as difficult to track down as it sounds, so best to stay far away from the possibility. You'll stay out of trouble by follwing these rules, violating them only with great care: #. Use objects of simple immutable types as keys in BTree-based data structures. #. Within a single BTree-based data structure, use objects of a single type as keys. Don't use multiple key types in a single structure. #. If you want to use class instances as keys, and there's any possibility that the structure may be stored in a database, it's crucial that the class define a :meth:`__cmp__` method, and that the method is carefully implemented. Any part of a comparison implementation that relies (explicitly or implicitly) on an address-based comparison result will eventually cause serious failure. #. Do not use :class:`Persistent` objects as keys, or objects of a subclass of :class:`Persistent`. That last item may be surprising. It stems from details of how conflict resolution is implemented: the states passed to conflict resolution do not materialize persistent subobjects (if a persistent object P is a key in a BTree, then P is a subobject of the bucket containing P). Instead, if an object O references a persistent subobject P directly, and O is involved in a conflict, the states passed to conflict resolution contain an instance of an internal :class:`PersistentReference` stub class everywhere O references P. Two :class:`PersistentReference` instances compare equal if and only if they "represent" the same persistent object; when they're not equal, they compare by memory address, and, as explained before, memory-based comparison must never happen in a sane persistent BTree. Note that it doesn't help in this case if your :class:`Persistent` subclass defines a sane :meth:`__cmp__` method: conflict resolution doesn't know about your class, and so also doesn't know about its :meth:`__cmp__` method. It only sees instances of the internal :class:`PersistentReference` stub class. Iteration and Mutation ---------------------- As with a Python dictionary or list, you should not mutate a BTree-based data structure while iterating over it, except that it's fine to replace the value associated with an existing key while iterating. You won't create internal damage in the structure if you try to remove, or add new keys, while iterating, but the results are undefined and unpredictable. A weak attempt is made to raise :exc:`RuntimeError` if the size of a BTree-based structure changes while iterating, but it doesn't catch most such cases, and is also unreliable. Example .. doctest:: :options: +SKIP >>> from BTrees.IIBTree import * >>> s = IISet(range(10)) >>> list(s) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> for i in s: # the output is undefined ... print i, ... s.remove(i) 0 2 4 6 8 Traceback (most recent call last): File "", line 1, in ? RuntimeError: the bucket being iterated changed size >>> list(s) # this output is also undefined [1, 3, 5, 7, 9] >>> Also as with Python dictionaries and lists, the safe and predictable way to mutate a BTree-based structure while iterating over it is to iterate over a copy of the keys. Example .. doctest:: >>> from BTrees.IIBTree import * >>> s = IISet(range(10)) >>> for i in list(s.keys()): # this is well defined ... print i, ... s.remove(i) 0 1 2 3 4 5 6 7 8 9 >>> list(s) [] >>> BTree node sizes ---------------- BTrees (and TreeSets) are made up of a tree of Buckets (and Sets) and internal nodes. There are maximum sizes of these notes configured for the various key and value types: ======== ========== ========================== ============================= Key Type Value Type Maximum Bucket or Set Size Maximum BTree or TreeSet Size ======== ========== ========================== ============================= Integer Float 120 500 Integer Integer 120 500 Integer Object 60 500 Long Float 120 500 Long Long 120 500 Long Object 60 500 Object Integer 60 250 Object Long 60 250 Object Object 30 250 ======== ========== ========================== ============================= For your application, especially when using object keys or values, you may want to override the default sizes. You can do this by subclassing any of the BTree (or TreeSet) classes and specifying new values for ``max_leaf_size`` or ``max_internal_size`` in your subclass:: import BTrees.OOBTree class MyBTree(BTrees.OOBTree.BTree): max_leaf_size = 500 max_internal_size = 1000 ``max_leaf_size`` is used for leaf nodes in a BTree, either Buckets or Sets. ``max_internal_size`` is used for internal nodes, either BTrees or TreeSets. BTree Diagnostic Tools ---------------------- A BTree (or TreeSet) is a complex data structure, really a graph of variable- size nodes, connected in multiple ways via three distinct kinds of C pointers. There are some tools available to help check internal consistency of a BTree as a whole. Most generally useful is the :mod:`BTrees.check` module. The :func:`check.check` function examines a BTree (or Bucket, Set, or TreeSet) for value-based consistency, such as that the keys are in strictly increasing order. See the function docstring for details. The :func:`check.display` function displays the internal structure of a BTree. BTrees and TreeSets also have a :meth:`_check` method. This verifies that the (possibly many) internal pointers in a BTree or TreeSet are mutually consistent, and raises :exc:`AssertionError` if they're not. If a :func:`check.check` or :meth:`_check` call fails, it may point to a bug in the implementation of BTrees or conflict resolution, or may point to database corruption. Repairing a damaged BTree is usually best done by making a copy of it. For example, if *self.data* is bound to a corrupted IOBTree, .. doctest:: self.data = IOBTree(self.data) usually suffices. If object identity needs to be preserved, .. doctest:: acopy = IOBTree(self.data) self.data.clear() self.data.update(acopy) does the same, but leaves *self.data* bound to the same object. Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` BTrees-4.3.1/MANIFEST.in0000664000175000017500000000045612510775554015627 0ustar tseavertseaver00000000000000include *.txt include *.rst include *.py include buildout.cfg include tox.ini include .travis.yml include .coveragerc recursive-include docs * recursive-include BTrees * global-exclude *.dll global-exclude *.pyc global-exclude *.pyo global-exclude *.so global-exclude coverage.xml prune docs/_build