pax_global_header00006660000000000000000000000064122330025360014506gustar00rootroot0000000000000052 comment=9aff798fc56f34065d453261ad5ad7c25cef5ea3 pymongo-2.6.3/000077500000000000000000000000001223300253600132065ustar00rootroot00000000000000pymongo-2.6.3/LICENSE000066400000000000000000000261361223300253600142230ustar00rootroot00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. pymongo-2.6.3/MANIFEST.in000066400000000000000000000004001223300253600147360ustar00rootroot00000000000000include README.rst include LICENSE include distribute_setup.py recursive-include doc *.rst recursive-include doc *.py recursive-include tools *.py include tools/README.rst recursive-include test *.pem recursive-include test *.py recursive-include bson *.h pymongo-2.6.3/PKG-INFO000066400000000000000000000124641223300253600143120ustar00rootroot00000000000000Metadata-Version: 1.1 Name: pymongo Version: 2.6.3 Summary: Python driver for MongoDB Home-page: http://github.com/mongodb/mongo-python-driver Author: Bernie Hackett Author-email: bernie@10gen.com License: Apache License, Version 2.0 Description: ======= PyMongo ======= :Info: See `the mongo site `_ for more information. See `github `_ for the latest source. :Author: Mike Dirolf :Maintainer: Bernie Hackett About ===== The PyMongo distribution contains tools for interacting with MongoDB database from Python. The ``bson`` package is an implementation of the `BSON format `_ for Python. The ``pymongo`` package is a native Python driver for MongoDB. The ``gridfs`` package is a `gridfs `_ implementation on top of ``pymongo``. Issues / Questions / Feedback ============================= Any issues with, questions about, or feedback for PyMongo should be sent to the mongodb-user list on Google Groups. For confirmed issues or feature requests, open a case on `jira `_. Please do not e-mail any of the PyMongo developers directly with issues or questions - you're more likely to get an answer on the list. Installation ============ If you have `distribute `_ installed you should be able to do **easy_install pymongo** to install PyMongo. Otherwise you can download the project source and do **python setup.py install** to install. Dependencies ============ The PyMongo distribution is supported and tested on Python 2.x (where x >= 4) and Python 3.x (where x >= 1). PyMongo versions <= 1.3 also supported Python 2.3, but that is no longer supported. Additional dependencies are: - (to generate documentation) sphinx_ - (to auto-discover tests) `nose `_ Examples ======== Here's a basic example (for more see the *examples* section of the docs): .. code-block:: pycon >>> import pymongo >>> client = pymongo.MongoClient("localhost", 27017) >>> db = client.test >>> db.name u'test' >>> db.my_collection Collection(Database(MongoClient('localhost', 27017), u'test'), u'my_collection') >>> db.my_collection.save({"x": 10}) ObjectId('4aba15ebe23f6b53b0000000') >>> db.my_collection.save({"x": 8}) ObjectId('4aba160ee23f6b543e000000') >>> db.my_collection.save({"x": 11}) ObjectId('4aba160ee23f6b543e000002') >>> db.my_collection.find_one() {u'x': 10, u'_id': ObjectId('4aba15ebe23f6b53b0000000')} >>> for item in db.my_collection.find(): ... print item["x"] ... 10 8 11 >>> db.my_collection.create_index("x") u'x_1' >>> for item in db.my_collection.find().sort("x", pymongo.ASCENDING): ... print item["x"] ... 8 10 11 >>> [item["x"] for item in db.my_collection.find().limit(2).skip(1)] [8, 11] Documentation ============= You will need sphinx_ installed to generate the documentation. Documentation can be generated by running **python setup.py doc**. Generated documentation can be found in the *doc/build/html/* directory. Testing ======= The easiest way to run the tests is to install `nose `_ (**easy_install nose**) and run **nosetests** or **python setup.py test** in the root of the distribution. Tests are located in the *test/* directory. .. _sphinx: http://sphinx.pocoo.org/ Keywords: mongo,mongodb,pymongo,gridfs,bson Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Apache Software License Classifier: Operating System :: MacOS :: MacOS X Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: POSIX Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.4 Classifier: Programming Language :: Python :: 2.5 Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.1 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: Jython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Topic :: Database pymongo-2.6.3/README.rst000066400000000000000000000063061223300253600147020ustar00rootroot00000000000000======= PyMongo ======= :Info: See `the mongo site `_ for more information. See `github `_ for the latest source. :Author: Mike Dirolf :Maintainer: Bernie Hackett About ===== The PyMongo distribution contains tools for interacting with MongoDB database from Python. The ``bson`` package is an implementation of the `BSON format `_ for Python. The ``pymongo`` package is a native Python driver for MongoDB. The ``gridfs`` package is a `gridfs `_ implementation on top of ``pymongo``. Issues / Questions / Feedback ============================= Any issues with, questions about, or feedback for PyMongo should be sent to the mongodb-user list on Google Groups. For confirmed issues or feature requests, open a case on `jira `_. Please do not e-mail any of the PyMongo developers directly with issues or questions - you're more likely to get an answer on the list. Installation ============ If you have `distribute `_ installed you should be able to do **easy_install pymongo** to install PyMongo. Otherwise you can download the project source and do **python setup.py install** to install. Dependencies ============ The PyMongo distribution is supported and tested on Python 2.x (where x >= 4) and Python 3.x (where x >= 1). PyMongo versions <= 1.3 also supported Python 2.3, but that is no longer supported. Additional dependencies are: - (to generate documentation) sphinx_ - (to auto-discover tests) `nose `_ Examples ======== Here's a basic example (for more see the *examples* section of the docs): .. code-block:: pycon >>> import pymongo >>> client = pymongo.MongoClient("localhost", 27017) >>> db = client.test >>> db.name u'test' >>> db.my_collection Collection(Database(MongoClient('localhost', 27017), u'test'), u'my_collection') >>> db.my_collection.save({"x": 10}) ObjectId('4aba15ebe23f6b53b0000000') >>> db.my_collection.save({"x": 8}) ObjectId('4aba160ee23f6b543e000000') >>> db.my_collection.save({"x": 11}) ObjectId('4aba160ee23f6b543e000002') >>> db.my_collection.find_one() {u'x': 10, u'_id': ObjectId('4aba15ebe23f6b53b0000000')} >>> for item in db.my_collection.find(): ... print item["x"] ... 10 8 11 >>> db.my_collection.create_index("x") u'x_1' >>> for item in db.my_collection.find().sort("x", pymongo.ASCENDING): ... print item["x"] ... 8 10 11 >>> [item["x"] for item in db.my_collection.find().limit(2).skip(1)] [8, 11] Documentation ============= You will need sphinx_ installed to generate the documentation. Documentation can be generated by running **python setup.py doc**. Generated documentation can be found in the *doc/build/html/* directory. Testing ======= The easiest way to run the tests is to install `nose `_ (**easy_install nose**) and run **nosetests** or **python setup.py test** in the root of the distribution. Tests are located in the *test/* directory. .. _sphinx: http://sphinx.pocoo.org/ pymongo-2.6.3/bson/000077500000000000000000000000001223300253600141475ustar00rootroot00000000000000pymongo-2.6.3/bson/__init__.py000066400000000000000000000520241223300253600162630ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """BSON (Binary JSON) encoding and decoding. """ import calendar import datetime import re import struct import sys from bson.binary import (Binary, OLD_UUID_SUBTYPE, JAVA_LEGACY, CSHARP_LEGACY) from bson.code import Code from bson.dbref import DBRef from bson.errors import (InvalidBSON, InvalidDocument, InvalidStringData) from bson.max_key import MaxKey from bson.min_key import MinKey from bson.objectid import ObjectId from bson.py3compat import b, binary_type from bson.son import SON, RE_TYPE from bson.timestamp import Timestamp from bson.tz_util import utc try: from bson import _cbson _use_c = True except ImportError: _use_c = False try: import uuid _use_uuid = True except ImportError: _use_uuid = False PY3 = sys.version_info[0] == 3 MAX_INT32 = 2147483647 MIN_INT32 = -2147483648 MAX_INT64 = 9223372036854775807 MIN_INT64 = -9223372036854775808 EPOCH_AWARE = datetime.datetime.fromtimestamp(0, utc) EPOCH_NAIVE = datetime.datetime.utcfromtimestamp(0) # Create constants compatible with all versions of # python from 2.4 forward. In 2.x b("foo") is just # "foo". In 3.x it becomes b"foo". EMPTY = b("") ZERO = b("\x00") ONE = b("\x01") BSONNUM = b("\x01") # Floating point BSONSTR = b("\x02") # UTF-8 string BSONOBJ = b("\x03") # Embedded document BSONARR = b("\x04") # Array BSONBIN = b("\x05") # Binary BSONUND = b("\x06") # Undefined BSONOID = b("\x07") # ObjectId BSONBOO = b("\x08") # Boolean BSONDAT = b("\x09") # UTC Datetime BSONNUL = b("\x0A") # Null BSONRGX = b("\x0B") # Regex BSONREF = b("\x0C") # DBRef BSONCOD = b("\x0D") # Javascript code BSONSYM = b("\x0E") # Symbol BSONCWS = b("\x0F") # Javascript code with scope BSONINT = b("\x10") # 32bit int BSONTIM = b("\x11") # Timestamp BSONLON = b("\x12") # 64bit int BSONMIN = b("\xFF") # Min key BSONMAX = b("\x7F") # Max key def _get_int(data, position, as_class=None, tz_aware=False, uuid_subtype=OLD_UUID_SUBTYPE, unsigned=False): format = unsigned and "I" or "i" try: value = struct.unpack("<%s" % format, data[position:position + 4])[0] except struct.error: raise InvalidBSON() position += 4 return value, position def _get_c_string(data, position, length=None): if length is None: try: end = data.index(ZERO, position) except ValueError: raise InvalidBSON() else: end = position + length value = data[position:end].decode("utf-8") position = end + 1 return value, position def _make_c_string(string, check_null=False): if isinstance(string, unicode): if check_null and "\x00" in string: raise InvalidDocument("BSON keys / regex patterns must not " "contain a NULL character") return string.encode("utf-8") + ZERO else: if check_null and ZERO in string: raise InvalidDocument("BSON keys / regex patterns must not " "contain a NULL character") try: string.decode("utf-8") return string + ZERO except UnicodeError: raise InvalidStringData("strings in documents must be valid " "UTF-8: %r" % string) def _get_number(data, position, as_class, tz_aware, uuid_subtype): num = struct.unpack(" MAX_INT64 or value < MIN_INT64: raise OverflowError("BSON can only handle up to 8-byte ints") if value > MAX_INT32 or value < MIN_INT32: return BSONLON + name + struct.pack(" MAX_INT64 or value < MIN_INT64: raise OverflowError("BSON can only handle up to 8-byte ints") return BSONLON + name + struct.pack("ob_type) #endif #if PY_MAJOR_VERSION >= 3 #define GETSTATE(m) ((struct module_state*)PyModule_GetState(m)) #else #define GETSTATE(m) (&_state) static struct module_state _state; #endif #if PY_VERSION_HEX < 0x02050000 #define WARN(category, message) \ PyErr_Warn((category), (message)) #else #define WARN(category, message) \ PyErr_WarnEx((category), (message), 1) #endif /* Maximum number of regex flags */ #define FLAGS_SIZE 7 #if defined(WIN32) || defined(_MSC_VER) /* This macro is basically an implementation of asprintf for win32 * We get the length of the int as string and malloc a buffer for it, * returning -1 if that malloc fails. We then actually print to the * buffer to get the string value as an int. Like asprintf, the result * must be explicitly free'd when done being used. */ #if defined(_MSC_VER) && (_MSC_VER >= 1400) #define INT2STRING(buffer, i) \ _snprintf_s((buffer), \ _scprintf("%d", (i)) + 1, \ _scprintf("%d", (i)) + 1, \ "%d", \ (i)) #define STRCAT(dest, n, src) strcat_s((dest), (n), (src)) #else #define INT2STRING(buffer, i) \ _snprintf((buffer), \ _scprintf("%d", (i)) + 1, \ "%d", \ (i)) #define STRCAT(dest, n, src) strcat((dest), (src)) #endif #else #define INT2STRING(buffer, i) snprintf((buffer), sizeof((buffer)), "%d", (i)) #define STRCAT(dest, n, src) strcat((dest), (src)) #endif #define JAVA_LEGACY 5 #define CSHARP_LEGACY 6 #define BSON_MAX_SIZE 2147483647 /* The smallest possible BSON document, i.e. "{}" */ #define BSON_MIN_SIZE 5 /* Get an error class from the bson.errors module. * * Returns a new ref */ static PyObject* _error(char* name) { PyObject* error; PyObject* errors = PyImport_ImportModule("bson.errors"); if (!errors) { return NULL; } error = PyObject_GetAttrString(errors, name); Py_DECREF(errors); return error; } /* Safely downcast from Py_ssize_t to int, setting an * exception and returning -1 on error. */ static int _downcast_and_check(Py_ssize_t size, int extra) { if (size > BSON_MAX_SIZE || ((BSON_MAX_SIZE - extra) < size)) { PyObject* InvalidStringData = _error("InvalidStringData"); if (InvalidStringData) { PyErr_SetString(InvalidStringData, "String length must be <= 2147483647"); Py_DECREF(InvalidStringData); } return -1; } return (int)size + extra; } static PyObject* elements_to_dict(PyObject* self, const char* string, unsigned max, PyObject* as_class, unsigned char tz_aware, unsigned char uuid_subtype); static int _write_element_to_buffer(PyObject* self, buffer_t buffer, int type_byte, PyObject* value, unsigned char check_keys, unsigned char uuid_subtype, unsigned char first_attempt); /* Date stuff */ static PyObject* datetime_from_millis(long long millis) { /* To encode a datetime instance like datetime(9999, 12, 31, 23, 59, 59, 999999) * we follow these steps: * 1. Calculate a timestamp in seconds: 253402300799 * 2. Multiply that by 1000: 253402300799000 * 3. Add in microseconds divided by 1000 253402300799999 * * (Note: BSON doesn't support microsecond accuracy, hence the rounding.) * * To decode we could do: * 1. Get seconds: timestamp / 1000: 253402300799 * 2. Get micros: (timestamp % 1000) * 1000: 999000 * Resulting in datetime(9999, 12, 31, 23, 59, 59, 999000) -- the expected result * * Now what if the we encode (1, 1, 1, 1, 1, 1, 111111)? * 1. and 2. gives: -62135593139000 * 3. Gives us: -62135593138889 * * Now decode: * 1. Gives us: -62135593138 * 2. Gives us: -889000 * Resulting in datetime(1, 1, 1, 1, 1, 2, 15888216) -- an invalid result * * If instead to decode we do: * diff = ((millis % 1000) + 1000) % 1000: 111 * seconds = (millis - diff) / 1000: -62135593139 * micros = diff * 1000 111000 * Resulting in datetime(1, 1, 1, 1, 1, 1, 111000) -- the expected result */ int diff = (int)(((millis % 1000) + 1000) % 1000); int microseconds = diff * 1000; Time64_T seconds = (millis - diff) / 1000; struct TM timeinfo; gmtime64_r(&seconds, &timeinfo); return PyDateTime_FromDateAndTime(timeinfo.tm_year + 1900, timeinfo.tm_mon + 1, timeinfo.tm_mday, timeinfo.tm_hour, timeinfo.tm_min, timeinfo.tm_sec, microseconds); } static long long millis_from_datetime(PyObject* datetime) { struct TM timeinfo; long long millis; timeinfo.tm_year = PyDateTime_GET_YEAR(datetime) - 1900; timeinfo.tm_mon = PyDateTime_GET_MONTH(datetime) - 1; timeinfo.tm_mday = PyDateTime_GET_DAY(datetime); timeinfo.tm_hour = PyDateTime_DATE_GET_HOUR(datetime); timeinfo.tm_min = PyDateTime_DATE_GET_MINUTE(datetime); timeinfo.tm_sec = PyDateTime_DATE_GET_SECOND(datetime); millis = timegm64(&timeinfo) * 1000; millis += PyDateTime_DATE_GET_MICROSECOND(datetime) / 1000; return millis; } /* Just make this compatible w/ the old API. */ int buffer_write_bytes(buffer_t buffer, const char* data, int size) { if (buffer_write(buffer, data, size)) { PyErr_NoMemory(); return 0; } return 1; } #if PY_MAJOR_VERSION >= 3 static int write_unicode(buffer_t buffer, PyObject* py_string) { int size; const char* data; PyObject* encoded = PyUnicode_AsUTF8String(py_string); if (!encoded) { return 0; } data = PyBytes_AsString(encoded); if (!data) { Py_DECREF(encoded); return 0; } if ((size = _downcast_and_check(PyBytes_Size(encoded), 1)) == -1){ Py_DECREF(encoded); return 0; } if (!buffer_write_bytes(buffer, (const char*)&size, 4)) { Py_DECREF(encoded); return 0; } if (!buffer_write_bytes(buffer, data, size)) { Py_DECREF(encoded); return 0; } Py_DECREF(encoded); return 1; } #endif /* returns 0 on failure */ static int write_string(buffer_t buffer, PyObject* py_string) { int size; const char* data; #if PY_MAJOR_VERSION >= 3 if (PyUnicode_Check(py_string)){ return write_unicode(buffer, py_string); } data = PyBytes_AsString(py_string); #else data = PyString_AsString(py_string); #endif if (!data) { return 0; } #if PY_MAJOR_VERSION >= 3 if ((size = _downcast_and_check(PyBytes_Size(py_string), 1)) == -1) #else if ((size = _downcast_and_check(PyString_Size(py_string), 1)) == -1) #endif return 0; if (!buffer_write_bytes(buffer, (const char*)&size, 4)) { return 0; } if (!buffer_write_bytes(buffer, data, size)) { return 0; } return 1; } /* Reload a cached Python object. * * Returns non-zero on failure. */ static int _reload_object(PyObject** object, char* module_name, char* object_name) { PyObject* module; module = PyImport_ImportModule(module_name); if (!module) { return 1; } *object = PyObject_GetAttrString(module, object_name); Py_DECREF(module); return (*object) ? 0 : 2; } /* Reload all cached Python objects. * * Returns non-zero on failure. */ static int _reload_python_objects(PyObject* module) { PyObject* empty_string; PyObject* compiled; struct module_state *state = GETSTATE(module); if (_reload_object(&state->Binary, "bson.binary", "Binary") || _reload_object(&state->Code, "bson.code", "Code") || _reload_object(&state->ObjectId, "bson.objectid", "ObjectId") || _reload_object(&state->DBRef, "bson.dbref", "DBRef") || _reload_object(&state->Timestamp, "bson.timestamp", "Timestamp") || _reload_object(&state->MinKey, "bson.min_key", "MinKey") || _reload_object(&state->MaxKey, "bson.max_key", "MaxKey") || _reload_object(&state->UTC, "bson.tz_util", "utc") || _reload_object(&state->RECompile, "re", "compile")) { return 1; } /* If we couldn't import uuid then we must be on 2.4. Just ignore. */ if (_reload_object(&state->UUID, "uuid", "UUID") == 1) { state->UUID = NULL; PyErr_Clear(); } /* Reload our REType hack too. */ #if PY_MAJOR_VERSION >= 3 empty_string = PyBytes_FromString(""); #else empty_string = PyString_FromString(""); #endif if (empty_string == NULL) { state->REType = NULL; return 1; } compiled = PyObject_CallFunction(state->RECompile, "O", empty_string); if (compiled == NULL) { state->REType = NULL; Py_DECREF(empty_string); return 1; } Py_INCREF(Py_TYPE(compiled)); state->REType = Py_TYPE(compiled); Py_DECREF(empty_string); Py_DECREF(compiled); return 0; } static int write_element_to_buffer(PyObject* self, buffer_t buffer, int type_byte, PyObject* value, unsigned char check_keys, unsigned char uuid_subtype, unsigned char first_attempt) { int result; if(Py_EnterRecursiveCall(" while encoding an object to BSON ")) return 0; result = _write_element_to_buffer(self, buffer, type_byte, value, check_keys, uuid_subtype, first_attempt); Py_LeaveRecursiveCall(); return result; } static void _fix_java(const char* in, char* out) { int i, j; for (i = 0, j = 7; i < j; i++, j--) { out[i] = in[j]; out[j] = in[i]; } for (i = 8, j = 15; i < j; i++, j--) { out[i] = in[j]; out[j] = in[i]; } } /* TODO our platform better be little-endian w/ 4-byte ints! */ /* Write a single value to the buffer (also write it's type_byte, for which * space has already been reserved. * * returns 0 on failure */ static int _write_element_to_buffer(PyObject* self, buffer_t buffer, int type_byte, PyObject* value, unsigned char check_keys, unsigned char uuid_subtype, unsigned char first_attempt) { struct module_state *state = GETSTATE(self); if (PyBool_Check(value)) { #if PY_MAJOR_VERSION >= 3 const long bool = PyLong_AsLong(value); #else const long bool = PyInt_AsLong(value); #endif const char c = bool ? 0x01 : 0x00; *(buffer_get_buffer(buffer) + type_byte) = 0x08; return buffer_write_bytes(buffer, &c, 1); } #if PY_MAJOR_VERSION >= 3 else if (PyLong_Check(value)) { const long long_value = PyLong_AsLong(value); #else else if (PyInt_Check(value)) { const long long_value = PyInt_AsLong(value); #endif const int int_value = (int)long_value; if (PyErr_Occurred() || long_value != int_value) { /* Overflow */ long long long_long_value; PyErr_Clear(); long_long_value = PyLong_AsLongLong(value); if (PyErr_Occurred()) { /* Overflow AGAIN */ PyErr_SetString(PyExc_OverflowError, "MongoDB can only handle up to 8-byte ints"); return 0; } *(buffer_get_buffer(buffer) + type_byte) = 0x12; return buffer_write_bytes(buffer, (const char*)&long_long_value, 8); } *(buffer_get_buffer(buffer) + type_byte) = 0x10; return buffer_write_bytes(buffer, (const char*)&int_value, 4); #if PY_MAJOR_VERSION < 3 } else if (PyLong_Check(value)) { const long long long_long_value = PyLong_AsLongLong(value); if (PyErr_Occurred()) { /* Overflow */ PyErr_SetString(PyExc_OverflowError, "MongoDB can only handle up to 8-byte ints"); return 0; } *(buffer_get_buffer(buffer) + type_byte) = 0x12; return buffer_write_bytes(buffer, (const char*)&long_long_value, 8); #endif } else if (PyFloat_Check(value)) { const double d = PyFloat_AsDouble(value); *(buffer_get_buffer(buffer) + type_byte) = 0x01; return buffer_write_bytes(buffer, (const char*)&d, 8); } else if (value == Py_None) { *(buffer_get_buffer(buffer) + type_byte) = 0x0A; return 1; } else if (PyDict_Check(value)) { *(buffer_get_buffer(buffer) + type_byte) = 0x03; return write_dict(self, buffer, value, check_keys, uuid_subtype, 0); } else if (PyList_Check(value) || PyTuple_Check(value)) { Py_ssize_t items, i; int start_position, length_location, length; char zero = 0; *(buffer_get_buffer(buffer) + type_byte) = 0x04; start_position = buffer_get_position(buffer); /* save space for length */ length_location = buffer_save_space(buffer, 4); if (length_location == -1) { PyErr_NoMemory(); return 0; } if ((items = PySequence_Size(value)) > BSON_MAX_SIZE) { PyObject* BSONError = _error("BSONError"); if (BSONError) { PyErr_SetString(BSONError, "Too many items to serialize."); Py_DECREF(BSONError); } return 0; } for(i = 0; i < items; i++) { int list_type_byte = buffer_save_space(buffer, 1); char name[16]; PyObject* item_value; if (list_type_byte == -1) { PyErr_NoMemory(); return 0; } INT2STRING(name, (int)i); if (!buffer_write_bytes(buffer, name, (int)strlen(name) + 1)) { return 0; } if (!(item_value = PySequence_GetItem(value, i))) return 0; if (!write_element_to_buffer(self, buffer, list_type_byte, item_value, check_keys, uuid_subtype, 1)) { Py_DECREF(item_value); return 0; } Py_DECREF(item_value); } /* write null byte and fill in length */ if (!buffer_write_bytes(buffer, &zero, 1)) { return 0; } length = buffer_get_position(buffer) - start_position; memcpy(buffer_get_buffer(buffer) + length_location, &length, 4); return 1; } else if (PyObject_IsInstance(value, state->Binary)) { PyObject* subtype_object; long subtype; const char* data; int size; *(buffer_get_buffer(buffer) + type_byte) = 0x05; subtype_object = PyObject_GetAttrString(value, "subtype"); if (!subtype_object) { return 0; } #if PY_MAJOR_VERSION >= 3 subtype = PyLong_AsLong(subtype_object); #else subtype = PyInt_AsLong(subtype_object); #endif if (subtype == -1) { Py_DECREF(subtype_object); return 0; } #if PY_MAJOR_VERSION >= 3 size = _downcast_and_check(PyBytes_Size(value), 0); #else size = _downcast_and_check(PyString_Size(value), 0); #endif if (size == -1) { Py_DECREF(subtype_object); return 0; } Py_DECREF(subtype_object); if (subtype == 2) { #if PY_MAJOR_VERSION >= 3 int other_size = _downcast_and_check(PyBytes_Size(value), 4); #else int other_size = _downcast_and_check(PyString_Size(value), 4); #endif if (other_size == -1) return 0; if (!buffer_write_bytes(buffer, (const char*)&other_size, 4)) { return 0; } if (!buffer_write_bytes(buffer, (const char*)&subtype, 1)) { return 0; } } if (!buffer_write_bytes(buffer, (const char*)&size, 4)) { return 0; } if (subtype != 2) { if (!buffer_write_bytes(buffer, (const char*)&subtype, 1)) { return 0; } } #if PY_MAJOR_VERSION >= 3 data = PyBytes_AsString(value); #else data = PyString_AsString(value); #endif if (!data) { return 0; } if (!buffer_write_bytes(buffer, data, size)) { return 0; } return 1; } else if (state->UUID && PyObject_IsInstance(value, state->UUID)) { /* Just a special case of Binary above, but * simpler to do as a separate case. */ PyObject* bytes; /* Could be bytes, bytearray, str... */ const char* data; /* UUID is always 16 bytes */ int size = 16; int subtype; if (uuid_subtype == JAVA_LEGACY || uuid_subtype == CSHARP_LEGACY) { subtype = 3; } else { subtype = uuid_subtype; } *(buffer_get_buffer(buffer) + type_byte) = 0x05; if (!buffer_write_bytes(buffer, (const char*)&size, 4)) { return 0; } if (!buffer_write_bytes(buffer, (const char*)&subtype, 1)) { return 0; } if (uuid_subtype == CSHARP_LEGACY) { /* Legacy C# byte order */ bytes = PyObject_GetAttrString(value, "bytes_le"); } else { bytes = PyObject_GetAttrString(value, "bytes"); } if (!bytes) { return 0; } #if PY_MAJOR_VERSION >= 3 /* Work around http://bugs.python.org/issue7380 */ if (PyByteArray_Check(bytes)) { data = PyByteArray_AsString(bytes); } else { data = PyBytes_AsString(bytes); } #else data = PyString_AsString(bytes); #endif if (data == NULL) { Py_DECREF(bytes); return 0; } if (uuid_subtype == JAVA_LEGACY) { /* Store in legacy java byte order. */ char as_legacy_java[16]; _fix_java(data, as_legacy_java); if (!buffer_write_bytes(buffer, as_legacy_java, size)) { Py_DECREF(bytes); return 0; } } else { if (!buffer_write_bytes(buffer, data, size)) { Py_DECREF(bytes); return 0; } } Py_DECREF(bytes); return 1; } else if (PyObject_IsInstance(value, state->Code)) { int start_position, length_location, length; PyObject* scope = PyObject_GetAttrString(value, "scope"); if (!scope) { return 0; } if (!PyDict_Size(scope)) { Py_DECREF(scope); *(buffer_get_buffer(buffer) + type_byte) = 0x0D; return write_string(buffer, value); } *(buffer_get_buffer(buffer) + type_byte) = 0x0F; start_position = buffer_get_position(buffer); /* save space for length */ length_location = buffer_save_space(buffer, 4); if (length_location == -1) { PyErr_NoMemory(); Py_DECREF(scope); return 0; } if (!write_string(buffer, value)) { Py_DECREF(scope); return 0; } if (!write_dict(self, buffer, scope, 0, uuid_subtype, 0)) { Py_DECREF(scope); return 0; } Py_DECREF(scope); length = buffer_get_position(buffer) - start_position; memcpy(buffer_get_buffer(buffer) + length_location, &length, 4); return 1; #if PY_MAJOR_VERSION >= 3 /* Python3 special case. Store bytes as BSON binary subtype 0. */ } else if (PyBytes_Check(value)) { int subtype = 0; int size; const char* data = PyBytes_AsString(value); if (!data) return 0; if ((size = _downcast_and_check(PyBytes_Size(value), 0)) == -1) return 0; *(buffer_get_buffer(buffer) + type_byte) = 0x05; if (!buffer_write_bytes(buffer, (const char*)&size, 4)) { return 0; } if (!buffer_write_bytes(buffer, (const char*)&subtype, 1)) { return 0; } if (!buffer_write_bytes(buffer, data, size)) { return 0; } return 1; #else /* PyString_Check only works in Python 2.x. */ } else if (PyString_Check(value)) { result_t status; const char* data; int size; if (!(data = PyString_AsString(value))) return 0; if ((size = _downcast_and_check(PyString_Size(value), 0)) == -1) return 0; *(buffer_get_buffer(buffer) + type_byte) = 0x02; status = check_string((const unsigned char*)data, size, 1, 0); if (status == NOT_UTF_8) { PyObject* InvalidStringData = _error("InvalidStringData"); if (InvalidStringData) { PyObject* repr = PyObject_Repr(value); char* repr_as_cstr = repr ? PyString_AsString(repr) : NULL; if (repr_as_cstr) { PyObject *message = PyString_FromFormat( "strings in documents must be valid UTF-8: %s", repr_as_cstr); if (message) { PyErr_SetObject(InvalidStringData, message); Py_DECREF(message); } } else { /* repr(value) failed, use a generic message. */ PyErr_SetString( InvalidStringData, "strings in documents must be valid UTF-8"); } Py_XDECREF(repr); Py_DECREF(InvalidStringData); } return 0; } return write_string(buffer, value); #endif } else if (PyUnicode_Check(value)) { PyObject* encoded; int result; *(buffer_get_buffer(buffer) + type_byte) = 0x02; encoded = PyUnicode_AsUTF8String(value); if (!encoded) { return 0; } result = write_string(buffer, encoded); Py_DECREF(encoded); return result; } else if (PyDateTime_Check(value)) { long long millis; PyObject* utcoffset = PyObject_CallMethod(value, "utcoffset", NULL); if (utcoffset == NULL) return 0; if (utcoffset != Py_None) { PyObject* result = PyNumber_Subtract(value, utcoffset); Py_DECREF(utcoffset); if (!result) { return 0; } millis = millis_from_datetime(result); Py_DECREF(result); } else { millis = millis_from_datetime(value); } *(buffer_get_buffer(buffer) + type_byte) = 0x09; return buffer_write_bytes(buffer, (const char*)&millis, 8); } else if (PyObject_IsInstance(value, state->ObjectId)) { const char* data; PyObject* pystring = PyObject_GetAttrString(value, "_ObjectId__id"); if (!pystring) { return 0; } #if PY_MAJOR_VERSION >= 3 data = PyBytes_AsString(pystring); #else data = PyString_AsString(pystring); #endif if (!data) { Py_DECREF(pystring); return 0; } if (!buffer_write_bytes(buffer, data, 12)) { Py_DECREF(pystring); return 0; } Py_DECREF(pystring); *(buffer_get_buffer(buffer) + type_byte) = 0x07; return 1; } else if (PyObject_IsInstance(value, state->DBRef)) { PyObject* as_doc = PyObject_CallMethod(value, "as_doc", NULL); if (!as_doc) { return 0; } if (!write_dict(self, buffer, as_doc, 0, uuid_subtype, 0)) { Py_DECREF(as_doc); return 0; } Py_DECREF(as_doc); *(buffer_get_buffer(buffer) + type_byte) = 0x03; return 1; } else if (PyObject_IsInstance(value, state->Timestamp)) { PyObject* obj; long i; obj = PyObject_GetAttrString(value, "inc"); if (!obj) { return 0; } #if PY_MAJOR_VERSION >= 3 i = PyLong_AsLong(obj); #else i = PyInt_AsLong(obj); #endif Py_DECREF(obj); if (!buffer_write_bytes(buffer, (const char*)&i, 4)) { return 0; } obj = PyObject_GetAttrString(value, "time"); if (!obj) { return 0; } #if PY_MAJOR_VERSION >= 3 i = PyLong_AsLong(obj); #else i = PyInt_AsLong(obj); #endif Py_DECREF(obj); if (!buffer_write_bytes(buffer, (const char*)&i, 4)) { return 0; } *(buffer_get_buffer(buffer) + type_byte) = 0x11; return 1; } else if (PyObject_TypeCheck(value, state->REType)) { PyObject* py_flags; PyObject* py_pattern; PyObject* encoded_pattern; long int_flags; char flags[FLAGS_SIZE]; char check_utf8 = 0; const char* pattern_data; int pattern_length, flags_length; result_t status; py_flags = PyObject_GetAttrString(value, "flags"); if (!py_flags) { return 0; } #if PY_MAJOR_VERSION >= 3 int_flags = PyLong_AsLong(py_flags); #else int_flags = PyInt_AsLong(py_flags); #endif Py_DECREF(py_flags); py_pattern = PyObject_GetAttrString(value, "pattern"); if (!py_pattern) { return 0; } if (PyUnicode_Check(py_pattern)) { encoded_pattern = PyUnicode_AsUTF8String(py_pattern); Py_DECREF(py_pattern); if (!encoded_pattern) { return 0; } } else { encoded_pattern = py_pattern; check_utf8 = 1; } #if PY_MAJOR_VERSION >= 3 if (!(pattern_data = PyBytes_AsString(encoded_pattern))) { Py_DECREF(encoded_pattern); return 0; } if ((pattern_length = _downcast_and_check(PyBytes_Size(encoded_pattern), 0)) == -1) { Py_DECREF(encoded_pattern); return 0; } #else if (!(pattern_data = PyString_AsString(encoded_pattern))) { Py_DECREF(encoded_pattern); return 0; } if ((pattern_length = _downcast_and_check(PyString_Size(encoded_pattern), 0)) == -1) { Py_DECREF(encoded_pattern); return 0; } #endif status = check_string((const unsigned char*)pattern_data, pattern_length, check_utf8, 1); if (status == NOT_UTF_8) { PyObject* InvalidStringData = _error("InvalidStringData"); if (InvalidStringData) { PyErr_SetString(InvalidStringData, "regex patterns must be valid UTF-8"); Py_DECREF(InvalidStringData); } Py_DECREF(encoded_pattern); return 0; } else if (status == HAS_NULL) { PyObject* InvalidDocument = _error("InvalidDocument"); if (InvalidDocument) { PyErr_SetString(InvalidDocument, "regex patterns must not contain the NULL byte"); Py_DECREF(InvalidDocument); } Py_DECREF(encoded_pattern); return 0; } if (!buffer_write_bytes(buffer, pattern_data, pattern_length + 1)) { Py_DECREF(encoded_pattern); return 0; } Py_DECREF(encoded_pattern); flags[0] = 0; /* TODO don't hardcode these */ if (int_flags & 2) { STRCAT(flags, FLAGS_SIZE, "i"); } if (int_flags & 4) { STRCAT(flags, FLAGS_SIZE, "l"); } if (int_flags & 8) { STRCAT(flags, FLAGS_SIZE, "m"); } if (int_flags & 16) { STRCAT(flags, FLAGS_SIZE, "s"); } if (int_flags & 32) { STRCAT(flags, FLAGS_SIZE, "u"); } if (int_flags & 64) { STRCAT(flags, FLAGS_SIZE, "x"); } flags_length = (int)strlen(flags) + 1; if (!buffer_write_bytes(buffer, flags, flags_length)) { return 0; } *(buffer_get_buffer(buffer) + type_byte) = 0x0B; return 1; } else if (PyObject_IsInstance(value, state->MinKey)) { *(buffer_get_buffer(buffer) + type_byte) = 0xFF; return 1; } else if (PyObject_IsInstance(value, state->MaxKey)) { *(buffer_get_buffer(buffer) + type_byte) = 0x7F; return 1; } else if (first_attempt) { /* Try reloading the modules and having one more go at it. */ if (WARN(PyExc_RuntimeWarning, "couldn't encode - reloading python " "modules and trying again. if you see this without getting " "an InvalidDocument exception please see http://api.mongodb" ".org/python/current/faq.html#does-pymongo-work-with-mod-" "wsgi") == -1) { return 0; } if (_reload_python_objects(self)) { return 0; } return write_element_to_buffer(self, buffer, type_byte, value, check_keys, uuid_subtype, 0); } { PyObject* InvalidDocument = _error("InvalidDocument"); if (InvalidDocument) { PyObject* repr = PyObject_Repr(value); if (repr) { #if PY_MAJOR_VERSION >= 3 PyObject* errmsg = PyUnicode_FromString("Cannot encode object: "); #else PyObject* errmsg = PyString_FromString("Cannot encode object: "); #endif if (errmsg) { #if PY_MAJOR_VERSION >= 3 PyObject* error = PyUnicode_Concat(errmsg, repr); if (error) { PyErr_SetObject(InvalidDocument, error); Py_DECREF(error); } Py_DECREF(errmsg); Py_DECREF(repr); #else PyString_ConcatAndDel(&errmsg, repr); if (errmsg) { PyErr_SetObject(InvalidDocument, errmsg); Py_DECREF(errmsg); } #endif } else { Py_DECREF(repr); } } Py_DECREF(InvalidDocument); } return 0; } } static int check_key_name(const char* name, const Py_ssize_t name_length) { int i; if (name_length > 0 && name[0] == '$') { PyObject* InvalidDocument = _error("InvalidDocument"); if (InvalidDocument) { #if PY_MAJOR_VERSION >= 3 PyObject* errmsg = PyUnicode_FromFormat( "key '%s' must not start with '$'", name); #else PyObject* errmsg = PyString_FromFormat( "key '%s' must not start with '$'", name); #endif if (errmsg) { PyErr_SetObject(InvalidDocument, errmsg); Py_DECREF(errmsg); } Py_DECREF(InvalidDocument); } return 0; } for (i = 0; i < name_length; i++) { if (name[i] == '.') { PyObject* InvalidDocument = _error("InvalidDocument"); if (InvalidDocument) { #if PY_MAJOR_VERSION >= 3 PyObject* errmsg = PyUnicode_FromFormat( "key '%s' must not contain '.'", name); #else PyObject* errmsg = PyString_FromFormat( "key '%s' must not contain '.'", name); #endif if (errmsg) { PyErr_SetObject(InvalidDocument, errmsg); Py_DECREF(errmsg); } Py_DECREF(InvalidDocument); } return 0; } } return 1; } /* Write a (key, value) pair to the buffer. * * Returns 0 on failure */ int write_pair(PyObject* self, buffer_t buffer, const char* name, Py_ssize_t name_length, PyObject* value, unsigned char check_keys, unsigned char uuid_subtype, unsigned char allow_id) { int type_byte; int length; if ((length = _downcast_and_check(name_length, 1)) == -1) return 0; /* Don't write any _id elements unless we're explicitly told to - * _id has to be written first so we do so, but don't bother * deleting it from the dictionary being written. */ if (!allow_id && strcmp(name, "_id") == 0) { return 1; } type_byte = buffer_save_space(buffer, 1); if (type_byte == -1) { PyErr_NoMemory(); return 0; } if (check_keys && !check_key_name(name, name_length)) { return 0; } if (!buffer_write_bytes(buffer, name, length)) { return 0; } if (!write_element_to_buffer(self, buffer, type_byte, value, check_keys, uuid_subtype, 1)) { return 0; } return 1; } int decode_and_write_pair(PyObject* self, buffer_t buffer, PyObject* key, PyObject* value, unsigned char check_keys, unsigned char uuid_subtype, unsigned char top_level) { PyObject* encoded; const char* data; int size; if (PyUnicode_Check(key)) { result_t status; encoded = PyUnicode_AsUTF8String(key); if (!encoded) { return 0; } #if PY_MAJOR_VERSION >= 3 if (!(data = PyBytes_AsString(encoded))) { Py_DECREF(encoded); return 0; } if ((size = _downcast_and_check(PyBytes_Size(encoded), 0)) == -1) { Py_DECREF(encoded); return 0; } #else if (!(data = PyString_AsString(encoded))) { Py_DECREF(encoded); return 0; } if ((size = _downcast_and_check(PyString_Size(encoded), 0)) == -1) { Py_DECREF(encoded); return 0; } #endif status = check_string((const unsigned char*)data, size, 0, 1); if (status == HAS_NULL) { PyObject* InvalidDocument = _error("InvalidDocument"); if (InvalidDocument) { PyErr_SetString(InvalidDocument, "Key names must not contain the NULL byte"); Py_DECREF(InvalidDocument); } Py_DECREF(encoded); return 0; } #if PY_MAJOR_VERSION < 3 } else if (PyString_Check(key)) { result_t status; encoded = key; Py_INCREF(encoded); if (!(data = PyString_AsString(encoded))) { Py_DECREF(encoded); return 0; } if ((size = _downcast_and_check(PyString_Size(encoded), 0)) == -1) { Py_DECREF(encoded); return 0; } status = check_string((const unsigned char*)data, size, 1, 1); if (status == NOT_UTF_8) { PyObject* InvalidStringData = _error("InvalidStringData"); if (InvalidStringData) { PyErr_SetString(InvalidStringData, "strings in documents must be valid UTF-8"); Py_DECREF(InvalidStringData); } Py_DECREF(encoded); return 0; } else if (status == HAS_NULL) { PyObject* InvalidDocument = _error("InvalidDocument"); if (InvalidDocument) { PyErr_SetString(InvalidDocument, "Key names must not contain the NULL byte"); Py_DECREF(InvalidDocument); } Py_DECREF(encoded); return 0; } #endif } else { PyObject* InvalidDocument = _error("InvalidDocument"); if (InvalidDocument) { PyObject* repr = PyObject_Repr(key); if (repr) { #if PY_MAJOR_VERSION >= 3 PyObject* errmsg = PyUnicode_FromString( "documents must have only string keys, key was "); #else PyObject* errmsg = PyString_FromString( "documents must have only string keys, key was "); #endif if (errmsg) { #if PY_MAJOR_VERSION >= 3 PyObject* error = PyUnicode_Concat(errmsg, repr); if (error) { PyErr_SetObject(InvalidDocument, error); Py_DECREF(error); } Py_DECREF(errmsg); Py_DECREF(repr); #else PyString_ConcatAndDel(&errmsg, repr); if (errmsg) { PyErr_SetObject(InvalidDocument, errmsg); Py_DECREF(errmsg); } #endif } else { Py_DECREF(repr); } } Py_DECREF(InvalidDocument); } return 0; } /* If top_level is True, don't allow writing _id here - it was already written. */ #if PY_MAJOR_VERSION >= 3 if (!write_pair(self, buffer, data, PyBytes_Size(encoded), value, check_keys, uuid_subtype, !top_level)) { #else if (!write_pair(self, buffer, data, PyString_Size(encoded), value, check_keys, uuid_subtype, !top_level)) { #endif Py_DECREF(encoded); return 0; } Py_DECREF(encoded); return 1; } /* returns 0 on failure */ int write_dict(PyObject* self, buffer_t buffer, PyObject* dict, unsigned char check_keys, unsigned char uuid_subtype, unsigned char top_level) { PyObject* key; PyObject* iter; char zero = 0; int length; int length_location; if (!PyDict_Check(dict)) { PyObject* repr = PyObject_Repr(dict); if (repr) { #if PY_MAJOR_VERSION >= 3 PyObject* errmsg = PyUnicode_FromString( "encoder expected a mapping type but got: "); if (errmsg) { PyObject* error = PyUnicode_Concat(errmsg, repr); if (error) { PyErr_SetObject(PyExc_TypeError, error); Py_DECREF(error); } Py_DECREF(errmsg); Py_DECREF(repr); } #else PyObject* errmsg = PyString_FromString( "encoder expected a mapping type but got: "); if (errmsg) { PyString_ConcatAndDel(&errmsg, repr); if (errmsg) { PyErr_SetObject(PyExc_TypeError, errmsg); Py_DECREF(errmsg); } } #endif else { Py_DECREF(repr); } } else { PyErr_SetString(PyExc_TypeError, "encoder expected a mapping type"); } return 0; } length_location = buffer_save_space(buffer, 4); if (length_location == -1) { PyErr_NoMemory(); return 0; } /* Write _id first if this is a top level doc. */ if (top_level) { PyObject* _id = PyDict_GetItemString(dict, "_id"); if (_id) { /* Don't bother checking keys, but do make sure we're allowed to * write _id */ if (!write_pair(self, buffer, "_id", 3, _id, 0, uuid_subtype, 1)) { return 0; } } } iter = PyObject_GetIter(dict); if (iter == NULL) { return 0; } while ((key = PyIter_Next(iter)) != NULL) { PyObject* value = PyDict_GetItem(dict, key); if (!value) { PyErr_SetObject(PyExc_KeyError, key); Py_DECREF(key); Py_DECREF(iter); return 0; } if (!decode_and_write_pair(self, buffer, key, value, check_keys, uuid_subtype, top_level)) { Py_DECREF(key); Py_DECREF(iter); return 0; } Py_DECREF(key); } Py_DECREF(iter); /* write null byte and fill in length */ if (!buffer_write_bytes(buffer, &zero, 1)) { return 0; } length = buffer_get_position(buffer) - length_location; memcpy(buffer_get_buffer(buffer) + length_location, &length, 4); return 1; } static PyObject* _cbson_dict_to_bson(PyObject* self, PyObject* args) { PyObject* dict; PyObject* result; unsigned char check_keys; unsigned char uuid_subtype; unsigned char top_level = 1; buffer_t buffer; if (!PyArg_ParseTuple(args, "Obb|b", &dict, &check_keys, &uuid_subtype, &top_level)) { return NULL; } buffer = buffer_new(); if (!buffer) { PyErr_NoMemory(); return NULL; } if (!write_dict(self, buffer, dict, check_keys, uuid_subtype, top_level)) { buffer_free(buffer); return NULL; } /* objectify buffer */ #if PY_MAJOR_VERSION >= 3 result = Py_BuildValue("y#", buffer_get_buffer(buffer), buffer_get_position(buffer)); #else result = Py_BuildValue("s#", buffer_get_buffer(buffer), buffer_get_position(buffer)); #endif buffer_free(buffer); return result; } static PyObject* get_value(PyObject* self, const char* buffer, unsigned* position, int type, unsigned max, PyObject* as_class, unsigned char tz_aware, unsigned char uuid_subtype) { struct module_state *state = GETSTATE(self); PyObject* value; PyObject* error; switch (type) { case 1: { double d; if (max < 8) { goto invalid; } memcpy(&d, buffer + *position, 8); value = PyFloat_FromDouble(d); if (!value) { return NULL; } *position += 8; break; } case 2: case 14: { unsigned value_length; if (max < 4) { goto invalid; } memcpy(&value_length, buffer + *position, 4); /* Encoded string length + string */ if (max < 4 + value_length) { goto invalid; } *position += 4; /* Strings must end in \0 */ if (buffer[*position + value_length - 1]) { goto invalid; } value = PyUnicode_DecodeUTF8(buffer + *position, value_length - 1, "strict"); if (!value) { return NULL; } *position += value_length; break; } case 3: { PyObject* collection; unsigned size; if (max < 4) { goto invalid; } memcpy(&size, buffer + *position, 4); if (size < BSON_MIN_SIZE || max < size) { goto invalid; } /* Check for bad eoo */ if (buffer[*position + size - 1]) { goto invalid; } value = elements_to_dict(self, buffer + *position + 4, size - 5, as_class, tz_aware, uuid_subtype); if (!value) { return NULL; } /* Decoding for DBRefs */ collection = PyDict_GetItemString(value, "$ref"); if (collection) { /* DBRef */ PyObject* dbref; PyObject* id; PyObject* database; Py_INCREF(collection); PyDict_DelItemString(value, "$ref"); id = PyDict_GetItemString(value, "$id"); if (id == NULL) { id = Py_None; Py_INCREF(id); } else { Py_INCREF(id); PyDict_DelItemString(value, "$id"); } database = PyDict_GetItemString(value, "$db"); if (database == NULL) { database = Py_None; Py_INCREF(database); } else { Py_INCREF(database); PyDict_DelItemString(value, "$db"); } dbref = PyObject_CallFunctionObjArgs(state->DBRef, collection, id, database, value, NULL); Py_DECREF(value); value = dbref; Py_DECREF(id); Py_DECREF(collection); Py_DECREF(database); if (!value) { return NULL; } } *position += size; break; } case 4: { unsigned size, end; if (max < 4) { goto invalid; } memcpy(&size, buffer + *position, 4); if (max < size) { goto invalid; } end = *position + size - 1; /* Check for bad eoo */ if (buffer[end]) { goto invalid; } *position += 4; value = PyList_New(0); if (!value) { return NULL; } while (*position < end) { PyObject* to_append; int bson_type = (int)buffer[(*position)++]; size_t key_size = strlen(buffer + *position); if (max < key_size) { Py_DECREF(value); goto invalid; } /* just skip the key, they're in order. */ *position += (unsigned)key_size + 1; if (Py_EnterRecursiveCall(" while decoding a list value")) { Py_DECREF(value); return NULL; } to_append = get_value(self, buffer, position, bson_type, max - (unsigned)key_size, as_class, tz_aware, uuid_subtype); Py_LeaveRecursiveCall(); if (!to_append) { Py_DECREF(value); return NULL; } PyList_Append(value, to_append); Py_DECREF(to_append); } (*position)++; break; } case 5: { PyObject* data; PyObject* st; unsigned length, subtype; if (max < 4) { goto invalid; } memcpy(&length, buffer + *position, 4); if (max < length) { goto invalid; } subtype = (unsigned char)buffer[*position + 4]; #if PY_MAJOR_VERSION >= 3 /* Python3 special case. Decode BSON binary subtype 0 to bytes. */ if (subtype == 0) { value = PyBytes_FromStringAndSize(buffer + *position + 5, length); *position += length + 5; break; } if (subtype == 2) { data = PyBytes_FromStringAndSize(buffer + *position + 9, length - 4); } else { data = PyBytes_FromStringAndSize(buffer + *position + 5, length); } #else if (subtype == 2) { data = PyString_FromStringAndSize(buffer + *position + 9, length - 4); } else { data = PyString_FromStringAndSize(buffer + *position + 5, length); } #endif if (!data) { return NULL; } if ((subtype == 3 || subtype == 4) && state->UUID) { // Encode as UUID, not Binary PyObject* kwargs; PyObject* args = PyTuple_New(0); if (!args) { Py_DECREF(data); return NULL; } kwargs = PyDict_New(); if (!kwargs) { Py_DECREF(data); Py_DECREF(args); return NULL; } assert(length == 16); // UUID should always be 16 bytes if (uuid_subtype == CSHARP_LEGACY) { /* Legacy C# byte order */ if ((PyDict_SetItemString(kwargs, "bytes_le", data)) == -1) goto uuiderror; } else { if (uuid_subtype == JAVA_LEGACY) { /* Convert from legacy java byte order */ char big_endian[16]; _fix_java(buffer + *position + 5, big_endian); /* Free the previously created PyString object */ Py_DECREF(data); #if PY_MAJOR_VERSION >= 3 data = PyBytes_FromStringAndSize(big_endian, length); #else data = PyString_FromStringAndSize(big_endian, length); #endif if (data == NULL) goto uuiderror; } if ((PyDict_SetItemString(kwargs, "bytes", data)) == -1) goto uuiderror; } value = PyObject_Call(state->UUID, args, kwargs); Py_DECREF(args); Py_DECREF(kwargs); Py_DECREF(data); if (!value) { return NULL; } *position += length + 5; break; uuiderror: Py_DECREF(args); Py_DECREF(kwargs); Py_XDECREF(data); return NULL; } #if PY_MAJOR_VERSION >= 3 st = PyLong_FromLong(subtype); #else st = PyInt_FromLong(subtype); #endif if (!st) { Py_DECREF(data); return NULL; } value = PyObject_CallFunctionObjArgs(state->Binary, data, st, NULL); Py_DECREF(st); Py_DECREF(data); if (!value) { return NULL; } *position += length + 5; break; } case 6: case 10: { value = Py_None; Py_INCREF(value); break; } case 7: { if (max < 12) { goto invalid; } #if PY_MAJOR_VERSION >= 3 value = PyObject_CallFunction(state->ObjectId, "y#", buffer + *position, 12); #else value = PyObject_CallFunction(state->ObjectId, "s#", buffer + *position, 12); #endif if (!value) { return NULL; } *position += 12; break; } case 8: { value = buffer[(*position)++] ? Py_True : Py_False; Py_INCREF(value); break; } case 9: { PyObject* naive; PyObject* replace; PyObject* args; PyObject* kwargs; if (max < 8) { goto invalid; } naive = datetime_from_millis(*(long long*)(buffer + *position)); *position += 8; if (!tz_aware) { /* In the naive case, we're done here. */ value = naive; break; } if (!naive) { return NULL; } replace = PyObject_GetAttrString(naive, "replace"); Py_DECREF(naive); if (!replace) { return NULL; } args = PyTuple_New(0); if (!args) { Py_DECREF(replace); return NULL; } kwargs = PyDict_New(); if (!kwargs) { Py_DECREF(replace); Py_DECREF(args); return NULL; } if (PyDict_SetItemString(kwargs, "tzinfo", state->UTC) == -1) { Py_DECREF(replace); Py_DECREF(args); Py_DECREF(kwargs); return NULL; } value = PyObject_Call(replace, args, kwargs); Py_DECREF(replace); Py_DECREF(args); Py_DECREF(kwargs); break; } case 11: { PyObject* pattern; int flags; size_t flags_length, i; size_t pattern_length = strlen(buffer + *position); if (pattern_length > BSON_MAX_SIZE || max < pattern_length) { goto invalid; } pattern = PyUnicode_DecodeUTF8(buffer + *position, pattern_length, "strict"); if (!pattern) { return NULL; } *position += (unsigned)pattern_length + 1; flags_length = strlen(buffer + *position); if (flags_length > BSON_MAX_SIZE || (BSON_MAX_SIZE - pattern_length) < flags_length) { Py_DECREF(pattern); goto invalid; } if (max < pattern_length + flags_length) { Py_DECREF(pattern); goto invalid; } flags = 0; for (i = 0; i < flags_length; i++) { if (buffer[*position + i] == 'i') { flags |= 2; } else if (buffer[*position + i] == 'l') { flags |= 4; } else if (buffer[*position + i] == 'm') { flags |= 8; } else if (buffer[*position + i] == 's') { flags |= 16; } else if (buffer[*position + i] == 'u') { flags |= 32; } else if (buffer[*position + i] == 'x') { flags |= 64; } } *position += (unsigned)flags_length + 1; value = PyObject_CallFunction(state->RECompile, "Oi", pattern, flags); Py_DECREF(pattern); break; } case 12: { unsigned coll_length; PyObject* collection; PyObject* id; if (max < 4) { goto invalid; } memcpy(&coll_length, buffer + *position, 4); /* Encoded string length + string + 12 byte ObjectId */ if (max < 4 + coll_length + 12) { goto invalid; } *position += 4; /* Strings must end in \0 */ if (buffer[*position + coll_length - 1]) { goto invalid; } collection = PyUnicode_DecodeUTF8(buffer + *position, coll_length - 1, "strict"); if (!collection) { return NULL; } *position += coll_length; id = PyObject_CallFunction(state->ObjectId, "s#", buffer + *position, 12); if (!id) { Py_DECREF(collection); return NULL; } *position += 12; value = PyObject_CallFunctionObjArgs(state->DBRef, collection, id, NULL); Py_DECREF(collection); Py_DECREF(id); break; } case 13: { PyObject* code; unsigned value_length; if (max < 4) { goto invalid; } memcpy(&value_length, buffer + *position, 4); /* Encoded string length + string */ if (max < 4 + value_length) { goto invalid; } *position += 4; /* Strings must end in \0 */ if (buffer[*position + value_length - 1]) { goto invalid; } code = PyUnicode_DecodeUTF8(buffer + *position, value_length - 1, "strict"); if (!code) { return NULL; } *position += value_length; value = PyObject_CallFunctionObjArgs(state->Code, code, NULL, NULL); Py_DECREF(code); break; } case 15: { unsigned c_w_s_size; unsigned code_size; unsigned scope_size; PyObject* code; PyObject* scope; if (max < 8) { goto invalid; } memcpy(&c_w_s_size, buffer + *position, 4); *position += 4; if (max < c_w_s_size) { goto invalid; } memcpy(&code_size, buffer + *position, 4); /* code_w_scope length + code length + code + scope length */ if (max < 4 + 4 + code_size + 4) { goto invalid; } *position += 4; /* Strings must end in \0 */ if (buffer[*position + code_size - 1]) { goto invalid; } code = PyUnicode_DecodeUTF8(buffer + *position, code_size - 1, "strict"); if (!code) { return NULL; } *position += code_size; memcpy(&scope_size, buffer + *position, 4); if (scope_size < BSON_MIN_SIZE) { Py_DECREF(code); goto invalid; } /* code length + code + scope length + scope */ if ((4 + code_size + 4 + scope_size) != c_w_s_size) { Py_DECREF(code); goto invalid; } /* Check for bad eoo */ if (buffer[*position + scope_size - 1]) { goto invalid; } scope = elements_to_dict(self, buffer + *position + 4, scope_size - 5, (PyObject*)&PyDict_Type, tz_aware, uuid_subtype); if (!scope) { Py_DECREF(code); return NULL; } *position += scope_size; value = PyObject_CallFunctionObjArgs(state->Code, code, scope, NULL); Py_DECREF(code); Py_DECREF(scope); break; } case 16: { int i; if (max < 4) { goto invalid; } memcpy(&i, buffer + *position, 4); #if PY_MAJOR_VERSION >= 3 value = PyLong_FromLong(i); #else value = PyInt_FromLong(i); #endif if (!value) { return NULL; } *position += 4; break; } case 17: { unsigned int time, inc; if (max < 8) { goto invalid; } memcpy(&inc, buffer + *position, 4); memcpy(&time, buffer + *position + 4, 4); value = PyObject_CallFunction(state->Timestamp, "II", time, inc); if (!value) { return NULL; } *position += 8; break; } case 18: { long long ll; if (max < 8) { goto invalid; } memcpy(&ll, buffer + *position, 8); value = PyLong_FromLongLong(ll); if (!value) { return NULL; } *position += 8; break; } case -1: { value = PyObject_CallFunctionObjArgs(state->MinKey, NULL); break; } case 127: { value = PyObject_CallFunctionObjArgs(state->MaxKey, NULL); break; } default: { PyObject* InvalidDocument = _error("InvalidDocument"); if (InvalidDocument) { PyErr_SetString(InvalidDocument, "no c decoder for this type yet"); Py_DECREF(InvalidDocument); } return NULL; } } return value; invalid: error = _error("InvalidBSON"); if (error) { PyErr_SetString(error, "invalid length or type code"); Py_DECREF(error); } return NULL; } static PyObject* _elements_to_dict(PyObject* self, const char* string, unsigned max, PyObject* as_class, unsigned char tz_aware, unsigned char uuid_subtype) { unsigned position = 0; PyObject* dict = PyObject_CallObject(as_class, NULL); if (!dict) { return NULL; } while (position < max) { PyObject* name; PyObject* value; int type = (int)string[position++]; size_t name_length = strlen(string + position); if (name_length > BSON_MAX_SIZE || position + name_length >= max) { PyObject* InvalidBSON = _error("InvalidBSON"); if (InvalidBSON) { PyErr_SetNone(InvalidBSON); Py_DECREF(InvalidBSON); } Py_DECREF(dict); return NULL; } name = PyUnicode_DecodeUTF8(string + position, name_length, "strict"); if (!name) { Py_DECREF(dict); return NULL; } position += (unsigned)name_length + 1; value = get_value(self, string, &position, type, max - position, as_class, tz_aware, uuid_subtype); if (!value) { Py_DECREF(name); Py_DECREF(dict); return NULL; } PyObject_SetItem(dict, name, value); Py_DECREF(name); Py_DECREF(value); } return dict; } static PyObject* elements_to_dict(PyObject* self, const char* string, unsigned max, PyObject* as_class, unsigned char tz_aware, unsigned char uuid_subtype) { PyObject* result; if (Py_EnterRecursiveCall(" while decoding a BSON document")) return NULL; result = _elements_to_dict(self, string, max, as_class, tz_aware, uuid_subtype); Py_LeaveRecursiveCall(); return result; } static PyObject* _cbson_bson_to_dict(PyObject* self, PyObject* args) { int size; Py_ssize_t total_size; const char* string; PyObject* bson; PyObject* as_class; unsigned char tz_aware; unsigned char uuid_subtype; PyObject* dict; PyObject* remainder; PyObject* result; if (!PyArg_ParseTuple(args, "OObb", &bson, &as_class, &tz_aware, &uuid_subtype)) { return NULL; } #if PY_MAJOR_VERSION >= 3 if (!PyBytes_Check(bson)) { PyErr_SetString(PyExc_TypeError, "argument to _bson_to_dict must be a bytes object"); #else if (!PyString_Check(bson)) { PyErr_SetString(PyExc_TypeError, "argument to _bson_to_dict must be a string"); #endif return NULL; } #if PY_MAJOR_VERSION >= 3 total_size = PyBytes_Size(bson); #else total_size = PyString_Size(bson); #endif if (total_size < BSON_MIN_SIZE) { PyObject* InvalidBSON = _error("InvalidBSON"); if (InvalidBSON) { PyErr_SetString(InvalidBSON, "not enough data for a BSON document"); Py_DECREF(InvalidBSON); } return NULL; } #if PY_MAJOR_VERSION >= 3 string = PyBytes_AsString(bson); #else string = PyString_AsString(bson); #endif if (!string) { return NULL; } memcpy(&size, string, 4); if (size < BSON_MIN_SIZE) { PyObject* InvalidBSON = _error("InvalidBSON"); if (InvalidBSON) { PyErr_SetString(InvalidBSON, "invalid message size"); Py_DECREF(InvalidBSON); } return NULL; } if (total_size < size || total_size > BSON_MAX_SIZE) { PyObject* InvalidBSON = _error("InvalidBSON"); if (InvalidBSON) { PyErr_SetString(InvalidBSON, "objsize too large"); Py_DECREF(InvalidBSON); } return NULL; } if (size != total_size || string[size - 1]) { PyObject* InvalidBSON = _error("InvalidBSON"); if (InvalidBSON) { PyErr_SetString(InvalidBSON, "bad eoo"); Py_DECREF(InvalidBSON); } return NULL; } dict = elements_to_dict(self, string + 4, (unsigned)size - 5, as_class, tz_aware, uuid_subtype); if (!dict) { return NULL; } #if PY_MAJOR_VERSION >= 3 remainder = PyBytes_FromStringAndSize(string + size, total_size - size); #else remainder = PyString_FromStringAndSize(string + size, total_size - size); #endif if (!remainder) { Py_DECREF(dict); return NULL; } result = Py_BuildValue("OO", dict, remainder); Py_DECREF(dict); Py_DECREF(remainder); return result; } static PyObject* _cbson_decode_all(PyObject* self, PyObject* args) { int size; Py_ssize_t total_size; const char* string; PyObject* bson; PyObject* dict; PyObject* result; PyObject* as_class = (PyObject*)&PyDict_Type; unsigned char tz_aware = 1; unsigned char uuid_subtype = 3; if (!PyArg_ParseTuple(args, "O|Obb", &bson, &as_class, &tz_aware, &uuid_subtype)) { return NULL; } #if PY_MAJOR_VERSION >= 3 if (!PyBytes_Check(bson)) { PyErr_SetString(PyExc_TypeError, "argument to decode_all must be a bytes object"); #else if (!PyString_Check(bson)) { PyErr_SetString(PyExc_TypeError, "argument to decode_all must be a string"); #endif return NULL; } #if PY_MAJOR_VERSION >= 3 total_size = PyBytes_Size(bson); string = PyBytes_AsString(bson); #else total_size = PyString_Size(bson); string = PyString_AsString(bson); #endif if (!string) { return NULL; } if (!(result = PyList_New(0))) return NULL; while (total_size > 0) { if (total_size < BSON_MIN_SIZE) { PyObject* InvalidBSON = _error("InvalidBSON"); if (InvalidBSON) { PyErr_SetString(InvalidBSON, "not enough data for a BSON document"); Py_DECREF(InvalidBSON); } Py_DECREF(result); return NULL; } memcpy(&size, string, 4); if (size < BSON_MIN_SIZE) { PyObject* InvalidBSON = _error("InvalidBSON"); if (InvalidBSON) { PyErr_SetString(InvalidBSON, "invalid message size"); Py_DECREF(InvalidBSON); } Py_DECREF(result); return NULL; } if (total_size < size) { PyObject* InvalidBSON = _error("InvalidBSON"); if (InvalidBSON) { PyErr_SetString(InvalidBSON, "objsize too large"); Py_DECREF(InvalidBSON); } Py_DECREF(result); return NULL; } if (string[size - 1]) { PyObject* InvalidBSON = _error("InvalidBSON"); if (InvalidBSON) { PyErr_SetString(InvalidBSON, "bad eoo"); Py_DECREF(InvalidBSON); } Py_DECREF(result); return NULL; } dict = elements_to_dict(self, string + 4, (unsigned)size - 5, as_class, tz_aware, uuid_subtype); if (!dict) { Py_DECREF(result); return NULL; } PyList_Append(result, dict); Py_DECREF(dict); string += size; total_size -= size; } return result; } static PyMethodDef _CBSONMethods[] = { {"_dict_to_bson", _cbson_dict_to_bson, METH_VARARGS, "convert a dictionary to a string containing its BSON representation."}, {"_bson_to_dict", _cbson_bson_to_dict, METH_VARARGS, "convert a BSON string to a SON object."}, {"decode_all", _cbson_decode_all, METH_VARARGS, "convert binary data to a sequence of documents."}, {NULL, NULL, 0, NULL} }; #if PY_MAJOR_VERSION >= 3 #define INITERROR return NULL static int _cbson_traverse(PyObject *m, visitproc visit, void *arg) { Py_VISIT(GETSTATE(m)->Binary); Py_VISIT(GETSTATE(m)->Code); Py_VISIT(GETSTATE(m)->ObjectId); Py_VISIT(GETSTATE(m)->DBRef); Py_VISIT(GETSTATE(m)->RECompile); Py_VISIT(GETSTATE(m)->UUID); Py_VISIT(GETSTATE(m)->Timestamp); Py_VISIT(GETSTATE(m)->MinKey); Py_VISIT(GETSTATE(m)->MaxKey); Py_VISIT(GETSTATE(m)->UTC); Py_VISIT(GETSTATE(m)->REType); return 0; } static int _cbson_clear(PyObject *m) { Py_CLEAR(GETSTATE(m)->Binary); Py_CLEAR(GETSTATE(m)->Code); Py_CLEAR(GETSTATE(m)->ObjectId); Py_CLEAR(GETSTATE(m)->DBRef); Py_CLEAR(GETSTATE(m)->RECompile); Py_CLEAR(GETSTATE(m)->UUID); Py_CLEAR(GETSTATE(m)->Timestamp); Py_CLEAR(GETSTATE(m)->MinKey); Py_CLEAR(GETSTATE(m)->MaxKey); Py_CLEAR(GETSTATE(m)->UTC); Py_CLEAR(GETSTATE(m)->REType); return 0; } static struct PyModuleDef moduledef = { PyModuleDef_HEAD_INIT, "_cbson", NULL, sizeof(struct module_state), _CBSONMethods, NULL, _cbson_traverse, _cbson_clear, NULL }; PyMODINIT_FUNC PyInit__cbson(void) #else #define INITERROR return PyMODINIT_FUNC init_cbson(void) #endif { PyObject *m; PyObject *c_api_object; static void *_cbson_API[_cbson_API_POINTER_COUNT]; PyDateTime_IMPORT; if (PyDateTimeAPI == NULL) { INITERROR; } /* Export C API */ _cbson_API[_cbson_buffer_write_bytes_INDEX] = (void *) buffer_write_bytes; _cbson_API[_cbson_write_dict_INDEX] = (void *) write_dict; _cbson_API[_cbson_write_pair_INDEX] = (void *) write_pair; _cbson_API[_cbson_decode_and_write_pair_INDEX] = (void *) decode_and_write_pair; #if PY_VERSION_HEX >= 0x03010000 /* PyCapsule is new in python 3.1 */ c_api_object = PyCapsule_New((void *) _cbson_API, "_cbson._C_API", NULL); #else c_api_object = PyCObject_FromVoidPtr((void *) _cbson_API, NULL); #endif if (c_api_object == NULL) INITERROR; #if PY_MAJOR_VERSION >= 3 m = PyModule_Create(&moduledef); #else m = Py_InitModule("_cbson", _CBSONMethods); #endif if (m == NULL) { Py_DECREF(c_api_object); INITERROR; } /* Import several python objects */ if (_reload_python_objects(m)) { Py_DECREF(c_api_object); #if PY_MAJOR_VERSION >= 3 Py_DECREF(m); #endif INITERROR; } if (PyModule_AddObject(m, "_C_API", c_api_object) < 0) { Py_DECREF(c_api_object); #if PY_MAJOR_VERSION >= 3 Py_DECREF(m); #endif INITERROR; } #if PY_MAJOR_VERSION >= 3 return m; #endif } pymongo-2.6.3/bson/_cbsonmodule.h000066400000000000000000000060571223300253600170010ustar00rootroot00000000000000/* * Copyright 2012 10gen, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef _CBSONMODULE_H #define _CBSONMODULE_H /* Py_ssize_t was new in python 2.5. See conversion * guidlines in http://www.python.org/dev/peps/pep-0353 * */ #if PY_VERSION_HEX < 0x02050000 && !defined(PY_SSIZE_T_MIN) typedef int Py_ssize_t; #define PY_SSIZE_T_MAX INT_MAX #define PY_SSIZE_T_MIN INT_MIN #endif /* C API functions */ #define _cbson_buffer_write_bytes_INDEX 0 #define _cbson_buffer_write_bytes_RETURN int #define _cbson_buffer_write_bytes_PROTO (buffer_t buffer, const char* data, int size) #define _cbson_write_dict_INDEX 1 #define _cbson_write_dict_RETURN int #define _cbson_write_dict_PROTO (PyObject* self, buffer_t buffer, PyObject* dict, unsigned char check_keys, unsigned char uuid_subtype, unsigned char top_level) #define _cbson_write_pair_INDEX 2 #define _cbson_write_pair_RETURN int #define _cbson_write_pair_PROTO (PyObject* self, buffer_t buffer, const char* name, Py_ssize_t name_length, PyObject* value, unsigned char check_keys, unsigned char uuid_subtype, unsigned char allow_id) #define _cbson_decode_and_write_pair_INDEX 3 #define _cbson_decode_and_write_pair_RETURN int #define _cbson_decode_and_write_pair_PROTO (PyObject* self, buffer_t buffer, PyObject* key, PyObject* value, unsigned char check_keys, unsigned char uuid_subtype, unsigned char top_level) /* Total number of C API pointers */ #define _cbson_API_POINTER_COUNT 4 #ifdef _CBSON_MODULE /* This section is used when compiling _cbsonmodule */ static _cbson_buffer_write_bytes_RETURN buffer_write_bytes _cbson_buffer_write_bytes_PROTO; static _cbson_write_dict_RETURN write_dict _cbson_write_dict_PROTO; static _cbson_write_pair_RETURN write_pair _cbson_write_pair_PROTO; static _cbson_decode_and_write_pair_RETURN decode_and_write_pair _cbson_decode_and_write_pair_PROTO; #else /* This section is used in modules that use _cbsonmodule's API */ static void **_cbson_API; #define buffer_write_bytes (*(_cbson_buffer_write_bytes_RETURN (*)_cbson_buffer_write_bytes_PROTO) _cbson_API[_cbson_buffer_write_bytes_INDEX]) #define write_dict (*(_cbson_write_dict_RETURN (*)_cbson_write_dict_PROTO) _cbson_API[_cbson_write_dict_INDEX]) #define write_pair (*(_cbson_write_pair_RETURN (*)_cbson_write_pair_PROTO) _cbson_API[_cbson_write_pair_INDEX]) #define decode_and_write_pair (*(_cbson_decode_and_write_pair_RETURN (*)_cbson_decode_and_write_pair_PROTO) _cbson_API[_cbson_decode_and_write_pair_INDEX]) #define _cbson_IMPORT _cbson_API = (void **)PyCapsule_Import("_cbson._C_API", 0) #endif #endif // _CBSONMODULE_H pymongo-2.6.3/bson/binary.py000066400000000000000000000150771223300253600160170ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. try: from uuid import UUID except ImportError: # Python2.4 doesn't have a uuid module. pass from bson.py3compat import PY3, binary_type """Tools for representing BSON binary data. """ BINARY_SUBTYPE = 0 """BSON binary subtype for binary data. This is the default subtype for binary data. .. versionadded:: 1.5 """ FUNCTION_SUBTYPE = 1 """BSON binary subtype for functions. .. versionadded:: 1.5 """ OLD_BINARY_SUBTYPE = 2 """Old BSON binary subtype for binary data. This is the old default subtype, the current default is :data:`BINARY_SUBTYPE`. .. versionadded:: 1.7 """ OLD_UUID_SUBTYPE = 3 """Old BSON binary subtype for a UUID. :class:`uuid.UUID` instances will automatically be encoded by :mod:`bson` using this subtype. .. versionadded:: 2.1 """ UUID_SUBTYPE = 4 """BSON binary subtype for a UUID. This is the new BSON binary subtype for UUIDs. The current default is :data:`OLD_UUID_SUBTYPE` but will change to this in a future release. .. versionchanged:: 2.1 Changed to subtype 4. .. versionadded:: 1.5 """ JAVA_LEGACY = 5 """Used with :attr:`pymongo.collection.Collection.uuid_subtype` to specify that UUIDs should be stored in the legacy byte order used by the Java driver. :class:`uuid.UUID` instances will automatically be encoded by :mod:`bson` using :data:`OLD_UUID_SUBTYPE`. .. versionadded:: 2.3 """ CSHARP_LEGACY = 6 """Used with :attr:`pymongo.collection.Collection.uuid_subtype` to specify that UUIDs should be stored in the legacy byte order used by the C# driver. :class:`uuid.UUID` instances will automatically be encoded by :mod:`bson` using :data:`OLD_UUID_SUBTYPE`. .. versionadded:: 2.3 """ ALL_UUID_SUBTYPES = (OLD_UUID_SUBTYPE, UUID_SUBTYPE, JAVA_LEGACY, CSHARP_LEGACY) MD5_SUBTYPE = 5 """BSON binary subtype for an MD5 hash. .. versionadded:: 1.5 """ USER_DEFINED_SUBTYPE = 128 """BSON binary subtype for any user defined structure. .. versionadded:: 1.5 """ class Binary(binary_type): """Representation of BSON binary data. This is necessary because we want to represent Python strings as the BSON string type. We need to wrap binary data so we can tell the difference between what should be considered binary data and what should be considered a string when we encode to BSON. Raises TypeError if `data` is not an instance of :class:`str` (:class:`bytes` in python 3) or `subtype` is not an instance of :class:`int`. Raises ValueError if `subtype` is not in [0, 256). .. note:: In python 3 instances of Binary with subtype 0 will be decoded directly to :class:`bytes`. :Parameters: - `data`: the binary data to represent - `subtype` (optional): the `binary subtype `_ to use """ def __new__(cls, data, subtype=BINARY_SUBTYPE): if not isinstance(data, binary_type): raise TypeError("data must be an " "instance of %s" % (binary_type.__name__,)) if not isinstance(subtype, int): raise TypeError("subtype must be an instance of int") if subtype >= 256 or subtype < 0: raise ValueError("subtype must be contained in [0, 256)") self = binary_type.__new__(cls, data) self.__subtype = subtype return self @property def subtype(self): """Subtype of this binary data. """ return self.__subtype def __getnewargs__(self): # Work around http://bugs.python.org/issue7382 data = super(Binary, self).__getnewargs__()[0] if PY3 and not isinstance(data, binary_type): data = data.encode('latin-1') return data, self.__subtype def __eq__(self, other): if isinstance(other, Binary): return ((self.__subtype, binary_type(self)) == (other.subtype, binary_type(other))) # We don't return NotImplemented here because if we did then # Binary("foo") == "foo" would return True, since Binary is a # subclass of str... return False def __ne__(self, other): return not self == other def __repr__(self): return "Binary(%s, %s)" % (binary_type.__repr__(self), self.__subtype) class UUIDLegacy(Binary): """UUID wrapper to support working with UUIDs stored as legacy BSON binary subtype 3. .. doctest:: >>> import uuid >>> from bson.binary import Binary, UUIDLegacy, UUID_SUBTYPE >>> my_uuid = uuid.uuid4() >>> coll = db.test >>> coll.uuid_subtype = UUID_SUBTYPE >>> coll.insert({'uuid': Binary(my_uuid.bytes, 3)}) ObjectId('...') >>> coll.find({'uuid': my_uuid}).count() 0 >>> coll.find({'uuid': UUIDLegacy(my_uuid)}).count() 1 >>> coll.find({'uuid': UUIDLegacy(my_uuid)})[0]['uuid'] UUID('...') >>> >>> # Convert from subtype 3 to subtype 4 >>> doc = coll.find_one({'uuid': UUIDLegacy(my_uuid)}) >>> coll.save(doc) ObjectId('...') >>> coll.find({'uuid': UUIDLegacy(my_uuid)}).count() 0 >>> coll.find({'uuid': {'$in': [UUIDLegacy(my_uuid), my_uuid]}}).count() 1 >>> coll.find_one({'uuid': my_uuid})['uuid'] UUID('...') Raises TypeError if `obj` is not an instance of :class:`~uuid.UUID`. :Parameters: - `obj`: An instance of :class:`~uuid.UUID`. """ def __new__(cls, obj): if not isinstance(obj, UUID): raise TypeError("obj must be an instance of uuid.UUID") # Python 3.0(.1) returns a bytearray instance for bytes (3.1 and # newer just return a bytes instance). Convert that to binary_type # for compatibility. self = Binary.__new__(cls, binary_type(obj.bytes), OLD_UUID_SUBTYPE) self.__uuid = obj return self def __getnewargs__(self): # Support copy and deepcopy return (self.__uuid,) @property def uuid(self): """UUID instance wrapped by this UUIDLegacy instance. """ return self.__uuid def __repr__(self): return "UUIDLegacy('%s')" % self.__uuid pymongo-2.6.3/bson/buffer.c000066400000000000000000000077041223300253600155740ustar00rootroot00000000000000/* * Copyright 2009-2012 10gen, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #include #include #include "buffer.h" #define INITIAL_BUFFER_SIZE 256 struct buffer { char* buffer; int size; int position; }; /* Allocate and return a new buffer. * Return NULL on allocation failure. */ buffer_t buffer_new(void) { buffer_t buffer; buffer = (buffer_t)malloc(sizeof(struct buffer)); if (buffer == NULL) { return NULL; } buffer->size = INITIAL_BUFFER_SIZE; buffer->position = 0; buffer->buffer = (char*)malloc(sizeof(char) * INITIAL_BUFFER_SIZE); if (buffer->buffer == NULL) { free(buffer); return NULL; } return buffer; } /* Free the memory allocated for `buffer`. * Return non-zero on failure. */ int buffer_free(buffer_t buffer) { if (buffer == NULL) { return 1; } free(buffer->buffer); free(buffer); return 0; } /* Grow `buffer` to at least `min_length`. * Return non-zero on allocation failure. */ static int buffer_grow(buffer_t buffer, int min_length) { int old_size = 0; int size = buffer->size; char* old_buffer = buffer->buffer; if (size >= min_length) { return 0; } while (size < min_length) { old_size = size; size *= 2; if (size <= old_size) { /* Size did not increase. Could be an overflow * or size < 1. Just go with min_length. */ size = min_length; } } buffer->buffer = (char*)realloc(buffer->buffer, sizeof(char) * size); if (buffer->buffer == NULL) { free(old_buffer); free(buffer); return 1; } buffer->size = size; return 0; } /* Assure that `buffer` has at least `size` free bytes (and grow if needed). * Return non-zero on allocation failure. */ static int buffer_assure_space(buffer_t buffer, int size) { if (buffer->position + size <= buffer->size) { return 0; } return buffer_grow(buffer, buffer->position + size); } /* Save `size` bytes from the current position in `buffer` (and grow if needed). * Return offset for writing, or -1 on allocation failure. */ buffer_position buffer_save_space(buffer_t buffer, int size) { int position = buffer->position; if (buffer_assure_space(buffer, size) != 0) { return -1; } buffer->position += size; return position; } /* Write `size` bytes from `data` to `buffer` (and grow if needed). * Return non-zero on allocation failure. */ int buffer_write(buffer_t buffer, const char* data, int size) { if (buffer_assure_space(buffer, size) != 0) { return 1; } memcpy(buffer->buffer + buffer->position, data, size); buffer->position += size; return 0; } /* Write `size` bytes from `data` to `buffer` at position `position`. * Does not change the internal position of `buffer`. * Return non-zero if buffer isn't large enough for write. */ int buffer_write_at_position(buffer_t buffer, buffer_position position, const char* data, int size) { if (position + size > buffer->size) { buffer_free(buffer); return 1; } memcpy(buffer->buffer + position, data, size); return 0; } int buffer_get_position(buffer_t buffer) { return buffer->position; } char* buffer_get_buffer(buffer_t buffer) { return buffer->buffer; } void buffer_update_position(buffer_t buffer, buffer_position new_position) { buffer->position = new_position; } pymongo-2.6.3/bson/buffer.h000066400000000000000000000040121223300253600155660ustar00rootroot00000000000000/* * Copyright 2009-2012 10gen, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef BUFFER_H #define BUFFER_H /* Note: if any of these functions return a failure condition then the buffer * has already been freed. */ /* A buffer */ typedef struct buffer* buffer_t; /* A position in the buffer */ typedef int buffer_position; /* Allocate and return a new buffer. * Return NULL on allocation failure. */ buffer_t buffer_new(void); /* Free the memory allocated for `buffer`. * Return non-zero on failure. */ int buffer_free(buffer_t buffer); /* Save `size` bytes from the current position in `buffer` (and grow if needed). * Return offset for writing, or -1 on allocation failure. */ buffer_position buffer_save_space(buffer_t buffer, int size); /* Write `size` bytes from `data` to `buffer` (and grow if needed). * Return non-zero on allocation failure. */ int buffer_write(buffer_t buffer, const char* data, int size); /* Write `size` bytes from `data` to `buffer` at position `position`. * Does not change the internal position of `buffer`. * Return non-zero if buffer isn't large enough for write. */ int buffer_write_at_position(buffer_t buffer, buffer_position position, const char* data, int size); /* Getters for the internals of a buffer_t. * Should try to avoid using these as much as possible * since they break the abstraction. */ buffer_position buffer_get_position(buffer_t buffer); char* buffer_get_buffer(buffer_t buffer); void buffer_update_position(buffer_t buffer, buffer_position new_position); #endif pymongo-2.6.3/bson/code.py000066400000000000000000000050411223300253600154330ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tools for representing JavaScript code in BSON. """ class Code(str): """BSON's JavaScript code type. Raises :class:`TypeError` if `code` is not an instance of :class:`basestring` (:class:`str` in python 3) or `scope` is not ``None`` or an instance of :class:`dict`. Scope variables can be set by passing a dictionary as the `scope` argument or by using keyword arguments. If a variable is set as a keyword argument it will override any setting for that variable in the `scope` dictionary. :Parameters: - `code`: string containing JavaScript code to be evaluated - `scope` (optional): dictionary representing the scope in which `code` should be evaluated - a mapping from identifiers (as strings) to values - `**kwargs` (optional): scope variables can also be passed as keyword arguments .. versionadded:: 1.9 Ability to pass scope values using keyword arguments. """ def __new__(cls, code, scope=None, **kwargs): if not isinstance(code, basestring): raise TypeError("code must be an " "instance of %s" % (basestring.__name__,)) self = str.__new__(cls, code) try: self.__scope = code.scope except AttributeError: self.__scope = {} if scope is not None: if not isinstance(scope, dict): raise TypeError("scope must be an instance of dict") self.__scope.update(scope) self.__scope.update(kwargs) return self @property def scope(self): """Scope dictionary for this instance. """ return self.__scope def __repr__(self): return "Code(%s, %r)" % (str.__repr__(self), self.__scope) def __eq__(self, other): if isinstance(other, Code): return (self.__scope, str(self)) == (other.__scope, str(other)) return False def __ne__(self, other): return not self == other pymongo-2.6.3/bson/dbref.py000066400000000000000000000114011223300253600156000ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tools for manipulating DBRefs (references to MongoDB documents).""" from copy import deepcopy from bson.son import SON class DBRef(object): """A reference to a document stored in MongoDB. """ def __init__(self, collection, id, database=None, _extra={}, **kwargs): """Initialize a new :class:`DBRef`. Raises :class:`TypeError` if `collection` or `database` is not an instance of :class:`basestring` (:class:`str` in python 3). `database` is optional and allows references to documents to work across databases. Any additional keyword arguments will create additional fields in the resultant embedded document. :Parameters: - `collection`: name of the collection the document is stored in - `id`: the value of the document's ``"_id"`` field - `database` (optional): name of the database to reference - `**kwargs` (optional): additional keyword arguments will create additional, custom fields .. versionchanged:: 1.8 Now takes keyword arguments to specify additional fields. .. versionadded:: 1.1.1 The `database` parameter. .. mongodoc:: dbrefs """ if not isinstance(collection, basestring): raise TypeError("collection must be an " "instance of %s" % (basestring.__name__,)) if database is not None and not isinstance(database, basestring): raise TypeError("database must be an " "instance of %s" % (basestring.__name__,)) self.__collection = collection self.__id = id self.__database = database kwargs.update(_extra) self.__kwargs = kwargs @property def collection(self): """Get the name of this DBRef's collection as unicode. """ return self.__collection @property def id(self): """Get this DBRef's _id. """ return self.__id @property def database(self): """Get the name of this DBRef's database. Returns None if this DBRef doesn't specify a database. .. versionadded:: 1.1.1 """ return self.__database def __getattr__(self, key): try: return self.__kwargs[key] except KeyError: raise AttributeError(key) # Have to provide __setstate__ to avoid # infinite recursion since we override # __getattr__. def __setstate__(self, state): self.__dict__.update(state) def as_doc(self): """Get the SON document representation of this DBRef. Generally not needed by application developers """ doc = SON([("$ref", self.collection), ("$id", self.id)]) if self.database is not None: doc["$db"] = self.database doc.update(self.__kwargs) return doc def __repr__(self): extra = "".join([", %s=%r" % (k, v) for k, v in self.__kwargs.iteritems()]) if self.database is None: return "DBRef(%r, %r%s)" % (self.collection, self.id, extra) return "DBRef(%r, %r, %r%s)" % (self.collection, self.id, self.database, extra) def __eq__(self, other): if isinstance(other, DBRef): us = (self.__database, self.__collection, self.__id, self.__kwargs) them = (other.__database, other.__collection, other.__id, other.__kwargs) return us == them return NotImplemented def __ne__(self, other): return not self == other def __hash__(self): """Get a hash value for this :class:`DBRef`. .. versionadded:: 1.1 """ return hash((self.__collection, self.__id, self.__database, tuple(sorted(self.__kwargs.items())))) def __deepcopy__(self, memo): """Support function for `copy.deepcopy()`. .. versionadded:: 1.10 """ return DBRef(deepcopy(self.__collection, memo), deepcopy(self.__id, memo), deepcopy(self.__database, memo), deepcopy(self.__kwargs, memo)) pymongo-2.6.3/bson/encoding_helpers.c000066400000000000000000000105761223300253600176340ustar00rootroot00000000000000/* * Copyright 2009-2012 10gen, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #include "encoding_helpers.h" /* * Portions Copyright 2001 Unicode, Inc. * * Disclaimer * * This source code is provided as is by Unicode, Inc. No claims are * made as to fitness for any particular purpose. No warranties of any * kind are expressed or implied. The recipient agrees to determine * applicability of information provided. If this file has been * purchased on magnetic or optical media from Unicode, Inc., the * sole remedy for any claim will be exchange of defective media * within 90 days of receipt. * * Limitations on Rights to Redistribute This Code * * Unicode, Inc. hereby grants the right to freely use the information * supplied in this file in the creation of products supporting the * Unicode Standard, and to make copies of this file in any form * for internal or external distribution as long as this notice * remains attached. */ /* * Index into the table below with the first byte of a UTF-8 sequence to * get the number of trailing bytes that are supposed to follow it. */ static const char trailingBytesForUTF8[256] = { 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2, 3,3,3,3,3,3,3,3,4,4,4,4,5,5,5,5 }; /* --------------------------------------------------------------------- */ /* * Utility routine to tell whether a sequence of bytes is legal UTF-8. * This must be called with the length pre-determined by the first byte. * The length can be set by: * length = trailingBytesForUTF8[*source]+1; * and the sequence is illegal right away if there aren't that many bytes * available. * If presented with a length > 4, this returns 0. The Unicode * definition of UTF-8 goes up to 4-byte sequences. */ static unsigned char isLegalUTF8(const unsigned char* source, int length) { unsigned char a; const unsigned char* srcptr = source + length; switch (length) { default: return 0; /* Everything else falls through when "true"... */ case 4: if ((a = (*--srcptr)) < 0x80 || a > 0xBF) return 0; case 3: if ((a = (*--srcptr)) < 0x80 || a > 0xBF) return 0; case 2: if ((a = (*--srcptr)) > 0xBF) return 0; switch (*source) { /* no fall-through in this inner switch */ case 0xE0: if (a < 0xA0) return 0; break; case 0xF0: if (a < 0x90) return 0; break; case 0xF4: if (a > 0x8F) return 0; break; default: if (a < 0x80) return 0; } case 1: if (*source >= 0x80 && *source < 0xC2) return 0; if (*source > 0xF4) return 0; } return 1; } result_t check_string(const unsigned char* string, const int length, const char check_utf8, const char check_null) { int position = 0; /* By default we go character by character. Will be different for checking * UTF-8 */ int sequence_length = 1; if (!check_utf8 && !check_null) { return VALID; } while (position < length) { if (check_null && *(string + position) == 0) { return HAS_NULL; } if (check_utf8) { sequence_length = trailingBytesForUTF8[*(string + position)] + 1; if ((position + sequence_length) > length) { return NOT_UTF_8; } if (!isLegalUTF8(string + position, sequence_length)) { return NOT_UTF_8; } } position += sequence_length; } return VALID; } pymongo-2.6.3/bson/encoding_helpers.h000066400000000000000000000015411223300253600176310ustar00rootroot00000000000000/* * Copyright 2009-2012 10gen, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #ifndef ENCODING_HELPERS_H #define ENCODING_HELPERS_H typedef enum { VALID, NOT_UTF_8, HAS_NULL } result_t; result_t check_string(const unsigned char* string, const int length, const char check_utf8, const char check_null); #endif pymongo-2.6.3/bson/errors.py000066400000000000000000000022021223300253600160310ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Exceptions raised by the BSON package.""" class BSONError(Exception): """Base class for all BSON exceptions. """ class InvalidBSON(BSONError): """Raised when trying to create a BSON object from invalid data. """ class InvalidStringData(BSONError): """Raised when trying to encode a string containing non-UTF8 data. """ class InvalidDocument(BSONError): """Raised when trying to create a BSON object from an invalid document. """ class InvalidId(BSONError): """Raised when trying to create an ObjectId from invalid data. """ pymongo-2.6.3/bson/json_util.py000066400000000000000000000166431223300253600165410ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tools for using Python's :mod:`json` module with BSON documents. This module provides two helper methods `dumps` and `loads` that wrap the native :mod:`json` methods and provide explicit BSON conversion to and from json. This allows for specialized encoding and decoding of BSON documents into `Mongo Extended JSON `_'s *Strict* mode. This lets you encode / decode BSON documents to JSON even when they use special BSON types. Example usage (serialization):: .. doctest:: >>> from bson import Binary, Code >>> from bson.json_util import dumps >>> dumps([{'foo': [1, 2]}, ... {'bar': {'hello': 'world'}}, ... {'code': Code("function x() { return 1; }")}, ... {'bin': Binary("\x00\x01\x02\x03\x04")}]) '[{"foo": [1, 2]}, {"bar": {"hello": "world"}}, {"code": {"$scope": {}, "$code": "function x() { return 1; }"}}, {"bin": {"$type": "00", "$binary": "AAECAwQ="}}]' Example usage (deserialization):: .. doctest:: >>> from bson.json_util import loads >>> loads('[{"foo": [1, 2]}, {"bar": {"hello": "world"}}, {"code": {"$scope": {}, "$code": "function x() { return 1; }"}}, {"bin": {"$type": "00", "$binary": "AAECAwQ="}}]') [{u'foo': [1, 2]}, {u'bar': {u'hello': u'world'}}, {u'code': Code('function x() { return 1; }', {})}, {u'bin': Binary('\x00\x01\x02\x03\x04', 0)}] Alternatively, you can manually pass the `default` to :func:`json.dumps`. It won't handle :class:`~bson.binary.Binary` and :class:`~bson.code.Code` instances (as they are extended strings you can't provide custom defaults), but it will be faster as there is less recursion. .. versionchanged:: 2.3 Added dumps and loads helpers to automatically handle conversion to and from json and supports :class:`~bson.binary.Binary` and :class:`~bson.code.Code` .. versionchanged:: 1.9 Handle :class:`uuid.UUID` instances, whenever possible. .. versionchanged:: 1.8 Handle timezone aware datetime instances on encode, decode to timezone aware datetime instances. .. versionchanged:: 1.8 Added support for encoding/decoding :class:`~bson.max_key.MaxKey` and :class:`~bson.min_key.MinKey`, and for encoding :class:`~bson.timestamp.Timestamp`. .. versionchanged:: 1.2 Added support for encoding/decoding datetimes and regular expressions. """ import base64 import calendar import datetime import re json_lib = True try: import json except ImportError: try: import simplejson as json except ImportError: json_lib = False import bson from bson import EPOCH_AWARE, RE_TYPE from bson.binary import Binary from bson.code import Code from bson.dbref import DBRef from bson.max_key import MaxKey from bson.min_key import MinKey from bson.objectid import ObjectId from bson.timestamp import Timestamp from bson.py3compat import PY3, binary_type, string_types _RE_OPT_TABLE = { "i": re.I, "l": re.L, "m": re.M, "s": re.S, "u": re.U, "x": re.X, } def dumps(obj, *args, **kwargs): """Helper function that wraps :class:`json.dumps`. Recursive function that handles all BSON types including :class:`~bson.binary.Binary` and :class:`~bson.code.Code`. """ if not json_lib: raise Exception("No json library available") return json.dumps(_json_convert(obj), *args, **kwargs) def loads(s, *args, **kwargs): """Helper function that wraps :class:`json.loads`. Automatically passes the object_hook for BSON type conversion. """ if not json_lib: raise Exception("No json library available") kwargs['object_hook'] = object_hook return json.loads(s, *args, **kwargs) def _json_convert(obj): """Recursive helper method that converts BSON types so they can be converted into json. """ if hasattr(obj, 'iteritems') or hasattr(obj, 'items'): # PY3 support return dict(((k, _json_convert(v)) for k, v in obj.iteritems())) elif hasattr(obj, '__iter__') and not isinstance(obj, string_types): return list((_json_convert(v) for v in obj)) try: return default(obj) except TypeError: return obj def object_hook(dct): if "$oid" in dct: return ObjectId(str(dct["$oid"])) if "$ref" in dct: return DBRef(dct["$ref"], dct["$id"], dct.get("$db", None)) if "$date" in dct: secs = float(dct["$date"]) / 1000.0 return EPOCH_AWARE + datetime.timedelta(seconds=secs) if "$regex" in dct: flags = 0 # PyMongo always adds $options but some other tools may not. for opt in dct.get("$options", ""): flags |= _RE_OPT_TABLE.get(opt, 0) return re.compile(dct["$regex"], flags) if "$minKey" in dct: return MinKey() if "$maxKey" in dct: return MaxKey() if "$binary" in dct: if isinstance(dct["$type"], int): dct["$type"] = "%02x" % dct["$type"] subtype = int(dct["$type"], 16) if subtype >= 0xffffff80: # Handle mongoexport values subtype = int(dct["$type"][6:], 16) return Binary(base64.b64decode(dct["$binary"].encode()), subtype) if "$code" in dct: return Code(dct["$code"], dct.get("$scope")) if bson.has_uuid() and "$uuid" in dct: return bson.uuid.UUID(dct["$uuid"]) return dct def default(obj): if isinstance(obj, ObjectId): return {"$oid": str(obj)} if isinstance(obj, DBRef): return _json_convert(obj.as_doc()) if isinstance(obj, datetime.datetime): # TODO share this code w/ bson.py? if obj.utcoffset() is not None: obj = obj - obj.utcoffset() millis = int(calendar.timegm(obj.timetuple()) * 1000 + obj.microsecond / 1000) return {"$date": millis} if isinstance(obj, RE_TYPE): flags = "" if obj.flags & re.IGNORECASE: flags += "i" if obj.flags & re.LOCALE: flags += "l" if obj.flags & re.MULTILINE: flags += "m" if obj.flags & re.DOTALL: flags += "s" if obj.flags & re.UNICODE: flags += "u" if obj.flags & re.VERBOSE: flags += "x" return {"$regex": obj.pattern, "$options": flags} if isinstance(obj, MinKey): return {"$minKey": 1} if isinstance(obj, MaxKey): return {"$maxKey": 1} if isinstance(obj, Timestamp): return {"t": obj.time, "i": obj.inc} if isinstance(obj, Code): return {'$code': "%s" % obj, '$scope': obj.scope} if isinstance(obj, Binary): return {'$binary': base64.b64encode(obj).decode(), '$type': "%02x" % obj.subtype} if PY3 and isinstance(obj, binary_type): return {'$binary': base64.b64encode(obj).decode(), '$type': "00"} if bson.has_uuid() and isinstance(obj, bson.uuid.UUID): return {"$uuid": obj.hex} raise TypeError("%r is not JSON serializable" % obj) pymongo-2.6.3/bson/max_key.py000066400000000000000000000016561223300253600161660ustar00rootroot00000000000000# Copyright 2010-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Representation for the MongoDB internal MaxKey type. """ class MaxKey(object): """MongoDB internal MaxKey type. """ def __eq__(self, other): if isinstance(other, MaxKey): return True return NotImplemented def __ne__(self, other): return not self == other def __repr__(self): return "MaxKey()" pymongo-2.6.3/bson/min_key.py000066400000000000000000000016561223300253600161640ustar00rootroot00000000000000# Copyright 2010-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Representation for the MongoDB internal MinKey type. """ class MinKey(object): """MongoDB internal MinKey type. """ def __eq__(self, other): if isinstance(other, MinKey): return True return NotImplemented def __ne__(self, other): return not self == other def __repr__(self): return "MinKey()" pymongo-2.6.3/bson/objectid.py000066400000000000000000000215221223300253600163060ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tools for working with MongoDB `ObjectIds `_. """ import binascii import calendar import datetime try: import hashlib _md5func = hashlib.md5 except ImportError: # for Python < 2.5 import md5 _md5func = md5.new import os import random import socket import struct import threading import time from bson.errors import InvalidId from bson.py3compat import (PY3, b, binary_type, text_type, bytes_from_hex, string_types) from bson.tz_util import utc EMPTY = b("") ZERO = b("\x00") def _machine_bytes(): """Get the machine portion of an ObjectId. """ machine_hash = _md5func() if PY3: # gethostname() returns a unicode string in python 3.x # while update() requires a byte string. machine_hash.update(socket.gethostname().encode()) else: # Calling encode() here will fail with non-ascii hostnames machine_hash.update(socket.gethostname()) return machine_hash.digest()[0:3] class ObjectId(object): """A MongoDB ObjectId. """ _inc = random.randint(0, 0xFFFFFF) _inc_lock = threading.Lock() _machine_bytes = _machine_bytes() __slots__ = ('__id') def __init__(self, oid=None): """Initialize a new ObjectId. If `oid` is ``None``, create a new (unique) ObjectId. If `oid` is an instance of (:class:`basestring` (:class:`str` or :class:`bytes` in python 3), :class:`ObjectId`) validate it and use that. Otherwise, a :class:`TypeError` is raised. If `oid` is invalid, :class:`~bson.errors.InvalidId` is raised. :Parameters: - `oid` (optional): a valid ObjectId (12 byte binary or 24 character hex string) .. versionadded:: 1.2.1 The `oid` parameter can be a ``unicode`` instance (that contains only hexadecimal digits). .. mongodoc:: objectids """ if oid is None: self.__generate() else: self.__validate(oid) @classmethod def from_datetime(cls, generation_time): """Create a dummy ObjectId instance with a specific generation time. This method is useful for doing range queries on a field containing :class:`ObjectId` instances. .. warning:: It is not safe to insert a document containing an ObjectId generated using this method. This method deliberately eliminates the uniqueness guarantee that ObjectIds generally provide. ObjectIds generated with this method should be used exclusively in queries. `generation_time` will be converted to UTC. Naive datetime instances will be treated as though they already contain UTC. An example using this helper to get documents where ``"_id"`` was generated before January 1, 2010 would be: >>> gen_time = datetime.datetime(2010, 1, 1) >>> dummy_id = ObjectId.from_datetime(gen_time) >>> result = collection.find({"_id": {"$lt": dummy_id}}) :Parameters: - `generation_time`: :class:`~datetime.datetime` to be used as the generation time for the resulting ObjectId. .. versionchanged:: 1.8 Properly handle timezone aware values for `generation_time`. .. versionadded:: 1.6 """ if generation_time.utcoffset() is not None: generation_time = generation_time - generation_time.utcoffset() ts = calendar.timegm(generation_time.timetuple()) oid = struct.pack(">i", int(ts)) + ZERO * 8 return cls(oid) @classmethod def is_valid(cls, oid): """Checks if a `oid` string is valid or not. :Parameters: - `oid`: the object id to validate .. versionadded:: 2.3 """ try: ObjectId(oid) return True except (InvalidId, TypeError): return False def __generate(self): """Generate a new value for this ObjectId. """ oid = EMPTY # 4 bytes current time oid += struct.pack(">i", int(time.time())) # 3 bytes machine oid += ObjectId._machine_bytes # 2 bytes pid oid += struct.pack(">H", os.getpid() % 0xFFFF) # 3 bytes inc ObjectId._inc_lock.acquire() oid += struct.pack(">i", ObjectId._inc)[1:4] ObjectId._inc = (ObjectId._inc + 1) % 0xFFFFFF ObjectId._inc_lock.release() self.__id = oid def __validate(self, oid): """Validate and use the given id for this ObjectId. Raises TypeError if id is not an instance of (:class:`basestring` (:class:`str` or :class:`bytes` in python 3), ObjectId) and InvalidId if it is not a valid ObjectId. :Parameters: - `oid`: a valid ObjectId """ if isinstance(oid, ObjectId): self.__id = oid.__id elif isinstance(oid, string_types): if len(oid) == 12: if isinstance(oid, binary_type): self.__id = oid else: raise InvalidId("%s is not a valid ObjectId" % oid) elif len(oid) == 24: try: self.__id = bytes_from_hex(oid) except (TypeError, ValueError): raise InvalidId("%s is not a valid ObjectId" % oid) else: raise InvalidId("%s is not a valid ObjectId" % oid) else: raise TypeError("id must be an instance of (%s, %s, ObjectId), " "not %s" % (binary_type.__name__, text_type.__name__, type(oid))) @property def binary(self): """12-byte binary representation of this ObjectId. """ return self.__id @property def generation_time(self): """A :class:`datetime.datetime` instance representing the time of generation for this :class:`ObjectId`. The :class:`datetime.datetime` is timezone aware, and represents the generation time in UTC. It is precise to the second. .. versionchanged:: 1.8 Now return an aware datetime instead of a naive one. .. versionadded:: 1.2 """ t = struct.unpack(">i", self.__id[0:4])[0] return datetime.datetime.fromtimestamp(t, utc) def __getstate__(self): """return value of object for pickling. needed explicitly because __slots__() defined. """ return self.__id def __setstate__(self, value): """explicit state set from pickling """ # Provide backwards compatability with OIDs # pickled with pymongo-1.9 or older. if isinstance(value, dict): oid = value["_ObjectId__id"] else: oid = value # ObjectIds pickled in python 2.x used `str` for __id. # In python 3.x this has to be converted to `bytes` # by encoding latin-1. if PY3 and isinstance(oid, text_type): self.__id = oid.encode('latin-1') else: self.__id = oid def __str__(self): if PY3: return binascii.hexlify(self.__id).decode() return binascii.hexlify(self.__id) def __repr__(self): return "ObjectId('%s')" % (str(self),) def __eq__(self, other): if isinstance(other, ObjectId): return self.__id == other.__id return NotImplemented def __ne__(self, other): if isinstance(other, ObjectId): return self.__id != other.__id return NotImplemented def __lt__(self, other): if isinstance(other, ObjectId): return self.__id < other.__id return NotImplemented def __le__(self, other): if isinstance(other, ObjectId): return self.__id <= other.__id return NotImplemented def __gt__(self, other): if isinstance(other, ObjectId): return self.__id > other.__id return NotImplemented def __ge__(self, other): if isinstance(other, ObjectId): return self.__id >= other.__id return NotImplemented def __hash__(self): """Get a hash value for this :class:`ObjectId`. .. versionadded:: 1.1 """ return hash(self.__id) pymongo-2.6.3/bson/py3compat.py000066400000000000000000000034431223300253600164440ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you # may not use this file except in compliance with the License. You # may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. See the License for the specific language governing # permissions and limitations under the License. """Utility functions and definitions for python3 compatibility.""" import sys PY3 = sys.version_info[0] == 3 if PY3: import codecs from io import BytesIO as StringIO def b(s): # BSON and socket operations deal in binary data. In # python 3 that means instances of `bytes`. In python # 2.6 and 2.7 you can create an alias for `bytes` using # the b prefix (e.g. b'foo'). Python 2.4 and 2.5 don't # provide this marker so we provide this compat function. # In python 3.x b('foo') results in b'foo'. # See http://python3porting.com/problems.html#nicer-solutions return codecs.latin_1_encode(s)[0] def bytes_from_hex(h): return bytes.fromhex(h) binary_type = bytes text_type = str else: try: from cStringIO import StringIO except ImportError: from StringIO import StringIO def b(s): # See comments above. In python 2.x b('foo') is just 'foo'. return s def bytes_from_hex(h): return h.decode('hex') binary_type = str # 2to3 will convert this to "str". That's okay # since we won't ever get here under python3. text_type = unicode string_types = (binary_type, text_type) pymongo-2.6.3/bson/son.py000066400000000000000000000201711223300253600153210ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tools for creating and manipulating SON, the Serialized Ocument Notation. Regular dictionaries can be used instead of SON objects, but not when the order of keys is important. A SON object can be used just like a normal Python dictionary.""" import copy import re # This sort of sucks, but seems to be as good as it gets... # This is essentially the same as re._pattern_type RE_TYPE = type(re.compile("")) class SON(dict): """SON data. A subclass of dict that maintains ordering of keys and provides a few extra niceties for dealing with SON. SON objects can be converted to and from BSON. The mapping from Python types to BSON types is as follows: =================================== ============= =================== Python Type BSON Type Supported Direction =================================== ============= =================== None null both bool boolean both int [#int]_ int32 / int64 py -> bson long int64 both float number (real) both string string py -> bson unicode string both list array both dict / `SON` object both datetime.datetime [#dt]_ [#dt2]_ date both compiled re regex both `bson.binary.Binary` binary both `bson.objectid.ObjectId` oid both `bson.dbref.DBRef` dbref both None undefined bson -> py unicode code bson -> py `bson.code.Code` code py -> bson unicode symbol bson -> py bytes (Python 3) [#bytes]_ binary both =================================== ============= =================== Note that to save binary data it must be wrapped as an instance of `bson.binary.Binary`. Otherwise it will be saved as a BSON string and retrieved as unicode. .. [#int] A Python int will be saved as a BSON int32 or BSON int64 depending on its size. A BSON int32 will always decode to a Python int. In Python 2.x a BSON int64 will always decode to a Python long. In Python 3.x a BSON int64 will decode to a Python int since there is no longer a long type. .. [#dt] datetime.datetime instances will be rounded to the nearest millisecond when saved .. [#dt2] all datetime.datetime instances are treated as *naive*. clients should always use UTC. .. [#bytes] The bytes type from Python 3.x is encoded as BSON binary with subtype 0. In Python 3.x it will be decoded back to bytes. In Python 2.x it will be decoded to an instance of :class:`~bson.binary.Binary` with subtype 0. """ def __init__(self, data=None, **kwargs): self.__keys = [] dict.__init__(self) self.update(data) self.update(kwargs) def __new__(cls, *args, **kwargs): instance = super(SON, cls).__new__(cls, *args, **kwargs) instance.__keys = [] return instance def __repr__(self): result = [] for key in self.__keys: result.append("(%r, %r)" % (key, self[key])) return "SON([%s])" % ", ".join(result) def __setitem__(self, key, value): if key not in self: self.__keys.append(key) dict.__setitem__(self, key, value) def __delitem__(self, key): self.__keys.remove(key) dict.__delitem__(self, key) def keys(self): return list(self.__keys) def copy(self): other = SON() other.update(self) return other # TODO this is all from UserDict.DictMixin. it could probably be made more # efficient. # second level definitions support higher levels def __iter__(self): for k in self.keys(): yield k def has_key(self, key): return key in self.keys() def __contains__(self, key): return key in self.keys() # third level takes advantage of second level definitions def iteritems(self): for k in self: yield (k, self[k]) def iterkeys(self): return self.__iter__() # fourth level uses definitions from lower levels def itervalues(self): for _, v in self.iteritems(): yield v def values(self): return [v for _, v in self.iteritems()] def items(self): return [(key, self[key]) for key in self] def clear(self): for key in self.keys(): del self[key] def setdefault(self, key, default=None): try: return self[key] except KeyError: self[key] = default return default def pop(self, key, *args): if len(args) > 1: raise TypeError("pop expected at most 2 arguments, got "\ + repr(1 + len(args))) try: value = self[key] except KeyError: if args: return args[0] raise del self[key] return value def popitem(self): try: k, v = self.iteritems().next() except StopIteration: raise KeyError('container is empty') del self[k] return (k, v) def update(self, other=None, **kwargs): # Make progressively weaker assumptions about "other" if other is None: pass elif hasattr(other, 'iteritems'): # iteritems saves memory and lookups for k, v in other.iteritems(): self[k] = v elif hasattr(other, 'keys'): for k in other.keys(): self[k] = other[k] else: for k, v in other: self[k] = v if kwargs: self.update(kwargs) def get(self, key, default=None): try: return self[key] except KeyError: return default def __eq__(self, other): """Comparison to another SON is order-sensitive while comparison to a regular dictionary is order-insensitive. """ if isinstance(other, SON): return len(self) == len(other) and self.items() == other.items() return self.to_dict() == other def __ne__(self, other): return not self == other def __len__(self): return len(self.keys()) def to_dict(self): """Convert a SON document to a normal Python dictionary instance. This is trickier than just *dict(...)* because it needs to be recursive. """ def transform_value(value): if isinstance(value, list): return [transform_value(v) for v in value] if isinstance(value, SON): value = dict(value) if isinstance(value, dict): for k, v in value.iteritems(): value[k] = transform_value(v) return value return transform_value(dict(self)) def __deepcopy__(self, memo): out = SON() val_id = id(self) if val_id in memo: return memo.get(val_id) memo[val_id] = out for k, v in self.iteritems(): if not isinstance(v, RE_TYPE): v = copy.deepcopy(v, memo) out[k] = v return out pymongo-2.6.3/bson/time64.c000066400000000000000000000522061223300253600154300ustar00rootroot00000000000000/* Copyright (c) 2007-2010 Michael G Schwern This software originally derived from Paul Sheer's pivotal_gmtime_r.c. The MIT License: Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ /* Programmers who have available to them 64-bit time values as a 'long long' type can use localtime64_r() and gmtime64_r() which correctly converts the time even on 32-bit systems. Whether you have 64-bit time values will depend on the operating system. localtime64_r() is a 64-bit equivalent of localtime_r(). gmtime64_r() is a 64-bit equivalent of gmtime_r(). */ #ifdef _MSC_VER #define _CRT_SECURE_NO_WARNINGS #endif #include #include #include #include #include #include #include "time64.h" #include "time64_limits.h" /* Spec says except for stftime() and the _r() functions, these all return static memory. Stabbings! */ static struct TM Static_Return_Date; static const int days_in_month[2][12] = { {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}, {31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}, }; static const int julian_days_by_month[2][12] = { {0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334}, {0, 31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335}, }; static const int length_of_year[2] = { 365, 366 }; /* Some numbers relating to the gregorian cycle */ static const Year years_in_gregorian_cycle = 400; #define days_in_gregorian_cycle ((365 * 400) + 100 - 4 + 1) static const Time64_T seconds_in_gregorian_cycle = days_in_gregorian_cycle * 60LL * 60LL * 24LL; /* Year range we can trust the time funcitons with */ #define MAX_SAFE_YEAR 2037 #define MIN_SAFE_YEAR 1971 /* 28 year Julian calendar cycle */ #define SOLAR_CYCLE_LENGTH 28 /* Year cycle from MAX_SAFE_YEAR down. */ static const int safe_years_high[SOLAR_CYCLE_LENGTH] = { 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2010, 2011, 2012, 2013, 2014, 2015 }; /* Year cycle from MIN_SAFE_YEAR up */ static const int safe_years_low[SOLAR_CYCLE_LENGTH] = { 1996, 1997, 1998, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, }; /* This isn't used, but it's handy to look at */ static const int dow_year_start[SOLAR_CYCLE_LENGTH] = { 5, 0, 1, 2, /* 0 2016 - 2019 */ 3, 5, 6, 0, /* 4 */ 1, 3, 4, 5, /* 8 1996 - 1998, 1971*/ 6, 1, 2, 3, /* 12 1972 - 1975 */ 4, 6, 0, 1, /* 16 */ 2, 4, 5, 6, /* 20 2036, 2037, 2010, 2011 */ 0, 2, 3, 4 /* 24 2012, 2013, 2014, 2015 */ }; /* Let's assume people are going to be looking for dates in the future. Let's provide some cheats so you can skip ahead. This has a 4x speed boost when near 2008. */ /* Number of days since epoch on Jan 1st, 2008 GMT */ #define CHEAT_DAYS (1199145600 / 24 / 60 / 60) #define CHEAT_YEARS 108 #define IS_LEAP(n) ((!(((n) + 1900) % 400) || (!(((n) + 1900) % 4) && (((n) + 1900) % 100))) != 0) #define WRAP(a,b,m) ((a) = ((a) < 0 ) ? ((b)--, (a) + (m)) : (a)) #ifdef USE_SYSTEM_LOCALTIME # define SHOULD_USE_SYSTEM_LOCALTIME(a) ( \ (a) <= SYSTEM_LOCALTIME_MAX && \ (a) >= SYSTEM_LOCALTIME_MIN \ ) #else # define SHOULD_USE_SYSTEM_LOCALTIME(a) (0) #endif #ifdef USE_SYSTEM_GMTIME # define SHOULD_USE_SYSTEM_GMTIME(a) ( \ (a) <= SYSTEM_GMTIME_MAX && \ (a) >= SYSTEM_GMTIME_MIN \ ) #else # define SHOULD_USE_SYSTEM_GMTIME(a) (0) #endif /* Multi varadic macros are a C99 thing, alas */ #ifdef TIME_64_DEBUG # define TIME64_TRACE(format) (fprintf(stderr, format)) # define TIME64_TRACE1(format, var1) (fprintf(stderr, format, var1)) # define TIME64_TRACE2(format, var1, var2) (fprintf(stderr, format, var1, var2)) # define TIME64_TRACE3(format, var1, var2, var3) (fprintf(stderr, format, var1, var2, var3)) #else # define TIME64_TRACE(format) ((void)0) # define TIME64_TRACE1(format, var1) ((void)0) # define TIME64_TRACE2(format, var1, var2) ((void)0) # define TIME64_TRACE3(format, var1, var2, var3) ((void)0) #endif static int is_exception_century(Year year) { int is_exception = ((year % 100 == 0) && !(year % 400 == 0)); TIME64_TRACE1("# is_exception_century: %s\n", is_exception ? "yes" : "no"); return(is_exception); } /* Compare two dates. The result is like cmp. Ignores things like gmtoffset and dst */ int cmp_date( const struct TM* left, const struct tm* right ) { if( left->tm_year > right->tm_year ) return 1; else if( left->tm_year < right->tm_year ) return -1; if( left->tm_mon > right->tm_mon ) return 1; else if( left->tm_mon < right->tm_mon ) return -1; if( left->tm_mday > right->tm_mday ) return 1; else if( left->tm_mday < right->tm_mday ) return -1; if( left->tm_hour > right->tm_hour ) return 1; else if( left->tm_hour < right->tm_hour ) return -1; if( left->tm_min > right->tm_min ) return 1; else if( left->tm_min < right->tm_min ) return -1; if( left->tm_sec > right->tm_sec ) return 1; else if( left->tm_sec < right->tm_sec ) return -1; return 0; } /* Check if a date is safely inside a range. The intention is to check if its a few days inside. */ int date_in_safe_range( const struct TM* date, const struct tm* min, const struct tm* max ) { if( cmp_date(date, min) == -1 ) return 0; if( cmp_date(date, max) == 1 ) return 0; return 1; } /* timegm() is not in the C or POSIX spec, but it is such a useful extension I would be remiss in leaving it out. Also I need it for localtime64() */ Time64_T timegm64(const struct TM *date) { Time64_T days = 0; Time64_T seconds = 0; Year year; Year orig_year = (Year)date->tm_year; int cycles = 0; if( orig_year > 100 ) { cycles = (int)((orig_year - 100) / 400); orig_year -= cycles * 400; days += (Time64_T)cycles * days_in_gregorian_cycle; } else if( orig_year < -300 ) { cycles = (int)((orig_year - 100) / 400); orig_year -= cycles * 400; days += (Time64_T)cycles * days_in_gregorian_cycle; } TIME64_TRACE3("# timegm/ cycles: %d, days: %lld, orig_year: %lld\n", cycles, days, orig_year); if( orig_year > 70 ) { year = 70; while( year < orig_year ) { days += length_of_year[IS_LEAP(year)]; year++; } } else if ( orig_year < 70 ) { year = 69; do { days -= length_of_year[IS_LEAP(year)]; year--; } while( year >= orig_year ); } days += julian_days_by_month[IS_LEAP(orig_year)][date->tm_mon]; days += date->tm_mday - 1; seconds = days * 60 * 60 * 24; seconds += date->tm_hour * 60 * 60; seconds += date->tm_min * 60; seconds += date->tm_sec; return(seconds); } #ifndef NDEBUG static int check_tm(struct TM *tm) { /* Don't forget leap seconds */ assert(tm->tm_sec >= 0); assert(tm->tm_sec <= 61); assert(tm->tm_min >= 0); assert(tm->tm_min <= 59); assert(tm->tm_hour >= 0); assert(tm->tm_hour <= 23); assert(tm->tm_mday >= 1); assert(tm->tm_mday <= days_in_month[IS_LEAP(tm->tm_year)][tm->tm_mon]); assert(tm->tm_mon >= 0); assert(tm->tm_mon <= 11); assert(tm->tm_wday >= 0); assert(tm->tm_wday <= 6); assert(tm->tm_yday >= 0); assert(tm->tm_yday <= length_of_year[IS_LEAP(tm->tm_year)]); #ifdef HAS_TM_TM_GMTOFF assert(tm->tm_gmtoff >= -24 * 60 * 60); assert(tm->tm_gmtoff <= 24 * 60 * 60); #endif return 1; } #endif /* The exceptional centuries without leap years cause the cycle to shift by 16 */ static Year cycle_offset(Year year) { const Year start_year = 2000; Year year_diff = year - start_year; Year exceptions; if( year > start_year ) year_diff--; exceptions = year_diff / 100; exceptions -= year_diff / 400; TIME64_TRACE3("# year: %lld, exceptions: %lld, year_diff: %lld\n", year, exceptions, year_diff); return exceptions * 16; } /* For a given year after 2038, pick the latest possible matching year in the 28 year calendar cycle. A matching year... 1) Starts on the same day of the week. 2) Has the same leap year status. This is so the calendars match up. Also the previous year must match. When doing Jan 1st you might wind up on Dec 31st the previous year when doing a -UTC time zone. Finally, the next year must have the same start day of week. This is for Dec 31st with a +UTC time zone. It doesn't need the same leap year status since we only care about January 1st. */ static int safe_year(const Year year) { int safe_year = 0; Year year_cycle; if( year >= MIN_SAFE_YEAR && year <= MAX_SAFE_YEAR ) { return (int)year; } year_cycle = year + cycle_offset(year); /* safe_years_low is off from safe_years_high by 8 years */ if( year < MIN_SAFE_YEAR ) year_cycle -= 8; /* Change non-leap xx00 years to an equivalent */ if( is_exception_century(year) ) year_cycle += 11; /* Also xx01 years, since the previous year will be wrong */ if( is_exception_century(year - 1) ) year_cycle += 17; year_cycle %= SOLAR_CYCLE_LENGTH; if( year_cycle < 0 ) year_cycle = SOLAR_CYCLE_LENGTH + year_cycle; assert( year_cycle >= 0 ); assert( year_cycle < SOLAR_CYCLE_LENGTH ); if( year < MIN_SAFE_YEAR ) safe_year = safe_years_low[year_cycle]; else if( year > MAX_SAFE_YEAR ) safe_year = safe_years_high[year_cycle]; else assert(0); TIME64_TRACE3("# year: %lld, year_cycle: %lld, safe_year: %d\n", year, year_cycle, safe_year); assert(safe_year <= MAX_SAFE_YEAR && safe_year >= MIN_SAFE_YEAR); return safe_year; } void copy_tm_to_TM64(const struct tm *src, struct TM *dest) { if( src == NULL ) { memset(dest, 0, sizeof(*dest)); } else { # ifdef USE_TM64 dest->tm_sec = src->tm_sec; dest->tm_min = src->tm_min; dest->tm_hour = src->tm_hour; dest->tm_mday = src->tm_mday; dest->tm_mon = src->tm_mon; dest->tm_year = (Year)src->tm_year; dest->tm_wday = src->tm_wday; dest->tm_yday = src->tm_yday; dest->tm_isdst = src->tm_isdst; # ifdef HAS_TM_TM_GMTOFF dest->tm_gmtoff = src->tm_gmtoff; # endif # ifdef HAS_TM_TM_ZONE dest->tm_zone = src->tm_zone; # endif # else /* They're the same type */ memcpy(dest, src, sizeof(*dest)); # endif } } void copy_TM64_to_tm(const struct TM *src, struct tm *dest) { if( src == NULL ) { memset(dest, 0, sizeof(*dest)); } else { # ifdef USE_TM64 dest->tm_sec = src->tm_sec; dest->tm_min = src->tm_min; dest->tm_hour = src->tm_hour; dest->tm_mday = src->tm_mday; dest->tm_mon = src->tm_mon; dest->tm_year = (int)src->tm_year; dest->tm_wday = src->tm_wday; dest->tm_yday = src->tm_yday; dest->tm_isdst = src->tm_isdst; # ifdef HAS_TM_TM_GMTOFF dest->tm_gmtoff = src->tm_gmtoff; # endif # ifdef HAS_TM_TM_ZONE dest->tm_zone = src->tm_zone; # endif # else /* They're the same type */ memcpy(dest, src, sizeof(*dest)); # endif } } /* Simulate localtime_r() to the best of our ability */ struct tm * fake_localtime_r(const time_t *time, struct tm *result) { const struct tm *static_result = localtime(time); assert(result != NULL); if( static_result == NULL ) { memset(result, 0, sizeof(*result)); return NULL; } else { memcpy(result, static_result, sizeof(*result)); return result; } } /* Simulate gmtime_r() to the best of our ability */ struct tm * fake_gmtime_r(const time_t *time, struct tm *result) { const struct tm *static_result = gmtime(time); assert(result != NULL); if( static_result == NULL ) { memset(result, 0, sizeof(*result)); return NULL; } else { memcpy(result, static_result, sizeof(*result)); return result; } } static Time64_T seconds_between_years(Year left_year, Year right_year) { int increment = (left_year > right_year) ? 1 : -1; Time64_T seconds = 0; int cycles; if( left_year > 2400 ) { cycles = (int)((left_year - 2400) / 400); left_year -= cycles * 400; seconds += cycles * seconds_in_gregorian_cycle; } else if( left_year < 1600 ) { cycles = (int)((left_year - 1600) / 400); left_year += cycles * 400; seconds += cycles * seconds_in_gregorian_cycle; } while( left_year != right_year ) { seconds += length_of_year[IS_LEAP(right_year - 1900)] * 60 * 60 * 24; right_year += increment; } return seconds * increment; } Time64_T mktime64(const struct TM *input_date) { struct tm safe_date; struct TM date; Time64_T time; Year year = input_date->tm_year + 1900; if( date_in_safe_range(input_date, &SYSTEM_MKTIME_MIN, &SYSTEM_MKTIME_MAX) ) { copy_TM64_to_tm(input_date, &safe_date); return (Time64_T)mktime(&safe_date); } /* Have to make the year safe in date else it won't fit in safe_date */ date = *input_date; date.tm_year = safe_year(year) - 1900; copy_TM64_to_tm(&date, &safe_date); time = (Time64_T)mktime(&safe_date); time += seconds_between_years(year, (Year)(safe_date.tm_year + 1900)); return time; } /* Because I think mktime() is a crappy name */ Time64_T timelocal64(const struct TM *date) { return mktime64(date); } struct TM *gmtime64_r (const Time64_T *in_time, struct TM *p) { int v_tm_sec, v_tm_min, v_tm_hour, v_tm_mon, v_tm_wday; Time64_T v_tm_tday; int leap; Time64_T m; Time64_T time = *in_time; Year year = 70; int cycles = 0; assert(p != NULL); /* Use the system gmtime() if time_t is small enough */ if( SHOULD_USE_SYSTEM_GMTIME(*in_time) ) { time_t safe_time = (time_t)*in_time; struct tm safe_date; GMTIME_R(&safe_time, &safe_date); copy_tm_to_TM64(&safe_date, p); assert(check_tm(p)); return p; } #ifdef HAS_TM_TM_GMTOFF p->tm_gmtoff = 0; #endif p->tm_isdst = 0; #ifdef HAS_TM_TM_ZONE p->tm_zone = "UTC"; #endif v_tm_sec = (int)(time % 60); time /= 60; v_tm_min = (int)(time % 60); time /= 60; v_tm_hour = (int)(time % 24); time /= 24; v_tm_tday = time; WRAP (v_tm_sec, v_tm_min, 60); WRAP (v_tm_min, v_tm_hour, 60); WRAP (v_tm_hour, v_tm_tday, 24); v_tm_wday = (int)((v_tm_tday + 4) % 7); if (v_tm_wday < 0) v_tm_wday += 7; m = v_tm_tday; if (m >= CHEAT_DAYS) { year = CHEAT_YEARS; m -= CHEAT_DAYS; } if (m >= 0) { /* Gregorian cycles, this is huge optimization for distant times */ cycles = (int)(m / (Time64_T) days_in_gregorian_cycle); if( cycles ) { m -= (cycles * (Time64_T) days_in_gregorian_cycle); year += (cycles * years_in_gregorian_cycle); } /* Years */ leap = IS_LEAP (year); while (m >= (Time64_T) length_of_year[leap]) { m -= (Time64_T) length_of_year[leap]; year++; leap = IS_LEAP (year); } /* Months */ v_tm_mon = 0; while (m >= (Time64_T) days_in_month[leap][v_tm_mon]) { m -= (Time64_T) days_in_month[leap][v_tm_mon]; v_tm_mon++; } } else { year--; /* Gregorian cycles */ cycles = (int)((m / (Time64_T) days_in_gregorian_cycle) + 1); if( cycles ) { m -= (cycles * (Time64_T) days_in_gregorian_cycle); year += (cycles * years_in_gregorian_cycle); } /* Years */ leap = IS_LEAP (year); while (m < (Time64_T) -length_of_year[leap]) { m += (Time64_T) length_of_year[leap]; year--; leap = IS_LEAP (year); } /* Months */ v_tm_mon = 11; while (m < (Time64_T) -days_in_month[leap][v_tm_mon]) { m += (Time64_T) days_in_month[leap][v_tm_mon]; v_tm_mon--; } m += (Time64_T) days_in_month[leap][v_tm_mon]; } p->tm_year = (int)year; if( p->tm_year != year ) { #ifdef EOVERFLOW errno = EOVERFLOW; #endif return NULL; } /* At this point m is less than a year so casting to an int is safe */ p->tm_mday = (int) m + 1; p->tm_yday = julian_days_by_month[leap][v_tm_mon] + (int)m; p->tm_sec = v_tm_sec; p->tm_min = v_tm_min; p->tm_hour = v_tm_hour; p->tm_mon = v_tm_mon; p->tm_wday = v_tm_wday; assert(check_tm(p)); return p; } struct TM *localtime64_r (const Time64_T *time, struct TM *local_tm) { time_t safe_time; struct tm safe_date; struct TM gm_tm; Year orig_year; int month_diff; assert(local_tm != NULL); /* Use the system localtime() if time_t is small enough */ if( SHOULD_USE_SYSTEM_LOCALTIME(*time) ) { safe_time = (time_t)*time; TIME64_TRACE1("Using system localtime for %lld\n", *time); LOCALTIME_R(&safe_time, &safe_date); copy_tm_to_TM64(&safe_date, local_tm); assert(check_tm(local_tm)); return local_tm; } if( gmtime64_r(time, &gm_tm) == NULL ) { TIME64_TRACE1("gmtime64_r returned null for %lld\n", *time); return NULL; } orig_year = gm_tm.tm_year; if (gm_tm.tm_year > (2037 - 1900) || gm_tm.tm_year < (1970 - 1900) ) { TIME64_TRACE1("Mapping tm_year %lld to safe_year\n", (Year)gm_tm.tm_year); gm_tm.tm_year = safe_year((Year)(gm_tm.tm_year + 1900)) - 1900; } safe_time = (time_t)timegm64(&gm_tm); if( LOCALTIME_R(&safe_time, &safe_date) == NULL ) { TIME64_TRACE1("localtime_r(%d) returned NULL\n", (int)safe_time); return NULL; } copy_tm_to_TM64(&safe_date, local_tm); local_tm->tm_year = (int)orig_year; if( local_tm->tm_year != orig_year ) { TIME64_TRACE2("tm_year overflow: tm_year %lld, orig_year %lld\n", (Year)local_tm->tm_year, (Year)orig_year); #ifdef EOVERFLOW errno = EOVERFLOW; #endif return NULL; } month_diff = local_tm->tm_mon - gm_tm.tm_mon; /* When localtime is Dec 31st previous year and gmtime is Jan 1st next year. */ if( month_diff == 11 ) { local_tm->tm_year--; } /* When localtime is Jan 1st, next year and gmtime is Dec 31st, previous year. */ if( month_diff == -11 ) { local_tm->tm_year++; } /* GMT is Jan 1st, xx01 year, but localtime is still Dec 31st in a non-leap xx00. There is one point in the cycle we can't account for which the safe xx00 year is a leap year. So we need to correct for Dec 31st comming out as the 366th day of the year. */ if( !IS_LEAP(local_tm->tm_year) && local_tm->tm_yday == 365 ) local_tm->tm_yday--; assert(check_tm(local_tm)); return local_tm; } int valid_tm_wday( const struct TM* date ) { if( 0 <= date->tm_wday && date->tm_wday <= 6 ) return 1; else return 0; } int valid_tm_mon( const struct TM* date ) { if( 0 <= date->tm_mon && date->tm_mon <= 11 ) return 1; else return 0; } /* Non-thread safe versions of the above */ struct TM *localtime64(const Time64_T *time) { #ifdef _MSC_VER _tzset(); #else tzset(); #endif return localtime64_r(time, &Static_Return_Date); } struct TM *gmtime64(const Time64_T *time) { return gmtime64_r(time, &Static_Return_Date); } pymongo-2.6.3/bson/time64.h000066400000000000000000000027541223300253600154400ustar00rootroot00000000000000#ifndef TIME64_H # define TIME64_H #include #include "time64_config.h" /* Set our custom types */ typedef INT_64_T Int64; typedef Int64 Time64_T; typedef Int64 Year; /* A copy of the tm struct but with a 64 bit year */ struct TM64 { int tm_sec; int tm_min; int tm_hour; int tm_mday; int tm_mon; Year tm_year; int tm_wday; int tm_yday; int tm_isdst; #ifdef HAS_TM_TM_GMTOFF long tm_gmtoff; #endif #ifdef HAS_TM_TM_ZONE char *tm_zone; #endif }; /* Decide which tm struct to use */ #ifdef USE_TM64 #define TM TM64 #else #define TM tm #endif /* Declare public functions */ struct TM *gmtime64_r (const Time64_T *, struct TM *); struct TM *localtime64_r (const Time64_T *, struct TM *); struct TM *gmtime64 (const Time64_T *); struct TM *localtime64 (const Time64_T *); Time64_T timegm64 (const struct TM *); Time64_T mktime64 (const struct TM *); Time64_T timelocal64 (const struct TM *); /* Not everyone has gm/localtime_r(), provide a replacement */ #ifdef HAS_LOCALTIME_R # define LOCALTIME_R(clock, result) localtime_r(clock, result) #else # define LOCALTIME_R(clock, result) fake_localtime_r(clock, result) #endif #ifdef HAS_GMTIME_R # define GMTIME_R(clock, result) gmtime_r(clock, result) #else # define GMTIME_R(clock, result) fake_gmtime_r(clock, result) #endif #endif pymongo-2.6.3/bson/time64_config.h000066400000000000000000000032221223300253600167540ustar00rootroot00000000000000/* Configuration ------------- Define as appropriate for your system. Sensible defaults provided. */ #ifndef TIME64_CONFIG_H # define TIME64_CONFIG_H /* Debugging TIME_64_DEBUG Define if you want debugging messages */ /* #define TIME_64_DEBUG */ /* INT_64_T A 64 bit integer type to use to store time and others. Must be defined. */ #define INT_64_T long long /* USE_TM64 Should we use a 64 bit safe replacement for tm? This will let you go past year 2 billion but the struct will be incompatible with tm. Conversion functions will be provided. */ /* #define USE_TM64 */ /* Availability of system functions. HAS_GMTIME_R Define if your system has gmtime_r() HAS_LOCALTIME_R Define if your system has localtime_r() HAS_TIMEGM Define if your system has timegm(), a GNU extension. */ #if !defined(WIN32) && !defined(_MSC_VER) #define HAS_GMTIME_R #define HAS_LOCALTIME_R #endif /* #define HAS_TIMEGM */ /* Details of non-standard tm struct elements. HAS_TM_TM_GMTOFF True if your tm struct has a "tm_gmtoff" element. A BSD extension. HAS_TM_TM_ZONE True if your tm struct has a "tm_zone" element. A BSD extension. */ /* #define HAS_TM_TM_GMTOFF */ /* #define HAS_TM_TM_ZONE */ /* USE_SYSTEM_LOCALTIME USE_SYSTEM_GMTIME USE_SYSTEM_MKTIME USE_SYSTEM_TIMEGM Should we use the system functions if the time is inside their range? Your system localtime() is probably more accurate, but our gmtime() is fast and safe. */ #define USE_SYSTEM_LOCALTIME /* #define USE_SYSTEM_GMTIME */ #define USE_SYSTEM_MKTIME /* #define USE_SYSTEM_TIMEGM */ #endif /* TIME64_CONFIG_H */ pymongo-2.6.3/bson/time64_limits.h000066400000000000000000000027251223300253600170170ustar00rootroot00000000000000/* Maximum and minimum inputs your system's respective time functions can correctly handle. time64.h will use your system functions if the input falls inside these ranges and corresponding USE_SYSTEM_* constant is defined. */ #ifndef TIME64_LIMITS_H #define TIME64_LIMITS_H /* Max/min for localtime() */ #define SYSTEM_LOCALTIME_MAX 2147483647 #define SYSTEM_LOCALTIME_MIN -2147483647-1 /* Max/min for gmtime() */ #define SYSTEM_GMTIME_MAX 2147483647 #define SYSTEM_GMTIME_MIN -2147483647-1 /* Max/min for mktime() */ static const struct tm SYSTEM_MKTIME_MAX = { 7, 14, 19, 18, 0, 138, 1, 17, 0 #ifdef HAS_TM_TM_GMTOFF ,-28800 #endif #ifdef HAS_TM_TM_ZONE ,"PST" #endif }; static const struct tm SYSTEM_MKTIME_MIN = { 52, 45, 12, 13, 11, 1, 5, 346, 0 #ifdef HAS_TM_TM_GMTOFF ,-28800 #endif #ifdef HAS_TM_TM_ZONE ,"PST" #endif }; /* Max/min for timegm() */ #ifdef HAS_TIMEGM static const struct tm SYSTEM_TIMEGM_MAX = { 7, 14, 3, 19, 0, 138, 2, 18, 0 #ifdef HAS_TM_TM_GMTOFF ,0 #endif #ifdef HAS_TM_TM_ZONE ,"UTC" #endif }; static const struct tm SYSTEM_TIMEGM_MIN = { 52, 45, 20, 13, 11, 1, 5, 346, 0 #ifdef HAS_TM_TM_GMTOFF ,0 #endif #ifdef HAS_TM_TM_ZONE ,"UTC" #endif }; #endif /* HAS_TIMEGM */ #endif /* TIME64_LIMITS_H */ pymongo-2.6.3/bson/timestamp.py000066400000000000000000000062671223300253600165370ustar00rootroot00000000000000# Copyright 2010-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tools for representing MongoDB internal Timestamps. """ import calendar import datetime from bson.tz_util import utc UPPERBOUND = 4294967296 class Timestamp(object): """MongoDB internal timestamps used in the opLog. """ def __init__(self, time, inc): """Create a new :class:`Timestamp`. This class is only for use with the MongoDB opLog. If you need to store a regular timestamp, please use a :class:`~datetime.datetime`. Raises :class:`TypeError` if `time` is not an instance of :class: `int` or :class:`~datetime.datetime`, or `inc` is not an instance of :class:`int`. Raises :class:`ValueError` if `time` or `inc` is not in [0, 2**32). :Parameters: - `time`: time in seconds since epoch UTC, or a naive UTC :class:`~datetime.datetime`, or an aware :class:`~datetime.datetime` - `inc`: the incrementing counter .. versionchanged:: 1.7 `time` can now be a :class:`~datetime.datetime` instance. """ if isinstance(time, datetime.datetime): if time.utcoffset() is not None: time = time - time.utcoffset() time = int(calendar.timegm(time.timetuple())) if not isinstance(time, (int, long)): raise TypeError("time must be an instance of int") if not isinstance(inc, (int, long)): raise TypeError("inc must be an instance of int") if not 0 <= time < UPPERBOUND: raise ValueError("time must be contained in [0, 2**32)") if not 0 <= inc < UPPERBOUND: raise ValueError("inc must be contained in [0, 2**32)") self.__time = time self.__inc = inc @property def time(self): """Get the time portion of this :class:`Timestamp`. """ return self.__time @property def inc(self): """Get the inc portion of this :class:`Timestamp`. """ return self.__inc def __eq__(self, other): if isinstance(other, Timestamp): return (self.__time == other.time and self.__inc == other.inc) else: return NotImplemented def __ne__(self, other): return not self == other def __repr__(self): return "Timestamp(%s, %s)" % (self.__time, self.__inc) def as_datetime(self): """Return a :class:`~datetime.datetime` instance corresponding to the time portion of this :class:`Timestamp`. .. versionchanged:: 1.8 The returned datetime is now timezone aware. """ return datetime.datetime.fromtimestamp(self.__time, utc) pymongo-2.6.3/bson/tz_util.py000066400000000000000000000027541223300253600162230ustar00rootroot00000000000000# Copyright 2010-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Timezone related utilities for BSON.""" from datetime import (timedelta, tzinfo) ZERO = timedelta(0) class FixedOffset(tzinfo): """Fixed offset timezone, in minutes east from UTC. Implementation based from the Python `standard library documentation `_. Defining __getinitargs__ enables pickling / copying. """ def __init__(self, offset, name): if isinstance(offset, timedelta): self.__offset = offset else: self.__offset = timedelta(minutes=offset) self.__name = name def __getinitargs__(self): return self.__offset, self.__name def utcoffset(self, dt): return self.__offset def tzname(self, dt): return self.__name def dst(self, dt): return ZERO utc = FixedOffset(0, "UTC") """Fixed offset timezone representing UTC.""" pymongo-2.6.3/distribute_setup.py000066400000000000000000000415171223300253600171660ustar00rootroot00000000000000#!python """Bootstrap distribute installation If you want to use setuptools in your package's setup.py, just include this file in the same directory with it, and add this to the top of your setup.py:: from distribute_setup import use_setuptools use_setuptools() If you want to require a specific version of setuptools, set a download mirror, or use an alternate download directory, you can do so by supplying the appropriate options to ``use_setuptools()``. This file can also be run as a script to install or upgrade setuptools. """ import os import shutil import sys import time import fnmatch import tempfile import tarfile import optparse from distutils import log try: from site import USER_SITE except ImportError: USER_SITE = None try: import subprocess def _python_cmd(*args): args = (sys.executable,) + args return subprocess.call(args) == 0 except ImportError: # will be used for python 2.3 def _python_cmd(*args): args = (sys.executable,) + args # quoting arguments if windows if sys.platform == 'win32': def quote(arg): if ' ' in arg: return '"%s"' % arg return arg args = [quote(arg) for arg in args] return os.spawnl(os.P_WAIT, sys.executable, *args) == 0 DEFAULT_VERSION = "0.6.30" DEFAULT_URL = "http://pypi.python.org/packages/source/d/distribute/" SETUPTOOLS_FAKED_VERSION = "0.6c11" SETUPTOOLS_PKG_INFO = """\ Metadata-Version: 1.0 Name: setuptools Version: %s Summary: xxxx Home-page: xxx Author: xxx Author-email: xxx License: xxx Description: xxx """ % SETUPTOOLS_FAKED_VERSION def _install(tarball, install_args=()): # extracting the tarball tmpdir = tempfile.mkdtemp() log.warn('Extracting in %s', tmpdir) old_wd = os.getcwd() try: os.chdir(tmpdir) tar = tarfile.open(tarball) _extractall(tar) tar.close() # going in the directory subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0]) os.chdir(subdir) log.warn('Now working in %s', subdir) # installing log.warn('Installing Distribute') if not _python_cmd('setup.py', 'install', *install_args): log.warn('Something went wrong during the installation.') log.warn('See the error message above.') # exitcode will be 2 return 2 finally: os.chdir(old_wd) shutil.rmtree(tmpdir) def _build_egg(egg, tarball, to_dir): # extracting the tarball tmpdir = tempfile.mkdtemp() log.warn('Extracting in %s', tmpdir) old_wd = os.getcwd() try: os.chdir(tmpdir) tar = tarfile.open(tarball) _extractall(tar) tar.close() # going in the directory subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0]) os.chdir(subdir) log.warn('Now working in %s', subdir) # building an egg log.warn('Building a Distribute egg in %s', to_dir) _python_cmd('setup.py', '-q', 'bdist_egg', '--dist-dir', to_dir) finally: os.chdir(old_wd) shutil.rmtree(tmpdir) # returning the result log.warn(egg) if not os.path.exists(egg): raise IOError('Could not build the egg.') def _do_download(version, download_base, to_dir, download_delay): egg = os.path.join(to_dir, 'distribute-%s-py%d.%d.egg' % (version, sys.version_info[0], sys.version_info[1])) if not os.path.exists(egg): tarball = download_setuptools(version, download_base, to_dir, download_delay) _build_egg(egg, tarball, to_dir) sys.path.insert(0, egg) import setuptools setuptools.bootstrap_install_from = egg def use_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, download_delay=15, no_fake=True): # making sure we use the absolute path to_dir = os.path.abspath(to_dir) was_imported = 'pkg_resources' in sys.modules or \ 'setuptools' in sys.modules try: try: import pkg_resources if not hasattr(pkg_resources, '_distribute'): if not no_fake: _fake_setuptools() raise ImportError except ImportError: return _do_download(version, download_base, to_dir, download_delay) try: pkg_resources.require("distribute>=" + version) return except pkg_resources.VersionConflict: e = sys.exc_info()[1] if was_imported: sys.stderr.write( "The required version of distribute (>=%s) is not available,\n" "and can't be installed while this script is running. Please\n" "install a more recent version first, using\n" "'easy_install -U distribute'." "\n\n(Currently using %r)\n" % (version, e.args[0])) sys.exit(2) else: del pkg_resources, sys.modules['pkg_resources'] # reload ok return _do_download(version, download_base, to_dir, download_delay) except pkg_resources.DistributionNotFound: return _do_download(version, download_base, to_dir, download_delay) finally: if not no_fake: _create_fake_setuptools_pkg_info(to_dir) def download_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir, delay=15): """Download distribute from a specified location and return its filename `version` should be a valid distribute version number that is available as an egg for download under the `download_base` URL (which should end with a '/'). `to_dir` is the directory where the egg will be downloaded. `delay` is the number of seconds to pause before an actual download attempt. """ # making sure we use the absolute path to_dir = os.path.abspath(to_dir) try: from urllib.request import urlopen except ImportError: from urllib2 import urlopen tgz_name = "distribute-%s.tar.gz" % version url = download_base + tgz_name saveto = os.path.join(to_dir, tgz_name) src = dst = None if not os.path.exists(saveto): # Avoid repeated downloads try: log.warn("Downloading %s", url) src = urlopen(url) # Read/write all in one block, so we don't create a corrupt file # if the download is interrupted. data = src.read() dst = open(saveto, "wb") dst.write(data) finally: if src: src.close() if dst: dst.close() return os.path.realpath(saveto) def _no_sandbox(function): def __no_sandbox(*args, **kw): try: from setuptools.sandbox import DirectorySandbox if not hasattr(DirectorySandbox, '_old'): def violation(*args): pass DirectorySandbox._old = DirectorySandbox._violation DirectorySandbox._violation = violation patched = True else: patched = False except ImportError: patched = False try: return function(*args, **kw) finally: if patched: DirectorySandbox._violation = DirectorySandbox._old del DirectorySandbox._old return __no_sandbox def _patch_file(path, content): """Will backup the file then patch it""" existing_content = open(path).read() if existing_content == content: # already patched log.warn('Already patched.') return False log.warn('Patching...') _rename_path(path) f = open(path, 'w') try: f.write(content) finally: f.close() return True _patch_file = _no_sandbox(_patch_file) def _same_content(path, content): return open(path).read() == content def _rename_path(path): new_name = path + '.OLD.%s' % time.time() log.warn('Renaming %s to %s', path, new_name) os.rename(path, new_name) return new_name def _remove_flat_installation(placeholder): if not os.path.isdir(placeholder): log.warn('Unkown installation at %s', placeholder) return False found = False for file in os.listdir(placeholder): if fnmatch.fnmatch(file, 'setuptools*.egg-info'): found = True break if not found: log.warn('Could not locate setuptools*.egg-info') return log.warn('Moving elements out of the way...') pkg_info = os.path.join(placeholder, file) if os.path.isdir(pkg_info): patched = _patch_egg_dir(pkg_info) else: patched = _patch_file(pkg_info, SETUPTOOLS_PKG_INFO) if not patched: log.warn('%s already patched.', pkg_info) return False # now let's move the files out of the way for element in ('setuptools', 'pkg_resources.py', 'site.py'): element = os.path.join(placeholder, element) if os.path.exists(element): _rename_path(element) else: log.warn('Could not find the %s element of the ' 'Setuptools distribution', element) return True _remove_flat_installation = _no_sandbox(_remove_flat_installation) def _after_install(dist): log.warn('After install bootstrap.') placeholder = dist.get_command_obj('install').install_purelib _create_fake_setuptools_pkg_info(placeholder) def _create_fake_setuptools_pkg_info(placeholder): if not placeholder or not os.path.exists(placeholder): log.warn('Could not find the install location') return pyver = '%s.%s' % (sys.version_info[0], sys.version_info[1]) setuptools_file = 'setuptools-%s-py%s.egg-info' % \ (SETUPTOOLS_FAKED_VERSION, pyver) pkg_info = os.path.join(placeholder, setuptools_file) if os.path.exists(pkg_info): log.warn('%s already exists', pkg_info) return log.warn('Creating %s', pkg_info) try: f = open(pkg_info, 'w') except EnvironmentError: log.warn("Don't have permissions to write %s, skipping", pkg_info) return try: f.write(SETUPTOOLS_PKG_INFO) finally: f.close() pth_file = os.path.join(placeholder, 'setuptools.pth') log.warn('Creating %s', pth_file) f = open(pth_file, 'w') try: f.write(os.path.join(os.curdir, setuptools_file)) finally: f.close() _create_fake_setuptools_pkg_info = _no_sandbox( _create_fake_setuptools_pkg_info ) def _patch_egg_dir(path): # let's check if it's already patched pkg_info = os.path.join(path, 'EGG-INFO', 'PKG-INFO') if os.path.exists(pkg_info): if _same_content(pkg_info, SETUPTOOLS_PKG_INFO): log.warn('%s already patched.', pkg_info) return False _rename_path(path) os.mkdir(path) os.mkdir(os.path.join(path, 'EGG-INFO')) pkg_info = os.path.join(path, 'EGG-INFO', 'PKG-INFO') f = open(pkg_info, 'w') try: f.write(SETUPTOOLS_PKG_INFO) finally: f.close() return True _patch_egg_dir = _no_sandbox(_patch_egg_dir) def _before_install(): log.warn('Before install bootstrap.') _fake_setuptools() def _under_prefix(location): if 'install' not in sys.argv: return True args = sys.argv[sys.argv.index('install') + 1:] for index, arg in enumerate(args): for option in ('--root', '--prefix'): if arg.startswith('%s=' % option): top_dir = arg.split('root=')[-1] return location.startswith(top_dir) elif arg == option: if len(args) > index: top_dir = args[index + 1] return location.startswith(top_dir) if arg == '--user' and USER_SITE is not None: return location.startswith(USER_SITE) return True def _fake_setuptools(): log.warn('Scanning installed packages') try: import pkg_resources except ImportError: # we're cool log.warn('Setuptools or Distribute does not seem to be installed.') return ws = pkg_resources.working_set try: setuptools_dist = ws.find( pkg_resources.Requirement.parse('setuptools', replacement=False) ) except TypeError: # old distribute API setuptools_dist = ws.find( pkg_resources.Requirement.parse('setuptools') ) if setuptools_dist is None: log.warn('No setuptools distribution found') return # detecting if it was already faked setuptools_location = setuptools_dist.location log.warn('Setuptools installation detected at %s', setuptools_location) # if --root or --preix was provided, and if # setuptools is not located in them, we don't patch it if not _under_prefix(setuptools_location): log.warn('Not patching, --root or --prefix is installing Distribute' ' in another location') return # let's see if its an egg if not setuptools_location.endswith('.egg'): log.warn('Non-egg installation') res = _remove_flat_installation(setuptools_location) if not res: return else: log.warn('Egg installation') pkg_info = os.path.join(setuptools_location, 'EGG-INFO', 'PKG-INFO') if (os.path.exists(pkg_info) and _same_content(pkg_info, SETUPTOOLS_PKG_INFO)): log.warn('Already patched.') return log.warn('Patching...') # let's create a fake egg replacing setuptools one res = _patch_egg_dir(setuptools_location) if not res: return log.warn('Patching complete.') _relaunch() def _relaunch(): log.warn('Relaunching...') # we have to relaunch the process # pip marker to avoid a relaunch bug _cmd1 = ['-c', 'install', '--single-version-externally-managed'] _cmd2 = ['-c', 'install', '--record'] if sys.argv[:3] == _cmd1 or sys.argv[:3] == _cmd2: sys.argv[0] = 'setup.py' args = [sys.executable] + sys.argv sys.exit(subprocess.call(args)) def _extractall(self, path=".", members=None): """Extract all members from the archive to the current working directory and set owner, modification time and permissions on directories afterwards. `path' specifies a different directory to extract to. `members' is optional and must be a subset of the list returned by getmembers(). """ import copy import operator from tarfile import ExtractError directories = [] if members is None: members = self for tarinfo in members: if tarinfo.isdir(): # Extract directories with a safe mode. directories.append(tarinfo) tarinfo = copy.copy(tarinfo) tarinfo.mode = 448 # decimal for oct 0700 self.extract(tarinfo, path) # Reverse sort directories. if sys.version_info < (2, 4): def sorter(dir1, dir2): return cmp(dir1.name, dir2.name) directories.sort(sorter) directories.reverse() else: directories.sort(key=operator.attrgetter('name'), reverse=True) # Set correct owner, mtime and filemode on directories. for tarinfo in directories: dirpath = os.path.join(path, tarinfo.name) try: self.chown(tarinfo, dirpath) self.utime(tarinfo, dirpath) self.chmod(tarinfo, dirpath) except ExtractError: e = sys.exc_info()[1] if self.errorlevel > 1: raise else: self._dbg(1, "tarfile: %s" % e) def _build_install_args(options): """ Build the arguments to 'python setup.py install' on the distribute package """ install_args = [] if options.user_install: if sys.version_info < (2, 6): log.warn("--user requires Python 2.6 or later") raise SystemExit(1) install_args.append('--user') return install_args def _parse_args(): """ Parse the command line for options """ parser = optparse.OptionParser() parser.add_option( '--user', dest='user_install', action='store_true', default=False, help='install in user site package (requires Python 2.6 or later)') parser.add_option( '--download-base', dest='download_base', metavar="URL", default=DEFAULT_URL, help='alternative URL from where to download the distribute package') options, args = parser.parse_args() # positional arguments are ignored return options def main(version=DEFAULT_VERSION): """Install or upgrade setuptools and EasyInstall""" options = _parse_args() tarball = download_setuptools(download_base=options.download_base) return _install(tarball, _build_install_args(options)) if __name__ == '__main__': sys.exit(main()) pymongo-2.6.3/doc/000077500000000000000000000000001223300253600137535ustar00rootroot00000000000000pymongo-2.6.3/doc/__init__.py000066400000000000000000000000011223300253600160530ustar00rootroot00000000000000 pymongo-2.6.3/doc/api/000077500000000000000000000000001223300253600145245ustar00rootroot00000000000000pymongo-2.6.3/doc/api/bson/000077500000000000000000000000001223300253600154655ustar00rootroot00000000000000pymongo-2.6.3/doc/api/bson/binary.rst000066400000000000000000000013231223300253600175020ustar00rootroot00000000000000:mod:`binary` -- Tools for representing binary data to be stored in MongoDB =========================================================================== .. automodule:: bson.binary :synopsis: Tools for representing binary data to be stored in MongoDB .. autodata:: BINARY_SUBTYPE .. autodata:: FUNCTION_SUBTYPE .. autodata:: OLD_BINARY_SUBTYPE .. autodata:: OLD_UUID_SUBTYPE .. autodata:: UUID_SUBTYPE .. autodata:: JAVA_LEGACY .. autodata:: CSHARP_LEGACY .. autodata:: MD5_SUBTYPE .. autodata:: USER_DEFINED_SUBTYPE .. autoclass:: Binary(data[, subtype=BINARY_SUBTYPE]) :members: :show-inheritance: .. autoclass:: UUIDLegacy(obj) :members: :show-inheritance: pymongo-2.6.3/doc/api/bson/code.rst000066400000000000000000000004351223300253600171330ustar00rootroot00000000000000:mod:`code` -- Tools for representing JavaScript code ===================================================== .. automodule:: bson.code :synopsis: Tools for representing JavaScript code .. autoclass:: Code(code[, scope=None[, **kwargs]]) :members: :show-inheritance: pymongo-2.6.3/doc/api/bson/dbref.rst000066400000000000000000000004651223300253600173060ustar00rootroot00000000000000:mod:`dbref` -- Tools for manipulating DBRefs (references to documents stored in MongoDB) ========================================================================================= .. automodule:: bson.dbref :synopsis: Tools for manipulating DBRefs (references to documents stored in MongoDB) :members: pymongo-2.6.3/doc/api/bson/errors.rst000066400000000000000000000003351223300253600175340ustar00rootroot00000000000000:mod:`errors` -- Exceptions raised by the :mod:`bson` package ================================================================ .. automodule:: bson.errors :synopsis: Exceptions raised by the bson package :members: pymongo-2.6.3/doc/api/bson/index.rst000066400000000000000000000005521223300253600173300ustar00rootroot00000000000000:mod:`bson` -- BSON (Binary JSON) Encoding and Decoding ======================================================= .. automodule:: bson :synopsis: BSON (Binary JSON) Encoding and Decoding :members: Sub-modules: .. toctree:: :maxdepth: 2 binary code dbref errors json_util max_key min_key objectid son timestamp tz_util pymongo-2.6.3/doc/api/bson/json_util.rst000066400000000000000000000005121223300253600202230ustar00rootroot00000000000000:mod:`json_util` -- Tools for using Python's :mod:`json` module with BSON documents ====================================================================================== .. versionadded:: 1.1.1 .. automodule:: bson.json_util :synopsis: Tools for using Python's json module with BSON documents :members: :undoc-members: pymongo-2.6.3/doc/api/bson/max_key.rst000066400000000000000000000004171223300253600176560ustar00rootroot00000000000000:mod:`max_key` -- Representation for the MongoDB internal MaxKey type ===================================================================== .. versionadded:: 1.7 .. automodule:: bson.max_key :synopsis: Representation for the MongoDB internal MaxKey type :members: pymongo-2.6.3/doc/api/bson/min_key.rst000066400000000000000000000004171223300253600176540ustar00rootroot00000000000000:mod:`min_key` -- Representation for the MongoDB internal MinKey type ===================================================================== .. versionadded:: 1.7 .. automodule:: bson.min_key :synopsis: Representation for the MongoDB internal MinKey type :members: pymongo-2.6.3/doc/api/bson/objectid.rst000066400000000000000000000012771223300253600200110ustar00rootroot00000000000000:mod:`objectid` -- Tools for working with MongoDB ObjectIds =========================================================== .. automodule:: bson.objectid :synopsis: Tools for working with MongoDB ObjectIds .. autoclass:: bson.objectid.ObjectId([oid=None]) :members: .. describe:: str(o) Get a hex encoded version of :class:`ObjectId` `o`. The following property always holds: .. testsetup:: from bson.objectid import ObjectId .. doctest:: >>> o = ObjectId() >>> o == ObjectId(str(o)) True This representation is useful for urls or other places where ``o.binary`` is inappropriate. pymongo-2.6.3/doc/api/bson/son.rst000066400000000000000000000003361223300253600170200ustar00rootroot00000000000000:mod:`son` -- Tools for working with SON, an ordered mapping ============================================================ .. automodule:: bson.son :synopsis: Tools for working with SON, an ordered mapping :members: pymongo-2.6.3/doc/api/bson/timestamp.rst000066400000000000000000000004221223300253600202200ustar00rootroot00000000000000:mod:`timestamp` -- Tools for representing MongoDB internal Timestamps ====================================================================== .. versionadded:: 1.5 .. automodule:: bson.timestamp :synopsis: Tools for representing MongoDB internal Timestamps :members: pymongo-2.6.3/doc/api/bson/tz_util.rst000066400000000000000000000003521223300253600177110ustar00rootroot00000000000000:mod:`tz_util` -- Utilities for dealing with timezones in Python ================================================================ .. automodule:: bson.tz_util :synopsis: Utilities for dealing with timezones in Python :members: pymongo-2.6.3/doc/api/gridfs/000077500000000000000000000000001223300253600160025ustar00rootroot00000000000000pymongo-2.6.3/doc/api/gridfs/errors.rst000066400000000000000000000003441223300253600200510ustar00rootroot00000000000000:mod:`errors` -- Exceptions raised by the :mod:`gridfs` package ================================================================= .. automodule:: gridfs.errors :synopsis: Exceptions raised by the gridfs package :members: pymongo-2.6.3/doc/api/gridfs/grid_file.rst000066400000000000000000000006761223300253600204710ustar00rootroot00000000000000:mod:`grid_file` -- Tools for representing files stored in GridFS ================================================================= .. automodule:: gridfs.grid_file :synopsis: Tools for representing files stored in GridFS .. autoclass:: GridIn :members: .. autoattribute:: _id .. autoclass:: GridOut :members: .. autoattribute:: _id .. automethod:: __iter__ .. autoclass:: GridFile :members: pymongo-2.6.3/doc/api/gridfs/index.rst000066400000000000000000000003631223300253600176450ustar00rootroot00000000000000:mod:`gridfs` -- Tools for working with GridFS ============================================== .. automodule:: gridfs :synopsis: Tools for working with GridFS :members: Sub-modules: .. toctree:: :maxdepth: 2 errors grid_file pymongo-2.6.3/doc/api/index.rst000066400000000000000000000007451223300253600163730ustar00rootroot00000000000000API Documentation ================= The PyMongo distribution contains three top-level packages for interacting with MongoDB. :mod:`bson` is an implementation of the `BSON format `_, :mod:`pymongo` is a full-featured driver for MongoDB, and :mod:`gridfs` is a set of tools for working with the `GridFS `_ storage specification. .. toctree:: :maxdepth: 2 bson/index pymongo/index gridfs/index pymongo-2.6.3/doc/api/pymongo/000077500000000000000000000000001223300253600162145ustar00rootroot00000000000000pymongo-2.6.3/doc/api/pymongo/collection.rst000066400000000000000000000047741223300253600211150ustar00rootroot00000000000000:mod:`collection` -- Collection level operations ================================================ .. automodule:: pymongo.collection :synopsis: Collection level operations .. autodata:: pymongo.ASCENDING .. autodata:: pymongo.DESCENDING .. autodata:: pymongo.GEO2D .. autodata:: pymongo.GEOHAYSTACK .. autodata:: pymongo.GEOSPHERE .. autodata:: pymongo.HASHED .. autoclass:: pymongo.collection.Collection(database, name[, create=False[, **kwargs]]]) .. describe:: c[name] || c.name Get the `name` sub-collection of :class:`Collection` `c`. Raises :class:`~pymongo.errors.InvalidName` if an invalid collection name is used. .. autoattribute:: full_name .. autoattribute:: name .. autoattribute:: database .. autoattribute:: read_preference .. autoattribute:: tag_sets .. autoattribute:: secondary_acceptable_latency_ms .. autoattribute:: write_concern .. autoattribute:: uuid_subtype .. automethod:: insert(doc_or_docs[, manipulate=True[, safe=None[, check_keys=True[, continue_on_error=False[, **kwargs]]]]]) .. automethod:: save(to_save[, manipulate=True[, safe=None[, check_keys=True[, **kwargs]]]]) .. automethod:: update(spec, document[, upsert=False[, manipulate=False[, safe=None[, multi=False[, check_keys=True[, **kwargs]]]]]]) .. automethod:: remove([spec_or_id=None[, safe=None[, **kwargs]]]) .. automethod:: drop .. automethod:: find([spec=None[, fields=None[, skip=0[, limit=0[, timeout=True[, snapshot=False[, tailable=False[, sort=None[, max_scan=None[, as_class=None[, slave_okay=False[, await_data=False[, partial=False[, manipulate=True[, read_preference=ReadPreference.PRIMARY[, exhaust=False[,**kwargs]]]]]]]]]]]]]]]]]) .. automethod:: find_one([spec_or_id=None[, *args[, **kwargs]]]) .. automethod:: count .. automethod:: create_index .. automethod:: ensure_index .. automethod:: drop_index .. automethod:: drop_indexes .. automethod:: reindex .. automethod:: index_information .. automethod:: options .. automethod:: aggregate .. automethod:: group .. automethod:: rename .. automethod:: distinct .. automethod:: map_reduce .. automethod:: inline_map_reduce .. automethod:: find_and_modify .. autoattribute:: slave_okay .. autoattribute:: safe .. automethod:: get_lasterror_options .. automethod:: set_lasterror_options .. automethod:: unset_lasterror_options pymongo-2.6.3/doc/api/pymongo/connection.rst000066400000000000000000000037051223300253600211120ustar00rootroot00000000000000:mod:`connection` -- Tools for connecting to MongoDB ==================================================== .. automodule:: pymongo.connection :synopsis: Tools for connecting to MongoDB .. autoclass:: pymongo.connection.Connection([host='localhost'[, port=27017[, max_pool_size=10[, network_timeout=None[, document_class=dict[, tz_aware=False[, **kwargs]]]]]]]) .. automethod:: disconnect .. automethod:: close .. automethod:: alive .. describe:: c[db_name] || c.db_name Get the `db_name` :class:`~pymongo.database.Database` on :class:`Connection` `c`. Raises :class:`~pymongo.errors.InvalidName` if an invalid database name is used. .. autoattribute:: host .. autoattribute:: port .. autoattribute:: is_primary .. autoattribute:: is_mongos .. autoattribute:: max_pool_size .. autoattribute:: nodes .. autoattribute:: auto_start_request .. autoattribute:: document_class .. autoattribute:: tz_aware .. autoattribute:: max_bson_size .. autoattribute:: max_message_size .. autoattribute:: read_preference .. autoattribute:: tag_sets .. autoattribute:: secondary_acceptable_latency_ms .. autoattribute:: write_concern .. autoattribute:: slave_okay .. autoattribute:: safe .. autoattribute:: is_locked .. automethod:: database_names .. automethod:: drop_database .. automethod:: copy_database(from_name, to_name[, from_host=None[, username=None[, password=None]]]) .. automethod:: get_default_database .. automethod:: server_info .. automethod:: start_request .. automethod:: end_request .. automethod:: close_cursor .. automethod:: kill_cursors .. automethod:: set_cursor_manager .. automethod:: fsync .. automethod:: unlock .. automethod:: get_lasterror_options .. automethod:: set_lasterror_options .. automethod:: unset_lasterror_options pymongo-2.6.3/doc/api/pymongo/cursor.rst000066400000000000000000000013141223300253600202620ustar00rootroot00000000000000:mod:`cursor` -- Tools for iterating over MongoDB query results =============================================================== .. automodule:: pymongo.cursor :synopsis: Tools for iterating over MongoDB query results .. autoclass:: pymongo.cursor.Cursor(collection, spec=None, fields=None, skip=0, limit=0, timeout=True, snapshot=False, tailable=False, sort=None, max_scan=None, as_class=None, slave_okay=False, await_data=False, partial=False, manipulate=True, read_preference=ReadPreference.PRIMARY, tag_sets=[{}], secondary_acceptable_latency_ms=None, exhaust=False, network_timeout=None) :members: .. describe:: c[index] See :meth:`__getitem__`. .. automethod:: __getitem__ pymongo-2.6.3/doc/api/pymongo/cursor_manager.rst000066400000000000000000000005131223300253600217540ustar00rootroot00000000000000:mod:`cursor_manager` -- Managers to handle when cursors are killed after being closed -- DEPRECATED ==================================================================================================== .. automodule:: pymongo.cursor_manager :synopsis: Managers to handle when cursors are killed after being closed :members: pymongo-2.6.3/doc/api/pymongo/database.rst000066400000000000000000000023131223300253600205110ustar00rootroot00000000000000:mod:`database` -- Database level operations ============================================ .. automodule:: pymongo.database :synopsis: Database level operations .. autodata:: pymongo.auth.MECHANISMS .. autodata:: pymongo.OFF .. autodata:: pymongo.SLOW_ONLY .. autodata:: pymongo.ALL .. autoclass:: pymongo.database.Database :members: .. describe:: db[collection_name] || db.collection_name Get the `collection_name` :class:`~pymongo.collection.Collection` of :class:`Database` `db`. Raises :class:`~pymongo.errors.InvalidName` if an invalid collection name is used. .. note:: Use dictionary style access if `collection_name` is an attribute of the :class:`Database` class eg: db[`collection_name`]. .. autoattribute:: read_preference .. autoattribute:: tag_sets .. autoattribute:: secondary_acceptable_latency_ms .. autoattribute:: write_concern .. autoattribute:: slave_okay .. autoattribute:: safe .. automethod:: get_lasterror_options .. automethod:: set_lasterror_options .. automethod:: unset_lasterror_options .. autoclass:: pymongo.database.SystemJS :members: pymongo-2.6.3/doc/api/pymongo/errors.rst000066400000000000000000000003461223300253600202650ustar00rootroot00000000000000:mod:`errors` -- Exceptions raised by the :mod:`pymongo` package ================================================================ .. automodule:: pymongo.errors :synopsis: Exceptions raised by the pymongo package :members: pymongo-2.6.3/doc/api/pymongo/index.rst000066400000000000000000000013441223300253600200570ustar00rootroot00000000000000:mod:`pymongo` -- Python driver for MongoDB =========================================== .. automodule:: pymongo :synopsis: Python driver for MongoDB .. autodata:: version .. data:: MongoClient Alias for :class:`pymongo.mongo_client.MongoClient`. .. data:: MongoReplicaSetClient Alias for :class:`pymongo.mongo_replica_set_client.MongoReplicaSetClient`. .. autoclass:: pymongo.read_preferences.ReadPreference .. autofunction:: has_c Sub-modules: .. toctree:: :maxdepth: 2 connection database collection cursor errors master_slave_connection message mongo_client mongo_replica_set_client pool replica_set_connection son_manipulator cursor_manager uri_parser pymongo-2.6.3/doc/api/pymongo/master_slave_connection.rst000066400000000000000000000007701223300253600236560ustar00rootroot00000000000000:mod:`master_slave_connection` -- Master-slave connection to MongoDB ==================================================================== .. automodule:: pymongo.master_slave_connection :synopsis: Master-slave connection to MongoDB .. autoclass:: pymongo.master_slave_connection.MasterSlaveConnection :members: .. autoattribute:: safe .. automethod:: get_lasterror_options .. automethod:: set_lasterror_options .. automethod:: unset_lasterror_options pymongo-2.6.3/doc/api/pymongo/message.rst000066400000000000000000000003661223300253600203770ustar00rootroot00000000000000:mod:`message` -- Tools for creating messages to be sent to MongoDB =================================================================== .. automodule:: pymongo.message :synopsis: Tools for creating messages to be sent to MongoDB :members: pymongo-2.6.3/doc/api/pymongo/mongo_client.rst000066400000000000000000000033571223300253600214330ustar00rootroot00000000000000:mod:`mongo_client` -- Tools for connecting to MongoDB ====================================================== .. automodule:: pymongo.mongo_client :synopsis: Tools for connecting to MongoDB .. autoclass:: pymongo.mongo_client.MongoClient([host='localhost'[, port=27017[, max_pool_size=10[, document_class=dict[, tz_aware=False[, **kwargs]]]]]]) .. automethod:: disconnect .. automethod:: close .. automethod:: alive .. describe:: c[db_name] || c.db_name Get the `db_name` :class:`~pymongo.database.Database` on :class:`MongoClient` `c`. Raises :class:`~pymongo.errors.InvalidName` if an invalid database name is used. .. autoattribute:: host .. autoattribute:: port .. autoattribute:: is_primary .. autoattribute:: is_mongos .. autoattribute:: max_pool_size .. autoattribute:: nodes .. autoattribute:: auto_start_request .. autoattribute:: document_class .. autoattribute:: tz_aware .. autoattribute:: max_bson_size .. autoattribute:: max_message_size .. autoattribute:: read_preference .. autoattribute:: tag_sets .. autoattribute:: secondary_acceptable_latency_ms .. autoattribute:: write_concern .. autoattribute:: is_locked .. automethod:: database_names .. automethod:: drop_database .. automethod:: copy_database(from_name, to_name[, from_host=None[, username=None[, password=None]]]) .. automethod:: get_default_database .. automethod:: server_info .. automethod:: start_request .. automethod:: end_request .. automethod:: close_cursor .. automethod:: kill_cursors .. automethod:: set_cursor_manager .. automethod:: fsync .. automethod:: unlock pymongo-2.6.3/doc/api/pymongo/mongo_replica_set_client.rst000066400000000000000000000031421223300253600237750ustar00rootroot00000000000000:mod:`mongo_replica_set_client` -- Tools for connecting to a MongoDB replica set ================================================================================ .. automodule:: pymongo.mongo_replica_set_client :synopsis: Tools for connecting to a MongoDB replica set .. autoclass:: pymongo.mongo_replica_set_client.MongoReplicaSetClient([hosts_or_uri[, max_pool_size=10[, document_class=dict[, tz_aware=False[, **kwargs]]]]]) .. automethod:: disconnect .. automethod:: close .. automethod:: alive .. describe:: c[db_name] || c.db_name Get the `db_name` :class:`~pymongo.database.Database` on :class:`MongoReplicaSetClient` `c`. Raises :class:`~pymongo.errors.InvalidName` if an invalid database name is used. .. autoattribute:: seeds .. autoattribute:: hosts .. autoattribute:: primary .. autoattribute:: secondaries .. autoattribute:: arbiters .. autoattribute:: is_mongos .. autoattribute:: max_pool_size .. autoattribute:: document_class .. autoattribute:: tz_aware .. autoattribute:: max_bson_size .. autoattribute:: max_message_size .. autoattribute:: auto_start_request .. autoattribute:: read_preference .. autoattribute:: tag_sets .. autoattribute:: secondary_acceptable_latency_ms .. autoattribute:: write_concern .. automethod:: database_names .. automethod:: drop_database .. automethod:: copy_database(from_name, to_name[, from_host=None[, username=None[, password=None]]]) .. automethod:: get_default_database .. automethod:: close_cursor pymongo-2.6.3/doc/api/pymongo/pool.rst000066400000000000000000000003351223300253600177200ustar00rootroot00000000000000:mod:`pool` -- Pool module for use with a MongoDB client. ============================================================== .. automodule:: pymongo.pool :synopsis: Pool module for use with a MongoDB client. :members: pymongo-2.6.3/doc/api/pymongo/replica_set_connection.rst000066400000000000000000000033741223300253600234660ustar00rootroot00000000000000:mod:`replica_set_connection` -- Tools for connecting to a MongoDB replica set ============================================================================== .. automodule:: pymongo.replica_set_connection :synopsis: Tools for connecting to a MongoDB replica set .. autoclass:: pymongo.replica_set_connection.ReplicaSetConnection([hosts_or_uri[, max_pool_size=10[, document_class=dict[, tz_aware=False[, **kwargs]]]]]) .. automethod:: disconnect .. automethod:: close .. automethod:: alive .. describe:: c[db_name] || c.db_name Get the `db_name` :class:`~pymongo.database.Database` on :class:`ReplicaSetConnection` `c`. Raises :class:`~pymongo.errors.InvalidName` if an invalid database name is used. .. autoattribute:: seeds .. autoattribute:: hosts .. autoattribute:: primary .. autoattribute:: secondaries .. autoattribute:: arbiters .. autoattribute:: is_mongos .. autoattribute:: max_pool_size .. autoattribute:: document_class .. autoattribute:: tz_aware .. autoattribute:: max_bson_size .. autoattribute:: max_message_size .. autoattribute:: auto_start_request .. autoattribute:: read_preference .. autoattribute:: tag_sets .. autoattribute:: secondary_acceptable_latency_ms .. autoattribute:: write_concern .. autoattribute:: safe .. automethod:: database_names .. automethod:: drop_database .. automethod:: copy_database(from_name, to_name[, from_host=None[, username=None[, password=None]]]) .. automethod:: get_default_database .. automethod:: close_cursor .. automethod:: get_lasterror_options .. automethod:: set_lasterror_options .. automethod:: unset_lasterror_options pymongo-2.6.3/doc/api/pymongo/son_manipulator.rst000066400000000000000000000005201223300253600221550ustar00rootroot00000000000000:mod:`son_manipulator` -- Manipulators that can edit SON documents as they are saved or retrieved ================================================================================================= .. automodule:: pymongo.son_manipulator :synopsis: Manipulators that can edit SON documents as they are saved or retrieved :members: pymongo-2.6.3/doc/api/pymongo/uri_parser.rst000066400000000000000000000003501223300253600211170ustar00rootroot00000000000000:mod:`uri_parser` -- Tools to parse and validate a MongoDB URI ============================================================== .. automodule:: pymongo.uri_parser :synopsis: Tools to parse and validate a MongoDB URI. :members: pymongo-2.6.3/doc/changelog.rst000066400000000000000000001356441223300253600164510ustar00rootroot00000000000000Changelog ========= Changes in Version 2.6.3 ------------------------ Version 2.6.3 fixes issues reported since the release of 2.6.2, most importantly a semaphore leak when a connection to the server fails. Issues Resolved ............... See the `PyMongo 2.6.3 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.6.3 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/13098 Changes in Version 2.6.2 ------------------------ Version 2.6.2 fixes a :exc:`TypeError` problem when max_pool_size=None is used in Python 3. Issues Resolved ............... See the `PyMongo 2.6.2 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.6.2 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/12910 Changes in Version 2.6.1 ------------------------ Version 2.6.1 fixes a reference leak in the :meth:`~pymongo.collection.Collection.insert` method. Issues Resolved ............... See the `PyMongo 2.6.1 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.6.1 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/12905 Changes in Version 2.6 ---------------------- Version 2.6 includes some frequently requested improvements and adds support for some early MongoDB 2.6 features. Special thanks go to Justin Patrin for his work on the connection pool in this release. Important new features: - The ``max_pool_size`` option for :class:`~pymongo.mongo_client.MongoClient` and :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` now actually caps the number of sockets the pool will open concurrently. Once the pool has reached :attr:`~pymongo.mongo_client.MongoClient.max_pool_size` operations will block waiting for a socket to become available. If ``waitQueueTimeoutMS`` is set, an operation that blocks waiting for a socket will raise :exc:`~pymongo.errors.ConnectionFailure` after the timeout. By default ``waitQueueTimeoutMS`` is not set. See :ref:`connection-pooling` for more information. - The :meth:`~pymongo.collection.Collection.insert` method automatically splits large batches of documents into multiple insert messages based on :attr:`~pymongo.mongo_client.MongoClient.max_message_size` - Support for the exhaust cursor flag. See :meth:`~pymongo.collection.Collection.find` for details and caveats. - Support for the PLAIN and MONGODB-X509 authentication mechanisms. See :doc:`the authentication docs ` for more information. - Support aggregation output as a :class:`~pymongo.cursor.Cursor`. See :meth:`~pymongo.collection.Collection.aggregate` for details. .. warning:: SIGNIFICANT BEHAVIOR CHANGE in 2.6. Previously, `max_pool_size` would limit only the idle sockets the pool would hold onto, not the number of open sockets. The default has also changed, from 10 to 100. If you pass a value for ``max_pool_size`` make sure it is large enough for the expected load. (Sockets are only opened when needed, so there is no cost to having a ``max_pool_size`` larger than necessary. Err towards a larger value.) If your application accepts the default, continue to do so. See :ref:`connection-pooling` for more information. Issues Resolved ............... See the `PyMongo 2.6 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.6 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/12380 Changes in Version 2.5.2 ------------------------ Version 2.5.2 fixes a NULL pointer dereference issue when decoding an invalid :class:`~bson.dbref.DBRef`. Issues Resolved ............... See the `PyMongo 2.5.2 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.5.2 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/12581 Changes in Version 2.5.1 ------------------------ Version 2.5.1 is a minor release that fixes issues discovered after the release of 2.5. Most importantly, this release addresses some race conditions in replica set monitoring. Issues Resolved ............... See the `PyMongo 2.5.1 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.5.1 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/12484 Changes in Version 2.5 ---------------------- Version 2.5 includes changes to support new features in MongoDB 2.4. Important new features: - Support for :ref:`GSSAPI (Kerberos) authentication `. - Support for SSL certificate validation with hostname matching. - Support for delegated and role based authentication. - New GEOSPHERE (2dsphere) and HASHED index constants. .. note:: :meth:`~pymongo.database.Database.authenticate` now raises a subclass of :class:`~pymongo.errors.PyMongoError` if authentication fails due to invalid credentials or configuration issues. Issues Resolved ............... See the `PyMongo 2.5 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.5 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/11981 Changes in Version 2.4.2 ------------------------ Version 2.4.2 is a minor release that fixes issues discovered after the release of 2.4.1. Most importantly, PyMongo will no longer select a replica set member for read operations that is not in primary or secondary state. Issues Resolved ............... See the `PyMongo 2.4.2 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.4.2 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/12299 Changes in Version 2.4.1 ------------------------ Version 2.4.1 is a minor release that fixes issues discovered after the release of 2.4. Most importantly, this release fixes a regression using :meth:`~pymongo.collection.Collection.aggregate`, and possibly other commands, with mongos. Issues Resolved ............... See the `PyMongo 2.4.1 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.4.1 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/12286 Changes in Version 2.4 ---------------------- Version 2.4 includes a few important new features and a large number of bug fixes. Important new features: - New :class:`~pymongo.mongo_client.MongoClient` and :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` classes - these connection classes do acknowledged write operations (previously referred to as 'safe' writes) by default. :class:`~pymongo.connection.Connection` and :class:`~pymongo.replica_set_connection.ReplicaSetConnection` are deprecated but still support the old default fire-and-forget behavior. - A new write concern API implemented as a :attr:`~pymongo.collection.Collection.write_concern` attribute on the connection, :class:`~pymongo.database.Database`, or :class:`~pymongo.collection.Collection` classes. - :class:`~pymongo.mongo_client.MongoClient` (and :class:`~pymongo.connection.Connection`) now support Unix Domain Sockets. - :class:`~pymongo.cursor.Cursor` can be copied with functions from the :mod:`copy` module. - The :meth:`~pymongo.database.Database.set_profiling_level` method now supports a `slow_ms` option. - The replica set monitor task (used by :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` and :class:`~pymongo.replica_set_connection.ReplicaSetConnection`) is a daemon thread once again, meaning you won't have to call :meth:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient.close` before exiting the python interactive shell. .. warning:: The constructors for :class:`~pymongo.mongo_client.MongoClient`, :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient`, :class:`~pymongo.connection.Connection`, and :class:`~pymongo.replica_set_connection.ReplicaSetConnection` now raise :exc:`~pymongo.errors.ConnectionFailure` instead of its subclass :exc:`~pymongo.errors.AutoReconnect` if the server is unavailable. Applications that expect to catch :exc:`~pymongo.errors.AutoReconnect` should now catch :exc:`~pymongo.errors.ConnectionFailure` while creating a new connection. Issues Resolved ............... See the `PyMongo 2.4 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.4 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/11485 Changes in Version 2.3 ---------------------- Version 2.3 adds support for new features and behavior changes in MongoDB 2.2. Important New Features: - Support for expanded read preferences including directing reads to tagged servers - See :ref:`secondary-reads` for more information. - Support for mongos failover - See :ref:`mongos-high-availability` for more information. - A new :meth:`~pymongo.collection.Collection.aggregate` method to support MongoDB's new `aggregation framework `_. - Support for legacy Java and C# byte order when encoding and decoding UUIDs. - Support for connecting directly to an arbiter. .. warning:: Starting with MongoDB 2.2 the getLastError command requires authentication when the server's `authentication features `_ are enabled. Changes to PyMongo were required to support this behavior change. Users of authentication must upgrade to PyMongo 2.3 (or newer) for "safe" write operations to function correctly. Issues Resolved ............... See the `PyMongo 2.3 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.3 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/11146 Changes in Version 2.2.1 ------------------------ Version 2.2.1 is a minor release that fixes issues discovered after the release of 2.2. Most importantly, this release fixes an incompatibility with mod_wsgi 2.x that could cause connections to leak. Users of mod_wsgi 2.x are strongly encouraged to upgrade from PyMongo 2.2. Issues Resolved ............... See the `PyMongo 2.2.1 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.2.1 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/11185 Changes in Version 2.2 ----------------------- Version 2.2 adds a few more frequently requested features and fixes a number of bugs. Special thanks go to Alex Grönholm for his contributions to Python 3 support and maintaining the original pymongo3 port. Christoph Simon, Wouter Bolsterlee, Mike O'Brien, and Chris Tompkinson also contributed to this release. Important New Features: - Support for Python 3 - See the :doc:`python3` for more information. - Support for Gevent - See :doc:`examples/gevent` for more information. - Improved connection pooling - See :doc:`examples/requests` for more information. .. warning:: A number of methods and method parameters that were deprecated in PyMongo 1.9 or older versions have been removed in this release. The full list of changes can be found in the following JIRA ticket: https://jira.mongodb.org/browse/PYTHON-305 BSON module aliases from the pymongo package that were deprecated in PyMongo 1.9 have also been removed in this release. See the following JIRA ticket for details: https://jira.mongodb.org/browse/PYTHON-304 As a result of this cleanup some minor code changes may be required to use this release. Issues Resolved ............... See the `PyMongo 2.2 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.2 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/10584 Changes in Version 2.1.1 ------------------------ Version 2.1.1 is a minor release that fixes a few issues discovered after the release of 2.1. You can now use :class:`~pymongo.replica_set_connection.ReplicaSetConnection` to run inline map reduce commands on secondaries. See :meth:`~pymongo.collection.Collection.inline_map_reduce` for details. Special thanks go to Samuel Clay and Ross Lawley for their contributions to this release. Issues Resolved ............... See the `PyMongo 2.1.1 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.1.1 release notes in JIRA: https://jira.mongodb.org/browse/PYTHON/fixforversion/11081 Changes in Version 2.1 ------------------------ Version 2.1 adds a few frequently requested features and includes the usual round of bug fixes and improvements. Special thanks go to Alexey Borzenkov, Dan Crosta, Kostya Rybnikov, Flavio Percoco Premoli, Jonas Haag, and Jesse Davis for their contributions to this release. Important New Features: - ReplicaSetConnection - :class:`~pymongo.replica_set_connection.ReplicaSetConnection` can be used to distribute reads to secondaries in a replica set. It supports automatic failover handling and periodically checks the state of the replica set to handle issues like primary stepdown or secondaries being removed for backup operations. Read preferences are defined through :class:`~pymongo.read_preferences.ReadPreference`. - PyMongo supports the new BSON binary subtype 4 for UUIDs. The default subtype to use can be set through :attr:`~pymongo.collection.Collection.uuid_subtype` The current default remains :attr:`~bson.binary.OLD_UUID_SUBTYPE` but will be changed to :attr:`~bson.binary.UUID_SUBTYPE` in a future release. - The getLastError option 'w' can be set to a string, allowing for options like "majority" available in newer version of MongoDB. - Added support for the MongoDB URI options socketTimeoutMS and connectTimeoutMS. - Added support for the ContinueOnError insert flag. - Added basic SSL support. - Added basic support for Jython. - Secondaries can be used for :meth:`~pymongo.cursor.Cursor.count`, :meth:`~pymongo.cursor.Cursor.distinct`, :meth:`~pymongo.collection.Collection.group`, and querying :class:`~gridfs.GridFS`. - Added document_class and tz_aware options to :class:`~pymongo.master_slave_connection.MasterSlaveConnection` Issues Resolved ............... See the `PyMongo 2.1 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.1 release notes in JIRA: https://jira.mongodb.org/secure/ReleaseNote.jspa?projectId=10004&version=10583 Changes in Version 2.0.1 ------------------------ Version 2.0.1 fixes a regression in :class:`~gridfs.grid_file.GridIn` when writing pre-chunked strings. Thanks go to Alexey Borzenkov for reporting the issue and submitting a patch. Issues Resolved ............... - `PYTHON-271 `_: Regression in GridFS leads to serious loss of data. Changes in Version 2.0 ---------------------- Version 2.0 adds a large number of features and fixes a number of issues. Special thanks go to James Murty, Abhay Vardhan, David Pisoni, Ryan Smith-Roberts, Andrew Pendleton, Mher Movsisyan, Reed O'Brien, Michael Schurter, Josip Delic and Jonas Haag for their contributions to this release. Important New Features: - PyMongo now performs automatic per-socket database authentication. You no longer have to re-authenticate for each new thread or after a replica set failover. Authentication credentials are cached by the driver until the application calls :meth:`~pymongo.database.Database.logout`. - slave_okay can be set independently at the connection, database, collection or query level. Each level will inherit the slave_okay setting from the previous level and each level can override the previous level's setting. - safe and getLastError options (e.g. w, wtimeout, etc.) can be set independently at the connection, database, collection or query level. Each level will inherit settings from the previous level and each level can override the previous level's setting. - PyMongo now supports the `await_data` and `partial` cursor flags. If the `await_data` flag is set on a `tailable` cursor the server will block for some extra time waiting for more data to return. The `partial` flag tells a mongos to return partial data for a query if not all shards are available. - :meth:`~pymongo.collection.Collection.map_reduce` will accept a `dict` or instance of :class:`~bson.son.SON` as the `out` parameter. - The URI parser has been moved into its own module and can be used directly by application code. - AutoReconnect exception now provides information about the error that actually occured instead of a generic failure message. - A number of new helper methods have been added with options for setting and unsetting cursor flags, re-indexing a collection, fsync and locking a server, and getting the server's current operations. API changes: - If only one host:port pair is specified :class:`~pymongo.connection.Connection` will make a direct connection to only that host. Please note that `slave_okay` must be `True` in order to query from a secondary. - If more than one host:port pair is specified or the `replicaset` option is used PyMongo will treat the specified host:port pair(s) as a seed list and connect using replica set behavior. .. warning:: The default subtype for :class:`~bson.binary.Binary` has changed from :const:`~bson.binary.OLD_BINARY_SUBTYPE` (2) to :const:`~bson.binary.BINARY_SUBTYPE` (0). Issues Resolved ............... See the `PyMongo 2.0 release notes in JIRA`_ for the list of resolved issues in this release. .. _PyMongo 2.0 release notes in JIRA: https://jira.mongodb.org/secure/ReleaseNote.jspa?projectId=10004&version=10274 Changes in Version 1.11 ----------------------- Version 1.11 adds a few new features and fixes a few more bugs. New Features: - Basic IPv6 support: pymongo prefers IPv4 but will try IPv6. You can also specify an IPv6 address literal in the `host` parameter or a MongoDB URI provided it is enclosed in '[' and ']'. - max_pool_size option: previously pymongo had a hard coded pool size of 10 connections. With this change you can specify a different pool size as a parameter to :class:`~pymongo.connection.Connection` (max_pool_size=) or in the MongoDB URI (maxPoolSize=). - Find by metadata in GridFS: You can know specify query fields as keyword parameters for :meth:`~gridfs.GridFS.get_version` and :meth:`~gridfs.GridFS.get_last_version`. - Per-query slave_okay option: slave_okay=True is now a valid keyword argument for :meth:`~pymongo.collection.Collection.find` and :meth:`~pymongo.collection.Collection.find_one`. API changes: - :meth:`~pymongo.database.Database.validate_collection` now returns a dict instead of a string. This change was required to deal with an API change on the server. This method also now takes the optional `scandata` and `full` parameters. See the documentation for more details. .. warning:: The `pool_size`, `auto_start_request`, and `timeout` parameters for :class:`~pymongo.connection.Connection` have been completely removed in this release. They were deprecated in pymongo-1.4 and have had no effect since then. Please make sure that your code doesn't currently pass these parameters when creating a Connection instance. Issues resolved ............... - `PYTHON-241 `_: Support setting slaveok at the cursor level. - `PYTHON-240 `_: Queries can sometimes permanently fail after a replica set fail over. - `PYTHON-238 `_: error after few million requests - `PYTHON-237 `_: Basic IPv6 support. - `PYTHON-236 `_: Restore option to specify pool size in Connection. - `PYTHON-212 `_: pymongo does not recover after stale config - `PYTHON-138 `_: Find method for GridFS Changes in Version 1.10.1 ------------------------- Version 1.10.1 is primarily a bugfix release. It fixes a regression in version 1.10 that broke pickling of ObjectIds. A number of other bugs have been fixed as well. There are two behavior changes to be aware of: - If a read slave raises :class:`~pymongo.errors.AutoReconnect` :class:`~pymongo.master_slave_connection.MasterSlaveConnection` will now retry the query on each slave until it is successful or all slaves have raised :class:`~pymongo.errors.AutoReconnect`. Any other exception will immediately be raised. The order that the slaves are tried is random. Previously the read would be sent to one randomly chosen slave and :class:`~pymongo.errors.AutoReconnect` was immediately raised in case of a connection failure. - A Python `long` is now always BSON encoded as an int64. Previously the encoding was based only on the value of the field and a `long` with a value less than `2147483648` or greater than `-2147483649` would always be BSON encoded as an int32. Issues resolved ............... - `PYTHON-234 `_: Fix setup.py to raise exception if any when building extensions - `PYTHON-233 `_: Add information to build and test with extensions on windows - `PYTHON-232 `_: Traceback when hashing a DBRef instance - `PYTHON-231 `_: Traceback when pickling a DBRef instance - `PYTHON-230 `_: Pickled ObjectIds are not compatible between pymongo 1.9 and 1.10 - `PYTHON-228 `_: Cannot pickle bson.ObjectId - `PYTHON-227 `_: Traceback when calling find() on system.js - `PYTHON-216 `_: MasterSlaveConnection is missing disconnect() method - `PYTHON-186 `_: When storing integers, type is selected according to value instead of type - `PYTHON-173 `_: as_class option is not propogated by Cursor.clone - `PYTHON-113 `_: Redunducy in MasterSlaveConnection Changes in Version 1.10 ----------------------- Version 1.10 includes changes to support new features in MongoDB 1.8.x. Highlights include a modified map/reduce API including an inline map/reduce helper method, a new find_and_modify helper, and the ability to query the server for the maximum BSON document size it supports. - added :meth:`~pymongo.collection.Collection.find_and_modify`. - added :meth:`~pymongo.collection.Collection.inline_map_reduce`. - changed :meth:`~pymongo.collection.Collection.map_reduce`. .. warning:: MongoDB versions greater than 1.7.4 no longer generate temporary collections for map/reduce results. An output collection name must be provided and the output will replace any existing output collection with the same name. :meth:`~pymongo.collection.Collection.map_reduce` now requires the `out` parameter. Issues resolved ............... - PYTHON-225: :class:`~pymongo.objectid.ObjectId` class definition should use __slots__. - PYTHON-223: Documentation fix. - PYTHON-220: Documentation fix. - PYTHON-219: KeyError in :meth:`~pymongo.collection.Collection.find_and_modify` - PYTHON-213: Query server for maximum BSON document size. - PYTHON-208: Fix :class:`~pymongo.connection.Connection` __repr__. - PYTHON-207: Changes to Map/Reduce API. - PYTHON-205: Accept slaveOk in the URI to match the URI docs. - PYTHON-203: When slave_okay=True and we only specify one host don't autodetect other set members. - PYTHON-194: Show size when whining about a document being too large. - PYTHON-184: Raise :class:`~pymongo.errors.DuplicateKeyError` for duplicate keys in capped collections. - PYTHON-178: Don't segfault when trying to encode a recursive data structure. - PYTHON-177: Don't segfault when decoding dicts with broken iterators. - PYTHON-172: Fix a typo. - PYTHON-170: Add :meth:`~pymongo.collection.Collection.find_and_modify`. - PYTHON-169: Support deepcopy of DBRef. - PYTHON-167: Duplicate of PYTHON-166. - PYTHON-166: Fixes a concurrency issue. - PYTHON-158: Add code and err string to `db assertion` messages. Changes in Version 1.9 ---------------------- Version 1.9 adds a new package to the PyMongo distribution, :mod:`bson`. :mod:`bson` contains all of the `BSON `_ encoding and decoding logic, and the BSON types that were formerly in the :mod:`pymongo` package. The following modules have been renamed: - :mod:`pymongo.bson` -> :mod:`bson` - :mod:`pymongo._cbson` -> :mod:`bson._cbson` and :mod:`pymongo._cmessage` - :mod:`pymongo.binary` -> :mod:`bson.binary` - :mod:`pymongo.code` -> :mod:`bson.code` - :mod:`pymongo.dbref` -> :mod:`bson.dbref` - :mod:`pymongo.json_util` -> :mod:`bson.json_util` - :mod:`pymongo.max_key` -> :mod:`bson.max_key` - :mod:`pymongo.min_key` -> :mod:`bson.min_key` - :mod:`pymongo.objectid` -> :mod:`bson.objectid` - :mod:`pymongo.son` -> :mod:`bson.son` - :mod:`pymongo.timestamp` -> :mod:`bson.timestamp` - :mod:`pymongo.tz_util` -> :mod:`bson.tz_util` In addition, the following exception classes have been renamed: - :class:`pymongo.errors.InvalidBSON` -> :class:`bson.errors.InvalidBSON` - :class:`pymongo.errors.InvalidStringData` -> :class:`bson.errors.InvalidStringData` - :class:`pymongo.errors.InvalidDocument` -> :class:`bson.errors.InvalidDocument` - :class:`pymongo.errors.InvalidId` -> :class:`bson.errors.InvalidId` The above exceptions now inherit from :class:`bson.errors.BSONError` rather than :class:`pymongo.errors.PyMongoError`. .. note:: All of the renamed modules and exceptions above have aliases created with the old names, so these changes should not break existing code. The old names will eventually be deprecated and then removed, so users should begin migrating towards the new names now. .. warning:: The change to the exception hierarchy mentioned above is possibly breaking. If your code is catching :class:`~pymongo.errors.PyMongoError`, then the exceptions raised by :mod:`bson` will not be caught, even though they would have been caught previously. Before upgrading, it is recommended that users check for any cases like this. - the C extension now shares buffer.c/h with the Ruby driver - :mod:`bson` no longer raises :class:`~pymongo.errors.InvalidName`, all occurrences have been replaced with :class:`~bson.errors.InvalidDocument`. - renamed :meth:`bson._to_dicts` to :meth:`~bson.decode_all`. - renamed :meth:`~bson.BSON.from_dict` to :meth:`~bson.BSON.encode` and :meth:`~bson.BSON.to_dict` to :meth:`~bson.BSON.decode`. - added :meth:`~pymongo.cursor.Cursor.batch_size`. - allow updating (some) file metadata after a :class:`~gridfs.grid_file.GridIn` instance has been closed. - performance improvements for reading from GridFS. - special cased slice with the same start and stop to return an empty cursor. - allow writing :class:`unicode` to GridFS if an :attr:`encoding` attribute has been specified for the file. - added :meth:`gridfs.GridFS.get_version`. - scope variables for :class:`~bson.code.Code` can now be specified as keyword arguments. - added :meth:`~gridfs.grid_file.GridOut.readline` to :class:`~gridfs.grid_file.GridOut`. - make a best effort to transparently auto-reconnect if a :class:`~pymongo.connection.Connection` has been idle for a while. - added :meth:`~pymongo.database.SystemJS.list` to :class:`~pymongo.database.SystemJS`. - added `file_document` argument to :meth:`~gridfs.grid_file.GridOut` to allow initializing from an existing file document. - raise :class:`~pymongo.errors.TimeoutError` even if the ``getLastError`` command was run manually and not through "safe" mode. - added :class:`uuid` support to :mod:`~bson.json_util`. Changes in Version 1.8.1 ------------------------ - fixed a typo in the C extension that could cause safe-mode operations to report a failure (:class:`SystemError`) even when none occurred. - added a :meth:`__ne__` implementation to any class where we define :meth:`__eq__`. Changes in Version 1.8 ---------------------- Version 1.8 adds support for connecting to replica sets, specifying per-operation values for `w` and `wtimeout`, and decoding to timezone-aware datetimes. - fixed a reference leak in the C extension when decoding a :class:`~bson.dbref.DBRef`. - added support for `w`, `wtimeout`, and `fsync` (and any other options for `getLastError`) to "safe mode" operations. - added :attr:`~pymongo.connection.Connection.nodes` property. - added a maximum pool size of 10 sockets. - added support for replica sets. - DEPRECATED :meth:`~pymongo.connection.Connection.from_uri` and :meth:`~pymongo.connection.Connection.paired`, both are supplanted by extended functionality in :meth:`~pymongo.connection.Connection`. - added tz aware support for datetimes in :class:`~bson.objectid.ObjectId`, :class:`~bson.timestamp.Timestamp` and :mod:`~bson.json_util` methods. - added :meth:`~pymongo.collection.Collection.drop` helper. - reuse the socket used for finding the master when a :class:`~pymongo.connection.Connection` is first created. - added support for :class:`~bson.min_key.MinKey`, :class:`~bson.max_key.MaxKey` and :class:`~bson.timestamp.Timestamp` to :mod:`~bson.json_util`. - added support for decoding datetimes as aware (UTC) - it is highly recommended to enable this by setting the `tz_aware` parameter to :meth:`~pymongo.connection.Connection` to ``True``. - added `network_timeout` option for individual calls to :meth:`~pymongo.collection.Collection.find` and :meth:`~pymongo.collection.Collection.find_one`. - added :meth:`~gridfs.GridFS.exists` to check if a file exists in GridFS. - added support for additional keys in :class:`~bson.dbref.DBRef` instances. - added :attr:`~pymongo.errors.OperationFailure.code` attribute to :class:`~pymongo.errors.OperationFailure` exceptions. - fixed serialization of int and float subclasses in the C extension. Changes in Version 1.7 ---------------------- Version 1.7 is a recommended upgrade for all PyMongo users. The full release notes are below, and some more in depth discussion of the highlights is `here `_. - no longer attempt to build the C extension on big-endian systems. - added :class:`~bson.min_key.MinKey` and :class:`~bson.max_key.MaxKey`. - use unsigned for :class:`~bson.timestamp.Timestamp` in BSON encoder/decoder. - support ``True`` as ``"ok"`` in command responses, in addition to ``1.0`` - necessary for server versions **>= 1.5.X** - BREAKING change to :meth:`~pymongo.collection.Collection.index_information` to add support for querying unique status and other index information. - added :attr:`~pymongo.connection.Connection.document_class`, to specify class for returned documents. - added `as_class` argument for :meth:`~pymongo.collection.Collection.find`, and in the BSON decoder. - added support for creating :class:`~bson.timestamp.Timestamp` instances using a :class:`~datetime.datetime`. - allow `dropTarget` argument for :class:`~pymongo.collection.Collection.rename`. - handle aware :class:`~datetime.datetime` instances, by converting to UTC. - added support for :class:`~pymongo.cursor.Cursor.max_scan`. - raise :class:`~gridfs.errors.FileExists` exception when creating a duplicate GridFS file. - use `y2038 `_ for time handling in the C extension - eliminates 2038 problems when extension is installed. - added `sort` parameter to :meth:`~pymongo.collection.Collection.find` - finalized deprecation of changes from versions **<= 1.4** - take any non-:class:`dict` as an ``"_id"`` query for :meth:`~pymongo.collection.Collection.find_one` or :meth:`~pymongo.collection.Collection.remove` - added ability to pass a :class:`dict` for `fields` argument to :meth:`~pymongo.collection.Collection.find` (supports ``"$slice"`` and field negation) - simplified code to find master, since paired setups don't always have a remote - fixed bug in C encoder for certain invalid types (like :class:`~pymongo.collection.Collection` instances). - don't transparently map ``"filename"`` key to :attr:`name` attribute for GridFS. Changes in Version 1.6 ---------------------- The biggest change in version 1.6 is a complete re-implementation of :mod:`gridfs` with a lot of improvements over the old implementation. There are many details and examples of using the new API in `this blog post `_. The old API has been removed in this version, so existing code will need to be modified before upgrading to 1.6. - fixed issue where connection pool was being shared across :class:`~pymongo.connection.Connection` instances. - more improvements to Python code caching in C extension - should improve behavior on mod_wsgi. - added :meth:`~bson.objectid.ObjectId.from_datetime`. - complete rewrite of :mod:`gridfs` support. - improvements to the :meth:`~pymongo.database.Database.command` API. - fixed :meth:`~pymongo.collection.Collection.drop_indexes` behavior on non-existent collections. - disallow empty bulk inserts. Changes in Version 1.5.2 ------------------------ - fixed response handling to ignore unknown response flags in queries. - handle server versions containing '-pre-'. Changes in Version 1.5.1 ------------------------ - added :data:`~gridfs.grid_file.GridFile._id` property for :class:`~gridfs.grid_file.GridFile` instances. - fix for making a :class:`~pymongo.connection.Connection` (with `slave_okay` set) directly to a slave in a replica pair. - accept kwargs for :meth:`~pymongo.collection.Collection.create_index` and :meth:`~pymongo.collection.Collection.ensure_index` to support all indexing options. - add :data:`pymongo.GEO2D` and support for geo indexing. - improvements to Python code caching in C extension - should improve behavior on mod_wsgi. Changes in Version 1.5 ---------------------- - added subtype constants to :mod:`~bson.binary` module. - DEPRECATED `options` argument to :meth:`~pymongo.collection.Collection` and :meth:`~pymongo.database.Database.create_collection` in favor of kwargs. - added :meth:`~pymongo.has_c` to check for C extension. - added :meth:`~pymongo.connection.Connection.copy_database`. - added :data:`~pymongo.cursor.Cursor.alive` to tell when a cursor might have more data to return (useful for tailable cursors). - added :class:`~bson.timestamp.Timestamp` to better support dealing with internal MongoDB timestamps. - added `name` argument for :meth:`~pymongo.collection.Collection.create_index` and :meth:`~pymongo.collection.Collection.ensure_index`. - fixed connection pooling w/ fork - :meth:`~pymongo.connection.Connection.paired` takes all kwargs that are allowed for :meth:`~pymongo.connection.Connection`. - :meth:`~pymongo.collection.Collection.insert` returns list for bulk inserts of size one. - fixed handling of :class:`datetime.datetime` instances in :mod:`~bson.json_util`. - added :meth:`~pymongo.connection.Connection.from_uri` to support MongoDB connection uri scheme. - fixed chunk number calculation when unaligned in :mod:`gridfs`. - :meth:`~pymongo.database.Database.command` takes a string for simple commands. - added :data:`~pymongo.database.Database.system_js` helper for dealing with server-side JS. - don't wrap queries containing ``"$query"`` (support manual use of ``"$min"``, etc.). - added :class:`~gridfs.errors.GridFSError` as base class for :mod:`gridfs` exceptions. Changes in Version 1.4 ---------------------- Perhaps the most important change in version 1.4 is that we have decided to **no longer support Python 2.3**. The most immediate reason for this is to allow some improvements to connection pooling. This will also allow us to use some new (as in Python 2.4 ;) idioms and will help begin the path towards supporting Python 3.0. If you need to use Python 2.3 you should consider using version 1.3 of this driver, although that will no longer be actively supported. Other changes: - move ``"_id"`` to front only for top-level documents (fixes some corner cases). - :meth:`~pymongo.collection.Collection.update` and :meth:`~pymongo.collection.Collection.remove` return the entire response to the *lastError* command when safe is ``True``. - completed removal of things that were deprecated in version 1.2 or earlier. - enforce that collection names do not contain the NULL byte. - fix to allow using UTF-8 collection names with the C extension. - added :class:`~pymongo.errors.PyMongoError` as base exception class for all :mod:`~pymongo.errors`. this changes the exception hierarchy somewhat, and is a BREAKING change if you depend on :class:`~pymongo.errors.ConnectionFailure` being a :class:`IOError` or :class:`~bson.errors.InvalidBSON` being a :class:`ValueError`, for example. - added :class:`~pymongo.errors.DuplicateKeyError` for calls to :meth:`~pymongo.collection.Collection.insert` or :meth:`~pymongo.collection.Collection.update` with `safe` set to ``True``. - removed :mod:`~pymongo.thread_util`. - added :meth:`~pymongo.database.Database.add_user` and :meth:`~pymongo.database.Database.remove_user` helpers. - fix for :meth:`~pymongo.database.Database.authenticate` when using non-UTF-8 names or passwords. - minor fixes for :class:`~pymongo.master_slave_connection.MasterSlaveConnection`. - clean up all cases where :class:`~pymongo.errors.ConnectionFailure` is raised. - simplification of connection pooling - makes driver ~2x faster for simple benchmarks. see :ref:`connection-pooling` for more information. - DEPRECATED `pool_size`, `auto_start_request` and `timeout` parameters to :class:`~pymongo.connection.Connection`. DEPRECATED :meth:`~pymongo.connection.Connection.start_request`. - use :meth:`socket.sendall`. - removed :meth:`~bson.son.SON.from_xml` as it was only being used for some internal testing - also eliminates dependency on :mod:`elementtree`. - implementation of :meth:`~pymongo.message.update` in C. - deprecate :meth:`~pymongo.database.Database._command` in favor of :meth:`~pymongo.database.Database.command`. - send all commands without wrapping as ``{"query": ...}``. - support string as `key` argument to :meth:`~pymongo.collection.Collection.group` (keyf) and run all groups as commands. - support for equality testing for :class:`~bson.code.Code` instances. - allow the NULL byte in strings and disallow it in key names or regex patterns Changes in Version 1.3 ---------------------- - DEPRECATED running :meth:`~pymongo.collection.Collection.group` as :meth:`~pymongo.database.Database.eval`, also changed default for :meth:`~pymongo.collection.Collection.group` to running as a command - remove :meth:`pymongo.cursor.Cursor.__len__`, which was deprecated in 1.1.1 - needed to do this aggressively due to it's presence breaking **Django** template *for* loops - DEPRECATED :meth:`~pymongo.connection.Connection.host`, :meth:`~pymongo.connection.Connection.port`, :meth:`~pymongo.database.Database.connection`, :meth:`~pymongo.database.Database.name`, :meth:`~pymongo.collection.Collection.database`, :meth:`~pymongo.collection.Collection.name` and :meth:`~pymongo.collection.Collection.full_name` in favor of :attr:`~pymongo.connection.Connection.host`, :attr:`~pymongo.connection.Connection.port`, :attr:`~pymongo.database.Database.connection`, :attr:`~pymongo.database.Database.name`, :attr:`~pymongo.collection.Collection.database`, :attr:`~pymongo.collection.Collection.name` and :attr:`~pymongo.collection.Collection.full_name`, respectively. The deprecation schedule for this change will probably be faster than usual, as it carries some performance implications. - added :meth:`~pymongo.connection.Connection.disconnect` Changes in Version 1.2.1 ------------------------ - added :doc:`changelog` to docs - added ``setup.py doc --test`` to run doctests for tutorial, examples - moved most examples to Sphinx docs (and remove from *examples/* directory) - raise :class:`~bson.errors.InvalidId` instead of :class:`TypeError` when passing a 24 character string to :class:`~bson.objectid.ObjectId` that contains non-hexadecimal characters - allow :class:`unicode` instances for :class:`~bson.objectid.ObjectId` init Changes in Version 1.2 ---------------------- - `spec` parameter for :meth:`~pymongo.collection.Collection.remove` is now optional to allow for deleting all documents in a :class:`~pymongo.collection.Collection` - always wrap queries with ``{query: ...}`` even when no special options - get around some issues with queries on fields named ``query`` - enforce 4MB document limit on the client side - added :meth:`~pymongo.collection.Collection.map_reduce` helper - see :doc:`example ` - added :meth:`~pymongo.cursor.Cursor.distinct` method on :class:`~pymongo.cursor.Cursor` instances to allow distinct with queries - fix for :meth:`~pymongo.cursor.Cursor.__getitem__` after :meth:`~pymongo.cursor.Cursor.skip` - allow any UTF-8 string in :class:`~bson.BSON` encoder, not just ASCII subset - added :attr:`~bson.objectid.ObjectId.generation_time` - removed support for legacy :class:`~bson.objectid.ObjectId` format - pretty sure this was never used, and is just confusing - DEPRECATED :meth:`~bson.objectid.ObjectId.url_encode` and :meth:`~bson.objectid.ObjectId.url_decode` in favor of :meth:`str` and :meth:`~bson.objectid.ObjectId`, respectively - allow *oplog.$main* as a valid collection name - some minor fixes for installation process - added support for datetime and regex in :mod:`~bson.json_util` Changes in Version 1.1.2 ------------------------ - improvements to :meth:`~pymongo.collection.Collection.insert` speed (using C for insert message creation) - use random number for request_id - fix some race conditions with :class:`~pymongo.errors.AutoReconnect` Changes in Version 1.1.1 ------------------------ - added `multi` parameter for :meth:`~pymongo.collection.Collection.update` - fix unicode regex patterns with C extension - added :meth:`~pymongo.collection.Collection.distinct` - added `database` support for :class:`~bson.dbref.DBRef` - added :mod:`~bson.json_util` with helpers for encoding / decoding special types to JSON - DEPRECATED :meth:`pymongo.cursor.Cursor.__len__` in favor of :meth:`~pymongo.cursor.Cursor.count` with `with_limit_and_skip` set to ``True`` due to performance regression - switch documentation to Sphinx Changes in Version 1.1 ---------------------- - added :meth:`__hash__` for :class:`~bson.dbref.DBRef` and :class:`~bson.objectid.ObjectId` - bulk :meth:`~pymongo.collection.Collection.insert` works with any iterable - fix :class:`~bson.objectid.ObjectId` generation when using :mod:`multiprocessing` - added :attr:`~pymongo.cursor.Cursor.collection` - added `network_timeout` parameter for :meth:`~pymongo.connection.Connection` - DEPRECATED `slave_okay` parameter for individual queries - fix for `safe` mode when multi-threaded - added `safe` parameter for :meth:`~pymongo.collection.Collection.remove` - added `tailable` parameter for :meth:`~pymongo.collection.Collection.find` Changes in Version 1.0 ---------------------- - fixes for :class:`~pymongo.master_slave_connection.MasterSlaveConnection` - added `finalize` parameter for :meth:`~pymongo.collection.Collection.group` - improvements to :meth:`~pymongo.collection.Collection.insert` speed - improvements to :mod:`gridfs` speed - added :meth:`~pymongo.cursor.Cursor.__getitem__` and :meth:`~pymongo.cursor.Cursor.__len__` for :class:`~pymongo.cursor.Cursor` instances Changes in Version 0.16 ----------------------- - support for encoding/decoding :class:`uuid.UUID` instances - fix for :meth:`~pymongo.cursor.Cursor.explain` with limits Changes in Version 0.15.2 ------------------------- - documentation changes only Changes in Version 0.15.1 ------------------------- - various performance improvements - API CHANGE no longer need to specify direction for :meth:`~pymongo.collection.Collection.create_index` and :meth:`~pymongo.collection.Collection.ensure_index` when indexing a single key - support for encoding :class:`tuple` instances as :class:`list` instances Changes in Version 0.15 ----------------------- - fix string representation of :class:`~bson.objectid.ObjectId` instances - added `timeout` parameter for :meth:`~pymongo.collection.Collection.find` - allow scope for `reduce` function in :meth:`~pymongo.collection.Collection.group` Changes in Version 0.14.2 ------------------------- - minor bugfixes Changes in Version 0.14.1 ------------------------- - :meth:`~gridfs.grid_file.GridFile.seek` and :meth:`~gridfs.grid_file.GridFile.tell` for (read mode) :class:`~gridfs.grid_file.GridFile` instances Changes in Version 0.14 ----------------------- - support for long in :class:`~bson.BSON` - added :meth:`~pymongo.collection.Collection.rename` - added `snapshot` parameter for :meth:`~pymongo.collection.Collection.find` Changes in Version 0.13 ----------------------- - better :class:`~pymongo.master_slave_connection.MasterSlaveConnection` support - API CHANGE :meth:`~pymongo.collection.Collection.insert` and :meth:`~pymongo.collection.Collection.save` both return inserted ``_id`` - DEPRECATED passing an index name to :meth:`~pymongo.cursor.Cursor.hint` Changes in Version 0.12 ----------------------- - improved :class:`~bson.objectid.ObjectId` generation - added :class:`~pymongo.errors.AutoReconnect` exception for when reconnection is possible - make :mod:`gridfs` thread-safe - fix for :mod:`gridfs` with non :class:`~bson.objectid.ObjectId` ``_id`` Changes in Version 0.11.3 ------------------------- - don't allow NULL bytes in string encoder - fixes for Python 2.3 Changes in Version 0.11.2 ------------------------- - PEP 8 - updates for :meth:`~pymongo.collection.Collection.group` - VS build Changes in Version 0.11.1 ------------------------- - fix for connection pooling under Python 2.5 Changes in Version 0.11 ----------------------- - better build failure detection - driver support for selecting fields in sub-documents - disallow insertion of invalid key names - added `timeout` parameter for :meth:`~pymongo.connection.Connection` Changes in Version 0.10.3 ------------------------- - fix bug with large :meth:`~pymongo.cursor.Cursor.limit` - better exception when modules get reloaded out from underneath the C extension - better exception messages when calling a :class:`~pymongo.collection.Collection` or :class:`~pymongo.database.Database` instance Changes in Version 0.10.2 ------------------------- - support subclasses of :class:`dict` in C encoder Changes in Version 0.10.1 ------------------------- - alias :class:`~pymongo.connection.Connection` as :attr:`pymongo.Connection` - raise an exception rather than silently overflowing in encoder Changes in Version 0.10 ----------------------- - added :meth:`~pymongo.collection.Collection.ensure_index` Changes in Version 0.9.7 ------------------------ - allow sub-collections of *$cmd* as valid :class:`~pymongo.collection.Collection` names - add version as :attr:`pymongo.version` - add ``--no_ext`` command line option to *setup.py* .. toctree:: :hidden: python3 examples/gevent examples/requests pymongo-2.6.3/doc/conf.py000066400000000000000000000117231223300253600152560ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # PyMongo documentation build configuration file # # This file is execfile()d with the current directory set to its containing dir. import sys, os sys.path[0:0] = [os.path.abspath('..')] import pymongo # -- General configuration ----------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.coverage', 'sphinx.ext.todo', 'doc.mongo_extensions'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The master toctree document. master_doc = 'index' # General information about the project. project = u'PyMongo' copyright = u'2008 - 2012, 10gen, Inc.' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. version = pymongo.version # The full version, including alpha/beta/rc tags. release = pymongo.version # List of documents that shouldn't be included in the build. unused_docs = [] # List of directories, relative to source directory, that shouldn't be searched # for source files. exclude_trees = ['_build'] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # If true, the current module name will be prepended to all description # unit titles (such as .. function::). add_module_names = True # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # -- Options for extensions ---------------------------------------------------- autoclass_content = 'init' doctest_path = os.path.abspath('..') doctest_test_doctest_blocks = False doctest_global_setup = """ from pymongo.mongo_client import MongoClient client = MongoClient() client.drop_database("doctest_test") db = client.doctest_test """ # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. Major themes that come with # Sphinx are currently 'default' and 'sphinxdoc'. html_theme = 'default' html_theme_options = {'collapsiblesidebar': True} # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". #html_static_path = ['_static'] # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = '' # Output file base name for HTML help builder. htmlhelp_basename = 'PyMongo' + release.replace('.', '_') # -- Options for LaTeX output -------------------------------------------------- # The paper size ('letter' or 'a4'). #latex_paper_size = 'letter' # The font size ('10pt', '11pt' or '12pt'). #latex_font_size = '10pt' # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ ('index', 'PyMongo.tex', u'PyMongo Documentation', u'Michael Dirolf', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # Additional stuff for the LaTeX preamble. #latex_preamble = '' # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_use_modindex = True pymongo-2.6.3/doc/contributors.rst000066400000000000000000000035251223300253600172470ustar00rootroot00000000000000Contributors ============ The following is a list of people who have contributed to **PyMongo**. If you belong here and are missing please let us know (or send a pull request after adding yourself to the list): - Mike Dirolf (mdirolf) - Jeff Jenkins (jeffjenkins) - Jim Jones - Eliot Horowitz (erh) - Michael Stephens (mikejs) - Joakim Sernbrant (serbaut) - Alexander Artemenko (svetlyak40wt) - Mathias Stearn (RedBeard0531) - Fajran Iman Rusadi (fajran) - Brad Clements (bkc) - Andrey Fedorov (andreyf) - Joshua Roesslein (joshthecoder) - Gregg Lind (gregglind) - Michael Schurter (schmichael) - Daniel Lundin - Michael Richardson (mtrichardson) - Dan McKinley (mcfunley) - David Wolever (wolever) - Carlos Valiente (carletes) - Jehiah Czebotar (jehiah) - Drew Perttula (drewp) - Carl Baatz (c-w-b) - Johan Bergstrom (jbergstroem) - Jonas Haag (jonashaag) - Kristina Chodorow (kchodorow) - Andrew Sibley (sibsibsib) - Flavio Percoco Premoli (FlaPer87) - Ken Kurzweil (kurzweil) - Christian Wyglendowski (dowski) - James Murty (jmurty) - Brendan W. McAdams (bwmcadams) - Bernie Hackett (behackett) - Reed O'Brien (reedobrien) - Francisco Souza (fsouza) - Alexey I. Froloff (raorn) - Steve Lacy (slacy) - Richard Shea (shearic) - Vladimir Sidorenko (gearheart) - Aaron Westendorf (awestendorf) - Dan Crosta (dcrosta) - Ryan Smith-Roberts (rmsr) - David Pisoni (gefilte) - Abhay Vardhan (abhayv) - Alexey Borzenkov (snaury) - Kostya Rybnikov (k-bx) - A Jesse Jiryu Davis (ajdavis) - Samuel Clay (samuelclay) - Ross Lawley (rozza) - Wouter Bolsterlee (wbolster) - Alex Grönholm (agronholm) - Christoph Simon (kalanzun) - Chris Tompkinson (tompko) - Mike O'Brien (mpobrien) - T Dampier (dampier) - Michael Henson (hensom) - Craig Hobbs (craigahobbs) - Emily Stolfo (estolfo) - Sam Helman (shelman) - Justin Patrin (reversefold) - Xiuming Chen (cxmcc) - Tyler Jones (thomascirca) pymongo-2.6.3/doc/examples/000077500000000000000000000000001223300253600155715ustar00rootroot00000000000000pymongo-2.6.3/doc/examples/aggregation.rst000066400000000000000000000146721223300253600206240ustar00rootroot00000000000000Aggregation Examples ==================== There are several methods of performing aggregations in MongoDB. These examples cover the new aggregation framework, using map reduce and using the group method. .. testsetup:: from pymongo import MongoClient client = MongoClient() client.drop_database('aggregation_example') Setup ----- To start, we'll insert some example data which we can perform aggregations on: .. doctest:: >>> from pymongo import MongoClient >>> db = MongoClient().aggregation_example >>> db.things.insert({"x": 1, "tags": ["dog", "cat"]}) ObjectId('...') >>> db.things.insert({"x": 2, "tags": ["cat"]}) ObjectId('...') >>> db.things.insert({"x": 2, "tags": ["mouse", "cat", "dog"]}) ObjectId('...') >>> db.things.insert({"x": 3, "tags": []}) ObjectId('...') Aggregation Framework --------------------- This example shows how to use the :meth:`~pymongo.collection.Collection.aggregate` method to use the aggregation framework. We'll perform a simple aggregation to count the number of occurrences for each tag in the ``tags`` array, across the entire collection. To achieve this we need to pass in three operations to the pipeline. First, we need to unwind the ``tags`` array, then group by the tags and sum them up, finally we sort by count. As python dictionaries don't maintain order you should use :class:`~bson.son.SON` or :class:`collections.OrderedDict` where explicit ordering is required eg "$sort": .. note:: aggregate requires server version **>= 2.1.0**. The PyMongo :meth:`~pymongo.collection.Collection.aggregate` helper requires PyMongo version **>= 2.3**. .. doctest:: >>> from bson.son import SON >>> db.things.aggregate([ ... {"$unwind": "$tags"}, ... {"$group": {"_id": "$tags", "count": {"$sum": 1}}}, ... {"$sort": SON([("count", -1), ("_id", -1)])} ... ]) ... {u'ok': 1.0, u'result': [{u'count': 3, u'_id': u'cat'}, {u'count': 2, u'_id': u'dog'}, {u'count': 1, u'_id': u'mouse'}]} As well as simple aggregations the aggregation framework provides projection capabilities to reshape the returned data. Using projections and aggregation, you can add computed fields, create new virtual sub-objects, and extract sub-fields into the top-level of results. .. seealso:: The full documentation for MongoDB's `aggregation framework `_ Map/Reduce ---------- Another option for aggregation is to use the map reduce framework. Here we will define **map** and **reduce** functions to also count he number of occurrences for each tag in the ``tags`` array, across the entire collection. Our **map** function just emits a single `(key, 1)` pair for each tag in the array: .. doctest:: >>> from bson.code import Code >>> mapper = Code(""" ... function () { ... this.tags.forEach(function(z) { ... emit(z, 1); ... }); ... } ... """) The **reduce** function sums over all of the emitted values for a given key: .. doctest:: >>> reducer = Code(""" ... function (key, values) { ... var total = 0; ... for (var i = 0; i < values.length; i++) { ... total += values[i]; ... } ... return total; ... } ... """) .. note:: We can't just return ``values.length`` as the **reduce** function might be called iteratively on the results of other reduce steps. Finally, we call :meth:`~pymongo.collection.Collection.map_reduce` and iterate over the result collection: .. doctest:: >>> result = db.things.map_reduce(mapper, reducer, "myresults") >>> for doc in result.find(): ... print doc ... {u'_id': u'cat', u'value': 3.0} {u'_id': u'dog', u'value': 2.0} {u'_id': u'mouse', u'value': 1.0} Advanced Map/Reduce ------------------- PyMongo's API supports all of the features of MongoDB's map/reduce engine. One interesting feature is the ability to get more detailed results when desired, by passing `full_response=True` to :meth:`~pymongo.collection.Collection.map_reduce`. This returns the full response to the map/reduce command, rather than just the result collection: .. doctest:: >>> db.things.map_reduce(mapper, reducer, "myresults", full_response=True) {u'counts': {u'input': 4, u'reduce': 2, u'emit': 6, u'output': 3}, u'timeMillis': ..., u'ok': ..., u'result': u'...'} All of the optional map/reduce parameters are also supported, simply pass them as keyword arguments. In this example we use the `query` parameter to limit the documents that will be mapped over: .. doctest:: >>> result = db.things.map_reduce(mapper, reducer, "myresults", query={"x": {"$lt": 2}}) >>> for doc in result.find(): ... print doc ... {u'_id': u'cat', u'value': 1.0} {u'_id': u'dog', u'value': 1.0} With MongoDB 1.8.0 or newer you can use :class:`~bson.son.SON` or :class:`collections.OrderedDict` to specify a different database to store the result collection: .. doctest:: >>> from bson.son import SON >>> db.things.map_reduce(mapper, reducer, out=SON([("replace", "results"), ("db", "outdb")]), full_response=True) {u'counts': {u'input': 4, u'reduce': 2, u'emit': 6, u'output': 3}, u'timeMillis': ..., u'ok': ..., u'result': {u'db': ..., u'collection': ...}} .. seealso:: The full list of options for MongoDB's `map reduce engine `_ Group ----- The :meth:`~pymongo.collection.Collection.group` method provides some of the same functionality as SQL's GROUP BY. Simpler than a map reduce you need to provide a key to group by, an initial value for the aggregation and a reduce function. .. note:: Doesn't work with sharded MongoDB configurations, use aggregation or map/reduce instead of group(). Here we are doing a simple group and count of the occurrences ``x`` values: .. doctest:: >>> reducer = Code(""" ... function(obj, prev){ ... prev.count++; ... } ... """) ... >>> from bson.son import SON >>> results = db.things.group(key={"x":1}, condition={}, initial={"count": 0}, reduce=reducer) >>> for doc in results: ... print doc {u'count': 1.0, u'x': 1.0} {u'count': 2.0, u'x': 2.0} {u'count': 1.0, u'x': 3.0} .. seealso:: The full list of options for MongoDB's `group method `_ pymongo-2.6.3/doc/examples/authentication.rst000066400000000000000000000153551223300253600213530ustar00rootroot00000000000000Authentication Examples ======================= MongoDB supports several different authentication mechanisms. These examples cover all authentication methods currently supported by PyMongo, documenting Python module and MongoDB version dependencies. MONGODB-CR ---------- MONGODB-CR is the default authentication mechanism supported by a MongoDB cluster configured for authentication. Authentication is per-database and credentials can be specified through the MongoDB URI or passed to the :meth:`~pymongo.database.Database.authenticate` method:: >>> from pymongo import MongoClient >>> client = pymongo.MongoClient('example.com') >>> client.the_database.authenticate('user', 'password') True >>> >>> uri = "mongodb://user:password@example.com/the_database" >>> client = pymongo.MongoClient(uri) >>> When using MongoDB's delegated authentication features, a separate authentication source can be specified (using PyMongo 2.5 or newer):: >>> from pymongo import MongoClient >>> client = pymongo.MongoClient('example.com') >>> client.the_database.authenticate('user', ... 'password', ... source='source_database') True >>> >>> uri = "mongodb://user:password@example.com/?authSource=source_database" >>> client = pymongo.MongoClient(uri) >>> MONGODB-X509 ------------ .. versionadded:: 2.6 The MONGODB-X509 mechanism authenticates a username derived from the distinguished subject name of the X.509 certificate presented by the driver during SSL negotiation. This authentication method requires the use of SSL connections with certificate validation and is available in MongoDB 2.5.1 and newer:: >>> import ssl >>> from pymongo import MongoClient >>> client = pymongo.MongoClient('example.com', ... ssl=True, ... ssl_certfile='/path/to/client.pem', ... ssl_cert_reqs=ssl.CERT_REQUIRED, ... ssl_ca_certs='/path/to/ca.pem') >>> client.the_database.authenticate("", ... mechanism='MONGODB-X509') True >>> MONGODB-X509 authenticates against the $external virtual database, so you do not have to specify a database in the URI:: >>> uri = "mongodb://@example.com/?authMechanism=MONGODB-X509" >>> client = pymongo.MongoClient(uri, ... ssl=True, ... ssl_certfile='/path/to/client.pem', ... ssl_cert_reqs=ssl.CERT_REQUIRED, ... ssl_ca_certs='/path/to/ca.pem') >>> .. note:: If you are using CPython 2.4 or 2.5 you must install the python `ssl module`_ using easy_install or pip. .. _ssl module: https://pypi.python.org/pypi/ssl/ .. _use_kerberos: GSSAPI (Kerberos) ----------------- .. versionadded:: 2.5 GSSAPI (Kerberos) authentication is available in the Enterprise Edition of MongoDB, version 2.4 and newer. To authenticate using GSSAPI you must first install the python `kerberos module`_ using easy_install or pip. Make sure you run kinit before using the following authentication methods:: $ kinit mongodbuser@EXAMPLE.COM mongodbuser@EXAMPLE.COM's Password: $ klist Credentials cache: FILE:/tmp/krb5cc_1000 Principal: mongodbuser@EXAMPLE.COM Issued Expires Principal Feb 9 13:48:51 2013 Feb 9 23:48:51 2013 krbtgt/EXAMPLE.COM@EXAMPLE.COM Now authenticate using the MongoDB URI. GSSAPI authenticates against the $external virtual database so you do not have to specify a database in the URI:: >>> # Note: the kerberos principal must be url encoded. >>> import pymongo >>> uri = "mongodb://mongodbuser%40EXAMPLE.COM@example.com/?authMechanism=GSSAPI" >>> client = pymongo.MongoClient(uri) >>> or using :meth:`~pymongo.database.Database.authenticate`:: >>> import pymongo >>> client = pymongo.MongoClient('example.com') >>> db = client.test >>> db.authenticate('mongodbuser@EXAMPLE.COM', mechanism='GSSAPI') True The default service name used by MongoDB and PyMongo is `mongodb`. You can specify a custom service name with the ``gssapiServiceName`` option:: >>> import pymongo >>> uri = "mongodb://mongodbuser%40EXAMPLE.COM@example.com/?authMechanism=GSSAPI&gssapiServiceName=myservicename" >>> client = pymongo.MongoClient(uri) >>> >>> client = pymongo.MongoClient('example.com') >>> db = client.test >>> db.authenticate('mongodbuser@EXAMPLE.COM', mechanism='GSSAPI', gssapiServiceName='myservicename') True .. note:: Kerberos support is only provided in environments supported by the python `kerberos module`_. This currently limits support to CPython 2.x and Unix environments. .. _kerberos module: http://pypi.python.org/pypi/kerberos SASL PLAIN (RFC 4616) --------------------- .. versionadded:: 2.6 MongoDB Enterprise Edition versions 2.5.0 and newer support the SASL PLAIN authentication mechanism, initially intended for delegating authentication to an LDAP server. Using the PLAIN mechanism is very similar to MONGODB-CR. These examples use the $external virtual database for LDAP support:: >>> from pymongo import MongoClient >>> client = pymongo.MongoClient('example.com') >>> client.the_database.authenticate('user', ... 'password', ... source='$external', ... mechanism='PLAIN') True >>> >>> uri = "mongodb://user:password@example.com/?authMechanism=PLAIN&authSource=$external" >>> client = pymongo.MongoClient(uri) >>> SASL PLAIN is a clear-text authentication mechanism. We **strongly** recommend that you connect to MongoDB using SSL with certificate validation when using the SASL PLAIN mechanism:: >>> import ssl >>> from pymongo import MongoClient >>> client = pymongo.MongoClient('example.com', ... ssl=True, ... ssl_certfile='/path/to/client.pem', ... ssl_cert_reqs=ssl.CERT_REQUIRED, ... ssl_ca_certs='/path/to/ca.pem') >>> client.the_database.authenticate('user', ... 'password', ... source='$external', ... mechanism='PLAIN') True >>> >>> uri = "mongodb://user:password@example.com/?authMechanism=PLAIN&authSource=$external" >>> client = pymongo.MongoClient(uri, ... ssl=True, ... ssl_certfile='/path/to/client.pem', ... ssl_cert_reqs=ssl.CERT_REQUIRED, ... ssl_ca_certs='/path/to/ca.pem') >>> pymongo-2.6.3/doc/examples/custom_type.rst000066400000000000000000000151631223300253600207040ustar00rootroot00000000000000Custom Type Example =================== This is an example of using a custom type with PyMongo. The example here is a bit contrived, but shows how to use a :class:`~pymongo.son_manipulator.SONManipulator` to manipulate documents as they are saved or retrieved from MongoDB. More specifically, it shows a couple different mechanisms for working with custom datatypes in PyMongo. Setup ----- We'll start by getting a clean database to use for the example: .. doctest:: >>> from pymongo.mongo_client import MongoClient >>> client = MongoClient() >>> client.drop_database("custom_type_example") >>> db = client.custom_type_example Since the purpose of the example is to demonstrate working with custom types, we'll need a custom datatype to use. Here we define the aptly named :class:`Custom` class, which has a single method, :meth:`x`: .. doctest:: >>> class Custom(object): ... def __init__(self, x): ... self.__x = x ... ... def x(self): ... return self.__x ... >>> foo = Custom(10) >>> foo.x() 10 When we try to save an instance of :class:`Custom` with PyMongo, we'll get an :class:`~bson.errors.InvalidDocument` exception: .. doctest:: >>> db.test.insert({"custom": Custom(5)}) Traceback (most recent call last): InvalidDocument: cannot convert value of type to bson Manual Encoding --------------- One way to work around this is to manipulate our data into something we *can* save with PyMongo. To do so we define two methods, :meth:`encode_custom` and :meth:`decode_custom`: .. doctest:: >>> def encode_custom(custom): ... return {"_type": "custom", "x": custom.x()} ... >>> def decode_custom(document): ... assert document["_type"] == "custom" ... return Custom(document["x"]) ... We can now manually encode and decode :class:`Custom` instances and use them with PyMongo: .. doctest:: >>> db.test.insert({"custom": encode_custom(Custom(5))}) ObjectId('...') >>> db.test.find_one() {u'_id': ObjectId('...'), u'custom': {u'x': 5, u'_type': u'custom'}} >>> decode_custom(db.test.find_one()["custom"]) >>> decode_custom(db.test.find_one()["custom"]).x() 5 Automatic Encoding and Decoding ------------------------------- Needless to say, that was a little unwieldy. Let's make this a bit more seamless by creating a new :class:`~pymongo.son_manipulator.SONManipulator`. :class:`~pymongo.son_manipulator.SONManipulator` instances allow you to specify transformations to be applied automatically by PyMongo: .. doctest:: >>> from pymongo.son_manipulator import SONManipulator >>> class Transform(SONManipulator): ... def transform_incoming(self, son, collection): ... for (key, value) in son.items(): ... if isinstance(value, Custom): ... son[key] = encode_custom(value) ... elif isinstance(value, dict): # Make sure we recurse into sub-docs ... son[key] = self.transform_incoming(value, collection) ... return son ... ... def transform_outgoing(self, son, collection): ... for (key, value) in son.items(): ... if isinstance(value, dict): ... if "_type" in value and value["_type"] == "custom": ... son[key] = decode_custom(value) ... else: # Again, make sure to recurse into sub-docs ... son[key] = self.transform_outgoing(value, collection) ... return son ... Now we add our manipulator to the :class:`~pymongo.database.Database`: .. doctest:: >>> db.add_son_manipulator(Transform()) After doing so we can save and restore :class:`Custom` instances seamlessly: .. doctest:: >>> db.test.remove() # remove whatever has already been saved {...} >>> db.test.insert({"custom": Custom(5)}) ObjectId('...') >>> db.test.find_one() {u'_id': ObjectId('...'), u'custom': } >>> db.test.find_one()["custom"].x() 5 If we get a new :class:`~pymongo.database.Database` instance we'll clear out the :class:`~pymongo.son_manipulator.SONManipulator` instance we added: .. doctest:: >>> db = client.custom_type_example This allows us to see what was actually saved to the database: .. doctest:: >>> db.test.find_one() {u'_id': ObjectId('...'), u'custom': {u'x': 5, u'_type': u'custom'}} which is the same format that we encode to with our :meth:`encode_custom` method! Binary Encoding --------------- We can take this one step further by encoding to binary, using a user defined subtype. This allows us to identify what to decode without resorting to tricks like the ``_type`` field used above. We'll start by defining the methods :meth:`to_binary` and :meth:`from_binary`, which convert :class:`Custom` instances to and from :class:`~bson.binary.Binary` instances: .. note:: You could just pickle the instance and save that. What we do here is a little more lightweight. .. doctest:: >>> from bson.binary import Binary >>> def to_binary(custom): ... return Binary(str(custom.x()), 128) ... >>> def from_binary(binary): ... return Custom(int(binary)) ... Next we'll create another :class:`~pymongo.son_manipulator.SONManipulator`, this time using the methods we just defined: .. doctest:: >>> class TransformToBinary(SONManipulator): ... def transform_incoming(self, son, collection): ... for (key, value) in son.items(): ... if isinstance(value, Custom): ... son[key] = to_binary(value) ... elif isinstance(value, dict): ... son[key] = self.transform_incoming(value, collection) ... return son ... ... def transform_outgoing(self, son, collection): ... for (key, value) in son.items(): ... if isinstance(value, Binary) and value.subtype == 128: ... son[key] = from_binary(value) ... elif isinstance(value, dict): ... son[key] = self.transform_outgoing(value, collection) ... return son ... Now we'll empty the :class:`~pymongo.database.Database` and add the new manipulator: .. doctest:: >>> db.test.remove() {...} >>> db.add_son_manipulator(TransformToBinary()) After doing so we can save and restore :class:`Custom` instances seamlessly: .. doctest:: >>> db.test.insert({"custom": Custom(5)}) ObjectId('...') >>> db.test.find_one() {u'_id': ObjectId('...'), u'custom': } >>> db.test.find_one()["custom"].x() 5 We can see what's actually being saved to the database (and verify that it is using a :class:`~bson.binary.Binary` instance) by clearing out the manipulators and repeating our :meth:`~pymongo.collection.Collection.find_one`: .. doctest:: >>> db = client.custom_type_example >>> db.test.find_one() {u'_id': ObjectId('...'), u'custom': Binary('5', 128)} pymongo-2.6.3/doc/examples/geo.rst000066400000000000000000000045511223300253600171020ustar00rootroot00000000000000Geospatial Indexing Example =========================== .. testsetup:: from pymongo import MongoClient client = MongoClient() client.drop_database('geo_example') This example shows how to create and use a :data:`~pymongo.GEO2D` index in PyMongo. .. note:: 2D indexes require server version **>= 1.3.4**. Support for 2D indexes also requires PyMongo version **>= 1.5.1**. .. mongodoc:: geo Creating a Geospatial Index --------------------------- Creating a geospatial index in pymongo is easy: .. doctest:: >>> from pymongo import MongoClient, GEO2D >>> db = MongoClient().geo_example >>> db.places.create_index([("loc", GEO2D)]) u'loc_2d' Inserting Places ---------------- Locations in MongoDB are represented using either embedded documents or lists where the first two elements are coordinates. Here, we'll insert a couple of example locations: .. doctest:: >>> db.places.insert({"loc": [2, 5]}) ObjectId('...') >>> db.places.insert({"loc": [30, 5]}) ObjectId('...') >>> db.places.insert({"loc": [1, 2]}) ObjectId('...') >>> db.places.insert({"loc": [4, 4]}) ObjectId('...') Querying -------- Using the geospatial index we can find documents near another point: .. doctest:: >>> for doc in db.places.find({"loc": {"$near": [3, 6]}}).limit(3): ... repr(doc) ... "{u'loc': [2, 5], u'_id': ObjectId('...')}" "{u'loc': [4, 4], u'_id': ObjectId('...')}" "{u'loc': [1, 2], u'_id': ObjectId('...')}" It's also possible to query for all items within a given rectangle (specified by lower-left and upper-right coordinates): .. doctest:: >>> for doc in db.places.find({"loc": {"$within": {"$box": [[2, 2], [5, 6]]}}}): ... repr(doc) ... "{u'loc': [4, 4], u'_id': ObjectId('...')}" "{u'loc': [2, 5], u'_id': ObjectId('...')}" Or circle (specified by center point and radius): .. doctest:: >>> for doc in db.places.find({"loc": {"$within": {"$center": [[0, 0], 6]}}}): ... repr(doc) ... "{u'loc': [1, 2], u'_id': ObjectId('...')}" "{u'loc': [4, 4], u'_id': ObjectId('...')}" "{u'loc': [2, 5], u'_id': ObjectId('...')}" geoNear queries are also supported using :class:`~bson.son.SON`: .. doctest:: >>> from bson.son import SON >>> db.command(SON([('geoNear', 'places'), ('near', [1, 2])])) {u'ok': 1.0, u'near': u'1100000000000001100111100111100000000001100111100111', u'ns': u'geo_example.places', u'stats': ...} pymongo-2.6.3/doc/examples/gevent.rst000066400000000000000000000036341223300253600176210ustar00rootroot00000000000000Gevent ====== PyMongo supports `Gevent `_. Simply call Gevent's ``monkey.patch_all()`` before loading any other modules: .. doctest:: >>> # You must call patch_all() *before* importing any other modules >>> from gevent import monkey; monkey.patch_all() >>> from pymongo import MongoClient >>> client = MongoClient() PyMongo's Gevent support means that :meth:`~pymongo.mongo_client.MongoClient.start_request()` ensures the current greenlet (not merely the current thread) uses the same socket for all operations until :meth:`~pymongo.mongo_client.MongoClient.end_request()` is called. See the :doc:`requests documentation ` for details on requests in PyMongo. Using Gevent With Threads ------------------------- If you need to use standard Python threads in the same process as Gevent and greenlets, run ``monkey.patch_socket()``, rather than ``monkey.patch_all()``, and create a :class:`~pymongo.mongo_client.MongoClient` with ``use_greenlets=True``. The :class:`~pymongo.mongo_client.MongoClient` will use a special greenlet-aware connection pool. .. doctest:: >>> from gevent import monkey; monkey.patch_socket() >>> from pymongo import MongoClient >>> client = MongoClient(use_greenlets=True) An instance of :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` created with ``use_greenlets=True`` will also use a greenlet-aware pool. Additionally, it will use a background greenlet instead of a background thread to monitor the state of the replica set. .. doctest:: >>> from gevent import monkey; monkey.patch_socket() >>> from pymongo.mongo_replica_set_client import MongoReplicaSetClient >>> rsc = MongoReplicaSetClient( ... 'mongodb://localhost:27017,localhost:27018,localhost:27019', ... replicaSet='repl0', use_greenlets=True) Setting ``use_greenlets`` is unnecessary under normal circumstances; simply call ``patch_all`` to use Gevent with PyMongo.pymongo-2.6.3/doc/examples/gridfs.rst000066400000000000000000000043721223300253600176070ustar00rootroot00000000000000GridFS Example ============== .. testsetup:: from pymongo import MongoClient client = MongoClient() client.drop_database('gridfs_example') This example shows how to use :mod:`gridfs` to store large binary objects (e.g. files) in MongoDB. .. seealso:: The API docs for :mod:`gridfs`. .. seealso:: `This blog post `_ for some motivation behind this API. Setup ----- We start by creating a :class:`~gridfs.GridFS` instance to use: .. doctest:: >>> from pymongo import MongoClient >>> import gridfs >>> >>> db = MongoClient().gridfs_example >>> fs = gridfs.GridFS(db) Every :class:`~gridfs.GridFS` instance is created with and will operate on a specific :class:`~pymongo.database.Database` instance. Saving and Retrieving Data -------------------------- The simplest way to work with :mod:`gridfs` is to use its key/value interface (the :meth:`~gridfs.GridFS.put` and :meth:`~gridfs.GridFS.get` methods). To write data to GridFS, use :meth:`~gridfs.GridFS.put`: .. doctest:: >>> a = fs.put("hello world") :meth:`~gridfs.GridFS.put` creates a new file in GridFS, and returns the value of the file document's ``"_id"`` key. Given that ``"_id"`` we can use :meth:`~gridfs.GridFS.get` to get back the contents of the file: .. doctest:: >>> fs.get(a).read() 'hello world' :meth:`~gridfs.GridFS.get` returns a file-like object, so we get the file's contents by calling :meth:`~gridfs.grid_file.GridOut.read`. In addition to putting a :class:`str` as a GridFS file, we can also put any file-like object (an object with a :meth:`read` method). GridFS will handle reading the file in chunk-sized segments automatically. We can also add additional attributes to the file as keyword arguments: .. doctest:: >>> b = fs.put(fs.get(a), filename="foo", bar="baz") >>> out = fs.get(b) >>> out.read() 'hello world' >>> out.filename u'foo' >>> out.bar u'baz' >>> out.upload_date datetime.datetime(...) The attributes we set in :meth:`~gridfs.GridFS.put` are stored in the file document, and retrievable after calling :meth:`~gridfs.GridFS.get`. Some attributes (like ``"filename"``) are special and are defined in the GridFS specification - see that document for more details. pymongo-2.6.3/doc/examples/high_availability.rst000066400000000000000000000327101223300253600217770ustar00rootroot00000000000000High Availability and PyMongo ============================= PyMongo makes it easy to write highly available applications whether you use a `single replica set `_ or a `large sharded cluster `_. Connecting to a Replica Set --------------------------- PyMongo makes working with `replica sets `_ easy. Here we'll launch a new replica set and show how to handle both initialization and normal connections with PyMongo. .. note:: Replica sets require server version **>= 1.6.0**. Support for connecting to replica sets also requires PyMongo version **>= 1.8.0**. .. mongodoc:: rs Starting a Replica Set ~~~~~~~~~~~~~~~~~~~~~~ The main `replica set documentation `_ contains extensive information about setting up a new replica set or migrating an existing MongoDB setup, be sure to check that out. Here, we'll just do the bare minimum to get a three node replica set setup locally. .. warning:: Replica sets should always use multiple nodes in production - putting all set members on the same physical node is only recommended for testing and development. We start three ``mongod`` processes, each on a different port and with a different dbpath, but all using the same replica set name "foo". In the example we use the hostname "morton.local", so replace that with your hostname when running: .. code-block:: bash $ hostname morton.local $ mongod --replSet foo/morton.local:27018,morton.local:27019 --rest .. code-block:: bash $ mongod --port 27018 --dbpath /data/db1 --replSet foo/morton.local:27017 --rest .. code-block:: bash $ mongod --port 27019 --dbpath /data/db2 --replSet foo/morton.local:27017 --rest Initializing the Set ~~~~~~~~~~~~~~~~~~~~ At this point all of our nodes are up and running, but the set has yet to be initialized. Until the set is initialized no node will become the primary, and things are essentially "offline". To initialize the set we need to connect to a single node and run the initiate command. Since we don't have a primary yet, we'll need to tell PyMongo that it's okay to connect to a slave/secondary:: >>> from pymongo import MongoClient >>> c = MongoClient("morton.local:27017", slave_okay=True) .. note:: We could have connected to any of the other nodes instead, but only the node we initiate from is allowed to contain any initial data. After connecting, we run the initiate command to get things started (here we just use an implicit configuration, for more advanced configuration options see the replica set documentation):: >>> c.admin.command("replSetInitiate") {u'info': u'Config now saved locally. Should come online in about a minute.', u'info2': u'no configuration explicitly specified -- making one', u'ok': 1.0} The three ``mongod`` servers we started earlier will now coordinate and come online as a replica set. Connecting to a Replica Set ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The initial connection as made above is a special case for an uninitialized replica set. Normally we'll want to connect differently. A connection to a replica set can be made using the normal :meth:`~pymongo.mongo_client.MongoClient` constructor, specifying one or more members of the set. For example, any of the following will create a connection to the set we just created:: >>> MongoClient("morton.local", replicaset='foo') MongoClient([u'morton.local:27019', 'morton.local:27017', u'morton.local:27018']) >>> MongoClient("morton.local:27018", replicaset='foo') MongoClient([u'morton.local:27019', u'morton.local:27017', 'morton.local:27018']) >>> MongoClient("morton.local", 27019, replicaset='foo') MongoClient(['morton.local:27019', u'morton.local:27017', u'morton.local:27018']) >>> MongoClient(["morton.local:27018", "morton.local:27019"]) MongoClient(['morton.local:27019', u'morton.local:27017', 'morton.local:27018']) >>> MongoClient("mongodb://morton.local:27017,morton.local:27018,morton.local:27019") MongoClient(['morton.local:27019', 'morton.local:27017', 'morton.local:27018']) The nodes passed to :meth:`~pymongo.mongo_client.MongoClient` are called the *seeds*. If only one host is specified the `replicaset` parameter must be used to indicate this isn't a connection to a single node. As long as at least one of the seeds is online, the driver will be able to "discover" all of the nodes in the set and make a connection to the current primary. Handling Failover ~~~~~~~~~~~~~~~~~ When a failover occurs, PyMongo will automatically attempt to find the new primary node and perform subsequent operations on that node. This can't happen completely transparently, however. Here we'll perform an example failover to illustrate how everything behaves. First, we'll connect to the replica set and perform a couple of basic operations:: >>> db = MongoClient("morton.local", replicaSet='foo').test >>> db.test.save({"x": 1}) ObjectId('...') >>> db.test.find_one() {u'x': 1, u'_id': ObjectId('...')} By checking the host and port, we can see that we're connected to *morton.local:27017*, which is the current primary:: >>> db.connection.host 'morton.local' >>> db.connection.port 27017 Now let's bring down that node and see what happens when we run our query again:: >>> db.test.find_one() Traceback (most recent call last): pymongo.errors.AutoReconnect: ... We get an :class:`~pymongo.errors.AutoReconnect` exception. This means that the driver was not able to connect to the old primary (which makes sense, as we killed the server), but that it will attempt to automatically reconnect on subsequent operations. When this exception is raised our application code needs to decide whether to retry the operation or to simply continue, accepting the fact that the operation might have failed. On subsequent attempts to run the query we might continue to see this exception. Eventually, however, the replica set will failover and elect a new primary (this should take a couple of seconds in general). At that point the driver will connect to the new primary and the operation will succeed:: >>> db.test.find_one() {u'x': 1, u'_id': ObjectId('...')} >>> db.connection.host 'morton.local' >>> db.connection.port 27018 MongoReplicaSetClient ~~~~~~~~~~~~~~~~~~~~~ Using a :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` instead of a simple :class:`~pymongo.mongo_client.MongoClient` offers two key features: secondary reads and replica set health monitoring. To connect using :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` just provide a host:port pair and the name of the replica set:: >>> from pymongo import MongoReplicaSetClient >>> MongoReplicaSetClient("morton.local:27017", replicaSet='foo') MongoReplicaSetClient([u'morton.local:27019', u'morton.local:27017', u'morton.local:27018']) .. _secondary-reads: Secondary Reads ''''''''''''''' By default an instance of MongoReplicaSetClient will only send queries to the primary member of the replica set. To use secondaries for queries we have to change the :class:`~pymongo.read_preferences.ReadPreference`:: >>> db = MongoReplicaSetClient("morton.local:27017", replicaSet='foo').test >>> from pymongo.read_preferences import ReadPreference >>> db.read_preference = ReadPreference.SECONDARY_PREFERRED Now all queries will be sent to the secondary members of the set. If there are no secondary members the primary will be used as a fallback. If you have queries you would prefer to never send to the primary you can specify that using the ``SECONDARY`` read preference:: >>> db.read_preference = ReadPreference.SECONDARY Read preference can be set on a client, database, collection, or on a per-query basis, e.g.:: >>> db.collection.find_one(read_preference=ReadPreference.PRIMARY) Reads are configured using three options: **read_preference**, **tag_sets**, and **secondary_acceptable_latency_ms**. **read_preference**: - ``PRIMARY``: Read from the primary. This is the default, and provides the strongest consistency. If no primary is available, raise :class:`~pymongo.errors.AutoReconnect`. - ``PRIMARY_PREFERRED``: Read from the primary if available, or if there is none, read from a secondary matching your choice of ``tag_sets`` and ``secondary_acceptable_latency_ms``. - ``SECONDARY``: Read from a secondary matching your choice of ``tag_sets`` and ``secondary_acceptable_latency_ms``. If no matching secondary is available, raise :class:`~pymongo.errors.AutoReconnect`. - ``SECONDARY_PREFERRED``: Read from a secondary matching your choice of ``tag_sets`` and ``secondary_acceptable_latency_ms`` if available, otherwise from primary (regardless of the primary's tags and latency). - ``NEAREST``: Read from any member matching your choice of ``tag_sets`` and ``secondary_acceptable_latency_ms``. **tag_sets**: Replica-set members can be `tagged `_ according to any criteria you choose. By default, MongoReplicaSetClient ignores tags when choosing a member to read from, but it can be configured with the ``tag_sets`` parameter. ``tag_sets`` must be a list of dictionaries, each dict providing tag values that the replica set member must match. MongoReplicaSetClient tries each set of tags in turn until it finds a set of tags with at least one matching member. For example, to prefer reads from the New York data center, but fall back to the San Francisco data center, tag your replica set members according to their location and create a MongoReplicaSetClient like so: >>> rsc = MongoReplicaSetClient( ... "morton.local:27017", ... replicaSet='foo' ... read_preference=ReadPreference.SECONDARY, ... tag_sets=[{'dc': 'ny'}, {'dc': 'sf'}] ... ) MongoReplicaSetClient tries to find secondaries in New York, then San Francisco, and raises :class:`~pymongo.errors.AutoReconnect` if none are available. As an additional fallback, specify a final, empty tag set, ``{}``, which means "read from any member that matches the mode, ignoring tags." **secondary_acceptable_latency_ms**: If multiple members match the mode and tag sets, MongoReplicaSetClient reads from among the nearest members, chosen according to ping time. By default, only members whose ping times are within 15 milliseconds of the nearest are used for queries. You can choose to distribute reads among members with higher latencies by setting ``secondary_acceptable_latency_ms`` to a larger number. In that case, MongoReplicaSetClient distributes reads among matching members within ``secondary_acceptable_latency_ms`` of the closest member's ping time. .. note:: ``secondary_acceptable_latency_ms`` is ignored when talking to a replica set *through* a mongos. The equivalent is the localThreshold_ command line option. .. _localThreshold: http://docs.mongodb.org/manual/reference/mongos/#cmdoption-mongos--localThreshold Health Monitoring ''''''''''''''''' When MongoReplicaSetClient is initialized it launches a background task to monitor the replica set for changes in: * Health: detect when a member goes down or comes up, or if a different member becomes primary * Configuration: detect changes in tags * Latency: track a moving average of each member's ping time Replica-set monitoring ensures queries are continually routed to the proper members as the state of the replica set changes. It is critical to call :meth:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient.close` to terminate the monitoring task before your process exits. .. _mongos-high-availability: High Availability and mongos ---------------------------- An instance of :class:`~pymongo.mongo_client.MongoClient` can be configured to automatically connect to a different mongos if the instance it is currently connected to fails. If a failure occurs, PyMongo will attempt to find the nearest mongos to perform subsequent operations. As with a replica set this can't happen completely transparently, Here we'll perform an example failover to illustrate how everything behaves. First, we'll connect to a sharded cluster, using a seed list, and perform a couple of basic operations:: >>> db = MongoClient('morton.local:30000,morton.local:30001,morton.local:30002').test >>> db.test.save({"x": 1}) ObjectId('...') >>> db.test.find_one() {u'x': 1, u'_id': ObjectId('...')} Each member of the seed list passed to MongoClient must be a mongos. By checking the host, port, and is_mongos attributes we can see that we're connected to *morton.local:30001*, a mongos:: >>> db.connection.host 'morton.local' >>> db.connection.port 30001 >>> db.connection.is_mongos True Now let's shut down that mongos instance and see what happens when we run our query again:: >>> db.test.find_one() Traceback (most recent call last): pymongo.errors.AutoReconnect: ... As in the replica set example earlier in this document, we get an :class:`~pymongo.errors.AutoReconnect` exception. This means that the driver was not able to connect to the original mongos at port 30001 (which makes sense, since we shut it down), but that it will attempt to connect to a new mongos on subsequent operations. When this exception is raised our application code needs to decide whether to retry the operation or to simply continue, accepting the fact that the operation might have failed. As long as one of the seed list members is still available the next operation will succeed:: >>> db.test.find_one() {u'x': 1, u'_id': ObjectId('...')} >>> db.connection.host 'morton.local' >>> db.connection.port 30002 >>> db.connection.is_mongos True pymongo-2.6.3/doc/examples/index.rst000066400000000000000000000010701223300253600174300ustar00rootroot00000000000000Examples ======== The examples in this section are intended to give in depth overviews of how to accomplish specific tasks with MongoDB and PyMongo. Unless otherwise noted, all examples assume that a MongoDB instance is running on the default host and port. Assuming you have `downloaded and installed `_ MongoDB, you can start it like so: .. code-block:: bash $ mongod .. toctree:: :maxdepth: 1 aggregation authentication custom_type geo gevent gridfs high_availability requests pymongo-2.6.3/doc/examples/requests.rst000066400000000000000000000101501223300253600201730ustar00rootroot00000000000000Requests ======== PyMongo supports the idea of a *request*: a series of operations executed with a single socket, which are guaranteed to be processed on the server in the same order as they ran on the client. Requests are not usually necessary with PyMongo. By default, the methods :meth:`~pymongo.collection.Collection.insert`, :meth:`~pymongo.collection.Collection.update`, :meth:`~pymongo.collection.Collection.save`, and :meth:`~pymongo.collection.Collection.remove` block until they receive acknowledgment from the server, so ordered execution is already guaranteed. You can be certain the next :meth:`~pymongo.collection.Collection.find` or :meth:`~pymongo.collection.Collection.count`, for example, is executed on the server after the writes complete. This is called "read-your-writes consistency." An example of when a request is necessary is if a series of documents are inserted with ``w=0`` for performance reasons, and you want to query those documents immediately afterward: With ``w=0`` the writes can queue up at the server and might not be immediately visible in query results. Wrapping the inserts and queries within :meth:`~pymongo.mongo_client.MongoClient.start_request` and :meth:`~pymongo.mongo_client.MongoClient.end_request` forces a query to be on the same socket as the inserts, so the query won't execute until the inserts are complete on the server side. Example ------- Let's consider a collection of web-analytics counters. We want to count the number of page views our site has served for each combination of browser, region, and OS, and then show the user the number of page views from his or her region, *including* the user's own visit. We have three ways to do so reliably: 1. Simply update the counters with an acknowledged write (the default), and then ``find`` all counters for the visitor's region. This will ensure that the ``update`` completes before the ``find`` begins, but it comes with a performance penalty that may be unacceptable for analytics. 2. Create the :class:`~pymongo.mongo_client.MongoClient` with ``w=0`` and ``auto_start_request=True`` to do unacknowledged writes and ensure each thread gets its own socket. 3. Explicitly call :meth:`~pymongo.mongo_client.MongoClient.start_request`, then do the unacknowledged updates and the queries within the request. This third method looks like: .. testsetup:: from pymongo import MongoClient client = MongoClient() counts = client.requests_example.counts counts.drop() region, browser, os = 'US', 'Firefox', 'Mac OS X' .. doctest:: >>> client = MongoClient() >>> counts = client.requests_example.counts >>> region, browser, os = 'US', 'Firefox', 'Mac OS X' >>> request = client.start_request() >>> try: ... counts.update( ... {'region': region, 'browser': browser, 'os': os}, ... {'$inc': {'n': 1 }}, ... upsert=True, ... w=0) # unacknowledged write ... ... # This always runs after update has completed: ... count = sum([p['n'] for p in counts.find({'region': region})]) ... finally: ... request.end() >>> print count 1 Requests can also be used as context managers, with the `with statement `_, which makes the previous example more terse: .. doctest:: >>> client.in_request() False >>> with client.start_request(): ... # MongoClient is now in request ... counts.update( ... {'region': region, 'browser': browser, 'os': os}, ... {'$inc': {'n': 1 }}, ... upsert=True, ... safe=False) ... print sum([p['n'] for p in counts.find({'region': region})]) 2 >>> client.in_request() # request automatically ended False Requests And ``max_pool_size`` ------------------------------ A thread in a request retains exclusive access to a socket until its request ends or the thread dies; thus, applications in which more than 100 threads are in requests at once should disable the ``max_pool_size`` option:: client = MongoClient(host, port, max_pool_size=None) Failure to increase or disable ``max_pool_size`` in such an application can leave threads forever waiting for sockets. See :ref:`connection-pooling` pymongo-2.6.3/doc/faq.rst000066400000000000000000000354011223300253600152570ustar00rootroot00000000000000Frequently Asked Questions ========================== .. contents:: Is PyMongo thread-safe? ----------------------- PyMongo is thread-safe and even provides built-in connection pooling for threaded applications. .. _connection-pooling: How does connection pooling work in PyMongo? -------------------------------------------- Every :class:`~pymongo.mongo_client.MongoClient` instance has a built-in connection pool. The pool begins with one open connection. If necessary to support concurrent access to MongoDB from multiple threads in your application, the client opens new connections on demand. By default, there is no thread-affinity for connections. In versions before 2.6, the default ``max_pool_size`` was 10, and it did not actually bound the number of open connections; it only determined the number of connections that would be kept open when no longer in use. Starting with PyMongo 2.6, the size of the connection pool is capped at ``max_pool_size``, which now defaults to 100. When a thread in your application begins an operation on MongoDB, if all other connections are in use and the pool has reached its maximum, the thread pauses, waiting for a connection to be returned to the pool by another thread. The default configuration for a :class:`~pymongo.mongo_client.MongoClient` works for most applications:: client = MongoClient(host, port) Create this client **once** when your program starts up, and reuse it for all operations. It is a common mistake to create a new client for each request, which is very inefficient. To support extremely high numbers of concurrent MongoDB operations within one process, increase ``max_pool_size``:: client = MongoClient(host, port, max_pool_size=200) ... or make it unbounded:: client = MongoClient(host, port, max_pool_size=None) By default, any number of threads are allowed to wait for connections to become available, and they can wait any length of time. Override ``waitQueueMultiple`` to cap the number of waiting threads. E.g., to keep the number of waiters less than or equal to 500:: client = MongoClient(host, port, max_pool_size=50, waitQueueMultiple=10) When 500 threads are waiting for a socket, the 501st that needs a connection raises :exc:`~pymongo.errors.ExceededMaxWaiters`. Use this option to bound the amount of queueing in your application during a load spike, at the cost of additional exceptions. Once the pool reaches its max size, additional threads are allowed to wait indefinitely for connections to become available, unless you set ``waitQueueTimeoutMS``:: client = MongoClient(host, port, waitQueueTimeoutMS=100) A thread that waits more than 100ms (in this example) for a connection raises :exc:`~pymongo.errors.ConnectionFailure`. Use this option if it is more important to bound the duration of operations during a load spike than it is to complete every operation. When :meth:`~pymongo.mongo_client.MongoClient.disconnect` is called by any thread, all sockets are closed. :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` maintains one connection pool per server in your replica set. .. seealso:: :doc:`examples/requests` Does PyMongo support Python 3? ------------------------------ Starting with version 2.2 PyMongo supports Python 3.x where x >= 1. See the :doc:`python3` for details. Does PyMongo support asynchronous frameworks like Gevent, Tornado, or Twisted? ------------------------------------------------------------------------------ PyMongo fully supports :doc:`Gevent `. To use MongoDB with `Tornado `_ see the `Motor `_ project. For `Twisted `_, see `TxMongo `_. Compared to PyMongo, TxMongo is less stable, lack features, and is less actively maintained. What does *OperationFailure* cursor id not valid at server mean? ---------------------------------------------------------------- Cursors in MongoDB can timeout on the server if they've been open for a long time without any operations being performed on them. This can lead to an :class:`~pymongo.errors.OperationFailure` exception being raised when attempting to iterate the cursor. How do I change the timeout value for cursors? ---------------------------------------------- MongoDB doesn't support custom timeouts for cursors, but cursor timeouts can be turned off entirely. Pass ``timeout=False`` to :meth:`~pymongo.collection.Collection.find`. How can I store :mod:`decimal.Decimal` instances? ------------------------------------------------- MongoDB only supports IEEE 754 floating points - the same as the Python float type. The only way PyMongo could store Decimal instances would be to convert them to this standard, so you'd really only be storing floats anyway - we force users to do this conversion explicitly so that they are aware that it is happening. I'm saving ``9.99`` but when I query my document contains ``9.9900000000000002`` - what's going on here? -------------------------------------------------------------------------------------------------------- The database representation is ``9.99`` as an IEEE floating point (which is common to MongoDB and Python as well as most other modern languages). The problem is that ``9.99`` cannot be represented exactly with a double precision floating point - this is true in some versions of Python as well: >>> 9.99 9.9900000000000002 The result that you get when you save ``9.99`` with PyMongo is exactly the same as the result you'd get saving it with the JavaScript shell or any of the other languages (and as the data you're working with when you type ``9.99`` into a Python program). Can you add attribute style access for documents? ------------------------------------------------- This request has come up a number of times but we've decided not to implement anything like this. The relevant `jira case `_ has some information about the decision, but here is a brief summary: 1. This will pollute the attribute namespace for documents, so could lead to subtle bugs / confusing errors when using a key with the same name as a dictionary method. 2. The only reason we even use SON objects instead of regular dictionaries is to maintain key ordering, since the server requires this for certain operations. So we're hesitant to needlessly complicate SON (at some point it's hypothetically possible we might want to revert back to using dictionaries alone, without breaking backwards compatibility for everyone). 3. It's easy (and Pythonic) for new users to deal with documents, since they behave just like dictionaries. If we start changing their behavior it adds a barrier to entry for new users - another class to learn. What is the correct way to handle time zones with PyMongo? ---------------------------------------------------------- Prior to PyMongo version 1.7, the correct way is to only save naive :class:`~datetime.datetime` instances, and to save all dates as UTC. In versions >= 1.7, the driver will automatically convert aware datetimes to UTC before saving them. By default, datetimes retrieved from the server (no matter what version of the driver you're using) will be naive and represent UTC. In newer versions of the driver you can set the :class:`~pymongo.mongo_client.MongoClient` `tz_aware` parameter to ``True``, which will cause all :class:`~datetime.datetime` instances returned from that MongoClient to be aware (UTC). This setting is recommended, as it can force application code to handle timezones properly. .. warning:: Be careful not to save naive :class:`~datetime.datetime` instances that are not UTC (i.e. the result of calling :meth:`datetime.datetime.now`). Something like :mod:`pytz` can be used to convert dates to localtime after retrieving them from the database. How can I save a :mod:`datetime.date` instance? ----------------------------------------------- PyMongo doesn't support saving :mod:`datetime.date` instances, since there is no BSON type for dates without times. Rather than having the driver enforce a convention for converting :mod:`datetime.date` instances to :mod:`datetime.datetime` instances for you, any conversion should be performed in your client code. .. _web-application-querying-by-objectid: When I query for a document by ObjectId in my web application I get no result ----------------------------------------------------------------------------- It's common in web applications to encode documents' ObjectIds in URLs, like:: "/posts/50b3bda58a02fb9a84d8991e" Your web framework will pass the ObjectId portion of the URL to your request handler as a string, so it must be converted to :class:`~bson.objectid.ObjectId` before it is passed to :meth:`~pymongo.collection.Collection.find_one`. It is a common mistake to forget to do this conversion. Here's how to do it correctly in Flask_ (other web frameworks are similar):: from pymongo import MongoClient from bson.objectid import ObjectId from flask import Flask, render_template client = MongoClient() app = Flask(__name__) @app.route("/posts/<_id>") def show_post(_id): # NOTE!: converting _id from string to ObjectId before passing to find_one post = client.db.posts.find_one({'_id': ObjectId(_id)}) return render_template('post.html', post=post) if __name__ == "__main__": app.run() .. _Flask: http://flask.pocoo.org/ .. seealso:: :ref:`querying-by-objectid` How can I use PyMongo from Django? ---------------------------------- `Django `_ is a popular Python web framework. Django includes an ORM, :mod:`django.db`. Currently, there's no official MongoDB backend for Django. `django-mongodb-engine `_ is an unofficial MongoDB backend that supports Django aggregations, (atomic) updates, embedded objects, Map/Reduce and GridFS. It allows you to use most of Django's built-in features, including the ORM, admin, authentication, site and session frameworks and caching. However, it's easy to use MongoDB (and PyMongo) from Django without using a Django backend. Certain features of Django that require :mod:`django.db` (admin, authentication and sessions) will not work using just MongoDB, but most of what Django provides can still be used. We have written a demo `Django + MongoDB project `_. The README for that project describes some of what you need to do to use MongoDB from Django. The main point is that your persistence code will go directly into your views, rather than being defined in separate models. The README also gives instructions for how to change settings.py to disable the features that won't work with MongoDB. One project which should make working with MongoDB and Django easier is `mango `_. Mango is a set of MongoDB backends for Django sessions and authentication (bypassing :mod:`django.db` entirely). .. _using-with-mod-wsgi: Does PyMongo work with **mod_wsgi**? ------------------------------------ `mod_wsgi `_ is a popular Apache module used for hosting Python applications conforming to the `wsgi `_ specification. There is a potential issue when deploying PyMongo applications with mod_wsgi involving PyMongo's C extension and mod_wsgi's multiple sub interpreters. One tricky issue that we've seen when deploying PyMongo applications with mod_wsgi is documented `here `_, in the **Multiple Python Sub Interpreters** section. When running PyMongo with the C extension enabled it is possible to see strange failures when encoding due to the way mod_wsgi handles module reloading with multiple sub interpreters. There are several possible ways to work around this issue: 1. Run mod_wsgi in daemon mode with each WSGI application assigned to its own daemon process. 2. Force all WSGI applications to run in the same application group. 3. Install PyMongo :ref:`without the C extension ` (this will carry a performance penalty, but is the most immediate solution to this problem). How can I use something like Python's :mod:`json` module to encode my documents to JSON? ---------------------------------------------------------------------------------------- The :mod:`json` module won't work out of the box with all documents from PyMongo as PyMongo supports some special types (like :class:`~bson.objectid.ObjectId` and :class:`~bson.dbref.DBRef`) that are not supported in JSON. We've added some utilities for working with :mod:`json` and :mod:`simplejson` in the :mod:`~bson.json_util` module. .. _year-2038-problem: Why do I get an error for dates on or after 2038? ------------------------------------------------- On Unix systems, dates are represented as seconds from 1 January 1970 and usually stored in the C :mod:`time_t` type. On most 32-bit operating systems :mod:`time_t` is a signed 4 byte integer which means it can't handle dates after 19 January 2038; this is known as the `year 2038 problem `_. Neither MongoDB nor Python uses :mod:`time_t` to represent dates internally so do not suffer from this problem, but Python's :mod:`datetime.datetime.fromtimestamp()` does, which means it is susceptible. Previous to version 2.0, PyMongo used :mod:`datetime.datetime.fromtimestamp()` in its pure Python :mod:`bson` module. Therefore, on 32-bit systems you may get an error retrieving dates after 2038 from MongoDB using older versions of PyMongo with the pure Python version of :mod:`bson`. This problem was fixed in the pure Python implementation of :mod:`bson` by commit ``b19ab334af2a29353529`` (8 June 2011 - PyMongo 2.0). The C implementation of :mod:`bson` also used to suffer from this problem but it was fixed in commit ``566bc9fb7be6f9ab2604`` (10 May 2010 - PyMongo 1.7). Why do I get OverflowError decoding dates stored by another language's driver? ------------------------------------------------------------------------------ PyMongo decodes BSON datetime values to instances of Python's :class:`datetime.datetime`. Instances of :class:`datetime.datetime` are limited to years between :data:`datetime.MINYEAR` (usually 1) and :data:`datetime.MAXYEAR` (usually 9999). Some MongoDB drivers (e.g. the PHP driver) can store BSON datetimes with year values far outside those supported by :class:`datetime.datetime`. There are a few ways to work around this issue. One option is to filter out documents with values outside of the range supported by :class:`datetime.datetime`:: >>> from datetime import datetime >>> coll = client.test.dates >>> cur = coll.find({'dt': {'$gte': datetime.min, '$lte': datetime.max}}) Another option, assuming you don't need the datetime field, is to filter out just that field:: >>> cur = coll.find({}, fields={'dt': False}) pymongo-2.6.3/doc/index.rst000066400000000000000000000046771223300253600156320ustar00rootroot00000000000000PyMongo |release| Documentation =============================== Overview -------- **PyMongo** is a Python distribution containing tools for working with `MongoDB `_, and is the recommended way to work with MongoDB from Python. This documentation attempts to explain everything you need to know to use **PyMongo**. .. todo:: a list of PyMongo's features :doc:`installation` Instructions on how to get the distribution. :doc:`tutorial` Start here for a quick overview. :doc:`examples/index` Examples of how to perform specific tasks. :doc:`faq` Some questions that come up often. :doc:`python3` Frequently asked questions about python 3 support. :doc:`api/index` The complete API documentation, organized by module. :doc:`tools` A listing of Python tools and libraries that have been written for MongoDB. Getting Help ------------ If you're having trouble or have questions about PyMongo, the best place to ask is the `MongoDB user group `_. Once you get an answer, it'd be great if you could work it back into this documentation and contribute! Issues ------ All issues should be reported (and can be tracked / voted for / commented on) at the main `MongoDB JIRA bug tracker `_, in the "Python Driver" project. Contributing ------------ **PyMongo** has a large :doc:`community ` and contributions are always encouraged. Contributions can be as simple as minor tweaks to this documentation. To contribute, fork the project on `github `_ and send a pull request. Changes ------- See the :doc:`changelog` for a full list of changes to PyMongo. For older versions of the documentation please see the `archive list `_. About This Documentation ------------------------ This documentation is generated using the `Sphinx `_ documentation generator. The source files for the documentation are located in the *doc/* directory of the **PyMongo** distribution. To generate the docs locally run the following command from the root directory of the **PyMongo** source: .. code-block:: bash $ python setup.py doc Indices and tables ------------------ * :ref:`genindex` * :ref:`modindex` * :ref:`search` .. toctree:: :hidden: installation tutorial examples/index faq api/index tools contributors changelog python3 pymongo-2.6.3/doc/installation.rst000066400000000000000000000160171223300253600172130ustar00rootroot00000000000000Installing / Upgrading ====================== .. highlight:: bash **PyMongo** is in the `Python Package Index `_. Microsoft Windows ----------------- We recommend using the `MS Windows installers` available from the `Python Package Index `_. Installing with pip ------------------- We prefer `pip `_ to install pymongo on platforms other than Windows:: $ pip install pymongo To get a specific version of pymongo:: $ pip install pymongo==2.1.1 To upgrade using pip:: $ pip install --upgrade pymongo Installing with easy_install ---------------------------- If you must install pymongo using `setuptools `_ do:: $ easy_install pymongo To upgrade do:: $ easy_install -U pymongo Dependencies for installing C Extensions on Unix ------------------------------------------------ 10gen does not provide statically linked binary packages for Unix flavors other than OSX. To build the optional C extensions you must have the GNU C compiler (gcc) installed. Depending on your flavor of Unix (or Linux distribution) you may also need a python development package that provides the necessary header files for your version of Python. The package name may vary from distro to distro. Debian and Ubuntu users should issue the following command:: $ sudo apt-get install build-essential python-dev RedHat, CentOS, and Fedora users should issue the following command:: $ sudo yum install gcc python-devel OSX --- 10gen provides pre-built egg packages for Apple provided Python versions on Snow Leopard (2.5, 2.6), Lion (2.5, 2.6, 2.7) and Mountain Lion (2.5, 2.6, 2.7). If you want to install PyMongo for other Python versions (or from source) you will have to install the following to build the C extensions: **Snow Leopard (10.6)** - Xcode 3 with 'UNIX Development Support'. **Snow Leopard Xcode 4**: The Python versions shipped with OSX 10.6.x are universal binaries. They support i386, PPC, and (in the case of python2.6) x86_64. Xcode 4 removed support for PPC, causing the distutils version shipped with Apple's builds of Python to fail to build the C extensions if you have Xcode 4 installed. There is a workaround:: # For Apple-supplied Python2.6 (installed at /usr/bin/python2.6) and # some builds from python.org $ env ARCHFLAGS='-arch i386 -arch x86_64' python -m easy_install pymongo # For 32-bit-only Python (/usr/bin/python2.5) and some builds # from python.org $ env ARCHFLAGS='-arch i386' python -m easy_install pymongo See `http://bugs.python.org/issue11623 `_ for a more detailed explanation. **Lion (10.7)** - PyMongo's C extensions can be built against versions of Python >= 3.2 downloaded from python.org. Building against versions older than 3.2.3 requires **Xcode 4.1**. Any version of Xcode 4 can be used to build the C extensions against 3.2.3 and newer. In all cases Xcode must be installed with 'UNIX Development Support'. See the following for more information: http://bugs.python.org/issue13590 http://hg.python.org/cpython/file/v3.2.3/Misc/NEWS#l198 **Mountain Lion (10.8)** - PyMongo's C extensions can be built against versions of Python >= 3.3rc1 downloaded from python.org with no special requirements. If you want to build against the python.org provided 3.2.3 you must have MacOSX10.6.sdk in /Developer/SDKs. See the following for more information: http://bugs.python.org/issue14499 Installing from source ---------------------- If you'd rather install directly from the source (i.e. to stay on the bleeding edge), install the C extension dependencies then check out the latest source from github and install the driver from the resulting tree:: $ git clone git://github.com/mongodb/mongo-python-driver.git pymongo $ cd pymongo/ $ python setup.py install Installing from source on Windows ................................. .. note:: 10gen provides pre-built exe installers for 32-bit and 64-bit Windows. We recommend that users install those packages (`available from pypi `_). If you want to install PyMongo with C extensions from source the following directions apply to both CPython and ActiveState's ActivePython: 64-bit Windows ~~~~~~~~~~~~~~ For Python 3.3 install Visual Studio 2010. For Python 3.2 and older install Visual Studio 2008. In either case you must use the full version as Visual C++ Express does not provide 64-bit compilers. Make sure that you check the "x64 Compilers and Tools" option under Visual C++. 32-bit Windows ~~~~~~~~~~~~~~ For Python 3.3 install Visual C++ 2010 Express. For Python 2.6 through 3.2 install Visual C++ 2008 Express SP1. For Python 2.4 or 2.5 you must install `MingW32 `_ then run the following command to install:: python setup.py build -c mingw32 install .. _install-no-c: Installing Without C Extensions ------------------------------- By default, the driver attempts to build and install optional C extensions (used for increasing performance) when it is installed. If any extension fails to build the driver will be installed anyway but a warning will be printed. In :ref:`certain cases `, you might wish to install the driver without the C extensions, even if the extensions build properly. This can be done using a command line option to *setup.py*:: $ python setup.py --no_ext install Building PyMongo egg Packages ----------------------------- Some organizations do not allow compilers and other build tools on production systems. To install PyMongo on these systems with C extensions you may need to build custom egg packages. Make sure that you have installed the dependencies listed above for your operating system then run the following command in the PyMongo source directory:: $ python setup.py bdist_egg The egg package can be found in the dist/ subdirectory. The file name will resemble “pymongo-2.2-py2.7-linux-x86_64.egg” but may have a different name depending on your platform and the version of python you use to compile. .. warning:: These “binary distributions,” will only work on systems that resemble the environment on which you built the package. In other words, ensure that operating systems and versions of Python and architecture (i.e. “32” or “64” bit) match. Copy this file to the target system and issue the following command to install the package:: $ sudo easy_install pymongo-2.2-py2.7-linux-x86_64.egg Installing a release candidate ------------------------------ 10gen may occasionally tag a release candidate for testing by the community before final release. These releases will not be uploaded to pypi but can be found on the `github tags page `_. They can be installed by passing the full URL for the tag to pip:: $ pip install https://github.com/mongodb/mongo-python-driver/tarball/2.2rc1 or easy_install:: $ easy_install https://github.com/mongodb/mongo-python-driver/tarball/2.2rc1 pymongo-2.6.3/doc/mongo_extensions.py000066400000000000000000000061341223300253600177270ustar00rootroot00000000000000# Copyright 2009-2010 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """MongoDB specific extensions to Sphinx.""" from docutils import nodes from sphinx import addnodes from sphinx.util.compat import (Directive, make_admonition) class mongodoc(nodes.Admonition, nodes.Element): pass class mongoref(nodes.reference): pass def visit_mongodoc_node(self, node): self.visit_admonition(node, "seealso") def depart_mongodoc_node(self, node): self.depart_admonition(node) def visit_mongoref_node(self, node): atts = {"class": "reference external", "href": node["refuri"], "name": node["name"]} self.body.append(self.starttag(node, 'a', '', **atts)) def depart_mongoref_node(self, node): self.body.append('') if not isinstance(node.parent, nodes.TextElement): self.body.append('\n') class MongodocDirective(Directive): has_content = True required_arguments = 0 optional_arguments = 0 final_argument_whitespace = False option_spec = {} def run(self): return make_admonition(mongodoc, self.name, ['See general MongoDB documentation'], self.options, self.content, self.lineno, self.content_offset, self.block_text, self.state, self.state_machine) def process_mongodoc_nodes(app, doctree, fromdocname): for node in doctree.traverse(mongodoc): anchor = None for name in node.parent.parent.traverse(addnodes.desc_signature): anchor = name["ids"][0] break if not anchor: for name in node.parent.traverse(nodes.section): anchor = name["ids"][0] break for para in node.traverse(nodes.paragraph): tag = str(para.traverse()[1]) link = mongoref("", "") link["refuri"] = "http://dochub.mongodb.org/core/%s" % tag link["name"] = anchor link.append(nodes.emphasis(tag, tag)) new_para = nodes.paragraph() new_para += link node.replace(para, new_para) def setup(app): app.add_node(mongodoc, html=(visit_mongodoc_node, depart_mongodoc_node), latex=(visit_mongodoc_node, depart_mongodoc_node), text=(visit_mongodoc_node, depart_mongodoc_node)) app.add_node(mongoref, html=(visit_mongoref_node, depart_mongoref_node)) app.add_directive("mongodoc", MongodocDirective) app.connect("doctree-resolved", process_mongodoc_nodes) pymongo-2.6.3/doc/python3.rst000066400000000000000000000160551223300253600161200ustar00rootroot00000000000000Python 3 FAQ ============ .. contents:: What Python 3 versions are supported? ------------------------------------- PyMongo supports Python 3.x where x >= 1. We **do not** support Python 3.0.x. It has many problems (some that directly impact PyMongo) and was `end-of-lifed`_ with the release of Python 3.1. .. _end-of-lifed: http://www.python.org/download/releases/3.0.1/ Are there any PyMongo behavior changes with Python 3? ----------------------------------------------------- Only one intentional change. Instances of :class:`bytes` are encoded as BSON type 5 (Binary data) with subtype 0. In Python 3 they are decoded back to :class:`bytes`. In Python 2 they will be decoded to :class:`~bson.binary.Binary` with subtype 0. For example, let's insert a :class:`bytes` instance using Python 3 then read it back. Notice the byte string is decoded back to :class:`bytes`:: Python 3.1.4 (default, Mar 21 2012, 14:34:01) [GCC 4.5.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pymongo >>> c = pymongo.MongoClient() >>> c.test.bintest.insert({'binary': b'this is a byte string'}) ObjectId('4f9086b1fba5222021000000') >>> c.test.bintest.find_one() {'binary': b'this is a byte string', '_id': ObjectId('4f9086b1fba5222021000000')} Now retrieve the same document in Python 2. Notice the byte string is decoded to :class:`~bson.binary.Binary`:: Python 2.7.3 (default, Apr 12 2012, 10:35:17) [GCC 4.5.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pymongo >>> c = pymongo.MongoClient() >>> c.test.bintest.find_one() {u'binary': Binary('this is a byte string', 0), u'_id': ObjectId('4f9086b1fba5222021000000')} Why can't I share pickled ObjectIds between some versions of Python 2 and 3? ---------------------------------------------------------------------------- Instances of :class:`~bson.objectid.ObjectId` pickled using Python 2 can always be unpickled using Python 3. Due to `http://bugs.python.org/issue13505 `_ you must use Python 3.2.3 or newer to pickle instances of ObjectId if you need to unpickle them in Python 2. If you pickled an ObjectId using Python 2 and want to unpickle it using Python 3 you must pass ``encoding='latin-1'`` to pickle.loads:: Python 2.7.3 (default, Apr 12 2012, 10:35:17) [GCC 4.5.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pickle >>> from bson.objectid import ObjectId >>> oid = ObjectId() >>> oid ObjectId('4f919ba2fba5225b84000000') >>> pickle.dumps(oid) 'ccopy_reg\n_reconstructor\np0\n(cbson.objectid\...' Python 3.1.4 (default, Mar 21 2012, 14:34:01) [GCC 4.5.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pickle >>> pickle.loads(b'ccopy_reg\n_reconstructor\np0\n(cbson.objectid\...', encoding='latin-1') ObjectId('4f919ba2fba5225b84000000') If you need to pickle ObjectIds using Python 3 and unpickle them using Python 2 you must use Python 3.2.3 or newer and ``protocol <= 2``:: Python 3.2.3 (v3.2.3:3d0686d90f55, Apr 10 2012, 11:25:50) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pickle >>> from bson.objectid import ObjectId >>> oid = ObjectId() >>> oid ObjectId('4f96f20c430ee6bd06000000') >>> pickle.dumps(oid, protocol=2) b'\x80\x02cbson.objectid\nObjectId\nq\x00)\x81q\x01c_codecs\nencode\...' Python 2.4.4 (#1, Oct 18 2006, 10:34:39) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pickle >>> pickle.loads('\x80\x02cbson.objectid\nObjectId\nq\x00)\x81q\x01c_codecs\nencode\...') ObjectId('4f96f20c430ee6bd06000000') Unfortunately this won't work if you pickled the ObjectId using a Python 3 version older than 3.2.3:: Python 3.2.2 (default, Mar 21 2012, 14:32:23) [GCC 4.5.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pickle >>> from bson.objectid import ObjectId >>> oid = ObjectId() >>> pickle.dumps(oid, protocol=2) b'\x80\x02cbson.objectid\nObjectId\nq\x00)\x81q\x01c__builtin__\nbytes\...' Python 2.4.6 (#1, Apr 12 2012, 14:48:24) [GCC 4.5.3] on linux3 Type "help", "copyright", "credits" or "license" for more information. >>> import pickle >>> pickle.loads('\x80\x02cbson.objectid\nObjectId\nq\x00)\x81q\x01c__builtin__\nbytes\...') Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.4/pickle.py", line 1394, in loads return Unpickler(file).load() File "/usr/lib/python2.4/pickle.py", line 872, in load dispatch[key](self) File "/usr/lib/python2.4/pickle.py", line 1104, in load_global klass = self.find_class(module, name) File "/usr/lib/python2.4/pickle.py", line 1140, in find_class klass = getattr(mod, name) AttributeError: 'module' object has no attribute 'bytes' .. warning:: Unpickling in Python 2.6 or 2.7 an ObjectId pickled in a Python 3 version older than 3.2.3 will seem to succeed but the resulting ObjectId instance will contain garbage data. >>> pickle.loads('\x80\x02cbson.objectid\nObjectId\nq\x00)\x81q\x01c__builtin__\nbytes\...) ObjectId('5b37392c203135302c203234362c2034352c203235312c203136352c2033342c203532...') Why do I get a syntax error importing pymongo after installing from source? --------------------------------------------------------------------------- PyMongo makes use of the 2to3 tool to translate much of its code to valid Python 3 syntax at install time. The translated modules are written to the build subdirectory before being installed, leaving the original source files intact. If you start the python interactive shell from the top level source directory after running ``python setup.py install`` the untranslated modules will be the first thing in your path. Importing pymongo will result in an exception similar to:: Python 3.1.5 (default, Jun 2 2012, 12:24:49) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pymongo Traceback (most recent call last): File "", line 1, in File "pymongo/__init__.py", line 58, in version = get_version_string() File "pymongo/__init__.py", line 54, in get_version_string if isinstance(version_tuple[-1], basestring): NameError: global name 'basestring' is not defined Note the path in the traceback (``pymongo/__init__.py``). Changing out of the source directory takes the untranslated modules out of your path:: $ cd .. $ python Python 3.1.5 (default, Jun 2 2012, 12:24:49) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pymongo >>> pymongo.__file__ '/home/behackett/py3k/lib/python3.1/site-packages/pymongo-2.2-py3.1-linux-x86_64.egg/pymongo/__init__.py' pymongo-2.6.3/doc/tools.rst000066400000000000000000000145151223300253600156530ustar00rootroot00000000000000Tools ===== Many tools have been written for working with **PyMongo**. If you know of or have created a tool for working with MongoDB from Python please list it here. .. note:: We try to keep this list current. As such, projects that have not been updated recently or appear to be unmaintained will occasionally be removed from the list or moved to the back (to keep the list from becoming too intimidating). If a project gets removed that is still being developed or is in active use please let us know or add it back. ORM-like Layers --------------- Some people have found that they prefer to work with a layer that has more features than PyMongo provides. Often, things like models and validation are desired. To that end, several different ORM-like layers have been written by various authors. It is our recommendation that new users begin by working directly with PyMongo, as described in the rest of this documentation. Many people have found that the features of PyMongo are enough for their needs. Even if you eventually come to the decision to use one of these layers, the time spent working directly with the driver will have increased your understanding of how MongoDB actually works. Humongolus `Humongolus `_ is a lightweight ORM framework for Python and MongoDB. The name comes from the combination of MongoDB and `Homunculus `_ (the concept of a miniature though fully formed human body). Humongolus allows you to create models/schemas with robust validation. It attempts to be as pythonic as possible and exposes the pymongo cursor objects whenever possible. The code is available for download `at github `_. Tutorials and usage examples are also available at GitHub. MongoKit The `MongoKit `_ framework is an ORM-like layer on top of PyMongo. There is also a MongoKit `google group `_. Ming `Ming `_ (the Merciless) is a library that allows you to enforce schemas on a MongoDB database in your Python application. It was developed by `SourceForge `_ in the course of their migration to MongoDB. See the `introductory blog post `_ for more details. MongoAlchemy `MongoAlchemy `_ is another ORM-like layer on top of PyMongo. Its API is inspired by `SQLAlchemy `_. The code is available `on github `_; for more information, see `the tutorial `_. MongoEngine `MongoEngine `_ is another ORM-like layer on top of PyMongo. It allows you to define schemas for documents and query collections using syntax inspired by the Django ORM. The code is available on `github `_; for more information, see the `tutorial `_. Minimongo `minimongo `_ is a lightweight, pythonic interface to MongoDB. It retains pymongo's query and update API, and provides a number of additional features, including a simple document-oriented interface, connection pooling, index management, and collection & database naming helpers. The `source is on github `_. Manga `Manga `_ aims to be a simpler ORM-like layer on top of PyMongo. The syntax for defining schema is inspired be the Django ORM, but Pymongo's query language is maintained. The source `is on github `_. Framework Tools --------------- This section lists tools and adapters that have been designed to work with various Python frameworks and libraries. * `Django MongoDB Engine `_ is a MongoDB database backend for Django that completely integrates with its ORM. For more information `see the tutorial `_. * `mango `_ provides MongoDB backends for Django sessions and authentication (bypassing :mod:`django.db` entirely). * `Django MongoEngine `_ is a MongoDB backend for Django, an `example: `_. For more information ``_ * `mongodb_beaker `_ is a project to enable using MongoDB as a backend for `beaker's `_ caching / session system. `The source is on github `_. * `MongoLog `_ is a Python logging handler that stores logs in MongoDB using a capped collection. * `c5t `_ is a content-management system using TurboGears and MongoDB. * `rod.recipe.mongodb `_ is a ZC Buildout recipe for downloading and installing MongoDB. * `repoze-what-plugins-mongodb `_ is a project working to support a plugin for using MongoDB as a backend for :mod:`repoze.what`. * `mongobox `_ is a tool to run a sandboxed MongoDB instance from within a python app. * `Flask-MongoAlchemy `_ Add Flask support for MongoDB using MongoAlchemy. * `Flask-MongoKit `_ Flask extension to better integrate MongoKit into Flask. * `Flask-PyMongo `_ Flask-PyMongo bridges Flask and PyMongo. Alternative Drivers ------------------- These are alternatives to PyMongo. * `Motor `_ is a full-featured, non-blocking MongoDB driver for Python Tornado applications. * `TxMongo `_ is an asynchronous Python driver for MongoDB, although it is not currently recommended for production use. pymongo-2.6.3/doc/tutorial.rst000066400000000000000000000316071223300253600163570ustar00rootroot00000000000000Tutorial ======== .. testsetup:: from pymongo import MongoClient client = MongoClient() client.drop_database('test-database') This tutorial is intended as an introduction to working with **MongoDB** and **PyMongo**. Prerequisites ------------- Before we start, make sure that you have the **PyMongo** distribution :doc:`installed `. In the Python shell, the following should run without raising an exception: .. doctest:: >>> import pymongo This tutorial also assumes that a MongoDB instance is running on the default host and port. Assuming you have `downloaded and installed `_ MongoDB, you can start it like so: .. code-block:: bash $ mongod Making a Connection with MongoClient ------------------------------------ The first step when working with **PyMongo** is to create a :class:`~pymongo.mongo_client.MongoClient` to the running **mongod** instance. Doing so is easy: .. doctest:: >>> from pymongo import MongoClient >>> client = MongoClient() The above code will connect on the default host and port. We can also specify the host and port explicitly, as follows: .. doctest:: >>> client = MongoClient('localhost', 27017) Or use the MongoDB URI format: .. doctest:: >>> client = MongoClient('mongodb://localhost:27017/') Getting a Database ------------------ A single instance of MongoDB can support multiple independent `databases `_. When working with PyMongo you access databases using attribute style access on :class:`~pymongo.mongo_client.MongoClient` instances: .. doctest:: >>> db = client.test_database If your database name is such that using attribute style access won't work (like ``test-database``), you can use dictionary style access instead: .. doctest:: >>> db = client['test-database'] Getting a Collection -------------------- A `collection `_ is a group of documents stored in MongoDB, and can be thought of as roughly the equivalent of a table in a relational database. Getting a collection in PyMongo works the same as getting a database: .. doctest:: >>> collection = db.test_collection or (using dictionary style access): .. doctest:: >>> collection = db['test-collection'] An important note about collections (and databases) in MongoDB is that they are created lazily - none of the above commands have actually performed any operations on the MongoDB server. Collections and databases are created when the first document is inserted into them. Documents --------- Data in MongoDB is represented (and stored) using JSON-style documents. In PyMongo we use dictionaries to represent documents. As an example, the following dictionary might be used to represent a blog post: .. doctest:: >>> import datetime >>> post = {"author": "Mike", ... "text": "My first blog post!", ... "tags": ["mongodb", "python", "pymongo"], ... "date": datetime.datetime.utcnow()} Note that documents can contain native Python types (like :class:`datetime.datetime` instances) which will be automatically converted to and from the appropriate `BSON `_ types. .. todo:: link to table of Python <-> BSON types Inserting a Document -------------------- To insert a document into a collection we can use the :meth:`~pymongo.collection.Collection.insert` method: .. doctest:: >>> posts = db.posts >>> post_id = posts.insert(post) >>> post_id ObjectId('...') When a document is inserted a special key, ``"_id"``, is automatically added if the document doesn't already contain an ``"_id"`` key. The value of ``"_id"`` must be unique across the collection. :meth:`~pymongo.collection.Collection.insert` returns the value of ``"_id"`` for the inserted document. For more information, see the `documentation on _id `_. .. todo:: notes on the differences between save and insert After inserting the first document, the *posts* collection has actually been created on the server. We can verify this by listing all of the collections in our database: .. doctest:: >>> db.collection_names() [u'system.indexes', u'posts'] .. note:: The *system.indexes* collection is a special internal collection that was created automatically. Getting a Single Document With :meth:`~pymongo.collection.Collection.find_one` ------------------------------------------------------------------------------ The most basic type of query that can be performed in MongoDB is :meth:`~pymongo.collection.Collection.find_one`. This method returns a single document matching a query (or ``None`` if there are no matches). It is useful when you know there is only one matching document, or are only interested in the first match. Here we use :meth:`~pymongo.collection.Collection.find_one` to get the first document from the posts collection: .. doctest:: >>> posts.find_one() {u'date': datetime.datetime(...), u'text': u'My first blog post!', u'_id': ObjectId('...'), u'author': u'Mike', u'tags': [u'mongodb', u'python', u'pymongo']} The result is a dictionary matching the one that we inserted previously. .. note:: The returned document contains an ``"_id"``, which was automatically added on insert. :meth:`~pymongo.collection.Collection.find_one` also supports querying on specific elements that the resulting document must match. To limit our results to a document with author "Mike" we do: .. doctest:: >>> posts.find_one({"author": "Mike"}) {u'date': datetime.datetime(...), u'text': u'My first blog post!', u'_id': ObjectId('...'), u'author': u'Mike', u'tags': [u'mongodb', u'python', u'pymongo']} If we try with a different author, like "Eliot", we'll get no result: .. doctest:: >>> posts.find_one({"author": "Eliot"}) >>> .. _querying-by-objectid: Querying By ObjectId -------------------- We can also find a post by its ``_id``, which in our example is an ObjectId: .. doctest:: >>> post_id ObjectId(...) >>> posts.find_one({"_id": post_id}) {u'date': datetime.datetime(...), u'text': u'My first blog post!', u'_id': ObjectId('...'), u'author': u'Mike', u'tags': [u'mongodb', u'python', u'pymongo']} Note that an ObjectId is not the same as its string representation: .. doctest:: >>> post_id_as_str = str(post_id) >>> posts.find_one({"_id": post_id_as_str}) # No result >>> A common task in web applications is to get an ObjectId from the request URL and find the matching document. It's necessary in this case to **convert the ObjectId from a string** before passing it to ``find_one``:: from bson.objectid import ObjectId # The web framework gets post_id from the URL and passes it as a string def get(post_id): # Convert from string to ObjectId: document = client.db.collection.find_one({'_id': ObjectId(post_id)}) .. seealso:: :ref:`web-application-querying-by-objectid` A Note On Unicode Strings ------------------------- You probably noticed that the regular Python strings we stored earlier look different when retrieved from the server (e.g. u'Mike' instead of 'Mike'). A short explanation is in order. MongoDB stores data in `BSON format `_. BSON strings are UTF-8 encoded so PyMongo must ensure that any strings it stores contain only valid UTF-8 data. Regular strings () are validated and stored unaltered. Unicode strings () are encoded UTF-8 first. The reason our example string is represented in the Python shell as u'Mike' instead of 'Mike' is that PyMongo decodes each BSON string to a Python unicode string, not a regular str. `You can read more about Python unicode strings here `_. Bulk Inserts ------------ In order to make querying a little more interesting, let's insert a few more documents. In addition to inserting a single document, we can also perform *bulk insert* operations, by passing an iterable as the first argument to :meth:`~pymongo.collection.Collection.insert`. This will insert each document in the iterable, sending only a single command to the server: .. doctest:: >>> new_posts = [{"author": "Mike", ... "text": "Another post!", ... "tags": ["bulk", "insert"], ... "date": datetime.datetime(2009, 11, 12, 11, 14)}, ... {"author": "Eliot", ... "title": "MongoDB is fun", ... "text": "and pretty easy too!", ... "date": datetime.datetime(2009, 11, 10, 10, 45)}] >>> posts.insert(new_posts) [ObjectId('...'), ObjectId('...')] There are a couple of interesting things to note about this example: - The call to :meth:`~pymongo.collection.Collection.insert` now returns two :class:`~bson.objectid.ObjectId` instances, one for each inserted document. - ``new_posts[1]`` has a different "shape" than the other posts - there is no ``"tags"`` field and we've added a new field, ``"title"``. This is what we mean when we say that MongoDB is *schema-free*. Querying for More Than One Document ----------------------------------- To get more than a single document as the result of a query we use the :meth:`~pymongo.collection.Collection.find` method. :meth:`~pymongo.collection.Collection.find` returns a :class:`~pymongo.cursor.Cursor` instance, which allows us to iterate over all matching documents. For example, we can iterate over every document in the ``posts`` collection: .. doctest:: >>> for post in posts.find(): ... post ... {u'date': datetime.datetime(...), u'text': u'My first blog post!', u'_id': ObjectId('...'), u'author': u'Mike', u'tags': [u'mongodb', u'python', u'pymongo']} {u'date': datetime.datetime(2009, 11, 12, 11, 14), u'text': u'Another post!', u'_id': ObjectId('...'), u'author': u'Mike', u'tags': [u'bulk', u'insert']} {u'date': datetime.datetime(2009, 11, 10, 10, 45), u'text': u'and pretty easy too!', u'_id': ObjectId('...'), u'author': u'Eliot', u'title': u'MongoDB is fun'} Just like we did with :meth:`~pymongo.collection.Collection.find_one`, we can pass a document to :meth:`~pymongo.collection.Collection.find` to limit the returned results. Here, we get only those documents whose author is "Mike": .. doctest:: >>> for post in posts.find({"author": "Mike"}): ... post ... {u'date': datetime.datetime(...), u'text': u'My first blog post!', u'_id': ObjectId('...'), u'author': u'Mike', u'tags': [u'mongodb', u'python', u'pymongo']} {u'date': datetime.datetime(2009, 11, 12, 11, 14), u'text': u'Another post!', u'_id': ObjectId('...'), u'author': u'Mike', u'tags': [u'bulk', u'insert']} Counting -------- If we just want to know how many documents match a query we can perform a :meth:`~pymongo.cursor.Cursor.count` operation instead of a full query. We can get a count of all of the documents in a collection: .. doctest:: >>> posts.count() 3 or just of those documents that match a specific query: .. doctest:: >>> posts.find({"author": "Mike"}).count() 2 Range Queries ------------- MongoDB supports many different types of `advanced queries `_. As an example, lets perform a query where we limit results to posts older than a certain date, but also sort the results by author: .. doctest:: >>> d = datetime.datetime(2009, 11, 12, 12) >>> for post in posts.find({"date": {"$lt": d}}).sort("author"): ... print post ... {u'date': datetime.datetime(2009, 11, 10, 10, 45), u'text': u'and pretty easy too!', u'_id': ObjectId('...'), u'author': u'Eliot', u'title': u'MongoDB is fun'} {u'date': datetime.datetime(2009, 11, 12, 11, 14), u'text': u'Another post!', u'_id': ObjectId('...'), u'author': u'Mike', u'tags': [u'bulk', u'insert']} Here we use the special ``"$lt"`` operator to do a range query, and also call :meth:`~pymongo.cursor.Cursor.sort` to sort the results by author. Indexing -------- To make the above query fast we can add a compound index on ``"date"`` and ``"author"``. To start, lets use the :meth:`~pymongo.cursor.Cursor.explain` method to get some information about how the query is being performed without the index: .. doctest:: >>> posts.find({"date": {"$lt": d}}).sort("author").explain()["cursor"] u'BasicCursor' >>> posts.find({"date": {"$lt": d}}).sort("author").explain()["nscanned"] 3 We can see that the query is using the *BasicCursor* and scanning over all 3 documents in the collection. Now let's add a compound index and look at the same information: .. doctest:: >>> from pymongo import ASCENDING, DESCENDING >>> posts.create_index([("date", DESCENDING), ("author", ASCENDING)]) u'date_-1_author_1' >>> posts.find({"date": {"$lt": d}}).sort("author").explain()["cursor"] u'BtreeCursor date_-1_author_1' >>> posts.find({"date": {"$lt": d}}).sort("author").explain()["nscanned"] 2 Now the query is using a *BtreeCursor* (the index) and only scanning over the 2 matching documents. .. seealso:: The MongoDB documentation on `indexes `_ pymongo-2.6.3/gridfs/000077500000000000000000000000001223300253600144645ustar00rootroot00000000000000pymongo-2.6.3/gridfs/__init__.py000066400000000000000000000262251223300253600166040ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """GridFS is a specification for storing large objects in Mongo. The :mod:`gridfs` package is an implementation of GridFS on top of :mod:`pymongo`, exposing a file-like interface. .. mongodoc:: gridfs """ from gridfs.errors import (NoFile, UnsupportedAPI) from gridfs.grid_file import (GridIn, GridOut) from pymongo import (ASCENDING, DESCENDING) from pymongo.database import Database class GridFS(object): """An instance of GridFS on top of a single Database. """ def __init__(self, database, collection="fs"): """Create a new instance of :class:`GridFS`. Raises :class:`TypeError` if `database` is not an instance of :class:`~pymongo.database.Database`. :Parameters: - `database`: database to use - `collection` (optional): root collection to use .. versionadded:: 1.6 The `collection` parameter. .. mongodoc:: gridfs """ if not isinstance(database, Database): raise TypeError("database must be an instance of Database") self.__database = database self.__collection = database[collection] self.__files = self.__collection.files self.__chunks = self.__collection.chunks connection = database.connection if not hasattr(connection, 'is_primary') or connection.is_primary: self.__chunks.ensure_index([("files_id", ASCENDING), ("n", ASCENDING)], unique=True) def new_file(self, **kwargs): """Create a new file in GridFS. Returns a new :class:`~gridfs.grid_file.GridIn` instance to which data can be written. Any keyword arguments will be passed through to :meth:`~gridfs.grid_file.GridIn`. If the ``"_id"`` of the file is manually specified, it must not already exist in GridFS. Otherwise :class:`~gridfs.errors.FileExists` is raised. :Parameters: - `**kwargs` (optional): keyword arguments for file creation .. versionadded:: 1.6 """ return GridIn(self.__collection, **kwargs) def put(self, data, **kwargs): """Put data in GridFS as a new file. Equivalent to doing:: try: f = new_file(**kwargs) f.write(data) finally f.close() `data` can be either an instance of :class:`str` (:class:`bytes` in python 3) or a file-like object providing a :meth:`read` method. If an `encoding` keyword argument is passed, `data` can also be a :class:`unicode` (:class:`str` in python 3) instance, which will be encoded as `encoding` before being written. Any keyword arguments will be passed through to the created file - see :meth:`~gridfs.grid_file.GridIn` for possible arguments. Returns the ``"_id"`` of the created file. If the ``"_id"`` of the file is manually specified, it must not already exist in GridFS. Otherwise :class:`~gridfs.errors.FileExists` is raised. :Parameters: - `data`: data to be written as a file. - `**kwargs` (optional): keyword arguments for file creation .. versionadded:: 1.9 The ability to write :class:`unicode`, if an `encoding` has been specified as a keyword argument. .. versionadded:: 1.6 """ grid_file = GridIn(self.__collection, **kwargs) # Start a request - necessary if w=0, harmless otherwise request = self.__collection.database.connection.start_request() try: try: grid_file.write(data) finally: grid_file.close() finally: # Ensure request is ended even if close() throws error request.end() return grid_file._id def get(self, file_id): """Get a file from GridFS by ``"_id"``. Returns an instance of :class:`~gridfs.grid_file.GridOut`, which provides a file-like interface for reading. :Parameters: - `file_id`: ``"_id"`` of the file to get .. versionadded:: 1.6 """ return GridOut(self.__collection, file_id) def get_version(self, filename=None, version=-1, **kwargs): """Get a file from GridFS by ``"filename"`` or metadata fields. Returns a version of the file in GridFS whose filename matches `filename` and whose metadata fields match the supplied keyword arguments, as an instance of :class:`~gridfs.grid_file.GridOut`. Version numbering is a convenience atop the GridFS API provided by MongoDB. If more than one file matches the query (either by `filename` alone, by metadata fields, or by a combination of both), then version ``-1`` will be the most recently uploaded matching file, ``-2`` the second most recently uploaded, etc. Version ``0`` will be the first version uploaded, ``1`` the second version, etc. So if three versions have been uploaded, then version ``0`` is the same as version ``-3``, version ``1`` is the same as version ``-2``, and version ``2`` is the same as version ``-1``. Raises :class:`~gridfs.errors.NoFile` if no such version of that file exists. An index on ``{filename: 1, uploadDate: -1}`` will automatically be created when this method is called the first time. :Parameters: - `filename`: ``"filename"`` of the file to get, or `None` - `version` (optional): version of the file to get (defaults to -1, the most recent version uploaded) - `**kwargs` (optional): find files by custom metadata. .. versionchanged:: 1.11 `filename` defaults to None; .. versionadded:: 1.11 Accept keyword arguments to find files by custom metadata. .. versionadded:: 1.9 """ connection = self.__database.connection if not hasattr(connection, 'is_primary') or connection.is_primary: self.__files.ensure_index([("filename", ASCENDING), ("uploadDate", DESCENDING)]) query = kwargs if filename is not None: query["filename"] = filename cursor = self.__files.find(query) if version < 0: skip = abs(version) - 1 cursor.limit(-1).skip(skip).sort("uploadDate", DESCENDING) else: cursor.limit(-1).skip(version).sort("uploadDate", ASCENDING) try: grid_file = cursor.next() return GridOut(self.__collection, file_document=grid_file) except StopIteration: raise NoFile("no version %d for filename %r" % (version, filename)) def get_last_version(self, filename=None, **kwargs): """Get the most recent version of a file in GridFS by ``"filename"`` or metadata fields. Equivalent to calling :meth:`get_version` with the default `version` (``-1``). :Parameters: - `filename`: ``"filename"`` of the file to get, or `None` - `**kwargs` (optional): find files by custom metadata. .. versionchanged:: 1.11 `filename` defaults to None; .. versionadded:: 1.11 Accept keyword arguments to find files by custom metadata. See :meth:`get_version`. .. versionadded:: 1.6 """ return self.get_version(filename=filename, **kwargs) # TODO add optional safe mode for chunk removal? def delete(self, file_id): """Delete a file from GridFS by ``"_id"``. Removes all data belonging to the file with ``"_id"``: `file_id`. .. warning:: Any processes/threads reading from the file while this method is executing will likely see an invalid/corrupt file. Care should be taken to avoid concurrent reads to a file while it is being deleted. .. note:: Deletes of non-existent files are considered successful since the end result is the same: no file with that _id remains. :Parameters: - `file_id`: ``"_id"`` of the file to delete .. versionadded:: 1.6 """ self.__files.remove({"_id": file_id}, **self.__files._get_wc_override()) self.__chunks.remove({"files_id": file_id}) def list(self): """List the names of all files stored in this instance of :class:`GridFS`. .. versionchanged:: 1.6 Removed the `collection` argument. """ return self.__files.distinct("filename") def exists(self, document_or_id=None, **kwargs): """Check if a file exists in this instance of :class:`GridFS`. The file to check for can be specified by the value of its ``_id`` key, or by passing in a query document. A query document can be passed in as dictionary, or by using keyword arguments. Thus, the following three calls are equivalent: >>> fs.exists(file_id) >>> fs.exists({"_id": file_id}) >>> fs.exists(_id=file_id) As are the following two calls: >>> fs.exists({"filename": "mike.txt"}) >>> fs.exists(filename="mike.txt") And the following two: >>> fs.exists({"foo": {"$gt": 12}}) >>> fs.exists(foo={"$gt": 12}) Returns ``True`` if a matching file exists, ``False`` otherwise. Calls to :meth:`exists` will not automatically create appropriate indexes; application developers should be sure to create indexes if needed and as appropriate. :Parameters: - `document_or_id` (optional): query document, or _id of the document to check for - `**kwargs` (optional): keyword arguments are used as a query document, if they're present. .. versionadded:: 1.8 """ if kwargs: return self.__files.find_one(kwargs, ["_id"]) is not None return self.__files.find_one(document_or_id, ["_id"]) is not None def open(self, *args, **kwargs): """No longer supported. .. versionchanged:: 1.6 The open method is no longer supported. """ raise UnsupportedAPI("The open method is no longer supported.") def remove(self, *args, **kwargs): """No longer supported. .. versionchanged:: 1.6 The remove method is no longer supported. """ raise UnsupportedAPI("The remove method is no longer supported. " "Please use the delete method instead.") pymongo-2.6.3/gridfs/errors.py000066400000000000000000000031021223300253600163460ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Exceptions raised by the :mod:`gridfs` package""" from pymongo.errors import PyMongoError class GridFSError(PyMongoError): """Base class for all GridFS exceptions. .. versionadded:: 1.5 """ class CorruptGridFile(GridFSError): """Raised when a file in :class:`~gridfs.GridFS` is malformed. """ class NoFile(GridFSError): """Raised when trying to read from a non-existent file. .. versionadded:: 1.6 """ class FileExists(GridFSError): """Raised when trying to create a file that already exists. .. versionadded:: 1.7 """ class UnsupportedAPI(GridFSError): """Raised when trying to use the old GridFS API. In version 1.6 of the PyMongo distribution there were backwards incompatible changes to the GridFS API. Upgrading shouldn't be difficult, but the old API is no longer supported (with no deprecation period). This exception will be raised when attempting to use unsupported constructs from the old API. .. versionadded:: 1.6 """ pymongo-2.6.3/gridfs/grid_file.py000066400000000000000000000507311223300253600167700ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tools for representing files stored in GridFS.""" import datetime import math import os from bson.binary import Binary from bson.objectid import ObjectId from bson.py3compat import b, binary_type, string_types, text_type, StringIO from gridfs.errors import (CorruptGridFile, FileExists, NoFile, UnsupportedAPI) from pymongo import ASCENDING from pymongo.collection import Collection from pymongo.errors import DuplicateKeyError try: _SEEK_SET = os.SEEK_SET _SEEK_CUR = os.SEEK_CUR _SEEK_END = os.SEEK_END # before 2.5 except AttributeError: _SEEK_SET = 0 _SEEK_CUR = 1 _SEEK_END = 2 EMPTY = b("") NEWLN = b("\n") """Default chunk size, in bytes.""" DEFAULT_CHUNK_SIZE = 256 * 1024 def _create_property(field_name, docstring, read_only=False, closed_only=False): """Helper for creating properties to read/write to files. """ def getter(self): if closed_only and not self._closed: raise AttributeError("can only get %r on a closed file" % field_name) # Protect against PHP-237 if field_name == 'length': return self._file.get(field_name, 0) return self._file.get(field_name, None) def setter(self, value): if self._closed: self._coll.files.update({"_id": self._file["_id"]}, {"$set": {field_name: value}}, **self._coll._get_wc_override()) self._file[field_name] = value if read_only: docstring = docstring + "\n\nThis attribute is read-only." elif closed_only: docstring = "%s\n\n%s" % (docstring, "This attribute is read-only and " "can only be read after :meth:`close` " "has been called.") if not read_only and not closed_only: return property(getter, setter, doc=docstring) return property(getter, doc=docstring) class GridIn(object): """Class to write data to GridFS. """ def __init__(self, root_collection, **kwargs): """Write a file to GridFS Application developers should generally not need to instantiate this class directly - instead see the methods provided by :class:`~gridfs.GridFS`. Raises :class:`TypeError` if `root_collection` is not an instance of :class:`~pymongo.collection.Collection`. Any of the file level options specified in the `GridFS Spec `_ may be passed as keyword arguments. Any additional keyword arguments will be set as additional fields on the file document. Valid keyword arguments include: - ``"_id"``: unique ID for this file (default: :class:`~bson.objectid.ObjectId`) - this ``"_id"`` must not have already been used for another file - ``"filename"``: human name for the file - ``"contentType"`` or ``"content_type"``: valid mime-type for the file - ``"chunkSize"`` or ``"chunk_size"``: size of each of the chunks, in bytes (default: 256 kb) - ``"encoding"``: encoding used for this file. In Python 2, any :class:`unicode` that is written to the file will be converted to a :class:`str`. In Python 3, any :class:`str` that is written to the file will be converted to :class:`bytes`. If you turn off write-acknowledgment for performance reasons, it is critical to wrap calls to :meth:`write` and :meth:`close` within a single request: >>> from pymongo import MongoClient >>> from gridfs import GridFS >>> client = MongoClient(w=0) # turn off write acknowledgment >>> fs = GridFS(client) >>> gridin = fs.new_file() >>> request = client.start_request() >>> try: ... for i in range(10): ... gridin.write('foo') ... gridin.close() ... finally: ... request.end() In Python 2.5 and later this code can be simplified with a with-statement, see :doc:`/examples/requests` for more information. :Parameters: - `root_collection`: root collection to write to - `**kwargs` (optional): file level options (see above) """ if not isinstance(root_collection, Collection): raise TypeError("root_collection must be an " "instance of Collection") # Handle alternative naming if "content_type" in kwargs: kwargs["contentType"] = kwargs.pop("content_type") if "chunk_size" in kwargs: kwargs["chunkSize"] = kwargs.pop("chunk_size") # Defaults kwargs["_id"] = kwargs.get("_id", ObjectId()) kwargs["chunkSize"] = kwargs.get("chunkSize", DEFAULT_CHUNK_SIZE) root_collection.chunks.ensure_index([("files_id", ASCENDING), ("n", ASCENDING)], unique=True) object.__setattr__(self, "_coll", root_collection) object.__setattr__(self, "_chunks", root_collection.chunks) object.__setattr__(self, "_file", kwargs) object.__setattr__(self, "_buffer", StringIO()) object.__setattr__(self, "_position", 0) object.__setattr__(self, "_chunk_number", 0) object.__setattr__(self, "_closed", False) @property def closed(self): """Is this file closed? """ return self._closed _id = _create_property("_id", "The ``'_id'`` value for this file.", read_only=True) filename = _create_property("filename", "Name of this file.") name = _create_property("filename", "Alias for `filename`.") content_type = _create_property("contentType", "Mime-type for this file.") length = _create_property("length", "Length (in bytes) of this file.", closed_only=True) chunk_size = _create_property("chunkSize", "Chunk size for this file.", read_only=True) upload_date = _create_property("uploadDate", "Date that this file was uploaded.", closed_only=True) md5 = _create_property("md5", "MD5 of the contents of this file " "(generated on the server).", closed_only=True) def __getattr__(self, name): if name in self._file: return self._file[name] raise AttributeError("GridIn object has no attribute '%s'" % name) def __setattr__(self, name, value): # For properties of this instance like _buffer, or descriptors set on # the class like filename, use regular __setattr__ if name in self.__dict__ or name in self.__class__.__dict__: object.__setattr__(self, name, value) else: # All other attributes are part of the document in db.fs.files. # Store them to be sent to server on close() or if closed, send # them now. self._file[name] = value if self._closed: self._coll.files.update({"_id": self._file["_id"]}, {"$set": {name: value}}, **self._coll._get_wc_override()) def __flush_data(self, data): """Flush `data` to a chunk. """ if not data: return assert(len(data) <= self.chunk_size) chunk = {"files_id": self._file["_id"], "n": self._chunk_number, "data": Binary(data)} try: self._chunks.insert(chunk) except DuplicateKeyError: self._raise_file_exists(self._file['_id']) self._chunk_number += 1 self._position += len(data) def __flush_buffer(self): """Flush the buffer contents out to a chunk. """ self.__flush_data(self._buffer.getvalue()) self._buffer.close() self._buffer = StringIO() def __flush(self): """Flush the file to the database. """ try: self.__flush_buffer() db = self._coll.database # See PYTHON-417, "Sharded GridFS fails with exception: chunks out # of order." Inserts via mongos, even if they use a single # connection, can succeed out-of-order due to the writebackListener. # We mustn't call "filemd5" until all inserts are complete, which # we ensure by calling getLastError (and ignoring the result). db.error() md5 = db.command( "filemd5", self._id, root=self._coll.name)["md5"] self._file["md5"] = md5 self._file["length"] = self._position self._file["uploadDate"] = datetime.datetime.utcnow() return self._coll.files.insert(self._file, **self._coll._get_wc_override()) except DuplicateKeyError: self._raise_file_exists(self._id) def _raise_file_exists(self, file_id): """Raise a FileExists exception for the given file_id.""" raise FileExists("file with _id %r already exists" % file_id) def close(self): """Flush the file and close it. A closed file cannot be written any more. Calling :meth:`close` more than once is allowed. """ if not self._closed: self.__flush() object.__setattr__(self, "_closed", True) def write(self, data): """Write data to the file. There is no return value. `data` can be either a string of bytes or a file-like object (implementing :meth:`read`). If the file has an :attr:`encoding` attribute, `data` can also be a :class:`unicode` (:class:`str` in python 3) instance, which will be encoded as :attr:`encoding` before being written. Due to buffering, the data may not actually be written to the database until the :meth:`close` method is called. Raises :class:`ValueError` if this file is already closed. Raises :class:`TypeError` if `data` is not an instance of :class:`str` (:class:`bytes` in python 3), a file-like object, or an instance of :class:`unicode` (:class:`str` in python 3). Unicode data is only allowed if the file has an :attr:`encoding` attribute. :Parameters: - `data`: string of bytes or file-like object to be written to the file .. versionadded:: 1.9 The ability to write :class:`unicode`, if the file has an :attr:`encoding` attribute. """ if self._closed: raise ValueError("cannot write to a closed file") try: # file-like read = data.read except AttributeError: # string if not isinstance(data, string_types): raise TypeError("can only write strings or file-like objects") if isinstance(data, unicode): try: data = data.encode(self.encoding) except AttributeError: raise TypeError("must specify an encoding for file in " "order to write %s" % (text_type.__name__,)) read = StringIO(data).read if self._buffer.tell() > 0: # Make sure to flush only when _buffer is complete space = self.chunk_size - self._buffer.tell() if space: to_write = read(space) self._buffer.write(to_write) if len(to_write) < space: return # EOF or incomplete self.__flush_buffer() to_write = read(self.chunk_size) while to_write and len(to_write) == self.chunk_size: self.__flush_data(to_write) to_write = read(self.chunk_size) self._buffer.write(to_write) def writelines(self, sequence): """Write a sequence of strings to the file. Does not add seperators. """ for line in sequence: self.write(line) def __enter__(self): """Support for the context manager protocol. """ return self def __exit__(self, exc_type, exc_val, exc_tb): """Support for the context manager protocol. Close the file and allow exceptions to propagate. """ self.close() # propagate exceptions return False class GridOut(object): """Class to read data out of GridFS. """ def __init__(self, root_collection, file_id=None, file_document=None): """Read a file from GridFS Application developers should generally not need to instantiate this class directly - instead see the methods provided by :class:`~gridfs.GridFS`. Either `file_id` or `file_document` must be specified, `file_document` will be given priority if present. Raises :class:`TypeError` if `root_collection` is not an instance of :class:`~pymongo.collection.Collection`. :Parameters: - `root_collection`: root collection to read from - `file_id`: value of ``"_id"`` for the file to read - `file_document`: file document from `root_collection.files` .. versionadded:: 1.9 The `file_document` parameter. """ if not isinstance(root_collection, Collection): raise TypeError("root_collection must be an " "instance of Collection") self.__chunks = root_collection.chunks files = root_collection.files self._file = file_document or files.find_one({"_id": file_id}) if not self._file: raise NoFile("no file in gridfs collection %r with _id %r" % (files, file_id)) self.__buffer = EMPTY self.__position = 0 _id = _create_property("_id", "The ``'_id'`` value for this file.", True) filename = _create_property("filename", "Name of this file.", True) name = _create_property("filename", "Alias for `filename`.", True) content_type = _create_property("contentType", "Mime-type for this file.", True) length = _create_property("length", "Length (in bytes) of this file.", True) chunk_size = _create_property("chunkSize", "Chunk size for this file.", True) upload_date = _create_property("uploadDate", "Date that this file was first uploaded.", True) aliases = _create_property("aliases", "List of aliases for this file.", True) metadata = _create_property("metadata", "Metadata attached to this file.", True) md5 = _create_property("md5", "MD5 of the contents of this file " "(generated on the server).", True) def __getattr__(self, name): if name in self._file: return self._file[name] raise AttributeError("GridOut object has no attribute '%s'" % name) def read(self, size=-1): """Read at most `size` bytes from the file (less if there isn't enough data). The bytes are returned as an instance of :class:`str` (:class:`bytes` in python 3). If `size` is negative or omitted all data is read. :Parameters: - `size` (optional): the number of bytes to read """ if size == 0: return "" remainder = int(self.length) - self.__position if size < 0 or size > remainder: size = remainder received = len(self.__buffer) chunk_number = int((received + self.__position) / self.chunk_size) chunks = [] while received < size: chunk = self.__chunks.find_one({"files_id": self._id, "n": chunk_number}) if not chunk: raise CorruptGridFile("no chunk #%d" % chunk_number) if received: chunk_data = chunk["data"] else: chunk_data = chunk["data"][self.__position % self.chunk_size:] received += len(chunk_data) chunks.append(chunk_data) chunk_number += 1 data = EMPTY.join([self.__buffer] + chunks) self.__position += size to_return = data[:size] self.__buffer = data[size:] return to_return def readline(self, size=-1): """Read one line or up to `size` bytes from the file. :Parameters: - `size` (optional): the maximum number of bytes to read .. versionadded:: 1.9 """ bytes = EMPTY while len(bytes) != size: byte = self.read(1) bytes += byte if byte == EMPTY or byte == NEWLN: break return bytes def tell(self): """Return the current position of this file. """ return self.__position def seek(self, pos, whence=_SEEK_SET): """Set the current position of this file. :Parameters: - `pos`: the position (or offset if using relative positioning) to seek to - `whence` (optional): where to seek from. :attr:`os.SEEK_SET` (``0``) for absolute file positioning, :attr:`os.SEEK_CUR` (``1``) to seek relative to the current position, :attr:`os.SEEK_END` (``2``) to seek relative to the file's end. """ if whence == _SEEK_SET: new_pos = pos elif whence == _SEEK_CUR: new_pos = self.__position + pos elif whence == _SEEK_END: new_pos = int(self.length) + pos else: raise IOError(22, "Invalid value for `whence`") if new_pos < 0: raise IOError(22, "Invalid value for `pos` - must be positive") self.__position = new_pos self.__buffer = EMPTY def __iter__(self): """Return an iterator over all of this file's data. The iterator will return chunk-sized instances of :class:`str` (:class:`bytes` in python 3). This can be useful when serving files using a webserver that handles such an iterator efficiently. """ return GridOutIterator(self, self.__chunks) def close(self): """Make GridOut more generically file-like.""" pass def __enter__(self): """Makes it possible to use :class:`GridOut` files with the context manager protocol. """ return self def __exit__(self, exc_type, exc_val, exc_tb): """Makes it possible to use :class:`GridOut` files with the context manager protocol. """ return False class GridOutIterator(object): def __init__(self, grid_out, chunks): self.__id = grid_out._id self.__chunks = chunks self.__current_chunk = 0 self.__max_chunk = math.ceil(float(grid_out.length) / grid_out.chunk_size) def __iter__(self): return self def next(self): if self.__current_chunk >= self.__max_chunk: raise StopIteration chunk = self.__chunks.find_one({"files_id": self.__id, "n": self.__current_chunk}) if not chunk: raise CorruptGridFile("no chunk #%d" % self.__current_chunk) self.__current_chunk += 1 return binary_type(chunk["data"]) class GridFile(object): """No longer supported. .. versionchanged:: 1.6 The GridFile class is no longer supported. """ def __init__(self, *args, **kwargs): raise UnsupportedAPI("The GridFile class is no longer supported. " "Please use GridIn or GridOut instead.") pymongo-2.6.3/pymongo.egg-info/000077500000000000000000000000001223300253600163705ustar00rootroot00000000000000pymongo-2.6.3/pymongo.egg-info/PKG-INFO000066400000000000000000000124641223300253600174740ustar00rootroot00000000000000Metadata-Version: 1.1 Name: pymongo Version: 2.6.3 Summary: Python driver for MongoDB Home-page: http://github.com/mongodb/mongo-python-driver Author: Bernie Hackett Author-email: bernie@10gen.com License: Apache License, Version 2.0 Description: ======= PyMongo ======= :Info: See `the mongo site `_ for more information. See `github `_ for the latest source. :Author: Mike Dirolf :Maintainer: Bernie Hackett About ===== The PyMongo distribution contains tools for interacting with MongoDB database from Python. The ``bson`` package is an implementation of the `BSON format `_ for Python. The ``pymongo`` package is a native Python driver for MongoDB. The ``gridfs`` package is a `gridfs `_ implementation on top of ``pymongo``. Issues / Questions / Feedback ============================= Any issues with, questions about, or feedback for PyMongo should be sent to the mongodb-user list on Google Groups. For confirmed issues or feature requests, open a case on `jira `_. Please do not e-mail any of the PyMongo developers directly with issues or questions - you're more likely to get an answer on the list. Installation ============ If you have `distribute `_ installed you should be able to do **easy_install pymongo** to install PyMongo. Otherwise you can download the project source and do **python setup.py install** to install. Dependencies ============ The PyMongo distribution is supported and tested on Python 2.x (where x >= 4) and Python 3.x (where x >= 1). PyMongo versions <= 1.3 also supported Python 2.3, but that is no longer supported. Additional dependencies are: - (to generate documentation) sphinx_ - (to auto-discover tests) `nose `_ Examples ======== Here's a basic example (for more see the *examples* section of the docs): .. code-block:: pycon >>> import pymongo >>> client = pymongo.MongoClient("localhost", 27017) >>> db = client.test >>> db.name u'test' >>> db.my_collection Collection(Database(MongoClient('localhost', 27017), u'test'), u'my_collection') >>> db.my_collection.save({"x": 10}) ObjectId('4aba15ebe23f6b53b0000000') >>> db.my_collection.save({"x": 8}) ObjectId('4aba160ee23f6b543e000000') >>> db.my_collection.save({"x": 11}) ObjectId('4aba160ee23f6b543e000002') >>> db.my_collection.find_one() {u'x': 10, u'_id': ObjectId('4aba15ebe23f6b53b0000000')} >>> for item in db.my_collection.find(): ... print item["x"] ... 10 8 11 >>> db.my_collection.create_index("x") u'x_1' >>> for item in db.my_collection.find().sort("x", pymongo.ASCENDING): ... print item["x"] ... 8 10 11 >>> [item["x"] for item in db.my_collection.find().limit(2).skip(1)] [8, 11] Documentation ============= You will need sphinx_ installed to generate the documentation. Documentation can be generated by running **python setup.py doc**. Generated documentation can be found in the *doc/build/html/* directory. Testing ======= The easiest way to run the tests is to install `nose `_ (**easy_install nose**) and run **nosetests** or **python setup.py test** in the root of the distribution. Tests are located in the *test/* directory. .. _sphinx: http://sphinx.pocoo.org/ Keywords: mongo,mongodb,pymongo,gridfs,bson Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Apache Software License Classifier: Operating System :: MacOS :: MacOS X Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: POSIX Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.4 Classifier: Programming Language :: Python :: 2.5 Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.1 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: Jython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Topic :: Database pymongo-2.6.3/pymongo.egg-info/SOURCES.txt000066400000000000000000000066731223300253600202700ustar00rootroot00000000000000LICENSE MANIFEST.in README.rst distribute_setup.py setup.py bson/__init__.py bson/_cbsonmodule.c bson/_cbsonmodule.h bson/binary.py bson/buffer.c bson/buffer.h bson/code.py bson/dbref.py bson/encoding_helpers.c bson/encoding_helpers.h bson/errors.py bson/json_util.py bson/max_key.py bson/min_key.py bson/objectid.py bson/py3compat.py bson/son.py bson/time64.c bson/time64.h bson/time64_config.h bson/time64_limits.h bson/timestamp.py bson/tz_util.py doc/__init__.py doc/changelog.rst doc/conf.py doc/contributors.rst doc/faq.rst doc/index.rst doc/installation.rst doc/mongo_extensions.py doc/python3.rst doc/tools.rst doc/tutorial.rst doc/api/index.rst doc/api/bson/binary.rst doc/api/bson/code.rst doc/api/bson/dbref.rst doc/api/bson/errors.rst doc/api/bson/index.rst doc/api/bson/json_util.rst doc/api/bson/max_key.rst doc/api/bson/min_key.rst doc/api/bson/objectid.rst doc/api/bson/son.rst doc/api/bson/timestamp.rst doc/api/bson/tz_util.rst doc/api/gridfs/errors.rst doc/api/gridfs/grid_file.rst doc/api/gridfs/index.rst doc/api/pymongo/collection.rst doc/api/pymongo/connection.rst doc/api/pymongo/cursor.rst doc/api/pymongo/cursor_manager.rst doc/api/pymongo/database.rst doc/api/pymongo/errors.rst doc/api/pymongo/index.rst doc/api/pymongo/master_slave_connection.rst doc/api/pymongo/message.rst doc/api/pymongo/mongo_client.rst doc/api/pymongo/mongo_replica_set_client.rst doc/api/pymongo/pool.rst doc/api/pymongo/replica_set_connection.rst doc/api/pymongo/son_manipulator.rst doc/api/pymongo/uri_parser.rst doc/examples/aggregation.rst doc/examples/authentication.rst doc/examples/custom_type.rst doc/examples/geo.rst doc/examples/gevent.rst doc/examples/gridfs.rst doc/examples/high_availability.rst doc/examples/index.rst doc/examples/requests.rst gridfs/__init__.py gridfs/errors.py gridfs/grid_file.py pymongo/__init__.py pymongo/_cmessagemodule.c pymongo/auth.py pymongo/collection.py pymongo/common.py pymongo/connection.py pymongo/cursor.py pymongo/cursor_manager.py pymongo/database.py pymongo/errors.py pymongo/helpers.py pymongo/master_slave_connection.py pymongo/message.py pymongo/mongo_client.py pymongo/mongo_replica_set_client.py pymongo/pool.py pymongo/read_preferences.py pymongo/replica_set_connection.py pymongo/son_manipulator.py pymongo/ssl_match_hostname.py pymongo/thread_util.py pymongo/uri_parser.py pymongo.egg-info/PKG-INFO pymongo.egg-info/SOURCES.txt pymongo.egg-info/dependency_links.txt pymongo.egg-info/top_level.txt test/__init__.py test/qcheck.py test/test_auth.py test/test_binary.py test/test_bson.py test/test_client.py test/test_code.py test/test_collection.py test/test_common.py test/test_cursor.py test/test_database.py test/test_dbref.py test/test_errors.py test/test_grid_file.py test/test_gridfs.py test/test_json_util.py test/test_legacy_connections.py test/test_master_slave_connection.py test/test_objectid.py test/test_pooling.py test/test_pooling_base.py test/test_pooling_gevent.py test/test_pymongo.py test/test_read_preferences.py test/test_replica_set_client.py test/test_son.py test/test_son_manipulator.py test/test_ssl.py test/test_thread_util.py test/test_threads.py test/test_threads_replica_set_client.py test/test_timestamp.py test/test_uri_parser.py test/utils.py test/version.py test/certificates/ca.pem test/certificates/client.pem test/high_availability/ha_tools.py test/high_availability/test_ha.py test/mod_wsgi_test/test_client.py test/slow/test_high_concurrency.py tools/README.rst tools/benchmark.py tools/clean.py tools/fail_if_no_c.pypymongo-2.6.3/pymongo.egg-info/dependency_links.txt000066400000000000000000000000011223300253600224360ustar00rootroot00000000000000 pymongo-2.6.3/pymongo.egg-info/top_level.txt000066400000000000000000000000241223300253600211160ustar00rootroot00000000000000bson pymongo gridfs pymongo-2.6.3/pymongo/000077500000000000000000000000001223300253600146765ustar00rootroot00000000000000pymongo-2.6.3/pymongo/__init__.py000066400000000000000000000052321223300253600170110ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Python driver for MongoDB.""" ASCENDING = 1 """Ascending sort order.""" DESCENDING = -1 """Descending sort order.""" GEO2D = "2d" """Index specifier for a 2-dimensional `geospatial index`_. .. versionadded:: 1.5.1 .. note:: Geo-spatial indexing requires server version **>= 1.3.3**. .. _geospatial index: http://docs.mongodb.org/manual/core/geospatial-indexes/ """ GEOHAYSTACK = "geoHaystack" """Index specifier for a 2-dimensional `haystack index`_. .. versionadded:: 2.1 .. note:: Geo-spatial indexing requires server version **>= 1.5.6**. .. _haystack index: http://docs.mongodb.org/manual/core/geospatial-indexes/#haystack-indexes """ GEOSPHERE = "2dsphere" """Index specifier for a `spherical geospatial index`_. .. versionadded:: 2.5 .. note:: 2dsphere indexing requires server version **>= 2.4.0**. .. _spherical geospatial index: http://docs.mongodb.org/manual/release-notes/2.4/#new-geospatial-indexes-with-geojson-and-improved-spherical-geometry """ HASHED = "hashed" """Index specifier for a `hashed index`_. .. versionadded:: 2.5 .. note:: hashed indexing requires server version **>= 2.4.0**. .. _hashed index: http://docs.mongodb.org/manual/release-notes/2.4/#new-hashed-index-and-sharding-with-a-hashed-shard-key """ OFF = 0 """No database profiling.""" SLOW_ONLY = 1 """Only profile slow operations.""" ALL = 2 """Profile all operations.""" version_tuple = (2, 6, 3) def get_version_string(): if isinstance(version_tuple[-1], basestring): return '.'.join(map(str, version_tuple[:-1])) + version_tuple[-1] return '.'.join(map(str, version_tuple)) version = get_version_string() """Current version of PyMongo.""" from pymongo.connection import Connection from pymongo.mongo_client import MongoClient from pymongo.mongo_replica_set_client import MongoReplicaSetClient from pymongo.replica_set_connection import ReplicaSetConnection from pymongo.read_preferences import ReadPreference def has_c(): """Is the C extension installed? .. versionadded:: 1.5 """ try: from pymongo import _cmessage return True except ImportError: return False pymongo-2.6.3/pymongo/_cmessagemodule.c000066400000000000000000000735341223300253600202120ustar00rootroot00000000000000/* * Copyright 2009-2012 10gen, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ /* * This file contains C implementations of some of the functions * needed by the message module. If possible, these implementations * should be used to speed up message creation. */ #include "Python.h" #include "_cbsonmodule.h" #include "buffer.h" struct module_state { PyObject* _cbson; }; /* See comments about module initialization in _cbsonmodule.c */ #if PY_MAJOR_VERSION >= 3 #define GETSTATE(m) ((struct module_state*)PyModule_GetState(m)) #else #define GETSTATE(m) (&_state) static struct module_state _state; #endif #if PY_MAJOR_VERSION >= 3 #define BYTES_FORMAT_STRING "y#" #else #define BYTES_FORMAT_STRING "s#" #endif #define DOC_TOO_LARGE_FMT "BSON document too large (%d bytes)" \ " - the connected server supports" \ " BSON document sizes up to %ld bytes." /* Get an error class from the pymongo.errors module. * * Returns a new ref */ static PyObject* _error(char* name) { PyObject* error; PyObject* errors = PyImport_ImportModule("pymongo.errors"); if (!errors) { return NULL; } error = PyObject_GetAttrString(errors, name); Py_DECREF(errors); return error; } /* add a lastError message on the end of the buffer. * returns 0 on failure */ static int add_last_error(PyObject* self, buffer_t buffer, int request_id, char* ns, int nslen, PyObject* args) { struct module_state *state = GETSTATE(self); int message_start; int document_start; int message_length; int document_length; PyObject* key; PyObject* value; Py_ssize_t pos = 0; PyObject* one; char *p = strchr(ns, '.'); /* Length of the database portion of ns. */ nslen = p ? (int)(p - ns) : nslen; message_start = buffer_save_space(buffer, 4); if (message_start == -1) { PyErr_NoMemory(); return 0; } if (!buffer_write_bytes(buffer, (const char*)&request_id, 4) || !buffer_write_bytes(buffer, "\x00\x00\x00\x00" /* responseTo */ "\xd4\x07\x00\x00" /* opcode */ "\x00\x00\x00\x00", /* options */ 12) || !buffer_write_bytes(buffer, ns, nslen) || /* database */ !buffer_write_bytes(buffer, ".$cmd\x00" /* collection name */ "\x00\x00\x00\x00" /* skip */ "\xFF\xFF\xFF\xFF", /* limit (-1) */ 14)) { return 0; } /* save space for length */ document_start = buffer_save_space(buffer, 4); if (document_start == -1) { PyErr_NoMemory(); return 0; } /* getlasterror: 1 */ if (!(one = PyLong_FromLong(1))) return 0; if (!write_pair(state->_cbson, buffer, "getlasterror", 12, one, 0, 4, 1)) { Py_DECREF(one); return 0; } Py_DECREF(one); /* getlasterror options */ while (PyDict_Next(args, &pos, &key, &value)) { if (!decode_and_write_pair(state->_cbson, buffer, key, value, 0, 4, 0)) { return 0; } } /* EOD */ if (!buffer_write_bytes(buffer, "\x00", 1)) { return 0; } message_length = buffer_get_position(buffer) - message_start; document_length = buffer_get_position(buffer) - document_start; memcpy(buffer_get_buffer(buffer) + message_start, &message_length, 4); memcpy(buffer_get_buffer(buffer) + document_start, &document_length, 4); return 1; } static int init_insert_buffer(buffer_t buffer, int request_id, int options, const char* coll_name, int coll_name_len) { /* Save space for message length */ int length_location = buffer_save_space(buffer, 4); if (length_location == -1) { PyErr_NoMemory(); return length_location; } if (!buffer_write_bytes(buffer, (const char*)&request_id, 4) || !buffer_write_bytes(buffer, "\x00\x00\x00\x00" "\xd2\x07\x00\x00", 8) || !buffer_write_bytes(buffer, (const char*)&options, 4) || !buffer_write_bytes(buffer, coll_name, coll_name_len + 1)) { return -1; } return length_location; } static PyObject* _cbson_insert_message(PyObject* self, PyObject* args) { /* Note: As of PyMongo 2.6, this function is no longer used. It * is being kept (with tests) for backwards compatibility with 3rd * party libraries that may currently be using it, but will likely * be removed in a future release. */ struct module_state *state = GETSTATE(self); /* NOTE just using a random number as the request_id */ int request_id = rand(); char* collection_name = NULL; int collection_name_length; PyObject* docs; PyObject* doc; PyObject* iterator; int before, cur_size, max_size = 0; int options = 0; unsigned char check_keys; unsigned char safe; unsigned char continue_on_error; unsigned char uuid_subtype; PyObject* last_error_args; buffer_t buffer; int length_location, message_length; PyObject* result; if (!PyArg_ParseTuple(args, "et#ObbObb", "utf-8", &collection_name, &collection_name_length, &docs, &check_keys, &safe, &last_error_args, &continue_on_error, &uuid_subtype)) { return NULL; } if (continue_on_error) { options += 1; } buffer = buffer_new(); if (!buffer) { PyErr_NoMemory(); PyMem_Free(collection_name); return NULL; } length_location = init_insert_buffer(buffer, request_id, options, collection_name, collection_name_length); if (length_location == -1) { PyMem_Free(collection_name); buffer_free(buffer); return NULL; } iterator = PyObject_GetIter(docs); if (iterator == NULL) { PyObject* InvalidOperation = _error("InvalidOperation"); if (InvalidOperation) { PyErr_SetString(InvalidOperation, "input is not iterable"); Py_DECREF(InvalidOperation); } buffer_free(buffer); PyMem_Free(collection_name); return NULL; } while ((doc = PyIter_Next(iterator)) != NULL) { before = buffer_get_position(buffer); if (!write_dict(state->_cbson, buffer, doc, check_keys, uuid_subtype, 1)) { Py_DECREF(doc); Py_DECREF(iterator); buffer_free(buffer); PyMem_Free(collection_name); return NULL; } Py_DECREF(doc); cur_size = buffer_get_position(buffer) - before; max_size = (cur_size > max_size) ? cur_size : max_size; } Py_DECREF(iterator); if (PyErr_Occurred()) { buffer_free(buffer); PyMem_Free(collection_name); return NULL; } if (!max_size) { PyObject* InvalidOperation = _error("InvalidOperation"); if (InvalidOperation) { PyErr_SetString(InvalidOperation, "cannot do an empty bulk insert"); Py_DECREF(InvalidOperation); } buffer_free(buffer); PyMem_Free(collection_name); return NULL; } message_length = buffer_get_position(buffer) - length_location; memcpy(buffer_get_buffer(buffer) + length_location, &message_length, 4); if (safe) { if (!add_last_error(self, buffer, request_id, collection_name, collection_name_length, last_error_args)) { buffer_free(buffer); PyMem_Free(collection_name); return NULL; } } PyMem_Free(collection_name); /* objectify buffer */ result = Py_BuildValue("i" BYTES_FORMAT_STRING "i", request_id, buffer_get_buffer(buffer), buffer_get_position(buffer), max_size); buffer_free(buffer); return result; } PyDoc_STRVAR(_cbson_insert_message_doc, "Create an insert message to be sent to MongoDB\n\ \n\ Note: As of PyMongo 2.6, this function is no longer used. It\n\ is being kept (with tests) for backwards compatibility with 3rd\n\ party libraries that may currently be using it, but will likely\n\ be removed in a future release."); static PyObject* _cbson_update_message(PyObject* self, PyObject* args) { /* NOTE just using a random number as the request_id */ struct module_state *state = GETSTATE(self); int request_id = rand(); char* collection_name = NULL; int collection_name_length; int before, cur_size, max_size = 0; PyObject* doc; PyObject* spec; unsigned char multi; unsigned char upsert; unsigned char safe; unsigned char check_keys; unsigned char uuid_subtype; PyObject* last_error_args; int options; buffer_t buffer; int length_location, message_length; PyObject* result; if (!PyArg_ParseTuple(args, "et#bbOObObb", "utf-8", &collection_name, &collection_name_length, &upsert, &multi, &spec, &doc, &safe, &last_error_args, &check_keys, &uuid_subtype)) { return NULL; } options = 0; if (upsert) { options += 1; } if (multi) { options += 2; } buffer = buffer_new(); if (!buffer) { PyErr_NoMemory(); PyMem_Free(collection_name); return NULL; } // save space for message length length_location = buffer_save_space(buffer, 4); if (length_location == -1) { PyMem_Free(collection_name); PyErr_NoMemory(); return NULL; } if (!buffer_write_bytes(buffer, (const char*)&request_id, 4) || !buffer_write_bytes(buffer, "\x00\x00\x00\x00" "\xd1\x07\x00\x00" "\x00\x00\x00\x00", 12) || !buffer_write_bytes(buffer, collection_name, collection_name_length + 1) || !buffer_write_bytes(buffer, (const char*)&options, 4)) { buffer_free(buffer); PyMem_Free(collection_name); return NULL; } before = buffer_get_position(buffer); if (!write_dict(state->_cbson, buffer, spec, 0, uuid_subtype, 1)) { buffer_free(buffer); PyMem_Free(collection_name); return NULL; } max_size = buffer_get_position(buffer) - before; before = buffer_get_position(buffer); if (!write_dict(state->_cbson, buffer, doc, check_keys, uuid_subtype, 1)) { buffer_free(buffer); PyMem_Free(collection_name); return NULL; } cur_size = buffer_get_position(buffer) - before; max_size = (cur_size > max_size) ? cur_size : max_size; message_length = buffer_get_position(buffer) - length_location; memcpy(buffer_get_buffer(buffer) + length_location, &message_length, 4); if (safe) { if (!add_last_error(self, buffer, request_id, collection_name, collection_name_length, last_error_args)) { buffer_free(buffer); PyMem_Free(collection_name); return NULL; } } PyMem_Free(collection_name); /* objectify buffer */ result = Py_BuildValue("i" BYTES_FORMAT_STRING "i", request_id, buffer_get_buffer(buffer), buffer_get_position(buffer), max_size); buffer_free(buffer); return result; } static PyObject* _cbson_query_message(PyObject* self, PyObject* args) { /* NOTE just using a random number as the request_id */ struct module_state *state = GETSTATE(self); int request_id = rand(); unsigned int options; char* collection_name = NULL; int collection_name_length; int begin, cur_size, max_size = 0; int num_to_skip; int num_to_return; PyObject* query; PyObject* field_selector = Py_None; unsigned char uuid_subtype = 3; buffer_t buffer; int length_location, message_length; PyObject* result; if (!PyArg_ParseTuple(args, "Iet#iiO|Ob", &options, "utf-8", &collection_name, &collection_name_length, &num_to_skip, &num_to_return, &query, &field_selector, &uuid_subtype)) { return NULL; } buffer = buffer_new(); if (!buffer) { PyErr_NoMemory(); PyMem_Free(collection_name); return NULL; } // save space for message length length_location = buffer_save_space(buffer, 4); if (length_location == -1) { PyMem_Free(collection_name); PyErr_NoMemory(); return NULL; } if (!buffer_write_bytes(buffer, (const char*)&request_id, 4) || !buffer_write_bytes(buffer, "\x00\x00\x00\x00\xd4\x07\x00\x00", 8) || !buffer_write_bytes(buffer, (const char*)&options, 4) || !buffer_write_bytes(buffer, collection_name, collection_name_length + 1) || !buffer_write_bytes(buffer, (const char*)&num_to_skip, 4) || !buffer_write_bytes(buffer, (const char*)&num_to_return, 4)) { buffer_free(buffer); PyMem_Free(collection_name); return NULL; } begin = buffer_get_position(buffer); if (!write_dict(state->_cbson, buffer, query, 0, uuid_subtype, 1)) { buffer_free(buffer); PyMem_Free(collection_name); return NULL; } max_size = buffer_get_position(buffer) - begin; if (field_selector != Py_None) { begin = buffer_get_position(buffer); if (!write_dict(state->_cbson, buffer, field_selector, 0, uuid_subtype, 1)) { buffer_free(buffer); PyMem_Free(collection_name); return NULL; } cur_size = buffer_get_position(buffer) - begin; max_size = (cur_size > max_size) ? cur_size : max_size; } PyMem_Free(collection_name); message_length = buffer_get_position(buffer) - length_location; memcpy(buffer_get_buffer(buffer) + length_location, &message_length, 4); /* objectify buffer */ result = Py_BuildValue("i" BYTES_FORMAT_STRING "i", request_id, buffer_get_buffer(buffer), buffer_get_position(buffer), max_size); buffer_free(buffer); return result; } static PyObject* _cbson_get_more_message(PyObject* self, PyObject* args) { /* NOTE just using a random number as the request_id */ int request_id = rand(); char* collection_name = NULL; int collection_name_length; int num_to_return; long long cursor_id; buffer_t buffer; int length_location, message_length; PyObject* result; if (!PyArg_ParseTuple(args, "et#iL", "utf-8", &collection_name, &collection_name_length, &num_to_return, &cursor_id)) { return NULL; } buffer = buffer_new(); if (!buffer) { PyErr_NoMemory(); PyMem_Free(collection_name); return NULL; } // save space for message length length_location = buffer_save_space(buffer, 4); if (length_location == -1) { PyMem_Free(collection_name); PyErr_NoMemory(); return NULL; } if (!buffer_write_bytes(buffer, (const char*)&request_id, 4) || !buffer_write_bytes(buffer, "\x00\x00\x00\x00" "\xd5\x07\x00\x00" "\x00\x00\x00\x00", 12) || !buffer_write_bytes(buffer, collection_name, collection_name_length + 1) || !buffer_write_bytes(buffer, (const char*)&num_to_return, 4) || !buffer_write_bytes(buffer, (const char*)&cursor_id, 8)) { buffer_free(buffer); PyMem_Free(collection_name); return NULL; } PyMem_Free(collection_name); message_length = buffer_get_position(buffer) - length_location; memcpy(buffer_get_buffer(buffer) + length_location, &message_length, 4); /* objectify buffer */ result = Py_BuildValue("i" BYTES_FORMAT_STRING, request_id, buffer_get_buffer(buffer), buffer_get_position(buffer)); buffer_free(buffer); return result; } static PyObject* _cbson_do_batched_insert(PyObject* self, PyObject* args) { struct module_state *state = GETSTATE(self); /* NOTE just using a random number as the request_id */ int request_id = rand(); int options = 0, max_size = 0; int length_location, message_length; int collection_name_length; char* collection_name = NULL; PyObject* docs; PyObject* doc; PyObject* iterator; PyObject* client; PyObject* last_error_args; PyObject* result; PyObject* max_bson_size_obj; PyObject* max_message_size_obj; PyObject* send_message_result; unsigned char check_keys; unsigned char safe; unsigned char continue_on_error; unsigned char uuid_subtype; long max_bson_size; long max_message_size; buffer_t buffer; PyObject *exc_type = NULL, *exc_value = NULL, *exc_trace = NULL; if (!PyArg_ParseTuple(args, "et#ObbObbO", "utf-8", &collection_name, &collection_name_length, &docs, &check_keys, &safe, &last_error_args, &continue_on_error, &uuid_subtype, &client)) { return NULL; } if (continue_on_error) { options += 1; } max_bson_size_obj = PyObject_GetAttrString(client, "max_bson_size"); #if PY_MAJOR_VERSION >= 3 max_bson_size = PyLong_AsLong(max_bson_size_obj); #else max_bson_size = PyInt_AsLong(max_bson_size_obj); #endif Py_XDECREF(max_bson_size_obj); if (max_bson_size == -1) { PyMem_Free(collection_name); return NULL; } max_message_size_obj = PyObject_GetAttrString(client, "max_message_size"); #if PY_MAJOR_VERSION >= 3 max_message_size = PyLong_AsLong(max_message_size_obj); #else max_message_size = PyInt_AsLong(max_message_size_obj); #endif Py_XDECREF(max_message_size_obj); if (max_message_size == -1) { PyMem_Free(collection_name); return NULL; } buffer = buffer_new(); if (!buffer) { PyErr_NoMemory(); PyMem_Free(collection_name); return NULL; } length_location = init_insert_buffer(buffer, request_id, options, collection_name, collection_name_length); if (length_location == -1) { goto insertfail; } iterator = PyObject_GetIter(docs); if (iterator == NULL) { PyObject* InvalidOperation = _error("InvalidOperation"); if (InvalidOperation) { PyErr_SetString(InvalidOperation, "input is not iterable"); Py_DECREF(InvalidOperation); } goto insertfail; } while ((doc = PyIter_Next(iterator)) != NULL) { int before = buffer_get_position(buffer); int cur_size; if (!write_dict(state->_cbson, buffer, doc, check_keys, uuid_subtype, 1)) { Py_DECREF(doc); goto iterfail; } Py_DECREF(doc); cur_size = buffer_get_position(buffer) - before; max_size = (cur_size > max_size) ? cur_size : max_size; if (cur_size > max_bson_size) { PyObject* InvalidDocument = _error("InvalidDocument"); if (InvalidDocument) { #if PY_MAJOR_VERSION >= 3 PyObject* error = PyUnicode_FromFormat(DOC_TOO_LARGE_FMT, cur_size, max_bson_size); #else PyObject* error = PyString_FromFormat(DOC_TOO_LARGE_FMT, cur_size, max_bson_size); #endif if (error) { PyErr_SetObject(InvalidDocument, error); Py_DECREF(error); } Py_DECREF(InvalidDocument); } goto iterfail; } /* We have enough data, send this batch. */ if (buffer_get_position(buffer) > max_message_size) { int new_request_id = rand(); int message_start; PyObject* send_gle = Py_False; buffer_t new_buffer = buffer_new(); if (!new_buffer) { PyErr_NoMemory(); goto iterfail; } message_start = init_insert_buffer(new_buffer, new_request_id, options, collection_name, collection_name_length); if (message_start == -1) { buffer_free(new_buffer); goto iterfail; } /* Copy the overflow encoded document into the new buffer. */ if (!buffer_write_bytes(new_buffer, (const char*)buffer_get_buffer(buffer) + before, cur_size)) { buffer_free(new_buffer); goto iterfail; } /* Roll back to the beginning of this document. */ buffer_update_position(buffer, before); message_length = buffer_get_position(buffer) - length_location; memcpy(buffer_get_buffer(buffer) + length_location, &message_length, 4); /* If we are doing unacknowledged writes *and* continue_on_error * is True it's pointless (and slower) to send GLE. */ if (safe || !continue_on_error) { send_gle = Py_True; if (!add_last_error(self, buffer, request_id, collection_name, collection_name_length, last_error_args)) { buffer_free(new_buffer); goto iterfail; } } /* Objectify buffer */ result = Py_BuildValue("i" BYTES_FORMAT_STRING, request_id, buffer_get_buffer(buffer), buffer_get_position(buffer)); buffer_free(buffer); buffer = new_buffer; request_id = new_request_id; length_location = message_start; send_message_result = PyObject_CallMethod(client, "_send_message", "NO", result, send_gle); if (!send_message_result) { PyObject *etype = NULL, *evalue = NULL, *etrace = NULL; PyObject* OperationFailure; PyErr_Fetch(&etype, &evalue, &etrace); OperationFailure = _error("OperationFailure"); if (OperationFailure) { if (PyErr_GivenExceptionMatches(etype, OperationFailure)) { if (!safe || continue_on_error) { Py_DECREF(OperationFailure); if (!safe) { /* We're doing unacknowledged writes and * continue_on_error is False. Just return. */ Py_DECREF(etype); Py_DECREF(evalue); Py_DECREF(etrace); Py_DECREF(iterator); buffer_free(buffer); PyMem_Free(collection_name); Py_RETURN_NONE; } /* continue_on_error is True, store the error * details to re-raise after the final batch */ Py_XDECREF(exc_type); Py_XDECREF(exc_value); Py_XDECREF(exc_trace); exc_type = etype; exc_value = evalue; exc_trace = etrace; continue; } } Py_DECREF(OperationFailure); } /* This isn't OperationFailure, we couldn't * import OperationFailure, or we are doing * acknowledged writes. Re-raise immediately. */ PyErr_Restore(etype, evalue, etrace); goto iterfail; } else { Py_DECREF(send_message_result); } } } Py_DECREF(iterator); if (PyErr_Occurred()) { goto insertfail; } if (!max_size) { PyObject* InvalidOperation = _error("InvalidOperation"); if (InvalidOperation) { PyErr_SetString(InvalidOperation, "cannot do an empty bulk insert"); Py_DECREF(InvalidOperation); } goto insertfail; } message_length = buffer_get_position(buffer) - length_location; memcpy(buffer_get_buffer(buffer) + length_location, &message_length, 4); if (safe) { if (!add_last_error(self, buffer, request_id, collection_name, collection_name_length, last_error_args)) { goto insertfail; } } PyMem_Free(collection_name); /* objectify buffer */ result = Py_BuildValue("i" BYTES_FORMAT_STRING, request_id, buffer_get_buffer(buffer), buffer_get_position(buffer)); buffer_free(buffer); /* Send the last (or only) batch */ send_message_result = PyObject_CallMethod(client, "_send_message", "NN", result, PyBool_FromLong((long)safe)); if (!send_message_result) { Py_XDECREF(exc_type); Py_XDECREF(exc_value); Py_XDECREF(exc_trace); return NULL; } else { Py_DECREF(send_message_result); } if (exc_type) { /* Re-raise any previously stored exception * due to continue_on_error being True */ PyErr_Restore(exc_type, exc_value, exc_trace); return NULL; } Py_RETURN_NONE; iterfail: Py_DECREF(iterator); insertfail: Py_XDECREF(exc_type); Py_XDECREF(exc_value); Py_XDECREF(exc_trace); buffer_free(buffer); PyMem_Free(collection_name); return NULL; } static PyMethodDef _CMessageMethods[] = { {"_insert_message", _cbson_insert_message, METH_VARARGS, _cbson_insert_message_doc}, {"_update_message", _cbson_update_message, METH_VARARGS, "create an update message to be sent to MongoDB"}, {"_query_message", _cbson_query_message, METH_VARARGS, "create a query message to be sent to MongoDB"}, {"_get_more_message", _cbson_get_more_message, METH_VARARGS, "create a get more message to be sent to MongoDB"}, {"_do_batched_insert", _cbson_do_batched_insert, METH_VARARGS, "insert a batch of documents, splitting the batch as needed"}, {NULL, NULL, 0, NULL} }; #if PY_MAJOR_VERSION >= 3 #define INITERROR return NULL static int _cmessage_traverse(PyObject *m, visitproc visit, void *arg) { Py_VISIT(GETSTATE(m)->_cbson); return 0; } static int _cmessage_clear(PyObject *m) { Py_CLEAR(GETSTATE(m)->_cbson); return 0; } static struct PyModuleDef moduledef = { PyModuleDef_HEAD_INIT, "_cmessage", NULL, sizeof(struct module_state), _CMessageMethods, NULL, _cmessage_traverse, _cmessage_clear, NULL }; PyMODINIT_FUNC PyInit__cmessage(void) #else #define INITERROR return PyMODINIT_FUNC init_cmessage(void) #endif { PyObject *_cbson; PyObject *c_api_object; PyObject *m; struct module_state *state; /* Store a reference to the _cbson module since it's needed to call some * of its functions */ _cbson = PyImport_ImportModule("bson._cbson"); if (_cbson == NULL) { INITERROR; } /* Import C API of _cbson * The header file accesses _cbson_API to call the functions */ c_api_object = PyObject_GetAttrString(_cbson, "_C_API"); if (c_api_object == NULL) { Py_DECREF(_cbson); INITERROR; } #if PY_VERSION_HEX >= 0x03010000 _cbson_API = (void **)PyCapsule_GetPointer(c_api_object, "_cbson._C_API"); #else _cbson_API = (void **)PyCObject_AsVoidPtr(c_api_object); #endif if (_cbson_API == NULL) { Py_DECREF(c_api_object); Py_DECREF(_cbson); INITERROR; } #if PY_MAJOR_VERSION >= 3 m = PyModule_Create(&moduledef); #else m = Py_InitModule("_cmessage", _CMessageMethods); #endif if (m == NULL) { Py_DECREF(c_api_object); Py_DECREF(_cbson); INITERROR; } state = GETSTATE(m); state->_cbson = _cbson; Py_DECREF(c_api_object); #if PY_MAJOR_VERSION >= 3 return m; #endif } pymongo-2.6.3/pymongo/auth.py000066400000000000000000000174301223300253600162160ustar00rootroot00000000000000# Copyright 2013 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Authentication helpers.""" try: import hashlib _MD5 = hashlib.md5 except ImportError: # for Python < 2.5 import md5 _MD5 = md5.new HAVE_KERBEROS = True try: import kerberos except ImportError: HAVE_KERBEROS = False from bson.binary import Binary from bson.son import SON from pymongo.errors import ConfigurationError, OperationFailure MECHANISMS = ('GSSAPI', 'MONGODB-CR', 'MONGODB-X509', 'PLAIN') """The authentication mechanisms supported by PyMongo.""" def _build_credentials_tuple(mech, source, user, passwd, extra): """Build and return a mechanism specific credentials tuple. """ if mech == 'GSSAPI': gsn = extra.get('gssapiservicename', 'mongodb') # No password, source is always $external. return (mech, '$external', user, gsn) elif mech == 'MONGODB-X509': return (mech, '$external', user) return (mech, source, user, passwd) def _password_digest(username, password): """Get a password digest to use for authentication. """ if not isinstance(password, basestring): raise TypeError("password must be an instance " "of %s" % (basestring.__name__,)) if len(password) == 0: raise TypeError("password can't be empty") if not isinstance(username, basestring): raise TypeError("username must be an instance " "of %s" % (basestring.__name__,)) md5hash = _MD5() data = "%s:mongo:%s" % (username, password) md5hash.update(data.encode('utf-8')) return unicode(md5hash.hexdigest()) def _auth_key(nonce, username, password): """Get an auth key to use for authentication. """ digest = _password_digest(username, password) md5hash = _MD5() data = "%s%s%s" % (nonce, unicode(username), digest) md5hash.update(data.encode('utf-8')) return unicode(md5hash.hexdigest()) def _authenticate_gssapi(credentials, sock_info, cmd_func): """Authenticate using GSSAPI. """ try: dummy, username, gsn = credentials # Starting here and continuing through the while loop below - establish # the security context. See RFC 4752, Section 3.1, first paragraph. result, ctx = kerberos.authGSSClientInit(gsn + '@' + sock_info.host, kerberos.GSS_C_MUTUAL_FLAG) if result != kerberos.AUTH_GSS_COMPLETE: raise OperationFailure('Kerberos context failed to initialize.') try: # pykerberos uses a weird mix of exceptions and return values # to indicate errors. # 0 == continue, 1 == complete, -1 == error # Only authGSSClientStep can return 0. if kerberos.authGSSClientStep(ctx, '') != 0: raise OperationFailure('Unknown kerberos ' 'failure in step function.') # Start a SASL conversation with mongod/s # Note: pykerberos deals with base64 encoded byte strings. # Since mongo accepts base64 strings as the payload we don't # have to use bson.binary.Binary. payload = kerberos.authGSSClientResponse(ctx) cmd = SON([('saslStart', 1), ('mechanism', 'GSSAPI'), ('payload', payload), ('autoAuthorize', 1)]) response, _ = cmd_func(sock_info, '$external', cmd) # Limit how many times we loop to catch protocol / library issues for _ in xrange(10): result = kerberos.authGSSClientStep(ctx, str(response['payload'])) if result == -1: raise OperationFailure('Unknown kerberos ' 'failure in step function.') payload = kerberos.authGSSClientResponse(ctx) or '' cmd = SON([('saslContinue', 1), ('conversationId', response['conversationId']), ('payload', payload)]) response, _ = cmd_func(sock_info, '$external', cmd) if result == kerberos.AUTH_GSS_COMPLETE: break else: raise OperationFailure('Kerberos ' 'authentication failed to complete.') # Once the security context is established actually authenticate. # See RFC 4752, Section 3.1, last two paragraphs. if kerberos.authGSSClientUnwrap(ctx, str(response['payload'])) != 1: raise OperationFailure('Unknown kerberos ' 'failure during GSS_Unwrap step.') if kerberos.authGSSClientWrap(ctx, kerberos.authGSSClientResponse(ctx), username) != 1: raise OperationFailure('Unknown kerberos ' 'failure during GSS_Wrap step.') payload = kerberos.authGSSClientResponse(ctx) cmd = SON([('saslContinue', 1), ('conversationId', response['conversationId']), ('payload', payload)]) response, _ = cmd_func(sock_info, '$external', cmd) finally: kerberos.authGSSClientClean(ctx) except kerberos.KrbError, exc: raise OperationFailure(str(exc)) def _authenticate_plain(credentials, sock_info, cmd_func): """Authenticate using SASL PLAIN (RFC 4616) """ source, username, password = credentials payload = ('\x00%s\x00%s' % (username, password)).encode('utf-8') cmd = SON([('saslStart', 1), ('mechanism', 'PLAIN'), ('payload', Binary(payload)), ('autoAuthorize', 1)]) cmd_func(sock_info, source, cmd) def _authenticate_x509(credentials, sock_info, cmd_func): """Authenticate using MONGODB-X509. """ dummy, username = credentials query = SON([('authenticate', 1), ('mechanism', 'MONGODB-X509'), ('user', username)]) cmd_func(sock_info, '$external', query) def _authenticate_mongo_cr(credentials, sock_info, cmd_func): """Authenticate using MONGODB-CR. """ source, username, password = credentials # Get a nonce response, _ = cmd_func(sock_info, source, {'getnonce': 1}) nonce = response['nonce'] key = _auth_key(nonce, username, password) # Actually authenticate query = SON([('authenticate', 1), ('user', username), ('nonce', nonce), ('key', key)]) cmd_func(sock_info, source, query) _AUTH_MAP = { 'GSSAPI': _authenticate_gssapi, 'MONGODB-CR': _authenticate_mongo_cr, 'MONGODB-X509': _authenticate_x509, 'PLAIN': _authenticate_plain, } def authenticate(credentials, sock_info, cmd_func): """Authenticate sock_info. """ mechanism = credentials[0] if mechanism == 'GSSAPI': if not HAVE_KERBEROS: raise ConfigurationError('The "kerberos" module must be ' 'installed to use GSSAPI authentication.') auth_func = _AUTH_MAP.get(mechanism) auth_func(credentials[1:], sock_info, cmd_func) pymongo-2.6.3/pymongo/collection.py000066400000000000000000002007411223300253600174070ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Collection level utilities for Mongo.""" import warnings from bson.binary import ALL_UUID_SUBTYPES, OLD_UUID_SUBTYPE from bson.code import Code from bson.son import SON from pymongo import (common, helpers, message) from pymongo.cursor import Cursor from pymongo.errors import ConfigurationError, InvalidName try: from collections import OrderedDict ordered_types = (SON, OrderedDict) except ImportError: ordered_types = SON def _gen_index_name(keys): """Generate an index name from the set of fields it is over. """ return u"_".join([u"%s_%s" % item for item in keys]) class Collection(common.BaseObject): """A Mongo collection. """ def __init__(self, database, name, create=False, **kwargs): """Get / create a Mongo collection. Raises :class:`TypeError` if `name` is not an instance of :class:`basestring` (:class:`str` in python 3). Raises :class:`~pymongo.errors.InvalidName` if `name` is not a valid collection name. Any additional keyword arguments will be used as options passed to the create command. See :meth:`~pymongo.database.Database.create_collection` for valid options. If `create` is ``True`` or additional keyword arguments are present a create command will be sent. Otherwise, a create command will not be sent and the collection will be created implicitly on first use. :Parameters: - `database`: the database to get a collection from - `name`: the name of the collection to get - `create` (optional): if ``True``, force collection creation even without options being set - `**kwargs` (optional): additional keyword arguments will be passed as options for the create collection command .. versionchanged:: 2.2 Removed deprecated argument: options .. versionadded:: 2.1 uuid_subtype attribute .. versionchanged:: 1.5 deprecating `options` in favor of kwargs .. versionadded:: 1.5 the `create` parameter .. mongodoc:: collections """ super(Collection, self).__init__( slave_okay=database.slave_okay, read_preference=database.read_preference, tag_sets=database.tag_sets, secondary_acceptable_latency_ms=( database.secondary_acceptable_latency_ms), safe=database.safe, **database.write_concern) if not isinstance(name, basestring): raise TypeError("name must be an instance " "of %s" % (basestring.__name__,)) if not name or ".." in name: raise InvalidName("collection names cannot be empty") if "$" in name and not (name.startswith("oplog.$main") or name.startswith("$cmd")): raise InvalidName("collection names must not " "contain '$': %r" % name) if name[0] == "." or name[-1] == ".": raise InvalidName("collection names must not start " "or end with '.': %r" % name) if "\x00" in name: raise InvalidName("collection names must not contain the " "null character") self.__database = database self.__name = unicode(name) self.__uuid_subtype = OLD_UUID_SUBTYPE self.__full_name = u"%s.%s" % (self.__database.name, self.__name) if create or kwargs: self.__create(kwargs) def __create(self, options): """Sends a create command with the given options. """ if options: if "size" in options: options["size"] = float(options["size"]) self.__database.command("create", self.__name, **options) else: self.__database.command("create", self.__name) def __getattr__(self, name): """Get a sub-collection of this collection by name. Raises InvalidName if an invalid collection name is used. :Parameters: - `name`: the name of the collection to get """ return Collection(self.__database, u"%s.%s" % (self.__name, name)) def __getitem__(self, name): return self.__getattr__(name) def __repr__(self): return "Collection(%r, %r)" % (self.__database, self.__name) def __eq__(self, other): if isinstance(other, Collection): us = (self.__database, self.__name) them = (other.__database, other.__name) return us == them return NotImplemented def __ne__(self, other): return not self == other @property def full_name(self): """The full name of this :class:`Collection`. The full name is of the form `database_name.collection_name`. .. versionchanged:: 1.3 ``full_name`` is now a property rather than a method. """ return self.__full_name @property def name(self): """The name of this :class:`Collection`. .. versionchanged:: 1.3 ``name`` is now a property rather than a method. """ return self.__name @property def database(self): """The :class:`~pymongo.database.Database` that this :class:`Collection` is a part of. .. versionchanged:: 1.3 ``database`` is now a property rather than a method. """ return self.__database def __get_uuid_subtype(self): return self.__uuid_subtype def __set_uuid_subtype(self, subtype): if subtype not in ALL_UUID_SUBTYPES: raise ConfigurationError("Not a valid setting for uuid_subtype.") self.__uuid_subtype = subtype uuid_subtype = property(__get_uuid_subtype, __set_uuid_subtype, doc="""This attribute specifies which BSON Binary subtype is used when storing UUIDs. Historically UUIDs have been stored as BSON Binary subtype 3. This attribute is used to switch to the newer BSON binary subtype 4. It can also be used to force legacy byte order and subtype compatibility with the Java and C# drivers. See the :mod:`bson.binary` module for all options.""") def save(self, to_save, manipulate=True, safe=None, check_keys=True, **kwargs): """Save a document in this collection. If `to_save` already has an ``"_id"`` then an :meth:`update` (upsert) operation is performed and any existing document with that ``"_id"`` is overwritten. Otherwise an :meth:`insert` operation is performed. In this case if `manipulate` is ``True`` an ``"_id"`` will be added to `to_save` and this method returns the ``"_id"`` of the saved document. If `manipulate` is ``False`` the ``"_id"`` will be added by the server but this method will return ``None``. Raises :class:`TypeError` if `to_save` is not an instance of :class:`dict`. Write concern options can be passed as keyword arguments, overriding any global defaults. Valid options include w=, wtimeout=, j=, or fsync=. See the parameter list below for a detailed explanation of these options. By default an acknowledgment is requested from the server that the save was successful, raising :class:`~pymongo.errors.OperationFailure` if an error occurred. **Passing ``w=0`` disables write acknowledgement and all other write concern options.** :Parameters: - `to_save`: the document to be saved - `manipulate` (optional): manipulate the document before saving it? - `safe` (optional): **DEPRECATED** - Use `w` instead. - `check_keys` (optional): check if keys start with '$' or contain '.', raising :class:`~pymongo.errors.InvalidName` in either case. - `w` (optional): (integer or string) If this is a replica set, write operations will block until they have been replicated to the specified number or tagged set of servers. `w=` always includes the replica set primary (e.g. w=3 means write to the primary and wait until replicated to **two** secondaries). **Passing w=0 disables write acknowledgement and all other write concern options.** - `wtimeout` (optional): (integer) Used in conjunction with `w`. Specify a value in milliseconds to control how long to wait for write propagation to complete. If replication does not complete in the given timeframe, a timeout exception is raised. - `j` (optional): If ``True`` block until write operations have been committed to the journal. Ignored if the server is running without journaling. - `fsync` (optional): If ``True`` force the database to fsync all files before returning. When used with `j` the server awaits the next group commit before returning. :Returns: - The ``'_id'`` value of `to_save` or ``[None]`` if `manipulate` is ``False`` and `to_save` has no '_id' field. .. versionadded:: 1.8 Support for passing `getLastError` options as keyword arguments. .. mongodoc:: insert """ if not isinstance(to_save, dict): raise TypeError("cannot save object of type %s" % type(to_save)) if "_id" not in to_save: return self.insert(to_save, manipulate, safe, check_keys, **kwargs) else: self.update({"_id": to_save["_id"]}, to_save, True, manipulate, safe, check_keys=check_keys, **kwargs) return to_save.get("_id", None) def insert(self, doc_or_docs, manipulate=True, safe=None, check_keys=True, continue_on_error=False, **kwargs): """Insert a document(s) into this collection. If `manipulate` is ``True``, the document(s) are manipulated using any :class:`~pymongo.son_manipulator.SONManipulator` instances that have been added to this :class:`~pymongo.database.Database`. In this case an ``"_id"`` will be added if the document(s) does not already contain one and the ``"id"`` (or list of ``"_id"`` values for more than one document) will be returned. If `manipulate` is ``False`` and the document(s) does not include an ``"_id"`` one will be added by the server. The server does not return the ``"_id"`` it created so ``None`` is returned. Write concern options can be passed as keyword arguments, overriding any global defaults. Valid options include w=, wtimeout=, j=, or fsync=. See the parameter list below for a detailed explanation of these options. By default an acknowledgment is requested from the server that the insert was successful, raising :class:`~pymongo.errors.OperationFailure` if an error occurred. **Passing ``w=0`` disables write acknowledgement and all other write concern options.** :Parameters: - `doc_or_docs`: a document or list of documents to be inserted - `manipulate` (optional): If ``True`` manipulate the documents before inserting. - `safe` (optional): **DEPRECATED** - Use `w` instead. - `check_keys` (optional): If ``True`` check if keys start with '$' or contain '.', raising :class:`~pymongo.errors.InvalidName` in either case. - `continue_on_error` (optional): If ``True``, the database will not stop processing a bulk insert if one fails (e.g. due to duplicate IDs). This makes bulk insert behave similarly to a series of single inserts, except lastError will be set if any insert fails, not just the last one. If multiple errors occur, only the most recent will be reported by :meth:`~pymongo.database.Database.error`. - `w` (optional): (integer or string) If this is a replica set, write operations will block until they have been replicated to the specified number or tagged set of servers. `w=` always includes the replica set primary (e.g. w=3 means write to the primary and wait until replicated to **two** secondaries). **Passing w=0 disables write acknowledgement and all other write concern options.** - `wtimeout` (optional): (integer) Used in conjunction with `w`. Specify a value in milliseconds to control how long to wait for write propagation to complete. If replication does not complete in the given timeframe, a timeout exception is raised. - `j` (optional): If ``True`` block until write operations have been committed to the journal. Ignored if the server is running without journaling. - `fsync` (optional): If ``True`` force the database to fsync all files before returning. When used with `j` the server awaits the next group commit before returning. :Returns: - The ``'_id'`` value (or list of '_id' values) of `doc_or_docs` or ``[None]`` if manipulate is ``False`` and the documents passed as `doc_or_docs` do not include an '_id' field. .. note:: `continue_on_error` requires server version **>= 1.9.1** .. versionadded:: 2.1 Support for continue_on_error. .. versionadded:: 1.8 Support for passing `getLastError` options as keyword arguments. .. versionchanged:: 1.1 Bulk insert works with any iterable .. mongodoc:: insert """ # Batch inserts require us to know the connected master's # max_bson_size and max_message_size. We have to be connected # to a master to know that. self.database.connection._ensure_connected(True) docs = doc_or_docs return_one = False if isinstance(docs, dict): return_one = True docs = [docs] if manipulate: docs = [self.__database._fix_incoming(doc, self) for doc in docs] safe, options = self._get_write_mode(safe, **kwargs) message._do_batched_insert(self.__full_name, docs, check_keys, safe, options, continue_on_error, self.__uuid_subtype, self.database.connection) ids = [doc.get("_id", None) for doc in docs] if return_one: return ids[0] else: return ids def update(self, spec, document, upsert=False, manipulate=False, safe=None, multi=False, check_keys=True, **kwargs): """Update a document(s) in this collection. Raises :class:`TypeError` if either `spec` or `document` is not an instance of ``dict`` or `upsert` is not an instance of ``bool``. Write concern options can be passed as keyword arguments, overriding any global defaults. Valid options include w=, wtimeout=, j=, or fsync=. See the parameter list below for a detailed explanation of these options. By default an acknowledgment is requested from the server that the update was successful, raising :class:`~pymongo.errors.OperationFailure` if an error occurred. **Passing ``w=0`` disables write acknowledgement and all other write concern options.** There are many useful `update modifiers`_ which can be used when performing updates. For example, here we use the ``"$set"`` modifier to modify some fields in a matching document: .. doctest:: >>> db.test.insert({"x": "y", "a": "b"}) ObjectId('...') >>> list(db.test.find()) [{u'a': u'b', u'x': u'y', u'_id': ObjectId('...')}] >>> db.test.update({"x": "y"}, {"$set": {"a": "c"}}) {...} >>> list(db.test.find()) [{u'a': u'c', u'x': u'y', u'_id': ObjectId('...')}] :Parameters: - `spec`: a ``dict`` or :class:`~bson.son.SON` instance specifying elements which must be present for a document to be updated - `document`: a ``dict`` or :class:`~bson.son.SON` instance specifying the document to be used for the update or (in the case of an upsert) insert - see docs on MongoDB `update modifiers`_ - `upsert` (optional): perform an upsert if ``True`` - `manipulate` (optional): manipulate the document before updating? If ``True`` all instances of :mod:`~pymongo.son_manipulator.SONManipulator` added to this :class:`~pymongo.database.Database` will be applied to the document before performing the update. - `check_keys` (optional): check if keys in `document` start with '$' or contain '.', raising :class:`~pymongo.errors.InvalidName`. Only applies to document replacement, not modification through $ operators. - `safe` (optional): **DEPRECATED** - Use `w` instead. - `multi` (optional): update all documents that match `spec`, rather than just the first matching document. The default value for `multi` is currently ``False``, but this might eventually change to ``True``. It is recommended that you specify this argument explicitly for all update operations in order to prepare your code for that change. - `w` (optional): (integer or string) If this is a replica set, write operations will block until they have been replicated to the specified number or tagged set of servers. `w=` always includes the replica set primary (e.g. w=3 means write to the primary and wait until replicated to **two** secondaries). **Passing w=0 disables write acknowledgement and all other write concern options.** - `wtimeout` (optional): (integer) Used in conjunction with `w`. Specify a value in milliseconds to control how long to wait for write propagation to complete. If replication does not complete in the given timeframe, a timeout exception is raised. - `j` (optional): If ``True`` block until write operations have been committed to the journal. Ignored if the server is running without journaling. - `fsync` (optional): If ``True`` force the database to fsync all files before returning. When used with `j` the server awaits the next group commit before returning. :Returns: - A document (dict) describing the effect of the update or ``None`` if write acknowledgement is disabled. .. versionadded:: 1.8 Support for passing `getLastError` options as keyword arguments. .. versionchanged:: 1.4 Return the response to *lastError* if `safe` is ``True``. .. versionadded:: 1.1.1 The `multi` parameter. .. _update modifiers: http://www.mongodb.org/display/DOCS/Updating .. mongodoc:: update """ if not isinstance(spec, dict): raise TypeError("spec must be an instance of dict") if not isinstance(document, dict): raise TypeError("document must be an instance of dict") if not isinstance(upsert, bool): raise TypeError("upsert must be an instance of bool") if manipulate: document = self.__database._fix_incoming(document, self) safe, options = self._get_write_mode(safe, **kwargs) if document: # If a top level key begins with '$' this is a modify operation # and we should skip key validation. It doesn't matter which key # we check here. Passing a document with a mix of top level keys # starting with and without a '$' is invalid and the server will # raise an appropriate exception. first = (document.iterkeys()).next() if first.startswith('$'): check_keys = False return self.__database.connection._send_message( message.update(self.__full_name, upsert, multi, spec, document, safe, options, check_keys, self.__uuid_subtype), safe) def drop(self): """Alias for :meth:`~pymongo.database.Database.drop_collection`. The following two calls are equivalent: >>> db.foo.drop() >>> db.drop_collection("foo") .. versionadded:: 1.8 """ self.__database.drop_collection(self.__name) def remove(self, spec_or_id=None, safe=None, **kwargs): """Remove a document(s) from this collection. .. warning:: Calls to :meth:`remove` should be performed with care, as removed data cannot be restored. If `spec_or_id` is ``None``, all documents in this collection will be removed. This is not equivalent to calling :meth:`~pymongo.database.Database.drop_collection`, however, as indexes will not be removed. Write concern options can be passed as keyword arguments, overriding any global defaults. Valid options include w=, wtimeout=, j=, or fsync=. See the parameter list below for a detailed explanation of these options. By default an acknowledgment is requested from the server that the remove was successful, raising :class:`~pymongo.errors.OperationFailure` if an error occurred. **Passing ``w=0`` disables write acknowledgement and all other write concern options.** :Parameters: - `spec_or_id` (optional): a dictionary specifying the documents to be removed OR any other type specifying the value of ``"_id"`` for the document to be removed - `safe` (optional): **DEPRECATED** - Use `w` instead. - `w` (optional): (integer or string) If this is a replica set, write operations will block until they have been replicated to the specified number or tagged set of servers. `w=` always includes the replica set primary (e.g. w=3 means write to the primary and wait until replicated to **two** secondaries). **Passing w=0 disables write acknowledgement and all other write concern options.** - `wtimeout` (optional): (integer) Used in conjunction with `w`. Specify a value in milliseconds to control how long to wait for write propagation to complete. If replication does not complete in the given timeframe, a timeout exception is raised. - `j` (optional): If ``True`` block until write operations have been committed to the journal. Ignored if the server is running without journaling. - `fsync` (optional): If ``True`` force the database to fsync all files before returning. When used with `j` the server awaits the next group commit before returning. :Returns: - A document (dict) describing the effect of the remove or ``None`` if write acknowledgement is disabled. .. versionadded:: 1.8 Support for passing `getLastError` options as keyword arguments. .. versionchanged:: 1.7 Accept any type other than a ``dict`` instance for removal by ``"_id"``, not just :class:`~bson.objectid.ObjectId` instances. .. versionchanged:: 1.4 Return the response to *lastError* if `safe` is ``True``. .. versionchanged:: 1.2 The `spec_or_id` parameter is now optional. If it is not specified *all* documents in the collection will be removed. .. versionadded:: 1.1 The `safe` parameter. .. mongodoc:: remove """ if spec_or_id is None: spec_or_id = {} if not isinstance(spec_or_id, dict): spec_or_id = {"_id": spec_or_id} safe, options = self._get_write_mode(safe, **kwargs) return self.__database.connection._send_message( message.delete(self.__full_name, spec_or_id, safe, options, self.__uuid_subtype), safe) def find_one(self, spec_or_id=None, *args, **kwargs): """Get a single document from the database. All arguments to :meth:`find` are also valid arguments for :meth:`find_one`, although any `limit` argument will be ignored. Returns a single document, or ``None`` if no matching document is found. :Parameters: - `spec_or_id` (optional): a dictionary specifying the query to be performed OR any other type to be used as the value for a query for ``"_id"``. - `*args` (optional): any additional positional arguments are the same as the arguments to :meth:`find`. - `**kwargs` (optional): any additional keyword arguments are the same as the arguments to :meth:`find`. .. versionchanged:: 1.7 Allow passing any of the arguments that are valid for :meth:`find`. .. versionchanged:: 1.7 Accept any type other than a ``dict`` instance as an ``"_id"`` query, not just :class:`~bson.objectid.ObjectId` instances. """ if spec_or_id is not None and not isinstance(spec_or_id, dict): spec_or_id = {"_id": spec_or_id} for result in self.find(spec_or_id, *args, **kwargs).limit(-1): return result return None def find(self, *args, **kwargs): """Query the database. The `spec` argument is a prototype document that all results must match. For example: >>> db.test.find({"hello": "world"}) only matches documents that have a key "hello" with value "world". Matches can have other keys *in addition* to "hello". The `fields` argument is used to specify a subset of fields that should be included in the result documents. By limiting results to a certain subset of fields you can cut down on network traffic and decoding time. Raises :class:`TypeError` if any of the arguments are of improper type. Returns an instance of :class:`~pymongo.cursor.Cursor` corresponding to this query. :Parameters: - `spec` (optional): a SON object specifying elements which must be present for a document to be included in the result set - `fields` (optional): a list of field names that should be returned in the result set or a dict specifying the fields to include or exclude. If `fields` is a list "_id" will always be returned. Use a dict to exclude fields from the result (e.g. fields={'_id': False}). - `skip` (optional): the number of documents to omit (from the start of the result set) when returning the results - `limit` (optional): the maximum number of results to return - `timeout` (optional): if True (the default), any returned cursor is closed by the server after 10 minutes of inactivity. If set to False, the returned cursor will never time out on the server. Care should be taken to ensure that cursors with timeout turned off are properly closed. - `snapshot` (optional): if True, snapshot mode will be used for this query. Snapshot mode assures no duplicates are returned, or objects missed, which were present at both the start and end of the query's execution. For details, see the `snapshot documentation `_. - `tailable` (optional): the result of this find call will be a tailable cursor - tailable cursors aren't closed when the last data is retrieved but are kept open and the cursors location marks the final document's position. if more data is received iteration of the cursor will continue from the last document received. For details, see the `tailable cursor documentation `_. - `sort` (optional): a list of (key, direction) pairs specifying the sort order for this query. See :meth:`~pymongo.cursor.Cursor.sort` for details. - `max_scan` (optional): limit the number of documents examined when performing the query - `as_class` (optional): class to use for documents in the query result (default is :attr:`~pymongo.mongo_client.MongoClient.document_class`) - `slave_okay` (optional): if True, allows this query to be run against a replica secondary. - `await_data` (optional): if True, the server will block for some extra time before returning, waiting for more data to return. Ignored if `tailable` is False. - `partial` (optional): if True, mongos will return partial results if some shards are down instead of returning an error. - `manipulate`: (optional): If True (the default), apply any outgoing SON manipulators before returning. - `network_timeout` (optional): specify a timeout to use for this query, which will override the :class:`~pymongo.mongo_client.MongoClient`-level default - `read_preference` (optional): The read preference for this query. - `tag_sets` (optional): The tag sets for this query. - `secondary_acceptable_latency_ms` (optional): Any replica-set member whose ping time is within secondary_acceptable_latency_ms of the nearest member may accept reads. Default 15 milliseconds. **Ignored by mongos** and must be configured on the command line. See the localThreshold_ option for more information. - `exhaust` (optional): If ``True`` create an "exhaust" cursor. MongoDB will stream batched results to the client without waiting for the client to request each batch, reducing latency. .. note:: There are a number of caveats to using the `exhaust` parameter: 1. The `exhaust` and `limit` options are incompatible and can not be used together. 2. The `exhaust` option is not supported by mongos and can not be used with a sharded cluster. 3. A :class:`~pymongo.cursor.Cursor` instance created with the `exhaust` option requires an exclusive :class:`~socket.socket` connection to MongoDB. If the :class:`~pymongo.cursor.Cursor` is discarded without being completely iterated the underlying :class:`~socket.socket` connection will be closed and discarded without being returned to the connection pool. 4. A :class:`~pymongo.cursor.Cursor` instance created with the `exhaust` option in a :doc:`request ` **must** be completely iterated before executing any other operation. 5. The `network_timeout` option is ignored when using the `exhaust` option. .. note:: The `manipulate` parameter may default to False in a future release. .. note:: The `max_scan` parameter requires server version **>= 1.5.1** .. versionadded:: 2.3 The `tag_sets` and `secondary_acceptable_latency_ms` parameters. .. versionadded:: 1.11+ The `await_data`, `partial`, and `manipulate` parameters. .. versionadded:: 1.8 The `network_timeout` parameter. .. versionadded:: 1.7 The `sort`, `max_scan` and `as_class` parameters. .. versionchanged:: 1.7 The `fields` parameter can now be a dict or any iterable in addition to a list. .. versionadded:: 1.1 The `tailable` parameter. .. mongodoc:: find .. _localThreshold: http://docs.mongodb.org/manual/reference/mongos/#cmdoption-mongos--localThreshold """ if not 'slave_okay' in kwargs: kwargs['slave_okay'] = self.slave_okay if not 'read_preference' in kwargs: kwargs['read_preference'] = self.read_preference if not 'tag_sets' in kwargs: kwargs['tag_sets'] = self.tag_sets if not 'secondary_acceptable_latency_ms' in kwargs: kwargs['secondary_acceptable_latency_ms'] = ( self.secondary_acceptable_latency_ms) return Cursor(self, *args, **kwargs) def count(self): """Get the number of documents in this collection. To get the number of documents matching a specific query use :meth:`pymongo.cursor.Cursor.count`. """ return self.find().count() def create_index(self, key_or_list, cache_for=300, **kwargs): """Creates an index on this collection. Takes either a single key or a list of (key, direction) pairs. The key(s) must be an instance of :class:`basestring` (:class:`str` in python 3), and the directions must be one of (:data:`~pymongo.ASCENDING`, :data:`~pymongo.DESCENDING`, :data:`~pymongo.GEO2D`). Returns the name of the created index. To create a single key index on the key ``'mike'`` we just use a string argument: >>> my_collection.create_index("mike") For a compound index on ``'mike'`` descending and ``'eliot'`` ascending we need to use a list of tuples: >>> my_collection.create_index([("mike", pymongo.DESCENDING), ... ("eliot", pymongo.ASCENDING)]) All optional index creation parameters should be passed as keyword arguments to this method. Valid options include: - `name`: custom name to use for this index - if none is given, a name will be generated - `unique`: should this index guarantee uniqueness? - `dropDups` or `drop_dups`: should we drop duplicates - `background`: if this index should be created in the background - `sparse`: if True, omit from the index any documents that lack the indexed field - `bucketSize` or `bucket_size`: for use with geoHaystack indexes. Number of documents to group together within a certain proximity to a given longitude and latitude. - `min`: minimum value for keys in a :data:`~pymongo.GEO2D` index - `max`: maximum value for keys in a :data:`~pymongo.GEO2D` index - `expireAfterSeconds`: Used to create an expiring (TTL) collection. MongoDB will automatically delete documents from this collection after seconds. The indexed field must be a UTC datetime or the data will not expire. .. note:: `expireAfterSeconds` requires server version **>= 2.1.2** :Parameters: - `key_or_list`: a single key or a list of (key, direction) pairs specifying the index to create - `cache_for` (optional): time window (in seconds) during which this index will be recognized by subsequent calls to :meth:`ensure_index` - see documentation for :meth:`ensure_index` for details - `**kwargs` (optional): any additional index creation options (see the above list) should be passed as keyword arguments - `ttl` (deprecated): Use `cache_for` instead. .. versionchanged:: 2.3 The `ttl` parameter has been deprecated to avoid confusion with TTL collections. Use `cache_for` instead. .. versionchanged:: 2.2 Removed deprecated argument: deprecated_unique .. versionchanged:: 1.5.1 Accept kwargs to support all index creation options. .. versionadded:: 1.5 The `name` parameter. .. seealso:: :meth:`ensure_index` .. mongodoc:: indexes """ if 'ttl' in kwargs: cache_for = kwargs.pop('ttl') warnings.warn("ttl is deprecated. Please use cache_for instead.", DeprecationWarning, stacklevel=2) # The types supported by datetime.timedelta. 2to3 removes long. if not isinstance(cache_for, (int, long, float)): raise TypeError("cache_for must be an integer or float.") keys = helpers._index_list(key_or_list) index_doc = helpers._index_document(keys) index = {"key": index_doc, "ns": self.__full_name} name = "name" in kwargs and kwargs["name"] or _gen_index_name(keys) index["name"] = name if "drop_dups" in kwargs: kwargs["dropDups"] = kwargs.pop("drop_dups") if "bucket_size" in kwargs: kwargs["bucketSize"] = kwargs.pop("bucket_size") index.update(kwargs) self.__database.system.indexes.insert(index, manipulate=False, check_keys=False, **self._get_wc_override()) self.__database.connection._cache_index(self.__database.name, self.__name, name, cache_for) return name def ensure_index(self, key_or_list, cache_for=300, **kwargs): """Ensures that an index exists on this collection. Takes either a single key or a list of (key, direction) pairs. The key(s) must be an instance of :class:`basestring` (:class:`str` in python 3), and the direction(s) must be one of (:data:`~pymongo.ASCENDING`, :data:`~pymongo.DESCENDING`, :data:`~pymongo.GEO2D`). See :meth:`create_index` for a detailed example. Unlike :meth:`create_index`, which attempts to create an index unconditionally, :meth:`ensure_index` takes advantage of some caching within the driver such that it only attempts to create indexes that might not already exist. When an index is created (or ensured) by PyMongo it is "remembered" for `cache_for` seconds. Repeated calls to :meth:`ensure_index` within that time limit will be lightweight - they will not attempt to actually create the index. Care must be taken when the database is being accessed through multiple clients at once. If an index is created using this client and deleted using another, any call to :meth:`ensure_index` within the cache window will fail to re-create the missing index. Returns the name of the created index if an index is actually created. Returns ``None`` if the index already exists. All optional index creation parameters should be passed as keyword arguments to this method. Valid options include: - `name`: custom name to use for this index - if none is given, a name will be generated - `unique`: should this index guarantee uniqueness? - `dropDups` or `drop_dups`: should we drop duplicates during index creation when creating a unique index? - `background`: if this index should be created in the background - `sparse`: if True, omit from the index any documents that lack the indexed field - `bucketSize` or `bucket_size`: for use with geoHaystack indexes. Number of documents to group together within a certain proximity to a given longitude and latitude. - `min`: minimum value for keys in a :data:`~pymongo.GEO2D` index - `max`: maximum value for keys in a :data:`~pymongo.GEO2D` index - `expireAfterSeconds`: Used to create an expiring (TTL) collection. MongoDB will automatically delete documents from this collection after seconds. The indexed field must be a UTC datetime or the data will not expire. .. note:: `expireAfterSeconds` requires server version **>= 2.1.2** :Parameters: - `key_or_list`: a single key or a list of (key, direction) pairs specifying the index to create - `cache_for` (optional): time window (in seconds) during which this index will be recognized by subsequent calls to :meth:`ensure_index` - `**kwargs` (optional): any additional index creation options (see the above list) should be passed as keyword arguments - `ttl` (deprecated): Use `cache_for` instead. .. versionchanged:: 2.3 The `ttl` parameter has been deprecated to avoid confusion with TTL collections. Use `cache_for` instead. .. versionchanged:: 2.2 Removed deprecated argument: deprecated_unique .. versionchanged:: 1.5.1 Accept kwargs to support all index creation options. .. versionadded:: 1.5 The `name` parameter. .. seealso:: :meth:`create_index` """ if "name" in kwargs: name = kwargs["name"] else: keys = helpers._index_list(key_or_list) name = kwargs["name"] = _gen_index_name(keys) if not self.__database.connection._cached(self.__database.name, self.__name, name): return self.create_index(key_or_list, cache_for, **kwargs) return None def drop_indexes(self): """Drops all indexes on this collection. Can be used on non-existant collections or collections with no indexes. Raises OperationFailure on an error. """ self.__database.connection._purge_index(self.__database.name, self.__name) self.drop_index(u"*") def drop_index(self, index_or_name): """Drops the specified index on this collection. Can be used on non-existant collections or collections with no indexes. Raises OperationFailure on an error. `index_or_name` can be either an index name (as returned by `create_index`), or an index specifier (as passed to `create_index`). An index specifier should be a list of (key, direction) pairs. Raises TypeError if index is not an instance of (str, unicode, list). .. warning:: if a custom name was used on index creation (by passing the `name` parameter to :meth:`create_index` or :meth:`ensure_index`) the index **must** be dropped by name. :Parameters: - `index_or_name`: index (or name of index) to drop """ name = index_or_name if isinstance(index_or_name, list): name = _gen_index_name(index_or_name) if not isinstance(name, basestring): raise TypeError("index_or_name must be an index name or list") self.__database.connection._purge_index(self.__database.name, self.__name, name) self.__database.command("dropIndexes", self.__name, index=name, allowable_errors=["ns not found"]) def reindex(self): """Rebuilds all indexes on this collection. .. warning:: reindex blocks all other operations (indexes are built in the foreground) and will be slow for large collections. .. versionadded:: 1.11+ """ return self.__database.command("reIndex", self.__name) def index_information(self): """Get information on this collection's indexes. Returns a dictionary where the keys are index names (as returned by create_index()) and the values are dictionaries containing information about each index. The dictionary is guaranteed to contain at least a single key, ``"key"`` which is a list of (key, direction) pairs specifying the index (as passed to create_index()). It will also contain any other information in `system.indexes`, except for the ``"ns"`` and ``"name"`` keys, which are cleaned. Example output might look like this: >>> db.test.ensure_index("x", unique=True) u'x_1' >>> db.test.index_information() {u'_id_': {u'key': [(u'_id', 1)]}, u'x_1': {u'unique': True, u'key': [(u'x', 1)]}} .. versionchanged:: 1.7 The values in the resultant dictionary are now dictionaries themselves, whose ``"key"`` item contains the list that was the value in previous versions of PyMongo. """ raw = self.__database.system.indexes.find({"ns": self.__full_name}, {"ns": 0}, as_class=SON) info = {} for index in raw: index["key"] = index["key"].items() index = dict(index) info[index.pop("name")] = index return info def options(self): """Get the options set on this collection. Returns a dictionary of options and their values - see :meth:`~pymongo.database.Database.create_collection` for more information on the possible options. Returns an empty dictionary if the collection has not been created yet. """ result = self.__database.system.namespaces.find_one( {"name": self.__full_name}) if not result: return {} options = result.get("options", {}) if "create" in options: del options["create"] return options def aggregate(self, pipeline, **kwargs): """Perform an aggregation using the aggregation framework on this collection. With :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` or :class:`~pymongo.master_slave_connection.MasterSlaveConnection`, if the `read_preference` attribute of this instance is not set to :attr:`pymongo.read_preferences.ReadPreference.PRIMARY` or the (deprecated) `slave_okay` attribute of this instance is set to `True` the `aggregate command`_ will be sent to a secondary or slave. :Parameters: - `pipeline`: a single command or list of aggregation commands - `**kwargs`: send arbitrary parameters to the aggregate command .. note:: Requires server version **>= 2.1.0**. With server version **>= 2.5.1**, pass ``cursor={}`` to retrieve unlimited aggregation results with a :class:`~pymongo.cursor.Cursor`:: pipeline = [{'$project': {'name': {'$toUpper': '$name'}}}] cursor = collection.aggregate(pipeline, cursor={}) for doc in cursor: print doc .. versionchanged:: 2.6 Added cursor support. .. versionadded:: 2.3 .. _aggregate command: http://docs.mongodb.org/manual/applications/aggregation """ if not isinstance(pipeline, (dict, list, tuple)): raise TypeError("pipeline must be a dict, list or tuple") if isinstance(pipeline, dict): pipeline = [pipeline] use_master = not self.slave_okay and not self.read_preference command_kwargs = { 'pipeline': pipeline, 'read_preference': self.read_preference, 'tag_sets': self.tag_sets, 'secondary_acceptable_latency_ms': ( self.secondary_acceptable_latency_ms), 'slave_okay': self.slave_okay, '_use_master': use_master} command_kwargs.update(kwargs) command_response = self.__database.command( "aggregate", self.__name, **command_kwargs) if 'cursor' in command_response: cursor_info = command_response['cursor'] return Cursor( self, _first_batch=cursor_info['firstBatch'], _cursor_id=cursor_info['id']) else: return command_response # TODO key and condition ought to be optional, but deprecation # could be painful as argument order would have to change. def group(self, key, condition, initial, reduce, finalize=None): """Perform a query similar to an SQL *group by* operation. Returns an array of grouped items. The `key` parameter can be: - ``None`` to use the entire document as a key. - A :class:`list` of keys (each a :class:`basestring` (:class:`str` in python 3)) to group by. - A :class:`basestring` (:class:`str` in python 3), or :class:`~bson.code.Code` instance containing a JavaScript function to be applied to each document, returning the key to group by. With :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` or :class:`~pymongo.master_slave_connection.MasterSlaveConnection`, if the `read_preference` attribute of this instance is not set to :attr:`pymongo.read_preferences.ReadPreference.PRIMARY` or :attr:`pymongo.read_preferences.ReadPreference.PRIMARY_PREFERRED`, or the (deprecated) `slave_okay` attribute of this instance is set to `True`, the group command will be sent to a secondary or slave. :Parameters: - `key`: fields to group by (see above description) - `condition`: specification of rows to be considered (as a :meth:`find` query specification) - `initial`: initial value of the aggregation counter object - `reduce`: aggregation function as a JavaScript string - `finalize`: function to be called on each object in output list. .. versionchanged:: 2.2 Removed deprecated argument: command .. versionchanged:: 1.4 The `key` argument can now be ``None`` or a JavaScript function, in addition to a :class:`list` of keys. .. versionchanged:: 1.3 The `command` argument now defaults to ``True`` and is deprecated. """ group = {} if isinstance(key, basestring): group["$keyf"] = Code(key) elif key is not None: group = {"key": helpers._fields_list_to_dict(key)} group["ns"] = self.__name group["$reduce"] = Code(reduce) group["cond"] = condition group["initial"] = initial if finalize is not None: group["finalize"] = Code(finalize) use_master = not self.slave_okay and not self.read_preference return self.__database.command("group", group, uuid_subtype=self.__uuid_subtype, read_preference=self.read_preference, tag_sets=self.tag_sets, secondary_acceptable_latency_ms=( self.secondary_acceptable_latency_ms), slave_okay=self.slave_okay, _use_master=use_master)["retval"] def rename(self, new_name, **kwargs): """Rename this collection. If operating in auth mode, client must be authorized as an admin to perform this operation. Raises :class:`TypeError` if `new_name` is not an instance of :class:`basestring` (:class:`str` in python 3). Raises :class:`~pymongo.errors.InvalidName` if `new_name` is not a valid collection name. :Parameters: - `new_name`: new name for this collection - `**kwargs` (optional): any additional rename options should be passed as keyword arguments (i.e. ``dropTarget=True``) .. versionadded:: 1.7 support for accepting keyword arguments for rename options """ if not isinstance(new_name, basestring): raise TypeError("new_name must be an instance " "of %s" % (basestring.__name__,)) if not new_name or ".." in new_name: raise InvalidName("collection names cannot be empty") if new_name[0] == "." or new_name[-1] == ".": raise InvalidName("collecion names must not start or end with '.'") if "$" in new_name and not new_name.startswith("oplog.$main"): raise InvalidName("collection names must not contain '$'") new_name = "%s.%s" % (self.__database.name, new_name) self.__database.connection.admin.command("renameCollection", self.__full_name, to=new_name, **kwargs) def distinct(self, key): """Get a list of distinct values for `key` among all documents in this collection. Raises :class:`TypeError` if `key` is not an instance of :class:`basestring` (:class:`str` in python 3). To get the distinct values for a key in the result set of a query use :meth:`~pymongo.cursor.Cursor.distinct`. :Parameters: - `key`: name of key for which we want to get the distinct values .. note:: Requires server version **>= 1.1.0** .. versionadded:: 1.1.1 """ return self.find().distinct(key) def map_reduce(self, map, reduce, out, full_response=False, **kwargs): """Perform a map/reduce operation on this collection. If `full_response` is ``False`` (default) returns a :class:`~pymongo.collection.Collection` instance containing the results of the operation. Otherwise, returns the full response from the server to the `map reduce command`_. :Parameters: - `map`: map function (as a JavaScript string) - `reduce`: reduce function (as a JavaScript string) - `out`: output collection name or `out object` (dict). See the `map reduce command`_ documentation for available options. Note: `out` options are order sensitive. :class:`~bson.son.SON` can be used to specify multiple options. e.g. SON([('replace', ), ('db', )]) - `full_response` (optional): if ``True``, return full response to this command - otherwise just return the result collection - `**kwargs` (optional): additional arguments to the `map reduce command`_ may be passed as keyword arguments to this helper method, e.g.:: >>> db.test.map_reduce(map, reduce, "myresults", limit=2) .. note:: Requires server version **>= 1.1.1** .. seealso:: :doc:`/examples/aggregation` .. versionchanged:: 2.2 Removed deprecated arguments: merge_output and reduce_output .. versionchanged:: 1.11+ DEPRECATED The merge_output and reduce_output parameters. .. versionadded:: 1.2 .. _map reduce command: http://www.mongodb.org/display/DOCS/MapReduce .. mongodoc:: mapreduce """ if not isinstance(out, (basestring, dict)): raise TypeError("'out' must be an instance of " "%s or dict" % (basestring.__name__,)) if isinstance(out, dict) and out.get('inline'): must_use_master = False else: must_use_master = True response = self.__database.command("mapreduce", self.__name, uuid_subtype=self.__uuid_subtype, map=map, reduce=reduce, read_preference=self.read_preference, tag_sets=self.tag_sets, secondary_acceptable_latency_ms=( self.secondary_acceptable_latency_ms), out=out, _use_master=must_use_master, **kwargs) if full_response or not response.get('result'): return response elif isinstance(response['result'], dict): dbase = response['result']['db'] coll = response['result']['collection'] return self.__database.connection[dbase][coll] else: return self.__database[response["result"]] def inline_map_reduce(self, map, reduce, full_response=False, **kwargs): """Perform an inline map/reduce operation on this collection. Perform the map/reduce operation on the server in RAM. A result collection is not created. The result set is returned as a list of documents. If `full_response` is ``False`` (default) returns the result documents in a list. Otherwise, returns the full response from the server to the `map reduce command`_. With :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` or :class:`~pymongo.master_slave_connection.MasterSlaveConnection`, if the `read_preference` attribute of this instance is not set to :attr:`pymongo.read_preferences.ReadPreference.PRIMARY` or :attr:`pymongo.read_preferences.ReadPreference.PRIMARY_PREFERRED`, or the (deprecated) `slave_okay` attribute of this instance is set to `True`, the inline map reduce will be run on a secondary or slave. :Parameters: - `map`: map function (as a JavaScript string) - `reduce`: reduce function (as a JavaScript string) - `full_response` (optional): if ``True``, return full response to this command - otherwise just return the result collection - `**kwargs` (optional): additional arguments to the `map reduce command`_ may be passed as keyword arguments to this helper method, e.g.:: >>> db.test.inline_map_reduce(map, reduce, limit=2) .. note:: Requires server version **>= 1.7.4** .. versionadded:: 1.10 """ use_master = not self.slave_okay and not self.read_preference res = self.__database.command("mapreduce", self.__name, uuid_subtype=self.__uuid_subtype, read_preference=self.read_preference, tag_sets=self.tag_sets, secondary_acceptable_latency_ms=( self.secondary_acceptable_latency_ms), slave_okay=self.slave_okay, _use_master=use_master, map=map, reduce=reduce, out={"inline": 1}, **kwargs) if full_response: return res else: return res.get("results") def find_and_modify(self, query={}, update=None, upsert=False, sort=None, full_response=False, **kwargs): """Update and return an object. This is a thin wrapper around the findAndModify_ command. The positional arguments are designed to match the first three arguments to :meth:`update` however most options should be passed as named parameters. Either `update` or `remove` arguments are required, all others are optional. Returns either the object before or after modification based on `new` parameter. If no objects match the `query` and `upsert` is false, returns ``None``. If upserting and `new` is false, returns ``{}``. If the full_response parameter is ``True``, the return value will be the entire response object from the server, including the 'ok' and 'lastErrorObject' fields, rather than just the modified object. This is useful mainly because the 'lastErrorObject' document holds information about the command's execution. :Parameters: - `query`: filter for the update (default ``{}``) - `update`: see second argument to :meth:`update` (no default) - `upsert`: insert if object doesn't exist (default ``False``) - `sort`: a list of (key, direction) pairs specifying the sort order for this query. See :meth:`~pymongo.cursor.Cursor.sort` for details. - `full_response`: return the entire response object from the server (default ``False``) - `remove`: remove rather than updating (default ``False``) - `new`: return updated rather than original object (default ``False``) - `fields`: see second argument to :meth:`find` (default all) - `**kwargs`: any other options the findAndModify_ command supports can be passed here. .. mongodoc:: findAndModify .. _findAndModify: http://dochub.mongodb.org/core/findAndModify .. note:: Requires server version **>= 1.3.0** .. versionchanged:: 2.5 Added the optional full_response parameter .. versionchanged:: 2.4 Deprecated the use of mapping types for the sort parameter .. versionadded:: 1.10 """ if (not update and not kwargs.get('remove', None)): raise ValueError("Must either update or remove") if (update and kwargs.get('remove', None)): raise ValueError("Can't do both update and remove") # No need to include empty args if query: kwargs['query'] = query if update: kwargs['update'] = update if upsert: kwargs['upsert'] = upsert if sort: # Accept a list of tuples to match Cursor's sort parameter. if isinstance(sort, list): kwargs['sort'] = helpers._index_document(sort) # Accept OrderedDict, SON, and dict with len == 1 so we # don't break existing code already using find_and_modify. elif (isinstance(sort, ordered_types) or isinstance(sort, dict) and len(sort) == 1): warnings.warn("Passing mapping types for `sort` is deprecated," " use a list of (key, direction) pairs instead", DeprecationWarning, stacklevel=2) kwargs['sort'] = sort else: raise TypeError("sort must be a list of (key, direction) " "pairs, a dict of len 1, or an instance of " "SON or OrderedDict") no_obj_error = "No matching object found" out = self.__database.command("findAndModify", self.__name, allowable_errors=[no_obj_error], uuid_subtype=self.__uuid_subtype, **kwargs) if not out['ok']: if out["errmsg"] == no_obj_error: return None else: # Should never get here b/c of allowable_errors raise ValueError("Unexpected Error: %s" % (out,)) if full_response: return out else: return out.get('value') def __iter__(self): return self def next(self): raise TypeError("'Collection' object is not iterable") def __call__(self, *args, **kwargs): """This is only here so that some API misusages are easier to debug. """ if "." not in self.__name: raise TypeError("'Collection' object is not callable. If you " "meant to call the '%s' method on a 'Database' " "object it is failing because no such method " "exists." % self.__name) raise TypeError("'Collection' object is not callable. If you meant to " "call the '%s' method on a 'Collection' object it is " "failing because no such method exists." % self.__name.split(".")[-1]) pymongo-2.6.3/pymongo/common.py000066400000000000000000000555601223300253600165530ustar00rootroot00000000000000# Copyright 2011-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you # may not use this file except in compliance with the License. You # may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. See the License for the specific language governing # permissions and limitations under the License. """Functions and classes common to multiple pymongo modules.""" import sys import warnings from pymongo import read_preferences from pymongo.auth import MECHANISMS from pymongo.read_preferences import ReadPreference from pymongo.errors import ConfigurationError HAS_SSL = True try: import ssl except ImportError: HAS_SSL = False # Jython 2.7 includes an incomplete ssl module. See PYTHON-498. if sys.platform.startswith('java'): HAS_SSL = False def raise_config_error(key, dummy): """Raise ConfigurationError with the given key name.""" raise ConfigurationError("Unknown option %s" % (key,)) def validate_boolean(option, value): """Validates that 'value' is 'true' or 'false'. """ if isinstance(value, bool): return value elif isinstance(value, basestring): if value not in ('true', 'false'): raise ConfigurationError("The value of %s must be " "'true' or 'false'" % (option,)) return value == 'true' raise TypeError("Wrong type for %s, value must be a boolean" % (option,)) def validate_integer(option, value): """Validates that 'value' is an integer (or basestring representation). """ if isinstance(value, (int, long)): return value elif isinstance(value, basestring): if not value.isdigit(): raise ConfigurationError("The value of %s must be " "an integer" % (option,)) return int(value) raise TypeError("Wrong type for %s, value must be an integer" % (option,)) def validate_positive_integer(option, value): """Validate that 'value' is a positive integer. """ val = validate_integer(option, value) if val < 0: raise ConfigurationError("The value of %s must be " "a positive integer" % (option,)) return val def validate_readable(option, value): """Validates that 'value' is file-like and readable. """ # First make sure its a string py3.3 open(True, 'r') succeeds # Used in ssl cert checking due to poor ssl module error reporting value = validate_basestring(option, value) open(value, 'r').close() return value def validate_cert_reqs(option, value): """Validate the cert reqs are valid. It must be None or one of the three values ``ssl.CERT_NONE``, ``ssl.CERT_OPTIONAL`` or ``ssl.CERT_REQUIRED``""" if value is None: return value if HAS_SSL: if value in (ssl.CERT_NONE, ssl.CERT_OPTIONAL, ssl.CERT_REQUIRED): return value raise ConfigurationError("The value of %s must be one of: " "`ssl.CERT_NONE`, `ssl.CERT_OPTIONAL` or " "`ssl.CERT_REQUIRED" % (option,)) else: raise ConfigurationError("The value of %s is set but can't be " "validated. The ssl module is not available" % (option,)) def validate_positive_integer_or_none(option, value): """Validate that 'value' is a positive integer or None. """ if value is None: return value return validate_positive_integer(option, value) def validate_basestring(option, value): """Validates that 'value' is an instance of `basestring`. """ if isinstance(value, basestring): return value raise TypeError("Wrong type for %s, value must be an " "instance of %s" % (option, basestring.__name__)) def validate_int_or_basestring(option, value): """Validates that 'value' is an integer or string. """ if isinstance(value, (int, long)): return value elif isinstance(value, basestring): if value.isdigit(): return int(value) return value raise TypeError("Wrong type for %s, value must be an " "integer or a string" % (option,)) def validate_positive_float(option, value): """Validates that 'value' is a float, or can be converted to one, and is positive. """ err = ConfigurationError("%s must be a positive int or float" % (option,)) try: value = float(value) except (ValueError, TypeError): raise err # float('inf') doesn't work in 2.4 or 2.5 on Windows, so just cap floats at # one billion - this is a reasonable approximation for infinity if not 0 < value < 1e9: raise err return value def validate_timeout_or_none(option, value): """Validates a timeout specified in milliseconds returning a value in floating point seconds. """ if value is None: return value return validate_positive_float(option, value) / 1000.0 def validate_read_preference(dummy, value): """Validate read preference for a ReplicaSetConnection. """ if value in read_preferences.modes: return value # Also allow string form of enum for uri_parser try: return read_preferences.mongos_enum(value) except ValueError: raise ConfigurationError("Not a valid read preference") def validate_tag_sets(dummy, value): """Validate tag sets for a ReplicaSetConnection. """ if value is None: return [{}] if not isinstance(value, list): raise ConfigurationError(( "Tag sets %s invalid, must be a list" ) % repr(value)) if len(value) == 0: raise ConfigurationError(( "Tag sets %s invalid, must be None or contain at least one set of" " tags") % repr(value)) for tags in value: if not isinstance(tags, dict): raise ConfigurationError( "Tag set %s invalid, must be a dict" % repr(tags)) return value def validate_auth_mechanism(option, value): """Validate the authMechanism URI option. """ if value not in MECHANISMS: raise ConfigurationError("%s must be in " "%s" % (option, MECHANISMS)) return value # jounal is an alias for j, # wtimeoutms is an alias for wtimeout VALIDATORS = { 'replicaset': validate_basestring, 'slaveok': validate_boolean, 'slave_okay': validate_boolean, 'safe': validate_boolean, 'w': validate_int_or_basestring, 'wtimeout': validate_integer, 'wtimeoutms': validate_integer, 'fsync': validate_boolean, 'j': validate_boolean, 'journal': validate_boolean, 'connecttimeoutms': validate_timeout_or_none, 'sockettimeoutms': validate_timeout_or_none, 'waitqueuetimeoutms': validate_timeout_or_none, 'waitqueuemultiple': validate_positive_integer_or_none, 'ssl': validate_boolean, 'ssl_keyfile': validate_readable, 'ssl_certfile': validate_readable, 'ssl_cert_reqs': validate_cert_reqs, 'ssl_ca_certs': validate_readable, 'readpreference': validate_read_preference, 'read_preference': validate_read_preference, 'tag_sets': validate_tag_sets, 'secondaryacceptablelatencyms': validate_positive_float, 'secondary_acceptable_latency_ms': validate_positive_float, 'auto_start_request': validate_boolean, 'use_greenlets': validate_boolean, 'authmechanism': validate_auth_mechanism, 'authsource': validate_basestring, 'gssapiservicename': validate_basestring, } _AUTH_OPTIONS = frozenset(['gssapiservicename']) def validate_auth_option(option, value): """Validate optional authentication parameters. """ lower, value = validate(option, value) if lower not in _AUTH_OPTIONS: raise ConfigurationError('Unknown ' 'authentication option: %s' % (option,)) return lower, value def validate(option, value): """Generic validation function. """ lower = option.lower() validator = VALIDATORS.get(lower, raise_config_error) value = validator(option, value) return lower, value SAFE_OPTIONS = frozenset([ 'w', 'wtimeout', 'wtimeoutms', 'fsync', 'j', 'journal' ]) class WriteConcern(dict): def __init__(self, *args, **kwargs): """A subclass of dict that overrides __setitem__ to validate write concern options. """ super(WriteConcern, self).__init__(*args, **kwargs) def __setitem__(self, key, value): if key not in SAFE_OPTIONS: raise ConfigurationError("%s is not a valid write " "concern option." % (key,)) key, value = validate(key, value) super(WriteConcern, self).__setitem__(key, value) class BaseObject(object): """A base class that provides attributes and methods common to multiple pymongo classes. SHOULD NOT BE USED BY DEVELOPERS EXTERNAL TO 10GEN """ def __init__(self, **options): self.__slave_okay = False self.__read_pref = ReadPreference.PRIMARY self.__tag_sets = [{}] self.__secondary_acceptable_latency_ms = 15 self.__safe = None self.__write_concern = WriteConcern() self.__set_options(options) if (self.__read_pref == ReadPreference.PRIMARY and self.__tag_sets != [{}] ): raise ConfigurationError( "ReadPreference PRIMARY cannot be combined with tags") # If safe hasn't been implicitly set by write concerns then set it. if self.__safe is None: if options.get("w") == 0: self.__safe = False else: self.__safe = validate_boolean('safe', options.get("safe", True)) # Note: 'safe' is always passed by Connection and ReplicaSetConnection # Always do the most "safe" thing, but warn about conflicts. if self.__safe and options.get('w') == 0: warnings.warn("Conflicting write concerns. 'w' set to 0 " "but other options have enabled write concern. " "Please set 'w' to a value other than 0.", UserWarning) def __set_safe_option(self, option, value): """Validates and sets getlasterror options for this object (Connection, Database, Collection, etc.) """ if value is None: self.__write_concern.pop(option, None) else: self.__write_concern[option] = value if option != "w" or value != 0: self.__safe = True def __set_options(self, options): """Validates and sets all options passed to this object.""" for option, value in options.iteritems(): if option in ('slave_okay', 'slaveok'): self.__slave_okay = validate_boolean(option, value) elif option in ('read_preference', "readpreference"): self.__read_pref = validate_read_preference(option, value) elif option == 'tag_sets': self.__tag_sets = validate_tag_sets(option, value) elif option in ( 'secondaryacceptablelatencyms', 'secondary_acceptable_latency_ms' ): self.__secondary_acceptable_latency_ms = \ validate_positive_float(option, value) elif option in SAFE_OPTIONS: if option == 'journal': self.__set_safe_option('j', value) elif option == 'wtimeoutms': self.__set_safe_option('wtimeout', value) else: self.__set_safe_option(option, value) def __set_write_concern(self, value): """Property setter for write_concern.""" if not isinstance(value, dict): raise ConfigurationError("write_concern must be an " "instance of dict or a subclass.") # Make a copy here to avoid users accidentally setting the # same dict on multiple instances. wc = WriteConcern() for k, v in value.iteritems(): # Make sure we validate each option. wc[k] = v self.__write_concern = wc def __get_write_concern(self): """The default write concern for this instance. Supports dict style access for getting/setting write concern options. Valid options include: - `w`: (integer or string) If this is a replica set, write operations will block until they have been replicated to the specified number or tagged set of servers. `w=` always includes the replica set primary (e.g. w=3 means write to the primary and wait until replicated to **two** secondaries). **Setting w=0 disables write acknowledgement and all other write concern options.** - `wtimeout`: (integer) Used in conjunction with `w`. Specify a value in milliseconds to control how long to wait for write propagation to complete. If replication does not complete in the given timeframe, a timeout exception is raised. - `j`: If ``True`` block until write operations have been committed to the journal. Ignored if the server is running without journaling. - `fsync`: If ``True`` force the database to fsync all files before returning. When used with `j` the server awaits the next group commit before returning. >>> m = pymongo.MongoClient() >>> m.write_concern {} >>> m.write_concern = {'w': 2, 'wtimeout': 1000} >>> m.write_concern {'wtimeout': 1000, 'w': 2} >>> m.write_concern['j'] = True >>> m.write_concern {'wtimeout': 1000, 'j': True, 'w': 2} >>> m.write_concern = {'j': True} >>> m.write_concern {'j': True} >>> # Disable write acknowledgement and write concern ... >>> m.write_concern['w'] = 0 .. note:: Accessing :attr:`write_concern` returns its value (a subclass of :class:`dict`), not a copy. .. warning:: If you are using :class:`~pymongo.connection.Connection` or :class:`~pymongo.replica_set_connection.ReplicaSetConnection` make sure you explicitly set ``w`` to 1 (or a greater value) or :attr:`safe` to ``True``. Unlike calling :meth:`set_lasterror_options`, setting an option in :attr:`write_concern` does not implicitly set :attr:`safe` to ``True``. """ # To support dict style access we have to return the actual # WriteConcern here, not a copy. return self.__write_concern write_concern = property(__get_write_concern, __set_write_concern) def __get_slave_okay(self): """DEPRECATED. Use :attr:`read_preference` instead. .. versionchanged:: 2.1 Deprecated slave_okay. .. versionadded:: 2.0 """ return self.__slave_okay def __set_slave_okay(self, value): """Property setter for slave_okay""" warnings.warn("slave_okay is deprecated. Please use " "read_preference instead.", DeprecationWarning, stacklevel=2) self.__slave_okay = validate_boolean('slave_okay', value) slave_okay = property(__get_slave_okay, __set_slave_okay) def __get_read_pref(self): """The read preference mode for this instance. See :class:`~pymongo.read_preferences.ReadPreference` for available options. .. versionadded:: 2.1 """ return self.__read_pref def __set_read_pref(self, value): """Property setter for read_preference""" self.__read_pref = validate_read_preference('read_preference', value) read_preference = property(__get_read_pref, __set_read_pref) def __get_acceptable_latency(self): """Any replica-set member whose ping time is within secondary_acceptable_latency_ms of the nearest member may accept reads. Defaults to 15 milliseconds. See :class:`~pymongo.read_preferences.ReadPreference`. .. versionadded:: 2.3 .. note:: ``secondary_acceptable_latency_ms`` is ignored when talking to a replica set *through* a mongos. The equivalent is the localThreshold_ command line option. .. _localThreshold: http://docs.mongodb.org/manual/reference/mongos/#cmdoption-mongos--localThreshold """ return self.__secondary_acceptable_latency_ms def __set_acceptable_latency(self, value): """Property setter for secondary_acceptable_latency_ms""" self.__secondary_acceptable_latency_ms = (validate_positive_float( 'secondary_acceptable_latency_ms', value)) secondary_acceptable_latency_ms = property( __get_acceptable_latency, __set_acceptable_latency) def __get_tag_sets(self): """Set ``tag_sets`` to a list of dictionaries like [{'dc': 'ny'}] to read only from members whose ``dc`` tag has the value ``"ny"``. To specify a priority-order for tag sets, provide a list of tag sets: ``[{'dc': 'ny'}, {'dc': 'la'}, {}]``. A final, empty tag set, ``{}``, means "read from any member that matches the mode, ignoring tags." ReplicaSetConnection tries each set of tags in turn until it finds a set of tags with at least one matching member. .. seealso:: `Data-Center Awareness `_ .. versionadded:: 2.3 """ return self.__tag_sets def __set_tag_sets(self, value): """Property setter for tag_sets""" self.__tag_sets = validate_tag_sets('tag_sets', value) tag_sets = property(__get_tag_sets, __set_tag_sets) def __get_safe(self): """**DEPRECATED:** Use the 'w' :attr:`write_concern` option instead. Use getlasterror with every write operation? .. versionadded:: 2.0 """ return self.__safe def __set_safe(self, value): """Property setter for safe""" warnings.warn("safe is deprecated. Please use the" " 'w' write_concern option instead.", DeprecationWarning, stacklevel=2) self.__safe = validate_boolean('safe', value) safe = property(__get_safe, __set_safe) def get_lasterror_options(self): """DEPRECATED: Use :attr:`write_concern` instead. Returns a dict of the getlasterror options set on this instance. .. versionchanged:: 2.4 Deprecated get_lasterror_options. .. versionadded:: 2.0 """ warnings.warn("get_lasterror_options is deprecated. Please use " "write_concern instead.", DeprecationWarning, stacklevel=2) return self.__write_concern.copy() def set_lasterror_options(self, **kwargs): """DEPRECATED: Use :attr:`write_concern` instead. Set getlasterror options for this instance. Valid options include j=, w=, wtimeout=, and fsync=. Implies safe=True. :Parameters: - `**kwargs`: Options should be passed as keyword arguments (e.g. w=2, fsync=True) .. versionchanged:: 2.4 Deprecated set_lasterror_options. .. versionadded:: 2.0 """ warnings.warn("set_lasterror_options is deprecated. Please use " "write_concern instead.", DeprecationWarning, stacklevel=2) for key, value in kwargs.iteritems(): self.__set_safe_option(key, value) def unset_lasterror_options(self, *options): """DEPRECATED: Use :attr:`write_concern` instead. Unset getlasterror options for this instance. If no options are passed unsets all getlasterror options. This does not set `safe` to False. :Parameters: - `*options`: The list of options to unset. .. versionchanged:: 2.4 Deprecated unset_lasterror_options. .. versionadded:: 2.0 """ warnings.warn("unset_lasterror_options is deprecated. Please use " "write_concern instead.", DeprecationWarning, stacklevel=2) if len(options): for option in options: self.__write_concern.pop(option, None) else: self.__write_concern = WriteConcern() def _get_wc_override(self): """Get write concern override. Used in internal methods that **must** do acknowledged write ops. We don't want to override user write concern options if write concern is already enabled. """ if self.safe and self.__write_concern.get('w') != 0: return {} return {'w': 1} def _get_write_mode(self, safe=None, **options): """Get the current write mode. Determines if the current write is safe or not based on the passed in or inherited safe value, write_concern values, or passed options. :Parameters: - `safe`: check that the operation succeeded? - `**options`: overriding write concern options. .. versionadded:: 2.3 """ # Don't ever send w=1 to the server. def pop1(dct): if dct.get('w') == 1: dct.pop('w') return dct if safe is not None: warnings.warn("The safe parameter is deprecated. Please use " "write concern options instead.", DeprecationWarning, stacklevel=3) validate_boolean('safe', safe) # Passed options override collection level defaults. if safe is not None or options: if safe or options: if not options: options = self.__write_concern.copy() # Backwards compatability edge case. Call getLastError # with no options if safe=True was passed but collection # level defaults have been disabled with w=0. # These should be equivalent: # Connection(w=0).foo.bar.insert({}, safe=True) # MongoClient(w=0).foo.bar.insert({}, w=1) if options.get('w') == 0: return True, {} # Passing w=0 overrides passing safe=True. return options.get('w') != 0, pop1(options) return False, {} # Fall back to collection level defaults. # w=0 takes precedence over self.safe = True if self.__write_concern.get('w') == 0: return False, {} elif self.safe or self.__write_concern.get('w', 0) != 0: return True, pop1(self.__write_concern.copy()) return False, {} pymongo-2.6.3/pymongo/connection.py000066400000000000000000000262611223300253600174160ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you # may not use this file except in compliance with the License. You # may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. See the License for the specific language governing # permissions and limitations under the License. """Tools for connecting to MongoDB. .. warning:: **DEPRECATED:** Please use :mod:`~pymongo.mongo_client` instead. .. seealso:: Module :mod:`~pymongo.master_slave_connection` for connecting to master-slave clusters, and :doc:`/examples/high_availability` for an example of how to connect to a replica set, or specify a list of mongos instances for automatic failover. To get a :class:`~pymongo.database.Database` instance from a :class:`Connection` use either dictionary-style or attribute-style access: .. doctest:: >>> from pymongo import Connection >>> c = Connection() >>> c.test_database Database(Connection('localhost', 27017), u'test_database') >>> c['test-database'] Database(Connection('localhost', 27017), u'test-database') """ from pymongo.mongo_client import MongoClient from pymongo.errors import ConfigurationError class Connection(MongoClient): """Connection to MongoDB. """ def __init__(self, host=None, port=None, max_pool_size=None, network_timeout=None, document_class=dict, tz_aware=False, _connect=True, **kwargs): """Create a new connection to a single MongoDB instance at *host:port*. .. warning:: **DEPRECATED:** :class:`Connection` is deprecated. Please use :class:`~pymongo.mongo_client.MongoClient` instead. The resultant connection object has connection-pooling built in. It also performs auto-reconnection when necessary. If an operation fails because of a connection error, :class:`~pymongo.errors.ConnectionFailure` is raised. If auto-reconnection will be performed, :class:`~pymongo.errors.AutoReconnect` will be raised. Application code should handle this exception (recognizing that the operation failed) and then continue to execute. Raises :class:`TypeError` if port is not an instance of ``int``. Raises :class:`~pymongo.errors.ConnectionFailure` if the connection cannot be made. The `host` parameter can be a full `mongodb URI `_, in addition to a simple hostname. It can also be a list of hostnames or URIs. Any port specified in the host string(s) will override the `port` parameter. If multiple mongodb URIs containing database or auth information are passed, the last database, username, and password present will be used. For username and passwords reserved characters like ':', '/', '+' and '@' must be escaped following RFC 2396. :Parameters: - `host` (optional): hostname or IP address of the instance to connect to, or a mongodb URI, or a list of hostnames / mongodb URIs. If `host` is an IPv6 literal it must be enclosed in '[' and ']' characters following the RFC2732 URL syntax (e.g. '[::1]' for localhost) - `port` (optional): port number on which to connect - `max_pool_size` (optional): The maximum number of connections that the pool will open simultaneously. If this is set, operations will block if there are `max_pool_size` outstanding connections from the pool. By default the pool size is unlimited. - `network_timeout` (optional): timeout (in seconds) to use for socket operations - default is no timeout - `document_class` (optional): default class to use for documents returned from queries on this connection - `tz_aware` (optional): if ``True``, :class:`~datetime.datetime` instances returned as values in a document by this :class:`Connection` will be timezone aware (otherwise they will be naive) | **Other optional parameters can be passed as keyword arguments:** - `socketTimeoutMS`: (integer) How long (in milliseconds) a send or receive on a socket can take before timing out. - `connectTimeoutMS`: (integer) How long (in milliseconds) a connection can take to be opened before timing out. - `auto_start_request`: If ``True`` (the default), each thread that accesses this Connection has a socket allocated to it for the thread's lifetime. This ensures consistent reads, even if you read after an unsafe write. - `use_greenlets`: if ``True``, :meth:`start_request()` will ensure that the current greenlet uses the same socket for all operations until :meth:`end_request()` | **Write Concern options:** - `safe`: :class:`Connection` **disables** acknowledgement of write operations. Use ``safe=True`` to enable write acknowledgement. - `w`: (integer or string) If this is a replica set, write operations will block until they have been replicated to the specified number or tagged set of servers. `w=` always includes the replica set primary (e.g. w=3 means write to the primary and wait until replicated to **two** secondaries). Implies safe=True. - `wtimeout`: (integer) Used in conjunction with `w`. Specify a value in milliseconds to control how long to wait for write propagation to complete. If replication does not complete in the given timeframe, a timeout exception is raised. Implies safe=True. - `j`: If ``True`` block until write operations have been committed to the journal. Ignored if the server is running without journaling. Implies safe=True. - `fsync`: If ``True`` force the database to fsync all files before returning. When used with `j` the server awaits the next group commit before returning. Implies safe=True. | **Replica-set keyword arguments for connecting with a replica-set - either directly or via a mongos:** | (ignored by standalone mongod instances) - `slave_okay` or `slaveOk` (deprecated): Use `read_preference` instead. - `replicaSet`: (string) The name of the replica-set to connect to. The driver will verify that the replica-set it connects to matches this name. Implies that the hosts specified are a seed list and the driver should attempt to find all members of the set. *Ignored by mongos*. - `read_preference`: The read preference for this client. If connecting to a secondary then a read preference mode *other* than PRIMARY is required - otherwise all queries will throw a :class:`~pymongo.errors.AutoReconnect` "not master" error. See :class:`~pymongo.read_preferences.ReadPreference` for all available read preference options. - `tag_sets`: Ignored unless connecting to a replica-set via mongos. Specify a priority-order for tag sets, provide a list of tag sets: ``[{'dc': 'ny'}, {'dc': 'la'}, {}]``. A final, empty tag set, ``{}``, means "read from any member that matches the mode, ignoring tags. | **SSL configuration:** - `ssl`: If ``True``, create the connection to the server using SSL. - `ssl_keyfile`: The private keyfile used to identify the local connection against mongod. If included with the ``certfile` then only the ``ssl_certfile`` is needed. Implies ``ssl=True``. - `ssl_certfile`: The certificate file used to identify the local connection against mongod. Implies ``ssl=True``. - `ssl_cert_reqs`: The parameter cert_reqs specifies whether a certificate is required from the other side of the connection, and whether it will be validated if provided. It must be one of the three values ``ssl.CERT_NONE`` (certificates ignored), ``ssl.CERT_OPTIONAL`` (not required, but validated if provided), or ``ssl.CERT_REQUIRED`` (required and validated). If the value of this parameter is not ``ssl.CERT_NONE``, then the ``ssl_ca_certs`` parameter must point to a file of CA certificates. Implies ``ssl=True``. - `ssl_ca_certs`: The ca_certs file contains a set of concatenated "certification authority" certificates, which are used to validate certificates passed from the other end of the connection. Implies ``ssl=True``. .. seealso:: :meth:`end_request` .. versionchanged:: 2.5 Added additional ssl options .. versionchanged:: 2.3 Added support for failover between mongos seed list members. .. versionchanged:: 2.2 Added `auto_start_request` option back. Added `use_greenlets` option. .. versionchanged:: 2.1 Support `w` = integer or string. Added `ssl` option. DEPRECATED slave_okay/slaveOk. .. versionchanged:: 2.0 `slave_okay` is a pure keyword argument. Added support for safe, and getlasterror options as keyword arguments. .. versionchanged:: 1.11 Added `max_pool_size`. Completely removed previously deprecated `pool_size`, `auto_start_request` and `timeout` parameters. .. versionchanged:: 1.8 The `host` parameter can now be a full `mongodb URI `_, in addition to a simple hostname. It can also be a list of hostnames or URIs. .. versionadded:: 1.8 The `tz_aware` parameter. .. versionadded:: 1.7 The `document_class` parameter. .. versionadded:: 1.1 The `network_timeout` parameter. .. mongodoc:: connections """ if network_timeout is not None: if (not isinstance(network_timeout, (int, float)) or network_timeout <= 0): raise ConfigurationError("network_timeout must " "be a positive integer") kwargs['socketTimeoutMS'] = network_timeout * 1000 kwargs['auto_start_request'] = kwargs.get('auto_start_request', True) kwargs['safe'] = kwargs.get('safe', False) super(Connection, self).__init__(host, port, max_pool_size, document_class, tz_aware, _connect, **kwargs) def __repr__(self): if len(self.nodes) == 1: return "Connection(%r, %r)" % (self.host, self.port) else: return "Connection(%r)" % ["%s:%d" % n for n in self.nodes] def next(self): raise TypeError("'Connection' object is not iterable") pymongo-2.6.3/pymongo/cursor.py000066400000000000000000001100051223300253600165620ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Cursor class to iterate over Mongo query results.""" import copy from collections import deque from bson import RE_TYPE from bson.code import Code from bson.son import SON from pymongo import helpers, message, read_preferences from pymongo.read_preferences import ReadPreference, secondary_ok_commands from pymongo.errors import (InvalidOperation, AutoReconnect) _QUERY_OPTIONS = { "tailable_cursor": 2, "slave_okay": 4, "oplog_replay": 8, "no_timeout": 16, "await_data": 32, "exhaust": 64, "partial": 128} # This has to be an old style class due to # http://bugs.jython.org/issue1057 class _SocketManager: """Used with exhaust cursors to ensure the socket is returned. """ def __init__(self, sock, pool): self.sock = sock self.pool = pool self.__closed = False def __del__(self): self.close() def close(self): """Return this instance's socket to the connection pool. """ if not self.__closed: self.__closed = True self.pool.maybe_return_socket(self.sock) self.sock, self.pool = None, None # TODO might be cool to be able to do find().include("foo") or # find().exclude(["bar", "baz"]) or find().slice("a", 1, 2) as an # alternative to the fields specifier. class Cursor(object): """A cursor / iterator over Mongo query results. """ def __init__(self, collection, spec=None, fields=None, skip=0, limit=0, timeout=True, snapshot=False, tailable=False, sort=None, max_scan=None, as_class=None, slave_okay=False, await_data=False, partial=False, manipulate=True, read_preference=ReadPreference.PRIMARY, tag_sets=[{}], secondary_acceptable_latency_ms=None, exhaust=False, _must_use_master=False, _uuid_subtype=None, _first_batch=None, _cursor_id=None, **kwargs): """Create a new cursor. Should not be called directly by application developers - see :meth:`~pymongo.collection.Collection.find` instead. .. mongodoc:: cursors """ self.__id = _cursor_id self.__is_command_cursor = _cursor_id is not None if spec is None: spec = {} if not isinstance(spec, dict): raise TypeError("spec must be an instance of dict") if not isinstance(skip, int): raise TypeError("skip must be an instance of int") if not isinstance(limit, int): raise TypeError("limit must be an instance of int") if not isinstance(timeout, bool): raise TypeError("timeout must be an instance of bool") if not isinstance(snapshot, bool): raise TypeError("snapshot must be an instance of bool") if not isinstance(tailable, bool): raise TypeError("tailable must be an instance of bool") if not isinstance(slave_okay, bool): raise TypeError("slave_okay must be an instance of bool") if not isinstance(await_data, bool): raise TypeError("await_data must be an instance of bool") if not isinstance(partial, bool): raise TypeError("partial must be an instance of bool") if not isinstance(exhaust, bool): raise TypeError("exhaust must be an instance of bool") if fields is not None: if not fields: fields = {"_id": 1} if not isinstance(fields, dict): fields = helpers._fields_list_to_dict(fields) if as_class is None: as_class = collection.database.connection.document_class self.__collection = collection self.__spec = spec self.__fields = fields self.__skip = skip self.__limit = limit self.__batch_size = 0 # Exhaust cursor support if self.__collection.database.connection.is_mongos and exhaust: raise InvalidOperation('Exhaust cursors are ' 'not supported by mongos') if limit and exhaust: raise InvalidOperation("Can't use limit and exhaust together.") self.__exhaust = exhaust self.__exhaust_mgr = None # This is ugly. People want to be able to do cursor[5:5] and # get an empty result set (old behavior was an # exception). It's hard to do that right, though, because the # server uses limit(0) to mean 'no limit'. So we set __empty # in that case and check for it when iterating. We also unset # it anytime we change __limit. self.__empty = False self.__snapshot = snapshot self.__ordering = sort and helpers._index_document(sort) or None self.__max_scan = max_scan self.__explain = False self.__hint = None self.__as_class = as_class self.__slave_okay = slave_okay self.__manipulate = manipulate self.__read_preference = read_preference self.__tag_sets = tag_sets self.__secondary_acceptable_latency_ms = secondary_acceptable_latency_ms self.__tz_aware = collection.database.connection.tz_aware self.__must_use_master = _must_use_master self.__uuid_subtype = _uuid_subtype or collection.uuid_subtype self.__data = deque(_first_batch or []) self.__connection_id = None self.__retrieved = 0 self.__killed = False self.__query_flags = 0 if tailable: self.__query_flags |= _QUERY_OPTIONS["tailable_cursor"] if not timeout: self.__query_flags |= _QUERY_OPTIONS["no_timeout"] if tailable and await_data: self.__query_flags |= _QUERY_OPTIONS["await_data"] if exhaust: self.__query_flags |= _QUERY_OPTIONS["exhaust"] if partial: self.__query_flags |= _QUERY_OPTIONS["partial"] # this is for passing network_timeout through if it's specified # need to use kwargs as None is a legit value for network_timeout self.__kwargs = kwargs @property def collection(self): """The :class:`~pymongo.collection.Collection` that this :class:`Cursor` is iterating. .. versionadded:: 1.1 """ return self.__collection def __del__(self): if self.__id and not self.__killed: self.__die() def rewind(self): """Rewind this cursor to its unevaluated state. Reset this cursor if it has been partially or completely evaluated. Any options that are present on the cursor will remain in effect. Future iterating performed on this cursor will cause new queries to be sent to the server, even if the resultant data has already been retrieved by this cursor. """ self.__check_not_command_cursor('rewind') self.__data = deque() self.__id = None self.__connection_id = None self.__retrieved = 0 self.__killed = False return self def clone(self): """Get a clone of this cursor. Returns a new Cursor instance with options matching those that have been set on the current instance. The clone will be completely unevaluated, even if the current instance has been partially or completely evaluated. """ return self.__clone(True) def __clone(self, deepcopy=True): self.__check_not_command_cursor('clone') clone = Cursor(self.__collection) values_to_clone = ("spec", "fields", "skip", "limit", "snapshot", "ordering", "explain", "hint", "batch_size", "max_scan", "as_class", "slave_okay", "manipulate", "read_preference", "tag_sets", "secondary_acceptable_latency_ms", "must_use_master", "uuid_subtype", "query_flags", "kwargs") data = dict((k, v) for k, v in self.__dict__.iteritems() if k.startswith('_Cursor__') and k[9:] in values_to_clone) if deepcopy: data = self.__deepcopy(data) clone.__dict__.update(data) return clone def __die(self): """Closes this cursor. """ if self.__id and not self.__killed: if self.__exhaust and self.__exhaust_mgr: # If this is an exhaust cursor and we haven't completely # exhausted the result set we *must* close the socket # to stop the server from sending more data. self.__exhaust_mgr.sock.close() else: connection = self.__collection.database.connection if self.__connection_id is not None: connection.close_cursor(self.__id, self.__connection_id) else: connection.close_cursor(self.__id) if self.__exhaust and self.__exhaust_mgr: self.__exhaust_mgr.close() self.__killed = True def close(self): """Explicitly close / kill this cursor. Required for PyPy, Jython and other Python implementations that don't use reference counting garbage collection. """ self.__die() def __query_spec(self): """Get the spec to use for a query. """ operators = {} if self.__ordering: operators["$orderby"] = self.__ordering if self.__explain: operators["$explain"] = True if self.__hint: operators["$hint"] = self.__hint if self.__snapshot: operators["$snapshot"] = True if self.__max_scan: operators["$maxScan"] = self.__max_scan # Only set $readPreference if it's something other than # PRIMARY to avoid problems with mongos versions that # don't support read preferences. if (self.__collection.database.connection.is_mongos and self.__read_preference != ReadPreference.PRIMARY): has_tags = self.__tag_sets and self.__tag_sets != [{}] # For maximum backwards compatibility, don't set $readPreference # for SECONDARY_PREFERRED unless tags are in use. Just rely on # the slaveOkay bit (set automatically if read preference is not # PRIMARY), which has the same behavior. if (self.__read_preference != ReadPreference.SECONDARY_PREFERRED or has_tags): read_pref = { 'mode': read_preferences.mongos_mode(self.__read_preference) } if has_tags: read_pref['tags'] = self.__tag_sets operators['$readPreference'] = read_pref if operators: # Make a shallow copy so we can cleanly rewind or clone. spec = self.__spec.copy() # Only commands that can be run on secondaries should have any # operators added to the spec. Command queries can be issued # by db.command or calling find_one on $cmd directly if self.collection.name == "$cmd": # Don't change commands that can't be sent to secondaries command_name = spec and spec.keys()[0].lower() or "" if command_name not in secondary_ok_commands: return spec elif command_name == 'mapreduce': # mapreduce shouldn't be changed if its not inline out = spec.get('out') if not isinstance(out, dict) or not out.get('inline'): return spec # White-listed commands must be wrapped in $query. if "$query" not in spec: # $query has to come first spec = SON([("$query", spec)]) if not isinstance(spec, SON): # Ensure the spec is SON. As order is important this will # ensure its set before merging in any extra operators. spec = SON(spec) spec.update(operators) return spec # Have to wrap with $query if "query" is the first key. # We can't just use $query anytime "query" is a key as # that breaks commands like count and find_and_modify. # Checking spec.keys()[0] covers the case that the spec # was passed as an instance of SON or OrderedDict. elif ("query" in self.__spec and (len(self.__spec) == 1 or self.__spec.keys()[0] == "query")): return SON({"$query": self.__spec}) return self.__spec def __query_options(self): """Get the query options string to use for this query. """ options = self.__query_flags if (self.__slave_okay or self.__read_preference != ReadPreference.PRIMARY ): options |= _QUERY_OPTIONS["slave_okay"] return options def __check_okay_to_chain(self): """Check if it is okay to chain more options onto this cursor. """ if self.__retrieved or self.__id is not None: raise InvalidOperation("cannot set options after executing query") def __check_not_command_cursor(self, method_name): """Check if calling a method on this cursor is valid. """ if self.__is_command_cursor: raise InvalidOperation( "cannot call %s on a command cursor" % method_name) def add_option(self, mask): """Set arbitary query flags using a bitmask. To set the tailable flag: cursor.add_option(2) """ if not isinstance(mask, int): raise TypeError("mask must be an int") self.__check_okay_to_chain() if mask & _QUERY_OPTIONS["slave_okay"]: self.__slave_okay = True if mask & _QUERY_OPTIONS["exhaust"]: if self.__limit: raise InvalidOperation("Can't use limit and exhaust together.") if self.__collection.database.connection.is_mongos: raise InvalidOperation('Exhaust cursors are ' 'not supported by mongos') self.__exhaust = True self.__query_flags |= mask return self def remove_option(self, mask): """Unset arbitrary query flags using a bitmask. To unset the tailable flag: cursor.remove_option(2) """ if not isinstance(mask, int): raise TypeError("mask must be an int") self.__check_okay_to_chain() if mask & _QUERY_OPTIONS["slave_okay"]: self.__slave_okay = False if mask & _QUERY_OPTIONS["exhaust"]: self.__exhaust = False self.__query_flags &= ~mask return self def limit(self, limit): """Limits the number of results to be returned by this cursor. Raises TypeError if limit is not an instance of int. Raises InvalidOperation if this cursor has already been used. The last `limit` applied to this cursor takes precedence. A limit of ``0`` is equivalent to no limit. :Parameters: - `limit`: the number of results to return .. mongodoc:: limit """ if not isinstance(limit, int): raise TypeError("limit must be an int") if self.__exhaust: raise InvalidOperation("Can't use limit and exhaust together.") self.__check_okay_to_chain() self.__empty = False self.__limit = limit return self def batch_size(self, batch_size): """Limits the number of documents returned in one batch. Each batch requires a round trip to the server. It can be adjusted to optimize performance and limit data transfer. .. note:: batch_size can not override MongoDB's internal limits on the amount of data it will return to the client in a single batch (i.e if you set batch size to 1,000,000,000, MongoDB will currently only return 4-16MB of results per batch). Raises :class:`TypeError` if `batch_size` is not an instance of :class:`int`. Raises :class:`ValueError` if `batch_size` is less than ``0``. Raises :class:`~pymongo.errors.InvalidOperation` if this :class:`Cursor` has already been used. The last `batch_size` applied to this cursor takes precedence. :Parameters: - `batch_size`: The size of each batch of results requested. .. versionadded:: 1.9 """ if not isinstance(batch_size, int): raise TypeError("batch_size must be an int") if batch_size < 0: raise ValueError("batch_size must be >= 0") self.__check_okay_to_chain() self.__batch_size = batch_size == 1 and 2 or batch_size return self def skip(self, skip): """Skips the first `skip` results of this cursor. Raises TypeError if skip is not an instance of int. Raises InvalidOperation if this cursor has already been used. The last `skip` applied to this cursor takes precedence. :Parameters: - `skip`: the number of results to skip """ if not isinstance(skip, (int, long)): raise TypeError("skip must be an int") self.__check_okay_to_chain() self.__skip = skip return self def __getitem__(self, index): """Get a single document or a slice of documents from this cursor. Raises :class:`~pymongo.errors.InvalidOperation` if this cursor has already been used. To get a single document use an integral index, e.g.:: >>> db.test.find()[50] An :class:`IndexError` will be raised if the index is negative or greater than the amount of documents in this cursor. Any limit previously applied to this cursor will be ignored. To get a slice of documents use a slice index, e.g.:: >>> db.test.find()[20:25] This will return this cursor with a limit of ``5`` and skip of ``20`` applied. Using a slice index will override any prior limits or skips applied to this cursor (including those applied through previous calls to this method). Raises :class:`IndexError` when the slice has a step, a negative start value, or a stop value less than or equal to the start value. :Parameters: - `index`: An integer or slice index to be applied to this cursor """ self.__check_okay_to_chain() self.__empty = False if isinstance(index, slice): if index.step is not None: raise IndexError("Cursor instances do not support slice steps") skip = 0 if index.start is not None: if index.start < 0: raise IndexError("Cursor instances do not support" "negative indices") skip = index.start if index.stop is not None: limit = index.stop - skip if limit < 0: raise IndexError("stop index must be greater than start" "index for slice %r" % index) if limit == 0: self.__empty = True else: limit = 0 self.__skip = skip self.__limit = limit return self if isinstance(index, (int, long)): if index < 0: raise IndexError("Cursor instances do not support negative" "indices") clone = self.clone() clone.skip(index + self.__skip) clone.limit(-1) # use a hard limit for doc in clone: return doc raise IndexError("no such item for Cursor instance") raise TypeError("index %r cannot be applied to Cursor " "instances" % index) def max_scan(self, max_scan): """Limit the number of documents to scan when performing the query. Raises :class:`~pymongo.errors.InvalidOperation` if this cursor has already been used. Only the last :meth:`max_scan` applied to this cursor has any effect. :Parameters: - `max_scan`: the maximum number of documents to scan .. note:: Requires server version **>= 1.5.1** .. versionadded:: 1.7 """ self.__check_okay_to_chain() self.__max_scan = max_scan return self def sort(self, key_or_list, direction=None): """Sorts this cursor's results. Takes either a single key and a direction, or a list of (key, direction) pairs. The key(s) must be an instance of ``(str, unicode)``, and the direction(s) must be one of (:data:`~pymongo.ASCENDING`, :data:`~pymongo.DESCENDING`). Raises :class:`~pymongo.errors.InvalidOperation` if this cursor has already been used. Only the last :meth:`sort` applied to this cursor has any effect. :Parameters: - `key_or_list`: a single key or a list of (key, direction) pairs specifying the keys to sort on - `direction` (optional): only used if `key_or_list` is a single key, if not given :data:`~pymongo.ASCENDING` is assumed """ self.__check_okay_to_chain() keys = helpers._index_list(key_or_list, direction) self.__ordering = helpers._index_document(keys) return self def count(self, with_limit_and_skip=False): """Get the size of the results set for this query. Returns the number of documents in the results set for this query. Does not take :meth:`limit` and :meth:`skip` into account by default - set `with_limit_and_skip` to ``True`` if that is the desired behavior. Raises :class:`~pymongo.errors.OperationFailure` on a database error. With :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` or :class:`~pymongo.master_slave_connection.MasterSlaveConnection`, if `read_preference` is not :attr:`pymongo.read_preferences.ReadPreference.PRIMARY` or :attr:`pymongo.read_preferences.ReadPreference.PRIMARY_PREFERRED`, or (deprecated) `slave_okay` is `True`, the count command will be sent to a secondary or slave. :Parameters: - `with_limit_and_skip` (optional): take any :meth:`limit` or :meth:`skip` that has been applied to this cursor into account when getting the count .. note:: The `with_limit_and_skip` parameter requires server version **>= 1.1.4-** .. note:: ``count`` ignores ``network_timeout``. For example, the timeout is ignored in the following code:: collection.find({}, network_timeout=1).count() .. versionadded:: 1.1.1 The `with_limit_and_skip` parameter. :meth:`~pymongo.cursor.Cursor.__len__` was deprecated in favor of calling :meth:`count` with `with_limit_and_skip` set to ``True``. """ self.__check_not_command_cursor('count') command = {"query": self.__spec, "fields": self.__fields} command['read_preference'] = self.__read_preference command['tag_sets'] = self.__tag_sets command['secondary_acceptable_latency_ms'] = ( self.__secondary_acceptable_latency_ms) command['slave_okay'] = self.__slave_okay use_master = not self.__slave_okay and not self.__read_preference command['_use_master'] = use_master if with_limit_and_skip: if self.__limit: command["limit"] = self.__limit if self.__skip: command["skip"] = self.__skip database = self.__collection.database r = database.command("count", self.__collection.name, allowable_errors=["ns missing"], uuid_subtype=self.__uuid_subtype, **command) if r.get("errmsg", "") == "ns missing": return 0 return int(r["n"]) def distinct(self, key): """Get a list of distinct values for `key` among all documents in the result set of this query. Raises :class:`TypeError` if `key` is not an instance of :class:`basestring` (:class:`str` in python 3). With :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` or :class:`~pymongo.master_slave_connection.MasterSlaveConnection`, if `read_preference` is not :attr:`pymongo.read_preferences.ReadPreference.PRIMARY` or (deprecated) `slave_okay` is `True` the distinct command will be sent to a secondary or slave. :Parameters: - `key`: name of key for which we want to get the distinct values .. note:: Requires server version **>= 1.1.3+** .. seealso:: :meth:`pymongo.collection.Collection.distinct` .. versionadded:: 1.2 """ self.__check_not_command_cursor('distinct') if not isinstance(key, basestring): raise TypeError("key must be an instance " "of %s" % (basestring.__name__,)) options = {"key": key} if self.__spec: options["query"] = self.__spec options['read_preference'] = self.__read_preference options['tag_sets'] = self.__tag_sets options['secondary_acceptable_latency_ms'] = ( self.__secondary_acceptable_latency_ms) options['slave_okay'] = self.__slave_okay use_master = not self.__slave_okay and not self.__read_preference options['_use_master'] = use_master database = self.__collection.database return database.command("distinct", self.__collection.name, uuid_subtype=self.__uuid_subtype, **options)["values"] def explain(self): """Returns an explain plan record for this cursor. .. mongodoc:: explain """ self.__check_not_command_cursor('explain') c = self.clone() c.__explain = True # always use a hard limit for explains if c.__limit: c.__limit = -abs(c.__limit) return c.next() def hint(self, index): """Adds a 'hint', telling Mongo the proper index to use for the query. Judicious use of hints can greatly improve query performance. When doing a query on multiple fields (at least one of which is indexed) pass the indexed field as a hint to the query. Hinting will not do anything if the corresponding index does not exist. Raises :class:`~pymongo.errors.InvalidOperation` if this cursor has already been used. `index` should be an index as passed to :meth:`~pymongo.collection.Collection.create_index` (e.g. ``[('field', ASCENDING)]``). If `index` is ``None`` any existing hints for this query are cleared. The last hint applied to this cursor takes precedence over all others. :Parameters: - `index`: index to hint on (as an index specifier) """ self.__check_okay_to_chain() if index is None: self.__hint = None return self self.__hint = helpers._index_document(index) return self def where(self, code): """Adds a $where clause to this query. The `code` argument must be an instance of :class:`basestring` (:class:`str` in python 3) or :class:`~bson.code.Code` containing a JavaScript expression. This expression will be evaluated for each document scanned. Only those documents for which the expression evaluates to *true* will be returned as results. The keyword *this* refers to the object currently being scanned. Raises :class:`TypeError` if `code` is not an instance of :class:`basestring` (:class:`str` in python 3). Raises :class:`~pymongo.errors.InvalidOperation` if this :class:`Cursor` has already been used. Only the last call to :meth:`where` applied to a :class:`Cursor` has any effect. :Parameters: - `code`: JavaScript expression to use as a filter """ self.__check_okay_to_chain() if not isinstance(code, Code): code = Code(code) self.__spec["$where"] = code return self def __send_message(self, message): """Send a query or getmore message and handles the response. If message is ``None`` this is an exhaust cursor, which reads the next result batch off the exhaust socket instead of sending getMore messages to the server. """ client = self.__collection.database.connection if message: kwargs = {"_must_use_master": self.__must_use_master} kwargs["read_preference"] = self.__read_preference kwargs["tag_sets"] = self.__tag_sets kwargs["secondary_acceptable_latency_ms"] = ( self.__secondary_acceptable_latency_ms) kwargs['exhaust'] = self.__exhaust if self.__connection_id is not None: kwargs["_connection_to_use"] = self.__connection_id kwargs.update(self.__kwargs) try: res = client._send_message_with_response(message, **kwargs) self.__connection_id, (response, sock, pool) = res if self.__exhaust: self.__exhaust_mgr = _SocketManager(sock, pool) except AutoReconnect: # Don't try to send kill cursors on another socket # or to another server. It can cause a _pinValue # assertion on some server releases if we get here # due to a socket timeout. self.__killed = True raise else: # exhaust cursor - no getMore message response = client._exhaust_next(self.__exhaust_mgr.sock) try: response = helpers._unpack_response(response, self.__id, self.__as_class, self.__tz_aware, self.__uuid_subtype) except AutoReconnect: # Don't send kill cursors to another server after a "not master" # error. It's completely pointless. self.__killed = True client.disconnect() raise self.__id = response["cursor_id"] # starting from doesn't get set on getmore's for tailable cursors if not (self.__query_flags & _QUERY_OPTIONS["tailable_cursor"]): assert response["starting_from"] == self.__retrieved, ( "Result batch started from %s, expected %s" % ( response['starting_from'], self.__retrieved)) self.__retrieved += response["number_returned"] self.__data = deque(response["data"]) if self.__limit and self.__id and self.__limit <= self.__retrieved: self.__die() # Don't wait for garbage collection to call __del__, return the # socket to the pool now. if self.__exhaust and self.__id == 0: self.__exhaust_mgr.close() def _refresh(self): """Refreshes the cursor with more data from Mongo. Returns the length of self.__data after refresh. Will exit early if self.__data is already non-empty. Raises OperationFailure when the cursor cannot be refreshed due to an error on the query. """ if len(self.__data) or self.__killed: return len(self.__data) if self.__id is None: # Query ntoreturn = self.__batch_size if self.__limit: if self.__batch_size: ntoreturn = min(self.__limit, self.__batch_size) else: ntoreturn = self.__limit self.__send_message( message.query(self.__query_options(), self.__collection.full_name, self.__skip, ntoreturn, self.__query_spec(), self.__fields, self.__uuid_subtype)) if not self.__id: self.__killed = True elif self.__id: # Get More if self.__limit: limit = self.__limit - self.__retrieved if self.__batch_size: limit = min(limit, self.__batch_size) else: limit = self.__batch_size # Exhaust cursors don't send getMore messages. if self.__exhaust: self.__send_message(None) else: self.__send_message( message.get_more(self.__collection.full_name, limit, self.__id)) else: # Cursor id is zero nothing else to return self.__killed = True return len(self.__data) @property def alive(self): """Does this cursor have the potential to return more data? This is mostly useful with `tailable cursors `_ since they will stop iterating even though they *may* return more results in the future. .. versionadded:: 1.5 """ return bool(len(self.__data) or (not self.__killed)) @property def cursor_id(self): """Returns the id of the cursor Useful if you need to manage cursor ids and want to handle killing cursors manually using :meth:`~pymongo.mongo_client.MongoClient.kill_cursors` .. versionadded:: 2.2 """ return self.__id def __iter__(self): return self def next(self): if self.__empty: raise StopIteration db = self.__collection.database if len(self.__data) or self._refresh(): if self.__manipulate: return db._fix_outgoing(self.__data.popleft(), self.__collection) else: return self.__data.popleft() else: raise StopIteration def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): self.__die() def __copy__(self): """Support function for `copy.copy()`. .. versionadded:: 2.4 """ return self.__clone(deepcopy=False) def __deepcopy__(self, memo): """Support function for `copy.deepcopy()`. .. versionadded:: 2.4 """ return self.__clone(deepcopy=True) def __deepcopy(self, x, memo=None): """Deepcopy helper for the data dictionary or list. Regular expressions cannot be deep copied but as they are immutable we don't have to copy them when cloning. """ if not hasattr(x, 'items'): y, is_list, iterator = [], True, enumerate(x) else: y, is_list, iterator = {}, False, x.iteritems() if memo is None: memo = {} val_id = id(x) if val_id in memo: return memo.get(val_id) memo[val_id] = y for key, value in iterator: if isinstance(value, (dict, list)) and not isinstance(value, SON): value = self.__deepcopy(value, memo) elif not isinstance(value, RE_TYPE): value = copy.deepcopy(value, memo) if is_list: y.append(value) else: if not isinstance(key, RE_TYPE): key = copy.deepcopy(key, memo) y[key] = value return y pymongo-2.6.3/pymongo/cursor_manager.py000066400000000000000000000054341223300253600202650ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """DEPRECATED - Different managers to handle when cursors are killed after they are closed. New cursor managers should be defined as subclasses of CursorManager and can be installed on a connection by calling `pymongo.connection.Connection.set_cursor_manager`. .. versionchanged:: 2.1+ Deprecated. """ import weakref class CursorManager(object): """The default cursor manager. This manager will kill cursors one at a time as they are closed. """ def __init__(self, connection): """Instantiate the manager. :Parameters: - `connection`: a Mongo Connection """ self.__connection = weakref.ref(connection) def close(self, cursor_id): """Close a cursor by killing it immediately. Raises TypeError if cursor_id is not an instance of (int, long). :Parameters: - `cursor_id`: cursor id to close """ if not isinstance(cursor_id, (int, long)): raise TypeError("cursor_id must be an instance of (int, long)") self.__connection().kill_cursors([cursor_id]) class BatchCursorManager(CursorManager): """A cursor manager that kills cursors in batches. """ def __init__(self, connection): """Instantiate the manager. :Parameters: - `connection`: a Mongo Connection """ self.__dying_cursors = [] self.__max_dying_cursors = 20 self.__connection = weakref.ref(connection) CursorManager.__init__(self, connection) def __del__(self): """Cleanup - be sure to kill any outstanding cursors. """ self.__connection().kill_cursors(self.__dying_cursors) def close(self, cursor_id): """Close a cursor by killing it in a batch. Raises TypeError if cursor_id is not an instance of (int, long). :Parameters: - `cursor_id`: cursor id to close """ if not isinstance(cursor_id, (int, long)): raise TypeError("cursor_id must be an instance of (int, long)") self.__dying_cursors.append(cursor_id) if len(self.__dying_cursors) > self.__max_dying_cursors: self.__connection().kill_cursors(self.__dying_cursors) self.__dying_cursors = [] pymongo-2.6.3/pymongo/database.py000066400000000000000000001026631223300253600170240ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Database level operations.""" from bson.binary import OLD_UUID_SUBTYPE from bson.code import Code from bson.dbref import DBRef from bson.son import SON from pymongo import auth, common, helpers from pymongo.collection import Collection from pymongo.errors import (CollectionInvalid, InvalidName, OperationFailure) from pymongo.son_manipulator import ObjectIdInjector from pymongo import read_preferences as rp def _check_name(name): """Check if a database name is valid. """ if not name: raise InvalidName("database name cannot be the empty string") for invalid_char in [" ", ".", "$", "/", "\\", "\x00"]: if invalid_char in name: raise InvalidName("database names cannot contain the " "character %r" % invalid_char) class Database(common.BaseObject): """A Mongo database. """ def __init__(self, connection, name): """Get a database by connection and name. Raises :class:`TypeError` if `name` is not an instance of :class:`basestring` (:class:`str` in python 3). Raises :class:`~pymongo.errors.InvalidName` if `name` is not a valid database name. :Parameters: - `connection`: a client instance - `name`: database name .. mongodoc:: databases """ super(Database, self).__init__(slave_okay=connection.slave_okay, read_preference=connection.read_preference, tag_sets=connection.tag_sets, secondary_acceptable_latency_ms=( connection.secondary_acceptable_latency_ms), safe=connection.safe, **connection.write_concern) if not isinstance(name, basestring): raise TypeError("name must be an instance " "of %s" % (basestring.__name__,)) if name != '$external': _check_name(name) self.__name = unicode(name) self.__connection = connection self.__incoming_manipulators = [] self.__incoming_copying_manipulators = [] self.__outgoing_manipulators = [] self.__outgoing_copying_manipulators = [] self.add_son_manipulator(ObjectIdInjector()) def add_son_manipulator(self, manipulator): """Add a new son manipulator to this database. Newly added manipulators will be applied before existing ones. :Parameters: - `manipulator`: the manipulator to add """ def method_overwritten(instance, method): return getattr(instance, method) != \ getattr(super(instance.__class__, instance), method) if manipulator.will_copy(): if method_overwritten(manipulator, "transform_incoming"): self.__incoming_copying_manipulators.insert(0, manipulator) if method_overwritten(manipulator, "transform_outgoing"): self.__outgoing_copying_manipulators.insert(0, manipulator) else: if method_overwritten(manipulator, "transform_incoming"): self.__incoming_manipulators.insert(0, manipulator) if method_overwritten(manipulator, "transform_outgoing"): self.__outgoing_manipulators.insert(0, manipulator) @property def system_js(self): """A :class:`SystemJS` helper for this :class:`Database`. See the documentation for :class:`SystemJS` for more details. .. versionadded:: 1.5 """ return SystemJS(self) @property def connection(self): """The client instance for this :class:`Database`. .. versionchanged:: 1.3 ``connection`` is now a property rather than a method. """ return self.__connection @property def name(self): """The name of this :class:`Database`. .. versionchanged:: 1.3 ``name`` is now a property rather than a method. """ return self.__name @property def incoming_manipulators(self): """List all incoming SON manipulators installed on this instance. .. versionadded:: 2.0 """ return [manipulator.__class__.__name__ for manipulator in self.__incoming_manipulators] @property def incoming_copying_manipulators(self): """List all incoming SON copying manipulators installed on this instance. .. versionadded:: 2.0 """ return [manipulator.__class__.__name__ for manipulator in self.__incoming_copying_manipulators] @property def outgoing_manipulators(self): """List all outgoing SON manipulators installed on this instance. .. versionadded:: 2.0 """ return [manipulator.__class__.__name__ for manipulator in self.__outgoing_manipulators] @property def outgoing_copying_manipulators(self): """List all outgoing SON copying manipulators installed on this instance. .. versionadded:: 2.0 """ return [manipulator.__class__.__name__ for manipulator in self.__outgoing_copying_manipulators] def __eq__(self, other): if isinstance(other, Database): us = (self.__connection, self.__name) them = (other.__connection, other.__name) return us == them return NotImplemented def __ne__(self, other): return not self == other def __repr__(self): return "Database(%r, %r)" % (self.__connection, self.__name) def __getattr__(self, name): """Get a collection of this database by name. Raises InvalidName if an invalid collection name is used. :Parameters: - `name`: the name of the collection to get """ return Collection(self, name) def __getitem__(self, name): """Get a collection of this database by name. Raises InvalidName if an invalid collection name is used. :Parameters: - `name`: the name of the collection to get """ return self.__getattr__(name) def create_collection(self, name, **kwargs): """Create a new :class:`~pymongo.collection.Collection` in this database. Normally collection creation is automatic. This method should only be used to specify options on creation. :class:`~pymongo.errors.CollectionInvalid` will be raised if the collection already exists. Options should be passed as keyword arguments to this method. Any of the following options are valid: - "size": desired initial size for the collection (in bytes). For capped collections this size is the max size of the collection. - "capped": if True, this is a capped collection - "max": maximum number of objects if capped (optional) :Parameters: - `name`: the name of the collection to create - `**kwargs` (optional): additional keyword arguments will be passed as options for the create collection command .. versionchanged:: 2.2 Removed deprecated argument: options .. versionchanged:: 1.5 deprecating `options` in favor of kwargs """ opts = {"create": True} opts.update(kwargs) if name in self.collection_names(): raise CollectionInvalid("collection %s already exists" % name) return Collection(self, name, **opts) def _fix_incoming(self, son, collection): """Apply manipulators to an incoming SON object before it gets stored. :Parameters: - `son`: the son object going into the database - `collection`: the collection the son object is being saved in """ for manipulator in self.__incoming_manipulators: son = manipulator.transform_incoming(son, collection) for manipulator in self.__incoming_copying_manipulators: son = manipulator.transform_incoming(son, collection) return son def _fix_outgoing(self, son, collection): """Apply manipulators to a SON object as it comes out of the database. :Parameters: - `son`: the son object coming out of the database - `collection`: the collection the son object was saved in """ for manipulator in reversed(self.__outgoing_manipulators): son = manipulator.transform_outgoing(son, collection) for manipulator in reversed(self.__outgoing_copying_manipulators): son = manipulator.transform_outgoing(son, collection) return son def command(self, command, value=1, check=True, allowable_errors=[], uuid_subtype=OLD_UUID_SUBTYPE, **kwargs): """Issue a MongoDB command. Send command `command` to the database and return the response. If `command` is an instance of :class:`basestring` (:class:`str` in python 3) then the command {`command`: `value`} will be sent. Otherwise, `command` must be an instance of :class:`dict` and will be sent as is. Any additional keyword arguments will be added to the final command document before it is sent. For example, a command like ``{buildinfo: 1}`` can be sent using: >>> db.command("buildinfo") For a command where the value matters, like ``{collstats: collection_name}`` we can do: >>> db.command("collstats", collection_name) For commands that take additional arguments we can use kwargs. So ``{filemd5: object_id, root: file_root}`` becomes: >>> db.command("filemd5", object_id, root=file_root) :Parameters: - `command`: document representing the command to be issued, or the name of the command (for simple commands only). .. note:: the order of keys in the `command` document is significant (the "verb" must come first), so commands which require multiple keys (e.g. `findandmodify`) should use an instance of :class:`~bson.son.SON` or a string and kwargs instead of a Python `dict`. - `value` (optional): value to use for the command verb when `command` is passed as a string - `check` (optional): check the response for errors, raising :class:`~pymongo.errors.OperationFailure` if there are any - `allowable_errors`: if `check` is ``True``, error messages in this list will be ignored by error-checking - `uuid_subtype` (optional): The BSON binary subtype to use for a UUID used in this command. - `read_preference`: The read preference for this connection. See :class:`~pymongo.read_preferences.ReadPreference` for available options. - `tag_sets`: Read from replica-set members with these tags. To specify a priority-order for tag sets, provide a list of tag sets: ``[{'dc': 'ny'}, {'dc': 'la'}, {}]``. A final, empty tag set, ``{}``, means "read from any member that matches the mode, ignoring tags." ReplicaSetConnection tries each set of tags in turn until it finds a set of tags with at least one matching member. - `secondary_acceptable_latency_ms`: Any replica-set member whose ping time is within secondary_acceptable_latency_ms of the nearest member may accept reads. Default 15 milliseconds. **Ignored by mongos** and must be configured on the command line. See the localThreshold_ option for more information. - `**kwargs` (optional): additional keyword arguments will be added to the command document before it is sent .. note:: ``command`` ignores the ``network_timeout`` parameter. .. versionchanged:: 2.3 Added `tag_sets` and `secondary_acceptable_latency_ms` options. .. versionchanged:: 2.2 Added support for `as_class` - the class you want to use for the resulting documents .. versionchanged:: 1.6 Added the `value` argument for string commands, and keyword arguments for additional command options. .. versionchanged:: 1.5 `command` can be a string in addition to a full document. .. versionadded:: 1.4 .. mongodoc:: commands .. _localThreshold: http://docs.mongodb.org/manual/reference/mongos/#cmdoption-mongos--localThreshold """ if isinstance(command, basestring): command = SON([(command, value)]) command_name = command.keys()[0].lower() must_use_master = kwargs.pop('_use_master', False) if command_name not in rp.secondary_ok_commands: must_use_master = True # Special-case: mapreduce can go to secondaries only if inline if command_name == 'mapreduce': out = command.get('out') or kwargs.get('out') if not isinstance(out, dict) or not out.get('inline'): must_use_master = True extra_opts = { 'as_class': kwargs.pop('as_class', None), 'slave_okay': kwargs.pop('slave_okay', self.slave_okay), '_must_use_master': must_use_master, '_uuid_subtype': uuid_subtype } extra_opts['read_preference'] = kwargs.pop( 'read_preference', self.read_preference) extra_opts['tag_sets'] = kwargs.pop( 'tag_sets', self.tag_sets) extra_opts['secondary_acceptable_latency_ms'] = kwargs.pop( 'secondary_acceptable_latency_ms', self.secondary_acceptable_latency_ms) fields = kwargs.get('fields') if fields is not None and not isinstance(fields, dict): kwargs['fields'] = helpers._fields_list_to_dict(fields) command.update(kwargs) result = self["$cmd"].find_one(command, **extra_opts) if check: msg = "command %s failed: %%s" % repr(command).replace("%", "%%") helpers._check_command_response(result, self.connection.disconnect, msg, allowable_errors) return result def collection_names(self, include_system_collections=True): """Get a list of all the collection names in this database. :Parameters: - `include_system_collections` (optional): if ``False`` list will not include system collections (e.g ``system.indexes``) """ results = self["system.namespaces"].find(_must_use_master=True) names = [r["name"] for r in results] names = [n[len(self.__name) + 1:] for n in names if n.startswith(self.__name + ".") and "$" not in n] if not include_system_collections: names = [n for n in names if not n.startswith("system.")] return names def drop_collection(self, name_or_collection): """Drop a collection. :Parameters: - `name_or_collection`: the name of a collection to drop or the collection object itself """ name = name_or_collection if isinstance(name, Collection): name = name.name if not isinstance(name, basestring): raise TypeError("name_or_collection must be an instance of " "%s or Collection" % (basestring.__name__,)) self.__connection._purge_index(self.__name, name) self.command("drop", unicode(name), allowable_errors=["ns not found"]) def validate_collection(self, name_or_collection, scandata=False, full=False): """Validate a collection. Returns a dict of validation info. Raises CollectionInvalid if validation fails. With MongoDB < 1.9 the result dict will include a `result` key with a string value that represents the validation results. With MongoDB >= 1.9 the `result` key no longer exists and the results are split into individual fields in the result dict. :Parameters: - `name_or_collection`: A Collection object or the name of a collection to validate. - `scandata`: Do extra checks beyond checking the overall structure of the collection. - `full`: Have the server do a more thorough scan of the collection. Use with `scandata` for a thorough scan of the structure of the collection and the individual documents. Ignored in MongoDB versions before 1.9. .. versionchanged:: 1.11 validate_collection previously returned a string. .. versionadded:: 1.11 Added `scandata` and `full` options. """ name = name_or_collection if isinstance(name, Collection): name = name.name if not isinstance(name, basestring): raise TypeError("name_or_collection must be an instance of " "%s or Collection" % (basestring.__name__,)) result = self.command("validate", unicode(name), scandata=scandata, full=full) valid = True # Pre 1.9 results if "result" in result: info = result["result"] if info.find("exception") != -1 or info.find("corrupt") != -1: raise CollectionInvalid("%s invalid: %s" % (name, info)) # Sharded results elif "raw" in result: for _, res in result["raw"].iteritems(): if "result" in res: info = res["result"] if (info.find("exception") != -1 or info.find("corrupt") != -1): raise CollectionInvalid("%s invalid: " "%s" % (name, info)) elif not res.get("valid", False): valid = False break # Post 1.9 non-sharded results. elif not result.get("valid", False): valid = False if not valid: raise CollectionInvalid("%s invalid: %r" % (name, result)) return result def current_op(self, include_all=False): """Get information on operations currently running. :Parameters: - `include_all` (optional): if ``True`` also list currently idle operations in the result """ if include_all: return self['$cmd.sys.inprog'].find_one({"$all": True}) else: return self['$cmd.sys.inprog'].find_one() def profiling_level(self): """Get the database's current profiling level. Returns one of (:data:`~pymongo.OFF`, :data:`~pymongo.SLOW_ONLY`, :data:`~pymongo.ALL`). .. mongodoc:: profiling """ result = self.command("profile", -1) assert result["was"] >= 0 and result["was"] <= 2 return result["was"] def set_profiling_level(self, level, slow_ms=None): """Set the database's profiling level. :Parameters: - `level`: Specifies a profiling level, see list of possible values below. - `slow_ms`: Optionally modify the threshold for the profile to consider a query or operation. Even if the profiler is off queries slower than the `slow_ms` level will get written to the logs. Possible `level` values: +----------------------------+------------------------------------+ | Level | Setting | +============================+====================================+ | :data:`~pymongo.OFF` | Off. No profiling. | +----------------------------+------------------------------------+ | :data:`~pymongo.SLOW_ONLY` | On. Only includes slow operations. | +----------------------------+------------------------------------+ | :data:`~pymongo.ALL` | On. Includes all operations. | +----------------------------+------------------------------------+ Raises :class:`ValueError` if level is not one of (:data:`~pymongo.OFF`, :data:`~pymongo.SLOW_ONLY`, :data:`~pymongo.ALL`). .. mongodoc:: profiling """ if not isinstance(level, int) or level < 0 or level > 2: raise ValueError("level must be one of (OFF, SLOW_ONLY, ALL)") if slow_ms is not None and not isinstance(slow_ms, int): raise TypeError("slow_ms must be an integer") if slow_ms is not None: self.command("profile", level, slowms=slow_ms) else: self.command("profile", level) def profiling_info(self): """Returns a list containing current profiling information. .. mongodoc:: profiling """ return list(self["system.profile"].find()) def error(self): """Get a database error if one occured on the last operation. Return None if the last operation was error-free. Otherwise return the error that occurred. """ error = self.command("getlasterror") error_msg = error.get("err", "") if error_msg is None: return None if error_msg.startswith("not master"): self.__connection.disconnect() return error def last_status(self): """Get status information from the last operation. Returns a SON object with status information. """ return self.command("getlasterror") def previous_error(self): """Get the most recent error to have occurred on this database. Only returns errors that have occurred since the last call to `Database.reset_error_history`. Returns None if no such errors have occurred. """ error = self.command("getpreverror") if error.get("err", 0) is None: return None return error def reset_error_history(self): """Reset the error history of this database. Calls to `Database.previous_error` will only return errors that have occurred since the most recent call to this method. """ self.command("reseterror") def __iter__(self): return self def next(self): raise TypeError("'Database' object is not iterable") def add_user(self, name, password=None, read_only=None, **kwargs): """Create user `name` with password `password`. Add a new user with permissions for this :class:`Database`. .. note:: Will change the password if user `name` already exists. :Parameters: - `name`: the name of the user to create - `password` (optional): the password of the user to create. Can not be used with the ``userSource`` argument. - `read_only` (optional): if ``True`` the user will be read only - `**kwargs` (optional): optional fields for the user document (e.g. ``userSource``, ``otherDBRoles``, or ``roles``). See ``_ for more information. .. note:: The use of optional keyword arguments like ``userSource``, ``otherDBRoles``, or ``roles`` requires MongoDB >= 2.4.0 .. versionchanged:: 2.5 Added kwargs support for optional fields introduced in MongoDB 2.4 .. versionchanged:: 2.2 Added support for read only users .. versionadded:: 1.4 """ user = self.system.users.find_one({"user": name}) or {"user": name} if password is not None: user["pwd"] = auth._password_digest(name, password) if read_only is not None: user["readOnly"] = common.validate_boolean('read_only', read_only) user.update(kwargs) try: self.system.users.save(user, **self._get_wc_override()) except OperationFailure, e: # First admin user add fails gle in MongoDB >= 2.1.2 # See SERVER-4225 for more information. if 'login' in str(e): pass else: raise def remove_user(self, name): """Remove user `name` from this :class:`Database`. User `name` will no longer have permissions to access this :class:`Database`. :Parameters: - `name`: the name of the user to remove .. versionadded:: 1.4 """ self.system.users.remove({"user": name}, **self._get_wc_override()) def authenticate(self, name, password=None, source=None, mechanism='MONGODB-CR', **kwargs): """Authenticate to use this database. Authentication lasts for the life of the underlying client instance, or until :meth:`logout` is called. Raises :class:`TypeError` if (required) `name`, (optional) `password`, or (optional) `source` is not an instance of :class:`basestring` (:class:`str` in python 3). .. note:: - This method authenticates the current connection, and will also cause all new :class:`~socket.socket` connections in the underlying client instance to be authenticated automatically. - Authenticating more than once on the same database with different credentials is not supported. You must call :meth:`logout` before authenticating with new credentials. - When sharing a client instance between multiple threads, all threads will share the authentication. If you need different authentication profiles for different purposes you must use distinct client instances. - To get authentication to apply immediately to all existing sockets you may need to reset this client instance's sockets using :meth:`~pymongo.mongo_client.MongoClient.disconnect`. :Parameters: - `name`: the name of the user to authenticate. - `password` (optional): the password of the user to authenticate. Not used with GSSAPI or MONGODB-X509 authentication. - `source` (optional): the database to authenticate on. If not specified the current database is used. - `mechanism` (optional): See :data:`~pymongo.auth.MECHANISMS` for options. Defaults to MONGODB-CR (MongoDB Challenge Response protocol) - `gssapiServiceName` (optional): Used with the GSSAPI mechanism to specify the service name portion of the service principal name. Defaults to 'mongodb'. .. versionchanged:: 2.5 Added the `source` and `mechanism` parameters. :meth:`authenticate` now raises a subclass of :class:`~pymongo.errors.PyMongoError` if authentication fails due to invalid credentials or configuration issues. .. mongodoc:: authenticate """ if not isinstance(name, basestring): raise TypeError("name must be an instance " "of %s" % (basestring.__name__,)) if password is not None and not isinstance(password, basestring): raise TypeError("password must be an instance " "of %s" % (basestring.__name__,)) if source is not None and not isinstance(source, basestring): raise TypeError("source must be an instance " "of %s" % (basestring.__name__,)) common.validate_auth_mechanism('mechanism', mechanism) validated_options = {} for option, value in kwargs.iteritems(): normalized, val = common.validate_auth_option(option, value) validated_options[normalized] = val credentials = auth._build_credentials_tuple(mechanism, source or self.name, unicode(name), password and unicode(password) or None, validated_options) self.connection._cache_credentials(self.name, credentials) return True def logout(self): """Deauthorize use of this database for this client instance. .. note:: Other databases may still be authenticated, and other existing :class:`~socket.socket` connections may remain authenticated for this database unless you reset all sockets with :meth:`~pymongo.mongo_client.MongoClient.disconnect`. """ # Sockets will be deauthenticated as they are used. self.connection._purge_credentials(self.name) def dereference(self, dbref): """Dereference a :class:`~bson.dbref.DBRef`, getting the document it points to. Raises :class:`TypeError` if `dbref` is not an instance of :class:`~bson.dbref.DBRef`. Returns a document, or ``None`` if the reference does not point to a valid document. Raises :class:`ValueError` if `dbref` has a database specified that is different from the current database. :Parameters: - `dbref`: the reference """ if not isinstance(dbref, DBRef): raise TypeError("cannot dereference a %s" % type(dbref)) if dbref.database is not None and dbref.database != self.__name: raise ValueError("trying to dereference a DBRef that points to " "another database (%r not %r)" % (dbref.database, self.__name)) return self[dbref.collection].find_one({"_id": dbref.id}) def eval(self, code, *args): """Evaluate a JavaScript expression in MongoDB. Useful if you need to touch a lot of data lightly; in such a scenario the network transfer of the data could be a bottleneck. The `code` argument must be a JavaScript function. Additional positional arguments will be passed to that function when it is run on the server. Raises :class:`TypeError` if `code` is not an instance of :class:`basestring` (:class:`str` in python 3) or `Code`. Raises :class:`~pymongo.errors.OperationFailure` if the eval fails. Returns the result of the evaluation. :Parameters: - `code`: string representation of JavaScript code to be evaluated - `args` (optional): additional positional arguments are passed to the `code` being evaluated """ if not isinstance(code, Code): code = Code(code) result = self.command("$eval", code, args=args) return result.get("retval", None) def __call__(self, *args, **kwargs): """This is only here so that some API misusages are easier to debug. """ raise TypeError("'Database' object is not callable. If you meant to " "call the '%s' method on a '%s' object it is " "failing because no such method exists." % ( self.__name, self.__connection.__class__.__name__)) class SystemJS(object): """Helper class for dealing with stored JavaScript. """ def __init__(self, database): """Get a system js helper for the database `database`. An instance of :class:`SystemJS` can be created with an instance of :class:`Database` through :attr:`Database.system_js`, manual instantiation of this class should not be necessary. :class:`SystemJS` instances allow for easy manipulation and access to server-side JavaScript: .. doctest:: >>> db.system_js.add1 = "function (x) { return x + 1; }" >>> db.system.js.find({"_id": "add1"}).count() 1 >>> db.system_js.add1(5) 6.0 >>> del db.system_js.add1 >>> db.system.js.find({"_id": "add1"}).count() 0 .. note:: Requires server version **>= 1.1.1** .. versionadded:: 1.5 """ # can't just assign it since we've overridden __setattr__ object.__setattr__(self, "_db", database) def __setattr__(self, name, code): self._db.system.js.save({"_id": name, "value": Code(code)}, **self._db._get_wc_override()) def __setitem__(self, name, code): self.__setattr__(name, code) def __delattr__(self, name): self._db.system.js.remove({"_id": name}, **self._db._get_wc_override()) def __delitem__(self, name): self.__delattr__(name) def __getattr__(self, name): return lambda *args: self._db.eval(Code("function() { " "return this[name].apply(" "this, arguments); }", scope={'name': name}), *args) def __getitem__(self, name): return self.__getattr__(name) def list(self): """Get a list of the names of the functions stored in this database. .. versionadded:: 1.9 """ return [x["_id"] for x in self._db.system.js.find(fields=["_id"])] pymongo-2.6.3/pymongo/errors.py000066400000000000000000000060451223300253600165710ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Exceptions raised by PyMongo.""" from bson.errors import * try: from ssl import CertificateError except ImportError: from pymongo.ssl_match_hostname import CertificateError class PyMongoError(Exception): """Base class for all PyMongo exceptions. .. versionadded:: 1.4 """ class ConnectionFailure(PyMongoError): """Raised when a connection to the database cannot be made or is lost. """ class AutoReconnect(ConnectionFailure): """Raised when a connection to the database is lost and an attempt to auto-reconnect will be made. In order to auto-reconnect you must handle this exception, recognizing that the operation which caused it has not necessarily succeeded. Future operations will attempt to open a new connection to the database (and will continue to raise this exception until the first successful connection is made). """ def __init__(self, message='', errors=None): self.errors = errors or [] ConnectionFailure.__init__(self, message) class ConfigurationError(PyMongoError): """Raised when something is incorrectly configured. """ class OperationFailure(PyMongoError): """Raised when a database operation fails. .. versionadded:: 1.8 The :attr:`code` attribute. """ def __init__(self, error, code=None): self.code = code PyMongoError.__init__(self, error) class TimeoutError(OperationFailure): """Raised when a database operation times out. .. versionadded:: 1.8 """ class DuplicateKeyError(OperationFailure): """Raised when a safe insert or update fails due to a duplicate key error. .. note:: Requires server version **>= 1.3.0** .. versionadded:: 1.4 """ class InvalidOperation(PyMongoError): """Raised when a client attempts to perform an invalid operation. """ class InvalidName(PyMongoError): """Raised when an invalid name is used. """ class CollectionInvalid(PyMongoError): """Raised when collection validation fails. """ class InvalidURI(ConfigurationError): """Raised when trying to parse an invalid mongodb URI. .. versionadded:: 1.5 """ class UnsupportedOption(ConfigurationError): """Exception for unsupported options. .. versionadded:: 2.0 """ class ExceededMaxWaiters(Exception): """Raised when a thread tries to get a connection from a pool and ``max_pool_size * waitQueueMultiple`` threads are already waiting. .. versionadded:: 2.6 """ pass pymongo-2.6.3/pymongo/helpers.py000066400000000000000000000147621223300253600167240ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Bits and pieces used by the driver that don't really fit elsewhere.""" import random import struct import bson import pymongo from bson.binary import OLD_UUID_SUBTYPE from bson.son import SON from pymongo.errors import (AutoReconnect, DuplicateKeyError, OperationFailure, TimeoutError) def _index_list(key_or_list, direction=None): """Helper to generate a list of (key, direction) pairs. Takes such a list, or a single key, or a single key and direction. """ if direction is not None: return [(key_or_list, direction)] else: if isinstance(key_or_list, basestring): return [(key_or_list, pymongo.ASCENDING)] elif not isinstance(key_or_list, list): raise TypeError("if no direction is specified, " "key_or_list must be an instance of list") return key_or_list def _index_document(index_list): """Helper to generate an index specifying document. Takes a list of (key, direction) pairs. """ if isinstance(index_list, dict): raise TypeError("passing a dict to sort/create_index/hint is not " "allowed - use a list of tuples instead. did you " "mean %r?" % list(index_list.iteritems())) elif not isinstance(index_list, list): raise TypeError("must use a list of (key, direction) pairs, " "not: " + repr(index_list)) if not len(index_list): raise ValueError("key_or_list must not be the empty list") index = SON() for (key, value) in index_list: if not isinstance(key, basestring): raise TypeError("first item in each key pair must be a string") if not isinstance(value, (basestring, int)): raise TypeError("second item in each key pair must be 1, -1, " "'2d', 'geoHaystack', or another valid MongoDB " "index specifier.") index[key] = value return index def _unpack_response(response, cursor_id=None, as_class=dict, tz_aware=False, uuid_subtype=OLD_UUID_SUBTYPE): """Unpack a response from the database. Check the response for errors and unpack, returning a dictionary containing the response data. :Parameters: - `response`: byte string as returned from the database - `cursor_id` (optional): cursor_id we sent to get this response - used for raising an informative exception when we get cursor id not valid at server response - `as_class` (optional): class to use for resulting documents """ response_flag = struct.unpack("= 1") for slave in slaves: if not isinstance(slave, MongoClient): raise TypeError("slave %r is not an instance of MongoClient" % slave) super(MasterSlaveConnection, self).__init__(read_preference=ReadPreference.SECONDARY, safe=master.safe, **master.write_concern) self.__master = master self.__slaves = slaves self.__document_class = document_class self.__tz_aware = tz_aware self.__request_counter = thread_util.Counter(master.use_greenlets) @property def master(self): return self.__master @property def slaves(self): return self.__slaves @property def is_mongos(self): """If this MasterSlaveConnection is connected to mongos (always False) .. versionadded:: 2.3 """ return False @property def use_greenlets(self): """Whether calling :meth:`start_request` assigns greenlet-local, rather than thread-local, sockets. .. versionadded:: 2.4.2 """ return self.master.use_greenlets def get_document_class(self): return self.__document_class def set_document_class(self, klass): self.__document_class = klass document_class = property(get_document_class, set_document_class, doc="""Default class to use for documents returned on this connection.""") @property def tz_aware(self): return self.__tz_aware @property def max_bson_size(self): """Return the maximum size BSON object the connected master accepts in bytes. Defaults to 4MB in server < 1.7.4. .. versionadded:: 2.6 """ return self.master.max_bson_size @property def max_message_size(self): """Return the maximum message size the connected master accepts in bytes. .. versionadded:: 2.6 """ return self.master.max_message_size def disconnect(self): """Disconnect from MongoDB. Disconnecting will call disconnect on all master and slave connections. .. seealso:: Module :mod:`~pymongo.mongo_client` .. versionadded:: 1.10.1 """ self.__master.disconnect() for slave in self.__slaves: slave.disconnect() def set_cursor_manager(self, manager_class): """Set the cursor manager for this connection. Helper to set cursor manager for each individual `MongoClient` instance that make up this `MasterSlaveConnection`. """ self.__master.set_cursor_manager(manager_class) for slave in self.__slaves: slave.set_cursor_manager(manager_class) def _ensure_connected(self, sync): """Ensure the master is connected to a mongod/s. """ self.__master._ensure_connected(sync) # _connection_to_use is a hack that we need to include to make sure # that killcursor operations can be sent to the same instance on which # the cursor actually resides... def _send_message(self, message, with_last_error=False, _connection_to_use=None): """Say something to Mongo. Sends a message on the Master connection. This is used for inserts, updates, and deletes. Raises ConnectionFailure if the message cannot be sent. Returns the request id of the sent message. :Parameters: - `operation`: opcode of the message - `data`: data to send - `safe`: perform a getLastError after sending the message """ if _connection_to_use is None or _connection_to_use == -1: return self.__master._send_message(message, with_last_error) return self.__slaves[_connection_to_use]._send_message( message, with_last_error, check_primary=False) # _connection_to_use is a hack that we need to include to make sure # that getmore operations can be sent to the same instance on which # the cursor actually resides... def _send_message_with_response(self, message, _connection_to_use=None, _must_use_master=False, **kwargs): """Receive a message from Mongo. Sends the given message and returns a (connection_id, response) pair. :Parameters: - `operation`: opcode of the message to send - `data`: data to send """ if _connection_to_use is not None: if _connection_to_use == -1: member = self.__master conn = -1 else: member = self.__slaves[_connection_to_use] conn = _connection_to_use return (conn, member._send_message_with_response(message, **kwargs)[1]) # _must_use_master is set for commands, which must be sent to the # master instance. any queries in a request must be sent to the # master since that is where writes go. if _must_use_master or self.in_request(): return (-1, self.__master._send_message_with_response(message, **kwargs)[1]) # Iterate through the slaves randomly until we have success. Raise # reconnect if they all fail. for connection_id in helpers.shuffled(xrange(len(self.__slaves))): try: slave = self.__slaves[connection_id] return (connection_id, slave._send_message_with_response(message, **kwargs)[1]) except AutoReconnect: pass raise AutoReconnect("failed to connect to slaves") def start_request(self): """Start a "request". Start a sequence of operations in which order matters. Note that all operations performed within a request will be sent using the Master connection. """ self.__request_counter.inc() self.master.start_request() def in_request(self): return bool(self.__request_counter.get()) def end_request(self): """End the current "request". See documentation for `MongoClient.end_request`. """ self.__request_counter.dec() self.master.end_request() def __eq__(self, other): if isinstance(other, MasterSlaveConnection): us = (self.__master, self.slaves) them = (other.__master, other.__slaves) return us == them return NotImplemented def __ne__(self, other): return not self == other def __repr__(self): return "MasterSlaveConnection(%r, %r)" % (self.__master, self.__slaves) def __getattr__(self, name): """Get a database by name. Raises InvalidName if an invalid database name is used. :Parameters: - `name`: the name of the database to get """ return Database(self, name) def __getitem__(self, name): """Get a database by name. Raises InvalidName if an invalid database name is used. :Parameters: - `name`: the name of the database to get """ return self.__getattr__(name) def close_cursor(self, cursor_id, connection_id): """Close a single database cursor. Raises TypeError if cursor_id is not an instance of (int, long). What closing the cursor actually means depends on this connection's cursor manager. :Parameters: - `cursor_id`: cursor id to close - `connection_id`: id of the `MongoClient` instance where the cursor was opened """ if connection_id == -1: return self.__master.close_cursor(cursor_id) return self.__slaves[connection_id].close_cursor(cursor_id) def database_names(self): """Get a list of all database names. """ return self.__master.database_names() def drop_database(self, name_or_database): """Drop a database. :Parameters: - `name_or_database`: the name of a database to drop or the object itself """ return self.__master.drop_database(name_or_database) def __iter__(self): return self def next(self): raise TypeError("'MasterSlaveConnection' object is not iterable") def _cached(self, database_name, collection_name, index_name): return self.__master._cached(database_name, collection_name, index_name) def _cache_index(self, database_name, collection_name, index_name, cache_for): return self.__master._cache_index(database_name, collection_name, index_name, cache_for) def _purge_index(self, database_name, collection_name=None, index_name=None): return self.__master._purge_index(database_name, collection_name, index_name) pymongo-2.6.3/pymongo/message.py000066400000000000000000000213641223300253600167020ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tools for creating `messages `_ to be sent to MongoDB. .. note:: This module is for internal use and is generally not needed by application developers. .. versionadded:: 1.1.2 """ import random import struct import bson from bson.binary import OLD_UUID_SUBTYPE from bson.py3compat import b from bson.son import SON try: from pymongo import _cmessage _use_c = True except ImportError: _use_c = False from pymongo.errors import InvalidDocument, InvalidOperation, OperationFailure __ZERO = b("\x00\x00\x00\x00") EMPTY = b("") MAX_INT32 = 2147483647 MIN_INT32 = -2147483648 def __last_error(namespace, args): """Data to send to do a lastError. """ cmd = SON([("getlasterror", 1)]) cmd.update(args) splitns = namespace.split('.', 1) return query(0, splitns[0] + '.$cmd', 0, -1, cmd) def __pack_message(operation, data): """Takes message data and adds a message header based on the operation. Returns the resultant message string. """ request_id = random.randint(MIN_INT32, MAX_INT32) message = struct.pack(" client.max_bson_size: raise InvalidDocument("BSON document too large (%d bytes)" " - the connected server supports" " BSON document sizes up to %d" " bytes." % (encoded_length, client.max_bson_size)) message_length += encoded_length if message_length < client.max_message_size: data.append(encoded) continue # We have enough data, send this message. send_safe = safe or not continue_on_error try: client._send_message(_insert_message(EMPTY.join(data), send_safe), send_safe) # Exception type could be OperationFailure or a subtype # (e.g. DuplicateKeyError) except OperationFailure, exc: # Like it says, continue on error... if continue_on_error: # Store exception details to re-raise after the final batch. last_error = exc # With unacknowledged writes just return at the first error. elif not safe: return # With acknowledged writes raise immediately. else: raise message_length = len(begin) + encoded_length data = [begin, encoded] client._send_message(_insert_message(EMPTY.join(data), safe), safe) # Re-raise any exception stored due to continue_on_error if last_error is not None: raise last_error if _use_c: _do_batched_insert = _cmessage._do_batched_insert pymongo-2.6.3/pymongo/mongo_client.py000066400000000000000000001471131223300253600177340ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you # may not use this file except in compliance with the License. You # may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. See the License for the specific language governing # permissions and limitations under the License. """Tools for connecting to MongoDB. .. seealso:: Module :mod:`~pymongo.master_slave_connection` for connecting to master-slave clusters, and :doc:`/examples/high_availability` for an example of how to connect to a replica set, or specify a list of mongos instances for automatic failover. To get a :class:`~pymongo.database.Database` instance from a :class:`MongoClient` use either dictionary-style or attribute-style access: .. doctest:: >>> from pymongo import MongoClient >>> c = MongoClient() >>> c.test_database Database(MongoClient('localhost', 27017), u'test_database') >>> c['test-database'] Database(MongoClient('localhost', 27017), u'test-database') """ import datetime import random import socket import struct import time import warnings from bson.py3compat import b from pymongo import (auth, common, database, helpers, message, pool, uri_parser) from pymongo.common import HAS_SSL from pymongo.cursor_manager import CursorManager from pymongo.errors import (AutoReconnect, ConfigurationError, ConnectionFailure, DuplicateKeyError, InvalidDocument, InvalidURI, OperationFailure) EMPTY = b("") def _partition_node(node): """Split a host:port string returned from mongod/s into a (host, int(port)) pair needed for socket.connect(). """ host = node port = 27017 idx = node.rfind(':') if idx != -1: host, port = node[:idx], int(node[idx + 1:]) if host.startswith('['): host = host[1:-1] return host, port class MongoClient(common.BaseObject): """Connection to MongoDB. """ HOST = "localhost" PORT = 27017 __max_bson_size = 4 * 1024 * 1024 def __init__(self, host=None, port=None, max_pool_size=100, document_class=dict, tz_aware=False, _connect=True, **kwargs): """Create a new connection to a single MongoDB instance at *host:port*. The resultant client object has connection-pooling built in. It also performs auto-reconnection when necessary. If an operation fails because of a connection error, :class:`~pymongo.errors.ConnectionFailure` is raised. If auto-reconnection will be performed, :class:`~pymongo.errors.AutoReconnect` will be raised. Application code should handle this exception (recognizing that the operation failed) and then continue to execute. Raises :class:`TypeError` if port is not an instance of ``int``. Raises :class:`~pymongo.errors.ConnectionFailure` if the connection cannot be made. The `host` parameter can be a full `mongodb URI `_, in addition to a simple hostname. It can also be a list of hostnames or URIs. Any port specified in the host string(s) will override the `port` parameter. If multiple mongodb URIs containing database or auth information are passed, the last database, username, and password present will be used. For username and passwords reserved characters like ':', '/', '+' and '@' must be escaped following RFC 2396. :Parameters: - `host` (optional): hostname or IP address of the instance to connect to, or a mongodb URI, or a list of hostnames / mongodb URIs. If `host` is an IPv6 literal it must be enclosed in '[' and ']' characters following the RFC2732 URL syntax (e.g. '[::1]' for localhost) - `port` (optional): port number on which to connect - `max_pool_size` (optional): The maximum number of connections that the pool will open simultaneously. If this is set, operations will block if there are `max_pool_size` outstanding connections from the pool. Defaults to 100. - `document_class` (optional): default class to use for documents returned from queries on this client - `tz_aware` (optional): if ``True``, :class:`~datetime.datetime` instances returned as values in a document by this :class:`MongoClient` will be timezone aware (otherwise they will be naive) | **Other optional parameters can be passed as keyword arguments:** - `socketTimeoutMS`: (integer) How long (in milliseconds) a send or receive on a socket can take before timing out. - `connectTimeoutMS`: (integer) How long (in milliseconds) a connection can take to be opened before timing out. - `waitQueueTimeoutMS`: (integer) How long (in milliseconds) a thread will wait for a socket from the pool if the pool has no free sockets. - `waitQueueMultiple`: (integer) Multiplied by max_pool_size to give the number of threads allowed to wait for a socket at one time. - `auto_start_request`: If ``True``, each thread that accesses this :class:`MongoClient` has a socket allocated to it for the thread's lifetime. This ensures consistent reads, even if you read after an unacknowledged write. Defaults to ``False`` - `use_greenlets`: If ``True``, :meth:`start_request()` will ensure that the current greenlet uses the same socket for all operations until :meth:`end_request()` | **Write Concern options:** - `w`: (integer or string) If this is a replica set, write operations will block until they have been replicated to the specified number or tagged set of servers. `w=` always includes the replica set primary (e.g. w=3 means write to the primary and wait until replicated to **two** secondaries). Passing w=0 **disables write acknowledgement** and all other write concern options. - `wtimeout`: (integer) Used in conjunction with `w`. Specify a value in milliseconds to control how long to wait for write propagation to complete. If replication does not complete in the given timeframe, a timeout exception is raised. - `j`: If ``True`` block until write operations have been committed to the journal. Ignored if the server is running without journaling. - `fsync`: If ``True`` force the database to fsync all files before returning. When used with `j` the server awaits the next group commit before returning. | **Replica set keyword arguments for connecting with a replica set - either directly or via a mongos:** | (ignored by standalone mongod instances) - `replicaSet`: (string) The name of the replica set to connect to. The driver will verify that the replica set it connects to matches this name. Implies that the hosts specified are a seed list and the driver should attempt to find all members of the set. *Ignored by mongos*. - `read_preference`: The read preference for this client. If connecting to a secondary then a read preference mode *other* than PRIMARY is required - otherwise all queries will throw :class:`~pymongo.errors.AutoReconnect` "not master". See :class:`~pymongo.read_preferences.ReadPreference` for all available read preference options. - `tag_sets`: Ignored unless connecting to a replica set via mongos. Specify a priority-order for tag sets, provide a list of tag sets: ``[{'dc': 'ny'}, {'dc': 'la'}, {}]``. A final, empty tag set, ``{}``, means "read from any member that matches the mode, ignoring tags. | **SSL configuration:** - `ssl`: If ``True``, create the connection to the server using SSL. - `ssl_keyfile`: The private keyfile used to identify the local connection against mongod. If included with the ``certfile` then only the ``ssl_certfile`` is needed. Implies ``ssl=True``. - `ssl_certfile`: The certificate file used to identify the local connection against mongod. Implies ``ssl=True``. - `ssl_cert_reqs`: Specifies whether a certificate is required from the other side of the connection, and whether it will be validated if provided. It must be one of the three values ``ssl.CERT_NONE`` (certificates ignored), ``ssl.CERT_OPTIONAL`` (not required, but validated if provided), or ``ssl.CERT_REQUIRED`` (required and validated). If the value of this parameter is not ``ssl.CERT_NONE``, then the ``ssl_ca_certs`` parameter must point to a file of CA certificates. Implies ``ssl=True``. - `ssl_ca_certs`: The ca_certs file contains a set of concatenated "certification authority" certificates, which are used to validate certificates passed from the other end of the connection. Implies ``ssl=True``. .. seealso:: :meth:`end_request` .. mongodoc:: connections .. versionchanged:: 2.5 Added additional ssl options .. versionadded:: 2.4 """ if host is None: host = self.HOST if isinstance(host, basestring): host = [host] if port is None: port = self.PORT if not isinstance(port, int): raise TypeError("port must be an instance of int") seeds = set() username = None password = None self.__default_database_name = None opts = {} for entity in host: if "://" in entity: if entity.startswith("mongodb://"): res = uri_parser.parse_uri(entity, port) seeds.update(res["nodelist"]) username = res["username"] or username password = res["password"] or password self.__default_database_name = ( res["database"] or self.__default_database_name) opts = res["options"] else: idx = entity.find("://") raise InvalidURI("Invalid URI scheme: " "%s" % (entity[:idx],)) else: seeds.update(uri_parser.split_hosts(entity, port)) if not seeds: raise ConfigurationError("need to specify at least one host") self.__nodes = seeds self.__host = None self.__port = None self.__is_primary = False self.__is_mongos = False # _pool_class option is for deep customization of PyMongo, e.g. Motor. # SHOULD NOT BE USED BY DEVELOPERS EXTERNAL TO 10GEN. pool_class = kwargs.pop('_pool_class', pool.Pool) options = {} for option, value in kwargs.iteritems(): option, value = common.validate(option, value) options[option] = value options.update(opts) self.__max_pool_size = common.validate_positive_integer_or_none( 'max_pool_size', max_pool_size) self.__cursor_manager = CursorManager(self) self.__repl = options.get('replicaset') if len(seeds) == 1 and not self.__repl: self.__direct = True else: self.__direct = False self.__nodes = set() self.__net_timeout = options.get('sockettimeoutms') self.__conn_timeout = options.get('connecttimeoutms') self.__wait_queue_timeout = options.get('waitqueuetimeoutms') self.__wait_queue_multiple = options.get('waitqueuemultiple') self.__use_ssl = options.get('ssl', None) self.__ssl_keyfile = options.get('ssl_keyfile', None) self.__ssl_certfile = options.get('ssl_certfile', None) self.__ssl_cert_reqs = options.get('ssl_cert_reqs', None) self.__ssl_ca_certs = options.get('ssl_ca_certs', None) ssl_kwarg_keys = [k for k in kwargs.keys() if k.startswith('ssl_')] if self.__use_ssl == False and ssl_kwarg_keys: raise ConfigurationError("ssl has not been enabled but the " "following ssl parameters have been set: " "%s. Please set `ssl=True` or remove." % ', '.join(ssl_kwarg_keys)) if self.__ssl_cert_reqs and not self.__ssl_ca_certs: raise ConfigurationError("If `ssl_cert_reqs` is not " "`ssl.CERT_NONE` then you must " "include `ssl_ca_certs` to be able " "to validate the server.") if ssl_kwarg_keys and self.__use_ssl is None: # ssl options imply ssl = True self.__use_ssl = True if self.__use_ssl and not HAS_SSL: raise ConfigurationError("The ssl module is not available. If you " "are using a python version previous to " "2.6 you must install the ssl package " "from PyPI.") self.__use_greenlets = options.get('use_greenlets', False) self.__pool = pool_class( None, self.__max_pool_size, self.__net_timeout, self.__conn_timeout, self.__use_ssl, use_greenlets=self.__use_greenlets, ssl_keyfile=self.__ssl_keyfile, ssl_certfile=self.__ssl_certfile, ssl_cert_reqs=self.__ssl_cert_reqs, ssl_ca_certs=self.__ssl_ca_certs, wait_queue_timeout=self.__wait_queue_timeout, wait_queue_multiple=self.__wait_queue_multiple) self.__document_class = document_class self.__tz_aware = common.validate_boolean('tz_aware', tz_aware) self.__auto_start_request = options.get('auto_start_request', False) # cache of existing indexes used by ensure_index ops self.__index_cache = {} self.__auth_credentials = {} super(MongoClient, self).__init__(**options) if self.slave_okay: warnings.warn("slave_okay is deprecated. Please " "use read_preference instead.", DeprecationWarning, stacklevel=2) if _connect: try: self.__find_node(seeds) except AutoReconnect, e: # ConnectionFailure makes more sense here than AutoReconnect raise ConnectionFailure(str(e)) if username: mechanism = options.get('authmechanism', 'MONGODB-CR') source = ( options.get('authsource') or self.__default_database_name or 'admin') credentials = auth._build_credentials_tuple(mechanism, source, unicode(username), unicode(password), options) try: self._cache_credentials(source, credentials, _connect) except OperationFailure, exc: raise ConfigurationError(str(exc)) def _cached(self, dbname, coll, index): """Test if `index` is cached. """ cache = self.__index_cache now = datetime.datetime.utcnow() return (dbname in cache and coll in cache[dbname] and index in cache[dbname][coll] and now < cache[dbname][coll][index]) def _cache_index(self, database, collection, index, cache_for): """Add an index to the index cache for ensure_index operations. """ now = datetime.datetime.utcnow() expire = datetime.timedelta(seconds=cache_for) + now if database not in self.__index_cache: self.__index_cache[database] = {} self.__index_cache[database][collection] = {} self.__index_cache[database][collection][index] = expire elif collection not in self.__index_cache[database]: self.__index_cache[database][collection] = {} self.__index_cache[database][collection][index] = expire else: self.__index_cache[database][collection][index] = expire def _purge_index(self, database_name, collection_name=None, index_name=None): """Purge an index from the index cache. If `index_name` is None purge an entire collection. If `collection_name` is None purge an entire database. """ if not database_name in self.__index_cache: return if collection_name is None: del self.__index_cache[database_name] return if not collection_name in self.__index_cache[database_name]: return if index_name is None: del self.__index_cache[database_name][collection_name] return if index_name in self.__index_cache[database_name][collection_name]: del self.__index_cache[database_name][collection_name][index_name] def _cache_credentials(self, source, credentials, connect=True): """Add credentials to the database authentication cache for automatic login when a socket is created. If `connect` is True, verify the credentials on the server first. """ if source in self.__auth_credentials: # Nothing to do if we already have these credentials. if credentials == self.__auth_credentials[source]: return raise OperationFailure('Another user is already authenticated ' 'to this database. You must logout first.') if connect: sock_info = self.__socket() try: # Since __check_auth was called in __socket # there is no need to call it here. auth.authenticate(credentials, sock_info, self.__simple_command) sock_info.authset.add(credentials) finally: self.__pool.maybe_return_socket(sock_info) self.__auth_credentials[source] = credentials def _purge_credentials(self, source): """Purge credentials from the database authentication cache. """ if source in self.__auth_credentials: del self.__auth_credentials[source] def __check_auth(self, sock_info): """Authenticate using cached database credentials. """ if self.__auth_credentials or sock_info.authset: cached = set(self.__auth_credentials.itervalues()) authset = sock_info.authset.copy() # Logout any credentials that no longer exist in the cache. for credentials in authset - cached: self.__simple_command(sock_info, credentials[1], {'logout': 1}) sock_info.authset.discard(credentials) for credentials in cached - authset: auth.authenticate(credentials, sock_info, self.__simple_command) sock_info.authset.add(credentials) @property def host(self): """Current connected host. .. versionchanged:: 1.3 ``host`` is now a property rather than a method. """ return self.__host @property def port(self): """Current connected port. .. versionchanged:: 1.3 ``port`` is now a property rather than a method. """ return self.__port @property def is_primary(self): """If this instance is connected to a standalone, a replica set primary, or the master of a master-slave set. .. versionadded:: 2.3 """ return self.__is_primary @property def is_mongos(self): """If this instance is connected to mongos. .. versionadded:: 2.3 """ return self.__is_mongos @property def max_pool_size(self): """The maximum number of sockets the pool will open concurrently. When the pool has reached `max_pool_size`, operations block waiting for a socket to be returned to the pool. If ``waitQueueTimeoutMS`` is set, a blocked operation will raise :exc:`~pymongo.errors.ConnectionFailure` after a timeout. By default ``waitQueueTimeoutMS`` is not set. .. warning:: SIGNIFICANT BEHAVIOR CHANGE in 2.6. Previously, this parameter would limit only the idle sockets the pool would hold onto, not the number of open sockets. The default has also changed to 100. .. versionchanged:: 2.6 .. versionadded:: 1.11 """ return self.__max_pool_size @property def use_greenlets(self): """Whether calling :meth:`start_request` assigns greenlet-local, rather than thread-local, sockets. .. versionadded:: 2.4.2 """ return self.__use_greenlets @property def nodes(self): """List of all known nodes. Includes both nodes specified when this instance was created, as well as nodes discovered through the replica set discovery mechanism. .. versionadded:: 1.8 """ return self.__nodes @property def auto_start_request(self): """Is auto_start_request enabled? """ return self.__auto_start_request def get_document_class(self): return self.__document_class def set_document_class(self, klass): self.__document_class = klass document_class = property(get_document_class, set_document_class, doc="""Default class to use for documents returned from this client. .. versionadded:: 1.7 """) @property def tz_aware(self): """Does this client return timezone-aware datetimes? .. versionadded:: 1.8 """ return self.__tz_aware @property def max_bson_size(self): """Return the maximum size BSON object the connected server accepts in bytes. Defaults to 4MB in server < 1.7.4. .. versionadded:: 1.10 """ return self.__max_bson_size @property def max_message_size(self): """Return the maximum message size the connected server accepts in bytes. .. versionadded:: 2.6 """ return self.__max_message_size def __simple_command(self, sock_info, dbname, spec): """Send a command to the server. """ rqst_id, msg, _ = message.query(0, dbname + '.$cmd', 0, -1, spec) start = time.time() try: sock_info.sock.sendall(msg) response = self.__receive_message_on_socket(1, rqst_id, sock_info) except: sock_info.close() raise end = time.time() response = helpers._unpack_response(response)['data'][0] msg = "command %r failed: %%s" % spec helpers._check_command_response(response, None, msg) return response, end - start def __try_node(self, node): """Try to connect to this node and see if it works for our connection type. Returns ((host, port), ismaster, isdbgrid, res_time). :Parameters: - `node`: The (host, port) pair to try. """ self.disconnect() self.__host, self.__port = node # Call 'ismaster' directly so we can get a response time. sock_info = self.__socket() try: response, res_time = self.__simple_command(sock_info, 'admin', {'ismaster': 1}) finally: self.__pool.maybe_return_socket(sock_info) # Are we talking to a mongos? isdbgrid = response.get('msg', '') == 'isdbgrid' if "maxBsonObjectSize" in response: self.__max_bson_size = response["maxBsonObjectSize"] if "maxMessageSizeBytes" in response: self.__max_message_size = response["maxMessageSizeBytes"] else: self.__max_message_size = 2 * self.max_bson_size # Replica Set? if not self.__direct: # Check that this host is part of the given replica set. if self.__repl: set_name = response.get('setName') # The 'setName' field isn't returned by mongod before 1.6.2 # so we can't assume that if it's missing this host isn't in # the specified set. if set_name and set_name != self.__repl: raise ConfigurationError("%s:%d is not a member of " "replica set %s" % (node[0], node[1], self.__repl)) if "hosts" in response: self.__nodes = set([_partition_node(h) for h in response["hosts"]]) else: # The user passed a seed list of standalone or # mongos instances. self.__nodes.add(node) if response["ismaster"]: return node, True, isdbgrid, res_time elif "primary" in response: candidate = _partition_node(response["primary"]) return self.__try_node(candidate) # Explain why we aren't using this connection. raise AutoReconnect('%s:%d is not primary or master' % node) # Direct connection if response.get("arbiterOnly", False) and not self.__direct: raise ConfigurationError("%s:%d is an arbiter" % node) return node, response['ismaster'], isdbgrid, res_time def __pick_nearest(self, candidates): """Return the 'nearest' candidate based on response time. """ latency = self.secondary_acceptable_latency_ms # Only used for mongos high availability, res_time is in seconds. fastest = min([res_time for candidate, res_time in candidates]) near_candidates = [ candidate for candidate, res_time in candidates if res_time - fastest < latency / 1000.0 ] node = random.choice(near_candidates) # Clear the pool from the last choice. self.disconnect() self.__host, self.__port = node return node def __find_node(self, seeds=None): """Find a host, port pair suitable for our connection type. If only one host was supplied to __init__ see if we can connect to it. Don't check if the host is a master/primary so we can make a direct connection to read from a secondary or send commands to an arbiter. If more than one host was supplied treat them as a seed list for connecting to a replica set or to support high availability for mongos. If connecting to a replica set try to find the primary and fail if we can't, possibly updating any replSet information on success. If a mongos seed list was provided find the "nearest" mongos and return it. Otherwise we iterate through the list trying to find a host we can send write operations to. Sets __host and __port so that :attr:`host` and :attr:`port` will return the address of the connected host. Sets __is_primary to True if this is a primary or master, else False. Sets __is_mongos to True if the connection is to a mongos. """ errors = [] mongos_candidates = [] candidates = seeds or self.__nodes.copy() for candidate in candidates: try: node, ismaster, isdbgrid, res_time = self.__try_node(candidate) self.__is_primary = ismaster self.__is_mongos = isdbgrid # No need to calculate nearest if we only have one mongos. if isdbgrid and not self.__direct: mongos_candidates.append((node, res_time)) continue elif len(mongos_candidates): raise ConfigurationError("Seed list cannot contain a mix " "of mongod and mongos instances.") return node except OperationFailure: # The server is available but something failed, probably auth. raise except Exception, why: errors.append(str(why)) # If we have a mongos seed list, pick the "nearest" member. if len(mongos_candidates): self.__is_mongos = True return self.__pick_nearest(mongos_candidates) # Otherwise, try any hosts we discovered that were not in the seed list. for candidate in self.__nodes - candidates: try: node, ismaster, isdbgrid, _ = self.__try_node(candidate) self.__is_primary = ismaster self.__is_mongos = isdbgrid return node except Exception, why: errors.append(str(why)) # Couldn't find a suitable host. self.disconnect() raise AutoReconnect(', '.join(errors)) def __socket(self): """Get a SocketInfo from the pool. """ host, port = (self.__host, self.__port) if host is None or (port is None and '/' not in host): host, port = self.__find_node() try: if self.auto_start_request and not self.in_request(): self.start_request() sock_info = self.__pool.get_socket((host, port)) except socket.error, why: self.disconnect() # Check if a unix domain socket if host.endswith('.sock'): host_details = "%s:" % host else: host_details = "%s:%d:" % (host, port) raise AutoReconnect("could not connect to " "%s %s" % (host_details, str(why))) try: self.__check_auth(sock_info) except OperationFailure: self.__pool.maybe_return_socket(sock_info) raise return sock_info def _ensure_connected(self, dummy): """Ensure this client instance is connected to a mongod/s. """ host, port = (self.__host, self.__port) if host is None or (port is None and '/' not in host): self.__find_node() def disconnect(self): """Disconnect from MongoDB. Disconnecting will close all underlying sockets in the connection pool. If this instance is used again it will be automatically re-opened. Care should be taken to make sure that :meth:`disconnect` is not called in the middle of a sequence of operations in which ordering is important. This could lead to unexpected results. .. seealso:: :meth:`end_request` .. versionadded:: 1.3 """ self.__pool.reset() self.__host = None self.__port = None def close(self): """Alias for :meth:`disconnect` Disconnecting will close all underlying sockets in the connection pool. If this instance is used again it will be automatically re-opened. Care should be taken to make sure that :meth:`disconnect` is not called in the middle of a sequence of operations in which ordering is important. This could lead to unexpected results. .. seealso:: :meth:`end_request` .. versionadded:: 2.1 """ self.disconnect() def alive(self): """Return ``False`` if there has been an error communicating with the server, else ``True``. This method attempts to check the status of the server with minimal I/O. The current thread / greenlet retrieves a socket from the pool (its request socket if it's in a request, or a random idle socket if it's not in a request) and checks whether calling `select`_ on it raises an error. If there are currently no idle sockets, :meth:`alive` will attempt to actually connect to the server. A more certain way to determine server availability is:: client.admin.command('ping') .. _select: http://docs.python.org/2/library/select.html#select.select """ # In the common case, a socket is available and was used recently, so # calling select() on it is a reasonable attempt to see if the OS has # reported an error. Note this can be wasteful: __socket implicitly # calls select() if the socket hasn't been checked in the last second, # or it may create a new socket, in which case calling select() is # redundant. sock_info = None try: try: sock_info = self.__socket() return not pool._closed(sock_info.sock) except (socket.error, ConnectionFailure): return False finally: self.__pool.maybe_return_socket(sock_info) def set_cursor_manager(self, manager_class): """Set this client's cursor manager. Raises :class:`TypeError` if `manager_class` is not a subclass of :class:`~pymongo.cursor_manager.CursorManager`. A cursor manager handles closing cursors. Different managers can implement different policies in terms of when to actually kill a cursor that has been closed. :Parameters: - `manager_class`: cursor manager to use .. versionchanged:: 2.1+ Deprecated support for external cursor managers. """ warnings.warn("Support for external cursor managers is deprecated " "and will be removed in PyMongo 3.0.", DeprecationWarning, stacklevel=2) manager = manager_class(self) if not isinstance(manager, CursorManager): raise TypeError("manager_class must be a subclass of " "CursorManager") self.__cursor_manager = manager def __check_response_to_last_error(self, response): """Check a response to a lastError message for errors. `response` is a byte string representing a response to the message. If it represents an error response we raise OperationFailure. Return the response as a document. """ response = helpers._unpack_response(response) assert response["number_returned"] == 1 error = response["data"][0] helpers._check_command_response(error, self.disconnect) error_msg = error.get("err", "") if error_msg is None: return error if error_msg.startswith("not master"): self.disconnect() raise AutoReconnect(error_msg) details = error # mongos returns the error code in an error object # for some errors. if "errObjects" in error: for errobj in error["errObjects"]: if errobj["err"] == error_msg: details = errobj break if "code" in details: if details["code"] in (11000, 11001, 12582): raise DuplicateKeyError(details["err"], details["code"]) else: raise OperationFailure(details["err"], details["code"]) else: raise OperationFailure(details["err"]) def __check_bson_size(self, message): """Make sure the message doesn't include BSON documents larger than the connected server will accept. :Parameters: - `message`: message to check """ if len(message) == 3: (request_id, data, max_doc_size) = message if max_doc_size > self.__max_bson_size: raise InvalidDocument("BSON document too large (%d bytes)" " - the connected server supports" " BSON document sizes up to %d" " bytes." % (max_doc_size, self.__max_bson_size)) return (request_id, data) else: # get_more and kill_cursors messages # don't include BSON documents. return message def _send_message(self, message, with_last_error=False, check_primary=True): """Say something to Mongo. Raises ConnectionFailure if the message cannot be sent. Raises OperationFailure if `with_last_error` is ``True`` and the response to the getLastError call returns an error. Return the response from lastError, or ``None`` if `with_last_error` is ``False``. :Parameters: - `message`: message to send - `with_last_error`: check getLastError status after sending the message - `check_primary`: don't try to write to a non-primary; see kill_cursors for an exception to this rule """ if check_primary and not with_last_error and not self.is_primary: # The write won't succeed, bail as if we'd done a getLastError raise AutoReconnect("not master") sock_info = self.__socket() try: try: (request_id, data) = self.__check_bson_size(message) sock_info.sock.sendall(data) # Safe mode. We pack the message together with a lastError # message and send both. We then get the response (to the # lastError) and raise OperationFailure if it is an error # response. rv = None if with_last_error: response = self.__receive_message_on_socket(1, request_id, sock_info) rv = self.__check_response_to_last_error(response) return rv except OperationFailure: raise except (ConnectionFailure, socket.error), e: self.disconnect() raise AutoReconnect(str(e)) except: sock_info.close() raise finally: self.__pool.maybe_return_socket(sock_info) def __receive_data_on_socket(self, length, sock_info): """Lowest level receive operation. Takes length to receive and repeatedly calls recv until able to return a buffer of that length, raising ConnectionFailure on error. """ message = EMPTY while length: chunk = sock_info.sock.recv(length) if chunk == EMPTY: raise ConnectionFailure("connection closed") length -= len(chunk) message += chunk return message def __receive_message_on_socket(self, operation, rqst_id, sock_info): """Receive a message in response to `rqst_id` on `sock`. Returns the response data with the header removed. """ header = self.__receive_data_on_socket(16, sock_info) length = struct.unpack(">> client = pymongo.MongoClient(auto_start_request=False) >>> db = client.test >>> _id = db.test_collection.insert({}) >>> with client.start_request(): ... for i in range(100): ... db.test_collection.update({'_id': _id}, {'$set': {'i':i}}) ... ... # Definitely read the document after the final update completes ... print db.test_collection.find({'_id': _id}) If a thread or greenlet calls start_request multiple times, an equal number of calls to :meth:`end_request` is required to end the request. .. versionchanged:: 2.4 Now counts the number of calls to start_request and doesn't end request until an equal number of calls to end_request. .. versionadded:: 2.2 The :class:`~pymongo.pool.Request` return value. :meth:`start_request` previously returned None """ self.__pool.start_request() return pool.Request(self) def in_request(self): """True if this thread is in a request, meaning it has a socket reserved for its exclusive use. """ return self.__pool.in_request() def end_request(self): """Undo :meth:`start_request`. If :meth:`end_request` is called as many times as :meth:`start_request`, the request is over and this thread's connection returns to the pool. Extra calls to :meth:`end_request` have no effect. Ending a request allows the :class:`~socket.socket` that has been reserved for this thread by :meth:`start_request` to be returned to the pool. Other threads will then be able to re-use that :class:`~socket.socket`. If your application uses many threads, or has long-running threads that infrequently perform MongoDB operations, then judicious use of this method can lead to performance gains. Care should be taken, however, to make sure that :meth:`end_request` is not called in the middle of a sequence of operations in which ordering is important. This could lead to unexpected results. """ self.__pool.end_request() def __eq__(self, other): if isinstance(other, self.__class__): us = (self.__host, self.__port) them = (other.__host, other.__port) return us == them return NotImplemented def __ne__(self, other): return not self == other def __repr__(self): if len(self.__nodes) == 1: return "MongoClient(%r, %r)" % (self.__host, self.__port) else: return "MongoClient(%r)" % ["%s:%d" % n for n in self.__nodes] def __getattr__(self, name): """Get a database by name. Raises :class:`~pymongo.errors.InvalidName` if an invalid database name is used. :Parameters: - `name`: the name of the database to get """ return database.Database(self, name) def __getitem__(self, name): """Get a database by name. Raises :class:`~pymongo.errors.InvalidName` if an invalid database name is used. :Parameters: - `name`: the name of the database to get """ return self.__getattr__(name) def close_cursor(self, cursor_id): """Close a single database cursor. Raises :class:`TypeError` if `cursor_id` is not an instance of ``(int, long)``. What closing the cursor actually means depends on this client's cursor manager. :Parameters: - `cursor_id`: id of cursor to close """ if not isinstance(cursor_id, (int, long)): raise TypeError("cursor_id must be an instance of (int, long)") self.__cursor_manager.close(cursor_id) def kill_cursors(self, cursor_ids): """Send a kill cursors message with the given ids. Raises :class:`TypeError` if `cursor_ids` is not an instance of ``list``. :Parameters: - `cursor_ids`: list of cursor ids to kill """ if not isinstance(cursor_ids, list): raise TypeError("cursor_ids must be a list") return self._send_message( message.kill_cursors(cursor_ids), check_primary=False) def server_info(self): """Get information about the MongoDB server we're connected to. """ return self.admin.command("buildinfo") def database_names(self): """Get a list of the names of all databases on the connected server. """ return [db["name"] for db in self.admin.command("listDatabases")["databases"]] def drop_database(self, name_or_database): """Drop a database. Raises :class:`TypeError` if `name_or_database` is not an instance of :class:`basestring` (:class:`str` in python 3) or Database. :Parameters: - `name_or_database`: the name of a database to drop, or a :class:`~pymongo.database.Database` instance representing the database to drop """ name = name_or_database if isinstance(name, database.Database): name = name.name if not isinstance(name, basestring): raise TypeError("name_or_database must be an instance of " "%s or Database" % (basestring.__name__,)) self._purge_index(name) self[name].command("dropDatabase") def copy_database(self, from_name, to_name, from_host=None, username=None, password=None): """Copy a database, potentially from another host. Raises :class:`TypeError` if `from_name` or `to_name` is not an instance of :class:`basestring` (:class:`str` in python 3). Raises :class:`~pymongo.errors.InvalidName` if `to_name` is not a valid database name. If `from_host` is ``None`` the current host is used as the source. Otherwise the database is copied from `from_host`. If the source database requires authentication, `username` and `password` must be specified. :Parameters: - `from_name`: the name of the source database - `to_name`: the name of the target database - `from_host` (optional): host name to copy from - `username` (optional): username for source database - `password` (optional): password for source database .. note:: Specifying `username` and `password` requires server version **>= 1.3.3+**. .. versionadded:: 1.5 """ if not isinstance(from_name, basestring): raise TypeError("from_name must be an instance " "of %s" % (basestring.__name__,)) if not isinstance(to_name, basestring): raise TypeError("to_name must be an instance " "of %s" % (basestring.__name__,)) database._check_name(to_name) command = {"fromdb": from_name, "todb": to_name} if from_host is not None: command["fromhost"] = from_host try: self.start_request() if username is not None: nonce = self.admin.command("copydbgetnonce", fromhost=from_host)["nonce"] command["username"] = username command["nonce"] = nonce command["key"] = auth._auth_key(nonce, username, password) return self.admin.command("copydb", **command) finally: self.end_request() def get_default_database(self): """Get the database named in the MongoDB connection URI. >>> uri = 'mongodb://host/my_database' >>> client = MongoClient(uri) >>> db = client.get_default_database() >>> assert db.name == 'my_database' Useful in scripts where you want to choose which database to use based only on the URI in a configuration file. """ if self.__default_database_name is None: raise ConfigurationError('No default database defined') return self[self.__default_database_name] @property def is_locked(self): """Is this server locked? While locked, all write operations are blocked, although read operations may still be allowed. Use :meth:`unlock` to unlock. .. versionadded:: 2.0 """ ops = self.admin.current_op() return bool(ops.get('fsyncLock', 0)) def fsync(self, **kwargs): """Flush all pending writes to datafiles. :Parameters: Optional parameters can be passed as keyword arguments: - `lock`: If True lock the server to disallow writes. - `async`: If True don't block while synchronizing. .. warning:: `async` and `lock` can not be used together. .. warning:: MongoDB does not support the `async` option on Windows and will raise an exception on that platform. .. versionadded:: 2.0 """ self.admin.command("fsync", **kwargs) def unlock(self): """Unlock a previously locked server. .. versionadded:: 2.0 """ self.admin['$cmd'].sys.unlock.find_one() def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): self.disconnect() def __iter__(self): return self def next(self): raise TypeError("'MongoClient' object is not iterable") pymongo-2.6.3/pymongo/mongo_replica_set_client.py000066400000000000000000002145351223300253600223110ustar00rootroot00000000000000# Copyright 2011-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you # may not use this file except in compliance with the License. You # may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. See the License for the specific language governing # permissions and limitations under the License. """Tools for connecting to a MongoDB replica set. .. seealso:: :doc:`/examples/high_availability` for more examples of how to connect to a replica set. To get a :class:`~pymongo.database.Database` instance from a :class:`MongoReplicaSetClient` use either dictionary-style or attribute-style access: .. doctest:: >>> from pymongo import MongoReplicaSetClient >>> c = MongoReplicaSetClient('localhost:27017', replicaSet='repl0') >>> c.test_database Database(MongoReplicaSetClient([u'...', u'...']), u'test_database') >>> c['test_database'] Database(MongoReplicaSetClient([u'...', u'...']), u'test_database') """ import atexit import datetime import socket import struct import threading import time import warnings import weakref from bson.py3compat import b from pymongo import (auth, common, database, helpers, message, pool, thread_util, uri_parser) from pymongo.read_preferences import ( ReadPreference, select_member, modes, MovingAverage) from pymongo.errors import (AutoReconnect, ConfigurationError, ConnectionFailure, DuplicateKeyError, InvalidDocument, OperationFailure, InvalidOperation) EMPTY = b("") MAX_BSON_SIZE = 4 * 1024 * 1024 MAX_RETRY = 3 # Member states PRIMARY = 1 SECONDARY = 2 OTHER = 3 MONITORS = set() def register_monitor(monitor): ref = weakref.ref(monitor, _on_monitor_deleted) MONITORS.add(ref) def _on_monitor_deleted(ref): """Remove the weakreference from the set of active MONITORS. We no longer care about keeping track of it """ MONITORS.remove(ref) def shutdown_monitors(): # Keep a local copy of MONITORS as # shutting down threads has a side effect # of removing them from the MONITORS set() monitors = list(MONITORS) for ref in monitors: monitor = ref() if monitor: monitor.shutdown() monitor.join() atexit.register(shutdown_monitors) def _partition_node(node): """Split a host:port string returned from mongod/s into a (host, int(port)) pair needed for socket.connect(). """ host = node port = 27017 idx = node.rfind(':') if idx != -1: host, port = node[:idx], int(node[idx + 1:]) if host.startswith('['): host = host[1:-1] return host, port # Concurrency notes: A MongoReplicaSetClient keeps its view of the replica-set # state in an RSState instance. RSStates are immutable, except for # host-pinning. Pools, which are internally thread / greenlet safe, can be # copied from old to new RSStates safely. The client updates its view of the # set's state not by modifying its RSState but by replacing it with an updated # copy. # In __init__, MongoReplicaSetClient gets a list of potential members called # 'seeds' from its initial parameters, and calls refresh(). refresh() iterates # over the the seeds in arbitrary order looking for a member it can connect to. # Once it finds one, it calls 'ismaster' and sets self.__hosts to the list of # members in the response, and connects to the rest of the members. refresh() # sets the MongoReplicaSetClient's RSState. Finally, __init__ launches the # replica-set monitor. # The monitor calls refresh() every 30 seconds, or whenever the client has # encountered an error that prompts it to wake the monitor. # Every method that accesses the RSState multiple times within the method makes # a local reference first and uses that throughout, so it's isolated from a # concurrent method replacing the RSState with an updated copy. This technique # avoids the need to lock around accesses to the RSState. class RSState(object): def __init__( self, threadlocal, host_to_member=None, arbiters=None, writer=None, error_message='No primary available'): """An immutable snapshot of the client's view of the replica set state. :Parameters: - `threadlocal`: Thread- or greenlet-local storage - `host_to_member`: Optional dict: (host, port) -> Member instance - `arbiters`: Optional sequence of arbiters as (host, port) - `writer`: Optional (host, port) of primary - `error_message`: Optional error if `writer` is None """ self._threadlocal = threadlocal # threading.local or gevent local self._arbiters = frozenset(arbiters or []) # set of (host, port) self._writer = writer # (host, port) of the primary, or None self._error_message = error_message self._host_to_member = host_to_member or {} self._hosts = frozenset(self._host_to_member) self._members = frozenset(self._host_to_member.values()) if writer and self._host_to_member[writer].up: self._primary_member = self._host_to_member[writer] else: self._primary_member = None def clone_with_host_down(self, host, error_message): """Get a clone, marking as "down" the member with the given (host, port) """ members = self._host_to_member.copy() down_member = members.pop(host, None) if down_member: members[host] = down_member.clone_down() if host == self.writer: # The primary went down; record the error message. return RSState( self._threadlocal, members, self._arbiters, None, error_message) else: # Some other host went down. Keep our current primary or, if it's # already down, keep our current error message. return RSState( self._threadlocal, members, self._arbiters, self._writer, self._error_message) def clone_without_writer(self, threadlocal): """Get a clone without a primary. Unpins all threads. :Parameters: - `threadlocal`: Thread- or greenlet-local storage """ return RSState( threadlocal, self._host_to_member.copy(), self._arbiters, None) @property def arbiters(self): """Set of (host, port) pairs.""" return self._arbiters @property def writer(self): """(host, port) of primary, or None.""" return self._writer @property def primary_member(self): return self._primary_member @property def hosts(self): """Set of (host, port) tuples of data members of the replica set.""" return self._hosts @property def members(self): """Set of Member instances.""" return self._members @property def error_message(self): """The error, if any, raised when trying to connect to the primary""" return self._error_message @property def secondaries(self): """Set of (host, port) pairs.""" # Unlike the other properties, this isn't cached because it isn't used # in regular operations. return set([ host for host, member in self._host_to_member.items() if member.is_secondary]) def get(self, host): """Return a Member instance or None for the given (host, port).""" return self._host_to_member.get(host) def pin_host(self, host, mode, tag_sets, latency): """Pin this thread / greenlet to a member. `host` is a (host, port) pair. The remaining parameters are a read preference. """ # Fun fact: Unlike in thread_util.ThreadIdent, we needn't lock around # assignment here. Assignment to a threadlocal is only unsafe if it # can cause other Python code to run implicitly. self._threadlocal.host = host self._threadlocal.read_preference = (mode, tag_sets, latency) def keep_pinned_host(self, mode, tag_sets, latency): """Does a read pref match the last used by this thread / greenlet?""" return self._threadlocal.read_preference == (mode, tag_sets, latency) @property def pinned_host(self): """The (host, port) last used by this thread / greenlet, or None.""" return getattr(self._threadlocal, 'host', None) def unpin_host(self): """Forget this thread / greenlet's last used member.""" self._threadlocal.host = self._threadlocal.read_preference = None @property def threadlocal(self): return self._threadlocal def __str__(self): return '' % ( ', '.join(str(member) for member in self._host_to_member.itervalues()), self.writer and '%s:%s' % self.writer or None) class Monitor(object): """Base class for replica set monitors. """ _refresh_interval = 30 def __init__(self, rsc, event_class): self.rsc = weakref.proxy(rsc, self.shutdown) self.timer = event_class() self.refreshed = event_class() self.started_event = event_class() self.stopped = False def start_sync(self): """Start the Monitor and block until it's really started. """ self.start() # Implemented in subclasses. self.started_event.wait(5) def shutdown(self, dummy=None): """Signal the monitor to shutdown. """ self.stopped = True self.timer.set() def schedule_refresh(self): """Refresh immediately """ if not self.isAlive(): raise InvalidOperation( "Monitor thread is dead: Perhaps started before a fork?") self.refreshed.clear() self.timer.set() def wait_for_refresh(self, timeout_seconds): """Block until a scheduled refresh completes """ self.refreshed.wait(timeout_seconds) def monitor(self): """Run until the RSC is collected or an unexpected error occurs. """ self.started_event.set() while True: self.timer.wait(Monitor._refresh_interval) if self.stopped: break self.timer.clear() try: try: self.rsc.refresh() finally: self.refreshed.set() except AutoReconnect: pass # RSC has been collected or there # was an unexpected error. except: break def isAlive(self): raise NotImplementedError() class MonitorThread(threading.Thread, Monitor): """Thread based replica set monitor. """ def __init__(self, rsc): Monitor.__init__(self, rsc, threading.Event) threading.Thread.__init__(self) self.setName("ReplicaSetMonitorThread") # Track whether the thread has started. (Greenlets track this already.) self.started = False def start(self): self.started = True super(MonitorThread, self).start() def run(self): """Override Thread's run method. """ self.monitor() have_gevent = False try: from gevent import Greenlet from gevent.event import Event # Used by ReplicaSetConnection from gevent.local import local as gevent_local have_gevent = True class MonitorGreenlet(Monitor, Greenlet): """Greenlet based replica set monitor. """ def __init__(self, rsc): Monitor.__init__(self, rsc, Event) Greenlet.__init__(self) # Don't override `run` in a Greenlet. Add _run instead. # Refer to gevent's Greenlet docs and source for more # information. def _run(self): """Define Greenlet's _run method. """ self.monitor() def isAlive(self): # Gevent defines bool(Greenlet) as True if it's alive. return bool(self) except ImportError: pass class Member(object): """Immutable representation of one member of a replica set. :Parameters: - `host`: A (host, port) pair - `connection_pool`: A Pool instance - `ismaster_response`: A dict, MongoDB's ismaster response - `ping_time`: A MovingAverage instance - `up`: Whether we think this member is available """ # For unittesting only. Use under no circumstances! _host_to_ping_time = {} def __init__(self, host, connection_pool, ismaster_response, ping_time, up): self.host = host self.pool = connection_pool self.ismaster_response = ismaster_response self.ping_time = ping_time self.up = up if ismaster_response['ismaster']: self.state = PRIMARY elif ismaster_response.get('secondary'): self.state = SECONDARY else: self.state = OTHER self.tags = ismaster_response.get('tags', {}) self.max_bson_size = ismaster_response.get( 'maxBsonObjectSize', MAX_BSON_SIZE) self.max_message_size = ismaster_response.get( 'maxMessageSizeBytes', 2 * self.max_bson_size) def clone_with(self, ismaster_response, ping_time_sample): """Get a clone updated with ismaster response and a single ping time. """ ping_time = self.ping_time.clone_with(ping_time_sample) return Member(self.host, self.pool, ismaster_response, ping_time, True) def clone_down(self): """Get a clone of this Member, but with up=False. """ return Member( self.host, self.pool, self.ismaster_response, self.ping_time, False) @property def is_primary(self): return self.state == PRIMARY @property def is_secondary(self): return self.state == SECONDARY def get_avg_ping_time(self): """Get a moving average of this member's ping times. """ if self.host in Member._host_to_ping_time: # Simulate ping times for unittesting return Member._host_to_ping_time[self.host] return self.ping_time.get() def matches_mode(self, mode): if mode == ReadPreference.PRIMARY and not self.is_primary: return False if mode == ReadPreference.SECONDARY and not self.is_secondary: return False # If we're not primary or secondary, then we're in a state like # RECOVERING and we don't match any mode return self.is_primary or self.is_secondary def matches_tags(self, tags): """Return True if this member's tags are a superset of the passed-in tags. E.g., if this member is tagged {'dc': 'ny', 'rack': '1'}, then it matches {'dc': 'ny'}. """ for key, value in tags.items(): if key not in self.tags or self.tags[key] != value: return False return True def matches_tag_sets(self, tag_sets): """Return True if this member matches any of the tag sets, e.g. [{'dc': 'ny'}, {'dc': 'la'}, {}] """ for tags in tag_sets: if self.matches_tags(tags): return True return False def __str__(self): return '' % ( self.host[0], self.host[1], self.is_primary, self.up) class MongoReplicaSetClient(common.BaseObject): """Connection to a MongoDB replica set. """ def __init__(self, hosts_or_uri=None, max_pool_size=100, document_class=dict, tz_aware=False, _connect=True, **kwargs): """Create a new connection to a MongoDB replica set. The resultant client object has connection-pooling built in. It also performs auto-reconnection when necessary. If an operation fails because of a connection error, :class:`~pymongo.errors.ConnectionFailure` is raised. If auto-reconnection will be performed, :class:`~pymongo.errors.AutoReconnect` will be raised. Application code should handle this exception (recognizing that the operation failed) and then continue to execute. Raises :class:`~pymongo.errors.ConnectionFailure` if the connection cannot be made. The `hosts_or_uri` parameter can be a full `mongodb URI `_, in addition to a string of `host:port` pairs (e.g. 'host1:port1,host2:port2'). If `hosts_or_uri` is None 'localhost:27017' will be used. .. note:: Instances of :class:`MongoReplicaSetClient` start a background task to monitor the state of the replica set. This allows it to quickly respond to changes in replica set configuration. Before discarding an instance of :class:`MongoReplicaSetClient` make sure you call :meth:`~close` to ensure that the monitor task is cleanly shut down. .. note:: A :class:`MongoReplicaSetClient` created before a call to ``os.fork()`` is invalid after the fork. Applications should either fork before creating the client, or recreate the client after a fork. :Parameters: - `hosts_or_uri` (optional): A MongoDB URI or string of `host:port` pairs. If a host is an IPv6 literal it must be enclosed in '[' and ']' characters following the RFC2732 URL syntax (e.g. '[::1]' for localhost) - `max_pool_size` (optional): The maximum number of connections each pool will open simultaneously. If this is set, operations will block if there are `max_pool_size` outstanding connections from the pool. Defaults to 100. - `document_class` (optional): default class to use for documents returned from queries on this client - `tz_aware` (optional): if ``True``, :class:`~datetime.datetime` instances returned as values in a document by this :class:`MongoReplicaSetClient` will be timezone aware (otherwise they will be naive) - `replicaSet`: (required) The name of the replica set to connect to. The driver will verify that each host it connects to is a member of this replica set. Can be passed as a keyword argument or as a MongoDB URI option. | **Other optional parameters can be passed as keyword arguments:** - `host`: For compatibility with :class:`~mongo_client.MongoClient`. If both `host` and `hosts_or_uri` are specified `host` takes precedence. - `port`: For compatibility with :class:`~mongo_client.MongoClient`. The default port number to use for hosts. - `socketTimeoutMS`: (integer) How long (in milliseconds) a send or receive on a socket can take before timing out. - `connectTimeoutMS`: (integer) How long (in milliseconds) a connection can take to be opened before timing out. - `waitQueueTimeoutMS`: (integer) How long (in milliseconds) a thread will wait for a socket from the pool if the pool has no free sockets. Defaults to ``None`` (no timeout). - `waitQueueMultiple`: (integer) Multiplied by max_pool_size to give the number of threads allowed to wait for a socket at one time. Defaults to ``None`` (no waiters). - `auto_start_request`: If ``True``, each thread that accesses this :class:`MongoReplicaSetClient` has a socket allocated to it for the thread's lifetime, for each member of the set. For :class:`~pymongo.read_preferences.ReadPreference` PRIMARY, auto_start_request=True ensures consistent reads, even if you read after an unacknowledged write. For read preferences other than PRIMARY, there are no consistency guarantees. Default to ``False``. - `use_greenlets`: If ``True``, use a background Greenlet instead of a background thread to monitor state of replica set. Additionally, :meth:`start_request()` assigns a greenlet-local, rather than thread-local, socket. `use_greenlets` with :class:`MongoReplicaSetClient` requires `Gevent `_ to be installed. | **Write Concern options:** - `w`: (integer or string) Write operations will block until they have been replicated to the specified number or tagged set of servers. `w=` always includes the replica set primary (e.g. w=3 means write to the primary and wait until replicated to **two** secondaries). Passing w=0 **disables write acknowledgement** and all other write concern options. - `wtimeout`: (integer) Used in conjunction with `w`. Specify a value in milliseconds to control how long to wait for write propagation to complete. If replication does not complete in the given timeframe, a timeout exception is raised. - `j`: If ``True`` block until write operations have been committed to the journal. Ignored if the server is running without journaling. - `fsync`: If ``True`` force the database to fsync all files before returning. When used with `j` the server awaits the next group commit before returning. | **Read preference options:** - `read_preference`: The read preference for this client. See :class:`~pymongo.read_preferences.ReadPreference` for available options. - `tag_sets`: Read from replica-set members with these tags. To specify a priority-order for tag sets, provide a list of tag sets: ``[{'dc': 'ny'}, {'dc': 'la'}, {}]``. A final, empty tag set, ``{}``, means "read from any member that matches the mode, ignoring tags." :class:`MongoReplicaSetClient` tries each set of tags in turn until it finds a set of tags with at least one matching member. - `secondary_acceptable_latency_ms`: (integer) Any replica-set member whose ping time is within secondary_acceptable_latency_ms of the nearest member may accept reads. Default 15 milliseconds. **Ignored by mongos** and must be configured on the command line. See the localThreshold_ option for more information. | **SSL configuration:** - `ssl`: If ``True``, create the connection to the servers using SSL. - `ssl_keyfile`: The private keyfile used to identify the local connection against mongod. If included with the ``certfile` then only the ``ssl_certfile`` is needed. Implies ``ssl=True``. - `ssl_certfile`: The certificate file used to identify the local connection against mongod. Implies ``ssl=True``. - `ssl_cert_reqs`: Specifies whether a certificate is required from the other side of the connection, and whether it will be validated if provided. It must be one of the three values ``ssl.CERT_NONE`` (certificates ignored), ``ssl.CERT_OPTIONAL`` (not required, but validated if provided), or ``ssl.CERT_REQUIRED`` (required and validated). If the value of this parameter is not ``ssl.CERT_NONE``, then the ``ssl_ca_certs`` parameter must point to a file of CA certificates. Implies ``ssl=True``. - `ssl_ca_certs`: The ca_certs file contains a set of concatenated "certification authority" certificates, which are used to validate certificates passed from the other end of the connection. Implies ``ssl=True``. .. versionchanged:: 2.5 Added additional ssl options .. versionadded:: 2.4 .. _localThreshold: http://docs.mongodb.org/manual/reference/mongos/#cmdoption-mongos--localThreshold """ self.__opts = {} self.__seeds = set() self.__index_cache = {} self.__auth_credentials = {} self.__max_pool_size = common.validate_positive_integer_or_none( 'max_pool_size', max_pool_size) self.__tz_aware = common.validate_boolean('tz_aware', tz_aware) self.__document_class = document_class self.__monitor = None # Compatibility with mongo_client.MongoClient host = kwargs.pop('host', hosts_or_uri) port = kwargs.pop('port', 27017) if not isinstance(port, int): raise TypeError("port must be an instance of int") username = None password = None self.__default_database_name = None options = {} if host is None: self.__seeds.add(('localhost', port)) elif '://' in host: res = uri_parser.parse_uri(host, port) self.__seeds.update(res['nodelist']) username = res['username'] password = res['password'] self.__default_database_name = res['database'] options = res['options'] else: self.__seeds.update(uri_parser.split_hosts(host, port)) # _pool_class and _monitor_class are for deep customization of PyMongo, # e.g. Motor. SHOULD NOT BE USED BY DEVELOPERS EXTERNAL TO 10GEN. self.pool_class = kwargs.pop('_pool_class', pool.Pool) monitor_class = kwargs.pop('_monitor_class', None) for option, value in kwargs.iteritems(): option, value = common.validate(option, value) self.__opts[option] = value self.__opts.update(options) self.__use_greenlets = self.__opts.get('use_greenlets', False) if self.__use_greenlets and not have_gevent: raise ConfigurationError( "The gevent module is not available. " "Install the gevent package from PyPI.") self.__rs_state = RSState(self.__make_threadlocal()) self.__request_counter = thread_util.Counter(self.__use_greenlets) self.__auto_start_request = self.__opts.get('auto_start_request', False) if self.__auto_start_request: self.start_request() self.__name = self.__opts.get('replicaset') if not self.__name: raise ConfigurationError("the replicaSet " "keyword parameter is required.") self.__net_timeout = self.__opts.get('sockettimeoutms') self.__conn_timeout = self.__opts.get('connecttimeoutms') self.__wait_queue_timeout = self.__opts.get('waitqueuetimeoutms') self.__wait_queue_multiple = self.__opts.get('waitqueuemultiple') self.__use_ssl = self.__opts.get('ssl', None) self.__ssl_keyfile = self.__opts.get('ssl_keyfile', None) self.__ssl_certfile = self.__opts.get('ssl_certfile', None) self.__ssl_cert_reqs = self.__opts.get('ssl_cert_reqs', None) self.__ssl_ca_certs = self.__opts.get('ssl_ca_certs', None) ssl_kwarg_keys = [k for k in kwargs.keys() if k.startswith('ssl_')] if not self.__use_ssl and ssl_kwarg_keys: raise ConfigurationError("ssl has not been enabled but the " "following ssl parameters have been set: " "%s. Please set `ssl=True` or remove." % ', '.join(ssl_kwarg_keys)) if self.__ssl_cert_reqs and not self.__ssl_ca_certs: raise ConfigurationError("If `ssl_cert_reqs` is not " "`ssl.CERT_NONE` then you must " "include `ssl_ca_certs` to be able " "to validate the server.") if ssl_kwarg_keys and self.__use_ssl is None: # ssl options imply ssl = True self.__use_ssl = True if self.__use_ssl and not common.HAS_SSL: raise ConfigurationError("The ssl module is not available. If you " "are using a python version previous to " "2.6 you must install the ssl package " "from PyPI.") super(MongoReplicaSetClient, self).__init__(**self.__opts) if self.slave_okay: warnings.warn("slave_okay is deprecated. Please " "use read_preference instead.", DeprecationWarning, stacklevel=2) if _connect: try: self.refresh() except AutoReconnect, e: # ConnectionFailure makes more sense here than AutoReconnect raise ConnectionFailure(str(e)) if username: mechanism = options.get('authmechanism', 'MONGODB-CR') source = ( options.get('authsource') or self.__default_database_name or 'admin') credentials = auth._build_credentials_tuple(mechanism, source, unicode(username), unicode(password), options) try: self._cache_credentials(source, credentials, _connect) except OperationFailure, exc: raise ConfigurationError(str(exc)) # Start the monitor after we know the configuration is correct. if monitor_class: self.__monitor = monitor_class(self) elif self.__use_greenlets: self.__monitor = MonitorGreenlet(self) else: self.__monitor = MonitorThread(self) self.__monitor.setDaemon(True) register_monitor(self.__monitor) if _connect: # Wait for the monitor to really start. Otherwise if we return to # caller and caller forks immediately, the monitor could think it's # still alive in the child process when it really isn't. # See http://bugs.python.org/issue18418. self.__monitor.start_sync() def _cached(self, dbname, coll, index): """Test if `index` is cached. """ cache = self.__index_cache now = datetime.datetime.utcnow() return (dbname in cache and coll in cache[dbname] and index in cache[dbname][coll] and now < cache[dbname][coll][index]) def _cache_index(self, dbase, collection, index, cache_for): """Add an index to the index cache for ensure_index operations. """ now = datetime.datetime.utcnow() expire = datetime.timedelta(seconds=cache_for) + now if dbase not in self.__index_cache: self.__index_cache[dbase] = {} self.__index_cache[dbase][collection] = {} self.__index_cache[dbase][collection][index] = expire elif collection not in self.__index_cache[dbase]: self.__index_cache[dbase][collection] = {} self.__index_cache[dbase][collection][index] = expire else: self.__index_cache[dbase][collection][index] = expire def _purge_index(self, database_name, collection_name=None, index_name=None): """Purge an index from the index cache. If `index_name` is None purge an entire collection. If `collection_name` is None purge an entire database. """ if not database_name in self.__index_cache: return if collection_name is None: del self.__index_cache[database_name] return if not collection_name in self.__index_cache[database_name]: return if index_name is None: del self.__index_cache[database_name][collection_name] return if index_name in self.__index_cache[database_name][collection_name]: del self.__index_cache[database_name][collection_name][index_name] def _cache_credentials(self, source, credentials, connect=True): """Add credentials to the database authentication cache for automatic login when a socket is created. If `connect` is True, verify the credentials on the server first. Raises OperationFailure if other credentials are already stored for this source. """ if source in self.__auth_credentials: # Nothing to do if we already have these credentials. if credentials == self.__auth_credentials[source]: return raise OperationFailure('Another user is already authenticated ' 'to this database. You must logout first.') if connect: # Try to authenticate even during failover. member = select_member( self.__rs_state.members, ReadPreference.PRIMARY_PREFERRED) if not member: raise AutoReconnect( "No replica set members available for authentication") sock_info = self.__socket(member) try: # Since __check_auth was called in __socket # there is no need to call it here. auth.authenticate(credentials, sock_info, self.__simple_command) sock_info.authset.add(credentials) finally: member.pool.maybe_return_socket(sock_info) self.__auth_credentials[source] = credentials def _purge_credentials(self, source): """Purge credentials from the database authentication cache. """ if source in self.__auth_credentials: del self.__auth_credentials[source] def __check_auth(self, sock_info): """Authenticate using cached database credentials. """ if self.__auth_credentials or sock_info.authset: cached = set(self.__auth_credentials.itervalues()) authset = sock_info.authset.copy() # Logout any credentials that no longer exist in the cache. for credentials in authset - cached: self.__simple_command(sock_info, credentials[1], {'logout': 1}) sock_info.authset.discard(credentials) for credentials in cached - authset: auth.authenticate(credentials, sock_info, self.__simple_command) sock_info.authset.add(credentials) @property def seeds(self): """The seed list used to connect to this replica set. A sequence of (host, port) pairs. """ return self.__seeds @property def hosts(self): """All active and passive (priority 0) replica set members known to this client. This does not include hidden or slaveDelay members, or arbiters. A sequence of (host, port) pairs. """ return self.__rs_state.hosts @property def primary(self): """The (host, port) of the current primary of the replica set. Returns None if there is no primary. """ return self.__rs_state.writer @property def secondaries(self): """The secondary members known to this client. A sequence of (host, port) pairs. """ return self.__rs_state.secondaries @property def arbiters(self): """The arbiters known to this client. A sequence of (host, port) pairs. """ return self.__rs_state.arbiters @property def is_mongos(self): """If this instance is connected to mongos (always False). .. versionadded:: 2.3 """ return False @property def max_pool_size(self): """The maximum number of sockets the pool will open concurrently. When the pool has reached `max_pool_size`, operations block waiting for a socket to be returned to the pool. If ``waitQueueTimeoutMS`` is set, a blocked operation will raise :exc:`~pymongo.errors.ConnectionFailure` after a timeout. By default ``waitQueueTimeoutMS`` is not set. .. warning:: SIGNIFICANT BEHAVIOR CHANGE in 2.6. Previously, this parameter would limit only the idle sockets the pool would hold onto, not the number of open sockets. The default has also changed to 100. .. versionchanged:: 2.6 """ return self.__max_pool_size @property def use_greenlets(self): """Whether calling :meth:`start_request` assigns greenlet-local, rather than thread-local, sockets. .. versionadded:: 2.4.2 """ return self.__use_greenlets def get_document_class(self): """document_class getter""" return self.__document_class def set_document_class(self, klass): """document_class setter""" self.__document_class = klass document_class = property(get_document_class, set_document_class, doc="""Default class to use for documents returned from this client. """) @property def tz_aware(self): """Does this client return timezone-aware datetimes? """ return self.__tz_aware @property def max_bson_size(self): """Returns the maximum size BSON object the connected primary accepts in bytes. Defaults to 4MB in server < 1.7.4. Returns 0 if no primary is available. """ rs_state = self.__rs_state if rs_state.primary_member: return rs_state.primary_member.max_bson_size return 0 @property def max_message_size(self): """Returns the maximum message size the connected primary accepts in bytes. Returns 0 if no primary is available. """ rs_state = self.__rs_state if rs_state.primary_member: return rs_state.primary_member.max_message_size return 0 @property def auto_start_request(self): """Is auto_start_request enabled? """ return self.__auto_start_request def __simple_command(self, sock_info, dbname, spec): """Send a command to the server. Returns (response, ping_time in seconds). """ rqst_id, msg, _ = message.query(0, dbname + '.$cmd', 0, -1, spec) start = time.time() try: sock_info.sock.sendall(msg) response = self.__recv_msg(1, rqst_id, sock_info) except: sock_info.close() raise end = time.time() response = helpers._unpack_response(response)['data'][0] msg = "command %r failed: %%s" % spec helpers._check_command_response(response, None, msg) return response, end - start def __is_master(self, host): """Directly call ismaster. Returns (response, connection_pool, ping_time in seconds). """ connection_pool = self.pool_class( host, self.__max_pool_size, self.__net_timeout, self.__conn_timeout, self.__use_ssl, wait_queue_timeout=self.__wait_queue_timeout, wait_queue_multiple=self.__wait_queue_multiple, use_greenlets=self.__use_greenlets, ssl_keyfile=self.__ssl_keyfile, ssl_certfile=self.__ssl_certfile, ssl_cert_reqs=self.__ssl_cert_reqs, ssl_ca_certs=self.__ssl_ca_certs) if self.in_request(): connection_pool.start_request() sock_info = connection_pool.get_socket() try: response, ping_time = self.__simple_command( sock_info, 'admin', {'ismaster': 1} ) connection_pool.maybe_return_socket(sock_info) return response, connection_pool, ping_time except (ConnectionFailure, socket.error): connection_pool.discard_socket(sock_info) raise def __schedule_refresh(self, sync=False): """Awake the monitor to update our view of the replica set's state. If `sync` is True, block until the refresh completes. If multiple application threads call __schedule_refresh while refresh is in progress, the work of refreshing the state is only performed once. """ self.__monitor.schedule_refresh() if sync: self.__monitor.wait_for_refresh(timeout_seconds=5) def __make_threadlocal(self): if self.__use_greenlets: return gevent_local() else: return threading.local() def refresh(self): """Iterate through the existing host list, or possibly the seed list, to update the list of hosts and arbiters in this replica set. """ # Only one thread / greenlet calls refresh() at a time: the one # running __init__() or the monitor. We won't modify the state, only # replace it at the end. rs_state = self.__rs_state errors = [] if rs_state.hosts: # Try first those hosts we think are up, then the down ones. nodes = sorted( rs_state.hosts, key=lambda host: rs_state.get(host).up) else: nodes = self.__seeds hosts = set() # This will become the new RSState. members = {} arbiters = set() writer = None # Look for first member from which we can get a list of all members. for node in nodes: member, sock_info = rs_state.get(node), None try: if member: sock_info = self.__socket(member, force=True) response, ping_time = self.__simple_command( sock_info, 'admin', {'ismaster': 1}) member.pool.maybe_return_socket(sock_info) new_member = member.clone_with(response, ping_time) else: response, pool, ping_time = self.__is_master(node) new_member = Member( node, pool, response, MovingAverage([ping_time]), True) # Check that this host is part of the given replica set. set_name = response.get('setName') # The 'setName' field isn't returned by mongod before 1.6.2 # so we can't assume that if it's missing this host isn't in # the specified set. if set_name and set_name != self.__name: host, port = node raise ConfigurationError("%s:%d is not a member of " "replica set %s" % (host, port, self.__name)) if "arbiters" in response: arbiters = set([ _partition_node(h) for h in response["arbiters"]]) if "hosts" in response: hosts.update([_partition_node(h) for h in response["hosts"]]) if "passives" in response: hosts.update([_partition_node(h) for h in response["passives"]]) # Start off the new 'members' dict with this member # but don't add seed list members. if node in hosts: members[node] = new_member if response['ismaster']: writer = node except (ConnectionFailure, socket.error), why: if member: member.pool.discard_socket(sock_info) errors.append("%s:%d: %s" % (node[0], node[1], str(why))) if hosts: break else: if errors: raise AutoReconnect(', '.join(errors)) raise ConfigurationError('No suitable hosts found') # Ensure we have a pool for each member, and find the primary. for host in hosts: if host in members: # This member was the first we connected to, in the loop above. continue member, sock_info = rs_state.get(host), None try: if member: sock_info = self.__socket(member, force=True) res, ping_time = self.__simple_command( sock_info, 'admin', {'ismaster': 1}) member.pool.maybe_return_socket(sock_info) new_member = member.clone_with(res, ping_time) else: res, connection_pool, ping_time = self.__is_master(host) new_member = Member( host, connection_pool, res, MovingAverage([ping_time]), True) members[host] = new_member except (ConnectionFailure, socket.error): if member: member.pool.discard_socket(sock_info) continue if res['ismaster']: writer = host if writer == rs_state.writer: threadlocal = self.__rs_state.threadlocal else: # We unpin threads from members if the primary has changed, since # no monotonic consistency can be promised now anyway. threadlocal = self.__make_threadlocal() # Replace old state with new. self.__rs_state = RSState(threadlocal, members, arbiters, writer) def __find_primary(self): """Returns a connection to the primary of this replica set, if one exists, or raises AutoReconnect. """ primary = self.__rs_state.primary_member if primary: return primary # We had a failover. self.__schedule_refresh(sync=True) # Try again. This time copy the RSState reference so we're guaranteed # primary_member and error_message are from the same state. rs_state = self.__rs_state if rs_state.primary_member: return rs_state.primary_member # Couldn't find the primary. raise AutoReconnect(rs_state.error_message) def __socket(self, member, force=False): """Get a SocketInfo from the pool. """ if self.auto_start_request and not self.in_request(): self.start_request() sock_info = member.pool.get_socket(force=force) try: self.__check_auth(sock_info) except OperationFailure: member.pool.maybe_return_socket(sock_info) raise return sock_info def _ensure_connected(self, sync=False): """Ensure this client instance is connected to a primary. """ # This may be the first time we're connecting to the set. if self.__monitor and not self.__monitor.started: try: self.__monitor.start() # Minor race condition. It's possible that two (or more) # threads could call monitor.start() consecutively. Just pass. except RuntimeError: pass if sync: rs_state = self.__rs_state if not rs_state.primary_member: self.__schedule_refresh(sync) def disconnect(self): """Disconnect from the replica set primary, unpin all members, and refresh our view of the replica set. """ rs_state = self.__rs_state if rs_state.primary_member: rs_state.primary_member.pool.reset() threadlocal = self.__make_threadlocal() self.__rs_state = rs_state.clone_without_writer(threadlocal) self.__schedule_refresh() def close(self): """Close this client instance. This method first terminates the replica set monitor, then disconnects from all members of the replica set. .. warning:: This method stops the replica set monitor task. The replica set monitor is required to properly handle replica set configuration changes, including a failure of the primary. Once :meth:`~close` is called this client instance must not be reused. .. versionchanged:: 2.2.1 The :meth:`close` method now terminates the replica set monitor. """ if self.__monitor: self.__monitor.shutdown() # Use a reasonable timeout. self.__monitor.join(1.0) self.__monitor = None self.__rs_state = RSState(self.__make_threadlocal()) def alive(self): """Return ``False`` if there has been an error communicating with the primary, else ``True``. This method attempts to check the status of the primary with minimal I/O. The current thread / greenlet retrieves a socket (its request socket if it's in a request, or a random idle socket if it's not in a request) from the primary's connection pool and checks whether calling select_ on it raises an error. If there are currently no idle sockets, or if there is no known primary, :meth:`alive` will attempt to actually find and connect to the primary. A more certain way to determine primary availability is to ping it:: client.admin.command('ping') .. _select: http://docs.python.org/2/library/select.html#select.select """ # In the common case, a socket is available and was used recently, so # calling select() on it is a reasonable attempt to see if the OS has # reported an error. Note this can be wasteful: __socket implicitly # calls select() if the socket hasn't been checked in the last second, # or it may create a new socket, in which case calling select() is # redundant. member, sock_info = None, None try: try: member = self.__find_primary() sock_info = self.__socket(member) return not pool._closed(sock_info.sock) except (socket.error, ConnectionFailure): return False finally: if member and sock_info: member.pool.maybe_return_socket(sock_info) def __check_response_to_last_error(self, response): """Check a response to a lastError message for errors. `response` is a byte string representing a response to the message. If it represents an error response we raise OperationFailure. Return the response as a document. """ response = helpers._unpack_response(response) assert response["number_returned"] == 1 error = response["data"][0] helpers._check_command_response(error, self.disconnect) error_msg = error.get("err", "") if error_msg is None: return error if error_msg.startswith("not master"): self.disconnect() raise AutoReconnect(error_msg) if "code" in error: if error["code"] in (11000, 11001, 12582): raise DuplicateKeyError(error["err"], error["code"]) else: raise OperationFailure(error["err"], error["code"]) else: raise OperationFailure(error["err"]) def __recv_data(self, length, sock_info): """Lowest level receive operation. Takes length to receive and repeatedly calls recv until able to return a buffer of that length, raising ConnectionFailure on error. """ message = EMPTY while length: chunk = sock_info.sock.recv(length) if chunk == EMPTY: raise ConnectionFailure("connection closed") length -= len(chunk) message += chunk return message def __recv_msg(self, operation, rqst_id, sock): """Receive a message in response to `rqst_id` on `sock`. Returns the response data with the header removed. """ header = self.__recv_data(16, sock) length = struct.unpack(" max_size: raise InvalidDocument("BSON document too large (%d bytes)" " - the connected server supports" " BSON document sizes up to %d" " bytes." % (max_doc_size, max_size)) return (request_id, data) # get_more and kill_cursors messages # don't include BSON documents. return msg def _send_message(self, msg, with_last_error=False, _connection_to_use=None): """Say something to Mongo. Raises ConnectionFailure if the message cannot be sent. Raises OperationFailure if `with_last_error` is ``True`` and the response to the getLastError call returns an error. Return the response from lastError, or ``None`` if `with_last_error` is ``False``. :Parameters: - `msg`: message to send - `with_last_error`: check getLastError status after sending the message """ self._ensure_connected() if _connection_to_use in (None, -1): member = self.__find_primary() else: member = self.__rs_state.get(_connection_to_use) sock_info = None try: try: sock_info = self.__socket(member) rqst_id, data = self.__check_bson_size( msg, member.max_bson_size) sock_info.sock.sendall(data) # Safe mode. We pack the message together with a lastError # message and send both. We then get the response (to the # lastError) and raise OperationFailure if it is an error # response. rv = None if with_last_error: response = self.__recv_msg(1, rqst_id, sock_info) rv = self.__check_response_to_last_error(response) return rv except OperationFailure: raise except(ConnectionFailure, socket.error), why: member.pool.discard_socket(sock_info) if _connection_to_use in (None, -1): self.disconnect() raise AutoReconnect(str(why)) except: sock_info.close() raise finally: member.pool.maybe_return_socket(sock_info) def __send_and_receive(self, member, msg, **kwargs): """Send a message on the given socket and return the response data. Can raise socket.error. """ sock_info = None exhaust = kwargs.get('exhaust') rqst_id, data = self.__check_bson_size(msg, member.max_bson_size) try: sock_info = self.__socket(member) if not exhaust and "network_timeout" in kwargs: sock_info.sock.settimeout(kwargs['network_timeout']) sock_info.sock.sendall(data) response = self.__recv_msg(1, rqst_id, sock_info) if not exhaust: if "network_timeout" in kwargs: sock_info.sock.settimeout(self.__net_timeout) member.pool.maybe_return_socket(sock_info) return response, sock_info, member.pool except: if sock_info is not None: sock_info.close() member.pool.maybe_return_socket(sock_info) raise def __try_read(self, member, msg, **kwargs): """Attempt a read from a member; on failure mark the member "down" and wake up the monitor thread to refresh as soon as possible. """ try: return self.__send_and_receive(member, msg, **kwargs) except socket.timeout, e: # Could be one slow query, don't refresh. host, port = member.host raise AutoReconnect("%s:%d: %s" % (host, port, e)) except (socket.error, ConnectionFailure), why: # Try to replace our RSState with a clone where this member is # marked "down", to reduce exceptions on other threads, or repeated # exceptions on this thread. We accept that there's a race # condition (another thread could be replacing our state with a # different version concurrently) but this approach is simple and # lock-free. self.__rs_state = self.__rs_state.clone_with_host_down( member.host, str(why)) self.__schedule_refresh() host, port = member.host raise AutoReconnect("%s:%d: %s" % (host, port, why)) def _send_message_with_response(self, msg, _connection_to_use=None, _must_use_master=False, **kwargs): """Send a message to Mongo and return the response. Sends the given message and returns (host used, response). :Parameters: - `msg`: (request_id, data) pair making up the message to send - `_connection_to_use`: Optional (host, port) of member for message, used by Cursor for getMore and killCursors messages. - `_must_use_master`: If True, send to primary. """ self._ensure_connected() rs_state = self.__rs_state tag_sets = kwargs.get('tag_sets', [{}]) mode = kwargs.get('read_preference', ReadPreference.PRIMARY) if _must_use_master: mode = ReadPreference.PRIMARY tag_sets = [{}] if not rs_state.primary_member: # Primary was down last we checked. Start a refresh if one is not # already in progress. If caller requested the primary, wait to # see if it's up, otherwise continue with known-good members. sync = (mode == ReadPreference.PRIMARY) self.__schedule_refresh(sync=sync) rs_state = self.__rs_state latency = kwargs.get( 'secondary_acceptable_latency_ms', self.secondary_acceptable_latency_ms) try: if _connection_to_use is not None: if _connection_to_use == -1: member = rs_state.primary_member error_message = rs_state.error_message else: member = rs_state.get(_connection_to_use) error_message = '%s:%s not available' % _connection_to_use if not member: raise AutoReconnect(error_message) return member.pool.pair, self.__try_read( member, msg, **kwargs) except AutoReconnect: if _connection_to_use in (-1, rs_state.writer): # Primary's down. Refresh. self.disconnect() raise # To provide some monotonic consistency, we use the same member as # long as this thread is in a request and all reads use the same # mode, tags, and latency. The member gets unpinned if pref changes, # if member changes state, if we detect a failover, or if this thread # calls end_request(). errors = [] pinned_host = rs_state.pinned_host pinned_member = rs_state.get(pinned_host) if (pinned_member and pinned_member.matches_mode(mode) and pinned_member.matches_tag_sets(tag_sets) # TODO: REMOVE? and rs_state.keep_pinned_host(mode, tag_sets, latency)): try: return ( pinned_member.host, self.__try_read(pinned_member, msg, **kwargs)) except AutoReconnect, why: if _must_use_master or mode == ReadPreference.PRIMARY: self.disconnect() raise else: errors.append(str(why)) # No pinned member, or pinned member down or doesn't match read pref rs_state.unpin_host() members = list(rs_state.members) while len(errors) < MAX_RETRY: member = select_member( members=members, mode=mode, tag_sets=tag_sets, latency=latency) if not member: # Ran out of members to try break try: # Sets member.up False on failure, so select_member won't try # it again. response = self.__try_read(member, msg, **kwargs) # Success if self.in_request(): # Keep reading from this member in this thread / greenlet # unless read preference changes rs_state.pin_host(member.host, mode, tag_sets, latency) return member.host, response except AutoReconnect, why: errors.append(str(why)) members.remove(member) # Ran out of tries if mode == ReadPreference.PRIMARY: msg = "No replica set primary available for query" elif mode == ReadPreference.SECONDARY: msg = "No replica set secondary available for query" else: msg = "No replica set members available for query" msg += " with ReadPreference %s" % modes[mode] if tag_sets != [{}]: msg += " and tags " + repr(tag_sets) raise AutoReconnect(msg, errors) def _exhaust_next(self, sock_info): """Used with exhaust cursors to get the next batch off the socket. """ return self.__recv_msg(1, None, sock_info) def start_request(self): """Ensure the current thread or greenlet always uses the same socket until it calls :meth:`end_request`. For :class:`~pymongo.read_preferences.ReadPreference` PRIMARY, auto_start_request=True ensures consistent reads, even if you read after an unacknowledged write. For read preferences other than PRIMARY, there are no consistency guarantees. In Python 2.6 and above, or in Python 2.5 with "from __future__ import with_statement", :meth:`start_request` can be used as a context manager: >>> client = pymongo.MongoReplicaSetClient() >>> db = client.test >>> _id = db.test_collection.insert({}) >>> with client.start_request(): ... for i in range(100): ... db.test_collection.update({'_id': _id}, {'$set': {'i':i}}) ... ... # Definitely read the document after the final update completes ... print db.test_collection.find({'_id': _id}) .. versionadded:: 2.2 The :class:`~pymongo.pool.Request` return value. :meth:`start_request` previously returned None """ # We increment our request counter's thread- or greenlet-local value # for every call to start_request; however, we only call each pool's # start_request once to start a request, and call each pool's # end_request once to end it. We don't let pools' request counters # exceed 1. This keeps things sane when we create and delete pools # within a request. if 1 == self.__request_counter.inc(): for member in self.__rs_state.members: member.pool.start_request() return pool.Request(self) def in_request(self): """True if :meth:`start_request` has been called, but not :meth:`end_request`, or if `auto_start_request` is True and :meth:`end_request` has not been called in this thread or greenlet. """ return bool(self.__request_counter.get()) def end_request(self): """Undo :meth:`start_request` and allow this thread's connections to replica set members to return to the pool. Calling :meth:`end_request` allows the :class:`~socket.socket` that has been reserved for this thread by :meth:`start_request` to be returned to the pool. Other threads will then be able to re-use that :class:`~socket.socket`. If your application uses many threads, or has long-running threads that infrequently perform MongoDB operations, then judicious use of this method can lead to performance gains. Care should be taken, however, to make sure that :meth:`end_request` is not called in the middle of a sequence of operations in which ordering is important. This could lead to unexpected results. """ rs_state = self.__rs_state if 0 == self.__request_counter.dec(): for member in rs_state.members: # No effect if not in a request member.pool.end_request() rs_state.unpin_host() def __eq__(self, other): # XXX: Implement this? return NotImplemented def __ne__(self, other): return NotImplemented def __repr__(self): return "MongoReplicaSetClient(%r)" % (["%s:%d" % n for n in self.hosts],) def __getattr__(self, name): """Get a database by name. Raises :class:`~pymongo.errors.InvalidName` if an invalid database name is used. :Parameters: - `name`: the name of the database to get """ return database.Database(self, name) def __getitem__(self, name): """Get a database by name. Raises :class:`~pymongo.errors.InvalidName` if an invalid database name is used. :Parameters: - `name`: the name of the database to get """ return self.__getattr__(name) def close_cursor(self, cursor_id, _conn_id): """Close a single database cursor. Raises :class:`TypeError` if `cursor_id` is not an instance of ``(int, long)``. What closing the cursor actually means depends on this client's cursor manager. :Parameters: - `cursor_id`: id of cursor to close """ if not isinstance(cursor_id, (int, long)): raise TypeError("cursor_id must be an instance of (int, long)") self._send_message(message.kill_cursors([cursor_id]), _connection_to_use=_conn_id) def server_info(self): """Get information about the MongoDB primary we're connected to. """ return self.admin.command("buildinfo") def database_names(self): """Get a list of the names of all databases on the connected server. """ return [db["name"] for db in self.admin.command("listDatabases")["databases"]] def drop_database(self, name_or_database): """Drop a database. Raises :class:`TypeError` if `name_or_database` is not an instance of :class:`basestring` (:class:`str` in python 3) or Database :Parameters: - `name_or_database`: the name of a database to drop, or a :class:`~pymongo.database.Database` instance representing the database to drop """ name = name_or_database if isinstance(name, database.Database): name = name.name if not isinstance(name, basestring): raise TypeError("name_or_database must be an instance of " "%s or Database" % (basestring.__name__,)) self._purge_index(name) self[name].command("dropDatabase") def copy_database(self, from_name, to_name, from_host=None, username=None, password=None): """Copy a database, potentially from another host. Raises :class:`TypeError` if `from_name` or `to_name` is not an instance of :class:`basestring` (:class:`str` in python 3). Raises :class:`~pymongo.errors.InvalidName` if `to_name` is not a valid database name. If `from_host` is ``None`` the current host is used as the source. Otherwise the database is copied from `from_host`. If the source database requires authentication, `username` and `password` must be specified. :Parameters: - `from_name`: the name of the source database - `to_name`: the name of the target database - `from_host` (optional): host name to copy from - `username` (optional): username for source database - `password` (optional): password for source database .. note:: Specifying `username` and `password` requires server version **>= 1.3.3+**. """ if not isinstance(from_name, basestring): raise TypeError("from_name must be an instance " "of %s" % (basestring.__name__,)) if not isinstance(to_name, basestring): raise TypeError("to_name must be an instance " "of %s" % (basestring.__name__,)) database._check_name(to_name) command = {"fromdb": from_name, "todb": to_name} if from_host is not None: command["fromhost"] = from_host try: self.start_request() if username is not None: nonce = self.admin.command("copydbgetnonce", fromhost=from_host)["nonce"] command["username"] = username command["nonce"] = nonce command["key"] = auth._auth_key(nonce, username, password) return self.admin.command("copydb", **command) finally: self.end_request() def get_default_database(self): """Get the database named in the MongoDB connection URI. >>> uri = 'mongodb://host/my_database' >>> client = MongoReplicaSetClient(uri) >>> db = client.get_default_database() >>> assert db.name == 'my_database' Useful in scripts where you want to choose which database to use based only on the URI in a configuration file. """ if self.__default_database_name is None: raise ConfigurationError('No default database defined') return self[self.__default_database_name] pymongo-2.6.3/pymongo/pool.py000066400000000000000000000506111223300253600162240ustar00rootroot00000000000000# Copyright 2011-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you # may not use this file except in compliance with the License. You # may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. See the License for the specific language governing # permissions and limitations under the License. import os import socket import sys import time import threading import weakref from pymongo import thread_util from pymongo.common import HAS_SSL from pymongo.errors import ConnectionFailure, ConfigurationError try: from ssl import match_hostname except ImportError: from pymongo.ssl_match_hostname import match_hostname if HAS_SSL: import ssl if sys.platform.startswith('java'): from select import cpython_compatible_select as select else: from select import select NO_REQUEST = None NO_SOCKET_YET = -1 def _closed(sock): """Return True if we know socket has been closed, False otherwise. """ try: rd, _, _ = select([sock], [], [], 0) # Any exception here is equally bad (select.error, ValueError, etc.). except: return True return len(rd) > 0 class SocketInfo(object): """Store a socket with some metadata """ def __init__(self, sock, pool_id, host=None): self.sock = sock self.host = host self.authset = set() self.closed = False self.last_checkout = time.time() self.forced = False # The pool's pool_id changes with each reset() so we can close sockets # created before the last reset. self.pool_id = pool_id def close(self): self.closed = True # Avoid exceptions on interpreter shutdown. try: self.sock.close() except: pass def __eq__(self, other): # Need to check if other is NO_REQUEST or NO_SOCKET_YET, and then check # if its sock is the same as ours return hasattr(other, 'sock') and self.sock == other.sock def __ne__(self, other): return not self == other def __hash__(self): return hash(self.sock) def __repr__(self): return "SocketInfo(%s)%s at %s" % ( repr(self.sock), self.closed and " CLOSED" or "", id(self) ) # Do *not* explicitly inherit from object or Jython won't call __del__ # http://bugs.jython.org/issue1057 class Pool: def __init__(self, pair, max_size, net_timeout, conn_timeout, use_ssl, use_greenlets, ssl_keyfile=None, ssl_certfile=None, ssl_cert_reqs=None, ssl_ca_certs=None, wait_queue_timeout=None, wait_queue_multiple=None): """ :Parameters: - `pair`: a (hostname, port) tuple - `max_size`: The maximum number of open sockets. Calls to `get_socket` will block if this is set, this pool has opened `max_size` sockets, and there are none idle. Set to `None` to disable. - `net_timeout`: timeout in seconds for operations on open connection - `conn_timeout`: timeout in seconds for establishing connection - `use_ssl`: bool, if True use an encrypted connection - `use_greenlets`: bool, if True then start_request() assigns a socket to the current greenlet - otherwise it is assigned to the current thread - `ssl_keyfile`: The private keyfile used to identify the local connection against mongod. If included with the ``certfile` then only the ``ssl_certfile`` is needed. Implies ``ssl=True``. - `ssl_certfile`: The certificate file used to identify the local connection against mongod. Implies ``ssl=True``. - `ssl_cert_reqs`: Specifies whether a certificate is required from the other side of the connection, and whether it will be validated if provided. It must be one of the three values ``ssl.CERT_NONE`` (certificates ignored), ``ssl.CERT_OPTIONAL`` (not required, but validated if provided), or ``ssl.CERT_REQUIRED`` (required and validated). If the value of this parameter is not ``ssl.CERT_NONE``, then the ``ssl_ca_certs`` parameter must point to a file of CA certificates. Implies ``ssl=True``. - `ssl_ca_certs`: The ca_certs file contains a set of concatenated "certification authority" certificates, which are used to validate certificates passed from the other end of the connection. Implies ``ssl=True``. - `wait_queue_timeout`: (integer) How long (in seconds) a thread will wait for a socket from the pool if the pool has no free sockets. - `wait_queue_multiple`: (integer) Multiplied by max_pool_size to give the number of threads allowed to wait for a socket at one time. """ # Only check a socket's health with _closed() every once in a while. # Can override for testing: 0 to always check, None to never check. self._check_interval_seconds = 1 self.sockets = set() self.lock = threading.Lock() # Keep track of resets, so we notice sockets created before the most # recent reset and close them. self.pool_id = 0 self.pid = os.getpid() self.pair = pair self.max_size = max_size self.net_timeout = net_timeout self.conn_timeout = conn_timeout self.wait_queue_timeout = wait_queue_timeout self.wait_queue_multiple = wait_queue_multiple self.use_ssl = use_ssl self.ssl_keyfile = ssl_keyfile self.ssl_certfile = ssl_certfile self.ssl_cert_reqs = ssl_cert_reqs self.ssl_ca_certs = ssl_ca_certs if HAS_SSL and use_ssl and not ssl_cert_reqs: self.ssl_cert_reqs = ssl.CERT_NONE # Map self._ident.get() -> request socket self._tid_to_sock = {} if use_greenlets and not thread_util.have_gevent: raise ConfigurationError( "The Gevent module is not available. " "Install the gevent package from PyPI." ) self._ident = thread_util.create_ident(use_greenlets) # Count the number of calls to start_request() per thread or greenlet self._request_counter = thread_util.Counter(use_greenlets) if self.wait_queue_multiple is None or self.max_size is None: max_waiters = None else: max_waiters = self.max_size * self.wait_queue_multiple self._socket_semaphore = thread_util.create_semaphore( self.max_size, max_waiters, use_greenlets) def reset(self): # Ignore this race condition -- if many threads are resetting at once, # the pool_id will definitely change, which is all we care about. self.pool_id += 1 self.pid = os.getpid() sockets = None try: # Swapping variables is not atomic. We need to ensure no other # thread is modifying self.sockets, or replacing it, in this # critical section. self.lock.acquire() sockets, self.sockets = self.sockets, set() finally: self.lock.release() for sock_info in sockets: sock_info.close() def create_connection(self, pair): """Connect to *pair* and return the socket object. This is a modified version of create_connection from CPython >=2.6. """ host, port = pair or self.pair # Check if dealing with a unix domain socket if host.endswith('.sock'): if not hasattr(socket, "AF_UNIX"): raise ConnectionFailure("UNIX-sockets are not supported " "on this system") sock = socket.socket(socket.AF_UNIX) try: sock.connect(host) return sock except socket.error, e: if sock is not None: sock.close() raise e # Don't try IPv6 if we don't support it. Also skip it if host # is 'localhost' (::1 is fine). Avoids slow connect issues # like PYTHON-356. family = socket.AF_INET if socket.has_ipv6 and host != 'localhost': family = socket.AF_UNSPEC err = None for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): af, socktype, proto, dummy, sa = res sock = None try: sock = socket.socket(af, socktype, proto) sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) sock.settimeout(self.conn_timeout or 20.0) sock.connect(sa) return sock except socket.error, e: err = e if sock is not None: sock.close() if err is not None: raise err else: # This likely means we tried to connect to an IPv6 only # host with an OS/kernel or Python interpreter that doesn't # support IPv6. The test case is Jython2.5.1 which doesn't # support IPv6 at all. raise socket.error('getaddrinfo failed') def connect(self, pair): """Connect to Mongo and return a new (connected) socket. Note that the pool does not keep a reference to the socket -- you must call return_socket() when you're done with it. """ sock = self.create_connection(pair) hostname = (pair or self.pair)[0] if self.use_ssl: try: sock = ssl.wrap_socket(sock, certfile=self.ssl_certfile, keyfile=self.ssl_keyfile, ca_certs=self.ssl_ca_certs, cert_reqs=self.ssl_cert_reqs) if self.ssl_cert_reqs: match_hostname(sock.getpeercert(), hostname) except ssl.SSLError: sock.close() raise ConnectionFailure("SSL handshake failed. MongoDB may " "not be configured with SSL support.") sock.settimeout(self.net_timeout) return SocketInfo(sock, self.pool_id, hostname) def get_socket(self, pair=None, force=False): """Get a socket from the pool. Returns a :class:`SocketInfo` object wrapping a connected :class:`socket.socket`, and a bool saying whether the socket was from the pool or freshly created. :Parameters: - `pair`: optional (hostname, port) tuple - `force`: optional boolean, forces a connection to be returned without blocking, even if `max_size` has been reached. """ # We use the pid here to avoid issues with fork / multiprocessing. # See test.test_client:TestClient.test_fork for an example of # what could go wrong otherwise if self.pid != os.getpid(): self.reset() # Have we opened a socket for this request? req_state = self._get_request_state() if req_state not in (NO_SOCKET_YET, NO_REQUEST): # There's a socket for this request, check it and return it checked_sock = self._check(req_state, pair) if checked_sock != req_state: self._set_request_state(checked_sock) checked_sock.last_checkout = time.time() return checked_sock forced = False # We're not in a request, just get any free socket or create one if force: # If we're doing an internal operation, attempt to play nicely with # max_size, but if there is no open "slot" force the connection # and mark it as forced so we don't release the semaphore without # having acquired it for this socket. if not self._socket_semaphore.acquire(False): forced = True elif not self._socket_semaphore.acquire(True, self.wait_queue_timeout): self._raise_wait_queue_timeout() # We've now acquired the semaphore and must release it on error. try: sock_info, from_pool = None, None try: try: # set.pop() isn't atomic in Jython less than 2.7, see # http://bugs.jython.org/issue1854 self.lock.acquire() sock_info, from_pool = self.sockets.pop(), True finally: self.lock.release() except KeyError: sock_info, from_pool = self.connect(pair), False if from_pool: sock_info = self._check(sock_info, pair) sock_info.forced = forced if req_state == NO_SOCKET_YET: # start_request has been called but we haven't assigned a # socket to the request yet. Let's use this socket for this # request until end_request. self._set_request_state(sock_info) except: if not forced: self._socket_semaphore.release() raise sock_info.last_checkout = time.time() return sock_info def start_request(self): if self._get_request_state() == NO_REQUEST: # Add a placeholder value so we know we're in a request, but we # have no socket assigned to the request yet. self._set_request_state(NO_SOCKET_YET) self._request_counter.inc() def in_request(self): return bool(self._request_counter.get()) def end_request(self): # Check if start_request has ever been called in this thread / greenlet count = self._request_counter.get() if count: self._request_counter.dec() if count == 1: # End request sock_info = self._get_request_state() self._set_request_state(NO_REQUEST) if sock_info not in (NO_REQUEST, NO_SOCKET_YET): self._return_socket(sock_info) def discard_socket(self, sock_info): """Close and discard the active socket. """ if sock_info not in (NO_REQUEST, NO_SOCKET_YET): sock_info.close() if sock_info == self._get_request_state(): # Discarding request socket; prepare to use a new request # socket on next get_socket(). self._set_request_state(NO_SOCKET_YET) def maybe_return_socket(self, sock_info): """Return the socket to the pool unless it's the request socket. """ # These sentinel values should only be used internally. assert sock_info not in (NO_REQUEST, NO_SOCKET_YET) if self.pid != os.getpid(): if not sock_info.forced: self._socket_semaphore.release() self.reset() else: if sock_info.closed: if sock_info.forced: sock_info.forced = False elif sock_info != self._get_request_state(): self._socket_semaphore.release() return if sock_info != self._get_request_state(): self._return_socket(sock_info) def _return_socket(self, sock_info): """Return socket to the pool. If pool is full the socket is discarded. """ try: self.lock.acquire() too_many_sockets = (self.max_size is not None and len(self.sockets) >= self.max_size) if not too_many_sockets and sock_info.pool_id == self.pool_id: self.sockets.add(sock_info) else: sock_info.close() finally: self.lock.release() if sock_info.forced: sock_info.forced = False else: self._socket_semaphore.release() def _check(self, sock_info, pair): """This side-effecty function checks if this pool has been reset since the last time this socket was used, or if the socket has been closed by some external network error, and if so, attempts to create a new socket. If this connection attempt fails we reset the pool and reraise the error. Checking sockets lets us avoid seeing *some* :class:`~pymongo.errors.AutoReconnect` exceptions on server hiccups, etc. We only do this if it's been > 1 second since the last socket checkout, to keep performance reasonable - we can't avoid AutoReconnects completely anyway. """ error = False # How long since socket was last checked out. age = time.time() - sock_info.last_checkout if sock_info.closed: error = True elif self.pool_id != sock_info.pool_id: sock_info.close() error = True elif (self._check_interval_seconds is not None and ( 0 == self._check_interval_seconds or age > self._check_interval_seconds)): if _closed(sock_info.sock): sock_info.close() error = True if not error: return sock_info else: try: return self.connect(pair) except socket.error: self.reset() raise def _set_request_state(self, sock_info): ident = self._ident tid = ident.get() if sock_info == NO_REQUEST: # Ending a request ident.unwatch(tid) self._tid_to_sock.pop(tid, None) else: self._tid_to_sock[tid] = sock_info if not ident.watching(): # Closure over tid, poolref, and ident. Don't refer directly to # self, otherwise there's a cycle. # Do not access threadlocals in this function, or any # function it calls! In the case of the Pool subclass and # mod_wsgi 2.x, on_thread_died() is triggered when mod_wsgi # calls PyThreadState_Clear(), which deferences the # ThreadVigil and triggers the weakref callback. Accessing # thread locals in this function, while PyThreadState_Clear() # is in progress can cause leaks, see PYTHON-353. poolref = weakref.ref(self) def on_thread_died(ref): try: ident.unwatch(tid) pool = poolref() if pool: # End the request request_sock = pool._tid_to_sock.pop(tid, None) # Was thread ever assigned a socket before it died? if request_sock not in (NO_REQUEST, NO_SOCKET_YET): pool._return_socket(request_sock) except: # Random exceptions on interpreter shutdown. pass ident.watch(on_thread_died) def _get_request_state(self): tid = self._ident.get() return self._tid_to_sock.get(tid, NO_REQUEST) def _raise_wait_queue_timeout(self): raise ConnectionFailure( 'Timed out waiting for socket from pool with max_size %r and' ' wait_queue_timeout %r' % ( self.max_size, self.wait_queue_timeout)) def __del__(self): # Avoid ResourceWarnings in Python 3 for sock_info in self.sockets: sock_info.close() for request_sock in self._tid_to_sock.values(): if request_sock not in (NO_REQUEST, NO_SOCKET_YET): request_sock.close() class Request(object): """ A context manager returned by :meth:`start_request`, so you can do `with client.start_request(): do_something()` in Python 2.5+. """ def __init__(self, connection): self.connection = connection def end(self): self.connection.end_request() def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): self.end() # Returning False means, "Don't suppress exceptions if any were # thrown within the block" return False pymongo-2.6.3/pymongo/read_preferences.py000066400000000000000000000146461223300253600205570ustar00rootroot00000000000000# Copyright 2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License", # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Utilities for choosing which member of a replica set to read from.""" import random from pymongo.errors import ConfigurationError class ReadPreference: """An enum that defines the read preference modes supported by PyMongo. Used in three cases: :class:`~pymongo.mongo_client.MongoClient` connected to a single host: * `PRIMARY`: Queries are allowed if the host is standalone or the replica set primary. * All other modes allow queries to standalone servers, to the primary, or to secondaries. :class:`~pymongo.mongo_client.MongoClient` connected to a mongos, with a sharded cluster of replica sets: * `PRIMARY`: Queries are sent to the primary of a shard. * `PRIMARY_PREFERRED`: Queries are sent to the primary if available, otherwise a secondary. * `SECONDARY`: Queries are distributed among shard secondaries. An error is raised if no secondaries are available. * `SECONDARY_PREFERRED`: Queries are distributed among shard secondaries, or the primary if no secondary is available. * `NEAREST`: Queries are distributed among all members of a shard. :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient`: * `PRIMARY`: Queries are sent to the primary of the replica set. * `PRIMARY_PREFERRED`: Queries are sent to the primary if available, otherwise a secondary. * `SECONDARY`: Queries are distributed among secondaries. An error is raised if no secondaries are available. * `SECONDARY_PREFERRED`: Queries are distributed among secondaries, or the primary if no secondary is available. * `NEAREST`: Queries are distributed among all members. """ PRIMARY = 0 PRIMARY_PREFERRED = 1 SECONDARY = 2 SECONDARY_ONLY = 2 SECONDARY_PREFERRED = 3 NEAREST = 4 # For formatting error messages modes = { ReadPreference.PRIMARY: 'PRIMARY', ReadPreference.PRIMARY_PREFERRED: 'PRIMARY_PREFERRED', ReadPreference.SECONDARY: 'SECONDARY', ReadPreference.SECONDARY_PREFERRED: 'SECONDARY_PREFERRED', ReadPreference.NEAREST: 'NEAREST', } _mongos_modes = [ 'primary', 'primaryPreferred', 'secondary', 'secondaryPreferred', 'nearest', ] def mongos_mode(mode): return _mongos_modes[mode] def mongos_enum(enum): return _mongos_modes.index(enum) def select_primary(members): for member in members: if member.is_primary: if member.up: return member else: return None return None def select_member_with_tags(members, tags, secondary_only, latency): candidates = [] for candidate in members: if not candidate.up: continue if secondary_only and candidate.is_primary: continue if not (candidate.is_primary or candidate.is_secondary): # In RECOVERING or similar state continue if candidate.matches_tags(tags): candidates.append(candidate) if not candidates: return None # ping_time is in seconds fastest = min([candidate.get_avg_ping_time() for candidate in candidates]) near_candidates = [ candidate for candidate in candidates if candidate.get_avg_ping_time() - fastest < latency / 1000.] return random.choice(near_candidates) def select_member( members, mode=ReadPreference.PRIMARY, tag_sets=None, latency=15 ): """Return a Member or None. """ if tag_sets is None: tag_sets = [{}] # For brevity PRIMARY = ReadPreference.PRIMARY PRIMARY_PREFERRED = ReadPreference.PRIMARY_PREFERRED SECONDARY = ReadPreference.SECONDARY SECONDARY_PREFERRED = ReadPreference.SECONDARY_PREFERRED NEAREST = ReadPreference.NEAREST if mode == PRIMARY: if tag_sets != [{}]: raise ConfigurationError("PRIMARY cannot be combined with tags") return select_primary(members) elif mode == PRIMARY_PREFERRED: # Recurse. candidate_primary = select_member(members, PRIMARY, [{}], latency) if candidate_primary: return candidate_primary else: return select_member(members, SECONDARY, tag_sets, latency) elif mode == SECONDARY: for tags in tag_sets: candidate = select_member_with_tags(members, tags, True, latency) if candidate: return candidate return None elif mode == SECONDARY_PREFERRED: # Recurse. candidate_secondary = select_member( members, SECONDARY, tag_sets, latency) if candidate_secondary: return candidate_secondary else: return select_member(members, PRIMARY, [{}], latency) elif mode == NEAREST: for tags in tag_sets: candidate = select_member_with_tags(members, tags, False, latency) if candidate: return candidate # Ran out of tags. return None else: raise ConfigurationError("Invalid mode %s" % repr(mode)) """Commands that may be sent to replica-set secondaries, depending on ReadPreference and tags. All other commands are always run on the primary. """ secondary_ok_commands = frozenset([ "group", "aggregate", "collstats", "dbstats", "count", "distinct", "geonear", "geosearch", "geowalk", "mapreduce", "getnonce", "authenticate", "text", ]) class MovingAverage(object): def __init__(self, samples): """Immutable structure to track a 5-sample moving average. """ self.samples = samples[-5:] assert self.samples self.average = sum(self.samples) / float(len(self.samples)) def clone_with(self, sample): """Get a copy of this instance plus a new sample""" return MovingAverage(self.samples + [sample]) def get(self): return self.average pymongo-2.6.3/pymongo/replica_set_connection.py000066400000000000000000000271031223300253600217640ustar00rootroot00000000000000# Copyright 2011-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you # may not use this file except in compliance with the License. You # may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. See the License for the specific language governing # permissions and limitations under the License. """Tools for connecting to a MongoDB replica set. .. warning:: **DEPRECATED:** Please use :mod:`~pymongo.mongo_replica_set_client` instead. .. seealso:: :doc:`/examples/high_availability` for more examples of how to connect to a replica set. To get a :class:`~pymongo.database.Database` instance from a :class:`ReplicaSetConnection` use either dictionary-style or attribute-style access: .. doctest:: >>> from pymongo import ReplicaSetConnection >>> c = ReplicaSetConnection('localhost:27017', replicaSet='repl0') >>> c.test_database Database(ReplicaSetConnection([u'...', u'...']), u'test_database') >>> c['test_database'] Database(ReplicaSetConnection([u'...', u'...']), u'test_database') """ from pymongo.mongo_replica_set_client import MongoReplicaSetClient from pymongo.errors import ConfigurationError class ReplicaSetConnection(MongoReplicaSetClient): """Connection to a MongoDB replica set. """ def __init__(self, hosts_or_uri=None, max_pool_size=None, document_class=dict, tz_aware=False, **kwargs): """Create a new connection to a MongoDB replica set. .. warning:: **DEPRECATED:** :class:`ReplicaSetConnection` is deprecated. Please use :class:`~pymongo.mongo_replica_set_client.MongoReplicaSetClient` instead The resultant connection object has connection-pooling built in. It also performs auto-reconnection when necessary. If an operation fails because of a connection error, :class:`~pymongo.errors.ConnectionFailure` is raised. If auto-reconnection will be performed, :class:`~pymongo.errors.AutoReconnect` will be raised. Application code should handle this exception (recognizing that the operation failed) and then continue to execute. Raises :class:`~pymongo.errors.ConnectionFailure` if the connection cannot be made. The `hosts_or_uri` parameter can be a full `mongodb URI `_, in addition to a string of `host:port` pairs (e.g. 'host1:port1,host2:port2'). If `hosts_or_uri` is None 'localhost:27017' will be used. .. note:: Instances of :class:`~ReplicaSetConnection` start a background task to monitor the state of the replica set. This allows it to quickly respond to changes in replica set configuration. Before discarding an instance of :class:`~ReplicaSetConnection` make sure you call :meth:`~close` to ensure that the monitor task is cleanly shut down. :Parameters: - `hosts_or_uri` (optional): A MongoDB URI or string of `host:port` pairs. If a host is an IPv6 literal it must be enclosed in '[' and ']' characters following the RFC2732 URL syntax (e.g. '[::1]' for localhost) - `max_pool_size` (optional): The maximum number of connections each pool will open simultaneously. If this is set, operations will block if there are `max_pool_size` outstanding connections from the pool. By default the pool size is unlimited. - `document_class` (optional): default class to use for documents returned from queries on this connection - `tz_aware` (optional): if ``True``, :class:`~datetime.datetime` instances returned as values in a document by this :class:`ReplicaSetConnection` will be timezone aware (otherwise they will be naive) - `replicaSet`: (required) The name of the replica set to connect to. The driver will verify that each host it connects to is a member of this replica set. Can be passed as a keyword argument or as a MongoDB URI option. | **Other optional parameters can be passed as keyword arguments:** - `host`: For compatibility with connection.Connection. If both `host` and `hosts_or_uri` are specified `host` takes precedence. - `port`: For compatibility with connection.Connection. The default port number to use for hosts. - `network_timeout`: For compatibility with connection.Connection. The timeout (in seconds) to use for socket operations - default is no timeout. If both `network_timeout` and `socketTimeoutMS` are specified `network_timeout` takes precedence, matching connection.Connection. - `socketTimeoutMS`: (integer) How long (in milliseconds) a send or receive on a socket can take before timing out. - `connectTimeoutMS`: (integer) How long (in milliseconds) a connection can take to be opened before timing out. - `waitQueueTimeoutMS`: (integer) How long (in milliseconds) a thread will wait for a socket from the pool if the pool has no free sockets. Defaults to ``None`` (no timeout). - `waitQueueMultiple`: (integer) Multiplied by max_pool_size to give the number of threads allowed to wait for a socket at one time. Defaults to ``None`` (no waiters). - `auto_start_request`: If ``True`` (the default), each thread that accesses this :class:`ReplicaSetConnection` has a socket allocated to it for the thread's lifetime, for each member of the set. For :class:`~pymongo.read_preferences.ReadPreference` PRIMARY, auto_start_request=True ensures consistent reads, even if you read after an unsafe write. For read preferences other than PRIMARY, there are no consistency guarantees. - `use_greenlets`: if ``True``, use a background Greenlet instead of a background thread to monitor state of replica set. Additionally, :meth:`start_request()` will ensure that the current greenlet uses the same socket for all operations until :meth:`end_request()`. `use_greenlets` with ReplicaSetConnection requires `Gevent `_ to be installed. | **Write Concern options:** - `safe`: :class:`ReplicaSetConnection` **disables** acknowledgement of write operations. Use ``safe=True`` to enable write acknowledgement. - `w`: (integer or string) Write operations will block until they have been replicated to the specified number or tagged set of servers. `w=` always includes the replica set primary (e.g. w=3 means write to the primary and wait until replicated to **two** secondaries). Implies safe=True. - `wtimeout`: (integer) Used in conjunction with `w`. Specify a value in milliseconds to control how long to wait for write propagation to complete. If replication does not complete in the given timeframe, a timeout exception is raised. Implies safe=True. - `j`: If ``True`` block until write operations have been committed to the journal. Ignored if the server is running without journaling. Implies safe=True. - `fsync`: If ``True`` force the database to fsync all files before returning. When used with `j` the server awaits the next group commit before returning. Implies safe=True. | **Read preference options:** - `slave_okay` or `slaveOk` (deprecated): Use `read_preference` instead. - `read_preference`: The read preference for this connection. See :class:`~pymongo.read_preferences.ReadPreference` for available - `tag_sets`: Read from replica-set members with these tags. To specify a priority-order for tag sets, provide a list of tag sets: ``[{'dc': 'ny'}, {'dc': 'la'}, {}]``. A final, empty tag set, ``{}``, means "read from any member that matches the mode, ignoring tags." :class:`MongoReplicaSetClient` tries each set of tags in turn until it finds a set of tags with at least one matching member. - `secondary_acceptable_latency_ms`: (integer) Any replica-set member whose ping time is within secondary_acceptable_latency_ms of the nearest member may accept reads. Default 15 milliseconds. **Ignored by mongos** and must be configured on the command line. See the localThreshold_ option for more information. | **SSL configuration:** - `ssl`: If ``True``, create the connection to the servers using SSL. - `ssl_keyfile`: The private keyfile used to identify the local connection against mongod. If included with the ``certfile` then only the ``ssl_certfile`` is needed. Implies ``ssl=True``. - `ssl_certfile`: The certificate file used to identify the local connection against mongod. Implies ``ssl=True``. - `ssl_cert_reqs`: Specifies whether a certificate is required from the other side of the connection, and whether it will be validated if provided. It must be one of the three values ``ssl.CERT_NONE`` (certificates ignored), ``ssl.CERT_OPTIONAL`` (not required, but validated if provided), or ``ssl.CERT_REQUIRED`` (required and validated). If the value of this parameter is not ``ssl.CERT_NONE``, then the ``ssl_ca_certs`` parameter must point to a file of CA certificates. Implies ``ssl=True``. - `ssl_ca_certs`: The ca_certs file contains a set of concatenated "certification authority" certificates, which are used to validate certificates passed from the other end of the connection. Implies ``ssl=True``. .. versionchanged:: 2.5 Added additional ssl options .. versionchanged:: 2.3 Added `tag_sets` and `secondary_acceptable_latency_ms` options. .. versionchanged:: 2.2 Added `auto_start_request` and `use_greenlets` options. Added support for `host`, `port`, and `network_timeout` keyword arguments for compatibility with connection.Connection. .. versionadded:: 2.1 .. _localThreshold: http://docs.mongodb.org/manual/reference/mongos/#cmdoption-mongos--localThreshold """ network_timeout = kwargs.pop('network_timeout', None) if network_timeout is not None: if (not isinstance(network_timeout, (int, float)) or network_timeout <= 0): raise ConfigurationError("network_timeout must " "be a positive integer") kwargs['socketTimeoutMS'] = network_timeout * 1000 kwargs['auto_start_request'] = kwargs.get('auto_start_request', True) kwargs['safe'] = kwargs.get('safe', False) super(ReplicaSetConnection, self).__init__( hosts_or_uri, max_pool_size, document_class, tz_aware, **kwargs) def __repr__(self): return "ReplicaSetConnection(%r)" % (["%s:%d" % n for n in self.hosts],) pymongo-2.6.3/pymongo/son_manipulator.py000066400000000000000000000135201223300253600204630ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Manipulators that can edit SON objects as they enter and exit a database. New manipulators should be defined as subclasses of SONManipulator and can be installed on a database by calling `pymongo.database.Database.add_son_manipulator`.""" from bson.dbref import DBRef from bson.objectid import ObjectId from bson.son import SON class SONManipulator(object): """A base son manipulator. This manipulator just saves and restores objects without changing them. """ def will_copy(self): """Will this SON manipulator make a copy of the incoming document? Derived classes that do need to make a copy should override this method, returning True instead of False. All non-copying manipulators will be applied first (so that the user's document will be updated appropriately), followed by copying manipulators. """ return False def transform_incoming(self, son, collection): """Manipulate an incoming SON object. :Parameters: - `son`: the SON object to be inserted into the database - `collection`: the collection the object is being inserted into """ if self.will_copy(): return SON(son) return son def transform_outgoing(self, son, collection): """Manipulate an outgoing SON object. :Parameters: - `son`: the SON object being retrieved from the database - `collection`: the collection this object was stored in """ if self.will_copy(): return SON(son) return son class ObjectIdInjector(SONManipulator): """A son manipulator that adds the _id field if it is missing. """ def transform_incoming(self, son, collection): """Add an _id field if it is missing. """ if not "_id" in son: son["_id"] = ObjectId() return son # This is now handled during BSON encoding (for performance reasons), # but I'm keeping this here as a reference for those implementing new # SONManipulators. class ObjectIdShuffler(SONManipulator): """A son manipulator that moves _id to the first position. """ def will_copy(self): """We need to copy to be sure that we are dealing with SON, not a dict. """ return True def transform_incoming(self, son, collection): """Move _id to the front if it's there. """ if not "_id" in son: return son transformed = SON({"_id": son["_id"]}) transformed.update(son) return transformed class NamespaceInjector(SONManipulator): """A son manipulator that adds the _ns field. """ def transform_incoming(self, son, collection): """Add the _ns field to the incoming object """ son["_ns"] = collection.name return son class AutoReference(SONManipulator): """Transparently reference and de-reference already saved embedded objects. This manipulator should probably only be used when the NamespaceInjector is also being used, otherwise it doesn't make too much sense - documents can only be auto-referenced if they have an *_ns* field. NOTE: this will behave poorly if you have a circular reference. TODO: this only works for documents that are in the same database. To fix this we'll need to add a DatabaseInjector that adds *_db* and then make use of the optional *database* support for DBRefs. """ def __init__(self, db): self.database = db def will_copy(self): """We need to copy so the user's document doesn't get transformed refs. """ return True def transform_incoming(self, son, collection): """Replace embedded documents with DBRefs. """ def transform_value(value): if isinstance(value, dict): if "_id" in value and "_ns" in value: return DBRef(value["_ns"], transform_value(value["_id"])) else: return transform_dict(SON(value)) elif isinstance(value, list): return [transform_value(v) for v in value] return value def transform_dict(object): for (key, value) in object.items(): object[key] = transform_value(value) return object return transform_dict(SON(son)) def transform_outgoing(self, son, collection): """Replace DBRefs with embedded documents. """ def transform_value(value): if isinstance(value, DBRef): return self.database.dereference(value) elif isinstance(value, list): return [transform_value(v) for v in value] elif isinstance(value, dict): return transform_dict(SON(value)) return value def transform_dict(object): for (key, value) in object.items(): object[key] = transform_value(value) return object return transform_dict(SON(son)) # TODO make a generic translator for custom types. Take encode, decode, # should_encode and should_decode functions and just encode and decode where # necessary. See examples/custom_type.py for where this would be useful. # Alternatively it could take a should_encode, to_binary, from_binary and # binary subtype. pymongo-2.6.3/pymongo/ssl_match_hostname.py000066400000000000000000000050401223300253600211220ustar00rootroot00000000000000# Backport of the match_hostname logic introduced in python 3.2 # http://svn.python.org/projects/python/branches/release32-maint/Lib/ssl.py import re class CertificateError(ValueError): pass def _dnsname_to_pat(dn, max_wildcards=1): pats = [] for frag in dn.split(r'.'): if frag.count('*') > max_wildcards: # Issue #17980: avoid denials of service by refusing more # than one wildcard per fragment. A survery of established # policy among SSL implementations showed it to be a # reasonable choice. raise CertificateError( "too many wildcards in certificate DNS name: " + repr(dn)) if frag == '*': # When '*' is a fragment by itself, it matches a non-empty dotless # fragment. pats.append('[^.]+') else: # Otherwise, '*' matches any dotless fragment. frag = re.escape(frag) pats.append(frag.replace(r'\*', '[^.]*')) return re.compile(r'\A' + r'\.'.join(pats) + r'\Z', re.IGNORECASE) def match_hostname(cert, hostname): """Verify that *cert* (in decoded format as returned by SSLSocket.getpeercert()) matches the *hostname*. RFC 2818 rules are mostly followed, but IP addresses are not accepted for *hostname*. CertificateError is raised on failure. On success, the function returns nothing. """ if not cert: raise ValueError("empty or no certificate") dnsnames = [] san = cert.get('subjectAltName', ()) for key, value in san: if key == 'DNS': if _dnsname_to_pat(value).match(hostname): return dnsnames.append(value) if not san: # The subject is only checked when subjectAltName is empty for sub in cert.get('subject', ()): for key, value in sub: # XXX according to RFC 2818, the most specific Common Name # must be used. if key == 'commonName': if _dnsname_to_pat(value).match(hostname): return dnsnames.append(value) if len(dnsnames) > 1: raise CertificateError("hostname %r " "doesn't match either of %s" % (hostname, ', '.join(map(repr, dnsnames)))) elif len(dnsnames) == 1: raise CertificateError("hostname %r " "doesn't match %r" % (hostname, dnsnames[0])) else: raise CertificateError("no appropriate commonName or " "subjectAltName fields were found") pymongo-2.6.3/pymongo/thread_util.py000066400000000000000000000213001223300253600175500ustar00rootroot00000000000000# Copyright 2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Utilities to abstract the differences between threads and greenlets.""" import threading import sys import weakref try: from time import monotonic as _time except ImportError: from time import time as _time have_gevent = True try: import greenlet try: # gevent-1.0rc2 and later. from gevent.lock import BoundedSemaphore as GeventBoundedSemaphore except ImportError: from gevent.coros import BoundedSemaphore as GeventBoundedSemaphore from gevent.greenlet import SpawnedLink except ImportError: have_gevent = False from pymongo.errors import ExceededMaxWaiters # Do we have to work around http://bugs.python.org/issue1868? issue1868 = (sys.version_info[:3] <= (2, 7, 0)) class Ident(object): def __init__(self): self._refs = {} def watching(self): """Is the current thread or greenlet being watched for death?""" return self.get() in self._refs def unwatch(self, tid): self._refs.pop(tid, None) def get(self): """An id for this thread or greenlet""" raise NotImplementedError def watch(self, callback): """Run callback when this thread or greenlet dies. callback takes one meaningless argument. """ raise NotImplementedError class ThreadIdent(Ident): class _DummyLock(object): def acquire(self): pass def release(self): pass def __init__(self): super(ThreadIdent, self).__init__() self._local = threading.local() if issue1868: self._lock = threading.Lock() else: self._lock = ThreadIdent._DummyLock() # We watch for thread-death using a weakref callback to a thread local. # Weakrefs are permitted on subclasses of object but not object() itself. class ThreadVigil(object): pass def _make_vigil(self): # Threadlocals in Python <= 2.7.0 have race conditions when setting # attributes and possibly when getting them, too, leading to weakref # callbacks not getting called later. self._lock.acquire() try: vigil = getattr(self._local, 'vigil', None) if not vigil: self._local.vigil = vigil = ThreadIdent.ThreadVigil() finally: self._lock.release() return vigil def get(self): return id(self._make_vigil()) def watch(self, callback): vigil = self._make_vigil() self._refs[id(vigil)] = weakref.ref(vigil, callback) class GreenletIdent(Ident): def get(self): return id(greenlet.getcurrent()) def watch(self, callback): current = greenlet.getcurrent() tid = self.get() if hasattr(current, 'link'): # This is a Gevent Greenlet (capital G), which inherits from # greenlet and provides a 'link' method to detect when the # Greenlet exits. link = SpawnedLink(callback) current.rawlink(link) self._refs[tid] = link else: # This is a non-Gevent greenlet (small g), or it's the main # greenlet. self._refs[tid] = weakref.ref(current, callback) def unwatch(self, tid): """ call unlink if link before """ link = self._refs.pop(tid, None) current = greenlet.getcurrent() if hasattr(current, 'unlink'): # This is a Gevent enhanced Greenlet. Remove the SpawnedLink we # linked to it. current.unlink(link) def create_ident(use_greenlets): if use_greenlets: return GreenletIdent() else: return ThreadIdent() class Counter(object): """A thread- or greenlet-local counter. """ def __init__(self, use_greenlets): self.ident = create_ident(use_greenlets) self._counters = {} def inc(self): # Copy these references so on_thread_died needn't close over self ident = self.ident _counters = self._counters tid = ident.get() _counters.setdefault(tid, 0) _counters[tid] += 1 if not ident.watching(): # Before the tid is possibly reused, remove it from _counters def on_thread_died(ref): ident.unwatch(tid) _counters.pop(tid, None) ident.watch(on_thread_died) return _counters[tid] def dec(self): tid = self.ident.get() if self._counters.get(tid, 0) > 0: self._counters[tid] -= 1 return self._counters[tid] else: return 0 def get(self): return self._counters.get(self.ident.get(), 0) ### Begin backport from CPython 3.2 for timeout support for Semaphore.acquire class Semaphore: # After Tim Peters' semaphore class, but not quite the same (no maximum) def __init__(self, value=1): if value < 0: raise ValueError("semaphore initial value must be >= 0") self._cond = threading.Condition(threading.Lock()) self._value = value def acquire(self, blocking=True, timeout=None): if not blocking and timeout is not None: raise ValueError("can't specify timeout for non-blocking acquire") rc = False endtime = None self._cond.acquire() while self._value == 0: if not blocking: break if timeout is not None: if endtime is None: endtime = _time() + timeout else: timeout = endtime - _time() if timeout <= 0: break self._cond.wait(timeout) else: self._value = self._value - 1 rc = True self._cond.release() return rc __enter__ = acquire def release(self): self._cond.acquire() self._value = self._value + 1 self._cond.notify() self._cond.release() def __exit__(self, t, v, tb): self.release() @property def counter(self): return self._value class BoundedSemaphore(Semaphore): """Semaphore that checks that # releases is <= # acquires""" def __init__(self, value=1): Semaphore.__init__(self, value) self._initial_value = value def release(self): if self._value >= self._initial_value: raise ValueError("Semaphore released too many times") return Semaphore.release(self) ### End backport from CPython 3.2 class DummySemaphore(object): def __init__(self, value=None): pass def acquire(self, blocking=True, timeout=None): return True def release(self): pass class MaxWaitersBoundedSemaphore(object): def __init__(self, semaphore_class, value=1, max_waiters=1): self.waiter_semaphore = semaphore_class(max_waiters) self.semaphore = semaphore_class(value) def acquire(self, blocking=True, timeout=None): if not self.waiter_semaphore.acquire(False): raise ExceededMaxWaiters() try: return self.semaphore.acquire(blocking, timeout) finally: self.waiter_semaphore.release() def __getattr__(self, name): return getattr(self.semaphore, name) class MaxWaitersBoundedSemaphoreThread(MaxWaitersBoundedSemaphore): def __init__(self, value=1, max_waiters=1): MaxWaitersBoundedSemaphore.__init__( self, BoundedSemaphore, value, max_waiters) if have_gevent: class MaxWaitersBoundedSemaphoreGevent(MaxWaitersBoundedSemaphore): def __init__(self, value=1, max_waiters=1): MaxWaitersBoundedSemaphore.__init__( self, GeventBoundedSemaphore, value, max_waiters) def create_semaphore(max_size, max_waiters, use_greenlets): if max_size is None: return DummySemaphore() elif use_greenlets: if max_waiters is None: return GeventBoundedSemaphore(max_size) else: return MaxWaitersBoundedSemaphoreGevent(max_size, max_waiters) else: if max_waiters is None: return BoundedSemaphore(max_size) else: return MaxWaitersBoundedSemaphoreThread(max_size, max_waiters) pymongo-2.6.3/pymongo/uri_parser.py000066400000000000000000000234311223300253600174260ustar00rootroot00000000000000# Copyright 2011-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you # may not use this file except in compliance with the License. You # may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. See the License for the specific language governing # permissions and limitations under the License. """Tools to parse and validate a MongoDB URI.""" from urllib import unquote_plus from pymongo.common import validate from pymongo.errors import (ConfigurationError, InvalidURI, UnsupportedOption) SCHEME = 'mongodb://' SCHEME_LEN = len(SCHEME) DEFAULT_PORT = 27017 def _partition(entity, sep): """Python2.4 doesn't have a partition method so we provide our own that mimics str.partition from later releases. Split the string at the first occurrence of sep, and return a 3-tuple containing the part before the separator, the separator itself, and the part after the separator. If the separator is not found, return a 3-tuple containing the string itself, followed by two empty strings. """ parts = entity.split(sep, 1) if len(parts) == 2: return parts[0], sep, parts[1] else: return entity, '', '' def _rpartition(entity, sep): """Python2.4 doesn't have an rpartition method so we provide our own that mimics str.rpartition from later releases. Split the string at the last occurrence of sep, and return a 3-tuple containing the part before the separator, the separator itself, and the part after the separator. If the separator is not found, return a 3-tuple containing two empty strings, followed by the string itself. """ idx = entity.rfind(sep) if idx == -1: return '', '', entity return entity[:idx], sep, entity[idx + 1:] def parse_userinfo(userinfo): """Validates the format of user information in a MongoDB URI. Reserved characters like ':', '/', '+' and '@' must be escaped following RFC 2396. Returns a 2-tuple containing the unescaped username followed by the unescaped password. :Paramaters: - `userinfo`: A string of the form : .. versionchanged:: 2.2 Now uses `urllib.unquote_plus` so `+` characters must be escaped. """ if '@' in userinfo or userinfo.count(':') > 1: raise InvalidURI("':' or '@' characters in a username or password " "must be escaped according to RFC 2396.") user, _, passwd = _partition(userinfo, ":") # No password is expected with GSSAPI authentication. if not user: raise InvalidURI("The empty string is not valid username.") user = unquote_plus(user) passwd = unquote_plus(passwd) return user, passwd def parse_ipv6_literal_host(entity, default_port): """Validates an IPv6 literal host:port string. Returns a 2-tuple of IPv6 literal followed by port where port is default_port if it wasn't specified in entity. :Parameters: - `entity`: A string that represents an IPv6 literal enclosed in braces (e.g. '[::1]' or '[::1]:27017'). - `default_port`: The port number to use when one wasn't specified in entity. """ if entity.find(']') == -1: raise ConfigurationError("an IPv6 address literal must be " "enclosed in '[' and ']' according " "to RFC 2732.") i = entity.find(']:') if i == -1: return entity[1:-1], default_port return entity[1: i], entity[i + 2:] def parse_host(entity, default_port=DEFAULT_PORT): """Validates a host string Returns a 2-tuple of host followed by port where port is default_port if it wasn't specified in the string. :Parameters: - `entity`: A host or host:port string where host could be a hostname or IP address. - `default_port`: The port number to use when one wasn't specified in entity. """ host = entity port = default_port if entity[0] == '[': host, port = parse_ipv6_literal_host(entity, default_port) elif entity.find(':') != -1: if entity.count(':') > 1: raise ConfigurationError("Reserved characters such as ':' must be " "escaped according RFC 2396. An IPv6 " "address literal must be enclosed in '[' " "and ']' according to RFC 2732.") host, port = host.split(':', 1) if isinstance(port, basestring): if not port.isdigit(): raise ConfigurationError("Port number must be an integer.") port = int(port) return host, port def validate_options(opts): """Validates and normalizes options passed in a MongoDB URI. Returns a new dictionary of validated and normalized options. :Parameters: - `opts`: A dict of MongoDB URI options. """ normalized = {} for option, value in opts.iteritems(): option, value = validate(option, value) # str(option) to ensure that a unicode URI results in plain 'str' # option names. 'normalized' is then suitable to be passed as kwargs # in all Python versions. normalized[str(option)] = value return normalized def split_options(opts): """Takes the options portion of a MongoDB URI, validates each option and returns the options in a dictionary. The option names will be returned lowercase even if camelCase options are used. :Parameters: - `opt`: A string representing MongoDB URI options. """ and_idx = opts.find("&") semi_idx = opts.find(";") try: if and_idx >= 0 and semi_idx >= 0: raise InvalidURI("Can not mix '&' and ';' for option separators.") elif and_idx >= 0: options = dict([kv.split("=") for kv in opts.split("&")]) elif semi_idx >= 0: options = dict([kv.split("=") for kv in opts.split(";")]) elif opts.find("=") != -1: options = dict([opts.split("=")]) else: raise ValueError except ValueError: raise InvalidURI("MongoDB URI options are key=value pairs.") return validate_options(options) def split_hosts(hosts, default_port=DEFAULT_PORT): """Takes a string of the form host1[:port],host2[:port]... and splits it into (host, port) tuples. If [:port] isn't present the default_port is used. Returns a set of 2-tuples containing the host name (or IP) followed by port number. :Parameters: - `hosts`: A string of the form host1[:port],host2[:port],... - `default_port`: The port number to use when one wasn't specified for a host. """ nodes = [] for entity in hosts.split(','): if not entity: raise ConfigurationError("Empty host " "(or extra comma in host list).") port = default_port # Unix socket entities don't have ports if entity.endswith('.sock'): port = None nodes.append(parse_host(entity, port)) return nodes def parse_uri(uri, default_port=DEFAULT_PORT): """Parse and validate a MongoDB URI. Returns a dict of the form:: { 'nodelist': , 'username': or None, 'password': or None, 'database': or None, 'collection': or None, 'options': } :Parameters: - `uri`: The MongoDB URI to parse. - `default_port`: The port number to use when one wasn't specified for a host in the URI. """ if not uri.startswith(SCHEME): raise InvalidURI("Invalid URI scheme: URI " "must begin with '%s'" % (SCHEME,)) scheme_free = uri[SCHEME_LEN:] if not scheme_free: raise InvalidURI("Must provide at least one hostname or IP.") nodes = None user = None passwd = None dbase = None collection = None options = {} # Check for unix domain sockets in the uri if '.sock' in scheme_free: host_part, _, path_part = _rpartition(scheme_free, '/') try: parse_uri('%s%s' % (SCHEME, host_part)) except (ConfigurationError, InvalidURI): host_part = scheme_free path_part = "" else: host_part, _, path_part = _partition(scheme_free, '/') if not path_part and '?' in host_part: raise InvalidURI("A '/' is required between " "the host list and any options.") if '@' in host_part: userinfo, _, hosts = _rpartition(host_part, '@') user, passwd = parse_userinfo(userinfo) else: hosts = host_part nodes = split_hosts(hosts, default_port=default_port) if path_part: if path_part[0] == '?': opts = path_part[1:] else: dbase, _, opts = _partition(path_part, '?') if '.' in dbase: dbase, collection = dbase.split('.', 1) if opts: options = split_options(opts) return { 'nodelist': nodes, 'username': user, 'password': passwd, 'database': dbase, 'collection': collection, 'options': options } if __name__ == '__main__': import pprint import sys try: pprint.pprint(parse_uri(sys.argv[1])) except (InvalidURI, UnsupportedOption), e: print e sys.exit(0) pymongo-2.6.3/setup.cfg000066400000000000000000000000731223300253600150270ustar00rootroot00000000000000[egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 pymongo-2.6.3/setup.py000077500000000000000000000246341223300253600147340ustar00rootroot00000000000000import glob import os import subprocess import sys import warnings # Hack to silence atexit traceback in newer python versions. try: import multiprocessing except ImportError: pass try: from ConfigParser import SafeConfigParser except ImportError: # PY3 from configparser import SafeConfigParser # Don't force people to install distribute unless # we have to. try: from setuptools import setup, Feature except ImportError: from distribute_setup import use_setuptools use_setuptools() from setuptools import setup, Feature from distutils.cmd import Command from distutils.command.build_ext import build_ext from distutils.errors import CCompilerError from distutils.errors import DistutilsPlatformError, DistutilsExecError from distutils.core import Extension version = "2.6.3" f = open("README.rst") try: try: readme_content = f.read() except: readme_content = "" finally: f.close() PY3 = sys.version_info[0] == 3 nose_config_options = { 'with-xunit': '1', # Write out nosetests.xml for CI. 'py3where': 'build', # Tell nose where to find tests under PY3. } def write_nose_config(): """Write out setup.cfg. Since py3where has to be set for tests to run correctly in Python 3 we create this on the fly. """ config = SafeConfigParser() config.add_section('nosetests') for opt, val in nose_config_options.items(): config.set('nosetests', opt, val) try: cf = open('setup.cfg', 'w') config.write(cf) finally: cf.close() def should_run_tests(): if "test" in sys.argv or "nosetests" in sys.argv: return True return False class doc(Command): description = "generate or test documentation" user_options = [("test", "t", "run doctests instead of generating documentation")] boolean_options = ["test"] def initialize_options(self): self.test = False def finalize_options(self): pass def run(self): if self.test: path = "doc/_build/doctest" mode = "doctest" else: path = "doc/_build/%s" % version mode = "html" try: os.makedirs(path) except: pass status = subprocess.call(["sphinx-build", "-E", "-b", mode, "doc", path]) if status: raise RuntimeError("documentation step '%s' failed" % (mode,)) sys.stdout.write("\nDocumentation step '%s' performed, results here:\n" " %s/\n" % (mode, path)) if sys.platform == 'win32' and sys.version_info > (2, 6): # 2.6's distutils.msvc9compiler can raise an IOError when failing to # find the compiler build_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError, IOError) else: build_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError) class custom_build_ext(build_ext): """Allow C extension building to fail. The C extension speeds up BSON encoding, but is not essential. """ warning_message = """ ******************************************************************** WARNING: %s could not be compiled. No C extensions are essential for PyMongo to run, although they do result in significant speed improvements. %s Please see the installation docs for solutions to build issues: http://api.mongodb.org/python/current/installation.html Here are some hints for popular operating systems: If you are seeing this message on Linux you probably need to install GCC and/or the Python development package for your version of Python. Debian and Ubuntu users should issue the following command: $ sudo apt-get install build-essential python-dev RedHat, CentOS, and Fedora users should issue the following command: $ sudo yum install gcc python-devel If you are seeing this message on Microsoft Windows please install PyMongo using the MS Windows installer for your version of Python, available on pypi here: http://pypi.python.org/pypi/pymongo/#downloads If you are seeing this message on OSX please read the documentation here: http://api.mongodb.org/python/current/installation.html#osx ******************************************************************** """ def run(self): try: build_ext.run(self) except DistutilsPlatformError: e = sys.exc_info()[1] sys.stdout.write('%s\n' % str(e)) warnings.warn(self.warning_message % ("Extension modules", "There was an issue with " "your platform configuration" " - see above.")) def set_nose_options(self): # Under python 3 we need to tell nose where to find the # proper tests. if we built the C extensions this will be # someplace like build/lib.-- if PY3: ver = '.'.join(map(str, sys.version_info[:2])) lib_dirs = glob.glob(os.path.join('build', 'lib*' + ver)) if lib_dirs: nose_config_options['py3where'] = lib_dirs[0] write_nose_config() def build_extension(self, ext): name = ext.name if sys.version_info[:3] >= (2, 4, 0): try: build_ext.build_extension(self, ext) if should_run_tests(): self.set_nose_options() except build_errors: e = sys.exc_info()[1] sys.stdout.write('%s\n' % str(e)) warnings.warn(self.warning_message % ("The %s extension " "module" % (name,), "The output above " "this warning shows how " "the compilation " "failed.")) else: warnings.warn(self.warning_message % ("The %s extension " "module" % (name,), "Please use Python >= 2.4 " "to take advantage of the " "extension.")) c_ext = Feature( "optional C extensions", standard=True, ext_modules=[Extension('bson._cbson', include_dirs=['bson'], sources=['bson/_cbsonmodule.c', 'bson/time64.c', 'bson/buffer.c', 'bson/encoding_helpers.c']), Extension('pymongo._cmessage', include_dirs=['bson'], sources=['pymongo/_cmessagemodule.c', 'bson/buffer.c'])]) if "--no_ext" in sys.argv: sys.argv = [x for x in sys.argv if x != "--no_ext"] features = {} elif (sys.platform.startswith("java") or sys.platform == "cli" or "PyPy" in sys.version): sys.stdout.write(""" *****************************************************\n The optional C extensions are currently not supported\n by this python implementation.\n *****************************************************\n """) features = {} elif sys.byteorder == "big": sys.stdout.write(""" *****************************************************\n The optional C extensions are currently not supported\n on big endian platforms and will not be built.\n Performance may be degraded.\n *****************************************************\n """) features = {} else: features = {"c-ext": c_ext} extra_opts = { "packages": ["bson", "pymongo", "gridfs"], "test_suite": "nose.collector" } if PY3: extra_opts["use_2to3"] = True if should_run_tests(): # Distribute isn't smart enough to copy the # tests and run 2to3 on them. We don't want to # install the test suite so only do this if we # are testing. # https://bitbucket.org/tarek/distribute/issue/233 extra_opts["packages"].append("test") extra_opts['package_data'] = {"test": ["certificates/ca.pem", "certificates/client.pem"]} # Hack to make "python3.x setup.py nosetests" work in python 3 # otherwise it won't run 2to3 before running the tests. if "nosetests" in sys.argv: sys.argv.remove("nosetests") sys.argv.append("test") # All "nosetests" does is import and run nose.main. extra_opts["test_suite"] = "nose.main" # This may be called a second time if # we are testing with C extensions. if should_run_tests(): write_nose_config() setup( name="pymongo", version=version, description="Python driver for MongoDB ", long_description=readme_content, author="Mike Dirolf", author_email="mongodb-user@googlegroups.com", maintainer="Bernie Hackett", maintainer_email="bernie@10gen.com", url="http://github.com/mongodb/mongo-python-driver", keywords=["mongo", "mongodb", "pymongo", "gridfs", "bson"], install_requires=[], features=features, license="Apache License, Version 2.0", tests_require=["nose"], classifiers=[ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Operating System :: MacOS :: MacOS X", "Operating System :: Microsoft :: Windows", "Operating System :: POSIX", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.4", "Programming Language :: Python :: 2.5", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.1", "Programming Language :: Python :: 3.2", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: Jython", "Programming Language :: Python :: Implementation :: PyPy", "Topic :: Database"], cmdclass={"build_ext": custom_build_ext, "doc": doc}, **extra_opts ) pymongo-2.6.3/test/000077500000000000000000000000001223300253600141655ustar00rootroot00000000000000pymongo-2.6.3/test/__init__.py000066400000000000000000000032231223300253600162760ustar00rootroot00000000000000# Copyright 2010-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Clean up databases after running `nosetests`. """ import os import pymongo from pymongo.errors import ConnectionFailure # hostnames retrieved by MongoReplicaSetClient from isMaster will be of unicode # type in Python 2, so ensure these hostnames are unicodes, too. It makes tests # like `test_repr` predictable. host = unicode(os.environ.get("DB_IP", 'localhost')) port = int(os.environ.get("DB_PORT", 27017)) pair = '%s:%d' % (host, port) host2 = unicode(os.environ.get("DB_IP2", 'localhost')) port2 = int(os.environ.get("DB_PORT2", 27018)) host3 = unicode(os.environ.get("DB_IP3", 'localhost')) port3 = int(os.environ.get("DB_PORT3", 27019)) def teardown(): try: c = pymongo.MongoClient(host, port) except ConnectionFailure: # Tests where ssl=True can cause connection failures here. # Ignore and continue. return c.drop_database("pymongo-pooling-tests") c.drop_database("pymongo_test") c.drop_database("pymongo_test1") c.drop_database("pymongo_test2") c.drop_database("pymongo_test_mike") c.drop_database("pymongo_test_bernie") pymongo-2.6.3/test/certificates/000077500000000000000000000000001223300253600166325ustar00rootroot00000000000000pymongo-2.6.3/test/certificates/ca.pem000066400000000000000000000020761223300253600177250ustar00rootroot00000000000000-----BEGIN CERTIFICATE----- MIIC9DCCAl2gAwIBAgIJAJeYVdtunBOmMA0GCSqGSIb3DQEBBQUAMIGSMQswCQYD VQQGEwJVUzERMA8GA1UECAwITmV3IFlvcmsxFjAUBgNVBAcMDU5ldyBZb3JrIENp dHkxDjAMBgNVBAoMBTEwR2VuMQ8wDQYDVQQLDAZLZXJuZWwxGjAYBgNVBAMMEU15 IENlcnQgQXV0aG9yaXR5MRswGQYJKoZIhvcNAQkBFgxyb290QGxhemFydXMwHhcN MTIxMTI3MTkwMzM5WhcNMTMxMTI3MTkwMzM5WjCBkjELMAkGA1UEBhMCVVMxETAP BgNVBAgMCE5ldyBZb3JrMRYwFAYDVQQHDA1OZXcgWW9yayBDaXR5MQ4wDAYDVQQK DAUxMEdlbjEPMA0GA1UECwwGS2VybmVsMRowGAYDVQQDDBFNeSBDZXJ0IEF1dGhv cml0eTEbMBkGCSqGSIb3DQEJARYMcm9vdEBsYXphcnVzMIGfMA0GCSqGSIb3DQEB AQUAA4GNADCBiQKBgQDXHKZ5j5T969S5C/Gm6f2ah7gaik3zRzWm2ZoAcz/U6fBq rnha3bueXXBRWZ7d2HgN1a+JhjuYnffcdUSen9CFVxPiRCEgJmp2A8o90Kx5Bbcf 7zHobDOGs1EF3PQ2RKgXEOUjKZ/LZDbGhClsIYCD4SdFhRMqUcxc2lQMsWEaNwID AQABo1AwTjAdBgNVHQ4EFgQUB0EZOp9+xbciTre81d/k/Am4ZBYwHwYDVR0jBBgw FoAUB0EZOp9+xbciTre81d/k/Am4ZBYwDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0B AQUFAAOBgQB6aSQNTmD4gIQEcZiOXHJVpGOHeHBOxWteMFhcBpWvt0Cv8sqLZIVq x0eAC/tQFkAVEjT+T4S4UdtxgZ44RKCZPYI00qZsyz5bNoTE8kN/bmYNjyKMVFaG 1tU+elCdOstzBLjY1aHG1oQzbyqgoiSIDpfzjlyK/tBpckFGCz6c6A== -----END CERTIFICATE----- pymongo-2.6.3/test/certificates/client.pem000066400000000000000000000035141223300253600206160ustar00rootroot00000000000000-----BEGIN CERTIFICATE----- MIICkjCCAfsCCQCRlIP8LltShTANBgkqhkiG9w0BAQUFADCBkjELMAkGA1UEBhMC VVMxETAPBgNVBAgMCE5ldyBZb3JrMRYwFAYDVQQHDA1OZXcgWW9yayBDaXR5MQ4w DAYDVQQKDAUxMEdlbjEPMA0GA1UECwwGS2VybmVsMRowGAYDVQQDDBFNeSBDZXJ0 IEF1dGhvcml0eTEbMBkGCSqGSIb3DQEJARYMcm9vdEBsYXphcnVzMB4XDTEyMTIx MDE4NTEzN1oXDTEzMTIxMDE4NTEzN1owgYcxCzAJBgNVBAYTAlVTMREwDwYDVQQI DAhOZXcgWW9yazEWMBQGA1UEBwwNTmV3IFlvcmsgQ2l0eTEOMAwGA1UECgwFMTBH ZW4xDzANBgNVBAsMBktlcm5lbDEPMA0GA1UEAwwGY2xpZW50MRswGQYJKoZIhvcN AQkBFgxyb290QGxhemFydXMwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBALX6 DqSWRJBEJJRIRqG5X3cFHzse5jGIdV8fTqikaVitvuhs15z1njzfqBQZMJBCEvNb 4eaenXJRMBDkEOcbfy6ah+ZLLqGFy7b6OxTROfx++3fTgsCAjBaIWvtGKNkwdcdM 7PQ2jE5bL8vN/ufbH2sX451nVd+j6oAz0dTz7RvhAgMBAAEwDQYJKoZIhvcNAQEF BQADgYEAlOJmaiT3ZhUHfCgBQEjHUZ/mmMDbUrgq5ZfQSrW/r3c6u+k8s2LVqVut Qz3V8z2vSuIkaPZRgDESWhPisi7sihhbV6xm4YTQW4LDlrom41/SEQ5TLP+Vz4Uq avzrAdaQ6+zHbEB94TuWuE3vyWVIP0fT1PtzFjcOJUWzgjEIR7M= -----END CERTIFICATE----- -----BEGIN PRIVATE KEY----- MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBALX6DqSWRJBEJJRI RqG5X3cFHzse5jGIdV8fTqikaVitvuhs15z1njzfqBQZMJBCEvNb4eaenXJRMBDk EOcbfy6ah+ZLLqGFy7b6OxTROfx++3fTgsCAjBaIWvtGKNkwdcdM7PQ2jE5bL8vN /ufbH2sX451nVd+j6oAz0dTz7RvhAgMBAAECgYEAmHRy+g5uSJLeNmBK1EiSIwtm e8hKP+s7scJvyrdbDpEZJG2zQWtA82zIynXECsdgSwOKQQRXkaNU6oG3a3bM19uY 0CqFRb9EwOLIStp+CM5zLRGmUr73u/+JrBPUWWFJkJvINvTXt18CMnCmosTvygWB IBZqsuEXQ6JcejxzQ6UCQQDdVUNdE2JgHp1qrr5l8563dztcrfCxuVFtgsj6qnhd UrBAa388B9kn4yVAe2i55xFmtHsO9Bz3ViiDFO163SafAkEA0nq8PeZtcIlZ2c7+ 6/Vdw1uLE5APVG2H9VEZdaVvkwIIXo8WQfMwWo5MQyPjVyBhUGlDwnKa46AcuplJ 2XMtfwJBAIDrMfKb4Ng13OEP6Yz+yvr4MxZ3plQOqlRMMn53HubUzB6pvpGbzKwE DWWyvDxUT/lvtKHwJJMYlz5KyUygVecCQHr50RBNmLW+2muDILiWlOD2lIyqh/pp QJ2Zc8mkDkuTTXaKHZQM1byjFXXI+yRFu/Xyeu+abFsAiqiPtXFCdVsCQHai+Ykv H3y0mUJmwBVP2fBE3GiTGlaadM0auZKu7/ad+yo7Hv8Kibacwibzrj9PjT3mFSSF vujX1oWOaxAMVbE= -----END PRIVATE KEY----- pymongo-2.6.3/test/high_availability/000077500000000000000000000000001223300253600176365ustar00rootroot00000000000000pymongo-2.6.3/test/high_availability/ha_tools.py000066400000000000000000000271661223300253600220340ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tools for testing high availability in PyMongo.""" import os import random import shutil import signal import socket import subprocess import sys import time from stat import S_IRUSR import pymongo import pymongo.errors from pymongo.read_preferences import ReadPreference home = os.environ.get('HOME') default_dbpath = os.path.join(home, 'data', 'pymongo_high_availability') dbpath = os.environ.get('DBPATH', default_dbpath) default_logpath = os.path.join(home, 'log', 'pymongo_high_availability') logpath = os.environ.get('LOGPATH', default_logpath) hostname = os.environ.get('HOSTNAME', socket.gethostname()) port = int(os.environ.get('DBPORT', 27017)) mongod = os.environ.get('MONGOD', 'mongod') mongos = os.environ.get('MONGOS', 'mongos') set_name = os.environ.get('SETNAME', 'repl0') use_greenlets = bool(os.environ.get('GREENLETS')) ha_tools_debug = bool(os.environ.get('HA_TOOLS_DEBUG')) nodes = {} routers = {} cur_port = port try: from subprocess import DEVNULL # Python 3. except ImportError: DEVNULL = open(os.devnull, 'wb') def kill_members(members, sig, hosts=nodes): for member in sorted(members): try: if ha_tools_debug: print('killing %s' % (member,)), proc = hosts[member]['proc'] # Not sure if cygwin makes sense here... if sys.platform in ('win32', 'cygwin'): os.kill(proc.pid, signal.CTRL_C_EVENT) else: os.kill(proc.pid, sig) except OSError: if ha_tools_debug: print('%s already dead?' % (member,)) def kill_all_members(): kill_members(nodes.keys(), 2, nodes) kill_members(routers.keys(), 2, routers) def wait_for(proc, port_num): trys = 0 while proc.poll() is None and trys < 160: trys += 1 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) try: try: s.connect((hostname, port_num)) return True except (IOError, socket.error): time.sleep(0.25) finally: s.close() kill_all_members() return False def start_subprocess(cmd): """Run cmd (a list of strings) and return a Popen instance.""" return subprocess.Popen(cmd, stdout=DEVNULL, stderr=DEVNULL) def start_replica_set(members, auth=False, fresh=True): global cur_port if fresh: if os.path.exists(dbpath): try: shutil.rmtree(dbpath) except OSError: pass try: os.makedirs(dbpath) except OSError: exc = sys.exc_info()[1] print(exc) print("\tWhile creating %s" % (dbpath,)) if auth: key_file = os.path.join(dbpath, 'key.txt') if not os.path.exists(key_file): f = open(key_file, 'w') try: f.write("my super secret system password") finally: f.close() os.chmod(key_file, S_IRUSR) for i in range(len(members)): host = '%s:%d' % (hostname, cur_port) members[i].update({'_id': i, 'host': host}) path = os.path.join(dbpath, 'db' + str(i)) if not os.path.exists(path): os.makedirs(path) member_logpath = os.path.join(logpath, 'db' + str(i) + '.log') if not os.path.exists(os.path.dirname(member_logpath)): os.makedirs(os.path.dirname(member_logpath)) cmd = [mongod, '--dbpath', path, '--port', str(cur_port), '--replSet', set_name, '--nojournal', '--oplogSize', '64', '--logappend', '--logpath', member_logpath] if auth: cmd += ['--keyFile', key_file] if ha_tools_debug: print('starting %s' % (' '.join(cmd),)) proc = start_subprocess(cmd) nodes[host] = {'proc': proc, 'cmd': cmd} res = wait_for(proc, cur_port) cur_port += 1 if not res: return None config = {'_id': set_name, 'members': members} primary = members[0]['host'] c = pymongo.MongoClient(primary, use_greenlets=use_greenlets) try: if ha_tools_debug: print('rs.initiate(%s)' % (config,)) c.admin.command('replSetInitiate', config) except pymongo.errors.OperationFailure: # Already initialized from a previous run? if ha_tools_debug: exc = sys.exc_info()[1] print(exc) expected_arbiters = 0 for member in members: if member.get('arbiterOnly'): expected_arbiters += 1 expected_secondaries = len(members) - expected_arbiters - 1 # Wait for 8 minutes for replica set to come up patience = 8 for i in range(int(patience * 60 / 2)): time.sleep(2) try: if (get_primary() and len(get_secondaries()) == expected_secondaries and len(get_arbiters()) == expected_arbiters): break except pymongo.errors.ConnectionFailure: # Keep waiting pass if ha_tools_debug: print('waiting for RS %s' % (i,)) else: kill_all_members() raise Exception( "Replica set still not initalized after %s minutes" % patience) return primary, set_name def create_sharded_cluster(num_routers=3): global cur_port # Start a config server configdb_host = '%s:%d' % (hostname, cur_port) path = os.path.join(dbpath, 'configdb') if not os.path.exists(path): os.makedirs(path) configdb_logpath = os.path.join(logpath, 'configdb.log') cmd = [mongod, '--dbpath', path, '--port', str(cur_port), '--nojournal', '--logappend', '--logpath', configdb_logpath] proc = start_subprocess(cmd) nodes[configdb_host] = {'proc': proc, 'cmd': cmd} res = wait_for(proc, cur_port) if not res: return None # ...and a shard server cur_port = cur_port + 1 shard_host = '%s:%d' % (hostname, cur_port) path = os.path.join(dbpath, 'shard1') if not os.path.exists(path): os.makedirs(path) db_logpath = os.path.join(logpath, 'shard1.log') cmd = [mongod, '--dbpath', path, '--port', str(cur_port), '--nojournal', '--logappend', '--logpath', db_logpath] proc = start_subprocess(cmd) nodes[shard_host] = {'proc': proc, 'cmd': cmd} res = wait_for(proc, cur_port) if not res: return None # ...and a few mongos instances cur_port = cur_port + 1 for i in range(num_routers): cur_port = cur_port + i host = '%s:%d' % (hostname, cur_port) mongos_logpath = os.path.join(logpath, 'mongos' + str(i) + '.log') cmd = [mongos, '--port', str(cur_port), '--logappend', '--logpath', mongos_logpath, '--configdb', configdb_host] proc = start_subprocess(cmd) routers[host] = {'proc': proc, 'cmd': cmd} res = wait_for(proc, cur_port) if not res: return None # Add the shard client = pymongo.MongoClient(host) try: client.admin.command({'addshard': shard_host}) except pymongo.errors.OperationFailure: # Already configured. pass return get_mongos_seed_list() # Connect to a random member def get_client(): return pymongo.MongoClient( nodes.keys(), read_preference=ReadPreference.PRIMARY_PREFERRED, use_greenlets=use_greenlets) def get_mongos_seed_list(): members = routers.keys() return ','.join(members) def kill_mongos(host): kill_members([host], 2, hosts=routers) return host def restart_mongos(host): restart_members([host], True) def get_members_in_state(state): status = get_client().admin.command('replSetGetStatus') members = status['members'] return [k['name'] for k in members if k['state'] == state] def get_primary(): try: primaries = get_members_in_state(1) assert len(primaries) <= 1 if primaries: return primaries[0] except pymongo.errors.ConnectionFailure: pass return None def get_random_secondary(): secondaries = get_members_in_state(2) if len(secondaries): return random.choice(secondaries) return None def get_secondaries(): return get_members_in_state(2) def get_arbiters(): return get_members_in_state(7) def get_recovering(): return get_members_in_state(3) def get_passives(): return get_client().admin.command('ismaster').get('passives', []) def get_hosts(): return get_client().admin.command('ismaster').get('hosts', []) def get_hidden_members(): # Both 'hidden' and 'slaveDelay' secondaries = get_secondaries() readers = get_hosts() + get_passives() for member in readers: try: secondaries.remove(member) except: # Skip primary pass return secondaries def get_tags(member): config = get_client().local.system.replset.find_one() for m in config['members']: if m['host'] == member: return m.get('tags', {}) raise Exception('member %s not in config' % repr(member)) def kill_primary(sig=2): primary = get_primary() kill_members([primary], sig) return primary def kill_secondary(sig=2): secondary = get_random_secondary() kill_members([secondary], sig) return secondary def kill_all_secondaries(sig=2): secondaries = get_secondaries() kill_members(secondaries, sig) return secondaries def stepdown_primary(): primary = get_primary() if primary: if ha_tools_debug: print('stepping down primary: %s' % (primary,)) c = pymongo.MongoClient(primary, use_greenlets=use_greenlets) # replSetStepDown causes mongod to close all connections try: c.admin.command('replSetStepDown', 20) except Exception: if ha_tools_debug: exc = sys.exc_info()[1] print('Exception from replSetStepDown: %s' % (exc.message,)) if ha_tools_debug: print('\tcalled replSetStepDown') elif ha_tools_debug: print('stepdown_primary() found no primary') def set_maintenance(member, value): """Put a member into RECOVERING state if value is True, else normal state. """ c = pymongo.MongoClient(member, use_greenlets=use_greenlets) c.admin.command('replSetMaintenance', value) start = time.time() while value != (member in get_recovering()): assert (time.time() - start) <= 10, ( "Member %s never switched state" % member) time.sleep(0.25) def restart_members(members, router=False): restarted = [] for member in members: if router: cmd = routers[member]['cmd'] else: cmd = nodes[member]['cmd'] proc = start_subprocess(cmd) if router: routers[member]['proc'] = proc else: nodes[member]['proc'] = proc res = wait_for(proc, int(member.split(':')[1])) if res: restarted.append(member) return restarted pymongo-2.6.3/test/high_availability/test_ha.py000066400000000000000000001050111223300253600216350ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test replica set operations and failures.""" # These test methods exuberantly violate the "one assert per test" rule, because # each method requires running setUp, which takes about 30 seconds to bring up # a replica set. Thus each method asserts everything we want to assert for a # given replica-set configuration. import itertools import time import unittest import ha_tools from ha_tools import use_greenlets from pymongo.errors import AutoReconnect, OperationFailure, ConnectionFailure from pymongo.mongo_replica_set_client import Member, Monitor from pymongo.mongo_replica_set_client import MongoReplicaSetClient from pymongo.mongo_client import MongoClient, _partition_node from pymongo.read_preferences import ReadPreference, modes from test import utils from test.utils import one # Will be imported from time or gevent, below sleep = None # Override default 30-second interval for faster testing Monitor._refresh_interval = MONITOR_INTERVAL = 0.5 # To make the code terser, copy modes into module scope PRIMARY = ReadPreference.PRIMARY PRIMARY_PREFERRED = ReadPreference.PRIMARY_PREFERRED SECONDARY = ReadPreference.SECONDARY SECONDARY_PREFERRED = ReadPreference.SECONDARY_PREFERRED NEAREST = ReadPreference.NEAREST def partition_nodes(nodes): """Translate from ['host:port', ...] to [(host, port), ...]""" return [_partition_node(node) for node in nodes] class HATestCase(unittest.TestCase): """A test case for connections to replica sets or mongos.""" def tearDown(self): ha_tools.kill_all_members() ha_tools.nodes.clear() ha_tools.routers.clear() sleep(1) # Let members really die. class TestDirectConnection(HATestCase): def setUp(self): members = [{}, {}, {'arbiterOnly': True}] res = ha_tools.start_replica_set(members) self.seed, self.name = res def test_secondary_connection(self): self.c = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) self.assertTrue(bool(len(self.c.secondaries))) db = self.c.pymongo_test # Wait for replication... w = len(self.c.secondaries) + 1 db.test.remove({}, w=w) db.test.insert({'foo': 'bar'}, w=w) # Test direct connection to a primary or secondary primary_host, primary_port = ha_tools.get_primary().split(':') primary_port = int(primary_port) (secondary_host, secondary_port) = ha_tools.get_secondaries()[0].split(':') secondary_port = int(secondary_port) arbiter_host, arbiter_port = ha_tools.get_arbiters()[0].split(':') arbiter_port = int(arbiter_port) # MongoClient succeeds no matter the read preference for kwargs in [ {'read_preference': PRIMARY}, {'read_preference': PRIMARY_PREFERRED}, {'read_preference': SECONDARY}, {'read_preference': SECONDARY_PREFERRED}, {'read_preference': NEAREST}, {'slave_okay': True} ]: client = MongoClient(primary_host, primary_port, use_greenlets=use_greenlets, **kwargs) self.assertEqual(primary_host, client.host) self.assertEqual(primary_port, client.port) self.assertTrue(client.is_primary) # Direct connection to primary can be queried with any read pref self.assertTrue(client.pymongo_test.test.find_one()) client = MongoClient(secondary_host, secondary_port, use_greenlets=use_greenlets, **kwargs) self.assertEqual(secondary_host, client.host) self.assertEqual(secondary_port, client.port) self.assertFalse(client.is_primary) # Direct connection to secondary can be queried with any read pref # but PRIMARY if kwargs.get('read_preference') != PRIMARY: self.assertTrue(client.pymongo_test.test.find_one()) else: self.assertRaises( AutoReconnect, client.pymongo_test.test.find_one) # Since an attempt at an acknowledged write to a secondary from a # direct connection raises AutoReconnect('not master'), MongoClient # should do the same for unacknowledged writes. try: client.pymongo_test.test.insert({}, w=0) except AutoReconnect, e: self.assertEqual('not master', e.args[0]) else: self.fail( 'Unacknowledged insert into secondary client %s should' 'have raised exception' % (client,)) # Test direct connection to an arbiter client = MongoClient(arbiter_host, arbiter_port, **kwargs) self.assertEqual(arbiter_host, client.host) self.assertEqual(arbiter_port, client.port) self.assertFalse(client.is_primary) # See explanation above try: client.pymongo_test.test.insert({}, w=0) except AutoReconnect, e: self.assertEqual('not master', e.args[0]) else: self.fail( 'Unacknowledged insert into arbiter client %s should' 'have raised exception' % (client,)) def tearDown(self): self.c.close() super(TestDirectConnection, self).tearDown() class TestPassiveAndHidden(HATestCase): def setUp(self): members = [{}, {'priority': 0}, {'arbiterOnly': True}, {'priority': 0, 'hidden': True}, {'priority': 0, 'slaveDelay': 5} ] res = ha_tools.start_replica_set(members) self.seed, self.name = res def test_passive_and_hidden(self): self.c = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) passives = ha_tools.get_passives() passives = partition_nodes(passives) self.assertEqual(self.c.secondaries, set(passives)) for mode in SECONDARY, SECONDARY_PREFERRED: utils.assertReadFromAll(self, self.c, passives, mode) ha_tools.kill_members(ha_tools.get_passives(), 2) sleep(2 * MONITOR_INTERVAL) utils.assertReadFrom(self, self.c, self.c.primary, SECONDARY_PREFERRED) def tearDown(self): self.c.close() super(TestPassiveAndHidden, self).tearDown() class TestMonitorRemovesRecoveringMember(HATestCase): # Members in STARTUP2 or RECOVERING states are shown in the primary's # isMaster response, but aren't secondaries and shouldn't be read from. # Verify that if a secondary goes into RECOVERING mode, the Monitor removes # it from the set of readers. def setUp(self): members = [{}, {'priority': 0}, {'priority': 0}] res = ha_tools.start_replica_set(members) self.seed, self.name = res def test_monitor_removes_recovering_member(self): self.c = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) secondaries = ha_tools.get_secondaries() for mode in SECONDARY, SECONDARY_PREFERRED: partitioned_secondaries = partition_nodes(secondaries) utils.assertReadFromAll(self, self.c, partitioned_secondaries, mode) secondary, recovering_secondary = secondaries ha_tools.set_maintenance(recovering_secondary, True) sleep(2 * MONITOR_INTERVAL) for mode in SECONDARY, SECONDARY_PREFERRED: # Don't read from recovering member utils.assertReadFrom(self, self.c, _partition_node(secondary), mode) def tearDown(self): self.c.close() super(TestMonitorRemovesRecoveringMember, self).tearDown() class TestTriggeredRefresh(HATestCase): # Verify that if a secondary goes into RECOVERING mode or if the primary # changes, the next exception triggers an immediate refresh. def setUp(self): members = [{}, {}] res = ha_tools.start_replica_set(members) self.seed, self.name = res # Disable periodic refresh Monitor._refresh_interval = 1e6 def test_recovering_member_triggers_refresh(self): # To test that find_one() and count() trigger immediate refreshes, # we'll create a separate client for each self.c_find_one, self.c_count = [ MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets, read_preference=SECONDARY) for _ in xrange(2)] # We've started the primary and one secondary primary = ha_tools.get_primary() secondary = ha_tools.get_secondaries()[0] # Pre-condition: just make sure they all connected OK for c in self.c_find_one, self.c_count: self.assertEqual(one(c.secondaries), _partition_node(secondary)) ha_tools.set_maintenance(secondary, True) # Trigger a refresh in various ways self.assertRaises(AutoReconnect, self.c_find_one.test.test.find_one) self.assertRaises(AutoReconnect, self.c_count.test.test.count) # Wait for the immediate refresh to complete - we're not waiting for # the periodic refresh, which has been disabled sleep(1) for c in self.c_find_one, self.c_count: self.assertFalse(c.secondaries) self.assertEqual(_partition_node(primary), c.primary) def test_stepdown_triggers_refresh(self): c_find_one = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) # We've started the primary and one secondary primary = ha_tools.get_primary() secondary = ha_tools.get_secondaries()[0] self.assertEqual( one(c_find_one.secondaries), _partition_node(secondary)) ha_tools.stepdown_primary() # Make sure the stepdown completes sleep(1) # Trigger a refresh self.assertRaises(AutoReconnect, c_find_one.test.test.find_one) # Wait for the immediate refresh to complete - we're not waiting for # the periodic refresh, which has been disabled sleep(1) # We've detected the stepdown self.assertTrue( not c_find_one.primary or _partition_node(primary) != c_find_one.primary) def tearDown(self): Monitor._refresh_interval = MONITOR_INTERVAL super(TestTriggeredRefresh, self).tearDown() class TestHealthMonitor(HATestCase): def setUp(self): res = ha_tools.start_replica_set([{}, {}, {}]) self.seed, self.name = res def test_primary_failure(self): c = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) self.assertTrue(bool(len(c.secondaries))) primary = c.primary secondaries = c.secondaries # Wait for new primary to be elected def primary_changed(): for _ in xrange(30): if c.primary and c.primary != primary: return True sleep(1) return False killed = ha_tools.kill_primary() self.assertTrue(bool(len(killed))) self.assertTrue(primary_changed()) self.assertNotEqual(secondaries, c.secondaries) def test_secondary_failure(self): c = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) self.assertTrue(bool(len(c.secondaries))) primary = c.primary secondaries = c.secondaries def readers_changed(): for _ in xrange(20): if c.secondaries != secondaries: return True sleep(1) return False killed = ha_tools.kill_secondary() sleep(2 * MONITOR_INTERVAL) self.assertTrue(bool(len(killed))) self.assertEqual(primary, c.primary) self.assertTrue(readers_changed()) secondaries = c.secondaries ha_tools.restart_members([killed]) self.assertEqual(primary, c.primary) self.assertTrue(readers_changed()) def test_primary_stepdown(self): c = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) self.assertTrue(bool(len(c.secondaries))) primary = c.primary ha_tools.stepdown_primary() # Wait for new primary patience_seconds = 30 for _ in xrange(patience_seconds): sleep(1) rs_state = c._MongoReplicaSetClient__rs_state if rs_state.writer and rs_state.writer != primary: # New primary stepped up new_primary = _partition_node(ha_tools.get_primary()) self.assertEqual(new_primary, rs_state.writer) new_secondaries = partition_nodes(ha_tools.get_secondaries()) self.assertEqual(set(new_secondaries), rs_state.secondaries) break else: self.fail( "No new primary after %s seconds. Old primary was %s, current" " is %s" % (patience_seconds, primary, ha_tools.get_primary())) class TestWritesWithFailover(HATestCase): def setUp(self): res = ha_tools.start_replica_set([{}, {}, {}]) self.seed, self.name = res def test_writes_with_failover(self): c = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) primary = c.primary db = c.pymongo_test w = len(c.secondaries) + 1 db.test.remove({}, w=w) db.test.insert({'foo': 'bar'}, w=w) self.assertEqual('bar', db.test.find_one()['foo']) def try_write(): for _ in xrange(30): try: db.test.insert({'bar': 'baz'}) return True except AutoReconnect: sleep(1) return False killed = ha_tools.kill_primary(9) self.assertTrue(bool(len(killed))) self.assertTrue(try_write()) self.assertTrue(primary != c.primary) self.assertEqual('baz', db.test.find_one({'bar': 'baz'})['bar']) class TestReadWithFailover(HATestCase): def setUp(self): res = ha_tools.start_replica_set([{}, {}, {}]) self.seed, self.name = res def test_read_with_failover(self): c = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) self.assertTrue(bool(len(c.secondaries))) def iter_cursor(cursor): for _ in cursor: pass return True db = c.pymongo_test w = len(c.secondaries) + 1 db.test.remove({}, w=w) # Force replication db.test.insert([{'foo': i} for i in xrange(10)], w=w) self.assertEqual(10, db.test.count()) db.read_preference = SECONDARY_PREFERRED cursor = db.test.find().batch_size(5) cursor.next() self.assertEqual(5, cursor._Cursor__retrieved) self.assertTrue(cursor._Cursor__connection_id in c.secondaries) ha_tools.kill_primary() # Primary failure shouldn't interrupt the cursor self.assertTrue(iter_cursor(cursor)) self.assertEqual(10, cursor._Cursor__retrieved) class TestReadPreference(HATestCase): def setUp(self): members = [ # primary {'tags': {'dc': 'ny', 'name': 'primary'}}, # secondary {'tags': {'dc': 'la', 'name': 'secondary'}, 'priority': 0}, # other_secondary {'tags': {'dc': 'ny', 'name': 'other_secondary'}, 'priority': 0}, ] res = ha_tools.start_replica_set(members) self.seed, self.name = res primary = ha_tools.get_primary() self.primary = _partition_node(primary) self.primary_tags = ha_tools.get_tags(primary) # Make sure priority worked self.assertEqual('primary', self.primary_tags['name']) self.primary_dc = {'dc': self.primary_tags['dc']} secondaries = ha_tools.get_secondaries() (secondary, ) = [ s for s in secondaries if ha_tools.get_tags(s)['name'] == 'secondary'] self.secondary = _partition_node(secondary) self.secondary_tags = ha_tools.get_tags(secondary) self.secondary_dc = {'dc': self.secondary_tags['dc']} (other_secondary, ) = [ s for s in secondaries if ha_tools.get_tags(s)['name'] == 'other_secondary'] self.other_secondary = _partition_node(other_secondary) self.other_secondary_tags = ha_tools.get_tags(other_secondary) self.other_secondary_dc = {'dc': self.other_secondary_tags['dc']} self.c = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) self.db = self.c.pymongo_test self.w = len(self.c.secondaries) + 1 self.db.test.remove({}, w=self.w) self.db.test.insert( [{'foo': i} for i in xrange(10)], w=self.w) self.clear_ping_times() def set_ping_time(self, host, ping_time_seconds): Member._host_to_ping_time[host] = ping_time_seconds def clear_ping_times(self): Member._host_to_ping_time.clear() def test_read_preference(self): # We pass through four states: # # 1. A primary and two secondaries # 2. Primary down # 3. Primary up, one secondary down # 4. Primary up, all secondaries down # # For each state, we verify the behavior of PRIMARY, # PRIMARY_PREFERRED, SECONDARY, SECONDARY_PREFERRED, and NEAREST c = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) def assertReadFrom(member, *args, **kwargs): utils.assertReadFrom(self, c, member, *args, **kwargs) def assertReadFromAll(members, *args, **kwargs): utils.assertReadFromAll(self, c, members, *args, **kwargs) def unpartition_node(node): host, port = node return '%s:%s' % (host, port) # To make the code terser, copy hosts into local scope primary = self.primary secondary = self.secondary other_secondary = self.other_secondary bad_tag = {'bad': 'tag'} # 1. THREE MEMBERS UP ------------------------------------------------- # PRIMARY assertReadFrom(primary, PRIMARY) # PRIMARY_PREFERRED # Trivial: mode and tags both match assertReadFrom(primary, PRIMARY_PREFERRED, self.primary_dc) # Secondary matches but not primary, choose primary assertReadFrom(primary, PRIMARY_PREFERRED, self.secondary_dc) # Chooses primary, ignoring tag sets assertReadFrom(primary, PRIMARY_PREFERRED, self.primary_dc) # Chooses primary, ignoring tag sets assertReadFrom(primary, PRIMARY_PREFERRED, bad_tag) assertReadFrom(primary, PRIMARY_PREFERRED, [bad_tag, {}]) # SECONDARY assertReadFromAll([secondary, other_secondary], SECONDARY) # SECONDARY_PREFERRED assertReadFromAll([secondary, other_secondary], SECONDARY_PREFERRED) # Multiple tags assertReadFrom(secondary, SECONDARY_PREFERRED, self.secondary_tags) # Fall back to primary if it's the only one matching the tags assertReadFrom(primary, SECONDARY_PREFERRED, {'name': 'primary'}) # No matching secondaries assertReadFrom(primary, SECONDARY_PREFERRED, bad_tag) # Fall back from non-matching tag set to matching set assertReadFromAll([secondary, other_secondary], SECONDARY_PREFERRED, [bad_tag, {}]) assertReadFrom(other_secondary, SECONDARY_PREFERRED, [bad_tag, {'dc': 'ny'}]) # NEAREST self.clear_ping_times() assertReadFromAll([primary, secondary, other_secondary], NEAREST) assertReadFromAll([primary, other_secondary], NEAREST, [bad_tag, {'dc': 'ny'}]) self.set_ping_time(primary, 0) self.set_ping_time(secondary, .03) # 30 ms self.set_ping_time(other_secondary, 10) # Nearest member, no tags assertReadFrom(primary, NEAREST) # Tags override nearness assertReadFrom(primary, NEAREST, {'name': 'primary'}) assertReadFrom(secondary, NEAREST, self.secondary_dc) # Make secondary fast self.set_ping_time(primary, .03) # 30 ms self.set_ping_time(secondary, 0) assertReadFrom(secondary, NEAREST) # Other secondary fast self.set_ping_time(secondary, 10) self.set_ping_time(other_secondary, 0) assertReadFrom(other_secondary, NEAREST) # High secondaryAcceptableLatencyMS, should read from all members assertReadFromAll( [primary, secondary, other_secondary], NEAREST, secondary_acceptable_latency_ms=1000*1000) self.clear_ping_times() assertReadFromAll([primary, other_secondary], NEAREST, [{'dc': 'ny'}]) # 2. PRIMARY DOWN ----------------------------------------------------- killed = ha_tools.kill_primary() # Let monitor notice primary's gone sleep(2 * MONITOR_INTERVAL) # PRIMARY assertReadFrom(None, PRIMARY) # PRIMARY_PREFERRED # No primary, choose matching secondary assertReadFromAll([secondary, other_secondary], PRIMARY_PREFERRED) assertReadFrom(secondary, PRIMARY_PREFERRED, {'name': 'secondary'}) # No primary or matching secondary assertReadFrom(None, PRIMARY_PREFERRED, bad_tag) # SECONDARY assertReadFromAll([secondary, other_secondary], SECONDARY) # Only primary matches assertReadFrom(None, SECONDARY, {'name': 'primary'}) # No matching secondaries assertReadFrom(None, SECONDARY, bad_tag) # SECONDARY_PREFERRED assertReadFromAll([secondary, other_secondary], SECONDARY_PREFERRED) # Mode and tags both match assertReadFrom(secondary, SECONDARY_PREFERRED, {'name': 'secondary'}) # NEAREST self.clear_ping_times() assertReadFromAll([secondary, other_secondary], NEAREST) # 3. PRIMARY UP, ONE SECONDARY DOWN ----------------------------------- ha_tools.restart_members([killed]) for _ in range(30): if ha_tools.get_primary(): break sleep(1) else: self.fail("Primary didn't come back up") ha_tools.kill_members([unpartition_node(secondary)], 2) self.assertTrue(MongoClient( unpartition_node(primary), use_greenlets=use_greenlets, read_preference=PRIMARY_PREFERRED ).admin.command('ismaster')['ismaster']) sleep(2 * MONITOR_INTERVAL) # PRIMARY assertReadFrom(primary, PRIMARY) # PRIMARY_PREFERRED assertReadFrom(primary, PRIMARY_PREFERRED) # SECONDARY assertReadFrom(other_secondary, SECONDARY) assertReadFrom(other_secondary, SECONDARY, self.other_secondary_dc) # Only the down secondary matches assertReadFrom(None, SECONDARY, {'name': 'secondary'}) # SECONDARY_PREFERRED assertReadFrom(other_secondary, SECONDARY_PREFERRED) assertReadFrom( other_secondary, SECONDARY_PREFERRED, self.other_secondary_dc) # The secondary matching the tag is down, use primary assertReadFrom(primary, SECONDARY_PREFERRED, {'name': 'secondary'}) # NEAREST assertReadFromAll([primary, other_secondary], NEAREST) assertReadFrom(other_secondary, NEAREST, {'name': 'other_secondary'}) assertReadFrom(primary, NEAREST, {'name': 'primary'}) # 4. PRIMARY UP, ALL SECONDARIES DOWN --------------------------------- ha_tools.kill_members([unpartition_node(other_secondary)], 2) self.assertTrue(MongoClient( unpartition_node(primary), use_greenlets=use_greenlets, read_preference=PRIMARY_PREFERRED ).admin.command('ismaster')['ismaster']) # PRIMARY assertReadFrom(primary, PRIMARY) # PRIMARY_PREFERRED assertReadFrom(primary, PRIMARY_PREFERRED) assertReadFrom(primary, PRIMARY_PREFERRED, self.secondary_dc) # SECONDARY assertReadFrom(None, SECONDARY) assertReadFrom(None, SECONDARY, self.other_secondary_dc) assertReadFrom(None, SECONDARY, {'dc': 'ny'}) # SECONDARY_PREFERRED assertReadFrom(primary, SECONDARY_PREFERRED) assertReadFrom(primary, SECONDARY_PREFERRED, self.secondary_dc) assertReadFrom(primary, SECONDARY_PREFERRED, {'name': 'secondary'}) assertReadFrom(primary, SECONDARY_PREFERRED, {'dc': 'ny'}) # NEAREST assertReadFrom(primary, NEAREST) assertReadFrom(None, NEAREST, self.secondary_dc) assertReadFrom(None, NEAREST, {'name': 'secondary'}) # Even if primary's slow, still read from it self.set_ping_time(primary, 100) assertReadFrom(primary, NEAREST) assertReadFrom(None, NEAREST, self.secondary_dc) self.clear_ping_times() def test_pinning(self): # To make the code terser, copy modes into local scope PRIMARY = ReadPreference.PRIMARY PRIMARY_PREFERRED = ReadPreference.PRIMARY_PREFERRED SECONDARY = ReadPreference.SECONDARY SECONDARY_PREFERRED = ReadPreference.SECONDARY_PREFERRED NEAREST = ReadPreference.NEAREST c = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets, auto_start_request=True) # Verify that changing the mode unpins the member. We'll try it for # every relevant change of mode. for mode0, mode1 in itertools.permutations( (PRIMARY, SECONDARY, SECONDARY_PREFERRED, NEAREST), 2 ): # Try reading and then changing modes and reading again, see if we # read from a different host for _ in range(1000): # pin to this host host = utils.read_from_which_host(c, mode0) # unpin? new_host = utils.read_from_which_host(c, mode1) if host != new_host: # Reading with a different mode unpinned, hooray! break else: self.fail( "Changing from mode %s to mode %s never unpinned" % ( modes[mode0], modes[mode1])) # Now verify changing the tag_sets unpins the member. tags0 = [{'a': 'a'}, {}] tags1 = [{'a': 'x'}, {}] for _ in range(1000): host = utils.read_from_which_host(c, NEAREST, tags0) new_host = utils.read_from_which_host(c, NEAREST, tags1) if host != new_host: break else: self.fail( "Changing from tags %s to tags %s never unpinned" % ( tags0, tags1)) # Finally, verify changing the secondary_acceptable_latency_ms unpins # the member. for _ in range(1000): host = utils.read_from_which_host(c, SECONDARY, None, 15) new_host = utils.read_from_which_host(c, SECONDARY, None, 20) if host != new_host: break else: self.fail( "Changing secondary_acceptable_latency_ms from 15 to 20" " never unpinned") def tearDown(self): self.c.close() super(TestReadPreference, self).tearDown() class TestReplicaSetAuth(HATestCase): def setUp(self): members = [ {}, {'priority': 0}, {'priority': 0}, ] res = ha_tools.start_replica_set(members, auth=True) self.c = MongoReplicaSetClient(res[0], replicaSet=res[1], use_greenlets=use_greenlets) # Add an admin user to enable auth self.c.admin.add_user('admin', 'adminpass') self.c.admin.authenticate('admin', 'adminpass') self.db = self.c.pymongo_ha_auth self.db.add_user('user', 'userpass') self.c.admin.logout() def test_auth_during_failover(self): self.assertTrue(self.db.authenticate('user', 'userpass')) self.assertTrue(self.db.foo.insert({'foo': 'bar'}, safe=True, w=3, wtimeout=3000)) self.db.logout() self.assertRaises(OperationFailure, self.db.foo.find_one) primary = '%s:%d' % self.c.primary ha_tools.kill_members([primary], 2) # Let monitor notice primary's gone sleep(2 * MONITOR_INTERVAL) # Make sure we can still authenticate self.assertTrue(self.db.authenticate('user', 'userpass')) # And still query. self.db.read_preference = PRIMARY_PREFERRED self.assertEqual('bar', self.db.foo.find_one()['foo']) def tearDown(self): self.c.close() super(TestReplicaSetAuth, self).tearDown() class TestAlive(HATestCase): def setUp(self): members = [{}, {}] self.seed, self.name = ha_tools.start_replica_set(members) def test_alive(self): primary = ha_tools.get_primary() secondary = ha_tools.get_random_secondary() primary_cx = MongoClient(primary, use_greenlets=use_greenlets) secondary_cx = MongoClient(secondary, use_greenlets=use_greenlets) rsc = MongoReplicaSetClient( self.seed, replicaSet=self.name, use_greenlets=use_greenlets) try: self.assertTrue(primary_cx.alive()) self.assertTrue(secondary_cx.alive()) self.assertTrue(rsc.alive()) ha_tools.kill_primary() time.sleep(0.5) self.assertFalse(primary_cx.alive()) self.assertTrue(secondary_cx.alive()) self.assertFalse(rsc.alive()) ha_tools.kill_members([secondary], 2) time.sleep(0.5) self.assertFalse(primary_cx.alive()) self.assertFalse(secondary_cx.alive()) self.assertFalse(rsc.alive()) finally: rsc.close() class TestMongosHighAvailability(HATestCase): def setUp(self): seed_list = ha_tools.create_sharded_cluster() self.dbname = 'pymongo_mongos_ha' self.client = MongoClient(seed_list) self.client.drop_database(self.dbname) def test_mongos_ha(self): coll = self.client[self.dbname].test self.assertTrue(coll.insert({'foo': 'bar'})) first = '%s:%d' % (self.client.host, self.client.port) ha_tools.kill_mongos(first) # Fail first attempt self.assertRaises(AutoReconnect, coll.count) # Find new mongos self.assertEqual(1, coll.count()) second = '%s:%d' % (self.client.host, self.client.port) self.assertNotEqual(first, second) ha_tools.kill_mongos(second) # Fail first attempt self.assertRaises(AutoReconnect, coll.count) # Find new mongos self.assertEqual(1, coll.count()) third = '%s:%d' % (self.client.host, self.client.port) self.assertNotEqual(second, third) ha_tools.kill_mongos(third) # Fail first attempt self.assertRaises(AutoReconnect, coll.count) # We've killed all three, restart one. ha_tools.restart_mongos(first) # Find new mongos self.assertEqual(1, coll.count()) def tearDown(self): self.client.drop_database(self.dbname) super(TestMongosHighAvailability, self).tearDown() class TestReplicaSetRequest(HATestCase): def setUp(self): members = [{}, {}, {'arbiterOnly': True}] res = ha_tools.start_replica_set(members) self.c = MongoReplicaSetClient(res[0], replicaSet=res[1], use_greenlets=use_greenlets, auto_start_request=True) def test_request_during_failover(self): primary = _partition_node(ha_tools.get_primary()) secondary = _partition_node(ha_tools.get_random_secondary()) self.assertTrue(self.c.auto_start_request) self.assertTrue(self.c.in_request()) rs_state = self.c._MongoReplicaSetClient__rs_state primary_pool = rs_state.get(primary).pool secondary_pool = rs_state.get(secondary).pool # Trigger start_request on primary pool utils.assertReadFrom(self, self.c, primary, PRIMARY) self.assertTrue(primary_pool.in_request()) # Fail over ha_tools.kill_primary() patience_seconds = 60 for _ in range(patience_seconds): sleep(1) try: if ha_tools.ha_tools_debug: print 'Waiting for failover' if ha_tools.get_primary(): # We have a new primary break except ConnectionFailure: pass else: self.fail("Problem with test: No new primary after %s seconds" % patience_seconds) try: # Trigger start_request on secondary_pool, which is becoming new # primary self.c.test.test.find_one() except AutoReconnect: # We've noticed the failover now pass # The old secondary is now primary utils.assertReadFrom(self, self.c, secondary, PRIMARY) self.assertTrue(self.c.in_request()) self.assertTrue(secondary_pool.in_request()) def tearDown(self): self.c.close() super(TestReplicaSetRequest, self).tearDown() if __name__ == '__main__': if use_greenlets: print('Using Gevent') import gevent print('gevent version %s' % gevent.__version__) from gevent import monkey monkey.patch_socket() sleep = gevent.sleep else: sleep = time.sleep unittest.main() pymongo-2.6.3/test/mod_wsgi_test/000077500000000000000000000000001223300253600170345ustar00rootroot00000000000000pymongo-2.6.3/test/mod_wsgi_test/test_client.py000066400000000000000000000111041223300253600217200ustar00rootroot00000000000000# Copyright 2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test client for mod_wsgi application, see bug PYTHON-353. """ import sys import urllib2 import thread import threading import time from optparse import OptionParser def parse_args(): parser = OptionParser("""usage: %prog [options] mode url mode:\tparallel or serial""") # Should be enough that any connection leak will exhaust available file # descriptors. parser.add_option( "-n", "--nrequests", type="int", dest="nrequests", default=50 * 1000, help="Number of times to GET the URL, in total") parser.add_option( "-t", "--nthreads", type="int", dest="nthreads", default=100, help="Number of threads with mode 'parallel'") parser.add_option( "-q", "--quiet", action="store_false", dest="verbose", default=True, help="Don't print status messages to stdout") parser.add_option( "-c", "--continue", action="store_true", dest="continue_", default=False, help="Continue after HTTP errors") try: options, (mode, url) = parser.parse_args() except ValueError: parser.print_usage() sys.exit(1) if mode not in ('parallel', 'serial'): parser.print_usage() sys.exit(1) return options, mode, url def get(url): urllib2.urlopen(url).read().strip() class URLGetterThread(threading.Thread): # Class variables. counter_lock = threading.Lock() counter = 0 def __init__(self, options, url, nrequests_per_thread): super(URLGetterThread, self).__init__() self.options = options self.url = url self.nrequests_per_thread = nrequests_per_thread self.errors = 0 def run(self): for i in range(self.nrequests_per_thread): try: get(url) except Exception, e: print e if not options.continue_: thread.interrupt_main() thread.exit() self.errors += 1 URLGetterThread.counter_lock.acquire() URLGetterThread.counter += 1 counter = URLGetterThread.counter URLGetterThread.counter_lock.release() should_print = options.verbose and not counter % 1000 if should_print: print counter def main(options, mode, url): start_time = time.time() errors = 0 if mode == 'parallel': nrequests_per_thread = options.nrequests / options.nthreads if options.verbose: print ( 'Getting %s %s times total in %s threads, ' '%s times per thread' % ( url, nrequests_per_thread * options.nthreads, options.nthreads, nrequests_per_thread)) threads = [ URLGetterThread(options, url, nrequests_per_thread) for _ in range(options.nthreads) ] for t in threads: t.start() for t in threads: t.join() errors = sum([t.errors for t in threads]) nthreads_with_errors = len([t for t in threads if t.errors]) if nthreads_with_errors: print '%d threads had errors! %d errors in total' % ( nthreads_with_errors, errors) else: assert mode == 'serial' if options.verbose: print 'Getting %s %s times in one thread' % ( url, options.nrequests ) for i in range(1, options.nrequests + 1): try: get(url) except Exception, e: print e if not options.continue_: sys.exit(1) errors += 1 if options.verbose and not i % 1000: print i if errors: print '%d errors!' % errors if options.verbose: print 'Completed in %.2f seconds' % (time.time() - start_time) if errors: # Failure sys.exit(1) if __name__ == '__main__': options, mode, url = parse_args() main(options, mode, url) pymongo-2.6.3/test/qcheck.py000066400000000000000000000167261223300253600160110ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import random import traceback import datetime import re import sys sys.path[0:0] = [""] from bson.binary import Binary from bson.dbref import DBRef from bson.objectid import ObjectId from bson.py3compat import b, binary_type from bson.son import SON gen_target = 100 reduction_attempts = 10 examples = 5 PY3 = sys.version_info[0] == 3 def lift(value): return lambda: value def choose_lifted(generator_list): return lambda: random.choice(generator_list) def my_map(generator, function): return lambda: function(generator()) def choose(list): return lambda: random.choice(list)() def gen_range(start, stop): return lambda: random.randint(start, stop) def gen_int(): max_int = 2147483647 return lambda: random.randint(-max_int - 1, max_int) def gen_float(): return lambda: (random.random() - 0.5) * sys.maxint def gen_boolean(): return lambda: random.choice([True, False]) def gen_printable_char(): return lambda: chr(random.randint(32, 126)) def gen_printable_string(gen_length): return lambda: "".join(gen_list(gen_printable_char(), gen_length)()) if PY3: def gen_char(set=None): return lambda: bytes([random.randint(0, 255)]) else: def gen_char(set=None): return lambda: chr(random.randint(0, 255)) def gen_string(gen_length): return lambda: b("").join(gen_list(gen_char(), gen_length)()) def gen_unichar(): return lambda: unichr(random.randint(1, 0xFFF)) def gen_unicode(gen_length): return lambda: u"".join([x for x in gen_list(gen_unichar(), gen_length)() if x not in ".$"]) def gen_list(generator, gen_length): return lambda: [generator() for _ in range(gen_length())] def gen_datetime(): return lambda: datetime.datetime(random.randint(1970, 2037), random.randint(1, 12), random.randint(1, 28), random.randint(0, 23), random.randint(0, 59), random.randint(0, 59), random.randint(0, 999) * 1000) def gen_dict(gen_key, gen_value, gen_length): def a_dict(gen_key, gen_value, length): result = {} for _ in range(length): result[gen_key()] = gen_value() return result return lambda: a_dict(gen_key, gen_value, gen_length()) def gen_regexp(gen_length): # TODO our patterns only consist of one letter. # this is because of a bug in CPython's regex equality testing, # which I haven't quite tracked down, so I'm just ignoring it... pattern = lambda: u"".join(gen_list(choose_lifted(u"a"), gen_length)()) def gen_flags(): flags = 0 if random.random() > 0.5: flags = flags | re.IGNORECASE if random.random() > 0.5: flags = flags | re.MULTILINE if random.random() > 0.5: flags = flags | re.VERBOSE return flags return lambda: re.compile(pattern(), gen_flags()) def gen_objectid(): return lambda: ObjectId() def gen_dbref(): collection = gen_unicode(gen_range(0, 20)) return lambda: DBRef(collection(), gen_mongo_value(1, True)()) def gen_mongo_value(depth, ref): bintype = Binary if PY3: # If we used Binary in python3 tests would fail since we # decode BSON binary subtype 0 to bytes. Testing this with # bytes in python3 makes a lot more sense. # binary_type is `str` in python 2, `bytes` in python 3. bintype = binary_type choices = [gen_unicode(gen_range(0, 50)), gen_printable_string(gen_range(0, 50)), my_map(gen_string(gen_range(0, 1000)), bintype), gen_int(), gen_float(), gen_boolean(), gen_datetime(), gen_objectid(), lift(None)] if ref: choices.append(gen_dbref()) if depth > 0: choices.append(gen_mongo_list(depth, ref)) choices.append(gen_mongo_dict(depth, ref)) return choose(choices) def gen_mongo_list(depth, ref): return gen_list(gen_mongo_value(depth - 1, ref), gen_range(0, 10)) def gen_mongo_dict(depth, ref=True): return my_map(gen_dict(gen_unicode(gen_range(0, 20)), gen_mongo_value(depth - 1, ref), gen_range(0, 10)), SON) def simplify(case): # TODO this is a hack if isinstance(case, SON) and "$ref" not in case: simplified = SON(case) # make a copy! if random.choice([True, False]): # delete if not len(simplified.keys()): return (False, case) del simplified[random.choice(simplified.keys())] return (True, simplified) else: # simplify a value if not len(simplified.items()): return (False, case) (key, value) = random.choice(simplified.items()) (success, value) = simplify(value) simplified[key] = value return (success, success and simplified or case) if isinstance(case, list): simplified = list(case) if random.choice([True, False]): # delete if not len(simplified): return (False, case) simplified.pop(random.randrange(len(simplified))) return (True, simplified) else: # simplify an item if not len(simplified): return (False, case) index = random.randrange(len(simplified)) (success, value) = simplify(simplified[index]) simplified[index] = value return (success, success and simplified or case) return (False, case) def reduce(case, predicate, reductions=0): for _ in range(reduction_attempts): (reduced, simplified) = simplify(case) if reduced and not predicate(simplified): return reduce(simplified, predicate, reductions + 1) return (reductions, case) def isnt(predicate): return lambda x: not predicate(x) def check(predicate, generator): counter_examples = [] for _ in range(gen_target): case = generator() try: if not predicate(case): reduction = reduce(case, predicate) counter_examples.append("after %s reductions: %r" % reduction) except: counter_examples.append("%r : %s" % (case, traceback.format_exc())) return counter_examples def check_unittest(test, predicate, generator): counter_examples = check(predicate, generator) if counter_examples: failures = len(counter_examples) message = "\n".join([" -> %s" % f for f in counter_examples[:examples]]) message = ("found %d counter examples, displaying first %d:\n%s" % (failures, min(failures, examples), message)) test.fail(message) pymongo-2.6.3/test/slow/000077500000000000000000000000001223300253600151515ustar00rootroot00000000000000pymongo-2.6.3/test/slow/test_high_concurrency.py000066400000000000000000000024151223300253600221150ustar00rootroot00000000000000# Copyright 2013 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test built in connection-pooling with lots of threads.""" import unittest import sys sys.path[0:0] = [""] from test.test_pooling_base import _TestMaxPoolSize class TestPoolWithLotsOfThreads(_TestMaxPoolSize, unittest.TestCase): use_greenlets = False def test_max_pool_size_with_leaked_request_super_massive(self): # Like test_max_pool_size_with_leaked_request_massive but even more # threads. Tests that socket reclamation works under high load, # especially in Python <= 2.7.0. You may need to raise ulimit. # See http://bugs.python.org/issue1868. nthreads = 1000 self._test_max_pool_size( 2, 1, max_pool_size=2 * nthreads, nthreads=nthreads) pymongo-2.6.3/test/test_auth.py000066400000000000000000000334721223300253600165500ustar00rootroot00000000000000# Copyright 2013 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Authentication Tests.""" import os import sys import threading import unittest from urllib import quote_plus sys.path[0:0] = [""] from nose.plugins.skip import SkipTest from pymongo import MongoClient, MongoReplicaSetClient from pymongo.auth import HAVE_KERBEROS from pymongo.errors import OperationFailure from pymongo.read_preferences import ReadPreference from test import version, host, port from test.utils import is_mongos, server_started_with_auth # YOU MUST RUN KINIT BEFORE RUNNING GSSAPI TESTS. GSSAPI_HOST = os.environ.get('GSSAPI_HOST') GSSAPI_PORT = int(os.environ.get('GSSAPI_PORT', '27017')) PRINCIPAL = os.environ.get('PRINCIPAL') SASL_HOST = os.environ.get('SASL_HOST') SASL_PORT = int(os.environ.get('SASL_PORT', '27017')) SASL_USER = os.environ.get('SASL_USER') SASL_PASS = os.environ.get('SASL_PASS') SASL_DB = os.environ.get('SASL_DB', '$external') class AutoAuthenticateThread(threading.Thread): """Used in testing threaded authentication. """ def __init__(self, database): super(AutoAuthenticateThread, self).__init__() self.database = database self.success = True def run(self): try: self.database.command('dbstats') except OperationFailure: self.success = False class TestGSSAPI(unittest.TestCase): def setUp(self): if not HAVE_KERBEROS: raise SkipTest('Kerberos module not available.') if not GSSAPI_HOST or not PRINCIPAL: raise SkipTest('Must set GSSAPI_HOST and PRINCIPAL to test GSSAPI') def test_gssapi_simple(self): client = MongoClient(GSSAPI_HOST, GSSAPI_PORT) # Without gssapiServiceName self.assertTrue(client.test.authenticate(PRINCIPAL, mechanism='GSSAPI')) self.assertTrue(client.database_names()) uri = ('mongodb://%s@%s:%d/?authMechanism=' 'GSSAPI' % (quote_plus(PRINCIPAL), GSSAPI_HOST, GSSAPI_PORT)) client = MongoClient(uri) self.assertTrue(client.database_names()) # With gssapiServiceName self.assertTrue(client.test.authenticate(PRINCIPAL, mechanism='GSSAPI', gssapiServiceName='mongodb')) self.assertTrue(client.database_names()) uri = ('mongodb://%s@%s:%d/?authMechanism=' 'GSSAPI;gssapiServiceName=mongodb' % (quote_plus(PRINCIPAL), GSSAPI_HOST, GSSAPI_PORT)) client = MongoClient(uri) self.assertTrue(client.database_names()) set_name = client.admin.command('ismaster').get('setName') if set_name: client = MongoReplicaSetClient(GSSAPI_HOST, port=GSSAPI_PORT, replicaSet=set_name) # Without gssapiServiceName self.assertTrue(client.test.authenticate(PRINCIPAL, mechanism='GSSAPI')) self.assertTrue(client.database_names()) uri = ('mongodb://%s@%s:%d/?authMechanism=GSSAPI;replicaSet' '=%s' % (quote_plus(PRINCIPAL), GSSAPI_HOST, GSSAPI_PORT, str(set_name))) client = MongoReplicaSetClient(uri) self.assertTrue(client.database_names()) # With gssapiServiceName self.assertTrue(client.test.authenticate(PRINCIPAL, mechanism='GSSAPI', gssapiServiceName='mongodb')) self.assertTrue(client.database_names()) uri = ('mongodb://%s@%s:%d/?authMechanism=GSSAPI;replicaSet' '=%s;gssapiServiceName=mongodb' % (quote_plus(PRINCIPAL), GSSAPI_HOST, GSSAPI_PORT, str(set_name))) client = MongoReplicaSetClient(uri) self.assertTrue(client.database_names()) def test_gssapi_threaded(self): # Use auto_start_request=True to make sure each thread # uses a different socket. client = MongoClient(GSSAPI_HOST, auto_start_request=True) self.assertTrue(client.test.authenticate(PRINCIPAL, mechanism='GSSAPI')) threads = [] for _ in xrange(4): threads.append(AutoAuthenticateThread(client.foo)) for thread in threads: thread.start() for thread in threads: thread.join() self.assertTrue(thread.success) set_name = client.admin.command('ismaster').get('setName') if set_name: preference = ReadPreference.SECONDARY client = MongoReplicaSetClient(GSSAPI_HOST, replicaSet=set_name, read_preference=preference) self.assertTrue(client.test.authenticate(PRINCIPAL, mechanism='GSSAPI')) self.assertTrue(client.foo.command('dbstats')) threads = [] for _ in xrange(4): threads.append(AutoAuthenticateThread(client.foo)) for thread in threads: thread.start() for thread in threads: thread.join() self.assertTrue(thread.success) class TestSASL(unittest.TestCase): def setUp(self): if not SASL_HOST or not SASL_USER or not SASL_PASS: raise SkipTest('Must set SASL_HOST, ' 'SASL_USER, and SASL_PASS to test SASL') def test_sasl_plain(self): client = MongoClient(SASL_HOST, SASL_PORT) self.assertTrue(client.test.authenticate(SASL_USER, SASL_PASS, SASL_DB, 'PLAIN')) uri = ('mongodb://%s:%s@%s:%d/?authMechanism=PLAIN;' 'authSource=%s' % (quote_plus(SASL_USER), quote_plus(SASL_PASS), SASL_HOST, SASL_PORT, SASL_DB)) client = MongoClient(uri) set_name = client.admin.command('ismaster').get('setName') if set_name: client = MongoReplicaSetClient(SASL_HOST, port=SASL_PORT, replicaSet=set_name) self.assertTrue(client.test.authenticate(SASL_USER, SASL_PASS, SASL_DB, 'PLAIN')) uri = ('mongodb://%s:%s@%s:%d/?authMechanism=PLAIN;' 'authSource=%s;replicaSet=%s' % (quote_plus(SASL_USER), quote_plus(SASL_PASS), SASL_HOST, SASL_PORT, SASL_DB, str(set_name))) client = MongoReplicaSetClient(uri) class TestAuthURIOptions(unittest.TestCase): def setUp(self): client = MongoClient(host, port) # Sharded auth not supported before MongoDB 2.0 if is_mongos(client) and not version.at_least(client, (2, 0, 0)): raise SkipTest("Auth with sharding requires MongoDB >= 2.0.0") if not server_started_with_auth(client): raise SkipTest('Authentication is not enabled on server') response = client.admin.command('ismaster') self.set_name = str(response.get('setName', '')) client.pymongo_test.add_user('user', 'pass') client.admin.add_user('admin', 'pass') if self.set_name: # GLE requires authentication. client.admin.authenticate('admin', 'pass') # Make sure the admin user is replicated after calling add_user # above. This avoids a race in the MRSC tests below. Adding a # user is just an insert into system.users. client.admin.command('getLastError', w=len(response['hosts'])) self.client = client def tearDown(self): self.client.admin.authenticate('admin', 'pass') self.client.pymongo_test.system.users.remove() self.client.admin.system.users.remove() self.client.admin.logout() def test_uri_options(self): # Test default to admin client = MongoClient('mongodb://admin:pass@%s:%d' % (host, port)) self.assertTrue(client.admin.command('dbstats')) if self.set_name: uri = ('mongodb://admin:pass' '@%s:%d/?replicaSet=%s' % (host, port, self.set_name)) client = MongoReplicaSetClient(uri) self.assertTrue(client.admin.command('dbstats')) client.read_preference = ReadPreference.SECONDARY self.assertTrue(client.admin.command('dbstats')) # Test explicit database uri = 'mongodb://user:pass@%s:%d/pymongo_test' % (host, port) client = MongoClient(uri) self.assertRaises(OperationFailure, client.admin.command, 'dbstats') self.assertTrue(client.pymongo_test.command('dbstats')) if self.set_name: uri = ('mongodb://user:pass@%s:%d' '/pymongo_test?replicaSet=%s' % (host, port, self.set_name)) client = MongoReplicaSetClient(uri) self.assertRaises(OperationFailure, client.admin.command, 'dbstats') self.assertTrue(client.pymongo_test.command('dbstats')) client.read_preference = ReadPreference.SECONDARY self.assertTrue(client.pymongo_test.command('dbstats')) # Test authSource uri = ('mongodb://user:pass@%s:%d' '/pymongo_test2?authSource=pymongo_test' % (host, port)) client = MongoClient(uri) self.assertRaises(OperationFailure, client.pymongo_test2.command, 'dbstats') self.assertTrue(client.pymongo_test.command('dbstats')) if self.set_name: uri = ('mongodb://user:pass@%s:%d/pymongo_test2?replicaSet=' '%s;authSource=pymongo_test' % (host, port, self.set_name)) client = MongoReplicaSetClient(uri) self.assertRaises(OperationFailure, client.pymongo_test2.command, 'dbstats') self.assertTrue(client.pymongo_test.command('dbstats')) client.read_preference = ReadPreference.SECONDARY self.assertTrue(client.pymongo_test.command('dbstats')) class TestDelegatedAuth(unittest.TestCase): def setUp(self): self.client = MongoClient(host, port) if not version.at_least(self.client, (2, 4, 0)): raise SkipTest('Delegated authentication requires MongoDB >= 2.4.0') if not server_started_with_auth(self.client): raise SkipTest('Authentication is not enabled on server') # Give admin all priviledges. self.client.admin.add_user('admin', 'pass', roles=['readAnyDatabase', 'readWriteAnyDatabase', 'userAdminAnyDatabase', 'dbAdminAnyDatabase', 'clusterAdmin']) def tearDown(self): self.client.admin.authenticate('admin', 'pass') self.client.pymongo_test.system.users.remove() self.client.pymongo_test2.system.users.remove() self.client.pymongo_test2.foo.remove() self.client.admin.system.users.remove() self.client.admin.logout() def test_delegated_auth(self): self.client.admin.authenticate('admin', 'pass') self.client.pymongo_test2.foo.remove() self.client.pymongo_test2.foo.insert({}) # User definition with no roles in pymongo_test. self.client.pymongo_test.add_user('user', 'pass', roles=[]) # Delegate auth to pymongo_test. self.client.pymongo_test2.add_user('user', userSource='pymongo_test', roles=['read']) self.client.admin.logout() self.assertRaises(OperationFailure, self.client.pymongo_test2.foo.find_one) # Auth must occur on the db where the user is defined. self.assertRaises(OperationFailure, self.client.pymongo_test2.authenticate, 'user', 'pass') # Auth directly self.assertTrue(self.client.pymongo_test.authenticate('user', 'pass')) self.assertTrue(self.client.pymongo_test2.foo.find_one()) self.client.pymongo_test.logout() self.assertRaises(OperationFailure, self.client.pymongo_test2.foo.find_one) # Auth using source self.assertTrue(self.client.pymongo_test2.authenticate( 'user', 'pass', source='pymongo_test')) self.assertTrue(self.client.pymongo_test2.foo.find_one()) # Must logout from the db authenticate was called on. self.client.pymongo_test2.logout() self.assertRaises(OperationFailure, self.client.pymongo_test2.foo.find_one) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_binary.py000066400000000000000000000300721223300253600170640ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tests for the Binary wrapper.""" import base64 import copy import pickle import sys import unittest try: import uuid should_test_uuid = True except ImportError: should_test_uuid = False sys.path[0:0] = [""] import bson from bson.binary import * from bson.py3compat import b, binary_type from bson.son import SON from nose.plugins.skip import SkipTest from test.test_client import get_client class TestBinary(unittest.TestCase): def setUp(self): pass def test_binary(self): a_string = "hello world" a_binary = Binary(b("hello world")) self.assertTrue(a_binary.startswith(b("hello"))) self.assertTrue(a_binary.endswith(b("world"))) self.assertTrue(isinstance(a_binary, Binary)) self.assertFalse(isinstance(a_string, Binary)) def test_exceptions(self): self.assertRaises(TypeError, Binary, None) self.assertRaises(TypeError, Binary, u"hello") self.assertRaises(TypeError, Binary, 5) self.assertRaises(TypeError, Binary, 10.2) self.assertRaises(TypeError, Binary, b("hello"), None) self.assertRaises(TypeError, Binary, b("hello"), "100") self.assertRaises(ValueError, Binary, b("hello"), -1) self.assertRaises(ValueError, Binary, b("hello"), 256) self.assertTrue(Binary(b("hello"), 0)) self.assertTrue(Binary(b("hello"), 255)) def test_subtype(self): one = Binary(b("hello")) self.assertEqual(one.subtype, 0) two = Binary(b("hello"), 2) self.assertEqual(two.subtype, 2) three = Binary(b("hello"), 100) self.assertEqual(three.subtype, 100) def test_equality(self): two = Binary(b("hello")) three = Binary(b("hello"), 100) self.assertNotEqual(two, three) self.assertEqual(three, Binary(b("hello"), 100)) self.assertEqual(two, Binary(b("hello"))) self.assertNotEqual(two, Binary(b("hello "))) self.assertNotEqual(b("hello"), Binary(b("hello"))) # Explicitly test inequality self.assertFalse(three != Binary(b("hello"), 100)) self.assertFalse(two != Binary(b("hello"))) def test_repr(self): one = Binary(b("hello world")) self.assertEqual(repr(one), "Binary(%s, 0)" % (repr(b("hello world")),)) two = Binary(b("hello world"), 2) self.assertEqual(repr(two), "Binary(%s, 2)" % (repr(b("hello world")),)) three = Binary(b("\x08\xFF")) self.assertEqual(repr(three), "Binary(%s, 0)" % (repr(b("\x08\xFF")),)) four = Binary(b("\x08\xFF"), 2) self.assertEqual(repr(four), "Binary(%s, 2)" % (repr(b("\x08\xFF")),)) five = Binary(b("test"), 100) self.assertEqual(repr(five), "Binary(%s, 100)" % (repr(b("test")),)) def test_legacy_java_uuid(self): if not should_test_uuid: raise SkipTest("No uuid module") # Generated by the Java driver from_java = b('bAAAAAdfaWQAUCBQxkVm+XdxJ9tOBW5ld2d1aWQAEAAAAAMIQkfACFu' 'Z/0RustLOU/G6Am5ld2d1aWRzdHJpbmcAJQAAAGZmOTk1YjA4LWMwND' 'ctNDIwOC1iYWYxLTUzY2VkMmIyNmU0NAAAbAAAAAdfaWQAUCBQxkVm+' 'XdxJ9tPBW5ld2d1aWQAEAAAAANgS/xhRXXv8kfIec+dYdyCAm5ld2d1' 'aWRzdHJpbmcAJQAAAGYyZWY3NTQ1LTYxZmMtNGI2MC04MmRjLTYxOWR' 'jZjc5Yzg0NwAAbAAAAAdfaWQAUCBQxkVm+XdxJ9tQBW5ld2d1aWQAEA' 'AAAAPqREIbhZPUJOSdHCJIgaqNAm5ld2d1aWRzdHJpbmcAJQAAADI0Z' 'DQ5Mzg1LTFiNDItNDRlYS04ZGFhLTgxNDgyMjFjOWRlNAAAbAAAAAdf' 'aWQAUCBQxkVm+XdxJ9tRBW5ld2d1aWQAEAAAAANjQBn/aQuNfRyfNyx' '29COkAm5ld2d1aWRzdHJpbmcAJQAAADdkOGQwYjY5LWZmMTktNDA2My' '1hNDIzLWY0NzYyYzM3OWYxYwAAbAAAAAdfaWQAUCBQxkVm+XdxJ9tSB' 'W5ld2d1aWQAEAAAAAMtSv/Et1cAQUFHUYevqxaLAm5ld2d1aWRzdHJp' 'bmcAJQAAADQxMDA1N2I3LWM0ZmYtNGEyZC04YjE2LWFiYWY4NzUxNDc' '0MQAA') data = base64.b64decode(from_java) # Test decoding docs = bson.decode_all(data, SON, False, OLD_UUID_SUBTYPE) for d in docs: self.assertNotEqual(d['newguid'], uuid.UUID(d['newguidstring'])) docs = bson.decode_all(data, SON, False, UUID_SUBTYPE) for d in docs: self.assertNotEqual(d['newguid'], uuid.UUID(d['newguidstring'])) docs = bson.decode_all(data, SON, False, CSHARP_LEGACY) for d in docs: self.assertNotEqual(d['newguid'], uuid.UUID(d['newguidstring'])) docs = bson.decode_all(data, SON, False, JAVA_LEGACY) for d in docs: self.assertEqual(d['newguid'], uuid.UUID(d['newguidstring'])) # Test encoding encoded = b('').join([bson.BSON.encode(doc, uuid_subtype=OLD_UUID_SUBTYPE) for doc in docs]) self.assertNotEqual(data, encoded) encoded = b('').join([bson.BSON.encode(doc, uuid_subtype=UUID_SUBTYPE) for doc in docs]) self.assertNotEqual(data, encoded) encoded = b('').join([bson.BSON.encode(doc, uuid_subtype=CSHARP_LEGACY) for doc in docs]) self.assertNotEqual(data, encoded) encoded = b('').join([bson.BSON.encode(doc, uuid_subtype=JAVA_LEGACY) for doc in docs]) self.assertEqual(data, encoded) # Test insert and find client = get_client() client.pymongo_test.drop_collection('java_uuid') coll = client.pymongo_test.java_uuid coll.uuid_subtype = JAVA_LEGACY coll.insert(docs) self.assertEqual(5, coll.count()) for d in coll.find(): self.assertEqual(d['newguid'], uuid.UUID(d['newguidstring'])) coll.uuid_subtype = OLD_UUID_SUBTYPE for d in coll.find(): self.assertNotEqual(d['newguid'], d['newguidstring']) client.pymongo_test.drop_collection('java_uuid') def test_legacy_csharp_uuid(self): if not should_test_uuid: raise SkipTest("No uuid module") # Generated by the .net driver from_csharp = b('ZAAAABBfaWQAAAAAAAVuZXdndWlkABAAAAAD+MkoCd/Jy0iYJ7Vhl' 'iF3BAJuZXdndWlkc3RyaW5nACUAAAAwOTI4YzlmOC1jOWRmLTQ4Y2' 'ItOTgyNy1iNTYxOTYyMTc3MDQAAGQAAAAQX2lkAAEAAAAFbmV3Z3V' 'pZAAQAAAAA9MD0oXQe6VOp7mK4jkttWUCbmV3Z3VpZHN0cmluZwAl' 'AAAAODVkMjAzZDMtN2JkMC00ZWE1LWE3YjktOGFlMjM5MmRiNTY1A' 'ABkAAAAEF9pZAACAAAABW5ld2d1aWQAEAAAAAPRmIO2auc/Tprq1Z' 'oQ1oNYAm5ld2d1aWRzdHJpbmcAJQAAAGI2ODM5OGQxLWU3NmEtNGU' 'zZi05YWVhLWQ1OWExMGQ2ODM1OAAAZAAAABBfaWQAAwAAAAVuZXdn' 'dWlkABAAAAADISpriopuTEaXIa7arYOCFAJuZXdndWlkc3RyaW5nA' 'CUAAAA4YTZiMmEyMS02ZThhLTQ2NGMtOTcyMS1hZWRhYWQ4MzgyMT' 'QAAGQAAAAQX2lkAAQAAAAFbmV3Z3VpZAAQAAAAA98eg0CFpGlPihP' 'MwOmYGOMCbmV3Z3VpZHN0cmluZwAlAAAANDA4MzFlZGYtYTQ4NS00' 'ZjY5LThhMTMtY2NjMGU5OTgxOGUzAAA=') data = base64.b64decode(from_csharp) # Test decoding docs = bson.decode_all(data, SON, False, OLD_UUID_SUBTYPE) for d in docs: self.assertNotEqual(d['newguid'], uuid.UUID(d['newguidstring'])) docs = bson.decode_all(data, SON, False, UUID_SUBTYPE) for d in docs: self.assertNotEqual(d['newguid'], uuid.UUID(d['newguidstring'])) docs = bson.decode_all(data, SON, False, JAVA_LEGACY) for d in docs: self.assertNotEqual(d['newguid'], uuid.UUID(d['newguidstring'])) docs = bson.decode_all(data, SON, False, CSHARP_LEGACY) for d in docs: self.assertEqual(d['newguid'], uuid.UUID(d['newguidstring'])) # Test encoding encoded = b('').join([bson.BSON.encode(doc, uuid_subtype=OLD_UUID_SUBTYPE) for doc in docs]) self.assertNotEqual(data, encoded) encoded = b('').join([bson.BSON.encode(doc, uuid_subtype=UUID_SUBTYPE) for doc in docs]) self.assertNotEqual(data, encoded) encoded = b('').join([bson.BSON.encode(doc, uuid_subtype=JAVA_LEGACY) for doc in docs]) self.assertNotEqual(data, encoded) encoded = b('').join([bson.BSON.encode(doc, uuid_subtype=CSHARP_LEGACY) for doc in docs]) self.assertEqual(data, encoded) # Test insert and find client = get_client() client.pymongo_test.drop_collection('csharp_uuid') coll = client.pymongo_test.csharp_uuid coll.uuid_subtype = CSHARP_LEGACY coll.insert(docs) self.assertEqual(5, coll.count()) for d in coll.find(): self.assertEqual(d['newguid'], uuid.UUID(d['newguidstring'])) coll.uuid_subtype = OLD_UUID_SUBTYPE for d in coll.find(): self.assertNotEqual(d['newguid'], d['newguidstring']) client.pymongo_test.drop_collection('csharp_uuid') def test_uuid_queries(self): if not should_test_uuid: raise SkipTest("No uuid module") c = get_client() coll = c.pymongo_test.test coll.drop() uu = uuid.uuid4() # Wrap uu.bytes in binary_type to work # around http://bugs.python.org/issue7380. coll.insert({'uuid': Binary(binary_type(uu.bytes), 3)}) self.assertEqual(1, coll.count()) # Test UUIDLegacy queries. coll.uuid_subtype = 4 self.assertEqual(0, coll.find({'uuid': uu}).count()) cur = coll.find({'uuid': UUIDLegacy(uu)}) self.assertEqual(1, cur.count()) retrieved = cur.next() self.assertEqual(uu, retrieved['uuid']) # Test regular UUID queries (using subtype 4). coll.insert({'uuid': uu}) self.assertEqual(2, coll.count()) cur = coll.find({'uuid': uu}) self.assertEqual(1, cur.count()) retrieved = cur.next() self.assertEqual(uu, retrieved['uuid']) # Test both. cur = coll.find({'uuid': {'$in': [uu, UUIDLegacy(uu)]}}) self.assertEqual(2, cur.count()) coll.drop() def test_pickle(self): b1 = Binary(b('123'), 2) # For testing backwards compatibility with pre-2.4 pymongo if PY3: p = b("\x80\x03cbson.binary\nBinary\nq\x00C\x03123q\x01\x85q" "\x02\x81q\x03}q\x04X\x10\x00\x00\x00_Binary__subtypeq" "\x05K\x02sb.") else: p = b("ccopy_reg\n_reconstructor\np0\n(cbson.binary\nBinary\np1\nc" "__builtin__\nstr\np2\nS'123'\np3\ntp4\nRp5\n(dp6\nS'_Binary" "__subtype'\np7\nI2\nsb.") if not sys.version.startswith('3.0'): self.assertEqual(b1, pickle.loads(p)) for proto in xrange(pickle.HIGHEST_PROTOCOL + 1): self.assertEqual(b1, pickle.loads(pickle.dumps(b1, proto))) if should_test_uuid: uu = uuid.uuid4() uul = UUIDLegacy(uu) self.assertEqual(uul, copy.copy(uul)) self.assertEqual(uul, copy.deepcopy(uul)) for proto in xrange(pickle.HIGHEST_PROTOCOL + 1): self.assertEqual(uul, pickle.loads(pickle.dumps(uul, proto))) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_bson.py000066400000000000000000000473771223300253600165610ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the bson module.""" import unittest import datetime import re import sys try: import uuid should_test_uuid = True except ImportError: should_test_uuid = False sys.path[0:0] = [""] from nose.plugins.skip import SkipTest import bson from bson import (BSON, decode_all, is_valid) from bson.binary import Binary, UUIDLegacy from bson.code import Code from bson.objectid import ObjectId from bson.dbref import DBRef from bson.py3compat import b from bson.son import SON from bson.timestamp import Timestamp from bson.errors import (InvalidDocument, InvalidStringData) from bson.max_key import MaxKey from bson.min_key import MinKey from bson.tz_util import (FixedOffset, utc) from test import qcheck PY3 = sys.version_info[0] == 3 class TestBSON(unittest.TestCase): def setUp(self): pass def test_basic_validation(self): self.assertRaises(TypeError, is_valid, 100) self.assertRaises(TypeError, is_valid, u"test") self.assertRaises(TypeError, is_valid, 10.4) self.assertFalse(is_valid(b("test"))) # the simplest valid BSON document self.assertTrue(is_valid(b("\x05\x00\x00\x00\x00"))) self.assertTrue(is_valid(BSON(b("\x05\x00\x00\x00\x00")))) # failure cases self.assertFalse(is_valid(b("\x04\x00\x00\x00\x00"))) self.assertFalse(is_valid(b("\x05\x00\x00\x00\x01"))) self.assertFalse(is_valid(b("\x05\x00\x00\x00"))) self.assertFalse(is_valid(b("\x05\x00\x00\x00\x00\x00"))) self.assertFalse(is_valid(b("\x07\x00\x00\x00\x02a\x00\x78\x56\x34\x12"))) self.assertFalse(is_valid(b("\x09\x00\x00\x00\x10a\x00\x05\x00"))) self.assertFalse(is_valid(b("\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"))) self.assertFalse(is_valid(b("\x13\x00\x00\x00\x02foo\x00" "\x04\x00\x00\x00bar\x00\x00"))) self.assertFalse(is_valid(b("\x18\x00\x00\x00\x03foo\x00\x0f\x00\x00" "\x00\x10bar\x00\xff\xff\xff\x7f\x00\x00"))) self.assertFalse(is_valid(b("\x15\x00\x00\x00\x03foo\x00\x0c" "\x00\x00\x00\x08bar\x00\x01\x00\x00"))) self.assertFalse(is_valid(b("\x1c\x00\x00\x00\x03foo\x00" "\x12\x00\x00\x00\x02bar\x00" "\x05\x00\x00\x00baz\x00\x00\x00"))) self.assertFalse(is_valid(b("\x10\x00\x00\x00\x02a\x00" "\x04\x00\x00\x00abc\xff\x00"))) def test_random_data_is_not_bson(self): qcheck.check_unittest(self, qcheck.isnt(is_valid), qcheck.gen_string(qcheck.gen_range(0, 40))) def test_basic_decode(self): self.assertEqual({"test": u"hello world"}, BSON(b("\x1B\x00\x00\x00\x0E\x74\x65\x73\x74\x00\x0C" "\x00\x00\x00\x68\x65\x6C\x6C\x6F\x20\x77\x6F" "\x72\x6C\x64\x00\x00")).decode()) self.assertEqual([{"test": u"hello world"}, {}], decode_all(b("\x1B\x00\x00\x00\x0E\x74\x65\x73\x74" "\x00\x0C\x00\x00\x00\x68\x65\x6C\x6C" "\x6f\x20\x77\x6F\x72\x6C\x64\x00\x00" "\x05\x00\x00\x00\x00"))) def test_data_timestamp(self): self.assertEqual({"test": Timestamp(4, 20)}, BSON(b("\x13\x00\x00\x00\x11\x74\x65\x73\x74\x00\x14" "\x00\x00\x00\x04\x00\x00\x00\x00")).decode()) def test_basic_encode(self): self.assertRaises(TypeError, BSON.encode, 100) self.assertRaises(TypeError, BSON.encode, "hello") self.assertRaises(TypeError, BSON.encode, None) self.assertRaises(TypeError, BSON.encode, []) self.assertEqual(BSON.encode({}), BSON(b("\x05\x00\x00\x00\x00"))) self.assertEqual(BSON.encode({"test": u"hello world"}), b("\x1B\x00\x00\x00\x02\x74\x65\x73\x74\x00\x0C\x00" "\x00\x00\x68\x65\x6C\x6C\x6F\x20\x77\x6F\x72\x6C" "\x64\x00\x00")) self.assertEqual(BSON.encode({u"mike": 100}), b("\x0F\x00\x00\x00\x10\x6D\x69\x6B\x65\x00\x64\x00" "\x00\x00\x00")) self.assertEqual(BSON.encode({"hello": 1.5}), b("\x14\x00\x00\x00\x01\x68\x65\x6C\x6C\x6F\x00\x00" "\x00\x00\x00\x00\x00\xF8\x3F\x00")) self.assertEqual(BSON.encode({"true": True}), b("\x0C\x00\x00\x00\x08\x74\x72\x75\x65\x00\x01\x00")) self.assertEqual(BSON.encode({"false": False}), b("\x0D\x00\x00\x00\x08\x66\x61\x6C\x73\x65\x00\x00" "\x00")) self.assertEqual(BSON.encode({"empty": []}), b("\x11\x00\x00\x00\x04\x65\x6D\x70\x74\x79\x00\x05" "\x00\x00\x00\x00\x00")) self.assertEqual(BSON.encode({"none": {}}), b("\x10\x00\x00\x00\x03\x6E\x6F\x6E\x65\x00\x05\x00" "\x00\x00\x00\x00")) self.assertEqual(BSON.encode({"test": Binary(b("test"), 0)}), b("\x14\x00\x00\x00\x05\x74\x65\x73\x74\x00\x04\x00" "\x00\x00\x00\x74\x65\x73\x74\x00")) self.assertEqual(BSON.encode({"test": Binary(b("test"), 2)}), b("\x18\x00\x00\x00\x05\x74\x65\x73\x74\x00\x08\x00" "\x00\x00\x02\x04\x00\x00\x00\x74\x65\x73\x74\x00")) self.assertEqual(BSON.encode({"test": Binary(b("test"), 128)}), b("\x14\x00\x00\x00\x05\x74\x65\x73\x74\x00\x04\x00" "\x00\x00\x80\x74\x65\x73\x74\x00")) self.assertEqual(BSON.encode({"test": None}), b("\x0B\x00\x00\x00\x0A\x74\x65\x73\x74\x00\x00")) self.assertEqual(BSON.encode({"date": datetime.datetime(2007, 1, 8, 0, 30, 11)}), b("\x13\x00\x00\x00\x09\x64\x61\x74\x65\x00\x38\xBE" "\x1C\xFF\x0F\x01\x00\x00\x00")) self.assertEqual(BSON.encode({"regex": re.compile(b("a*b"), re.IGNORECASE)}), b("\x12\x00\x00\x00\x0B\x72\x65\x67\x65\x78\x00\x61" "\x2A\x62\x00\x69\x00\x00")) self.assertEqual(BSON.encode({"$where": Code("test")}), b("\x16\x00\x00\x00\r$where\x00\x05\x00\x00\x00test" "\x00\x00")) self.assertEqual(BSON.encode({"$field": Code("function(){ return true;}", scope=None)}), b("+\x00\x00\x00\r$field\x00\x1a\x00\x00\x00" "function(){ return true;}\x00\x00")) self.assertEqual(BSON.encode({"$field": Code("return function(){ return x; }", scope={'x': False})}), b("=\x00\x00\x00\x0f$field\x000\x00\x00\x00\x1f\x00" "\x00\x00return function(){ return x; }\x00\t\x00" "\x00\x00\x08x\x00\x00\x00\x00")) a = ObjectId(b("\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0A\x0B")) self.assertEqual(BSON.encode({"oid": a}), b("\x16\x00\x00\x00\x07\x6F\x69\x64\x00\x00\x01\x02" "\x03\x04\x05\x06\x07\x08\x09\x0A\x0B\x00")) self.assertEqual(BSON.encode({"ref": DBRef("coll", a)}), b("\x2F\x00\x00\x00\x03ref\x00\x25\x00\x00\x00\x02" "$ref\x00\x05\x00\x00\x00coll\x00\x07$id\x00\x00" "\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0A\x0B\x00" "\x00")) def test_encode_then_decode(self): def helper(dict): self.assertEqual(dict, (BSON.encode(dict)).decode()) helper({}) helper({"test": u"hello"}) self.assertTrue(isinstance(BSON.encode({"hello": "world"}) .decode()["hello"], unicode)) helper({"mike": -10120}) helper({"long": long(10)}) helper({"really big long": 2147483648}) helper({u"hello": 0.0013109}) helper({"something": True}) helper({"false": False}) helper({"an array": [1, True, 3.8, u"world"]}) helper({"an object": {"test": u"something"}}) helper({"a binary": Binary(b("test"), 100)}) helper({"a binary": Binary(b("test"), 128)}) helper({"a binary": Binary(b("test"), 254)}) helper({"another binary": Binary(b("test"), 2)}) helper(SON([(u'test dst', datetime.datetime(1993, 4, 4, 2))])) helper(SON([(u'test negative dst', datetime.datetime(1, 1, 1, 1, 1, 1))])) helper({"big float": float(10000000000)}) helper({"ref": DBRef("coll", 5)}) helper({"ref": DBRef("coll", 5, foo="bar", bar=4)}) helper({"ref": DBRef("coll", 5, "foo")}) helper({"ref": DBRef("coll", 5, "foo", foo="bar")}) helper({"ref": Timestamp(1, 2)}) helper({"foo": MinKey()}) helper({"foo": MaxKey()}) helper({"$field": Code("function(){ return true; }")}) helper({"$field": Code("return function(){ return x; }", scope={'x': False})}) doc_class = dict # Work around http://bugs.jython.org/issue1728 if (sys.platform.startswith('java') and sys.version_info[:3] >= (2, 5, 2)): doc_class = SON def encode_then_decode(doc): return doc == (BSON.encode(doc)).decode(as_class=doc_class) qcheck.check_unittest(self, encode_then_decode, qcheck.gen_mongo_dict(3)) def test_bytes_as_keys(self): doc = {b("foo"): 'bar'} # Since `bytes` are stored as Binary you can't use them # as keys in python 3.x. Using binary data as a key makes # no sense in BSON anyway and little sense in python. if PY3: self.assertRaises(InvalidDocument, BSON.encode, doc) else: self.assertTrue(BSON.encode(doc)) def test_datetime_encode_decode(self): # Negative timestamps dt1 = datetime.datetime(1, 1, 1, 1, 1, 1, 111000) dt2 = BSON.encode({"date": dt1}).decode()["date"] self.assertEqual(dt1, dt2) dt1 = datetime.datetime(1959, 6, 25, 12, 16, 59, 999000) dt2 = BSON.encode({"date": dt1}).decode()["date"] self.assertEqual(dt1, dt2) # Positive timestamps dt1 = datetime.datetime(9999, 12, 31, 23, 59, 59, 999000) dt2 = BSON.encode({"date": dt1}).decode()["date"] self.assertEqual(dt1, dt2) dt1 = datetime.datetime(2011, 6, 14, 10, 47, 53, 444000) dt2 = BSON.encode({"date": dt1}).decode()["date"] self.assertEqual(dt1, dt2) def test_aware_datetime(self): aware = datetime.datetime(1993, 4, 4, 2, tzinfo=FixedOffset(555, "SomeZone")) as_utc = (aware - aware.utcoffset()).replace(tzinfo=utc) self.assertEqual(datetime.datetime(1993, 4, 3, 16, 45, tzinfo=utc), as_utc) after = BSON.encode({"date": aware}).decode(tz_aware=True)["date"] self.assertEqual(utc, after.tzinfo) self.assertEqual(as_utc, after) def test_naive_decode(self): aware = datetime.datetime(1993, 4, 4, 2, tzinfo=FixedOffset(555, "SomeZone")) naive_utc = (aware - aware.utcoffset()).replace(tzinfo=None) self.assertEqual(datetime.datetime(1993, 4, 3, 16, 45), naive_utc) after = BSON.encode({"date": aware}).decode()["date"] self.assertEqual(None, after.tzinfo) self.assertEqual(naive_utc, after) def test_dst(self): d = {"x": datetime.datetime(1993, 4, 4, 2)} self.assertEqual(d, BSON.encode(d).decode()) def test_bad_encode(self): if not PY3: # Python3 treats this as a unicode string which won't raise # an exception. If we passed the string as bytes instead we # still wouldn't get an error since we store bytes as BSON # binary subtype 0. self.assertRaises(InvalidStringData, BSON.encode, {"lalala": '\xf4\xe0\xf0\xe1\xc0 Color Touch'}) evil_list = {'a': []} evil_list['a'].append(evil_list) evil_dict = {} evil_dict['a'] = evil_dict for evil_data in [evil_dict, evil_list]: self.assertRaises(RuntimeError, BSON.encode, evil_data) def test_overflow(self): self.assertTrue(BSON.encode({"x": 9223372036854775807L})) self.assertRaises(OverflowError, BSON.encode, {"x": 9223372036854775808L}) self.assertTrue(BSON.encode({"x": -9223372036854775808L})) self.assertRaises(OverflowError, BSON.encode, {"x": -9223372036854775809L}) def test_small_long_encode_decode(self): if PY3: raise SkipTest("No long type in Python 3.") encoded1 = BSON.encode({'x': 256}) decoded1 = BSON.decode(encoded1)['x'] self.assertEqual(256, decoded1) self.assertEqual(type(256), type(decoded1)) encoded2 = BSON.encode({'x': 256L}) decoded2 = BSON.decode(encoded2)['x'] self.assertEqual(256L, decoded2) self.assertEqual(type(256L), type(decoded2)) self.assertNotEqual(type(decoded1), type(decoded2)) def test_tuple(self): self.assertEqual({"tuple": [1, 2]}, BSON.encode({"tuple": (1, 2)}).decode()) def test_uuid(self): if not should_test_uuid: raise SkipTest("No uuid module") id = uuid.uuid4() transformed_id = (BSON.encode({"id": id})).decode()["id"] self.assertTrue(isinstance(transformed_id, uuid.UUID)) self.assertEqual(id, transformed_id) self.assertNotEqual(uuid.uuid4(), transformed_id) def test_uuid_legacy(self): if not should_test_uuid: raise SkipTest("No uuid module") id = uuid.uuid4() legacy = UUIDLegacy(id) self.assertEqual(3, legacy.subtype) transformed = (BSON.encode({"uuid": legacy})).decode()["uuid"] self.assertTrue(isinstance(transformed, uuid.UUID)) self.assertEqual(id, transformed) self.assertNotEqual(UUIDLegacy(uuid.uuid4()), UUIDLegacy(transformed)) # The C extension was segfaulting on unicode RegExs, so we have this test # that doesn't really test anything but the lack of a segfault. def test_unicode_regex(self): regex = re.compile(u'revisi\xf3n') BSON.encode({"regex": regex}).decode() def test_non_string_keys(self): self.assertRaises(InvalidDocument, BSON.encode, {8.9: "test"}) def test_utf8(self): w = {u"aéあ": u"aéあ"} self.assertEqual(w, BSON.encode(w).decode()) iso8859_bytes = u"aé".encode("iso-8859-1") y = {"hello": iso8859_bytes} if PY3: # Stored as BSON binary subtype 0. out = BSON.encode(y).decode() self.assertTrue(isinstance(out['hello'], bytes)) self.assertEqual(out['hello'], iso8859_bytes) else: # Python 2. try: BSON.encode(y) except InvalidStringData, e: self.assertTrue(repr(iso8859_bytes) in str(e)) # The next two tests only make sense in python 2.x since # you can't use `bytes` type as document keys in python 3.x. x = {u"aéあ".encode("utf-8"): u"aéあ".encode("utf-8")} self.assertEqual(w, BSON.encode(x).decode()) z = {iso8859_bytes: "hello"} self.assertRaises(InvalidStringData, BSON.encode, z) def test_null_character(self): doc = {"a": "\x00"} self.assertEqual(doc, BSON.encode(doc).decode()) # This test doesn't make much sense in Python2 # since {'a': '\x00'} == {'a': u'\x00'}. # Decoding here actually returns {'a': '\x00'} doc = {"a": u"\x00"} self.assertEqual(doc, BSON.encode(doc).decode()) self.assertRaises(InvalidDocument, BSON.encode, {b("\x00"): "a"}) self.assertRaises(InvalidDocument, BSON.encode, {u"\x00": "a"}) self.assertRaises(InvalidDocument, BSON.encode, {"a": re.compile(b("ab\x00c"))}) self.assertRaises(InvalidDocument, BSON.encode, {"a": re.compile(u"ab\x00c")}) def test_move_id(self): self.assertEqual(b("\x19\x00\x00\x00\x02_id\x00\x02\x00\x00\x00a\x00" "\x02a\x00\x02\x00\x00\x00a\x00\x00"), BSON.encode(SON([("a", "a"), ("_id", "a")]))) self.assertEqual(b("\x2c\x00\x00\x00" "\x02_id\x00\x02\x00\x00\x00b\x00" "\x03b\x00" "\x19\x00\x00\x00\x02a\x00\x02\x00\x00\x00a\x00" "\x02_id\x00\x02\x00\x00\x00a\x00\x00\x00"), BSON.encode(SON([("b", SON([("a", "a"), ("_id", "a")])), ("_id", "b")]))) def test_dates(self): doc = {"early": datetime.datetime(1686, 5, 5), "late": datetime.datetime(2086, 5, 5)} try: self.assertEqual(doc, BSON.encode(doc).decode()) except ValueError: # Ignore ValueError when no C ext, since it's probably # a problem w/ 32-bit Python - we work around this in the # C ext, though. if bson.has_c(): raise def test_custom_class(self): self.assertTrue(isinstance(BSON.encode({}).decode(), dict)) self.assertFalse(isinstance(BSON.encode({}).decode(), SON)) self.assertTrue(isinstance(BSON.encode({}).decode(SON), SON)) self.assertEqual(1, BSON.encode({"x": 1}).decode(SON)["x"]) x = BSON.encode({"x": [{"y": 1}]}) self.assertTrue(isinstance(x.decode(SON)["x"][0], SON)) def test_subclasses(self): # make sure we can serialize subclasses of native Python types. class _myint(int): pass class _myfloat(float): pass class _myunicode(unicode): pass d = {'a': _myint(42), 'b': _myfloat(63.9), 'c': _myunicode('hello world') } d2 = BSON.encode(d).decode() for key, value in d2.iteritems(): orig_value = d[key] orig_type = orig_value.__class__.__bases__[0] self.assertEqual(type(value), orig_type) self.assertEqual(value, orig_type(value)) def test_ordered_dict(self): try: from collections import OrderedDict except ImportError: raise SkipTest("No OrderedDict") d = OrderedDict([("one", 1), ("two", 2), ("three", 3), ("four", 4)]) self.assertEqual(d, BSON.encode(d).decode(as_class=OrderedDict)) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_client.py000066400000000000000000000757151223300253600170730ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the mongo_client module.""" import datetime import os import threading import socket import sys import time import thread import unittest sys.path[0:0] = [""] from nose.plugins.skip import SkipTest from bson.son import SON from bson.tz_util import utc from pymongo.mongo_client import MongoClient from pymongo.database import Database from pymongo.pool import SocketInfo from pymongo import thread_util from pymongo.errors import (ConfigurationError, ConnectionFailure, InvalidName, OperationFailure, PyMongoError) from test import version, host, port from test.utils import (assertRaisesExactly, delay, is_mongos, server_is_master_with_slave, server_started_with_auth, TestRequestMixin) def get_client(*args, **kwargs): return MongoClient(host, port, *args, **kwargs) class TestClient(unittest.TestCase, TestRequestMixin): def test_types(self): self.assertRaises(TypeError, MongoClient, 1) self.assertRaises(TypeError, MongoClient, 1.14) self.assertRaises(TypeError, MongoClient, "localhost", "27017") self.assertRaises(TypeError, MongoClient, "localhost", 1.14) self.assertRaises(TypeError, MongoClient, "localhost", []) self.assertRaises(ConfigurationError, MongoClient, []) def test_constants(self): MongoClient.HOST = host MongoClient.PORT = port self.assertTrue(MongoClient()) MongoClient.HOST = "somedomainthatdoesntexist.org" MongoClient.PORT = 123456789 assertRaisesExactly( ConnectionFailure, MongoClient, connectTimeoutMS=600) self.assertTrue(MongoClient(host, port)) MongoClient.HOST = host MongoClient.PORT = port self.assertTrue(MongoClient()) def test_init_disconnected(self): c = MongoClient(host, port, _connect=False) # No errors c.is_primary c.is_mongos c.max_pool_size c.use_greenlets c.nodes c.auto_start_request c.get_document_class() c.tz_aware c.max_bson_size self.assertEqual(None, c.host) self.assertEqual(None, c.port) c.pymongo_test.test.find_one() # Auto-connect. self.assertEqual(host, c.host) self.assertEqual(port, c.port) bad_host = "somedomainthatdoesntexist.org" c = MongoClient(bad_host, port, connectTimeoutMS=1,_connect=False) self.assertRaises(ConnectionFailure, c.pymongo_test.test.find_one) def test_init_disconnected_with_auth(self): uri = "mongodb://user:pass@somedomainthatdoesntexist" c = MongoClient(uri, connectTimeoutMS=1, _connect=False) self.assertRaises(ConnectionFailure, c.pymongo_test.test.find_one) def test_connect(self): # Check that the exception is a ConnectionFailure, not a subclass like # AutoReconnect assertRaisesExactly( ConnectionFailure, MongoClient, "somedomainthatdoesntexist.org", connectTimeoutMS=600) assertRaisesExactly( ConnectionFailure, MongoClient, host, 123456789) self.assertTrue(MongoClient(host, port)) def test_equality(self): client = MongoClient(host, port) self.assertEqual(client, MongoClient(host, port)) # Explicitly test inequality self.assertFalse(client != MongoClient(host, port)) def test_host_w_port(self): self.assertTrue(MongoClient("%s:%d" % (host, port))) assertRaisesExactly( ConnectionFailure, MongoClient, "%s:1234567" % (host,), port) def test_repr(self): # Making host a str avoids the 'u' prefix in Python 2, so the repr is # the same in Python 2 and 3. self.assertEqual(repr(MongoClient(str(host), port)), "MongoClient('%s', %d)" % (host, port)) def test_getters(self): self.assertEqual(MongoClient(host, port).host, host) self.assertEqual(MongoClient(host, port).port, port) self.assertEqual(set([(host, port)]), MongoClient(host, port).nodes) def test_use_greenlets(self): self.assertFalse(MongoClient(host, port).use_greenlets) if thread_util.have_gevent: self.assertTrue( MongoClient( host, port, use_greenlets=True).use_greenlets) def test_get_db(self): client = MongoClient(host, port) def make_db(base, name): return base[name] self.assertRaises(InvalidName, make_db, client, "") self.assertRaises(InvalidName, make_db, client, "te$t") self.assertRaises(InvalidName, make_db, client, "te.t") self.assertRaises(InvalidName, make_db, client, "te\\t") self.assertRaises(InvalidName, make_db, client, "te/t") self.assertRaises(InvalidName, make_db, client, "te st") self.assertTrue(isinstance(client.test, Database)) self.assertEqual(client.test, client["test"]) self.assertEqual(client.test, Database(client, "test")) def test_database_names(self): client = MongoClient(host, port) client.pymongo_test.test.save({"dummy": u"object"}) client.pymongo_test_mike.test.save({"dummy": u"object"}) dbs = client.database_names() self.assertTrue("pymongo_test" in dbs) self.assertTrue("pymongo_test_mike" in dbs) def test_drop_database(self): client = MongoClient(host, port) self.assertRaises(TypeError, client.drop_database, 5) self.assertRaises(TypeError, client.drop_database, None) raise SkipTest("This test often fails due to SERVER-2329") client.pymongo_test.test.save({"dummy": u"object"}) dbs = client.database_names() self.assertTrue("pymongo_test" in dbs) client.drop_database("pymongo_test") dbs = client.database_names() self.assertTrue("pymongo_test" not in dbs) client.pymongo_test.test.save({"dummy": u"object"}) dbs = client.database_names() self.assertTrue("pymongo_test" in dbs) client.drop_database(client.pymongo_test) dbs = client.database_names() self.assertTrue("pymongo_test" not in dbs) def test_copy_db(self): c = MongoClient(host, port) # We test copy twice; once starting in a request and once not. In # either case the copy should succeed (because it starts a request # internally) and should leave us in the same state as before the copy. c.start_request() self.assertRaises(TypeError, c.copy_database, 4, "foo") self.assertRaises(TypeError, c.copy_database, "foo", 4) self.assertRaises(InvalidName, c.copy_database, "foo", "$foo") c.pymongo_test.test.drop() c.drop_database("pymongo_test1") c.drop_database("pymongo_test2") c.pymongo_test.test.insert({"foo": "bar"}) # Due to SERVER-2329, databases may not disappear from a master in a # master-slave pair if not server_is_master_with_slave(c): self.assertFalse("pymongo_test1" in c.database_names()) self.assertFalse("pymongo_test2" in c.database_names()) c.copy_database("pymongo_test", "pymongo_test1") # copy_database() didn't accidentally end the request self.assertTrue(c.in_request()) self.assertTrue("pymongo_test1" in c.database_names()) self.assertEqual("bar", c.pymongo_test1.test.find_one()["foo"]) c.end_request() self.assertFalse(c.in_request()) c.copy_database("pymongo_test", "pymongo_test2", "%s:%d" % (host, port)) # copy_database() didn't accidentally restart the request self.assertFalse(c.in_request()) self.assertTrue("pymongo_test2" in c.database_names()) self.assertEqual("bar", c.pymongo_test2.test.find_one()["foo"]) if version.at_least(c, (1, 3, 3, 1)): c.drop_database("pymongo_test1") c.pymongo_test.add_user("mike", "password") self.assertRaises(OperationFailure, c.copy_database, "pymongo_test", "pymongo_test1", username="foo", password="bar") if not server_is_master_with_slave(c): self.assertFalse("pymongo_test1" in c.database_names()) self.assertRaises(OperationFailure, c.copy_database, "pymongo_test", "pymongo_test1", username="mike", password="bar") if not server_is_master_with_slave(c): self.assertFalse("pymongo_test1" in c.database_names()) if not is_mongos(c): # See SERVER-6427 c.copy_database("pymongo_test", "pymongo_test1", username="mike", password="password") self.assertTrue("pymongo_test1" in c.database_names()) self.assertEqual("bar", c.pymongo_test1.test.find_one()["foo"]) def test_iteration(self): client = MongoClient(host, port) def iterate(): [a for a in client] self.assertRaises(TypeError, iterate) def test_disconnect(self): c = MongoClient(host, port) coll = c.pymongo_test.bar c.disconnect() c.disconnect() coll.count() c.disconnect() c.disconnect() coll.count() def test_from_uri(self): c = MongoClient(host, port) self.assertEqual(c, MongoClient("mongodb://%s:%d" % (host, port))) self.assertTrue(MongoClient( "mongodb://%s:%d" % (host, port), slave_okay=True).slave_okay) self.assertTrue(MongoClient( "mongodb://%s:%d/?slaveok=true;w=2" % (host, port)).slave_okay) def test_get_default_database(self): c = MongoClient("mongodb://%s:%d/foo" % (host, port), _connect=False) self.assertEqual(Database(c, 'foo'), c.get_default_database()) def test_get_default_database_error(self): # URI with no database. c = MongoClient("mongodb://%s:%d/" % (host, port), _connect=False) self.assertRaises(ConfigurationError, c.get_default_database) def test_get_default_database_with_authsource(self): # Ensure we distinguish database name from authSource. uri = "mongodb://%s:%d/foo?authSource=src" % (host, port) c = MongoClient(uri, _connect=False) self.assertEqual(Database(c, 'foo'), c.get_default_database()) def test_auth_from_uri(self): c = MongoClient(host, port) # Sharded auth not supported before MongoDB 2.0 if is_mongos(c) and not version.at_least(c, (2, 0, 0)): raise SkipTest("Auth with sharding requires MongoDB >= 2.0.0") c.admin.system.users.remove({}) c.pymongo_test.system.users.remove({}) try: c.admin.add_user("admin", "pass") c.admin.authenticate("admin", "pass") c.pymongo_test.add_user("user", "pass") self.assertRaises(ConfigurationError, MongoClient, "mongodb://foo:bar@%s:%d" % (host, port)) self.assertRaises(ConfigurationError, MongoClient, "mongodb://admin:bar@%s:%d" % (host, port)) self.assertRaises(ConfigurationError, MongoClient, "mongodb://user:pass@%s:%d" % (host, port)) MongoClient("mongodb://admin:pass@%s:%d" % (host, port)) self.assertRaises(ConfigurationError, MongoClient, "mongodb://admin:pass@%s:%d/pymongo_test" % (host, port)) self.assertRaises(ConfigurationError, MongoClient, "mongodb://user:foo@%s:%d/pymongo_test" % (host, port)) MongoClient("mongodb://user:pass@%s:%d/pymongo_test" % (host, port)) # Auth with lazy connection. MongoClient( "mongodb://user:pass@%s:%d/pymongo_test" % (host, port), _connect=False).pymongo_test.test.find_one() # Wrong password. bad_client = MongoClient( "mongodb://user:wrong@%s:%d/pymongo_test" % (host, port), _connect=False) # If auth fails with lazy connection, MongoClient raises # AutoReconnect instead of the more appropriate OperationFailure, # PYTHON-517. self.assertRaises( PyMongoError, bad_client.pymongo_test.test.find_one) finally: # Clean up. c.admin.system.users.remove({}) c.pymongo_test.system.users.remove({}) def test_lazy_auth_raises_operation_failure(self): # Check if we have the prerequisites to run this test. c = MongoClient(host, port) if not server_started_with_auth(c): raise SkipTest('Authentication is not enabled on server') if is_mongos(c) and not version.at_least(c, (2, 0, 0)): raise SkipTest("Auth with sharding requires MongoDB >= 2.0.0") lazy_client = MongoClient( "mongodb://user:wrong@%s:%d/pymongo_test" % (host, port), _connect=False) assertRaisesExactly( OperationFailure, lazy_client.test.collection.find_one) def test_unix_socket(self): if not hasattr(socket, "AF_UNIX"): raise SkipTest("UNIX-sockets are not supported on this system") if (sys.platform == 'darwin' and server_started_with_auth(MongoClient(host, port))): raise SkipTest("SERVER-8492") mongodb_socket = '/tmp/mongodb-27017.sock' if not os.access(mongodb_socket, os.R_OK): raise SkipTest("Socket file is not accessable") self.assertTrue(MongoClient("mongodb://%s" % mongodb_socket)) client = MongoClient("mongodb://%s" % mongodb_socket) client.pymongo_test.test.save({"dummy": "object"}) # Confirm we can read via the socket dbs = client.database_names() self.assertTrue("pymongo_test" in dbs) # Confirm it fails with a missing socket self.assertRaises(ConnectionFailure, MongoClient, "mongodb:///tmp/none-existent.sock") def test_fork(self): # Test using a client before and after a fork. if sys.platform == "win32": raise SkipTest("Can't fork on windows") try: from multiprocessing import Process, Pipe except ImportError: raise SkipTest("No multiprocessing module") db = MongoClient(host, port).pymongo_test # Failure occurs if the client is used before the fork db.test.find_one() db.connection.end_request() def loop(pipe): while True: try: db.test.insert({"a": "b"}) for _ in db.test.find(): pass except: pipe.send(True) os._exit(1) cp1, cc1 = Pipe() cp2, cc2 = Pipe() p1 = Process(target=loop, args=(cc1,)) p2 = Process(target=loop, args=(cc2,)) p1.start() p2.start() p1.join(1) p2.join(1) p1.terminate() p2.terminate() p1.join() p2.join() cc1.close() cc2.close() # recv will only have data if the subprocess failed try: cp1.recv() self.fail() except EOFError: pass try: cp2.recv() self.fail() except EOFError: pass def test_document_class(self): c = MongoClient(host, port) db = c.pymongo_test db.test.insert({"x": 1}) self.assertEqual(dict, c.document_class) self.assertTrue(isinstance(db.test.find_one(), dict)) self.assertFalse(isinstance(db.test.find_one(), SON)) c.document_class = SON self.assertEqual(SON, c.document_class) self.assertTrue(isinstance(db.test.find_one(), SON)) self.assertFalse(isinstance(db.test.find_one(as_class=dict), SON)) c = MongoClient(host, port, document_class=SON) db = c.pymongo_test self.assertEqual(SON, c.document_class) self.assertTrue(isinstance(db.test.find_one(), SON)) self.assertFalse(isinstance(db.test.find_one(as_class=dict), SON)) c.document_class = dict self.assertEqual(dict, c.document_class) self.assertTrue(isinstance(db.test.find_one(), dict)) self.assertFalse(isinstance(db.test.find_one(), SON)) def test_timeouts(self): client = MongoClient(host, port, connectTimeoutMS=10500) self.assertEqual(10.5, client._MongoClient__pool.conn_timeout) client = MongoClient(host, port, socketTimeoutMS=10500) self.assertEqual(10.5, client._MongoClient__pool.net_timeout) def test_network_timeout_validation(self): c = get_client(socketTimeoutMS=10 * 1000) self.assertEqual(10, c._MongoClient__net_timeout) c = get_client(socketTimeoutMS=None) self.assertEqual(None, c._MongoClient__net_timeout) self.assertRaises(ConfigurationError, get_client, socketTimeoutMS=0) self.assertRaises(ConfigurationError, get_client, socketTimeoutMS=-1) self.assertRaises(ConfigurationError, get_client, socketTimeoutMS=1e10) self.assertRaises(ConfigurationError, get_client, socketTimeoutMS='foo') # network_timeout is gone from MongoClient, remains in deprecated # Connection self.assertRaises(ConfigurationError, get_client, network_timeout=10) def test_network_timeout(self): no_timeout = MongoClient(host, port) timeout_sec = 1 timeout = MongoClient( host, port, socketTimeoutMS=1000 * timeout_sec) no_timeout.pymongo_test.drop_collection("test") no_timeout.pymongo_test.test.insert({"x": 1}) # A $where clause that takes a second longer than the timeout where_func = delay(timeout_sec + 1) def get_x(db): doc = db.test.find().where(where_func).next() return doc["x"] self.assertEqual(1, get_x(no_timeout.pymongo_test)) self.assertRaises(ConnectionFailure, get_x, timeout.pymongo_test) def get_x_timeout(db, t): doc = db.test.find(network_timeout=t).where(where_func).next() return doc["x"] self.assertEqual(1, get_x_timeout(timeout.pymongo_test, None)) self.assertRaises(ConnectionFailure, get_x_timeout, no_timeout.pymongo_test, 0.1) def test_waitQueueTimeoutMS(self): client = MongoClient(host, port, waitQueueTimeoutMS=2000) self.assertEqual(client._MongoClient__pool.wait_queue_timeout, 2) def test_waitQueueMultiple(self): client = MongoClient(host, port, max_pool_size=3, waitQueueMultiple=2) pool = client._MongoClient__pool self.assertEqual(pool.wait_queue_multiple, 2) self.assertEqual(pool._socket_semaphore.waiter_semaphore.counter, 6) def test_tz_aware(self): self.assertRaises(ConfigurationError, MongoClient, tz_aware='foo') aware = MongoClient(host, port, tz_aware=True) naive = MongoClient(host, port) aware.pymongo_test.drop_collection("test") now = datetime.datetime.utcnow() aware.pymongo_test.test.insert({"x": now}) self.assertEqual(None, naive.pymongo_test.test.find_one()["x"].tzinfo) self.assertEqual(utc, aware.pymongo_test.test.find_one()["x"].tzinfo) self.assertEqual( aware.pymongo_test.test.find_one()["x"].replace(tzinfo=None), naive.pymongo_test.test.find_one()["x"]) def test_ipv6(self): try: client = MongoClient("[::1]") except: # Either mongod was started without --ipv6 # or the OS doesn't support it (or both). raise SkipTest("No IPv6") # Try a few simple things MongoClient("mongodb://[::1]:%d" % (port,)) MongoClient("mongodb://[::1]:%d/?slaveOk=true" % (port,)) MongoClient("[::1]:%d,localhost:%d" % (port, port)) client = MongoClient("localhost:%d,[::1]:%d" % (port, port)) client.pymongo_test.test.save({"dummy": u"object"}) client.pymongo_test_bernie.test.save({"dummy": u"object"}) dbs = client.database_names() self.assertTrue("pymongo_test" in dbs) self.assertTrue("pymongo_test_bernie" in dbs) def test_fsync_lock_unlock(self): c = get_client() if is_mongos(c): raise SkipTest('fsync/lock not supported by mongos') res = c.admin.command('getCmdLineOpts') if '--master' in res['argv'] and version.at_least(c, (2, 3, 0)): raise SkipTest('SERVER-7714') self.assertFalse(c.is_locked) # async flushing not supported on windows... if sys.platform not in ('cygwin', 'win32'): c.fsync(async=True) self.assertFalse(c.is_locked) c.fsync(lock=True) self.assertTrue(c.is_locked) locked = True c.unlock() for _ in xrange(5): locked = c.is_locked if not locked: break time.sleep(1) self.assertFalse(locked) def test_contextlib(self): if sys.version_info < (2, 6): raise SkipTest("With statement requires Python >= 2.6") import contextlib client = get_client(auto_start_request=False) client.pymongo_test.drop_collection("test") client.pymongo_test.test.insert({"foo": "bar"}) # The socket used for the previous commands has been returned to the # pool self.assertEqual(1, len(client._MongoClient__pool.sockets)) # We need exec here because if the Python version is less than 2.6 # these with-statements won't even compile. exec """ with contextlib.closing(client): self.assertEqual("bar", client.pymongo_test.test.find_one()["foo"]) self.assertEqual(0, len(client._MongoClient__pool.sockets)) """ exec """ with get_client() as client: self.assertEqual("bar", client.pymongo_test.test.find_one()["foo"]) # Calling client.close() has reset the pool self.assertEqual(0, len(client._MongoClient__pool.sockets)) """ def test_with_start_request(self): client = get_client() pool = client._MongoClient__pool # No request started self.assertNoRequest(pool) self.assertDifferentSock(pool) # Start a request request_context_mgr = client.start_request() self.assertTrue( isinstance(request_context_mgr, object) ) self.assertNoSocketYet(pool) self.assertSameSock(pool) self.assertRequestSocket(pool) # End request request_context_mgr.__exit__(None, None, None) self.assertNoRequest(pool) self.assertDifferentSock(pool) # Test the 'with' statement if sys.version_info >= (2, 6): # We need exec here because if the Python version is less than 2.6 # these with-statements won't even compile. exec """ with client.start_request() as request: self.assertEqual(client, request.connection) self.assertNoSocketYet(pool) self.assertSameSock(pool) self.assertRequestSocket(pool) """ # Request has ended self.assertNoRequest(pool) self.assertDifferentSock(pool) def test_auto_start_request(self): for bad_horrible_value in (None, 5, 'hi!'): self.assertRaises( (TypeError, ConfigurationError), lambda: get_client(auto_start_request=bad_horrible_value) ) # auto_start_request should default to False client = get_client() self.assertFalse(client.auto_start_request) client = get_client(auto_start_request=True) self.assertTrue(client.auto_start_request) self.assertTrue(client.in_request()) pool = client._MongoClient__pool # Request started already, just from MongoClient constructor - it's a # bit weird, but MongoClient does some socket stuff when it initializes # and it ends up with a request socket self.assertRequestSocket(pool) self.assertSameSock(pool) client.end_request() self.assertNoRequest(pool) self.assertDifferentSock(pool) # Trigger auto_start_request client.pymongo_test.test.find_one() self.assertRequestSocket(pool) self.assertSameSock(pool) def test_nested_request(self): # auto_start_request is False client = get_client() pool = client._MongoClient__pool self.assertFalse(client.in_request()) # Start and end request client.start_request() self.assertInRequestAndSameSock(client, pool) client.end_request() self.assertNotInRequestAndDifferentSock(client, pool) # Double-nesting client.start_request() client.start_request() client.end_request() self.assertInRequestAndSameSock(client, pool) client.end_request() self.assertNotInRequestAndDifferentSock(client, pool) # Extra end_request calls have no effect - count stays at zero client.end_request() self.assertNotInRequestAndDifferentSock(client, pool) client.start_request() self.assertInRequestAndSameSock(client, pool) client.end_request() self.assertNotInRequestAndDifferentSock(client, pool) def test_request_threads(self): client = get_client(auto_start_request=False) pool = client._MongoClient__pool self.assertNotInRequestAndDifferentSock(client, pool) started_request, ended_request = threading.Event(), threading.Event() checked_request = threading.Event() thread_done = [False] # Starting a request in one thread doesn't put the other thread in a # request def f(): self.assertNotInRequestAndDifferentSock(client, pool) client.start_request() self.assertInRequestAndSameSock(client, pool) started_request.set() checked_request.wait() checked_request.clear() self.assertInRequestAndSameSock(client, pool) client.end_request() self.assertNotInRequestAndDifferentSock(client, pool) ended_request.set() checked_request.wait() thread_done[0] = True t = threading.Thread(target=f) t.setDaemon(True) t.start() # It doesn't matter in what order the main thread or t initially get # to started_request.set() / wait(); by waiting here we ensure that t # has called client.start_request() before we assert on the next line. started_request.wait() self.assertNotInRequestAndDifferentSock(client, pool) checked_request.set() ended_request.wait() self.assertNotInRequestAndDifferentSock(client, pool) checked_request.set() t.join() self.assertNotInRequestAndDifferentSock(client, pool) self.assertTrue(thread_done[0], "Thread didn't complete") def test_interrupt_signal(self): if sys.platform.startswith('java'): # We can't figure out how to raise an exception on a thread that's # blocked on a socket, whether that's the main thread or a worker, # without simply killing the whole thread in Jython. This suggests # PYTHON-294 can't actually occur in Jython. raise SkipTest("Can't test interrupts in Jython") # Test fix for PYTHON-294 -- make sure MongoClient closes its # socket if it gets an interrupt while waiting to recv() from it. c = get_client() db = c.pymongo_test # A $where clause which takes 1.5 sec to execute where = delay(1.5) # Need exactly 1 document so find() will execute its $where clause once db.drop_collection('foo') db.foo.insert({'_id': 1}) def interrupter(): # Raises KeyboardInterrupt in the main thread time.sleep(0.25) thread.interrupt_main() thread.start_new_thread(interrupter, ()) raised = False try: # Will be interrupted by a KeyboardInterrupt. db.foo.find({'$where': where}).next() except KeyboardInterrupt: raised = True # Can't use self.assertRaises() because it doesn't catch system # exceptions self.assertTrue(raised, "Didn't raise expected KeyboardInterrupt") # Raises AssertionError due to PYTHON-294 -- Mongo's response to the # previous find() is still waiting to be read on the socket, so the # request id's don't match. self.assertEqual( {'_id': 1}, db.foo.find().next() ) def test_operation_failure_without_request(self): # Ensure MongoClient doesn't close socket after it gets an error # response to getLastError. PYTHON-395. c = get_client() pool = c._MongoClient__pool self.assertEqual(1, len(pool.sockets)) old_sock_info = iter(pool.sockets).next() c.pymongo_test.test.drop() c.pymongo_test.test.insert({'_id': 'foo'}) self.assertRaises( OperationFailure, c.pymongo_test.test.insert, {'_id': 'foo'}) self.assertEqual(1, len(pool.sockets)) new_sock_info = iter(pool.sockets).next() self.assertEqual(old_sock_info, new_sock_info) def test_operation_failure_with_request(self): # Ensure MongoClient doesn't close socket after it gets an error # response to getLastError. PYTHON-395. c = get_client(auto_start_request=True) pool = c._MongoClient__pool # MongoClient has reserved a socket for this thread self.assertTrue(isinstance(pool._get_request_state(), SocketInfo)) old_sock_info = pool._get_request_state() c.pymongo_test.test.drop() c.pymongo_test.test.insert({'_id': 'foo'}) self.assertRaises( OperationFailure, c.pymongo_test.test.insert, {'_id': 'foo'}) # OperationFailure doesn't affect the request socket self.assertEqual(old_sock_info, pool._get_request_state()) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_code.py000066400000000000000000000064501223300253600165150ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tests for the Code wrapper.""" import unittest import sys sys.path[0:0] = [""] from bson.code import Code class TestCode(unittest.TestCase): def setUp(self): pass def test_types(self): self.assertRaises(TypeError, Code, 5) self.assertRaises(TypeError, Code, None) self.assertRaises(TypeError, Code, "aoeu", 5) self.assertRaises(TypeError, Code, u"aoeu", 5) self.assertTrue(Code("aoeu")) self.assertTrue(Code(u"aoeu")) self.assertTrue(Code("aoeu", {})) self.assertTrue(Code(u"aoeu", {})) def test_read_only(self): c = Code("blah") def set_c(): c.scope = 5 self.assertRaises(AttributeError, set_c) def test_code(self): a_string = "hello world" a_code = Code("hello world") self.assertTrue(a_code.startswith("hello")) self.assertTrue(a_code.endswith("world")) self.assertTrue(isinstance(a_code, Code)) self.assertFalse(isinstance(a_string, Code)) self.assertEqual(a_code.scope, {}) a_code.scope["my_var"] = 5 self.assertEqual(a_code.scope, {"my_var": 5}) def test_repr(self): c = Code("hello world") self.assertEqual(repr(c), "Code('hello world', {})") c.scope["foo"] = "bar" self.assertEqual(repr(c), "Code('hello world', {'foo': 'bar'})") c = Code("hello world", {"blah": 3}) self.assertEqual(repr(c), "Code('hello world', {'blah': 3})") c = Code("\x08\xFF") self.assertEqual(repr(c), "Code(%s, {})" % (repr("\x08\xFF"),)) def test_equality(self): b = Code("hello") c = Code("hello", {"foo": 5}) self.assertNotEqual(b, c) self.assertEqual(c, Code("hello", {"foo": 5})) self.assertNotEqual(c, Code("hello", {"foo": 6})) self.assertEqual(b, Code("hello")) self.assertEqual(b, Code("hello", {})) self.assertNotEqual(b, Code("hello ")) self.assertNotEqual("hello", Code("hello")) # Explicitly test inequality self.assertFalse(c != Code("hello", {"foo": 5})) self.assertFalse(b != Code("hello")) self.assertFalse(b != Code("hello", {})) def test_scope_preserved(self): a = Code("hello") b = Code("hello", {"foo": 5}) self.assertEqual(a, Code(a)) self.assertEqual(b, Code(b)) self.assertNotEqual(a, Code(b)) self.assertNotEqual(b, Code(a)) def test_scope_kwargs(self): self.assertEqual({"a": 1}, Code("", a=1).scope) self.assertEqual({"a": 1}, Code("", {"a": 2}, a=1).scope) self.assertEqual({"a": 1, "b": 2, "c": 3}, Code("", {"b": 2}, a=1, c=3).scope) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_collection.py000066400000000000000000002503731223300253600177430ustar00rootroot00000000000000# -*- coding: utf-8 -*- # Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the collection module.""" import itertools import re import sys import threading import time import unittest import warnings from nose.plugins.skip import SkipTest sys.path[0:0] = [""] from bson.binary import Binary, UUIDLegacy, OLD_UUID_SUBTYPE, UUID_SUBTYPE from bson.code import Code from bson.dbref import DBRef from bson.objectid import ObjectId from bson.py3compat import b from bson.son import SON from pymongo import (ASCENDING, DESCENDING, GEO2D, GEOHAYSTACK, GEOSPHERE, HASHED) from pymongo import message as message_module from pymongo.collection import Collection from pymongo.cursor import Cursor from pymongo.son_manipulator import SONManipulator from pymongo.errors import (ConfigurationError, DuplicateKeyError, InvalidDocument, InvalidName, InvalidOperation, OperationFailure, TimeoutError) from test.test_client import get_client from test.utils import is_mongos, joinall, enable_text_search from test import (qcheck, version) have_uuid = True try: import uuid except ImportError: have_uuid = False class TestCollection(unittest.TestCase): def setUp(self): self.client = get_client() self.db = self.client.pymongo_test def tearDown(self): self.db.drop_collection("test_large_limit") self.db = None self.client = None def test_collection(self): self.assertRaises(TypeError, Collection, self.db, 5) def make_col(base, name): return base[name] self.assertRaises(InvalidName, make_col, self.db, "") self.assertRaises(InvalidName, make_col, self.db, "te$t") self.assertRaises(InvalidName, make_col, self.db, ".test") self.assertRaises(InvalidName, make_col, self.db, "test.") self.assertRaises(InvalidName, make_col, self.db, "tes..t") self.assertRaises(InvalidName, make_col, self.db.test, "") self.assertRaises(InvalidName, make_col, self.db.test, "te$t") self.assertRaises(InvalidName, make_col, self.db.test, ".test") self.assertRaises(InvalidName, make_col, self.db.test, "test.") self.assertRaises(InvalidName, make_col, self.db.test, "tes..t") self.assertRaises(InvalidName, make_col, self.db.test, "tes\x00t") self.assertTrue(isinstance(self.db.test, Collection)) self.assertEqual(self.db.test, self.db["test"]) self.assertEqual(self.db.test, Collection(self.db, "test")) self.assertEqual(self.db.test.mike, self.db["test.mike"]) self.assertEqual(self.db.test["mike"], self.db["test.mike"]) self.db.drop_collection('test') self.assertFalse('test' in self.db.collection_names()) # No exception self.db.drop_collection('test') def test_create_index(self): db = self.db self.assertRaises(TypeError, db.test.create_index, 5) self.assertRaises(TypeError, db.test.create_index, {"hello": 1}) self.assertRaises(TypeError, db.test.ensure_index, {"hello": 1}, cache_for='foo') self.assertRaises(TypeError, db.test.ensure_index, {"hello": 1}, ttl='foo') self.assertRaises(ValueError, db.test.create_index, []) db.test.drop_indexes() self.assertEqual(db.system.indexes.find({"ns": u"pymongo_test.test"}) .count(), 1) db.test.create_index("hello") db.test.create_index([("hello", DESCENDING), ("world", ASCENDING)]) count = 0 for _ in db.system.indexes.find({"ns": u"pymongo_test.test"}): count += 1 self.assertEqual(count, 3) db.test.drop_indexes() ix = db.test.create_index([("hello", DESCENDING), ("world", ASCENDING)], name="hello_world") self.assertEqual(ix, "hello_world") db.test.drop_indexes() self.assertEqual(db.system.indexes.find({"ns": u"pymongo_test.test"}) .count(), 1) db.test.create_index("hello") self.assertTrue(u"hello_1" in [a["name"] for a in db.system.indexes .find({"ns": u"pymongo_test.test"})]) db.test.drop_indexes() self.assertEqual(db.system.indexes.find({"ns": u"pymongo_test.test"}) .count(), 1) db.test.create_index([("hello", DESCENDING), ("world", ASCENDING)]) self.assertTrue(u"hello_-1_world_1" in [a["name"] for a in db.system.indexes .find({"ns": u"pymongo_test.test"})]) db.test.drop() db.test.insert({'a': 1}) db.test.insert({'a': 1}) self.assertRaises(DuplicateKeyError, db.test.create_index, 'a', unique=True) def test_ensure_index(self): db = self.db self.assertRaises(TypeError, db.test.ensure_index, {"hello": 1}) self.assertRaises(TypeError, db.test.ensure_index, {"hello": 1}, cache_for='foo') self.assertRaises(TypeError, db.test.ensure_index, {"hello": 1}, ttl='foo') db.test.drop_indexes() self.assertEqual("hello_1", db.test.create_index("hello")) self.assertEqual("hello_1", db.test.create_index("hello")) self.assertEqual("goodbye_1", db.test.ensure_index("goodbye")) self.assertEqual(None, db.test.ensure_index("goodbye")) db.test.drop_indexes() self.assertEqual("foo", db.test.ensure_index("goodbye", name="foo")) self.assertEqual(None, db.test.ensure_index("goodbye", name="foo")) db.test.drop_indexes() self.assertEqual("goodbye_1", db.test.ensure_index("goodbye")) self.assertEqual(None, db.test.ensure_index("goodbye")) db.test.drop_index("goodbye_1") self.assertEqual("goodbye_1", db.test.ensure_index("goodbye")) self.assertEqual(None, db.test.ensure_index("goodbye")) db.drop_collection("test") self.assertEqual("goodbye_1", db.test.ensure_index("goodbye")) self.assertEqual(None, db.test.ensure_index("goodbye")) db.test.drop_index("goodbye_1") self.assertEqual("goodbye_1", db.test.create_index("goodbye")) self.assertEqual(None, db.test.ensure_index("goodbye")) db.test.drop_index("goodbye_1") self.assertEqual("goodbye_1", db.test.ensure_index("goodbye", cache_for=1)) time.sleep(1.2) self.assertEqual("goodbye_1", db.test.ensure_index("goodbye")) db.test.drop_index("goodbye_1") self.assertEqual("goodbye_1", db.test.create_index("goodbye", cache_for=1)) time.sleep(1.2) self.assertEqual("goodbye_1", db.test.ensure_index("goodbye")) # Make sure the expiration time is updated. self.assertEqual(None, db.test.ensure_index("goodbye")) # Clean up indexes for later tests db.test.drop_indexes() def test_deprecated_ttl_index_kwarg(self): db = self.db # In Python 2.6+ we could use the catch_warnings context # manager to test this warning nicely. As we can't do that # we must test raising errors before the ignore filter is applied. warnings.simplefilter("error", DeprecationWarning) self.assertRaises(DeprecationWarning, lambda: db.test.ensure_index("goodbye", ttl=10)) warnings.resetwarnings() warnings.simplefilter("ignore") self.assertEqual("goodbye_1", db.test.ensure_index("goodbye", ttl=10)) self.assertEqual(None, db.test.ensure_index("goodbye")) def test_ensure_unique_index_threaded(self): coll = self.db.test_unique_threaded coll.drop() coll.insert(({'foo': i} for i in xrange(10000))) class Indexer(threading.Thread): def run(self): try: coll.ensure_index('foo', unique=True) coll.insert({'foo': 'bar'}) coll.insert({'foo': 'bar'}) except OperationFailure: pass threads = [] for _ in xrange(10): t = Indexer() t.setDaemon(True) threads.append(t) for i in xrange(10): threads[i].start() joinall(threads) self.assertEqual(10001, coll.count()) coll.drop() def test_index_on_binary(self): db = self.db db.drop_collection("test") db.test.save({"bin": Binary(b("def"))}) db.test.save({"bin": Binary(b("abc"))}) db.test.save({"bin": Binary(b("ghi"))}) self.assertEqual(db.test.find({"bin": Binary(b("abc"))}) .explain()["nscanned"], 3) db.test.create_index("bin") self.assertEqual(db.test.find({"bin": Binary(b("abc"))}) .explain()["nscanned"], 1) def test_drop_index(self): db = self.db db.test.drop_indexes() db.test.create_index("hello") name = db.test.create_index("goodbye") self.assertEqual(db.system.indexes.find({"ns": u"pymongo_test.test"}) .count(), 3) self.assertEqual(name, "goodbye_1") db.test.drop_index(name) self.assertEqual(db.system.indexes.find({"ns": u"pymongo_test.test"}) .count(), 2) self.assertTrue(u"hello_1" in [a["name"] for a in db.system.indexes .find({"ns": u"pymongo_test.test"})]) db.test.drop_indexes() db.test.create_index("hello") name = db.test.create_index("goodbye") self.assertEqual(db.system.indexes.find({"ns": u"pymongo_test.test"}) .count(), 3) self.assertEqual(name, "goodbye_1") db.test.drop_index([("goodbye", ASCENDING)]) self.assertEqual(db.system.indexes.find({"ns": u"pymongo_test.test"}) .count(), 2) self.assertTrue(u"hello_1" in [a["name"] for a in db.system.indexes .find({"ns": u"pymongo_test.test"})]) def test_reindex(self): db = self.db db.drop_collection("test") db.test.insert({"foo": "bar", "who": "what", "when": "how"}) db.test.create_index("foo") db.test.create_index("who") db.test.create_index("when") info = db.test.index_information() def check_result(result): self.assertEqual(4, result['nIndexes']) self.assertEqual(4, result['nIndexesWas']) indexes = result['indexes'] names = [idx['name'] for idx in indexes] for name in names: self.assertTrue(name in info) for key in info: self.assertTrue(key in names) reindexed = db.test.reindex() if 'raw' in reindexed: # mongos for result in reindexed['raw'].itervalues(): check_result(result) else: check_result(reindexed) def test_index_info(self): db = self.db db.test.drop_indexes() db.test.remove({}) db.test.save({}) # create collection self.assertEqual(len(db.test.index_information()), 1) self.assertTrue("_id_" in db.test.index_information()) db.test.create_index("hello") self.assertEqual(len(db.test.index_information()), 2) self.assertEqual(db.test.index_information()["hello_1"]["key"], [("hello", ASCENDING)]) db.test.create_index([("hello", DESCENDING), ("world", ASCENDING)], unique=True) self.assertEqual(db.test.index_information()["hello_1"]["key"], [("hello", ASCENDING)]) self.assertEqual(len(db.test.index_information()), 3) self.assertEqual([("hello", DESCENDING), ("world", ASCENDING)], db.test.index_information()["hello_-1_world_1"]["key"] ) self.assertEqual(True, db.test.index_information()["hello_-1_world_1"]["unique"]) def test_index_geo2d(self): db = self.db db.test.drop_indexes() self.assertEqual('loc_2d', db.test.create_index([("loc", GEO2D)])) index_info = db.test.index_information()['loc_2d'] self.assertEqual([('loc', '2d')], index_info['key']) def test_index_haystack(self): if is_mongos(self.db.connection): raise SkipTest("geoSearch is not supported by mongos") db = self.db db.test.drop_indexes() db.test.remove() _id = db.test.insert({ "pos": {"long": 34.2, "lat": 33.3}, "type": "restaurant" }) db.test.insert({ "pos": {"long": 34.2, "lat": 37.3}, "type": "restaurant" }) db.test.insert({ "pos": {"long": 59.1, "lat": 87.2}, "type": "office" }) db.test.create_index( [("pos", GEOHAYSTACK), ("type", ASCENDING)], bucket_size=1 ) results = db.command(SON([ ("geoSearch", "test"), ("near", [33, 33]), ("maxDistance", 6), ("search", {"type": "restaurant"}), ("limit", 30), ]))['results'] self.assertEqual(2, len(results)) self.assertEqual({ "_id": _id, "pos": {"long": 34.2, "lat": 33.3}, "type": "restaurant" }, results[0]) def test_index_text(self): if not version.at_least(self.client, (2, 3, 2)): raise SkipTest("Text search requires server >=2.3.2.") if is_mongos(self.client): raise SkipTest("setParameter does not work through mongos") enable_text_search(self.client) db = self.db db.test.drop_indexes() self.assertEqual("t_text", db.test.create_index([("t", "text")])) index_info = db.test.index_information()["t_text"] self.assertTrue("weights" in index_info) db.test.drop_indexes() def test_index_2dsphere(self): if not version.at_least(self.client, (2, 3, 2)): raise SkipTest("2dsphere indexing requires server >=2.3.2.") db = self.db db.test.drop_indexes() self.assertEqual("geo_2dsphere", db.test.create_index([("geo", GEOSPHERE)])) poly = {"type": "Polygon", "coordinates": [[[40,5], [40,6], [41,6], [41,5], [40,5]]]} query = {"geo": {"$within": {"$geometry": poly}}} self.assertTrue( db.test.find(query).explain()['cursor'].startswith('S2Cursor')) db.test.drop_indexes() def test_index_hashed(self): if not version.at_least(self.client, (2, 3, 2)): raise SkipTest("hashed indexing requires server >=2.3.2.") db = self.db db.test.drop_indexes() self.assertEqual("a_hashed", db.test.create_index([("a", HASHED)])) self.assertEqual("BtreeCursor a_hashed", db.test.find({'a': 1}).explain()['cursor']) db.test.drop_indexes() def test_index_sparse(self): db = self.db db.test.drop_indexes() db.test.create_index([('key', ASCENDING)], sparse=True) self.assertTrue(db.test.index_information()['key_1']['sparse']) def test_index_background(self): db = self.db db.test.drop_indexes() db.test.create_index([('keya', ASCENDING)]) db.test.create_index([('keyb', ASCENDING)], background=False) db.test.create_index([('keyc', ASCENDING)], background=True) self.assertFalse('background' in db.test.index_information()['keya_1']) self.assertFalse(db.test.index_information()['keyb_1']['background']) self.assertTrue(db.test.index_information()['keyc_1']['background']) def _drop_dups_setup(self, db): db.drop_collection('test') db.test.insert({'i': 1}) db.test.insert({'i': 2}) db.test.insert({'i': 2}) # duplicate db.test.insert({'i': 3}) def test_index_drop_dups(self): # Try dropping duplicates db = self.db self._drop_dups_setup(db) if version.at_least(db.connection, (1, 9, 2)): # No error, just drop the duplicate db.test.create_index( [('i', ASCENDING)], unique=True, drop_dups=True ) else: # https://jira.mongodb.org/browse/SERVER-2054 "Creating an index # with dropDups shouldn't assert". On Mongo < 1.9.2, the duplicate # is dropped & the index created, but an error is thrown. def test_create(): db.test.create_index( [('i', ASCENDING)], unique=True, drop_dups=True ) self.assertRaises(DuplicateKeyError, test_create) # Duplicate was dropped self.assertEqual(3, db.test.count()) # Index was created, plus the index on _id self.assertEqual(2, len(db.test.index_information())) def test_index_dont_drop_dups(self): # Try *not* dropping duplicates db = self.db self._drop_dups_setup(db) # There's a duplicate def test_create(): db.test.create_index( [('i', ASCENDING)], unique=True, drop_dups=False ) self.assertRaises(DuplicateKeyError, test_create) # Duplicate wasn't dropped self.assertEqual(4, db.test.count()) # Index wasn't created, only the default index on _id self.assertEqual(1, len(db.test.index_information())) def test_field_selection(self): db = self.db db.drop_collection("test") doc = {"a": 1, "b": 5, "c": {"d": 5, "e": 10}} db.test.insert(doc) # Test field inclusion doc = db.test.find({}, ["_id"]).next() self.assertEqual(doc.keys(), ["_id"]) doc = db.test.find({}, ["a"]).next() l = doc.keys() l.sort() self.assertEqual(l, ["_id", "a"]) doc = db.test.find({}, ["b"]).next() l = doc.keys() l.sort() self.assertEqual(l, ["_id", "b"]) doc = db.test.find({}, ["c"]).next() l = doc.keys() l.sort() self.assertEqual(l, ["_id", "c"]) doc = db.test.find({}, ["a"]).next() self.assertEqual(doc["a"], 1) doc = db.test.find({}, ["b"]).next() self.assertEqual(doc["b"], 5) doc = db.test.find({}, ["c"]).next() self.assertEqual(doc["c"], {"d": 5, "e": 10}) # Test inclusion of fields with dots doc = db.test.find({}, ["c.d"]).next() self.assertEqual(doc["c"], {"d": 5}) doc = db.test.find({}, ["c.e"]).next() self.assertEqual(doc["c"], {"e": 10}) doc = db.test.find({}, ["b", "c.e"]).next() self.assertEqual(doc["c"], {"e": 10}) doc = db.test.find({}, ["b", "c.e"]).next() l = doc.keys() l.sort() self.assertEqual(l, ["_id", "b", "c"]) doc = db.test.find({}, ["b", "c.e"]).next() self.assertEqual(doc["b"], 5) # Test field exclusion doc = db.test.find({}, {"a": False, "b": 0}).next() l = doc.keys() l.sort() self.assertEqual(l, ["_id", "c"]) doc = db.test.find({}, {"_id": False}).next() l = doc.keys() self.assertFalse("_id" in l) def test_options(self): db = self.db db.drop_collection("test") db.test.save({}) self.assertEqual(db.test.options(), {}) self.assertEqual(db.test.doesnotexist.options(), {}) db.drop_collection("test") if version.at_least(db.connection, (1, 9)): db.create_collection("test", capped=True, size=1000) self.assertEqual(db.test.options(), {"capped": True, 'size': 1000}) else: db.create_collection("test", capped=True) self.assertEqual(db.test.options(), {"capped": True}) db.drop_collection("test") def test_insert_find_one(self): db = self.db db.test.remove({}) self.assertEqual(0, len(list(db.test.find()))) doc = {"hello": u"world"} id = db.test.insert(doc) self.assertEqual(1, len(list(db.test.find()))) self.assertEqual(doc, db.test.find_one()) self.assertEqual(doc["_id"], id) self.assertTrue(isinstance(id, ObjectId)) doc_class = None # Work around http://bugs.jython.org/issue1728 if (sys.platform.startswith('java') and sys.version_info[:3] >= (2, 5, 2)): doc_class = SON def remove_insert_find_one(doc): db.test.remove({}) db.test.insert(doc) # SON equality is order sensitive. return db.test.find_one(as_class=doc_class) == doc.to_dict() qcheck.check_unittest(self, remove_insert_find_one, qcheck.gen_mongo_dict(3)) def test_generator_insert(self): db = self.db db.test.remove({}) self.assertEqual(db.test.find().count(), 0) db.test.insert(({'a': i} for i in xrange(5)), manipulate=False) self.assertEqual(5, db.test.count()) db.test.remove({}) def test_remove_all(self): self.db.test.remove() self.assertEqual(0, self.db.test.count()) self.db.test.insert({"x": 1}) self.db.test.insert({"y": 1}) self.assertEqual(2, self.db.test.count()) self.db.test.remove() self.assertEqual(0, self.db.test.count()) def test_find_w_fields(self): db = self.db db.test.remove({}) db.test.insert({"x": 1, "mike": "awesome", "extra thing": "abcdefghijklmnopqrstuvwxyz"}) self.assertEqual(1, db.test.count()) doc = db.test.find({}).next() self.assertTrue("x" in doc) doc = db.test.find({}).next() self.assertTrue("mike" in doc) doc = db.test.find({}).next() self.assertTrue("extra thing" in doc) doc = db.test.find({}, ["x", "mike"]).next() self.assertTrue("x" in doc) doc = db.test.find({}, ["x", "mike"]).next() self.assertTrue("mike" in doc) doc = db.test.find({}, ["x", "mike"]).next() self.assertFalse("extra thing" in doc) doc = db.test.find({}, ["mike"]).next() self.assertFalse("x" in doc) doc = db.test.find({}, ["mike"]).next() self.assertTrue("mike" in doc) doc = db.test.find({}, ["mike"]).next() self.assertFalse("extra thing" in doc) def test_fields_specifier_as_dict(self): db = self.db db.test.remove({}) db.test.insert({"x": [1, 2, 3], "mike": "awesome"}) self.assertEqual([1, 2, 3], db.test.find_one()["x"]) if version.at_least(db.connection, (1, 5, 1)): self.assertEqual([2, 3], db.test.find_one(fields={"x": {"$slice": -2}})["x"]) self.assertTrue("x" not in db.test.find_one(fields={"x": 0})) self.assertTrue("mike" in db.test.find_one(fields={"x": 0})) def test_find_w_regex(self): db = self.db db.test.remove({}) db.test.insert({"x": "hello_world"}) db.test.insert({"x": "hello_mike"}) db.test.insert({"x": "hello_mikey"}) db.test.insert({"x": "hello_test"}) self.assertEqual(db.test.find().count(), 4) self.assertEqual(db.test.find({"x": re.compile("^hello.*")}).count(), 4) self.assertEqual(db.test.find({"x": re.compile("ello")}).count(), 4) self.assertEqual(db.test.find({"x": re.compile("^hello$")}).count(), 0) self.assertEqual(db.test.find({"x": re.compile("^hello_mi.*$")}).count(), 2) def test_id_can_be_anything(self): db = self.db db.test.remove({}) auto_id = {"hello": "world"} db.test.insert(auto_id) self.assertTrue(isinstance(auto_id["_id"], ObjectId)) numeric = {"_id": 240, "hello": "world"} db.test.insert(numeric) self.assertEqual(numeric["_id"], 240) object = {"_id": numeric, "hello": "world"} db.test.insert(object) self.assertEqual(object["_id"], numeric) for x in db.test.find(): self.assertEqual(x["hello"], u"world") self.assertTrue("_id" in x) def test_iteration(self): db = self.db def iterate(): [a for a in db.test] self.assertRaises(TypeError, iterate) def test_invalid_key_names(self): db = self.db db.test.drop() db.test.insert({"hello": "world"}) db.test.insert({"hello": {"hello": "world"}}) self.assertRaises(InvalidDocument, db.test.insert, {"$hello": "world"}) self.assertRaises(InvalidDocument, db.test.insert, {"hello": {"$hello": "world"}}) db.test.insert({"he$llo": "world"}) db.test.insert({"hello": {"hello$": "world"}}) self.assertRaises(InvalidDocument, db.test.insert, {".hello": "world"}) self.assertRaises(InvalidDocument, db.test.insert, {"hello": {".hello": "world"}}) self.assertRaises(InvalidDocument, db.test.insert, {"hello.": "world"}) self.assertRaises(InvalidDocument, db.test.insert, {"hello": {"hello.": "world"}}) self.assertRaises(InvalidDocument, db.test.insert, {"hel.lo": "world"}) self.assertRaises(InvalidDocument, db.test.insert, {"hello": {"hel.lo": "world"}}) def test_insert_multiple(self): db = self.db db.drop_collection("test") doc1 = {"hello": u"world"} doc2 = {"hello": u"mike"} self.assertEqual(db.test.find().count(), 0) ids = db.test.insert([doc1, doc2]) self.assertEqual(db.test.find().count(), 2) self.assertEqual(doc1, db.test.find_one({"hello": u"world"})) self.assertEqual(doc2, db.test.find_one({"hello": u"mike"})) self.assertEqual(2, len(ids)) self.assertEqual(doc1["_id"], ids[0]) self.assertEqual(doc2["_id"], ids[1]) id = db.test.insert([{"hello": 1}]) self.assertTrue(isinstance(id, list)) self.assertEqual(1, len(id)) self.assertRaises(InvalidOperation, db.test.insert, []) def test_insert_multiple_with_duplicate(self): db = self.db db.drop_collection("test") db.test.ensure_index([('i', ASCENDING)], unique=True) # No error db.test.insert([{'i': i} for i in range(5, 10)], w=0) db.test.remove() # No error db.test.insert([{'i': 1}] * 2, w=0) self.assertEqual(1, db.test.count()) self.assertRaises( DuplicateKeyError, lambda: db.test.insert([{'i': 2}] * 2), ) db.drop_collection("test") db.write_concern['w'] = 0 db.test.ensure_index([('i', ASCENDING)], unique=True) # No error db.test.insert([{'i': 1}] * 2) self.assertEqual(1, db.test.count()) # Implied safe self.assertRaises( DuplicateKeyError, lambda: db.test.insert([{'i': 2}] * 2, j=True), ) # Explicit safe self.assertRaises( DuplicateKeyError, lambda: db.test.insert([{'i': 2}] * 2, w=1), ) # Misconfigured value for safe self.assertRaises( TypeError, lambda: db.test.insert([{'i': 2}] * 2, safe=1), ) def test_insert_iterables(self): db = self.db self.assertRaises(TypeError, db.test.insert, 4) self.assertRaises(TypeError, db.test.insert, None) self.assertRaises(TypeError, db.test.insert, True) db.drop_collection("test") self.assertEqual(db.test.find().count(), 0) ids = db.test.insert(({"hello": u"world"}, {"hello": u"world"})) self.assertEqual(db.test.find().count(), 2) db.drop_collection("test") self.assertEqual(db.test.find().count(), 0) ids = db.test.insert(itertools.imap(lambda x: {"hello": "world"}, itertools.repeat(None, 10))) self.assertEqual(db.test.find().count(), 10) def test_insert_manipulate_false(self): # Test three aspects of insert with manipulate=False: # 1. The return value is None or [None] as appropriate. # 2. _id is not set on the passed-in document object. # 3. _id is not sent to server. if not version.at_least(self.db.connection, (2, 0)): raise SkipTest('Need at least MongoDB 2.0') collection_name = 'test_insert_manipulate_false' try: self.db.drop_collection(collection_name) # A capped collection, so server doesn't set _id automatically. collection = self.db.create_collection( collection_name, capped=True, autoIndexId=False, size=1000) oid = ObjectId() doc = {'a': oid} # The return value is None. self.assertTrue(collection.insert(doc, manipulate=False) is None) # insert() shouldn't set _id on the passed-in document object. self.assertEqual({'a': oid}, doc) # _id is not sent to server. self.assertEqual(doc, collection.find_one()) # Bulk insert. The return value is a list of None. self.assertEqual([None], collection.insert([{}], manipulate=False)) ids = collection.insert([{}, {}], manipulate=False) self.assertEqual([None, None], ids) finally: self.db.drop_collection(collection_name) def test_save(self): self.db.drop_collection("test") # Save a doc with autogenerated id id = self.db.test.save({"hello": "world"}) self.assertEqual(self.db.test.find_one()["_id"], id) self.assertTrue(isinstance(id, ObjectId)) # Save a doc with explicit id self.db.test.save({"_id": "explicit_id", "hello": "bar"}) doc = self.db.test.find_one({"_id": "explicit_id"}) self.assertEqual(doc['_id'], 'explicit_id') self.assertEqual(doc['hello'], 'bar') # Save docs with _id field already present (shouldn't create new docs) self.assertEqual(2, self.db.test.count()) self.db.test.save({'_id': id, 'hello': 'world'}) self.assertEqual(2, self.db.test.count()) self.db.test.save({'_id': 'explicit_id', 'hello': 'baz'}) self.assertEqual(2, self.db.test.count()) self.assertEqual( 'baz', self.db.test.find_one({'_id': 'explicit_id'})['hello'] ) # Safe mode self.db.test.create_index("hello", unique=True) # No exception, even though we duplicate the first doc's "hello" value self.db.test.save({'_id': 'explicit_id', 'hello': 'world'}, w=0) self.assertRaises( DuplicateKeyError, self.db.test.save, {'_id': 'explicit_id', 'hello': 'world'}) def test_save_with_invalid_key(self): self.db.drop_collection("test") self.assertTrue(self.db.test.insert({"hello": "world"})) doc = self.db.test.find_one() doc['a.b'] = 'c' self.assertRaises(InvalidDocument, self.db.test.save, doc) def test_unique_index(self): db = self.db db.drop_collection("test") db.test.create_index("hello") db.test.save({"hello": "world"}) db.test.save({"hello": "mike"}) db.test.save({"hello": "world"}) self.assertFalse(db.error()) db.drop_collection("test") db.test.create_index("hello", unique=True) db.test.save({"hello": "world"}) db.test.save({"hello": "mike"}) db.test.save({"hello": "world"}, w=0) self.assertTrue(db.error()) def test_duplicate_key_error(self): db = self.db db.drop_collection("test") db.test.create_index("x", unique=True) db.test.insert({"_id": 1, "x": 1}) db.test.insert({"_id": 2, "x": 2}) # No error db.test.insert({"_id": 1, "x": 1}, safe=False) db.test.save({"_id": 1, "x": 1}, safe=False) db.test.insert({"_id": 2, "x": 2}, safe=False) db.test.save({"_id": 2, "x": 2}, safe=False) db.test.insert({"_id": 1, "x": 1}, w=0) db.test.save({"_id": 1, "x": 1}, w=0) db.test.insert({"_id": 2, "x": 2}, w=0) db.test.save({"_id": 2, "x": 2}, w=0) # But all those statements didn't do anything self.assertEqual(2, db.test.count()) expected_error = OperationFailure if version.at_least(db.connection, (1, 3)): expected_error = DuplicateKeyError self.assertRaises(expected_error, db.test.insert, {"_id": 1}) self.assertRaises(expected_error, db.test.insert, {"x": 1}) self.assertRaises(expected_error, db.test.save, {"x": 2}) self.assertRaises(expected_error, db.test.update, {"x": 1}, {"$inc": {"x": 1}}) def test_continue_on_error(self): db = self.db if not version.at_least(db.connection, (1, 9, 1)): raise SkipTest("continue_on_error requires MongoDB >= 1.9.1") db.drop_collection("test") oid = db.test.insert({"one": 1}) self.assertEqual(1, db.test.count()) docs = [] docs.append({"_id": oid, "two": 2}) docs.append({"three": 3}) docs.append({"four": 4}) docs.append({"five": 5}) db.test.insert(docs, manipulate=False, w=0) self.assertEqual(11000, db.error()['code']) self.assertEqual(1, db.test.count()) db.test.insert(docs, manipulate=False, continue_on_error=True, w=0) self.assertEqual(11000, db.error()['code']) self.assertEqual(4, db.test.count()) db.drop_collection("test") oid = db.test.insert({"_id": oid, "one": 1}, w=0) self.assertEqual(1, db.test.count()) docs[0].pop("_id") docs[2]["_id"] = oid db.test.insert(docs, manipulate=False, w=0) self.assertEqual(11000, db.error()['code']) self.assertEqual(3, db.test.count()) db.test.insert(docs, manipulate=False, continue_on_error=True, w=0) self.assertEqual(11000, db.error()['code']) self.assertEqual(6, db.test.count()) def test_error_code(self): try: self.db.test.update({}, {"$thismodifierdoesntexist": 1}) self.fail() except OperationFailure, e: if version.at_least(self.db.connection, (1, 3)): if e.code not in (10147, 17009): self.fail() def test_index_on_subfield(self): db = self.db db.drop_collection("test") db.test.insert({"hello": {"a": 4, "b": 5}}) db.test.insert({"hello": {"a": 7, "b": 2}}) db.test.insert({"hello": {"a": 4, "b": 10}}) db.drop_collection("test") db.test.create_index("hello.a", unique=True) db.test.insert({"hello": {"a": 4, "b": 5}}) db.test.insert({"hello": {"a": 7, "b": 2}}) self.assertRaises(DuplicateKeyError, db.test.insert, {"hello": {"a": 4, "b": 10}}) def test_safe_insert(self): db = self.db db.drop_collection("test") a = {"hello": "world"} db.test.insert(a) db.test.insert(a, w=0) self.assertTrue("E11000" in db.error()["err"]) self.assertRaises(OperationFailure, db.test.insert, a) def test_update(self): db = self.db db.drop_collection("test") id1 = db.test.save({"x": 5}) db.test.update({}, {"$inc": {"x": 1}}) self.assertEqual(db.test.find_one(id1)["x"], 6) id2 = db.test.save({"x": 1}) db.test.update({"x": 6}, {"$inc": {"x": 1}}) self.assertEqual(db.test.find_one(id1)["x"], 7) self.assertEqual(db.test.find_one(id2)["x"], 1) def test_multi_update(self): db = self.db if not version.at_least(db.connection, (1, 1, 3, -1)): raise SkipTest("multi-update requires MongoDB >= 1.1.3") db.drop_collection("test") db.test.save({"x": 4, "y": 3}) db.test.save({"x": 5, "y": 5}) db.test.save({"x": 4, "y": 4}) db.test.update({"x": 4}, {"$set": {"y": 5}}, multi=True) self.assertEqual(3, db.test.count()) for doc in db.test.find(): self.assertEqual(5, doc["y"]) self.assertEqual(2, db.test.update({"x": 4}, {"$set": {"y": 6}}, multi=True)["n"]) def test_upsert(self): db = self.db db.drop_collection("test") db.test.update({"page": "/"}, {"$inc": {"count": 1}}, upsert=True) db.test.update({"page": "/"}, {"$inc": {"count": 1}}, upsert=True) self.assertEqual(1, db.test.count()) self.assertEqual(2, db.test.find_one()["count"]) def test_safe_update(self): db = self.db v113minus = version.at_least(db.connection, (1, 1, 3, -1)) v19 = version.at_least(db.connection, (1, 9)) db.drop_collection("test") db.test.create_index("x", unique=True) db.test.insert({"x": 5}) id = db.test.insert({"x": 4}) self.assertEqual( None, db.test.update({"_id": id}, {"$inc": {"x": 1}}, w=0)) if v19: self.assertTrue("E11000" in db.error()["err"]) elif v113minus: self.assertTrue(db.error()["err"].startswith("E11001")) else: self.assertTrue(db.error()["err"].startswith("E12011")) self.assertRaises(OperationFailure, db.test.update, {"_id": id}, {"$inc": {"x": 1}}) self.assertEqual(1, db.test.update({"_id": id}, {"$inc": {"x": 2}})["n"]) self.assertEqual(0, db.test.update({"_id": "foo"}, {"$inc": {"x": 2}})["n"]) def test_update_with_invalid_keys(self): self.db.drop_collection("test") self.assertTrue(self.db.test.insert({"hello": "world"})) doc = self.db.test.find_one() doc['a.b'] = 'c' # Replace self.assertRaises(InvalidDocument, self.db.test.update, {"hello": "world"}, doc) # Upsert self.assertRaises(InvalidDocument, self.db.test.update, {"foo": "bar"}, doc, upsert=True) # Check that the last two ops didn't actually modify anything self.assertTrue('a.b' not in self.db.test.find_one()) # Modify shouldn't check keys... self.assertTrue(self.db.test.update({"hello": "world"}, {"$set": {"foo.bar": "baz"}}, upsert=True)) # I know this seems like testing the server but I'd like to be notified # by CI if the server's behavior changes here. doc = SON([("$set", {"foo.bar": "bim"}), ("hello", "world")]) self.assertRaises(OperationFailure, self.db.test.update, {"hello": "world"}, doc, upsert=True) # This is going to cause keys to be checked and raise InvalidDocument. # That's OK assuming the server's behavior in the previous assert # doesn't change. If the behavior changes checking the first key for # '$' in update won't be good enough anymore. doc = SON([("hello", "world"), ("$set", {"foo.bar": "bim"})]) self.assertRaises(InvalidDocument, self.db.test.update, {"hello": "world"}, doc, upsert=True) # Replace with empty document self.assertNotEqual(0, self.db.test.update({"hello": "world"}, {})['n']) def test_safe_save(self): db = self.db db.drop_collection("test") db.test.create_index("hello", unique=True) db.test.save({"hello": "world"}) db.test.save({"hello": "world"}, w=0) self.assertTrue("E11000" in db.error()["err"]) self.assertRaises(OperationFailure, db.test.save, {"hello": "world"}) def test_safe_remove(self): db = self.db db.drop_collection("test") db.create_collection("test", capped=True, size=1000) db.test.insert({"x": 1}) self.assertEqual(1, db.test.count()) self.assertEqual(None, db.test.remove({"x": 1}, w=0)) self.assertEqual(1, db.test.count()) if version.at_least(db.connection, (1, 1, 3, -1)): self.assertRaises(OperationFailure, db.test.remove, {"x": 1}) else: # Just test that it doesn't blow up db.test.remove({"x": 1}) db.drop_collection("test") db.test.insert({"x": 1}) db.test.insert({"x": 1}) self.assertEqual(2, db.test.remove({})["n"]) self.assertEqual(0, db.test.remove({})["n"]) def test_last_error_options(self): if not version.at_least(self.client, (1, 5, 1)): raise SkipTest("getLastError options require MongoDB >= 1.5.1") # XXX: Fix this if we ever have a replica set unittest env. # mongo >=1.7.6 errors with 'norepl' when w=2+ # and we aren't replicated. if not version.at_least(self.client, (1, 7, 6)): self.assertRaises(TimeoutError, self.db.test.save, {"x": 1}, w=2, wtimeout=1) self.assertRaises(TimeoutError, self.db.test.insert, {"x": 1}, w=2, wtimeout=1) self.assertRaises(TimeoutError, self.db.test.update, {"x": 1}, {"y": 2}, w=2, wtimeout=1) self.assertRaises(TimeoutError, self.db.test.remove, {"x": 1}, w=2, wtimeout=1) self.db.test.save({"x": 1}, w=1, wtimeout=1) self.db.test.insert({"x": 1}, w=1, wtimeout=1) self.db.test.remove({"x": 1}, w=1, wtimeout=1) self.db.test.update({"x": 1}, {"y": 2}, w=1, wtimeout=1) def test_manual_last_error(self): self.db.test.save({"x": 1}, w=0) self.db.command("getlasterror", w=1, wtimeout=1) def test_count(self): db = self.db db.drop_collection("test") self.assertEqual(db.test.count(), 0) db.test.save({}) db.test.save({}) self.assertEqual(db.test.count(), 2) db.test.save({'foo': 'bar'}) db.test.save({'foo': 'baz'}) self.assertEqual(db.test.find({'foo': 'bar'}).count(), 1) self.assertEqual(db.test.find({'foo': re.compile(r'ba.*')}).count(), 2) def test_aggregate(self): if not version.at_least(self.db.connection, (2, 1, 0)): raise SkipTest("The aggregate command requires MongoDB >= 2.1.0") db = self.db db.drop_collection("test") db.test.save({'foo': [1, 2]}) self.assertRaises(TypeError, db.test.aggregate, "wow") pipeline = {"$project": {"_id": False, "foo": True}} expected = {'ok': 1.0, 'result': [{'foo': [1, 2]}]} self.assertEqual(expected, db.test.aggregate(pipeline)) self.assertEqual(expected, db.test.aggregate([pipeline])) self.assertEqual(expected, db.test.aggregate((pipeline,))) def test_aggregation_cursor_validation(self): if not version.at_least(self.db.connection, (2, 5, 1)): raise SkipTest("Aggregation cursor requires MongoDB >= 2.5.1") db = self.db projection = {'$project': {'_id': '$_id'}} cursor = db.test.aggregate(projection, cursor={}) self.assertTrue(isinstance(cursor, Cursor)) self.assertRaises(InvalidOperation, cursor.rewind) self.assertRaises(InvalidOperation, cursor.clone) self.assertRaises(InvalidOperation, cursor.count) self.assertRaises(InvalidOperation, cursor.explain) def test_aggregation_cursor(self): if not version.at_least(self.db.connection, (2, 5, 1)): raise SkipTest("Aggregation cursor requires MongoDB >= 2.5.1") db = self.db # A small collection which returns only an initial batch, # and a larger one that requires a getMore. for collection_size in (10, 1000): db.drop_collection("test") db.test.insert([{'_id': i} for i in range(collection_size)]) expected_sum = sum(range(collection_size)) cursor = db.test.aggregate( {'$project': {'_id': '$_id'}}, cursor={}) self.assertEqual( expected_sum, sum(doc['_id'] for doc in cursor)) def test_group(self): db = self.db db.drop_collection("test") def group_checker(args, expected): eval = db.test.group(*args) self.assertEqual(eval, expected) self.assertEqual([], db.test.group([], {}, {"count": 0}, "function (obj, prev) { prev.count++; }" )) db.test.save({"a": 2}) db.test.save({"b": 5}) db.test.save({"a": 1}) self.assertEqual([{"count": 3}], db.test.group([], {}, {"count": 0}, "function (obj, prev) { prev.count++; }" )) self.assertEqual([{"count": 1}], db.test.group([], {"a": {"$gt": 1}}, {"count": 0}, "function (obj, prev) { prev.count++; }" )) db.test.save({"a": 2, "b": 3}) self.assertEqual([{"a": 2, "count": 2}, {"a": None, "count": 1}, {"a": 1, "count": 1}], db.test.group(["a"], {}, {"count": 0}, "function (obj, prev) { prev.count++; }" )) # modifying finalize self.assertEqual([{"a": 2, "count": 3}, {"a": None, "count": 2}, {"a": 1, "count": 2}], db.test.group(["a"], {}, {"count": 0}, "function (obj, prev) " "{ prev.count++; }", "function (obj) { obj.count++; }")) # returning finalize self.assertEqual([2, 1, 1], db.test.group(["a"], {}, {"count": 0}, "function (obj, prev) " "{ prev.count++; }", "function (obj) { return obj.count; }")) # keyf self.assertEqual([2, 2], db.test.group("function (obj) { if (obj.a == 2) " "{ return {a: true} }; " "return {b: true}; }", {}, {"count": 0}, "function (obj, prev) " "{ prev.count++; }", "function (obj) { return obj.count; }")) # no key self.assertEqual([{"count": 4}], db.test.group(None, {}, {"count": 0}, "function (obj, prev) { prev.count++; }" )) self.assertRaises(OperationFailure, db.test.group, [], {}, {}, "5 ++ 5") def test_group_with_scope(self): db = self.db db.drop_collection("test") db.test.save({"a": 1}) db.test.save({"b": 1}) reduce_function = "function (obj, prev) { prev.count += inc_value; }" self.assertEqual(2, db.test.group([], {}, {"count": 0}, Code(reduce_function, {"inc_value": 1}))[0]['count']) self.assertEqual(4, db.test.group([], {}, {"count": 0}, Code(reduce_function, {"inc_value": 2}))[0]['count']) self.assertEqual(1, db.test.group([], {}, {"count": 0}, Code(reduce_function, {"inc_value": 0.5}))[0]['count']) if version.at_least(db.connection, (1, 1)): self.assertEqual(2, db.test.group([], {}, {"count": 0}, Code(reduce_function, {"inc_value": 1}), )[0]['count']) self.assertEqual(4, db.test.group([], {}, {"count": 0}, Code(reduce_function, {"inc_value": 2}), )[0]['count']) self.assertEqual(1, db.test.group([], {}, {"count": 0}, Code(reduce_function, {"inc_value": 0.5}), )[0]['count']) def test_large_limit(self): db = self.db db.drop_collection("test_large_limit") db.test_large_limit.create_index([('x', 1)]) for i in range(2000): doc = {"x": i, "y": "mongomongo" * 1000} db.test_large_limit.insert(doc) # Wait for insert to complete; often mysteriously failing in Jenkins st = time.time() while ( len(list(db.test_large_limit.find())) < 2000 and time.time() - st < 30 ): time.sleep(1) self.assertEqual(2000, len(list(db.test_large_limit.find()))) i = 0 y = 0 for doc in db.test_large_limit.find(limit=1900).sort([('x', 1)]): i += 1 y += doc["x"] self.assertEqual(1900, i) self.assertEqual((1900 * 1899) / 2, y) def test_find_kwargs(self): db = self.db db.drop_collection("test") for i in range(10): db.test.insert({"x": i}) self.assertEqual(10, db.test.count()) sum = 0 for x in db.test.find({}, skip=4, limit=2): sum += x["x"] self.assertEqual(9, sum) def test_rename(self): db = self.db db.drop_collection("test") db.drop_collection("foo") self.assertRaises(TypeError, db.test.rename, 5) self.assertRaises(InvalidName, db.test.rename, "") self.assertRaises(InvalidName, db.test.rename, "te$t") self.assertRaises(InvalidName, db.test.rename, ".test") self.assertRaises(InvalidName, db.test.rename, "test.") self.assertRaises(InvalidName, db.test.rename, "tes..t") self.assertEqual(0, db.test.count()) self.assertEqual(0, db.foo.count()) for i in range(10): db.test.insert({"x": i}) self.assertEqual(10, db.test.count()) db.test.rename("foo") self.assertEqual(0, db.test.count()) self.assertEqual(10, db.foo.count()) x = 0 for doc in db.foo.find(): self.assertEqual(x, doc["x"]) x += 1 db.test.insert({}) self.assertRaises(OperationFailure, db.foo.rename, "test") db.foo.rename("test", dropTarget=True) # doesn't really test functionality, just that the option is set correctly def test_snapshot(self): db = self.db self.assertRaises(TypeError, db.test.find, snapshot=5) list(db.test.find(snapshot=True)) self.assertRaises(OperationFailure, list, db.test.find(snapshot=True).sort("foo", 1)) def test_find_one(self): db = self.db db.drop_collection("test") id = db.test.save({"hello": "world", "foo": "bar"}) self.assertEqual("world", db.test.find_one()["hello"]) self.assertEqual(db.test.find_one(id), db.test.find_one()) self.assertEqual(db.test.find_one(None), db.test.find_one()) self.assertEqual(db.test.find_one({}), db.test.find_one()) self.assertEqual(db.test.find_one({"hello": "world"}), db.test.find_one()) self.assertTrue("hello" in db.test.find_one(fields=["hello"])) self.assertTrue("hello" not in db.test.find_one(fields=["foo"])) self.assertEqual(["_id"], db.test.find_one(fields=[]).keys()) self.assertEqual(None, db.test.find_one({"hello": "foo"})) self.assertEqual(None, db.test.find_one(ObjectId())) def test_find_one_non_objectid(self): db = self.db db.drop_collection("test") db.test.save({"_id": 5}) self.assertTrue(db.test.find_one(5)) self.assertFalse(db.test.find_one(6)) def test_remove_non_objectid(self): db = self.db db.drop_collection("test") db.test.save({"_id": 5}) self.assertEqual(1, db.test.count()) db.test.remove(5) self.assertEqual(0, db.test.count()) def test_find_one_with_find_args(self): db = self.db db.drop_collection("test") db.test.save({"x": 1}) db.test.save({"x": 2}) db.test.save({"x": 3}) self.assertEqual(1, db.test.find_one()["x"]) self.assertEqual(2, db.test.find_one(skip=1, limit=2)["x"]) def test_find_with_sort(self): db = self.db db.drop_collection("test") db.test.save({"x": 2}) db.test.save({"x": 1}) db.test.save({"x": 3}) self.assertEqual(2, db.test.find_one()["x"]) self.assertEqual(1, db.test.find_one(sort=[("x", 1)])["x"]) self.assertEqual(3, db.test.find_one(sort=[("x", -1)])["x"]) def to_list(foo): return [bar["x"] for bar in foo] self.assertEqual([2, 1, 3], to_list(db.test.find())) self.assertEqual([1, 2, 3], to_list(db.test.find(sort=[("x", 1)]))) self.assertEqual([3, 2, 1], to_list(db.test.find(sort=[("x", -1)]))) self.assertRaises(TypeError, db.test.find, sort=5) self.assertRaises(TypeError, db.test.find, sort="hello") self.assertRaises(ValueError, db.test.find, sort=["hello", 1]) def test_insert_adds_id(self): doc = {"hello": "world"} self.db.test.insert(doc) self.assertTrue("_id" in doc) docs = [{"hello": "world"}, {"hello": "world"}] self.db.test.insert(docs) for doc in docs: self.assertTrue("_id" in doc) def test_save_adds_id(self): doc = {"hello": "jesse"} self.db.test.save(doc) self.assertTrue("_id" in doc) # TODO doesn't actually test functionality, just that it doesn't blow up def test_cursor_timeout(self): list(self.db.test.find(timeout=False)) list(self.db.test.find(timeout=True)) def test_exhaust(self): if is_mongos(self.db.connection): self.assertRaises(InvalidOperation, self.db.test.find, exhaust=True) return self.assertRaises(TypeError, self.db.test.find, exhaust=5) # Limit is incompatible with exhaust. self.assertRaises(InvalidOperation, self.db.test.find, exhaust=True, limit=5) cur = self.db.test.find(exhaust=True) self.assertRaises(InvalidOperation, cur.limit, 5) cur = self.db.test.find(limit=5) self.assertRaises(InvalidOperation, cur.add_option, 64) cur = self.db.test.find() cur.add_option(64) self.assertRaises(InvalidOperation, cur.limit, 5) self.db.drop_collection("test") # Insert enough documents to require more than one batch self.db.test.insert([{'i': i} for i in xrange(150)]) client = get_client(max_pool_size=1) socks = client._MongoClient__pool.sockets self.assertEqual(1, len(socks)) # Make sure the socket is returned after exhaustion. cur = client[self.db.name].test.find(exhaust=True) cur.next() self.assertEqual(0, len(socks)) for doc in cur: pass self.assertEqual(1, len(socks)) # Same as previous but don't call next() for doc in client[self.db.name].test.find(exhaust=True): pass self.assertEqual(1, len(socks)) # If the Cursor intance is discarded before being # completely interated we have to close and # discard the socket. cur = client[self.db.name].test.find(exhaust=True) cur.next() self.assertEqual(0, len(socks)) if sys.platform.startswith('java') or 'PyPy' in sys.version: # Don't wait for GC or use gc.collect(), it's unreliable. cur.close() cur = None # The socket should be discarded. self.assertEqual(0, len(socks)) def test_distinct(self): if not version.at_least(self.db.connection, (1, 1)): raise SkipTest("distinct command requires MongoDB >= 1.1") self.db.drop_collection("test") test = self.db.test test.save({"a": 1}) test.save({"a": 2}) test.save({"a": 2}) test.save({"a": 2}) test.save({"a": 3}) distinct = test.distinct("a") distinct.sort() self.assertEqual([1, 2, 3], distinct) distinct = test.find({'a': {'$gt': 1}}).distinct("a") distinct.sort() self.assertEqual([2, 3], distinct) self.db.drop_collection("test") test.save({"a": {"b": "a"}, "c": 12}) test.save({"a": {"b": "b"}, "c": 12}) test.save({"a": {"b": "c"}, "c": 12}) test.save({"a": {"b": "c"}, "c": 12}) distinct = test.distinct("a.b") distinct.sort() self.assertEqual(["a", "b", "c"], distinct) def test_query_on_query_field(self): self.db.drop_collection("test") self.db.test.save({"query": "foo"}) self.db.test.save({"bar": "foo"}) self.assertEqual(1, self.db.test.find({"query": {"$ne": None}}).count()) self.assertEqual(1, len(list(self.db.test.find({"query": {"$ne": None}}))) ) def test_min_query(self): self.db.drop_collection("test") self.db.test.save({"x": 1}) self.db.test.save({"x": 2}) self.db.test.create_index("x") self.assertEqual(1, len(list(self.db.test.find({"$min": {"x": 2}, "$query": {}})))) self.assertEqual(2, self.db.test.find({"$min": {"x": 2}, "$query": {}})[0]["x"]) def test_insert_large_document(self): max_size = self.db.connection.max_bson_size half_size = int(max_size / 2) if version.at_least(self.db.connection, (1, 7, 4)): self.assertEqual(max_size, 16777216) self.assertRaises(InvalidDocument, self.db.test.insert, {"foo": "x" * max_size}) self.assertRaises(InvalidDocument, self.db.test.save, {"foo": "x" * max_size}) self.assertRaises(InvalidDocument, self.db.test.insert, [{"x": 1}, {"foo": "x" * max_size}]) self.db.test.insert([{"foo": "x" * half_size}, {"foo": "x" * half_size}]) self.db.test.insert({"bar": "x"}) self.assertRaises(InvalidDocument, self.db.test.update, {"bar": "x"}, {"bar": "x" * (max_size - 14)}) self.db.test.update({"bar": "x"}, {"bar": "x" * (max_size - 15)}) def test_insert_large_batch(self): max_bson_size = self.db.connection.max_bson_size big_string = 'x' * (max_bson_size - 100) self.db.test.drop() self.assertEqual(0, self.db.test.count()) # Batch insert that requires 2 batches batch = [{'x': big_string}, {'x': big_string}, {'x': big_string}, {'x': big_string}] self.assertTrue(self.db.test.insert(batch, w=1)) self.assertEqual(4, self.db.test.count()) batch[1]['_id'] = batch[0]['_id'] # Test that inserts fail after first error, acknowledged. self.db.test.drop() self.assertRaises(DuplicateKeyError, self.db.test.insert, batch, w=1) self.assertEqual(1, self.db.test.count()) # Test that inserts fail after first error, unacknowledged. self.db.test.drop() self.assertTrue(self.db.test.insert(batch, w=0)) self.assertEqual(1, self.db.test.count()) # 2 batches, 2 errors, acknowledged, continue on error self.db.test.drop() batch[3]['_id'] = batch[2]['_id'] try: self.db.test.insert(batch, continue_on_error=True, w=1) except OperationFailure, e: # Make sure we report the last error, not the first. self.assertTrue(str(batch[2]['_id']) in str(e)) else: self.fail('OpreationFailure not raised.') # Only the first and third documents should be inserted. self.assertEqual(2, self.db.test.count()) # 2 batches, 2 errors, unacknowledged, continue on error self.db.test.drop() self.assertTrue(self.db.test.insert(batch, continue_on_error=True, w=0)) # Only the first and third documents should be inserted. self.assertEqual(2, self.db.test.count()) # Starting in PyMongo 2.6 we no longer use message.insert for inserts, but # message.insert is part of the public API. Do minimal testing here; there # isn't really a better place. def test_insert_message_creation(self): send = self.db.connection._send_message name = "%s.%s" % (self.db.name, "test") def do_insert(args): send(message_module.insert(*args), args[3]) self.db.drop_collection("test") self.db.test.insert({'_id': 0}, w=1) self.assertTrue(1, self.db.test.count()) simple_args = (name, [{'_id': 0}], True, False, {}, False, 3) gle_args = (name, [{'_id': 0}], True, True, {'w': 1}, False, 3) coe_args = (name, [{'_id': 0}, {'_id': 1}], True, True, {'w': 1}, True, 3) self.assertEqual(None, do_insert(simple_args)) self.assertTrue(1, self.db.test.count()) self.assertRaises(DuplicateKeyError, do_insert, gle_args) self.assertTrue(1, self.db.test.count()) self.assertRaises(DuplicateKeyError, do_insert, coe_args) self.assertTrue(2, self.db.test.count()) if have_uuid: doc = {'_id': 2, 'uuid': uuid.uuid4()} uuid_sub_args = (name, [doc], True, True, {'w': 1}, True, 6) do_insert(uuid_sub_args) coll = self.db.test self.assertNotEqual(doc, coll.find_one({'_id': 2})) coll.uuid_subtype = 6 self.assertEqual(doc, coll.find_one({'_id': 2})) def test_map_reduce(self): if not version.at_least(self.db.connection, (1, 1, 1)): raise SkipTest("mapReduce command requires MongoDB >= 1.1.1") db = self.db db.drop_collection("test") db.test.insert({"id": 1, "tags": ["dog", "cat"]}) db.test.insert({"id": 2, "tags": ["cat"]}) db.test.insert({"id": 3, "tags": ["mouse", "cat", "dog"]}) db.test.insert({"id": 4, "tags": []}) map = Code("function () {" " this.tags.forEach(function(z) {" " emit(z, 1);" " });" "}") reduce = Code("function (key, values) {" " var total = 0;" " for (var i = 0; i < values.length; i++) {" " total += values[i];" " }" " return total;" "}") result = db.test.map_reduce(map, reduce, out='mrunittests') self.assertEqual(3, result.find_one({"_id": "cat"})["value"]) self.assertEqual(2, result.find_one({"_id": "dog"})["value"]) self.assertEqual(1, result.find_one({"_id": "mouse"})["value"]) if version.at_least(self.db.connection, (1, 7, 4)): db.test.insert({"id": 5, "tags": ["hampster"]}) result = db.test.map_reduce(map, reduce, out='mrunittests') self.assertEqual(1, result.find_one({"_id": "hampster"})["value"]) db.test.remove({"id": 5}) result = db.test.map_reduce(map, reduce, out={'merge': 'mrunittests'}) self.assertEqual(3, result.find_one({"_id": "cat"})["value"]) self.assertEqual(1, result.find_one({"_id": "hampster"})["value"]) result = db.test.map_reduce(map, reduce, out={'reduce': 'mrunittests'}) self.assertEqual(6, result.find_one({"_id": "cat"})["value"]) self.assertEqual(4, result.find_one({"_id": "dog"})["value"]) self.assertEqual(2, result.find_one({"_id": "mouse"})["value"]) self.assertEqual(1, result.find_one({"_id": "hampster"})["value"]) result = db.test.map_reduce( map, reduce, out={'replace': 'mrunittests'} ) self.assertEqual(3, result.find_one({"_id": "cat"})["value"]) self.assertEqual(2, result.find_one({"_id": "dog"})["value"]) self.assertEqual(1, result.find_one({"_id": "mouse"})["value"]) if (is_mongos(self.db.connection) and not version.at_least(self.db.connection, (2, 1, 2))): pass else: result = db.test.map_reduce(map, reduce, out=SON([('replace', 'mrunittests'), ('db', 'mrtestdb') ])) self.assertEqual(3, result.find_one({"_id": "cat"})["value"]) self.assertEqual(2, result.find_one({"_id": "dog"})["value"]) self.assertEqual(1, result.find_one({"_id": "mouse"})["value"]) self.client.drop_database('mrtestdb') full_result = db.test.map_reduce(map, reduce, out='mrunittests', full_response=True) self.assertEqual(6, full_result["counts"]["emit"]) result = db.test.map_reduce(map, reduce, out='mrunittests', limit=2) self.assertEqual(2, result.find_one({"_id": "cat"})["value"]) self.assertEqual(1, result.find_one({"_id": "dog"})["value"]) self.assertEqual(None, result.find_one({"_id": "mouse"})) if version.at_least(self.db.connection, (1, 7, 4)): result = db.test.map_reduce(map, reduce, out={'inline': 1}) self.assertTrue(isinstance(result, dict)) self.assertTrue('results' in result) self.assertTrue(result['results'][1]["_id"] in ("cat", "dog", "mouse")) result = db.test.inline_map_reduce(map, reduce) self.assertTrue(isinstance(result, list)) self.assertEqual(3, len(result)) self.assertTrue(result[1]["_id"] in ("cat", "dog", "mouse")) full_result = db.test.inline_map_reduce(map, reduce, full_response=True) self.assertEqual(6, full_result["counts"]["emit"]) def test_messages_with_unicode_collection_names(self): db = self.db db[u"Employés"].insert({"x": 1}) db[u"Employés"].update({"x": 1}, {"x": 2}) db[u"Employés"].remove({}) db[u"Employés"].find_one() list(db[u"Employés"].find()) def test_drop_indexes_non_existant(self): self.db.drop_collection("test") self.db.test.drop_indexes() # This is really a bson test but easier to just reproduce it here... # (Shame on me) def test_bad_encode(self): c = self.db.test warnings.simplefilter("ignore") self.assertRaises(InvalidDocument, c.save, {"x": c}) warnings.simplefilter("default") def test_bad_dbref(self): c = self.db.test c.drop() # Incomplete DBRefs. self.assertRaises( InvalidDocument, c.insert, {'ref': {'$ref': 'collection'}}) self.assertRaises( InvalidDocument, c.insert, {'ref': {'$id': ObjectId()}}) ref_only = {'ref': {'$ref': 'collection'}} id_only = {'ref': {'$id': ObjectId()}} # Starting with MongoDB 2.5.2 this is no longer possible # from insert, update, or findAndModify. if not version.at_least(self.db.connection, (2, 5, 2)): # Force insert of ref without $id. c.insert(ref_only, check_keys=False) self.assertEqual(DBRef('collection', id=None), c.find_one()['ref']) c.drop() # DBRef without $ref is decoded as normal subdocument. c.insert(id_only, check_keys=False) self.assertEqual(id_only, c.find_one()) def test_as_class(self): c = self.db.test c.drop() c.insert({"x": 1}) doc = c.find().next() self.assertTrue(isinstance(doc, dict)) doc = c.find().next() self.assertFalse(isinstance(doc, SON)) doc = c.find(as_class=SON).next() self.assertTrue(isinstance(doc, SON)) self.assertTrue(isinstance(c.find_one(), dict)) self.assertFalse(isinstance(c.find_one(), SON)) self.assertTrue(isinstance(c.find_one(as_class=SON), SON)) self.assertEqual(1, c.find_one(as_class=SON)["x"]) doc = c.find(as_class=SON).next() self.assertEqual(1, doc["x"]) def test_find_and_modify(self): c = self.db.test c.drop() c.insert({'_id': 1, 'i': 1}) # Test that we raise DuplicateKeyError when appropriate. # MongoDB doesn't have a code field for DuplicateKeyError # from commands before 2.2. if version.at_least(self.db.connection, (2, 2)): c.ensure_index('i', unique=True) self.assertRaises(DuplicateKeyError, c.find_and_modify, query={'i': 1, 'j': 1}, update={'$set': {'k': 1}}, upsert=True) c.drop_indexes() # Test correct findAndModify self.assertEqual({'_id': 1, 'i': 1}, c.find_and_modify({'_id': 1}, {'$inc': {'i': 1}})) self.assertEqual({'_id': 1, 'i': 3}, c.find_and_modify({'_id': 1}, {'$inc': {'i': 1}}, new=True)) self.assertEqual({'_id': 1, 'i': 3}, c.find_and_modify({'_id': 1}, remove=True)) self.assertEqual(None, c.find_one({'_id': 1})) self.assertEqual(None, c.find_and_modify({'_id': 1}, {'$inc': {'i': 1}})) # The return value changed in 2.1.2. See SERVER-6226. if version.at_least(self.db.connection, (2, 1, 2)): self.assertEqual(None, c.find_and_modify({'_id': 1}, {'$inc': {'i': 1}}, upsert=True)) else: self.assertEqual({}, c.find_and_modify({'_id': 1}, {'$inc': {'i': 1}}, upsert=True)) self.assertEqual({'_id': 1, 'i': 2}, c.find_and_modify({'_id': 1}, {'$inc': {'i': 1}}, upsert=True, new=True)) self.assertEqual({'_id': 1, 'i': 2}, c.find_and_modify({'_id': 1}, {'$inc': {'i': 1}}, fields=['i'])) self.assertEqual({'_id': 1, 'i': 4}, c.find_and_modify({'_id': 1}, {'$inc': {'i': 1}}, new=True, fields={'i': 1})) # Test with full_response=True # No lastErrorObject from mongos until 2.0 if (not is_mongos(self.db.connection) or version.at_least(self.db.connection, (2, 0))): result = c.find_and_modify({'_id': 1}, {'$inc': {'i': 1}}, new=True, upsert=True, full_response=True, fields={'i': 1}) self.assertEqual({'_id': 1, 'i': 5}, result["value"]) self.assertEqual(True, result["lastErrorObject"]["updatedExisting"]) result = c.find_and_modify({'_id': 2}, {'$inc': {'i': 1}}, new=True, upsert=True, full_response=True, fields={'i': 1}) self.assertEqual({'_id': 2, 'i': 1}, result["value"]) self.assertEqual(False, result["lastErrorObject"]["updatedExisting"]) class ExtendedDict(dict): pass result = c.find_and_modify({'_id': 1}, {'$inc': {'i': 1}}, new=True, fields={'i': 1}) self.assertFalse(isinstance(result, ExtendedDict)) result = c.find_and_modify({'_id': 1}, {'$inc': {'i': 1}}, new=True, fields={'i': 1}, as_class=ExtendedDict) self.assertTrue(isinstance(result, ExtendedDict)) def test_find_and_modify_with_sort(self): c = self.db.test c.drop() for j in xrange(5): c.insert({'j': j, 'i': 0}) sort={'j': DESCENDING} self.assertEqual(4, c.find_and_modify({}, {'$inc': {'i': 1}}, sort=sort)['j']) sort={'j': ASCENDING} self.assertEqual(0, c.find_and_modify({}, {'$inc': {'i': 1}}, sort=sort)['j']) sort=[('j', DESCENDING)] self.assertEqual(4, c.find_and_modify({}, {'$inc': {'i': 1}}, sort=sort)['j']) sort=[('j', ASCENDING)] self.assertEqual(0, c.find_and_modify({}, {'$inc': {'i': 1}}, sort=sort)['j']) sort=SON([('j', DESCENDING)]) self.assertEqual(4, c.find_and_modify({}, {'$inc': {'i': 1}}, sort=sort)['j']) sort=SON([('j', ASCENDING)]) self.assertEqual(0, c.find_and_modify({}, {'$inc': {'i': 1}}, sort=sort)['j']) try: from collections import OrderedDict sort=OrderedDict([('j', DESCENDING)]) self.assertEqual(4, c.find_and_modify({}, {'$inc': {'i': 1}}, sort=sort)['j']) sort=OrderedDict([('j', ASCENDING)]) self.assertEqual(0, c.find_and_modify({}, {'$inc': {'i': 1}}, sort=sort)['j']) except ImportError: pass # Test that a standard dict with two keys is rejected. sort={'j': DESCENDING, 'foo': DESCENDING} self.assertRaises(TypeError, c.find_and_modify, {}, {'$inc': {'i': 1}}, sort=sort) def test_find_with_nested(self): if not version.at_least(self.db.connection, (2, 0, 0)): raise SkipTest("nested $and and $or requires MongoDB >= 2.0") c = self.db.test c.drop() c.insert([{'i': i} for i in range(5)]) # [0, 1, 2, 3, 4] self.assertEqual( [2], [i['i'] for i in c.find({ '$and': [ { # This clause gives us [1,2,4] '$or': [ {'i': {'$lte': 2}}, {'i': {'$gt': 3}}, ], }, { # This clause gives us [2,3] '$or': [ {'i': 2}, {'i': 3}, ] }, ] })] ) self.assertEqual( [0, 1, 2], [i['i'] for i in c.find({ '$or': [ { # This clause gives us [2] '$and': [ {'i': {'$gte': 2}}, {'i': {'$lt': 3}}, ], }, { # This clause gives us [0,1] '$and': [ {'i': {'$gt': -100}}, {'i': {'$lt': 2}}, ] }, ] })] ) def test_disabling_manipulators(self): class IncByTwo(SONManipulator): def transform_outgoing(self, son, collection): if 'foo' in son: son['foo'] += 2 return son db = self.client.pymongo_test db.add_son_manipulator(IncByTwo()) c = db.test c.drop() c.insert({'foo': 0}) self.assertEqual(2, c.find_one()['foo']) self.assertEqual(0, c.find_one(manipulate=False)['foo']) self.assertEqual(2, c.find_one(manipulate=True)['foo']) c.remove({}) def test_uuid_subtype(self): if not have_uuid: raise SkipTest("No uuid module") coll = self.client.pymongo_test.uuid coll.drop() def change_subtype(collection, subtype): collection.uuid_subtype = subtype # Test property self.assertEqual(OLD_UUID_SUBTYPE, coll.uuid_subtype) self.assertRaises(ConfigurationError, change_subtype, coll, 7) self.assertRaises(ConfigurationError, change_subtype, coll, 2) # Test basic query uu = uuid.uuid4() # Insert as binary subtype 3 coll.insert({'uu': uu}) self.assertEqual(uu, coll.find_one({'uu': uu})['uu']) coll.uuid_subtype = UUID_SUBTYPE self.assertEqual(UUID_SUBTYPE, coll.uuid_subtype) self.assertEqual(None, coll.find_one({'uu': uu})) self.assertEqual(uu, coll.find_one({'uu': UUIDLegacy(uu)})['uu']) # Test Cursor.count self.assertEqual(0, coll.find({'uu': uu}).count()) coll.uuid_subtype = OLD_UUID_SUBTYPE self.assertEqual(1, coll.find({'uu': uu}).count()) # Test remove coll.uuid_subtype = UUID_SUBTYPE coll.remove({'uu': uu}) self.assertEqual(1, coll.count()) coll.uuid_subtype = OLD_UUID_SUBTYPE coll.remove({'uu': uu}) self.assertEqual(0, coll.count()) # Test save coll.insert({'_id': uu, 'i': 0}) self.assertEqual(1, coll.count()) self.assertEqual(1, coll.find({'_id': uu}).count()) self.assertEqual(0, coll.find_one({'_id': uu})['i']) doc = coll.find_one({'_id': uu}) doc['i'] = 1 coll.save(doc) self.assertEqual(1, coll.find_one({'_id': uu})['i']) # Test update coll.uuid_subtype = UUID_SUBTYPE coll.update({'_id': uu}, {'$set': {'i': 2}}) coll.uuid_subtype = OLD_UUID_SUBTYPE self.assertEqual(1, coll.find_one({'_id': uu})['i']) coll.update({'_id': uu}, {'$set': {'i': 2}}) self.assertEqual(2, coll.find_one({'_id': uu})['i']) # Test Cursor.distinct self.assertEqual([2], coll.find({'_id': uu}).distinct('i')) coll.uuid_subtype = UUID_SUBTYPE self.assertEqual([], coll.find({'_id': uu}).distinct('i')) # Test find_and_modify self.assertEqual(None, coll.find_and_modify({'_id': uu}, {'$set': {'i': 5}})) coll.uuid_subtype = OLD_UUID_SUBTYPE self.assertEqual(2, coll.find_and_modify({'_id': uu}, {'$set': {'i': 5}})['i']) self.assertEqual(5, coll.find_one({'_id': uu})['i']) # Test command db = self.client.pymongo_test no_obj_error = "No matching object found" result = db.command('findAndModify', 'uuid', allowable_errors=[no_obj_error], uuid_subtype=UUID_SUBTYPE, query={'_id': uu}, update={'$set': {'i': 6}}) self.assertEqual(None, result.get('value')) self.assertEqual(5, db.command('findAndModify', 'uuid', update={'$set': {'i': 6}}, query={'_id': uu})['value']['i']) self.assertEqual(6, db.command('findAndModify', 'uuid', update={'$set': {'i': 7}}, query={'_id': UUIDLegacy(uu)} )['value']['i']) # Test (inline)_map_reduce coll.drop() coll.insert({"_id": uu, "x": 1, "tags": ["dog", "cat"]}) coll.insert({"_id": uuid.uuid4(), "x": 3, "tags": ["mouse", "cat", "dog"]}) map = Code("function () {" " this.tags.forEach(function(z) {" " emit(z, 1);" " });" "}") reduce = Code("function (key, values) {" " var total = 0;" " for (var i = 0; i < values.length; i++) {" " total += values[i];" " }" " return total;" "}") coll.uuid_subtype = UUID_SUBTYPE q = {"_id": uu} if version.at_least(self.db.connection, (1, 7, 4)): result = coll.inline_map_reduce(map, reduce, query=q) self.assertEqual([], result) result = coll.map_reduce(map, reduce, "results", query=q) self.assertEqual(0, db.results.count()) coll.uuid_subtype = OLD_UUID_SUBTYPE q = {"_id": uu} if version.at_least(self.db.connection, (1, 7, 4)): result = coll.inline_map_reduce(map, reduce, query=q) self.assertEqual(2, len(result)) result = coll.map_reduce(map, reduce, "results", query=q) self.assertEqual(2, db.results.count()) db.drop_collection("result") coll.drop() # Test group coll.insert({"_id": uu, "a": 2}) coll.insert({"_id": uuid.uuid4(), "a": 1}) reduce = "function (obj, prev) { prev.count++; }" coll.uuid_subtype = UUID_SUBTYPE self.assertEqual([], coll.group([], {"_id": uu}, {"count": 0}, reduce)) coll.uuid_subtype = OLD_UUID_SUBTYPE self.assertEqual([{"count": 1}], coll.group([], {"_id": uu}, {"count": 0}, reduce)) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_common.py000066400000000000000000000331051223300253600170700ustar00rootroot00000000000000# Copyright 2011-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the pymongo common module.""" import os import sys import unittest import warnings sys.path[0:0] = [""] from nose.plugins.skip import SkipTest from bson.objectid import ObjectId from bson.son import SON from pymongo.connection import Connection from pymongo.mongo_client import MongoClient from pymongo.mongo_replica_set_client import MongoReplicaSetClient from pymongo.errors import ConfigurationError, OperationFailure from test import host, port, pair from test.utils import drop_collections class TestCommon(unittest.TestCase): def test_baseobject(self): # In Python 2.6+ we could use the catch_warnings context # manager to test this warning nicely. As we can't do that # we must test raising errors before the ignore filter is applied. warnings.simplefilter("error", UserWarning) self.assertRaises(UserWarning, lambda: MongoClient(host, port, wtimeout=1000, w=0)) try: MongoClient(host, port, wtimeout=1000, w=1) except UserWarning: self.fail() try: MongoClient(host, port, wtimeout=1000) except UserWarning: self.fail() warnings.resetwarnings() warnings.simplefilter("ignore") # Connection tests c = Connection(pair) self.assertFalse(c.slave_okay) self.assertFalse(c.safe) self.assertEqual({}, c.get_lasterror_options()) db = c.pymongo_test db.drop_collection("test") self.assertFalse(db.slave_okay) self.assertFalse(db.safe) self.assertEqual({}, db.get_lasterror_options()) coll = db.test self.assertFalse(coll.slave_okay) self.assertFalse(coll.safe) self.assertEqual({}, coll.get_lasterror_options()) self.assertEqual((False, {}), coll._get_write_mode()) coll.safe = False coll.write_concern.update(w=1) self.assertEqual((True, {}), coll._get_write_mode()) coll.write_concern.update(w=3) self.assertEqual((True, {'w': 3}), coll._get_write_mode()) coll.safe = True coll.write_concern.update(w=0) self.assertEqual((False, {}), coll._get_write_mode()) coll = db.test cursor = coll.find() self.assertFalse(cursor._Cursor__slave_okay) cursor = coll.find(slave_okay=True) self.assertTrue(cursor._Cursor__slave_okay) # MongoClient test c = MongoClient(pair) self.assertFalse(c.slave_okay) self.assertTrue(c.safe) self.assertEqual({}, c.get_lasterror_options()) db = c.pymongo_test db.drop_collection("test") self.assertFalse(db.slave_okay) self.assertTrue(db.safe) self.assertEqual({}, db.get_lasterror_options()) coll = db.test self.assertFalse(coll.slave_okay) self.assertTrue(coll.safe) self.assertEqual({}, coll.get_lasterror_options()) self.assertEqual((True, {}), coll._get_write_mode()) coll.safe = False coll.write_concern.update(w=1) self.assertEqual((True, {}), coll._get_write_mode()) coll.write_concern.update(w=3) self.assertEqual((True, {'w': 3}), coll._get_write_mode()) coll.safe = True coll.write_concern.update(w=0) self.assertEqual((False, {}), coll._get_write_mode()) coll = db.test cursor = coll.find() self.assertFalse(cursor._Cursor__slave_okay) cursor = coll.find(slave_okay=True) self.assertTrue(cursor._Cursor__slave_okay) # Setting any safe operations overrides explicit safe self.assertTrue(MongoClient(host, port, wtimeout=1000, safe=False).safe) c = MongoClient(pair, slaveok=True, w='majority', wtimeout=300, fsync=True, j=True) self.assertTrue(c.slave_okay) self.assertTrue(c.safe) d = {'w': 'majority', 'wtimeout': 300, 'fsync': True, 'j': True} self.assertEqual(d, c.get_lasterror_options()) db = c.pymongo_test self.assertTrue(db.slave_okay) self.assertTrue(db.safe) self.assertEqual(d, db.get_lasterror_options()) coll = db.test self.assertTrue(coll.slave_okay) self.assertTrue(coll.safe) self.assertEqual(d, coll.get_lasterror_options()) cursor = coll.find() self.assertTrue(cursor._Cursor__slave_okay) cursor = coll.find(slave_okay=False) self.assertFalse(cursor._Cursor__slave_okay) c = MongoClient('mongodb://%s/?' 'w=2;wtimeoutMS=300;fsync=true;' 'journal=true' % (pair,)) self.assertTrue(c.safe) d = {'w': 2, 'wtimeout': 300, 'fsync': True, 'j': True} self.assertEqual(d, c.get_lasterror_options()) c = MongoClient('mongodb://%s/?' 'slaveok=true;w=1;wtimeout=300;' 'fsync=true;j=true' % (pair,)) self.assertTrue(c.slave_okay) self.assertTrue(c.safe) d = {'w': 1, 'wtimeout': 300, 'fsync': True, 'j': True} self.assertEqual(d, c.get_lasterror_options()) self.assertEqual(d, c.write_concern) db = c.pymongo_test self.assertTrue(db.slave_okay) self.assertTrue(db.safe) self.assertEqual(d, db.get_lasterror_options()) self.assertEqual(d, db.write_concern) coll = db.test self.assertTrue(coll.slave_okay) self.assertTrue(coll.safe) self.assertEqual(d, coll.get_lasterror_options()) self.assertEqual(d, coll.write_concern) cursor = coll.find() self.assertTrue(cursor._Cursor__slave_okay) cursor = coll.find(slave_okay=False) self.assertFalse(cursor._Cursor__slave_okay) c.unset_lasterror_options() self.assertTrue(c.slave_okay) self.assertTrue(c.safe) c.safe = False self.assertFalse(c.safe) c.slave_okay = False self.assertFalse(c.slave_okay) self.assertEqual({}, c.get_lasterror_options()) self.assertEqual({}, c.write_concern) db = c.pymongo_test self.assertFalse(db.slave_okay) self.assertFalse(db.safe) self.assertEqual({}, db.get_lasterror_options()) self.assertEqual({}, db.write_concern) coll = db.test self.assertFalse(coll.slave_okay) self.assertFalse(coll.safe) self.assertEqual({}, coll.get_lasterror_options()) self.assertEqual({}, coll.write_concern) cursor = coll.find() self.assertFalse(cursor._Cursor__slave_okay) cursor = coll.find(slave_okay=True) self.assertTrue(cursor._Cursor__slave_okay) coll.set_lasterror_options(j=True) self.assertEqual({'j': True}, coll.get_lasterror_options()) self.assertEqual({'j': True}, coll.write_concern) self.assertEqual({}, db.get_lasterror_options()) self.assertEqual({}, db.write_concern) self.assertFalse(db.safe) self.assertEqual({}, c.get_lasterror_options()) self.assertEqual({}, c.write_concern) self.assertFalse(c.safe) db.set_lasterror_options(w='majority') self.assertEqual({'j': True}, coll.get_lasterror_options()) self.assertEqual({'j': True}, coll.write_concern) self.assertEqual({'w': 'majority'}, db.get_lasterror_options()) self.assertEqual({'w': 'majority'}, db.write_concern) self.assertEqual({}, c.get_lasterror_options()) self.assertEqual({}, c.write_concern) self.assertFalse(c.safe) db.slave_okay = True self.assertTrue(db.slave_okay) self.assertFalse(c.slave_okay) self.assertFalse(coll.slave_okay) cursor = coll.find() self.assertFalse(cursor._Cursor__slave_okay) cursor = db.coll2.find() self.assertTrue(cursor._Cursor__slave_okay) cursor = db.coll2.find(slave_okay=False) self.assertFalse(cursor._Cursor__slave_okay) self.assertRaises(ConfigurationError, coll.set_lasterror_options, foo=20) self.assertRaises(TypeError, coll._BaseObject__set_slave_okay, 20) self.assertRaises(TypeError, coll._BaseObject__set_safe, 20) coll.remove() self.assertEqual(None, coll.find_one(slave_okay=True)) coll.unset_lasterror_options() coll.set_lasterror_options(w=4, wtimeout=10) # Fails if we don't have 4 active nodes or we don't have replication... self.assertRaises(OperationFailure, coll.insert, {'foo': 'bar'}) # Succeeds since we override the lasterror settings per query. self.assertTrue(coll.insert({'foo': 'bar'}, fsync=True)) drop_collections(db) warnings.resetwarnings() def test_write_concern(self): c = MongoClient(pair) self.assertEqual({}, c.write_concern) wc = {'w': 2, 'wtimeout': 1000} c.write_concern = wc self.assertEqual(wc, c.write_concern) wc = {'w': 3, 'wtimeout': 1000} c.write_concern['w'] = 3 self.assertEqual(wc, c.write_concern) wc = {'w': 3} del c.write_concern['wtimeout'] self.assertEqual(wc, c.write_concern) wc = {'w': 3, 'wtimeout': 1000} c = MongoClient(pair, w=3, wtimeout=1000) self.assertEqual(wc, c.write_concern) wc = {'w': 2, 'wtimeout': 1000} c.write_concern = wc self.assertEqual(wc, c.write_concern) db = c.pymongo_test self.assertEqual(wc, db.write_concern) coll = db.test self.assertEqual(wc, coll.write_concern) coll.write_concern = {'j': True} self.assertEqual({'j': True}, coll.write_concern) self.assertEqual(wc, db.write_concern) wc = SON([('w', 2)]) coll.write_concern = wc self.assertEqual(wc.to_dict(), coll.write_concern) def f(): c.write_concern = {'foo': 'bar'} self.assertRaises(ConfigurationError, f) def f(): c.write_concern['foo'] = 'bar' self.assertRaises(ConfigurationError, f) def f(): c.write_concern = [('foo', 'bar')] self.assertRaises(ConfigurationError, f) def test_mongo_client(self): m = MongoClient(pair, w=0) coll = m.pymongo_test.write_concern_test coll.drop() doc = {"_id": ObjectId()} coll.insert(doc) self.assertTrue(coll.insert(doc, safe=False)) self.assertTrue(coll.insert(doc, w=0)) self.assertTrue(coll.insert(doc)) self.assertRaises(OperationFailure, coll.insert, doc, safe=True) self.assertRaises(OperationFailure, coll.insert, doc, w=1) m = MongoClient(pair) coll = m.pymongo_test.write_concern_test self.assertTrue(coll.insert(doc, safe=False)) self.assertTrue(coll.insert(doc, w=0)) self.assertRaises(OperationFailure, coll.insert, doc) self.assertRaises(OperationFailure, coll.insert, doc, safe=True) self.assertRaises(OperationFailure, coll.insert, doc, w=1) m = MongoClient("mongodb://%s/" % (pair,)) self.assertTrue(m.safe) coll = m.pymongo_test.write_concern_test self.assertRaises(OperationFailure, coll.insert, doc) m = MongoClient("mongodb://%s/?w=0" % (pair,)) self.assertFalse(m.safe) coll = m.pymongo_test.write_concern_test self.assertTrue(coll.insert(doc)) # Equality tests self.assertEqual(m, MongoClient("mongodb://%s/?w=0" % (pair,))) self.assertFalse(m != MongoClient("mongodb://%s/?w=0" % (pair,))) def test_mongo_replica_set_client(self): c = MongoClient(pair) ismaster = c.admin.command('ismaster') if 'setName' in ismaster: setname = str(ismaster.get('setName')) else: raise SkipTest("Not connected to a replica set.") m = MongoReplicaSetClient(pair, replicaSet=setname, w=0) coll = m.pymongo_test.write_concern_test coll.drop() doc = {"_id": ObjectId()} coll.insert(doc) self.assertTrue(coll.insert(doc, safe=False)) self.assertTrue(coll.insert(doc, w=0)) self.assertTrue(coll.insert(doc)) self.assertRaises(OperationFailure, coll.insert, doc, safe=True) self.assertRaises(OperationFailure, coll.insert, doc, w=1) m = MongoReplicaSetClient(pair, replicaSet=setname) coll = m.pymongo_test.write_concern_test self.assertTrue(coll.insert(doc, safe=False)) self.assertTrue(coll.insert(doc, w=0)) self.assertRaises(OperationFailure, coll.insert, doc) self.assertRaises(OperationFailure, coll.insert, doc, safe=True) self.assertRaises(OperationFailure, coll.insert, doc, w=1) m = MongoReplicaSetClient("mongodb://%s/?replicaSet=%s" % (pair, setname)) self.assertTrue(m.safe) coll = m.pymongo_test.write_concern_test self.assertRaises(OperationFailure, coll.insert, doc) m = MongoReplicaSetClient("mongodb://%s/?replicaSet=%s;w=0" % (pair, setname)) self.assertFalse(m.safe) coll = m.pymongo_test.write_concern_test self.assertTrue(coll.insert(doc)) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_cursor.py000066400000000000000000000732221223300253600171210ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the cursor module.""" import copy import itertools import random import re import sys import unittest sys.path[0:0] = [""] from nose.plugins.skip import SkipTest from bson.code import Code from bson.son import SON from pymongo import (ASCENDING, DESCENDING) from pymongo.database import Database from pymongo.errors import (InvalidOperation, OperationFailure) from test import version from test.test_client import get_client from test.utils import is_mongos class TestCursor(unittest.TestCase): def setUp(self): self.db = Database(get_client(), "pymongo_test") def test_explain(self): a = self.db.test.find() b = a.explain() for _ in a: break c = a.explain() del b["millis"] b.pop("oldPlan", None) del c["millis"] c.pop("oldPlan", None) self.assertEqual(b, c) self.assertTrue("cursor" in b) def test_hint(self): db = self.db self.assertRaises(TypeError, db.test.find().hint, 5.5) db.test.drop() for i in range(100): db.test.insert({"num": i, "foo": i}) self.assertRaises(OperationFailure, db.test.find({"num": 17, "foo": 17}) .hint([("num", ASCENDING)]).explain) self.assertRaises(OperationFailure, db.test.find({"num": 17, "foo": 17}) .hint([("foo", ASCENDING)]).explain) index = db.test.create_index("num") spec = [("num", ASCENDING)] self.assertEqual(db.test.find({}).explain()["cursor"], "BasicCursor") self.assertEqual(db.test.find({}).hint(spec).explain()["cursor"], "BtreeCursor %s" % index) self.assertEqual(db.test.find({}).hint(spec).hint(None) .explain()["cursor"], "BasicCursor") self.assertRaises(OperationFailure, db.test.find({"num": 17, "foo": 17}) .hint([("foo", ASCENDING)]).explain) a = db.test.find({"num": 17}) a.hint(spec) for _ in a: break self.assertRaises(InvalidOperation, a.hint, spec) self.assertRaises(TypeError, db.test.find().hint, index) def test_limit(self): db = self.db self.assertRaises(TypeError, db.test.find().limit, None) self.assertRaises(TypeError, db.test.find().limit, "hello") self.assertRaises(TypeError, db.test.find().limit, 5.5) db.test.drop() for i in range(100): db.test.save({"x": i}) count = 0 for _ in db.test.find(): count += 1 self.assertEqual(count, 100) count = 0 for _ in db.test.find().limit(20): count += 1 self.assertEqual(count, 20) count = 0 for _ in db.test.find().limit(99): count += 1 self.assertEqual(count, 99) count = 0 for _ in db.test.find().limit(1): count += 1 self.assertEqual(count, 1) count = 0 for _ in db.test.find().limit(0): count += 1 self.assertEqual(count, 100) count = 0 for _ in db.test.find().limit(0).limit(50).limit(10): count += 1 self.assertEqual(count, 10) a = db.test.find() a.limit(10) for _ in a: break self.assertRaises(InvalidOperation, a.limit, 5) def test_batch_size(self): db = self.db db.test.drop() for x in range(200): db.test.save({"x": x}) self.assertRaises(TypeError, db.test.find().batch_size, None) self.assertRaises(TypeError, db.test.find().batch_size, "hello") self.assertRaises(TypeError, db.test.find().batch_size, 5.5) self.assertRaises(ValueError, db.test.find().batch_size, -1) a = db.test.find() for _ in a: break self.assertRaises(InvalidOperation, a.batch_size, 5) def cursor_count(cursor, expected_count): count = 0 for _ in cursor: count += 1 self.assertEqual(expected_count, count) cursor_count(db.test.find().batch_size(0), 200) cursor_count(db.test.find().batch_size(1), 200) cursor_count(db.test.find().batch_size(2), 200) cursor_count(db.test.find().batch_size(5), 200) cursor_count(db.test.find().batch_size(100), 200) cursor_count(db.test.find().batch_size(500), 200) cursor_count(db.test.find().batch_size(0).limit(1), 1) cursor_count(db.test.find().batch_size(1).limit(1), 1) cursor_count(db.test.find().batch_size(2).limit(1), 1) cursor_count(db.test.find().batch_size(5).limit(1), 1) cursor_count(db.test.find().batch_size(100).limit(1), 1) cursor_count(db.test.find().batch_size(500).limit(1), 1) cursor_count(db.test.find().batch_size(0).limit(10), 10) cursor_count(db.test.find().batch_size(1).limit(10), 10) cursor_count(db.test.find().batch_size(2).limit(10), 10) cursor_count(db.test.find().batch_size(5).limit(10), 10) cursor_count(db.test.find().batch_size(100).limit(10), 10) cursor_count(db.test.find().batch_size(500).limit(10), 10) def test_limit_and_batch_size(self): db = self.db db.test.drop() for x in range(500): db.test.save({"x": x}) curs = db.test.find().limit(0).batch_size(10) curs.next() self.assertEqual(10, curs._Cursor__retrieved) curs = db.test.find().limit(-2).batch_size(0) curs.next() self.assertEqual(2, curs._Cursor__retrieved) curs = db.test.find().limit(-4).batch_size(5) curs.next() self.assertEqual(4, curs._Cursor__retrieved) curs = db.test.find().limit(50).batch_size(500) curs.next() self.assertEqual(50, curs._Cursor__retrieved) curs = db.test.find().batch_size(500) curs.next() self.assertEqual(500, curs._Cursor__retrieved) curs = db.test.find().limit(50) curs.next() self.assertEqual(50, curs._Cursor__retrieved) # these two might be shaky, as the default # is set by the server. as of 2.0.0-rc0, 101 # or 1MB (whichever is smaller) is default # for queries without ntoreturn curs = db.test.find() curs.next() self.assertEqual(101, curs._Cursor__retrieved) curs = db.test.find().limit(0).batch_size(0) curs.next() self.assertEqual(101, curs._Cursor__retrieved) def test_skip(self): db = self.db self.assertRaises(TypeError, db.test.find().skip, None) self.assertRaises(TypeError, db.test.find().skip, "hello") self.assertRaises(TypeError, db.test.find().skip, 5.5) db.drop_collection("test") for i in range(100): db.test.save({"x": i}) for i in db.test.find(): self.assertEqual(i["x"], 0) break for i in db.test.find().skip(20): self.assertEqual(i["x"], 20) break for i in db.test.find().skip(99): self.assertEqual(i["x"], 99) break for i in db.test.find().skip(1): self.assertEqual(i["x"], 1) break for i in db.test.find().skip(0): self.assertEqual(i["x"], 0) break for i in db.test.find().skip(0).skip(50).skip(10): self.assertEqual(i["x"], 10) break for i in db.test.find().skip(1000): self.fail() a = db.test.find() a.skip(10) for _ in a: break self.assertRaises(InvalidOperation, a.skip, 5) def test_sort(self): db = self.db self.assertRaises(TypeError, db.test.find().sort, 5) self.assertRaises(ValueError, db.test.find().sort, []) self.assertRaises(TypeError, db.test.find().sort, [], ASCENDING) self.assertRaises(TypeError, db.test.find().sort, [("hello", DESCENDING)], DESCENDING) db.test.drop() unsort = range(10) random.shuffle(unsort) for i in unsort: db.test.save({"x": i}) asc = [i["x"] for i in db.test.find().sort("x", ASCENDING)] self.assertEqual(asc, range(10)) asc = [i["x"] for i in db.test.find().sort("x")] self.assertEqual(asc, range(10)) asc = [i["x"] for i in db.test.find().sort([("x", ASCENDING)])] self.assertEqual(asc, range(10)) expect = range(10) expect.reverse() desc = [i["x"] for i in db.test.find().sort("x", DESCENDING)] self.assertEqual(desc, expect) desc = [i["x"] for i in db.test.find().sort([("x", DESCENDING)])] self.assertEqual(desc, expect) desc = [i["x"] for i in db.test.find().sort("x", ASCENDING).sort("x", DESCENDING)] self.assertEqual(desc, expect) expected = [(1, 5), (2, 5), (0, 3), (7, 3), (9, 2), (2, 1), (3, 1)] shuffled = list(expected) random.shuffle(shuffled) db.test.drop() for (a, b) in shuffled: db.test.save({"a": a, "b": b}) result = [(i["a"], i["b"]) for i in db.test.find().sort([("b", DESCENDING), ("a", ASCENDING)])] self.assertEqual(result, expected) a = db.test.find() a.sort("x", ASCENDING) for _ in a: break self.assertRaises(InvalidOperation, a.sort, "x", ASCENDING) def test_count(self): db = self.db db.test.drop() self.assertEqual(0, db.test.find().count()) for i in range(10): db.test.save({"x": i}) self.assertEqual(10, db.test.find().count()) self.assertTrue(isinstance(db.test.find().count(), int)) self.assertEqual(10, db.test.find().limit(5).count()) self.assertEqual(10, db.test.find().skip(5).count()) self.assertEqual(1, db.test.find({"x": 1}).count()) self.assertEqual(5, db.test.find({"x": {"$lt": 5}}).count()) a = db.test.find() b = a.count() for _ in a: break self.assertEqual(b, a.count()) self.assertEqual(0, db.test.acollectionthatdoesntexist.find().count()) def test_where(self): db = self.db db.test.drop() a = db.test.find() self.assertRaises(TypeError, a.where, 5) self.assertRaises(TypeError, a.where, None) self.assertRaises(TypeError, a.where, {}) for i in range(10): db.test.save({"x": i}) self.assertEqual(3, len(list(db.test.find().where('this.x < 3')))) self.assertEqual(3, len(list(db.test.find().where(Code('this.x < 3'))))) self.assertEqual(3, len(list(db.test.find().where(Code('this.x < i', {"i": 3}))))) self.assertEqual(10, len(list(db.test.find()))) self.assertEqual(3, db.test.find().where('this.x < 3').count()) self.assertEqual(10, db.test.find().count()) self.assertEqual(3, db.test.find().where(u'this.x < 3').count()) self.assertEqual([0, 1, 2], [a["x"] for a in db.test.find().where('this.x < 3')]) self.assertEqual([], [a["x"] for a in db.test.find({"x": 5}).where('this.x < 3')]) self.assertEqual([5], [a["x"] for a in db.test.find({"x": 5}).where('this.x > 3')]) cursor = db.test.find().where('this.x < 3').where('this.x > 7') self.assertEqual([8, 9], [a["x"] for a in cursor]) a = db.test.find() b = a.where('this.x > 3') for _ in a: break self.assertRaises(InvalidOperation, a.where, 'this.x < 3') def test_rewind(self): self.db.test.save({"x": 1}) self.db.test.save({"x": 2}) self.db.test.save({"x": 3}) cursor = self.db.test.find().limit(2) count = 0 for _ in cursor: count += 1 self.assertEqual(2, count) count = 0 for _ in cursor: count += 1 self.assertEqual(0, count) cursor.rewind() count = 0 for _ in cursor: count += 1 self.assertEqual(2, count) cursor.rewind() count = 0 for _ in cursor: break cursor.rewind() for _ in cursor: count += 1 self.assertEqual(2, count) self.assertEqual(cursor, cursor.rewind()) def test_clone(self): self.db.test.save({"x": 1}) self.db.test.save({"x": 2}) self.db.test.save({"x": 3}) cursor = self.db.test.find().limit(2) count = 0 for _ in cursor: count += 1 self.assertEqual(2, count) count = 0 for _ in cursor: count += 1 self.assertEqual(0, count) cursor = cursor.clone() cursor2 = cursor.clone() count = 0 for _ in cursor: count += 1 self.assertEqual(2, count) for _ in cursor2: count += 1 self.assertEqual(4, count) cursor.rewind() count = 0 for _ in cursor: break cursor = cursor.clone() for _ in cursor: count += 1 self.assertEqual(2, count) self.assertNotEqual(cursor, cursor.clone()) class MyClass(dict): pass cursor = self.db.test.find(as_class=MyClass) for e in cursor: self.assertEqual(type(MyClass()), type(e)) cursor = self.db.test.find(as_class=MyClass) self.assertEqual(type(MyClass()), type(cursor[0])) # Just test attributes cursor = self.db.test.find({"x": re.compile("^hello.*")}, skip=1, timeout=False, snapshot=True, tailable=True, as_class=MyClass, slave_okay=True, await_data=True, partial=True, manipulate=False, fields={'_id': False}).limit(2) cursor.add_option(128) cursor2 = cursor.clone() self.assertEqual(cursor._Cursor__skip, cursor2._Cursor__skip) self.assertEqual(cursor._Cursor__limit, cursor2._Cursor__limit) self.assertEqual(cursor._Cursor__snapshot, cursor2._Cursor__snapshot) self.assertEqual(type(cursor._Cursor__as_class), type(cursor2._Cursor__as_class)) self.assertEqual(cursor._Cursor__slave_okay, cursor2._Cursor__slave_okay) self.assertEqual(cursor._Cursor__manipulate, cursor2._Cursor__manipulate) self.assertEqual(cursor._Cursor__query_flags, cursor2._Cursor__query_flags) # Shallow copies can so can mutate cursor2 = copy.copy(cursor) cursor2._Cursor__fields['cursor2'] = False self.assertTrue('cursor2' in cursor._Cursor__fields) # Deepcopies and shouldn't mutate cursor3 = copy.deepcopy(cursor) cursor3._Cursor__fields['cursor3'] = False self.assertFalse('cursor3' in cursor._Cursor__fields) cursor4 = cursor.clone() cursor4._Cursor__fields['cursor4'] = False self.assertFalse('cursor4' in cursor._Cursor__fields) # Test memo when deepcopying queries query = {"hello": "world"} query["reflexive"] = query cursor = self.db.test.find(query) cursor2 = copy.deepcopy(cursor) self.assertNotEqual(id(cursor._Cursor__spec), id(cursor2._Cursor__spec)) self.assertEqual(id(cursor2._Cursor__spec['reflexive']), id(cursor2._Cursor__spec)) self.assertEqual(len(cursor2._Cursor__spec), 2) # Ensure hints are cloned as the correct type cursor = self.db.test.find().hint([('z', 1), ("a", 1)]) cursor2 = copy.deepcopy(cursor) self.assertTrue(isinstance(cursor2._Cursor__hint, SON)) self.assertEqual(cursor._Cursor__hint, cursor2._Cursor__hint) def test_deepcopy_cursor_littered_with_regexes(self): cursor = self.db.test.find({"x": re.compile("^hmmm.*"), "y": [re.compile("^hmm.*")], "z": {"a": [re.compile("^hm.*")]}, re.compile("^key.*"): {"a": [re.compile("^hm.*")]}}) cursor2 = copy.deepcopy(cursor) self.assertEqual(cursor._Cursor__spec, cursor2._Cursor__spec) def test_add_remove_option(self): cursor = self.db.test.find() self.assertEqual(0, cursor._Cursor__query_options()) cursor.add_option(2) cursor2 = self.db.test.find(tailable=True) self.assertEqual(2, cursor2._Cursor__query_options()) self.assertEqual(cursor._Cursor__query_options(), cursor2._Cursor__query_options()) cursor.add_option(32) cursor2 = self.db.test.find(tailable=True, await_data=True) self.assertEqual(34, cursor2._Cursor__query_options()) self.assertEqual(cursor._Cursor__query_options(), cursor2._Cursor__query_options()) cursor.add_option(128) cursor2 = self.db.test.find(tailable=True, await_data=True).add_option(128) self.assertEqual(162, cursor2._Cursor__query_options()) self.assertEqual(cursor._Cursor__query_options(), cursor2._Cursor__query_options()) self.assertEqual(162, cursor._Cursor__query_options()) cursor.add_option(128) self.assertEqual(162, cursor._Cursor__query_options()) cursor.remove_option(128) cursor2 = self.db.test.find(tailable=True, await_data=True) self.assertEqual(34, cursor2._Cursor__query_options()) self.assertEqual(cursor._Cursor__query_options(), cursor2._Cursor__query_options()) cursor.remove_option(32) cursor2 = self.db.test.find(tailable=True) self.assertEqual(2, cursor2._Cursor__query_options()) self.assertEqual(cursor._Cursor__query_options(), cursor2._Cursor__query_options()) self.assertEqual(2, cursor._Cursor__query_options()) cursor.remove_option(32) self.assertEqual(2, cursor._Cursor__query_options()) # Slave OK cursor = self.db.test.find(slave_okay=True) self.assertEqual(4, cursor._Cursor__query_options()) cursor2 = self.db.test.find().add_option(4) self.assertEqual(cursor._Cursor__query_options(), cursor2._Cursor__query_options()) self.assertTrue(cursor._Cursor__slave_okay) cursor.remove_option(4) self.assertEqual(0, cursor._Cursor__query_options()) self.assertFalse(cursor._Cursor__slave_okay) # Timeout cursor = self.db.test.find(timeout=False) self.assertEqual(16, cursor._Cursor__query_options()) cursor2 = self.db.test.find().add_option(16) self.assertEqual(cursor._Cursor__query_options(), cursor2._Cursor__query_options()) cursor.remove_option(16) self.assertEqual(0, cursor._Cursor__query_options()) # Tailable / Await data cursor = self.db.test.find(tailable=True, await_data=True) self.assertEqual(34, cursor._Cursor__query_options()) cursor2 = self.db.test.find().add_option(34) self.assertEqual(cursor._Cursor__query_options(), cursor2._Cursor__query_options()) cursor.remove_option(32) self.assertEqual(2, cursor._Cursor__query_options()) # Exhaust - which mongos doesn't support if not is_mongos(self.db.connection): cursor = self.db.test.find(exhaust=True) self.assertEqual(64, cursor._Cursor__query_options()) cursor2 = self.db.test.find().add_option(64) self.assertEqual(cursor._Cursor__query_options(), cursor2._Cursor__query_options()) self.assertTrue(cursor._Cursor__exhaust) cursor.remove_option(64) self.assertEqual(0, cursor._Cursor__query_options()) self.assertFalse(cursor._Cursor__exhaust) # Partial cursor = self.db.test.find(partial=True) self.assertEqual(128, cursor._Cursor__query_options()) cursor2 = self.db.test.find().add_option(128) self.assertEqual(cursor._Cursor__query_options(), cursor2._Cursor__query_options()) cursor.remove_option(128) self.assertEqual(0, cursor._Cursor__query_options()) def test_count_with_fields(self): self.db.test.drop() self.db.test.save({"x": 1}) if not version.at_least(self.db.connection, (1, 1, 3, -1)): for _ in self.db.test.find({}, ["a"]): self.fail() self.assertEqual(0, self.db.test.find({}, ["a"]).count()) else: self.assertEqual(1, self.db.test.find({}, ["a"]).count()) def test_bad_getitem(self): self.assertRaises(TypeError, lambda x: self.db.test.find()[x], "hello") self.assertRaises(TypeError, lambda x: self.db.test.find()[x], 5.5) self.assertRaises(TypeError, lambda x: self.db.test.find()[x], None) def test_getitem_slice_index(self): self.db.drop_collection("test") for i in range(100): self.db.test.save({"i": i}) count = itertools.count self.assertRaises(IndexError, lambda: self.db.test.find()[-1:]) self.assertRaises(IndexError, lambda: self.db.test.find()[1:2:2]) for a, b in zip(count(0), self.db.test.find()): self.assertEqual(a, b['i']) self.assertEqual(100, len(list(self.db.test.find()[0:]))) for a, b in zip(count(0), self.db.test.find()[0:]): self.assertEqual(a, b['i']) self.assertEqual(80, len(list(self.db.test.find()[20:]))) for a, b in zip(count(20), self.db.test.find()[20:]): self.assertEqual(a, b['i']) for a, b in zip(count(99), self.db.test.find()[99:]): self.assertEqual(a, b['i']) for i in self.db.test.find()[1000:]: self.fail() self.assertEqual(5, len(list(self.db.test.find()[20:25]))) self.assertEqual(5, len(list(self.db.test.find()[20L:25L]))) for a, b in zip(count(20), self.db.test.find()[20:25]): self.assertEqual(a, b['i']) self.assertEqual(80, len(list(self.db.test.find()[40:45][20:]))) for a, b in zip(count(20), self.db.test.find()[40:45][20:]): self.assertEqual(a, b['i']) self.assertEqual(80, len(list(self.db.test.find()[40:45].limit(0).skip(20)) ) ) for a, b in zip(count(20), self.db.test.find()[40:45].limit(0).skip(20)): self.assertEqual(a, b['i']) self.assertEqual(80, len(list(self.db.test.find().limit(10).skip(40)[20:])) ) for a, b in zip(count(20), self.db.test.find().limit(10).skip(40)[20:]): self.assertEqual(a, b['i']) self.assertEqual(1, len(list(self.db.test.find()[:1]))) self.assertEqual(5, len(list(self.db.test.find()[:5]))) self.assertEqual(1, len(list(self.db.test.find()[99:100]))) self.assertEqual(1, len(list(self.db.test.find()[99:1000]))) self.assertEqual(0, len(list(self.db.test.find()[10:10]))) self.assertEqual(0, len(list(self.db.test.find()[:0]))) self.assertEqual(80, len(list(self.db.test.find()[10:10].limit(0).skip(20)) ) ) self.assertRaises(IndexError, lambda: self.db.test.find()[10:8]) def test_getitem_numeric_index(self): self.db.drop_collection("test") for i in range(100): self.db.test.save({"i": i}) self.assertEqual(0, self.db.test.find()[0]['i']) self.assertEqual(50, self.db.test.find()[50]['i']) self.assertEqual(50, self.db.test.find().skip(50)[0]['i']) self.assertEqual(50, self.db.test.find().skip(49)[1]['i']) self.assertEqual(50, self.db.test.find()[50L]['i']) self.assertEqual(99, self.db.test.find()[99]['i']) self.assertRaises(IndexError, lambda x: self.db.test.find()[x], -1) self.assertRaises(IndexError, lambda x: self.db.test.find()[x], 100) self.assertRaises(IndexError, lambda x: self.db.test.find().skip(50)[x], 50) def test_count_with_limit_and_skip(self): if not version.at_least(self.db.connection, (1, 1, 4, -1)): raise SkipTest("count with limit / skip requires MongoDB >= 1.1.4") def check_len(cursor, length): self.assertEqual(len(list(cursor)), cursor.count(True)) self.assertEqual(length, cursor.count(True)) self.db.drop_collection("test") for i in range(100): self.db.test.save({"i": i}) check_len(self.db.test.find(), 100) check_len(self.db.test.find().limit(10), 10) check_len(self.db.test.find().limit(110), 100) check_len(self.db.test.find().skip(10), 90) check_len(self.db.test.find().skip(110), 0) check_len(self.db.test.find().limit(10).skip(10), 10) check_len(self.db.test.find()[10:20], 10) check_len(self.db.test.find().limit(10).skip(95), 5) check_len(self.db.test.find()[95:105], 5) def test_len(self): self.assertRaises(TypeError, len, self.db.test.find()) def test_properties(self): self.assertEqual(self.db.test, self.db.test.find().collection) def set_coll(): self.db.test.find().collection = "hello" self.assertRaises(AttributeError, set_coll) def test_get_more(self): db = self.db db.drop_collection("test") db.test.insert([{'i': i} for i in range(10)]) self.assertEqual(10, len(list(db.test.find().batch_size(5)))) def test_tailable(self): db = self.db db.drop_collection("test") db.create_collection("test", capped=True, size=1000) cursor = db.test.find(tailable=True) db.test.insert({"x": 1}) count = 0 for doc in cursor: count += 1 self.assertEqual(1, doc["x"]) self.assertEqual(1, count) db.test.insert({"x": 2}) count = 0 for doc in cursor: count += 1 self.assertEqual(2, doc["x"]) self.assertEqual(1, count) db.test.insert({"x": 3}) count = 0 for doc in cursor: count += 1 self.assertEqual(3, doc["x"]) self.assertEqual(1, count) self.assertEqual(3, db.test.count()) db.drop_collection("test") def test_distinct(self): if not version.at_least(self.db.connection, (1, 1, 3, 1)): raise SkipTest("distinct with query requires MongoDB >= 1.1.3") self.db.drop_collection("test") self.db.test.save({"a": 1}) self.db.test.save({"a": 2}) self.db.test.save({"a": 2}) self.db.test.save({"a": 2}) self.db.test.save({"a": 3}) distinct = self.db.test.find({"a": {"$lt": 3}}).distinct("a") distinct.sort() self.assertEqual([1, 2], distinct) self.db.drop_collection("test") self.db.test.save({"a": {"b": "a"}, "c": 12}) self.db.test.save({"a": {"b": "b"}, "c": 8}) self.db.test.save({"a": {"b": "c"}, "c": 12}) self.db.test.save({"a": {"b": "c"}, "c": 8}) distinct = self.db.test.find({"c": 8}).distinct("a.b") distinct.sort() self.assertEqual(["b", "c"], distinct) def test_max_scan(self): if not version.at_least(self.db.connection, (1, 5, 1)): raise SkipTest("maxScan requires MongoDB >= 1.5.1") self.db.drop_collection("test") for _ in range(100): self.db.test.insert({}) self.assertEqual(100, len(list(self.db.test.find()))) self.assertEqual(50, len(list(self.db.test.find(max_scan=50)))) self.assertEqual(50, len(list(self.db.test.find() .max_scan(90).max_scan(50)))) def test_with_statement(self): if sys.version_info < (2, 6): raise SkipTest("With statement requires Python >= 2.6") self.db.drop_collection("test") for _ in range(100): self.db.test.insert({}) c1 = self.db.test.find() exec """ with self.db.test.find() as c2: self.assertTrue(c2.alive) self.assertFalse(c2.alive) with self.db.test.find() as c2: self.assertEqual(100, len(list(c2))) self.assertFalse(c2.alive) """ self.assertTrue(c1.alive) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_database.py000066400000000000000000000701461223300253600173520ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the database module.""" import datetime import os import sys sys.path[0:0] = [""] import unittest from nose.plugins.skip import SkipTest from bson.code import Code from bson.dbref import DBRef from bson.objectid import ObjectId from bson.son import SON from pymongo import (ALL, auth, OFF, SLOW_ONLY) from pymongo.collection import Collection from pymongo.database import Database from pymongo.errors import (CollectionInvalid, ConfigurationError, InvalidName, OperationFailure) from pymongo.son_manipulator import (AutoReference, NamespaceInjector, ObjectIdShuffler) from test import version from test.utils import is_mongos, server_started_with_auth from test.test_client import get_client class TestDatabase(unittest.TestCase): def setUp(self): self.client = get_client() def test_name(self): self.assertRaises(TypeError, Database, self.client, 4) self.assertRaises(InvalidName, Database, self.client, "my db") self.assertRaises(InvalidName, Database, self.client, "my\x00db") self.assertRaises(InvalidName, Database, self.client, u"my\u0000db") self.assertEqual("name", Database(self.client, "name").name) def test_equality(self): self.assertNotEqual(Database(self.client, "test"), Database(self.client, "mike")) self.assertEqual(Database(self.client, "test"), Database(self.client, "test")) # Explicitly test inequality self.assertFalse(Database(self.client, "test") != Database(self.client, "test")) def test_repr(self): self.assertEqual(repr(Database(self.client, "pymongo_test")), "Database(%r, %s)" % (self.client, repr(u"pymongo_test"))) def test_get_coll(self): db = Database(self.client, "pymongo_test") self.assertEqual(db.test, db["test"]) self.assertEqual(db.test, Collection(db, "test")) self.assertNotEqual(db.test, Collection(db, "mike")) self.assertEqual(db.test.mike, db["test.mike"]) def test_create_collection(self): db = Database(self.client, "pymongo_test") db.test.insert({"hello": "world"}) self.assertRaises(CollectionInvalid, db.create_collection, "test") db.drop_collection("test") self.assertRaises(TypeError, db.create_collection, 5) self.assertRaises(TypeError, db.create_collection, None) self.assertRaises(InvalidName, db.create_collection, "coll..ection") test = db.create_collection("test") test.save({"hello": u"world"}) self.assertEqual(db.test.find_one()["hello"], "world") self.assertTrue(u"test" in db.collection_names()) db.drop_collection("test.foo") db.create_collection("test.foo") self.assertTrue(u"test.foo" in db.collection_names()) self.assertEqual(db.test.foo.options(), {}) self.assertRaises(CollectionInvalid, db.create_collection, "test.foo") def test_collection_names(self): db = Database(self.client, "pymongo_test") db.test.save({"dummy": u"object"}) db.test.mike.save({"dummy": u"object"}) colls = db.collection_names() self.assertTrue("test" in colls) self.assertTrue("test.mike" in colls) for coll in colls: self.assertTrue("$" not in coll) colls_without_systems = db.collection_names(False) for coll in colls_without_systems: self.assertTrue(not coll.startswith("system.")) def test_drop_collection(self): db = Database(self.client, "pymongo_test") self.assertRaises(TypeError, db.drop_collection, 5) self.assertRaises(TypeError, db.drop_collection, None) db.test.save({"dummy": u"object"}) self.assertTrue("test" in db.collection_names()) db.drop_collection("test") self.assertFalse("test" in db.collection_names()) db.test.save({"dummy": u"object"}) self.assertTrue("test" in db.collection_names()) db.drop_collection(u"test") self.assertFalse("test" in db.collection_names()) db.test.save({"dummy": u"object"}) self.assertTrue("test" in db.collection_names()) db.drop_collection(db.test) self.assertFalse("test" in db.collection_names()) db.test.save({"dummy": u"object"}) self.assertTrue("test" in db.collection_names()) db.test.drop() self.assertFalse("test" in db.collection_names()) db.test.drop() db.drop_collection(db.test.doesnotexist) def test_validate_collection(self): db = self.client.pymongo_test self.assertRaises(TypeError, db.validate_collection, 5) self.assertRaises(TypeError, db.validate_collection, None) db.test.save({"dummy": u"object"}) self.assertRaises(OperationFailure, db.validate_collection, "test.doesnotexist") self.assertRaises(OperationFailure, db.validate_collection, db.test.doesnotexist) self.assertTrue(db.validate_collection("test")) self.assertTrue(db.validate_collection(db.test)) self.assertTrue(db.validate_collection(db.test, full=True)) self.assertTrue(db.validate_collection(db.test, scandata=True)) self.assertTrue(db.validate_collection(db.test, scandata=True, full=True)) self.assertTrue(db.validate_collection(db.test, True, True)) def test_profiling_levels(self): if is_mongos(self.client): raise SkipTest('profile is not supported by mongos') db = self.client.pymongo_test self.assertEqual(db.profiling_level(), OFF) # default self.assertRaises(ValueError, db.set_profiling_level, 5.5) self.assertRaises(ValueError, db.set_profiling_level, None) self.assertRaises(ValueError, db.set_profiling_level, -1) self.assertRaises(TypeError, db.set_profiling_level, SLOW_ONLY, 5.5) self.assertRaises(TypeError, db.set_profiling_level, SLOW_ONLY, '1') db.set_profiling_level(SLOW_ONLY) self.assertEqual(db.profiling_level(), SLOW_ONLY) db.set_profiling_level(ALL) self.assertEqual(db.profiling_level(), ALL) db.set_profiling_level(OFF) self.assertEqual(db.profiling_level(), OFF) db.set_profiling_level(SLOW_ONLY, 50) self.assertEqual(50, db.command("profile", -1)['slowms']) db.set_profiling_level(ALL, -1) self.assertEqual(-1, db.command("profile", -1)['slowms']) db.set_profiling_level(OFF, 100) # back to default self.assertEqual(100, db.command("profile", -1)['slowms']) def test_profiling_info(self): if is_mongos(self.client): raise SkipTest('profile is not supported by mongos') db = self.client.pymongo_test db.set_profiling_level(ALL) db.test.find() db.set_profiling_level(OFF) info = db.profiling_info() self.assertTrue(isinstance(info, list)) # Check if we're going to fail because of SERVER-4754, in which # profiling info isn't collected if mongod was started with --auth if server_started_with_auth(self.client): raise SkipTest( "We need SERVER-4754 fixed for the rest of this test to pass" ) self.assertTrue(len(info) >= 1) # These basically clue us in to server changes. if version.at_least(db.connection, (1, 9, 1, -1)): self.assertTrue(isinstance(info[0]['responseLength'], int)) self.assertTrue(isinstance(info[0]['millis'], int)) self.assertTrue(isinstance(info[0]['client'], basestring)) self.assertTrue(isinstance(info[0]['user'], basestring)) self.assertTrue(isinstance(info[0]['ntoreturn'], int)) self.assertTrue(isinstance(info[0]['ns'], basestring)) self.assertTrue(isinstance(info[0]['op'], basestring)) else: self.assertTrue(isinstance(info[0]["info"], basestring)) self.assertTrue(isinstance(info[0]["millis"], float)) self.assertTrue(isinstance(info[0]["ts"], datetime.datetime)) def test_iteration(self): db = self.client.pymongo_test def iterate(): [a for a in db] self.assertRaises(TypeError, iterate) def test_errors(self): if is_mongos(self.client): raise SkipTest('getpreverror not supported by mongos') db = self.client.pymongo_test db.reset_error_history() self.assertEqual(None, db.error()) self.assertEqual(None, db.previous_error()) db.command("forceerror", check=False) self.assertTrue(db.error()) self.assertTrue(db.previous_error()) db.command("forceerror", check=False) self.assertTrue(db.error()) prev_error = db.previous_error() self.assertEqual(prev_error["nPrev"], 1) del prev_error["nPrev"] prev_error.pop("lastOp", None) error = db.error() error.pop("lastOp", None) # getLastError includes "connectionId" in recent # server versions, getPrevError does not. error.pop("connectionId", None) self.assertEqual(error, prev_error) db.test.find_one() self.assertEqual(None, db.error()) self.assertTrue(db.previous_error()) self.assertEqual(db.previous_error()["nPrev"], 2) db.reset_error_history() self.assertEqual(None, db.error()) self.assertEqual(None, db.previous_error()) def test_command(self): db = self.client.admin self.assertEqual(db.command("buildinfo"), db.command({"buildinfo": 1})) def test_command_ignores_network_timeout(self): # command() should ignore network_timeout. if not version.at_least(self.client, (1, 9, 0)): raise SkipTest("Need sleep() to test command with network timeout") db = self.client.pymongo_test # No errors. db.test.remove() db.test.insert({}) cursor = db.test.find( {'$where': 'sleep(100); return true'}, network_timeout=0.001) self.assertEqual(1, cursor.count()) # mongos doesn't support the eval command if not is_mongos(self.client): db.command('eval', 'sleep(100)', network_timeout=0.001) def test_last_status(self): db = self.client.pymongo_test db.test.remove({}) db.test.save({"i": 1}) db.test.update({"i": 1}, {"$set": {"i": 2}}) self.assertTrue(db.last_status()["updatedExisting"]) db.test.update({"i": 1}, {"$set": {"i": 500}}) self.assertFalse(db.last_status()["updatedExisting"]) def test_password_digest(self): self.assertRaises(TypeError, auth._password_digest, 5) self.assertRaises(TypeError, auth._password_digest, True) self.assertRaises(TypeError, auth._password_digest, None) self.assertTrue(isinstance(auth._password_digest("mike", "password"), unicode)) self.assertEqual(auth._password_digest("mike", "password"), u"cd7e45b3b2767dc2fa9b6b548457ed00") self.assertEqual(auth._password_digest("mike", "password"), auth._password_digest(u"mike", u"password")) self.assertEqual(auth._password_digest("Gustave", u"Dor\xe9"), u"81e0e2364499209f466e75926a162d73") def test_authenticate_add_remove_user(self): if (is_mongos(self.client) and not version.at_least(self.client, (2, 0, 0))): raise SkipTest("Auth with sharding requires MongoDB >= 2.0.0") db = self.client.pymongo_test db.system.users.remove({}) db.remove_user("mike") self.assertRaises(TypeError, db.add_user, "user", '') self.assertRaises(TypeError, db.add_user, "user", 'password', 15) self.assertRaises(ConfigurationError, db.add_user, "user", 'password', 'True') db.add_user("mike", "password") self.assertRaises(TypeError, db.authenticate, 5, "password") self.assertRaises(TypeError, db.authenticate, "mike", 5) self.assertRaises(OperationFailure, db.authenticate, "mike", "not a real password") self.assertRaises(OperationFailure, db.authenticate, "faker", "password") self.assertTrue(db.authenticate("mike", "password")) self.assertTrue(db.authenticate(u"mike", u"password")) db.logout() db.remove_user("mike") self.assertRaises(OperationFailure, db.authenticate, "mike", "password") self.assertRaises(OperationFailure, db.authenticate, "Gustave", u"Dor\xe9") db.add_user("Gustave", u"Dor\xe9") self.assertTrue(db.authenticate("Gustave", u"Dor\xe9")) db.logout() db.add_user("Gustave", "password") self.assertRaises(OperationFailure, db.authenticate, "Gustave", u"Dor\xe9") self.assertTrue(db.authenticate("Gustave", u"password")) db.logout() db.add_user("Ross", "password", read_only=True) self.assertTrue(db.authenticate("Ross", u"password")) self.assertTrue(db.system.users.find({"readOnly": True}).count()) db.logout() def test_authenticate_and_safe(self): if (is_mongos(self.client) and not version.at_least(self.client, (2, 0, 0))): raise SkipTest("Auth with sharding requires MongoDB >= 2.0.0") db = self.client.auth_test db.system.users.remove({}) db.add_user("bernie", "password") db.authenticate("bernie", "password") db.test.remove({}) self.assertTrue(db.test.insert({"bim": "baz"})) self.assertEqual(1, db.test.count()) self.assertEqual(1, db.test.update({"bim": "baz"}, {"$set": {"bim": "bar"}}).get('n')) self.assertEqual(1, db.test.remove({}).get('n')) self.assertEqual(0, db.test.count()) self.client.drop_database("auth_test") def test_authenticate_and_request(self): if (is_mongos(self.client) and not version.at_least(self.client, (2, 0, 0))): raise SkipTest("Auth with sharding requires MongoDB >= 2.0.0") # Database.authenticate() needs to be in a request - check that it # always runs in a request, and that it restores the request state # (in or not in a request) properly when it's finished. self.assertFalse(self.client.auto_start_request) db = self.client.pymongo_test db.system.users.remove({}) db.remove_user("mike") db.add_user("mike", "password") self.assertFalse(self.client.in_request()) self.assertTrue(db.authenticate("mike", "password")) self.assertFalse(self.client.in_request()) request_cx = get_client(auto_start_request=True) request_db = request_cx.pymongo_test self.assertTrue(request_cx.in_request()) self.assertTrue(request_db.authenticate("mike", "password")) self.assertTrue(request_cx.in_request()) # just make sure there are no exceptions here db.logout() db.collection.find_one() request_db.logout() request_db.collection.find_one() def test_authenticate_multiple(self): client = get_client() if (is_mongos(client) and not version.at_least(self.client, (2, 0, 0))): raise SkipTest("Auth with sharding requires MongoDB >= 2.0.0") if not server_started_with_auth(client): raise SkipTest("Authentication is not enabled on server") # Setup users_db = client.pymongo_test admin_db = client.admin other_db = client.pymongo_test1 users_db.system.users.remove() admin_db.system.users.remove() users_db.test.remove() other_db.test.remove() admin_db.add_user('admin', 'pass') self.assertTrue(admin_db.authenticate('admin', 'pass')) admin_db.add_user('ro-admin', 'pass', read_only=True) users_db.add_user('user', 'pass') admin_db.logout() self.assertRaises(OperationFailure, users_db.test.find_one) # Regular user should be able to query its own db, but # no other. users_db.authenticate('user', 'pass') self.assertEqual(0, users_db.test.count()) self.assertRaises(OperationFailure, other_db.test.find_one) # Admin read-only user should be able to query any db, # but not write. admin_db.authenticate('ro-admin', 'pass') self.assertEqual(0, other_db.test.count()) self.assertRaises(OperationFailure, other_db.test.insert, {}) # Force close all sockets client.disconnect() # We should still be able to write to the regular user's db self.assertTrue(users_db.test.remove()) # And read from other dbs... self.assertEqual(0, other_db.test.count()) # But still not write to other dbs... self.assertRaises(OperationFailure, other_db.test.insert, {}) # Cleanup admin_db.logout() users_db.logout() self.assertTrue(admin_db.authenticate('admin', 'pass')) self.assertTrue(admin_db.system.users.remove()) self.assertEqual(0, admin_db.system.users.count()) self.assertTrue(users_db.system.users.remove()) def test_id_ordering(self): # PyMongo attempts to have _id show up first # when you iterate key/value pairs in a document. # This isn't reliable since python dicts don't # guarantee any particular order. This will never # work right in Jython or Python >= 3.3 with # hash randomization enabled. db = self.client.pymongo_test db.test.remove({}) db.test.insert(SON([("hello", "world"), ("_id", 5)])) if ((sys.version_info >= (3, 3) and os.environ.get('PYTHONHASHSEED') != '0') or sys.platform.startswith('java')): # See http://bugs.python.org/issue13703 for why we # use as_class=SON in certain environments. cursor = db.test.find(as_class=SON) else: cursor = db.test.find() for x in cursor: for (k, v) in x.items(): self.assertEqual(k, "_id") break def test_deref(self): db = self.client.pymongo_test db.test.remove({}) self.assertRaises(TypeError, db.dereference, 5) self.assertRaises(TypeError, db.dereference, "hello") self.assertRaises(TypeError, db.dereference, None) self.assertEqual(None, db.dereference(DBRef("test", ObjectId()))) obj = {"x": True} key = db.test.save(obj) self.assertEqual(obj, db.dereference(DBRef("test", key))) self.assertEqual(obj, db.dereference(DBRef("test", key, "pymongo_test"))) self.assertRaises(ValueError, db.dereference, DBRef("test", key, "foo")) self.assertEqual(None, db.dereference(DBRef("test", 4))) obj = {"_id": 4} db.test.save(obj) self.assertEqual(obj, db.dereference(DBRef("test", 4))) def test_eval(self): db = self.client.pymongo_test db.test.remove({}) self.assertRaises(TypeError, db.eval, None) self.assertRaises(TypeError, db.eval, 5) self.assertRaises(TypeError, db.eval, []) self.assertEqual(3, db.eval("function (x) {return x;}", 3)) self.assertEqual(3, db.eval(u"function (x) {return x;}", 3)) self.assertEqual(None, db.eval("function (x) {db.test.save({y:x});}", 5)) self.assertEqual(db.test.find_one()["y"], 5) self.assertEqual(5, db.eval("function (x, y) {return x + y;}", 2, 3)) self.assertEqual(5, db.eval("function () {return 5;}")) self.assertEqual(5, db.eval("2 + 3;")) self.assertEqual(5, db.eval(Code("2 + 3;"))) self.assertRaises(OperationFailure, db.eval, Code("return i;")) self.assertEqual(2, db.eval(Code("return i;", {"i": 2}))) self.assertEqual(5, db.eval(Code("i + 3;", {"i": 2}))) self.assertRaises(OperationFailure, db.eval, "5 ++ 5;") # TODO some of these tests belong in the collection level testing. def test_save_find_one(self): db = Database(self.client, "pymongo_test") db.test.remove({}) a_doc = SON({"hello": u"world"}) a_key = db.test.save(a_doc) self.assertTrue(isinstance(a_doc["_id"], ObjectId)) self.assertEqual(a_doc["_id"], a_key) self.assertEqual(a_doc, db.test.find_one({"_id": a_doc["_id"]})) self.assertEqual(a_doc, db.test.find_one(a_key)) self.assertEqual(None, db.test.find_one(ObjectId())) self.assertEqual(a_doc, db.test.find_one({"hello": u"world"})) self.assertEqual(None, db.test.find_one({"hello": u"test"})) b = db.test.find_one() b["hello"] = u"mike" db.test.save(b) self.assertNotEqual(a_doc, db.test.find_one(a_key)) self.assertEqual(b, db.test.find_one(a_key)) self.assertEqual(b, db.test.find_one()) count = 0 for _ in db.test.find(): count += 1 self.assertEqual(count, 1) def test_long(self): db = self.client.pymongo_test db.test.remove({}) db.test.save({"x": 9223372036854775807L}) self.assertEqual(9223372036854775807L, db.test.find_one()["x"]) def test_remove(self): db = self.client.pymongo_test db.test.remove({}) one = db.test.save({"x": 1}) db.test.save({"x": 2}) db.test.save({"x": 3}) length = 0 for _ in db.test.find(): length += 1 self.assertEqual(length, 3) db.test.remove(one) length = 0 for _ in db.test.find(): length += 1 self.assertEqual(length, 2) db.test.remove(db.test.find_one()) db.test.remove(db.test.find_one()) self.assertEqual(db.test.find_one(), None) one = db.test.save({"x": 1}) db.test.save({"x": 2}) db.test.save({"x": 3}) self.assertTrue(db.test.find_one({"x": 2})) db.test.remove({"x": 2}) self.assertFalse(db.test.find_one({"x": 2})) self.assertTrue(db.test.find_one()) db.test.remove({}) self.assertFalse(db.test.find_one()) def test_save_a_bunch(self): db = self.client.pymongo_test db.test.remove({}) for i in xrange(1000): db.test.save({"x": i}) count = 0 for _ in db.test.find(): count += 1 self.assertEqual(1000, count) # test that kill cursors doesn't assert or anything for _ in xrange(62): for _ in db.test.find(): break def test_auto_ref_and_deref(self): db = self.client.pymongo_test db.add_son_manipulator(AutoReference(db)) db.add_son_manipulator(NamespaceInjector()) db.test.a.remove({}) db.test.b.remove({}) db.test.c.remove({}) a = {"hello": u"world"} db.test.a.save(a) b = {"test": a} db.test.b.save(b) c = {"another test": b} db.test.c.save(c) a["hello"] = "mike" db.test.a.save(a) self.assertEqual(db.test.a.find_one(), a) self.assertEqual(db.test.b.find_one()["test"], a) self.assertEqual(db.test.c.find_one()["another test"]["test"], a) self.assertEqual(db.test.b.find_one(), b) self.assertEqual(db.test.c.find_one()["another test"], b) self.assertEqual(db.test.c.find_one(), c) # some stuff the user marc wanted to be able to do, make sure it works def test_marc(self): db = self.client.pymongo_test db.add_son_manipulator(AutoReference(db)) db.add_son_manipulator(NamespaceInjector()) db.drop_collection("users") db.drop_collection("messages") message_1 = {"title": "foo"} db.messages.save(message_1) message_2 = {"title": "bar"} db.messages.save(message_2) user = {"name": "marc", "messages": [message_1, message_2]} db.users.save(user) message = db.messages.find_one() db.messages.update(message, {"title": "buzz"}) self.assertEqual("buzz", db.users.find_one()["messages"][0]["title"]) self.assertEqual("bar", db.users.find_one()["messages"][1]["title"]) def test_system_js(self): db = self.client.pymongo_test db.system.js.remove() self.assertEqual(0, db.system.js.count()) db.system_js.add = "function(a, b) { return a + b; }" self.assertEqual('add', db.system.js.find_one()['_id']) self.assertEqual(1, db.system.js.count()) self.assertEqual(6, db.system_js.add(1, 5)) del db.system_js.add self.assertEqual(0, db.system.js.count()) db.system_js['add'] = "function(a, b) { return a + b; }" self.assertEqual('add', db.system.js.find_one()['_id']) self.assertEqual(1, db.system.js.count()) self.assertEqual(6, db.system_js['add'](1, 5)) del db.system_js['add'] self.assertEqual(0, db.system.js.count()) if version.at_least(db.connection, (1, 3, 2, -1)): self.assertRaises(OperationFailure, db.system_js.add, 1, 5) # TODO right now CodeWScope doesn't work w/ system js # db.system_js.scope = Code("return hello;", {"hello": 8}) # self.assertEqual(8, db.system_js.scope()) self.assertRaises(OperationFailure, db.system_js.non_existant) # XXX: Broken in V8, works in SpiderMonkey if not version.at_least(db.connection, (2, 3, 0)): db.system_js.no_param = Code("return 5;") self.assertEqual(5, db.system_js.no_param()) def test_system_js_list(self): db = self.client.pymongo_test db.system.js.remove() self.assertEqual([], db.system_js.list()) db.system_js.foo = "function() { return 'blah'; }" self.assertEqual(["foo"], db.system_js.list()) db.system_js.bar = "function() { return 'baz'; }" self.assertEqual(set(["foo", "bar"]), set(db.system_js.list())) del db.system_js.foo self.assertEqual(["bar"], db.system_js.list()) def test_manipulator_properties(self): db = self.client.foo self.assertEqual(['ObjectIdInjector'], db.incoming_manipulators) self.assertEqual([], db.incoming_copying_manipulators) self.assertEqual([], db.outgoing_manipulators) self.assertEqual([], db.outgoing_copying_manipulators) db.add_son_manipulator(AutoReference(db)) db.add_son_manipulator(NamespaceInjector()) db.add_son_manipulator(ObjectIdShuffler()) self.assertEqual(2, len(db.incoming_manipulators)) for name in db.incoming_manipulators: self.assertTrue(name in ('ObjectIdInjector', 'NamespaceInjector')) self.assertEqual(2, len(db.incoming_copying_manipulators)) for name in db.incoming_copying_manipulators: self.assertTrue(name in ('ObjectIdShuffler', 'AutoReference')) self.assertEqual([], db.outgoing_manipulators) self.assertEqual(['AutoReference'], db.outgoing_copying_manipulators) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_dbref.py000066400000000000000000000131331223300253600166610ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tests for the dbref module.""" import os import pickle import unittest import sys sys.path[0:0] = [""] from bson.dbref import DBRef from bson.objectid import ObjectId from bson.py3compat import b from copy import deepcopy class TestDBRef(unittest.TestCase): def setUp(self): pass def test_creation(self): a = ObjectId() self.assertRaises(TypeError, DBRef) self.assertRaises(TypeError, DBRef, "coll") self.assertRaises(TypeError, DBRef, 4, a) self.assertRaises(TypeError, DBRef, 1.5, a) self.assertRaises(TypeError, DBRef, a, a) self.assertRaises(TypeError, DBRef, None, a) self.assertRaises(TypeError, DBRef, "coll", a, 5) self.assertTrue(DBRef("coll", a)) self.assertTrue(DBRef(u"coll", a)) self.assertTrue(DBRef(u"coll", 5)) self.assertTrue(DBRef(u"coll", 5, "database")) def test_read_only(self): a = DBRef("coll", ObjectId()) def foo(): a.collection = "blah" def bar(): a.id = "aoeu" self.assertEqual("coll", a.collection) a.id self.assertEqual(None, a.database) self.assertRaises(AttributeError, foo) self.assertRaises(AttributeError, bar) def test_repr(self): self.assertEqual(repr(DBRef("coll", ObjectId("1234567890abcdef12345678"))), "DBRef('coll', ObjectId('1234567890abcdef12345678'))") self.assertEqual(repr(DBRef(u"coll", ObjectId("1234567890abcdef12345678"))), "DBRef(%s, ObjectId('1234567890abcdef12345678'))" % (repr(u'coll'),) ) self.assertEqual(repr(DBRef("coll", 5, foo="bar")), "DBRef('coll', 5, foo='bar')") self.assertEqual(repr(DBRef("coll", ObjectId("1234567890abcdef12345678"), "foo")), "DBRef('coll', ObjectId('1234567890abcdef12345678'), " "'foo')") # This assert will fail in Python 3.3+ unless # hash randomization is disabled. if (sys.version_info < (3, 3) or os.environ.get('PYTHONHASHSEED') == '0'): self.assertEqual(repr(DBRef("coll", 5, "baz", foo="bar", baz=4)), "DBRef('coll', 5, 'baz', foo='bar', baz=4)") def test_equality(self): obj_id = ObjectId("1234567890abcdef12345678") self.assertEqual(DBRef('foo', 5), DBRef('foo', 5)) self.assertEqual(DBRef("coll", obj_id), DBRef(u"coll", obj_id)) self.assertNotEqual(DBRef("coll", obj_id), DBRef(u"coll", obj_id, "foo")) self.assertNotEqual(DBRef("coll", obj_id), DBRef("col", obj_id)) self.assertNotEqual(DBRef("coll", obj_id), DBRef("coll", ObjectId(b("123456789011")))) self.assertNotEqual(DBRef("coll", obj_id), 4) self.assertEqual(DBRef("coll", obj_id, "foo"), DBRef(u"coll", obj_id, "foo")) self.assertNotEqual(DBRef("coll", obj_id, "foo"), DBRef(u"coll", obj_id, "bar")) # Explicitly test inequality self.assertFalse(DBRef('foo', 5) != DBRef('foo', 5)) self.assertFalse(DBRef("coll", obj_id) != DBRef(u"coll", obj_id)) self.assertFalse(DBRef("coll", obj_id, "foo") != DBRef(u"coll", obj_id, "foo")) def test_kwargs(self): self.assertEqual(DBRef("coll", 5, foo="bar"), DBRef("coll", 5, foo="bar")) self.assertNotEqual(DBRef("coll", 5, foo="bar"), DBRef("coll", 5)) self.assertNotEqual(DBRef("coll", 5, foo="bar"), DBRef("coll", 5, foo="baz")) self.assertEqual("bar", DBRef("coll", 5, foo="bar").foo) self.assertRaises(AttributeError, getattr, DBRef("coll", 5, foo="bar"), "bar") def test_deepcopy(self): a = DBRef('coll', 'asdf', 'db', x=[1]) b = deepcopy(a) self.assertEqual(a, b) self.assertNotEqual(id(a), id(b.x)) self.assertEqual(a.x, b.x) self.assertNotEqual(id(a.x), id(b.x)) b.x[0] = 2 self.assertEqual(a.x, [1]) self.assertEqual(b.x, [2]) def test_pickling(self): dbr = DBRef('coll', 5, foo='bar') for protocol in [0, 1, 2, -1]: pkl = pickle.dumps(dbr, protocol=protocol) dbr2 = pickle.loads(pkl) self.assertEqual(dbr, dbr2) def test_dbref_hash(self): dbref_1a = DBRef('collection', 'id', 'database') dbref_1b = DBRef('collection', 'id', 'database') self.assertEqual(hash(dbref_1a), hash(dbref_1b)) dbref_2a = DBRef('collection', 'id', 'database', custom='custom') dbref_2b = DBRef('collection', 'id', 'database', custom='custom') self.assertEqual(hash(dbref_2a), hash(dbref_2b)) self.assertNotEqual(hash(dbref_1a), hash(dbref_2a)) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_errors.py000066400000000000000000000016221223300253600171130ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the errors module.""" import unittest import sys sys.path[0:0] = [""] from pymongo import MongoClient from pymongo.errors import PyMongoError class TestErrors(unittest.TestCase): def test_base_exception(self): self.assertRaises(PyMongoError, MongoClient, port=0) if __name__ == '__main__': unittest.main() pymongo-2.6.3/test/test_grid_file.py000066400000000000000000000426551223300253600175360ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tests for the grid_file module. """ import datetime import sys import unittest sys.path[0:0] = [""] from nose.plugins.skip import SkipTest from bson.objectid import ObjectId from bson.py3compat import b, StringIO from gridfs import GridFS from gridfs.grid_file import (DEFAULT_CHUNK_SIZE, _SEEK_CUR, _SEEK_END, GridIn, GridFile, GridOut) from gridfs.errors import (NoFile, UnsupportedAPI) from test.test_client import get_client from test import qcheck class TestGridFile(unittest.TestCase): def setUp(self): self.db = get_client().pymongo_test self.db.fs.files.remove({}) self.db.fs.chunks.remove({}) def test_basic(self): f = GridIn(self.db.fs, filename="test") f.write(b("hello world")) f.close() self.assertEqual(1, self.db.fs.files.find().count()) self.assertEqual(1, self.db.fs.chunks.find().count()) g = GridOut(self.db.fs, f._id) self.assertEqual(b("hello world"), g.read()) # make sure it's still there... g = GridOut(self.db.fs, f._id) self.assertEqual(b("hello world"), g.read()) f = GridIn(self.db.fs, filename="test") f.close() self.assertEqual(2, self.db.fs.files.find().count()) self.assertEqual(1, self.db.fs.chunks.find().count()) g = GridOut(self.db.fs, f._id) self.assertEqual(b(""), g.read()) def test_md5(self): f = GridIn(self.db.fs) f.write(b("hello world\n")) f.close() self.assertEqual("6f5902ac237024bdd0c176cb93063dc4", f.md5) def test_alternate_collection(self): self.db.alt.files.remove({}) self.db.alt.chunks.remove({}) f = GridIn(self.db.alt) f.write(b("hello world")) f.close() self.assertEqual(1, self.db.alt.files.find().count()) self.assertEqual(1, self.db.alt.chunks.find().count()) g = GridOut(self.db.alt, f._id) self.assertEqual(b("hello world"), g.read()) # test that md5 still works... self.assertEqual("5eb63bbbe01eeed093cb22bb8f5acdc3", g.md5) def test_grid_file(self): self.assertRaises(UnsupportedAPI, GridFile) def test_grid_in_default_opts(self): self.assertRaises(TypeError, GridIn, "foo") a = GridIn(self.db.fs) self.assertTrue(isinstance(a._id, ObjectId)) self.assertRaises(AttributeError, setattr, a, "_id", 5) self.assertEqual(None, a.filename) self.assertEqual(None, a.name) a.filename = "my_file" self.assertEqual("my_file", a.filename) self.assertEqual("my_file", a.name) self.assertEqual(None, a.content_type) a.content_type = "text/html" self.assertEqual("text/html", a.content_type) self.assertRaises(AttributeError, getattr, a, "length") self.assertRaises(AttributeError, setattr, a, "length", 5) self.assertEqual(256 * 1024, a.chunk_size) self.assertRaises(AttributeError, setattr, a, "chunk_size", 5) self.assertRaises(AttributeError, getattr, a, "upload_date") self.assertRaises(AttributeError, setattr, a, "upload_date", 5) self.assertRaises(AttributeError, getattr, a, "aliases") a.aliases = ["foo"] self.assertEqual(["foo"], a.aliases) self.assertRaises(AttributeError, getattr, a, "metadata") a.metadata = {"foo": 1} self.assertEqual({"foo": 1}, a.metadata) self.assertRaises(AttributeError, getattr, a, "md5") self.assertRaises(AttributeError, setattr, a, "md5", 5) a.close() a.forty_two = 42 self.assertEqual(42, a.forty_two) self.assertTrue(isinstance(a._id, ObjectId)) self.assertRaises(AttributeError, setattr, a, "_id", 5) self.assertEqual("my_file", a.filename) self.assertEqual("my_file", a.name) self.assertEqual("text/html", a.content_type) self.assertEqual(0, a.length) self.assertRaises(AttributeError, setattr, a, "length", 5) self.assertEqual(256 * 1024, a.chunk_size) self.assertRaises(AttributeError, setattr, a, "chunk_size", 5) self.assertTrue(isinstance(a.upload_date, datetime.datetime)) self.assertRaises(AttributeError, setattr, a, "upload_date", 5) self.assertEqual(["foo"], a.aliases) self.assertEqual({"foo": 1}, a.metadata) self.assertEqual("d41d8cd98f00b204e9800998ecf8427e", a.md5) self.assertRaises(AttributeError, setattr, a, "md5", 5) # Make sure custom attributes that were set both before and after # a.close() are reflected in b. PYTHON-411. b = GridFS(self.db).get_last_version(filename=a.filename) self.assertEqual(a.metadata, b.metadata) self.assertEqual(a.aliases, b.aliases) self.assertEqual(a.forty_two, b.forty_two) def test_grid_in_custom_opts(self): self.assertRaises(TypeError, GridIn, "foo") a = GridIn(self.db.fs, _id=5, filename="my_file", contentType="text/html", chunkSize=1000, aliases=["foo"], metadata={"foo": 1, "bar": 2}, bar=3, baz="hello") self.assertEqual(5, a._id) self.assertEqual("my_file", a.filename) self.assertEqual("my_file", a.name) self.assertEqual("text/html", a.content_type) self.assertEqual(1000, a.chunk_size) self.assertEqual(["foo"], a.aliases) self.assertEqual({"foo": 1, "bar": 2}, a.metadata) self.assertEqual(3, a.bar) self.assertEqual("hello", a.baz) self.assertRaises(AttributeError, getattr, a, "mike") b = GridIn(self.db.fs, content_type="text/html", chunk_size=1000, baz=100) self.assertEqual("text/html", b.content_type) self.assertEqual(1000, b.chunk_size) self.assertEqual(100, b.baz) def test_grid_out_default_opts(self): self.assertRaises(TypeError, GridOut, "foo") self.assertRaises(NoFile, GridOut, self.db.fs, 5) a = GridIn(self.db.fs) a.close() b = GridOut(self.db.fs, a._id) self.assertEqual(a._id, b._id) self.assertEqual(0, b.length) self.assertEqual(None, b.content_type) self.assertEqual(None, b.name) self.assertEqual(None, b.filename) self.assertEqual(256 * 1024, b.chunk_size) self.assertTrue(isinstance(b.upload_date, datetime.datetime)) self.assertEqual(None, b.aliases) self.assertEqual(None, b.metadata) self.assertEqual("d41d8cd98f00b204e9800998ecf8427e", b.md5) for attr in ["_id", "name", "content_type", "length", "chunk_size", "upload_date", "aliases", "metadata", "md5"]: self.assertRaises(AttributeError, setattr, b, attr, 5) def test_grid_out_custom_opts(self): one = GridIn(self.db.fs, _id=5, filename="my_file", contentType="text/html", chunkSize=1000, aliases=["foo"], metadata={"foo": 1, "bar": 2}, bar=3, baz="hello") one.write(b("hello world")) one.close() two = GridOut(self.db.fs, 5) self.assertEqual("my_file", two.name) self.assertEqual("my_file", two.filename) self.assertEqual(5, two._id) self.assertEqual(11, two.length) self.assertEqual("text/html", two.content_type) self.assertEqual(1000, two.chunk_size) self.assertTrue(isinstance(two.upload_date, datetime.datetime)) self.assertEqual(["foo"], two.aliases) self.assertEqual({"foo": 1, "bar": 2}, two.metadata) self.assertEqual(3, two.bar) self.assertEqual("5eb63bbbe01eeed093cb22bb8f5acdc3", two.md5) for attr in ["_id", "name", "content_type", "length", "chunk_size", "upload_date", "aliases", "metadata", "md5"]: self.assertRaises(AttributeError, setattr, two, attr, 5) def test_grid_out_file_document(self): one = GridIn(self.db.fs) one.write(b("foo bar")) one.close() two = GridOut(self.db.fs, file_document=self.db.fs.files.find_one()) self.assertEqual(b("foo bar"), two.read()) three = GridOut(self.db.fs, 5, file_document=self.db.fs.files.find_one()) self.assertEqual(b("foo bar"), three.read()) self.assertRaises(NoFile, GridOut, self.db.fs, file_document={}) def test_write_file_like(self): one = GridIn(self.db.fs) one.write(b("hello world")) one.close() two = GridOut(self.db.fs, one._id) three = GridIn(self.db.fs) three.write(two) three.close() four = GridOut(self.db.fs, three._id) self.assertEqual(b("hello world"), four.read()) five = GridIn(self.db.fs, chunk_size=2) five.write(b("hello")) buffer = StringIO(b(" world")) five.write(buffer) five.write(b(" and mongodb")) five.close() self.assertEqual(b("hello world and mongodb"), GridOut(self.db.fs, five._id).read()) def test_write_lines(self): a = GridIn(self.db.fs) a.writelines([b("hello "), b("world")]) a.close() self.assertEqual(b("hello world"), GridOut(self.db.fs, a._id).read()) def test_close(self): f = GridIn(self.db.fs) f.close() self.assertRaises(ValueError, f.write, "test") f.close() def test_multi_chunk_file(self): random_string = qcheck.gen_string(qcheck.lift(300000))() f = GridIn(self.db.fs) f.write(random_string) f.close() self.assertEqual(1, self.db.fs.files.find().count()) self.assertEqual(2, self.db.fs.chunks.find().count()) g = GridOut(self.db.fs, f._id) self.assertEqual(random_string, g.read()) def test_small_chunks(self): self.files = 0 self.chunks = 0 def helper(data): f = GridIn(self.db.fs, chunkSize=1) f.write(data) f.close() self.files += 1 self.chunks += len(data) self.assertEqual(self.files, self.db.fs.files.find().count()) self.assertEqual(self.chunks, self.db.fs.chunks.find().count()) g = GridOut(self.db.fs, f._id) self.assertEqual(data, g.read()) g = GridOut(self.db.fs, f._id) self.assertEqual(data, g.read(10) + g.read(10)) return True qcheck.check_unittest(self, helper, qcheck.gen_string(qcheck.gen_range(0, 20))) def test_seek(self): f = GridIn(self.db.fs, chunkSize=3) f.write(b("hello world")) f.close() g = GridOut(self.db.fs, f._id) self.assertEqual(b("hello world"), g.read()) g.seek(0) self.assertEqual(b("hello world"), g.read()) g.seek(1) self.assertEqual(b("ello world"), g.read()) self.assertRaises(IOError, g.seek, -1) g.seek(-3, _SEEK_END) self.assertEqual(b("rld"), g.read()) g.seek(0, _SEEK_END) self.assertEqual(b(""), g.read()) self.assertRaises(IOError, g.seek, -100, _SEEK_END) g.seek(3) g.seek(3, _SEEK_CUR) self.assertEqual(b("world"), g.read()) self.assertRaises(IOError, g.seek, -100, _SEEK_CUR) def test_tell(self): f = GridIn(self.db.fs, chunkSize=3) f.write(b("hello world")) f.close() g = GridOut(self.db.fs, f._id) self.assertEqual(0, g.tell()) g.read(0) self.assertEqual(0, g.tell()) g.read(1) self.assertEqual(1, g.tell()) g.read(2) self.assertEqual(3, g.tell()) g.read() self.assertEqual(g.length, g.tell()) def test_multiple_reads(self): f = GridIn(self.db.fs, chunkSize=3) f.write(b("hello world")) f.close() g = GridOut(self.db.fs, f._id) self.assertEqual(b("he"), g.read(2)) self.assertEqual(b("ll"), g.read(2)) self.assertEqual(b("o "), g.read(2)) self.assertEqual(b("wo"), g.read(2)) self.assertEqual(b("rl"), g.read(2)) self.assertEqual(b("d"), g.read(2)) self.assertEqual(b(""), g.read(2)) def test_readline(self): f = GridIn(self.db.fs, chunkSize=5) f.write(b("""Hello world, How are you? Hope all is well. Bye""")) f.close() g = GridOut(self.db.fs, f._id) self.assertEqual(b("H"), g.read(1)) self.assertEqual(b("ello world,\n"), g.readline()) self.assertEqual(b("How a"), g.readline(5)) self.assertEqual(b(""), g.readline(0)) self.assertEqual(b("re you?\n"), g.readline()) self.assertEqual(b("Hope all is well.\n"), g.readline(1000)) self.assertEqual(b("Bye"), g.readline()) self.assertEqual(b(""), g.readline()) def test_iterator(self): f = GridIn(self.db.fs) f.close() g = GridOut(self.db.fs, f._id) self.assertEqual([], list(g)) f = GridIn(self.db.fs) f.write(b("hello world")) f.close() g = GridOut(self.db.fs, f._id) self.assertEqual([b("hello world")], list(g)) self.assertEqual(b("hello"), g.read(5)) self.assertEqual([b("hello world")], list(g)) self.assertEqual(b(" worl"), g.read(5)) f = GridIn(self.db.fs, chunk_size=2) f.write(b("hello world")) f.close() g = GridOut(self.db.fs, f._id) self.assertEqual([b("he"), b("ll"), b("o "), b("wo"), b("rl"), b("d")], list(g)) def test_read_chunks_unaligned_buffer_size(self): in_data = b("This is a text that doesn't " "quite fit in a single 16-byte chunk.") f = GridIn(self.db.fs, chunkSize=16) f.write(in_data) f.close() g = GridOut(self.db.fs, f._id) out_data = b('') while 1: s = g.read(13) if not s: break out_data += s self.assertEqual(in_data, out_data) def test_write_unicode(self): f = GridIn(self.db.fs) self.assertRaises(TypeError, f.write, u"foo") f = GridIn(self.db.fs, encoding="utf-8") f.write(u"foo") f.close() g = GridOut(self.db.fs, f._id) self.assertEqual(b("foo"), g.read()) f = GridIn(self.db.fs, encoding="iso-8859-1") f.write(u"aé") f.close() g = GridOut(self.db.fs, f._id) self.assertEqual(u"aé".encode("iso-8859-1"), g.read()) def test_set_after_close(self): f = GridIn(self.db.fs, _id="foo", bar="baz") self.assertEqual("foo", f._id) self.assertEqual("baz", f.bar) self.assertRaises(AttributeError, getattr, f, "baz") self.assertRaises(AttributeError, getattr, f, "uploadDate") self.assertRaises(AttributeError, setattr, f, "_id", 5) f.bar = "foo" f.baz = 5 self.assertEqual("foo", f._id) self.assertEqual("foo", f.bar) self.assertEqual(5, f.baz) self.assertRaises(AttributeError, getattr, f, "uploadDate") f.close() self.assertEqual("foo", f._id) self.assertEqual("foo", f.bar) self.assertEqual(5, f.baz) self.assertTrue(f.uploadDate) self.assertRaises(AttributeError, setattr, f, "_id", 5) f.bar = "a" f.baz = "b" self.assertRaises(AttributeError, setattr, f, "upload_date", 5) g = GridOut(self.db.fs, f._id) self.assertEqual("a", g.bar) self.assertEqual("b", g.baz) # Versions 2.0.1 and older saved a _closed field for some reason. self.assertRaises(AttributeError, getattr, g, "_closed") def test_context_manager(self): if sys.version_info < (2, 6): raise SkipTest("With statement requires Python >= 2.6") contents = b("Imagine this is some important data...") # Hack around python2.4 an 2.5 not supporting 'with' syntax exec """ with GridIn(self.db.fs, filename="important") as infile: infile.write(contents) with GridOut(self.db.fs, infile._id) as outfile: self.assertEqual(contents, outfile.read()) """ def test_prechunked_string(self): def write_me(s, chunk_size): buf = StringIO(s) infile = GridIn(self.db.fs) while True: to_write = buf.read(chunk_size) if to_write == b(''): break infile.write(to_write) infile.close() buf.close() outfile = GridOut(self.db.fs, infile._id) data = outfile.read() self.assertEqual(s, data) s = b('x' * DEFAULT_CHUNK_SIZE * 4) # Test with default chunk size write_me(s, DEFAULT_CHUNK_SIZE) # Multiple write_me(s, DEFAULT_CHUNK_SIZE * 3) # Custom write_me(s, 262300) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_gridfs.py000066400000000000000000000344521223300253600170640ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tests for the gridfs package. """ import sys sys.path[0:0] = [""] from pymongo.mongo_client import MongoClient from pymongo.errors import AutoReconnect from pymongo.read_preferences import ReadPreference from test.test_replica_set_client import TestReplicaSetClientBase import datetime import unittest import threading import time import gridfs from bson.py3compat import b, StringIO from gridfs.errors import (FileExists, NoFile) from test.test_client import get_client from test.utils import joinall class JustWrite(threading.Thread): def __init__(self, fs, n): threading.Thread.__init__(self) self.fs = fs self.n = n self.setDaemon(True) def run(self): for _ in range(self.n): file = self.fs.new_file(filename="test") file.write(b("hello")) file.close() class JustRead(threading.Thread): def __init__(self, fs, n, results): threading.Thread.__init__(self) self.fs = fs self.n = n self.results = results self.setDaemon(True) def run(self): for _ in range(self.n): file = self.fs.get("test") data = file.read() self.results.append(data) assert data == b("hello") class TestGridfs(unittest.TestCase): def setUp(self): self.db = get_client().pymongo_test self.db.drop_collection("fs.files") self.db.drop_collection("fs.chunks") self.db.drop_collection("alt.files") self.db.drop_collection("alt.chunks") self.fs = gridfs.GridFS(self.db) self.alt = gridfs.GridFS(self.db, "alt") def test_gridfs(self): self.assertRaises(TypeError, gridfs.GridFS, "foo") self.assertRaises(TypeError, gridfs.GridFS, self.db, 5) def test_basic(self): oid = self.fs.put(b("hello world")) self.assertEqual(b("hello world"), self.fs.get(oid).read()) self.assertEqual(1, self.db.fs.files.count()) self.assertEqual(1, self.db.fs.chunks.count()) self.fs.delete(oid) self.assertRaises(NoFile, self.fs.get, oid) self.assertEqual(0, self.db.fs.files.count()) self.assertEqual(0, self.db.fs.chunks.count()) self.assertRaises(NoFile, self.fs.get, "foo") oid = self.fs.put(b("hello world"), _id="foo") self.assertEqual("foo", oid) self.assertEqual(b("hello world"), self.fs.get("foo").read()) def test_list(self): self.assertEqual([], self.fs.list()) self.fs.put(b("hello world")) self.assertEqual([], self.fs.list()) self.fs.put(b(""), filename="mike") self.fs.put(b("foo"), filename="test") self.fs.put(b(""), filename="hello world") self.assertEqual(set(["mike", "test", "hello world"]), set(self.fs.list())) def test_empty_file(self): oid = self.fs.put(b("")) self.assertEqual(b(""), self.fs.get(oid).read()) self.assertEqual(1, self.db.fs.files.count()) self.assertEqual(0, self.db.fs.chunks.count()) raw = self.db.fs.files.find_one() self.assertEqual(0, raw["length"]) self.assertEqual(oid, raw["_id"]) self.assertTrue(isinstance(raw["uploadDate"], datetime.datetime)) self.assertEqual(256 * 1024, raw["chunkSize"]) self.assertTrue(isinstance(raw["md5"], basestring)) def test_alt_collection(self): oid = self.alt.put(b("hello world")) self.assertEqual(b("hello world"), self.alt.get(oid).read()) self.assertEqual(1, self.db.alt.files.count()) self.assertEqual(1, self.db.alt.chunks.count()) self.alt.delete(oid) self.assertRaises(NoFile, self.alt.get, oid) self.assertEqual(0, self.db.alt.files.count()) self.assertEqual(0, self.db.alt.chunks.count()) self.assertRaises(NoFile, self.alt.get, "foo") oid = self.alt.put(b("hello world"), _id="foo") self.assertEqual("foo", oid) self.assertEqual(b("hello world"), self.alt.get("foo").read()) self.alt.put(b(""), filename="mike") self.alt.put(b("foo"), filename="test") self.alt.put(b(""), filename="hello world") self.assertEqual(set(["mike", "test", "hello world"]), set(self.alt.list())) def test_threaded_reads(self): self.fs.put(b("hello"), _id="test") threads = [] results = [] for i in range(10): threads.append(JustRead(self.fs, 10, results)) threads[i].start() joinall(threads) self.assertEqual( 100 * [b('hello')], results ) def test_threaded_writes(self): threads = [] for i in range(10): threads.append(JustWrite(self.fs, 10)) threads[i].start() joinall(threads) f = self.fs.get_last_version("test") self.assertEqual(f.read(), b("hello")) # Should have created 100 versions of 'test' file self.assertEqual( 100, self.db.fs.files.find({'filename':'test'}).count() ) def test_get_last_version(self): one = self.fs.put(b("foo"), filename="test") time.sleep(0.01) two = self.fs.new_file(filename="test") two.write(b("bar")) two.close() time.sleep(0.01) two = two._id three = self.fs.put(b("baz"), filename="test") self.assertEqual(b("baz"), self.fs.get_last_version("test").read()) self.fs.delete(three) self.assertEqual(b("bar"), self.fs.get_last_version("test").read()) self.fs.delete(two) self.assertEqual(b("foo"), self.fs.get_last_version("test").read()) self.fs.delete(one) self.assertRaises(NoFile, self.fs.get_last_version, "test") def test_get_last_version_with_metadata(self): one = self.fs.put(b("foo"), filename="test", author="author") time.sleep(0.01) two = self.fs.put(b("bar"), filename="test", author="author") self.assertEqual(b("bar"), self.fs.get_last_version(author="author").read()) self.fs.delete(two) self.assertEqual(b("foo"), self.fs.get_last_version(author="author").read()) self.fs.delete(one) one = self.fs.put(b("foo"), filename="test", author="author1") time.sleep(0.01) two = self.fs.put(b("bar"), filename="test", author="author2") self.assertEqual(b("foo"), self.fs.get_last_version(author="author1").read()) self.assertEqual(b("bar"), self.fs.get_last_version(author="author2").read()) self.assertEqual(b("bar"), self.fs.get_last_version(filename="test").read()) self.assertRaises(NoFile, self.fs.get_last_version, author="author3") self.assertRaises(NoFile, self.fs.get_last_version, filename="nottest", author="author1") self.fs.delete(one) self.fs.delete(two) def test_get_version(self): self.fs.put(b("foo"), filename="test") time.sleep(0.01) self.fs.put(b("bar"), filename="test") time.sleep(0.01) self.fs.put(b("baz"), filename="test") time.sleep(0.01) self.assertEqual(b("foo"), self.fs.get_version("test", 0).read()) self.assertEqual(b("bar"), self.fs.get_version("test", 1).read()) self.assertEqual(b("baz"), self.fs.get_version("test", 2).read()) self.assertEqual(b("baz"), self.fs.get_version("test", -1).read()) self.assertEqual(b("bar"), self.fs.get_version("test", -2).read()) self.assertEqual(b("foo"), self.fs.get_version("test", -3).read()) self.assertRaises(NoFile, self.fs.get_version, "test", 3) self.assertRaises(NoFile, self.fs.get_version, "test", -4) def test_get_version_with_metadata(self): one = self.fs.put(b("foo"), filename="test", author="author1") time.sleep(0.01) two = self.fs.put(b("bar"), filename="test", author="author1") time.sleep(0.01) three = self.fs.put(b("baz"), filename="test", author="author2") self.assertEqual(b("foo"), self.fs.get_version(filename="test", author="author1", version=-2).read()) self.assertEqual(b("bar"), self.fs.get_version(filename="test", author="author1", version=-1).read()) self.assertEqual(b("foo"), self.fs.get_version(filename="test", author="author1", version=0).read()) self.assertEqual(b("bar"), self.fs.get_version(filename="test", author="author1", version=1).read()) self.assertEqual(b("baz"), self.fs.get_version(filename="test", author="author2", version=0).read()) self.assertEqual(b("baz"), self.fs.get_version(filename="test", version=-1).read()) self.assertEqual(b("baz"), self.fs.get_version(filename="test", version=2).read()) self.assertRaises(NoFile, self.fs.get_version, filename="test", author="author3") self.assertRaises(NoFile, self.fs.get_version, filename="test", author="author1", version=2) self.fs.delete(one) self.fs.delete(two) self.fs.delete(three) def test_put_filelike(self): oid = self.fs.put(StringIO(b("hello world")), chunk_size=1) self.assertEqual(11, self.db.fs.chunks.count()) self.assertEqual(b("hello world"), self.fs.get(oid).read()) def test_file_exists(self): db = get_client(w=1).pymongo_test fs = gridfs.GridFS(db) oid = fs.put(b("hello")) self.assertRaises(FileExists, fs.put, b("world"), _id=oid) one = fs.new_file(_id=123) one.write(b("some content")) one.close() two = fs.new_file(_id=123) self.assertRaises(FileExists, two.write, b('x' * 262146)) def test_exists(self): oid = self.fs.put(b("hello")) self.assertTrue(self.fs.exists(oid)) self.assertTrue(self.fs.exists({"_id": oid})) self.assertTrue(self.fs.exists(_id=oid)) self.assertFalse(self.fs.exists(filename="mike")) self.assertFalse(self.fs.exists("mike")) oid = self.fs.put(b("hello"), filename="mike", foo=12) self.assertTrue(self.fs.exists(oid)) self.assertTrue(self.fs.exists({"_id": oid})) self.assertTrue(self.fs.exists(_id=oid)) self.assertTrue(self.fs.exists(filename="mike")) self.assertTrue(self.fs.exists({"filename": "mike"})) self.assertTrue(self.fs.exists(foo=12)) self.assertTrue(self.fs.exists({"foo": 12})) self.assertTrue(self.fs.exists(foo={"$gt": 11})) self.assertTrue(self.fs.exists({"foo": {"$gt": 11}})) self.assertFalse(self.fs.exists(foo=13)) self.assertFalse(self.fs.exists({"foo": 13})) self.assertFalse(self.fs.exists(foo={"$gt": 12})) self.assertFalse(self.fs.exists({"foo": {"$gt": 12}})) def test_put_unicode(self): self.assertRaises(TypeError, self.fs.put, u"hello") oid = self.fs.put(u"hello", encoding="utf-8") self.assertEqual(b("hello"), self.fs.get(oid).read()) self.assertEqual("utf-8", self.fs.get(oid).encoding) oid = self.fs.put(u"aé", encoding="iso-8859-1") self.assertEqual(u"aé".encode("iso-8859-1"), self.fs.get(oid).read()) self.assertEqual("iso-8859-1", self.fs.get(oid).encoding) def test_missing_length_iter(self): # Test fix that guards against PHP-237 self.fs.put(b(""), filename="empty") doc = self.db.fs.files.find_one({"filename": "empty"}) doc.pop("length") self.db.fs.files.save(doc) f = self.fs.get_last_version(filename="empty") def iterate_file(grid_file): for chunk in grid_file: pass return True self.assertTrue(iterate_file(f)) def test_request(self): c = self.db.connection c.start_request() n = 5 for i in range(n): file = self.fs.new_file(filename="test") file.write(b("hello")) file.close() c.end_request() self.assertEqual( n, self.db.fs.files.find({'filename':'test'}).count() ) def test_gridfs_request(self): self.assertFalse(self.db.connection.in_request()) self.fs.put(b("hello world")) # Request started and ended by put(), we're back to original state self.assertFalse(self.db.connection.in_request()) class TestGridfsReplicaSet(TestReplicaSetClientBase): def test_gridfs_replica_set(self): rsc = self._get_client( w=self.w, wtimeout=5000, read_preference=ReadPreference.SECONDARY) try: fs = gridfs.GridFS(rsc.pymongo_test) oid = fs.put(b('foo')) content = fs.get(oid).read() self.assertEqual(b('foo'), content) finally: rsc.close() def test_gridfs_secondary(self): primary_host, primary_port = self.primary primary_connection = MongoClient(primary_host, primary_port) secondary_host, secondary_port = self.secondaries[0] for secondary_connection in [ MongoClient(secondary_host, secondary_port, slave_okay=True), MongoClient(secondary_host, secondary_port, read_preference=ReadPreference.SECONDARY), ]: primary_connection.pymongo_test.drop_collection("fs.files") primary_connection.pymongo_test.drop_collection("fs.chunks") # Should detect it's connected to secondary and not attempt to # create index fs = gridfs.GridFS(secondary_connection.pymongo_test) # This won't detect secondary, raises error self.assertRaises(AutoReconnect, fs.put, b('foo')) def tearDown(self): rsc = self._get_client() rsc.pymongo_test.drop_collection('fs.files') rsc.pymongo_test.drop_collection('fs.chunks') rsc.close() if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_json_util.py000066400000000000000000000137371223300253600176170ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test some utilities for working with JSON and PyMongo.""" import unittest import datetime import re import sys from nose.plugins.skip import SkipTest sys.path[0:0] = [""] import bson from bson.py3compat import b from bson import json_util from bson.binary import Binary, MD5_SUBTYPE, USER_DEFINED_SUBTYPE from bson.code import Code from bson.dbref import DBRef from bson.max_key import MaxKey from bson.min_key import MinKey from bson.objectid import ObjectId from bson.timestamp import Timestamp from bson.tz_util import utc from test.test_client import get_client PY3 = sys.version_info[0] == 3 class TestJsonUtil(unittest.TestCase): def setUp(self): if not json_util.json_lib: raise SkipTest("No json or simplejson module") self.db = get_client().pymongo_test def round_tripped(self, doc): return json_util.loads(json_util.dumps(doc)) def round_trip(self, doc): self.assertEqual(doc, self.round_tripped(doc)) def test_basic(self): self.round_trip({"hello": "world"}) def test_objectid(self): self.round_trip({"id": ObjectId()}) def test_dbref(self): self.round_trip({"ref": DBRef("foo", 5)}) self.round_trip({"ref": DBRef("foo", 5, "db")}) self.round_trip({"ref": DBRef("foo", ObjectId())}) self.round_trip({"ref": DBRef("foo", ObjectId(), "db")}) def test_datetime(self): # only millis, not micros self.round_trip({"date": datetime.datetime(2009, 12, 9, 15, 49, 45, 191000, utc)}) def test_regex(self): res = self.round_tripped({"r": re.compile("a*b", re.IGNORECASE)})["r"] self.assertEqual("a*b", res.pattern) if PY3: # re.UNICODE is a default in python 3. self.assertEqual(re.IGNORECASE | re.UNICODE, res.flags) else: self.assertEqual(re.IGNORECASE, res.flags) all_options = re.I|re.L|re.M|re.S|re.U|re.X regex = re.compile("a*b", all_options) res = self.round_tripped({"r": regex})["r"] self.assertEqual(all_options, res.flags) # Some tools may not add $options if no flags are set. res = json_util.loads('{"r": {"$regex": "a*b"}}')['r'] expected_flags = 0 if PY3: expected_flags = re.U self.assertEqual(expected_flags, res.flags) def test_minkey(self): self.round_trip({"m": MinKey()}) def test_maxkey(self): self.round_trip({"m": MaxKey()}) def test_timestamp(self): res = json_util.json.dumps({"ts": Timestamp(4, 13)}, default=json_util.default) dct = json_util.json.loads(res) self.assertEqual(dct['ts']['t'], 4) self.assertEqual(dct['ts']['i'], 13) def test_uuid(self): if not bson.has_uuid(): raise SkipTest("No uuid module") self.round_trip( {'uuid': bson.uuid.UUID( 'f47ac10b-58cc-4372-a567-0e02b2c3d479')}) def test_binary(self): bin_type_dict = {"bin": Binary(b("\x00\x01\x02\x03\x04"))} md5_type_dict = { "md5": Binary(b(' n7\x18\xaf\t/\xd1\xd1/\x80\xca\xe7q\xcc\xac'), MD5_SUBTYPE)} custom_type_dict = {"custom": Binary(b("hello"), USER_DEFINED_SUBTYPE)} self.round_trip(bin_type_dict) self.round_trip(md5_type_dict) self.round_trip(custom_type_dict) # PYTHON-443 ensure old type formats are supported json_bin_dump = json_util.dumps(bin_type_dict) self.assertTrue('"$type": "00"' in json_bin_dump) self.assertEqual(bin_type_dict, json_util.loads('{"bin": {"$type": 0, "$binary": "AAECAwQ="}}')) json_bin_dump = json_util.dumps(md5_type_dict) self.assertTrue('"$type": "05"' in json_bin_dump) self.assertEqual(md5_type_dict, json_util.loads('{"md5": {"$type": 5, "$binary":' ' "IG43GK8JL9HRL4DK53HMrA=="}}')) json_bin_dump = json_util.dumps(custom_type_dict) self.assertTrue('"$type": "80"' in json_bin_dump) self.assertEqual(custom_type_dict, json_util.loads('{"custom": {"$type": 128, "$binary":' ' "aGVsbG8="}}')) # Handle mongoexport where subtype >= 128 self.assertEqual(128, json_util.loads('{"custom": {"$type": "ffffff80", "$binary":' ' "aGVsbG8="}}')['custom'].subtype) self.assertEqual(255, json_util.loads('{"custom": {"$type": "ffffffff", "$binary":' ' "aGVsbG8="}}')['custom'].subtype) def test_code(self): self.round_trip({"code": Code("function x() { return 1; }")}) self.round_trip({"code": Code("function y() { return z; }", z=2)}) def test_cursor(self): db = self.db db.drop_collection("test") docs = [ {'foo': [1, 2]}, {'bar': {'hello': 'world'}}, {'code': Code("function x() { return 1; }")}, {'bin': Binary(b("\x00\x01\x02\x03\x04"))}, {'dbref': {'_ref': DBRef('simple', ObjectId('509b8db456c02c5ab7e63c34'))}} ] db.test.insert(docs) reloaded_docs = json_util.loads(json_util.dumps(db.test.find())) for doc in docs: self.assertTrue(doc in reloaded_docs) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_legacy_connections.py000066400000000000000000000101071223300253600214430ustar00rootroot00000000000000# Copyright 2013 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test deprecated client classes Connection and ReplicaSetConnection.""" import sys import unittest sys.path[0:0] = [""] from bson import ObjectId import pymongo from pymongo.connection import Connection from pymongo.replica_set_connection import ReplicaSetConnection from pymongo.errors import ConfigurationError from test import host, port, pair from test.test_replica_set_client import TestReplicaSetClientBase from test.utils import get_pool class TestConnection(unittest.TestCase): def test_connection(self): c = Connection(host, port) self.assertTrue(c.auto_start_request) self.assertEqual(None, c.max_pool_size) self.assertFalse(c.slave_okay) self.assertFalse(c.safe) self.assertEqual({}, c.get_lasterror_options()) # Connection's writes are unacknowledged by default doc = {"_id": ObjectId()} coll = c.pymongo_test.write_concern_test coll.drop() coll.insert(doc) coll.insert(doc) c = Connection("mongodb://%s:%s/?safe=true" % (host, port)) self.assertTrue(c.safe) # To preserve legacy Connection's behavior, max_size should be None. # Pool should handle this without error. self.assertEqual(None, c._MongoClient__pool.max_size) c.end_request() # Connection's network_timeout argument is translated into # socketTimeoutMS self.assertEqual(123, Connection( host, port, network_timeout=123)._MongoClient__net_timeout) for network_timeout in 'foo', 0, -1: self.assertRaises( ConfigurationError, Connection, host, port, network_timeout=network_timeout) def test_connection_alias(self): # Testing that pymongo module imports connection.Connection self.assertEqual(Connection, pymongo.Connection) class TestReplicaSetConnection(TestReplicaSetClientBase): def test_replica_set_connection(self): c = ReplicaSetConnection(pair, replicaSet=self.name) self.assertTrue(c.auto_start_request) self.assertEqual(None, c.max_pool_size) self.assertFalse(c.slave_okay) self.assertFalse(c.safe) self.assertEqual({}, c.get_lasterror_options()) # ReplicaSetConnection's writes are unacknowledged by default doc = {"_id": ObjectId()} coll = c.pymongo_test.write_concern_test coll.drop() coll.insert(doc) coll.insert(doc) c = ReplicaSetConnection("mongodb://%s:%s/?replicaSet=%s&safe=true" % ( host, port, self.name)) self.assertTrue(c.safe) # To preserve legacy ReplicaSetConnection's behavior, max_size should # be None. Pool should handle this without error. pool = get_pool(c) self.assertEqual(None, pool.max_size) c.end_request() # ReplicaSetConnection's network_timeout argument is translated into # socketTimeoutMS self.assertEqual(123, ReplicaSetConnection( pair, replicaSet=self.name, network_timeout=123 )._MongoReplicaSetClient__net_timeout) for network_timeout in 'foo', 0, -1: self.assertRaises( ConfigurationError, ReplicaSetConnection, pair, replicaSet=self.name, network_timeout=network_timeout) def test_replica_set_connection_alias(self): # Testing that pymongo module imports ReplicaSetConnection self.assertEqual(ReplicaSetConnection, pymongo.ReplicaSetConnection) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_master_slave_connection.py000066400000000000000000000441751223300253600225150ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test for master slave connections.""" import datetime import os import sys import threading import time import unittest sys.path[0:0] = [""] from nose.plugins.skip import SkipTest from bson.son import SON from bson.tz_util import utc from pymongo import ReadPreference, thread_util from pymongo.errors import ConnectionFailure, InvalidName from pymongo.errors import CollectionInvalid, OperationFailure from pymongo.errors import AutoReconnect from pymongo.database import Database from pymongo.mongo_client import MongoClient from pymongo.collection import Collection from pymongo.master_slave_connection import MasterSlaveConnection from test import host, port, host2, port2, host3, port3 from test.utils import TestRequestMixin class TestMasterSlaveConnection(unittest.TestCase, TestRequestMixin): def setUp(self): self.master = MongoClient(host, port) self.slaves = [] try: self.slaves.append(MongoClient( host2, port2, read_preference=ReadPreference.SECONDARY)) except ConnectionFailure: pass try: self.slaves.append(MongoClient( host3, port3, read_preference=ReadPreference.SECONDARY)) except ConnectionFailure: pass if not self.slaves: raise SkipTest("Not connected to master-slave set") self.client = MasterSlaveConnection(self.master, self.slaves) self.db = self.client.pymongo_test def tearDown(self): try: self.db.test.drop_indexes() except Exception: # Tests like test_disconnect can monkey with the client in ways # that make this fail pass super(TestMasterSlaveConnection, self).tearDown() def test_types(self): self.assertRaises(TypeError, MasterSlaveConnection, 1) self.assertRaises(TypeError, MasterSlaveConnection, self.master, 1) self.assertRaises(TypeError, MasterSlaveConnection, self.master, [1]) def test_use_greenlets(self): self.assertFalse(self.client.use_greenlets) if thread_util.have_gevent: master = MongoClient(host, port, use_greenlets=True) slaves = [ MongoClient(slave.host, slave.port, use_greenlets=True) for slave in self.slaves] self.assertTrue( MasterSlaveConnection(master, slaves).use_greenlets) def test_repr(self): self.assertEqual(repr(self.client), "MasterSlaveConnection(%r, %r)" % (self.master, self.slaves)) def test_disconnect(self): class MongoClient(object): def __init__(self): self._disconnects = 0 def disconnect(self): self._disconnects += 1 self.client._MasterSlaveConnection__master = MongoClient() self.client._MasterSlaveConnection__slaves = [MongoClient(), MongoClient()] self.client.disconnect() self.assertEqual(1, self.client._MasterSlaveConnection__master._disconnects) self.assertEqual(1, self.client._MasterSlaveConnection__slaves[0]._disconnects) self.assertEqual(1, self.client._MasterSlaveConnection__slaves[1]._disconnects) def test_continue_until_slave_works(self): class Slave(object): calls = 0 def __init__(self, fail): self._fail = fail def _send_message_with_response(self, *args, **kwargs): Slave.calls += 1 if self._fail: raise AutoReconnect() return (None, 'sent') class NotRandomList(object): last_idx = -1 def __init__(self): self._items = [Slave(True), Slave(True), Slave(False), Slave(True)] def __len__(self): return len(self._items) def __getitem__(self, idx): NotRandomList.last_idx = idx return self._items.pop(0) self.client._MasterSlaveConnection__slaves = NotRandomList() response = self.client._send_message_with_response('message') self.assertEqual((NotRandomList.last_idx, 'sent'), response) self.assertNotEqual(-1, NotRandomList.last_idx) self.assertEqual(3, Slave.calls) def test_raise_autoreconnect_if_all_slaves_fail(self): class Slave(object): calls = 0 def __init__(self, fail): self._fail = fail def _send_message_with_response(self, *args, **kwargs): Slave.calls += 1 if self._fail: raise AutoReconnect() return 'sent' class NotRandomList(object): def __init__(self): self._items = [Slave(True), Slave(True), Slave(True), Slave(True)] def __len__(self): return len(self._items) def __getitem__(self, idx): return self._items.pop(0) self.client._MasterSlaveConnection__slaves = NotRandomList() self.assertRaises(AutoReconnect, self.client._send_message_with_response, 'message') self.assertEqual(4, Slave.calls) def test_get_db(self): def make_db(base, name): return base[name] self.assertRaises(InvalidName, make_db, self.client, "") self.assertRaises(InvalidName, make_db, self.client, "te$t") self.assertRaises(InvalidName, make_db, self.client, "te.t") self.assertRaises(InvalidName, make_db, self.client, "te\\t") self.assertRaises(InvalidName, make_db, self.client, "te/t") self.assertRaises(InvalidName, make_db, self.client, "te st") self.assertTrue(isinstance(self.client.test, Database)) self.assertEqual(self.client.test, self.client["test"]) self.assertEqual(self.client.test, Database(self.client, "test")) def test_database_names(self): self.client.pymongo_test.test.save({"dummy": u"object"}) self.client.pymongo_test_mike.test.save({"dummy": u"object"}) dbs = self.client.database_names() self.assertTrue("pymongo_test" in dbs) self.assertTrue("pymongo_test_mike" in dbs) def test_drop_database(self): self.assertRaises(TypeError, self.client.drop_database, 5) self.assertRaises(TypeError, self.client.drop_database, None) raise SkipTest("This test often fails due to SERVER-2329") self.client.pymongo_test.test.save({"dummy": u"object"}) dbs = self.client.database_names() self.assertTrue("pymongo_test" in dbs) self.client.drop_database("pymongo_test") dbs = self.client.database_names() self.assertTrue("pymongo_test" not in dbs) self.client.pymongo_test.test.save({"dummy": u"object"}) dbs = self.client.database_names() self.assertTrue("pymongo_test" in dbs) self.client.drop_database(self.client.pymongo_test) dbs = self.client.database_names() self.assertTrue("pymongo_test" not in dbs) def test_iteration(self): def iterate(): [a for a in self.client] self.assertRaises(TypeError, iterate) def test_insert_find_one_in_request(self): count = 0 for i in range(100): self.client.start_request() self.db.test.remove({}) self.db.test.insert({"x": i}) try: if i != self.db.test.find_one()["x"]: count += 1 except: count += 1 self.client.end_request() self.assertFalse(count) def test_nested_request(self): client = self.client def assertRequest(in_request): self.assertEqual(in_request, client.in_request()) self.assertEqual(in_request, client.master.in_request()) # MasterSlaveConnection is special, alas - it has no auto_start_request # and it begins *not* in a request. When it's in a request, it sends # all queries to primary. self.assertFalse(client.in_request()) self.assertFalse(client.master.in_request()) # Start and end request client.start_request() assertRequest(True) client.end_request() assertRequest(False) # Double-nesting client.start_request() client.start_request() client.end_request() assertRequest(True) client.end_request() assertRequest(False) def test_request_threads(self): client = self.client # In a request, all ops go through master pool = client.master._MongoClient__pool client.master.end_request() self.assertNotInRequestAndDifferentSock(client, pool) started_request, ended_request = threading.Event(), threading.Event() checked_request = threading.Event() thread_done = [False] # Starting a request in one thread doesn't put the other thread in a # request def f(): self.assertNotInRequestAndDifferentSock(client, pool) client.start_request() self.assertInRequestAndSameSock(client, pool) started_request.set() checked_request.wait() checked_request.clear() self.assertInRequestAndSameSock(client, pool) client.end_request() self.assertNotInRequestAndDifferentSock(client, pool) ended_request.set() checked_request.wait() thread_done[0] = True t = threading.Thread(target=f) t.setDaemon(True) t.start() started_request.wait() self.assertNotInRequestAndDifferentSock(client, pool) checked_request.set() ended_request.wait() self.assertNotInRequestAndDifferentSock(client, pool) checked_request.set() t.join() self.assertNotInRequestAndDifferentSock(client, pool) self.assertTrue(thread_done[0], "Thread didn't complete") # This was failing because commands were being sent to the slaves def test_create_collection(self): self.client.pymongo_test.test.drop() collection = self.db.create_collection('test') self.assertTrue(isinstance(collection, Collection)) self.assertRaises(CollectionInvalid, self.db.create_collection, 'test') # Believe this was failing for the same reason... def test_unique_index(self): self.client.pymongo_test.test.drop() self.db.test.create_index('username', unique=True) self.db.test.save({'username': 'mike'}) self.assertRaises(OperationFailure, self.db.test.save, {'username': 'mike'}) # NOTE this test is non-deterministic, but I expect # some failures unless the db is pulling instantaneously... def test_insert_find_one_with_slaves(self): count = 0 for i in range(100): self.db.test.remove({}) self.db.test.insert({"x": i}) try: if i != self.db.test.find_one()["x"]: count += 1 except: count += 1 self.assertTrue(count) # NOTE this test is non-deterministic, but hopefully we pause long enough # for the slaves to pull... def test_insert_find_one_with_pause(self): count = 0 self.db.test.remove({}) self.db.test.insert({"x": 5586}) time.sleep(11) for _ in range(10): try: if 5586 != self.db.test.find_one()["x"]: count += 1 except: count += 1 self.assertFalse(count) def test_kill_cursor_explicit(self): c = self.client c.slave_okay = True db = c.pymongo_test test = db.master_slave_test_kill_cursor_explicit test.drop() for i in range(20): test.insert({"i": i}, w=1 + len(self.slaves)) st = time.time() while time.time() - st < 120: # Wait for replication -- the 'w' parameter should obviate this # loop but it's not working reliably in Jenkins right now if list(test.find({"i": 19})): break time.sleep(0.5) else: self.fail("Replication timeout, test coll has %s records" % ( len(list(test.find())) )) # Partially evaluate cursor so it's left alive, then kill it cursor = test.find().batch_size(10) self.assertNotEqual( cursor._Cursor__connection_id, -1, "Expected cursor connected to a slave, not master") self.assertTrue(cursor.next()) self.assertNotEqual(0, cursor.cursor_id) cursor_id = cursor.cursor_id # Cursor dead on server - trigger a getMore on the same cursor_id and # check that the server returns an error. cursor2 = cursor.clone() cursor2._Cursor__id = cursor_id if (sys.platform.startswith('java') or 'PyPy' in sys.version): # Explicitly kill cursor. cursor.close() else: # Implicitly kill it in CPython. del cursor self.assertRaises(OperationFailure, lambda: list(cursor2)) def test_base_object(self): c = self.client self.assertFalse(c.slave_okay) self.assertTrue(bool(c.read_preference)) self.assertTrue(c.safe) self.assertEqual({}, c.get_lasterror_options()) db = c.pymongo_test self.assertFalse(db.slave_okay) self.assertTrue(bool(c.read_preference)) self.assertTrue(db.safe) self.assertEqual({}, db.get_lasterror_options()) coll = db.test coll.drop() self.assertFalse(coll.slave_okay) self.assertTrue(bool(c.read_preference)) self.assertTrue(coll.safe) self.assertEqual({}, coll.get_lasterror_options()) cursor = coll.find() self.assertFalse(cursor._Cursor__slave_okay) self.assertTrue(bool(cursor._Cursor__read_preference)) w = 1 + len(self.slaves) wtimeout=10000 # Wait 10 seconds for replication to complete c.set_lasterror_options(w=w, wtimeout=wtimeout) self.assertFalse(c.slave_okay) self.assertTrue(bool(c.read_preference)) self.assertTrue(c.safe) self.assertEqual({'w': w, 'wtimeout': wtimeout}, c.get_lasterror_options()) db = c.pymongo_test self.assertFalse(db.slave_okay) self.assertTrue(bool(c.read_preference)) self.assertTrue(db.safe) self.assertEqual({'w': w, 'wtimeout': wtimeout}, db.get_lasterror_options()) coll = db.test self.assertFalse(coll.slave_okay) self.assertTrue(bool(c.read_preference)) self.assertTrue(coll.safe) self.assertEqual({'w': w, 'wtimeout': wtimeout}, coll.get_lasterror_options()) cursor = coll.find() self.assertFalse(cursor._Cursor__slave_okay) self.assertTrue(bool(cursor._Cursor__read_preference)) coll.insert({'foo': 'bar'}) self.assertEqual(1, coll.find({'foo': 'bar'}).count()) self.assertTrue(coll.find({'foo': 'bar'})) coll.remove({'foo': 'bar'}) self.assertEqual(0, coll.find({'foo': 'bar'}).count()) c.safe = False c.unset_lasterror_options() self.assertFalse(self.client.slave_okay) self.assertTrue(bool(self.client.read_preference)) self.assertFalse(self.client.safe) self.assertEqual({}, self.client.get_lasterror_options()) def test_document_class(self): c = MasterSlaveConnection(self.master, self.slaves) db = c.pymongo_test w = 1 + len(self.slaves) db.test.insert({"x": 1}, w=w) self.assertEqual(dict, c.document_class) self.assertTrue(isinstance(db.test.find_one(), dict)) self.assertFalse(isinstance(db.test.find_one(), SON)) c.document_class = SON self.assertEqual(SON, c.document_class) self.assertTrue(isinstance(db.test.find_one(), SON)) self.assertFalse(isinstance(db.test.find_one(as_class=dict), SON)) c = MasterSlaveConnection(self.master, self.slaves, document_class=SON) db = c.pymongo_test self.assertEqual(SON, c.document_class) self.assertTrue(isinstance(db.test.find_one(), SON)) self.assertFalse(isinstance(db.test.find_one(as_class=dict), SON)) c.document_class = dict self.assertEqual(dict, c.document_class) self.assertTrue(isinstance(db.test.find_one(), dict)) self.assertFalse(isinstance(db.test.find_one(), SON)) def test_tz_aware(self): dt = datetime.datetime.utcnow() client = MasterSlaveConnection(self.master, self.slaves) self.assertEqual(False, client.tz_aware) db = client.pymongo_test w = 1 + len(self.slaves) db.tztest.insert({'dt': dt}, w=w) self.assertEqual(None, db.tztest.find_one()['dt'].tzinfo) client = MasterSlaveConnection(self.master, self.slaves, tz_aware=True) self.assertEqual(True, client.tz_aware) db = client.pymongo_test db.tztest.insert({'dt': dt}, w=w) self.assertEqual(utc, db.tztest.find_one()['dt'].tzinfo) client = MasterSlaveConnection(self.master, self.slaves, tz_aware=False) self.assertEqual(False, client.tz_aware) db = client.pymongo_test db.tztest.insert({'dt': dt}, w=w) self.assertEqual(None, db.tztest.find_one()['dt'].tzinfo) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_objectid.py000066400000000000000000000165571223300253600173770ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tests for the objectid module.""" import datetime import pickle import warnings import unittest import sys import time sys.path[0:0] = [""] from nose.plugins.skip import SkipTest from bson.errors import InvalidId from bson.objectid import ObjectId from bson.py3compat import b, binary_type from bson.tz_util import (FixedOffset, utc) PY3 = sys.version_info[0] == 3 def oid(x): return ObjectId() class TestObjectId(unittest.TestCase): def setUp(self): pass def test_creation(self): self.assertRaises(TypeError, ObjectId, 4) self.assertRaises(TypeError, ObjectId, 175.0) self.assertRaises(TypeError, ObjectId, {"test": 4}) self.assertRaises(TypeError, ObjectId, ["something"]) self.assertRaises(InvalidId, ObjectId, "") self.assertRaises(InvalidId, ObjectId, "12345678901") self.assertRaises(InvalidId, ObjectId, "1234567890123") self.assertTrue(ObjectId()) self.assertTrue(ObjectId(b("123456789012"))) a = ObjectId() self.assertTrue(ObjectId(a)) def test_unicode(self): a = ObjectId() self.assertEqual(a, ObjectId(unicode(a))) self.assertEqual(ObjectId("123456789012123456789012"), ObjectId(u"123456789012123456789012")) self.assertRaises(InvalidId, ObjectId, u"hello") def test_from_hex(self): ObjectId("123456789012123456789012") self.assertRaises(InvalidId, ObjectId, "123456789012123456789G12") self.assertRaises(InvalidId, ObjectId, u"123456789012123456789G12") def test_repr_str(self): self.assertEqual(repr(ObjectId("1234567890abcdef12345678")), "ObjectId('1234567890abcdef12345678')") self.assertEqual(str(ObjectId("1234567890abcdef12345678")), "1234567890abcdef12345678") self.assertEqual(str(ObjectId(b("123456789012"))), "313233343536373839303132") self.assertEqual(ObjectId("1234567890abcdef12345678").binary, b('\x124Vx\x90\xab\xcd\xef\x124Vx')) self.assertEqual(str(ObjectId(b('\x124Vx\x90\xab\xcd\xef\x124Vx'))), "1234567890abcdef12345678") def test_equality(self): a = ObjectId() self.assertEqual(a, ObjectId(a)) self.assertEqual(ObjectId(b("123456789012")), ObjectId(b("123456789012"))) self.assertNotEqual(ObjectId(), ObjectId()) self.assertNotEqual(ObjectId(b("123456789012")), b("123456789012")) # Explicitly test inequality self.assertFalse(a != ObjectId(a)) self.assertFalse(ObjectId(b("123456789012")) != ObjectId(b("123456789012"))) def test_binary_str_equivalence(self): a = ObjectId() self.assertEqual(a, ObjectId(a.binary)) self.assertEqual(a, ObjectId(str(a))) def test_multiprocessing(self): # multiprocessing on windows is weird and I don't feel like figuring it # out right now. this should fix buildbot. if sys.platform == "win32": raise SkipTest("Can't fork on Windows") try: import multiprocessing except ImportError: raise SkipTest("No multiprocessing module") pool = multiprocessing.Pool(2) ids = pool.map(oid, range(20)) pool.close() pool.join() map = {} for id in ids: self.assertTrue(id not in map) map[id] = True def test_generation_time(self): d1 = datetime.datetime.utcnow() d2 = ObjectId().generation_time self.assertEqual(utc, d2.tzinfo) d2 = d2.replace(tzinfo=None) self.assertTrue(d2 - d1 < datetime.timedelta(seconds=2)) def test_from_datetime(self): if 'PyPy 1.8.0' in sys.version: # See https://bugs.pypy.org/issue1092 raise SkipTest("datetime.timedelta is broken in pypy 1.8.0") d = datetime.datetime.utcnow() d = d - datetime.timedelta(microseconds=d.microsecond) oid = ObjectId.from_datetime(d) self.assertEqual(d, oid.generation_time.replace(tzinfo=None)) self.assertEqual("0" * 16, str(oid)[8:]) aware = datetime.datetime(1993, 4, 4, 2, tzinfo=FixedOffset(555, "SomeZone")) as_utc = (aware - aware.utcoffset()).replace(tzinfo=utc) oid = ObjectId.from_datetime(aware) self.assertEqual(as_utc, oid.generation_time) def test_pickling(self): orig = ObjectId() for protocol in [0, 1, 2, -1]: pkl = pickle.dumps(orig, protocol=protocol) self.assertEqual(orig, pickle.loads(pkl)) def test_pickle_backwards_compatability(self): # For a full discussion see http://bugs.python.org/issue6137 if sys.version.startswith('3.0'): raise SkipTest("Python 3.0.x can't unpickle " "objects pickled in Python 2.x.") # This string was generated by pickling an ObjectId in pymongo # version 1.9 pickled_with_1_9 = b( "ccopy_reg\n_reconstructor\np0\n" "(cbson.objectid\nObjectId\np1\nc__builtin__\n" "object\np2\nNtp3\nRp4\n" "(dp5\nS'_ObjectId__id'\np6\n" "S'M\\x9afV\\x13v\\xc0\\x0b\\x88\\x00\\x00\\x00'\np7\nsb.") # We also test against a hardcoded "New" pickle format so that we # make sure we're backward compatible with the current version in # the future as well. pickled_with_1_10 = b( "ccopy_reg\n_reconstructor\np0\n" "(cbson.objectid\nObjectId\np1\nc__builtin__\n" "object\np2\nNtp3\nRp4\n" "S'M\\x9afV\\x13v\\xc0\\x0b\\x88\\x00\\x00\\x00'\np5\nb." ) if PY3: # Have to load using 'latin-1' since these were pickled in python2.x. oid_1_9 = pickle.loads(pickled_with_1_9, encoding='latin-1') oid_1_10 = pickle.loads(pickled_with_1_10, encoding='latin-1') else: oid_1_9 = pickle.loads(pickled_with_1_9) oid_1_10 = pickle.loads(pickled_with_1_10) self.assertEqual(oid_1_9, ObjectId("4d9a66561376c00b88000000")) self.assertEqual(oid_1_9, oid_1_10) def test_is_valid(self): self.assertFalse(ObjectId.is_valid(4)) self.assertFalse(ObjectId.is_valid(175.0)) self.assertFalse(ObjectId.is_valid({"test": 4})) self.assertFalse(ObjectId.is_valid(["something"])) self.assertFalse(ObjectId.is_valid("")) self.assertFalse(ObjectId.is_valid("12345678901")) self.assertFalse(ObjectId.is_valid("1234567890123")) self.assertTrue(ObjectId.is_valid(b("123456789012"))) self.assertTrue(ObjectId.is_valid("123456789012123456789012")) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_pooling.py000066400000000000000000000127431223300253600172540ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test built in connection-pooling with threads.""" import sys import thread import time import unittest sys.path[0:0] = [""] from nose.plugins.skip import SkipTest from test import host, port from test.test_pooling_base import ( _TestPooling, _TestMaxPoolSize, _TestMaxOpenSockets, _TestPoolSocketSharing, _TestWaitQueueMultiple, one) class TestPoolingThreads(_TestPooling, unittest.TestCase): use_greenlets = False def test_request_with_fork(self): if sys.platform == "win32": raise SkipTest("Can't test forking on Windows") try: from multiprocessing import Process, Pipe except ImportError: raise SkipTest("No multiprocessing module") coll = self.c.pymongo_test.test coll.remove() coll.insert({'_id': 1}) coll.find_one() self.assert_pool_size(1) self.c.start_request() self.assert_pool_size(1) coll.find_one() self.assert_pool_size(0) self.assert_request_with_socket() def f(pipe): # We can still query server without error self.assertEqual({'_id':1}, coll.find_one()) # Pool has detected that we forked, but resumed the request self.assert_request_with_socket() self.assert_pool_size(0) pipe.send("success") parent_conn, child_conn = Pipe() p = Process(target=f, args=(child_conn,)) p.start() p.join(1) p.terminate() child_conn.close() self.assertEqual("success", parent_conn.recv()) def test_primitive_thread(self): p = self.get_pool((host, port), 10, None, None, False) # Test that start/end_request work with a thread begun from thread # module, rather than threading module lock = thread.allocate_lock() lock.acquire() sock_ids = [] def run_in_request(): p.start_request() sock0 = p.get_socket() sock1 = p.get_socket() sock_ids.extend([id(sock0), id(sock1)]) p.maybe_return_socket(sock0) p.maybe_return_socket(sock1) p.end_request() lock.release() thread.start_new_thread(run_in_request, ()) # Join thread acquired = False for i in range(30): time.sleep(0.5) acquired = lock.acquire(0) if acquired: break self.assertTrue(acquired, "Thread is hung") self.assertEqual(sock_ids[0], sock_ids[1]) def test_pool_with_fork(self): # Test that separate MongoClients have separate Pools, and that the # driver can create a new MongoClient after forking if sys.platform == "win32": raise SkipTest("Can't test forking on Windows") try: from multiprocessing import Process, Pipe except ImportError: raise SkipTest("No multiprocessing module") a = self.get_client(auto_start_request=False) a.pymongo_test.test.remove() a.pymongo_test.test.insert({'_id':1}) a.pymongo_test.test.find_one() self.assertEqual(1, len(a._MongoClient__pool.sockets)) a_sock = one(a._MongoClient__pool.sockets) def loop(pipe): c = self.get_client(auto_start_request=False) self.assertEqual(1,len(c._MongoClient__pool.sockets)) c.pymongo_test.test.find_one() self.assertEqual(1,len(c._MongoClient__pool.sockets)) pipe.send(one(c._MongoClient__pool.sockets).sock.getsockname()) cp1, cc1 = Pipe() cp2, cc2 = Pipe() p1 = Process(target=loop, args=(cc1,)) p2 = Process(target=loop, args=(cc2,)) p1.start() p2.start() p1.join(1) p2.join(1) p1.terminate() p2.terminate() p1.join() p2.join() cc1.close() cc2.close() b_sock = cp1.recv() c_sock = cp2.recv() self.assertTrue(a_sock.sock.getsockname() != b_sock) self.assertTrue(a_sock.sock.getsockname() != c_sock) self.assertTrue(b_sock != c_sock) # a_sock, created by parent process, is still in the pool d_sock = a._MongoClient__pool.get_socket((a.host, a.port)) self.assertEqual(a_sock, d_sock) d_sock.close() class TestMaxPoolSizeThreads(_TestMaxPoolSize, unittest.TestCase): use_greenlets = False def test_max_pool_size_with_leaked_request_massive(self): nthreads = 50 self._test_max_pool_size( 2, 1, max_pool_size=2 * nthreads, nthreads=nthreads) class TestPoolSocketSharingThreads(_TestPoolSocketSharing, unittest.TestCase): use_greenlets = False class TestMaxOpenSocketsThreads(_TestMaxOpenSockets, unittest.TestCase): use_greenlets = False class TestWaitQueueMultipleThreads(_TestWaitQueueMultiple, unittest.TestCase): use_greenlets = False if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_pooling_base.py000066400000000000000000001222341223300253600202430ustar00rootroot00000000000000# Copyright 2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Base classes to test built-in connection-pooling with threads or greenlets. """ import gc import random import socket import sys import thread import threading import time sys.path[0:0] = [""] from nose.plugins.skip import SkipTest import pymongo.pool from pymongo.mongo_client import MongoClient from pymongo.pool import Pool, NO_REQUEST, NO_SOCKET_YET, SocketInfo from pymongo.errors import ConfigurationError, ConnectionFailure from pymongo.errors import ExceededMaxWaiters from test import version, host, port from test.test_client import get_client from test.utils import delay, is_mongos, one N = 50 DB = "pymongo-pooling-tests" if sys.version_info[0] >= 3: from imp import reload try: import gevent from gevent import Greenlet, monkey, hub import gevent.coros, gevent.event has_gevent = True except ImportError: has_gevent = False def gc_collect_until_done(threads, timeout=60): start = time.time() running = list(threads) while running: assert (time.time() - start) < timeout, "Threads timed out" for t in running: t.thread.join(0.1) if not t.alive: running.remove(t) gc.collect() class MongoThread(object): """A thread, or a greenlet, that uses a MongoClient""" def __init__(self, test_case): self.use_greenlets = test_case.use_greenlets self.client = test_case.c self.db = self.client[DB] self.ut = test_case self.passed = False def start(self): if self.use_greenlets: # A Gevent extended Greenlet self.thread = Greenlet(self.run) else: self.thread = threading.Thread(target=self.run) self.thread.setDaemon(True) # Don't hang whole test if thread hangs self.thread.start() @property def alive(self): if self.use_greenlets: return not self.thread.dead else: return self.thread.isAlive() def join(self): self.thread.join(10) if self.use_greenlets: msg = "Greenlet timeout" else: msg = "Thread timeout" assert not self.alive, msg self.thread = None def run(self): self.run_mongo_thread() # No exceptions thrown self.passed = True def run_mongo_thread(self): raise NotImplementedError() class SaveAndFind(MongoThread): def run_mongo_thread(self): for _ in xrange(N): rand = random.randint(0, N) _id = self.db.sf.save({"x": rand}) self.ut.assertEqual(rand, self.db.sf.find_one(_id)["x"]) class Unique(MongoThread): def run_mongo_thread(self): for _ in xrange(N): self.client.start_request() self.db.unique.insert({}) # no error self.client.end_request() class NonUnique(MongoThread): def run_mongo_thread(self): for _ in xrange(N): self.client.start_request() self.db.unique.insert({"_id": "jesse"}, w=0) self.ut.assertNotEqual(None, self.db.error()) self.client.end_request() class Disconnect(MongoThread): def run_mongo_thread(self): for _ in xrange(N): self.client.disconnect() class NoRequest(MongoThread): def run_mongo_thread(self): self.client.start_request() errors = 0 for _ in xrange(N): self.db.unique.insert({"_id": "jesse"}, w=0) if not self.db.error(): errors += 1 self.client.end_request() self.ut.assertEqual(0, errors) def run_cases(ut, cases): threads = [] nruns = 10 if ( ut.use_greenlets and sys.platform == 'darwin' and gevent.version_info[0] < 1 ): # Gevent 0.13.6 bug on Mac, Greenlet.join() hangs if more than # about 35 Greenlets share a MongoClient. Apparently fixed in # recent Gevent development. nruns = 5 for case in cases: for i in range(nruns): t = case(ut) t.start() threads.append(t) for t in threads: t.join() for t in threads: assert t.passed, "%s.run_mongo_thread() threw an exception" % repr(t) class OneOp(MongoThread): def __init__(self, ut): super(OneOp, self).__init__(ut) def run_mongo_thread(self): pool = self.client._MongoClient__pool assert len(pool.sockets) == 1, "Expected 1 socket, found %d" % ( len(pool.sockets) ) sock_info = one(pool.sockets) self.client.start_request() # start_request() hasn't yet moved the socket from the general pool into # the request assert len(pool.sockets) == 1 assert one(pool.sockets) == sock_info self.client[DB].test.find_one() # find_one() causes the socket to be used in the request, so now it's # bound to this thread assert len(pool.sockets) == 0 assert pool._get_request_state() == sock_info self.client.end_request() # The socket is back in the pool assert len(pool.sockets) == 1 assert one(pool.sockets) == sock_info class CreateAndReleaseSocket(MongoThread): """A thread or greenlet that acquires a socket, waits for all other threads to reach rendezvous point, then terminates. """ class Rendezvous(object): def __init__(self, nthreads, use_greenlets): self.nthreads = nthreads self.nthreads_run = 0 self.use_greenlets = use_greenlets if use_greenlets: self.lock = gevent.coros.RLock() else: self.lock = threading.Lock() self.reset_ready() def reset_ready(self): if self.use_greenlets: self.ready = gevent.event.Event() else: self.ready = threading.Event() def __init__(self, ut, client, start_request, end_request, rendezvous): super(CreateAndReleaseSocket, self).__init__(ut) self.client = client self.start_request = start_request self.end_request = end_request self.rendezvous = rendezvous def run_mongo_thread(self): # Do an operation that requires a socket. # test_max_pool_size uses this to spin up lots of threads requiring # lots of simultaneous connections, to ensure that Pool obeys its # max_size configuration and closes extra sockets as they're returned. for i in range(self.start_request): self.client.start_request() # Use a socket self.client[DB].test.find_one() # Don't finish until all threads reach this point r = self.rendezvous r.lock.acquire() r.nthreads_run += 1 if r.nthreads_run == r.nthreads: # Everyone's here, let them finish r.ready.set() r.lock.release() else: r.lock.release() r.ready.wait(2) # Wait two seconds assert r.ready.isSet(), "Rendezvous timed out" for i in range(self.end_request): self.client.end_request() class CreateAndReleaseSocketNoRendezvous(MongoThread): """A thread or greenlet that acquires a socket and terminates without waiting for other threads to reach rendezvous point. """ class Rendezvous(object): def __init__(self, nthreads, use_greenlets): self.nthreads = nthreads self.nthreads_run = 0 if use_greenlets: self.lock = gevent.coros.RLock() self.ready = gevent.event.Event() else: self.lock = threading.Lock() self.ready = threading.Event() def __init__(self, ut, client, start_request, end_request): super(CreateAndReleaseSocketNoRendezvous, self).__init__(ut) self.client = client self.start_request = start_request self.end_request = end_request def run_mongo_thread(self): # Do an operation that requires a socket. # test_max_pool_size uses this to spin up lots of threads requiring # lots of simultaneous connections, to ensure that Pool obeys its # max_size configuration and closes extra sockets as they're returned. for i in range(self.start_request): self.client.start_request() # Use a socket self.client[DB].test.find_one() for i in range(self.end_request): self.client.end_request() class _TestPoolingBase(object): """Base class for all client-pool tests. Doesn't inherit from unittest.TestCase, and its name is prefixed with "_" to avoid being run by nose. Real tests double-inherit from this base and from TestCase. """ use_greenlets = False def setUp(self): if self.use_greenlets: if not has_gevent: raise SkipTest("Gevent not installed") # Note we don't do patch_thread() or patch_all() - we're # testing here that patch_thread() is unnecessary for # the client pool to work properly. monkey.patch_socket() self.c = self.get_client(auto_start_request=False) # reset the db db = self.c[DB] db.unique.drop() db.test.drop() db.unique.insert({"_id": "jesse"}) db.test.insert([{} for i in range(10)]) def tearDown(self): self.c.close() if self.use_greenlets: # Undo patch reload(socket) def get_client(self, *args, **kwargs): opts = kwargs.copy() opts['use_greenlets'] = self.use_greenlets return get_client(*args, **opts) def get_pool(self, *args, **kwargs): kwargs['use_greenlets'] = self.use_greenlets return Pool(*args, **kwargs) def sleep(self, seconds): if self.use_greenlets: gevent.sleep(seconds) else: time.sleep(seconds) def assert_no_request(self): self.assertEqual( NO_REQUEST, self.c._MongoClient__pool._get_request_state() ) def assert_request_without_socket(self): self.assertEqual( NO_SOCKET_YET, self.c._MongoClient__pool._get_request_state() ) def assert_request_with_socket(self): self.assertTrue(isinstance( self.c._MongoClient__pool._get_request_state(), SocketInfo )) def assert_pool_size(self, pool_size): self.assertEqual( pool_size, len(self.c._MongoClient__pool.sockets) ) class _TestPooling(_TestPoolingBase): """Basic pool tests, to be run both with threads and with greenlets.""" def test_max_pool_size_validation(self): self.assertRaises( ConfigurationError, MongoClient, host=host, port=port, max_pool_size=-1 ) self.assertRaises( ConfigurationError, MongoClient, host=host, port=port, max_pool_size='foo' ) c = MongoClient(host=host, port=port, max_pool_size=100) self.assertEqual(c.max_pool_size, 100) def test_no_disconnect(self): run_cases(self, [NoRequest, NonUnique, Unique, SaveAndFind]) def test_simple_disconnect(self): # MongoClient just created, expect 1 free socket self.assert_pool_size(1) self.assert_no_request() self.c.start_request() self.assert_request_without_socket() cursor = self.c[DB].stuff.find() # Cursor hasn't actually caused a request yet, so there's still 1 free # socket. self.assert_pool_size(1) self.assert_request_without_socket() # Actually make a request to server, triggering a socket to be # allocated to the request list(cursor) self.assert_pool_size(0) self.assert_request_with_socket() # Pool returns to its original state self.c.end_request() self.assert_no_request() self.assert_pool_size(1) self.c.disconnect() self.assert_pool_size(0) self.assert_no_request() def test_disconnect(self): run_cases(self, [SaveAndFind, Disconnect, Unique]) def test_independent_pools(self): # Test for regression of very early PyMongo bug: separate pools shared # state. p = self.get_pool((host, port), 10, None, None, False) self.c.start_request() self.c.pymongo_test.test.find_one() self.assertEqual(set(), p.sockets) self.c.end_request() self.assert_pool_size(1) self.assertEqual(set(), p.sockets) def test_dependent_pools(self): self.assert_pool_size(1) self.c.start_request() self.assert_request_without_socket() self.c.pymongo_test.test.find_one() self.assert_request_with_socket() self.assert_pool_size(0) self.c.end_request() self.assert_pool_size(1) t = OneOp(self) t.start() t.join() self.assertTrue(t.passed, "OneOp.run() threw exception") self.assert_pool_size(1) self.c.pymongo_test.test.find_one() self.assert_pool_size(1) def test_multiple_connections(self): a = self.get_client(auto_start_request=False) b = self.get_client(auto_start_request=False) self.assertEqual(1, len(a._MongoClient__pool.sockets)) self.assertEqual(1, len(b._MongoClient__pool.sockets)) a.start_request() a.pymongo_test.test.find_one() self.assertEqual(0, len(a._MongoClient__pool.sockets)) a.end_request() self.assertEqual(1, len(a._MongoClient__pool.sockets)) self.assertEqual(1, len(b._MongoClient__pool.sockets)) a_sock = one(a._MongoClient__pool.sockets) b.end_request() self.assertEqual(1, len(a._MongoClient__pool.sockets)) self.assertEqual(1, len(b._MongoClient__pool.sockets)) b.start_request() b.pymongo_test.test.find_one() self.assertEqual(1, len(a._MongoClient__pool.sockets)) self.assertEqual(0, len(b._MongoClient__pool.sockets)) b.end_request() b_sock = one(b._MongoClient__pool.sockets) b.pymongo_test.test.find_one() a.pymongo_test.test.find_one() self.assertEqual(b_sock, b._MongoClient__pool.get_socket((b.host, b.port))) self.assertEqual(a_sock, a._MongoClient__pool.get_socket((a.host, a.port))) a_sock.close() b_sock.close() def test_request(self): # Check that Pool gives two different sockets in two calls to # get_socket() -- doesn't automatically put us in a request any more cx_pool = self.get_pool( pair=(host,port), max_size=10, net_timeout=1000, conn_timeout=1000, use_ssl=False ) sock0 = cx_pool.get_socket() sock1 = cx_pool.get_socket() self.assertNotEqual(sock0, sock1) # Now in a request, we'll get the same socket both times cx_pool.start_request() sock2 = cx_pool.get_socket() sock3 = cx_pool.get_socket() self.assertEqual(sock2, sock3) # Pool didn't keep reference to sock0 or sock1; sock2 and 3 are new self.assertNotEqual(sock0, sock2) self.assertNotEqual(sock1, sock2) # Return the request sock to pool cx_pool.end_request() sock4 = cx_pool.get_socket() sock5 = cx_pool.get_socket() # Not in a request any more, we get different sockets self.assertNotEqual(sock4, sock5) # end_request() returned sock2 to pool self.assertEqual(sock4, sock2) for s in [sock0, sock1, sock2, sock3, sock4, sock5]: s.close() def test_reset_and_request(self): # reset() is called after a fork, or after a socket error. Ensure that # a new request is begun if a request was in progress when the reset() # occurred, otherwise no request is begun. p = self.get_pool((host, port), 10, None, None, False) self.assertFalse(p.in_request()) p.start_request() self.assertTrue(p.in_request()) p.reset() self.assertTrue(p.in_request()) p.end_request() self.assertFalse(p.in_request()) p.reset() self.assertFalse(p.in_request()) def test_pool_reuses_open_socket(self): # Test Pool's _check_closed() method doesn't close a healthy socket cx_pool = self.get_pool((host,port), 10, None, None, False) cx_pool._check_interval_seconds = 0 # Always check. sock_info = cx_pool.get_socket() cx_pool.maybe_return_socket(sock_info) new_sock_info = cx_pool.get_socket() self.assertEqual(sock_info, new_sock_info) cx_pool.maybe_return_socket(new_sock_info) self.assertEqual(1, len(cx_pool.sockets)) def test_pool_removes_dead_socket(self): # Test that Pool removes dead socket and the socket doesn't return # itself PYTHON-344 cx_pool = self.get_pool((host,port), 10, None, None, False) cx_pool._check_interval_seconds = 0 # Always check. sock_info = cx_pool.get_socket() # Simulate a closed socket without telling the SocketInfo it's closed sock_info.sock.close() self.assertTrue(pymongo.pool._closed(sock_info.sock)) cx_pool.maybe_return_socket(sock_info) new_sock_info = cx_pool.get_socket() self.assertEqual(0, len(cx_pool.sockets)) self.assertNotEqual(sock_info, new_sock_info) cx_pool.maybe_return_socket(new_sock_info) self.assertEqual(1, len(cx_pool.sockets)) def test_pool_removes_dead_request_socket_after_check(self): # Test that Pool keeps request going even if a socket dies in request cx_pool = self.get_pool((host,port), 10, None, None, False) cx_pool._check_interval_seconds = 0 # Always check. cx_pool.start_request() # Get the request socket sock_info = cx_pool.get_socket() self.assertEqual(0, len(cx_pool.sockets)) self.assertEqual(sock_info, cx_pool._get_request_state()) sock_info.sock.close() cx_pool.maybe_return_socket(sock_info) # Although the request socket died, we're still in a request with a # new socket new_sock_info = cx_pool.get_socket() self.assertTrue(cx_pool.in_request()) self.assertNotEqual(sock_info, new_sock_info) self.assertEqual(new_sock_info, cx_pool._get_request_state()) cx_pool.maybe_return_socket(new_sock_info) self.assertEqual(new_sock_info, cx_pool._get_request_state()) self.assertEqual(0, len(cx_pool.sockets)) cx_pool.end_request() self.assertEqual(1, len(cx_pool.sockets)) def test_pool_removes_dead_request_socket(self): # Test that Pool keeps request going even if a socket dies in request cx_pool = self.get_pool((host,port), 10, None, None, False) cx_pool.start_request() # Get the request socket sock_info = cx_pool.get_socket() self.assertEqual(0, len(cx_pool.sockets)) self.assertEqual(sock_info, cx_pool._get_request_state()) # Unlike in test_pool_removes_dead_request_socket_after_check, we # set sock_info.closed and *don't* wait for it to be checked. sock_info.close() cx_pool.maybe_return_socket(sock_info) # Although the request socket died, we're still in a request with a # new socket new_sock_info = cx_pool.get_socket() self.assertTrue(cx_pool.in_request()) self.assertNotEqual(sock_info, new_sock_info) self.assertEqual(new_sock_info, cx_pool._get_request_state()) cx_pool.maybe_return_socket(new_sock_info) self.assertEqual(new_sock_info, cx_pool._get_request_state()) self.assertEqual(0, len(cx_pool.sockets)) cx_pool.end_request() self.assertEqual(1, len(cx_pool.sockets)) def test_pool_removes_dead_socket_after_request(self): # Test that Pool handles a socket dying that *used* to be the request # socket. cx_pool = self.get_pool((host,port), 10, None, None, False) cx_pool._check_interval_seconds = 0 # Always check. cx_pool.start_request() # Get the request socket sock_info = cx_pool.get_socket() self.assertEqual(sock_info, cx_pool._get_request_state()) cx_pool.maybe_return_socket(sock_info) # End request cx_pool.end_request() self.assertEqual(1, len(cx_pool.sockets)) # Kill old request socket sock_info.sock.close() # Dead socket detected and removed new_sock_info = cx_pool.get_socket() self.assertFalse(cx_pool.in_request()) self.assertNotEqual(sock_info, new_sock_info) self.assertEqual(0, len(cx_pool.sockets)) self.assertFalse(pymongo.pool._closed(new_sock_info.sock)) cx_pool.maybe_return_socket(new_sock_info) self.assertEqual(1, len(cx_pool.sockets)) def test_dead_request_socket_with_max_size(self): # When a pool replaces a dead request socket, the semaphore it uses # to enforce max_size should remain unaffected. cx_pool = self.get_pool( (host, port), 1, None, None, False, wait_queue_timeout=1) cx_pool._check_interval_seconds = 0 # Always check. cx_pool.start_request() # Get and close the request socket. request_sock_info = cx_pool.get_socket() request_sock_info.sock.close() cx_pool.maybe_return_socket(request_sock_info) # Detects closed socket and creates new one, semaphore value still 0. request_sock_info_2 = cx_pool.get_socket() self.assertNotEqual(request_sock_info, request_sock_info_2) cx_pool.maybe_return_socket(request_sock_info_2) cx_pool.end_request() # Semaphore value now 1; we can get a socket. sock_info = cx_pool.get_socket() # Clean up. cx_pool.maybe_return_socket(sock_info) def test_socket_reclamation(self): if sys.platform.startswith('java'): raise SkipTest("Jython can't do socket reclamation") # Check that if a thread starts a request and dies without ending # the request, that the socket is reclaimed into the pool. cx_pool = self.get_pool( pair=(host,port), max_size=10, net_timeout=1000, conn_timeout=1000, use_ssl=False, ) self.assertEqual(0, len(cx_pool.sockets)) lock = None the_sock = [None] def leak_request(): self.assertEqual(NO_REQUEST, cx_pool._get_request_state()) cx_pool.start_request() self.assertEqual(NO_SOCKET_YET, cx_pool._get_request_state()) sock_info = cx_pool.get_socket() self.assertEqual(sock_info, cx_pool._get_request_state()) the_sock[0] = id(sock_info.sock) cx_pool.maybe_return_socket(sock_info) if not self.use_greenlets: lock.release() if self.use_greenlets: g = Greenlet(leak_request) g.start() g.join(1) self.assertTrue(g.ready(), "Greenlet is hung") # In Gevent after 0.13.8, join() returns before the Greenlet.link # callback fires. Give it a moment to reclaim the socket. gevent.sleep(0.1) else: lock = thread.allocate_lock() lock.acquire() # Start a thread WITHOUT a threading.Thread - important to test that # Pool can deal with primitive threads. thread.start_new_thread(leak_request, ()) # Join thread acquired = lock.acquire() self.assertTrue(acquired, "Thread is hung") # Make sure thread is really gone time.sleep(1) if 'PyPy' in sys.version: gc.collect() # Access the thread local from the main thread to trigger the # ThreadVigil's delete callback, returning the request socket to # the pool. # In Python 2.7.0 and lesser, a dead thread's locals are deleted # and those locals' weakref callbacks are fired only when another # thread accesses the locals and finds the thread state is stale, # see http://bugs.python.org/issue1868. Accessing the thread # local from the main thread is a necessary part of this test, and # realistic: in a multithreaded web server a new thread will access # Pool._ident._local soon after an old thread has died. cx_pool._ident.get() # Pool reclaimed the socket self.assertEqual(1, len(cx_pool.sockets)) self.assertEqual(the_sock[0], id(one(cx_pool.sockets).sock)) self.assertEqual(0, len(cx_pool._tid_to_sock)) class _TestMaxPoolSize(_TestPoolingBase): """Test that connection pool keeps proper number of idle sockets open, no matter how start/end_request are called. To be run both with threads and with greenlets. """ def _test_max_pool_size( self, start_request, end_request, max_pool_size=4, nthreads=10): """Start `nthreads` threads. Each calls start_request `start_request` times, then find_one and waits at a barrier; once all reach the barrier each calls end_request `end_request` times. The test asserts that the pool ends with min(max_pool_size, nthreads) sockets or, if start_request wasn't called, at least one socket. This tests both max_pool_size enforcement and that leaked request sockets are eventually returned to the pool when their threads end. You may need to increase ulimit -n on Mac. If you increase nthreads over about 35, note a Gevent 0.13.6 bug on Mac: Greenlet.join() hangs if more than about 35 Greenlets share a MongoClient. Apparently fixed in recent Gevent development. """ if start_request: if max_pool_size is not None and max_pool_size < nthreads: raise AssertionError("Deadlock") c = self.get_client( max_pool_size=max_pool_size, auto_start_request=False) rendezvous = CreateAndReleaseSocket.Rendezvous( nthreads, self.use_greenlets) threads = [] for i in range(nthreads): t = CreateAndReleaseSocket( self, c, start_request, end_request, rendezvous) threads.append(t) for t in threads: t.start() if 'PyPy' in sys.version: # With PyPy we need to kick off the gc whenever the threads hit the # rendezvous since nthreads > max_pool_size. gc_collect_until_done(threads) else: for t in threads: t.join() # join() returns before the thread state is cleared; give it time. self.sleep(1) for t in threads: self.assertTrue(t.passed) # Socket-reclamation doesn't work in Jython if not sys.platform.startswith('java'): cx_pool = c._MongoClient__pool # Socket-reclamation depends on timely garbage-collection if 'PyPy' in sys.version: gc.collect() if self.use_greenlets: # Wait for Greenlet.link() callbacks to execute the_hub = hub.get_hub() if hasattr(the_hub, 'join'): # Gevent 1.0 the_hub.join() else: # Gevent 0.13 and less the_hub.shutdown() if start_request: # Trigger final cleanup in Python <= 2.7.0. cx_pool._ident.get() expected_idle = min(max_pool_size, nthreads) message = ( '%d idle sockets (expected %d) and %d request sockets' ' (expected 0)' % ( len(cx_pool.sockets), expected_idle, len(cx_pool._tid_to_sock))) self.assertEqual( expected_idle, len(cx_pool.sockets), message) else: # Without calling start_request(), threads can safely share # sockets; the number running concurrently, and hence the # number of sockets needed, is between 1 and 10, depending # on thread-scheduling. self.assertTrue(len(cx_pool.sockets) >= 1) # thread.join completes slightly *before* thread locals are # cleaned up, so wait up to 5 seconds for them. self.sleep(0.1) cx_pool._ident.get() start = time.time() while ( not cx_pool.sockets and cx_pool._socket_semaphore.counter < max_pool_size and (time.time() - start) < 5 ): self.sleep(0.1) cx_pool._ident.get() if max_pool_size is not None: self.assertEqual( max_pool_size, cx_pool._socket_semaphore.counter) self.assertEqual(0, len(cx_pool._tid_to_sock)) def _test_max_pool_size_no_rendezvous(self, start_request, end_request): max_pool_size = 5 c = self.get_client( max_pool_size=max_pool_size, auto_start_request=False) # If you increase nthreads over about 35, note a # Gevent 0.13.6 bug on Mac, Greenlet.join() hangs if more than # about 35 Greenlets share a MongoClient. Apparently fixed in # recent Gevent development. # On the other hand, nthreads had better be much larger than # max_pool_size to ensure that max_pool_size sockets are actually # required at some point in this test's execution. nthreads = 30 if (sys.platform.startswith('java') and start_request > end_request and nthreads > max_pool_size): # Since Jython can't reclaim the socket and release the semaphore # after a thread leaks a request, we'll exhaust the semaphore and # deadlock. raise SkipTest("Jython can't do socket reclamation") threads = [] for i in range(nthreads): t = CreateAndReleaseSocketNoRendezvous( self, c, start_request, end_request) threads.append(t) for t in threads: t.start() if 'PyPy' in sys.version: # With PyPy we need to kick off the gc whenever the threads hit the # rendezvous since nthreads > max_pool_size. gc_collect_until_done(threads) else: for t in threads: t.join() for t in threads: self.assertTrue(t.passed) cx_pool = c._MongoClient__pool # Socket-reclamation depends on timely garbage-collection if 'PyPy' in sys.version: gc.collect() if self.use_greenlets: # Wait for Greenlet.link() callbacks to execute the_hub = hub.get_hub() if hasattr(the_hub, 'join'): # Gevent 1.0 the_hub.join() else: # Gevent 0.13 and less the_hub.shutdown() # thread.join completes slightly *before* thread locals are # cleaned up, so wait up to 5 seconds for them. self.sleep(0.1) cx_pool._ident.get() start = time.time() while ( not cx_pool.sockets and cx_pool._socket_semaphore.counter < max_pool_size and (time.time() - start) < 5 ): self.sleep(0.1) cx_pool._ident.get() self.assertTrue(len(cx_pool.sockets) >= 1) self.assertEqual(max_pool_size, cx_pool._socket_semaphore.counter) def test_max_pool_size(self): self._test_max_pool_size( start_request=0, end_request=0, nthreads=10, max_pool_size=4) def test_max_pool_size_none(self): self._test_max_pool_size( start_request=0, end_request=0, nthreads=10, max_pool_size=None) def test_max_pool_size_with_request(self): self._test_max_pool_size( start_request=1, end_request=1, nthreads=10, max_pool_size=10) def test_max_pool_size_with_multiple_request(self): self._test_max_pool_size( start_request=10, end_request=10, nthreads=10, max_pool_size=10) def test_max_pool_size_with_redundant_request(self): self._test_max_pool_size( start_request=2, end_request=1, nthreads=10, max_pool_size=10) def test_max_pool_size_with_redundant_request2(self): self._test_max_pool_size( start_request=20, end_request=1, nthreads=10, max_pool_size=10) def test_max_pool_size_with_redundant_request_no_rendezvous(self): self._test_max_pool_size_no_rendezvous(2, 1) def test_max_pool_size_with_redundant_request_no_rendezvous2(self): self._test_max_pool_size_no_rendezvous(20, 1) def test_max_pool_size_with_leaked_request(self): # Call start_request() but not end_request() -- when threads die, they # should return their request sockets to the pool. self._test_max_pool_size( start_request=1, end_request=0, nthreads=10, max_pool_size=10) def test_max_pool_size_with_leaked_request_no_rendezvous(self): self._test_max_pool_size_no_rendezvous(1, 0) def test_max_pool_size_with_end_request_only(self): # Call end_request() but not start_request() self._test_max_pool_size(0, 1) def test_max_pool_size_with_connection_failure(self): # The pool acquires its semaphore before attempting to connect; ensure # it releases the semaphore on connection failure. class TestPool(Pool): def connect(self, pair): raise socket.error() test_pool = TestPool( pair=('example.com', 27017), max_size=1, net_timeout=1, conn_timeout=1, use_ssl=False, wait_queue_timeout=1, use_greenlets=self.use_greenlets) # First call to get_socket fails; if pool doesn't release its semaphore # then the second call raises "ConnectionFailure: Timed out waiting for # socket from pool" instead of the socket.error. for i in range(2): self.assertRaises(socket.error, test_pool.get_socket) class SocketGetter(MongoThread): """Utility for _TestMaxOpenSockets and _TestWaitQueueMultiple""" def __init__(self, test_case, pool): super(SocketGetter, self).__init__(test_case) self.state = 'init' self.pool = pool self.sock = None def run(self): self.state = 'get_socket' self.sock = self.pool.get_socket() self.state = 'sock' class _TestMaxOpenSockets(_TestPoolingBase): """Test that connection pool doesn't open more than max_size sockets. To be run both with threads and with greenlets. """ def get_pool_with_wait_queue_timeout(self, wait_queue_timeout): return self.get_pool((host, port), 1, None, None, False, wait_queue_timeout=wait_queue_timeout, wait_queue_multiple=None) def test_wait_queue_timeout(self): wait_queue_timeout = 2 # Seconds pool = self.get_pool_with_wait_queue_timeout(wait_queue_timeout) sock_info = pool.get_socket() start = time.time() self.assertRaises(ConnectionFailure, pool.get_socket) duration = time.time() - start self.assertTrue( abs(wait_queue_timeout - duration) < 1, "Waited %.2f seconds for a socket, expected %f" % ( duration, wait_queue_timeout)) sock_info.close() def test_blocking(self): # Verify get_socket() with no wait_queue_timeout blocks forever. pool = self.get_pool_with_wait_queue_timeout(None) # Reach max_size. s1 = pool.get_socket() t = SocketGetter(self, pool) t.start() while t.state != 'get_socket': self.sleep(0.1) self.sleep(1) self.assertEqual(t.state, 'get_socket') pool.maybe_return_socket(s1) while t.state != 'sock': self.sleep(0.1) self.assertEqual(t.state, 'sock') self.assertEqual(t.sock, s1) s1.close() class _TestWaitQueueMultiple(_TestPoolingBase): """Test that connection pool doesn't allow more than waitQueueMultiple * max_size waiters. To be run both with threads and with greenlets. """ def get_pool_with_wait_queue_multiple(self, wait_queue_multiple): return self.get_pool((host, port), 2, None, None, False, wait_queue_timeout=None, wait_queue_multiple=wait_queue_multiple) def test_wait_queue_multiple(self): pool = self.get_pool_with_wait_queue_multiple(3) # Reach max_size sockets. socket_info_0 = pool.get_socket() socket_info_1 = pool.get_socket() # Reach max_size * wait_queue_multiple waiters. threads = [] for _ in xrange(6): t = SocketGetter(self, pool) t.start() threads.append(t) self.sleep(1) for t in threads: self.assertEqual(t.state, 'get_socket') self.assertRaises(ExceededMaxWaiters, pool.get_socket) socket_info_0.close() socket_info_1.close() def test_wait_queue_multiple_unset(self): pool = self.get_pool_with_wait_queue_multiple(None) socks = [] for _ in xrange(2): sock = pool.get_socket() socks.append(sock) threads = [] for _ in xrange(30): t = SocketGetter(self, pool) t.start() threads.append(t) self.sleep(1) for t in threads: self.assertEqual(t.state, 'get_socket') for socket_info in socks: socket_info.close() class _TestPoolSocketSharing(_TestPoolingBase): """Directly test that two simultaneous operations don't share a socket. To be run both with threads and with greenlets. """ def _test_pool(self, use_request): """ Test that the connection pool prevents both threads and greenlets from using a socket at the same time. Sequence: gr0: start a slow find() gr1: start a fast find() gr1: get results gr0: get results """ cx = get_client( use_greenlets=self.use_greenlets, auto_start_request=False ) db = cx.pymongo_test db.test.remove() db.test.insert({'_id': 1}) history = [] def find_fast(): if use_request: cx.start_request() history.append('find_fast start') # With greenlets and the old connection._Pool, this would throw # AssertionError: "This event is already used by another # greenlet" self.assertEqual({'_id': 1}, db.test.find_one()) history.append('find_fast done') if use_request: cx.end_request() def find_slow(): if use_request: cx.start_request() history.append('find_slow start') # Javascript function that pauses N seconds per document fn = delay(10) if (is_mongos(db.connection) or not version.at_least(db.connection, (1, 7, 2))): # mongos doesn't support eval so we have to use $where # which is less reliable in this context. self.assertEqual(1, db.test.find({"$where": fn}).count()) else: # 'nolock' allows find_fast to start and finish while we're # waiting for this to complete. self.assertEqual({'ok': 1.0, 'retval': True}, db.command('eval', fn, nolock=True)) history.append('find_slow done') if use_request: cx.end_request() if self.use_greenlets: gr0, gr1 = Greenlet(find_slow), Greenlet(find_fast) gr0.start() gr1.start_later(.1) else: gr0 = threading.Thread(target=find_slow) gr0.setDaemon(True) gr1 = threading.Thread(target=find_fast) gr1.setDaemon(True) gr0.start() time.sleep(.1) gr1.start() gr0.join() gr1.join() self.assertEqual([ 'find_slow start', 'find_fast start', 'find_fast done', 'find_slow done', ], history) def test_pool(self): self._test_pool(use_request=False) def test_pool_request(self): self._test_pool(use_request=True) pymongo-2.6.3/test/test_pooling_gevent.py000066400000000000000000000200231223300253600206120ustar00rootroot00000000000000# Copyright 2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you # may not use this file except in compliance with the License. You # may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. See the License for the specific language governing # permissions and limitations under the License. """Tests for connection-pooling with greenlets and Gevent""" import gc import time import unittest from nose.plugins.skip import SkipTest from pymongo import pool from pymongo.errors import ConfigurationError from test import host, port from test.utils import looplet from test.test_pooling_base import ( _TestPooling, _TestMaxPoolSize, _TestMaxOpenSockets, _TestPoolSocketSharing, _TestWaitQueueMultiple, has_gevent) class TestPoolingGevent(_TestPooling, unittest.TestCase): """Apply all the standard pool tests with greenlets and Gevent""" use_greenlets = True class TestPoolingGeventSpecial(unittest.TestCase): """Do a few special greenlet tests that don't use TestPoolingBase""" def test_greenlet_sockets(self): # Check that Pool gives two sockets to two greenlets try: import greenlet import gevent except ImportError: raise SkipTest('Gevent not installed') cx_pool = pool.Pool( pair=(host, port), max_size=10, net_timeout=1000, conn_timeout=1000, use_ssl=False, use_greenlets=True) socks = [] def get_socket(): cx_pool.start_request() socks.append(cx_pool.get_socket()) looplet([ greenlet.greenlet(get_socket), greenlet.greenlet(get_socket), ]) self.assertEqual(2, len(socks)) self.assertNotEqual(socks[0], socks[1]) def test_greenlet_sockets_with_request(self): # Verify two assumptions: that start_request() with two greenlets but # not use_greenlets fails, meaning that the two greenlets will # share one socket. Also check that start_request() with use_greenlets # succeeds, meaning that two greenlets will get different sockets. try: import greenlet import gevent except ImportError: raise SkipTest('Gevent not installed') pool_args = dict( pair=(host,port), max_size=10, net_timeout=1000, conn_timeout=1000, use_ssl=False, ) for use_greenlets, use_request, expect_success in [ (True, True, True), (True, False, False), (False, True, False), (False, False, False), ]: pool_args_cp = pool_args.copy() pool_args_cp['use_greenlets'] = use_greenlets cx_pool = pool.Pool(**pool_args_cp) # Map: greenlet -> socket greenlet2socks = {} main = greenlet.getcurrent() def get_socket_in_request(): # Get a socket from the pool twice, switching contexts each time if use_request: cx_pool.start_request() main.switch() for _ in range(2): sock = cx_pool.get_socket() cx_pool.maybe_return_socket(sock) greenlet2socks.setdefault( greenlet.getcurrent(), [] ).append(id(sock)) main.switch() cx_pool.end_request() greenlets = [ greenlet.greenlet(get_socket_in_request), greenlet.greenlet(get_socket_in_request), ] # Run both greenlets to completion looplet(greenlets) socks_for_gr0 = greenlet2socks[greenlets[0]] socks_for_gr1 = greenlet2socks[greenlets[1]] # Whether we expect requests to work or not, we definitely expect # greenlet2socks to have the same number of keys and values self.assertEqual(2, len(greenlet2socks)) self.assertEqual(2, len(socks_for_gr0)) self.assertEqual(2, len(socks_for_gr1)) # If we started a request, then there was a point at which we had # 2 active sockets, otherwise we always used one. if use_request and use_greenlets: self.assertEqual(2, len(cx_pool.sockets)) else: self.assertEqual(1, len(cx_pool.sockets)) # Again, regardless of whether requests work, a greenlet will get # the same socket each time it calls get_socket() within a request. # What we're really testing is that the two *different* greenlets # get *different* sockets from each other. self.assertEqual( socks_for_gr0[0], socks_for_gr0[1], "Expected greenlet 0 to get the same socket for each call " "to get_socket()" ) self.assertEqual( socks_for_gr1[0], socks_for_gr1[1], "Expected greenlet 1 to get the same socket for each call " "to get_socket()" ) if expect_success: # We passed use_greenlets=True, so start_request successfully # distinguished between the two greenlets. self.assertNotEqual( socks_for_gr0[0], socks_for_gr1[0], "Expected two greenlets to get two different sockets" ) else: # We passed use_greenlets=False, so start_request didn't # distinguish between the two greenlets, and it gave them both # the same socket. self.assertEqual( socks_for_gr0[0], socks_for_gr1[0], "Expected two greenlets to get same socket" ) class TestMaxPoolSizeGevent(_TestMaxPoolSize, unittest.TestCase): use_greenlets = True class TestPoolSocketSharingGevent(_TestPoolSocketSharing, unittest.TestCase): use_greenlets = True class TestMaxOpenSocketsGevent(_TestMaxOpenSockets, unittest.TestCase): use_greenlets = True class TestWaitQueueMultipleGevent(_TestWaitQueueMultiple, unittest.TestCase): use_greenlets = True class TestUseGreenletsWithoutGevent(unittest.TestCase): def test_use_greenlets_without_gevent(self): # Verify that Pool(use_greenlets=True) raises ConfigurationError if # Gevent is not installed, and that its destructor runs without error. if has_gevent: raise SkipTest( "Gevent is installed, can't test what happens calling " "Pool(use_greenlets=True) when Gevent is unavailable") # Possible outcomes of __del__. DID_NOT_RUN, RAISED, SUCCESS = range(3) outcome = [DID_NOT_RUN] class TestPool(pool.Pool): def __del__(self): try: pool.Pool.__del__(self) # Pool is old-style, no super() outcome[0] = SUCCESS except: outcome[0] = RAISED # Pool raises ConfigurationError, "The Gevent module is not available". self.assertRaises( ConfigurationError, TestPool, pair=(host, port), max_size=10, net_timeout=1000, conn_timeout=1000, use_ssl=False, use_greenlets=True) # Convince Jython or PyPy to call __del__. for _ in range(10): if outcome[0] == DID_NOT_RUN: gc.collect() time.sleep(0.1) if outcome[0] == DID_NOT_RUN: self.fail("Pool.__del__ didn't run") elif outcome[0] == RAISED: self.fail("Pool.__del__ raised exception") if __name__ == '__main__': unittest.main() pymongo-2.6.3/test/test_pymongo.py000066400000000000000000000020161223300253600172650ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the pymongo module itself.""" import unittest import os import sys sys.path[0:0] = [""] import pymongo from test import host, port class TestPyMongo(unittest.TestCase): def test_mongo_client_alias(self): # Testing that pymongo module imports mongo_client.MongoClient c = pymongo.MongoClient(host, port) self.assertEqual(c.host, host) self.assertEqual(c.port, port) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_read_preferences.py000066400000000000000000000576161223300253600211110ustar00rootroot00000000000000# Copyright 2011-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the replica_set_connection module.""" import random import sys import unittest from nose.plugins.skip import SkipTest sys.path[0:0] = [""] from bson.son import SON from pymongo.cursor import _QUERY_OPTIONS from pymongo.mongo_replica_set_client import MongoReplicaSetClient from pymongo.read_preferences import (ReadPreference, modes, MovingAverage, secondary_ok_commands) from pymongo.errors import ConfigurationError from test.test_replica_set_client import TestReplicaSetClientBase from test.test_client import get_client from test import version, utils, host, port class TestReadPreferencesBase(TestReplicaSetClientBase): def setUp(self): super(TestReadPreferencesBase, self).setUp() # Insert some data so we can use cursors in read_from_which_host c = self._get_client() c.pymongo_test.test.drop() c.pymongo_test.test.insert([{'_id': i} for i in range(10)], w=self.w) def tearDown(self): super(TestReadPreferencesBase, self).tearDown() c = self._get_client() c.pymongo_test.test.drop() def read_from_which_host(self, client): """Do a find() on the client and return which host was used """ cursor = client.pymongo_test.test.find() cursor.next() return cursor._Cursor__connection_id def read_from_which_kind(self, client): """Do a find() on the client and return 'primary' or 'secondary' depending on which the client used. """ connection_id = self.read_from_which_host(client) if connection_id == client.primary: return 'primary' elif connection_id in client.secondaries: return 'secondary' else: self.fail( 'Cursor used connection id %s, expected either primary ' '%s or secondaries %s' % ( connection_id, client.primary, client.secondaries)) def assertReadsFrom(self, expected, **kwargs): c = self._get_client(**kwargs) used = self.read_from_which_kind(c) self.assertEqual(expected, used, 'Cursor used %s, expected %s' % ( expected, used)) class TestReadPreferences(TestReadPreferencesBase): def test_mode_validation(self): # 'modes' are imported from read_preferences.py for mode in modes: self.assertEqual(mode, self._get_client( read_preference=mode).read_preference) self.assertRaises(ConfigurationError, self._get_client, read_preference='foo') def test_tag_sets_validation(self): # Can't use tags with PRIMARY self.assertRaises(ConfigurationError, self._get_client, tag_sets=[{'k': 'v'}]) # ... but empty tag sets are ok with PRIMARY self.assertEqual([{}], self._get_client(tag_sets=[{}]).tag_sets) S = ReadPreference.SECONDARY self.assertEqual([{}], self._get_client(read_preference=S).tag_sets) self.assertEqual([{'k': 'v'}], self._get_client( read_preference=S, tag_sets=[{'k': 'v'}]).tag_sets) self.assertEqual([{'k': 'v'}, {}], self._get_client( read_preference=S, tag_sets=[{'k': 'v'}, {}]).tag_sets) self.assertRaises(ConfigurationError, self._get_client, read_preference=S, tag_sets=[]) # One dict not ok, must be a list of dicts self.assertRaises(ConfigurationError, self._get_client, read_preference=S, tag_sets={'k': 'v'}) self.assertRaises(ConfigurationError, self._get_client, read_preference=S, tag_sets='foo') self.assertRaises(ConfigurationError, self._get_client, read_preference=S, tag_sets=['foo']) def test_latency_validation(self): self.assertEqual(17, self._get_client( secondary_acceptable_latency_ms=17 ).secondary_acceptable_latency_ms) self.assertEqual(42, self._get_client( secondaryAcceptableLatencyMS=42 ).secondary_acceptable_latency_ms) self.assertEqual(666, self._get_client( secondaryacceptablelatencyms=666 ).secondary_acceptable_latency_ms) def test_primary(self): self.assertReadsFrom('primary', read_preference=ReadPreference.PRIMARY) def test_primary_with_tags(self): # Tags not allowed with PRIMARY self.assertRaises(ConfigurationError, self._get_client, tag_sets=[{'dc': 'ny'}]) def test_primary_preferred(self): self.assertReadsFrom('primary', read_preference=ReadPreference.PRIMARY_PREFERRED) def test_secondary(self): self.assertReadsFrom('secondary', read_preference=ReadPreference.SECONDARY) def test_secondary_preferred(self): self.assertReadsFrom('secondary', read_preference=ReadPreference.SECONDARY_PREFERRED) def test_secondary_only(self): # Test deprecated mode SECONDARY_ONLY, which is now a synonym for # SECONDARY self.assertEqual( ReadPreference.SECONDARY, ReadPreference.SECONDARY_ONLY) def test_nearest(self): # With high secondaryAcceptableLatencyMS, expect to read from any # member c = self._get_client( read_preference=ReadPreference.NEAREST, secondaryAcceptableLatencyMS=10000, # 10 seconds auto_start_request=False) data_members = set(self.hosts).difference(set(self.arbiters)) # This is a probabilistic test; track which members we've read from so # far, and keep reading until we've used all the members or give up. # Chance of using only 2 of 3 members 10k times if there's no bug = # 3 * (2/3)**10000, very low. used = set() i = 0 while data_members.difference(used) and i < 10000: host = self.read_from_which_host(c) used.add(host) i += 1 not_used = data_members.difference(used) latencies = ', '.join( '%s: %dms' % (member.host, member.ping_time.get()) for member in c._MongoReplicaSetClient__rs_state.members) self.assertFalse(not_used, "Expected to use primary and all secondaries for mode NEAREST," " but didn't use %s\nlatencies: %s" % (not_used, latencies)) class ReadPrefTester(MongoReplicaSetClient): def __init__(self, *args, **kwargs): self.has_read_from = set() super(ReadPrefTester, self).__init__(*args, **kwargs) def _MongoReplicaSetClient__send_and_receive(self, member, *args, **kwargs): self.has_read_from.add(member) rsc = super(ReadPrefTester, self) return rsc._MongoReplicaSetClient__send_and_receive( member, *args, **kwargs) class TestCommandAndReadPreference(TestReplicaSetClientBase): def setUp(self): super(TestCommandAndReadPreference, self).setUp() # Need auto_start_request False to avoid pinning members. self.c = ReadPrefTester( '%s:%s' % (host, port), replicaSet=self.name, auto_start_request=False, # Effectively ignore members' ping times so we can test the effect # of ReadPreference modes only secondary_acceptable_latency_ms=1000*1000) def tearDown(self): self.c.close() # We create a lot of collections and indexes in these tests, so drop # the database self._get_client().drop_database('pymongo_test') super(TestCommandAndReadPreference, self).tearDown() def executed_on_which_member(self, client, fn, *args, **kwargs): client.has_read_from.clear() fn(*args, **kwargs) self.assertEqual(1, len(client.has_read_from)) member, = client.has_read_from return member def assertExecutedOn(self, state, client, fn, *args, **kwargs): member = self.executed_on_which_member(client, fn, *args, **kwargs) if state == 'primary': self.assertTrue(member.is_primary) elif state == 'secondary': self.assertFalse(member.is_primary) else: self.fail("Bad state %s" % repr(state)) def _test_fn(self, obedient, fn): if not obedient: for mode in modes: self.c.read_preference = mode # Run it a few times to make sure we don't just get lucky the # first time. for _ in range(10): self.assertExecutedOn('primary', self.c, fn) else: for mode, expected_state in [ (ReadPreference.PRIMARY, 'primary'), (ReadPreference.PRIMARY_PREFERRED, 'primary'), (ReadPreference.SECONDARY, 'secondary'), (ReadPreference.SECONDARY_PREFERRED, 'secondary'), (ReadPreference.NEAREST, 'any'), ]: self.c.read_preference = mode for _ in range(10): if expected_state in ('primary', 'secondary'): self.assertExecutedOn(expected_state, self.c, fn) elif expected_state == 'any': used = set() for _ in range(1000): member = self.executed_on_which_member( self.c, fn) used.add(member.host) if len(used) == len(self.c.secondaries) + 1: # Success break unused = self.c.secondaries.union( set([self.c.primary]) ).difference(used) if unused: self.fail( "Some members not used for NEAREST: %s" % ( unused)) def test_command(self): # Test generic 'command' method. Some commands obey read preference, # most don't. # Disobedient commands, always go to primary self._test_fn(False, lambda: self.c.pymongo_test.command('ping')) self._test_fn(False, lambda: self.c.admin.command('buildinfo')) # Obedient commands. self._test_fn(True, lambda: self.c.pymongo_test.command('group', { 'ns': 'test', 'key': {'a': 1}, '$reduce': 'function(obj, prev) { }', 'initial': {}})) self._test_fn(True, lambda: self.c.pymongo_test.command('dbStats')) # collStats fails if no collection self.c.pymongo_test.test.insert({}, w=self.w) self._test_fn(True, lambda: self.c.pymongo_test.command( 'collStats', 'test')) # Count self._test_fn(True, lambda: self.c.pymongo_test.command( 'count', 'test')) self._test_fn(True, lambda: self.c.pymongo_test.command( 'count', 'test', query={'a': 1})) self._test_fn(True, lambda: self.c.pymongo_test.command(SON([ ('count', 'test'), ('query', {'a': 1})]))) # Distinct self._test_fn(True, lambda: self.c.pymongo_test.command( 'distinct', 'test', key={'a': 1})) self._test_fn(True, lambda: self.c.pymongo_test.command( 'distinct', 'test', key={'a': 1}, query={'a': 1})) self._test_fn(True, lambda: self.c.pymongo_test.command(SON([ ('distinct', 'test'), ('key', {'a': 1}), ('query', {'a': 1})]))) # Geo stuff. Make sure a 2d index is created and replicated self.c.pymongo_test.system.indexes.insert({ 'key' : { 'location' : '2d' }, 'ns' : 'pymongo_test.test', 'name' : 'location_2d' }, w=self.w) self.c.pymongo_test.system.indexes.insert(SON([ ('ns', 'pymongo_test.test'), ('key', SON([('location', 'geoHaystack'), ('key', 1)])), ('bucketSize', 100), ('name', 'location_geoHaystack'), ]), w=self.w) self._test_fn(True, lambda: self.c.pymongo_test.command( 'geoNear', 'test', near=[0, 0])) self._test_fn(True, lambda: self.c.pymongo_test.command(SON([ ('geoNear', 'test'), ('near', [0, 0])]))) self._test_fn(True, lambda: self.c.pymongo_test.command( 'geoSearch', 'test', near=[33, 33], maxDistance=6, search={'type': 'restaurant' }, limit=30)) self._test_fn(True, lambda: self.c.pymongo_test.command(SON([ ('geoSearch', 'test'), ('near', [33, 33]), ('maxDistance', 6), ('search', {'type': 'restaurant'}), ('limit', 30)]))) if version.at_least(self.c, (2, 1, 0)): self._test_fn(True, lambda: self.c.pymongo_test.command(SON([ ('aggregate', 'test'), ('pipeline', []) ]))) # Text search. if version.at_least(self.c, (2, 3, 2)): utils.enable_text_search(self.c) db = self.c.pymongo_test # Only way to create an index and wait for all members to build it. index = { 'ns': 'pymongo_test.test', 'name': 't_text', 'key': {'t': 'text'}} db.system.indexes.insert( index, manipulate=False, check_keys=False, w=self.w) self._test_fn(True, lambda: self.c.pymongo_test.command(SON([ ('text', 'test'), ('search', 'foo')]))) self.c.pymongo_test.test.drop_indexes() def test_map_reduce_command(self): # mapreduce fails if no collection self.c.pymongo_test.test.insert({}, w=self.w) # Non-inline mapreduce always goes to primary, doesn't obey read prefs. # Test with command in a SON and with kwargs self._test_fn(False, lambda: self.c.pymongo_test.command(SON([ ('mapreduce', 'test'), ('map', 'function() { }'), ('reduce', 'function() { }'), ('out', 'mr_out') ]))) self._test_fn(False, lambda: self.c.pymongo_test.command( 'mapreduce', 'test', map='function() { }', reduce='function() { }', out='mr_out')) self._test_fn(False, lambda: self.c.pymongo_test.command( 'mapreduce', 'test', map='function() { }', reduce='function() { }', out={'replace': 'some_collection'})) # Inline mapreduce obeys read prefs self._test_fn(True, lambda: self.c.pymongo_test.command( 'mapreduce', 'test', map='function() { }', reduce='function() { }', out={'inline': True})) self._test_fn(True, lambda: self.c.pymongo_test.command(SON([ ('mapreduce', 'test'), ('map', 'function() { }'), ('reduce', 'function() { }'), ('out', {'inline': True}) ]))) def test_create_collection(self): # Collections should be created on primary, obviously self._test_fn(False, lambda: self.c.pymongo_test.command( 'create', 'some_collection%s' % random.randint(0, sys.maxint))) self._test_fn(False, lambda: self.c.pymongo_test.create_collection( 'some_collection%s' % random.randint(0, sys.maxint))) def test_drop_collection(self): self._test_fn(False, lambda: self.c.pymongo_test.drop_collection( 'some_collection')) self._test_fn(False, lambda: self.c.pymongo_test.some_collection.drop()) def test_group(self): self._test_fn(True, lambda: self.c.pymongo_test.test.group( {'a': 1}, {}, {}, 'function() { }')) def test_map_reduce(self): # mapreduce fails if no collection self.c.pymongo_test.test.insert({}, w=self.w) self._test_fn(False, lambda: self.c.pymongo_test.test.map_reduce( 'function() { }', 'function() { }', 'mr_out')) self._test_fn(True, lambda: self.c.pymongo_test.test.map_reduce( 'function() { }', 'function() { }', {'inline': 1})) def test_inline_map_reduce(self): # mapreduce fails if no collection self.c.pymongo_test.test.insert({}, w=self.w) self._test_fn(True, lambda: self.c.pymongo_test.test.inline_map_reduce( 'function() { }', 'function() { }')) self._test_fn(True, lambda: self.c.pymongo_test.test.inline_map_reduce( 'function() { }', 'function() { }', full_response=True)) def test_count(self): self._test_fn(True, lambda: self.c.pymongo_test.test.count()) self._test_fn(True, lambda: self.c.pymongo_test.test.find().count()) def test_distinct(self): self._test_fn(True, lambda: self.c.pymongo_test.test.distinct('a')) self._test_fn(True, lambda: self.c.pymongo_test.test.find().distinct('a')) def test_aggregate(self): if version.at_least(self.c, (2, 1, 0)): self._test_fn(True, lambda: self.c.pymongo_test.test.aggregate([])) class TestMovingAverage(unittest.TestCase): def test_empty_init(self): self.assertRaises(AssertionError, MovingAverage, []) def test_moving_average(self): avg = MovingAverage([10]) self.assertEqual(10, avg.get()) avg2 = avg.clone_with(20) self.assertEqual(15, avg2.get()) avg3 = avg2.clone_with(30) self.assertEqual(20, avg3.get()) avg4 = avg3.clone_with(-100) self.assertEqual((10 + 20 + 30 - 100) / 4., avg4.get()) avg5 = avg4.clone_with(17) self.assertEqual((10 + 20 + 30 - 100 + 17) / 5., avg5.get()) avg6 = avg5.clone_with(43) self.assertEqual((20 + 30 - 100 + 17 + 43) / 5., avg6.get()) avg7 = avg6.clone_with(-1111) self.assertEqual((30 - 100 + 17 + 43 - 1111) / 5., avg7.get()) class TestMongosConnection(unittest.TestCase): def test_mongos_connection(self): c = get_client() is_mongos = utils.is_mongos(c) # Test default mode, PRIMARY cursor = c.pymongo_test.test.find() if is_mongos: # We only set $readPreference if it's something other than # PRIMARY to avoid problems with mongos versions that don't # support read preferences. self.assertEqual( None, cursor._Cursor__query_spec().get('$readPreference') ) else: self.assertFalse( '$readPreference' in cursor._Cursor__query_spec()) # Copy these constants for brevity PRIMARY_PREFERRED = ReadPreference.PRIMARY_PREFERRED SECONDARY = ReadPreference.SECONDARY SECONDARY_PREFERRED = ReadPreference.SECONDARY_PREFERRED NEAREST = ReadPreference.NEAREST SLAVE_OKAY = _QUERY_OPTIONS['slave_okay'] # Test non-PRIMARY modes which can be combined with tags for kwarg, value, mongos_mode in ( ('read_preference', PRIMARY_PREFERRED, 'primaryPreferred'), ('read_preference', SECONDARY, 'secondary'), ('read_preference', SECONDARY_PREFERRED, 'secondaryPreferred'), ('read_preference', NEAREST, 'nearest'), ('slave_okay', True, 'secondaryPreferred'), ('slave_okay', False, 'primary') ): for tag_sets in ( None, [{}] ): # Create a client e.g. with read_preference=NEAREST or # slave_okay=True c = get_client(tag_sets=tag_sets, **{kwarg: value}) self.assertEqual(is_mongos, c.is_mongos) cursor = c.pymongo_test.test.find() if is_mongos: # We don't set $readPreference for SECONDARY_PREFERRED # unless tags are in use. slaveOkay has the same effect. if mongos_mode == 'secondaryPreferred': self.assertEqual( None, cursor._Cursor__query_spec().get('$readPreference')) self.assertTrue( cursor._Cursor__query_options() & SLAVE_OKAY) # Don't send $readPreference for PRIMARY either elif mongos_mode == 'primary': self.assertEqual( None, cursor._Cursor__query_spec().get('$readPreference')) self.assertFalse( cursor._Cursor__query_options() & SLAVE_OKAY) else: self.assertEqual( {'mode': mongos_mode}, cursor._Cursor__query_spec().get('$readPreference')) self.assertTrue( cursor._Cursor__query_options() & SLAVE_OKAY) else: self.assertFalse( '$readPreference' in cursor._Cursor__query_spec()) for tag_sets in ( [{'dc': 'la'}], [{'dc': 'la'}, {'dc': 'sf'}], [{'dc': 'la'}, {'dc': 'sf'}, {}], ): if kwarg == 'slave_okay': # Can't use tags with slave_okay True or False, need a # real read preference self.assertRaises( ConfigurationError, get_client, tag_sets=tag_sets, **{kwarg: value}) continue c = get_client(tag_sets=tag_sets, **{kwarg: value}) self.assertEqual(is_mongos, c.is_mongos) cursor = c.pymongo_test.test.find() if is_mongos: self.assertEqual( {'mode': mongos_mode, 'tags': tag_sets}, cursor._Cursor__query_spec().get('$readPreference')) else: self.assertFalse( '$readPreference' in cursor._Cursor__query_spec()) def test_only_secondary_ok_commands_have_read_prefs(self): c = get_client(read_preference=ReadPreference.SECONDARY) is_mongos = utils.is_mongos(c) if not is_mongos: raise SkipTest("Only mongos have read_prefs added to the spec") # Ensure secondary_ok_commands have readPreference for cmd in secondary_ok_commands: if cmd == 'mapreduce': # map reduce is a special case continue command = SON([(cmd, 1)]) cursor = c.pymongo_test["$cmd"].find(command.copy()) # White-listed commands also have to be wrapped in $query command = SON([('$query', command)]) command['$readPreference'] = {'mode': 'secondary'} self.assertEqual(command, cursor._Cursor__query_spec()) # map_reduce inline should have read prefs command = SON([('mapreduce', 'test'), ('out', {'inline': 1})]) cursor = c.pymongo_test["$cmd"].find(command.copy()) # White-listed commands also have to be wrapped in $query command = SON([('$query', command)]) command['$readPreference'] = {'mode': 'secondary'} self.assertEqual(command, cursor._Cursor__query_spec()) # map_reduce that outputs to a collection shouldn't have read prefs command = SON([('mapreduce', 'test'), ('out', {'mrtest': 1})]) cursor = c.pymongo_test["$cmd"].find(command.copy()) self.assertEqual(command, cursor._Cursor__query_spec()) # Other commands shouldn't be changed for cmd in ('drop', 'create', 'any-future-cmd'): command = SON([(cmd, 1)]) cursor = c.pymongo_test["$cmd"].find(command.copy()) self.assertEqual(command, cursor._Cursor__query_spec()) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_replica_set_client.py000066400000000000000000001240041223300253600214270ustar00rootroot00000000000000# Copyright 2011-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the replica_set_connection module.""" # TODO: anywhere we wait for refresh in tests, consider just refreshing w/ sync import copy import datetime import os import signal import socket import sys import time import thread import threading import traceback import unittest sys.path[0:0] = [""] from nose.plugins.skip import SkipTest from bson.son import SON from bson.tz_util import utc from pymongo.mongo_client import MongoClient from pymongo.read_preferences import ReadPreference from pymongo.mongo_replica_set_client import MongoReplicaSetClient from pymongo.mongo_replica_set_client import PRIMARY, SECONDARY, OTHER from pymongo.mongo_replica_set_client import _partition_node, have_gevent from pymongo.database import Database from pymongo.pool import SocketInfo from pymongo.errors import (AutoReconnect, ConfigurationError, ConnectionFailure, InvalidName, OperationFailure, InvalidOperation) from test import version, port, pair from test.utils import ( delay, assertReadFrom, assertReadFromAll, read_from_which_host, assertRaisesExactly, TestRequestMixin, one, server_started_with_auth, pools_from_rs_client, get_pool) class TestReplicaSetClientAgainstStandalone(unittest.TestCase): """This is a funny beast -- we want to run tests for MongoReplicaSetClient but only if the database at DB_IP and DB_PORT is a standalone. """ def setUp(self): client = MongoClient(pair) response = client.admin.command('ismaster') if 'setName' in response: raise SkipTest("Connected to a replica set, not a standalone mongod") def test_connect(self): self.assertRaises(ConfigurationError, MongoReplicaSetClient, pair, replicaSet='anything', connectTimeoutMS=600) class TestReplicaSetClientBase(unittest.TestCase): def setUp(self): client = MongoClient(pair) response = client.admin.command('ismaster') if 'setName' in response: self.name = str(response['setName']) self.w = len(response['hosts']) self.hosts = set([_partition_node(h) for h in response["hosts"]]) self.arbiters = set([_partition_node(h) for h in response.get("arbiters", [])]) repl_set_status = client.admin.command('replSetGetStatus') primary_info = [ m for m in repl_set_status['members'] if m['stateStr'] == 'PRIMARY' ][0] self.primary = _partition_node(primary_info['name']) self.secondaries = [ _partition_node(m['name']) for m in repl_set_status['members'] if m['stateStr'] == 'SECONDARY' ] else: raise SkipTest("Not connected to a replica set") def _get_client(self, **kwargs): return MongoReplicaSetClient(pair, replicaSet=self.name, **kwargs) class TestReplicaSetClient(TestReplicaSetClientBase, TestRequestMixin): def test_init_disconnected(self): c = self._get_client(_connect=False) # No errors c.seeds c.hosts c.arbiters c.is_mongos c.max_pool_size c.use_greenlets c.get_document_class c.tz_aware c.max_bson_size c.auto_start_request self.assertFalse(c.primary) self.assertFalse(c.secondaries) c.pymongo_test.test.find_one() # Auto-connect for read. self.assertTrue(c.primary) self.assertTrue(c.secondaries) c = self._get_client(_connect=False) c.pymongo_test.test.update({}, {}) # Auto-connect for write. self.assertTrue(c.primary) c = self._get_client(_connect=False) c.pymongo_test.test.insert({}) # Auto-connect for write. self.assertTrue(c.primary) c = self._get_client(_connect=False) c.pymongo_test.test.remove({}) # Auto-connect for write. self.assertTrue(c.primary) c = MongoReplicaSetClient( "somedomainthatdoesntexist.org", replicaSet="rs", connectTimeoutMS=1, _connect=False) self.assertRaises(ConnectionFailure, c.pymongo_test.test.find_one) def test_init_disconnected_with_auth_failure(self): c = MongoReplicaSetClient( "mongodb://user:pass@somedomainthatdoesntexist", replicaSet="rs", connectTimeoutMS=1, _connect=False) self.assertRaises(ConnectionFailure, c.pymongo_test.test.find_one) def test_init_disconnected_with_auth(self): c = self._get_client() c.admin.system.users.remove({}) c.pymongo_test.system.users.remove({}) try: c.admin.add_user("admin", "pass") c.admin.authenticate("admin", "pass") c.pymongo_test.add_user("user", "pass") # Auth with lazy connection. host = one(self.hosts) uri = "mongodb://user:pass@%s:%d/pymongo_test?replicaSet=%s" % ( host[0], host[1], self.name) authenticated_client = MongoReplicaSetClient(uri, _connect=False) authenticated_client.pymongo_test.test.find_one() # Wrong password. bad_uri = "mongodb://user:wrong@%s:%d/pymongo_test?replicaSet=%s" % ( host[0], host[1], self.name) bad_client = MongoReplicaSetClient(bad_uri, _connect=False) self.assertRaises( OperationFailure, bad_client.pymongo_test.test.find_one) finally: # Clean up. c.admin.system.users.remove({}) c.pymongo_test.system.users.remove({}) def test_connect(self): assertRaisesExactly(ConnectionFailure, MongoReplicaSetClient, "somedomainthatdoesntexist.org:27017", replicaSet=self.name, connectTimeoutMS=600) self.assertRaises(ConfigurationError, MongoReplicaSetClient, pair, replicaSet='fdlksjfdslkjfd') self.assertTrue(MongoReplicaSetClient(pair, replicaSet=self.name)) def test_repr(self): client = self._get_client() # Quirk: the RS client makes a frozenset of hosts from a dict's keys, # so we must do the same to achieve the same order. host_dict = dict([(host, 1) for host in self.hosts]) hosts_set = frozenset(host_dict) hosts_repr = ', '.join([ repr(unicode('%s:%s' % host)) for host in hosts_set]) self.assertEqual(repr(client), "MongoReplicaSetClient([%s])" % hosts_repr) def test_properties(self): c = MongoReplicaSetClient(pair, replicaSet=self.name) c.admin.command('ping') self.assertEqual(c.primary, self.primary) self.assertEqual(c.hosts, self.hosts) self.assertEqual(c.arbiters, self.arbiters) self.assertEqual(c.max_pool_size, 100) self.assertEqual(c.document_class, dict) self.assertEqual(c.tz_aware, False) # Make sure MRSC's properties are copied to Database and Collection for obj in c, c.pymongo_test, c.pymongo_test.test: self.assertEqual(obj.read_preference, ReadPreference.PRIMARY) self.assertEqual(obj.tag_sets, [{}]) self.assertEqual(obj.secondary_acceptable_latency_ms, 15) self.assertEqual(obj.slave_okay, False) self.assertEqual(obj.write_concern, {}) cursor = c.pymongo_test.test.find() self.assertEqual( ReadPreference.PRIMARY, cursor._Cursor__read_preference) self.assertEqual([{}], cursor._Cursor__tag_sets) self.assertEqual(15, cursor._Cursor__secondary_acceptable_latency_ms) self.assertEqual(False, cursor._Cursor__slave_okay) c.close() tag_sets = [{'dc': 'la', 'rack': '2'}, {'foo': 'bar'}] c = MongoReplicaSetClient(pair, replicaSet=self.name, max_pool_size=25, document_class=SON, tz_aware=True, slaveOk=False, read_preference=ReadPreference.SECONDARY, tag_sets=copy.deepcopy(tag_sets), secondary_acceptable_latency_ms=77) c.admin.command('ping') self.assertEqual(c.primary, self.primary) self.assertEqual(c.hosts, self.hosts) self.assertEqual(c.arbiters, self.arbiters) self.assertEqual(c.max_pool_size, 25) self.assertEqual(c.document_class, SON) self.assertEqual(c.tz_aware, True) for obj in c, c.pymongo_test, c.pymongo_test.test: self.assertEqual(obj.read_preference, ReadPreference.SECONDARY) self.assertEqual(obj.tag_sets, tag_sets) self.assertEqual(obj.secondary_acceptable_latency_ms, 77) self.assertEqual(obj.slave_okay, False) self.assertEqual(obj.safe, True) cursor = c.pymongo_test.test.find() self.assertEqual( ReadPreference.SECONDARY, cursor._Cursor__read_preference) self.assertEqual(tag_sets, cursor._Cursor__tag_sets) self.assertEqual(77, cursor._Cursor__secondary_acceptable_latency_ms) self.assertEqual(False, cursor._Cursor__slave_okay) cursor = c.pymongo_test.test.find( read_preference=ReadPreference.NEAREST, tag_sets=[{'dc':'ny'}, {}], secondary_acceptable_latency_ms=123) self.assertEqual( ReadPreference.NEAREST, cursor._Cursor__read_preference) self.assertEqual([{'dc':'ny'}, {}], cursor._Cursor__tag_sets) self.assertEqual(123, cursor._Cursor__secondary_acceptable_latency_ms) self.assertEqual(False, cursor._Cursor__slave_okay) if version.at_least(c, (1, 7, 4)): self.assertEqual(c.max_bson_size, 16777216) else: self.assertEqual(c.max_bson_size, 4194304) c.close() def test_use_greenlets(self): self.assertFalse( MongoReplicaSetClient(pair, replicaSet=self.name).use_greenlets) if have_gevent: self.assertTrue(MongoReplicaSetClient( pair, replicaSet=self.name, use_greenlets=True).use_greenlets) def test_get_db(self): client = self._get_client() def make_db(base, name): return base[name] self.assertRaises(InvalidName, make_db, client, "") self.assertRaises(InvalidName, make_db, client, "te$t") self.assertRaises(InvalidName, make_db, client, "te.t") self.assertRaises(InvalidName, make_db, client, "te\\t") self.assertRaises(InvalidName, make_db, client, "te/t") self.assertRaises(InvalidName, make_db, client, "te st") self.assertTrue(isinstance(client.test, Database)) self.assertEqual(client.test, client["test"]) self.assertEqual(client.test, Database(client, "test")) client.close() def test_auto_reconnect_exception_when_read_preference_is_secondary(self): c = self._get_client() db = c.pymongo_test def raise_socket_error(*args, **kwargs): raise socket.error old_sendall = socket.socket.sendall socket.socket.sendall = raise_socket_error try: cursor = db.test.find(read_preference=ReadPreference.SECONDARY) self.assertRaises(AutoReconnect, cursor.next) finally: socket.socket.sendall = old_sendall def test_lazy_auth_raises_operation_failure(self): # Check if we have the prerequisites to run this test. c = self._get_client() if not server_started_with_auth(c): raise SkipTest('Authentication is not enabled on server') lazy_client = MongoReplicaSetClient( "mongodb://user:wrong@%s/pymongo_test" % pair, replicaSet=self.name, _connect=False) assertRaisesExactly( OperationFailure, lazy_client.test.collection.find_one) def test_operations(self): c = self._get_client() # Check explicitly for a case we've commonly hit in tests: # a replica set is started with a tiny oplog, a previous # test does a big insert that leaves the secondaries # permanently "RECOVERING", and our insert(w=self.w) hangs # forever. rs_status = c.admin.command('replSetGetStatus') members = rs_status['members'] self.assertFalse( [m for m in members if m['stateStr'] == 'RECOVERING'], "Replica set is recovering, try a larger oplogSize next time" ) db = c.pymongo_test db.test.remove({}) self.assertEqual(0, db.test.count()) db.test.insert({'foo': 'x'}, w=self.w, wtimeout=10000) self.assertEqual(1, db.test.count()) cursor = db.test.find() doc = cursor.next() self.assertEqual('x', doc['foo']) # Ensure we read from the primary self.assertEqual(c.primary, cursor._Cursor__connection_id) cursor = db.test.find(read_preference=ReadPreference.SECONDARY) doc = cursor.next() self.assertEqual('x', doc['foo']) # Ensure we didn't read from the primary self.assertTrue(cursor._Cursor__connection_id in c.secondaries) self.assertEqual(1, db.test.count()) db.test.remove({}) self.assertEqual(0, db.test.count()) db.test.drop() c.close() def test_database_names(self): client = self._get_client() client.pymongo_test.test.save({"dummy": u"object"}) client.pymongo_test_mike.test.save({"dummy": u"object"}) dbs = client.database_names() self.assertTrue("pymongo_test" in dbs) self.assertTrue("pymongo_test_mike" in dbs) client.close() def test_drop_database(self): client = self._get_client() self.assertRaises(TypeError, client.drop_database, 5) self.assertRaises(TypeError, client.drop_database, None) client.pymongo_test.test.save({"dummy": u"object"}) dbs = client.database_names() self.assertTrue("pymongo_test" in dbs) client.drop_database("pymongo_test") dbs = client.database_names() self.assertTrue("pymongo_test" not in dbs) client.pymongo_test.test.save({"dummy": u"object"}) dbs = client.database_names() self.assertTrue("pymongo_test" in dbs) client.drop_database(client.pymongo_test) dbs = client.database_names() self.assertTrue("pymongo_test" not in dbs) client.close() def test_copy_db(self): c = self._get_client() # We test copy twice; once starting in a request and once not. In # either case the copy should succeed (because it starts a request # internally) and should leave us in the same state as before the copy. c.start_request() self.assertRaises(TypeError, c.copy_database, 4, "foo") self.assertRaises(TypeError, c.copy_database, "foo", 4) self.assertRaises(InvalidName, c.copy_database, "foo", "$foo") c.pymongo_test.test.drop() c.drop_database("pymongo_test1") c.drop_database("pymongo_test2") c.pymongo_test.test.insert({"foo": "bar"}) self.assertFalse("pymongo_test1" in c.database_names()) self.assertFalse("pymongo_test2" in c.database_names()) c.copy_database("pymongo_test", "pymongo_test1") # copy_database() didn't accidentally end the request self.assertTrue(c.in_request()) self.assertTrue("pymongo_test1" in c.database_names()) self.assertEqual("bar", c.pymongo_test1.test.find_one()["foo"]) c.end_request() self.assertFalse(c.in_request()) c.copy_database("pymongo_test", "pymongo_test2", pair) # copy_database() didn't accidentally restart the request self.assertFalse(c.in_request()) time.sleep(1) self.assertTrue("pymongo_test2" in c.database_names()) self.assertEqual("bar", c.pymongo_test2.test.find_one()["foo"]) if version.at_least(c, (1, 3, 3, 1)): c.drop_database("pymongo_test1") c.pymongo_test.add_user("mike", "password") self.assertRaises(OperationFailure, c.copy_database, "pymongo_test", "pymongo_test1", username="foo", password="bar") self.assertFalse("pymongo_test1" in c.database_names()) self.assertRaises(OperationFailure, c.copy_database, "pymongo_test", "pymongo_test1", username="mike", password="bar") self.assertFalse("pymongo_test1" in c.database_names()) c.copy_database("pymongo_test", "pymongo_test1", username="mike", password="password") self.assertTrue("pymongo_test1" in c.database_names()) time.sleep(2) self.assertEqual("bar", c.pymongo_test1.test.find_one()["foo"]) c.close() def test_get_default_database(self): host = one(self.hosts) uri = "mongodb://%s:%d/foo?replicaSet=%s" % ( host[0], host[1], self.name) c = MongoReplicaSetClient(uri, _connect=False) self.assertEqual(Database(c, 'foo'), c.get_default_database()) def test_get_default_database_error(self): host = one(self.hosts) # URI with no database. uri = "mongodb://%s:%d/?replicaSet=%s" % ( host[0], host[1], self.name) c = MongoReplicaSetClient(uri, _connect=False) self.assertRaises(ConfigurationError, c.get_default_database) def test_get_default_database_with_authsource(self): # Ensure we distinguish database name from authSource. host = one(self.hosts) uri = "mongodb://%s:%d/foo?replicaSet=%s&authSource=src" % ( host[0], host[1], self.name) c = MongoReplicaSetClient(uri, _connect=False) self.assertEqual(Database(c, 'foo'), c.get_default_database()) def test_iteration(self): client = self._get_client() def iterate(): [a for a in client] self.assertRaises(TypeError, iterate) client.close() def test_disconnect(self): c = self._get_client() coll = c.pymongo_test.bar c.disconnect() c.disconnect() coll.count() c.disconnect() c.disconnect() coll.count() def test_fork(self): # Test using a client before and after a fork. if sys.platform == "win32": raise SkipTest("Can't fork on Windows") try: from multiprocessing import Process, Pipe except ImportError: raise SkipTest("No multiprocessing module") db = self._get_client().pymongo_test # Failure occurs if the client is used before the fork db.test.find_one() def loop(pipe): while True: try: db.test.insert({"a": "b"}) for _ in db.test.find(): pass except: traceback.print_exc() pipe.send(True) os._exit(1) cp1, cc1 = Pipe() cp2, cc2 = Pipe() p1 = Process(target=loop, args=(cc1,)) p2 = Process(target=loop, args=(cc2,)) p1.start() p2.start() p1.join(1) p2.join(1) p1.terminate() p2.terminate() p1.join() p2.join() cc1.close() cc2.close() # recv will only have data if the subprocess failed try: cp1.recv() self.fail() except EOFError: pass try: cp2.recv() self.fail() except EOFError: pass db.connection.close() def test_fork_and_schedule_refresh(self): # After a fork the monitor thread is gone. # Verify that schedule_refresh throws InvalidOperation. if sys.platform == "win32": raise SkipTest("Can't fork on Windows") try: from multiprocessing import Process, Pipe except ImportError: raise SkipTest("No multiprocessing module") client = self._get_client() db = client.pymongo_test def f(pipe): try: # Trigger a refresh. self.assertRaises(InvalidOperation, client.disconnect) except: traceback.print_exc() pipe.send(True) os._exit(1) cp, cc = Pipe() p = Process(target=f, args=(cc,)) p.start() p.join(10) p.terminate() p.join() cc.close() # recv will only have data if the subprocess failed try: cp.recv() self.fail() except EOFError: pass db.connection.close() def test_document_class(self): c = self._get_client() db = c.pymongo_test db.test.insert({"x": 1}) self.assertEqual(dict, c.document_class) self.assertTrue(isinstance(db.test.find_one(), dict)) self.assertFalse(isinstance(db.test.find_one(), SON)) c.document_class = SON self.assertEqual(SON, c.document_class) self.assertTrue(isinstance(db.test.find_one(), SON)) self.assertFalse(isinstance(db.test.find_one(as_class=dict), SON)) c.close() c = self._get_client(document_class=SON) db = c.pymongo_test self.assertEqual(SON, c.document_class) self.assertTrue(isinstance(db.test.find_one(), SON)) self.assertFalse(isinstance(db.test.find_one(as_class=dict), SON)) c.document_class = dict self.assertEqual(dict, c.document_class) self.assertTrue(isinstance(db.test.find_one(), dict)) self.assertFalse(isinstance(db.test.find_one(), SON)) c.close() def test_network_timeout_validation(self): c = self._get_client(socketTimeoutMS=10 * 1000) self.assertEqual(10, c._MongoReplicaSetClient__net_timeout) c = self._get_client(socketTimeoutMS=None) self.assertEqual(None, c._MongoReplicaSetClient__net_timeout) self.assertRaises(ConfigurationError, self._get_client, socketTimeoutMS=0) self.assertRaises(ConfigurationError, self._get_client, socketTimeoutMS=-1) self.assertRaises(ConfigurationError, self._get_client, socketTimeoutMS=1e10) self.assertRaises(ConfigurationError, self._get_client, socketTimeoutMS='foo') # network_timeout is gone from MongoReplicaSetClient, remains in # deprecated ReplicaSetConnection self.assertRaises(ConfigurationError, self._get_client, network_timeout=10) def test_network_timeout(self): no_timeout = self._get_client() timeout_sec = 1 timeout = self._get_client(socketTimeoutMS=timeout_sec*1000) no_timeout.pymongo_test.drop_collection("test") no_timeout.pymongo_test.test.insert({"x": 1}) # A $where clause that takes a second longer than the timeout where_func = delay(1 + timeout_sec) def get_x(db): doc = db.test.find().where(where_func).next() return doc["x"] self.assertEqual(1, get_x(no_timeout.pymongo_test)) self.assertRaises(ConnectionFailure, get_x, timeout.pymongo_test) def get_x_timeout(db, t): doc = db.test.find(network_timeout=t).where(where_func).next() return doc["x"] self.assertEqual(1, get_x_timeout(timeout.pymongo_test, None)) self.assertRaises(ConnectionFailure, get_x_timeout, no_timeout.pymongo_test, 0.1) no_timeout.close() timeout.close() def test_socket_error_marks_member_down(self): # A socket error (besides timeout) changes a member's state to "down". c = self._get_client() collection = c.pymongo_test.test collection.insert({}, w=self.w) previous_writer = c._MongoReplicaSetClient__rs_state.writer def kill_sockets(): for pool in pools_from_rs_client(c): for socket_info in pool.sockets: socket_info.sock.close() kill_sockets() # Query the primary. self.assertRaises(ConnectionFailure, collection.find_one) # primary_member returns None if primary is marked "down". rs_state = c._MongoReplicaSetClient__rs_state self.assertEqual(None, rs_state.writer) self.assertFalse(rs_state.get(previous_writer).up) collection.find_one() # No error, we recovered. rs_state = c._MongoReplicaSetClient__rs_state self.assertTrue(rs_state.get(rs_state.writer).up) kill_sockets() # Query secondaries. Client marks them "down" as they fail, and tries # up to 3 of them before raising. self.assertRaises( ConnectionFailure, collection.find_one, read_preference=SECONDARY) # Secondaries were either removed from state or marked "down". rs_state = c._MongoReplicaSetClient__rs_state for secondary_host in rs_state.secondaries: self.assertFalse(rs_state.get(secondary_host).up) def test_timeout_does_not_mark_member_down(self): # If a query times out, the RS client shouldn't mark the member "down". c = self._get_client(socketTimeoutMS=3000) collection = c.pymongo_test.test collection.insert({}, w=self.w) # Query the primary. self.assertRaises( ConnectionFailure, collection.find_one, {'$where': delay(5)}) # primary_member returns None if primary is marked "down". rs_state = c._MongoReplicaSetClient__rs_state self.assertTrue(rs_state.primary_member) collection.find_one() # No error. # Query the secondary. self.assertRaises( ConnectionFailure, collection.find_one, {'$where': delay(5)}, read_preference=SECONDARY) rs_state = c._MongoReplicaSetClient__rs_state secondary_host = one(rs_state.secondaries) self.assertTrue(rs_state.get(secondary_host).up) collection.find_one(read_preference=SECONDARY) # No error. def test_waitQueueTimeoutMS(self): client = self._get_client(waitQueueTimeoutMS=2000) pool = get_pool(client) self.assertEqual(pool.wait_queue_timeout, 2) def test_waitQueueMultiple(self): client = self._get_client(max_pool_size=3, waitQueueMultiple=2) pool = get_pool(client) self.assertEqual(pool.wait_queue_multiple, 2) self.assertEqual(pool._socket_semaphore.waiter_semaphore.counter, 6) def test_tz_aware(self): self.assertRaises(ConfigurationError, MongoReplicaSetClient, tz_aware='foo', replicaSet=self.name) aware = self._get_client(tz_aware=True) naive = self._get_client() aware.pymongo_test.drop_collection("test") now = datetime.datetime.utcnow() aware.pymongo_test.test.insert({"x": now}) time.sleep(1) self.assertEqual(None, naive.pymongo_test.test.find_one()["x"].tzinfo) self.assertEqual(utc, aware.pymongo_test.test.find_one()["x"].tzinfo) self.assertEqual( aware.pymongo_test.test.find_one()["x"].replace(tzinfo=None), naive.pymongo_test.test.find_one()["x"]) def test_ipv6(self): try: client = MongoReplicaSetClient("[::1]:%d" % (port,), replicaSet=self.name) except: # Either mongod was started without --ipv6 # or the OS doesn't support it (or both). raise SkipTest("No IPv6") # Try a few simple things client = MongoReplicaSetClient("mongodb://[::1]:%d" % (port,), replicaSet=self.name) client = MongoReplicaSetClient("mongodb://[::1]:%d/?safe=true;" "replicaSet=%s" % (port, self.name)) client = MongoReplicaSetClient("[::1]:%d,localhost:" "%d" % (port, port), replicaSet=self.name) client = MongoReplicaSetClient("localhost:%d,[::1]:" "%d" % (port, port), replicaSet=self.name) client.pymongo_test.test.save({"dummy": u"object"}) client.pymongo_test_bernie.test.save({"dummy": u"object"}) dbs = client.database_names() self.assertTrue("pymongo_test" in dbs) self.assertTrue("pymongo_test_bernie" in dbs) client.close() def _test_kill_cursor_explicit(self, read_pref): c = self._get_client(read_preference=read_pref) db = c.pymongo_test db.drop_collection("test") test = db.test test.insert([{"i": i} for i in range(20)], w=1 + len(c.secondaries)) # Partially evaluate cursor so it's left alive, then kill it cursor = test.find().batch_size(10) cursor.next() self.assertNotEqual(0, cursor.cursor_id) connection_id = cursor._Cursor__connection_id writer = c._MongoReplicaSetClient__rs_state.writer if read_pref == ReadPreference.PRIMARY: msg = "Expected cursor's connection_id to be %s, got %s" % ( writer, connection_id) self.assertEqual(connection_id, writer, msg) else: self.assertNotEqual(connection_id, writer, "Expected cursor's connection_id not to be primary") cursor_id = cursor.cursor_id # Cursor dead on server - trigger a getMore on the same cursor_id and # check that the server returns an error. cursor2 = cursor.clone() cursor2._Cursor__id = cursor_id if (sys.platform.startswith('java') or 'PyPy' in sys.version): # Explicitly kill cursor. cursor.close() else: # Implicitly kill it in CPython. del cursor self.assertRaises(OperationFailure, lambda: list(cursor2)) def test_kill_cursor_explicit_primary(self): self._test_kill_cursor_explicit(ReadPreference.PRIMARY) def test_kill_cursor_explicit_secondary(self): self._test_kill_cursor_explicit(ReadPreference.SECONDARY) def test_interrupt_signal(self): if sys.platform.startswith('java'): raise SkipTest("Can't test interrupts in Jython") # Test fix for PYTHON-294 -- make sure client closes its socket if it # gets an interrupt while waiting to recv() from it. c = self._get_client() db = c.pymongo_test # A $where clause which takes 1.5 sec to execute where = delay(1.5) # Need exactly 1 document so find() will execute its $where clause once db.drop_collection('foo') db.foo.insert({'_id': 1}) old_signal_handler = None try: # Platform-specific hacks for raising a KeyboardInterrupt on the main # thread while find() is in-progress: On Windows, SIGALRM is unavailable # so we use second thread. In our Bamboo setup on Linux, the thread # technique causes an error in the test at sock.recv(): # TypeError: 'int' object is not callable # We don't know what causes this in Bamboo, so we hack around it. if sys.platform == 'win32': def interrupter(): time.sleep(0.25) # Raises KeyboardInterrupt in the main thread thread.interrupt_main() thread.start_new_thread(interrupter, ()) else: # Convert SIGALRM to SIGINT -- it's hard to schedule a SIGINT for one # second in the future, but easy to schedule SIGALRM. def sigalarm(num, frame): raise KeyboardInterrupt old_signal_handler = signal.signal(signal.SIGALRM, sigalarm) signal.alarm(1) raised = False try: # Will be interrupted by a KeyboardInterrupt. db.foo.find({'$where': where}).next() except KeyboardInterrupt: raised = True # Can't use self.assertRaises() because it doesn't catch system # exceptions self.assertTrue(raised, "Didn't raise expected ConnectionFailure") # Raises AssertionError due to PYTHON-294 -- Mongo's response to the # previous find() is still waiting to be read on the socket, so the # request id's don't match. self.assertEqual( {'_id': 1}, db.foo.find().next() ) finally: if old_signal_handler: signal.signal(signal.SIGALRM, old_signal_handler) def test_operation_failure_without_request(self): # Ensure MongoReplicaSetClient doesn't close socket after it gets an # error response to getLastError. PYTHON-395. c = self._get_client(auto_start_request=False) pool = get_pool(c) self.assertEqual(1, len(pool.sockets)) old_sock_info = iter(pool.sockets).next() c.pymongo_test.test.drop() c.pymongo_test.test.insert({'_id': 'foo'}) self.assertRaises( OperationFailure, c.pymongo_test.test.insert, {'_id': 'foo'}) self.assertEqual(1, len(pool.sockets)) new_sock_info = iter(pool.sockets).next() self.assertEqual(old_sock_info, new_sock_info) c.close() def test_operation_failure_with_request(self): # Ensure MongoReplicaSetClient doesn't close socket after it gets an # error response to getLastError. PYTHON-395. c = self._get_client(auto_start_request=True) c.pymongo_test.test.find_one() pool = get_pool(c) # Client reserved a socket for this thread self.assertTrue(isinstance(pool._get_request_state(), SocketInfo)) old_sock_info = pool._get_request_state() c.pymongo_test.test.drop() c.pymongo_test.test.insert({'_id': 'foo'}) self.assertRaises( OperationFailure, c.pymongo_test.test.insert, {'_id': 'foo'}) # OperationFailure doesn't affect the request socket self.assertEqual(old_sock_info, pool._get_request_state()) c.close() def test_auto_start_request(self): for bad_horrible_value in (None, 5, 'hi!'): self.assertRaises( (TypeError, ConfigurationError), lambda: self._get_client(auto_start_request=bad_horrible_value) ) client = self._get_client(auto_start_request=True) self.assertTrue(client.auto_start_request) pools = pools_from_rs_client(client) self.assertInRequestAndSameSock(client, pools) primary_pool = get_pool(client) # Trigger the RSC to actually start a request on primary pool client.pymongo_test.test.find_one() self.assertTrue(primary_pool.in_request()) # Trigger the RSC to actually start a request on secondary pool cursor = client.pymongo_test.test.find( read_preference=ReadPreference.SECONDARY) try: cursor.next() except StopIteration: # No results, no problem pass secondary = cursor._Cursor__connection_id rs_state = client._MongoReplicaSetClient__rs_state secondary_pool = rs_state.get(secondary).pool self.assertTrue(secondary_pool.in_request()) client.end_request() self.assertNotInRequestAndDifferentSock(client, pools) for pool in pools: self.assertFalse(pool.in_request()) client.start_request() self.assertInRequestAndSameSock(client, pools) client.close() client = self._get_client() pools = pools_from_rs_client(client) self.assertNotInRequestAndDifferentSock(client, pools) client.start_request() self.assertInRequestAndSameSock(client, pools) client.end_request() self.assertNotInRequestAndDifferentSock(client, pools) client.close() def test_nested_request(self): client = self._get_client(auto_start_request=True) try: pools = pools_from_rs_client(client) self.assertTrue(client.in_request()) # Start and end request - we're still in "outer" original request client.start_request() self.assertInRequestAndSameSock(client, pools) client.end_request() self.assertInRequestAndSameSock(client, pools) # Double-nesting client.start_request() client.start_request() self.assertEqual( 3, client._MongoReplicaSetClient__request_counter.get()) for pool in pools: # MRSC only called start_request() once per pool, although its # own counter is 2. self.assertEqual(1, pool._request_counter.get()) client.end_request() client.end_request() self.assertInRequestAndSameSock(client, pools) self.assertEqual( 1, client._MongoReplicaSetClient__request_counter.get()) for pool in pools: self.assertEqual(1, pool._request_counter.get()) # Finally, end original request client.end_request() for pool in pools: self.assertFalse(pool.in_request()) self.assertNotInRequestAndDifferentSock(client, pools) finally: client.close() def test_request_threads(self): client = self._get_client() try: pools = pools_from_rs_client(client) self.assertNotInRequestAndDifferentSock(client, pools) started_request, ended_request = threading.Event(), threading.Event() checked_request = threading.Event() thread_done = [False] # Starting a request in one thread doesn't put the other thread in a # request def f(): self.assertNotInRequestAndDifferentSock(client, pools) client.start_request() self.assertInRequestAndSameSock(client, pools) started_request.set() checked_request.wait() checked_request.clear() self.assertInRequestAndSameSock(client, pools) client.end_request() self.assertNotInRequestAndDifferentSock(client, pools) ended_request.set() checked_request.wait() thread_done[0] = True t = threading.Thread(target=f) t.setDaemon(True) t.start() started_request.wait() self.assertNotInRequestAndDifferentSock(client, pools) checked_request.set() ended_request.wait() self.assertNotInRequestAndDifferentSock(client, pools) checked_request.set() t.join() self.assertNotInRequestAndDifferentSock(client, pools) self.assertTrue(thread_done[0], "Thread didn't complete") finally: client.close() def test_schedule_refresh(self): client = self._get_client() new_rs_state = rs_state = client._MongoReplicaSetClient__rs_state for host in rs_state.hosts: new_rs_state = new_rs_state.clone_with_host_down(host, 'error!') client._MongoReplicaSetClient__rs_state = new_rs_state client._MongoReplicaSetClient__schedule_refresh(sync=True) rs_state = client._MongoReplicaSetClient__rs_state for member in rs_state.members: self.assertTrue( member.up, "MongoReplicaSetClient didn't detect member is up") client.close() def test_pinned_member(self): latency = 1000 * 1000 client = self._get_client(secondary_acceptable_latency_ms=latency) host = read_from_which_host(client, ReadPreference.SECONDARY) self.assertTrue(host in client.secondaries) # No pinning since we're not in a request assertReadFromAll( self, client, client.secondaries, ReadPreference.SECONDARY, None, latency) assertReadFromAll( self, client, list(client.secondaries) + [client.primary], ReadPreference.NEAREST, None, latency) client.start_request() host = read_from_which_host(client, ReadPreference.SECONDARY) self.assertTrue(host in client.secondaries) assertReadFrom(self, client, host, ReadPreference.SECONDARY) # Changing any part of read preference (mode, tag_sets, latency) # unpins the current host and pins to a new one primary = client.primary assertReadFrom(self, client, primary, ReadPreference.PRIMARY_PREFERRED) host = read_from_which_host(client, ReadPreference.NEAREST) assertReadFrom(self, client, host, ReadPreference.NEAREST) assertReadFrom(self, client, primary, ReadPreference.PRIMARY_PREFERRED) host = read_from_which_host(client, ReadPreference.SECONDARY_PREFERRED) self.assertTrue(host in client.secondaries) assertReadFrom(self, client, host, ReadPreference.SECONDARY_PREFERRED) # Unpin client.end_request() assertReadFromAll( self, client, list(client.secondaries) + [client.primary], ReadPreference.NEAREST, None, latency) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_son.py000066400000000000000000000127631223300253600164060ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tests for the son module.""" import copy import pickle import re import sys import unittest sys.path[0:0] = [""] from nose.plugins.skip import SkipTest from bson.py3compat import b from bson.son import SON class TestSON(unittest.TestCase): def setUp(self): pass def test_ordered_dict(self): a1 = SON() a1["hello"] = "world" a1["mike"] = "awesome" a1["hello_"] = "mike" self.assertEqual(a1.items(), [("hello", "world"), ("mike", "awesome"), ("hello_", "mike")]) b2 = SON({"hello": "world"}) self.assertEqual(b2["hello"], "world") self.assertRaises(KeyError, lambda: b2["goodbye"]) def test_equality(self): a1 = SON({"hello": "world"}) b2 = SON((('hello', 'world'), ('mike', 'awesome'), ('hello_', 'mike'))) self.assertEqual(a1, SON({"hello": "world"})) self.assertEqual(b2, SON((('hello', 'world'), ('mike', 'awesome'), ('hello_', 'mike')))) self.assertEqual(b2, dict((('hello_', 'mike'), ('mike', 'awesome'), ('hello', 'world')))) self.assertNotEqual(a1, b2) self.assertNotEqual(b2, SON((('hello_', 'mike'), ('mike', 'awesome'), ('hello', 'world')))) # Explicitly test inequality self.assertFalse(a1 != SON({"hello": "world"})) self.assertFalse(b2 != SON((('hello', 'world'), ('mike', 'awesome'), ('hello_', 'mike')))) self.assertFalse(b2 != dict((('hello_', 'mike'), ('mike', 'awesome'), ('hello', 'world')))) def test_to_dict(self): a1 = SON() b2 = SON([("blah", SON())]) c3 = SON([("blah", [SON()])]) d4 = SON([("blah", {"foo": SON()})]) self.assertEqual({}, a1.to_dict()) self.assertEqual({"blah": {}}, b2.to_dict()) self.assertEqual({"blah": [{}]}, c3.to_dict()) self.assertEqual({"blah": {"foo": {}}}, d4.to_dict()) self.assertEqual(dict, a1.to_dict().__class__) self.assertEqual(dict, b2.to_dict()["blah"].__class__) self.assertEqual(dict, c3.to_dict()["blah"][0].__class__) self.assertEqual(dict, d4.to_dict()["blah"]["foo"].__class__) def test_pickle(self): simple_son = SON([]) complex_son = SON([('son', simple_son), ('list', [simple_son, simple_son])]) for protocol in xrange(pickle.HIGHEST_PROTOCOL + 1): pickled = pickle.loads(pickle.dumps(complex_son, protocol=protocol)) self.assertEqual(pickled['son'], pickled['list'][0]) self.assertEqual(pickled['son'], pickled['list'][1]) def test_pickle_backwards_compatability(self): # For a full discussion see http://bugs.python.org/issue6137 if sys.version.startswith('3.0'): raise SkipTest("Python 3.0.x can't unpickle " "objects pickled in Python 2.x.") # This string was generated by pickling a SON object in pymongo # version 2.1.1 pickled_with_2_1_1 = b( "ccopy_reg\n_reconstructor\np0\n(cbson.son\nSON\np1\n" "c__builtin__\ndict\np2\n(dp3\ntp4\nRp5\n(dp6\n" "S'_SON__keys'\np7\n(lp8\nsb." ) son_2_1_1 = pickle.loads(pickled_with_2_1_1) self.assertEqual(son_2_1_1, SON([])) def test_copying(self): simple_son = SON([]) complex_son = SON([('son', simple_son), ('list', [simple_son, simple_son])]) regex_son = SON([("x", re.compile("^hello.*"))]) reflexive_son = SON([('son', simple_son)]) reflexive_son["reflexive"] = reflexive_son simple_son1 = copy.copy(simple_son) self.assertEqual(simple_son, simple_son1) complex_son1 = copy.copy(complex_son) self.assertEqual(complex_son, complex_son1) regex_son1 = copy.copy(regex_son) self.assertEqual(regex_son, regex_son1) reflexive_son1 = copy.copy(reflexive_son) self.assertEqual(reflexive_son, reflexive_son1) # Test deepcopying simple_son1 = copy.deepcopy(simple_son) self.assertEqual(simple_son, simple_son1) regex_son1 = copy.deepcopy(regex_son) self.assertEqual(regex_son, regex_son1) complex_son1 = copy.deepcopy(complex_son) self.assertEqual(complex_son, complex_son1) reflexive_son1 = copy.deepcopy(reflexive_son) self.assertEqual(reflexive_son.keys(), reflexive_son1.keys()) self.assertEqual(id(reflexive_son1), id(reflexive_son1["reflexive"])) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_son_manipulator.py000066400000000000000000000077661223300253600210300ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tests for SONManipulators. """ import unittest import sys sys.path[0:0] = [""] from bson.objectid import ObjectId from bson.son import SON from pymongo.database import Database from pymongo.son_manipulator import (NamespaceInjector, ObjectIdInjector, ObjectIdShuffler, SONManipulator) from test.test_client import get_client from test import qcheck class TestSONManipulator(unittest.TestCase): def setUp(self): self.db = Database(get_client(), "pymongo_test") def test_basic(self): manip = SONManipulator() collection = self.db.test def incoming_is_identity(son): return son == manip.transform_incoming(son, collection) qcheck.check_unittest(self, incoming_is_identity, qcheck.gen_mongo_dict(3)) def outgoing_is_identity(son): return son == manip.transform_outgoing(son, collection) qcheck.check_unittest(self, outgoing_is_identity, qcheck.gen_mongo_dict(3)) def test_id_injection(self): manip = ObjectIdInjector() collection = self.db.test def incoming_adds_id(son): son = manip.transform_incoming(son, collection) assert "_id" in son return True qcheck.check_unittest(self, incoming_adds_id, qcheck.gen_mongo_dict(3)) def outgoing_is_identity(son): return son == manip.transform_outgoing(son, collection) qcheck.check_unittest(self, outgoing_is_identity, qcheck.gen_mongo_dict(3)) def test_id_shuffling(self): manip = ObjectIdShuffler() collection = self.db.test def incoming_moves_id(son_in): son = manip.transform_incoming(son_in, collection) if not "_id" in son: return True for (k, v) in son.items(): self.assertEqual(k, "_id") break # Key order matters in SON equality test, # matching collections.OrderedDict if isinstance(son_in, SON): return son_in.to_dict() == son.to_dict() return son_in == son self.assertTrue(incoming_moves_id({})) self.assertTrue(incoming_moves_id({"_id": 12})) self.assertTrue(incoming_moves_id({"hello": "world", "_id": 12})) self.assertTrue(incoming_moves_id(SON([("hello", "world"), ("_id", 12)]))) def outgoing_is_identity(son): return son == manip.transform_outgoing(son, collection) qcheck.check_unittest(self, outgoing_is_identity, qcheck.gen_mongo_dict(3)) def test_ns_injection(self): manip = NamespaceInjector() collection = self.db.test def incoming_adds_ns(son): son = manip.transform_incoming(son, collection) assert "_ns" in son return son["_ns"] == collection.name qcheck.check_unittest(self, incoming_adds_ns, qcheck.gen_mongo_dict(3)) def outgoing_is_identity(son): return son == manip.transform_outgoing(son, collection) qcheck.check_unittest(self, outgoing_is_identity, qcheck.gen_mongo_dict(3)) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_ssl.py000066400000000000000000000424631223300253600164100ustar00rootroot00000000000000# Copyright 2011-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tests for SSL support.""" import os import socket import sys import unittest sys.path[0:0] = [""] from urllib import quote_plus from nose.plugins.skip import SkipTest from pymongo import MongoClient, MongoReplicaSetClient from pymongo.common import HAS_SSL from pymongo.errors import (ConfigurationError, ConnectionFailure, OperationFailure) from test import host, port, pair, version from test.utils import get_command_line CERT_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'certificates') CLIENT_PEM = os.path.join(CERT_PATH, 'client.pem') CA_PEM = os.path.join(CERT_PATH, 'ca.pem') SIMPLE_SSL = False CERT_SSL = False SERVER_IS_RESOLVABLE = False MONGODB_X509_USERNAME = os.environ.get('MONGODB_X509_USERNAME') # To fully test this start a mongod instance (built with SSL support) like so: # mongod --dbpath /path/to/data/directory --sslOnNormalPorts \ # --sslPEMKeyFile /path/to/mongo/jstests/libs/server.pem \ # --sslCAFile /path/to/mongo/jstests/libs/ca.pem \ # --sslCRLFile /path/to/mongo/jstests/libs/crl.pem \ # --sslWeakCertificateValidation # Also, make sure you have 'server' as an alias for localhost in /etc/hosts # # Note: For all tests to pass with MongoReplicaSetConnection the replica # set configuration must use 'server' for the hostname of all hosts. def is_server_resolvable(): """Returns True if 'server' is resolvable.""" socket_timeout = socket.getdefaulttimeout() socket.setdefaulttimeout(1) try: try: socket.gethostbyname('server') return True except socket.error: return False finally: socket.setdefaulttimeout(socket_timeout) if HAS_SSL: import ssl # Check this all once instead of before every test method below. # Is MongoDB configured for SSL? try: MongoClient(host, port, connectTimeoutMS=100, ssl=True) SIMPLE_SSL = True except ConnectionFailure: pass if SIMPLE_SSL: # Is MongoDB configured with server.pem, ca.pem, and crl.pem from # mongodb jstests/lib? try: MongoClient(host, port, connectTimeoutMS=100, ssl=True, ssl_certfile=CLIENT_PEM) CERT_SSL = True except ConnectionFailure: pass if CERT_SSL: SERVER_IS_RESOLVABLE = is_server_resolvable() class TestClientSSL(unittest.TestCase): def test_no_ssl_module(self): # Test that ConfigurationError is raised if the ssl # module isn't available. if HAS_SSL: raise SkipTest( "The ssl module is available, can't test what happens " "without it." ) # Explicit self.assertRaises(ConfigurationError, MongoClient, ssl=True) self.assertRaises(ConfigurationError, MongoReplicaSetClient, ssl=True) # Implied self.assertRaises(ConfigurationError, MongoClient, ssl_certfile=CLIENT_PEM) self.assertRaises(ConfigurationError, MongoReplicaSetClient, ssl_certfile=CLIENT_PEM) def test_config_ssl(self): """Tests various ssl configurations""" self.assertRaises(ConfigurationError, MongoClient, ssl='foo') self.assertRaises(ConfigurationError, MongoClient, ssl=False, ssl_certfile=CLIENT_PEM) self.assertRaises(TypeError, MongoClient, ssl=0) self.assertRaises(TypeError, MongoClient, ssl=5.5) self.assertRaises(TypeError, MongoClient, ssl=[]) self.assertRaises(ConfigurationError, MongoReplicaSetClient, ssl='foo') self.assertRaises(ConfigurationError, MongoReplicaSetClient, ssl=False, ssl_certfile=CLIENT_PEM) self.assertRaises(TypeError, MongoReplicaSetClient, ssl=0) self.assertRaises(TypeError, MongoReplicaSetClient, ssl=5.5) self.assertRaises(TypeError, MongoReplicaSetClient, ssl=[]) self.assertRaises(IOError, MongoClient, ssl_certfile="NoSuchFile") self.assertRaises(TypeError, MongoClient, ssl_certfile=True) self.assertRaises(TypeError, MongoClient, ssl_certfile=[]) self.assertRaises(IOError, MongoClient, ssl_keyfile="NoSuchFile") self.assertRaises(TypeError, MongoClient, ssl_keyfile=True) self.assertRaises(TypeError, MongoClient, ssl_keyfile=[]) self.assertRaises(IOError, MongoReplicaSetClient, ssl_keyfile="NoSuchFile") self.assertRaises(IOError, MongoReplicaSetClient, ssl_certfile="NoSuchFile") self.assertRaises(TypeError, MongoReplicaSetClient, ssl_certfile=True) # Test invalid combinations self.assertRaises(ConfigurationError, MongoClient, ssl=False, ssl_keyfile=CLIENT_PEM) self.assertRaises(ConfigurationError, MongoClient, ssl=False, ssl_certfile=CLIENT_PEM) self.assertRaises(ConfigurationError, MongoClient, ssl=False, ssl_keyfile=CLIENT_PEM, ssl_certfile=CLIENT_PEM) self.assertRaises(ConfigurationError, MongoReplicaSetClient, ssl=False, ssl_keyfile=CLIENT_PEM) self.assertRaises(ConfigurationError, MongoReplicaSetClient, ssl=False, ssl_certfile=CLIENT_PEM) self.assertRaises(ConfigurationError, MongoReplicaSetClient, ssl=False, ssl_keyfile=CLIENT_PEM, ssl_certfile=CLIENT_PEM) class TestSSL(unittest.TestCase): def setUp(self): if not HAS_SSL: raise SkipTest("The ssl module is not available.") if sys.version.startswith('3.0'): raise SkipTest("Python 3.0.x has problems " "with SSL and socket timeouts.") if not SIMPLE_SSL: raise SkipTest("No simple mongod available over SSL") def test_simple_ssl(self): # Expects the server to be running with ssl and with # no --sslPEMKeyFile or with --sslWeakCertificateValidation client = MongoClient(host, port, ssl=True) response = client.admin.command('ismaster') if 'setName' in response: client = MongoReplicaSetClient(pair, replicaSet=response['setName'], w=len(response['hosts']), ssl=True) db = client.pymongo_ssl_test db.test.drop() self.assertTrue(db.test.insert({'ssl': True})) self.assertTrue(db.test.find_one()['ssl']) client.drop_database('pymongo_ssl_test') def test_cert_ssl(self): # Expects the server to be running with the the server.pem, ca.pem # and crl.pem provided in mongodb and the server tests eg: # # --sslPEMKeyFile=jstests/libs/server.pem # --sslCAFile=jstests/libs/ca.pem # --sslCRLFile=jstests/libs/crl.pem # # Also requires an /etc/hosts entry where "server" is resolvable if not CERT_SSL: raise SkipTest("No mongod available over SSL with certs") client = MongoClient(host, port, ssl=True, ssl_certfile=CLIENT_PEM) response = client.admin.command('ismaster') if 'setName' in response: client = MongoReplicaSetClient(pair, replicaSet=response['setName'], w=len(response['hosts']), ssl=True, ssl_certfile=CLIENT_PEM) db = client.pymongo_ssl_test db.test.drop() self.assertTrue(db.test.insert({'ssl': True})) self.assertTrue(db.test.find_one()['ssl']) client.drop_database('pymongo_ssl_test') def test_cert_ssl_implicitly_set(self): # Expects the server to be running with the the server.pem, ca.pem # and crl.pem provided in mongodb and the server tests eg: # # --sslPEMKeyFile=jstests/libs/server.pem # --sslCAFile=jstests/libs/ca.pem # --sslCRLFile=jstests/libs/crl.pem # # Also requires an /etc/hosts entry where "server" is resolvable if not CERT_SSL: raise SkipTest("No mongod available over SSL with certs") client = MongoClient(host, port, ssl_certfile=CLIENT_PEM) response = client.admin.command('ismaster') if 'setName' in response: client = MongoReplicaSetClient(pair, replicaSet=response['setName'], w=len(response['hosts']), ssl_certfile=CLIENT_PEM) db = client.pymongo_ssl_test db.test.drop() self.assertTrue(db.test.insert({'ssl': True})) self.assertTrue(db.test.find_one()['ssl']) client.drop_database('pymongo_ssl_test') def test_cert_ssl_validation(self): # Expects the server to be running with the the server.pem, ca.pem # and crl.pem provided in mongodb and the server tests eg: # # --sslPEMKeyFile=jstests/libs/server.pem # --sslCAFile=jstests/libs/ca.pem # --sslCRLFile=jstests/libs/crl.pem # # Also requires an /etc/hosts entry where "server" is resolvable if not CERT_SSL: raise SkipTest("No mongod available over SSL with certs") if not SERVER_IS_RESOLVABLE: raise SkipTest("No hosts entry for 'server'. Cannot validate " "hostname in the certificate") client = MongoClient('server', ssl=True, ssl_certfile=CLIENT_PEM, ssl_cert_reqs=ssl.CERT_REQUIRED, ssl_ca_certs=CA_PEM) response = client.admin.command('ismaster') if 'setName' in response: if response['primary'].split(":")[0] != 'server': raise SkipTest("No hosts in the replicaset for 'server'. " "Cannot validate hostname in the certificate") client = MongoReplicaSetClient('server', replicaSet=response['setName'], w=len(response['hosts']), ssl=True, ssl_certfile=CLIENT_PEM, ssl_cert_reqs=ssl.CERT_REQUIRED, ssl_ca_certs=CA_PEM) db = client.pymongo_ssl_test db.test.drop() self.assertTrue(db.test.insert({'ssl': True})) self.assertTrue(db.test.find_one()['ssl']) client.drop_database('pymongo_ssl_test') def test_cert_ssl_validation_optional(self): # Expects the server to be running with the the server.pem, ca.pem # and crl.pem provided in mongodb and the server tests eg: # # --sslPEMKeyFile=jstests/libs/server.pem # --sslCAFile=jstests/libs/ca.pem # --sslCRLFile=jstests/libs/crl.pem # # Also requires an /etc/hosts entry where "server" is resolvable if not CERT_SSL: raise SkipTest("No mongod available over SSL with certs") if not SERVER_IS_RESOLVABLE: raise SkipTest("No hosts entry for 'server'. Cannot validate " "hostname in the certificate") client = MongoClient('server', ssl=True, ssl_certfile=CLIENT_PEM, ssl_cert_reqs=ssl.CERT_OPTIONAL, ssl_ca_certs=CA_PEM) response = client.admin.command('ismaster') if 'setName' in response: if response['primary'].split(":")[0] != 'server': raise SkipTest("No hosts in the replicaset for 'server'. " "Cannot validate hostname in the certificate") client = MongoReplicaSetClient('server', replicaSet=response['setName'], w=len(response['hosts']), ssl=True, ssl_certfile=CLIENT_PEM, ssl_cert_reqs=ssl.CERT_OPTIONAL, ssl_ca_certs=CA_PEM) db = client.pymongo_ssl_test db.test.drop() self.assertTrue(db.test.insert({'ssl': True})) self.assertTrue(db.test.find_one()['ssl']) client.drop_database('pymongo_ssl_test') def test_cert_ssl_validation_hostname_fail(self): # Expects the server to be running with the the server.pem, ca.pem # and crl.pem provided in mongodb and the server tests eg: # # --sslPEMKeyFile=jstests/libs/server.pem # --sslCAFile=jstests/libs/ca.pem # --sslCRLFile=jstests/libs/crl.pem if not CERT_SSL: raise SkipTest("No mongod available over SSL with certs") client = MongoClient(ssl=True, ssl_certfile=CLIENT_PEM) response = client.admin.command('ismaster') single_server = 'setName' not in response if single_server: try: MongoClient(pair, ssl=True, ssl_certfile=CLIENT_PEM, ssl_cert_reqs=ssl.CERT_REQUIRED, ssl_ca_certs=CA_PEM) self.fail("Invalid hostname should have failed") except: pass else: try: MongoReplicaSetClient(pair, replicaSet=response['setName'], w=len(response['hosts']), ssl=True, ssl_certfile=CLIENT_PEM, ssl_cert_reqs=ssl.CERT_OPTIONAL, ssl_ca_certs=CA_PEM) self.fail("Invalid hostname should have failed") except: pass def test_mongodb_x509_auth(self): # Expects the server to be running with the the server.pem, ca.pem # and crl.pem provided in mongodb and the server tests as well as # --auth # # --sslPEMKeyFile=jstests/libs/server.pem # --sslCAFile=jstests/libs/ca.pem # --sslCRLFile=jstests/libs/crl.pem # --auth if not MONGODB_X509_USERNAME: raise SkipTest("MONGODB_X509_USERNAME " "must be set to test MONGODB-X509") if not CERT_SSL: raise SkipTest("No mongod available over SSL with certs") client = MongoClient(host, port, ssl=True, ssl_certfile=CLIENT_PEM) if not version.at_least(client, (2, 5, 1)): raise SkipTest("MONGODB-X509 requires MongoDB 2.5.1 or newer") argv = get_command_line(client) if '--auth' not in argv: raise SkipTest("Mongo must be started with " "--auth to test MONGODB-X509") # Give admin all necessary priviledges. client.admin.add_user(MONGODB_X509_USERNAME, userSource='$external', roles=['readWriteAnyDatabase', 'userAdminAnyDatabase', 'dbAdminAnyDatabase']) client = MongoClient(host, port, ssl=True, ssl_certfile=CLIENT_PEM) coll = client.pymongo_test.test self.assertRaises(OperationFailure, coll.count) self.assertTrue(client.admin.authenticate(MONGODB_X509_USERNAME, mechanism='MONGODB-X509')) self.assertEqual(0, coll.count()) uri = ('mongodb://%s@%s:%d/?authMechanism=' 'MONGODB-X509' % (quote_plus(MONGODB_X509_USERNAME), host, port)) # SSL options aren't supported in the URI... self.assertTrue(MongoClient(uri, ssl=True, ssl_certfile=CLIENT_PEM)) # Cleanup client.admin.system.users.remove() client['$external'].logout() if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_thread_util.py000066400000000000000000000201751223300253600201070ustar00rootroot00000000000000# Copyright 2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the thread_util module.""" import gc import sys import threading import time import unittest sys.path[0:0] = [""] from nose.plugins.skip import SkipTest from pymongo import thread_util if thread_util.have_gevent: import greenlet # Plain greenlets. import gevent.greenlet # Gevent's enhanced Greenlets. import gevent.hub from test.utils import looplet, RendezvousThread class TestIdent(unittest.TestCase): """Ensure thread_util.Ident works for threads and greenlets. This has gotten intricate from refactoring: we have classes, Watched and Unwatched, that implement the logic for the two child threads / greenlets. For the greenlet case it's easy to ensure the two children are alive at once, so we run the Watched and Unwatched logic directly. For the thread case we mix in the RendezvousThread class so we're sure both children are alive when they call Ident.get(). 1. Store main thread's / greenlet's id 2. Start 2 child threads / greenlets 3. Store their values for Ident.get() 4. Children reach rendezvous point 5. Children call Ident.watch() 6. One of the children calls Ident.unwatch() 7. Children terminate 8. Assert that children got different ids from each other and from main, and assert watched child's callback was executed, and that unwatched child's callback was not """ def _test_ident(self, use_greenlets): ident = thread_util.create_ident(use_greenlets) ids = set([ident.get()]) unwatched_id = [] done = set([ident.get()]) # Start with main thread's / greenlet's id. died = set() class Watched(object): def __init__(self, ident): self._my_ident = ident def before_rendezvous(self): self.my_id = self._my_ident.get() ids.add(self.my_id) def after_rendezvous(self): assert not self._my_ident.watching() self._my_ident.watch(lambda ref: died.add(self.my_id)) assert self._my_ident.watching() done.add(self.my_id) class Unwatched(Watched): def before_rendezvous(self): Watched.before_rendezvous(self) unwatched_id.append(self.my_id) def after_rendezvous(self): Watched.after_rendezvous(self) self._my_ident.unwatch(self.my_id) assert not self._my_ident.watching() if use_greenlets: class WatchedGreenlet(Watched): def run(self): self.before_rendezvous() self.after_rendezvous() class UnwatchedGreenlet(Unwatched): def run(self): self.before_rendezvous() self.after_rendezvous() t_watched = greenlet.greenlet(WatchedGreenlet(ident).run) t_unwatched = greenlet.greenlet(UnwatchedGreenlet(ident).run) looplet([t_watched, t_unwatched]) else: class WatchedThread(Watched, RendezvousThread): def __init__(self, ident, state): Watched.__init__(self, ident) RendezvousThread.__init__(self, state) class UnwatchedThread(Unwatched, RendezvousThread): def __init__(self, ident, state): Unwatched.__init__(self, ident) RendezvousThread.__init__(self, state) state = RendezvousThread.create_shared_state(2) t_watched = WatchedThread(ident, state) t_watched.start() t_unwatched = UnwatchedThread(ident, state) t_unwatched.start() RendezvousThread.wait_for_rendezvous(state) RendezvousThread.resume_after_rendezvous(state) t_watched.join() t_unwatched.join() self.assertTrue(t_watched.passed) self.assertTrue(t_unwatched.passed) # Remove references, let weakref callbacks run del t_watched del t_unwatched # Trigger final cleanup in Python <= 2.7.0. # http://bugs.python.org/issue1868 ident.get() self.assertEqual(3, len(ids)) self.assertEqual(3, len(done)) # Make sure thread is really gone slept = 0 while not died and slept < 10: time.sleep(1) gc.collect() slept += 1 self.assertEqual(1, len(died)) self.assertFalse(unwatched_id[0] in died) def test_thread_ident(self): self._test_ident(False) def test_greenlet_ident(self): if not thread_util.have_gevent: raise SkipTest('greenlet not installed') self._test_ident(True) class TestGreenletIdent(unittest.TestCase): def setUp(self): if not thread_util.have_gevent: raise SkipTest("need Gevent") def test_unwatch_cleans_up(self): # GreenletIdent.unwatch() should remove the on_thread_died callback # from an enhanced Gevent Greenlet's list of links. callback_ran = [False] def on_greenlet_died(_): callback_ran[0] = True ident = thread_util.create_ident(use_greenlets=True) def watch_and_unwatch(): ident.watch(on_greenlet_died) ident.unwatch(ident.get()) g = gevent.greenlet.Greenlet(run=watch_and_unwatch) g.start() g.join(10) the_hub = gevent.hub.get_hub() if hasattr(the_hub, 'join'): # Gevent 1.0 the_hub.join() else: # Gevent 0.13 and less the_hub.shutdown() self.assertTrue(g.successful()) # unwatch() canceled the callback. self.assertFalse(callback_ran[0]) # No functools in Python 2.4 def my_partial(f, *args, **kwargs): def _f(*new_args, **new_kwargs): final_kwargs = kwargs.copy() final_kwargs.update(new_kwargs) return f(*(args + new_args), **final_kwargs) return _f class TestCounter(unittest.TestCase): def _test_counter(self, use_greenlets): counter = thread_util.Counter(use_greenlets) self.assertEqual(0, counter.dec()) self.assertEqual(0, counter.get()) self.assertEqual(0, counter.dec()) self.assertEqual(0, counter.get()) done = set() def f(n): for i in xrange(n): self.assertEqual(i, counter.get()) self.assertEqual(i + 1, counter.inc()) for i in xrange(n, 0, -1): self.assertEqual(i, counter.get()) self.assertEqual(i - 1, counter.dec()) self.assertEqual(0, counter.get()) # Extra decrements have no effect self.assertEqual(0, counter.dec()) self.assertEqual(0, counter.get()) self.assertEqual(0, counter.dec()) self.assertEqual(0, counter.get()) done.add(n) if use_greenlets: greenlets = [ greenlet.greenlet(my_partial(f, i)) for i in xrange(10)] looplet(greenlets) else: threads = [ threading.Thread(target=my_partial(f, i)) for i in xrange(10)] for t in threads: t.start() for t in threads: t.join() self.assertEqual(10, len(done)) def test_thread_counter(self): self._test_counter(False) def test_greenlet_counter(self): if not thread_util.have_gevent: raise SkipTest('greenlet not installed') self._test_counter(True) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_threads.py000066400000000000000000000273161223300253600172410ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test that pymongo is thread safe.""" import unittest import threading import traceback from nose.plugins.skip import SkipTest from test.utils import server_started_with_auth, joinall, RendezvousThread from test.test_client import get_client from test.utils import get_pool from pymongo.pool import SocketInfo, _closed from pymongo.errors import AutoReconnect, OperationFailure class AutoAuthenticateThreads(threading.Thread): def __init__(self, collection, num): threading.Thread.__init__(self) self.coll = collection self.num = num self.success = True self.setDaemon(True) def run(self): try: for i in xrange(self.num): self.coll.insert({'num':i}) self.coll.find_one({'num':i}) except Exception: traceback.print_exc() self.success = False class SaveAndFind(threading.Thread): def __init__(self, collection): threading.Thread.__init__(self) self.collection = collection self.setDaemon(True) def run(self): sum = 0 for document in self.collection.find(): sum += document["x"] assert sum == 499500, "sum was %d not 499500" % sum class Insert(threading.Thread): def __init__(self, collection, n, expect_exception): threading.Thread.__init__(self) self.collection = collection self.n = n self.expect_exception = expect_exception self.setDaemon(True) def run(self): for _ in xrange(self.n): error = True try: self.collection.insert({"test": "insert"}) error = False except: if not self.expect_exception: raise if self.expect_exception: assert error class Update(threading.Thread): def __init__(self, collection, n, expect_exception): threading.Thread.__init__(self) self.collection = collection self.n = n self.expect_exception = expect_exception self.setDaemon(True) def run(self): for _ in xrange(self.n): error = True try: self.collection.update({"test": "unique"}, {"$set": {"test": "update"}}) error = False except: if not self.expect_exception: raise if self.expect_exception: assert error class IgnoreAutoReconnect(threading.Thread): def __init__(self, collection, n): threading.Thread.__init__(self) self.c = collection self.n = n self.setDaemon(True) def run(self): for _ in range(self.n): try: self.c.find_one() except AutoReconnect: pass class FindPauseFind(RendezvousThread): """See test_server_disconnect() for details""" def __init__(self, collection, state): """Params: `collection`: A collection for testing `state`: A shared state object from RendezvousThread.shared_state() """ super(FindPauseFind, self).__init__(state) self.collection = collection def before_rendezvous(self): # acquire a socket list(self.collection.find()) self.pool = get_pool(self.collection.database.connection) socket_info = self.pool._get_request_state() assert isinstance(socket_info, SocketInfo) self.request_sock = socket_info.sock assert not _closed(self.request_sock) def after_rendezvous(self): # test_server_disconnect() has closed this socket, but that's ok # because it's not our request socket anymore assert _closed(self.request_sock) # if disconnect() properly closed all threads' request sockets, then # this won't raise AutoReconnect because it will acquire a new socket assert self.request_sock == self.pool._get_request_state().sock list(self.collection.find()) assert self.collection.database.connection.in_request() assert self.request_sock != self.pool._get_request_state().sock class BaseTestThreads(object): """ Base test class for TestThreads and TestThreadsReplicaSet. (This is not itself a unittest.TestCase, otherwise it'd be run twice -- once when nose imports this module, and once when nose imports test_threads_replica_set_connection.py, which imports this module.) """ def setUp(self): self.db = self._get_client().pymongo_test def tearDown(self): # Clear client reference so that RSC's monitor thread # dies. self.db = None def _get_client(self): """ Intended for overriding in TestThreadsReplicaSet. This method returns a MongoClient here, and a MongoReplicaSetClient in test_threads_replica_set_connection.py. """ # Regular test client return get_client() def test_threading(self): self.db.drop_collection("test") for i in xrange(1000): self.db.test.save({"x": i}) threads = [] for i in range(10): t = SaveAndFind(self.db.test) t.start() threads.append(t) joinall(threads) def test_safe_insert(self): self.db.drop_collection("test1") self.db.test1.insert({"test": "insert"}) self.db.drop_collection("test2") self.db.test2.insert({"test": "insert"}) self.db.test2.create_index("test", unique=True) self.db.test2.find_one() okay = Insert(self.db.test1, 2000, False) error = Insert(self.db.test2, 2000, True) error.start() okay.start() error.join() okay.join() def test_safe_update(self): self.db.drop_collection("test1") self.db.test1.insert({"test": "update"}) self.db.test1.insert({"test": "unique"}) self.db.drop_collection("test2") self.db.test2.insert({"test": "update"}) self.db.test2.insert({"test": "unique"}) self.db.test2.create_index("test", unique=True) self.db.test2.find_one() okay = Update(self.db.test1, 2000, False) error = Update(self.db.test2, 2000, True) error.start() okay.start() error.join() okay.join() def test_server_disconnect(self): # PYTHON-345, we need to make sure that threads' request sockets are # closed by disconnect(). # # 1. Create a client with auto_start_request=True # 2. Start N threads and do a find() in each to get a request socket # 3. Pause all threads # 4. In the main thread close all sockets, including threads' request # sockets # 5. In main thread, do a find(), which raises AutoReconnect and resets # pool # 6. Resume all threads, do a find() in them # # If we've fixed PYTHON-345, then only one AutoReconnect is raised, # and all the threads get new request sockets. cx = get_client(auto_start_request=True) collection = cx.db.pymongo_test # acquire a request socket for the main thread collection.find_one() pool = get_pool(collection.database.connection) socket_info = pool._get_request_state() assert isinstance(socket_info, SocketInfo) request_sock = socket_info.sock state = FindPauseFind.create_shared_state(nthreads=40) threads = [ FindPauseFind(collection, state) for _ in range(state.nthreads) ] # Each thread does a find(), thus acquiring a request socket for t in threads: t.start() # Wait for the threads to reach the rendezvous FindPauseFind.wait_for_rendezvous(state) try: # Simulate an event that closes all sockets, e.g. primary stepdown for t in threads: t.request_sock.close() # Finally, ensure the main thread's socket's last_checkout is # updated: collection.find_one() # ... and close it: request_sock.close() # Doing an operation on the client raises an AutoReconnect and # resets the pool behind the scenes self.assertRaises(AutoReconnect, collection.find_one) finally: # Let threads do a second find() FindPauseFind.resume_after_rendezvous(state) joinall(threads) for t in threads: self.assertTrue(t.passed, "%s threw exception" % t) class BaseTestThreadsAuth(object): """ Base test class for TestThreadsAuth and TestThreadsAuthReplicaSet. (This is not itself a unittest.TestCase, otherwise it'd be run twice -- once when nose imports this module, and once when nose imports test_threads_replica_set_connection.py, which imports this module.) """ def _get_client(self): """ Intended for overriding in TestThreadsAuthReplicaSet. This method returns a MongoClient here, and a MongoReplicaSetClient in test_threads_replica_set_connection.py. """ # Regular test client return get_client() def setUp(self): client = self._get_client() if not server_started_with_auth(client): raise SkipTest("Authentication is not enabled on server") self.client = client self.client.admin.system.users.remove({}) self.client.admin.add_user('admin-user', 'password') self.client.admin.authenticate("admin-user", "password") self.client.auth_test.system.users.remove({}) self.client.auth_test.add_user("test-user", "password") def tearDown(self): # Remove auth users from databases self.client.admin.authenticate("admin-user", "password") self.client.admin.system.users.remove({}) self.client.auth_test.system.users.remove({}) self.client.drop_database('auth_test') # Clear client reference so that RSC's monitor thread # dies. self.client = None def test_auto_auth_login(self): client = self._get_client() self.assertRaises(OperationFailure, client.auth_test.test.find_one) # Admin auth client = self._get_client() client.admin.authenticate("admin-user", "password") nthreads = 10 threads = [] for _ in xrange(nthreads): t = AutoAuthenticateThreads(client.auth_test.test, 100) t.start() threads.append(t) joinall(threads) for t in threads: self.assertTrue(t.success) # Database-specific auth client = self._get_client() client.auth_test.authenticate("test-user", "password") threads = [] for _ in xrange(nthreads): t = AutoAuthenticateThreads(client.auth_test.test, 100) t.start() threads.append(t) joinall(threads) for t in threads: self.assertTrue(t.success) class TestThreads(BaseTestThreads, unittest.TestCase): pass class TestThreadsAuth(BaseTestThreadsAuth, unittest.TestCase): pass if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_threads_replica_set_client.py000066400000000000000000000043441223300253600231450ustar00rootroot00000000000000# Copyright 2011-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test that pymongo is thread safe.""" import unittest from pymongo.mongo_replica_set_client import MongoReplicaSetClient from test.test_threads import BaseTestThreads, BaseTestThreadsAuth from test.test_replica_set_client import TestReplicaSetClientBase, pair class TestThreadsReplicaSet(TestReplicaSetClientBase, BaseTestThreads): def setUp(self): """ Prepare to test all the same things that TestThreads tests, but do it with a replica-set client """ TestReplicaSetClientBase.setUp(self) BaseTestThreads.setUp(self) def tearDown(self): TestReplicaSetClientBase.tearDown(self) BaseTestThreads.tearDown(self) def _get_client(self, **kwargs): return TestReplicaSetClientBase._get_client(self, **kwargs) class TestThreadsAuthReplicaSet(TestReplicaSetClientBase, BaseTestThreadsAuth): def setUp(self): """ Prepare to test all the same things that TestThreads tests, but do it with a replica-set client """ TestReplicaSetClientBase.setUp(self) BaseTestThreadsAuth.setUp(self) def tearDown(self): TestReplicaSetClientBase.tearDown(self) BaseTestThreadsAuth.tearDown(self) def _get_client(self): """ Override TestThreadsAuth, so its tests run on a MongoReplicaSetClient instead of a regular MongoClient. """ return MongoReplicaSetClient(pair, replicaSet=self.name) if __name__ == "__main__": suite = unittest.TestSuite([ unittest.makeSuite(TestThreadsReplicaSet), unittest.makeSuite(TestThreadsAuthReplicaSet) ]) unittest.TextTestRunner(verbosity=2).run(suite) pymongo-2.6.3/test/test_timestamp.py000066400000000000000000000047341223300253600176110ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Tests for the Timestamp class.""" import datetime import unittest import sys import copy import pickle sys.path[0:0] = [""] from bson.timestamp import Timestamp from bson.tz_util import utc class TestTimestamp(unittest.TestCase): def setUp(self): pass def test_timestamp(self): t = Timestamp(123, 456) self.assertEqual(t.time, 123) self.assertEqual(t.inc, 456) self.assertTrue(isinstance(t, Timestamp)) def test_datetime(self): d = datetime.datetime(2010, 5, 5, tzinfo=utc) t = Timestamp(d, 0) self.assertEqual(1273017600, t.time) self.assertEqual(d, t.as_datetime()) def test_datetime_copy_pickle(self): d = datetime.datetime(2010, 5, 5, tzinfo=utc) t = Timestamp(d, 0) dc = copy.deepcopy(d) self.assertEqual(dc, t.as_datetime()) for protocol in [0, 1, 2, -1]: pkl = pickle.dumps(d, protocol=protocol) dp = pickle.loads(pkl) self.assertEqual(dp, t.as_datetime()) def test_exceptions(self): self.assertRaises(TypeError, Timestamp) self.assertRaises(TypeError, Timestamp, None, 123) self.assertRaises(TypeError, Timestamp, 1.2, 123) self.assertRaises(TypeError, Timestamp, 123, None) self.assertRaises(TypeError, Timestamp, 123, 1.2) self.assertRaises(ValueError, Timestamp, 0, -1) self.assertRaises(ValueError, Timestamp, -1, 0) self.assertTrue(Timestamp(0, 0)) def test_equality(self): t = Timestamp(1, 1) self.assertNotEqual(t, Timestamp(0, 1)) self.assertNotEqual(t, Timestamp(1, 0)) self.assertEqual(t, Timestamp(1, 1)) # Explicitly test inequality self.assertFalse(t != Timestamp(1, 1)) def test_repr(self): t = Timestamp(0, 0) self.assertEqual(repr(t), "Timestamp(0, 0)") if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/test_uri_parser.py000066400000000000000000000401731223300253600177560ustar00rootroot00000000000000# Copyright 2011-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Test the pymongo uri_parser module.""" import copy import unittest import sys sys.path[0:0] = [""] from pymongo.uri_parser import (_partition, _rpartition, parse_userinfo, split_hosts, split_options, parse_uri) from pymongo.errors import ConfigurationError, InvalidURI from pymongo import ReadPreference class TestURI(unittest.TestCase): def test_partition(self): self.assertEqual(('foo', ':', 'bar'), _partition('foo:bar', ':')) self.assertEqual(('foobar', '', ''), _partition('foobar', ':')) def test_rpartition(self): self.assertEqual(('fo:o:', ':', 'bar'), _rpartition('fo:o::bar', ':')) self.assertEqual(('', '', 'foobar'), _rpartition('foobar', ':')) def test_validate_userinfo(self): self.assertRaises(InvalidURI, parse_userinfo, 'foo@') self.assertRaises(InvalidURI, parse_userinfo, ':password') self.assertRaises(InvalidURI, parse_userinfo, 'fo::o:p@ssword') self.assertRaises(InvalidURI, parse_userinfo, ':') self.assertTrue(parse_userinfo('user:password')) self.assertEqual(('us:r', 'p@ssword'), parse_userinfo('us%3Ar:p%40ssword')) self.assertEqual(('us er', 'p ssword'), parse_userinfo('us+er:p+ssword')) self.assertEqual(('us er', 'p ssword'), parse_userinfo('us%20er:p%20ssword')) self.assertEqual(('us+er', 'p+ssword'), parse_userinfo('us%2Ber:p%2Bssword')) self.assertEqual(('dev1@FOO.COM', ''), parse_userinfo('dev1%40FOO.COM')) self.assertEqual(('dev1@FOO.COM', ''), parse_userinfo('dev1%40FOO.COM:')) def test_split_hosts(self): self.assertRaises(ConfigurationError, split_hosts, 'localhost:27017,') self.assertRaises(ConfigurationError, split_hosts, ',localhost:27017') self.assertRaises(ConfigurationError, split_hosts, 'localhost:27017,,localhost:27018') self.assertEqual([('localhost', 27017), ('example.com', 27017)], split_hosts('localhost,example.com')) self.assertEqual([('localhost', 27018), ('example.com', 27019)], split_hosts('localhost:27018,example.com:27019')) self.assertEqual([('/tmp/mongodb-27017.sock', None)], split_hosts('/tmp/mongodb-27017.sock')) self.assertEqual([('/tmp/mongodb-27017.sock', None), ('example.com', 27017)], split_hosts('/tmp/mongodb-27017.sock,' 'example.com:27017')) self.assertEqual([('example.com', 27017), ('/tmp/mongodb-27017.sock', None)], split_hosts('example.com:27017,' '/tmp/mongodb-27017.sock')) self.assertRaises(ConfigurationError, split_hosts, '::1', 27017) self.assertRaises(ConfigurationError, split_hosts, '[::1:27017') self.assertRaises(ConfigurationError, split_hosts, '::1') self.assertRaises(ConfigurationError, split_hosts, '::1]:27017') self.assertEqual([('::1', 27017)], split_hosts('[::1]:27017')) self.assertEqual([('::1', 27017)], split_hosts('[::1]')) def test_split_options(self): self.assertRaises(ConfigurationError, split_options, 'foo') self.assertRaises(ConfigurationError, split_options, 'foo=bar') self.assertRaises(ConfigurationError, split_options, 'foo=bar;foo') self.assertRaises(ConfigurationError, split_options, 'socketTimeoutMS=foo') self.assertRaises(ConfigurationError, split_options, 'socketTimeoutMS=0.0') self.assertRaises(ConfigurationError, split_options, 'connectTimeoutMS=foo') self.assertRaises(ConfigurationError, split_options, 'connectTimeoutMS=0.0') self.assertRaises(ConfigurationError, split_options, 'connectTimeoutMS=1e100000') self.assertRaises(ConfigurationError, split_options, 'connectTimeoutMS=-1e100000') # On most platforms float('inf') and float('-inf') represent # +/- infinity, although on Python 2.4 and 2.5 on Windows those # expressions are invalid if not (sys.platform == "win32" and sys.version_info <= (2, 5)): self.assertRaises(ConfigurationError, split_options, 'connectTimeoutMS=inf') self.assertRaises(ConfigurationError, split_options, 'connectTimeoutMS=-inf') self.assertTrue(split_options('socketTimeoutMS=300')) self.assertTrue(split_options('connectTimeoutMS=300')) self.assertEqual({'sockettimeoutms': 0.3}, split_options('socketTimeoutMS=300')) self.assertEqual({'sockettimeoutms': 0.0001}, split_options('socketTimeoutMS=0.1')) self.assertEqual({'connecttimeoutms': 0.3}, split_options('connectTimeoutMS=300')) self.assertEqual({'connecttimeoutms': 0.0001}, split_options('connectTimeoutMS=0.1')) self.assertTrue(split_options('connectTimeoutMS=300')) self.assertTrue(isinstance(split_options('w=5')['w'], int)) self.assertTrue(isinstance(split_options('w=5.5')['w'], basestring)) self.assertTrue(split_options('w=foo')) self.assertTrue(split_options('w=majority')) self.assertRaises(ConfigurationError, split_options, 'wtimeoutms=foo') self.assertRaises(ConfigurationError, split_options, 'wtimeoutms=5.5') self.assertTrue(split_options('wtimeoutms=500')) self.assertRaises(ConfigurationError, split_options, 'fsync=foo') self.assertRaises(ConfigurationError, split_options, 'fsync=5.5') self.assertEqual({'fsync': True}, split_options('fsync=true')) self.assertEqual({'fsync': False}, split_options('fsync=false')) self.assertEqual({'authmechanism': 'GSSAPI'}, split_options('authMechanism=GSSAPI')) self.assertEqual({'authmechanism': 'MONGODB-CR'}, split_options('authMechanism=MONGODB-CR')) self.assertEqual({'authsource': 'foobar'}, split_options('authSource=foobar')) # maxPoolSize isn't yet a documented URI option. self.assertRaises(ConfigurationError, split_options, 'maxpoolsize=50') def test_parse_uri(self): self.assertRaises(InvalidURI, parse_uri, "http://foobar.com") self.assertRaises(InvalidURI, parse_uri, "http://foo@foobar.com") self.assertRaises(ConfigurationError, parse_uri, "mongodb://::1", 27017) orig = { 'nodelist': [("localhost", 27017)], 'username': None, 'password': None, 'database': None, 'collection': None, 'options': {} } res = copy.deepcopy(orig) self.assertEqual(res, parse_uri("mongodb://localhost")) res.update({'username': 'fred', 'password': 'foobar'}) self.assertEqual(res, parse_uri("mongodb://fred:foobar@localhost")) res.update({'database': 'baz'}) self.assertEqual(res, parse_uri("mongodb://fred:foobar@localhost/baz")) res = copy.deepcopy(orig) res['nodelist'] = [("example1.com", 27017), ("example2.com", 27017)] self.assertEqual(res, parse_uri("mongodb://example1.com:27017," "example2.com:27017")) res = copy.deepcopy(orig) res['nodelist'] = [("localhost", 27017), ("localhost", 27018), ("localhost", 27019)] self.assertEqual(res, parse_uri("mongodb://localhost," "localhost:27018,localhost:27019")) res = copy.deepcopy(orig) res['database'] = 'foo' self.assertEqual(res, parse_uri("mongodb://localhost/foo")) res = copy.deepcopy(orig) self.assertEqual(res, parse_uri("mongodb://localhost/")) res.update({'database': 'test', 'collection': 'yield_historical.in'}) self.assertEqual(res, parse_uri("mongodb://" "localhost/test.yield_historical.in")) res.update({'username': 'fred', 'password': 'foobar'}) self.assertEqual(res, parse_uri("mongodb://fred:foobar@localhost/" "test.yield_historical.in")) res = copy.deepcopy(orig) res['nodelist'] = [("example1.com", 27017), ("example2.com", 27017)] res.update({'database': 'test', 'collection': 'yield_historical.in'}) self.assertEqual(res, parse_uri("mongodb://example1.com:27017,example2.com" ":27017/test.yield_historical.in")) res = copy.deepcopy(orig) res['nodelist'] = [("::1", 27017)] res['options'] = {'slaveok': True} self.assertEqual(res, parse_uri("mongodb://[::1]:27017/?slaveOk=true")) res = copy.deepcopy(orig) res['nodelist'] = [("2001:0db8:85a3:0000:0000:8a2e:0370:7334", 27017)] res['options'] = {'slaveok': True} self.assertEqual(res, parse_uri( "mongodb://[2001:0db8:85a3:0000:0000" ":8a2e:0370:7334]:27017/?slaveOk=true")) res = copy.deepcopy(orig) res['nodelist'] = [("/tmp/mongodb-27017.sock", None)] self.assertEqual(res, parse_uri("mongodb:///tmp/mongodb-27017.sock")) res = copy.deepcopy(orig) res['nodelist'] = [("example2.com", 27017), ("/tmp/mongodb-27017.sock", None)] self.assertEqual(res, parse_uri("mongodb://example2.com," "/tmp/mongodb-27017.sock")) res = copy.deepcopy(orig) res['nodelist'] = [("shoe.sock.pants.co.uk", 27017), ("/tmp/mongodb-27017.sock", None)] res['database'] = "nethers_db" self.assertEqual(res, parse_uri("mongodb://shoe.sock.pants.co.uk," "/tmp/mongodb-27017.sock/nethers_db")) res = copy.deepcopy(orig) res['nodelist'] = [("/tmp/mongodb-27017.sock", None), ("example2.com", 27017)] res.update({'database': 'test', 'collection': 'yield_historical.in'}) self.assertEqual(res, parse_uri("mongodb:///tmp/mongodb-27017.sock," "example2.com:27017" "/test.yield_historical.in")) res = copy.deepcopy(orig) res['nodelist'] = [("/tmp/mongodb-27017.sock", None), ("example2.com", 27017)] res.update({'database': 'test', 'collection': 'yield_historical.sock'}) self.assertEqual(res, parse_uri("mongodb:///tmp/mongodb-27017.sock," "example2.com:27017" "/test.yield_historical.sock")) res = copy.deepcopy(orig) res['nodelist'] = [("example2.com", 27017)] res.update({'database': 'test', 'collection': 'yield_historical.sock'}) self.assertEqual(res, parse_uri("mongodb://example2.com:27017" "/test.yield_historical.sock")) res = copy.deepcopy(orig) res['nodelist'] = [("/tmp/mongodb-27017.sock", None)] res.update({'database': 'test', 'collection': 'mongodb-27017.sock'}) self.assertEqual(res, parse_uri("mongodb:///tmp/mongodb-27017.sock" "/test.mongodb-27017.sock")) res = copy.deepcopy(orig) res['nodelist'] = [('/tmp/mongodb-27020.sock', None), ("::1", 27017), ("2001:0db8:85a3:0000:0000:8a2e:0370:7334", 27018), ("192.168.0.212", 27019), ("localhost", 27018)] self.assertEqual(res, parse_uri("mongodb:///tmp/mongodb-27020.sock," "[::1]:27017,[2001:0db8:" "85a3:0000:0000:8a2e:0370:7334]," "192.168.0.212:27019,localhost", 27018)) res = copy.deepcopy(orig) res.update({'username': 'fred', 'password': 'foobar'}) res.update({'database': 'test', 'collection': 'yield_historical.in'}) res['options'] = {'slaveok': True} self.assertEqual(res, parse_uri("mongodb://fred:foobar@localhost/" "test.yield_historical.in?slaveok=true")) res = copy.deepcopy(orig) res['options'] = {'readpreference': ReadPreference.SECONDARY} self.assertEqual(res, parse_uri("mongodb://localhost/?readPreference=secondary")) # Various authentication tests res = copy.deepcopy(orig) res['options'] = {'authmechanism': 'MONGODB-CR'} res['username'] = 'user' res['password'] = 'password' self.assertEqual(res, parse_uri("mongodb://user:password@localhost/" "?authMechanism=MONGODB-CR")) res = copy.deepcopy(orig) res['options'] = {'authmechanism': 'MONGODB-CR', 'authsource': 'bar'} res['username'] = 'user' res['password'] = 'password' res['database'] = 'foo' self.assertEqual(res, parse_uri("mongodb://user:password@localhost/foo" "?authSource=bar;authMechanism=MONGODB-CR")) res = copy.deepcopy(orig) res['options'] = {'authmechanism': 'MONGODB-CR'} res['username'] = 'user' res['password'] = '' self.assertEqual(res, parse_uri("mongodb://user:@localhost/" "?authMechanism=MONGODB-CR")) res = copy.deepcopy(orig) res['username'] = 'user@domain.com' res['password'] = 'password' res['database'] = 'foo' self.assertEqual(res, parse_uri("mongodb://user%40domain.com:password" "@localhost/foo")) res = copy.deepcopy(orig) res['options'] = {'authmechanism': 'GSSAPI'} res['username'] = 'user@domain.com' res['password'] = 'password' res['database'] = 'foo' self.assertEqual(res, parse_uri("mongodb://user%40domain.com:password" "@localhost/foo?authMechanism=GSSAPI")) res = copy.deepcopy(orig) res['options'] = {'authmechanism': 'GSSAPI'} res['username'] = 'user@domain.com' res['password'] = '' res['database'] = 'foo' self.assertEqual(res, parse_uri("mongodb://user%40domain.com" "@localhost/foo?authMechanism=GSSAPI")) def test_parse_uri_unicode(self): # Ensure parsing a unicode returns option names that can be passed # as kwargs. In Python 2.4, keyword argument names must be ASCII. # In all Pythons, str is the type of valid keyword arg names. res = parse_uri(unicode("mongodb://localhost/?fsync=true")) for key in res['options']: self.assertTrue(isinstance(key, str)) if __name__ == "__main__": unittest.main() pymongo-2.6.3/test/utils.py000066400000000000000000000261661223300253600157120ustar00rootroot00000000000000# Copyright 2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Utilities for testing pymongo """ import threading from pymongo import MongoClient, MongoReplicaSetClient from pymongo.errors import AutoReconnect from pymongo.pool import NO_REQUEST, NO_SOCKET_YET, SocketInfo from test import host, port def one(s): """Get one element of a set""" return iter(s).next() def delay(sec): # Javascript sleep() only available in MongoDB since version ~1.9 return '''function() { var d = new Date((new Date()).getTime() + %s * 1000); while (d > (new Date())) { }; return true; }''' % sec def get_command_line(client): command_line = client.admin.command('getCmdLineOpts') assert command_line['ok'] == 1, "getCmdLineOpts() failed" return command_line['argv'] def server_started_with_auth(client): argv = get_command_line(client) return '--auth' in argv or '--keyFile' in argv def server_is_master_with_slave(client): return '--master' in get_command_line(client) def drop_collections(db): for coll in db.collection_names(): if not coll.startswith('system'): db.drop_collection(coll) def joinall(threads): """Join threads with a 5-minute timeout, assert joins succeeded""" for t in threads: t.join(300) assert not t.isAlive(), "Thread %s hung" % t def is_mongos(client): res = client.admin.command('ismaster') return res.get('msg', '') == 'isdbgrid' def enable_text_search(client): client.admin.command( 'setParameter', textSearchEnabled=True) if isinstance(client, MongoReplicaSetClient): for host, port in client.secondaries: MongoClient(host, port).admin.command( 'setParameter', textSearchEnabled=True) def assertRaisesExactly(cls, fn, *args, **kwargs): """ Unlike the standard assertRaises, this checks that a function raises a specific class of exception, and not a subclass. E.g., check that MongoClient() raises ConnectionFailure but not its subclass, AutoReconnect. """ try: fn(*args, **kwargs) except Exception, e: assert e.__class__ == cls, "got %s, expected %s" % ( e.__class__.__name__, cls.__name__) else: raise AssertionError("%s not raised" % cls) def looplet(greenlets): """World's smallest event loop; run until all greenlets are done """ while True: done = True for g in greenlets: if not g.dead: done = False g.switch() if done: return class RendezvousThread(threading.Thread): """A thread that starts and pauses at a rendezvous point before resuming. To be used in tests that must ensure that N threads are all alive simultaneously, regardless of thread-scheduling's vagaries. 1. Write a subclass of RendezvousThread and override before_rendezvous and / or after_rendezvous. 2. Create a state with RendezvousThread.shared_state(N) 3. Start N of your subclassed RendezvousThreads, passing the state to each one's __init__ 4. In the main thread, call RendezvousThread.wait_for_rendezvous 5. Test whatever you need to test while threads are paused at rendezvous point 6. In main thread, call RendezvousThread.resume_after_rendezvous 7. Join all threads from main thread 8. Assert that all threads' "passed" attribute is True 9. Test post-conditions """ class RendezvousState(object): def __init__(self, nthreads): # Number of threads total self.nthreads = nthreads # Number of threads that have arrived at rendezvous point self.arrived_threads = 0 self.arrived_threads_lock = threading.Lock() # Set when all threads reach rendezvous self.ev_arrived = threading.Event() # Set by resume_after_rendezvous() so threads can continue. self.ev_resume = threading.Event() @classmethod def create_shared_state(cls, nthreads): return RendezvousThread.RendezvousState(nthreads) def before_rendezvous(self): """Overridable: Do this before the rendezvous""" pass def after_rendezvous(self): """Overridable: Do this after the rendezvous. If it throws no exception, `passed` is set to True """ pass @classmethod def wait_for_rendezvous(cls, state): """Wait for all threads to reach rendezvous and pause there""" state.ev_arrived.wait(10) assert state.ev_arrived.isSet(), "Thread timeout" assert state.nthreads == state.arrived_threads @classmethod def resume_after_rendezvous(cls, state): """Tell all the paused threads to continue""" state.ev_resume.set() def __init__(self, state): """Params: `state`: A shared state object from RendezvousThread.shared_state() """ super(RendezvousThread, self).__init__() self.state = state self.passed = False # If this thread fails to terminate, don't hang the whole program self.setDaemon(True) def _rendezvous(self): """Pause until all threads arrive here""" s = self.state s.arrived_threads_lock.acquire() s.arrived_threads += 1 if s.arrived_threads == s.nthreads: s.arrived_threads_lock.release() s.ev_arrived.set() else: s.arrived_threads_lock.release() s.ev_arrived.wait() def run(self): try: self.before_rendezvous() finally: self._rendezvous() # all threads have passed the rendezvous, wait for # resume_after_rendezvous() self.state.ev_resume.wait() self.after_rendezvous() self.passed = True def read_from_which_host( rsc, mode, tag_sets=None, secondary_acceptable_latency_ms=15 ): """Read from a MongoReplicaSetClient with the given Read Preference mode, tags, and acceptable latency. Return the 'host:port' which was read from. :Parameters: - `rsc`: A MongoReplicaSetClient - `mode`: A ReadPreference - `tag_sets`: List of dicts of tags for data-center-aware reads - `secondary_acceptable_latency_ms`: a float """ db = rsc.pymongo_test db.read_preference = mode if isinstance(tag_sets, dict): tag_sets = [tag_sets] db.tag_sets = tag_sets or [{}] db.secondary_acceptable_latency_ms = secondary_acceptable_latency_ms cursor = db.test.find() try: try: cursor.next() except StopIteration: # No documents in collection, that's fine pass return cursor._Cursor__connection_id except AutoReconnect: return None def assertReadFrom(testcase, rsc, member, *args, **kwargs): """Check that a query with the given mode, tag_sets, and secondary_acceptable_latency_ms reads from the expected replica-set member :Parameters: - `testcase`: A unittest.TestCase - `rsc`: A MongoReplicaSetClient - `member`: A host:port expected to be used - `mode`: A ReadPreference - `tag_sets` (optional): List of dicts of tags for data-center-aware reads - `secondary_acceptable_latency_ms` (optional): a float """ for _ in range(10): testcase.assertEqual(member, read_from_which_host(rsc, *args, **kwargs)) def assertReadFromAll(testcase, rsc, members, *args, **kwargs): """Check that a query with the given mode, tag_sets, and secondary_acceptable_latency_ms reads from all members in a set, and only members in that set. :Parameters: - `testcase`: A unittest.TestCase - `rsc`: A MongoReplicaSetClient - `members`: Sequence of host:port expected to be used - `mode`: A ReadPreference - `tag_sets` (optional): List of dicts of tags for data-center-aware reads - `secondary_acceptable_latency_ms` (optional): a float """ members = set(members) used = set() for _ in range(100): used.add(read_from_which_host(rsc, *args, **kwargs)) testcase.assertEqual(members, used) def get_pool(client): if isinstance(client, MongoClient): return client._MongoClient__pool elif isinstance(client, MongoReplicaSetClient): rs_state = client._MongoReplicaSetClient__rs_state return rs_state.primary_member.pool else: raise TypeError(str(client)) def pools_from_rs_client(client): """Get Pool instances from a MongoReplicaSetClient or ReplicaSetConnection. """ return [ member.pool for member in client._MongoReplicaSetClient__rs_state.members] class TestRequestMixin(object): """Inherit from this class and from unittest.TestCase to get some convenient methods for testing connection pools and requests """ def get_sock(self, pool): # MongoClient calls Pool.get_socket((host, port)), whereas RSC sets # Pool.pair at construction-time and just calls Pool.get_socket(). # Deal with either case so we can use TestRequestMixin to test pools # from MongoClient and from RSC. if not pool.pair: sock_info = pool.get_socket((host, port)) else: sock_info = pool.get_socket() return sock_info def assertSameSock(self, pool): sock_info0 = self.get_sock(pool) sock_info1 = self.get_sock(pool) self.assertEqual(sock_info0, sock_info1) pool.maybe_return_socket(sock_info0) pool.maybe_return_socket(sock_info1) def assertDifferentSock(self, pool): sock_info0 = self.get_sock(pool) sock_info1 = self.get_sock(pool) self.assertNotEqual(sock_info0, sock_info1) pool.maybe_return_socket(sock_info0) pool.maybe_return_socket(sock_info1) def assertNoRequest(self, pool): self.assertEqual(NO_REQUEST, pool._get_request_state()) def assertNoSocketYet(self, pool): self.assertEqual(NO_SOCKET_YET, pool._get_request_state()) def assertRequestSocket(self, pool): self.assertTrue(isinstance(pool._get_request_state(), SocketInfo)) def assertInRequestAndSameSock(self, client, pools): self.assertTrue(client.in_request()) if not isinstance(pools, list): pools = [pools] for pool in pools: self.assertTrue(pool.in_request()) self.assertSameSock(pool) def assertNotInRequestAndDifferentSock(self, client, pools): self.assertFalse(client.in_request()) if not isinstance(pools, list): pools = [pools] for pool in pools: self.assertFalse(pool.in_request()) self.assertDifferentSock(pool) pymongo-2.6.3/test/version.py000066400000000000000000000033151223300253600162260ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Some tools for running tests based on MongoDB server version.""" def _padded(iter, length, padding=0): l = list(iter) if len(l) < length: for _ in range(length - len(l)): l.append(0) return l def _parse_version_string(version_string): mod = 0 if version_string.endswith("+"): version_string = version_string[0:-1] mod = 1 elif version_string.endswith("-pre-"): version_string = version_string[0:-5] mod = -1 elif version_string.endswith("-"): version_string = version_string[0:-1] mod = -1 # Deal with '-rcX' substrings if version_string.find('-rc') != -1: version_string = version_string[0:version_string.find('-rc')] mod = -1 version = [int(part) for part in version_string.split(".")] version = _padded(version, 3) version.append(mod) return tuple(version) # Note this is probably broken for very old versions of the database... def version(client): return _parse_version_string(client.server_info()["version"]) def at_least(client, min_version): return version(client) >= tuple(_padded(min_version, 4)) pymongo-2.6.3/tools/000077500000000000000000000000001223300253600143465ustar00rootroot00000000000000pymongo-2.6.3/tools/README.rst000066400000000000000000000001171223300253600160340ustar00rootroot00000000000000Tools ===== This directory contains tools for use with the ``pymongo`` module. pymongo-2.6.3/tools/benchmark.py000066400000000000000000000134761223300253600166650ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """MongoDB benchmarking suite.""" import time import sys sys.path[0:0] = [""] import datetime import cProfile from pymongo import mongo_client from pymongo import ASCENDING trials = 2 per_trial = 5000 batch_size = 100 small = {} medium = {"integer": 5, "number": 5.05, "boolean": False, "array": ["test", "benchmark"] } # this is similar to the benchmark data posted to the user list large = {"base_url": "http://www.example.com/test-me", "total_word_count": 6743, "access_time": datetime.datetime.utcnow(), "meta_tags": {"description": "i am a long description string", "author": "Holly Man", "dynamically_created_meta_tag": "who know\n what" }, "page_structure": {"counted_tags": 3450, "no_of_js_attached": 10, "no_of_images": 6 }, "harvested_words": ["10gen", "web", "open", "source", "application", "paas", "platform-as-a-service", "technology", "helps", "developers", "focus", "building", "mongodb", "mongo"] * 20 } def setup_insert(db, collection, object): db.drop_collection(collection) def insert(db, collection, object): for i in range(per_trial): to_insert = object.copy() to_insert["x"] = i db[collection].insert(to_insert) def insert_batch(db, collection, object): for i in range(per_trial / batch_size): db[collection].insert([object] * batch_size) def find_one(db, collection, x): for _ in range(per_trial): db[collection].find_one({"x": x}) def find(db, collection, x): for _ in range(per_trial): for _ in db[collection].find({"x": x}): pass def timed(name, function, args=[], setup=None): times = [] for _ in range(trials): if setup: setup(*args) start = time.time() function(*args) times.append(time.time() - start) best_time = min(times) print "%s%d" % (name + (60 - len(name)) * ".", per_trial / best_time) return best_time def main(): c = mongo_client.MongoClient(connectTimeoutMS=60*1000) # jack up timeout c.drop_database("benchmark") db = c.benchmark timed("insert (small, no index)", insert, [db, 'small_none', small], setup_insert) timed("insert (medium, no index)", insert, [db, 'medium_none', medium], setup_insert) timed("insert (large, no index)", insert, [db, 'large_none', large], setup_insert) db.small_index.create_index("x", ASCENDING) timed("insert (small, indexed)", insert, [db, 'small_index', small]) db.medium_index.create_index("x", ASCENDING) timed("insert (medium, indexed)", insert, [db, 'medium_index', medium]) db.large_index.create_index("x", ASCENDING) timed("insert (large, indexed)", insert, [db, 'large_index', large]) timed("batch insert (small, no index)", insert_batch, [db, 'small_bulk', small], setup_insert) timed("batch insert (medium, no index)", insert_batch, [db, 'medium_bulk', medium], setup_insert) timed("batch insert (large, no index)", insert_batch, [db, 'large_bulk', large], setup_insert) timed("find_one (small, no index)", find_one, [db, 'small_none', per_trial / 2]) timed("find_one (medium, no index)", find_one, [db, 'medium_none', per_trial / 2]) timed("find_one (large, no index)", find_one, [db, 'large_none', per_trial / 2]) timed("find_one (small, indexed)", find_one, [db, 'small_index', per_trial / 2]) timed("find_one (medium, indexed)", find_one, [db, 'medium_index', per_trial / 2]) timed("find_one (large, indexed)", find_one, [db, 'large_index', per_trial / 2]) timed("find (small, no index)", find, [db, 'small_none', per_trial / 2]) timed("find (medium, no index)", find, [db, 'medium_none', per_trial / 2]) timed("find (large, no index)", find, [db, 'large_none', per_trial / 2]) timed("find (small, indexed)", find, [db, 'small_index', per_trial / 2]) timed("find (medium, indexed)", find, [db, 'medium_index', per_trial / 2]) timed("find (large, indexed)", find, [db, 'large_index', per_trial / 2]) # timed("find range (small, no index)", find, # [db, 'small_none', # {"$gt": per_trial / 4, "$lt": 3 * per_trial / 4}]) # timed("find range (medium, no index)", find, # [db, 'medium_none', # {"$gt": per_trial / 4, "$lt": 3 * per_trial / 4}]) # timed("find range (large, no index)", find, # [db, 'large_none', # {"$gt": per_trial / 4, "$lt": 3 * per_trial / 4}]) timed("find range (small, indexed)", find, [db, 'small_index', {"$gt": per_trial / 2, "$lt": per_trial / 2 + batch_size}]) timed("find range (medium, indexed)", find, [db, 'medium_index', {"$gt": per_trial / 2, "$lt": per_trial / 2 + batch_size}]) timed("find range (large, indexed)", find, [db, 'large_index', {"$gt": per_trial / 2, "$lt": per_trial / 2 + batch_size}]) if __name__ == "__main__": # cProfile.run("main()") main() pymongo-2.6.3/tools/clean.py000066400000000000000000000021261223300253600160030ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Clean up script for build artifacts. Only really intended to be used by internal build scripts. """ import os import sys try: os.remove("pymongo/_cmessage.so") os.remove("bson/_cbson.so") except: pass try: os.remove("pymongo/_cmessage.pyd") os.remove("bson/_cbson.pyd") except: pass try: from pymongo import _cmessage sys.exit("could still import _cmessage") except ImportError: pass try: from bson import _cbson sys.exit("could still import _cbson") except ImportError: pass pymongo-2.6.3/tools/fail_if_no_c.py000066400000000000000000000015101223300253600173040ustar00rootroot00000000000000# Copyright 2009-2012 10gen, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Fail if the C extension module doesn't exist. Only really intended to be used by internal build scripts. """ import sys sys.path[0:0] = [""] import bson import pymongo if not pymongo.has_c() or not bson.has_c(): sys.exit("could not load C extensions")