django-picklefield-1.0.0/0000775000175000017500000000000013125310527015520 5ustar simonsimon00000000000000django-picklefield-1.0.0/src/0000775000175000017500000000000013125310527016307 5ustar simonsimon00000000000000django-picklefield-1.0.0/src/django_picklefield.egg-info/0000775000175000017500000000000013125310527023576 5ustar simonsimon00000000000000django-picklefield-1.0.0/src/django_picklefield.egg-info/SOURCES.txt0000664000175000017500000000053013125310527025460 0ustar simonsimon00000000000000README.rst setup.cfg setup.py src/django_picklefield.egg-info/PKG-INFO src/django_picklefield.egg-info/SOURCES.txt src/django_picklefield.egg-info/dependency_links.txt src/django_picklefield.egg-info/top_level.txt src/picklefield/__init__.py src/picklefield/compat.py src/picklefield/fields.py src/picklefield/models.py src/picklefield/tests.pydjango-picklefield-1.0.0/src/django_picklefield.egg-info/PKG-INFO0000664000175000017500000002460613125310527024703 0ustar simonsimon00000000000000Metadata-Version: 1.1 Name: django-picklefield Version: 1.0.0 Summary: Pickled object field for Django Home-page: http://github.com/gintas/django-picklefield Author: Gintautas Miliauskas Author-email: gintautas@miliauskas.lt License: UNKNOWN Description: .. image:: https://travis-ci.org/gintas/django-picklefield.svg?branch=master :target: https://travis-ci.org/gintas/django-picklefield .. image:: https://coveralls.io/repos/gintas/django-picklefield/badge.svg?branch=master&service=github :target: https://coveralls.io/github/gintas/django-picklefield?branch=master ----- About ----- **django-picklefield** provides an implementation of a pickled object field. Such fields can contain any picklable objects. The implementation is taken and adopted from `Django snippet #1694`_ by Taavi Taijala, which is in turn based on `Django snippet #513`_ by Oliver Beattie. django-picklefield is available under the MIT license. .. _Django snippet #1694: http://www.djangosnippets.org/snippets/1694/ .. _Django snippet #513: http://www.djangosnippets.org/snippets/513/ ----- Usage ----- First of all, you need to have **django-picklefield** installed; for your convenience, recent versions should be available from PyPI. To use, just define a field in your model:: >>> from picklefield.fields import PickledObjectField ... class SomeObject(models.Model): ... args = PickledObjectField() and assign whatever you like (as long as it's picklable) to the field:: >>> obj = SomeObject() >>> obj.args = ['fancy', {'objects': 'inside'}] >>> obj.save() ----- Notes ----- If you need to serialize an object with a PickledObjectField for transmission to the browser, you may need to subclass the field and override the ``value_to_string()`` method. Currently pickle fields are serialized as base64-encoded pickles, which allows reliable deserialization, but such a format is not convenient for parsing in the browser. By overriding ``value_to_string()`` you can choose a more convenient serialization format. Fields now accept the boolean key word argument `copy`, which defaults to `True`. The `copy` is necessary for lookups to work correctly. If you don't care about performing lookups on the picklefield, you can set `copy=False` to save on some memory usage. This an be especially beneficial for very large object trees. ------------- Running tests ------------- The full test suite can be run with `Tox`_:: >>> pip install tox >>> tox .. _Tox: https://testrun.org/tox/latest/ -------------- Original notes -------------- Here are the notes by taavi223, the original author: Incredibly useful for storing just about anything in the database (provided it is Pickle-able, of course) when there isn't a 'proper' field for the job. PickledObjectField is database-agnostic, and should work with any database backend you can throw at it. You can pass in any Python object and it will automagically be converted behind the scenes. You never have to manually pickle or unpickle anything. Also works fine when querying; supports exact, in, and isnull lookups. It should be noted, however, that calling QuerySet.values() will only return the encoded data, not the original Python object. This PickledObjectField has a few improvements over the one in snippet #513. This one solves the DjangoUnicodeDecodeError problem when saving an object containing non-ASCII data by base64 encoding the pickled output stream. This ensures that all stored data is ASCII, eliminating the problem. PickledObjectField will now optionally use zlib to compress (and uncompress) pickled objects on the fly. This can be set per-field using the keyword argument "compress=True". For most items this is probably not worth the small performance penalty, but for Models with larger objects, it can be a real space saver. You can also now specify the pickle protocol per-field, using the protocol keyword argument. The default of 2 should always work, unless you are trying to access the data from outside of the Django ORM. Worked around a rare issue when using the cPickle and performing lookups of complex data types. In short, cPickle would sometimes output different streams for the same object depending on how it was referenced. This of course could cause lookups for complex objects to fail, even when a matching object exists. See the docstrings and tests for more information. You can now use the isnull lookup and have it function as expected. A consequence of this is that by default, PickledObjectField has null=True set (you can of course pass null=False if you want to change that). If null=False is set (the default for fields), then you wouldn't be able to store a Python None value, since None values aren't pickled or encoded (this in turn is what makes the isnull lookup possible). You can now pass in an object as the default argument for the field without it being converted to a unicode string first. If you pass in a callable though, the field will still call it. It will not try to pickle and encode it. You can manually import dbsafe_encode and dbsafe_decode from fields.py if you want to encode and decode objects yourself. This is mostly useful for decoding values returned from calling QuerySet.values(), which are still encoded strings. Note: If you are trying to store other django models in the PickledObjectField, please see the comments for a discussion on the problems associated with doing that. The easy solution is to put django models into a list or tuple before assigning them to the PickledObjectField. Update 9/2/09: Fixed the value_to_string method so that serialization should now work as expected. Also added deepcopy back into dbsafe_encode, fixing #4 above, since deepcopy had somehow managed to remove itself. This means that lookups should once again work as expected in all situations. Also made the field editable=False by default (which I swear I already did once before!) since it is never a good idea to have a PickledObjectField be user editable. ------- Changes ------- Changes in version 1.0.0 ======================== * Added a new option to prevent a copy of the object before pickling: `copy=True` * Dropped support for Django 1.4 * Dropped support for Django 1.7 * Dropped support for Python 3.2 * Added support for Python 3.6 Changes in version 0.3.2 ======================== * Dropped support for Django 1.3. * Dropped support for Python 2.5. * Silenced deprecation warnings on Django 1.8+. Changes in version 0.3.1 ======================== * Favor the built in json module (thanks to Simon Charette). Changes in version 0.3.0 ======================== * Python 3 support (thanks to Rafal Stozek). Changes in version 0.2.0 ======================== * Allow pickling of subclasses of django.db.models.Model (thanks to Simon Charette). Changes in version 0.1.9 ======================== * Added `connection` and `prepared` parameters to `get_db_prep_value()` too (thanks to Matthew Schinckel). Changes in version 0.1.8 ======================== * Updated link to code repository. Changes in version 0.1.7 ======================== * Added `connection` and `prepared` parameters to `get_db_prep_lookup()` to get rid of deprecation warnings in Django 1.2. Changes in version 0.1.6 ======================== * Fixed South support (thanks aehlke@github). Changes in version 0.1.5 ======================== * Added support for South. * Changed default to null=False, as is common throughout Django. Changes in version 0.1.4 ======================== * Updated copyright statements. Changes in version 0.1.3 ======================== * Updated serialization tests (thanks to Michael Fladischer). Changes in version 0.1.2 ======================== * Added Simplified BSD licence. Changes in version 0.1.1 ======================== * Added test for serialization. * Added note about JSON serialization for browser. * Added support for different pickle protocol versions (thanks to Michael Fladischer). Changes in version 0.1 ====================== * First public release. -------- Feedback -------- There is a home page with instructions on how to access the code repository. Send feedback and suggestions to gintautas@miliauskas.lt . Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Framework :: Django Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: MIT License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 django-picklefield-1.0.0/src/django_picklefield.egg-info/dependency_links.txt0000664000175000017500000000000113125310527027644 0ustar simonsimon00000000000000 django-picklefield-1.0.0/src/django_picklefield.egg-info/top_level.txt0000664000175000017500000000001413125310527026323 0ustar simonsimon00000000000000picklefield django-picklefield-1.0.0/src/picklefield/0000775000175000017500000000000013125310527020562 5ustar simonsimon00000000000000django-picklefield-1.0.0/src/picklefield/__init__.py0000664000175000017500000000017713125306310022673 0ustar simonsimon00000000000000"""Pickle field implementation for Django.""" DEFAULT_PROTOCOL = 2 from picklefield.fields import PickledObjectField # noqa django-picklefield-1.0.0/src/picklefield/tests.py0000664000175000017500000002264113125306310022276 0ustar simonsimon00000000000000"""Unit tests for django-picklefield.""" import json from datetime import date from unittest import skipIf import django from django.core import serializers from django.db import models from django.test import TestCase from .fields import PickledObjectField, dbsafe_encode, wrap_conflictual_object S1 = 'Hello World' T1 = (1, 2, 3, 4, 5) L1 = [1, 2, 3, 4, 5] D1 = {1: 1, 2: 4, 3: 6, 4: 8, 5: 10} D2 = {1: 2, 2: 4, 3: 6, 4: 8, 5: 10} class TestCopyDataType(str): def __deepcopy__(self, memo): raise ValueError('Please dont copy me') class TestingModel(models.Model): pickle_field = PickledObjectField() compressed_pickle_field = PickledObjectField(compress=True) default_pickle_field = PickledObjectField(default=(D1, S1, T1, L1)) callable_pickle_field = PickledObjectField(default=date.today) non_copying_field = PickledObjectField(copy=False, default=TestCopyDataType('boom!')) class MinimalTestingModel(models.Model): pickle_field = PickledObjectField() class TestCustomDataType(str): pass class PickledObjectFieldTests(TestCase): def setUp(self): self.testing_data = (D2, S1, T1, L1, TestCustomDataType(S1), MinimalTestingModel) return super(PickledObjectFieldTests, self).setUp() def testDataIntegrity(self): """ Tests that data remains the same when saved to and fetched from the database, whether compression is enabled or not. """ for value in self.testing_data: model_test = TestingModel(pickle_field=value, compressed_pickle_field=value) model_test.save() model_test = TestingModel.objects.get(id__exact=model_test.id) # Make sure that both the compressed and uncompressed fields return # the same data, even thought it's stored differently in the DB. self.assertEqual(value, model_test.pickle_field) self.assertEqual(value, model_test.compressed_pickle_field) # Make sure we can also retrieve the model model_test.save() model_test.delete() # Make sure the default value for default_pickled_field gets stored # correctly and that it isn't converted to a string. model_test = TestingModel() model_test.save() model_test = TestingModel.objects.get(id__exact=model_test.id) self.assertEqual((D1, S1, T1, L1), model_test.default_pickle_field) self.assertEqual(date.today(), model_test.callable_pickle_field) def testLookups(self): """ Tests that lookups can be performed on data once stored in the database, whether compression is enabled or not. One problem with cPickle is that it will sometimes output different streams for the same object, depending on how they are referenced. It should be noted though, that this does not happen for every object, but usually only with more complex ones. >>> from pickle import dumps >>> t = ({1: 1, 2: 4, 3: 6, 4: 8, 5: 10}, \ ... 'Hello World', (1, 2, 3, 4, 5), [1, 2, 3, 4, 5]) >>> dumps(({1: 1, 2: 4, 3: 6, 4: 8, 5: 10}, \ ... 'Hello World', (1, 2, 3, 4, 5), [1, 2, 3, 4, 5])) "((dp0\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\nI10\nsS'Hello World'\np1\n(I1\nI2\nI3\nI4\nI5\ntp2\n(lp3\nI1\naI2\naI3\naI4\naI5\natp4\n." >>> dumps(t) "((dp0\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\nI10\nsS'Hello World'\np1\n(I1\nI2\nI3\nI4\nI5\ntp2\n(lp3\nI1\naI2\naI3\naI4\naI5\natp4\n." >>> # Both dumps() are the same using pickle. >>> from cPickle import dumps >>> t = ({1: 1, 2: 4, 3: 6, 4: 8, 5: 10}, 'Hello World', (1, 2, 3, 4, 5), [1, 2, 3, 4, 5]) >>> dumps(({1: 1, 2: 4, 3: 6, 4: 8, 5: 10}, 'Hello World', (1, 2, 3, 4, 5), [1, 2, 3, 4, 5])) "((dp1\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\nI10\nsS'Hello World'\np2\n(I1\nI2\nI3\nI4\nI5\ntp3\n(lp4\nI1\naI2\naI3\naI4\naI5\nat." >>> dumps(t) "((dp1\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\nI10\nsS'Hello World'\n(I1\nI2\nI3\nI4\nI5\nt(lp2\nI1\naI2\naI3\naI4\naI5\natp3\n." >>> # But with cPickle the two dumps() are not the same! >>> # Both will generate the same object when loads() is called though. We can solve this by calling deepcopy() on the value before pickling it, as this copies everything to a brand new data structure. >>> from cPickle import dumps >>> from copy import deepcopy >>> t = ({1: 1, 2: 4, 3: 6, 4: 8, 5: 10}, 'Hello World', (1, 2, 3, 4, 5), [1, 2, 3, 4, 5]) >>> dumps(deepcopy(({1: 1, 2: 4, 3: 6, 4: 8, 5: 10}, 'Hello World', (1, 2, 3, 4, 5), [1, 2, 3, 4, 5]))) "((dp1\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\nI10\nsS'Hello World'\np2\n(I1\nI2\nI3\nI4\nI5\ntp3\n(lp4\nI1\naI2\naI3\naI4\naI5\nat." >>> dumps(deepcopy(t)) "((dp1\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\nI10\nsS'Hello World'\np2\n(I1\nI2\nI3\nI4\nI5\ntp3\n(lp4\nI1\naI2\naI3\naI4\naI5\nat." >>> # Using deepcopy() beforehand means that now both dumps() are idential. >>> # It may not be necessary, but deepcopy() ensures that lookups will always work. Unfortunately calling copy() alone doesn't seem to fix the problem as it lies primarily with complex data types. >>> from cPickle import dumps >>> from copy import copy >>> t = ({1: 1, 2: 4, 3: 6, 4: 8, 5: 10}, 'Hello World', (1, 2, 3, 4, 5), [1, 2, 3, 4, 5]) >>> dumps(copy(({1: 1, 2: 4, 3: 6, 4: 8, 5: 10}, 'Hello World', (1, 2, 3, 4, 5), [1, 2, 3, 4, 5]))) "((dp1\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\nI10\nsS'Hello World'\np2\n(I1\nI2\nI3\nI4\nI5\ntp3\n(lp4\nI1\naI2\naI3\naI4\naI5\nat." >>> dumps(copy(t)) "((dp1\nI1\nI1\nsI2\nI4\nsI3\nI6\nsI4\nI8\nsI5\nI10\nsS'Hello World'\n(I1\nI2\nI3\nI4\nI5\nt(lp2\nI1\naI2\naI3\naI4\naI5\natp3\n." """ # noqa for value in self.testing_data: model_test = TestingModel(pickle_field=value, compressed_pickle_field=value) model_test.save() # Make sure that we can do an ``exact`` lookup by both the # pickle_field and the compressed_pickle_field. wrapped_value = wrap_conflictual_object(value) model_test = TestingModel.objects.get(pickle_field__exact=wrapped_value, compressed_pickle_field__exact=wrapped_value) self.assertEqual(value, model_test.pickle_field) self.assertEqual(value, model_test.compressed_pickle_field) # Make sure that ``in`` lookups also work correctly. model_test = TestingModel.objects.get(pickle_field__in=[wrapped_value], compressed_pickle_field__in=[wrapped_value]) self.assertEqual(value, model_test.pickle_field) self.assertEqual(value, model_test.compressed_pickle_field) # Make sure that ``is_null`` lookups are working. self.assertEqual(1, TestingModel.objects.filter(pickle_field__isnull=False).count()) self.assertEqual(0, TestingModel.objects.filter(pickle_field__isnull=True).count()) model_test.delete() # Make sure that lookups of the same value work, even when referenced # differently. See the above docstring for more info on the issue. value = (D1, S1, T1, L1) model_test = TestingModel(pickle_field=value, compressed_pickle_field=value) model_test.save() # Test lookup using an assigned variable. model_test = TestingModel.objects.get(pickle_field__exact=value) self.assertEqual(value, model_test.pickle_field) # Test lookup using direct input of a matching value. model_test = TestingModel.objects.get( pickle_field__exact=(D1, S1, T1, L1), compressed_pickle_field__exact=(D1, S1, T1, L1), ) self.assertEqual(value, model_test.pickle_field) model_test.delete() @skipIf(django.VERSION[:2] < (1, 5), 'Skip for older Django versions') def testLimitLookupsType(self): """ Test that picklefield supports lookup type limit """ with self.assertRaisesMessage(TypeError, 'Lookup type gte is not supported'): TestingModel.objects.filter(pickle_field__gte=1) def testSerialization(self): model = MinimalTestingModel(pk=1, pickle_field={'foo': 'bar'}) serialized = serializers.serialize('json', [model]) data = json.loads(serialized) # determine output at runtime, because pickle output in python 3 # is different (but compatible with python 2) p = dbsafe_encode({'foo': 'bar'}) self.assertEqual(data, [{ 'pk': 1, 'model': 'picklefield.minimaltestingmodel', 'fields': {"pickle_field": p}}, ]) for deserialized_test in serializers.deserialize('json', serialized): self.assertEqual(deserialized_test.object, model) def testNoCopy(self): TestingModel.objects.create( pickle_field='Copy Me', compressed_pickle_field='Copy Me', non_copying_field=TestCopyDataType('Dont Copy Me') ) with self.assertRaises(ValueError): TestingModel.objects.create( pickle_field=TestCopyDataType('BOOM!'), compressed_pickle_field='Copy Me', non_copying_field='Dont copy me' ) django-picklefield-1.0.0/src/picklefield/models.py0000664000175000017500000000000013125306310022400 0ustar simonsimon00000000000000django-picklefield-1.0.0/src/picklefield/fields.py0000664000175000017500000001520113125310442022375 0ustar simonsimon00000000000000"""Pickle field implementation for Django.""" from base64 import b64decode, b64encode from copy import deepcopy from zlib import compress, decompress from django.db import models from django.utils.encoding import force_text from . import DEFAULT_PROTOCOL from .compat import dumps, loads class PickledObject(str): """ A subclass of string so it can be told whether a string is a pickled object or not (if the object is an instance of this class then it must [well, should] be a pickled one). Only really useful for passing pre-encoded values to ``default`` with ``dbsafe_encode``, not that doing so is necessary. If you remove PickledObject and its references, you won't be able to pass in pre-encoded values anymore, but you can always just pass in the python objects themselves. """ class _ObjectWrapper(object): """ A class used to wrap object that have properties that may clash with the ORM internals. For example, objects with the `prepare_database_save` property such as `django.db.Model` subclasses won't work under certain conditions and the same apply for trying to retrieve any `callable` object. """ __slots__ = ('_obj',) def __init__(self, obj): self._obj = obj def wrap_conflictual_object(obj): if hasattr(obj, 'prepare_database_save') or callable(obj): obj = _ObjectWrapper(obj) return obj def dbsafe_encode(value, compress_object=False, pickle_protocol=DEFAULT_PROTOCOL, copy=True): # We use deepcopy() here to avoid a problem with cPickle, where dumps # can generate different character streams for same lookup value if # they are referenced differently. # The reason this is important is because we do all of our lookups as # simple string matches, thus the character streams must be the same # for the lookups to work properly. See tests.py for more information. if copy: # Copy can be very expensive if users aren't going to perform lookups # on the value anyway. value = deepcopy(value) value = dumps(value, protocol=pickle_protocol) if compress_object: value = compress(value) value = b64encode(value).decode() # decode bytes to str return PickledObject(value) def dbsafe_decode(value, compress_object=False): value = value.encode() # encode str to bytes value = b64decode(value) if compress_object: value = decompress(value) return loads(value) class PickledObjectField(models.Field): """ A field that will accept *any* python object and store it in the database. PickledObjectField will optionally compress its values if declared with the keyword argument ``compress=True``. Does not actually encode and compress ``None`` objects (although you can still do lookups using None). This way, it is still possible to use the ``isnull`` lookup type correctly. """ def __init__(self, *args, **kwargs): self.compress = kwargs.pop('compress', False) self.protocol = kwargs.pop('protocol', DEFAULT_PROTOCOL) self.copy = kwargs.pop('copy', True) kwargs.setdefault('editable', False) super(PickledObjectField, self).__init__(*args, **kwargs) def get_default(self): """ Returns the default value for this field. The default implementation on models.Field calls force_unicode on the default, which means you can't set arbitrary Python objects as the default. To fix this, we just return the value without calling force_unicode on it. Note that if you set a callable as a default, the field will still call it. It will *not* try to pickle and encode it. """ if self.has_default(): if callable(self.default): return self.default() return self.default # If the field doesn't have a default, then we punt to models.Field. return super(PickledObjectField, self).get_default() def to_python(self, value): """ B64decode and unpickle the object, optionally decompressing it. If an error is raised in de-pickling and we're sure the value is a definite pickle, the error is allowed to propagate. If we aren't sure if the value is a pickle or not, then we catch the error and return the original value instead. """ if value is not None: try: value = dbsafe_decode(value, self.compress) except: # If the value is a definite pickle; and an error is raised in # de-pickling it should be allowed to propogate. if isinstance(value, PickledObject): raise else: if isinstance(value, _ObjectWrapper): return value._obj return value def pre_save(self, model_instance, add): value = super(PickledObjectField, self).pre_save(model_instance, add) return wrap_conflictual_object(value) def from_db_value(self, value, expression, connection, context): return self.to_python(value) def get_db_prep_value(self, value, connection=None, prepared=False): """ Pickle and b64encode the object, optionally compressing it. The pickling protocol is specified explicitly (by default 2), rather than as -1 or HIGHEST_PROTOCOL, because we don't want the protocol to change over time. If it did, ``exact`` and ``in`` lookups would likely fail, since pickle would now be generating a different string. """ if value is not None and not isinstance(value, PickledObject): # We call force_text here explicitly, so that the encoded string # isn't rejected by the postgresql_psycopg2 backend. Alternatively, # we could have just registered PickledObject with the psycopg # marshaller (telling it to store it like it would a string), but # since both of these methods result in the same value being stored, # doing things this way is much easier. value = force_text(dbsafe_encode(value, self.compress, self.protocol, self.copy)) return value def value_to_string(self, obj): value = self.value_from_object(obj) return self.get_db_prep_value(value) def get_internal_type(self): return 'TextField' def get_lookup(self, lookup_name): """ We need to limit the lookup types. """ if lookup_name not in ['exact', 'in', 'isnull']: raise TypeError('Lookup type %s is not supported.' % lookup_name) return super(PickledObjectField, self).get_lookup(lookup_name) django-picklefield-1.0.0/src/picklefield/compat.py0000664000175000017500000000025613125310442022416 0ustar simonsimon00000000000000# python 3.x does not have cPickle module try: # cpython 2.x from cPickle import loads, dumps # noqa except ImportError: from pickle import loads, dumps # noqa django-picklefield-1.0.0/README.rst0000664000175000017500000001755213125310442017215 0ustar simonsimon00000000000000.. image:: https://travis-ci.org/gintas/django-picklefield.svg?branch=master :target: https://travis-ci.org/gintas/django-picklefield .. image:: https://coveralls.io/repos/gintas/django-picklefield/badge.svg?branch=master&service=github :target: https://coveralls.io/github/gintas/django-picklefield?branch=master ----- About ----- **django-picklefield** provides an implementation of a pickled object field. Such fields can contain any picklable objects. The implementation is taken and adopted from `Django snippet #1694`_ by Taavi Taijala, which is in turn based on `Django snippet #513`_ by Oliver Beattie. django-picklefield is available under the MIT license. .. _Django snippet #1694: http://www.djangosnippets.org/snippets/1694/ .. _Django snippet #513: http://www.djangosnippets.org/snippets/513/ ----- Usage ----- First of all, you need to have **django-picklefield** installed; for your convenience, recent versions should be available from PyPI. To use, just define a field in your model:: >>> from picklefield.fields import PickledObjectField ... class SomeObject(models.Model): ... args = PickledObjectField() and assign whatever you like (as long as it's picklable) to the field:: >>> obj = SomeObject() >>> obj.args = ['fancy', {'objects': 'inside'}] >>> obj.save() ----- Notes ----- If you need to serialize an object with a PickledObjectField for transmission to the browser, you may need to subclass the field and override the ``value_to_string()`` method. Currently pickle fields are serialized as base64-encoded pickles, which allows reliable deserialization, but such a format is not convenient for parsing in the browser. By overriding ``value_to_string()`` you can choose a more convenient serialization format. Fields now accept the boolean key word argument `copy`, which defaults to `True`. The `copy` is necessary for lookups to work correctly. If you don't care about performing lookups on the picklefield, you can set `copy=False` to save on some memory usage. This an be especially beneficial for very large object trees. ------------- Running tests ------------- The full test suite can be run with `Tox`_:: >>> pip install tox >>> tox .. _Tox: https://testrun.org/tox/latest/ -------------- Original notes -------------- Here are the notes by taavi223, the original author: Incredibly useful for storing just about anything in the database (provided it is Pickle-able, of course) when there isn't a 'proper' field for the job. PickledObjectField is database-agnostic, and should work with any database backend you can throw at it. You can pass in any Python object and it will automagically be converted behind the scenes. You never have to manually pickle or unpickle anything. Also works fine when querying; supports exact, in, and isnull lookups. It should be noted, however, that calling QuerySet.values() will only return the encoded data, not the original Python object. This PickledObjectField has a few improvements over the one in snippet #513. This one solves the DjangoUnicodeDecodeError problem when saving an object containing non-ASCII data by base64 encoding the pickled output stream. This ensures that all stored data is ASCII, eliminating the problem. PickledObjectField will now optionally use zlib to compress (and uncompress) pickled objects on the fly. This can be set per-field using the keyword argument "compress=True". For most items this is probably not worth the small performance penalty, but for Models with larger objects, it can be a real space saver. You can also now specify the pickle protocol per-field, using the protocol keyword argument. The default of 2 should always work, unless you are trying to access the data from outside of the Django ORM. Worked around a rare issue when using the cPickle and performing lookups of complex data types. In short, cPickle would sometimes output different streams for the same object depending on how it was referenced. This of course could cause lookups for complex objects to fail, even when a matching object exists. See the docstrings and tests for more information. You can now use the isnull lookup and have it function as expected. A consequence of this is that by default, PickledObjectField has null=True set (you can of course pass null=False if you want to change that). If null=False is set (the default for fields), then you wouldn't be able to store a Python None value, since None values aren't pickled or encoded (this in turn is what makes the isnull lookup possible). You can now pass in an object as the default argument for the field without it being converted to a unicode string first. If you pass in a callable though, the field will still call it. It will not try to pickle and encode it. You can manually import dbsafe_encode and dbsafe_decode from fields.py if you want to encode and decode objects yourself. This is mostly useful for decoding values returned from calling QuerySet.values(), which are still encoded strings. Note: If you are trying to store other django models in the PickledObjectField, please see the comments for a discussion on the problems associated with doing that. The easy solution is to put django models into a list or tuple before assigning them to the PickledObjectField. Update 9/2/09: Fixed the value_to_string method so that serialization should now work as expected. Also added deepcopy back into dbsafe_encode, fixing #4 above, since deepcopy had somehow managed to remove itself. This means that lookups should once again work as expected in all situations. Also made the field editable=False by default (which I swear I already did once before!) since it is never a good idea to have a PickledObjectField be user editable. ------- Changes ------- Changes in version 1.0.0 ======================== * Added a new option to prevent a copy of the object before pickling: `copy=True` * Dropped support for Django 1.4 * Dropped support for Django 1.7 * Dropped support for Python 3.2 * Added support for Python 3.6 Changes in version 0.3.2 ======================== * Dropped support for Django 1.3. * Dropped support for Python 2.5. * Silenced deprecation warnings on Django 1.8+. Changes in version 0.3.1 ======================== * Favor the built in json module (thanks to Simon Charette). Changes in version 0.3.0 ======================== * Python 3 support (thanks to Rafal Stozek). Changes in version 0.2.0 ======================== * Allow pickling of subclasses of django.db.models.Model (thanks to Simon Charette). Changes in version 0.1.9 ======================== * Added `connection` and `prepared` parameters to `get_db_prep_value()` too (thanks to Matthew Schinckel). Changes in version 0.1.8 ======================== * Updated link to code repository. Changes in version 0.1.7 ======================== * Added `connection` and `prepared` parameters to `get_db_prep_lookup()` to get rid of deprecation warnings in Django 1.2. Changes in version 0.1.6 ======================== * Fixed South support (thanks aehlke@github). Changes in version 0.1.5 ======================== * Added support for South. * Changed default to null=False, as is common throughout Django. Changes in version 0.1.4 ======================== * Updated copyright statements. Changes in version 0.1.3 ======================== * Updated serialization tests (thanks to Michael Fladischer). Changes in version 0.1.2 ======================== * Added Simplified BSD licence. Changes in version 0.1.1 ======================== * Added test for serialization. * Added note about JSON serialization for browser. * Added support for different pickle protocol versions (thanks to Michael Fladischer). Changes in version 0.1 ====================== * First public release. -------- Feedback -------- There is a home page with instructions on how to access the code repository. Send feedback and suggestions to gintautas@miliauskas.lt . django-picklefield-1.0.0/setup.py0000664000175000017500000000406113125310442017227 0ustar simonsimon00000000000000# Copyright (c) 2009-2010 Gintautas Miliauskas # Copyright (c) 2009, Taavi Taijala # Copyright (c) 2007, Oliver Beattie # # Permission is hereby granted, free of charge, to any person obtaining a copy # of this software and associated documentation files (the "Software"), to deal # in the Software without restriction, including without limitation the rights # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # copies of the Software, and to permit persons to whom the Software is # furnished to do so, subject to the following conditions: # # The above copyright notice and this permission notice shall be included in # all copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN # THE SOFTWARE. import codecs from setuptools import setup, find_packages long_description = codecs.open('README.rst', encoding='utf-8').read() setup( name='django-picklefield', version='1.0.0', description='Pickled object field for Django', long_description=long_description, author='Gintautas Miliauskas', author_email='gintautas@miliauskas.lt', url='http://github.com/gintas/django-picklefield', packages=find_packages('src'), package_dir={'': 'src'}, classifiers=[ 'Development Status :: 5 - Production/Stable', 'Framework :: Django', 'Intended Audience :: Developers', 'License :: OSI Approved :: MIT License', 'Programming Language :: Python', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', ] ) django-picklefield-1.0.0/PKG-INFO0000664000175000017500000002460613125310527016625 0ustar simonsimon00000000000000Metadata-Version: 1.1 Name: django-picklefield Version: 1.0.0 Summary: Pickled object field for Django Home-page: http://github.com/gintas/django-picklefield Author: Gintautas Miliauskas Author-email: gintautas@miliauskas.lt License: UNKNOWN Description: .. image:: https://travis-ci.org/gintas/django-picklefield.svg?branch=master :target: https://travis-ci.org/gintas/django-picklefield .. image:: https://coveralls.io/repos/gintas/django-picklefield/badge.svg?branch=master&service=github :target: https://coveralls.io/github/gintas/django-picklefield?branch=master ----- About ----- **django-picklefield** provides an implementation of a pickled object field. Such fields can contain any picklable objects. The implementation is taken and adopted from `Django snippet #1694`_ by Taavi Taijala, which is in turn based on `Django snippet #513`_ by Oliver Beattie. django-picklefield is available under the MIT license. .. _Django snippet #1694: http://www.djangosnippets.org/snippets/1694/ .. _Django snippet #513: http://www.djangosnippets.org/snippets/513/ ----- Usage ----- First of all, you need to have **django-picklefield** installed; for your convenience, recent versions should be available from PyPI. To use, just define a field in your model:: >>> from picklefield.fields import PickledObjectField ... class SomeObject(models.Model): ... args = PickledObjectField() and assign whatever you like (as long as it's picklable) to the field:: >>> obj = SomeObject() >>> obj.args = ['fancy', {'objects': 'inside'}] >>> obj.save() ----- Notes ----- If you need to serialize an object with a PickledObjectField for transmission to the browser, you may need to subclass the field and override the ``value_to_string()`` method. Currently pickle fields are serialized as base64-encoded pickles, which allows reliable deserialization, but such a format is not convenient for parsing in the browser. By overriding ``value_to_string()`` you can choose a more convenient serialization format. Fields now accept the boolean key word argument `copy`, which defaults to `True`. The `copy` is necessary for lookups to work correctly. If you don't care about performing lookups on the picklefield, you can set `copy=False` to save on some memory usage. This an be especially beneficial for very large object trees. ------------- Running tests ------------- The full test suite can be run with `Tox`_:: >>> pip install tox >>> tox .. _Tox: https://testrun.org/tox/latest/ -------------- Original notes -------------- Here are the notes by taavi223, the original author: Incredibly useful for storing just about anything in the database (provided it is Pickle-able, of course) when there isn't a 'proper' field for the job. PickledObjectField is database-agnostic, and should work with any database backend you can throw at it. You can pass in any Python object and it will automagically be converted behind the scenes. You never have to manually pickle or unpickle anything. Also works fine when querying; supports exact, in, and isnull lookups. It should be noted, however, that calling QuerySet.values() will only return the encoded data, not the original Python object. This PickledObjectField has a few improvements over the one in snippet #513. This one solves the DjangoUnicodeDecodeError problem when saving an object containing non-ASCII data by base64 encoding the pickled output stream. This ensures that all stored data is ASCII, eliminating the problem. PickledObjectField will now optionally use zlib to compress (and uncompress) pickled objects on the fly. This can be set per-field using the keyword argument "compress=True". For most items this is probably not worth the small performance penalty, but for Models with larger objects, it can be a real space saver. You can also now specify the pickle protocol per-field, using the protocol keyword argument. The default of 2 should always work, unless you are trying to access the data from outside of the Django ORM. Worked around a rare issue when using the cPickle and performing lookups of complex data types. In short, cPickle would sometimes output different streams for the same object depending on how it was referenced. This of course could cause lookups for complex objects to fail, even when a matching object exists. See the docstrings and tests for more information. You can now use the isnull lookup and have it function as expected. A consequence of this is that by default, PickledObjectField has null=True set (you can of course pass null=False if you want to change that). If null=False is set (the default for fields), then you wouldn't be able to store a Python None value, since None values aren't pickled or encoded (this in turn is what makes the isnull lookup possible). You can now pass in an object as the default argument for the field without it being converted to a unicode string first. If you pass in a callable though, the field will still call it. It will not try to pickle and encode it. You can manually import dbsafe_encode and dbsafe_decode from fields.py if you want to encode and decode objects yourself. This is mostly useful for decoding values returned from calling QuerySet.values(), which are still encoded strings. Note: If you are trying to store other django models in the PickledObjectField, please see the comments for a discussion on the problems associated with doing that. The easy solution is to put django models into a list or tuple before assigning them to the PickledObjectField. Update 9/2/09: Fixed the value_to_string method so that serialization should now work as expected. Also added deepcopy back into dbsafe_encode, fixing #4 above, since deepcopy had somehow managed to remove itself. This means that lookups should once again work as expected in all situations. Also made the field editable=False by default (which I swear I already did once before!) since it is never a good idea to have a PickledObjectField be user editable. ------- Changes ------- Changes in version 1.0.0 ======================== * Added a new option to prevent a copy of the object before pickling: `copy=True` * Dropped support for Django 1.4 * Dropped support for Django 1.7 * Dropped support for Python 3.2 * Added support for Python 3.6 Changes in version 0.3.2 ======================== * Dropped support for Django 1.3. * Dropped support for Python 2.5. * Silenced deprecation warnings on Django 1.8+. Changes in version 0.3.1 ======================== * Favor the built in json module (thanks to Simon Charette). Changes in version 0.3.0 ======================== * Python 3 support (thanks to Rafal Stozek). Changes in version 0.2.0 ======================== * Allow pickling of subclasses of django.db.models.Model (thanks to Simon Charette). Changes in version 0.1.9 ======================== * Added `connection` and `prepared` parameters to `get_db_prep_value()` too (thanks to Matthew Schinckel). Changes in version 0.1.8 ======================== * Updated link to code repository. Changes in version 0.1.7 ======================== * Added `connection` and `prepared` parameters to `get_db_prep_lookup()` to get rid of deprecation warnings in Django 1.2. Changes in version 0.1.6 ======================== * Fixed South support (thanks aehlke@github). Changes in version 0.1.5 ======================== * Added support for South. * Changed default to null=False, as is common throughout Django. Changes in version 0.1.4 ======================== * Updated copyright statements. Changes in version 0.1.3 ======================== * Updated serialization tests (thanks to Michael Fladischer). Changes in version 0.1.2 ======================== * Added Simplified BSD licence. Changes in version 0.1.1 ======================== * Added test for serialization. * Added note about JSON serialization for browser. * Added support for different pickle protocol versions (thanks to Michael Fladischer). Changes in version 0.1 ====================== * First public release. -------- Feedback -------- There is a home page with instructions on how to access the code repository. Send feedback and suggestions to gintautas@miliauskas.lt . Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Framework :: Django Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: MIT License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 django-picklefield-1.0.0/setup.cfg0000664000175000017500000000036113125310527017341 0ustar simonsimon00000000000000[flake8] max-line-length = 119 [isort] combine_as_imports = true include_trailing_comma = true known_third_party = picklefield,south multi_line_output = 5 not_skip = __init__.py [wheel] universal = 1 [egg_info] tag_build = tag_date = 0