pax_global_header 0000666 0000000 0000000 00000000064 15066622662 0014525 g ustar 00root root 0000000 0000000 52 comment=550a1cfa79c9f085c069dd33e774358acc371717
fralau-super-collections-550a1cf/ 0000775 0000000 0000000 00000000000 15066622662 0017055 5 ustar 00root root 0000000 0000000 fralau-super-collections-550a1cf/.github/ 0000775 0000000 0000000 00000000000 15066622662 0020415 5 ustar 00root root 0000000 0000000 fralau-super-collections-550a1cf/.github/workflows/ 0000775 0000000 0000000 00000000000 15066622662 0022452 5 ustar 00root root 0000000 0000000 fralau-super-collections-550a1cf/.github/workflows/greetings.yml 0000664 0000000 0000000 00000000666 15066622662 0025174 0 ustar 00root root 0000000 0000000 name: Greetings
on: [pull_request_target, issues]
jobs:
greeting:
runs-on: ubuntu-latest
permissions:
issues: write
pull-requests: write
steps:
- uses: actions/first-interaction@v1
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
issue-message: "Welcome to this project and thanks for your contribution!"
pr-message: "Message that will be displayed on users' first pull request"
fralau-super-collections-550a1cf/.github/workflows/test.yml 0000664 0000000 0000000 00000001315 15066622662 0024154 0 ustar 00root root 0000000 0000000 name: Run Pytest on Multiple Python Versions
on:
push:
branches:
- main
- master
pull_request:
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e .[test] # or just pip install pytest if you don't use extras
- name: Run tests
run: pytest
fralau-super-collections-550a1cf/.gitignore 0000664 0000000 0000000 00000000675 15066622662 0021055 0 ustar 00root root 0000000 0000000 # Temporary files
*~
.py~
.python-version
#Byte-compiled led / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# MkDocs
site/
# Mkdocs-Macros
__*/
# Other files (generated by mkdocs-macros or others)
cache*
# JetBrains PyCharm and other IntelliJ based IDEs
.idea/
fralau-super-collections-550a1cf/LICENSE 0000664 0000000 0000000 00000002065 15066622662 0020065 0 ustar 00root root 0000000 0000000 MIT License
Copyright (c) 2024 Laurent Franceschetti
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE. fralau-super-collections-550a1cf/MANIFEST.in 0000664 0000000 0000000 00000000036 15066622662 0020612 0 ustar 00root root 0000000 0000000 include test/*.json test/*.md
fralau-super-collections-550a1cf/README.md 0000664 0000000 0000000 00000037437 15066622662 0020352 0 ustar 00root root 0000000 0000000
# Python Super Collections
**Dictionaries and lists as you dreamed them when you were a kid.**
Instantly Convert json and YAML files into objects with attributes.
```python
import json
from super_collections import SuperDict
with open('my_file.json', 'r') as file:
data = json.load(file)
document = SuperDict(data)
print(document.author) # instead of document['author']
for document in document.blocks: # instead of document['blocks']
...
print(document.blocks[3].name) # instead of document['blocks'][3]['name'] -- eek! π€’
```
________
- [Python Super Collections](#python-super-collections)
- [What are SuperCollections?](#what-are-supercollections)
- [What is the problem?](#what-is-the-problem)
- [Definitions](#definitions)
- [How it works](#how-it-works)
- [Superdicts](#superdicts)
- [Superlists](#superlists)
- [Why Combining SuperDicts with SuperLists?](#why-combining-superdicts-with-superlists)
- [SuperCollection](#supercollection)
- [Factory function](#factory-function)
- [Install](#install)
- [From the repository](#from-the-repository)
- [Usage](#usage)
- [Export to JSON or Hjson](#export-to-json-or-hjson)
- [Export to YAML](#export-to-yaml)
- [Remarks](#remarks)
- [Restrictions](#restrictions)
- [Does it work?](#does-it-work)
- [When are superdictionaries and superlists _not_ recommended?](#when-are-superdictionaries-and-superlists-not-recommended)
- [Shelves and SuperShelves](#shelves-and-supershelves)
- [Metaphor of a shelf](#metaphor-of-a-shelf)
- [Definition of a shelf (data structure)](#definition-of-a-shelf-data-structure)
- [Iteration through a shelf](#iteration-through-a-shelf)
- [SuperShelf](#supershelf)
- [Related data structures and ideas](#related-data-structures-and-ideas)
- [Standard Python](#standard-python)
- [Dot notation on dictionaries](#dot-notation-on-dictionaries)
- [Using superlists to complement superdictionaries](#using-superlists-to-complement-superdictionaries)
## What are SuperCollections?
### What is the problem?
There are several packages that quickly convert json or YAML files into
dictionaries that contain dictionaries, lists etc.
If you want to properly use those data structures in Python, one solution is
to create specific classes.
But sometimes, it is overkill. You just want your app to quickly load
structured data and navigate through them.
That's where the **super-collections** package comes handy.
### Definitions
- A **SuperCollection** is a nested data structure that can encode
any type of information (in the same way as a JSON file).
It is essentially constituted of dictionaries and lists, where
dictionaries can contain lists and vice-versa, and the root can
be either a dictionary or a list.
- An **elementary datatype** is a non-mutable type: str, int, str,
float in Python that is used to build classes.
- A **SuperList** is a list that can contain SuperCollections, or
elementary datatypes.
- A **SuperDict** is a dict that can contain SuperCollections,
or elementary datatypes. A key advantage of SuperDicts over dictionaries,
is that its keys can be accessed as attributes (providing they are
valid Python identifiers and they don't conflict with pre-existing attributes).
## How it works
### Superdicts
> π **Definition** A **superdictionnary** is a dictionary
whose keys (at least those that are valid identifiers) are automatically accessible as attributes, with the **dot notation*.
```python
d = SuperDict({'foo':5, 'bar': 'hello'})
# instead of writing d['foo']
d.foo = 7
```
> Several other languages, such as Javascript, LUA, Ruby, and PHP offer that **dot notation**
> in some form or other. However, implementing that idea is not as
> straightforward as it seems.
> The idea of superdictionaries in Python has been around for some time
> (see the [superdict](https://github.com/itdxer/superdict) packagage by Yuri
> Shevchuk, 2015).
> π **Property** If a SuperDict object contains a value that is itself a dictionary, that dictionary is then converted in turn into a SuperDict.
If the object is not a dict or immediately compatible, it will try the following conversions:
- the methods `.asdict()`, `.dict()` or `dump()`, providing they actually generate a `dict`.
- if the object is a dataclass, it will apply the `asdict()` function on it.
### Superlists
A **superlist** is a list where all dictionary items have been
(automagically) converted to **superdictionnaries**.
> β οΈ **Superlists are indispensable** They were the missing piece of the jigsaw puzzle;
> without them, it is not possible to convert deep data structures into supercollections.
### Why Combining SuperDicts with SuperLists?
The structure of JSON, YAML or HTML data is generally a deeply nested combination of dictionaries and lists. Using superdictionaries alone would not be sufficient, since lists within the data contained in a list would still contain regular (unconverted) dictionaries; this would require you to switch back to the standard dictionary access method.
By combining superdictionaries and superlists,
it is possible to ensure that all nested dictionaries within lists will also be converted to SuperDicts, allowing for a consistent dot notation throughout the entire data structure.
> π‘ **Deep conversion** SuperLists objects, combined with SuperDicts make sure that the most complex
> datastructures (from json or YAML) can be recursively converted into
> well-behaved Python objects.
### SuperCollection
**SuperCollection** is an abstract class containing SuperList and SuperDict.
It means that SuperList and SuperDict objects are **instances** of
SuperCollection, but they do not inherit from it.
```python
obj1 = SuperList([1, 3, 5])
assert isinstance(obj1, SuperCollection)
obj2 = SuperDict({'a': 5, 'b':7})
assert isinstance(obj2, SuperCollection)
```
### Factory function
You can use the `super_collect()` function to create a SuperCollection
(SuperList or SuperDict) from an object.
**It is particularly useful for converting Python data structures such as
JSON files.**
```python
import json
with open(FILENAME, 'r', encoding='utf-8') as f:
data = json.load(f)
content = super_collect(data)
```
`super_collect()` is designed to work on any list or dict, but it will also attempt
to process other types:
- all **sequences** (see [definition](https://docs.python.org/3/glossary.html#term-sequence)),
in other words objects whose class is registered as
instance of `collections.abs.Sequence`
will be converted into lists. This applies to `tuple`,
`range`, `collections.UserList`, etc.
- Special types: `set`, `deque` as well as `ndarray` (Numpy or compatible)
and `Series` (Pandas and others; your mileage may vary).
- Otherwise, will try to generate a SuperDict.
This function is also available as a static method of the `SuperCollection` class:
```python
content = SuperCollection.collect(data)
```
## Install
### From the repository
```sh
pip install super-collections
```
## Usage
```python
from super_collections import SuperDict, SuperList
d = SuperDict({'foo':5, 'bar': 'hello'})
l = SuperList([5, 7, 'foo', {'foo': 5}])
```
You can cast any dictionary and list into its "Super" equivalent when you want, and you are off to the races.
**The casting is recursive** i.e. in the case above, you can assert:
```python
l[-1].foo == 5
```
All methods of dict and list are available.
Those objects are self documented. `d.properties()` is a generator
that lists all keys that are available as attributes.
The `__dir__()` method (accessible with `dir()`) is properly updated with
those additional properties.
```python
list(d.properties())
> ['foo', 'bar']
dir(d)
> ['__class__', ..., 'bar', 'clear', 'copy', 'foo', 'fromkeys', 'get', 'items', 'keys', 'pop', 'popitem', 'properties', 'setdefault', 'to_hjson', 'to_json', 'update', 'values']
```
This means the **auto-complete feature** might be available
for the attributes of a SuperDict within a code editor (if the dictionary was
statically declared in the code); or in an advanced REPL
(such as [bpython](https://bpython-interpreter.org/)).
The methods `dict.update(other_dict)` and `list.extend(other_list)`
automatically cast the contents into SuperDict and SuperList as needed.
## Export to JSON or Hjson
You can export a SuperDict or SuperList to JSON or [Hjson](https://hjson.github.io/), for debug purposes.
It is not guaranteed that it will preserve all the meaningful information you want, but all basic types
will be preserved.
> Python DateTimes are converted into ISO Dates.
```python
print (d.to_json())
print (d.to_hjson())
```
The module also exports a `json_encode()` function, which will attempt to serialize
any object in Python to JSON (in an opinionated way).
## Export to YAML
If you wish to use PyYAML and guarantee the SuperDict and SuperList behave exactly as dict and list,
use the `yaml_support()` function.
This works with both `dump()` and `safedump()`
```python
from super_collections import SuperDict, SuperList, yaml_support
yaml_support()
d = SuperDict({"x": 1})
l = SuperList(["a", "b"])
dumped_dict = yaml.dump(d)
dumped_list = yaml.dump(l)
```
## Remarks
### Restrictions
1. In a SuperDict, **only keys that are valid Python identifiers
can be accessed as attributes**. If 'bar' is a key of object `foo`,
you can write `foo.bar`; but you can't
write ~~`foo.hello world`~~ because 'hello world' is not a
valid Python identifier;
you will have to access that specific value with the "dictionary" notation:
`foo['hello world']`.
2. Similarly, you can't use pre-existing methods of the
`dict` class: `keys`, `items`, `update`, etc. as properties; as well as the
`properties` method itself (wich is specific to SuperDict).
In that case again, use the dictionary notation to access
the value (`d['items']`, etc.). Those keys that
cannot be accessed as attributes are said to be **masked**.
If you are uncertain which are available, just use `SuperDict.properties()`.
method.
3. Updating a single element (`d['foo']` for a SuperDict and `l[5]`
for a SuperList) does not perfom any casting. That's to avoid crazy
recursive situations, while giving
you fine grain control on what you want to do
(just cast with `SuperDict()` and `SuperList()`).
### Does it work?
Yes. It is tested with pytest. See the `test` directory for examples.
### When are superdictionaries and superlists _not_ recommended?
SuperDicts (and SuperLists) classes are most useful when the program you are
writing is consuming loosely structured data (json, YAML, HTML)
you have every reason to believe they
are sufficiently well-formed: typically data exported from existing APIs
or Web sources.
> β οΈ **Caution** super-collections may not be the best
> tool when source data come from a source whose quality
> is unsufficiently guaranteed for your needs, or is untrusted.
If you want to impose strongly formatted data structures in your code, one solution is
to create [dataclasses](https://docs.python.org/3/library/dataclasses.html); especially those of [Pydantic](https://docs.pydantic.dev/latest/concepts/dataclasses/), which make implicit and explicit
controls on the integrity of the source data.
## Shelves and SuperShelves
### Metaphor of a shelf
Imagine a real shelf in your home, with compartments. You place items in each compartment: books, boxes, a photo frame, etc.
Some compartments have labels ("History Books", "Photos"), others donβt. To refer to a compartment, you can use its
position ("the third from the left"), or refer to it by its label.
If it was a really big shelf, as in a library, and you were given the label of a compartment and no other information,
finding it would take some time, because you would have to scan the whole shelf each time.
To speed things up, you would have to keep a cardfile with cards for each label, indicating the position of the corresponding compartment in the shelf.
### Definition of a shelf (data structure)
A Shelf data structure works the same way. Conceptually, a shelf is very intuitive: you can think about it as a list where
you can _also_ use labels for fast retrieval.
Itβs a line of compartments, each holding one object.
If implements **dual addressing**: You can access an item by a **key** that can be either its index (2) or its label,
providing it exists ("Receipts").
In other words, a shelf works _both_ as a list and a dictionary, and has the attributes of both classes.
You can write:
```python
s = Shelf()
s.append("x")
s.append("y", label="thing")
```
or:
```python
s = Shelf()
s[0] = "X"
s["thing"] = "Y"
```
And then:
```python
assert s[0] == "X"
assert s[1] == "Y"
assert s["thing"] == "Y"
assert list(s) == ["X", "Y"]
assert list(s.values()) == ["X", "Y"]
assert list(s.keys()) == [0, "thing"]
```
Internally, a shelf is a **list of cells**, with a value and an optional **label**. It is complemented by a **cardfile**
that converts the labels of the cells into their index. However, you don't have to worry about it:
when you add an item to the shelf, change it or delete it, both the list and the carfile are kept up-to-date.
You can consult the internals:
```python
print(list(s.cells()))
```
It would return a list of mutable objects:
```
[Cell(value='X', label=None), Cell(value='Y', label='thing')]
```
> β οΈ **Caution** Do _not_ change the cells' labels directly,
> since the cardfile would not be updated.
### Iteration through a shelf
**When you iterate through a shelf, you return the values as in a list.**
> **β οΈ IMPORTANT NOTE:** This is distinct from a Python dictionary, where iteration returns the keys.
* If you want the keys use the `.keys()` method. For items that do not have a label, it returns the index.
* If you want the key, value pairs, use the `.items()` method.
* The `.values()` method is provided for compatibility with a dictionary.
### SuperShelf
A SuperShelf is essentially a shelf that behaves as a super-collection:
- it converts recursively all the content into shelves
- it exports the labels as attributes (when they are valid Python identifiers and not masked)
In other words it behaves as SuperList and a SuperDict at the same time.
## Related data structures and ideas
These projects contain ideas that inspired or predated super-collections.
### Standard Python
* `collections.namedtuple`: tuples with dot notation ([standard python class](https://docs.python.org/3/library/collections.html#collections.namedtuple))
* `types.SimpleNamespace`: objects with arbitrary attributes ([standard python class](https://docs.python.org/3/library/types.html#types.SimpleNamespace))
* All Python classes have a __dict__ attribute, used at the foundation to implement the dot notation in the language, with the relative standard methods (`__setattr__()`, etc.) and functions (`setattr()`, etc.).
* In modern Python, the `dict` class has ordered keys (by insertion order) and is subclassable.
### Dot notation on dictionaries
* [addict](https://github.com/mewwts/addict) (Github)
* [DotMap](https://github.com/drgrib/dotmap): subclasses and MutableMapping and OrderedDict (Github)
* [SuperDict](https://github.com/itdxer/superdict): subclasses `dict` (Github)
* [dotty_dict](https://github.com/pawelzny/dotty_dict): wrapper (Github)
### Using superlists to complement superdictionaries
* Packages that write to and read from files, such as [shelve](https://docs.python.org/3/library/shelve.html) (standard), json, YAML, [Beautifulsoup](https://code.launchpad.net/beautifulsoup/), etc. heavily rely
on a **combination of dictionaries and lists**. BeautifulSoup in particular supports dot notation.
* In general, **the construction of any syntactic or semantic tree requires both dictionaries and lists**.
fralau-super-collections-550a1cf/cleanup.sh 0000775 0000000 0000000 00000000743 15066622662 0021047 0 ustar 00root root 0000000 0000000 #!/bin/bash
# This may be useful to get a clean slate (for creating a distribution)
# Find and delete all __pycache__ directories
find . -type d -name "__pycache__" -exec rm -rf {} +
# Find and delete all directories starting with __
find . -type d -name "__*" -exec rm -rf {} +
# Find and delete all .egg-info directories
find . -type d -name "*.egg-info" -exec rm -rf {} +
# Find and delete all .egg files
find . -type f -name "*.egg" -exec rm -f {} +
echo "Cleanup complete!" fralau-super-collections-550a1cf/pyproject.toml 0000664 0000000 0000000 00000001660 15066622662 0021774 0 ustar 00root root 0000000 0000000 [build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "super_collections"
version = "0.6.2"
description = "file: README.md"
authors = [{ name = "Laurent Franceschetti" }]
license = { text = "MIT" }
dependencies = ["hjson"]
readme = "README.md"
requires-python = ">=3.8"
classifiers = [
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
]
[project.urls]
Source = "https://github.com/fralau/super-collections"
[tool.setuptools.packages.find]
where = ["src"]
[project.optional-dependencies]
test = ["pytest>=7.0", "rich", "pyyaml"]
[tool.pytest.ini_options]
testpaths = ["test"]
fralau-super-collections-550a1cf/src/ 0000775 0000000 0000000 00000000000 15066622662 0017644 5 ustar 00root root 0000000 0000000 fralau-super-collections-550a1cf/src/super_collections/ 0000775 0000000 0000000 00000000000 15066622662 0023400 5 ustar 00root root 0000000 0000000 fralau-super-collections-550a1cf/src/super_collections/__init__.py 0000664 0000000 0000000 00000047460 15066622662 0025524 0 ustar 00root root 0000000 0000000 """
Super Collections
Their purpose is to turn complex input such as json or YAML files into
Python objects accessible with attributes, and self documented.
The general idea is that those structured files are combinations
of lists and dictionaries.
(C) Laurent Franceschetti 2024
"""
import datetime
import json
import inspect
from typing import Any, Union, Optional, Dict
from abc import ABC, abstractmethod
import hjson
from .shelf import Shelf, Cell # makes import easier
# -------------------------------------
# Low-level fixtures
# -------------------------------------
from collections import UserDict, deque
DICT_TYPES = dict, UserDict
class JSONEncoder(json.JSONEncoder):
"""
Custom encoder for JSON serialization.
Used for debugging purposes.
It's purpose is to be extremely reliable.
"""
def default(self, obj: Any) -> Any:
TIME_FORMATS = (datetime.datetime, datetime.date, datetime.time)
if isinstance(obj, TIME_FORMATS):
return obj.isoformat()
elif isinstance(obj, UserDict):
# for objects used by some packages
return dict(obj)
elif inspect.isfunction(obj):
return f"Function: %s %s" % (inspect.signature(obj),
obj.__doc__)
try:
return super().default(obj)
except TypeError:
pass
# It all else fails, output as best as I can
# If the object wants to speak for itself, Iβll let it. If it canβt, Iβll describe it.
try:
return str(obj)
except Exception:
pass
try:
return repr(obj)
except Exception:
# If all else fails, return the object's type
return f"