construct-2.8.16/0000755000175000017500000000000013164134612015651 5ustar arkadiuszarkadiusz00000000000000construct-2.8.16/LICENSE0000644000175000017500000000223513151124063016653 0ustar arkadiuszarkadiusz00000000000000Copyright (C) 2006-2016 Arkadiusz Bulski (arek.bulski@gmail.com) Tomer Filiba (tomerfiliba@gmail.com) Corbin Simpson (MostAwesomeDude@gmail.com) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. construct-2.8.16/PKG-INFO0000644000175000017500000001300713164134612016747 0ustar arkadiuszarkadiusz00000000000000Metadata-Version: 1.1 Name: construct Version: 2.8.16 Summary: A powerful declarative parser/builder for binary data Home-page: http://construct.readthedocs.org Author: Arkadiusz Bulski, Tomer Filiba, Corbin Simpson Author-email: arek.bulski@gmail.com, tomerfiliba@gmail.com, MostAwesomeDude@gmail.com License: MIT Description: Construct 2.8 ============= Construct is a powerful **declarative** parser (and builder) for binary data. Instead of writing *imperative code* to parse a piece of data, you declaratively define a *data structure* that describes your data. As this data structure is not code, you can use it in one direction to *parse* data into Pythonic objects, and in the other direction, to *build* objects into binary data. The library provides both simple, atomic constructs (such as integers of various sizes), as well as composite ones which allow you form hierarchical and sequential structures of increasing complexity. Construct features **bit and byte granularity**, easy debugging and testing, an **easy-to-extend subclass system**, and lots of primitive constructs to make your work easier: * Fields: raw bytes or numerical types * Structs and Sequences: combine simpler constructs into more complex ones * Bitwise: splitting bytes into bit-grained fields * Adapters: change how data is represented * Arrays/Ranges: duplicate constructs * Meta-constructs: use the context (history) to compute the size of data * If/Switch: branch the computational path based on the context * On-demand (lazy) parsing: read and parse only what you require * Pointers: jump from here to there in the data stream Example --------- A ``Struct`` is a collection of ordered, named fields:: >>> format = Struct( ... "signature" / Const(b"BMP"), ... "width" / Int8ub, ... "height" / Int8ub, ... "pixels" / Array(this.width * this.height, Byte), ... ) >>> format.build(dict(width=3,height=2,pixels=[7,8,9,11,12,13])) b'BMP\x03\x02\x07\x08\t\x0b\x0c\r' >>> format.parse(b'BMP\x03\x02\x07\x08\t\x0b\x0c\r') Container(signature=b'BMP')(width=3)(height=2)(pixels=[7, 8, 9, 11, 12, 13]) A ``Sequence`` is a collection of ordered fields, and differs from a ``Range`` in that latter is homogenous:: >>> format = PascalString(Byte, encoding="utf8") >> GreedyRange(Byte) >>> format.build([u"lalaland", [255,1,2]]) b'\nlalaland\xff\x01\x02' >>> format.parse(b"\x004361789432197") ['', [52, 51, 54, 49, 55, 56, 57, 52, 51, 50, 49, 57, 55]] See more examples of `file formats `_ and `network protocols `_ in the repository. Sticky -------- Version 2.5.5 is legacy. If you are maintaining a project that depended on this library for a long time, you should probably use this version. This branch is not actively maintained, not even bugfixes. Version 2.8 was released on September, 2016. There are significant API and implementation changes. Fields are now name-less and operators / >> [] are used to create Structs Sequences and Ranges. Most classes changed interface and behavior. You should read entire documentation again. Development and support ------------------------- Please use the `github issues `_ to ask general questions, make feature requests, report issues and bugs, and to send in patches. Good quality extensions to test suite are highly welcomed. There is also a `mailing list `_ that was used for years but github issues should be preffered. Main documentation is at `construct.readthedocs.org `_, where you can find all kinds of examples. Source is on `github `_. Releases are available on `pypi `_. `Construct3 `_ is a different project. It is a rewrite from scratch and belongs to another (previous) developer, it diverged from this project years ago. As far as I can tell, it was never released and abandoned. Requirements -------------- Construct should run on any Python 2.7 3.3 3.4 3.5 3.6 and pypy pypy3 implementation. Best should be 3.6 because it supports ordered keyword arguments which comes handy when declaring Struct members or crafting Containers. Keywords: construct,declarative,data structure,binary,parser,builder,pack,unpack Platform: POSIX Platform: Windows Classifier: Development Status :: 5 - Production/Stable Classifier: License :: OSI Approved :: MIT License Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: Implementation :: PyPy Provides: construct construct-2.8.16/setup.py0000755000175000017500000000333213151124063017362 0ustar arkadiuszarkadiusz00000000000000#!/usr/bin/env python import os try: from setuptools import setup except ImportError: from distutils.core import setup HERE = os.path.dirname(__file__) exec(open(os.path.join(HERE, "construct", "version.py")).read()) setup( name = "construct", version = version_string, #@UndefinedVariable packages = [ 'construct', 'construct.lib', 'construct.examples', 'construct.examples.formats', 'construct.examples.formats.data', 'construct.examples.formats.executable', 'construct.examples.formats.filesystem', 'construct.examples.formats.graphics', 'construct.examples.protocols', ], license = "MIT", description = "A powerful declarative parser/builder for binary data", long_description = open(os.path.join(HERE, "README.rst")).read(), platforms = ["POSIX", "Windows"], url = "http://construct.readthedocs.org", author = "Arkadiusz Bulski, Tomer Filiba, Corbin Simpson", author_email = "arek.bulski@gmail.com, tomerfiliba@gmail.com, MostAwesomeDude@gmail.com", install_requires = [], requires = [], provides = ["construct"], keywords = "construct, declarative, data structure, binary, parser, builder, pack, unpack", classifiers = [ "Development Status :: 5 - Production/Stable", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: Implementation :: PyPy", ], ) construct-2.8.16/construct/0000755000175000017500000000000013164134612017675 5ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/core.py0000644000175000017500000037422713164134365021223 0ustar arkadiuszarkadiusz00000000000000# -*- coding: utf-8 -*- import struct as packer from struct import Struct as Packer from struct import error as PackerError from io import BytesIO, StringIO from binascii import hexlify, unhexlify import sys import collections from construct.lib import * from construct.expr import * #=============================================================================== # exceptions #=============================================================================== class ConstructError(Exception): pass class FieldError(ConstructError): pass class SizeofError(ConstructError): pass class AdaptationError(ConstructError): pass class RangeError(ConstructError): pass class SwitchError(ConstructError): pass class SelectError(ConstructError): pass class UnionError(ConstructError): pass class FocusedError(ConstructError): pass class TerminatedError(ConstructError): pass class OverwriteError(ConstructError): pass class PaddingError(ConstructError): pass class ConstError(ConstructError): pass class StringError(ConstructError): pass class ChecksumError(ConstructError): pass class ValidationError(ConstructError): pass class BitIntegerError(ConstructError): pass class MappingError(AdaptationError): pass class ExplicitError(ConstructError): pass #=============================================================================== # used internally #=============================================================================== def singleton(arg): return arg() def _read_stream(stream, length): if length < 0: raise ValueError("length must be >= 0", length) data = stream.read(length) if len(data) != length: raise FieldError("could not read enough bytes, expected %d, found %d" % (length, len(data))) return data def _write_stream(stream, length, data): if length < 0: raise ValueError("length must be >= 0", length) if len(data) != length: raise FieldError("could not write bytes, expected %d, found %d" % (length, len(data))) written = stream.write(data) if written is not None and written != length: raise FieldError("could not write bytes, written %d, should %d" % (written, length)) #=============================================================================== # abstract constructs #=============================================================================== class Construct(object): """ The mother of all constructs. This object is generally not directly instantiated, and it does not directly implement parsing and building, so it is largely only of interest to subclass implementors. There are also other abstract classes sitting on top of this one. The external user API: * ``parse()`` * ``parse_stream()`` * ``build()`` * ``build_stream()`` * ``sizeof()`` Subclass authors should not override the external methods. Instead, another API is available: * ``_parse()`` * ``_build()`` * ``_sizeof()`` And stateful copying: * ``__getstate__()`` * ``__setstate__()`` Attributes and Inheritance ========================== All constructs have a name and flags. The name is used for naming struct members and context dictionaries. Note that the name can be a string, or None by default. A single underscore "_" is a reserved name, used as up-level in nested containers. The name should be descriptive, short, and valid as a Python identifier, although these rules are not enforced. The flags specify additional behavioral information about this construct. Flags are used by enclosing constructs to determine a proper course of action. Flags are often inherited from inner subconstructs but that depends on each class behavior. """ __slots__ = ["name", "flagbuildnone", "flagembedded"] def __init__(self): self.name = None self.flagbuildnone = False self.flagembedded = False def __repr__(self): return "<%s: %s%s%s>" % (self.__class__.__name__, self.name, " +nonbuild" if self.flagbuildnone else "", " +embedded" if self.flagembedded else "") def __getstate__(self): """Obtain a dictionary representing this construct's state.""" attrs = {} if hasattr(self, "__dict__"): attrs.update(self.__dict__) slots = [] c = self.__class__ while c is not None: if hasattr(c, "__slots__"): slots.extend(c.__slots__) c = c.__base__ for name in slots: if hasattr(self, name): attrs[name] = getattr(self, name) return attrs def __setstate__(self, attrs): """Set this construct's state to a given state.""" for name, value in attrs.items(): setattr(self, name, value) def __copy__(self): """Returns a copy of this construct.""" self2 = object.__new__(self.__class__) self2.__setstate__(self, self.__getstate__()) return self2 def parse(self, data, context=None, **kw): """ Parse an in-memory buffer (often bytes object). Strings, buffers, memoryviews, and other complete buffers can be parsed with this method. """ return self.parse_stream(BytesIO(data), context, **kw) def parse_stream(self, stream, context=None, **kw): """ Parse a stream. Files, pipes, sockets, and other streaming sources of data are handled by this method. """ context2 = Container() if context is not None: context2.update(context) context2.update(kw) return self._parse(stream, context2, "(parsing)") def _parse(self, stream, context, path): """ Override in your subclass. :returns: some value, usually based on bytes read from the stream but sometimes it is computed from nothing or from context """ raise NotImplementedError def build(self, obj, context=None, **kw): """ Build an object in memory. :returns: bytes """ stream = BytesIO() self.build_stream(obj, stream, context, **kw) return stream.getvalue() def build_stream(self, obj, stream, context=None, **kw): """ Build an object directly into a stream. :returns: None """ context2 = Container() if context is not None: context2.update(context) context2.update(kw) self._build(obj, stream, context2, "(building)") def _build(self, obj, stream, context, path): """ Override in your subclass. :returns: None or a new value to put into the context, few fields use this internal functionality """ raise NotImplementedError def sizeof(self, context=None, **kw): """ Calculate the size of this object, optionally using a context. Some constructs have no fixed size and can only know their size for a given hunk of data. These constructs will raise an error if they are not passed a context. :param context: a container :returns: an integer for a fixed size field :raises SizeofError: the size could not be determined, ever or just with actual context """ if context is None: context = Container() context.update(kw) return self._sizeof(context, "sizeof") def _sizeof(self, context, path): """ Override in your subclass. :returns: an integer for a fixed size field :raises SizeofError: the size could not be determined, ever or just with actual context """ raise SizeofError def __getitem__(self, count): """ Used for making Ranges like Byte[5] or Byte[:]. """ if isinstance(count, slice): if count.step is not None: raise ValueError("slice must not contain a step: %r" % count) min = 0 if count.start is None else count.start max = sys.maxsize if count.stop is None else count.stop return Range(min, max, self) elif isinstance(count, int) or callable(count): return Range(count, count, self) else: raise TypeError("expected an int, a context lambda, or a slice thereof, but found %r" % count) def __rshift__(self, other): """ Used for making Sequences like (Byte >> Short). """ lhs = self.subcons if isinstance(self, Sequence) else [self] rhs = other.subcons if isinstance(other, Sequence) else [other] return Sequence(*(lhs + rhs)) def __rtruediv__(self, name): """ Used for renaming subcons, usually part of a Struct, like "index" / Byte. """ if name is not None: if not isinstance(name, stringtypes): raise TypeError("name must be b-string or u-string or None", name) return Renamed(name, self) __rdiv__ = __rtruediv__ def __add__(self, other): """ Used for making Structs, like ("index"/Byte + "prefix"/Byte). """ lhs = self.subcons if isinstance(self, Struct) else [self] rhs = other.subcons if isinstance(other, Struct) else [other] return Struct(*(lhs + rhs)) class Subconstruct(Construct): """ Abstract subconstruct (wraps an inner construct, inheriting its name and flags). Parsing and building is by default deferred to subcon, same as sizeof. :param subcon: the construct to wrap """ __slots__ = ["subcon"] def __init__(self, subcon): if not isinstance(subcon, Construct): raise TypeError("subcon should be a Construct field") super(Subconstruct, self).__init__() self.name = subcon.name self.subcon = subcon self.flagbuildnone = subcon.flagbuildnone self.flagembedded = subcon.flagembedded def _parse(self, stream, context, path): return self.subcon._parse(stream, context, path) def _build(self, obj, stream, context, path): return self.subcon._build(obj, stream, context, path) def _sizeof(self, context, path): return self.subcon._sizeof(context, path) class Adapter(Subconstruct): """ Abstract adapter parent class. Needs to implement ``_decode()`` for parsing and ``_encode()`` for building. :param subcon: the construct to wrap """ def _parse(self, stream, context, path): return self._decode(self.subcon._parse(stream, context, path), context) def _build(self, obj, stream, context, path): return self.subcon._build(self._encode(obj, context), stream, context, path) def _decode(self, obj, context): raise NotImplementedError def _encode(self, obj, context): raise NotImplementedError class SymmetricAdapter(Adapter): """ Abstract adapter parent class. Needs to implement ``_decode()`` only, for both parsing and building. :param subcon: the construct to wrap """ def _encode(self, obj, context): return self._decode(obj, context) class Validator(SymmetricAdapter): """ Abstract class that validates a condition on the encoded/decoded object. Needs to implement ``_validate()`` that returns a bool (or a truthy value) :param subcon: the subcon to validate """ def _decode(self, obj, context): if not self._validate(obj, context): raise ValidationError("object failed validation", obj) return obj def _validate(self, obj, context): raise NotImplementedError class Tunnel(Subconstruct): """ Abstract class that reads entire stream when parsing, and writes all data when building, but serves as an adapter as well. Needs to implement ``_decode()`` for parsing and ``_encode()`` for building. """ def _parse(self, stream, context, path): data = stream.read() # reads entire stream data = self._decode(data, context) return self.subcon.parse(data, context) def _build(self, obj, stream, context, path): data = self.subcon.build(obj, context) data = self._encode(data, context) _write_stream(stream, len(data), data) def _sizeof(self, context, path): raise SizeofError def _decode(self, data, context): raise NotImplementedError def _encode(self, data, context): raise NotImplementedError #=============================================================================== # bytes and bits #=============================================================================== class Bytes(Construct): """ Field consisting of a specified number of bytes. Builds from a bytes, or an integer (although deprecated and BytesInteger should be used instead). .. seealso:: Analog :func:`~construct.core.BytesInteger` that parses and builds from integers, as opposed to bytes. :param length: an integer or a context lambda that returns such an integer Example:: >>> d = Bytes(4) >>> d.parse(b'beef') b'beef' >>> d.build(b'beef') b'beef' >>> d.build(0) b'\x00\x00\x00\x00' >>> d.sizeof() 4 """ __slots__ = ["length"] def __init__(self, length): super(Bytes, self).__init__() self.length = length def _parse(self, stream, context, path): length = self.length(context) if callable(self.length) else self.length return _read_stream(stream, length) def _build(self, obj, stream, context, path): length = self.length(context) if callable(self.length) else self.length data = integer2bytes(obj, length) if isinstance(obj, int) else obj _write_stream(stream, length, data) return data def _sizeof(self, context, path): try: return self.length(context) if callable(self.length) else self.length except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") @singleton class GreedyBytes(Construct): """ Field that parses the stream to the end and builds into the stream as is. .. seealso:: Analog :func:`~construct.core.GreedyString` that parses and builds from strings using an encoding. Example:: >>> GreedyBytes.parse(b"asislight") b'asislight' >>> GreedyBytes.build(b"asislight") b'asislight' """ def _parse(self, stream, context, path): return stream.read() def _build(self, obj, stream, context, path): stream.write(obj) def Bitwise(subcon): """ Converts the stream from bytes to bits, and passes the bitstream to underlying subcon. .. seealso:: Analog :func:`~construct.core.Bytewise` that transforms subset of bits back to bytes. .. warning:: Do not use pointers inside this or other restreamed contexts. :param subcon: any field that works with bits like BitStruct or Bit Nibble Octet BitsInteger Example:: >>> d = Bitwise(Octet) >>> d.parse(b"\xff") 255 >>> d.build(1) b'\x01' >>> d.sizeof() 1 """ return Restreamed(subcon, bits2bytes, 8, bytes2bits, 1, lambda n: n//8) def Bytewise(subcon): """ Converts the stream from bits back to bytes. Must be used within Bitwise. :param subcon: any field that works with bytes Example:: >>> d = Bitwise(Bytewise(Byte)) >>> d.parse(b"\xff") 255 >>> d.build(255) b'\xff' >>> d.sizeof() 1 """ return Restreamed(subcon, bytes2bits, 1, bits2bytes, 8, lambda n: n*8) #=============================================================================== # integers and floats #=============================================================================== class FormatField(Bytes): """ Field that uses ``struct`` module to pack and unpack data. This is used to implement basic Int* and Float* fields alongside BytesInteger. See ``struct`` documentation for instructions on crafting format strings. :param endianity: endianness character like < > = :param format: format character like f d B H L Q b h l q Example:: >>> d = FormatField(">", "H") >>> d.parse(b"\x01\x00") 256 >>> d.build(256) b"\x01\x00" >>> d.sizeof() 2 """ __slots__ = ["fmtstr"] def __init__(self, endianity, format): if endianity not in (">", "<", "="): raise ValueError("endianity must be like: = < >", endianity) if len(format) != 1: raise ValueError("format must be like: f d B H L Q b h l q") super(FormatField, self).__init__(packer.calcsize(endianity + format)) self.fmtstr = endianity + format def _parse(self, stream, context, path): try: return packer.unpack(self.fmtstr, _read_stream(stream, self.sizeof()))[0] except Exception: raise FieldError("packer %r error during parsing" % self.fmtstr) def _build(self, obj, stream, context, path): try: _write_stream(stream, self.sizeof(), packer.pack(self.fmtstr, obj)) except Exception: raise FieldError("packer %r error during building, given value %s" % (self.fmtstr, obj)) class BytesInteger(Construct): """ Field that builds from integers as opposed to bytes. Similar to Int* fields but can have arbitrary size. .. seealso:: Analog :func:`~construct.core.BitsInteger` that operates on bits. :param length: number of bytes in the field, and integer or a context function that returns such an integer :param signed: whether the value is signed (two's complement), default is False (unsigned) :param swapped: whether to swap byte order (little endian), default is False (big endian) Example:: >>> d = BytesInteger(4) or Int32ub >>> d.parse(b"abcd") 1633837924 >>> d.build(1) b'\x00\x00\x00\x01' >>> d.sizeof() 4 """ __slots__ = ["length", "signed", "swapped"] def __init__(self, length, signed=False, swapped=False): super(BytesInteger, self).__init__() self.length = length self.signed = signed self.swapped = swapped def _parse(self, stream, context, path): length = self.length(context) if callable(self.length) else self.length data = _read_stream(stream, length) if self.swapped: data = swapbytes(data, 1) return bytes2integer(data, self.signed) def _build(self, obj, stream, context, path): if obj < 0 and not self.signed: raise BitIntegerError("object is negative, but field is not signed", obj) length = self.length(context) if callable(self.length) else self.length data = integer2bytes(obj, length) if self.swapped: data = swapbytes(data, 1) _write_stream(stream, len(data), data) def _sizeof(self, context, path): try: return self.length(context) if callable(self.length) else self.length except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") class BitsInteger(Construct): """ Field that builds from integers as opposed to bytes. Similar to Bit/Nibble/Octet fields but can have arbitrary sizes. Must be enclosed in Bitwise. :param length: number of bits in the field, an integer or a context function that returns such an integer :param signed: whether the value is signed (two's complement), default is False (unsigned) :param swapped: whether to swap byte order (little endian), default is False (big endian) Example:: >>> d = Bitwise(BitsInteger(8)) >>> d.parse(b"\x10") 16 >>> d.build(255) b'\xff' >>> d.sizeof() 1 """ __slots__ = ["length", "signed", "swapped"] def __init__(self, length, signed=False, swapped=False): super(BitsInteger, self).__init__() self.length = length self.signed = signed self.swapped = swapped def _parse(self, stream, context, path): length = self.length(context) if callable(self.length) else self.length data = _read_stream(stream, length) if self.swapped: data = swapbytes(data, 8) return bits2integer(data, self.signed) def _build(self, obj, stream, context, path): if obj < 0 and not self.signed: raise BitIntegerError("object is negative, but field is not signed", obj) length = self.length(context) if callable(self.length) else self.length data = integer2bits(obj, length) if self.swapped: data = swapbytes(data, 8) _write_stream(stream, len(data), data) def _sizeof(self, context, path): try: return self.length(context) if callable(self.length) else self.length except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") @singleton def Bit(): """A 1-bit integer, must be enclosed in a Bitwise (eg. BitStruct)""" return BitsInteger(1) @singleton def Nibble(): """A 4-bit integer, must be enclosed in a Bitwise (eg. BitStruct)""" return BitsInteger(4) @singleton def Octet(): """A 8-bit integer, must be enclosed in a Bitwise (eg. BitStruct)""" return BitsInteger(8) @singleton def Int8ub(): """Unsigned, big endian 8-bit integer""" return FormatField(">", "B") @singleton def Int16ub(): """Unsigned, big endian 16-bit integer""" return FormatField(">", "H") @singleton def Int32ub(): """Unsigned, big endian 32-bit integer""" return FormatField(">", "L") @singleton def Int64ub(): """Unsigned, big endian 64-bit integer""" return FormatField(">", "Q") @singleton def Int8sb(): """Signed, big endian 8-bit integer""" return FormatField(">", "b") @singleton def Int16sb(): """Signed, big endian 16-bit integer""" return FormatField(">", "h") @singleton def Int32sb(): """Signed, big endian 32-bit integer""" return FormatField(">", "l") @singleton def Int64sb(): """Signed, big endian 64-bit integer""" return FormatField(">", "q") @singleton def Int8ul(): """Unsigned, little endian 8-bit integer""" return FormatField("<", "B") @singleton def Int16ul(): """Unsigned, little endian 16-bit integer""" return FormatField("<", "H") @singleton def Int32ul(): """Unsigned, little endian 32-bit integer""" return FormatField("<", "L") @singleton def Int64ul(): """Unsigned, little endian 64-bit integer""" return FormatField("<", "Q") @singleton def Int8sl(): """Signed, little endian 8-bit integer""" return FormatField("<", "b") @singleton def Int16sl(): """Signed, little endian 16-bit integer""" return FormatField("<", "h") @singleton def Int32sl(): """Signed, little endian 32-bit integer""" return FormatField("<", "l") @singleton def Int64sl(): """Signed, little endian 64-bit integer""" return FormatField("<", "q") @singleton def Int8un(): """Unsigned, native endianity 8-bit integer""" return FormatField("=", "B") @singleton def Int16un(): """Unsigned, native endianity 16-bit integer""" return FormatField("=", "H") @singleton def Int32un(): """Unsigned, native endianity 32-bit integer""" return FormatField("=", "L") @singleton def Int64un(): """Unsigned, native endianity 64-bit integer""" return FormatField("=", "Q") @singleton def Int8sn(): """Signed, native endianity 8-bit integer""" return FormatField("=", "b") @singleton def Int16sn(): """Signed, native endianity 16-bit integer""" return FormatField("=", "h") @singleton def Int32sn(): """Signed, native endianity 32-bit integer""" return FormatField("=", "l") @singleton def Int64sn(): """Signed, native endianity 64-bit integer""" return FormatField("=", "q") Byte = Int8ub Short = Int16ub Int = Int32ub Long = Int64ub @singleton def Float32b(): """Big endian, 32-bit IEEE floating point number""" return FormatField(">", "f") @singleton def Float32l(): """Little endian, 32-bit IEEE floating point number""" return FormatField("<", "f") @singleton def Float32n(): """Native endianity, 32-bit IEEE floating point number""" return FormatField("=", "f") @singleton def Float64b(): """Big endian, 64-bit IEEE floating point number""" return FormatField(">", "d") @singleton def Float64l(): """Little endian, 64-bit IEEE floating point number""" return FormatField("<", "d") @singleton def Float64n(): """Native endianity, 64-bit IEEE floating point number""" return FormatField("=", "d") Single = Float32b Double = Float64b @singleton def Int24ub(): """A 3-byte big-endian unsigned integer, as used in ancient file formats.""" return BytesInteger(3) @singleton def Int24ul(): """A 3-byte little-endian unsigned integer, as used in ancient file formats.""" return BytesInteger(3, swapped=True) @singleton def Int24sb(): """A 3-byte big-endian signed integer, as used in ancient file formats.""" return BytesInteger(3, signed=True) @singleton def Int24sl(): """A 3-byte little-endian signed integer, as used in ancient file formats.""" return BytesInteger(3, signed=True, swapped=True) @singleton class VarInt(Construct): """ Varint encoded integer. Each 7 bits of the number are encoded in one byte of the stream, where leftmost (8th) bit is unset when byte is terminal. Can only encode non-negative numbers. Scheme defined at Google's site: https://developers.google.com/protocol-buffers/docs/encoding https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/ Example:: >>> VarInt.build(16) b'\x10' >>> VarInt.parse(_) 16 >>> VarInt.build(2**100) b'\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x04' >>> VarInt.parse(_) 1267650600228229401496703205376 or 2**100 """ def _parse(self, stream, context, path): acc = [] while True: b = byte2int(_read_stream(stream, 1)) acc.append(b & 0b01111111) if not b & 0b10000000: break num = 0 for b in reversed(acc): num = (num << 7) | b return num def _build(self, obj, stream, context, path): if obj < 0: raise ValueError("varint cannot build from negative number") while obj > 0b01111111: _write_stream(stream, 1, int2byte(0b10000000 | (obj & 0b01111111))) obj >>= 7 _write_stream(stream, 1, int2byte(obj)) #=============================================================================== # structures and sequences #=============================================================================== class Struct(Construct): """ Sequence of usually named constructs, similar to structs in C. The elements are parsed and built in the order they are defined. Some fields do not need to be named, since they are built without value anyway. See Const Padding Pass Terminated for examples of such fields. Size is the sum of all subcon sizes. Operator + can also be used to make Structs. .. seealso:: Can be nested easily, and embedded using :func:`~construct.core.Embedded` wrapper that merges members into parent's members. :param subcons: subcons that make up this structure Example:: >>> d = Struct("a"/Int8ul, "data"/Bytes(2), "data2"/Bytes(this.a)) >>> d.parse(b"\x01abc") Container(a=1)(data=b'ab')(data2=b'c') >>> d.build(_) b'\x01abc' >>> d.build(dict(a=5, data=b"??", data2=b"hello")) b'\x05??hello' >>> d = Struct(Const(b"MZ"), Padding(2), Pass, Terminated) >>> d.build({}) b'MZ\x00\x00' >>> d.parse(_) Container() >>> d.sizeof() 4 Alternative syntax (not recommended): >>> ("a"/Byte + "b"/Byte + "c"/Byte + "d"/Byte) Alternative syntax, note this works ONLY on python 3.6+: >>> Struct(a=Byte, b=Byte, c=Byte, d=Byte) """ __slots__ = ["subcons"] def __init__(self, *subcons, **kw): super(Struct, self).__init__() self.subcons = list(subcons) + list(k/v for k,v in kw.items()) def _parse(self, stream, context, path): obj = Container() context = Container(_ = context) for sc in self.subcons: try: subobj = sc._parse(stream, context, path) if sc.flagembedded: if subobj is not None: obj.update(subobj.items()) context.update(subobj.items()) else: if sc.name is not None: obj[sc.name] = subobj context[sc.name] = subobj except StopIteration: break return obj def _build(self, obj, stream, context, path): context = Container(_ = context) context.update(obj) for sc in self.subcons: try: if sc.flagembedded: subobj = obj elif sc.flagbuildnone: subobj = obj.get(sc.name, None) else: subobj = obj[sc.name] if sc.flagembedded: context.update(subobj) if sc.name is not None: context[sc.name] = subobj buildret = sc._build(subobj, stream, context, path) if buildret is not None: if sc.flagembedded: context.update(buildret) if sc.name is not None: context[sc.name] = buildret except StopIteration: break return context def _sizeof(self, context, path): try: def isStruct(sc): return isStruct(sc.subcon) if isinstance(sc, Renamed) else isinstance(sc, Struct) def nest(context, sc): if isStruct(sc) and not sc.flagembedded and sc.name in context: context2 = context[sc.name] context2["_"] = context return context2 else: return context return sum(sc._sizeof(nest(context, sc), path) for sc in self.subcons) except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") class Sequence(Struct): """ Sequence of unnamed constructs. The elements are parsed and built in the order they are defined. Size is the sum of all subcon sizes. Operator >> can also be used to make Sequences. .. seealso:: Can be nested easily, and embedded using :func:`~construct.core.Embedded` wrapper that merges entries into parent's entries. :param subcons: subcons that make up this sequence Example:: >>> d = (Byte >> Byte) >>> d.parse(b'\x01\x02') [1, 2] >>> d.build([1, 2]) b'\x01\x02' >>> d.sizeof() 2 >>> d = Sequence(Byte, CString(), Float32b) >>> d.build([255, b"hello", 123.0]) b'\xffhello\x00B\xf6\x00\x00' >>> d.parse(_) [255, b'hello', 123.0] Alternative syntax (not recommended): >>> (Byte >> "Byte >> "c"/Byte >> "d"/Byte) Alternative syntax, note this works ONLY on python 3.6+: >>> Sequence(a=Byte, b=Byte, c=Byte, d=Byte) """ def _parse(self, stream, context, path): obj = ListContainer() context = Container(_ = context) for i,sc in enumerate(self.subcons): try: subobj = sc._parse(stream, context, path) if sc.flagembedded: obj.extend(subobj) context[i] = subobj else: obj.append(subobj) if sc.name is not None: context[sc.name] = subobj context[i] = subobj except StopIteration: break return obj def _build(self, obj, stream, context, path): context = Container(_ = context) objiter = iter(obj) for i,sc in enumerate(self.subcons): try: if sc.flagembedded: subobj = objiter else: subobj = next(objiter) if sc.name is not None: context[sc.name] = subobj context[i] = subobj buildret = sc._build(subobj, stream, context, path) if buildret is not None: if sc.flagembedded: context.update(buildret) if sc.name is not None: context[sc.name] = buildret context[i] = buildret except StopIteration: break #=============================================================================== # arrays and repeaters #=============================================================================== class Range(Subconstruct): """ Homogenous array of elements. The array will iterate through between ``min`` to ``max`` times. If an exception occurs (EOF, validation error) then repeater exits cleanly. If less than ``min`` or more than ``max`` elements have been successfully processed, error is raised. Operator [] can be used to make instances. .. seealso:: Analog to :func:`~construct.core.GreedyRange` that parses until end of stream. :param min: the minimal count, an integer or a context lambda :param max: the maximal count, an integer or a context lambda :param subcon: the subcon to process individual elements :raises RangeError: when consumed or produced too little or too many elements Example:: >>> d = Range(3,5,Byte) or Byte[3:5] >>> d.parse(b'\x01\x02\x03\x04') [1,2,3,4] >>> d.build([1,2,3,4]) b'\x01\x02\x03\x04' >>> d.build([1,2]) construct.core.RangeError: expected from 3 to 5 elements, found 2 >>> d.build([1,2,3,4,5,6]) construct.core.RangeError: expected from 3 to 5 elements, found 6 Alternative syntax (recommended): >>> Byte[3:5], Byte[3:], Byte[:5], Byte[:] """ __slots__ = ["min", "max"] def __init__(self, min, max, subcon): super(Range, self).__init__(subcon) self.min = min self.max = max def _parse(self, stream, context, path): min = self.min(context) if callable(self.min) else self.min max = self.max(context) if callable(self.max) else self.max if not 0 <= min <= max <= sys.maxsize: raise RangeError("unsane min %s and max %s" % (min, max)) obj = ListContainer() context = Container(_ = context) try: while len(obj) < max: fallback = stream.tell() obj.append(self.subcon._parse(stream, context._, path)) context[len(obj)-1] = obj[-1] except StopIteration: pass except ExplicitError: raise except Exception: if len(obj) < min: raise RangeError("expected %d to %d, found %d" % (min, max, len(obj))) stream.seek(fallback) return obj def _build(self, obj, stream, context, path): min = self.min(context) if callable(self.min) else self.min max = self.max(context) if callable(self.max) else self.max if not 0 <= min <= max <= sys.maxsize: raise RangeError("unsane min %s and max %s" % (min, max)) if not isinstance(obj, collections.Sequence): raise RangeError("expected sequence type, found %s" % type(obj)) if not min <= len(obj) <= max: raise RangeError("expected from %d to %d elements, found %d" % (min, max, len(obj))) context = Container(_ = context) try: for i,subobj in enumerate(obj): context[i] = subobj self.subcon._build(subobj, stream, context._, path) except StopIteration: pass except ExplicitError: raise except Exception: if len(obj) < min: raise RangeError("expected %d to %d, found %d" % (min, max, len(obj))) else: raise def _sizeof(self, context, path): try: min = self.min(context) if callable(self.min) else self.min max = self.max(context) if callable(self.max) else self.max except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") if min == max: return min * self.subcon._sizeof(context, path) raise SizeofError("cannot calculate size, unless element count and size is fixed") def GreedyRange(subcon): """ Homogenous array of elements that parses until end of stream and builds from all elements. Operator [] can be used to make instances. :param subcon: the subcon to process individual elements Example:: >>> d = GreedyRange(Byte) or Byte[:] >>> d.build(range(8)) b'\x00\x01\x02\x03\x04\x05\x06\x07' >>> d.parse(_) [0, 1, 2, 3, 4, 5, 6, 7] Alternative syntax (recommended): >>> Byte[3:5], Byte[3:], Byte[:5], Byte[:] """ return Range(0, sys.maxsize, subcon) def Array(count, subcon): """ Homogenous array of elements. The array will iterate through exactly ``count`` elements. This is just a macro around `Range`. Operator [] can be used to make instances. :param count: an integer or a context function that returns such an integer :param subcon: the subcon to process individual elements Example:: >>> d = Array(5,5,Byte) or Byte[5] >>> d.build(range(5)) b'\x00\x01\x02\x03\x04' >>> d.parse(_) [0, 1, 2, 3, 4] Alternative syntax (recommended): >>> Byte[3:5], Byte[3:], Byte[:5], Byte[:] """ return Range(count, count, subcon) def PrefixedArray(lengthfield, subcon): """ Homogenous array prefixed by item count (as opposed to prefixed by byte count, see :func:`~construct.core.Prefixed`). :param lengthfield: field parsing and building an integer :param subcon: subcon to process individual elements Example:: >>> Prefixed(VarInt, GreedyRange(Int32ul)).parse(b"\x08abcdefgh") [1684234849, 1751606885] >>> PrefixedArray(VarInt, Int32ul).parse(b"\x02abcdefgh") [1684234849, 1751606885] """ return FocusedSeq(1, "count"/Rebuild(lengthfield, len_(this.items)), "items"/subcon[this.count], ) class RepeatUntil(Subconstruct): """ Homogenous array that repeats until the predicate indicates it to stop. Note that the last element (which caused the repeat to exit) is included in the return list. :param predicate: a predicate function that takes (obj, list, context) and returns True to break or False to continue (or a truthy value) :param subcon: the subcon used to parse and build each element Example:: >>> d = RepeatUntil(lambda x,lst,ctx: x>7, Byte) >>> d.build(range(20)) b'\x00\x01\x02\x03\x04\x05\x06\x07\x08' >>> d.parse(b"\x01\xff\x02") [1, 255] >>> d = RepeatUntil(lambda x,lst,ctx: lst[-2:]==[0,0], Byte) >>> d.parse(b"\x01\x00\x00\xff") [1, 0, 0] """ __slots__ = ["predicate"] def __init__(self, predicate, subcon): super(RepeatUntil, self).__init__(subcon) self.predicate = predicate def _parse(self, stream, context, path): obj = [] while True: subobj = self.subcon._parse(stream, context, path) obj.append(subobj) if self.predicate(subobj, obj, context): return ListContainer(obj) def _build(self, obj, stream, context, path): for i, subobj in enumerate(obj): self.subcon._build(subobj, stream, context, path) if self.predicate(subobj, obj[:i+1], context): break else: raise RangeError("expected any item to match predicate, when building") def _sizeof(self, context, path): raise SizeofError("cannot calculate size, amount depends on actual data") #=============================================================================== # subconstructs #=============================================================================== class Padded(Subconstruct): """ Appends additional null bytes to achieve a fixed length. Fails if actual data is longer than specified length. :raises PaddingError: when parsed or build data is longer than the length Example:: >>> d = Padded(4, Byte) >>> d.build(255) b'\xff\x00\x00\x00' >>> d.parse(_) 255 >>> d.sizeof() 4 >>> d = Padded(4, VarInt) >>> d.build(1) b'\x01\x00\x00\x00' >>> d.build(70000) b'\xf0\xa2\x04\x00' """ __slots__ = ["length", "pattern", "strict"] def __init__(self, length, subcon, pattern=b"\x00", strict=False): if not isinstance(pattern, bytes) or len(pattern) != 1: raise PaddingError("pattern expected to be bytes of length 1") super(Padded, self).__init__(subcon) self.length = length self.pattern = pattern self.strict = strict def _parse(self, stream, context, path): length = self.length(context) if callable(self.length) else self.length position1 = stream.tell() obj = self.subcon._parse(stream, context, path) position2 = stream.tell() padlen = length - (position2 - position1) if padlen < 0: raise PaddingError("subcon parsed more bytes than was allowed by length") pad = _read_stream(stream, padlen) if self.strict: if pad != self.pattern * padlen: raise PaddingError("expected %r times %r, found %r" % (self.pattern, padlen, pad)) return obj def _build(self, obj, stream, context, path): length = self.length(context) if callable(self.length) else self.length position1 = stream.tell() subobj = self.subcon._build(obj, stream, context, path) position2 = stream.tell() padlen = length - (position2 - position1) if padlen < 0: raise PaddingError("subcon built more bytes than was allowed by length") _write_stream(stream, padlen, self.pattern * padlen) return subobj def _sizeof(self, context, path): try: return self.length(context) if callable(self.length) else self.length except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") class Aligned(Subconstruct): """ Appends additional null bytes to achieve a length that is shortest multiple of a modulus. :param modulus: the modulus to final length, an integer or a context function returning such an integer :param subcon: the subcon to align :param pattern: optional, the padding pattern, a bytes character (default is \x00) Example:: >>> d = Aligned(4, Int16ub) >>> d.parse(b'\x00\x01\x00\x00') 1 >>> d.sizeof() 4 """ __slots__ = ["subcon", "modulus", "pattern"] def __init__(self, modulus, subcon, pattern=b"\x00"): if not isinstance(pattern, bytes) or len(pattern) != 1: raise PaddingError("pattern expected to be bytes character") super(Aligned, self).__init__(subcon) self.modulus = modulus self.pattern = pattern def _parse(self, stream, context, path): modulus = self.modulus(context) if callable(self.modulus) else self.modulus position1 = stream.tell() obj = self.subcon._parse(stream, context, path) position2 = stream.tell() pad = -(position2 - position1) % modulus _read_stream(stream, pad) return obj def _build(self, obj, stream, context, path): modulus = self.modulus(context) if callable(self.modulus) else self.modulus position1 = stream.tell() subobj = self.subcon._build(obj, stream, context, path) position2 = stream.tell() pad = -(position2 - position1) % modulus _write_stream(stream, pad, self.pattern * pad) return subobj def _sizeof(self, context, path): try: modulus = self.modulus(context) if callable(self.modulus) else self.modulus subconlen = self.subcon._sizeof(context, path) return subconlen + (-subconlen % modulus) except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") def AlignedStruct(modulus, *subcons, **kw): """ Makes a structure where each field is aligned to the same modulus (its a struct of aligned fields, not an aligned struct). .. seealso:: Uses :func:`~construct.core.Aligned` and `~construct.core.Struct`. :param modulus: passed to each member :param \*subcons: subcons that make up a Struct :param \*\*kw: named subcons Example:: >>> AlignedStruct(4, "a"/Int8ub, "b"/Int16ub).build(dict(a=1,b=5)) b'\x01\x00\x00\x00\x00\x05\x00\x00' >>> AlignedStruct(4, "a"/Int8ub, "b"/Int16ub).parse(_) Container(a=1)(b=5) >>> AlignedStruct(4, "a"/Int8ub, "b"/Int16ub).sizeof() 8 """ subcons = list(subcons) + list(k/v for k,v in kw.items()) return Struct(*[Aligned(modulus, sc) for sc in subcons]) def BitStruct(*subcons): """ Makes a structure inside a Bitwise. .. seealso:: Uses :func:`~construct.core.Bitwise` and :func:`~construct.core.Struct`. :param \*subcons: the subcons that make up this structure Example:: >>> d = BitStruct( ... "a" / Flag, ... "b" / Nibble, ... "c" / BitsInteger(10), ... "d" / Padding(1), ... ) >>> d.parse(b"\xbe\xef") Container(a=True)(b=7)(c=887)(d=None) >>> d.sizeof() 2 """ return Bitwise(Struct(*subcons)) def EmbeddedBitStruct(*subcons): """ Makes an embedded BitStruct. .. seealso:: Uses :func:`~construct.core.Bitwise` and :func:`~construct.core.Embedded` and :func:`~construct.core.Struct`. :param \*subcons: the subcons that make up this structure Example:: ??? """ return Bitwise(Embedded(Struct(*subcons))) #=============================================================================== # conditional #=============================================================================== class Union(Construct): """ Treats the same data as multiple constructs (similar to C union statement) so you can look at the data in multiple views. When parsing, all fields read the same data bytes, but stream remains at initial offset, unless parsefrom selects a subcon by index or name. When building, the first subcon that can find an entry in the dict (or builds from None, so it does not require an entry) is automatically selected. .. warning:: If you skip the `parsefrom` parameter then stream will be left back at the starting offset. Many users fail to use this properly. :param parsefrom: how to leave stream after parsing, can be integer index or string name selecting a subcon, None (leaves stream at initial offset, the default), a context lambda returning either of previously mentioned :param subcons: subconstructs (order and name sensitive) Example:: >>> d = Union(0, "raw"/Bytes(8), "ints"/Int32ub[2], "shorts"/Int16ub[4], "chars"/Byte[8]) >>> d.parse(b"12345678") Container(raw=b'12345678')(ints=[825373492, 892745528])(shorts=[12594, 13108, 13622, 14136])(chars=[49, 50, 51, 52, 53, 54, 55, 56]) >>> d.build(dict(chars=range(8))) b'\x00\x01\x02\x03\x04\x05\x06\x07' Alternative syntax, note this works ONLY on python 3.6+: >>> Union(0, raw=Bytes(8), ints=Int32ub[2], shorts=Int16ub[4], chars=Byte[8]) """ __slots__ = ["subcons","parsefrom"] def __init__(self, parsefrom, *subcons, **kw): if isinstance(parsefrom, Construct): raise UnionError("parsefrom should be either: None int str context-function") subcons = list(subcons) + list(k/v for k,v in kw.items()) super(Union, self).__init__() self.subcons = subcons self.parsefrom = parsefrom def _parse(self, stream, context, path): obj = Container() context = Container(_ = context) fallback = stream.tell() forwards = {} for i,sc in enumerate(self.subcons): if sc.flagembedded: subobj = list(sc._parse(stream, context, path).items()) obj.update(subobj) context.update(subobj) else: subobj = sc._parse(stream, context, path) if sc.name is not None: obj[sc.name] = subobj context[sc.name] = subobj forwards[i] = stream.tell() if sc.name is not None: forwards[sc.name] = stream.tell() stream.seek(fallback) parsefrom = self.parsefrom if callable(parsefrom): parsefrom = parsefrom(context) if isinstance(parsefrom, (int,str)): stream.seek(forwards[parsefrom]) return obj def _build(self, obj, stream, context, path): context = Container(_ = context) context.update(obj) for sc in self.subcons: if sc.flagbuildnone: subobj = obj.get(sc.name, None) buildret = sc._build(subobj, stream, context, path) if buildret is not None: if sc.flagembedded: context.update(buildret) if sc.name is not None: context[sc.name] = buildret return buildret elif sc.name in obj: context[sc.name] = obj[sc.name] buildret = sc._build(obj[sc.name], stream, context, path) if buildret is not None: if sc.flagembedded: context.update(buildret) if sc.name is not None: context[sc.name] = buildret return buildret else: raise UnionError("cannot build, none of subcons %s were found in the dictionary %s" % ([sc.name for sc in self.subcons], obj)) def _sizeof(self, context, path): parsefrom = self.parsefrom try: if callable(parsefrom): parsefrom = parsefrom(context) except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") if parsefrom is None: raise SizeofError("cannot calculate size") if isinstance(parsefrom, int): sc = self.subcons[parsefrom] return sc._sizeof(context, path) if isinstance(parsefrom, str): sc = {sc.name:sc for sc in self.subcons if sc.name is not None}[parsefrom] return sc._sizeof(context, path) raise UnionError("parsefrom should be either: None, an int, a str, or context function") class Select(Construct): """ Selects the first matching subconstruct. It will literally try each of the subconstructs, until one matches. :param subcons: the subcons to try (order sensitive) :param includename: indicates whether to include the name of the selected subcon in the return value of parsing, default is False Example:: >>> d = Select(Int32ub, CString(encoding="utf8")) >>> d.build(1) b'\x00\x00\x00\x01' >>> d.build(u"Афон") b'\xd0\x90\xd1\x84\xd0\xbe\xd0\xbd\x00' Alternative syntax, note this works ONLY on python 3.6+: >>> Select(num=Int32ub, text=CString(encoding="utf8")) """ __slots__ = ["subcons", "includename"] def __init__(self, *subcons, **kw): subcons = list(subcons) + list(k/v for k,v in kw.items() if k != "includename") super(Select, self).__init__() self.subcons = subcons self.flagbuildnone = all(sc.flagbuildnone for sc in subcons) self.flagembedded = all(sc.flagembedded for sc in subcons) self.includename = kw.pop("includename", False) def _parse(self, stream, context, path): for sc in self.subcons: fallback = stream.tell() try: obj = sc._parse(stream, context, path) except ExplicitError: raise except ConstructError: stream.seek(fallback) else: return (sc.name,obj) if self.includename else obj raise SelectError("no subconstruct matched") def _build(self, obj, stream, context, path): if self.includename: name, obj = obj for sc in self.subcons: if sc.name == name: return sc._build(obj, stream, context, path) else: for sc in self.subcons: try: data = sc.build(obj, context) except ExplicitError: raise except Exception: pass else: _write_stream(stream, len(data), data) return raise SelectError("no subconstruct matched", obj) def Optional(subcon): """ Makes an optional construct, that tries to parse the subcon. If parsing fails, returns None. If building fails, writes nothing. Size cannot be computed, because whether bytes are consumed or produced depends on actual data and context. :param subcon: the subcon to optionally parse or build Example:: >>> d = Optional(Int64ul) >>> d.parse(b"12345678") 4050765991979987505 >>> d.parse(b"") None >>> d.build(1) b'\x01\x00\x00\x00\x00\x00\x00\x00' >>> d.build(None) b'' """ return Select(subcon, Pass) class Switch(Construct): """ A conditional branch. Switch will choose the case to follow based on the return value of keyfunc. If no case is matched and no default value is given, SwitchError will be raised. .. warning:: You can use Embedded(Switch(...)) but not Switch(Embedded(...)). Same applies to If and IfThenElse macros. :param keyfunc: a context function that returns a key which will choose a case, or a constant :param cases: a dictionary mapping keys to subcons :param default: a default field to use when the key is not found in the cases. if not supplied, an exception will be raised when the key is not found. Pass can be used for do-nothing :param includekey: whether to include the key in the return value of parsing, default is False :raises SwitchError: when actual value is not in the dict nor a default is given Example:: >>> d = Switch(this.n, { 1:Int8ub, 2:Int16ub, 4:Int32ub }) >>> d.build(5, dict(n=1)) b'\x05' >>> d.build(5, dict(n=4)) b'\x00\x00\x00\x05' """ @singleton class NoDefault(Construct): def _parse(self, stream, context, path): raise SwitchError("no default case defined") def _build(self, obj, stream, context, path): raise SwitchError("no default case defined") def _sizeof(self, context, path): raise SwitchError("no default case defined") __slots__ = ["subcons", "keyfunc", "cases", "default", "includekey"] def __init__(self, keyfunc, cases, default=NoDefault, includekey=False): super(Switch, self).__init__() self.keyfunc = keyfunc self.cases = cases self.default = default self.includekey = includekey allcases = list(cases.values()) if default is not self.NoDefault: allcases.append(default) self.flagbuildnone = all(sc.flagbuildnone for sc in allcases) self.flagembedded = all(sc.flagembedded for sc in allcases) def _parse(self, stream, context, path): key = self.keyfunc(context) if callable(self.keyfunc) else self.keyfunc obj = self.cases.get(key, self.default)._parse(stream, context, path) return (key,obj) if self.includekey else obj def _build(self, obj, stream, context, path): if self.includekey: key,obj = obj else: key = self.keyfunc(context) if callable(self.keyfunc) else self.keyfunc case = self.cases.get(key, self.default) return case._build(obj, stream, context, path) def _sizeof(self, context, path): try: key = self.keyfunc(context) if callable(self.keyfunc) else self.keyfunc sc = self.cases.get(key, self.default) return sc._sizeof(context, path) except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") def IfThenElse(predicate, thensubcon, elsesubcon): """ An if-then-else conditional construct. One of the two subcons is used for parsing or building, depending whether the predicate returns a truthy or falsey value for given context. Constant truthy value can also be used. :param predicate: a context function that returns a bool (or truthy value) :param thensubcon: the subcon that will be used if the predicate indicates True :param elsesubcon: the subcon that will be used if the predicate indicates False Example:: >>> d = IfThenElse(this.x > 0, VarInt, Byte) >>> d.build(255, dict(x=1)) b'\xff\x01' >>> d.build(255, dict(x=0)) b'\xff' """ return Switch( lambda ctx: bool(predicate(ctx)) if callable(predicate) else bool(predicate), {True:thensubcon, False:elsesubcon}, ) def If(predicate, subcon): """ An if-then conditional construct. If the context predicate indicates True, the `subcon` will be used for parsing and building, otherwise parsing returns None and building is no-op. Note that the predicate has no access to parsed value, it computes only on context. :param predicate: a function taking context and returning a bool :param subcon: the subcon that will be used if the predicate returns True Example:: >>> d = If(this.x > 0, Byte) >>> d.build(255, dict(x=1)) b'\xff' >>> d.build(255, dict(x=0)) b'' """ return IfThenElse(predicate, subcon, Pass) #=============================================================================== # stream manipulation #=============================================================================== class Pointer(Subconstruct): """ Changes the stream position to a given offset, where the construction should take place, and restores the stream position when finished. Offset can also be nagative, indicating a position from EOF backwards. Size is defined as unspecified, instead of previous 0. .. seealso:: Analog to :func:`~construct.core.OnDemandPointer`, which also seeks to a given offset but returns a lazy lambda instead of constructing the value immediately. :param offset: an integer or a context function that returns a stream position, where the construction would take place :param subcon: the subcon to use at the offset Example:: >>> d = Pointer(8, Bytes(1)) >>> d.parse(b"abcdefghijkl") b'i' >>> d.build(b"Z") b'\x00\x00\x00\x00\x00\x00\x00\x00Z' """ __slots__ = ["offset"] def __init__(self, offset, subcon): super(Pointer, self).__init__(subcon) self.offset = offset def _parse(self, stream, context, path): offset = self.offset(context) if callable(self.offset) else self.offset fallback = stream.tell() stream.seek(offset, 2 if offset < 0 else 0) obj = self.subcon._parse(stream, context, path) stream.seek(fallback) return obj def _build(self, obj, stream, context, path): offset = self.offset(context) if callable(self.offset) else self.offset fallback = stream.tell() stream.seek(offset, 2 if offset < 0 else 0) buildret = self.subcon._build(obj, stream, context, path) stream.seek(fallback) return buildret def _sizeof(self, context, path): raise SizeofError class Peek(Subconstruct): """ Peeks at the stream. Parses without changing the stream position. If the end of the stream is reached when reading, returns None. Building is no-op. Size is defined as 0 because build does not put anything into the stream. .. seealso:: The :func:`~construct.core.Union` class. :param subcon: the subcon to peek at Example:: >>> d = Sequence(Peek(Int8ub), Peek(Int16ub)) >>> d.parse(b"\x01\x02") [1, 258] >>> d.sizeof() 0 """ def __init__(self, subcon): super(Peek, self).__init__(subcon) self.flagbuildnone = True def _parse(self, stream, context, path): fallback = stream.tell() try: return self.subcon._parse(stream, context, path) except ExplicitError: raise except FieldError: pass finally: stream.seek(fallback) def _build(self, obj, stream, context, path): pass def _sizeof(self, context, path): return 0 @singleton class Tell(Construct): """ Gets the stream position when parsing or building. Tell is useful for adjusting relative offsets to absolute positions, or to measure sizes of Constructs. To get an absolute pointer, use a Tell plus a relative offset. To get a size, place two Tells and measure their difference using a Compute. Size is defined as 0 because parsing and building does not consume or add into the stream. .. seealso:: Its better to use :func:`~construct.core.RawCopy` instead of manually extracting stream positions and bytes. Example:: >>> d = Struct("num"/VarInt, "offset"/Tell) >>> d.build(dict(num=88)) b'X' >>> d.parse(_) Container(num=88)(offset=1) """ def __init__(self): super(self.__class__, self).__init__() self.flagbuildnone = True def _parse(self, stream, context, path): return stream.tell() def _build(self, obj, stream, context, path): return stream.tell() def _sizeof(self, context, path): return 0 class Seek(Construct): """ Sets a new stream position when parsing or building. Seeks are useful when many other fields follow the jump. Pointer works when there is only one field to look at, but when there is more to be done, Seek may come useful. .. seealso:: Analog :func:`~construct.core.Pointer` wrapper that has same side effect but also processed a subcon. :param at: where to jump to, can be an integer or a context lambda returning such an integer :param whence: is the offset from beginning (0) or from current position (1) or from ending (2), can be an integer or a context lambda returning such an integer, default is 0 Example:: >>> d = (Seek(5) >> Byte) >>> d.parse(b"01234x") [5, 120] >>> d = (Bytes(10) >> Seek(5) >> Byte) >>> d.build([b"0123456789", None, 255]) b'01234\xff6789' """ __slots__ = ["at","whence"] def __init__(self, at, whence=0): super(Seek, self).__init__() self.at = at self.whence = whence self.flagbuildnone = True def _parse(self, stream, context, path): at = self.at(context) if callable(self.at) else self.at whence = self.whence(context) if callable(self.whence) else self.whence return stream.seek(at, whence) def _build(self, obj, stream, context, path): at = self.at(context) if callable(self.at) else self.at whence = self.whence(context) if callable(self.whence) else self.whence return stream.seek(at, whence) def _sizeof(self, context, path): raise SizeofError("Seek seeks the stream, sizeof is not meaningful") class Restreamed(Subconstruct): """ Transforms bytes between the underlying stream and the subcon. When the parsing or building is done, the wrapper stream is closed. If read buffer or write buffer is not empty, error is raised. .. seealso:: Both :func:`~construct.core.Bitwise` and :func:`~construct.core.Bytewise` are implemented using Restreamed. .. warning:: Remember that subcon must consume or produce an amount of bytes that is a multiple of encoding or decoding units. For example, in a Bitwise context you should process a multiple of 8 bits or the stream will fail after parsing/building. Also do NOT use pointers inside. :param subcon: the subcon which will operate on the buffer :param encoder: a function that takes bytes and returns bytes (used when building) :param encoderunit: ratio as integer, encoder takes that many bytes at once :param decoder: a function that takes bytes and returns bytes (used when parsing) :param decoderunit: ratio as integer, decoder takes that many bytes at once :param sizecomputer: a function that computes amount of bytes outputed by some bytes Example:: Bitwise <--> Restreamed(subcon, bits2bytes, 8, bytes2bits, 1, lambda n: n//8) Bytewise <--> Restreamed(subcon, bytes2bits, 1, bits2bytes, 8, lambda n: n*8) """ __slots__ = ["sizecomputer", "encoder", "encoderunit", "decoder", "decoderunit"] def __init__(self, subcon, encoder, encoderunit, decoder, decoderunit, sizecomputer): super(Restreamed, self).__init__(subcon) self.encoder = encoder self.encoderunit = encoderunit self.decoder = decoder self.decoderunit = decoderunit self.sizecomputer = sizecomputer def _parse(self, stream, context, path): stream2 = RestreamedBytesIO(stream, self.encoder, self.encoderunit, self.decoder, self.decoderunit) obj = self.subcon._parse(stream2, context, path) stream2.close() return obj def _build(self, obj, stream, context, path): stream2 = RestreamedBytesIO(stream, self.encoder, self.encoderunit, self.decoder, self.decoderunit) buildret = self.subcon._build(obj, stream2, context, path) stream2.close() return buildret def _sizeof(self, context, path): if self.sizecomputer is None: raise SizeofError("cannot calculate size") else: return self.sizecomputer(self.subcon._sizeof(context, path)) class Rebuffered(Subconstruct): """ Caches bytes from the underlying stream, so it becomes seekable and tellable. Also makes the stream blocking, in case it came from a socket or a pipe. Optionally, stream can forget bytes that went a certain amount of bytes beyond the current offset, allowing only a limited seeking capability while allowing to process an endless stream. .. warning:: Experimental implementation. May not be mature enough. :param subcon: the subcon which will operate on the buffered stream :param tailcutoff: optional, amount of bytes kept in buffer, by default buffers everything Example:: Rebuffered(..., tailcutoff=1024).parse_stream(endless_nonblocking_stream) """ __slots__ = ["stream2", "tailcutoff"] def __init__(self, subcon, tailcutoff=None): super(Rebuffered, self).__init__(subcon) self.stream2 = RebufferedBytesIO(None, tailcutoff=tailcutoff) def _parse(self, stream, context, path): self.stream2.substream = stream return self.subcon._parse(self.stream2, context, path) def _build(self, obj, stream, context, path): self.stream2.substream = stream return self.subcon._build(obj, self.stream2, context, path) #=============================================================================== # miscellaneous #=============================================================================== def Padding(length, pattern=b"\x00", strict=False): """ Padding field that adds bytes when building, discards bytes when parsing. :param length: length of the padding, an integer or a context function returning such an integer :param pattern: padding pattern as bytes character, default is \x00 :param strict: whether to verify during parsing that the stream contains the exact pattern, raises PaddingError if actual padding differs from the pattern, default is False :raises PaddingError: when strict is set and actual parsed pattern differs from specified Example:: >>> d = Padding(4, strict=True) >>> d.build(None) b'\x00\x00\x00\x00' >>> d.parse(b"****") construct.core.PaddingError: expected b'\x00\x00\x00\x00', found b'****' >>> d.sizeof() 4 """ return Padded(length, Pass, pattern=pattern, strict=strict) class Const(Subconstruct): """ Field enforcing a constant value. It is used for file signatures, to validate that the given pattern exists. When parsed, the value must match. Note that a variable length subcon may still provide positive verification. Const does not consume a precomputed amount of bytes (and hence does NOT require a fixed sized lenghtfield), but depends on the subcon to read the appropriate amount (eg. VarInt is acceptable). :param subcon: the subcon used to build value from, or a bytes value :param value: optional, the expected value :raises ConstError: when parsed data does not match specified value, or building from wrong value Example:: >>> d = Const(b"IHDR") >>> d.build(None) b'IHDR' >>> d.parse(b"JPEG") construct.core.ConstError: expected b'IHDR' but parsed b'JPEG' >>> d = Const(Int32ul, 16) >>> d.build(None) b'\x10\x00\x00\x00' """ __slots__ = ["value"] def __init__(self, subcon, value=None): if value is None: subcon, value = Bytes(len(subcon)), subcon if isinstance(subcon, str): subcon, value = Bytes(len(value)), value super(Const, self).__init__(subcon) self.value = value self.flagbuildnone = True def _parse(self, stream, context, path): obj = self.subcon._parse(stream, context, path) if obj != self.value: raise ConstError("expected %r but parsed %r" % (self.value, obj)) return obj def _build(self, obj, stream, context, path): if obj not in (None, self.value): raise ConstError("expected None or %r" % (self.value,)) return self.subcon._build(self.value, stream, context, path) def _sizeof(self, context, path): return self.subcon._sizeof(context, path) class Computed(Construct): """ Field computing a value. Underlying byte stream is unaffected. When parsing, the context function provides the value. Constant literal value can also be provided. Building does not require a value, the value gets computed. Size is defined as 0 because parsing and building does not consume or produce bytes. :param func: a context function or a constant value Example:: >>> d = Struct( ... "width" / Byte, ... "height" / Byte, ... "total" / Computed(this.width * this.height), ... ) >>> d.build(dict(width=4,height=5)) b'\x04\x05' >>> d.parse(b"12") Container(width=49)(height=50)(total=2450) >>> import os >>> d = Computed(lambda ctx: os.urandom(10)) >>> d.parse(b"") b'\x98\xc2\xec\x10\x07\xf5\x8e\x98\xc2\xec' """ __slots__ = ["func"] def __init__(self, func): super(Computed, self).__init__() self.func = func self.flagbuildnone = True def _parse(self, stream, context, path): return self.func(context) if callable(self.func) else self.func def _build(self, obj, stream, context, path): return self.func(context) if callable(self.func) else self.func def _sizeof(self, context, path): return 0 @singleton class Pass(Construct): """ No-op construct, useful as the default case for Switch. Returns None on parsing, puts nothing on building, size is 0 by definition. Building does not require a value, and any provided value gets discarded. Example:: >>> Pass.parse(b"") None >>> Pass.build(None) b'' >>> Pass.sizeof() 0 """ def __init__(self): super(self.__class__, self).__init__() self.flagbuildnone = True def _parse(self, stream, context, path): return None def _build(self, obj, stream, context, path): pass def _sizeof(self, context, path): return 0 @singleton class Terminated(Construct): """ Asserts the end of the stream has been reached at the point it was placed. You can use this to ensure no more unparsed data follows. This construct is only meaningful for parsing. Building does nothing. Example:: >>> Terminated.parse(b"") None >>> Terminated.parse(b"remaining") construct.core.TerminatedError: expected end of stream """ def __init__(self): super(self.__class__, self).__init__() self.flagbuildnone = True def _parse(self, stream, context, path): try: if stream.read(1): raise TerminatedError("expected end of stream") except IOError: # Restreamed.read(1) does not return empty string like BytesIO pass def _build(self, obj, stream, context, path): pass def _sizeof(self, context, path): return 0 @singleton class Error(Construct): """ Raises an exception when triggered by parse or build. Can be used as a sentinel that blows a whistle when a conditional branch goes the wrong way, or to raise an error explicitly the declarative way. :raises ExplicitError: when parsed or build Example:: >>> d = ("x"/Byte >> IfThenElse(this.x > 0, Byte, Error)) >>> d.parse(b"\xff\x05") construct.core.ExplicitError: Error field was activated during parsing """ def __init__(self): super(self.__class__, self).__init__() self.flagbuildnone = True def _parse(self, stream, context, path): raise ExplicitError("Error field was activated during parsing") def _build(self, obj, stream, context, path): raise ExplicitError("Error field was activated during building") @singleton class Numpy(Construct): """ Preserves numpy arrays (both shape, dtype and values). :raises ImportError: when numpy cannot be imported during init Example:: >>> import numpy >>> a = numpy.asarray([1,2,3]) >>> Numpy.build(a) b"\x93NUMPY\x01\x00F\x00"... >>> Numpy.parse(_) array([1, 2, 3]) """ def __init__(self): super(self.__class__, self).__init__() try: import numpy self.lib = numpy except ImportError: pass # in case import fails on Travis during singleton making def _parse(self, stream, context, path): return self.lib.load(stream) def _build(self, obj, stream, context, path): self.lib.save(stream, obj) class NamedTuple(Adapter): """ Both arrays, structs, and sequences can be mapped to a namedtuple from collections module. To create a named tuple, you need to provide a name and a sequence of fields, either a string with space-separated names or a list of string names. Just like the standard namedtuple. :raises AdaptationError: when subcon is not either Struct Sequence Range Example:: >>> d = NamedTuple("coord", "x y z", Byte[3]) >>> d = NamedTuple("coord", "x y z", Byte >> Byte >> Byte) >>> d = NamedTuple("coord", "x y z", Struct("x"/Byte, "y"/Byte, "z"/Byte)) >>> d.parse(b"123") coord(x=49, y=50, z=51) """ def __init__(self, tuplename, tuplefields, subcon): super(NamedTuple, self).__init__(subcon) self.factory = collections.namedtuple(tuplename, tuplefields) def _decode(self, obj, context): if isinstance(obj, list): return self.factory(*obj) if isinstance(obj, dict): return self.factory(**obj) raise AdaptationError("can only decode and encode from lists and dicts") def _encode(self, obj, context): if isinstance(self.subcon, (Sequence,Range)): return list(obj) if isinstance(self.subcon, Struct): return {sc.name:getattr(obj,sc.name) for sc in self.subcon.subcons if sc.name is not None} raise AdaptationError("can only decode and encode from lists and dicts") class Rebuild(Subconstruct): """ Parses the field like normal, but computes the value used for building from a context function. Constant value can also be used instead. Building does not require a value, because the value gets recomputed anyway. Size is the same as subcon size. .. seealso:: Useful for length and count fields when :class:`~construct.core.Prefixed` and :class:`~construct.core.PrefixedArray` cannot be used. Example:: >>> d = Struct( ... "count" / Rebuild(Byte, len_(this.items)), ... "items" / Byte[this.count], ... ) >>> d.build(dict(items=[1,2,3])) b'\x03\x01\x02\x03' """ __slots__ = ["func"] def __init__(self, subcon, func): super(Rebuild, self).__init__(subcon) self.func = func self.flagbuildnone = True def _build(self, obj, stream, context, path): obj = self.func(context) if callable(self.func) else self.func self.subcon._build(obj, stream, context, path) return obj class Default(Subconstruct): """ Allows to make a field have a default value, which comes handly when building a Struct from a dict with missing keys. Building does not require a value, but can accept one. Size is the same as subcon size. Example:: >>> d = Struct( ... "a" / Default(Byte, 0), ... ) >>> d.build(dict(a=1)) b'\x01' >>> d.build(dict()) b'\x00' """ __slots__ = ["value"] def __init__(self, subcon, value): super(Default, self).__init__(subcon) self.value = value self.flagbuildnone = True def _build(self, obj, stream, context, path): obj = self.value if obj is None else obj self.subcon._build(obj, stream, context, path) return obj #=============================================================================== # tunneling and swapping #=============================================================================== class RawCopy(Subconstruct): """ Returns a dict containing both parsed subcon, the raw bytes that were consumed by it, starting and ending offset in the stream, and the amount of bytes. Builds either from raw bytes or a value used by subcon. Context does contain a dict with data (if built from raw bytes) or with both (if built from value or parsed). :raises ConstructError: when building and neither data or value is given Example:: >>> d = RawCopy(Byte) >>> d.parse(b"\xff") Container(data=b'\xff')(value=255)(offset1=0)(offset2=1)(length=1) >>> d.build(dict(data=b"\xff")) '\xff' >>> d.build(dict(value=255)) '\xff' """ def _parse(self, stream, context, path): offset1 = stream.tell() obj = self.subcon._parse(stream, context, path) offset2 = stream.tell() stream.seek(offset1) data = _read_stream(stream, offset2-offset1) return Container(data=data, value=obj, offset1=offset1, offset2=offset2, length=(offset2-offset1)) def _build(self, obj, stream, context, path): if 'data' in obj: data = obj['data'] offset1 = stream.tell() _write_stream(stream, len(data), data) offset2 = stream.tell() return Container(obj, data=data, offset1=offset1, offset2=offset2, length=len(data)) if 'value' in obj: value = obj['value'] offset1 = stream.tell() ret = self.subcon._build(value, stream, context, path) value = value if ret is None else ret offset2 = stream.tell() stream.seek(offset1) data = _read_stream(stream, offset2-offset1) return Container(obj, data=data, value=value, offset1=offset1, offset2=offset2, length=(offset2-offset1)) raise ConstructError('both data and value keys are missing, cannot build') def ByteSwapped(subcon): """ Swap the byte order within boundaries of the given subcon. Requires a fixed sized subcon. :param subcon: the subcon on top of byte swapped bytes Example:: Int24ul <--> ByteSwapped(Int24ub) <--> BytesInteger(3, swapped=True) """ return Restreamed(subcon, lambda s: s[::-1], subcon.sizeof(), lambda s: s[::-1], subcon.sizeof(), lambda n: n) def BitsSwapped(subcon): """ Swap the bit order within each byte within boundaries of the given subcon. Does NOT require a fixed sized subcon. :param subcon: the subcon on top of bit swapped bytes Example:: >>> d = Bitwise(Bytes(8)) >>> d.parse(b"\x01") '\x00\x00\x00\x00\x00\x00\x00\x01' >>>> BitsSwapped(d).parse(b"\x01") '\x01\x00\x00\x00\x00\x00\x00\x00' """ return Restreamed(subcon, lambda s: bits2bytes(bytes2bits(s)[::-1]), 1, lambda s: bits2bytes(bytes2bits(s)[::-1]), 1, lambda n: n) class Prefixed(Subconstruct): """ Parses the length field. Then reads that amount of bytes and parses the subcon using only those bytes. Constructs that consume entire remaining stream are constrained to consuming only the specified amount of bytes. When building, data is prefixed by its length. Optionally, length field can include its own size. .. seealso:: The :class:`~construct.core.VarInt` encoding should be preferred over `Int*` fixed sized fields. VarInt is more compact and never overflows. :param lengthfield: a subcon used for storing the length :param subcon: the subcon used for storing the value :param includelength: optional, whether length field should include own size Example:: >>> Prefixed(VarInt, GreedyRange(Int32ul)).parse(b"\x08abcdefgh") [1684234849, 1751606885] >>> PrefixedArray(VarInt, Int32ul).parse(b"\x02abcdefgh") [1684234849, 1751606885] """ __slots__ = ["name", "lengthfield", "subcon", "includelength"] def __init__(self, lengthfield, subcon, includelength=False): super(Prefixed, self).__init__(subcon) self.lengthfield = lengthfield self.includelength = includelength def _parse(self, stream, context, path): length = self.lengthfield._parse(stream, context, path) if self.includelength: length -= self.lengthfield._sizeof(context, path) stream2 = BytesIO(_read_stream(stream, length)) return self.subcon._parse(stream2, context, path) def _build(self, obj, stream, context, path): stream2 = BytesIO() obj = self.subcon._build(obj, stream2, context, path) data = stream2.getvalue() length = len(data) if self.includelength: length += self.lengthfield._sizeof(context, path) self.lengthfield._build(length, stream, context, path) _write_stream(stream, len(data), data) return obj def _sizeof(self, context, path): return self.lengthfield._sizeof(context, path) + self.subcon._sizeof(context, path) class Checksum(Construct): """ Field that is build or validated by a hash of a given byte range. :param checksumfield: a subcon field that reads the checksum, usually Bytes(int) :param hashfunc: a function taking bytes and returning whatever checksumfield takes when building :param bytesfunc: a function taking context and returning the bytes or object to be hashed, usually like this.rawcopy1.data Example:: import hashlib d = Struct( "fields" / RawCopy(Struct( "a" / Byte, "b" / Byte, )), "checksum" / Checksum(Bytes(64), lambda data: hashlib.sha512(data).digest(), this.fields.data), ) d.build(dict(fields=dict(value=dict(a=1,b=2)))) -> b'\x01\x02\xbd\xd8\x1a\xb23\xbc\xebj\xd23\xcd'... """ __slots__ = ["checksumfield", "hashfunc", "bytesfunc"] def __init__(self, checksumfield, hashfunc, bytesfunc): super(Checksum, self).__init__() self.checksumfield = checksumfield self.hashfunc = hashfunc self.bytesfunc = bytesfunc self.flagbuildnone = True def _parse(self, stream, context, path): hash1 = self.checksumfield._parse(stream, context, path) hash2 = self.hashfunc(self.bytesfunc(context)) if hash1 != hash2: raise ChecksumError("wrong checksum, read %r, computed %r" % ( hash1 if not isinstance(hash1,bytes) else hexlify(hash1), hash2 if not isinstance(hash2,bytes) else hexlify(hash2), )) return hash1 def _build(self, obj, stream, context, path): hash2 = self.hashfunc(self.bytesfunc(context)) self.checksumfield._build(hash2, stream, context, path) return hash2 def _sizeof(self, context, path): return self.checksumfield._sizeof(context, path) class Compressed(Tunnel): """ Compresses and decompresses underlying stream when processing the subcon. When parsing, entire stream is consumed. When building, puts compressed bytes without marking the end. This construct should be used with :func:`~construct.core.Prefixed` or entire stream. :param subcon: the subcon used for storing the value :param encoding: any of the module names like zlib/gzip/bzip2/lzma, otherwise any of codecs module bytes<->bytes encodings, usually requires some Python version :param level: optional, an integer between 0..9, lzma discards it Example:: Prefixed(VarInt, Compressed(GreedyBytes, "zlib")) """ __slots__ = ["encoding", "level", "lib"] def __init__(self, subcon, encoding, level=None): super(Compressed, self).__init__(subcon) self.encoding = encoding self.level = level if self.encoding == "zlib": import zlib self.lib = zlib elif self.encoding == "gzip": import gzip self.lib = gzip elif self.encoding == "bzip2": import bz2 self.lib = bz2 elif self.encoding == "lzma": import lzma self.lib = lzma else: import codecs self.lib = codecs def _decode(self, data, context): if self.encoding in ("zlib", "gzip", "bzip2", "lzma"): return self.lib.decompress(data) return self.lib.decode(data, self.encoding) def _encode(self, data, context): if self.encoding in ("zlib", "gzip", "bzip2", "lzma"): if self.level is None or self.encoding == "lzma": return self.lib.compress(data) else: return self.lib.compress(data, self.level) return self.lib.encode(data, self.encoding) #=============================================================================== # lazy equivalents #=============================================================================== class LazyStruct(Construct): """ Equivalent to Struct construct, however fixed size members are parsed on demand, others are parsed immediately. If entire struct is fixed size then entire parse is essentially one stream seek. .. seealso:: Equivalent to :func:`~construct.core.Struct`. """ __slots__ = ["subcons", "offsetmap", "totalsize", "subsizes", "keys"] def __init__(self, *subcons, **kw): super(LazyStruct, self).__init__() self.subcons = list(subcons) + list(k/v for k,v in kw.items()) try: keys = Container() self.offsetmap = {} at = 0 for sc in self.subcons: if sc.flagembedded: raise SizeofError if sc.name is not None: keys[sc.name] = None self.offsetmap[sc.name] = (at, sc) at += sc.sizeof() self.totalsize = at self.keys = list(keys.keys()) except SizeofError: self.offsetmap = None self.totalsize = None self.subsizes = [] for sc in self.subcons: try: self.subsizes.append(sc.sizeof()) except SizeofError: self.subsizes.append(None) def _parse(self, stream, context, path): if self.offsetmap is not None: position = stream.tell() stream.seek(self.totalsize, 1) return LazyContainer(self.keys, self.offsetmap, {}, stream, position, context) context = Container(_ = context) offsetmap = {} keys = Container() values = {} position = stream.tell() for (sc,size) in zip(self.subcons, self.subsizes): if sc.flagembedded: subobj = list(sc._parse(stream, context, path).items()) keys.update(subobj) values.update(subobj) context.update(subobj) elif size is None: subobj = sc._parse(stream, context, path) if sc.name is not None: keys[sc.name] = None values[sc.name] = subobj context[sc.name] = subobj else: if sc.name is not None: keys[sc.name] = None offsetmap[sc.name] = (stream.tell(), sc) stream.seek(size, 1) return LazyContainer(list(keys.keys()), offsetmap, values, stream, 0, context) def _build(self, obj, stream, context, path): context = Container(_ = context) context.update(obj) for sc in self.subcons: if sc.flagembedded: subobj = obj elif sc.flagbuildnone: subobj = obj.get(sc.name, None) else: subobj = obj[sc.name] buildret = sc._build(subobj, stream, context, path) if buildret is not None: if sc.flagembedded: context.update(buildret) if sc.name is not None: context[sc.name] = buildret return context def _sizeof(self, context, path): if self.totalsize is not None: return self.totalsize else: raise SizeofError("cannot calculate size, not all members are fixed size") class LazyRange(Construct): """ Equivalent to Range construct, but members are parsed on demand. Works only with fixed size subcon. Entire parse is essentially one stream seek. .. seealso:: Equivalent to :func:`~construct.core.Range`. """ __slots__ = ["subcon", "min", "max", "subsize"] def __init__(self, min, max, subcon): super(LazyRange, self).__init__() self.subcon = subcon self.min = min self.max = max self.subsize = subcon.sizeof() def _parse(self, stream, context, path): currentmin = self.min(context) if callable(self.min) else self.min currentmax = self.max(context) if callable(self.max) else self.max if not 0 <= currentmin <= currentmax <= sys.maxsize: raise RangeError("unsane min %s and max %s" % (currentmin, currentmax)) starts = stream.tell() ends = stream.seek(0,2) remaining = ends - starts objcount = min(remaining//self.subsize, currentmax) if objcount < currentmin: raise RangeError("not enough bytes %d to read the min %d of %d bytes each" % (remaining, currentmin, self.subsize)) stream.seek(starts + objcount*self.subsize, 0) return LazyRangeContainer(self.subcon, self.subsize, objcount, stream, starts, context) def _build(self, obj, stream, context, path): currentmin = self.min(context) if callable(self.min) else self.min currentmax = self.max(context) if callable(self.max) else self.max if not 0 <= currentmin <= currentmax <= sys.maxsize: raise RangeError("unsane min %s and max %s" % (currentmin, currentmax)) if not isinstance(obj, collections.Sequence): raise RangeError("expected sequence type, found %s" % type(obj)) if not currentmin <= len(obj) <= currentmax: raise RangeError("expected from %d to %d elements, found %d" % (currentmin, currentmax, len(obj))) try: for i,subobj in enumerate(obj): context[i] = subobj self.subcon._build(subobj, stream, context, path) except ConstructError: if len(obj) < currentmin: raise RangeError("expected %d to %d, found %d" % (currentmin, currentmax, len(obj))) def _sizeof(self, context, path): try: currentmin = self.min(context) if callable(self.min) else self.min currentmax = self.max(context) if callable(self.max) else self.max if not 0 <= currentmin <= currentmax <= sys.maxsize: raise RangeError("unsane min %s and max %s" % (currentmin, currentmax)) if currentmin == currentmax: return self.min * self.subsize else: raise SizeofError("cannot calculate size, min not equal to max") except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") class LazySequence(Construct): """ Equivalent to Sequence construct, however fixed size members are parsed on demand, others are parsed immediately. If entire sequence is fixed size then entire parse is essentially one seek. .. seealso:: Equivalent to :func:`~construct.core.Sequence`. """ __slots__ = ["subcons", "offsetmap", "totalsize", "subsizes"] def __init__(self, *subcons, **kw): super(LazySequence, self).__init__() self.subcons = list(subcons) + list(k/v for k,v in kw.items()) try: self.offsetmap = {} at = 0 for i,sc in enumerate(self.subcons): if sc.flagembedded: raise SizeofError self.offsetmap[i] = (at, sc) at += sc.sizeof() self.totalsize = at except SizeofError: self.offsetmap = None self.totalsize = None self.subsizes = [] for sc in self.subcons: try: self.subsizes.append(sc.sizeof()) except SizeofError: self.subsizes.append(None) def _parse(self, stream, context, path): context = Container(_ = context) if self.totalsize is not None: position = stream.tell() stream.seek(self.totalsize, 1) return LazySequenceContainer(len(self.subcons), self.offsetmap, {}, stream, position, context) offsetmap = {} values = {} i = 0 for sc,size in zip(self.subcons, self.subsizes): if sc.flagembedded: subobj = list(sc._parse(stream, context, path)) for e in subobj: values[i] = e context[i] = e i += 1 elif size is None: obj = sc._parse(stream, context, path) values[i] = obj context[i] = obj i += 1 else: offsetmap[i] = (stream.tell(), sc) stream.seek(size, 1) i += 1 return LazySequenceContainer(i, offsetmap, values, stream, 0, context) def _build(self, obj, stream, context, path): context = Container(_ = context) objiter = iter(obj) for i,sc in enumerate(self.subcons): if sc.flagembedded: subobj = objiter else: subobj = next(objiter) if sc.name is not None: context[sc.name] = subobj context[i] = subobj buildret = sc._build(subobj, stream, context, path) if buildret is not None: if sc.flagembedded: context.update(buildret) if sc.name is not None: context[sc.name] = buildret context[i] = buildret def _sizeof(self, context, path): if self.totalsize is not None: return self.totalsize else: raise SizeofError("cannot calculate size, not all members are fixed size") class OnDemand(Subconstruct): """ Allows for on demand (lazy) parsing. When parsing, it will return a parameterless function that when called, will return the parsed value. Object is cached after first parsing, so non-deterministic subcons will be affected. Works only with fixed size subcon. :param subcon: the subcon to read/write on demand, must be fixed size Example:: >>> d = OnDemand(Byte) >>> d.parse(b"\xff") . at 0x7fdc241cfc80> >>> _() 255 >>> d.build(255) b'\xff' Can also re-build from the lambda returned at parsing. >>> d.parse(b"\xff") . at 0x7fcbd9855f28> >>> d.build(_) b'\xff' """ def _parse(self, stream, context, path): offset = stream.tell() stream.seek(self.subcon._sizeof(context, path), 1) cache = {} def effectuate(): if not cache: fallback = stream.tell() stream.seek(offset) obj = self.subcon._parse(stream, context, path) stream.seek(fallback) cache["parsed"] = obj return cache["parsed"] return effectuate def _build(self, obj, stream, context, path): obj = obj() if callable(obj) else obj return self.subcon._build(obj, stream, context, path) def OnDemandPointer(offset, subcon): """ CURRENTLY BROKEN. On demand pointer. Is both lazy and jumps to a position before reading. .. seealso:: Base :func:`~construct.core.OnDemand` and :func:`~construct.core.Pointer` construct. :param offset: an integer or a context function that returns such an integer :param subcon: subcon that will be parsed or build at the `offset` stream position Example:: >>> d = OnDemandPointer(lambda ctx: 2, Byte) >>> d.parse(b"\x01\x02\x03\x04\x05") .effectuate at 0x7f6f011ad510> >>> _() 3 """ return OnDemand(Pointer(offset, subcon)) class LazyBound(Construct): """ Lazy-bound construct that binds to the construct only at runtime. Useful for recursive data structures (like linked lists or trees), where a construct needs to refer to itself (while it does not exist yet). :param subconfunc: a context function returning a Construct (derived) instance, can also return Pass or itself Example:: >>> d = Struct( ... "value"/Byte, ... "next"/If(this.value > 0, LazyBound(lambda ctx: d)), ... ) ... >>> d.parse(b"\x05\x09\x00") Container(value=5)(next=Container(value=9)(next=Container(value=0)(next=None))) ... >>> print(d.parse(b"\x05\x09\x00")) Container: value = 5 next = Container: value = 9 next = Container: value = 0 next = None """ __slots__ = ["subconfunc"] def __init__(self, subconfunc): super(LazyBound, self).__init__() self.subconfunc = subconfunc def _parse(self, stream, context, path): return self.subconfunc(context)._parse(stream, context, path) def _build(self, obj, stream, context, path): return self.subconfunc(context)._build(obj, stream, context, path) def _sizeof(self, context, path): try: return self.subconfunc(context)._sizeof(context, path) except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") #=============================================================================== # special #=============================================================================== class Embedded(Subconstruct): """ Embeds a struct into the enclosing struct, merging fields. Can also embed sequences into sequences. Name is also inherited. .. warning:: You can use Embedded(Switch(...)) but not Switch(Embedded(...)). Sames applies to If and IfThenElse macros. :param subcon: the inner struct to embed inside outer struct or sequence Example:: >>> Struct("a"/Byte, Embedded(Struct("b"/Byte)), "c"/Byte).parse(b"abc") Container(a=97)(b=98)(c=99) >>> Struct("a"/Byte, Embedded(Struct("b"/Byte)), "c"/Byte).build(_) b'abc' """ def __init__(self, subcon): super(Embedded, self).__init__(subcon) self.flagembedded = True class Renamed(Subconstruct): """ Renames an existing construct. This creates a wrapper so underlying subcon retains it's original name, which in general means just a None. Can be used to give same construct few different names. Used internally by / operator. Also this wrapper is responsible for building a path (a chain of names) that gets attached to error message when parsing, building, or sizeof fails. A field that is not named does not appear on the path. :param newname: the new name :param subcon: the subcon to rename Example:: >>> "name" / Int32ul >>> Renamed("name", Int32ul) """ def __init__(self, newname, subcon): super(Renamed, self).__init__(subcon) self.name = newname def _parse(self, stream, context, path): try: path += " -> %s" % (self.name) return self.subcon._parse(stream, context, path) except ConstructError as e: if "\n" in str(e): raise raise e.__class__("%s\n %s" % (e, path)) def _build(self, obj, stream, context, path): try: path += " -> %s" % (self.name) return self.subcon._build(obj, stream, context, path) except ConstructError as e: if "\n" in str(e): raise raise e.__class__("%s\n %s" % (e, path)) def _sizeof(self, context, path): try: path += " -> %s" % (self.name) return self.subcon._sizeof(context, path) except ConstructError as e: if "\n" in str(e): raise raise e.__class__("%s\n %s" % (e, path)) #=============================================================================== # mappings #=============================================================================== class Mapping(Adapter): """ Adapter that maps objects to other objects. Translates objects before parsing and before building. :param subcon: the subcon to map :param decoding: the decoding (parsing) mapping as a dict :param encoding: the encoding (building) mapping as a dict :param decdefault: the default return value when object is not found in the mapping, if no object is given an exception is raised, if ``Pass`` is used, the unmapped object will be passed as-is :param encdefault: the default return value when object is not found in the mapping, if no object is given an exception is raised, if ``Pass`` is used, the unmapped object will be passed as-is Example:: ??? """ __slots__ = ["encoding", "decoding", "encdefault", "decdefault"] def __init__(self, subcon, decoding, encoding, decdefault=NotImplemented, encdefault=NotImplemented): super(Mapping, self).__init__(subcon) self.decoding = decoding self.encoding = encoding self.decdefault = decdefault self.encdefault = encdefault def _encode(self, obj, context): try: return self.encoding[obj] except ExplicitError: raise except (KeyError, TypeError): if self.encdefault is NotImplemented: raise MappingError("no encoding mapping for %r" % (obj,)) if self.encdefault is Pass: return obj return self.encdefault def _decode(self, obj, context): try: return self.decoding[obj] except ExplicitError: raise except (KeyError, TypeError): if self.decdefault is NotImplemented: raise MappingError("no decoding mapping for %r" % (obj,)) if self.decdefault is Pass: return obj return self.decdefault def SymmetricMapping(subcon, mapping, default=NotImplemented): """ Defines a symmetric mapping, same mapping is used on parsing and building. .. seealso:: Based on :func:`~construct.core.Mapping`. :param subcon: the subcon to map :param encoding: the mapping as a dict :param decdefault: the default return value when object is not found in the mapping, if no object is given an exception is raised, if ``Pass`` is used, the unmapped object will be passed as-is Example:: ??? """ return Mapping(subcon, encoding = mapping, decoding = dict((v,k) for k, v in mapping.items()), encdefault = default, decdefault = default, ) @singleton def Flag(): """ One byte (or one bit) field that maps to True or False. Other non-zero values are also considered True. Example:: >>> Flag.parse(b"\x01") True >>> Flag.build(True) b'\x01' """ return SymmetricMapping(Byte, {True : 1, False : 0}, default=True) def Enum(subcon, default=NotImplemented, **mapping): """ Set of named values mapping. Can build both from names and values. :param subcon: the subcon to map :param \*\*mapping: keyword arguments which serve as the encoding mapping :param default: an optional, keyword-only argument that specifies the default value to use when the mapping is undefined. if not given, and exception is raised when the mapping is undefined. use `Pass` topass the unmapped value as-is Example:: >>> d = Enum(Byte, a=1, b=2) >>> d.parse(b"\x01") 'a' >>> d.parse(b"\x08") construct.core.MappingError: no decoding mapping for 8 >>> d.build("a") b'\x01' >>> d.build(1) b'\x01' """ encmapping = mapping.copy() encmapping.update({v:v for v in mapping.values()}) return Mapping(subcon, encoding = encmapping, decoding = dict((v,k) for k, v in mapping.items()), encdefault = default, decdefault = default, ) class FlagsEnum(Adapter): """ Set of flag values mapping. Each flag is extracted from the number, resulting in a FlagsContainer dict that has each key assigned True or False. :param subcon: the subcon to extract :param \*\*flags: a dictionary mapping flag-names to their value Example:: >>> d = FlagsEnum(Byte, a=1, b=2, c=4, d=8) >>> d.parse(b"\x03") Container(c=False)(b=True)(a=True)(d=False) """ __slots__ = ["flags"] def __init__(self, subcon, **flags): super(FlagsEnum, self).__init__(subcon) self.flags = flags def _encode(self, obj, context): flags = 0 try: for name, value in obj.items(): if value: flags |= self.flags[name] except ExplicitError: raise except AttributeError: raise MappingError("not a mapping type: %r" % (obj,)) except KeyError: raise MappingError("unknown flag: %s" % name) return flags def _decode(self, obj, context): obj2 = FlagsContainer() for name, value in self.flags.items(): obj2[name] = bool(obj & value) return obj2 #=============================================================================== # adapters and validators #=============================================================================== class ExprAdapter(Adapter): """ A generic adapter that takes ``encoder`` and ``decoder`` as parameters. You can use ExprAdapter instead of writing a full-blown class when only a simple lambda is needed. :param subcon: the subcon to adapt :param encoder: a function that takes (obj, context) and returns an encoded version of obj, or None for identity :param decoder: a function that takes (obj, context) and returns an decoded version of obj, or None for identity Example:: Ident = ExprAdapter(Byte, encoder = lambda obj,ctx: obj+1, decoder = lambda obj,ctx: obj-1, ) """ __slots__ = ["_encode", "_decode"] def __init__(self, subcon, encoder, decoder): super(ExprAdapter, self).__init__(subcon) ident = lambda obj,ctx: obj self._encode = encoder if callable(encoder) else ident self._decode = decoder if callable(decoder) else ident class ExprSymmetricAdapter(ExprAdapter): def __init__(self, subcon, encoder): super(ExprAdapter, self).__init__(subcon) ident = lambda obj,ctx: obj self._encode = encoder if callable(encoder) else ident self._decode = self._encode class ExprValidator(Validator): """ A generic adapter that takes ``validator`` as parameter. You can use ExprValidator instead of writing a full-blown class when only a simple expression is needed. :param subcon: the subcon to adapt :param encoder: a function that takes (obj, context) and returns a bool Example:: OneOf = ExprValidator(Byte, validator = lambda obj,ctx: obj in [1,3,5]) """ def __init__(self, subcon, validator): super(ExprValidator, self).__init__(subcon) self._validate = validator def Hex(subcon): """ Adapter for hex-dumping bytes. It returns a hex dump when parsing, and un-dumps when building. Example:: >>> Hex(GreedyBytes).parse(b"abcd") b'61626364' >>> Hex(GreedyBytes).build("01020304") b'\x01\x02\x03\x04' """ return ExprAdapter(subcon, encoder = lambda obj,ctx: None if subcon.flagbuildnone else unhexlify(obj), decoder = lambda obj,ctx: hexlify(obj),) def HexDump(subcon, linesize=16): """ Adapter for hex-dumping bytes. It returns a hex dump when parsing, and un-dumps when building. :param linesize: default 16 bytes per line :param buildraw: by default build takes the same format that parse returns, set to build from a bytes directly Example:: >>> HexDump(Bytes(10)).parse(b"12345abc;/") '0000 31 32 33 34 35 61 62 63 3b 2f 12345abc;/ \n' """ return ExprAdapter(subcon, encoder = lambda obj,ctx: None if subcon.flagbuildnone else hexundump(obj, linesize=linesize), decoder = lambda obj,ctx: hexdump(obj, linesize=linesize),) class Slicing(Adapter): """ Adapter for slicing a list (getting a slice from that list). Works with Range and Sequence and their lazy equivalents. :param subcon: the subcon to slice :param count: expected number of elements, needed during building :param start: start index (or None for entire list) :param stop: stop index (or None for up-to-end) :param step: step (or 1 for every element) :param empty: value to fill the list with during building Example:: ??? """ __slots__ = ["count", "start", "stop", "step", "empty"] def __init__(self, subcon, count, start, stop, step=1, empty=None): super(Slicing, self).__init__(subcon) self.count = count self.start = start self.stop = stop self.step = step self.empty = empty def _encode(self, obj, context): if self.start is None: return obj elif self.stop is None: output = [self.empty] * self.count output[self.start::self.step] = obj else: output = [self.empty] * self.count output[self.start:self.stop:self.step] = obj return output def _decode(self, obj, context): return obj[self.start:self.stop:self.step] class Indexing(Adapter): """ Adapter for indexing a list (getting a single item from that list). Works with Range and Sequence and their lazy equivalents. :param subcon: the subcon to index :param count: expected number of elements, needed during building :param index: the index of the list to get :param empty: value to fill the list with during building Example:: ??? """ __slots__ = ["count", "index", "empty"] def __init__(self, subcon, count, index, empty=None): super(Indexing, self).__init__(subcon) self.count = count self.index = index self.empty = empty def _encode(self, obj, context): output = [self.empty] * self.count output[self.index] = obj return output def _decode(self, obj, context): return obj[self.index] class FocusedSeq(Construct): """ Parses and builds a sequence where only one subcon value is returned from parsing or taken into building, other fields are parsed and discarded or built from nothing. :param parsebuildfrom: which subcon to use, an integer index or string name, or a context lambda returning either :param \*subcons: a list of members :param \*\*kw: a list of members (works ONLY on python 3.6) Excample:: >>> d = FocusedSeq("num", Const(b"MZ"), "num"/Byte, Terminated) >>> d = FocusedSeq(1, Const(b"MZ"), "num"/Byte, Terminated) >>> d.parse(b"MZ\xff") 255 >>> d.build(255) b'MZ\xff' """ def __init__(self, parsebuildfrom, *subcons, **kw): subcons = list(subcons) + list(k/v for k,v in kw.items()) super(FocusedSeq, self).__init__() self.parsebuildfrom = parsebuildfrom self.subcons = subcons def _parse(self, stream, context, path): if callable(self.parsebuildfrom): self.parsebuildfrom = self.parsebuildfrom(context) if isinstance(self.parsebuildfrom, int): index = self.parsebuildfrom self.subcons[index] #IndexError check if isinstance(self.parsebuildfrom, str): index = [i for i,sc in enumerate(self.subcons) if sc.name == self.parsebuildfrom][0] for i,sc in enumerate(self.subcons): parseret = sc._parse(stream, context, path) context[i] = parseret if sc.name is not None: context[sc.name] = parseret if i == index: finalobj = parseret return finalobj def _build(self, obj, stream, context, path): if callable(self.parsebuildfrom): self.parsebuildfrom = self.parsebuildfrom(context) if isinstance(self.parsebuildfrom, int): index = self.parsebuildfrom self.subcons[index] #IndexError check if isinstance(self.parsebuildfrom, str): index = [i for i,sc in enumerate(self.subcons) if sc.name == self.parsebuildfrom][0] for i,sc in enumerate(self.subcons): if i == index: context[i] = obj if sc.name is not None: context[sc.name] = obj for i,sc in enumerate(self.subcons): buildret = sc._build(obj if i==index else None, stream, context, path) if buildret is not None: if sc.name is not None: context[sc.name] = buildret context[i] = buildret if i == index: finalobj = buildret return finalobj def _sizeof(self, context, path): try: if callable(self.parsebuildfrom): self.parsebuildfrom = self.parsebuildfrom(context) except (KeyError, AttributeError): raise SizeofError("cannot calculate size, key not found in context") if isinstance(self.parsebuildfrom, int): index = self.parsebuildfrom self.subcons[index] #IndexError check if isinstance(self.parsebuildfrom, str): index = [i for i,sc in enumerate(self.subcons) if sc.name == self.parsebuildfrom][0] return self.subcons[index]._sizeof(context, path) def OneOf(subcon, valids): """ Validates that the object is one of the listed values, both during parsing and building. Note that providing a set instead of a list may increase performance. :param subcon: a construct to validate :param valids: a collection implementing __contains__ :raises ValidationError: when actual value is not among valids Example:: >>> d = OneOf(Byte, [1,2,3]) >>> d.parse(b"\x01") 1 >>> d.parse(b"\xff") construct.core.ValidationError: ('object failed validation', 255) >>> d = OneOf(Bytes(2), b"1234567890") >>> d.parse(b"78") b'78' >>> d.parse(b"19") construct.core.ValidationError: ('invalid object', b'19') """ return ExprValidator(subcon, lambda obj,ctx: obj in valids) def NoneOf(subcon, invalids): """ Validates that the object is none of the listed values, both during parsing and building. :param subcon: a construct to validate :param valids: a collection implementing __contains__ :raises ValidationError: when actual value is among invalids .. seealso:: Analog of :func:`~construct.core.OneOf`. """ return ExprValidator(subcon, lambda obj,ctx: obj not in invalids) def Filter(predicate, subcon): """ Filters a list leaving only the elements that passed through the validator. :param subcon: a construct to validate, usually a Range Array Sequence :param predicate: a function taking (obj, context) and returning a bool Example:: >>> d = Filter(obj_ != 0, Byte[:]) >>> d.parse(b"\x00\x02\x00") [2] >>> d.build([0,1,0,2,0]) b'\x01\x02' """ return ExprSymmetricAdapter(subcon, lambda obj,ctx: [x for x in obj if predicate(x,ctx)]) class Check(Construct): """ Checks for a condition, and raises ValidationError if the check fails. :param func: a context function returning a bool (or truthy value) :raises ValidationError: when condition fails Example:: Check(lambda ctx: len(ctx.payload.data) == ctx.payload_len) Check(len_(this.payload.data) == this.payload_len) """ def __init__(self, func): super(Check, self).__init__() self.func = func self.flagbuildnone = True def _parse(self, stream, context, path): if not self.func(context): raise ValidationError("check failed during parsing") def _build(self, obj, stream, context, path): if not self.func(context): raise ValidationError("check failed during building") def _sizeof(self, context, path): return 0 class StopIf(Construct): """ Checks for a condition, and stops a Struct/Sequence/Range from parsing or building further. Example:: Struct('x'/Byte, StopIf(this.x == 0), 'y'/Byte) Sequence('x'/Byte, StopIf(this.x == 0), 'y'/Byte) GreedyRange(FocusedSeq(0, 'x'/Byte, StopIf(this.x == 0))) """ def __init__(self, condfunc): super(StopIf, self).__init__() self.condfunc = condfunc self.flagbuildnone = True def _parse(self, stream, context, path): if self.condfunc(context): raise StopIteration def _build(self, obj, stream, context, path): if self.condfunc(context): raise StopIteration def _sizeof(self, context, path): return SizeofError("Struct/Sequence/Range cannot compute size because StopIf is runtime-dependant") #=============================================================================== # strings #=============================================================================== globalstringencoding = None def setglobalstringencoding(encoding): """ Sets the encoding globally for all String/PascalString/CString/GreedyString instances. :param encoding: a string like "utf8", or None which means working with bytes (not unicode) """ global globalstringencoding globalstringencoding = encoding class StringEncoded(Adapter): """Used internally.""" __slots__ = ["encoding"] def __init__(self, subcon, encoding): super(StringEncoded, self).__init__(subcon) self.encoding = encoding def _decode(self, obj, context): encoding = self.encoding or globalstringencoding if encoding: if isinstance(encoding, str): obj = obj.decode(encoding) else: obj = encoding.decode(obj) return obj def _encode(self, obj, context): encoding = self.encoding or globalstringencoding if not isinstance(obj, bytes): if not encoding: raise StringError("no encoding provided when processing a unicode obj") if isinstance(encoding, str): obj = obj.encode(encoding) else: obj = encoding.encode(obj) return obj class StringPaddedTrimmed(Adapter): """Used internally.""" __slots__ = ["length", "padchar", "paddir", "trimdir"] def __init__(self, length, subcon, padchar=b"\x00", paddir="right", trimdir="right"): if not isinstance(padchar, bytes): raise StringError("padchar must be b-string character") super(StringPaddedTrimmed, self).__init__(subcon) self.length = length self.padchar = padchar self.paddir = paddir self.trimdir = trimdir def _decode(self, obj, context): if self.paddir == "right": obj = obj.rstrip(self.padchar) elif self.paddir == "left": obj = obj.lstrip(self.padchar) elif self.paddir == "center": obj = obj.strip(self.padchar) else: raise StringError("paddir must be one of: right left center") return obj def _encode(self, obj, context): length = self.length(context) if callable(self.length) else self.length if self.paddir == "right": obj = obj.ljust(length, self.padchar[0:1]) elif self.paddir == "left": obj = obj.rjust(length, self.padchar[0:1]) elif self.paddir == "center": obj = obj.center(length, self.padchar[0:1]) else: raise StringError("paddir must be one of: right left center") if len(obj) > length: if self.trimdir == "right": obj = obj[:length] elif self.trimdir == "left": obj = obj[-length:] else: raise StringError("expected a string of length %s given %s (%r)" % (length,len(obj),obj)) return obj def String(length, encoding=None, padchar=b"\x00", paddir="right", trimdir="right"): """ Configurable, fixed-length or variable-length string field. When parsing, the byte string is stripped of pad character (as specified) from the direction (as specified) then decoded (as specified). Length is a constant integer or a context function. When building, the string is encoded (as specified) then padded (as specified) from the direction (as specified) or trimmed (as specified). The padding character and direction must be specified for padding to work. The trim direction must be specified for trimming to work. If encoding is not specified, it works with bytes (not unicode strings). :param length: length in bytes (not unicode characters), as integer or context function :param encoding: encoding (eg. "utf8") or None for bytes :param padchar: bytes character to pad out strings (by default b"\x00") :param paddir: direction to pad out strings (one of: right left both) :param trimdir: direction to trim strings (one of: right left) Example:: >>> d = String(10) >>> d.build(b"hello") b'hello\x00\x00\x00\x00\x00' >>> d.parse(_) b'hello' >>> d.sizeof() 10 >>> d = String(10, encoding="utf8") >>> d.build(u"Афон") b'\xd0\x90\xd1\x84\xd0\xbe\xd0\xbd\x00\x00' >>> d.parse(_) u'Афон' >>> d = String(10, padchar=b"XYZ", paddir="center") >>> d.build(b"abc") b'XXXabcXXXX' >>> d.parse(b"XYZabcXYZY") b'abc' >>> d = String(10, trimdir="right") >>> d.build(b"12345678901234567890") b'1234567890' """ return StringEncoded( StringPaddedTrimmed(length, Bytes(length), padchar, paddir, trimdir), encoding) def PascalString(lengthfield, encoding=None): """ Length-prefixed string. The length field can be variable length (such as VarInt) or fixed length (such as Int64ul). VarInt is recommended for new designs. Stored length is in bytes, not characters. :param lengthfield: a field used to parse and build the length (eg. VarInt Int64ul) :param encoding: encoding (eg. "utf8"), or None for bytes Example:: >>> d = PascalString(VarInt, encoding="utf8") >>> d.build(u"Афон") b'\x08\xd0\x90\xd1\x84\xd0\xbe\xd0\xbd' >>> d.parse(_) u'Афон' """ return StringEncoded(Prefixed(lengthfield, GreedyBytes), encoding) def CString(terminators=b"\x00", encoding=None): """ String ending in a terminator byte. By default, the terminator is the \x00 byte character. Terminators field can be a longer bytes, and any one of the characters breaks parsing. First terminator byte is used when building. :param terminators: sequence of valid terminators, first is used when building, all are used when parsing :param encoding: encoding (eg. "utf8"), or None for bytes .. warning:: Do not use >1 byte encodings like UTF16 or UTF32 with CStrings, they are not safe. Example:: >>> d = CString(encoding="utf8") >>> d.build(u"Афон") b'\xd0\x90\xd1\x84\xd0\xbe\xd0\xbd\x00' >>> d.parse(_) u'Афон' """ return StringEncoded( ExprAdapter( RepeatUntil(lambda obj,lst,ctx: int2byte(obj) in terminators, Byte), encoder = lambda obj,ctx: iterateints(obj+terminators), decoder = lambda obj,ctx: b''.join(int2byte(c) for c in obj[:-1])), encoding) def GreedyString(encoding=None): """ String that reads the rest of the stream until EOF, and writes a given string as is. If no encoding is specified, this is essentially GreedyBytes. :param encoding: encoding (eg. "utf8"), or None for bytes .. seealso:: Analog to :class:`~construct.core.GreedyBytes` and the same when no enoding is used. Example:: >>> d = GreedyString(encoding="utf8") >>> d.build(u"Афон") b'\xd0\x90\xd1\x84\xd0\xbe\xd0\xbd' >>> d.parse(_) u'Афон' """ return StringEncoded(GreedyBytes, encoding) #=============================================================================== # end of file #=============================================================================== construct-2.8.16/construct/debug.py0000644000175000017500000002317413151124063021337 0ustar arkadiuszarkadiusz00000000000000""" Debugging utilities for constructs """ import sys import traceback import pdb import inspect from construct import * from construct.lib import * class Probe(Construct): r""" A probe: dumps the context, stack frames, and stream content to the screen to aid the debugging process. :param name: the display name :param show_stream: whether or not to show stream contents. default is True. the stream must be seekable. :param show_context: whether or not to show the context. default is True. :param show_stack: whether or not to show the upper stack frames. default is True. :param stream_lookahead: the number of bytes to dump when show_stack is set. default is 100. Example:: >>> Struct("count"/Byte, "items"/Byte[this.count], Probe()).parse(b"\x05abcde") ================================================================================ Probe EOF reached Container: count = 5 items = ListContainer: 97 98 99 100 101 ================================================================================ Container(count=5)(items=[97, 98, 99, 100, 101]) >>> (Byte >> Probe()).parse(b"?") ================================================================================ Probe EOF reached Container: 0 = 63 ================================================================================ [63, None] """ __slots__ = ["printname", "show_stream", "show_context", "show_stack", "stream_lookahead", "func"] counter = 0 def __init__(self, name=None, show_stream=True, show_context=True, show_stack=True, stream_lookahead=128, func=None): super(Probe, self).__init__() if name is None: Probe.counter += 1 name = "" % Probe.counter self.printname = name self.show_stream = show_stream self.show_context = show_context self.show_stack = show_stack self.stream_lookahead = stream_lookahead self.func = func self.flagbuildnone = True def __repr__(self): return "<%s: %s>" % (self.__class__.__name__, self.printname) def _parse(self, stream, context, path): self.printout(stream, context, path) def _build(self, obj, stream, context, path): self.printout(stream, context, path) def _sizeof(self, context, path): self.printout(None, context, path) return 0 def printout(self, stream, context, path): print("================================================================================") print("Probe %s" % self.printname) print("path is %s, func is %s" % (path, self.func)) if self.show_stream and stream is not None: fallback = stream.tell() datafollows = stream.read(self.stream_lookahead) stream.seek(fallback) if not datafollows: print("EOF reached") else: print(hexdump(datafollows, 32)) if self.show_context: if self.func: try: context = self.func(context) print(context) except Exception as e: print("Failed to compute `%r` on the context" % self.func) else: print(context) # if self.show_stack: # stack = ListContainer() # print("Stack: ") # frames = [s[0] for s in inspect.stack()][1:-1] # for f in reversed(frames): # a = Container() # a.__update__(f.f_locals) # stack.append(a) # # print(f.f_locals) # print(stack) print("================================================================================") def ProbeInto(func): r""" ProbeInto looks inside the context and extracts a part of it using a lambda instead of printing the entire context. Example:: >>> st = "junk"/RepeatUntil(obj_ == 0,Byte) + "num"/Byte + Probe() >>> st.parse(b"xcnzxmbjskahuiwerhquiehnsdjk\x00\xff") ================================================================================ Probe path is parsing EOF reached Container: junk = ListContainer: 120 99 110 122 120 109 98 106 115 107 97 104 117 105 119 101 114 104 113 117 105 101 104 110 115 100 106 107 0 num = 255 ================================================================================ Container(junk=[120, 99, 110, 122, 120, 109, 98, 106, 115, 107, 97, 104, 117, 105, 119, 101, 114, 104, 113, 117, 105, 101, 104, 110, 115, 100, 106, 107, 0])(num=255) >>> st = "junk"/RepeatUntil(obj_ == 0,Byte) + "num"/Byte + ProbeInto(this.num) >>> st.parse(b"xcnzxmbjskahuiwerhquiehnsdjk\x00\xff") ================================================================================ Probe path is parsing EOF reached 255 ================================================================================ Container(junk=[120, 99, 110, 122, 120, 109, 98, 106, 115, 107, 97, 104, 117, 105, 119, 101, 114, 104, 113, 117, 105, 101, 104, 110, 115, 100, 106, 107, 0])(num=255) """ return Probe(func=func) class Debugger(Subconstruct): r""" A pdb-based debugger. When an exception occurs in the subcon, a debugger will appear and allow you to debug the error (and even fix it on-the-fly). :param subcon: the subcon to debug Example:: >>> Debugger(Byte[3]).build([]) ================================================================================ Debugging exception of : File "/home/arkadiusz/Dokumenty/GitHub/construct/construct/debug.py", line 116, in _build obj.stack.append(a) File "/home/arkadiusz/Dokumenty/GitHub/construct/construct/core.py", line 1069, in _build raise RangeError("expected from %d to %d elements, found %d" % (self.min, self.max, len(obj))) construct.core.RangeError: expected from 3 to 3 elements, found 0 > /home/arkadiusz/Dokumenty/GitHub/construct/construct/core.py(1069)_build() -> raise RangeError("expected from %d to %d elements, found %d" % (self.min, self.max, len(obj))) (Pdb) ================================================================================ >>> format = Struct( ... "spam" / Debugger(Enum(Byte, A=1, B=2, C=3)), ... ) >>> format.parse(b"\xff") ================================================================================ Debugging exception of : File "/home/arkadiusz/Dokumenty/GitHub/construct/construct/core.py", line 2578, in _decode return self.decoding[obj] KeyError: 255 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/arkadiusz/Dokumenty/GitHub/construct/construct/debug.py", line 127, in _parse return self.subcon._parse(stream, context) File "/home/arkadiusz/Dokumenty/GitHub/construct/construct/core.py", line 308, in _parse return self._decode(self.subcon._parse(stream, context), context) File "/home/arkadiusz/Dokumenty/GitHub/construct/construct/core.py", line 2583, in _decode raise MappingError("no decoding mapping for %r" % (obj,)) construct.core.MappingError: no decoding mapping for 255 (you can set the value of 'self.retval', which will be returned) > /home/arkadiusz/Dokumenty/GitHub/construct/construct/core.py(2583)_decode() -> raise MappingError("no decoding mapping for %r" % (obj,)) (Pdb) self.retval = "???" (Pdb) q """ __slots__ = ["retval"] def _parse(self, stream, context, path): try: return self.subcon._parse(stream, context, path) except Exception: self.retval = NotImplemented self.handle_exc(path, msg="(you can set the value of 'self.retval', which will be returned)") if self.retval is NotImplemented: raise else: return self.retval def _build(self, obj, stream, context, path): try: self.subcon._build(obj, stream, context, path) except Exception: self.handle_exc(path) def _sizeof(self, context, path): try: self.subcon._sizeof(context, path) except Exception: self.handle_exc(path) def handle_exc(self, path, msg=None): print("================================================================================") print("Debugging exception of %s:" % self.subcon) print("path is %s" % path) print("".join(traceback.format_exception(*sys.exc_info())[1:])) if msg: print(msg) pdb.post_mortem(sys.exc_info()[2]) print("================================================================================") construct-2.8.16/construct/expr.py0000644000175000017500000001410313151124063021217 0ustar arkadiuszarkadiusz00000000000000import operator if not hasattr(operator, "div"): operator.div = operator.truediv opnames = { operator.add : "+", operator.sub : "-", operator.mul : "*", operator.div : "/", operator.floordiv : "//", operator.mod : "%", operator.pow : "**", operator.xor : "^", operator.lshift : "<<", operator.rshift : ">>", operator.and_ : "and", operator.or_ : "or", operator.not_ : "not", operator.neg : "-", operator.pos : "+", operator.contains : "in", operator.gt : ">", operator.ge : ">=", operator.lt : "<", operator.le : "<=", operator.eq : "==", operator.ne : "!=", } class ExprMixin(object): __slots__ = () def __add__(self, other): return BinExpr(operator.add, self, other) def __sub__(self, other): return BinExpr(operator.sub, self, other) def __mul__(self, other): return BinExpr(operator.mul, self, other) def __floordiv__(self, other): return BinExpr(operator.floordiv, self, other) def __truediv__(self, other): return BinExpr(operator.div, self, other) __div__ = __floordiv__ def __mod__(self, other): return BinExpr(operator.mod, self, other) def __pow__(self, other): return BinExpr(operator.pow, self, other) def __xor__(self, other): return BinExpr(operator.xor, self, other) def __rshift__(self, other): return BinExpr(operator.rshift, self, other) def __lshift__(self, other): return BinExpr(operator.rshift, self, other) def __and__(self, other): return BinExpr(operator.and_, self, other) def __or__(self, other): return BinExpr(operator.or_, self, other) def __radd__(self, other): return BinExpr(operator.add, other, self) def __rsub__(self, other): return BinExpr(operator.sub, other, self) def __rmul__(self, other): return BinExpr(operator.mul, other, self) def __rfloordiv__(self, other): return BinExpr(operator.floordiv, other, self) def __rtruediv__(self, other): return BinExpr(operator.div, other, self) __rdiv__ = __rfloordiv__ def __rmod__(self, other): return BinExpr(operator.mod, other, self) def __rpow__(self, other): return BinExpr(operator.pow, other, self) def __rxor__(self, other): return BinExpr(operator.xor, other, self) def __rrshift__(self, other): return BinExpr(operator.rshift, other, self) def __rlshift__(self, other): return BinExpr(operator.rshift, other, self) def __rand__(self, other): return BinExpr(operator.and_, other, self) def __ror__(self, other): return BinExpr(operator.or_, other, self) def __neg__(self): return UniExpr(operator.neg, self) def __pos__(self): return UniExpr(operator.pos, self) def __invert__(self): return UniExpr(operator.not_, self) __inv__ = __invert__ def __contains__(self, other): return BinExpr(operator.contains, self, other) def __gt__(self, other): return BinExpr(operator.gt, self, other) def __ge__(self, other): return BinExpr(operator.ge, self, other) def __lt__(self, other): return BinExpr(operator.lt, self, other) def __le__(self, other): return BinExpr(operator.le, self, other) def __eq__(self, other): return BinExpr(operator.eq, self, other) def __ne__(self, other): return BinExpr(operator.ne, self, other) class UniExpr(ExprMixin): __slots__ = ["op", "operand"] def __init__(self, op, operand): self.op = op self.operand = operand def __repr__(self): return "%s %r" % (opnames[self.op], self.operand) def __call__(self, objorcontext, *args): operand = self.operand(objorcontext) if callable(self.operand) else self.operand return self.op(operand) class BinExpr(ExprMixin): __slots__ = ["op", "lhs", "rhs"] def __init__(self, op, lhs, rhs): self.op = op self.lhs = lhs self.rhs = rhs def __repr__(self): return "(%r %s %r)" % (self.lhs, opnames[self.op], self.rhs) def __call__(self, objorcontext, *args): lhs = self.lhs(objorcontext) if callable(self.lhs) else self.lhs rhs = self.rhs(objorcontext) if callable(self.rhs) else self.rhs return self.op(lhs, rhs) class Path(ExprMixin): __slots__ = ["__name", "__parent"] def __init__(self, name, parent=None): self.__name = name self.__parent = parent def __repr__(self): if self.__parent is None: return self.__name return "%r.%s" % (self.__parent, self.__name) def __call__(self, context, *args): if self.__parent is None: return context context2 = self.__parent(context) return context2[self.__name] def __getattr__(self, name): return Path(name, self) def __getitem__(self, name): return Path(name, self) this = Path("this") class FuncExpr(ExprMixin): def __init__(self, func, operand): self.func = func self.operand = operand def __repr__(self): return "%s_(%r)" % (self.func.__name__, self.operand) def __call__(self, context, *args): operand = self.operand(context) if callable(self.operand) else self.operand return self.func(operand) class PathFunc(ExprMixin): def __init__(self, func): self.func = func def __repr__(self): return "%s_" % (self.func.__name__) def __call__(self, operand, *args): return FuncExpr(self.func, operand) if callable(operand) else operand len_ = PathFunc(len) sum_ = PathFunc(sum) min_ = PathFunc(min) max_ = PathFunc(max) abs_ = PathFunc(abs) class Path2(ExprMixin): def __init__(self, name=None, parent=None): self.__name = name self.__parent = parent def __repr__(self): return "obj_" def __call__(self, obj, *args): if self.__parent is None: return obj obj2 = self.__parent(obj) return obj2[self.__name] def __getattr__(self, name): return Path2(name, self) obj_ = Path2() construct-2.8.16/construct/lib/0000755000175000017500000000000013164134612020443 5ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/lib/binary.py0000644000175000017500000000755113151362557022320 0ustar arkadiuszarkadiusz00000000000000from construct.lib.py3compat import * def integer2bits(number, width): r""" Converts an integer into its binary representation in a b-string. Width is the amount of bits to generate. If width is larger than the actual amount of bits required to represent number in binary, sign-extension is used. If it's smaller, the representation is trimmed to width bits. Each bit is represented as either b'\x00' or b'\x01'. The most significant is first, big-endian. This is reverse to `bits2integer`. Examples: >>> integer2bits(19, 8) b'\x00\x00\x00\x01\x00\x00\x01\x01' """ if width < 1: raise ValueError("width must be positive") number = int(number) if number < 0: number += 1 << width bits = [b"\x00"] * width i = width - 1 while number and i >= 0: bits[i] = int2byte(number & 1) number >>= 1 i -= 1 return b"".join(bits) def integer2bytes(number, width): r""" Converts a b-string into an integer. This is reverse to `bytes2integer`. Examples: >>> integer2bytes(19,4) '\x00\x00\x00\x13' """ if width < 1: raise ValueError("width must be positive") number = int(number) if number < 0: number += 1 << (width * 8) acc = [b"\x00"] * width i = width - 1 while number > 0: acc[i] = int2byte(number & 255) number >>= 8 i -= 1 return b"".join(acc) def onebit2integer(b): if b in (b"0", b"\x00"): return 0 if b in (b"1", b"\x01"): return 1 raise ValueError(r"bit was not recognized as one of: 0 1 \x00 \x01") def bits2integer(data, signed=False): r""" Converts a b-string into an integer. Both b'0' and b'\x00' are considered zero, and both b'1' and b'\x01' are considered one. Set sign to interpret the number as a 2-s complement signed integer. This is reverse to `integer2bits`. Examples: >>> bits2integer(b"\x01\x00\x00\x01\x01") 19 >>> bits2integer(b"10011") 19 """ number = 0 for b in iteratebytes(data): number = (number << 1) | onebit2integer(b) if signed and onebit2integer(data[0:1]): bias = 1 << (len(data) -1) return number - bias*2 else: return number def bytes2integer(data, signed=False): r""" Converts a b-string into an integer. This is reverse to `integer2bytes`. Examples: >>> bytes2integer(b'\x00\x00\x00\x13') 19 """ number = 0 for b in iterateints(data): number = (number << 8) | b if signed and byte2int(bytes2bits(data[0:1])[0:1]): bias = 1 << (len(data)*8 -1) return number - bias*2 else: return number def bytes2bits(data): r""" Converts between bit and byte representations in b-strings. Example: >>> bytes2bits(b'ab') b"\x00\x01\x01\x00\x00\x00\x00\x01\x00\x01\x01\x00\x00\x00\x01\x00" """ return b"".join(integer2bits(c,8) for c in iterateints(data)) def bits2bytes(data): r""" Converts between bit and byte representations in b-strings. Example: >>> bits2bytes(b"\x00\x01\x01\x00\x00\x00\x00\x01\x00\x01\x01\x00\x00\x00\x01\x00") b'ab' """ if len(data) & 7: raise ValueError("data length must be a multiple of 8") return b"".join(int2byte(bits2integer(data[i:i+8])) for i in range(0,len(data),8)) def swapbytes(data, linesize=8): r""" Performs an endianness swap on a b-string. Example: >>> swapbytes(b'00011011', 2) b'11100100' >>> swapbytes(b'0000000011111111', 8) b'1111111100000000' """ if len(data) % linesize: raise ValueError("data length must be multiple of linesize") if linesize < 1: raise ValueError("linesize must be a positive number") return b"".join(data[i:i+linesize] for i in reversed(range(0,len(data),linesize))) construct-2.8.16/construct/lib/hex.py0000644000175000017500000000450113151362001021570 0ustar arkadiuszarkadiusz00000000000000from construct.lib.py3compat import byte2int, int2byte, bytes2str, iteratebytes, iterateints # Map an integer in the inclusive range 0-255 to its string byte representation _printable = [bytes2str(int2byte(i)) if 32 <= i < 128 else '.' for i in range(256)] _hexprint = [format(i, '02X') for i in range(256)] def hexdump(data, linesize): r""" Turns bytes into a unicode string of the format: >>>print(hexdump(b'0' * 100, 16)) 0000 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 0010 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 0020 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 0030 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 0040 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 0050 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 0060 30 30 30 30 0000 """ if len(data) < 16**4: fmt = "%%04X %%-%ds %%s" % (3 * linesize - 1,) elif len(data) < 16**8: fmt = "%%08X %%-%ds %%s" % (3 * linesize - 1,) else: raise ValueError("hexdump cannot process more than 16**8 or 4294967296 bytes") prettylines = [] for i in range(0, len(data), linesize): line = data[i:i+linesize] hextext = " ".join(_hexprint[b] for b in iterateints(line)) rawtext = "".join(_printable[b] for b in iterateints(line)) prettylines.append(fmt % (i, str(hextext), str(rawtext))) prettylines.append("") return "\n".join(prettylines) def hexundump(data, linesize): r""" Reverse of ``hexdump()``. """ raw = [] for line in data.split("\n"): line = line[line.find(" "):].lstrip() bytes = [int2byte(int(s,16)) for s in line[:3*linesize].split()] raw.extend(bytes) return b"".join(raw) class HexString(bytes): r""" Represents bytes that will be hex-dumped when parsing, and un-dumped when building. See hexdump(). """ def __init__(self, data, linesize=16): self.linesize = linesize def __new__(cls, data, *args, **kwargs): return bytes.__new__(cls, data) def __str__(self): if not self: return "''" return "\n" + "\n".join(hexdump(self, self.linesize)) construct-2.8.16/construct/lib/container.py0000644000175000017500000003311513151362425023003 0ustar arkadiuszarkadiusz00000000000000from construct.lib.py3compat import * globalfullprinting = None def setglobalfullprinting(enabled): r""" Sets full printing for all Container instances. When enabled, Container str produces full content of bytes and strings, otherwise and by default, it produces truncated output. :param enabled: bool to enable or disable full printing, or None to default """ global globalfullprinting globalfullprinting = enabled def getglobalfullprinting(): """Used internally.""" return bool(globalfullprinting) def recursion_lock(retval="", lock_name="__recursion_lock__"): """Used internally.""" def decorator(func): def wrapper(self, *args, **kw): if getattr(self, lock_name, False): return retval setattr(self, lock_name, True) try: return func(self, *args, **kw) finally: delattr(self, lock_name) wrapper.__name__ = func.__name__ return wrapper return decorator class Container(dict): r""" Generic ordered dictionary that allows both key and attribute access, and preserve key order by insertion. Also it uses __call__ method to chain add keys, because **kw does not preserve order. Struct and Sequence, and few others parsers returns a container, since their members have order so do keys. Example:: Container([ ("name","anonymous"), ("age",21) ]) Container(name="anonymous")(age=21) # Note that this syntax does NOT work before python 3.6 due to unordered keyword arguments: Container(name="anonymous", age=21) Container(container2) """ __slots__ = ["__keys_order__", "__recursion_lock__"] def __init__(self, *args, **kw): object.__setattr__(self, "__keys_order__", []) if isinstance(args, dict): for k, v in args.items(): self[k] = v return for arg in args: if isinstance(arg, dict): for k, v in arg.items(): self[k] = v else: for k, v in arg: self[k] = v for k, v in kw.items(): self[k] = v def __getstate__(self): return self.__keys_order__ def __setstate__(self, state): self.__keys_order__ = state def __getattr__(self, name): try: if name in self.__slots__: try: return object.__getattribute__(self, name) except AttributeError as e: if name == "__keys_order__": object.__setattr__(self, "__keys_order__", []) return [] else: raise e else: return self[name] except KeyError: raise AttributeError(name) def __setitem__(self, key, val): if key in self.__slots__: object.__setattr__(self, key, val) else: if key not in self: if not hasattr(self, "__keys_order__"): object.__setattr__(self, "__keys_order__", [key]) else: self.__keys_order__.append(key) dict.__setitem__(self, key, val) def __delitem__(self, key): """Removes an item from the Container in linear time O(n).""" if key in self.__slots__: object.__delattr__(self, key) else: dict.__delitem__(self, key) self.__keys_order__.remove(key) __delattr__ = __delitem__ __setattr__ = __setitem__ def __call__(self, **kw): """Chains adding new entries to the same container. See ctor.""" for k,v in kw.items(): self.__setitem__(k, v) return self def clear(self): dict.clear(self) del self.__keys_order__[:] def pop(self, key, *default): """Removes and returns the value for a given key, raises KeyError if not found.""" val = dict.pop(self, key, *default) self.__keys_order__.remove(key) return val def popitem(self): """Removes and returns the last key and value from order.""" k = self.__keys_order__.pop() v = dict.pop(self, k) return k, v def update(self, seqordict, **kw): if isinstance(seqordict, dict): for k, v in seqordict.items(): self[k] = v else: for k, v in seqordict: self[k] = v dict.update(self, kw) def copy(self): return Container(self.items()) __update__ = update __copy__ = copy def __len__(self): return len(self.__keys_order__) def keys(self): return iter(self.__keys_order__) def values(self): return (self[k] for k in self.__keys_order__) def items(self): return ((k, self[k]) for k in self.__keys_order__) __iter__ = keys def __dir__(self): """For auto completion of attributes based on container values.""" return list(self.keys()) + list(self.__class__.__dict__) + dir(super(Container, self)) def __eq__(self, other): if not isinstance(other, dict): return False if len(self) != len(other): return False for k,v in self.items(): if k not in other or v != other[k]: return False for k,v in other.items(): if k not in self or v != self[k]: return False return True def _search(self, compiled_pattern, search_all): items = [] for key in self.keys(): try: if type(self[key]) == Container or type(self[key]) == ListContainer: ret = self[key]._search(compiled_pattern, search_all) if ret is not None: if search_all: items.extend(ret) else: return ret elif compiled_pattern.match(key): if search_all: items.append(self[key]) else: return self[key] except: pass if search_all: return items else: return None def search(self, pattern): import re compiled_pattern = re.compile(pattern) return self._search(compiled_pattern, False) def search_all(self, pattern): import re compiled_pattern = re.compile(pattern) return self._search(compiled_pattern, True) @recursion_lock() def __repr__(self): parts = ["Container"] for k,v in self.items(): if not isinstance(k,str) or not k.startswith("_"): parts.extend(["(",str(k),"=",repr(v),")"]) if len(parts) == 1: parts.append("()") return "".join(parts) @recursion_lock() def __str__(self, indentation="\n "): fullprinting = getglobalfullprinting() printingcap = 64 text = ["Container: "] for k,v in self.items(): if not isinstance(k,str) or not k.startswith("_"): text.extend([indentation, str(k), " = "]) if isinstance(v, stringtypes): if len(v) <= printingcap or fullprinting: text.append("%s (total %d)" % (reprbytes(v), len(v))) else: text.append("%s... (truncated, total %d)" % (reprbytes(v[:printingcap]), len(v))) else: text.append(indentation.join(str(v).split("\n"))) return "".join(text) class FlagsContainer(Container): r""" Container made to represent a FlagsEnum, only equality skips order. Provides pretty-printing for flags. Only set flags are displayed. """ @recursion_lock() def __str__(self, indentation="\n "): text = ["FlagsContainer: "] for k,v in self.items(): if not k.startswith("_") and v: text.extend([indentation, k, " = "]) lines = str(v).split("\n") text.append(indentation.join(lines)) return "".join(text) class ListContainer(list): r""" A generic container for lists. Provides pretty-printing. """ @recursion_lock() def __str__(self, indentation="\n "): text = ["ListContainer: "] for k in self: text.extend([indentation]) lines = str(k).split("\n") text.append(indentation.join(lines)) return "".join(text) def _search(self, compiled_pattern, search_all): items = [] for item in self: try: ret = item._search(compiled_pattern, search_all) except: continue if ret is not None: if search_all: items.extend(ret) else: return ret if search_all: return items else: return None def search(self, pattern): import re compiled_pattern = re.compile(pattern) return self._search(compiled_pattern, False) def search_all(self, pattern): import re compiled_pattern = re.compile(pattern) return self._search(compiled_pattern, True) class LazyContainer(object): r""" Lazy equivalent to Container. Works the same but parses subcons on first access whenever possible. """ __slots__ = ["keysbackend", "offsetmap", "cached", "stream", "addoffset", "context"] def __init__(self, keysbackend, offsetmap, cached, stream, addoffset, context): self.keysbackend = keysbackend self.offsetmap = offsetmap self.cached = cached self.stream = stream self.addoffset = addoffset self.context = context def __getitem__(self, key): if key not in self.cached: at, sc = self.offsetmap[key] self.stream.seek(self.addoffset + at) self.cached[key] = sc._parse(self.stream, self.context, "lazy container") if len(self.cached) == len(self): self.offsetmap = None self.stream = None return self.cached[key] def __getattr__(self, name): try: return self[name] except KeyError: raise AttributeError(name) def __len__(self): return len(self.keysbackend) def keys(self): return iter(self.keysbackend) def values(self): return (self[name] for name in self.keysbackend) def items(self): return ((name,self[name]) for name in self.keysbackend) __iter__ = keys def __eq__(self, other): if not isinstance(other, dict): return False if len(self) != len(other): return False for k,v in self.items(): if k not in other or v != other[k]: return False for k,v in other.items(): if k not in self.keysbackend or v != self[k]: return False return True def __str__(self): return "" % (len(self),len(self.cached)) class LazyRangeContainer(ListContainer): r""" Lazy equivalent to ListContainer. Works the same but parses subcons on first access whenever possible. """ __slots__ = ["subcon", "subsize", "count", "stream", "addoffset", "context", "cached", "offsetmap"] def __init__(self, subcon, subsize, count, stream, addoffset, context): self.subcon = subcon self.subsize = subsize self.count = count self.stream = stream self.addoffset = addoffset self.context = context self.cached = {} def __getitem__(self, index): if not 0 <= index < len(self): raise ValueError("index %d out of range 0-%d" % (index,len(self)-1)) if index not in self.cached: self.stream.seek(self.addoffset + index * self.subsize) self.cached[index] = self.subcon._parse(self.stream, self.context, "lazy range container") if len(self.cached) == len(self): self.stream = None return self.cached[index] def __len__(self): return self.count def __iter__(self): return (self[i] for i in range(len(self))) def __eq__(self, other): return len(self)==len(other) and all(a==b for a,b in zip(self,other)) def __repr__(self): return "<%s: %d possible items, %d cached>" % (self.__class__.__name__, len(self), len(self.cached)) class LazySequenceContainer(LazyRangeContainer): r""" Lazy equivalent to ListContainer. Works the same but parses subcons on first access whenever possible. """ __slots__ = ["count", "offsetmap", "cached", "stream", "addoffset", "context"] def __init__(self, count, offsetmap, cached, stream, addoffset, context): self.count = count self.offsetmap = offsetmap self.cached = cached self.stream = stream self.addoffset = addoffset self.context = context def __getitem__(self, index): if not 0 <= index < len(self): raise ValueError("index %d out of range 0-%d" % (index,len(self)-1)) if index not in self.cached: at,sc = self.offsetmap[index] self.stream.seek(self.addoffset + at) self.cached[index] = sc._parse(self.stream, self.context, "lazy sequence container") if len(self.cached) == len(self): self.offsetmap = None self.stream = None return self.cached[index] def __len__(self): return self.count construct-2.8.16/construct/lib/py3compat.py0000644000175000017500000000557213151361771022751 0ustar arkadiuszarkadiusz00000000000000import sys PY = sys.version_info[:2] PY2 = sys.version_info[0] == 2 PY3 = sys.version_info[0] == 3 PY27 = sys.version_info[:2] == (2,7) PY32 = sys.version_info[:2] == (3,2) PY33 = sys.version_info[:2] == (3,3) PY34 = sys.version_info[:2] == (3,4) PY35 = sys.version_info[:2] == (3,5) PY36 = sys.version_info[:2] == (3,6) PY37 = sys.version_info[:2] == (3,7) PYPY = '__pypy__' in sys.builtin_module_names supportskwordered = PY >= (3,6) or PYPY if PY3: stringtypes = (bytes, str, ) def int2byte(i): """Converts int (0 through 255) into b'...' character.""" return bytes((i,)) def byte2int(b): """Converts b'...' character into int (0 through 255).""" return ord(b) def str2bytes(s): """Converts '...' str into b'...' bytes. On PY2 they are equivalent.""" return s.encode("utf8") def bytes2str(b): """Converts b'...' bytes into str. On PY2 they are equivalent.""" return b.decode("utf8") def str2unicode(s): """Converts '...' str into u'...' unicode string. On PY3 they are equivalent.""" return s def unicode2str(s): """Converts u'...' string into '...' str. On PY3 they are equivalent.""" return s def iteratebytes(s): """Iterates though b'...' string yielding characters as b'...' characters. On PY2 iter is the same.""" return map(int2byte, s) def iterateints(s): """Iterates though b'...' string yielding characters as ints. On PY3 iter is the same.""" return s def reprbytes(b): if isinstance(b, bytes): return repr(b)[2:-1] if isinstance(b, str): return repr(b)[1:-1] else: stringtypes = (str, unicode, ) def int2byte(i): """Converts int (0 through 255) into b'...' character.""" return chr(i) def byte2int(s): """Converts b'...' character into int (0 through 255).""" return ord(s) def str2bytes(s): """Converts '...' str into b'...' bytes. On PY2 they are equivalent.""" return s def bytes2str(b): """Converts b'...' bytes into str. On PY2 they are equivalent.""" return b def str2unicode(b): """Converts '...' str into u'...' unicode string. On PY3 they are equivalent.""" return b.encode("utf8") def unicode2str(s): """Converts u'...' string into '...' str. On PY3 they are equivalent.""" return s.decode("utf8") def iteratebytes(s): """Iterates though b'...' string yielding characters as b'...' characters. On PY2 iter is the same.""" return s def iterateints(s): """Iterates though b'...' string yielding characters as ints. On PY3 iter is the same.""" return map(byte2int, s) def reprbytes(b): if isinstance(b, str): return repr(b)[1:-1] if isinstance(b, unicode): return repr(b)[2:-1] construct-2.8.16/construct/lib/__init__.py0000644000175000017500000000243613151124063022554 0ustar arkadiuszarkadiusz00000000000000from construct.lib.container import Container, FlagsContainer, ListContainer, LazyContainer, LazyRangeContainer, LazySequenceContainer, setglobalfullprinting, getglobalfullprinting from construct.lib.binary import integer2bits, integer2bytes, onebit2integer, bits2integer, bytes2integer, bytes2bits, bits2bytes, swapbytes from construct.lib.bitstream import RestreamedBytesIO, RebufferedBytesIO from construct.lib.hex import HexString, hexdump, hexundump from construct.lib.py3compat import PY, PY2, PY3, PY27, PY32,PY33, PY34, PY35, PY36, PY37, PYPY, supportskwordered, stringtypes, int2byte, byte2int, str2bytes, bytes2str, str2unicode, unicode2str, iteratebytes, iterateints __all__ = [ 'Container', 'FlagsContainer', 'ListContainer', 'LazyContainer', 'LazyRangeContainer', 'LazySequenceContainer', 'integer2bits', 'integer2bytes', 'onebit2integer', 'bits2integer', 'bytes2integer', 'bytes2bits', 'bits2bytes', 'swapbytes', 'RestreamedBytesIO', 'RebufferedBytesIO', 'HexString', 'hexdump', 'hexundump', 'PY','PY2', 'PY3', 'PY27', 'PY32','PY33', 'PY34', 'PY35','PY36', 'PY37','PYPY', 'supportskwordered','stringtypes', 'int2byte', 'byte2int', 'str2bytes', 'bytes2str', 'str2unicode', 'unicode2str', 'iteratebytes', 'iterateints', 'setglobalfullprinting','getglobalfullprinting', ] construct-2.8.16/construct/lib/bitstream.py0000644000175000017500000001110213164127405023004 0ustar arkadiuszarkadiusz00000000000000from io import BlockingIOError from time import sleep from sys import maxsize class RestreamedBytesIO(object): __slots__ = ["substream", "encoder", "encoderunit", "decoder", "decoderunit", "rbuffer", "wbuffer","sincereadwritten"] def __init__(self, substream, encoder, encoderunit, decoder, decoderunit): self.substream = substream self.encoder = encoder self.encoderunit = encoderunit self.decoder = decoder self.decoderunit = decoderunit self.rbuffer = b"" self.wbuffer = b"" self.sincereadwritten = 0 def read(self, count): if count < 0: raise ValueError("count cannot be negative") while len(self.rbuffer) < count: data = self.substream.read(self.decoderunit) if data is None or len(data) == 0: raise IOError("Restreamed cannot satisfy read request of %d bytes" % (count,)) self.rbuffer += self.decoder(data) data, self.rbuffer = self.rbuffer[:count], self.rbuffer[count:] self.sincereadwritten += count return data def write(self, data): self.wbuffer += data datalen = len(data) while len(self.wbuffer) >= self.encoderunit: data, self.wbuffer = self.wbuffer[:self.encoderunit], self.wbuffer[self.encoderunit:] self.substream.write(self.encoder(data)) self.sincereadwritten += datalen return datalen def close(self): if len(self.rbuffer): raise ValueError("closing stream but %d unread bytes remain, %d is decoded unit" % (len(self.rbuffer), self.decoderunit)) if len(self.wbuffer): raise ValueError("closing stream but %d unwritten bytes remain, %d is encoded unit" % (len(self.wbuffer), self.encoderunit)) def seekable(self): return False def tell(self): """WARNING: tell is correct only on read-only and write-only instances.""" return self.sincereadwritten def tellable(self): return True class RebufferedBytesIO(object): __slots__ = ["substream","offset","rwbuffer","moved","tailcutoff"] def __init__(self, substream, tailcutoff=None): self.substream = substream self.offset = 0 self.rwbuffer = b"" self.moved = 0 self.tailcutoff = tailcutoff def read(self, count=None): if count is None: raise ValueError("count must be an int, reading until EOF not supported") startsat = self.offset endsat = startsat + count if startsat < self.moved: raise IOError("could not read because tail was cut off") while self.moved + len(self.rwbuffer) < endsat: try: newdata = self.substream.read(128*1024) except BlockingIOError: newdata = None if not newdata: sleep(0) continue self.rwbuffer += newdata data = self.rwbuffer[startsat-self.moved:endsat-self.moved] self.offset += count if self.tailcutoff is not None and self.moved < self.offset - self.tailcutoff: removed = self.offset - self.tailcutoff - self.moved self.moved += removed self.rwbuffer = self.rwbuffer[removed:] if len(data) < count: raise IOError("could not read enough bytes, something went wrong") return data def write(self, data): startsat = self.offset endsat = startsat + len(data) while self.moved + len(self.rwbuffer) < startsat: newdata = self.substream.read(128*1024) self.rwbuffer += newdata if not newdata: sleep(0) self.rwbuffer = self.rwbuffer[:startsat-self.moved] + data + self.rwbuffer[endsat-self.moved:] self.offset = endsat if self.tailcutoff is not None and self.moved < self.offset - self.tailcutoff: removed = self.offset - self.tailcutoff - self.moved self.moved += removed self.rwbuffer = self.rwbuffer[removed:] return len(data) def seek(self, at, whence=0): if whence == 0: self.offset = at return self.offset elif whence == 1: self.offset += at return self.offset else: raise ValueError("seeks only with whence 0 and 1") def seekable(self): return True def tell(self): return self.offset def tellable(self): return True def cachedfrom(self): return self.moved def cachedto(self): return self.moved + len(self.rwbuffer) construct-2.8.16/construct/examples/0000755000175000017500000000000013164134612021513 5ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/examples/protocols/0000755000175000017500000000000013164134612023537 5ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/examples/protocols/__init__.py0000644000175000017500000000000013151124063025631 0ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/examples/protocols/ipstack.py0000644000175000017500000005433713151124063025556 0ustar arkadiuszarkadiusz00000000000000""" TCP/IP Protocol Stack WARNING: before parsing the application layer over a TCP stream, you must first combine all the TCP frames into a stream. See utils.tcpip for some solutions. """ from construct import * from construct.lib import * #=============================================================================== # layer 2, Ethernet #=============================================================================== MacAddress = ExprAdapter(Byte[6], encoder = lambda obj,ctx: [int(part, 16) for part in obj.split("-")], decoder = lambda obj,ctx: "-".join("%02x" % b for b in obj), ) ethernet_header = "ethernet_header" / Struct( "destination" / MacAddress, "source" / MacAddress, "type" / Enum(Int16ub, IPv4 = 0x0800, ARP = 0x0806, RARP = 0x8035, X25 = 0x0805, IPX = 0x8137, IPv6 = 0x86DD, default = Pass, ), ) #=============================================================================== # layer 2, ARP #=============================================================================== # HwAddress = IfThenElse(this.hardware_type == "ETHERNET", # MacAddressAdapter(Bytes(this.hwaddr_length)), # Bytes(this.hwaddr_length) # ) HwAddress = Bytes(this.hwaddr_length) # ProtoAddress = IfThenElse(this.protocol_type == "IP", # IpAddressAdapter(Bytes(this.protoaddr_length)), # Bytes(this.protoaddr_length) # ) ProtoAddress = Bytes(this.protoaddr_length) arp_header = "arp_header" / Struct( "hardware_type" / Enum(Int16ub, ETHERNET = 1, EXPERIMENTAL_ETHERNET = 2, ProNET_TOKEN_RING = 4, CHAOS = 5, IEEE802 = 6, ARCNET = 7, HYPERCHANNEL = 8, ULTRALINK = 13, FRAME_RELAY = 15, FIBRE_CHANNEL = 18, IEEE1394 = 24, HIPARP = 28, ISO7816_3 = 29, ARPSEC = 30, IPSEC_TUNNEL = 31, INFINIBAND = 32, ), "protocol_type" / Enum(Int16ub, IP = 0x0800, ), "hwaddr_length" / Int8ub, "protoaddr_length" / Int8ub, "opcode" / Enum(Int16ub, REQUEST = 1, REPLY = 2, REQUEST_REVERSE = 3, REPLY_REVERSE = 4, DRARP_REQUEST = 5, DRARP_REPLY = 6, DRARP_ERROR = 7, InARP_REQUEST = 8, InARP_REPLY = 9, ARP_NAK = 10 ), "source_hwaddr" / HwAddress, "source_protoaddr" / ProtoAddress, "dest_hwaddr" / HwAddress, "dest_protoaddr" / ProtoAddress, ) #=============================================================================== # layer 2, Message Transport Part 2 (SS7 protocol stack) # (untested) #=============================================================================== mtp2_header = "mtp2_header" / BitStruct( "flag1" / Octet, "bsn" / BitsInteger(7), "bib" / Bit, "fsn" / BitsInteger(7), "sib" / Bit, "length" / Octet, "service_info" / Octet, "signalling_info" / Octet, "crc" / BitsInteger(16), "flag2" / Octet, ) #=============================================================================== # layer 3, IP v4 #=============================================================================== IpAddress = ExprAdapter(Byte[4], encoder = lambda obj,ctx: list(map(int, obj.split("."))), decoder = lambda obj,ctx: "{0}.{1}.{2}.{3}".format(*obj), ) def ProtocolEnum(code): return Enum(code, ICMP = 1, TCP = 6, UDP = 17, ) ipv4_header = "ip_header" / Struct( EmbeddedBitStruct( "version" / Const(Nibble, 4), "header_length" / ExprAdapter(Nibble, decoder = lambda obj, ctx: obj * 4, encoder = lambda obj, ctx: obj / 4 ), ), "tos" / BitStruct( "precedence" / BitsInteger(3), "minimize_delay" / Flag, "high_throuput" / Flag, "high_reliability" / Flag, "minimize_cost" / Flag, Padding(1), ), "total_length" / Int16ub, "payload_length" / Computed(this.total_length - this.header_length), "identification" / Int16ub, EmbeddedBitStruct( "flags" / Struct( Padding(1), "dont_fragment" / Flag, "more_fragments" / Flag, ), "frame_offset" / BitsInteger(13), ), "ttl" / Int8ub, "protocol" / ProtocolEnum(Int8ub), "checksum" / Int16ub, "source" / IpAddress, "destination" / IpAddress, "options" / Bytes(this.header_length - 20), ) #=============================================================================== # layer 3, IP v6 #=============================================================================== def ProtocolEnum(code): return Enum(code, ICMP = 1, TCP = 6, UDP = 17, ) Ipv6Address = ExprAdapter(Byte[16], encoder = lambda obj,ctx: [int(part, 16) for part in obj.split(":")], decoder = lambda obj,ctx: ":".join("%02x" % b for b in obj), ) ipv6_header = "ip_header" / Struct( EmbeddedBitStruct( "version" / OneOf(BitsInteger(4), [6]), "traffic_class" / BitsInteger(8), "flow_label" / BitsInteger(20), ), "payload_length" / Int16ub, "protocol" / ProtocolEnum(Int8ub), "hoplimit" / Int8ub, "ttl" / Computed(this.hoplimit), "source" / Ipv6Address, "destination" / Ipv6Address, ) #=============================================================================== # layer 3 # Message Transport Part 3 (SS7 protocol stack) # (untested) #=============================================================================== mtp3_header = "mtp3_header" / BitStruct( "service_indicator" / Nibble, "subservice" / Nibble, ) #=============================================================================== # layer 3 # Internet Control Message Protocol for IPv4 #=============================================================================== echo_payload = "echo_payload" / Struct( "identifier" / Int16ub, "sequence" / Int16ub, "data" / Bytes(32), # length is implementation dependent, is anyone using more than 32 bytes? ) dest_unreachable_payload = "dest_unreachable_payload" / Struct( Padding(2), "next_hop_mtu" / Int16ub, "host" / IpAddress, "echo" / Bytes(8), ) dest_unreachable_code = "code" / Enum(Byte, Network_unreachable_error = 0, Host_unreachable_error = 1, Protocol_unreachable_error = 2, Port_unreachable_error = 3, The_datagram_is_too_big = 4, Source_route_failed_error = 5, Destination_network_unknown_error = 6, Destination_host_unknown_error = 7, Source_host_isolated_error = 8, Desination_administratively_prohibited = 9, Host_administratively_prohibited2 = 10, Network_TOS_unreachable = 11, Host_TOS_unreachable = 12, ) icmp_header = "icmp_header" / Struct( "type" / Enum(Byte, Echo_reply = 0, Destination_unreachable = 3, Source_quench = 4, Redirect = 5, Alternate_host_address = 6, Echo_request = 8, Router_advertisement = 9, Router_solicitation = 10, Time_exceeded = 11, Parameter_problem = 12, Timestamp_request = 13, Timestamp_reply = 14, Information_request = 15, Information_reply = 16, Address_mask_request = 17, Address_mask_reply = 18, default = Pass, ), "code" / Switch(this.type, { "Destination_unreachable" : dest_unreachable_code, }, default = Byte ), "crc" / Int16ub, "payload" / Switch(this.type, { "Echo_reply" : echo_payload, "Echo_request" : echo_payload, "Destination_unreachable" : dest_unreachable_payload, }, default = Pass ) ) #=============================================================================== # layer 3 # Internet Group Management Protocol, Version 2 # # http://www.ietf.org/rfc/rfc2236.txt # jesse@housejunkie.ca #=============================================================================== igmp_type = "igmp_type" / Enum(Byte, MEMBERSHIP_QUERY = 0x11, MEMBERSHIP_REPORT_V1 = 0x12, MEMBERSHIP_REPORT_V2 = 0x16, LEAVE_GROUP = 0x17, ) igmpv2_header = "igmpv2_header" / Struct( igmp_type, "max_resp_time" / Byte, "checksum" / Int16ub, "group_address" / IpAddress, ) #=============================================================================== # layer 4 # Dynamic Host Configuration Protocol for IPv4 # # http://www.networksorcery.com/enp/protocol/dhcp.htm # http://www.networksorcery.com/enp/protocol/bootp/options.htm #=============================================================================== dhcp4_option = "dhcp_option" / Struct( "code" / Enum(Byte, Pad = 0, Subnet_Mask = 1, Time_Offset = 2, Router = 3, Time_Server = 4, Name_Server = 5, Domain_Name_Server = 6, Log_Server = 7, Quote_Server = 8, LPR_Server = 9, Impress_Server = 10, Resource_Location_Server = 11, Host_Name = 12, Boot_File_Size = 13, Merit_Dump_File = 14, Domain_Name = 15, Swap_Server = 16, Root_Path = 17, Extensions_Path = 18, IP_Forwarding_enabledisable = 19, Nonlocal_Source_Routing_enabledisable = 20, Policy_Filter = 21, Maximum_Datagram_Reassembly_Size = 22, Default_IP_TTL = 23, Path_MTU_Aging_Timeout = 24, Path_MTU_Plateau_Table = 25, Interface_MTU = 26, All_Subnets_are_Local = 27, Broadcast_Address = 28, Perform_Mask_Discovery = 29, Mask_supplier = 30, Perform_router_discovery = 31, Router_solicitation_address = 32, Static_routing_table = 33, Trailer_encapsulation = 34, ARP_cache_timeout = 35, Ethernet_encapsulation = 36, Default_TCP_TTL = 37, TCP_keepalive_interval = 38, TCP_keepalive_garbage = 39, Network_Information_Service_domain = 40, Network_Information_Servers = 41, NTP_servers = 42, Vendor_specific_information = 43, NetBIOS_over_TCPIP_name_server = 44, NetBIOS_over_TCPIP_Datagram_Distribution_Server = 45, NetBIOS_over_TCPIP_Node_Type = 46, NetBIOS_over_TCPIP_Scope = 47, X_Window_System_Font_Server = 48, X_Window_System_Display_Manager = 49, Requested_IP_Address = 50, IP_address_lease_time = 51, Option_overload = 52, DHCP_message_type = 53, Server_identifier = 54, Parameter_request_list = 55, Message = 56, Maximum_DHCP_message_size = 57, Renew_time_value = 58, Rebinding_time_value = 59, Class_identifier = 60, Client_identifier = 61, NetWareIP_Domain_Name = 62, NetWareIP_information = 63, Network_Information_Service_Domain = 64, Network_Information_Service_Servers = 65, TFTP_server_name = 66, Bootfile_name = 67, Mobile_IP_Home_Agent = 68, Simple_Mail_Transport_Protocol_Server = 69, Post_Office_Protocol_Server = 70, Network_News_Transport_Protocol_Server = 71, Default_World_Wide_Web_Server = 72, Default_Finger_Server = 73, Default_Internet_Relay_Chat_Server = 74, StreetTalk_Server = 75, StreetTalk_Directory_Assistance_Server = 76, User_Class_Information = 77, SLP_Directory_Agent = 78, SLP_Service_Scope = 79, Rapid_Commit = 80, Fully_Qualified_Domain_Name = 81, Relay_Agent_Information = 82, Internet_Storage_Name_Service = 83, NDS_servers = 85, NDS_tree_name = 86, NDS_context = 87, BCMCS_Controller_Domain_Name_list = 88, BCMCS_Controller_IPv4_address_list = 89, Authentication = 90, Client_last_transaction_time = 91, Associated_ip = 92, Client_System_Architecture_Type = 93, Client_Network_Interface_Identifier = 94, Lightweight_Directory_Access_Protocol = 95, Client_Machine_Identifier = 97, Open_Group_User_Authentication = 98, Autonomous_System_Number = 109, NetInfo_Parent_Server_Address = 112, NetInfo_Parent_Server_Tag = 113, URL = 114, Auto_Configure = 116, Name_Service_Search = 117, Subnet_Selection = 118, DNS_domain_search_list = 119, SIP_Servers_DHCP_Option = 120, Classless_Static_Route_Option = 121, CableLabs_Client_Configuration = 122, GeoConf = 123, ), "value" / If(this.code != "Pad", Prefixed(Byte, GreedyBytes)), ) dhcp4_header = "dhcp_header" / Struct( "opcode" / Enum(Byte, BootRequest = 1, BootReply = 2, ), "hardware_type" / Enum(Byte, Ethernet = 1, Experimental_Ethernet = 2, ProNET_Token_Ring = 4, Chaos = 5, IEEE_802 = 6, ARCNET = 7, Hyperchannel = 8, Lanstar = 9, ), "hardware_address_length" / Byte, "hop_count" / Byte, "transaction_id" / Int32ub, "elapsed_time" / Int16ub, "flags" / BitStruct( "broadcast" / Flag, Padding(15), ), "client_addr" / IpAddress, "your_addr" / IpAddress, "server_addr" / IpAddress, "relay_addr" / IpAddress, "client_hardware_addr" / Hex(Bytes(16)), "server_host_name" / Hex(Bytes(64)), "boot_filename" / Hex(Bytes(128)), # BOOTP/DHCP options # "The first four bytes contain the (decimal) values 99, 130, 83 and 99" "signature" / Const(b"\x63\x82\x53\x63"), "options" / GreedyRange(dhcp4_option), ) #=============================================================================== # layer 4 # Dynamic Host Configuration Protocol for IPv6 # # http://www.networksorcery.com/enp/rfc/rfc3315.txt #=============================================================================== dhcp6_option = "dhcp_option" / Struct( "code" / Enum(Int16ub, OPTION_CLIENTID = 1, OPTION_SERVERID = 2, OPTION_IA_NA = 3, OPTION_IA_TA = 4, OPTION_IAADDR = 5, OPTION_ORO = 6, OPTION_PREFERENCE = 7, OPTION_ELAPSED_TIME = 8, OPTION_RELAY_MSG = 9, OPTION_AUTH = 11, OPTION_UNICAST = 12, OPTION_STATUS_CODE = 13, OPTION_RAPID_COMMIT = 14, OPTION_USER_CLASS = 15, OPTION_VENDOR_CLASS = 16, OPTION_VENDOR_OPTS = 17, OPTION_INTERFACE_ID = 18, OPTION_RECONF_MSG = 19, OPTION_RECONF_ACCEPT = 20, SIP_SERVERS_DOMAIN_NAME_LIST = 21, SIP_SERVERS_IPV6_ADDRESS_LIST = 22, DNS_RECURSIVE_NAME_SERVER = 23, DOMAIN_SEARCH_LIST = 24, OPTION_IA_PD = 25, OPTION_IAPREFIX = 26, OPTION_NIS_SERVERS = 27, OPTION_NISP_SERVERS = 28, OPTION_NIS_DOMAIN_NAME = 29, OPTION_NISP_DOMAIN_NAME = 30, SNTP_SERVER_LIST = 31, INFORMATION_REFRESH_TIME = 32, BCMCS_CONTROLLER_DOMAIN_NAME_LIST = 33, BCMCS_CONTROLLER_IPV6_ADDRESS_LIST = 34, OPTION_GEOCONF_CIVIC = 36, OPTION_REMOTE_ID = 37, RELAY_AGENT_SUBSCRIBER_ID = 38, OPTION_CLIENT_FQDN = 39, ), "data" / Prefixed(Int16ub, GreedyBytes), ) client_message = "client_message" / BitStruct( "transaction_id" / BitsInteger(24), ) relay_message = "relay_message" / Struct( "hop_count" / Byte, "linkaddr" / Ipv6Address, "peeraddr" / Ipv6Address, ) dhcp6_message = "dhcp_message" / Struct( "msgtype" / Enum(Byte, # these are client-server messages SOLICIT = 1, ADVERTISE = 2, REQUEST = 3, CONFIRM = 4, RENEW = 5, REBIND = 6, REPLY = 7, RELEASE_ = 8, DECLINE_ = 9, RECONFIGURE = 10, INFORMATION_REQUEST = 11, # these two are relay messages RELAY_FORW = 12, RELAY_REPL = 13, ), # relay messages have a different structure from client-server messages "params" / Switch(this.msgtype, { "RELAY_FORW" : relay_message, "RELAY_REPL" : relay_message, }, default = client_message, ), "options" / GreedyRange(dhcp6_option), ) #=============================================================================== # layer 4 # ISDN User Part (SS7 protocol stack) #=============================================================================== isup_header = "isup_header" / Struct( "routing_label" / Bytes(5), "cic" / Int16ub, "message_type" / Int8ub, # mandatory fixed parameters # mandatory variable parameters # optional parameters ) #=============================================================================== # layer 4 # Transmission Control Protocol (TCP/IP protocol stack) #=============================================================================== tcp_header = "tcp_header" / Struct( "source" / Int16ub, "destination" / Int16ub, "seq" / Int32ub, "ack" / Int32ub, EmbeddedBitStruct( "header_length" / ExprAdapter(Nibble, encoder = lambda obj, ctx: obj / 4, decoder = lambda obj, ctx: obj * 4, ), Padding(3), "flags" / Struct( "ns" / Flag, "cwr" / Flag, "ece" / Flag, "urg" / Flag, "ack" / Flag, "psh" / Flag, "rst" / Flag, "syn" / Flag, "fin" / Flag, ), ), "window" / Int16ub, "checksum" / Int16ub, "urgent" / Int16ub, "options" / Bytes(this.header_length - 20), ) #=============================================================================== # layer 4 # User Datagram Protocol (TCP/IP protocol stack) #=============================================================================== udp_header = "udp_header" / Struct( "header_length" / Computed(lambda ctx: 8), "source" / Int16ub, "destination" / Int16ub, "payload_length" / ExprAdapter(Int16ub, encoder = lambda obj, ctx: obj + 8, decoder = lambda obj, ctx: obj - 8, ), "checksum" / Int16ub, ) #=============================================================================== # layer 4 # Domain Name System (TCP/IP protocol stack) #=============================================================================== class DnsStringAdapter(Adapter): def _decode(self, obj, context): return ".".join(obj[:-1]) def _encode(self, obj, context): return obj.split(".") + [""] class DnsNamesAdapter(Adapter): def _decode(self, obj, context): return [x.label if x.islabel else x.pointer & 0x3fff for x in obj] def _encode(self, obj, context): return [dict(ispointer=1,pointer=x|0xc000) if isinstance(x,int) else dict(islabel=1,label=x) for x in obj] dns_record_class = "class" / Enum(Int16ub, RESERVED = 0, INTERNET = 1, CHAOS = 3, HESIOD = 4, NONE = 254, ANY = 255, ) dns_record_type = "type" / Enum(Int16ub, IPv4 = 1, AUTHORITIVE_NAME_SERVER = 2, CANONICAL_NAME = 5, NULL = 10, MAIL_EXCHANGE = 15, TEXT = 16, X25 = 19, ISDN = 20, IPv6 = 28, UNSPECIFIED = 103, ALL = 255, ) query_record = "query_record" / Struct( "name" / DnsStringAdapter(RepeatUntil(len_(obj_)==0, PascalString(Byte, encoding="ascii"))), dns_record_type, dns_record_class, ) labelpointer = Struct( "firstbyte" / Peek(Byte), "islabel" / Computed(this.firstbyte & 0b11000000 == 0), "ispointer" / Computed(this.firstbyte & 0b11000000 == 0b11000000), Check(this.islabel | this.ispointer), "label" / If(this.islabel, PascalString(Byte, encoding="ascii")), "pointer" / If(this.ispointer, Int16ub), ) resource_record = "resource_record" / Struct( # http://www.zytrax.com/books/dns/ch15/#qname "names" / DnsNamesAdapter(RepeatUntil(obj_.ispointer | len_(obj_.label)==0, labelpointer)), dns_record_type, dns_record_class, "ttl" / Int32ub, "rdata" / Hex(Prefixed(Int16ub, GreedyBytes)), ) dns = "dns" / Struct( "id" / Int16ub, "flags" / BitStruct( "type" / Enum(Bit, QUERY = 0, RESPONSE = 1, ), "opcode" / Enum(Nibble, STANDARD_QUERY = 0, INVERSE_QUERY = 1, SERVER_STATUS_REQUEST = 2, NOTIFY = 4, UPDATE = 5, ), "authoritive_answer" / Flag, "truncation" / Flag, "recursion_desired" / Flag, "recursion_available" / Flag, Padding(1), "authenticated_data" / Flag, "checking_disabled" / Flag, "response_code" / Enum(Nibble, SUCCESS = 0, FORMAT_ERROR = 1, SERVER_FAILURE = 2, NAME_DOES_NOT_EXIST = 3, NOT_IMPLEMENTED = 4, REFUSED = 5, NAME_SHOULD_NOT_EXIST = 6, RR_SHOULD_NOT_EXIST = 7, RR_SHOULD_EXIST = 8, NOT_AUTHORITIVE = 9, NOT_ZONE = 10, ), ), "question_count" / Rebuild(Int16ub, len_(this.questions)), "answer_count" / Rebuild(Int16ub, len_(this.answers)), "authority_count" / Rebuild(Int16ub, len_(this.authorities)), "additional_count" / Rebuild(Int16ub, len_(this.additionals)), "questions" / query_record[this.question_count], "answers" / resource_record[this.answer_count], "authorities" / resource_record[this.authority_count], "additionals" / resource_record[this.additional_count], ) #=============================================================================== # entire IP stack #=============================================================================== layer4_tcp = "layer4_tcp" / Struct( "header" / tcp_header, "next" / HexDump(Bytes(this._.header.payload_length - this.header.header_length)), ) layer4_udp = "layer4_udp" / Struct( "header" / udp_header, "next" / HexDump(Bytes(this.header.payload_length)), ) layer3_payload = "next" / Switch(this.header.protocol, { "TCP" : layer4_tcp, "UDP" : layer4_udp, "ICMP" : icmp_header, }, default = Pass ) layer3_ipv4 = "layer3_ipv4" / Struct( "header" / ipv4_header, layer3_payload, ) layer3_ipv6 = "layer3_ipv6" / Struct( "header" / ipv6_header, layer3_payload, ) layer2_ethernet = "layer2_ethernet" / Struct( "header" / ethernet_header, "next" / Switch(this.header.type, { "IPv4" : layer3_ipv4, "IPv6" : layer3_ipv6, }, default = Pass, ), ) ip_stack = "ip_stack" / layer2_ethernet construct-2.8.16/construct/examples/formats/0000755000175000017500000000000013164134612023166 5ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/examples/formats/data/0000755000175000017500000000000013164134612024077 5ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/examples/formats/data/cap.py0000644000175000017500000000174113151124063025212 0ustar arkadiuszarkadiusz00000000000000############################################################## # WARNING: HEADER IS SKIPPED NOT PARSED, DATETIME CAN BE WRONG # https://wiki.wireshark.org/Development/LibpcapFileFormat ############################################################## """ tcpdump capture file """ from construct import * import time, datetime class MicrosecAdapter(Adapter): def _decode(self, obj, context): return datetime.datetime.fromtimestamp(obj[0] + obj[1] / 1000000.0) def _encode(self, obj, context): epoch = datetime.datetime.utcfromtimestamp(0) return [int((obj - epoch).total_seconds()), 0] # offset = time.mktime(*obj.timetuple()) # sec = int(offset) # usec = (offset - sec) * 1000000 # return (sec, usec) packet = Struct( "time" / MicrosecAdapter(Int32ul >> Int32ul), "length" / Int32ul, Padding(4), "data" / HexDump(Bytes(this.length)), ) cap_file = Struct( Padding(24), "packets" / GreedyRange(packet), ) construct-2.8.16/construct/examples/formats/data/snoop.py0000644000175000017500000000253213151124063025604 0ustar arkadiuszarkadiusz00000000000000""" what : snoop v2 capture file. how : http://tools.ietf.org/html/rfc1761 who : jesse@housejunkie.ca """ from construct import * import time class EpochTimeStampAdapter(Adapter): """ Convert epoch timestamp <-> localtime """ def _decode(self, obj, context): return time.ctime(obj) def _encode(self, obj, context): return int(time.mktime(time.strptime(obj))) packet_record = "packet_record" / Struct( "original_length" / Int32ub, "included_length" / Int32ub, "record_length" / Int32ub, "cumulative_drops" / Int32ub, "timestamp_seconds" / EpochTimeStampAdapter(Int32ub), "timestamp_microseconds" / Int32ub, "data" / HexDump(Bytes(this.included_length)), # 24 being the static length of the packet_record header Padding(this.record_length - this.included_length - 24), ) datalink_type = "datalink" / Enum(Int32ub, IEEE802dot3 = 0, IEEE802dot4 = 1, IEEE802dot5 = 2, IEEE802dot6 = 3, ETHERNET = 4, HDLC = 5, CHARSYNC = 6, IBMCHANNEL = 7, FDDI = 8, OTHER = 9, UNASSIGNED = 10, ) snoop_file = Struct( "signature" / Const(b"snoop\x00\x00\x00"), "version" / Int32ub, # snoop v1 is deprecated datalink_type, GreedyRange(packet_record), ) construct-2.8.16/construct/examples/formats/data/__init__.py0000644000175000017500000000011213151124063026175 0ustar arkadiuszarkadiusz00000000000000""" all sorts of raw data serialization (tcpdump capture files, etc.) """ construct-2.8.16/construct/examples/formats/executable/0000755000175000017500000000000013164134612025307 5ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/examples/formats/executable/__init__.py0000644000175000017500000000000013151124063027401 0ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/examples/formats/executable/pe32.py0000644000175000017500000002553713151124063026441 0ustar arkadiuszarkadiusz00000000000000""" Portable Executable (PE) 32 bit, little endian Used on MSWindows systems (including DOS) for EXEs and DLLs 1999 paper: http://download.microsoft.com/download/1/6/1/161ba512-40e2-4cc9-843a-923143f3456c/pecoff.doc 2006 with updates relevant for .NET: http://download.microsoft.com/download/9/c/5/9c5b2167-8017-4bae-9fde-d599bac8184a/pecoff_v8.doc """ import time from construct import * class UTCTimeStampAdapter(Adapter): def _decode(self, obj, context): return time.ctime(obj) def _encode(self, obj, context): return int(time.mktime(time.strptime(obj))) UTCTimeStamp = UTCTimeStampAdapter(Int32ul) class NamedSequence(Adapter): """ creates a mapping between the elements of a sequence and their respective names. this is useful for sequences of a variable length, where each element in the sequence has a name (as is the case with the data directories of the PE header) """ __slots__ = ["mapping", "rev_mapping"] prefix = "unnamed_" def __init__(self, subcon, mapping): super(NamedSequence, self).__init__(subcon) self.mapping = mapping self.rev_mapping = dict((v, k) for k, v in mapping.items()) def _encode(self, obj, context): obj2 = [None] * len(obj) for name, value in obj.items(): if name in self.rev_mapping: index = self.rev_mapping[name] elif name.startswith("__"): obj2.pop(-1) continue elif name.startswith(self.prefix): index = int(name.split(self.prefix)[1]) else: raise ValueError("no mapping defined for %r" % (name,)) obj2[index] = value return obj2 def _decode(self, obj, context): obj2 = Container() for i, item in enumerate(obj): if i in self.mapping: name = self.mapping[i] else: name = "%s%d" % (self.prefix, i) setattr(obj2, name, item) return obj2 msdos_header = "msdos_header" / Struct( Const(b"MZ"), "partPag" / Int16ul, "page_count" / Int16ul, "relocation_count" / Int16ul, "header_size" / Int16ul, "minmem" / Int16ul, "maxmem" / Int16ul, "relocation_stackseg" / Int16ul, "exe_stackptr" / Int16ul, "checksum" / Int16ul, "exe_ip" / Int16ul, "relocation_codeseg" / Int16ul, "table_offset" / Int16ul, "overlay" / Int16ul, Padding(8), "oem_id" / Int16ul, "oem_info" / Int16ul, Padding(20), "coff_header_pointer" / Int32ul, "_assembly_start" / Tell, "code" / OnDemand(HexDump(Bytes(this.coff_header_pointer - this._assembly_start))), ) symbol_table = "symbol_table" / Struct( "name" / String(8, padchar=b"\x00"), "value" / Int32ul, "section_number" / Enum(ExprAdapter(Int16sl, encoder = lambda obj,ctx: obj + 1, decoder = lambda obj,ctx: obj - 1, ), UNDEFINED = -1, ABSOLUTE = -2, DEBUG = -3, _default_ = Pass, ), "complex_type" / Enum(Int8ul, NULL = 0, POINTER = 1, FUNCTION = 2, ARRAY = 3, ), "base_type" / Enum(Int8ul, NULL = 0, VOID = 1, CHAR = 2, SHORT = 3, INT = 4, LONG = 5, FLOAT = 6, DOUBLE = 7, STRUCT = 8, UNION = 9, ENUM = 10, MOE = 11, BYTE = 12, WORD = 13, UINT = 14, DWORD = 15, ), "storage_class" / Enum(Int8ul, END_OF_FUNCTION = 255, NULL = 0, AUTOMATIC = 1, EXTERNAL = 2, STATIC = 3, REGISTER = 4, EXTERNAL_DEF = 5, LABEL = 6, UNDEFINED_LABEL = 7, MEMBER_OF_STRUCT = 8, ARGUMENT = 9, STRUCT_TAG = 10, MEMBER_OF_UNION = 11, UNION_TAG = 12, TYPE_DEFINITION = 13, UNDEFINED_STATIC = 14, ENUM_TAG = 15, MEMBER_OF_ENUM = 16, REGISTER_PARAM = 17, BIT_FIELD = 18, BLOCK = 100, FUNCTION = 101, END_OF_STRUCT = 102, FILE = 103, SECTION = 104, WEAK_EXTERNAL = 105, ), "number_of_aux_symbols" / Int8ul, "aux_symbols" / Array(this.number_of_aux_symbols, Bytes(18)) ) coff_header = "coff_header" / Struct( Const(b"PE\x00\x00"), "machine_type" / Enum(Int16ul, UNKNOWN = 0x0, AM33 = 0x1d3, AMD64 = 0x8664, ARM = 0x1c0, EBC = 0xebc, I386 = 0x14c, IA64 = 0x200, M32R = 0x9041, MIPS16 = 0x266, MIPSFPU = 0x366, MIPSFPU16 = 0x466, POWERPC = 0x1f0, POWERPCFP = 0x1f1, R4000 = 0x166, SH3 = 0x1a2, SH3DSP = 0x1a3, SH4 = 0x1a6, SH5= 0x1a8, THUMB = 0x1c2, WCEMIPSV2 = 0x169, _default_ = Pass ), "number_of_sections" / Int16ul, "time_stamp" / UTCTimeStamp, "symbol_table_pointer" / Int32ul, "number_of_symbols" / Int32ul, "optional_header_size" / Int16ul, "characteristics" / FlagsEnum(Int16ul, RELOCS_STRIPPED = 0x0001, EXECUTABLE_IMAGE = 0x0002, LINE_NUMS_STRIPPED = 0x0004, LOCAL_SYMS_STRIPPED = 0x0008, AGGRESSIVE_WS_TRIM = 0x0010, LARGE_ADDRESS_AWARE = 0x0020, MACHINE_16BIT = 0x0040, BYTES_REVERSED_LO = 0x0080, MACHINE_32BIT = 0x0100, DEBUG_STRIPPED = 0x0200, REMOVABLE_RUN_FROM_SWAP = 0x0400, SYSTEM = 0x1000, DLL = 0x2000, UNIPROCESSOR_ONLY = 0x4000, BIG_ENDIAN_MACHINE = 0x8000, ), "symbol_table" / Pointer(this.symbol_table_pointer, Array(this.number_of_symbols, symbol_table)) ) PEPlusField = IfThenElse(this.pe_type == "PE32_plus", Int64ul, Int32ul) optional_header = "optional_header" / Struct( # standard fields "pe_type" / Enum(Int16ul, PE32 = 0x10b, PE32_plus = 0x20b, ), "major_linker_version" / Int8ul, "minor_linker_version" / Int8ul, "code_size" / Int32ul, "initialized_data_size" / Int32ul, "uninitialized_data_size" / Int32ul, "entry_point_pointer" / Int32ul, "base_of_code" / Int32ul, # only in PE32 files "base_of_data" / If(this.pe_type == "PE32", Int32ul), # WinNT-specific fields "image_base" / PEPlusField, "section_aligment" / Int32ul, "file_alignment" / Int32ul, "major_os_version" / Int16ul, "minor_os_version" / Int16ul, "major_image_version" / Int16ul, "minor_image_version" / Int16ul, "major_subsystem_version" / Int16ul, "minor_subsystem_version" / Int16ul, Padding(4), "image_size" / Int32ul, "headers_size" / Int32ul, "checksum" / Int32ul, "subsystem" / Enum(Int16ul, UNKNOWN = 0, NATIVE = 1, WINDOWS_GUI = 2, WINDOWS_CUI = 3, POSIX_CIU = 7, WINDOWS_CE_GUI = 9, EFI_APPLICATION = 10, EFI_BOOT_SERVICE_DRIVER = 11, EFI_RUNTIME_DRIVER = 12, EFI_ROM = 13, XBOX = 14, _default_ = Pass ), "dll_characteristics" / FlagsEnum(Int16ul, NO_BIND = 0x0800, WDM_DRIVER = 0x2000, TERMINAL_SERVER_AWARE = 0x8000, ), "reserved_stack_size" / PEPlusField, "stack_commit_size" / PEPlusField, "reserved_heap_size" / PEPlusField, "heap_commit_size" / PEPlusField, "loader_flags" / Int32ul, "number_of_data_directories" / Int32ul, NamedSequence( Array(this.number_of_data_directories, "data_directories" / Struct( "address" / Int32ul, "size" / Int32ul, ) ), mapping = { 0 : 'export_table', 1 : 'import_table', 2 : 'resource_table', 3 : 'exception_table', 4 : 'certificate_table', 5 : 'base_relocation_table', 6 : 'debug', 7 : 'architecture', 8 : 'global_ptr', 9 : 'tls_table', 10 : 'load_config_table', 11 : 'bound_import', 12 : 'import_address_table', 13 : 'delay_import_descriptor', 14 : 'complus_runtime_header', } ), ) section = "section" / Struct( "name" / String(8, padchar=b"\x00"), "virtual_size" / Int32ul, "virtual_address" / Int32ul, "raw_data_size" / Int32ul, "raw_data_pointer" / Int32ul, "relocations_pointer" / Int32ul, "line_numbers_pointer" / Int32ul, "number_of_relocations" / Int16ul, "number_of_line_numbers" / Int16ul, "characteristics" / FlagsEnum(Int32ul, TYPE_REG = 0x00000000, TYPE_DSECT = 0x00000001, TYPE_NOLOAD = 0x00000002, TYPE_GROUP = 0x00000004, TYPE_NO_PAD = 0x00000008, TYPE_COPY = 0x00000010, CNT_CODE = 0x00000020, CNT_INITIALIZED_DATA = 0x00000040, CNT_UNINITIALIZED_DATA = 0x00000080, LNK_OTHER = 0x00000100, LNK_INFO = 0x00000200, TYPE_OVER = 0x00000400, LNK_REMOVE = 0x00000800, LNK_COMDAT = 0x00001000, MEM_FARDATA = 0x00008000, MEM_PURGEABLE = 0x00020000, MEM_16BIT = 0x00020000, MEM_LOCKED = 0x00040000, MEM_PRELOAD = 0x00080000, ALIGN_1BYTES = 0x00100000, ALIGN_2BYTES = 0x00200000, ALIGN_4BYTES = 0x00300000, ALIGN_8BYTES = 0x00400000, ALIGN_16BYTES = 0x00500000, ALIGN_32BYTES = 0x00600000, ALIGN_64BYTES = 0x00700000, ALIGN_128BYTES = 0x00800000, ALIGN_256BYTES = 0x00900000, ALIGN_512BYTES = 0x00A00000, ALIGN_1024BYTES = 0x00B00000, ALIGN_2048BYTES = 0x00C00000, ALIGN_4096BYTES = 0x00D00000, ALIGN_8192BYTES = 0x00E00000, LNK_NRELOC_OVFL = 0x01000000, MEM_DISCARDABLE = 0x02000000, MEM_NOT_CACHED = 0x04000000, MEM_NOT_PAGED = 0x08000000, MEM_SHARED = 0x10000000, MEM_EXECUTE = 0x20000000, MEM_READ = 0x40000000, MEM_WRITE = 0x80000000, ), "raw_data" / OnDemandPointer(this.raw_data_pointer, HexDump(Bytes(this.raw_data_size))), "line_numbers" / OnDemandPointer(this.line_numbers_pointer, Array(this.number_of_line_numbers, Struct( "type" / Int32ul, "line_number" / Int16ul, ) ) ), "relocations" / OnDemandPointer(this.relocations_pointer, Array(this.number_of_relocations, Struct( "virtual_address" / Int32ul, "symbol_table_index" / Int32ul, "type" / Int16ul, ) ) ), ) pe32_file = "pe32_file" / Struct( msdos_header, coff_header, "_start_of_optional_header" / Tell, optional_header, "_end_of_optional_header" / Tell, Padding(lambda ctx: min(0, ctx.coff_header.optional_header_size - ctx._end_of_optional_header + ctx._start_of_optional_header)), "sections" / Array(this.coff_header.number_of_sections, section) ) construct-2.8.16/construct/examples/formats/executable/elf32.py0000644000175000017500000001003213151363471026573 0ustar arkadiuszarkadiusz00000000000000""" Executable and Linkable Format (ELF), 32 bit, big or little endian. Used on *nix systems as a replacement of the older a.out format. Big-endian support kindly submitted by Craig McQueen (mcqueen-c#edsrd1!yzk!co!jp). """ from construct import * def elf32_body(ElfInt16, ElfInt32): elf32_program_header = Struct("program_header", Enum(ElfInt32("type"), NULL = 0, LOAD = 1, DYNAMIC = 2, INTERP = 3, NOTE = 4, SHLIB = 5, PHDR = 6, _default_ = Pass, ), ElfInt32("offset"), ElfInt32("vaddr"), ElfInt32("paddr"), ElfInt32("file_size"), ElfInt32("mem_size"), ElfInt32("flags"), ElfInt32("align"), ) elf32_section_header = Struct("section_header", ElfInt32("name_offset"), Pointer(lambda ctx: ctx._.strtab_data_offset + ctx.name_offset, CString("name") ), Enum(ElfInt32("type"), NULL = 0, PROGBITS = 1, SYMTAB = 2, STRTAB = 3, RELA = 4, HASH = 5, DYNAMIC = 6, NOTE = 7, NOBITS = 8, REL = 9, SHLIB = 10, DYNSYM = 11, _default_ = Pass, ), ElfInt32("flags"), ElfInt32("addr"), ElfInt32("offset"), ElfInt32("size"), ElfInt32("link"), ElfInt32("info"), ElfInt32("align"), ElfInt32("entry_size"), OnDemandPointer(lambda ctx: ctx.offset, HexDumpAdapter(Field("data", lambda ctx: ctx.size)) ), ) return Struct("body", Enum(ElfInt16("type"), NONE = 0, RELOCATABLE = 1, EXECUTABLE = 2, SHARED = 3, CORE = 4, ), Enum(ElfInt16("machine"), NONE = 0, M32 = 1, SPARC = 2, I386 = 3, Motorolla68K = 4, Motorolla88K = 5, Intel860 = 7, MIPS = 8, _default_ = Pass ), ElfInt32("version"), ElfInt32("entry"), ElfInt32("ph_offset"), ElfInt32("sh_offset"), ElfInt32("flags"), ElfInt16("header_size"), ElfInt16("ph_entry_size"), ElfInt16("ph_count"), ElfInt16("sh_entry_size"), ElfInt16("sh_count"), ElfInt16("strtab_section_index"), # calculate the string table data offset (pointer arithmetics) # ugh... anyway, we need it in order to read the section names, later on Pointer(lambda ctx: ctx.sh_offset + ctx.strtab_section_index * ctx.sh_entry_size + 16, ElfInt32("strtab_data_offset"), ), # program header table Rename("program_table", Pointer(lambda ctx: ctx.ph_offset, Array(lambda ctx: ctx.ph_count, elf32_program_header ) ) ), # section table Rename("sections", Pointer(lambda ctx: ctx.sh_offset, Array(lambda ctx: ctx.sh_count, elf32_section_header ) ) ), ) elf32_body_little_endian = elf32_body(ULInt16, ULInt32) elf32_body_big_endian = elf32_body(UBInt16, UBInt32) elf32_file = Struct("elf32_file", Struct("identifier", Const(b"\x7fELF"), Enum(Byte("file_class"), NONE = 0, CLASS32 = 1, CLASS64 = 2, ), Enum(Byte("encoding"), NONE = 0, LSB = 1, MSB = 2, ), Byte("version"), Padding(9), ), Embedded(IfThenElse("body", lambda ctx: ctx.identifier.encoding == "LSB", elf32_body_little_endian, elf32_body_big_endian, )), ) if __name__ == "__main__": obj = elf32_file.parse_stream(open("../../../tests/_ctypes_test.so", "rb")) #[s.data.value for s in obj.sections] print(obj) construct-2.8.16/construct/examples/formats/graphics/0000755000175000017500000000000013164134612024766 5ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/examples/formats/graphics/gif.py0000644000175000017500000000756513151364636026131 0ustar arkadiuszarkadiusz00000000000000""" Contributed by Dany Zatuchna (danzat at gmail) Implementation of the following grammar for the GIF89a file format ::= Header * Trailer ::= Logical Screen Descriptor [Global Color Table] ::= | ::= [Graphic Control Extension] ::= | Plain Text Extension ::= Image Descriptor [Local Color Table] Image Data ::= Application Extension | Comment Extension """ from construct import * data_sub_block = Struct("data_sub_block", ULInt8("size"), String("data", lambda ctx: ctx["size"]) ) gif_logical_screen = Struct("logical_screen", ULInt16("width"), ULInt16("height"), BitStruct("flags", Bit("global_color_table"), BitField("color_resolution", 3), Bit("sort_flag"), BitField("global_color_table_bpp", 3) ), ULInt8("bgcolor_index"), ULInt8("pixel_aspect_ratio"), If(lambda ctx: ctx["flags"]["global_color_table"], Array(lambda ctx: 2**(ctx["flags"]["global_color_table_bpp"] + 1), Struct("palette", ULInt8("R"), ULInt8("G"), ULInt8("B") ) ) ) ) gif_header = Struct("gif_header", Const("signature", b"GIF"), Const("version", b"89a"), ) application_extension = Struct("application_extension", Const(ULInt8("block_size"), 11), String("application_identifier", 8), String("application_auth_code", 3), data_sub_block, ULInt8("block_terminator") ) comment_extension = Struct("comment_extension", data_sub_block, ULInt8("block_terminator") ) graphic_control_extension = Struct("graphic_control_extension", Const(ULInt8("block_size"), 4), BitStruct("flags", BitField("reserved", 3), BitField("disposal_method", 3), Bit("user_input_flag"), Bit("transparent_color_flag"), ), ULInt16("delay"), ULInt8("transparent_color_index"), ULInt8("block_terminator") ) plain_text_extension = Struct("plain_text_extension", Const(ULInt8("block_size"), 12), ULInt16("text_left"), ULInt16("text_top"), ULInt16("text_width"), ULInt16("text_height"), ULInt8("cell_width"), ULInt8("cell_height"), ULInt8("foreground_index"), ULInt8("background_index"), data_sub_block, ULInt8("block_terminator") ) extension = Struct("extension", ULInt8("label"), Switch("ext", lambda ctx: ctx["label"], { 0xFF: application_extension, 0xFE: comment_extension, 0xF9: graphic_control_extension, 0x01: plain_text_extension }) ) image_descriptor = Struct("image_descriptor", ULInt16("left"), ULInt16("top"), ULInt16("width"), ULInt16("height"), BitStruct("flags", Bit("local_color_table"), Bit("interlace"), Bit("sort"), BitField("reserved", 2), BitField("local_color_table_bpp", 3) ), If(lambda ctx: ctx["flags"]["local_color_table"], Array(lambda ctx: 2**(ctx["flags"]["local_color_table_bpp"] + 1), Struct("palette", ULInt8("R"), ULInt8("G"), ULInt8("B") ) ) ), ULInt8("LZW_minimum_code_size"), RepeatUntil(lambda obj,lst,ctx: obj.size == 0, data_sub_block) ) gif_data = Struct("gif_data", ULInt8("introducer"), Switch("dat", lambda ctx: ctx["introducer"], { 0x21: extension, 0x2C: image_descriptor }) ) gif_file = Struct("gif_file", gif_header, gif_logical_screen, OptionalGreedyRange(gif_data), #Const(ULInt8("trailer"), 0x3B) ) if __name__ == "__main__": f = open("../../../tests/sample.gif", "rb") s = f.read() f.close() print(gif_file.parse(s)) construct-2.8.16/construct/examples/formats/graphics/emf.py0000644000175000017500000001054013151364531026110 0ustar arkadiuszarkadiusz00000000000000""" Enhanced Meta File """ from construct import * record_type = "record_type" / Enum(Int32ul, ABORTPATH = 68, ANGLEARC = 41, ARC = 45, ARCTO = 55, BEGINPATH = 59, BITBLT = 76, CHORD = 46, CLOSEFIGURE = 61, CREATEBRUSHINDIRECT = 39, CREATEDIBPATTERNBRUSHPT = 94, CREATEMONOBRUSH = 93, CREATEPALETTE = 49, CREATEPEN = 38, DELETEOBJECT = 40, ELLIPSE = 42, ENDPATH = 60, EOF = 14, EXCLUDECLIPRECT = 29, EXTCREATEFONTINDIRECTW = 82, EXTCREATEPEN = 95, EXTFLOODFILL = 53, EXTSELECTCLIPRGN = 75, EXTTEXTOUTA = 83, EXTTEXTOUTW = 84, FILLPATH = 62, FILLRGN = 71, FLATTENPATH = 65, FRAMERGN = 72, GDICOMMENT = 70, HEADER = 1, INTERSECTCLIPRECT = 30, INVERTRGN = 73, LINETO = 54, MASKBLT = 78, MODIFYWORLDTRANSFORM = 36, MOVETOEX = 27, OFFSETCLIPRGN = 26, PAINTRGN = 74, PIE = 47, PLGBLT = 79, POLYBEZIER = 2, POLYBEZIER16 = 85, POLYBEZIERTO = 5, POLYBEZIERTO16 = 88, POLYDRAW = 56, POLYDRAW16 = 92, POLYGON = 3, POLYGON16 = 86, POLYLINE = 4, POLYLINE16 = 87, POLYLINETO = 6, POLYLINETO16 = 89, POLYPOLYGON = 8, POLYPOLYGON16 = 91, POLYPOLYLINE = 7, POLYPOLYLINE16 = 90, POLYTEXTOUTA = 96, POLYTEXTOUTW = 97, REALIZEPALETTE = 52, RECTANGLE = 43, RESIZEPALETTE = 51, RESTOREDC = 34, ROUNDRECT = 44, SAVEDC = 33, SCALEVIEWPORTEXTEX = 31, SCALEWINDOWEXTEX = 32, SELECTCLIPPATH = 67, SELECTOBJECT = 37, SELECTPALETTE = 48, SETARCDIRECTION = 57, SETBKCOLOR = 25, SETBKMODE = 18, SETBRUSHORGEX = 13, SETCOLORADJUSTMENT = 23, SETDIBITSTODEVICE = 80, SETMAPMODE = 17, SETMAPPERFLAGS = 16, SETMETARGN = 28, SETMITERLIMIT = 58, SETPALETTEENTRIES = 50, SETPIXELV = 15, SETPOLYFILLMODE = 19, SETROP2 = 20, SETSTRETCHBLTMODE = 21, SETTEXTALIGN = 22, SETTEXTCOLOR = 24, SETVIEWPORTEXTEX = 11, SETVIEWPORTORGEX = 12, SETWINDOWEXTEX = 9, SETWINDOWORGEX = 10, SETWORLDTRANSFORM = 35, STRETCHBLT = 77, STRETCHDIBITS = 81, STROKEANDFILLPATH = 63, STROKEPATH = 64, WIDENPATH = 66, default=Pass ) generic_record = "records" / Struct( record_type, "record_size" / Int32ul, # Size of the record in bytes "params" / RawCopy(Array((this.record_size - 8) // 4, Int32ul)), ) header_record = "header_record" / Struct( Const(record_type, "HEADER"), "record_size" / Int32ul, # Size of the record in bytes "bounds_left" / Int32sl, # Left inclusive bounds "bounds_right" / Int32sl, # Right inclusive bounds "bounds_top" / Int32sl, # Top inclusive bounds "bounds_bottom" / Int32sl, # Bottom inclusive bounds "frame_left" / Int32sl, # Left side of inclusive picture frame "frame_right" / Int32sl, # Right side of inclusive picture frame "frame_top" / Int32sl, # Top side of inclusive picture frame "frame_bottom" / Int32sl, # Bottom side of inclusive picture frame "signature" / Const(Int32ul, 0x464D4520), "version" / Int32ul, # Version of the metafile "size" / Int32ul, # Size of the metafile in bytes "num_of_records" / Int32ul, # Number of records in the metafile "num_of_handles" / Int16ul, # Number of handles in the handle table Padding(2), "description_size" / Int32ul, # Size of description string in WORDs "description_offset" / Int32ul, # Offset of description string in metafile "num_of_palette_entries" / Int32ul, # Number of color palette entries "device_width_pixels" / Int32sl, # Width of reference device in pixels "device_height_pixels" / Int32sl, # Height of reference device in pixels "device_width_mm" / Int32sl, # Width of reference device in millimeters "device_height_mm" / Int32sl, # Height of reference device in millimeters "description" / Pointer(this.description_offset, String(this.description_size * 2), ), # padding up to end of record Padding(this.record_size - 88), ) emf_file = "emf_file" / Struct( header_record, Array(this.header_record.num_of_records - 1, generic_record), ) construct-2.8.16/construct/examples/formats/graphics/bmp.py0000644000175000017500000000540213156503400026113 0ustar arkadiuszarkadiusz00000000000000""" Windows/OS2 Bitmap (BMP) this could have been a perfect show-case file format, but they had to make it ugly (all sorts of alignments) """ from construct import * #=============================================================================== # pixels: uncompressed #=============================================================================== def UncompressedRows(subcon, align_to_byte = False): """argh! lines must be aligned to a 4-byte boundary, and bit-pixel lines must be aligned to full bytes...""" if align_to_byte: line_pixels = Bitwise(Aligned(8, Array(this.width, subcon))) else: line_pixels = Array(this.width, subcon) return Array(this.height, Aligned(4, line_pixels)) uncompressed_pixels = "uncompressed" / Switch(this.bpp, { 1 : UncompressedRows(Bit, align_to_byte = True), # index 4 : UncompressedRows(Nibble, align_to_byte = True), # index 8 : UncompressedRows(Byte), # index 24 : UncompressedRows(Byte[3]), # rgb } ) #=============================================================================== # pixels: Run Length Encoding (RLE) 8 bit #=============================================================================== class RunLengthAdapter(Adapter): def _encode(self, obj): return len(obj), obj[0] def _decode(self, obj): length, value = obj return [value] * length rle8pixel = "rle8pixel" / RunLengthAdapter(Byte >> Byte) #=============================================================================== # file structure #=============================================================================== bitmap_file = "bitmap_file" / Struct( "signature" / Const(b"BM"), "file_size" / Int32ul, Padding(4), "data_offset" / Int32ul, "header_size" / Int32ul, "version" / Enum(Computed(this.header_size), v2 = 12, v3 = 40, v4 = 108, default = Pass, ), "width" / Int32ul, "height" / Int32ul, "number_of_pixels" / Computed(this.width * this.height), "planes" / Int16ul, "bpp" / Int16ul, # bits per pixel "compression" / Enum(Int32ul, Uncompressed = 0, RLE8 = 1, RLE4 = 2, Bitfields = 3, JPEG = 4, PNG = 5, ), "image_data_size" / Int32ul, # in bytes "horizontal_dpi" / Int32ul, "vertical_dpi" / Int32ul, "colors_used" / Int32ul, "important_colors" / Int32ul, # palette (24 bit has no palette) "palette" / Array(lambda ctx: 2**ctx.bpp if ctx.bpp <= 8 else 0, Struct( "rgb" / Byte[3], Padding(1), ) ), "pixels" / Pointer(this.data_offset, Switch(this.compression, { "Uncompressed" : uncompressed_pixels, }, ), ), ) construct-2.8.16/construct/examples/formats/graphics/png.py0000644000175000017500000002131313151364726026133 0ustar arkadiuszarkadiusz00000000000000""" Portable Network Graphics (PNG) file format Official spec: http://www.w3.org/TR/PNG Original code contributed by Robin Munn (rmunn at pobox dot com) (although the code has been extensively reorganized to meet Construct's coding conventions) """ from construct import * #=============================================================================== # utils #=============================================================================== coord = Struct( "x" / Int32ub, "y" / Int32ub, ) compression_method = "compression_method" / Enum(Byte, deflate = 0, default=Pass) #=============================================================================== # 11.2.3: PLTE - Palette #=============================================================================== plte_info = "plte_info" / Struct( "num_entries" / Computed(this._.length / 3), "palette_entries" / Array(this.num_entries, Byte[3]), ) #=============================================================================== # 11.2.4: IDAT - Image data #=============================================================================== idat_info = "idat_info" / Bytes(this.length) #=============================================================================== # 11.3.2.1: tRNS - Transparency #=============================================================================== trns_info = "trns_info" / Switch(this._.image_header.color_type, { "greyscale": Int16ub, "truecolor": Int16ub[3], "indexed": Array(this.length, Byte), } ) #=============================================================================== # 11.3.3.1: cHRM - Primary chromacities and white point #=============================================================================== chrm_info = "chrm_info" / Struct( "white_point" / coord, "rgb" / coord[3], ) #=============================================================================== # 11.3.3.2: gAMA - Image gamma #=============================================================================== gama_info = "gama_info" / Struct( "gamma" / Int32ub, ) #=============================================================================== # 11.3.3.3: iCCP - Embedded ICC profile #=============================================================================== iccp_info = "iccp_info" / Struct( "name" / CString(), compression_method, "compressed_profile" / Bytes(lambda ctx: ctx._.length - (len(ctx.name) + 2)), ) #=============================================================================== # 11.3.3.4: sBIT - Significant bits #=============================================================================== sbit_info = "sbit_info" / Switch(this._.image_header.color_type, { "greyscale": Byte, "truecolor": Byte[3], "indexed": Byte[3], "greywithalpha": Byte[2], "truewithalpha": Byte[4], } ) #=============================================================================== # 11.3.3.5: sRGB - Standard RPG color space #=============================================================================== srgb_info = "rendering_intent" / Enum(Byte, perceptual = 0, relative_colorimetric = 1, saturation = 2, absolute_colorimetric = 3, default=Pass ) #=============================================================================== # 11.3.4.3: tEXt - Textual data #=============================================================================== text_info = "text_info" / Struct( "keyword" / CString(), "text" / Bytes(lambda ctx: ctx._.length - (len(ctx.keyword) + 1)), ) #=============================================================================== # 11.3.4.4: zTXt - Compressed textual data #=============================================================================== ztxt_info = "ztxt_info" / Struct( "keyword" / CString(), compression_method, # As with iCCP, length is chunk length, minus length of # keyword, minus two: one byte for the null terminator, # and one byte for the compression method. "compressed_text" / Bytes(lambda ctx: ctx._.length - (len(ctx.keyword) + 2)), ) #=============================================================================== # 11.3.4.5: iTXt - International textual data #=============================================================================== itxt_info = "itxt_info" / Struct( "keyword" / CString(), "compression_flag" / Byte, compression_method, "language_tag" / CString(), "translated_keyword" / CString(), "text" / Bytes(lambda ctx: ctx._.length - (len(ctx.keyword) + len(ctx.language_tag) + len(ctx.translated_keyword) + 5)), ) #=============================================================================== # 11.3.5.1: bKGD - Background color #=============================================================================== bkgd_info = "bkgd_info" / Switch(this._.image_header.color_type, { "greyscale": Int16ub[1], "greywithalpha": Int16ub[1], "truecolor": Int16ub[3], "truewithalpha": Int16ub[3], "indexed": Int8ub, } ) #=============================================================================== # 11.3.5.2: hIST - Image histogram #=============================================================================== hist_info = "frequency" / Array(this._.length / 2, Int16ub) #=============================================================================== # 11.3.5.3: pHYs - Physical pixel dimensions #=============================================================================== phys_info = "phys_info" / Struct( "pixels_per_unit_x" / Int32ub, "pixels_per_unit_y" / Int32ub, "unit" / Enum(Byte, unknown = 0, meter = 1, default = Pass), ) #=============================================================================== # 11.3.5.4: sPLT - Suggested palette #=============================================================================== def splt_info_data_length(ctx): if ctx.sample_depth == 8: entry_size = 6 else: entry_size = 10 return (ctx._.length - len(ctx.name) - 2) / entry_size splt_info = "data" / Struct( "name" / CString(), "sample_depth" / Byte, "table" / Array(splt_info_data_length, IfThenElse(this.sample_depth == 8, # Sample depth 8 Struct( "rgb" / Byte[3], "alpha" / Byte, "frequency" / Int16ub, ), # Sample depth 16 Struct( "rgb" / Byte[3], "alpha" / Int16ub, "frequency" / Int16ub, ), ), ), ) #=============================================================================== # 11.3.6.1: tIME - Image last-modification time #=============================================================================== time_info = "time_info" / Struct( "year" / Int16ub, "month" / Byte, "day" / Byte, "hour" / Byte, "minute" / Byte, "second" / Byte, ) #=============================================================================== # chunks #=============================================================================== default_chunk_info = HexDump(Bytes(this.length)) chunk = "chunk" / Struct( "length" / Int32ub, "type" / Bytes(4), "data" / Switch(this.type, { b"PLTE" : plte_info, b"IEND" : Pass, b"IDAT" : idat_info, b"tRNS" : trns_info, b"cHRM" : chrm_info, b"gAMA" : gama_info, b"iCCP" : iccp_info, b"sBIT" : sbit_info, b"sRGB" : srgb_info, b"tEXt" : text_info, b"zTXt" : ztxt_info, b"iTXt" : itxt_info, b"bKGD" : bkgd_info, b"hIST" : hist_info, b"pHYs" : phys_info, b"sPLT" : splt_info, b"tIME" : time_info, }, default = default_chunk_info, ), "crc" / Int32ub, ) image_header_chunk = "image_header" / Struct( "length" / Int32ub, "signature" / Const(b"IHDR"), "width" / Int32ub, "height" / Int32ub, "bit_depth" / Byte, "color_type" / Enum(Byte, greyscale = 0, truecolor = 2, indexed = 3, greywithalpha = 4, truewithalpha = 6, default = Pass, ), compression_method, # "adaptive filtering with five basic filter types" "filter_method" / Enum(Byte, adaptive5 = 0, default = Pass), "interlace_method" / Enum(Byte, none = 0, adam7 = 1, default = Pass), "crc" / Int32ub, ) #=============================================================================== # the complete PNG file #=============================================================================== png_file = "png" / Struct( "signature" / Const(b"\x89PNG\r\n\x1a\n"), image_header_chunk, "chunks" / GreedyRange(chunk), ) construct-2.8.16/construct/examples/formats/graphics/__init__.py0000644000175000017500000000000013151364660027071 0ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/examples/formats/graphics/wmf.py0000644000175000017500000000671713151364743026152 0ustar arkadiuszarkadiusz00000000000000""" Windows Meta File """ from construct import * wmf_record = Struct("records", ULInt32("size"), # size in words, including the size, function and params Enum(ULInt16("function"), AbortDoc = 0x0052, Aldus_Header = 0x0001, AnimatePalette = 0x0436, Arc = 0x0817, BitBlt = 0x0922, Chord = 0x0830, CLP_Header16 = 0x0002, CLP_Header32 = 0x0003, CreateBitmap = 0x06FE, CreateBitmapIndirect = 0x02FD, CreateBrush = 0x00F8, CreateBrushIndirect = 0x02FC, CreateFontIndirect = 0x02FB, CreatePalette = 0x00F7, CreatePatternBrush = 0x01F9, CreatePenIndirect = 0x02FA, CreateRegion = 0x06FF, DeleteObject = 0x01F0, DibBitblt = 0x0940, DibCreatePatternBrush = 0x0142, DibStretchBlt = 0x0B41, DrawText = 0x062F, Ellipse = 0x0418, EndDoc = 0x005E, EndPage = 0x0050, EOF = 0x0000, Escape = 0x0626, ExcludeClipRect = 0x0415, ExtFloodFill = 0x0548, ExtTextOut = 0x0A32, FillRegion = 0x0228, FloodFill = 0x0419, FrameRegion = 0x0429, Header = 0x0004, IntersectClipRect = 0x0416, InvertRegion = 0x012A, LineTo = 0x0213, MoveTo = 0x0214, OffsetClipRgn = 0x0220, OffsetViewportOrg = 0x0211, OffsetWindowOrg = 0x020F, PaintRegion = 0x012B, PatBlt = 0x061D, Pie = 0x081A, Polygon = 0x0324, Polyline = 0x0325, PolyPolygon = 0x0538, RealizePalette = 0x0035, Rectangle = 0x041B, ResetDC = 0x014C, ResizePalette = 0x0139, RestoreDC = 0x0127, RoundRect = 0x061C, SaveDC = 0x001E, ScaleViewportExt = 0x0412, ScaleWindowExt = 0x0410, SelectClipRegion = 0x012C, SelectObject = 0x012D, SelectPalette = 0x0234, SetBKColor = 0x0201, SetBKMode = 0x0102, SetDibToDev = 0x0D33, SelLayout = 0x0149, SetMapMode = 0x0103, SetMapperFlags = 0x0231, SetPalEntries = 0x0037, SetPixel = 0x041F, SetPolyFillMode = 0x0106, SetReLabs = 0x0105, SetROP2 = 0x0104, SetStretchBltMode = 0x0107, SetTextAlign = 0x012E, SetTextCharExtra = 0x0108, SetTextColor = 0x0209, SetTextJustification = 0x020A, SetViewportExt = 0x020E, SetViewportOrg = 0x020D, SetWindowExt = 0x020C, SetWindowOrg = 0x020B, StartDoc = 0x014D, StartPage = 0x004F, StretchBlt = 0x0B23, StretchDIB = 0x0F43, TextOut = 0x0521, _default_ = Pass, ), Array(lambda ctx: ctx.size - 3, ULInt16("params")), ) wmf_placeable_header = Struct("placeable_header", Const(ULInt32("key"), 0x9AC6CDD7), ULInt16("handle"), SLInt16("left"), SLInt16("top"), SLInt16("right"), SLInt16("bottom"), ULInt16("units_per_inch"), Padding(4), ULInt16("checksum") ) wmf_file = Struct("wmf_file", # --- optional placeable header --- Optional(wmf_placeable_header), # --- header --- Enum(ULInt16("type"), InMemory = 0, File = 1, ), Const(ULInt16("header_size"), 9), ULInt16("version"), ULInt32("size"), # file size is in words ULInt16("number_of_objects"), ULInt32("size_of_largest_record"), ULInt16("number_of_params"), # --- records --- GreedyRange(wmf_record) ) construct-2.8.16/construct/examples/formats/filesystem/0000755000175000017500000000000013164134612025352 5ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/examples/formats/filesystem/fat16.py0000644000175000017500000001610613151364312026647 0ustar arkadiuszarkadiusz00000000000000""" fat.py; ad-hoc fat16 reader by Bram Westerbaan references: http://en.wikipedia.org/wiki/File_Allocation_Table http://www.ecma-international.org/publications/standards/Ecma-107.htm example: with open("/dev/sdc1", "rb") as file: fs = FatFs(file) for rootdir in fs: print rootdir """ import numbers from io import BytesIO, BufferedReader from construct import * def Fat16Header(name): return Struct(name, Bytes("jumpInstruction", 3), Bytes("creatingSystemId", 8), ULInt16("sectorSize"), Byte("sectorsPerCluster"), ULInt16("reservedSectorCount"), Byte("fatCount"), ULInt16("rootdirEntryCount"), ULInt16("sectorCount_small"), Byte("mediaId"), ULInt16("sectorsPerFat"), ULInt16("sectorsPerTrack"), ULInt16("sideCount"), ULInt32("hiddenSectorCount"), ULInt32("sectorCount_large"), Byte("physicalDriveNumber"), Byte("currentHead"), Byte("extendedBootSignature"), Bytes("volumeId", 4), Bytes("volumeLabel", 11), Const(Bytes("fsType", 8), "FAT16 "), Bytes("bootCode", 448), Const(Bytes("bootSectorSignature", 2), "\x55\xaa"), ) def BootSector(name): header = Fat16Header("header") return Struct(name, Embed(header), Padding(lambda ctx: ctx.sectorSize - header.sizeof())) def FatEntry(name): return Enum(ULInt16(name), free_cluster = 0x0000, bad_cluster = 0xfff7, last_cluster = 0xffff, _default_ = Pass) def DirEntry(name): return Struct(name, Bytes("name", 8), Bytes("extension", 3), BitStruct("attributes", Flag("unused"), Flag("device"), Flag("archive"), Flag("subDirectory"), Flag("volumeLabel"), Flag("system"), Flag("hidden"), Flag("readonly")), # reserved Padding(10), ULInt16("timeRecorded"), ULInt16("dateRecorded"), ULInt16("firstCluster"), ULInt32("fileSize"), ) def PreDataRegion(name): rde = DirEntry("rootdirs") fe = FatEntry("fats") return Struct(name, Embed(BootSector("bootSector")), # the remaining reserved sectors Padding(lambda ctx: (ctx.reservedSectorCount - 1) * ctx.sectorSize), # file allocation tables Array(lambda ctx: (ctx.fatCount), Array(lambda ctx: ctx.sectorsPerFat * ctx.sectorSize / fe.sizeof(), fe)), # root directories Array(lambda ctx: (ctx.rootdirEntryCount*rde.sizeof()) / ctx.sectorSize, rde)) class File(object): def __init__(self, dirEntry, fs): self.fs = fs self.dirEntry = dirEntry @classmethod def fromDirEntry(cls, dirEntry, fs): if dirEntry.name[0] in "\x00\xe5\x2e": return None a = dirEntry.attributes #Long file name directory entry if a.volumeLabel and a.system and a.hidden and a.readonly: return None if a.subDirectory: return Directory(dirEntry, fs) return File(dirEntry, fs) @classmethod def fromDirEntries(cls, dirEntries, fs): return filter(None, [cls.fromDirEntry(de, fs) for de in dirEntries]) def toStream(self, stream): self.fs.fileToStream(self.dirEntry.firstCluster, stream) @property def name(self): return "%s.%s" % (self.dirEntry.name.rstrip(), self.dirEntry.extension) def __str__(self): return "&%s %s" % (self.dirEntry.firstCluster, self.name) class Directory(File): def __init__(self, dirEntry, fs, children=None): super(Directory, self).__init__(dirEntry, fs) self.children = children if not self.children: self.children = File.fromDirEntries(\ self.fs.getDirEntries(\ self.dirEntry.firstCluster), fs) @property def name(self): return self.dirEntry.name.rstrip() def __str__(self): return "&%s %s/" % (self.dirEntry.firstCluster, self.name) def __getitem__(self, name): for file in self.children: if file.name == name: return file def __iter__(self): return iter(self.children) class FatFs(Directory): def __init__(self, stream): self.stream = stream self.pdr = PreDataRegion("pdr").parse_stream(stream) super(FatFs, self).__init__(dirEntry = None, fs = self, children = File.fromDirEntries( self.pdr.rootdirs, self)) def fileToStream(self, clidx, stream): for clidx in self.getLinkedClusters(clidx): self.clusterToStream(clidx, stream) def clusterToStream(self, clidx, stream): start, todo = self.getClusterSlice(clidx) self.stream.seek(start) while todo > 0: read = self.stream.read(todo) if not len(read): print("failed to read %s bytes at %s" % (todo, self.stream.tell())) raise EOFError() todo -= len(read) stream.write(read) def getClusterSlice(self, clidx): startSector = self.pdr.reservedSectorCount \ + self.pdr.fatCount * self.pdr.sectorsPerFat \ + (self.pdr.rootdirEntryCount * 32) \ / self.pdr.sectorSize \ + (clidx-2) * self.pdr.sectorsPerCluster start = startSector * self.pdr.sectorSize length = self.pdr.sectorSize * self.pdr.sectorsPerCluster return (start, length) def getLinkedClusters(self, clidx): res = [] while clidx != "last_cluster": if not isinstance(clidx, numbers.Real): print(clidx) assert False assert 2 <= clidx <= 0xffef res.append(clidx) clidx = self.getNextCluster(clidx) assert clidx not in res return res def getNextCluster(self, clidx): ress = set([fat[clidx] for fat in self.pdr.fats]) if len(ress)==1: return ress.pop() print("inconsistencie between FATs: %s points to" % clidx) for i,fat in enumerate(self.pdr.fats): print("\t%s according to fat #%s" % (fat[clidx], i)) res = ress.pop() print ("assuming %s" % res) return res def getDirEntries(self, clidx): try: for de in self._getDirEntries(clidx): yield de except IOError: print("failed to read directory entries at %s" % clidx) def _getDirEntries(self, clidx): de = DirEntry("dirEntry") with BytesIO() as mem: self.fileToStream(clidx, mem) mem.seek(0) with BufferedReader(mem) as reader: while reader.peek(1): yield de.parse_stream(reader) def __str__(self): return "/" @property def name(self): return "" construct-2.8.16/construct/examples/formats/filesystem/mbr.py0000644000175000017500000000234213151364350026505 0ustar arkadiuszarkadiusz00000000000000""" Master Boot Record The first sector on disk, contains the partition table, bootloader, et al. http://www.win.tue.nl/~aeb/partitions/partition_types-1.html """ from construct import * mbr_format = "mbr" / Struct( "bootloader_code" / HexDump(Bytes(446)), "partitions" / Array(4, Struct( "state" / Enum(Byte, INACTIVE = 0x00, ACTIVE = 0x80, ), "beginning" / BitStruct( "head" / Octet, "sect" / BitsInteger(6), "cyl" / BitsInteger(10), ), "type" / Enum(Byte, Nothing = 0x00, FAT12 = 0x01, XENIX_ROOT = 0x02, XENIX_USR = 0x03, FAT16_old = 0x04, Extended_DOS = 0x05, FAT16 = 0x06, FAT32 = 0x0b, FAT32_LBA = 0x0c, NTFS = 0x07, LINUX_SWAP = 0x82, LINUX_NATIVE = 0x83, default=Pass, ), "ending" / BitStruct( "head" / Octet, "sect" / BitsInteger(6), "cyl" / BitsInteger(10), ), "sector_offset" / Int32ub, # offset from MBR in sectors "size" / Int32ub, # in sectors )), "signature" / Const(b"\x55\xAA"), ) construct-2.8.16/construct/examples/formats/filesystem/__init__.py0000644000175000017500000000000013151364340027450 0ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/examples/formats/filesystem/ext2.py0000644000175000017500000000570313151364007026612 0ustar arkadiuszarkadiusz00000000000000""" Extension 2 (ext2) used in Linux systems """ from construct import * Char = SLInt8 UChar = ULInt8 Short = SLInt16 UShort = ULInt16 Long = SLInt32 ULong = ULInt32 def BlockPointer(name): return Struct(name, ULong("block_number"), OnDemandPointer(lambda ctx: ctx["block_number"]), ) superblock = Struct("superblock", ULong('inodes_count'), ULong('blocks_count'), ULong('reserved_blocks_count'), ULong('free_blocks_count'), ULong('free_inodes_count'), ULong('first_data_block'), Enum(ULong('log_block_size'), OneKB = 0, TwoKB = 1, FourKB = 2, ), Long('log_frag_size'), ULong('blocks_per_group'), ULong('frags_per_group'), ULong('inodes_per_group'), ULong('mtime'), ULong('wtime'), UShort('mnt_count'), Short('max_mnt_count'), Const(UShort('magic'), 0xEF53), UShort('state'), UShort('errors'), Padding(2), ULong('lastcheck'), ULong('checkinterval'), ULong('creator_os'), ULong('rev_level'), Padding(235 * 4), ) group_descriptor = Struct("group_descriptor", ULong('block_bitmap'), ULong('inode_bitmap'), ULong('inode_table'), UShort('free_blocks_count'), UShort('free_inodes_count'), UShort('used_dirs_count'), Padding(14), ) inode = Struct("inode", FlagsEnum(UShort('mode'), IXOTH = 0x0001, IWOTH = 0x0002, IROTH = 0x0004, IRWXO = 0x0007, IXGRP = 0x0008, IWGRP = 0x0010, IRGRP = 0x0020, IRWXG = 0x0038, IXUSR = 0x0040, IWUSR = 0x0080, IRUSR = 0x0100, IRWXU = 0x01C0, ISVTX = 0x0200, ISGID = 0x0400, ISUID = 0x0800, IFIFO = 0x1000, IFCHR = 0x2000, IFDIR = 0x4000, IFBLK = 0x6000, IFREG = 0x8000, IFLNK = 0xC000, IFSOCK = 0xA000, IFMT = 0xF000, ), UShort('uid'), ULong('size'), ULong('atime'), ULong('ctime'), ULong('mtime'), ULong('dtime'), UShort('gid'), UShort('links_count'), ULong('blocks'), FlagsEnum(ULong('flags'), SecureDelete = 0x0001, AllowUndelete = 0x0002, Compressed = 0x0004, Synchronous = 0x0008, ), Padding(4), Array(12, ULong('blocks')), ULong("indirect1_block"), ULong("indirect2_block"), ULong("indirect3_block"), ULong('version'), ULong('file_acl'), ULong('dir_acl'), ULong('faddr'), UChar('frag'), Byte('fsize'), Padding(10) , ) # special inodes EXT2_BAD_INO = 1 EXT2_ROOT_INO = 2 EXT2_ACL_IDX_INO = 3 EXT2_ACL_DATA_INO = 4 EXT2_BOOT_LOADER_INO = 5 EXT2_UNDEL_DIR_INO = 6 EXT2_FIRST_INO = 11 directory_record = Struct("directory_entry", ULong("inode"), UShort("rec_length"), UShort("name_length"), Field("name", lambda ctx: ctx["name_length"]), Padding(lambda ctx: ctx["rec_length"] - ctx["name_length"]) ) if __name__ == "__main__": print (superblock.sizeof()) construct-2.8.16/construct/examples/formats/__init__.py0000644000175000017500000000066213151124063025276 0ustar arkadiuszarkadiusz00000000000000from construct.examples.formats.graphics.emf import emf_file from construct.examples.formats.graphics.png import png_file from construct.examples.formats.graphics.bmp import bitmap_file from construct.examples.formats.filesystem.mbr import mbr_format from construct.examples.formats.data.cap import cap_file from construct.examples.formats.data.snoop import snoop_file from construct.examples.formats.executable.pe32 import pe32_file construct-2.8.16/construct/examples/__init__.py0000644000175000017500000000000013151124063023605 0ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct/__init__.py0000644000175000017500000000662313164126674022026 0ustar arkadiuszarkadiusz00000000000000r""" Construct 2 -- Parsing Made Fun Homepage: https://github.com/construct/construct http://construct.readthedocs.org Hands-on example: >>> from construct import * >>> s = Struct( ... "a" / Byte, ... "b" / Short, ... ) >>> print s.parse(b"\x01\x02\x03") Container: a = 1 b = 515 >>> s.build(Container(a=1, b=0x0203)) b"\x01\x02\x03" """ from construct.core import * from construct.expr import this, Path, Path2, PathFunc, len_, sum_, min_, max_, abs_, obj_ from construct.debug import Probe, ProbeInto, Debugger from construct.version import version, version_string, release_date from construct import lib #=============================================================================== # metadata #=============================================================================== __author__ = "Arkadiusz Bulski , Tomer Filiba , Corbin Simpson " __version__ = version_string #=============================================================================== # aliases #=============================================================================== #=============================================================================== # exposed names #=============================================================================== __all__ = [ 'AdaptationError', 'Aligned', 'AlignedStruct', 'Array', 'Bit', 'BitIntegerError', 'BitStruct', 'Bitwise', 'CString', 'Construct', 'ConstructError', 'Container', 'Debugger', 'EmbeddedBitStruct', 'Enum', 'ExprAdapter', 'FieldError', 'Flag', 'FlagsContainer', 'FlagsEnum', 'Bytes', 'FormatField', 'GreedyRange', 'HexDump', 'HexString', 'If', 'IfThenElse', 'Indexing', 'LazyBound', 'LazyContainer', 'ListContainer', 'Mapping', 'MappingError', 'Nibble', 'NoneOf', 'Octet', 'OnDemand', 'OnDemandPointer', 'OneOf', 'Optional', 'OverwriteError', 'Packer', 'Padding', 'PaddingError', 'PascalString', 'Pass', 'Peek', 'Pointer', 'PrefixedArray', 'Probe', 'Range', 'RangeError', 'Renamed', 'RepeatUntil', 'Select', 'SelectError', 'Sequence', 'SizeofError', 'Slicing', 'String', 'Struct', 'Subconstruct', 'Switch', 'SwitchError', 'SymmetricMapping', 'Terminated', 'TerminatedError', 'UnionError', 'Union', 'ValidationError', 'Validator', 'Computed', 'Bytes', 'Tunnel', 'Embedded', 'Const', 'ConstError', 'VarInt', 'StringError', 'Checksum', 'ByteSwapped', 'LazyStruct', 'Numpy', 'Adapter', 'SymmetricAdapter', 'Tunnel', 'Compressed', 'GreedyBytes', 'Prefixed', 'Padded', 'GreedyString', 'RawCopy', 'LazyRange', 'LazySequence', 'LazySequenceContainer', 'BitsInteger', 'BytesInteger', '__author__', '__version__', 'Restreamed', 'RestreamedBytesIO', 'Bytewise', 'LazyRangeContainer', 'BitsSwapped', 'RebufferedBytesIO', 'Rebuffered', 'version', 'version_string','lib','Seek','Tell','setglobalstringencoding','globalstringencoding','NamedTuple','ExprValidator','Filter','Hex','Error','ExplicitError','release_date','Rebuild','Check','len_','sum_','min_','max_','abs_','obj_','singleton', 'this', 'Path','Path2','PathFunc','FocusedSeq','FocusedError','ExprSymmetricAdapter','ProbeInto','Default','ChecksumError','StopIf', 'setglobalfullprinting','getglobalfullprinting','Byte','Short','Int','Long', ] + ["Int%s%s%s" % (n,us,bln) for n in (8,16,32,64) for us in "us" for bln in "bln"] + ["Int24ub","Int24ul","Int24sb","Int24sl"] + ["Float%s%s" % (n,bl) for n in (32,64) for bl in "bl"] + ["Single","Double"] construct-2.8.16/construct/version.py0000644000175000017500000000011113164134560021727 0ustar arkadiuszarkadiusz00000000000000version = (2,8,16) version_string = "2.8.16" release_date = "2017.10.01" construct-2.8.16/setup.cfg0000644000175000017500000000010313164134612017464 0ustar arkadiuszarkadiusz00000000000000[bdist_wheel] universal = 1 [egg_info] tag_build = tag_date = 0 construct-2.8.16/MANIFEST.in0000644000175000017500000000004313151124063017377 0ustar arkadiuszarkadiusz00000000000000include README.rst include LICENSE construct-2.8.16/construct.egg-info/0000755000175000017500000000000013164134612021367 5ustar arkadiuszarkadiusz00000000000000construct-2.8.16/construct.egg-info/dependency_links.txt0000644000175000017500000000000113164134612025435 0ustar arkadiuszarkadiusz00000000000000 construct-2.8.16/construct.egg-info/SOURCES.txt0000644000175000017500000000242413164134612023255 0ustar arkadiuszarkadiusz00000000000000LICENSE MANIFEST.in README.rst setup.cfg setup.py construct/__init__.py construct/core.py construct/debug.py construct/expr.py construct/version.py construct.egg-info/PKG-INFO construct.egg-info/SOURCES.txt construct.egg-info/dependency_links.txt construct.egg-info/top_level.txt construct/examples/__init__.py construct/examples/formats/__init__.py construct/examples/formats/data/__init__.py construct/examples/formats/data/cap.py construct/examples/formats/data/snoop.py construct/examples/formats/executable/__init__.py construct/examples/formats/executable/elf32.py construct/examples/formats/executable/pe32.py construct/examples/formats/filesystem/__init__.py construct/examples/formats/filesystem/ext2.py construct/examples/formats/filesystem/fat16.py construct/examples/formats/filesystem/mbr.py construct/examples/formats/graphics/__init__.py construct/examples/formats/graphics/bmp.py construct/examples/formats/graphics/emf.py construct/examples/formats/graphics/gif.py construct/examples/formats/graphics/png.py construct/examples/formats/graphics/wmf.py construct/examples/protocols/__init__.py construct/examples/protocols/ipstack.py construct/lib/__init__.py construct/lib/binary.py construct/lib/bitstream.py construct/lib/container.py construct/lib/hex.py construct/lib/py3compat.pyconstruct-2.8.16/construct.egg-info/top_level.txt0000644000175000017500000000001213164134612024112 0ustar arkadiuszarkadiusz00000000000000construct construct-2.8.16/construct.egg-info/PKG-INFO0000644000175000017500000001300713164134612022465 0ustar arkadiuszarkadiusz00000000000000Metadata-Version: 1.1 Name: construct Version: 2.8.16 Summary: A powerful declarative parser/builder for binary data Home-page: http://construct.readthedocs.org Author: Arkadiusz Bulski, Tomer Filiba, Corbin Simpson Author-email: arek.bulski@gmail.com, tomerfiliba@gmail.com, MostAwesomeDude@gmail.com License: MIT Description: Construct 2.8 ============= Construct is a powerful **declarative** parser (and builder) for binary data. Instead of writing *imperative code* to parse a piece of data, you declaratively define a *data structure* that describes your data. As this data structure is not code, you can use it in one direction to *parse* data into Pythonic objects, and in the other direction, to *build* objects into binary data. The library provides both simple, atomic constructs (such as integers of various sizes), as well as composite ones which allow you form hierarchical and sequential structures of increasing complexity. Construct features **bit and byte granularity**, easy debugging and testing, an **easy-to-extend subclass system**, and lots of primitive constructs to make your work easier: * Fields: raw bytes or numerical types * Structs and Sequences: combine simpler constructs into more complex ones * Bitwise: splitting bytes into bit-grained fields * Adapters: change how data is represented * Arrays/Ranges: duplicate constructs * Meta-constructs: use the context (history) to compute the size of data * If/Switch: branch the computational path based on the context * On-demand (lazy) parsing: read and parse only what you require * Pointers: jump from here to there in the data stream Example --------- A ``Struct`` is a collection of ordered, named fields:: >>> format = Struct( ... "signature" / Const(b"BMP"), ... "width" / Int8ub, ... "height" / Int8ub, ... "pixels" / Array(this.width * this.height, Byte), ... ) >>> format.build(dict(width=3,height=2,pixels=[7,8,9,11,12,13])) b'BMP\x03\x02\x07\x08\t\x0b\x0c\r' >>> format.parse(b'BMP\x03\x02\x07\x08\t\x0b\x0c\r') Container(signature=b'BMP')(width=3)(height=2)(pixels=[7, 8, 9, 11, 12, 13]) A ``Sequence`` is a collection of ordered fields, and differs from a ``Range`` in that latter is homogenous:: >>> format = PascalString(Byte, encoding="utf8") >> GreedyRange(Byte) >>> format.build([u"lalaland", [255,1,2]]) b'\nlalaland\xff\x01\x02' >>> format.parse(b"\x004361789432197") ['', [52, 51, 54, 49, 55, 56, 57, 52, 51, 50, 49, 57, 55]] See more examples of `file formats `_ and `network protocols `_ in the repository. Sticky -------- Version 2.5.5 is legacy. If you are maintaining a project that depended on this library for a long time, you should probably use this version. This branch is not actively maintained, not even bugfixes. Version 2.8 was released on September, 2016. There are significant API and implementation changes. Fields are now name-less and operators / >> [] are used to create Structs Sequences and Ranges. Most classes changed interface and behavior. You should read entire documentation again. Development and support ------------------------- Please use the `github issues `_ to ask general questions, make feature requests, report issues and bugs, and to send in patches. Good quality extensions to test suite are highly welcomed. There is also a `mailing list `_ that was used for years but github issues should be preffered. Main documentation is at `construct.readthedocs.org `_, where you can find all kinds of examples. Source is on `github `_. Releases are available on `pypi `_. `Construct3 `_ is a different project. It is a rewrite from scratch and belongs to another (previous) developer, it diverged from this project years ago. As far as I can tell, it was never released and abandoned. Requirements -------------- Construct should run on any Python 2.7 3.3 3.4 3.5 3.6 and pypy pypy3 implementation. Best should be 3.6 because it supports ordered keyword arguments which comes handy when declaring Struct members or crafting Containers. Keywords: construct,declarative,data structure,binary,parser,builder,pack,unpack Platform: POSIX Platform: Windows Classifier: Development Status :: 5 - Production/Stable Classifier: License :: OSI Approved :: MIT License Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: Implementation :: PyPy Provides: construct construct-2.8.16/README.rst0000644000175000017500000001007713151137562017350 0ustar arkadiuszarkadiusz00000000000000Construct 2.8 ============= Construct is a powerful **declarative** parser (and builder) for binary data. Instead of writing *imperative code* to parse a piece of data, you declaratively define a *data structure* that describes your data. As this data structure is not code, you can use it in one direction to *parse* data into Pythonic objects, and in the other direction, to *build* objects into binary data. The library provides both simple, atomic constructs (such as integers of various sizes), as well as composite ones which allow you form hierarchical and sequential structures of increasing complexity. Construct features **bit and byte granularity**, easy debugging and testing, an **easy-to-extend subclass system**, and lots of primitive constructs to make your work easier: * Fields: raw bytes or numerical types * Structs and Sequences: combine simpler constructs into more complex ones * Bitwise: splitting bytes into bit-grained fields * Adapters: change how data is represented * Arrays/Ranges: duplicate constructs * Meta-constructs: use the context (history) to compute the size of data * If/Switch: branch the computational path based on the context * On-demand (lazy) parsing: read and parse only what you require * Pointers: jump from here to there in the data stream Example --------- A ``Struct`` is a collection of ordered, named fields:: >>> format = Struct( ... "signature" / Const(b"BMP"), ... "width" / Int8ub, ... "height" / Int8ub, ... "pixels" / Array(this.width * this.height, Byte), ... ) >>> format.build(dict(width=3,height=2,pixels=[7,8,9,11,12,13])) b'BMP\x03\x02\x07\x08\t\x0b\x0c\r' >>> format.parse(b'BMP\x03\x02\x07\x08\t\x0b\x0c\r') Container(signature=b'BMP')(width=3)(height=2)(pixels=[7, 8, 9, 11, 12, 13]) A ``Sequence`` is a collection of ordered fields, and differs from a ``Range`` in that latter is homogenous:: >>> format = PascalString(Byte, encoding="utf8") >> GreedyRange(Byte) >>> format.build([u"lalaland", [255,1,2]]) b'\nlalaland\xff\x01\x02' >>> format.parse(b"\x004361789432197") ['', [52, 51, 54, 49, 55, 56, 57, 52, 51, 50, 49, 57, 55]] See more examples of `file formats `_ and `network protocols `_ in the repository. Sticky -------- Version 2.5.5 is legacy. If you are maintaining a project that depended on this library for a long time, you should probably use this version. This branch is not actively maintained, not even bugfixes. Version 2.8 was released on September, 2016. There are significant API and implementation changes. Fields are now name-less and operators / >> [] are used to create Structs Sequences and Ranges. Most classes changed interface and behavior. You should read entire documentation again. Development and support ------------------------- Please use the `github issues `_ to ask general questions, make feature requests, report issues and bugs, and to send in patches. Good quality extensions to test suite are highly welcomed. There is also a `mailing list `_ that was used for years but github issues should be preffered. Main documentation is at `construct.readthedocs.org `_, where you can find all kinds of examples. Source is on `github `_. Releases are available on `pypi `_. `Construct3 `_ is a different project. It is a rewrite from scratch and belongs to another (previous) developer, it diverged from this project years ago. As far as I can tell, it was never released and abandoned. Requirements -------------- Construct should run on any Python 2.7 3.3 3.4 3.5 3.6 and pypy pypy3 implementation. Best should be 3.6 because it supports ordered keyword arguments which comes handy when declaring Struct members or crafting Containers.