././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1651864149.5370996 pidng-4.0.9/0000755000175000017500000000000000000000000010430 5ustar00pipi././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651466592.0 pidng-4.0.9/LICENSE0000644000175000017500000000205300000000000011435 0ustar00pipiMIT License Copyright (c) 2022 Csaba Nagy Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651466592.0 pidng-4.0.9/MANIFEST.in0000644000175000017500000000003100000000000012160 0ustar00pipirecursive-include src *.h././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1651864149.5370996 pidng-4.0.9/PKG-INFO0000644000175000017500000000432700000000000011533 0ustar00pipiMetadata-Version: 2.1 Name: pidng Version: 4.0.9 Summary: Python utility for creating Adobe DNG files from RAW image data. Home-page: https://github.com/schoolpost/PiDNG Author: Csaba Nagy License: UNKNOWN Description: PiDNG ========= ![](https://img.shields.io/badge/Version-4.0.9-green.svg) Create Adobe DNG RAW files using Python. **Features** ------------ - 8,10,12,14,16-bit precision - Lossless compression - DNG Tags ( extensible ) ### Works with any **Bayer RAW** Data including native support for **Raspberry Pi cameras**. - OV5467 ( Raspberry Pi Camera Module V1 ) - IMX219 ( Raspberry Pi Camera Module V2 ) - IMX477( Raspberry Pi High Quality Camera ) *** Instructions ------------ Requires: - Python3 - Numpy ### Install From PyPI: ``` python3 -mpip install PiDNG ``` Latest version from GitHub: ``` python3 -mpip install git+https://github.com/schoolpost/PiDNG.git ``` *** Credits ------------ Source referenced from: CanPi ( Jack ) | [color-matrices](https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=278828) Waveform80 | [picamera](https://github.com/waveform80/picamera) Krontech | [chronos-utils](https://github.com/krontech/chronos-utils) Andrew Baldwin | [MLVRawViewer](https://bitbucket.org/baldand/mlrawviewer) Platform: UNKNOWN Classifier: Development Status :: 4 - Beta Classifier: Topic :: Multimedia :: Graphics :: Capture :: Digital Camera Classifier: Topic :: Multimedia :: Graphics :: Graphics Conversion Classifier: Programming Language :: Python :: 3.6 Classifier: License :: OSI Approved :: MIT License Requires-Python: >=3.6 Description-Content-Type: text/markdown ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651864133.0 pidng-4.0.9/README.md0000644000175000017500000000224600000000000011713 0ustar00pipiPiDNG ========= ![](https://img.shields.io/badge/Version-4.0.9-green.svg) Create Adobe DNG RAW files using Python. **Features** ------------ - 8,10,12,14,16-bit precision - Lossless compression - DNG Tags ( extensible ) ### Works with any **Bayer RAW** Data including native support for **Raspberry Pi cameras**. - OV5467 ( Raspberry Pi Camera Module V1 ) - IMX219 ( Raspberry Pi Camera Module V2 ) - IMX477( Raspberry Pi High Quality Camera ) *** Instructions ------------ Requires: - Python3 - Numpy ### Install From PyPI: ``` python3 -mpip install PiDNG ``` Latest version from GitHub: ``` python3 -mpip install git+https://github.com/schoolpost/PiDNG.git ``` *** Credits ------------ Source referenced from: CanPi ( Jack ) | [color-matrices](https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=278828) Waveform80 | [picamera](https://github.com/waveform80/picamera) Krontech | [chronos-utils](https://github.com/krontech/chronos-utils) Andrew Baldwin | [MLVRawViewer](https://bitbucket.org/baldand/mlrawviewer) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1651864149.5370996 pidng-4.0.9/setup.cfg0000644000175000017500000000004600000000000012251 0ustar00pipi[egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651864134.0 pidng-4.0.9/setup.py0000644000175000017500000000222000000000000012136 0ustar00pipi# from distutils.core import setup, Extension from setuptools import setup, Extension, find_packages with open("README.md", "r", encoding="utf-8") as fh: long_description = fh.read() ljpeg92 = Extension('ljpegCompress', sources=[ "src/pidng/bitunpack.c", "src/pidng/liblj92/lj92.c"], extra_compile_args=['-std=gnu99'], extra_link_args=[]) setup( name="pidng", include_package_data=True, version="4.0.9", author="Csaba Nagy", description="Python utility for creating Adobe DNG files from RAW image data.", long_description=long_description, long_description_content_type="text/markdown", url="https://github.com/schoolpost/PiDNG", install_requires=[ 'numpy', ], classifiers=[ 'Development Status :: 4 - Beta', "Topic :: Multimedia :: Graphics :: Capture :: Digital Camera", "Topic :: Multimedia :: Graphics :: Graphics Conversion", "Programming Language :: Python :: 3.6", "License :: OSI Approved :: MIT License", ], ext_modules=[ljpeg92], package_dir={"": "src"}, packages=find_packages(where="src"), python_requires='>=3.6', )././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1651864149.5210998 pidng-4.0.9/src/0000755000175000017500000000000000000000000011217 5ustar00pipi././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1651864149.5290997 pidng-4.0.9/src/pidng/0000755000175000017500000000000000000000000012320 5ustar00pipi././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651466592.0 pidng-4.0.9/src/pidng/__init__.py0000644000175000017500000000000000000000000014417 0ustar00pipi././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651466592.0 pidng-4.0.9/src/pidng/bitunpack.c0000644000175000017500000000562600000000000014455 0ustar00pipi#include #include "liblj92/lj92.h" static PyObject* bitunpack_unpackljto16(PyObject* self, PyObject *args) { unsigned const char* input = 0; int inlen = 0; char* output; int outlen = 0; int outindex = 0; int outwrite = 0; int outskip = 0; unsigned const char* lin = 0; int linlen = 0; if (!PyArg_ParseTuple(args, "t#w#iiit#", &input, &inlen, &output, &outlen, &outindex, &outwrite, &outskip, &lin, &linlen)) return NULL; //printf("ljpeg decode inlen=%d,outlen=%d,outindex=%d,outwrite=%d,outskip=%d,linlen=%d\n",inlen,outlen,outindex,outwrite,outskip,linlen); int ret = 0; lj92 ljp; int iw,ih,ib; Py_BEGIN_ALLOW_THREADS; ret = lj92_open(&ljp,(uint8_t*)input,inlen,&iw,&ih,&ib); //printf("JPEG w:%d,h:%d,bits:%d\n",iw,ih,ib); if (ret==LJ92_ERROR_NONE) { if (linlen>0) { lj92_decode(ljp,(uint16_t*)(output+outindex),outwrite,outskip,(uint16_t*)lin,linlen); } else { lj92_decode(ljp,(uint16_t*)(output+outindex),outwrite,outskip,NULL,0); } //printf("Decoding complete\n"); } Py_END_ALLOW_THREADS; PyObject *stat = Py_BuildValue("I",ret); return stat; } static PyObject* bitunpack_pack16tolj(PyObject* self, PyObject *args) { unsigned const char* input = 0; int inlen = 0; int width = 0; int height = 0; int bitdepth = 0; int inindex = 0; int inread = 0; int inskip = 0; unsigned const char* delin = 0; int delinlen = 0; int ljPredictor = 6; if (!PyArg_ParseTuple(args, "y#iiiiiis#i", &input, &inlen, &width, &height, &bitdepth, &inindex, &inread, &inskip, &delin, &delinlen, &ljPredictor)) return NULL; if (width*height*sizeof(uint16_t) > inlen) return NULL; int ret = 0; uint8_t* encoded; int encodedLength; Py_BEGIN_ALLOW_THREADS; if (delinlen == 0) { ret = lj92_encode((uint16_t*)&input[inindex],width,height,bitdepth, inread,inskip,NULL,0,&encoded,&encodedLength,ljPredictor); } else { ret = lj92_encode((uint16_t*)&input[inindex],width,height,bitdepth, inread,inskip,(uint16_t*)delin,delinlen,&encoded,&encodedLength,ljPredictor); } if (ret != LJ92_ERROR_NONE) return NULL; Py_END_ALLOW_THREADS; PyObject* ba = PyByteArray_FromStringAndSize((char*)encoded,encodedLength); free(encoded); return ba; } static PyMethodDef methods[] = { { "unpackljto16", bitunpack_unpackljto16, METH_VARARGS, "Unpack a string of LJPEG values to 16bit values" }, { "pack16tolj", bitunpack_pack16tolj, METH_VARARGS, "Pack a string of 16bit values to LJPEG" }, { NULL, NULL, 0, NULL } }; static struct PyModuleDef ljpegCompress = { PyModuleDef_HEAD_INIT, "ljpegCompress", "", -1, methods }; PyMODINIT_FUNC PyInit_ljpegCompress(void) { return PyModule_Create(&ljpegCompress); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651787887.0 pidng-4.0.9/src/pidng/camdefs.py0000644000175000017500000002116400000000000014300 0ustar00pipiimport json import re import numpy as np from .dng import DNGTags, Tag from .defs import * class RaspberryPiCameraModels: Raspberry_Pi_Camera_V1 = "Raspberry Pi Camera V1" Raspberry_Pi_Camera_V2 = "Raspberry Pi Camera V2" Raspberry_Pi_High_Quality_Camera = "Raspberry Pi High Quality Camera" class BaseCameraModel(): def __init__(self) -> None: self.tags = DNGTags() self.model = "BaseCameraModel" def __settings__(self) -> None: pass # TODO def fromDict(dict : dict) -> None: pass # TODO def fromJson(jsn : str) -> None: parameters = json.loads(jsn) def __repr__(self) -> DNGTags: return self.tags def __str__(self) -> str: return str(self.model) class Picamera2Camera(BaseCameraModel): def __init__(self, fmt : dict, metadata: dict) -> None: super().__init__() self.model = "Picamera2Model" self.fmt = fmt self.metadata = metadata self.__settings__() def __settings__(self) -> None: width, height = self.fmt.get("size", (0,0)) fmt_str = self.fmt.get("format","").split("_")[0] bpp = int(re.search(r'\d+', fmt_str).group()) self.fmt["bpp"] = bpp black_levels = list() for val in self.metadata.get("SensorBlackLevels", (0)): black_levels.append((val >> (16 - bpp))) camera_calibration = [[1, 1], [0, 1], [0, 1], [0, 1], [1, 1], [0, 1], [0, 1], [0, 1], [1, 1]] color_gain_div = 10000 gain_r, gain_b = self.metadata.get("ColourGains",(color_gain_div, color_gain_div)) gain_matrix = np.array([[gain_r, 0, 0], [0, 1.0, 0], [0, 0, gain_b]]) gain_r = int(gain_r * color_gain_div) gain_b = int(gain_b * color_gain_div) as_shot_neutral = [[color_gain_div, gain_r], [color_gain_div, color_gain_div], [color_gain_div, gain_b]] ccm1 = list() ccm = self.metadata["ColourCorrectionMatrix"] # This maxtrix from http://www.brucelindbloom.com/index.html?Eqn_RGB_XYZ_Matrix.html rgb_to_xyz = np.array([[0.4124564, 0.3575761, 0.1804375], [0.2126729, 0.7151522, 0.0721750], [0.0193339, 0.1191920, 0.9503041]]) ccm_matrix = np.array(ccm).reshape((3, 3)) ccm = np.linalg.inv(rgb_to_xyz.dot(ccm_matrix).dot(gain_matrix)) for color in ccm.flatten().tolist(): ccm1.append((int(color*color_gain_div), color_gain_div)) ci1 = CalibrationIlluminant.D65 baseline_exp = 1 model = "PiDNG / PiCamera2" make = "RaspberryPi" profile_name = "PiDNG / PiCamera2 Profile" profile_embed = 3 self.orientation = 1 if "BGGR" in fmt_str: self.cfaPattern = CFAPattern.BGGR elif "GBRG" in fmt_str: self.cfaPattern = CFAPattern.GBRG elif "GRBG" in fmt_str: self.cfaPattern = CFAPattern.GRBG elif "RGGB" in fmt_str: self.cfaPattern = CFAPattern.RGGB exposure_time = int(1/(self.metadata["ExposureTime"] * 0.000001)) total_gain = self.metadata["AnalogueGain"] * self.metadata["DigitalGain"] iso = int(total_gain * 100) self.tags.set(Tag.PhotographicSensitivity, [iso]) self.tags.set(Tag.ExposureTime, [[1,exposure_time]]) self.tags.set(Tag.RawDataUniqueID, str(self.metadata["SensorTimestamp"]).encode("ascii")) self.tags.set(Tag.ImageWidth, width) self.tags.set(Tag.ImageLength, height) self.tags.set(Tag.TileWidth, width) self.tags.set(Tag.TileLength, height) self.tags.set(Tag.Orientation, self.orientation) self.tags.set(Tag.PhotometricInterpretation, PhotometricInterpretation.Color_Filter_Array) self.tags.set(Tag.SamplesPerPixel, 1) self.tags.set(Tag.BitsPerSample, bpp) self.tags.set(Tag.CFARepeatPatternDim, [2,2]) self.tags.set(Tag.CFAPattern, self.cfaPattern) self.tags.set(Tag.BlackLevelRepeatDim, [2,2]) self.tags.set(Tag.BlackLevel, black_levels) self.tags.set(Tag.WhiteLevel, ((1 << bpp) -1) ) self.tags.set(Tag.ColorMatrix1, ccm1) self.tags.set(Tag.CameraCalibration1, camera_calibration) self.tags.set(Tag.CameraCalibration2, camera_calibration) self.tags.set(Tag.CalibrationIlluminant1, ci1) self.tags.set(Tag.BaselineExposure, [[baseline_exp,1]]) self.tags.set(Tag.AsShotNeutral, as_shot_neutral) self.tags.set(Tag.Make, make) self.tags.set(Tag.Model, model) self.tags.set(Tag.ProfileName, profile_name) self.tags.set(Tag.ProfileEmbedPolicy, [profile_embed]) class RaspberryPiHqCamera(BaseCameraModel): def __init__(self, sensor_mode : int, cfaPattern=CFAPattern.BGGR, orientation=Orientation.Horizontal) -> None: super().__init__() self.model = RaspberryPiCameraModels.Raspberry_Pi_High_Quality_Camera self.mode = sensor_mode self.orientation = orientation self.cfaPattern = cfaPattern self.__settings__() def __settings__(self) -> None: width = None height = None bpp = None if self.mode == 1: width = 2028 height = 1080 if self.mode == 2: width = 2028 height = 1520 if self.mode == 3: width = 4056 height = 3040 if self.mode == 4: width = 1012 height = 760 if self.mode < 4: bpp = 12 else: bpp = 10 profile_name = "Repro 2_5D no LUT - D65 is really 5960K" profile_embed = 3 ccm1 = [[6759, 10000], [-2379, 10000], [751, 10000], [-4432, 10000], [13871, 10000], [5465, 10000], [-401, 10000], [1664, 10000], [7845, 10000]] ccm2 = [[5603, 10000], [-1351, 10000], [-600, 10000], [-2872, 10000], [11180, 10000], [2132, 10000], [600, 10000], [453, 10000], [5821, 10000]] fm1 = [[7889, 10000], [1273, 10000], [482, 10000], [2401, 10000], [9705, 10000], [-2106, 10000], [-26, 10000], [-4406, 10000], [12683, 10000]] fm2 = [[6591, 10000], [3034, 10000], [18, 10000], [1991, 10000], [10585, 10000], [-2575, 10000], [-493, 10000], [-919, 10000], [9663, 10000]] camera_calibration = [[1, 1], [0, 1], [0, 1], [0, 1], [1, 1], [0, 1], [0, 1], [0, 1], [1, 1]] gain_r = 2500 gain_b = 2000 as_shot_neutral = [[1000, gain_r], [1000, 1000], [1000, gain_b]] ci1 = CalibrationIlluminant.Standard_Light_A ci2 = CalibrationIlluminant.D65 model = " ".join(self.model.split(" ")[2:]) make = " ".join(self.model.split(" ")[:-3]) baseline_exp = 1 self.tags.set(Tag.ImageWidth, width) self.tags.set(Tag.ImageLength, height) self.tags.set(Tag.TileWidth, width) self.tags.set(Tag.TileLength, height) self.tags.set(Tag.Orientation, self.orientation) self.tags.set(Tag.PhotometricInterpretation, PhotometricInterpretation.Color_Filter_Array) self.tags.set(Tag.SamplesPerPixel, 1) self.tags.set(Tag.BitsPerSample, bpp) self.tags.set(Tag.CFARepeatPatternDim, [2,2]) self.tags.set(Tag.CFAPattern, self.cfaPattern) self.tags.set(Tag.BlackLevel, (4096 >> (16 - bpp))) self.tags.set(Tag.WhiteLevel, ((1 << bpp) -1) ) self.tags.set(Tag.ColorMatrix1, ccm1) self.tags.set(Tag.ColorMatrix2, ccm2) self.tags.set(Tag.ForwardMatrix1, fm1) self.tags.set(Tag.ForwardMatrix2, fm2) self.tags.set(Tag.CameraCalibration1, camera_calibration) self.tags.set(Tag.CameraCalibration2, camera_calibration) self.tags.set(Tag.CalibrationIlluminant1, ci1) self.tags.set(Tag.CalibrationIlluminant2, ci2) self.tags.set(Tag.BaselineExposure, [[baseline_exp,1]]) self.tags.set(Tag.AsShotNeutral, as_shot_neutral) self.tags.set(Tag.Make, make) self.tags.set(Tag.Model, model) self.tags.set(Tag.ProfileName, profile_name) self.tags.set(Tag.ProfileEmbedPolicy, [profile_embed]) # TODO class RaspberryPiCameraV2(BaseCameraModel): pass # TODO class RaspberryPiCameraV1(BaseCameraModel): pass ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651864024.0 pidng-4.0.9/src/pidng/core.py0000644000175000017500000001554500000000000013634 0ustar00pipiimport warnings import os import numpy as np import types from .dng import Tag, dngIFD, dngTag, DNG, DNGTags from .defs import Compression, DNGVersion from .packing import * from .camdefs import BaseCameraModel class DNGBASE: def __init__(self) -> None: self.compress = None self.path = None self.tags = None self.filter = None def __data_condition__(self, data : np.ndarray) -> None: if data.dtype != np.uint16: raise Exception("RAW Data is not in correct format. Must be uint16_t Numpy Array. ") def __tags_condition__(self, tags : DNGTags) -> None: if not tags.get(Tag.ImageWidth): raise Exception("No width is defined in tags.") if not tags.get(Tag.ImageLength): raise Exception("No height is defined in tags.") if not tags.get(Tag.BitsPerSample): raise Exception("Bit per pixel is not defined.") def __unpack_pixels__(self, data : np.ndarray) -> np.ndarray: return data def __filter__(self, rawFrame: np.ndarray, filter : types.FunctionType) -> np.ndarray: if not filter: return rawFrame processed = filter(rawFrame) if not isinstance(processed, np.ndarray): raise TypeError("return value is not a valid numpy array!") elif processed.shape != rawFrame.shape: raise ValueError("return array does not have the same shape!") if processed.dtype != np.uint16: raise ValueError("array data type is invalid!") return processed def __process__(self, rawFrame : np.ndarray, tags: DNGTags, compress : bool) -> bytearray: width = tags.get(Tag.ImageWidth).rawValue[0] length = tags.get(Tag.ImageLength).rawValue[0] bpp = tags.get(Tag.BitsPerSample).rawValue[0] compression_scheme = Compression.LJ92 if compress else Compression.Uncompressed if compress: from ljpegCompress import pack16tolj tile = pack16tolj(rawFrame, int(width*2), int(length/2), bpp, 0, 0, 0, "", 6) else: if bpp == 8: tile = rawFrame.astype('uint8').tobytes() elif bpp == 10: tile = pack10(rawFrame).tobytes() elif bpp == 12: tile = pack12(rawFrame).tobytes() elif bpp == 14: tile = pack14(rawFrame).tobytes() elif bpp == 16: tile = rawFrame.tobytes() dngTemplate = DNG() dngTemplate.ImageDataStrips.append(tile) # set up the FULL IFD mainIFD = dngIFD() mainTagStripOffset = dngTag( Tag.TileOffsets, [0 for tile in dngTemplate.ImageDataStrips]) mainIFD.tags.append(mainTagStripOffset) mainIFD.tags.append(dngTag(Tag.NewSubfileType, [0])) mainIFD.tags.append(dngTag(Tag.TileByteCounts, [len( tile) for tile in dngTemplate.ImageDataStrips])) mainIFD.tags.append(dngTag(Tag.Compression, [compression_scheme])) mainIFD.tags.append(dngTag(Tag.Software, "PiDNG")) mainIFD.tags.append(dngTag(Tag.DNGVersion, DNGVersion.V1_4)) mainIFD.tags.append(dngTag(Tag.DNGBackwardVersion, DNGVersion.V1_0)) for tag in tags.list(): try: mainIFD.tags.append(tag) except Exception as e: print("TAG Encoding Error!", e, tag) dngTemplate.IFDs.append(mainIFD) totalLength = dngTemplate.dataLen() mainTagStripOffset.setValue( [k for offset, k in dngTemplate.StripOffsets.items()]) buf = bytearray(totalLength) dngTemplate.setBuffer(buf) dngTemplate.write() return buf def options(self, tags : DNGTags, path : str, compress=False) -> None: self.__tags_condition__(tags) self.tags = tags self.compress = compress self.path = path def convert(self, image : np.ndarray, filename=""): if self.tags is None: raise Exception("Options have not been set!") # valdify incoming data self.__data_condition__(image) unpacked = self.__unpack_pixels__(image) filtered = self.__filter__(unpacked, self.filter) buf = self.__process__(filtered, self.tags, self.compress) file_output = False if len(filename) > 0: file_output = True if file_output: if not filename.endswith(".dng"): filename = filename + '.dng' outputDNG = os.path.join(self.path, filename) with open(outputDNG, "wb") as outfile: outfile.write(buf) return outputDNG else: return buf class RAW2DNG(DNGBASE): def __init__(self) -> None: super().__init__() class CAM2DNG(DNGBASE): def __init__(self, model : BaseCameraModel) -> None: super().__init__() self.model = model def options(self, path : str, compress=False) -> None: self.__tags_condition__(self.model.tags) self.tags = self.model.tags self.compress = compress self.path = path class RPICAM2DNG(CAM2DNG): def __data_condition__(self, data : np.ndarray) -> None: if data.dtype != np.uint8: warnings.warn("RAW Data is not in correct format. Already unpacked? ") def __unpack_pixels__(self, data : np.ndarray) -> np.ndarray: if data.dtype != np.uint8: return data width, height = self.model.fmt.get("size", (0,0)) stride = self.model.fmt.get("stride", 0) bpp = self.model.fmt.get("bpp", 8) # check to see if stored packed or unpacked format if "CSI2P" in self.model.fmt.get("format", ""): s_bpp = bpp # stored_bitperpixel else: s_bpp = 16 bytes_per_row = int(width * (s_bpp / 8)) data = data[:height, :bytes_per_row] if s_bpp == 10: data = data.astype(np.uint16) << 2 for byte in range(4): data[:, byte::5] |= ((data[:, 4::5] >> ((byte+1) * 2)) & 0b11) data = np.delete(data, np.s_[4::5], 1) elif s_bpp == 12: data = data.astype(np.uint16) shape = data.shape unpacked_data = np.zeros((shape[0], int(shape[1] / 3 * 2)), dtype=np.uint16) unpacked_data[:, ::2] = (data[:, ::3] << 4) + (data[:, 2::3] & 0x0F) unpacked_data[:, 1::2] = (data[:, 1::3] << 4) + ((data[:, 2::3] >> 4) & 0x0F) data = unpacked_data elif s_bpp == 16: data = np.ascontiguousarray(data).view(np.uint16) return data class PICAM2DNG(RPICAM2DNG): """For use within picamera2 library""" def options(self, compress=False) -> None: self.__tags_condition__(self.model.tags) self.tags = self.model.tags self.compress = compress self.path = "" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651466592.0 pidng-4.0.9/src/pidng/defs.py0000644000175000017500000000274300000000000013621 0ustar00pipi# Read more about tags and their definitions/values here: # https://exiftool.org/TagNames/EXIF.html # https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf class Compression: Uncompressed = 1 LJ92 = 7 #Lossless JPEG Lossy_JPEG = 34892 class PreviewColorSpace: Unknown = 0 Gray_Gamma_22 = 1 sRGB = 2 Adobe_RGB = 3 ProPhoto_RGB = 4 class Orientation: Horizontal = 1 MirrorH = 2 Rotate180 = 3 MirrorV = 4 class DNGVersion: V1_0 = [1, 0, 0, 0] V1_1 = [1, 1, 0, 0] V1_2 = [1, 2, 0, 0] V1_3 = [1, 3, 0, 0] V1_4 = [1, 4, 0, 0] V1_5 = [1, 5, 0, 0] V1_6 = [1, 6, 0, 0] class PhotometricInterpretation: WhiteIsZero = 0 BlackIsZero = 1 RGB = 2 Linear_Raw = 34892 Color_Filter_Array = 32803 class CFAPattern: BGGR = [2, 1, 1, 0] GBRG = [1, 2, 0, 1] GRBG = [1, 0, 2, 1] RGGB = [0, 1, 1, 2] class CalibrationIlluminant: Unknown = 0 Daylight = 1 Fluorescent = 2 Tungsten_Incandescent = 3 Flash = 4 Fine_Weather = 9 Cloudy = 10 Shade = 11 Daylight_Fluorescent = 12 Day_White_Fluorescent = 13 Cool_White_Fluorescent = 14 White_Fluorescent = 15 Warm_White_Fluorescent = 16 Standard_Light_A = 17 Standard_Light_B = 18 Standard_Light_C = 19 D55 = 20 D65 = 21 D75 = 22 D50 = 23 ISO_Studio_Tungsten = 24 Other = 255././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651466592.0 pidng-4.0.9/src/pidng/dng.py0000644000175000017500000003255100000000000013450 0ustar00pipiimport struct class Type: # TIFF Type Format = (Tag TYPE value, Size in bytes of one instance) Invalid = (0,0) # Should not be used Byte = (1,1) # 8-bit unsigned Ascii = (2,1) # 7-bit ASCII code Short = (3,2) # 16-bit unsigned Long = (4,4) # 32-bit unsigned Rational = (5,8) # 2 Longs, numerator:denominator Sbyte = (6,1) # 8 bit signed integer Undefined = (7,1) # 8 bit byte containing anything Sshort = (8,2) # 16 bit signed Slong = (9,4) # 32 bit signed Srational = (10,8) # 2 Slongs, numerator:denominator Float = (11,4) # 32bit float IEEE Double = (12,8) # 64bit double IEEE IFD = (13,4) # IFD (Same as Long) class Tag: Invalid = (0,Type.Invalid) NewSubfileType = (254,Type.Long) ImageWidth = (256,Type.Long) ImageLength = (257,Type.Long) BitsPerSample = (258,Type.Short) Compression = (259,Type.Short) PhotometricInterpretation = (262,Type.Short) FillOrder = (266,Type.Short) ImageDescription = (270,Type.Ascii) Make = (271,Type.Ascii) Model = (272,Type.Ascii) StripOffsets = (273,Type.Long) Orientation = (274,Type.Short) SamplesPerPixel = (277,Type.Short) RowsPerStrip = (278,Type.Short) StripByteCounts = (279,Type.Long) XResolution = (282,Type.Rational) YResolution = (283,Type.Rational) PlanarConfiguration = (284,Type.Short) ResolutionUnit = (296,Type.Short) Software = (305,Type.Ascii) DateTime = (306,Type.Ascii) Artist = (315,Type.Ascii) Predictor = (317,Type.Short) TileWidth = (322,Type.Short) TileLength = (323,Type.Short) TileOffsets = (324,Type.Long) TileByteCounts = (325,Type.Long) SubIFD = (330,Type.IFD) XMP_Metadata = (700,Type.Undefined) CFARepeatPatternDim = (33421,Type.Short) CFAPattern = (33422,Type.Byte) Copyright = (33432,Type.Ascii) ExposureTime = (33434,Type.Rational) FNumber = (33437,Type.Rational) EXIF_IFD = (34665,Type.IFD) ExposureProgram = (34850,Type.Short) PhotographicSensitivity = (34855,Type.Short) SensitivityType = (34864,Type.Short) ExifVersion = (36864,Type.Undefined) DateTimeOriginal = (36867,Type.Ascii) ShutterSpeedValue = (37377,Type.Srational) ApertureValue = (37378,Type.Rational) ExposureBiasValue = (37380,Type.Srational) MaxApertureValue = (37381,Type.Rational) SubjectDistance = (37382,Type.Rational) MeteringMode = (37383,Type.Short) Flash = (37385,Type.Short) FocalLength = (37386,Type.Rational) TIFF_EP_StandardID = (37398,Type.Byte) SubsecTime = (37520,Type.Ascii) SubsecTimeOriginal = (37521,Type.Ascii) FocalPlaneXResolution = (41486,Type.Rational) FocalPlaneYResolution = (41487,Type.Rational) FocalPlaneResolutionUnit = (41488,Type.Short) FocalLengthIn35mmFilm = (41989,Type.Short) EXIFPhotoBodySerialNumber = (42033,Type.Ascii) EXIFPhotoLensModel = (42036,Type.Ascii) DNGVersion = (50706,Type.Byte) DNGBackwardVersion = (50707,Type.Byte) UniqueCameraModel = (50708,Type.Ascii) CFAPlaneColor = (50710,Type.Byte) CFALayout = (50711,Type.Short) LinearizationTable = (50712,Type.Short) BlackLevelRepeatDim = (50713,Type.Short) BlackLevel = (50714,Type.Short) WhiteLevel = (50717,Type.Short) DefaultScale = (50718,Type.Rational) DefaultCropOrigin = (50719,Type.Long) DefaultCropSize = (50720,Type.Long) ColorMatrix1 = (50721,Type.Srational) ColorMatrix2 = (50722,Type.Srational) CameraCalibration1 = (50723,Type.Srational) CameraCalibration2 = (50724,Type.Srational) AnalogBalance = (50727,Type.Rational) AsShotNeutral = (50728,Type.Rational) BaselineExposure = (50730,Type.Srational) BaselineNoise = (50731,Type.Rational) BaselineSharpness = (50732,Type.Rational) BayerGreenSplit = (50733,Type.Long) LinearResponseLimit = (50734,Type.Rational) CameraSerialNumber = (50735,Type.Ascii) AntiAliasStrength = (50738,Type.Rational) ShadowScale = (50739,Type.Rational) DNGPrivateData = (50740,Type.Byte) MakerNoteSafety = (50741,Type.Short) CalibrationIlluminant1 = (50778,Type.Short) CalibrationIlluminant2 = (50779,Type.Short) BestQualityScale = (50780,Type.Rational) RawDataUniqueID = (50781,Type.Byte) ActiveArea = (50829,Type.Long) CameraCalibrationSignature = (50931,Type.Ascii) ProfileCalibrationSignature = (50932,Type.Ascii) NoiseReductionApplied = (50935,Type.Rational) ProfileName = (50936,Type.Ascii) ProfileHueSatMapDims = (50937,Type.Long) ProfileHueSatMapData1 = (50938,Type.Float) ProfileHueSatMapData2 = (50939,Type.Float) ProfileToneCurve = (50940,Type.Float) ProfileEmbedPolicy = (50941,Type.Long) ForwardMatrix1 = (50964, Type.Srational) ForwardMatrix2 = (50965, Type.Srational) PreviewApplicationName = (50966,Type.Ascii) PreviewApplicationVersion = (50967,Type.Ascii) PreviewSettingsDigest = (50969,Type.Byte) PreviewColorSpace = (50970,Type.Long) PreviewDateTime = (50971,Type.Ascii) NoiseProfile = (51041,Type.Double) TimeCodes = (51043,Type.Byte) FrameRate = (51044,Type.Srational) OpcodeList1 = (51008,Type.Undefined) OpcodeList2 = (51009,Type.Undefined) ReelName = (51081,Type.Ascii) BaselineExposureOffset = (51109,Type.Srational) # 1.4 Spec says rational but mentions negative values? DefaultBlackRender = (51110,Type.Long) NewRawImageDigest = (51111,Type.Byte) #CinemaDNG Tags FrameRate = (51044,Type.Srational) TimeCodes = (51043,Type.Byte) TStop = (51058,Type.Rational) ReelName = (51081,Type.Ascii) CameraLabel = (51105,Type.Ascii) class DNGTags: def __init__(self): self.__tags__ = dict() def set(self, tag : Tag, value): if isinstance(value, int): self.__tags__[tag] = dngTag(tag, [value]) else: self.__tags__[tag] = dngTag(tag, value) def get(self, tag): try: return self.__tags__[tag] except KeyError: return None def list(self): l = list() for k, v in self.__tags__.items(): l.append(v) return l class dngHeader(object): def __init__(self): self.IFDOffset = 8 def raw(self): return struct.pack(" #include #include #include #include "lj92.h" typedef uint8_t u8; typedef uint16_t u16; typedef uint32_t u32; //#define SLOW_HUFF //#define DEBUG typedef struct _ljp { u8* data; u8* dataend; int datalen; int scanstart; int ix; int x; // Width int y; // Height int bits; // Bit depth int writelen; // Write rows this long int skiplen; // Skip this many values after each row u16* linearize; // Linearization table int linlen; int sssshist[16]; // Huffman table - only one supported, and probably needed #ifdef SLOW_HUFF int* maxcode; int* mincode; int* valptr; u8* huffval; int* huffsize; int* huffcode; #else u16* hufflut; int huffbits; #endif // Parse state int cnt; u32 b; u16* image; u16* rowcache; u16* outrow[2]; } ljp; static int find(ljp* self) { int ix = self->ix; u8* data = self->data; while (data[ix] != 0xFF && ix<(self->datalen-1)) { ix += 1; } ix += 2; if (ix>=self->datalen) return -1; self->ix = ix; return data[ix-1]; } #define BEH(ptr) ((((int)(*&ptr))<<8)|(*(&ptr+1))) static int parseHuff(ljp* self) { int ret = LJ92_ERROR_CORRUPT; u8* huffhead = &self->data[self->ix]; // xstruct.unpack('>HB16B',self.data[self.ix:self.ix+19]) u8* bits = &huffhead[2]; bits[0] = 0; // Because table starts from 1 int hufflen = BEH(huffhead[0]); if ((self->ix + hufflen) >= self->datalen) return ret; #ifdef SLOW_HUFF u8* huffval = calloc(hufflen - 19,sizeof(u8)); if (huffval == NULL) return LJ92_ERROR_NO_MEMORY; self->huffval = huffval; for (int hix=0;hix<(hufflen-19);hix++) { huffval[hix] = self->data[self->ix+19+hix]; #ifdef DEBUG printf("huffval[%d]=%d\n",hix,huffval[hix]); #endif } self->ix += hufflen; // Generate huffman table int k = 0; int i = 1; int j = 1; int huffsize_needed = 1; // First calculate how long huffsize needs to be while (i<=16) { while (j<=bits[i]) { huffsize_needed++; k = k+1; j = j+1; } i = i+1; j = 1; } // Now allocate and do it int* huffsize = calloc(huffsize_needed,sizeof(int)); if (huffsize == NULL) return LJ92_ERROR_NO_MEMORY; self->huffsize = huffsize; k = 0; i = 1; j = 1; // First calculate how long huffsize needs to be int hsix = 0; while (i<=16) { while (j<=bits[i]) { huffsize[hsix++] = i; k = k+1; j = j+1; } i = i+1; j = 1; } huffsize[hsix++] = 0; // Calculate the size of huffcode array int huffcode_needed = 0; k = 0; int code = 0; int si = huffsize[0]; while (1) { while (huffsize[k] == si) { huffcode_needed++; code = code+1; k = k+1; } if (huffsize[k] == 0) break; while (huffsize[k] != si) { code = code << 1; si = si + 1; } } // Now fill it int* huffcode = calloc(huffcode_needed,sizeof(int)); if (huffcode == NULL) return LJ92_ERROR_NO_MEMORY; self->huffcode = huffcode; int hcix = 0; k = 0; code = 0; si = huffsize[0]; while (1) { while (huffsize[k] == si) { huffcode[hcix++] = code; code = code+1; k = k+1; } if (huffsize[k] == 0) break; while (huffsize[k] != si) { code = code << 1; si = si + 1; } } i = 0; j = 0; int* maxcode = calloc(17,sizeof(int)); if (maxcode == NULL) return LJ92_ERROR_NO_MEMORY; self->maxcode = maxcode; int* mincode = calloc(17,sizeof(int)); if (mincode == NULL) return LJ92_ERROR_NO_MEMORY; self->mincode = mincode; int* valptr = calloc(17,sizeof(int)); if (valptr == NULL) return LJ92_ERROR_NO_MEMORY; self->valptr = valptr; while (1) { while (1) { i++; if (i>16) break; if (bits[i]!=0) break; maxcode[i] = -1; } if (i>16) break; valptr[i] = j; mincode[i] = huffcode[j]; j = j+bits[i]-1; maxcode[i] = huffcode[j]; j++; } free(huffsize); self->huffsize = NULL; free(huffcode); self->huffcode = NULL; ret = LJ92_ERROR_NONE; #else /* Calculate huffman direct lut */ // How many bits in the table - find highest entry u8* huffvals = &self->data[self->ix+19]; int maxbits = 16; while (maxbits>0) { if (bits[maxbits]) break; maxbits--; } self->huffbits = maxbits; /* Now fill the lut */ u16* hufflut = malloc((1<hufflut = hufflut; int i = 0; int hv = 0; int rv = 0; int vl = 0; // i int hcode; int bitsused = 1; #ifdef DEBUG printf("%04x:%x:%d:%x\n",i,huffvals[hv],bitsused,1<<(maxbits-bitsused)); #endif while (i<1<maxbits) { break; // Done. Should never get here! } if (vl >= bits[bitsused]) { bitsused++; vl = 0; continue; } if (rv == 1 << (maxbits-bitsused)) { rv = 0; vl++; hv++; #ifdef DEBUG printf("%04x:%x:%d:%x\n",i,huffvals[hv],bitsused,1<<(maxbits-bitsused)); #endif continue; } hcode = huffvals[hv]; hufflut[i] = hcode<<8 | bitsused; //printf("%d %d %d\n",i,bitsused,hcode); i++; rv++; } ret = LJ92_ERROR_NONE; #endif return ret; } static int parseSof3(ljp* self) { if (self->ix+6 >= self->datalen) return LJ92_ERROR_CORRUPT; self->y = BEH(self->data[self->ix+3]); self->x = BEH(self->data[self->ix+5]); self->bits = self->data[self->ix+2]; self->ix += BEH(self->data[self->ix]); return LJ92_ERROR_NONE; } static int parseBlock(ljp* self,int marker) { self->ix += BEH(self->data[self->ix]); if (self->ix >= self->datalen) return LJ92_ERROR_CORRUPT; return LJ92_ERROR_NONE; } #ifdef SLOW_HUFF static int nextbit(ljp* self) { u32 b = self->b; if (self->cnt == 0) { u8* data = &self->data[self->ix]; u32 next = *data++; b = next; if (next == 0xff) { data++; self->ix++; } self->ix++; self->cnt = 8; } int bit = b >> 7; self->cnt--; self->b = (b << 1)&0xFF; return bit; } static int decode(ljp* self) { int i = 1; int code = nextbit(self); while (code > self->maxcode[i]) { i++; code = (code << 1) + nextbit(self); } int j = self->valptr[i]; j = j + code - self->mincode[i]; int value = self->huffval[j]; return value; } static int receive(ljp* self,int ssss) { int i = 0; int v = 0; while (i != ssss) { i++; v = (v<<1) + nextbit(self); } return v; } static int extend(ljp* self,int v,int t) { int vt = 1<<(t-1); if (v < vt) { vt = (-1 << t) + 1; v = v + vt; } return v; } #endif inline static int nextdiff(ljp* self, int Px) { #ifdef SLOW_HUFF int t = decode(self); int diff = receive(self,t); diff = extend(self,diff,t); //printf("%d %d %d %x\n",Px+diff,Px,diff,t);//,index,usedbits); #else u32 b = self->b; int cnt = self->cnt; int huffbits = self->huffbits; int ix = self->ix; int next; while (cnt < huffbits) { next = *(u16*)&self->data[ix]; int one = next&0xFF; int two = next>>8; b = (b<<16)|(one<<8)|two; cnt += 16; ix += 2; if (one==0xFF) { //printf("%x %x %x %x %d\n",one,two,b,b>>8,cnt); b >>= 8; cnt -= 8; } else if (two==0xFF) ix++; } int index = b >> (cnt - huffbits); u16 ssssused = self->hufflut[index]; int usedbits = ssssused&0xFF; int t = ssssused>>8; self->sssshist[t]++; cnt -= usedbits; int keepbitsmask = (1 << cnt)-1; b &= keepbitsmask; while (cnt < t) { next = *(u16*)&self->data[ix]; int one = next&0xFF; int two = next>>8; b = (b<<16)|(one<<8)|two; cnt += 16; ix += 2; if (one==0xFF) { b >>= 8; cnt -= 8; } else if (two==0xFF) ix++; } cnt -= t; int diff = b >> cnt; int vt = 1<<(t-1); if (diff < vt) { vt = (-1 << t) + 1; diff += vt; } keepbitsmask = (1 << cnt)-1; self->b = b & keepbitsmask; self->cnt = cnt; self->ix = ix; //printf("%d %d\n",t,diff); //printf("%d %d %d %x %x %d\n",Px+diff,Px,diff,t,index,usedbits); #ifdef DEBUG #endif #endif return diff; } static int parsePred6(ljp* self) { int ret = LJ92_ERROR_CORRUPT; self->ix = self->scanstart; //int compcount = self->data[self->ix+2]; self->ix += BEH(self->data[self->ix]); self->cnt = 0; self->b = 0; int write = self->writelen; // Now need to decode huffman coded values int c = 0; int pixels = self->y * self->x; u16* out = self->image; u16* temprow; u16* thisrow = self->outrow[0]; u16* lastrow = self->outrow[1]; // First pixel predicted from base value int diff; int Px; int col = 0; int row = 0; int left = 0; int linear; // First pixel diff = nextdiff(self,0); Px = 1 << (self->bits-1); left = Px + diff; if (self->linearize) linear = self->linearize[left]; else linear = left; thisrow[col++] = left; out[c++] = linear; if (self->ix >= self->datalen) return ret; --write; int rowcount = self->x-1; while (rowcount--) { diff = nextdiff(self,0); Px = left; left = Px + diff; if (self->linearize) linear = self->linearize[left]; else linear = left; thisrow[col++] = left; out[c++] = linear; //printf("%d %d %d %d %x\n",col-1,diff,left,thisrow[col-1],&thisrow[col-1]); if (self->ix >= self->datalen) return ret; if (--write==0) { out += self->skiplen; write = self->writelen; } } temprow = lastrow; lastrow = thisrow; thisrow = temprow; row++; //printf("%x %x\n",thisrow,lastrow); while (clinearize) { if (left>self->linlen) return LJ92_ERROR_CORRUPT; linear = self->linearize[left]; } else linear = left; thisrow[col++] = left; //printf("%d %d %d %d\n",col,diff,left,lastrow[col]); out[c++] = linear; if (self->ix >= self->datalen) break; rowcount = self->x-1; if (--write==0) { out += self->skiplen; write = self->writelen; } while (rowcount--) { diff = nextdiff(self,0); Px = lastrow[col] + ((left - lastrow[col-1])>>1); left = Px + diff; //printf("%d %d %d %d %d %x\n",col,diff,left,lastrow[col],lastrow[col-1],&lastrow[col]); if (self->linearize) { if (left>self->linlen) return LJ92_ERROR_CORRUPT; linear = self->linearize[left]; } else linear = left; thisrow[col++] = left; out[c++] = linear; if (--write==0) { out += self->skiplen; write = self->writelen; } } temprow = lastrow; lastrow = thisrow; thisrow = temprow; if (self->ix >= self->datalen) break; } if (c >= pixels) ret = LJ92_ERROR_NONE; return ret; } static int parseScan(ljp* self) { int ret = LJ92_ERROR_CORRUPT; memset(self->sssshist,0,sizeof(self->sssshist)); self->ix = self->scanstart; int compcount = self->data[self->ix+2]; int pred = self->data[self->ix+3+2*compcount]; if (pred<0 || pred>7) return ret; if (pred==6) return parsePred6(self); // Fast path self->ix += BEH(self->data[self->ix]); self->cnt = 0; self->b = 0; int write = self->writelen; // Now need to decode huffman coded values int c = 0; int pixels = self->y * self->x; u16* out = self->image; u16* thisrow = self->outrow[0]; u16* lastrow = self->outrow[1]; // First pixel predicted from base value int diff; int Px; int col = 0; int row = 0; int left = 0; while (cbits-1); } else if (row==0) { Px = left; } else if (col==0) { Px = lastrow[col]; // Use value above for first pixel in row } else { switch (pred) { case 0: Px = 0; break; // No prediction... should not be used case 1: Px = left; break; case 2: Px = lastrow[col]; break; case 3: Px = lastrow[col-1];break; case 4: Px = left + lastrow[col] - lastrow[col-1];break; case 5: Px = left + ((lastrow[col] - lastrow[col-1])>>1);break; case 6: Px = lastrow[col] + ((left - lastrow[col-1])>>1);break; case 7: Px = (left + lastrow[col])>>1;break; } } diff = nextdiff(self,Px); left = Px + diff; //printf("%d %d %d\n",c,diff,left); int linear; if (self->linearize) { if (left>self->linlen) return LJ92_ERROR_CORRUPT; linear = self->linearize[left]; } else linear = left; thisrow[col] = left; out[c++] = linear; if (++col==self->x) { col = 0; row++; u16* temprow = lastrow; lastrow = thisrow; thisrow = temprow; } if (--write==0) { out += self->skiplen; write = self->writelen; } if (self->ix >= self->datalen+2) break; } if (c >= pixels) ret = LJ92_ERROR_NONE; /*for (int h=0;h<17;h++) { printf("ssss:%d=%d (%f)\n",h,self->sssshist[h],(float)self->sssshist[h]/(float)(pixels)); }*/ return ret; } static int parseImage(ljp* self) { int ret = LJ92_ERROR_NONE; while (1) { int nextMarker = find(self); if (nextMarker == 0xc4) ret = parseHuff(self); else if (nextMarker == 0xc3) ret = parseSof3(self); else if (nextMarker == 0xfe)// Comment ret = parseBlock(self,nextMarker); else if (nextMarker == 0xd9) // End of image break; else if (nextMarker == 0xda) { self->scanstart = self->ix; ret = LJ92_ERROR_NONE; break; } else if (nextMarker == -1) { ret = LJ92_ERROR_CORRUPT; break; } else ret = parseBlock(self,nextMarker); if (ret != LJ92_ERROR_NONE) break; } return ret; } static int findSoI(ljp* self) { int ret = LJ92_ERROR_CORRUPT; if (find(self)==0xd8) ret = parseImage(self); return ret; } static void free_memory(ljp* self) { #ifdef SLOW_HUFF free(self->maxcode); self->maxcode = NULL; free(self->mincode); self->mincode = NULL; free(self->valptr); self->valptr = NULL; free(self->huffval); self->huffval = NULL; free(self->huffsize); self->huffsize = NULL; free(self->huffcode); self->huffcode = NULL; #else free(self->hufflut); self->hufflut = NULL; #endif free(self->rowcache); self->rowcache = NULL; } int lj92_open(lj92* lj, uint8_t* data, int datalen, int* width,int* height, int* bitdepth) { ljp* self = (ljp*)calloc(sizeof(ljp),1); if (self==NULL) return LJ92_ERROR_NO_MEMORY; self->data = (u8*)data; self->dataend = self->data + datalen; self->datalen = datalen; int ret = findSoI(self); if (ret == LJ92_ERROR_NONE) { u16* rowcache = calloc(self->x * 2,sizeof(u16)); if (rowcache == NULL) ret = LJ92_ERROR_NO_MEMORY; else { self->rowcache = rowcache; self->outrow[0] = rowcache; self->outrow[1] = &rowcache[self->x]; } } if (ret != LJ92_ERROR_NONE) { // Failed, clean up *lj = NULL; free_memory(self); free(self); } else { *width = self->x; *height = self->y; *bitdepth = self->bits; *lj = self; } return ret; } int lj92_decode(lj92 lj, uint16_t* target,int writeLength, int skipLength, uint16_t* linearize,int linearizeLength) { int ret = LJ92_ERROR_NONE; ljp* self = lj; if (self == NULL) return LJ92_ERROR_BAD_HANDLE; self->image = target; self->writelen = writeLength; self->skiplen = skipLength; self->linearize = linearize; self->linlen = linearizeLength; ret = parseScan(self); return ret; } void lj92_close(lj92 lj) { ljp* self = lj; if (self != NULL) free_memory(self); free(self); } /* Encoder implementation */ typedef struct _lje { uint16_t* image; int width; int height; int bitdepth; int readLength; int skipLength; uint16_t* delinearize; int delinearizeLength; uint8_t* encoded; int encodedWritten; int encodedLength; int hist[17]; // SSSS frequency histogram int bits[17]; int huffval[17]; u16 huffenc[17]; u16 huffbits[17]; int huffsym[17]; int ljPredictor; } lje; int frequencyScan(lje* self) { // Scan through the tile using the standard type 6 prediction // Need to cache the previous 2 row in target coordinates because of tiling uint16_t* pixel = self->image; int pixcount = self->width*self->height; int scan = self->readLength; uint16_t* rowcache = (uint16_t*)calloc(1,self->width*4); uint16_t* rows[2]; rows[0] = rowcache; rows[1] = &rowcache[self->width]; int col = 0; int row = 0; int Px = 0; int32_t diff = 0; int maxval = (1 << self->bitdepth); while (pixcount--) { uint16_t p = *pixel; if (self->delinearize) { if (p>=self->delinearizeLength) { free(rowcache); return LJ92_ERROR_TOO_WIDE; } p = self->delinearize[p]; } if (p>=maxval) { free(rowcache); return LJ92_ERROR_TOO_WIDE; } rows[1][col] = p; if ((row == 0)&&(col == 0)) Px = 1 << (self->bitdepth-1); else if (row == 0) Px = rows[1][col-1]; else if (col == 0) Px = rows[0][col]; else Px = rows[0][col] + ((rows[1][col-1] - rows[0][col-1])>>1); diff = rows[1][col] - Px; int ssss = 32 - __builtin_clz(abs(diff)); if (diff==0) ssss=0; self->hist[ssss]++; //printf("%d %d %d %d %d %d\n",col,row,p,Px,diff,ssss); pixel++; scan--; col++; if (scan==0) { pixel += self->skipLength; scan = self->readLength; } if (col==self->width) { uint16_t* tmprow = rows[1]; rows[1] = rows[0]; rows[0] = tmprow; col=0; row++; } } #ifdef DEBUG int sort[17]; for (int h=0;h<17;h++) { sort[h] = h; printf("%d:%d\n",h,self->hist[h]); } #endif free(rowcache); return LJ92_ERROR_NONE; } void createEncodeTable(lje* self) { float freq[18]; int codesize[18]; int others[18]; // Calculate frequencies float totalpixels = self->width * self->height; for (int i=0;i<17;i++) { freq[i] = (float)(self->hist[i])/totalpixels; #ifdef DEBUG printf("%d:%f\n",i,freq[i]); #endif codesize[i] = 0; others[i] = -1; } codesize[17] = 0; others[17] = -1; freq[17] = 1.0f; float v1f,v2f; int v1,v2; while (1) { v1f=3.0f; v1=-1; for (int i=0;i<18;i++) { if ((freq[i]<=v1f) && (freq[i]>0.0f)) { v1f = freq[i]; v1 = i; } } #ifdef DEBUG printf("v1:%d,%f\n",v1,v1f); #endif v2f=3.0f; v2=-1; for (int i=0;i<18;i++) { if (i==v1) continue; if ((freq[i]0.0f)) { v2f = freq[i]; v2 = i; } } if (v2==-1) break; // Done freq[v1] += freq[v2]; freq[v2] = 0.0f; while (1) { codesize[v1]++; if (others[v1]==-1) break; v1 = others[v1]; } others[v1] = v2; while (1) { codesize[v2]++; if (others[v2]==-1) break; v2 = others[v2]; } } int* bits = self->bits; memset(bits,0,sizeof(self->bits)); for (int i=0;i<18;i++) { if (codesize[i]!=0) { bits[codesize[i]]++; } } #ifdef DEBUG for (int i=0;i<17;i++) { printf("bits:%d,%d,%d\n",i,bits[i],codesize[i]); } #endif int* huffval = self->huffval; int i=1; int k=0; int j; memset(huffval,0,sizeof(self->huffval)); while (i<=32) { j=0; while (j<17) { if (codesize[j]==i) { huffval[k++] = j; } j++; } i++; } #ifdef DEBUG for (i=0;i<17;i++) { printf("i=%d,huffval[i]=%x\n",i,huffval[i]); } #endif int maxbits = 16; while (maxbits>0) { if (bits[maxbits]) break; maxbits--; } u16* huffenc = self->huffenc; u16* huffbits = self->huffbits; int* huffsym = self->huffsym; memset(huffenc,0,sizeof(self->huffenc)); memset(huffbits,0,sizeof(self->huffbits)); memset(self->huffsym,0,sizeof(self->huffsym)); i = 0; int hv = 0; int rv = 0; int vl = 0; // i //int hcode; int bitsused = 1; int sym = 0; //printf("%04x:%x:%d:%x\n",i,huffvals[hv],bitsused,1<<(maxbits-bitsused)); while (i<1<maxbits) { break; // Done. Should never get here! } if (vl >= bits[bitsused]) { bitsused++; vl = 0; continue; } if (rv == 1 << (maxbits-bitsused)) { rv = 0; vl++; hv++; //printf("%04x:%x:%d:%x\n",i,huffvals[hv],bitsused,1<<(maxbits-bitsused)); continue; } huffbits[sym] = bitsused; huffenc[sym++] = i>>(maxbits-bitsused); //printf("%d %d %d\n",i,bitsused,hcode); i+= (1<<(maxbits-bitsused)); rv = 1<<(maxbits-bitsused); } for (i=0;i<17;i++) { if (huffbits[i]>0) { huffsym[huffval[i]] = i; } #ifdef DEBUG printf("huffval[%d]=%d,huffenc[%d]=%x,bits=%d\n",i,huffval[i],i,huffenc[i],huffbits[i]); #endif if (huffbits[i]>0) { huffsym[huffval[i]] = i; } } #ifdef DEBUG for (i=0;i<17;i++) { printf("huffsym[%d]=%d\n",i,huffsym[i]); } #endif } void writeHeader(lje* self) { int w = self->encodedWritten; uint8_t* e = self->encoded; e[w++] = 0xff; e[w++] = 0xd8; //SOI e[w++] = 0xff; e[w++] = 0xc3; //SOF3 // Write SOF e[w++] = 0x0; e[w++] = 11; //Lf, frame header length e[w++] = self->bitdepth; e[w++] = self->height>>8; e[w++] = self->height&0xFF; e[w++] = self->width>>8; e[w++] = self->width&0xFF; e[w++] = 1; // Components e[w++] = 0; // Component ID e[w++] = 0x11; // Component X/Y e[w++] = 0; // Unused (Quantisation) e[w++] = 0xff; e[w++] = 0xc4; //HUFF // Write HUFF int count = 0; for (int i=0;i<17;i++) { count += self->bits[i]; } e[w++] = 0x0; e[w++] = 17+2+count; //Lf, frame header length e[w++] = 0; // Table ID for (int i=1;i<17;i++) { e[w++] = self->bits[i]; } for (int i=0;ihuffval[i]; } e[w++] = 0xff; e[w++] = 0xda; //SCAN // Write SCAN e[w++] = 0x0; e[w++] = 8; //Ls, scan header length e[w++] = 1; // Components e[w++] = 0; // e[w++] = 0; // e[w++] = self->ljPredictor; // Predictor e[w++] = 0; // e[w++] = 0; // self->encodedWritten = w; } void writePost(lje* self) { int w = self->encodedWritten; uint8_t* e = self->encoded; e[w++] = 0xff; e[w++] = 0xd9; //EOI self->encodedWritten = w; } void writeBody(lje* self) { // Scan through the tile using the standard type 6 prediction // Need to cache the previous 2 row in target coordinates because of tiling uint16_t* pixel = self->image; int pixcount = self->width*self->height; int scan = self->readLength; uint16_t* rowcache = (uint16_t*)calloc(1,self->width*4); uint16_t* rows[2]; rows[0] = rowcache; rows[1] = &rowcache[self->width]; int ljPredictor = self->ljPredictor; int col = 0; int row = 0; int Px = 0; int32_t diff = 0; int bitcount = 0; uint8_t* out = self->encoded; int w = self->encodedWritten; uint8_t next = 0; uint8_t nextbits = 8; while (pixcount--) { uint16_t p = *pixel; if (self->delinearize) p = self->delinearize[p]; rows[1][col] = p; if ((row == 0)&&(col == 0)) Px = 1 << (self->bitdepth-1); else if (row == 0) Px = rows[1][col-1]; else if (col == 0) Px = rows[0][col]; else if (ljPredictor == 1) Px = rows[1][col-1]; else if (ljPredictor == 2) Px = rows[0][col]; else if (ljPredictor == 3) Px = rows[0][col-1]; else if (ljPredictor == 4) Px = rows[1][col-1] + rows[0][col] - rows[0][col-1]; else if (ljPredictor == 5) Px = rows[1][col-1] + ((rows[0][col] - rows[0][col-1])>>1); else if (ljPredictor == 7) Px = ((rows[1][col-1] - rows[0][col-1])>>1); else if (ljPredictor == 6) Px = rows[0][col] + ((rows[1][col-1] - rows[0][col-1])>>1); else Px = rows[0][col] + ((rows[1][col-1] - rows[0][col-1])>>1); diff = rows[1][col] - Px % 65535; int ssss = 32 - __builtin_clz(abs(diff)); if (diff==0) ssss=0; //printf("%d %d %d %d %d\n",col,row,Px,diff,ssss); // Write the huffman code for the ssss value int huffcode = self->huffsym[ssss]; int huffenc = self->huffenc[huffcode]; int huffbits = self->huffbits[huffcode]; bitcount += huffbits + ssss; int vt = ssss>0?(1<<(ssss-1)):0; //printf("%d %d %d %d\n",rows[1][col],Px,diff,Px+diff); #ifdef DEBUG #endif if (diff < vt) diff += (1 << (ssss))-1; // Write the ssss while (huffbits>0) { int usebits = huffbits>nextbits?nextbits:huffbits; // Add top usebits from huffval to next usebits of nextbits int tophuff = huffenc >> (huffbits - usebits); next |= (tophuff << (nextbits-usebits)); nextbits -= usebits; huffbits -= usebits; huffenc &= (1<0) { int usebits = ssss>nextbits?nextbits:ssss; // Add top usebits from huffval to next usebits of nextbits int tophuff = diff >> (ssss - usebits); next |= (tophuff << (nextbits-usebits)); nextbits -= usebits; ssss -= usebits; diff &= (1<skipLength; scan = self->readLength; } if (col==self->width) { uint16_t* tmprow = rows[1]; rows[1] = rows[0]; rows[0] = tmprow; col=0; row++; } } // Flush the final bits if (nextbits<8) { out[w++] = next; if (next==0xff) out[w++] = 0x0; } #ifdef DEBUG int sort[17]; for (int h=0;h<17;h++) { sort[h] = h; printf("%d:%d\n",h,self->hist[h]); } printf("Total bytes: %d\n",bitcount>>3); #endif free(rowcache); self->encodedWritten = w; } /* Encoder * Read tile from an image and encode in one shot * Return the encoded data */ int lj92_encode(uint16_t* image, int width, int height, int bitdepth, int readLength, int skipLength, uint16_t* delinearize,int delinearizeLength, uint8_t** encoded, int* encodedLength, int ljPredictor) { int ret = LJ92_ERROR_NONE; lje* self = (lje*)calloc(sizeof(lje),1); if (self==NULL) return LJ92_ERROR_NO_MEMORY; self->image = image; self->width = width; self->height = height; self->bitdepth = bitdepth; self->readLength = readLength; self->skipLength = skipLength; self->delinearize = delinearize; self->delinearizeLength = delinearizeLength; self->encodedLength = width*height*3+200; self->encoded = malloc(self->encodedLength); self->ljPredictor = ljPredictor; if (self->encoded==NULL) { free(self); return LJ92_ERROR_NO_MEMORY; } // Scan through data to gather frequencies of ssss prefixes ret = frequencyScan(self); if (ret != LJ92_ERROR_NONE) { free(self->encoded); free(self); return ret; } // Create encoded table based on frequencies createEncodeTable(self); // Write JPEG head and scan header writeHeader(self); // Scan through and do the compression writeBody(self); // Finish writePost(self); #ifdef DEBUG printf("written:%d\n",self->encodedWritten); #endif self->encoded = realloc(self->encoded,self->encodedWritten); self->encodedLength = self->encodedWritten; *encoded = self->encoded; *encodedLength = self->encodedLength; free(self); return ret; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651466592.0 pidng-4.0.9/src/pidng/liblj92/lj92.h0000644000175000017500000000542300000000000014524 0ustar00pipi/* lj92.h (c) Andrew Baldwin 2014 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ #ifndef LJ92_H #define LJ92_H enum LJ92_ERRORS { LJ92_ERROR_NONE = 0, LJ92_ERROR_CORRUPT = -1, LJ92_ERROR_NO_MEMORY = -2, LJ92_ERROR_BAD_HANDLE = -3, LJ92_ERROR_TOO_WIDE = -4, }; typedef struct _ljp* lj92; /* Parse a lossless JPEG (1992) structure returning * - a handle that can be used to decode the data * - width/height/bitdepth of the data * Returns status code. * If status == LJ92_ERROR_NONE, handle must be closed with lj92_close */ int lj92_open(lj92* lj, // Return handle here uint8_t* data,int datalen, // The encoded data int* width,int* height,int* bitdepth); // Width, height and bitdepth /* Release a decoder object */ void lj92_close(lj92 lj); /* * Decode previously opened lossless JPEG (1992) into a 2D tile of memory * Starting at target, write writeLength 16bit values, then skip 16bit skipLength value before writing again * If linearize is not NULL, use table at linearize to convert data values from output value to target value * Data is only correct if LJ92_ERROR_NONE is returned */ int lj92_decode(lj92 lj, uint16_t* target, int writeLength, int skipLength, // The image is written to target as a tile uint16_t* linearize, int linearizeLength); // If not null, linearize the data using this table /* * Encode a grayscale image supplied as 16bit values within the given bitdepth * Read from tile in the image * Apply delinearization if given * Return the encoded lossless JPEG stream */ int lj92_encode(uint16_t* image, int width, int height, int bitdepth, int readLength, int skipLength, uint16_t* delinearize,int delinearizeLength, uint8_t** encoded, int* encodedLength, int ljPredictor); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651466592.0 pidng-4.0.9/src/pidng/packing.py0000644000175000017500000000317200000000000014311 0ustar00pipiimport numpy as np def pack10(data : np.ndarray) -> np.ndarray: out = np.zeros((data.shape[0], int(data.shape[1]*(1.25))), dtype=np.uint8) out[:, ::5] = data[:, ::4] >> 2 out[:, 1::5] = ((data[:, ::4] & 0b0000000000000011) << 6) out[:, 1::5] += data[:, 1::4] >> 4 out[:, 2::5] = ((data[:, 1::4] & 0b0000000000001111) << 4) out[:, 2::5] += data[:, 2::4] >> 6 out[:, 3::5] = ((data[:, 2::4] & 0b0000000000111111) << 2) out[:, 3::5] += data[:, 3::4] >> 8 out[:, 4::5] = data[:, 3::4] & 0b0000000011111111 return out def pack12(data : np.ndarray) -> np.ndarray: out = np.zeros((data.shape[0], int(data.shape[1]*(1.5))), dtype=np.uint8) out[:, ::3] = data[:, ::2] >> 4 out[:, 1::3] = ((data[:, ::2] & 0b0000000000001111) << 4) out[:, 1::3] += data[:, 1::2] >> 8 out[:, 2::3] = data[:, 1::2] & 0b0000001111111111 return out def pack14(data : np.ndarray) -> np.ndarray: out = np.zeros((data.shape[0], int(data.shape[1]*(1.75))), dtype=np.uint8) out[:, ::7] = data[:, ::6] >> 6 out[:, 1::7] = ((data[:, ::6] & 0b0000000000000011) << 6) out[:, 1::7] += data[:, 1::6] >> 8 out[:, 2::7] = ((data[:, 1::6] & 0b0000000000001111) << 4) out[:, 2::7] += data[:, 2::6] >> 6 out[:, 3::7] = ((data[:, 2::6] & 0b0000000000111111) << 2) out[:, 3::7] += data[:, 3::6] >> 8 out[:, 4::7] = ((data[:, 3::6] & 0b0000000000001111) << 4) out[:, 4::7] += data[:, 4::6] >> 6 out[:, 5::7] = ((data[:, 4::6] & 0b0000000000111111) << 2) out[:, 5::7] += data[:, 5::6] >> 8 out[:, 6::7] = data[:, 5::6] & 0b0000000011111111 return out././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1651864149.5330997 pidng-4.0.9/src/pidng.egg-info/0000755000175000017500000000000000000000000014012 5ustar00pipi././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651864149.0 pidng-4.0.9/src/pidng.egg-info/PKG-INFO0000644000175000017500000000432700000000000015115 0ustar00pipiMetadata-Version: 2.1 Name: pidng Version: 4.0.9 Summary: Python utility for creating Adobe DNG files from RAW image data. Home-page: https://github.com/schoolpost/PiDNG Author: Csaba Nagy License: UNKNOWN Description: PiDNG ========= ![](https://img.shields.io/badge/Version-4.0.9-green.svg) Create Adobe DNG RAW files using Python. **Features** ------------ - 8,10,12,14,16-bit precision - Lossless compression - DNG Tags ( extensible ) ### Works with any **Bayer RAW** Data including native support for **Raspberry Pi cameras**. - OV5467 ( Raspberry Pi Camera Module V1 ) - IMX219 ( Raspberry Pi Camera Module V2 ) - IMX477( Raspberry Pi High Quality Camera ) *** Instructions ------------ Requires: - Python3 - Numpy ### Install From PyPI: ``` python3 -mpip install PiDNG ``` Latest version from GitHub: ``` python3 -mpip install git+https://github.com/schoolpost/PiDNG.git ``` *** Credits ------------ Source referenced from: CanPi ( Jack ) | [color-matrices](https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=278828) Waveform80 | [picamera](https://github.com/waveform80/picamera) Krontech | [chronos-utils](https://github.com/krontech/chronos-utils) Andrew Baldwin | [MLVRawViewer](https://bitbucket.org/baldand/mlrawviewer) Platform: UNKNOWN Classifier: Development Status :: 4 - Beta Classifier: Topic :: Multimedia :: Graphics :: Capture :: Digital Camera Classifier: Topic :: Multimedia :: Graphics :: Graphics Conversion Classifier: Programming Language :: Python :: 3.6 Classifier: License :: OSI Approved :: MIT License Requires-Python: >=3.6 Description-Content-Type: text/markdown ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651864149.0 pidng-4.0.9/src/pidng.egg-info/SOURCES.txt0000644000175000017500000000063300000000000015700 0ustar00pipiLICENSE MANIFEST.in README.md setup.py src/pidng/__init__.py src/pidng/bitunpack.c src/pidng/camdefs.py src/pidng/core.py src/pidng/defs.py src/pidng/dng.py src/pidng/legacy.py src/pidng/packing.py src/pidng.egg-info/PKG-INFO src/pidng.egg-info/SOURCES.txt src/pidng.egg-info/dependency_links.txt src/pidng.egg-info/requires.txt src/pidng.egg-info/top_level.txt src/pidng/liblj92/lj92.c src/pidng/liblj92/lj92.h././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651864149.0 pidng-4.0.9/src/pidng.egg-info/dependency_links.txt0000644000175000017500000000000100000000000020060 0ustar00pipi ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651864149.0 pidng-4.0.9/src/pidng.egg-info/requires.txt0000644000175000017500000000000600000000000016406 0ustar00pipinumpy ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1651864149.0 pidng-4.0.9/src/pidng.egg-info/top_level.txt0000644000175000017500000000002400000000000016540 0ustar00pipiljpegCompress pidng