pax_global_header00006660000000000000000000000064142456324200014514gustar00rootroot0000000000000052 comment=a56f2bb57eecd49ebcdc9077234df8e76e725a6f karabo-bridge-py-0.6.2/000077500000000000000000000000001424563242000146405ustar00rootroot00000000000000karabo-bridge-py-0.6.2/.coveragerc000066400000000000000000000000271424563242000167600ustar00rootroot00000000000000[run] omit = */tests/* karabo-bridge-py-0.6.2/.github/000077500000000000000000000000001424563242000162005ustar00rootroot00000000000000karabo-bridge-py-0.6.2/.github/workflows/000077500000000000000000000000001424563242000202355ustar00rootroot00000000000000karabo-bridge-py-0.6.2/.github/workflows/tests.yml000066400000000000000000000015741424563242000221310ustar00rootroot00000000000000name: Tests on: push: branches: [ master ] pull_request: branches: [ master ] jobs: tests: runs-on: ubuntu-latest strategy: matrix: python-version: [3.6, 3.7, 3.8] steps: - name: Checkout uses: actions/checkout@v2 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} - uses: actions/cache@v1 with: path: ~/.cache/pip key: ${{ runner.os }}-${{ matrix.python-version }}-pip-${{ hashFiles('**/setup.py') }} - name: Install dependencies run: | python3 -m pip install --upgrade pip python3 -m pip install ".[test]" - name: Test with pytest run: | python3 -m pytest -v --cov=karabo_bridge - name: Upload coverage to Codecov uses: codecov/codecov-action@v1 karabo-bridge-py-0.6.2/.gitignore000066400000000000000000000001061424563242000166250ustar00rootroot00000000000000__pycache__ .pytest_cache/ *.egg-info/ /build/ /dist/ .idea .coverage karabo-bridge-py-0.6.2/LICENSE000066400000000000000000000030141424563242000156430ustar00rootroot00000000000000BSD 3-Clause License Copyright (c) 2017, European X-Ray Free-Electron Laser Facility GmbH All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. karabo-bridge-py-0.6.2/MANIFEST.in000066400000000000000000000000661424563242000164000ustar00rootroot00000000000000include LICENSE include README.md include .coveragerc karabo-bridge-py-0.6.2/README.rst000066400000000000000000000052121424563242000163270ustar00rootroot00000000000000=========================== European XFEL Karabo Bridge =========================== .. image:: https://github.com/European-XFEL/karabo-bridge-py/workflows/Tests/badge.svg :target: https://github.com/European-XFEL/karabo-bridge-py/actions?query=workflow%3ATests .. image:: https://codecov.io/gh/European-XFEL/karabo-bridge-py/branch/master/graph/badge.svg :target: https://codecov.io/gh/European-XFEL/karabo-bridge-py ``karabo_bridge`` is a Python 3 client to receive pipeline data from the Karabo control system used at `European XFEL `_. A simulated Karabo bridge server is included to allow testing code without a connection to a real Karabo server. Installing ---------- to install the package:: $ python3 -m pip install karabo-bridge or $ git clone https://github.com/European-XFEL/karabo-bridge-py.git $ cd ./karabo-bridge-py $ python3 -m pip install . How to use ---------- Request data from a karabo bridge server ++++++++++++++++++++++++++++++++++++++++ Use the ``Client`` class from karabo_brige to create a client and the ``next`` method to request data from the server. The function returns 2 dictionaries: the first one holds a train data and the second one holds the associated train metadata. Both dictionaries are keyed by source name, and the values are dictionaries containing parameters name and values for data and metadata information (source name, timestamp, trainId) for the metadata. Values are all built-in python types, or numpy arrays. .. code-block:: python >>> from karabo_bridge import Client >>> krb_client = Client('tcp://server-host-name:12345') >>> data, metadata = krb_client.next() >>> data.keys() dict_keys(['source1', 'source2', 'source3']) >>> data['source1'].keys() dict_keys(['param1', 'param2']) >>> metadata['source1'] {'source1': {'source': 'source1', 'timestamp': 1528476983.744877, 'timestamp.frac': '744877000000000000', 'timestamp.sec': '1528476983', 'timestamp.tid': 10000000073}} Use the Simulation server +++++++++++++++++++++++++ To start a simulation, call the ``start_gen`` function and provide a port to bind to. You can the use the ``Client`` class and connect to it to test the client without the need to use Karabo. .. code-block:: python >>> from karabo_bridge import start_gen >>> start_gen(1234) Server : emitted train: 10000000000 Server : emitted train: 10000000001 Server : emitted train: 10000000002 Server : emitted train: 10000000003 Server : emitted train: 10000000004 ... You can also run the simulated server from the command line:: $ karabo-bridge-server-sim 1234 karabo-bridge-py-0.6.2/examples/000077500000000000000000000000001424563242000164565ustar00rootroot00000000000000karabo-bridge-py-0.6.2/examples/demo.py000066400000000000000000000007701424563242000177600ustar00rootroot00000000000000from karabo_bridge import Client krb_client = Client("tcp://localhost:4545") for i in range(10): data, metadata = krb_client.next() print("Client : received train ID {}".format( metadata['SPB_DET_AGIPD1M-1/DET/detector']['timestamp.tid'])) det_data = data['SPB_DET_AGIPD1M-1/DET/detector'] print("Client : - detector image shape is {}, {} Mbytes".format( det_data['image.data'].shape, det_data['image.data'].nbytes/1024**2)) print("Client : Client stops reading here") karabo-bridge-py-0.6.2/examples/demo.sh000077500000000000000000000006261424563242000177450ustar00rootroot00000000000000#!/usr/bin/env bash # Start simulated experiment, which offers data as the KaraboBridge # would be during the experiment: echo "demo.sh: starting (simulated) server" karabo-bridge-server-sim 4545 & SIMULATION_PID=$! # Start client to read 10 trains echo "demo.sh: starting client" python3 demo.py # shutting down simulated experiment echo "demo.sh: killing simulated Karabo Bridge" kill $SIMULATION_PID karabo-bridge-py-0.6.2/karabo_bridge/000077500000000000000000000000001424563242000174135ustar00rootroot00000000000000karabo-bridge-py-0.6.2/karabo_bridge/__init__.py000066400000000000000000000036051424563242000215300ustar00rootroot00000000000000# coding: utf-8 """The karabo_bridge package. Copyright (c) 2017, European X-Ray Free-Electron Laser Facility GmbH All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You should have received a copy of the 3-Clause BSD License along with this program. If not, see """ __version__ = "0.6.2" from .cli import * from .client import * from .serializer import * from .server import * __all__ = (client.__all__ + serializer.__all__ + server.__all__) karabo-bridge-py-0.6.2/karabo_bridge/cli/000077500000000000000000000000001424563242000201625ustar00rootroot00000000000000karabo-bridge-py-0.6.2/karabo_bridge/cli/__init__.py000066400000000000000000000000001424563242000222610ustar00rootroot00000000000000karabo-bridge-py-0.6.2/karabo_bridge/cli/glimpse.py000066400000000000000000000147471424563242000222110ustar00rootroot00000000000000"""Inspect a single message from Karabo bridge. """ import argparse from collections.abc import Sequence from datetime import datetime import h5py import numpy as np from socket import gethostname from time import localtime, strftime, time from .. import Client def gen_filename(endpoint): """Generate a filename from endpoint with timestamp. return: str hostname_port_YearMonthDay_HourMinSecFrac.h5 """ now = datetime.now().strftime('%Y%m%d_%H%M%S%f')[:-4] base = endpoint.split('://', 1)[1] if base.startswith('localhost:'): base = gethostname().split('.')[0] + base[9:] base = base.replace(':', '_').replace('/', '_') return '{}_{}.h5'.format(base, now) def dict_to_hdf5(dic, endpoint): """Dump a dict to an HDF5 file. """ filename = gen_filename(endpoint) with h5py.File(filename, 'w') as handler: walk_dict_to_hdf5(dic, handler) print('dumped to', filename) def hdf5_to_dict(filepath, group='/'): """load the content of an hdf5 file to a dict. # TODO: how to split domain_type_dev : parameter : value ? """ if not h5py.is_hdf5(filepath): raise RuntimeError(filepath, 'is not a valid HDF5 file.') with h5py.File(filepath, 'r') as handler: dic = walk_hdf5_to_dict(handler[group]) return dic vlen_bytes = h5py.special_dtype(vlen=bytes) vlen_str = h5py.special_dtype(vlen=str) def walk_dict_to_hdf5(dic, h5): for key, value in sorted(dic.items()): if isinstance(value, dict): group = h5.create_group(key) walk_dict_to_hdf5(value, group) elif isinstance(value, (np.ndarray)): h5.create_dataset(key, data=value, dtype=value.dtype) elif isinstance(value, (int, float)): h5.create_dataset(key, data=value, dtype=type(value)) elif isinstance(value, str): # VLEN strings do not support embedded NULLs value = value.replace('\x00', '') ds = h5.create_dataset(key, shape=(len(value), ), dtype=vlen_str, maxshape=(None, )) ds[:len(value)] = value elif isinstance(value, bytes): # VLEN strings do not support embedded NULLs value = value.replace(b'\x00', b'') ds = h5.create_dataset(key, shape=(len(value), ), dtype=vlen_bytes, maxshape=(None, )) ds[:len(value)] = value else: print('not supported', type(value)) def walk_hdf5_to_dict(h5): dic = {} for key, value in h5.items(): if isinstance(value, h5py.Group): dic[key] = walk_hdf5_to_dict(value) elif isinstance(value, h5py.Dataset): dic[key] = value[()] else: print('what are you?', type(value)) return dic def print_one_train(client, verbosity=0): """Retrieve data for one train and print it. Returns the (data, metadata) dicts from the client. This is used by the -glimpse and -monitor command line tools. """ ts_before = time() data, meta = client.next() ts_after = time() if not data: print("Empty data") return train_id = list(meta.values())[0].get('timestamp.tid', 0) print("Train ID:", train_id, "--------------------------") delta = ts_after - ts_before print('Data from {} sources, REQ-REP took {:.2f} ms' .format(len(data), delta)) print() for i, (source, src_data) in enumerate(sorted(data.items()), start=1): src_metadata = meta.get(source, {}) tid = src_metadata.get('timestamp.tid', 0) print("Source {}: {!r} @ {}".format(i, source, tid)) try: ts = src_metadata['timestamp'] except KeyError: print("No timestamp") else: dt = strftime('%Y-%m-%d %H:%M:%S', localtime(ts)) delay = (ts_after - ts) * 1000 print('timestamp: {} ({}) | delay: {:.2f} ms' .format(dt, ts, delay)) if verbosity < 1: print("- data:", sorted(src_data)) print("- metadata:", sorted(src_metadata)) else: print('data:') pretty_print(src_data, verbosity=verbosity - 1) if src_metadata: print('metadata:') pretty_print(src_metadata) print() return data, meta def pretty_print(d, ind='', verbosity=0): """Pretty print a data dictionary from the bridge client """ assert isinstance(d, dict) for k, v in sorted(d.items()): str_base = '{} - [{}] {}'.format(ind, type(v).__name__, k) if isinstance(v, dict): print(str_base.replace('-', '+', 1)) pretty_print(v, ind=ind+' ', verbosity=verbosity) continue elif isinstance(v, np.ndarray): node = '{}, {}, {}'.format(str_base, v.dtype, v.shape) if verbosity >= 2: node += '\n{}'.format(v) elif isinstance(v, Sequence): if v and isinstance(v, (list, tuple)): itemtype = ' of ' + type(v[0]).__name__ pos = str_base.find(']') str_base = str_base[:pos] + itemtype + str_base[pos:] node = '{}, {}'.format(str_base, v) if verbosity < 1 and len(node) > 80: node = node[:77] + '...' else: node = '{}, {}'.format(str_base, v) print(node) def main(argv=None): ap = argparse.ArgumentParser( prog="karabo-bridge-glimpse", description="Get one Karabo bridge message and prints its data" "structure. optionally: save its data to an HDF5 file.") ap.add_argument('endpoint', help="ZMQ address to connect to, e.g. 'tcp://localhost:4545'") ap.add_argument('-z', '--server-socket', default='REP', choices=['REP', 'PUB', 'PUSH'], help='Socket type used by the karabo bridge server (default REP)') ap.add_argument('-s', '--save', action='store_true', help='Save the received train data to a HDF5 file') ap.add_argument('-v', '--verbose', action='count', default=0, help='Select verbosity (-vv for most verbose)') args = ap.parse_args(argv) # use the appropriate client socket type to match the server socket_map = {'REP': 'REQ', 'PUB': 'SUB', 'PUSH': 'PULL'} client = Client(args.endpoint, sock=socket_map[args.server_socket]) data, _ = print_one_train(client, verbosity=args.verbose + 1) if args.save: dict_to_hdf5(data, args.endpoint) karabo-bridge-py-0.6.2/karabo_bridge/cli/monitor.py000066400000000000000000000031441424563242000222250ustar00rootroot00000000000000#!/usr/bin/env python """Monitor messages coming from Karabo bridge.""" import argparse import gc from .glimpse import print_one_train from ..client import Client def main(argv=None): ap = argparse.ArgumentParser( prog="karabo-bridge-monitor", description="Monitor data from a Karabo bridge server") ap.add_argument('endpoint', help="ZMQ address to connect to, e.g. 'tcp://localhost:4545'") ap.add_argument('-z', '--server-socket', default='REP', choices=['REP', 'PUB', 'PUSH'], help='Socket type used by the karabo bridge server (default REP)') ap.add_argument('-v', '--verbose', action='count', default=0, help='Select verbosity (-vvv for most verbose)') ap.add_argument('--ntrains', help="Stop after N trains", metavar='N', type=int) args = ap.parse_args(argv) socket_map = {'REP': 'REQ', 'PUB': 'SUB', 'PUSH': 'PULL'} client = Client(args.endpoint, sock=socket_map[args.server_socket]) try: if args.ntrains is None: while True: print_one_train(client, verbosity=args.verbose) # Explicitly trigger garbage collection, # seems to be needed to avoid using lots of memory. gc.collect() else: for _ in range(args.ntrains): print_one_train(client, verbosity=args.verbose) # Explicitly trigger garbage collection, # seems to be needed to avoid using lots of memory. gc.collect() except KeyboardInterrupt: print('\nexit.') karabo-bridge-py-0.6.2/karabo_bridge/cli/simulation.py000066400000000000000000000041761424563242000227300ustar00rootroot00000000000000import argparse from karabo_bridge.server import start_gen def main(argv=None): ap = argparse.ArgumentParser( prog="karabo-bridge-server-sim", description="Run a Karabo bridge server producing simulated data." ) ap.add_argument( 'port', help="TCP port the server will bind" ) ap.add_argument( '-z', '--server-socket', default='REP', choices=['REP', 'PUB', 'PUSH'], help='Socket type used by the karabo bridge server (default REP)' ) ap.add_argument( '-d', '--detector', default='AGIPD', choices=['AGIPD', 'AGIPDModule', 'LPD'], help="Which kind of detector to simulate (default: AGIPD)" ) ap.add_argument( '-p', '--protocol', default='2.2', choices=['1.0', '2.1', '2.2'], help="Version of the Karabo Bridge protocol to send (default: 2.2)" ) ap.add_argument( '-s', '--serialisation', default='msgpack', choices=['msgpack', 'pickle'], help="Message serialisation format (default: msgpack)" ) ap.add_argument( '-r', '--raw', action='store_true', help='Simulate raw data if True, corrected data if False (default' 'False)' ) ap.add_argument( '-n', '--nsources', type=int, default=1, help='Number of simulated detector sources to send (default 1)' ) ap.add_argument( '-g', '--gen', default='random', choices=['random', 'zeros'], help='Generator function to generate simulated detector data' ) ap.add_argument( '--data-like', default='online', choices=['online', 'file'], help='Data array axes ordering: online -> (modules, fs, ss, pulses), ' 'file -> (pulses, modules, ss, fs)' ) ap.add_argument( '--debug', action='store_true', help='More verbose terminal logging' ) args = ap.parse_args(argv) start_gen(args.port, args.server_socket, args.serialisation, args.protocol, args.detector, args.raw, args.nsources, args.gen, args.data_like, debug=args.debug) if __name__ == '__main__': main() karabo-bridge-py-0.6.2/karabo_bridge/client.py000066400000000000000000000100141424563242000212370ustar00rootroot00000000000000# coding: utf-8 """ Karabo bridge client. Copyright (c) 2017, European X-Ray Free-Electron Laser Facility GmbH All rights reserved. You should have received a copy of the 3-Clause BSD License along with this program. If not, see """ import zmq from .serializer import deserialize __all__ = ['Client'] class Client: """Karabo bridge client for Karabo pipeline data. This class can request data to a Karabo bridge server. Create the client with:: from karabo_bridge import Client krb_client = Client("tcp://153.0.55.21:12345") then call ``data = krb_client.next()`` to request next available data container. Parameters ---------- endpoint : str server socket you want to connect to (only support TCP socket). sock : str, optional socket type - supported: REQ, SUB. ser : str, DEPRECATED Serialization protocol to use to decode the incoming message (default is msgpack) - supported: msgpack. context : zmq.Context To run the Client's sockets using a provided ZeroMQ context. timeout : int Timeout on :method:`next` (in seconds) Data transfered at the EuXFEL for Mega-pixels detectors can be very large. Setting a too small timeout might end in never getting data. Some example of transfer timing for 1Mpix detector (AGIPD, LPD): 32 pulses per train (125 MB): ~0.1 s 128 pulses per train (500 MB): ~0.4 s 350 pulses per train (1.37 GB): ~1 s Raises ------ NotImplementedError if socket type or serialization algorythm is not supported. ZMQError if provided endpoint is not valid. """ def __init__(self, endpoint, sock='REQ', ser='msgpack', timeout=None, context=None): if ser != 'msgpack': raise Exception('Only serialization supported is msgpack') self._context = context or zmq.Context() self._socket = None if sock == 'PULL': self._socket = self._context.socket(zmq.PULL) elif sock == 'REQ': self._socket = self._context.socket(zmq.REQ) elif sock == 'SUB': self._socket = self._context.socket(zmq.SUB) self._socket.setsockopt(zmq.SUBSCRIBE, b'') else: raise NotImplementedError('Unsupported socket: %s' % str(sock)) self._socket.setsockopt(zmq.LINGER, 0) self._socket.set_hwm(1) self._socket.connect(endpoint) if timeout is not None: self._socket.setsockopt(zmq.RCVTIMEO, int(timeout * 1000)) self._recv_ready = False self._pattern = self._socket.TYPE def next(self): """Request next data container. This function call is blocking. Returns ------- data : dict The data for this train, keyed by source name. meta : dict The metadata for this train, keyed by source name. This dictionary is populated for protocol version 1.0 and 2.2. For other protocol versions, metadata information is available in `data` dict. Raises ------ TimeoutError If timeout is reached before receiving data. """ if self._pattern == zmq.REQ and not self._recv_ready: self._socket.send(b'next') self._recv_ready = True try: msg = self._socket.recv_multipart(copy=False) except zmq.error.Again: raise TimeoutError( 'No data received from {} in the last {} ms'.format( self._socket.getsockopt_string(zmq.LAST_ENDPOINT), self._socket.getsockopt(zmq.RCVTIMEO))) self._recv_ready = False return deserialize(msg) def __enter__(self): return self def __exit__(self, exc_type, exc_value, traceback): self._context.destroy(linger=0) def __iter__(self): return self def __next__(self): return self.next() karabo-bridge-py-0.6.2/karabo_bridge/serializer.py000066400000000000000000000143751424563242000221500ustar00rootroot00000000000000from functools import partial from time import time import msgpack import msgpack_numpy import numpy as np import zmq __all__ = ['serialize', 'deserialize'] class Frame: def __init__(self, data): """Dummy zmq.Frame object deserialize() expects the message in ZMQ Frame objects. """ self.bytes = data self.buffer = data def timestamp(): """Generate dummy timestamp information based on machine time """ epoch = time() sec, frac = str(epoch).split('.') frac = frac.ljust(18, '0') return { 'timestamp': epoch, 'timestamp.sec': sec, 'timestamp.frac': frac, } def _serialize_old(data, metadata, dummy_timestamps): """Adapter for backward compatibility with protocol 1.0 """ ts = timestamp() new_data = {} for src, val in data.items(): # in version 1.0 metadata is contained in data[src]['metadata'] # We need to make a copy to not alter the original data dict new_data[src] = val.copy() meta = metadata[src].copy() if dummy_timestamps and 'timestamp' not in meta: meta.update(ts) new_data[src]['metadata'] = meta return [msgpack.packb(new_data, use_bin_type=True, default=msgpack_numpy.encode)] def serialize(data, metadata=None, protocol_version='2.2', dummy_timestamps=False): """Serializer for the Karabo bridge protocol Convert data/metadata to a list of bytestrings and/or memoryviews Parameters ---------- data : dict Contains train data. The dictionary has to follow the karabo_bridge protocol structure: - keys are source names - values are dict, where the keys are the parameter names and values must be python built-in types or numpy.ndarray. metadata : dict, optional Contains train metadata. The dictionary has to follow the karabo_bridge protocol structure: - keys are (str) source names - values (dict) should contain the following items: - 'timestamp' Unix time with subsecond resolution - 'timestamp.sec' Unix time with second resolution - 'timestamp.frac' fractional part with attosecond resolution - 'timestamp.tid' is European XFEL train unique ID :: { 'source': 'sourceName' # str 'timestamp': 1234.567890 # float 'timestamp.sec': '1234' # str 'timestamp.frac': '567890000000000000' # str 'timestamp.tid': 1234567890 # int } If the metadata dict is not provided it will be extracted from 'data' or an empty dict if 'metadata' key is missing from a data source. protocol_version: ('1.0' | '2.2') Which version of the bridge protocol to use. Defaults to the latest version implemented. dummy_timestamps: bool Some tools (such as OnDA) expect the timestamp information to be in the messages. We can't give accurate timestamps where these are not in the file, so this option generates fake timestamps from the time the data is fed in, if the real timestamp information is missing. returns ------- msg: list of bytes/memoryviews ojects binary conversion of data/metadata readable by the karabo_bridge """ if protocol_version not in {'1.0', '2.2'}: raise ValueError(f'Unknown protocol version {protocol_version}') if metadata is None: metadata = {src: v.get('metadata', {}) for src, v in data.items()} if protocol_version == '1.0': return _serialize_old(data, metadata, dummy_timestamps) pack = msgpack.Packer(use_bin_type=True).pack msg = [] ts = timestamp() for src, props in sorted(data.items()): src_meta = metadata[src].copy() if dummy_timestamps and 'timestamp' not in src_meta: src_meta.update(ts) main_data = {} arrays = [] for key, value in props.items(): if isinstance(value, np.ndarray): arrays.append((key, value)) elif isinstance(value, np.number): # Convert numpy type to native Python type main_data[key] = value.item() else: main_data[key] = value msg.extend([ pack({ 'source': src, 'content': 'msgpack', 'metadata': src_meta }), pack(main_data) ]) for key, array in arrays: if not array.flags['C_CONTIGUOUS']: array = np.ascontiguousarray(array) msg.extend([ pack({ 'source': src, 'content': 'array', 'path': key, 'dtype': str(array.dtype), 'shape': array.shape }), array.data, ]) return msg def deserialize(msg): """Deserializer for the karabo bridge protocol Parameters ---------- msg: list of zmq.Frame or list of byte objects Serialized data following the karabo_bridge protocol Returns ------- data : dict The data for a train, keyed by source name. meta : dict The metadata for a train, keyed by source name. """ unpack = partial(msgpack.loads, raw=False, max_bin_len=0x7fffffff) if not isinstance(msg[0], zmq.Frame): msg = [Frame(m) for m in msg] if len(msg) < 2: # protocol version 1.0 data = unpack(msg[-1].bytes, object_hook=msgpack_numpy.decode) meta = {} for key, value in data.items(): meta[key] = value.get('metadata', {}) return data, meta data, meta = {}, {} for header, payload in zip(*[iter(msg)]*2): md = unpack(header.bytes) source = md['source'] content = md['content'] if content == 'msgpack': data[source] = unpack(payload.bytes) meta[source] = md.get('metadata', {}) elif content == 'array': dtype, shape = md['dtype'], md['shape'] array = np.frombuffer(payload.buffer, dtype=dtype).reshape(shape) data[source].update({md['path']: array}) else: raise RuntimeError('Unknown message: %s' % md['content']) return data, meta karabo-bridge-py-0.6.2/karabo_bridge/server.py000066400000000000000000000214221424563242000212740ustar00rootroot00000000000000from functools import partial from queue import Queue from socket import gethostname from threading import Thread from time import time import zmq from .serializer import serialize from .simulation import data_generator __all__ = ['ServerInThread', 'start_gen'] class Sender: def __init__(self, endpoint, sock='REP', protocol_version='2.2', dummy_timestamps=False): self.dump = partial(serialize, protocol_version=protocol_version, dummy_timestamps=dummy_timestamps) self.zmq_context = zmq.Context() if sock == 'REP': self.server_socket = self.zmq_context.socket(zmq.REP) elif sock == 'PUB': self.server_socket = self.zmq_context.socket(zmq.PUB) elif sock == 'PUSH': self.server_socket = self.zmq_context.socket(zmq.PUSH) else: raise ValueError(f'Unsupported socket type: {sock}') self.server_socket.setsockopt(zmq.LINGER, 0) self.server_socket.set_hwm(1) self.server_socket.bind(endpoint) self.stopper_r = self.zmq_context.socket(zmq.PAIR) self.stopper_r.bind('inproc://sim-server-stop') self.stopper_w = self.zmq_context.socket(zmq.PAIR) self.stopper_w.connect('inproc://sim-server-stop') self.poller = zmq.Poller() self.poller.register(self.server_socket, zmq.POLLIN | zmq.POLLOUT) self.poller.register(self.stopper_r, zmq.POLLIN) @property def endpoint(self): endpoint = self.server_socket.getsockopt_string(zmq.LAST_ENDPOINT) endpoint = endpoint.replace('0.0.0.0', gethostname()) return endpoint def send(self, data, metadata=None): payload = self.dump(data, metadata) events = dict(self.poller.poll()) if self.stopper_r in events: self.stopper_r.recv() return True if events[self.server_socket] == zmq.POLLIN: msg = self.server_socket.recv() if msg != b'next': print(f'Unrecognised request: {msg}') self.server_socket.send(b'Error: bad request %b' % msg) return self.server_socket.send_multipart(payload, copy=False) class SimServer(Sender): def __init__(self, endpoint, sock='REP', ser='msgpack', protocol_version='2.2', detector='AGIPD', raw=False, nsources=1, datagen='random', data_like='online', *, debug=True): super().__init__(endpoint, sock=sock, protocol_version=protocol_version) if ser != 'msgpack': raise ValueError("Unknown serialisation format %s" % ser) self.data = data_generator( detector=detector, raw=raw, nsources=nsources, datagen=datagen, data_like=data_like, debug=debug) self.debug = debug def loop(self): print(f'Simulated Karabo-bridge server started on:\n' f'{self.endpoint}') timing_interval = 50 t_prev = time() n = 0 for data, meta in self.data: done = self.send(data, meta) if done: break if self.debug: print('Server : emitted train:', next(v for v in meta.values())['timestamp.tid']) n += 1 if n % timing_interval == 0: t_now = time() print('Sent {} trains in {:.2f} seconds ({:.2f} Hz)' ''.format(timing_interval, t_now - t_prev, timing_interval / (t_now - t_prev))) t_prev = t_now class ServerInThread(Sender): def __init__(self, endpoint, sock='REP', maxlen=10, protocol_version='2.2', dummy_timestamps=False): """ZeroMQ interface sending data over a TCP socket. example:: # Server: serve = ServerInThread(1234) serve.start() for tid, data in run.trains(): result = important_processing(data) serve.feed(result) # Client: from karabo_bridge import Client client = Client('tcp://server.hostname:1234') data = client.next() Parameters ---------- endpoint: str The address string. sock: str socket type - supported: REP, PUB, PUSH (default REP). maxlen: int, optional How many trains to cache before sending (default: 10) protocol_version: ('1.0' | '2.1') Which version of the bridge protocol to use. Defaults to the latest version implemented. dummy_timestamps: bool Some tools (such as OnDA) expect the timestamp information to be in the messages. We can't give accurate timestamps where these are not in the file, so this option generates fake timestamps from the time the data is fed in. """ super().__init__(endpoint, sock=sock, protocol_version=protocol_version, dummy_timestamps=dummy_timestamps) self.thread = Thread(target=self._run, daemon=True) self.buffer = Queue(maxsize=maxlen) def feed(self, data, metadata=None, block=True, timeout=None): """Push data to the sending queue. This blocks if the queue already has *maxlen* items waiting to be sent. Parameters ---------- data : dict Contains train data. The dictionary has to follow the karabo_bridge see :func:`~karabo_bridge.serializer.serialize` for details metadata : dict, optional Contains train metadata. If the metadata dict is not provided it will be extracted from 'data' or an empty dict if 'metadata' key is missing from a data source. see :func:`~karabo_bridge.serializer.serialize` for details block: bool If True, block if necessary until a free slot is available or 'timeout' has expired. If False and there is no free slot, raises 'queue.Full' (timeout is ignored) timeout: float In seconds, raises 'queue.Full' if no free slow was available within that time. """ self.buffer.put((data, metadata), block=block, timeout=timeout) def _run(self): while True: done = self.send(*self.buffer.get()) if done: break def start(self): self.thread.start() def stop(self): self.stopper_w.send(b'') if self.buffer.qsize() == 0: self.buffer.put(({},)) # release blocking queue self.thread.join() self.zmq_context.destroy(linger=0) def __enter__(self): self.start() return self def __exit__(self, exc_type, exc_val, exc_tb): self.stop() class SimServerInThread(SimServer, ServerInThread): def _run(self): self.loop() def start_gen(port, sock='REP', ser='msgpack', version='2.2', detector='AGIPD', raw=False, nsources=1, datagen='random', data_like='online', *, debug=True): """Karabo bridge server simulation. Simulate a Karabo Bridge server and send random data from a detector, either AGIPD or LPD. Parameters ---------- port: str The port to on which the server is bound. sock: str, optional socket type - supported: REP, PUB, PUSH. Default is REP. ser: str, optional The serialization algorithm, default is msgpack. version: str, optional The container version of the serialized data. detector: str, optional The data format to send, default is AGIPD detector. raw: bool, optional Generate raw data output if True, else CORRECTED. Default is False. nsources: int, optional Number of sources. datagen: string, optional Generator function used to generate detector data. Default is random. data_like: string optional ['online', 'file'] Data array axes ordering for Mhz detector. The data arrays's axes can have different ordering on online data. The calibration processing orders axes as (fs, ss, pulses), whereas data in files have (pulses, ss, fs). This option allow to chose between 2 ordering: - online: (modules, fs, ss, pulses) - file: (pulses, modules, ss, fs) Note that the real system can send data in both shape with a performance penalty for the file-like array shape. Default is online. """ endpoint = f'tcp://*:{port}' sender = SimServer( endpoint, sock=sock, ser=ser, protocol_version=version, detector=detector, raw=raw, nsources=nsources, datagen=datagen, data_like=data_like, debug=debug ) try: sender.loop() except KeyboardInterrupt: pass print('\nStopped.') karabo-bridge-py-0.6.2/karabo_bridge/simulation.py000066400000000000000000000144231424563242000221550ustar00rootroot00000000000000# coding: utf-8 """ Set of functions to simulate karabo bridge server and generate fake detector data. Copyright (c) 2017, European X-Ray Free-Electron Laser Facility GmbH All rights reserved. You should have received a copy of the 3-Clause BSD License along with this program. If not, see """ import copy from itertools import count from time import time import numpy as np class Detector: # Overridden in subclasses pulses = 0 # nimages per XRAY train modules = 0 # nb of super modules composing the detector mod_x = 0 # pixel count (y axis) of a single super module mod_y = 0 # pixel count (x axis) of a single super module pixel_size = 0 # [mm] distance = 0 # Sample to detector distance [mm] layout = np.array([[]]) # super module layout of the detector @staticmethod def getDetector(detector, source='', raw=False, gen='random', data_like='online'): if detector == 'AGIPD': if not raw: default = 'SPB_DET_AGIPD1M-1/CAL/APPEND_CORRECTED' else: default = 'SPB_DET_AGIPD1M-1/CAL/APPEND_RAW' source = source or default return AGIPD(source, raw=raw, gen=gen, data_like=data_like) elif detector == 'AGIPDModule': if not raw: raise NotImplementedError( 'Calib. Data for single Modules not available yet') source = source or 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' return AGIPDModule(source, raw=raw, gen=gen, data_like=data_like) elif detector == 'LPD': if not raw: default = 'FXE_DET_LPD1M-1/CAL/APPEND_CORRECTED' else: default = 'FXE_DET_LPD1M-1/CAL/APPEND_RAW' source = source or default return LPD(source, raw=raw, gen=gen) else: raise NotImplementedError('detector %r not available' % detector) def __init__(self, source='', raw=True, gen='random', data_like='online'): self.source = source or 'INST_DET_GENERIC/DET/detector' self.raw = raw self.data_like = data_like if gen == 'random': self.genfunc = self.random elif gen == 'zeros': self.genfunc = self.zeros else: raise NotImplementedError('gen func %r not implemented' % gen) @property def data_type(self): if self.raw: return np.uint16 else: return np.float32 def corr_passport(self): domain = self.source.partition('/')[0] passport = [ '%s/CAL/THRESHOLDING_Q1M1' % domain, '%s/CAL/OFFSET_CORR_Q1M1' % domain, '%s/CAL/RELGAIN_CORR_Q1M1' % domain ] return passport @property def data_shape(self): shape = () if self.modules == 1 else (self.modules, ) if self.data_like == 'online': shape += (self.mod_y, self.mod_x, self.pulses) else: shape = (self.pulses, ) + shape + (self.mod_x, self.mod_y) return shape def random(self): return np.random.uniform(low=1500, high=1600, size=self.data_shape).astype(self.data_type) def zeros(self): return np.zeros(self.data_shape, dtype=self.data_type) def module_position(self, ix): y, x = np.where(self.layout == ix) assert len(y) == len(x) == 1 return x[0], y[0] @staticmethod def gen_metadata(source, timestamp, trainId): sec, frac = str(timestamp).split('.') meta = {} meta[source] = { 'source': source, 'timestamp': timestamp, 'timestamp.tid': trainId, 'timestamp.sec': sec, 'timestamp.frac': frac.ljust(18, '0') # attosecond resolution } return meta def gen_data(self, trainId): data = {} timestamp = time() data['image.data'] = self.genfunc() base_src = '/'.join((self.source.rpartition('/')[0], '{}CH0:xtdf')) sources = [base_src.format(i) for i in range(16)] # TODO: cellId differ between AGIPD/LPD data['image.cellId'] = np.arange(self.pulses, dtype=np.uint16) if not self.raw: data['image.passport'] = self.corr_passport() if self.modules > 1: # More than one modules have sources data['sources'] = sources data['modulesPresent'] = [True for i in range(self.modules)] # Image gain has only entries for one module data['image.gain'] = np.zeros((self.mod_y, self.mod_x, self.pulses), dtype=np.uint16) # TODO: pulseId differ between AGIPD/LPD data['image.pulseId'] = np.arange(self.pulses, dtype=np.uint64) data['image.trainId'] = ( np.ones(self.pulses) * trainId).astype(np.uint64) meta = self.gen_metadata(self.source, timestamp, trainId) return {self.source: data}, meta class AGIPDModule(Detector): pulses = 64 modules = 1 mod_y = 128 mod_x = 512 pixel_size = 0.2 distance = 2000 layout = np.array([ [12, 0], [13, 1], [14, 2], [15, 3], [8, 4], [9, 5], [10, 6], [11, 7], ]) class AGIPD(AGIPDModule): modules = 16 class LPD(Detector): pulses = 300 modules = 16 mod_y = 256 mod_x = 256 pixel_size = 0.5 distance = 275 layout = np.array([ [15, 12, 3, 0], [14, 13, 2, 1], [11, 8, 7, 4], [10, 9, 6, 5], ]) def data_generator(detector='AGIPD', raw=False, nsources=1, datagen='random', data_like='online', *, debug=False): detector = Detector.getDetector(detector, raw=raw, gen=datagen, data_like=data_like) for train_id in count(start=10000000000): data, meta = detector.gen_data(train_id) if nsources > 1: source = detector.source for i in range(nsources): src = source + "-" + str(i+1) data[src] = copy.deepcopy(data[source]) meta[src] = copy.deepcopy(meta[source]) meta[src]['source'] = src del data[source] del meta[source] yield (data, meta) karabo-bridge-py-0.6.2/karabo_bridge/tests/000077500000000000000000000000001424563242000205555ustar00rootroot00000000000000karabo-bridge-py-0.6.2/karabo_bridge/tests/__init__.py000066400000000000000000000000001424563242000226540ustar00rootroot00000000000000karabo-bridge-py-0.6.2/karabo_bridge/tests/conftest.py000066400000000000000000000042741424563242000227630ustar00rootroot00000000000000from tempfile import TemporaryDirectory import numpy as np import pytest from karabo_bridge.server import ServerInThread, SimServerInThread @pytest.fixture(params=['1.0', '2.2']) def protocol_version(request): yield request.param @pytest.fixture(scope='function') def sim_server(protocol_version): with TemporaryDirectory() as td: endpoint = "ipc://{}/server".format(td) with SimServerInThread(endpoint, detector='AGIPDModule', raw=True, protocol_version=protocol_version) as s: yield s @pytest.fixture(scope='function') def sim_push_server(protocol_version): with TemporaryDirectory() as td: endpoint = "ipc://{}/push".format(td) with SimServerInThread(endpoint, sock='PUSH', detector='AGIPDModule', raw=True, protocol_version=protocol_version) as s: yield s @pytest.fixture(scope='function') def server(protocol_version): with TemporaryDirectory() as td: endpoint = "ipc://{}/server".format(td) with ServerInThread(endpoint, protocol_version=protocol_version) as s: yield s @pytest.fixture(scope='session') def data(): yield { 'source1': { 'parameter.1.value': 123, 'parameter.2.value': 1.23, 'list.of.int': [1, 2, 3], 'string.param': 'True', 'boolean': False, 'metadata': {'timestamp.tid': 9876543210, 'timestamp': 12345678}, }, 'XMPL/DET/MOD0': { 'image.data': np.random.randint(255, size=(2, 3, 4), dtype=np.uint8), 'something.else': ['a', 'bc', 'd'], }, } @pytest.fixture(scope='session') def metadata(): yield { 'source1': { 'source': 'source1', 'timestamp': 1585926035.7098422, 'timestamp.sec': '1585926035', 'timestamp.frac': '709842200000000000', 'timestamp.tid': 1000000 }, 'XMPL/DET/MOD0': { 'source': 'XMPL/DET/MOD0', 'timestamp': 1585926036.9098422, 'timestamp.sec': '1585926036', 'timestamp.frac': '909842200000000000', 'timestamp.tid': 1000010 } } karabo-bridge-py-0.6.2/karabo_bridge/tests/test_client.py000066400000000000000000000030401424563242000234410ustar00rootroot00000000000000from itertools import islice import pytest from karabo_bridge import Client def test_get_frame(sim_server, protocol_version): c = Client(sim_server.endpoint) data, metadata = c.next() assert 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' in data assert 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' in metadata if protocol_version == '1.0': assert all('metadata' in src for src in data.values()) def test_pull_socket(sim_push_server): c = Client(sim_push_server.endpoint, sock='PULL') data, metadata = c.next() assert 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' in data assert 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' in metadata def test_pair_socket(sim_server): with pytest.raises(NotImplementedError): c = Client(sim_server, sock='PAIR') def test_context_manager(sim_server): with Client(sim_server.endpoint) as c: data, metadata = c.next() assert 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' in data assert 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' in metadata assert c._socket.closed def test_iterator(sim_server): c = Client(sim_server.endpoint) for i, (data, metadata) in enumerate(islice(c, 3)): trainId = metadata['SPB_DET_AGIPD1M-1/DET/0CH0:xtdf']['timestamp.tid'] assert trainId == 10000000000 + i def test_timeout(): no_server = 'ipc://nodata' with Client(no_server, timeout=0.2) as c: for _ in range(3): with pytest.raises(TimeoutError) as info: tid, data = c.next() assert 'No data received from ipc://nodata in the last 200 ms' in str(info.value) karabo-bridge-py-0.6.2/karabo_bridge/tests/test_glimpse.py000066400000000000000000000012411424563242000236240ustar00rootroot00000000000000import os import h5py from testpath.tempdir import TemporaryWorkingDirectory from karabo_bridge.cli import glimpse def test_main(sim_server, capsys): glimpse.main([sim_server.endpoint]) out, err = capsys.readouterr() assert 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' in out def test_save(sim_server): with TemporaryWorkingDirectory() as td: glimpse.main(['--save', sim_server.endpoint]) files = os.listdir(td) assert len(files) == 1 path = os.path.join(td, files[0]) with h5py.File(path, 'r') as f: trainId = f['SPB_DET_AGIPD1M-1/DET/0CH0:xtdf/image.trainId'][:] assert trainId[0] == 10000000000 karabo-bridge-py-0.6.2/karabo_bridge/tests/test_monitor.py000066400000000000000000000003341424563242000236550ustar00rootroot00000000000000from karabo_bridge.cli import monitor def test_main(sim_server, capsys): monitor.main([sim_server.endpoint, '--ntrains', '1']) out, err = capsys.readouterr() assert 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' in out karabo-bridge-py-0.6.2/karabo_bridge/tests/test_serialize.py000066400000000000000000000027211424563242000241570ustar00rootroot00000000000000import numpy as np import pytest from karabo_bridge import serialize, deserialize from .utils import compare_nested_dict def test_serialize(data, protocol_version): msg = serialize(data, protocol_version=protocol_version) assert isinstance(msg, list) d, m = deserialize(msg) compare_nested_dict(data, d) assert m['source1'] == {'timestamp.tid': 9876543210, 'timestamp': 12345678} assert m['XMPL/DET/MOD0'] == {} def test_serialize_with_metadata(data, metadata, protocol_version): msg = serialize(data, metadata, protocol_version=protocol_version) d, m = deserialize(msg) compare_nested_dict(metadata, m) def test_serialize_with_dummy_timestamps(data, protocol_version): msg = serialize(data, protocol_version=protocol_version, dummy_timestamps=True) d, m = deserialize(msg) assert set(m['XMPL/DET/MOD0']) == {'timestamp', 'timestamp.sec', 'timestamp.frac'} assert set(m['source1']) == {'timestamp', 'timestamp.tid'} assert m['source1']['timestamp.tid'] == 9876543210 assert m['source1']['timestamp'] == 12345678 def test_serialize_with_metadata_and_dummy_timestamp(data, metadata, protocol_version): msg = serialize(data, metadata, protocol_version=protocol_version, dummy_timestamps=True) d, m = deserialize(msg) compare_nested_dict(metadata, m) def test_wrong_version(data): with pytest.raises(ValueError): serialize(data, protocol_version='3.0') karabo-bridge-py-0.6.2/karabo_bridge/tests/test_server.py000066400000000000000000000004561424563242000235010ustar00rootroot00000000000000from karabo_bridge import Client from .utils import compare_nested_dict def test_req_rep(server, data): for _ in range(3): server.feed(data) with Client(server.endpoint) as client: for _ in range(3): d, m = client.next() compare_nested_dict(data, d) karabo-bridge-py-0.6.2/karabo_bridge/tests/test_simulation.py000066400000000000000000000022541424563242000243550ustar00rootroot00000000000000import numpy as np from karabo_bridge.simulation import Detector source_lpd = 'FXE_DET_LPD1M-1/CAL/APPEND_CORRECTED' source_spb_module = 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' train_id = 10000000000 def test_lpd(): lpd = Detector.getDetector('LPD') data, meta = lpd.gen_data(train_id) assert len(data) == len(meta) == 1 assert source_lpd in data assert meta[source_lpd]['timestamp.tid'] == train_id img = data[source_lpd]['image.data'] assert img.shape == (16, 256, 256, 300) assert not np.any(img[(img<1500) | (img>1600)]) def test_gen(): agipd = Detector.getDetector('AGIPDModule', gen='zeros', raw=True) data, meta = agipd.gen_data(train_id) assert len(data) == len(meta) == 1 assert source_spb_module in data assert meta[source_spb_module]['timestamp.tid'] == train_id assert data[source_spb_module]['image.data'].shape == (128, 512, 64) assert not np.any(data[source_spb_module]['image.data']) def test_filelike_shape(): agipd = Detector.getDetector('AGIPDModule', gen='zeros', raw=True, data_like='file') data, meta = agipd.gen_data(train_id) assert data[source_spb_module]['image.data'].shape == (64, 512, 128) karabo-bridge-py-0.6.2/karabo_bridge/tests/utils.py000066400000000000000000000020101424563242000222600ustar00rootroot00000000000000import numpy as np def compare_nested_dict(d1, d2, path=''): for key in d1.keys(): if key not in d2: print(d1.keys()) print(d2.keys()) raise KeyError('key is missing in d2: {}{}'.format(path, key)) if isinstance(d1[key], dict): path += key + '.' compare_nested_dict(d1[key], d2[key], path) else: v1 = d1[key] v2 = d2[key] try: if isinstance(v1, np.ndarray): assert (v1 == v2).all() elif isinstance(v1, tuple) or isinstance(v2, tuple): # msgpack doesn't know about complex types, everything is # an array. So tuples are packed as array and then # unpacked as list by default. assert list(v1) == list(v2) else: assert v1 == v2 except AssertionError: raise AssertionError('diff: {}{}'.format(path, key), v1, v2)karabo-bridge-py-0.6.2/setup.py000077500000000000000000000037671424563242000163720ustar00rootroot00000000000000#!/usr/bin/env python import os.path as osp import re from setuptools import setup, find_packages import sys def get_script_path(): return osp.dirname(osp.realpath(sys.argv[0])) def read(*parts): return open(osp.join(get_script_path(), *parts)).read() def find_version(*parts): vers_file = read(*parts) match = re.search(r'^__version__ = "(\d+\.\d+\.\d+)"', vers_file, re.M) if match is not None: return match.group(1) raise RuntimeError("Unable to find version string.") setup(name="karabo_bridge", version=find_version("karabo_bridge", "__init__.py"), author="European XFEL GmbH", author_email="da-support@xfel.eu", maintainer="Thomas Michelat", url="https://github.com/European-XFEL/karabo-bridge-py", description=("Python 3 tools to request data from the Karabo control" "system."), long_description=read("README.rst"), license="BSD-3-Clause", install_requires=[ 'msgpack>=0.5.4', 'msgpack-numpy', 'numpy', 'pyzmq>=17.0.0', ], extras_require={ 'test': [ 'pytest', 'pytest-cov', 'h5py', 'testpath', ] }, packages=find_packages(), entry_points={ 'console_scripts': [ 'karabo-bridge-glimpse=karabo_bridge.cli.glimpse:main', 'karabo-bridge-monitor=karabo_bridge.cli.monitor:main', 'karabo-bridge-server-sim=karabo_bridge.cli.simulation:main', ], }, classifiers=[ 'Development Status :: 3 - Alpha', 'Environment :: Console', 'Intended Audience :: Developers', 'Intended Audience :: Science/Research', 'License :: OSI Approved :: BSD License', 'Operating System :: POSIX :: Linux', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', ])