pax_global_header00006660000000000000000000000064134552112260014513gustar00rootroot0000000000000052 comment=cdca778521151451be6e50c930dcb4d2117e96db debiancontributors-0.7.8/000077500000000000000000000000001345521122600154275ustar00rootroot00000000000000debiancontributors-0.7.8/.gitignore000066400000000000000000000000251345521122600174140ustar00rootroot00000000000000*.swp *.pyc MANIFEST debiancontributors-0.7.8/DATAMINING.rst000066400000000000000000000302401345521122600175730ustar00rootroot00000000000000 =================== dc-tool data mining =================== dc-tool has several methods of data mining that can be controlled via a configuration file. It works like this: 1. Read this documentation and create a configuration file to test. 2. Run ``dc-tool --mine=mysource.conf`` to perform data mining and print results to standard output. 3. When you are satisfied of the results, run ``dc-tool --mine=mysource.conf --post`` to post data to contributors.debian.org. Run that via cron and you have a full working data source. ------------------------- Configuration file syntax ------------------------- The configuration file follows the usual Debian RFC822/Yaml-like syntax. If the first group of options does not have a "contribution:" field, it is used for general configuration of the data source. All other sections define methods of mining the data you want. The data source configuration section ===================================== Example:: # You don't need this option if you call this file nm.debian.org.conf #source: nm.debian.org # Auhentication token used to post data. Use a leading '@' as in '@filename' # to use the contents of another file as auth token. Do not make this file # world-readable! auth_token: @secrets/auth_token.txt The general configuration section has three configurable keywords: ``source`` Data source name, as configured in contributors.debian.org. If omitted, dc-tool will use the configuration file name. If the file name ends in ``.ini``, ``.conf`` or ``.cfg``, the extension will be removed. ``auth_token`` The authentication token used for posting data to the site. Anyone with this authentication token can post data for this data source, so be careful not to give this file world-readable permissions. ``baseurl`` You never need this unless you want to test a local copy of the contributors.debian.org codebase: it defaults to ``https://contributors.debian.org/`` but you can change it to submit data to your local development version. Data mining sections ==================== Example:: contribution: committer # Data mining method method: gitdirs # Configuration specific to this method dirs: /srv/git.debian.org/git/collab-maint/*.git url: https://alioth.debian.org/users/{user}/ Each data mining section has at least two configurable keywords: ``contribution`` Contribution type for this data source, as configured in contributors.debian.org. You can have many sections with the same contribution types, and the results of their data mining will all be merged. ``method`` The mining method. There are several mining method available, each with its own configuration options, documented below. The rest of the options are specific to each data mining method. Below is a full documentation of them. Data mining methods =================== bts --- Parses the debbugs spool directories collecting contributions from mail headers. Example:: contribution: correspondant method: bts dirs: /srv/bugs.debian.org/spool/db-h/ /srv/bugs.debian.org/spool/archive/ url: https://bugs.debian.org/cgi-bin/pkgreport.cgi?correspondent={email} Configuration options ````````````````````` ``dirs`` : Glob, required, default: None. debbugs spool directories to scan. You can give one or more, and even use shell-style globbing. ``threshold`` : Integer, optional, default: 5. Minimum number of mails that need to exist in the BTS for an email address to be considered ``url`` : Char, optional, default: None. template used to build URLs to link to people's contributions. ``{email}`` will be replaced with the email address Option types ```````````` ``Char`` A string value. Can be any UTF-8 string. ``Glob`` A string with one or more filenames. Globbing is supported. Arguments can be quoted to deal with whitespace, but glob characters will always be expanded. ``Integer`` An integer value. files ----- Recursively scan directories using file attributes to detect contributions. Generates `login` types of identifiers, using the usernames of the system where it is run. Example:: contribution: committer method: files dirs: /srv/cvs.debian.org/cvs/webwml url: https://alioth.debian.org/users/{user}/ Configuration options ````````````````````` ``dirs`` : Glob, required, default: None. directories to scan. You can give one or more, and even use shell-style globbing. ``url`` : Char, optional, default: None. template used to build URLs to link to people's contributions. ``{user}`` will be replaced with the username Option types ```````````` ``Char`` A string value. Can be any UTF-8 string. ``Glob`` A string with one or more filenames. Globbing is supported. Arguments can be quoted to deal with whitespace, but glob characters will always be expanded. gitdirs ------- Scan git directories using file attributes to detect contributions. Generates `login` types of identifiers, using the usernames of the system where it is run. Example:: contribution: committer method: gitdirs dirs: /srv/git.debian.org/git/collab-maint/*.git url: https://alioth.debian.org/users/{user}/ Configuration options ````````````````````` ``dirs`` : Glob, required, default: None. ``.git`` directories to scan. You can give one or more, and even use shell-style globbing. ``url`` : Char, optional, default: None. template used to build URLs to link to people's contributions. ``{user}`` will be replaced with the username Option types ```````````` ``Char`` A string value. Can be any UTF-8 string. ``Glob`` A string with one or more filenames. Globbing is supported. Arguments can be quoted to deal with whitespace, but glob characters will always be expanded. gitlogs ------- Scan git logs, taking note of committer and author activity Generates `email` types of identifiers, trusting whatever is in the git log. Example:: contribution: committer method: gitlogs dirs: /srv/git.debian.org/git/collab-maint/*.git Configuration options ````````````````````` ``author_map`` : IdentMap, optional, default: None. Convert author emails using the given expressions ``dirs`` : Glob, required, default: None. ``.git`` directories to scan. You can give one or more, and even use shell-style globbing. ``subdir`` : Char, optional, default: None. Limit the scan to subdirectories in the repository. ``url`` : Char, optional, default: None. template used to build URLs to link to people's contributions. ``{email}`` will be replaced with the email address. Option types ```````````` ``Char`` A string value. Can be any UTF-8 string. ``Glob`` A string with one or more filenames. Globbing is supported. Arguments can be quoted to deal with whitespace, but glob characters will always be expanded. ``IdentMap`` A string with one or more identifier mapping expressions. Each expression is on a line by its own. Leading and trailing spaces do not matter. Lines can be in one of two forms: regexp replace regexp replace flags If regexp, replace or flags contain spaces, they can be shell-quoted. Regexp and replace use the syntax as found in re.sub. Flags are as found in re.X. For each mapping line, re.sub if called on each value found. mailfrom -------- Scan email address from From: headers in mailboxes Example:: contribution: developer method: mailfrom folders: /home/debian/lists/debian-devel-announce/* url: http://www.example.com/{email} Configuration options ````````````````````` ``blacklist`` : Emails, optional, default: None. if present, emails from this list will not be considered as contributors. ``folders`` : Glob, required, default: None. mail folders to scan. You can give one or more, and even use shell-style globbing. Mailbox, mailbox.gz and Maildir folders are supported. ``url`` : Char, optional, default: None. template used to build URLs to link to people's contributions. ``{email}`` will be replaced with the email address ``whitelist`` : Emails, optional, default: None. if present, only emails from this list will be considered as contributors. Option types ```````````` ``Char`` A string value. Can be any UTF-8 string. ``Emails`` A list of email addresses, like in email To: or Cc: headers. ``Glob`` A string with one or more filenames. Globbing is supported. Arguments can be quoted to deal with whitespace, but glob characters will always be expanded. mock ---- Generate random contributions for random people Example:: identifier_type: email method: mock count: 10000 Configuration options ````````````````````` ``count`` : Integer, optional, default: 1000. Number of contributions to generate. ``identifier_type`` : IdentifierType, optional, default: None. identifier type ``url`` : Char, optional, default: None. template used to build URLs to link to people's contributions. ``{email}`` will be replaced with the email address, ``{user}`` will be replaced with the user name, ``{fpr}`` will be replaced with the user key fingerprint. Option types ```````````` ``Char`` A string value. Can be any UTF-8 string. ``IdentifierType`` An identifier type. Can be one of: ``auto`` autodetect. "ident" or "Name " are accepted, and ident can be any email, login or OpenPGP fingerprint ``login`` debian.org or Alioth login name. ``email`` email address. ``fpr`` OpenPGP key fingerprint. ``Integer`` An integer value. postgres -------- Perform data mining using a SQL query on a Postgres database. This requires python-psycopg2 to be installed. Example:: contribution: uploader method: postgres db: service=projectb identifier: login query: SELECT s.install_date as date, u.uid as id, u.name as desc FROM source s JOIN fingerprint f ON s.sig_fpr = f.id JOIN uid u ON f.uid = u.id url: http://qa.debian.org/developer.php?login={id}&comaint=yes Configuration options ````````````````````` ``db`` : Char, required, default: None. database connection string. See `psycopg2.connect `_ for details. ``identifier`` : IdentifierType, optional, default: 'auto'. type of identifier that is found by this SQL query. ``query`` : Char, required, default: None. SQL query used to list contributions. SELECT column field names are significant: ``id`` is the contributor name, email, or fingerprint, depending on how ``identifier`` is configured. ``date`` is the contribution date, as a date or datetime. ``desc`` (optional) is a human-readable description for this ``id``, like a person's name. All other SELECT columns are ignored, but can be useful to provide values for the ``url`` template. ``url`` : Char, optional, default: None. template used to build URLs to link to people's contributions. Words in curly braces (like ``{id}``) will be expanded with the SELECT column of the same name. Option types ```````````` ``Char`` A string value. Can be any UTF-8 string. ``IdentifierType`` An identifier type. Can be one of: ``auto`` autodetect. "ident" or "Name " are accepted, and ident can be any email, login or OpenPGP fingerprint ``login`` debian.org or Alioth login name. ``email`` email address. ``fpr`` OpenPGP key fingerprint. svndirs ------- Scan subversion directories using file attributes to detect contributions. Generates `login` types of identifiers, using the usernames of the system where it is run. Example:: contribution: committer method: svndirs dirs: /srv/svn.debian.org/svn/collab-maint url: https://alioth.debian.org/users/{user}/ Configuration options ````````````````````` ``dirs`` : Glob, required, default: None. subversion directories to scan. You can give one or more, and even use shell-style globbing. ``url`` : Char, optional, default: None. template used to build URLs to link to people's contributions. ``{user}`` will be replaced with the username Option types ```````````` ``Char`` A string value. Can be any UTF-8 string. ``Glob`` A string with one or more filenames. Globbing is supported. Arguments can be quoted to deal with whitespace, but glob characters will always be expanded. debiancontributors-0.7.8/MANIFEST.in000066400000000000000000000002521345521122600171640ustar00rootroot00000000000000include MANIFEST.in include README.md include DATAMINING.rst include test/__init__.py include test/email/*.mbox include examples/* recursive-include test/bts_spool *.log debiancontributors-0.7.8/README.md000066400000000000000000000043311345521122600167070ustar00rootroot00000000000000debiancontributors python module ================================ See https://wiki.debian.org/DebianContributors for technical information about contributors.debian.org See https://wiki.debian.org/Teams/FrontDesk/DcSiteDevel for this project's page in the Debian wiki. ## Computing and posting data using dc-tool You can describe a data source and how to look for contribution data in a simple configuration file, then run this to perform data mining and submit data to the site, all in one go: dc-tool --mine myconfigfile --post Without --post, it prints the results of data mining on standard output: it is useful to test a data mining configuration. See DATAMINING.rst and the examples/ directory for documentation and examples for the data mining configuration. ## Posting data using dc-tool Assuming you [created a data source in the website][newds] called `myteam.debian.net` with authentication token 'foobar'. Assuming you have generated a file `submission.json`, with your submission. You can post it with `dc-tool` using: dc-tool --source myteam.debian.net --auth-token foobar --post submission.json dc-tool will validate the submission for you before posting it. You can check the contents of the submission by running `dc-tool` without any option: dc-tool submission.json ## Posting data using python code Assuming you [created a data source in the website][newds] called `myteam.debian.net` with authentication token 'foobar'. import debiancontributors as dc from datetime import Date # Create a Submission s = dc.Submission("myteam.debian.net") # Add contribution data to it s.add_contribution( dc.Identifier("email", "enrico@debian.org"), dc.Contribution("shave_yaks", Date(2013, 1, 1), Date(2013, 12, 23))) # Post it to the site success, info = s.post("foobar") if not success: import json print("submission failed:") print(json.dumps(info, indent=1)) ## Posting data the way you like Posting a submission is just a matter of building a JSON data structure and posting it to the site via HTTP POST. See [DebianContributors Implementation notes](https://wiki.debian.org/DebianContributors#Implementation_notes) for details. [newds]: https://wiki.debian.org/DebianContributors#Creating_a_new_data_source debiancontributors-0.7.8/dc-tool000077500000000000000000000115341345521122600167220ustar00rootroot00000000000000#!/usr/bin/python3 # coding: utf8 # Script for mining and/or posting data to contributors.debian.org # # Copyright (C) 2013--2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Affero General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Affero General Public License for more details. # # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals import sys import json import argparse import logging def read_json_sources(source_name, pathnames): import debiancontributors as dc # Read JSON data, parsing it to validate it submission = None if pathnames: for fname in pathnames: with open(fname, "r") as fd: try: s = dc.Submission.from_json(source_name, fd) except dc.parser.ClusterFail as e: for msg in e.errors: print("{}: {}".format(fname, msg), file=sys.stderr) return None except dc.parser.Fail as e: print("{}: {}".format(fname, e.msg), file=sys.stderr) return None if submission is None: submission = s else: submission.merge_with(s) else: submission = dc.Submission.from_json(source_name, sys.stdin) return submission if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--debug", help="enable debugging output", action="store_true") parser.add_argument("--verbose", help="enable verbose output", action="store_true") parser.add_argument("--baseurl", metavar="url", help="URL to post data to (default: %(default)s)", action="store", default='https://contributors.debian.org/') parser.add_argument("--source", help="Data source name") parser.add_argument("--auth-token", help="Authentication token. Use @file to use the file content as auth token.") parser.add_argument("--json", help="write the JSON submission to standard output", action="store_true") parser.add_argument("--post", help="POST contribution data to the site", action="store_true") parser.add_argument("--mine", metavar="conffile", action="store", default=None, help="Perform data mining using the given config file") parser.add_argument("--mine-documentation", action="store_true", default=False, help="Print data mining documentation in reStructuredText format") parser.add_argument("sources", metavar="source(s)", help="JSON file(s) to post if --mine is not provided", default=None, nargs="*") args = parser.parse_args() FORMAT = "%(asctime)-15s %(levelname)s %(message)s" if args.debug: verbose = True logging.basicConfig(level=logging.DEBUG, stream=sys.stderr, format=FORMAT) elif args.verbose: verbose = True logging.basicConfig(level=logging.INFO, stream=sys.stderr, format=FORMAT) else: verbose = False logging.basicConfig(level=logging.WARN, stream=sys.stderr, format=FORMAT) if args.mine_documentation: from debiancontributors import DataMine DataMine.print_documentation() sys.exit(0) if args.mine: from debiancontributors import DataMine miner = DataMine(args.mine, source_name=args.source) miner.scan() submission = miner.submission else: submission = read_json_sources(args.source, args.sources) if submission is None: sys.exit(1) # Override auth_token and baseurl from commandline if requested if args.auth_token: submission.set_auth_token(args.auth_token) if args.baseurl: submission.baseurl = args.baseurl if args.post: success, details = submission.post() if success: if verbose: json.dump(details, sys.stdout, indent=2) print() sys.exit(0) else: json.dump(details, sys.stderr, indent=2) print(file=sys.stderr) sys.exit(1) elif args.json: submission.to_json(sys.stdout, indent=1) else: import io with io.open(sys.stdout.fileno(), "wt", encoding="utf8", closefd=False) as out: submission.print_compact(out) sys.exit(0) debiancontributors-0.7.8/debian/000077500000000000000000000000001345521122600166515ustar00rootroot00000000000000debiancontributors-0.7.8/debian/changelog000066400000000000000000000127151345521122600205310ustar00rootroot00000000000000debiancontributors (0.7.8-1) UNRELEASED; urgency=medium * New upstream release. -- Daniele Tricoli Tue, 16 Apr 2019 01:20:42 +0200 debiancontributors (0.7.7-1) unstable; urgency=medium * Team upload. [ Enrico Zini ] * Fixed parsing compressed submissions in python3 * Convert bytes to stream of bytes * Fixed source merging [ Daniele Tricoli ] * Use items method instead of iteritems (Closes: 888588) - Thanks to clint@debian.org for submitting a patch. * debian/control - Update Vcs-Git and Vcs-Browser to salsa.debian.org [ Pierre-Elliott Bécue ] * New upstream release. * Update setup.py url to salsa.debian.org * debian/compat - Raise level to 11 * debian/control - Bump Standards-Version to 4.1.3 - Bump debhelper build depends version to 11 - Add a Homepage: tag - Make the descriptions of binary packages differ - Same for short descriptions * debian/copyright - Switch Format url to https - Update source URL to salsa.debian.org -- Pierre-Elliott Bécue Wed, 14 Feb 2018 14:27:21 +0100 debiancontributors (0.7.6-1) unstable; urgency=medium [ Daniele Tricoli ] * New upstream release. * Apply modifications from Enrico Zini: - Deal with bytes as input in python3. - Deal with the input stream to the parser being a stream of bytes. - Link bts urls with archive=both. * debian/control - Bump Standards-Version to 3.9.8 (no changes needed) -- Enrico Zini Sun, 11 Dec 2016 08:15:50 +0100 debiancontributors (0.7.5-3) unstable; urgency=medium * debian/compat - Bump debhelper compatibility level to 9. * debian/control - Set Debian Python Modules Team as Maintainer. - Set Enrico Zini and myself as uploaders. (Closes: #813236) - Update Vcs-* fields. - Bump Standards-Version to 3.9.7 (no changes needed). * debian/copyright - Update Source field. - Add stanza for debian directory. -- Daniele Tricoli Sat, 27 Feb 2016 19:06:34 +0100 debiancontributors (0.7.5-2) unstable; urgency=medium * Added break+replaces to smooth upgrades from 0.7.4. Closes: #814512 -- Enrico Zini Mon, 15 Feb 2016 18:01:06 +0100 debiancontributors (0.7.5-1) unstable; urgency=high * Do not package dc-tool in the python2 version. Closes: #814512 * Fixed packaging of examples. Closes: #808885 -- Enrico Zini Sat, 13 Feb 2016 16:10:20 +0100 debiancontributors (0.7.4-1) unstable; urgency=medium * Implemented git author mapping from gitlogs, to compensate for quirks in old histories * Show errors from server and not just a 400 -- Enrico Zini Fri, 05 Feb 2016 13:53:01 +0100 debiancontributors (0.7.3-1) unstable; urgency=medium * Fixes after testing it on stable -- Enrico Zini Fri, 05 Feb 2016 12:25:45 +0100 debiancontributors (0.7.2-1) unstable; urgency=medium * Ported to python3 * Fixed postgresql data source when an identifier url is not available * Updated standards-version, no changes required -- Enrico Zini Fri, 05 Feb 2016 11:29:57 +0100 debiancontributors (0.7.1-1) unstable; urgency=medium * Added a workaround for REQUESTS_CA_BUNDLE=/etc/ssl/ca-debian/ca-certificates.crt required on debian.org machines -- Enrico Zini Mon, 09 Mar 2015 17:55:06 +0100 debiancontributors (0.7-1~bpo70+2) wheezy-backports; urgency=medium * Added X-Python-Version: >= 2.7 -- Enrico Zini Mon, 31 Mar 2014 10:45:47 +0200 debiancontributors (0.7-1~bpo70+1) wheezy-backports; urgency=medium * Backported to wheezy -- Enrico Zini Sun, 30 Mar 2014 17:15:03 +0200 debiancontributors (0.7-1) unstable; urgency=high * svndirs: recursively scan db/revs directory instead of stopping at the first level * dc-tool: do not print submission results unless we have --verbose or --debug * Urgency set to high to speed up transition to testing/backports, since this package is mainly used in debian machines. -- Enrico Zini Fri, 21 Mar 2014 11:13:30 +0100 debiancontributors (0.6-1~bpo70+1) wheezy-backports; urgency=medium * Backported to wheezy -- Enrico Zini Sat, 18 Jan 2014 16:57:59 +0100 debiancontributors (0.6-1) unstable; urgency=medium * Allow to load auth_token from a separate file -- Enrico Zini Sat, 11 Jan 2014 22:29:18 +0100 debiancontributors (0.5-1) unstable; urgency=medium * Work with git 1.7 -- Enrico Zini Sat, 11 Jan 2014 19:15:00 +0100 debiancontributors (0.4-1) unstable; urgency=medium * New data mining methods: postgres, mailfrom * Documentation and examples updated -- Enrico Zini Sat, 11 Jan 2014 16:40:25 +0100 debiancontributors (0.3-1) unstable; urgency=medium * Redone data mining framework * Depend on python-debian * Data mining methods: files, gitdirs, gitlogs, svndirs * Install documentation in /usr/share/doc -- Enrico Zini Sat, 11 Jan 2014 02:03:23 +0100 debiancontributors (0.2-1) unstable; urgency=medium * Relicensed tests under LGPL. -- Enrico Zini Sat, 28 Dec 2013 08:46:34 +0100 debiancontributors (0.1-1) unstable; urgency=low * Initian version, packaged from a template created by stdeb 0.6.0+git Closes: #732991 -- Enrico Zini Mon, 23 Dec 2013 17:58:44 +0100 debiancontributors-0.7.8/debian/clean000066400000000000000000000000231345521122600176510ustar00rootroot00000000000000MANIFEST dc-tool.1 debiancontributors-0.7.8/debian/compat000066400000000000000000000000031345521122600200500ustar00rootroot0000000000000011 debiancontributors-0.7.8/debian/control000066400000000000000000000035471345521122600202650ustar00rootroot00000000000000Source: debiancontributors Maintainer: Debian Python Modules Team Uploaders: Enrico Zini , Daniele Tricoli Section: python Priority: optional Build-Depends: debhelper (>= 11), dh-python, python-all, python-setuptools, python-six, python-debian, python3-all, python3-setuptools, python3-six, python3-debian Build-Depends-Indep: help2man Standards-Version: 4.1.3 Homepage: https://salsa.debian.org/python-team/modules/python-debiancontributors Vcs-Git: https://salsa.debian.org/python-team/modules/python-debiancontributors.git Vcs-Browser: https://salsa.debian.org/python-team/modules/python-debiancontributors Package: python3-debiancontributors Architecture: all Depends: python3-requests (>= 2.0.0), python3-debian, ${misc:Depends}, ${python3:Depends} Breaks: python-debiancontributors (<< 0.7.5) Replaces: python-debiancontributors (<< 0.7.5) Description: Manage submissions to contributors.debian.org (Python3) This module contains the code to submit and parse contributions to contributors.debian.org. . It can be used to prepare submissions and submit them to the site, and it is used by the site to parse and validate them. . This package contains the Python3 version of the code. Package: python-debiancontributors Architecture: all Depends: python-requests (>= 2.0.0), python-debian, ${misc:Depends}, ${python:Depends} Description: Manage submissions to contributors.debian.org This module contains the code to submit and parse contributions to contributors.debian.org. . It can be used to prepare submissions and submit them to the site, and it is used by the site to parse and validate them. . This package contains the Python2 version of the code. debiancontributors-0.7.8/debian/copyright000066400000000000000000000011061345521122600206020ustar00rootroot00000000000000Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Upstream-Name: python-debiancontributors Upstream-Contact: Enrico Zini Source: https://salsa.debian.org/python-team/modules/python-debiancontributors Files: * Copyright: 2013 Enrico Zini License: LGPL-3+ Files: debian/* Copyright: 2013, Enrico Zini 2016, Daniele Tricoli License: LGPL-3+ License: LGPL-3+ The full text of the LGPL version 3 is distributed in /usr/share/common-licenses/LGPL-3 on Debian systems. debiancontributors-0.7.8/debian/docs000066400000000000000000000000311345521122600175160ustar00rootroot00000000000000README.md DATAMINING.rst debiancontributors-0.7.8/debian/examples000066400000000000000000000000131345521122600204040ustar00rootroot00000000000000examples/* debiancontributors-0.7.8/debian/manpages000066400000000000000000000000121345521122600203600ustar00rootroot00000000000000dc-tool.1 debiancontributors-0.7.8/debian/rules000077500000000000000000000014261345521122600177340ustar00rootroot00000000000000#!/usr/bin/make -f srcname = debiancontributors version = $(shell sed -nre 's/.*version[ ]*=[ ]*"([^"]+)".*/\1/p' setup.py) export PYBUILD_NAME=debiancontributors %: dh $@ --with python2,python3 --buildsystem=pybuild override_dh_auto_build: dh_auto_build COLUMNS=200 help2man --name='Manage contributions to contributors.debian.org' --section=1 --version-string="$(version)" --no-info ./dc-tool > dc-tool.1 debsrc: python3 setup.py sdist mv dist/$(srcname)-$(version).tar.gz ../$(srcname)_$(version).orig.tar.gz gbp buildpackage -S -us -uc bpodebsrc: if [ ! -e ../$(srcname)_$(version).orig.tar.gz ] ; then \ echo "Please copy the original $(srcname)_$(version).orig.tar.gz in .." >&2 ; \ exit 1 ; \ fi gbp buildpackage --git-debian-branch=backport -S -us -uc debiancontributors-0.7.8/debian/source/000077500000000000000000000000001345521122600201515ustar00rootroot00000000000000debiancontributors-0.7.8/debian/source/format000066400000000000000000000000141345521122600213570ustar00rootroot000000000000003.0 (quilt) debiancontributors-0.7.8/debian/source/options000066400000000000000000000000371345521122600215670ustar00rootroot00000000000000extend-diff-ignore="\.egg-info"debiancontributors-0.7.8/debiancontributors/000077500000000000000000000000001345521122600213275ustar00rootroot00000000000000debiancontributors-0.7.8/debiancontributors/__init__.py000066400000000000000000000017441345521122600234460ustar00rootroot00000000000000# coding: utf8 # Handle submissions to contributors.debian.org # # Copyright (C) 2013 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from .types import * from .submission import * from .datamine import * debiancontributors-0.7.8/debiancontributors/datamine.py000066400000000000000000000200771345521122600234710ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source data mining tools # # Copyright (C) 2013--2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from .submission import DEFAULT_BASE_URL, Submission from debian import deb822 import os.path import io import re import six import sys __all__ = ["Fail", "DataMine"] class Fail(BaseException): pass def read_config(fname): with io.open(fname, encoding="utf8") as fp: for par in deb822.Deb822.iter_paragraphs(fp): yield par def read_configstr(s): if not isinstance(s, six.text_type): raise TypeError("configuration contents must be a unicode string") with io.StringIO(s) as fp: for par in deb822.Deb822.iter_paragraphs(fp): yield par def load_scanners(): """ Load all scanners as a sequence of scanner classes """ from . import scanners from .scanner import Scanner import inspect for name, cls in inspect.getmembers(scanners, inspect.isclass): if not issubclass(cls, Scanner): continue # Make sure that NAME is set if cls.NAME is None: cls.NAME = name.lower() yield cls class DataMine(object): def __init__(self, configfname=None, configstr=None, source_name=None): """ Create a data miner for a data source reading a configuration file. If the first paragraph does not have a "contribution:" field, it is used for general data source configuration, like auth key, source name (if not the same as the file name), and base url (if the default is not ok) The source name is the value of source_name, if given. Else it is the value in general/name. Else it is the basename of configfname, with .conf or .cfg extension stripped, if present. """ # Read all the configuration as a dict { section: { key: value } } if configfname is not None: config = list(read_config(configfname)) elif configstr is not None: config = list(read_configstr(configstr)) else: raise TypeError("one of configfname or configstr should be provided") if not config: raise Fail("the configuration is empty") # Extract the general configuration name = source_name auth_token = None baseurl = None general = config[0] if "contribution" not in general: config = config[1:] if not name: name = general.get("source", None) auth_token = general.get("auth_token", None) baseurl = general.get("baseurl", DEFAULT_BASE_URL) # Default source with the config file name, without config-like # extensions if name is None: name = os.path.basename(configfname) name = re.sub(r".(?:cfg,conf)$", "", name) # Instantiate the submission that we are going to build self.submission = Submission(name, auth_token=auth_token, baseurl=baseurl) # Instantiate scanners self.scanners = [] scanner_factories = { x.NAME: x for x in load_scanners() } for cfg in config: # Contribution type ctype = cfg.get("contribution", None) if ctype is None: raise Fail("'contribution' field not found in data miner configuration") # Get scanner class 'method' configuration method = cfg.get("method", None) if method is None: raise Fail("'method' field not found in data miner configuration") scanner_cls = scanner_factories.get(method, None) if scanner_cls is None: raise Fail("'{}' configuration requests unsupported method: '{}'".format(ctype, method)) # Instantiate scanner self.scanners.append({ "ctype": ctype, "method": method, "scanner": scanner_cls(cfg), }) def scan(self): """ Run all data miners and add their output to the submission """ for s in self.scanners: ctype = s["ctype"] for ident, begin, until, url in s["scanner"].scan(): self.submission.add_contribution_data( ident, ctype, begin, until, url) @classmethod def print_documentation(cls, file=sys.stdout): print(""" =================== dc-tool data mining =================== dc-tool has several methods of data mining that can be controlled via a configuration file. It works like this: 1. Read this documentation and create a configuration file to test. 2. Run ``dc-tool --mine=mysource.conf`` to perform data mining and print results to standard output. 3. When you are satisfied of the results, run ``dc-tool --mine=mysource.conf --post`` to post data to contributors.debian.org. Run that via cron and you have a full working data source. ------------------------- Configuration file syntax ------------------------- The configuration file follows the usual Debian RFC822/Yaml-like syntax. If the first group of options does not have a "contribution:" field, it is used for general configuration of the data source. All other sections define methods of mining the data you want. The data source configuration section ===================================== Example:: # You don't need this option if you call this file nm.debian.org.conf #source: nm.debian.org # Auhentication token used to post data. Use a leading '@' as in '@filename' # to use the contents of another file as auth token. Do not make this file # world-readable! auth_token: @secrets/auth_token.txt The general configuration section has three configurable keywords: ``source`` Data source name, as configured in contributors.debian.org. If omitted, dc-tool will use the configuration file name. If the file name ends in ``.ini``, ``.conf`` or ``.cfg``, the extension will be removed. ``auth_token`` The authentication token used for posting data to the site. Anyone with this authentication token can post data for this data source, so be careful not to give this file world-readable permissions. ``baseurl`` You never need this unless you want to test a local copy of the contributors.debian.org codebase: it defaults to ``{DEFAULT_BASE_URL}`` but you can change it to submit data to your local development version. Data mining sections ==================== Example:: contribution: committer # Data mining method method: gitdirs # Configuration specific to this method dirs: /srv/git.debian.org/git/collab-maint/*.git url: https://alioth.debian.org/users/{{user}}/ Each data mining section has at least two configurable keywords: ``contribution`` Contribution type for this data source, as configured in contributors.debian.org. You can have many sections with the same contribution types, and the results of their data mining will all be merged. ``method`` The mining method. There are several mining method available, each with its own configuration options, documented below. The rest of the options are specific to each data mining method. Below is a full documentation of them. Data mining methods =================== """.format(DEFAULT_BASE_URL=DEFAULT_BASE_URL), file=file) for scanner in sorted(load_scanners(), key=lambda x:x.NAME): scanner.print_documentation(file=file) debiancontributors-0.7.8/debiancontributors/parser.py000066400000000000000000000200051345521122600231720ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source parser from untrusted data # # Copyright (C) 2013 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals import datetime import json import codecs import io import six class Fail(BaseException): """ Exception raised when a validation or lookup fails """ def __init__(self, code, msg): self.code = code self.msg = msg class ClusterFail(Fail): """ Exception raised to report a number of errors of the same kind """ def __init__(self, code, msg, errors): super(ClusterFail, self).__init__(code, msg) self.errors = errors def get_key(d, key): "Get a key from a dict" try: return d[key] except KeyError: raise Fail(400, "Key '{}' not found".format(key)) def get_key_int(d, key): "Get a key from a dict, as an int" try: return int(get_key(d, key)) except ValueError: raise Fail(400, "Key '{}' does not contain an integer value".format(key)) def get_key_string(d, key, empty=False): "Get a key from a dict, as a string" if empty: res = d.get(key, "") if not res: return "" else: res = get_key(d, key) try: res = str(res) except ValueError: raise Fail(400, "Key '{}' does not contain a string value".format(key)) if not empty and not res: raise Fail(400, "Key '{}' contains an empty string".format(key)) return res def get_key_unicode(d, key, empty=False): "Get a key from a dict, as a unicode, decoded from utf8 if necessary" if empty: res = d.get(key, "") if not res: return "" else: res = get_key(d, key) if not res: raise Fail(400, "Key '{}' contains an empty string".format(key)) if isinstance(res, six.text_type): return res if not isinstance(res, six.binary_type): raise Fail(400, "Key '{}' does not contain a string value".format(key)) try: return res.decode("utf8") except (UnicodeEncodeError, UnicodeDecodeError): escaped = res.decode(encoding="utf8", errors="replace") raise Fail(400, "Key '{}' contain {} which is not a valid UTF8 string".format(key, escaped)) def get_key_sequence(d, key): "Get a key from a dict, ensuring it is a list or tuple" res = get_key(d, key) if not isinstance(res, (list, tuple)): raise Fail(400, "Key '{}' does not contain an array".format(key)) return res def get_key_sequence_or_object(d, key): """ Get a key from a dict, ensuring it is a list or tuple, allowing singleton lists of objects to be just the object itself """ res = get_key(d, key) if isinstance(res, (list, tuple)): return res elif isinstance(res, dict): return [res] else: raise Fail(400, "Key '{}' does not contain an array or object".format(key)) def get_key_date_or_none(d, key): "Get a key from a dict, as a date, allowing None" res = get_key_string(d, key, empty=True) if not res: return None try: return datetime.datetime.strptime(res, "%Y-%m-%d").date() except ValueError: raise Fail(400, "Key '{}' does not contain a YYYY-MM-DD date".format(key)) def get_json(f, compression=None): """ Parse JSON from data from a file-like object, with optional decompression """ if compression: if compression == "gzip": import gzip try: with gzip.GzipFile(mode="rb", fileobj=f) as fd: try: reader = codecs.getreader("utf-8") return json.load(reader(fd)) except (ValueError, UnicodeDecodeError): raise Fail(400, "invalid JSON data") except IOError: raise Fail(400, "invalid gzip compressed data") elif compression == "xz": try: import lzma except ImportError: raise Fail(500, "but python-lzma is not installed to decode xz-compressed data") try: reader = codecs.getreader("utf-8") return json.load(reader(io.BytesIO(lzma.decompress(f.read())))) except IOError: raise Fail(400, "invalid xz compressed data") except (ValueError, UnicodeDecodeError): raise Fail(400, "invalid JSON data") else: raise Fail(500, "{} compression is not supported".format(compression)) else: try: reader = codecs.getreader("utf-8") return json.load(reader(f)) except (ValueError, UnicodeDecodeError): raise Fail(400, "invalid JSON data") class Parser(object): def parse_identifier(self, d): """ Parse a dict as an Identifier """ from .types import Identifier i_type = get_key_string(d, "type") i_id = get_key_unicode(d, "id") i_desc = get_key_unicode(d, "desc", True) res = Identifier(i_type, i_id, i_desc) res.validate() return res def parse_contribution(self, d): """ Parse a dict as a Contribution """ from .types import Contribution c_type = get_key_string(d, "type") c_begin = get_key_date_or_none(d, "begin") c_until = get_key_date_or_none(d, "end") c_url = get_key_unicode(d, "url", True) or None res = Contribution(c_type, c_begin, c_until, c_url) res.validate() return res def parse_submission(self, seq): """ Parse a sequence as a submission generate a sequence of (ids, contributions) """ if not isinstance(seq, (list, tuple)): raise Fail(400, "Submission is not an Array") errors = [] total_count = 0 for idx, rec in enumerate(seq): total_count += 1 if not isinstance(rec, dict): errors.append("#{}: submission is not an Array") continue # Parse identifiers try: s_ids = [self.parse_identifier(d) for d in get_key_sequence_or_object(rec, "id")] except Fail as f: errors.append("#{}: cannot parse identifier(s): {}".format(idx, f.msg)) continue if not s_ids: errors.append("#{}: identifier list is empty".format(idx)) continue # Parse contributions try: s_contribs = [self.parse_contribution(d) for d in get_key_sequence_or_object(rec, "contributions")] except Fail as f: errors.append("#{} for {}: cannot parse contribution(s): {}".format(idx, s_ids[0].id, f.msg)) continue if not s_contribs: errors.append("#{} for {}: contribution list is empty".format(idx, s_ids[0].id)) continue yield s_ids, s_contribs if errors: if len(errors) == 1: raise Fail(400, errors[0]) elif len(errors) == total_count: raise ClusterFail(400, "All submissions failed", errors) else: raise ClusterFail(400, "Some submission failed", errors) debiancontributors-0.7.8/debiancontributors/scanner.py000066400000000000000000000265541345521122600233460ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source core data structure # # Copyright (C) 2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals import sys import six class ValidationError(Exception): pass class ConfigField(object): def __init__(self, name=None, blank=True, default=None, help_text="", **kw): """ name: value name in the configuration blank: True if it can have an empty value, False if an empty or missing value should give an error default: value to use if the field was not found in the configuration. help_text: documentation for this field """ self.name = name self.blank = blank self.default = default self.help_text = help_text for k in kw.keys(): raise ValueError("Unknown ConfigField argument: '{}'".format(k)) def to_python(self, val): """ Validate and convert the None or string value from the configuration file to the Python value. If val is None (missing in the configuration) and we have a default value, try to use the default value. If val is a string, strips it. If blank is False, makes sure that the string is not empty. Returns a string (possibly empty, if blank=True) if the value was found in the config file. A ValidationError (or None if blank=True) if it was not found. """ # Handle value not found in the configuration if val is None: if self.blank: return self.default else: raise ValidationError("missing value") val = val.strip() if not self.blank and not val: raise ValidationError("empty value") return val def print_documentation(self, file=sys.stdout): from .scanners.utils.doc import docstring_trim, print_indented print("``{name}`` : {type}, {blank}, default: {default}.".format( name=self.name, type=self.type_name(), blank="optional" if self.blank else "required", default=repr(self.default), ), file=file) if self.help_text: ht = docstring_trim(self.help_text) print_indented(ht, indent=2, file=file) else: print(" currently undocumented.", file=file) print(file=file) @classmethod def type_name(cls): res = cls.__name__ if res.endswith("Field"): return res[:-5] else: return res class CharField(ConfigField): """ A string value. Can be any UTF-8 string. """ pass class IntegerField(ConfigField): """ An integer value. """ def to_python(self, val): val = super(IntegerField, self).to_python(val) try: val = int(val) except ValueError: raise ValidationError("invalid integer value: {}".format(val)) return val class IdentifierTypeField(CharField): """ An identifier type. Can be one of: ``auto`` autodetect. "ident" or "Name " are accepted, and ident can be any email, login or OpenPGP fingerprint ``login`` debian.org or Alioth login name. ``email`` email address. ``fpr`` OpenPGP key fingerprint. """ def to_python(self, val): from .types import Identifier res = super(IdentifierTypeField, self).to_python(val) if res == "auto": return res if res not in Identifier.TYPE_VALIDATORS: raise ValidationError("invalid identifier type. Use one of auto, {}".format( ", ".join(sorted(Identifier.TYPE_VALIDATORS.iterkeys())))) return res class GlobField(CharField): """ A string with one or more filenames. Globbing is supported. Arguments can be quoted to deal with whitespace, but glob characters will always be expanded. """ def to_python(self, val): """ Splits with shlex, expands with glob, returns a list of pathnames """ import shlex import glob val = super(GlobField, self).to_python(val) res = [] if val is None: return res for fname in shlex.split(val): res.extend(glob.glob(fname)) if not self.blank and not res: raise ValidationError("no such file or directory") return res class EmailsField(CharField): """ A list of email addresses, like in email To: or Cc: headers. """ def to_python(self, val): """ Parse everything using email.utils.getaddresses """ from email.utils import getaddresses val = super(EmailsField, self).to_python(val) if val is None: return [] res = [ email for name, email in getaddresses((val,)) ] if not self.blank and not res: raise ValidationError("no email addresses found") return res class IdentMapField(CharField): """ A string with one or more identifier mapping expressions. Each expression is on a line by its own. Leading and trailing spaces do not matter. Lines can be in one of two forms: regexp replace regexp replace flags If regexp, replace or flags contain spaces, they can be shell-quoted. Regexp and replace use the syntax as found in re.sub. Flags are as found in re.X. For each mapping line, re.sub if called on each value found. """ def to_python(self, val): """ Splits with shlex, expands with glob, returns a list of pathnames """ import shlex import re val = super(IdentMapField, self).to_python(val) res = [] if val is None: return res for line in val.splitlines(): line = line.strip() if not line: continue vals = shlex.split(line) if len(vals) == 2: match = re.compile(vals[0]) elif len(vals) == 3: flags = 0 for flag in vals[2]: val = getattr(re, flag.upper(), None) if val is None: raise ValidationError("unsupported flag {}".format(flag)) flags |= val match = re.compile(vals[0], flags) else: raise ValidationError("mapping line has {} fields, but only 2 or 3 are supported".format(len(vals))) res.append((match, vals[1])) if not self.blank and not res: raise ValidationError("no mapping expressions provided") return res class ScannerFields(type): """ Collects all class members that are instances of ConfigField, merges them to all the instances from the class parents, and set the results as the FIELD class member. """ def __new__(meta, name, parents, attrs): # Harvest config fields config_fields = {} # Collect fields from parents for p in parents: fields = getattr(p, "FIELDS", None) if fields is None: continue config_fields.update(fields.items()) # Add fields from ourselves for name, member in attrs.items(): if not isinstance(member, ConfigField): continue # Set the default for field names if member.name is None: member.name = name config_fields[name] = member # Add a FIELDS dict with all the fields attrs["FIELDS"] = config_fields return super(ScannerFields, meta).__new__(meta, name, parents, attrs) # https://wiki.python.org/moin/PortingToPy3k/BilingualQuickRef#metaclasses class ScannerBase(object): """ Base class for Scanner to have the ScannerFields as metaclass """ class Scanner(six.with_metaclass(ScannerFields, ScannerBase)): """ Base class for all data mining scanners Declarative definition of scanner configuration goes here. Any class members that are instances of ConfigField will be used to parse and validate the configuration. Their validated results will be set as object members. Example: # When instantiated, self.dirs will be a list of pathnames dirs = GlobField(blank=False, help_text="Directories to scan") All ConfigField instances found as class members, will be stored in the class FIELDS dict. For example, you can crudely document all the config options of a scanner like this: for name, field in MyScanner.FIELDS.items(): print("Config key {}, accessible as self.{}: {}".format( field.name, name, field.__doc__)) """ # Scanner name, used to refer to the scanner in the mining configuration. # Defaults to the class name. NAME = None def __init__(self, cfg): """ Initialize the scanner with the given configuration dictionary """ # Parse configuration using our field definition for name, field in self.FIELDS.items(): val = cfg.get(field.name, None) try: validated_val = field.to_python(val) except ValidationError as e: raise ValidationError("{} = {}: {}".format(name, val, str(e))) # Set the validated name=value pair as an object member setattr(self, name, validated_val) def scan(self): """ Perform scan and generate 4-tuples of (identifier, begin, end, url) Only identifier cannot be None, everything else can be. """ if False: yield None, None, None, None @classmethod def print_documentation(cls, file=sys.stdout): from .scanners.utils.doc import docstring_trim, print_indented print(cls.NAME, file=file) print("-" * len(cls.NAME), file=file) print(docstring_trim(cls.__doc__), file=file) print(file=file) if not cls.FIELDS: print("This scanning method has no specific configuration options", file=file) else: print("Configuration options", file=file) print("`````````````````````", file=file) print(file=file) types_used = {} for name, field in sorted(cls.FIELDS.items()): field.print_documentation(file=file) types_used.setdefault(field.__class__, name) print("Option types", file=file) print("````````````", file=file) print(file=file) for cls, name in sorted(types_used.items(), key=lambda x:x[0].type_name()): print("``{}``".format(cls.type_name()), file=file) ht = docstring_trim(cls.__doc__) print_indented(ht, indent=2, file=file) print(file=file) debiancontributors-0.7.8/debiancontributors/scanners/000077500000000000000000000000001345521122600231435ustar00rootroot00000000000000debiancontributors-0.7.8/debiancontributors/scanners/__init__.py000066400000000000000000000020701345521122600252530ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source data mining tools # # Copyright (C) 2013--2015 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from .git import * from .svn import * from .files import * from .postgres import * from .mailbox import * from .bts import * from .mock import * debiancontributors-0.7.8/debiancontributors/scanners/bts.py000066400000000000000000000053011345521122600243040ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source data mining tools # # Copyright (C) 2013--2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from .. import scanner from ..parser import Fail from ..types import * __all__ = ["Bts"] class Bts(scanner.Scanner): """ Parses the debbugs spool directories collecting contributions from mail headers. Example:: contribution: correspondant method: bts dirs: /srv/bugs.debian.org/spool/db-h/ /srv/bugs.debian.org/spool/archive/ url: https://bugs.debian.org/cgi-bin/pkgreport.cgi?correspondent={email} """ dirs = scanner.GlobField(blank=False, help_text=""" debbugs spool directories to scan. You can give one or more, and even use shell-style globbing. """) threshold = scanner.IntegerField(default=5, help_text=""" Minimum number of mails that need to exist in the BTS for an email address to be considered """) url = scanner.CharField(help_text=""" template used to build URLs to link to people's contributions. ``{email}`` will be replaced with the email address """) def scan(self): from .utils.bts import DebbugsScanner scan = DebbugsScanner() for d in self.dirs: scan.scan_dir(d) threshold = self.threshold tpl = self.url for email, (count, contrib) in scan.data.items(): if count < threshold: continue ident = Identifier("email", email) try: ident.validate() except Fail: continue yield ident, contrib.begin, contrib.end, ( tpl.format(email=email) if tpl else None ) debiancontributors-0.7.8/debiancontributors/scanners/files.py000066400000000000000000000043651345521122600246270ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source data mining tools # # Copyright (C) 2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from .. import scanner __all__ = ["Files"] class Files(scanner.Scanner): """ Recursively scan directories using file attributes to detect contributions. Generates `login` types of identifiers, using the usernames of the system where it is run. Example:: contribution: committer method: files dirs: /srv/cvs.debian.org/cvs/webwml url: https://alioth.debian.org/users/{user}/ """ dirs = scanner.GlobField(blank=False, help_text=""" directories to scan. You can give one or more, and even use shell-style globbing. """) url = scanner.CharField(help_text=""" template used to build URLs to link to people's contributions. ``{user}`` will be replaced with the username """) def scan(self): from .utils.filesystem import Filesystem scan = Filesystem() for d in self.dirs: scan.scan_all_files(d) if self.url: tpl = self.url for ident, begin, end in scan.contributions(): yield ident, begin, end, tpl.format(user=ident.id) else: for ident, begin, end in scan.contributions(): yield ident, begin, end, None debiancontributors-0.7.8/debiancontributors/scanners/git.py000066400000000000000000000115201345521122600242770ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source data mining tools # # Copyright (C) 2013--2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from .. import scanner from ..types import * import datetime import logging log = logging.getLogger(__name__) __all__ = ["GitDirs", "GitLogs"] class GitDirs(scanner.Scanner): """ Scan git directories using file attributes to detect contributions. Generates `login` types of identifiers, using the usernames of the system where it is run. Example:: contribution: committer method: gitdirs dirs: /srv/git.debian.org/git/collab-maint/*.git url: https://alioth.debian.org/users/{user}/ """ dirs = scanner.GlobField(blank=False, help_text=""" ``.git`` directories to scan. You can give one or more, and even use shell-style globbing. """) url = scanner.CharField(help_text=""" template used to build URLs to link to people's contributions. ``{user}`` will be replaced with the username """) def scan(self): from .utils.filesystem import Filesystem scan = Filesystem() for d in self.dirs: scan.scan_git_repo(d) if self.url: tpl = self.url for ident, begin, end in scan.contributions(): yield ident, begin, end, tpl.format(user=ident.id) else: for ident, begin, end in scan.contributions(): yield ident, begin, end, None class GitLogs(scanner.Scanner): """ Scan git logs, taking note of committer and author activity Generates `email` types of identifiers, trusting whatever is in the git log. Example:: contribution: committer method: gitlogs dirs: /srv/git.debian.org/git/collab-maint/*.git """ dirs = scanner.GlobField(blank=False, help_text=""" ``.git`` directories to scan. You can give one or more, and even use shell-style globbing. """) subdir = scanner.CharField(blank=True, help_text=""" Limit the scan to subdirectories in the repository. """) author_map = scanner.IdentMapField(blank=True, help_text=""" Convert author emails using the given expressions """) url = scanner.CharField(help_text=""" template used to build URLs to link to people's contributions. ``{email}`` will be replaced with the email address. """) def scan(self): from .utils.proc import stream_command_stdout from .utils.mine import Aggregate contribs = Aggregate() for d in self.dirs: try: cmd = ["git", "--git-dir", d, "log", "--all", "--pretty=tformat:%ae%x00%at%x00%ce%x00%ct"] if self.subdir: cmd += ["--", self.subdir] for line in stream_command_stdout(cmd): fields = line.split("\0") if len(fields) != 4: log.warn("Invalid git output in %s: %s", d, line.strip()) break ae, at, ce, ct = fields contribs.add(ae, int(at)) contribs.add(ce, int(ct)) except RuntimeError as e: log.warn("Git failed in %s: %s", d, str(e)) author_map = self.author_map for email, (begin, end) in contribs.items(): for match, repl in author_map: email = match.sub(repl, email) ident = Identifier("email", email) begin = datetime.date.fromtimestamp(begin) end = datetime.date.fromtimestamp(end) if self.url: yield ident, begin, end, self.url.format(email=email) else: yield ident, begin, end, None debiancontributors-0.7.8/debiancontributors/scanners/mailbox.py000066400000000000000000000101341345521122600251470ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data mining on emails # # Copyright (C) 2013--2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from ..types import Identifier from .utils.mine import Aggregate from .utils.email import get_mailbox from .. import scanner import email.utils import datetime import time __all__ = ["MailFrom"] class MailFrom(scanner.Scanner): """ Scan email address from From: headers in mailboxes Example:: contribution: developer method: mailfrom folders: /home/debian/lists/debian-devel-announce/* url: http://www.example.com/{email} """ folders = scanner.GlobField(blank=False, help_text=""" mail folders to scan. You can give one or more, and even use shell-style globbing. Mailbox, mailbox.gz and Maildir folders are supported. """) whitelist = scanner.EmailsField(help_text=""" if present, only emails from this list will be considered as contributors. """) blacklist = scanner.EmailsField(help_text=""" if present, emails from this list will not be considered as contributors. """) url = scanner.CharField(help_text=""" template used to build URLs to link to people's contributions. ``{email}`` will be replaced with the email address """) def scan(self): # Build a filter function from whitelist and blacklist whitelist = frozenset(self.whitelist) blacklist = frozenset(self.blacklist) if whitelist and blacklist: filter_func = lambda x: x in whitelist and x not in blacklist elif whitelist: filter_func = lambda x: x in whitelist elif blacklist: filter_func = lambda x: x not in blacklist else: filter_func = lambda x: True contribs = Aggregate() desc_by_email = {} for pathname in self.folders: folder = get_mailbox(pathname) try: for msg in folder: # Extract From address addr = msg.get("From", None) if addr is None: continue name, addr = email.utils.parseaddr(addr) if not filter_func(addr): continue if name: desc_by_email[addr] = name # Extract date date = msg.get("Date", None) if date is None: continue date = email.utils.parsedate(date) if date is None: continue ts = time.mktime(date) contribs.add(addr, ts) finally: folder.close() for addr, (begin, end) in contribs.items(): ident = Identifier("email", addr, desc_by_email.get(addr, None)) begin = datetime.date.fromtimestamp(begin) end = datetime.date.fromtimestamp(end) if self.url: yield ident, begin, end, self.url.format(email=addr) else: yield ident, begin, end, None debiancontributors-0.7.8/debiancontributors/scanners/mock.py000066400000000000000000000104471345521122600244540ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source data mining tools # # Copyright (C) 2013--2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from .. import scanner from ..parser import Fail from ..types import * import datetime import random __all__ = ["Mock"] class Generator(object): def __init__(self): self.allowed_chars = "abcdefghijklmnopqrstuvwxyz" self.fpr_allowed_chars = "ABCDEF0123456789" self.today = datetime.date.today() def make_string(self, string_len): return "".join(random.choice(self.allowed_chars) for i in range(string_len)) def make_username(self): return self.make_string(random.randrange(3, 15)) def make_fingerprint(self): return "".join(random.choice(self.fpr_allowed_chars) for i in range(40)) def make_email_domain(self): components = [] for i in range(random.randrange(1, 4)): components.append(self.make_string(random.randrange(3, 10))) components.append(self.make_string(random.randrange(2, 4))) return ".".join(components) def make_contribution_range(self): date_until = self.today - datetime.timedelta(days=random.triangular(1, 365 * 5)) date_since = date_until - datetime.timedelta(days=random.triangular(1, 365 * 10)) return date_since, date_until def make_person(self): res = { "user": self.make_username(), "name": " ".join((self.make_string(random.randrange(3, 8)).capitalize(), self.make_string(random.randrange(3, 8)).capitalize())), "fpr": self.make_fingerprint(), } res["email"] = "@".join((res["user"], self.make_email_domain())) return res def make_identifier(self, person, type): if type == "login": return Identifier(type, person["user"], person["name"]) elif type == "email": return Identifier(type, person["email"], person["name"]) elif type == "fpr": return Identifier(type, person["fpr"], person["name"]) elif type == "url": return Identifier(type, "https://example.org/users/" + person["user"]) else: raise KeyError("Identifier type {} is not supported".format(type)) class Mock(scanner.Scanner): """ Generate random contributions for random people Example:: identifier_type: email method: mock count: 10000 """ identifier_type = scanner.IdentifierTypeField(help_text=""" identifier type """) count = scanner.IntegerField(default=1000, help_text=""" Number of contributions to generate. """) url = scanner.CharField(help_text=""" template used to build URLs to link to people's contributions. ``{email}`` will be replaced with the email address, ``{user}`` will be replaced with the user name, ``{fpr}`` will be replaced with the user key fingerprint. """) def scan(self): gen = Generator() for i in range(self.count): person = gen.make_person() date_since, date_until = gen.make_contribution_range() url = self.url.format(**person) if self.url else None ident = gen.make_identifier(person, self.identifier_type) yield ident, date_since, date_until, url debiancontributors-0.7.8/debiancontributors/scanners/postgres.py000066400000000000000000000120351345521122600253640ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source data mining tools for dak # # Copyright (C) 2013--2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from ..types import * from .. import scanner import datetime import logging log = logging.getLogger(__name__) __all__ = ["Postgres"] class Postgres(scanner.Scanner): """ Perform data mining using a SQL query on a Postgres database. This requires python-psycopg2 to be installed. Example:: contribution: uploader method: postgres db: service=projectb identifier: login query: SELECT s.install_date as date, u.uid as id, u.name as desc FROM source s JOIN fingerprint f ON s.sig_fpr = f.id JOIN uid u ON f.uid = u.id url: http://qa.debian.org/developer.php?login={id}&comaint=yes """ db = scanner.CharField(blank=False, help_text=""" database connection string. See `psycopg2.connect `_ for details.""") identifier = scanner.IdentifierTypeField(default="auto", help_text=""" type of identifier that is found by this SQL query. """) query = scanner.CharField(blank=False, help_text=""" SQL query used to list contributions. SELECT column field names are significant: ``id`` is the contributor name, email, or fingerprint, depending on how ``identifier`` is configured. ``date`` is the contribution date, as a date or datetime. ``desc`` (optional) is a human-readable description for this ``id``, like a person's name. All other SELECT columns are ignored, but can be useful to provide values for the ``url`` template. """) url = scanner.CharField(help_text=""" template used to build URLs to link to people's contributions. Words in curly braces (like ``{id}``) will be expanded with the SELECT column of the same name. """) def scan(self): from .utils.mine import Aggregate import psycopg2 import psycopg2.extensions import psycopg2.extras psycopg2.extensions.register_type(psycopg2.extensions.UNICODE) psycopg2.extensions.register_type(psycopg2.extensions.UNICODEARRAY) db = psycopg2.connect(self.db) db.set_client_encoding('UTF8') cur = db.cursor(cursor_factory=psycopg2.extras.RealDictCursor) cur.execute(self.query) contribs = Aggregate() if self.url: url_by_id = {} else: url_by_id = None desc_by_id = {} for row in cur: # Validate id id = row.get("id", None) if not id: log.info("id is empty, skipping result row %r", row) continue # Validate date and turn it into a datetime.date date = row.get("date", None) if not date: log.info("date is empty, skipping result row %r", row) continue if isinstance(date, datetime.datetime): date = date.date() elif not isinstance(date, datetime.date): log.info("date is not a date I can understand, skipping result row %r", row) continue # Generate the URL if we didn't have one already if url_by_id is not None and id not in url_by_id: url_by_id[id] = self.url.format(**row) # Take note of desc if present desc = row.get("desc", None) if desc is not None: desc_by_id[id] = desc contribs.add(id, date) for id, (begin, end) in contribs.items(): if self.identifier == "auto": try: ident = Identifier.create_auto(id, default_desc=desc_by_id.get(id, None)) except ValueError as e: log.info("skipping identifier %s: %s", id, e) else: ident = Identifier(self.identifier, id, desc_by_id.get(id, None)) yield ident, begin, end, url_by_id.get(id, None) if url_by_id else None debiancontributors-0.7.8/debiancontributors/scanners/svn.py000066400000000000000000000044401345521122600243250ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source data mining tools # # Copyright (C) 2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from .. import scanner from ..types import * __all__ = ["SvnDirs"] class SvnDirs(scanner.Scanner): """ Scan subversion directories using file attributes to detect contributions. Generates `login` types of identifiers, using the usernames of the system where it is run. Example:: contribution: committer method: svndirs dirs: /srv/svn.debian.org/svn/collab-maint url: https://alioth.debian.org/users/{user}/ """ dirs = scanner.GlobField(blank=False, help_text=""" subversion directories to scan. You can give one or more, and even use shell-style globbing. """) url = scanner.CharField(help_text=""" template used to build URLs to link to people's contributions. ``{user}`` will be replaced with the username """) def scan(self): from .utils.filesystem import Filesystem scan = Filesystem() for d in self.dirs: scan.scan_svn_repo(d) if self.url: tpl = self.url for ident, begin, end in scan.contributions(): yield ident, begin, end, tpl.format(user=ident.id) else: for ident, begin, end in scan.contributions(): yield ident, begin, end, None debiancontributors-0.7.8/debiancontributors/scanners/utils/000077500000000000000000000000001345521122600243035ustar00rootroot00000000000000debiancontributors-0.7.8/debiancontributors/scanners/utils/__init__.py000066400000000000000000000016171345521122600264210ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data mining # # Copyright (C) 2013 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals debiancontributors-0.7.8/debiancontributors/scanners/utils/bts.py000066400000000000000000000112701345521122600254460ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data mining on debbugs spool directories # # Copyright (C) 2015 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from ...types import Identifier, Contribution import os import os.path import datetime import logging import email.utils log = logging.getLogger(__name__) __all__ = ["DebbugsScanner"] states = { 1: 'autocheck', 2: 'recips', 3: 'kill-end', 5: 'go', 6: 'html', 7: 'incoming-recv', } def read_incoming_recv(fh): """ Parse a log file and generate only incoming-recv lines (all lines of incoming emails). It generates an array of lines for each email. Lines are byte() strings. """ cur_lines = None for line in fh: if len(line) == 2: if line[0] == 7: cur_lines = [] elif line[0] in states: if cur_lines is not None: yield cur_lines cur_lines = None else: if cur_lines is not None: cur_lines.append(line) elif cur_lines is not None: cur_lines.append(line) def parse_bts_received(line): """ Parse the leading BTS log synthetic Received: line and return the date header for it, if any. If anything goes wrong, it returns None. """ pos = line.rfind(b";") if pos == -1: return None line = line[pos + 2:] try: parsed = email.utils.parsedate(line.decode("utf-8", errors="replace")) except IndexError: return None if parsed is None: return None try: date = datetime.date(parsed[0], parsed[1], parsed[2]) except ValueError: return None # Sanity cutoff date if date.year < 1995: return None return date def parse_from_header(line): """ Get the email address out of the From: header line """ # From: field realname, addr = email.utils.parseaddr(line[6:].decode("utf-8", errors="replace")) return addr def scan_file(fh): """ Scan a .log file and generate a (datetime.date, email_addr) couple for each email found. emails that do not parse correctly are ignored. The email address it generates is a str() object. """ for lines in read_incoming_recv(fh): # Parse the date from the leading synthetic Received field # Must start with /^Received: \(at \S+\) by \S+;/ if not lines[0].startswith(b"Received:"): continue date = parse_bts_received(lines[0]) if date is None: continue # Parse the email address from the From: field addr = None for l in lines[1:]: if l.startswith(b"From: "): # From: field addr = parse_from_header(l) break elif l == b"\n": # Stop at end of headers break if addr is None: continue yield date, addr class DebbugsScanner: def __init__(self): self.data = {} def scan_dir(self, pathname): """ Collect data from all {pathname}/??/*.log files """ rootfd = os.open(pathname, os.O_RDONLY) for idx, dn in enumerate(sorted(os.listdir(rootfd))): processed = 0 if len(dn) != 2: continue nnfd = os.open(dn, os.O_RDONLY, dir_fd=rootfd) for fn in os.listdir(nnfd): if not fn.endswith(".log"): continue fd = os.open(fn, os.O_RDONLY, dir_fd=nnfd) fh = os.fdopen(fd, "rb") for date, addr in scan_file(fh): c = self.data.get(addr, None) if c is None: self.data[addr] = [1, Contribution("correspondant", date, date)] else: c[0] += 1 c[1].extend_by_date(date) processed += 1 log.info("%s: %d emails processed", os.path.join(pathname, dn), processed) debiancontributors-0.7.8/debiancontributors/scanners/utils/doc.py000066400000000000000000000044121345521122600254230ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source data mining tools # # Copyright (C) 2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals import sys def docstring_trim(docstring): """ Deindent a docstring. This code has been taken from http://www.python.org/dev/peps/pep-0257/#handling-docstring-indentation and a docstring has been added because, hyronically, it did not have one. Also interesting is that the PEP has status Active since many years but this code is not in Python's standard library. """ if not docstring: return '' # Convert tabs to spaces (following the normal Python rules) # and split into a list of lines: lines = docstring.expandtabs().splitlines() # Determine minimum indentation (first line doesn't count): indent = 100000 for line in lines[1:]: stripped = line.lstrip() if stripped: indent = min(indent, len(line) - len(stripped)) # Remove indentation (first line is special): trimmed = [lines[0].strip()] if indent < 100000: for line in lines[1:]: trimmed.append(line[indent:].rstrip()) # Strip off trailing and leading blank lines: while trimmed and not trimmed[-1]: trimmed.pop() while trimmed and not trimmed[0]: trimmed.pop(0) # Return a single string: return '\n'.join(trimmed) def print_indented(s, indent=4, file=None): indent = " " * indent for line in s.split("\n"): print(indent, line, sep="", file=file) debiancontributors-0.7.8/debiancontributors/scanners/utils/email.py000066400000000000000000000035521345521122600257510ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data mining utilities # # Copyright (C) 2013--2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals import tempfile import mailbox import gzip import shutil import os.path import logging log = logging.getLogger(__name__) class CompressedMbox(mailbox.mbox): """ Read-only access of a compressed mbox using a temporary file for the uncompressed version """ def __init__(self, pathname): self.tempfile = tempfile.NamedTemporaryFile() with gzip.open(pathname) as fd: shutil.copyfileobj(fd, self.tempfile) self.tempfile.flush() mailbox.mbox.__init__(self, self.tempfile.name) def close(self): # mailbox.mbox is not a new-style object :'( mailbox.mbox.close(self) self.tempfile.close() def get_mailbox(pathname): """ Create the right Mailbox object for a pathname """ if os.path.isdir(pathname): return mailbox.Maildir(pathname) elif pathname.endswith(".gz"): return CompressedMbox(pathname) else: return mailbox.mbox(pathname) debiancontributors-0.7.8/debiancontributors/scanners/utils/filesystem.py000066400000000000000000000064001345521122600270410ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data mining on file systems # # Copyright (C) 2013--2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from ...types import Identifier from .mine import Aggregate import os import os.path import pwd import datetime import logging log = logging.getLogger(__name__) __all__ = ["Filesystem"] class Filesystem(object): """ Collect and aggregate contribution data from file inode information, and build contribution informations out of it """ def __init__(self): self.contribs = Aggregate() def scan_file(self, pathname): """ Add an information from the inode information of a file """ self.scan_stat(os.stat(pathname)) def scan_stat(self, st): """ Add an information from a stat structure """ self.contribs.add(st.st_uid, st.st_mtime) def scan_git_repo(self, gitdir): """ Add information from refs files in the given git repo gitdir: pathname to the bare repository or the .git directory """ for subdir in ("refs", "objects"): scanroot = os.path.join(gitdir, subdir) log.debug("git scanning at %s", scanroot) for root, dirs, files in os.walk(scanroot): for f in files: self.scan_file(os.path.join(root, f)) def scan_svn_repo(self, svnroot): """ Add information from commits in the given svn repo svnroot: pathname to the svn repository root dir """ scanroot = os.path.join(svnroot, "db/revs") for root, dirs, files in os.walk(scanroot): for f in files: self.scan_file(os.path.join(root, f)) def scan_all_files(self, root): """ Add information from commits in the given svn repo svnroot: pathname to the svn repository root dir """ for dirpath, dirnames, fnames in os.walk(root): for fname in fnames: self.scan_file(os.path.join(dirpath, fname)) def contributions(self): """ Generate (ident, begin, end) contributions """ for uid, stats in self.contribs.items(): try: pw = pwd.getpwuid(uid) ident = Identifier("login", pw.pw_name) begin = datetime.date.fromtimestamp(stats[0]) end = datetime.date.fromtimestamp(stats[1]) yield ident, begin, end except KeyError: pass debiancontributors-0.7.8/debiancontributors/scanners/utils/mine.py000066400000000000000000000025401345521122600256060ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data mining utilities # # Copyright (C) 2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals import logging log = logging.getLogger(__name__) __all__ = ["Aggregate"] class Aggregate(dict): """ Aggregate pairs of (key, val) in a dict { key: (minval, maxval) } """ def add(self, key, val): """ Add a (key, val) pair to the aggregation """ old = self.get(key, None) if old is None: self[key] = (val, val) else: self[key] = (min(old[0], val), max(old[1], val)) debiancontributors-0.7.8/debiancontributors/scanners/utils/proc.py000066400000000000000000000063401345521122600256230ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data mining using subprocesses # # Copyright (C) 2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals import codecs import io import fcntl import os import select import subprocess def stream_output(proc): """ Take a subprocess.Popen object and generate its output, as pairs of (tag, line) couples. Tag can be O for stdout, E for stderr and R for return value. Note that the output is not line-split. R is always the last bit that gets generated. """ fds = [proc.stdout, proc.stderr] tags = ["O", "E"] # Set both pipes as non-blocking for fd in fds: fcntl.fcntl(fd, fcntl.F_SETFL, os.O_NONBLOCK) # Multiplex stdout and stderr with different tags while len(fds) > 0: s = select.select(fds, (), ()) for fd in s[0]: idx = fds.index(fd) buf = fd.read() if buf: decoded = codecs.decode(buf, 'utf-8', 'replace') yield tags[idx], decoded else: fds.pop(idx) tags.pop(idx) res = proc.wait() yield "R", res class StreamStdoutKeepStderr(object): """ Stream lines of standard output from a Popen object, keeping all of its stderr inside a StringIO """ def __init__(self, proc): self.proc = proc self.stderr = io.StringIO() def __iter__(self): last_line = None for tag, buf in stream_output(self.proc): if tag == "O": for l in buf.splitlines(True): if last_line is not None: l = last_line + l last_line = None if l.endswith("\n"): yield l else: last_line = l elif tag == "E": self.stderr.write(buf) if last_line is not None: yield last_line def stream_command_stdout(cmd, **kw): try: proc = subprocess.Popen( cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kw) proc.stdin.close() lines = StreamStdoutKeepStderr(proc) for line in lines: yield line result = proc.wait() except: proc.terminate() raise if result != 0: raise RuntimeError("{} exited with status {}: {}".format( cmd[0], result, lines.stderr.getvalue().strip())) debiancontributors-0.7.8/debiancontributors/scanners/utils/stats.py000066400000000000000000000070351345521122600260200ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data mining on emails # # Copyright (C) 2013 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals import math def _smooth(days, decay_days): """ Return the smooth factor after 'days' days of decay, considering a maximum decay timespan of 'decay_days' """ # http://en.wikipedia.org/wiki/Bump_function if days >= decay_days: return 0 return math.e * math.exp(-1/(1-(days/decay_days)**2)) class ContributorFrequencyCheck(object): def __init__(self, ident, time_unit_length=86400, contribution_age=7, min_activity_time=60): self.ident = ident self.time_unit_length = time_unit_length self.decay_days = contribution_age self.min_activity_time = min_activity_time self.stamp_set = set() self.stamp_min = None self.stamp_max = None def add_stamp(self, ts): ts = ts // self.time_unit_length self.stamp_set.add(ts) if self.stamp_min is None: self.stamp_min = ts self.stamp_max = ts elif ts < self.stamp_min: self.stamp_min = ts elif ts > self.stamp_max: self.stamp_max = ts @property def contrib_range(self): return self.stamp_min * self.time_unit_length, self.stamp_max * self.time_unit_length def heat_function(self): # Precompute the smooth factors smooth_factors = [ _smooth(i, self.decay_days) for i in xrange(self.decay_days) ] # Compute the heat function over the whole activity timespan # TODO: can be optimizing using a deque of decay_days items, as a # moving window of the alst decay_days days as we progress on the time # axis for ts in xrange(int(self.stamp_min), int(self.stamp_max) + 1): val = 0.0 if ts in self.stamp_set: val += 1 for i in xrange(1, self.decay_days): if (ts - i) in self.stamp_set: val += smooth_factors[i] yield ts, val def is_contributor(self): threshold = 1 first_ts_above_threshold = None for ts, val in self.heat_function(): if val > threshold: if first_ts_above_threshold is None: first_ts_above_threshold = ts else: if ts - first_ts_above_threshold > self.min_activity_time: return True else: first_ts_above_threshold = None return False def test_dump_heat_function(self, fname=None): if fname is None: fname = self.ident.type + "-" + self.ident.id with open(fname, "w") as fd: for ts, val in self.heat_function(): print("{} {}".format(ts, val), file=fd) debiancontributors-0.7.8/debiancontributors/submission.py000066400000000000000000000225701345521122600241020ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source core data structure # # Copyright (C) 2013--2014 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from collections import defaultdict from .types import Contribution import sys import os import io import six import json __all__ = ["Submission"] DEFAULT_BASE_URL = "https://contributors.debian.org/" class Submission(object): """ A submission to contributors.debian.org """ def __init__(self, name, auth_token=None, baseurl=DEFAULT_BASE_URL): """ name: data source name """ # Data source name, as in the website self.name = name # Authentication token self.auth_token = None # Base URL self.baseurl = baseurl # List of contributions by identifier self.entries = defaultdict(dict) if auth_token is not None: self.set_auth_token(auth_token) def set_auth_token(self, auth_token): """ Set the auth token for this source. If auth_token starts with '@', the rest is treated as a pathname to a file that contains the token. """ if not auth_token: raise ValueError("auth_token is empty") if auth_token[0] == "@": with open(auth_token[1:], "r") as fd: self.auth_token = fd.read().strip() else: self.auth_token = auth_token def add_contribution(self, identifier, contrib): """ Add information about a contribution. identifier: Identifier for the user that made this contribution contrib: Contribution object """ entries = self.entries[identifier] old = entries.get(contrib.type, None) if old is None: entries[contrib.type] = contrib else: entries[contrib.type] = Contribution.merged(old, contrib) def add_contribution_data(self, identifier, type, begin=None, end=None, url=None): """ Add information about a contribution. identifier: Identifier for the user that made this contribution name: contribution name (chosen among the source contribution types) begin: start time of this contribution. None to reuse the last start time. end: end time of this contribution. None to mean 'now'. """ self.add_contribution(identifier, Contribution(type, begin, end, url)) def merge_with(self, submission): """ Merge another submission into this one """ if self.name != submission.name: raise ValueError("Merging submission for two different sources: {}!={}".format( self.name, submission.name)) for ident, contribs in submission.entries.items(): old = self.entries.get(ident, None) if old is None: self.entries[ident] = dict(contribs) else: self.entries[ident] = merge_contrib_dicts(old, contribs) def _gen_records(self): """ Generate DC records for serialization """ for ident, contributions in self.entries.items(): yield { "id": (ident.to_json(),), "contributions": [c.to_json() for c in contributions.values()], } def to_json(self, file=None, indent=None): """ Convert to JSON. file: if set to a file-like object, send data there. Else, return the JSON data as a string indent: passed as-is to the indent parameter of the encoder """ if file is not None: return json.dump(list(self._gen_records()), file, indent=indent) else: return json.dumps(list(self._gen_records()), indent=indent) def print_compact(self, file=sys.stdout): """ Make a compact dump of this source to the given file """ for ident, contributions in self.entries.items(): for ctype, c in sorted(contributions.items()): if ident.desc: lead = "{}:{} <{}>".format(ident.type, ident.desc, ident.id) else: lead = "{}:{}".format(ident.type, ident.id) print("{}: {} from {} to {}".format(lead, c.type, c.begin, c.end), file=file) if c.url: print("{}: {} url: {}".format(lead, c.type, c.url), file=file) def post(self): """ POST this submission to the contributors server Returns a couple (success, info). success: a bool, true if everything was imported correctly, false if there has been some problem. info: a dict with detailed status and error information, plus import statistics """ # Yuck! Python's stdlib cannot do file uploads :'( # We need to introduce an external dependency for it import requests from six.moves.urllib.parse import urljoin # Build the POST request to contributors.debian.org url = urljoin(self.baseurl, '/contributors/post') # Prepare the file to post try: import lzma compress_type = "xz" compress = lzma.compress except ImportError: import gzip compress_type = "gzip" def compress(data): out = io.BytesIO() with gzip.GzipFile(mode="wb", fileobj=out) as fd: fd.write(data) return out.getvalue() file_data = io.BytesIO(compress(self.to_json().encode("utf-8"))) files = { "data": file_data } # POST data data = { "source": self.name, "auth_token": self.auth_token, "data_compression": compress_type, } # POST everything to the server args = { "data": data, "files": files, } # If we are running on a machine that has special debian CA # certificates (like *.debian.org machines), use them if os.path.exists("/etc/ssl/ca-debian/ca-certificates.crt"): args["verify"] = "/etc/ssl/ca-debian/ca-certificates.crt" else: args["verify"] = True session = requests.Session() try: # Do a GET before the POST, to do HTTPS negotiation without a huge # payload. See #801506 res = session.get(url) res.raise_for_status() res = session.post(url, **args) res.raise_for_status() except requests.ConnectionError as e: return False, { "code": None, "errors": [ "Connection error: " + str(e) ] } except requests.HTTPError as e: try: parsed = json.loads(res.text) errors = parsed["errors"] except Exception: errors = [] return False, { "code": None, "errors": [ "Server responded with an HTTP error: " + six.text_type(e) ] + errors } # Whether the POST was successful or not, the response body contains # information and statistics in JSON format. response = res.json() if res.status_code == requests.codes.ok: return True, response else: return False, response @classmethod def from_json(cls, name, data): """ Build a Submission from previously generated JSON name: the data source name data: the JSON data, either in a string, in a file, or as a parsed data structure """ if isinstance(data, six.string_types): data = json.loads(data) elif hasattr(data, "read"): data = json.load(data) res = cls(name) from .parser import Parser parser = Parser() for ids, contribs in parser.parse_submission(data): for i in ids: res.entries[i] = {c.type: c for c in contribs} return res def merge_contrib_dicts(d1, d2): """ Merge two dicts of contributions from the same identifier. Contribution types that happen in both lists will have their timespans merged """ res = {} # Add elements from d1, merging them with d2 if they also exist in d2 for ctype, c1 in d1.items(): c2 = d2.get(ctype, None) if c2 is None: res[ctype] = c1 else: res[ctype] = Contribution.merged(c1, c2) # Add the elements that only exist in d2 for ctype, c2 in d2.items(): res.setdefault(ctype, c2) return res debiancontributors-0.7.8/debiancontributors/types.py000066400000000000000000000175121345521122600230530ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source core data structures # # Copyright (C) 2013 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from .parser import Fail import six import re __all__ = ["Identifier", "Contribution"] class Identifier(object): """ Information about a user identifier """ __slots__ = ("type", "id", "desc") # Validator regexps TYPE_VALIDATORS = { "login": re.compile(r"^[a-z0-9._-]+$"), # From http://www.regular-expressions.info/email.html "email": re.compile(r"^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}", re.I), "fpr": re.compile(r"^[A-F0-9]{32,40}$"), "url": re.compile(r"^https?://"), # "wiki": re.compile("^[A-Za-z]+$"), } def __init__(self, type, id, desc=None): self.type = type self.id = id self.desc = desc def __hash__(self): return hash(self.type) + hash(self.id) def __eq__(self, other): return (self.type == other.type and self.id == other.id) def to_json(self): """ Return a JSON-serializable structure for this identifier """ if self.desc: return { "type": self.type, "id": self.id, "desc": self.desc, } else: return { "type": self.type, "id": self.id, } def validate(self): """ Validate the contents of this Identifier, raising parser.Fail if anything fails. """ # Validate member types if not isinstance(self.type, six.string_types): raise Fail(400, "Identifier type is '{}' instead of a string".format(type(self.type))) if not isinstance(self.id, six.string_types): raise Fail(400, "Identifier id is '{}' instead of a string".format(type(self.id))) if self.desc is not None and not isinstance(self.desc, six.string_types): raise Fail(400, "Identifier desc is '{}' instead of None or a string".format(type(self.desc))) # Validator for this type type_validator = self.TYPE_VALIDATORS.get(self.type, None) if type_validator is None: raise Fail(400, "Invalid identifier type '{}'".format(self.type)) # Parse the ID and validate it if not type_validator.match(self.id): raise Fail(400, "{} '{}' is not a valid identifier".format(self.type, self.id)) @classmethod def create_auto(cls, s, default_desc=None): """ Autodetect identifier type and value from a string. 'desc' is the default description to use if not inferred automatically. """ from email.utils import getaddresses if "<" in s: # Use getaddresses instead of parseaddr because # parseaddr truncates the string at a stray command, instead of # declaring a failed parse: # parseaddr("a, ") gives ('', 'a') results = getaddresses((s,)) if len(results) == 1: # Parsing was ok desc, ident = results[0] else: # Something went wrong, possibly a stray comma. Trying again # wtih a regexp mo = re.match(r"^\s*(?:(.+)\s+)?<([^>]+)>\s*$", s) if mo: desc, ident = mo.group(1, 2) else: desc, ident = default_desc, s else: desc, ident = default_desc, s ident = ident.replace(" ", "") for type, regexp in cls.TYPE_VALIDATORS.items(): if regexp.match(ident): return cls(type, ident, desc) raise ValueError("cannot infer a valid Identifier from '{}'".format(s)) class Contribution(object): """ Information about a contribution. """ __slots__ = ("type", "begin", "end", "url") def __init__(self, type, begin=None, end=None, url=None): """ type: contribution type (as configured in contrbutors.debian.org for a source) begin: start time of this contribution. None to reuse the last start time. end: end time of this contribution. None to mean 'now'. url: URL used to list all contributions of this type from this person, if available. """ self.type = type self.begin = begin self.end = end self.url = url def __hash__(self): return hash(self.type) + hash(self.begin) + hash(self.end) def __eq__(self, other): return (self.type == other.type and self.begin == other.begin and self.end == other.end) def extend_by_date(self, date): """ Extend the date range to include the given date "Extend" is a bit imprecise: if the current end date is None (meaning 'today'), then it is set to 'date' (which could be before than today) """ if self.begin is None: self.begin = date else: self.begin = min(self.begin, date) if self.end is None: self.end = date else: self.end = max(self.end, date) def to_json(self): """ Return a JSON-serializable structure for this contribution """ res = {"type": self.type} if self.begin: res["begin"] = self.begin.strftime("%Y-%m-%d") if self.end: res["end"] = self.end.strftime("%Y-%m-%d") if self.url: res["url"] = self.url return res @classmethod def merged(cls, first, second): """ Build a Contribution with a merge of two existing ones """ if second.begin is None: begin = first.begin elif first.begin is None: begin = second.begin else: begin = min(first.begin, second.begin) if second.end is None: end = first.end elif first.end is None: end = second.end else: end = max(first.end, second.end) if first.url is None: url = second.url else: url = first.url return cls(first.type, begin, end, url) def validate(self): """ Validate the contents of this Identifier, raising parser.Fail if anything fails. """ # Validate member types if not isinstance(self.type, six.string_types): raise Fail(400, "Contribution type is '{}' instead of a string".format(type(self.type))) if self.begin is not None and not hasattr(self.begin, "strftime"): raise Fail(400, "Contribution begin is '{}' and does not look like a date or datetime".format( type(self.begin))) if self.end is not None and not hasattr(self.end, "strftime"): raise Fail(400, "Contribution end is '{}' and does not look like a date or datetime".format(type(self.end))) if self.url is not None and not isinstance(self.url, six.string_types): raise Fail(400, "Contribution URL is '{}' instead of None or a string".format(type(self.url))) debiancontributors-0.7.8/examples/000077500000000000000000000000001345521122600172455ustar00rootroot00000000000000debiancontributors-0.7.8/examples/bugs.debian.org000066400000000000000000000003111345521122600221320ustar00rootroot00000000000000contribution: correspondant method: bts dirs: /srv/bugs.debian.org/spool/db-h/ /srv/bugs.debian.org/spool/archive/ url: https://bugs.debian.org/cgi-bin/pkgreport.cgi?correspondent={email}&archive=both debiancontributors-0.7.8/examples/bugs.debian.org.conf000066400000000000000000000006111345521122600230610ustar00rootroot00000000000000contribution: bug-submission method: postgres db: service=udd query: select submitter_email as id, min(arrival), max(arrival) from all_bugs where arrival != 'epoch' and arrival <= 'now' group by submitter_email having count(*) > 1 and max(arrival) >= '-infinity' url: https://bugs.debian.org/cgi-bin/pkgreport.cgi?correspondent={id} debiancontributors-0.7.8/examples/collab-maint000066400000000000000000000004111345521122600215260ustar00rootroot00000000000000contribution: committer method: gitdirs dirs: /srv/git.debian.org/git/collab-maint/*.git url = https://alioth.debian.org/users/{user}/ contribution: committer method: svndirs dirs: /srv/svn.debian.org/svn/collab-maint url = https://alioth.debian.org/users/{user}/ debiancontributors-0.7.8/examples/debbits.conf000066400000000000000000000002141345521122600215250ustar00rootroot00000000000000source: debbits auth_token: here-your-token contribution: debbits editor method: gitlogs dirs: /srv/git.debian.org/git/debbits/debbits.git debiancontributors-0.7.8/examples/example.conf000066400000000000000000000007101345521122600215450ustar00rootroot00000000000000# Example data source definition. # This would submit data to a data source named 'example' on contributors.debian.org source: example auth_token: @example.auth_token # Data mining for contribution type 'committer' contribution: committer # Use the 'gitdirs' scanner method: gitdirs # Configuration of the gitdirs scanner. See dc-tool --mine-document for details. dirs: .git url: http://example.com/{user} contribution: committer method: gitlogs dirs: . debiancontributors-0.7.8/examples/ftp.debian.org000066400000000000000000000011371345521122600217720ustar00rootroot00000000000000contribution: upload method: postgres db: service=projectb query: SELECT s.install_date as date, u.uid as id, u.name as desc FROM source s JOIN fingerprint f ON s.sig_fpr = f.id JOIN uid u ON f.uid = u.id url: http://qa.debian.org/developer.php?login={id}&comaint=yes contribution: maint method: postgres db: service=projectb query: SELECT s.install_date as date, c.name as id FROM source s JOIN maintainer c ON s.changedby = c.id url: http://qa.debian.org/developer.php?login={id}&comaint=yes debiancontributors-0.7.8/examples/publicity.conf000066400000000000000000000002631345521122600221210ustar00rootroot00000000000000source: publicity auth_token: here-your-token contribution: publicity editor method: svndirs dirs: /srv/svn.debian.org/svn/publicity url: https://alioth.debian.org/users/{user}/ debiancontributors-0.7.8/examples/subsmine.conf000066400000000000000000000002321345521122600217360ustar00rootroot00000000000000source: debconf subtitle team auth_token: here-your-token contribution: subber method: gitlogs dirs: /srv/git.debian.org/git/debconfsubs/debconfsubs.git debiancontributors-0.7.8/examples/www.debian.org000066400000000000000000000001661345521122600220260ustar00rootroot00000000000000contribution: commit method: files dirs: /srv/cvs.debian.org/cvs/webwml url = https://alioth.debian.org/users/{user}/ debiancontributors-0.7.8/setup.py000066400000000000000000000024661345521122600171510ustar00rootroot00000000000000#!/usr/bin/env/python """ Copyright (C) 2013--2014 Enrico Zini This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this program. If not, see . """ from distutils.core import setup import sys if sys.version_info[0] < 3: scripts = [] else: scripts = ['dc-tool'] setup( name="debiancontributors", requires=['requests (>=2.0.0)'], version="0.7.8", description="Manage submissions to contributors.debian.org", author=["Enrico Zini"], author_email=["enrico@debian.org"], url="https://salsa.debian.org/python-team/modules/python-debiancontributors", license="http://www.gnu.org/licenses/lgpl-3.0.html", packages=["debiancontributors", "debiancontributors.scanners", "debiancontributors.scanners.utils"], scripts=scripts, ) debiancontributors-0.7.8/test/000077500000000000000000000000001345521122600164065ustar00rootroot00000000000000debiancontributors-0.7.8/test/__init__.py000066400000000000000000000016421345521122600205220ustar00rootroot00000000000000# coding: utf8 # Debian Contributors data source for git.debian.org # # Copyright (C) 2013 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals debiancontributors-0.7.8/test/bts_spool/000077500000000000000000000000001345521122600204125ustar00rootroot00000000000000debiancontributors-0.7.8/test/bts_spool/archive/000077500000000000000000000000001345521122600220335ustar00rootroot00000000000000debiancontributors-0.7.8/test/bts_spool/archive/99/000077500000000000000000000000001345521122600222745ustar00rootroot00000000000000debiancontributors-0.7.8/test/bts_spool/archive/99/778399.log000066400000000000000000001065271345521122600236040ustar00rootroot00000000000000 Report forwarded to debian-bugs-dist@lists.debian.org, Krystian Wlosek <kwlosek@gmail.com>:
Bug#778399; Package z88dk.   debian-bugs-dist@lists.debian.orgKrystian Wlosek  X-Loop: owner@bugs.debian.org Subject: Bug#778399: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability Reply-To: Luciano Bello , 778399@bugs.debian.org Resent-From: Luciano Bello Resent-To: debian-bugs-dist@lists.debian.org Resent-CC: Krystian Wlosek X-Loop: owner@bugs.debian.org Resent-Date: Sat, 14 Feb 2015 14:36:01 +0000 Resent-Message-ID: Resent-Sender: owner@bugs.debian.org X-Debian-PR-Message: report 778399 X-Debian-PR-Package: z88dk X-Debian-PR-Keywords: patch security X-Debian-PR-Source: z88dk Received: via spool by submit@bugs.debian.org id=B.142392441311613 (code B); Sat, 14 Feb 2015 14:36:01 +0000 Received: (at submit) by bugs.debian.org; 14 Feb 2015 14:33:33 +0000 X-Spam-Checker-Version: SpamAssassin 3.3.2-bugs.debian.org_2005_01_02 (2011-06-06) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=4.0 tests=BAYES_00,DIGITS_LETTERS, FOURLA,FROMDEVELOPER,HAS_PACKAGE,MURPHY_DRUGS_REL8,RCVD_IN_SBLXBL, RCVD_IN_SBLXBL_CBL autolearn=ham version=3.3.2-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 8; hammy, 151; neutral, 49; spammy, 0. spammytokens: hammytokens:0.000-+--Severity, 0.000-+--H*u:4.14.2, 0.000-+--H*UA:4.14.2, 0.000-+--H*u:x86_64, 0.000-+--UD:git Received: from sfa71.servidoraweb.net ([201.235.253.71] helo=nube.usla.org.ar) by buxtehude.debian.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1YMdmm-00030o-Ne for submit@bugs.debian.org; Sat, 14 Feb 2015 14:33:33 +0000 Received: by nube.usla.org.ar (Postfix, from userid 5001) id 1FD431AC9172; Sat, 14 Feb 2015 11:28:41 -0300 (ART) Received: from box.localnet (unknown [94.118.128.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by nube.usla.org.ar (Postfix) with ESMTPSA id AFAE51AC9170 for ; Sat, 14 Feb 2015 11:28:39 -0300 (ART) From: Luciano Bello To: submit@bugs.debian.org Date: Sat, 14 Feb 2015 15:33:21 +0100 Message-ID: <3441163.lIScrJmJKV@box> User-Agent: KMail/4.14.2 (Linux/3.14-2-amd64; KDE/4.14.2; x86_64; ; ) MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" ;X-Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAKlBMVEUNAgXu6OulW0diMy/Py8v38/zd4N716/KGSDy+eWLa2d0yHCEYCw7i5N7REpRjAAACTklEQVQ4jV3SMW/aQBQH8NvjxUPlqcvRwR7rYzBxFogjAZ1R5kgNUpEYOiD2c4eL2qVOBo5OqKhSiSpVcQY358Hq6luQqkod+C59d4cNzhsY/r977x4HaPp3V5Vtv7TrQjVsd1v7qNB4N606ftTpewXjsZGIe3fQ4+jYOcB3l2F3Cel87qjSAPKHYyiv68z3pWAMDQMOwvndvAHj3T+Pm+o2Ydrf5/yV04C3fgWLbQPedWr4bFouDXytIXltYGTgvh6Fn2yVz2YaJm8OdzwaMB2T82A/CLu6YzYysDl/aBFyw1thGjzBHZcH8EkofB6TODh1dMMe+gERqc9ZmNJTW+cVUEIY5SwO/A9XsGsNCXUZxQnGjNLVEawTD0IFGNOuU4+arBcaOE48fnMEm/UCq58jweqbdG0DG6j1gnLuEhZ4PKlHaehTOMqIhpUzOkDPx9hlsBjHNw0Y+JCrlbwmRIMHzAIGhXG7gkhDBqMoVk0NWEchzFfnMWt/a8AnEuBWABfFy7pjOByuoy8ypa6PW0QMDwB3RLfIIjFhYf57s5rvIRpGURT/RGWYkhw9bsyLKFAVnyFLlrI4yZowIHGBVGVkc3UEg5CQM5WfBOnwCHphSQjMlzKN0/bqRQU9AQlsJEISfEzFslq3D7ksy4DBuiyEf0vXwG2eA8iMqDdkuchFdqFBxahApaDwUrHIZSk61wogVmUJeGAclnBMiA7cD4dlgSz4TOHlhR5bivb1CM4iU1YWC1lI3SJ+XaBnVegOIbJmbCEp9/K8A8TSw/4D1hnU0kL6huUAAAAASUVORK5CYII= Delivered-To: submit@bugs.debian.org Package: z88dk Severity: important Tags: security patch The security team received a report from the CERT Coordination Center that the Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability. It looks like this package includes the affected code at that's the reason of this bug report. The patch is available here: http://gitweb.dragonflybsd.org/dragonfly.git/blobdiff/4d133046c59a851141519d03553a70e903b3eefc..2841837793bd095a82f477e9c370cfe6cfb3862c:/lib/libc/regex/regcomp.c Please, can you confirm if the binary packages are affected? Are stable and testing affected? More information, here: http://www.kb.cert.org/vuls/id/695940 https://guidovranken.wordpress.com/2015/02/04/full-disclosure-heap-overflow-in-h-spencers-regex-library-on-32-bit-systems/ A CVE id has been requested already and the report will be updated with it eventually. Cheers, luciano   Acknowledgement sent to Luciano Bello <luciano@debian.org>:
New Bug report received and forwarded. Copy sent to Krystian Wlosek <kwlosek@gmail.com>.   -t  Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.503 (Entity 5.503) Content-Type: text/plain; charset=utf-8 X-Loop: owner@bugs.debian.org From: owner@bugs.debian.org (Debian Bug Tracking System) To: Luciano Bello Subject: Bug#778399: Acknowledgement (Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability) Message-ID: References: <3441163.lIScrJmJKV@box> X-Debian-PR-Message: ack 778399 X-Debian-PR-Package: z88dk X-Debian-PR-Keywords: patch security X-Debian-PR-Source: z88dk Reply-To: 778399@bugs.debian.org Date: Sat, 14 Feb 2015 14:36:06 +0000 Thank you for filing a new Bug report with Debian. This is an automatically generated reply to let you know your message has been received. Your message is being forwarded to the package maintainers and other interested parties for their attention; they will reply in due course. Your message has been sent to the package maintainer(s): Krystian Wlosek If you wish to submit further information on this problem, please send it to 778399@bugs.debian.org. Please do not send mail to owner@bugs.debian.org unless you wish to report a problem with the Bug-tracking system. --=20 778399: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D778399 Debian Bug Tracking System Contact owner@bugs.debian.org with problems   Received: (at submit) by bugs.debian.org; 14 Feb 2015 14:33:33 +0000 From luciano@debian.org Sat Feb 14 14:33:33 2015 X-Spam-Checker-Version: SpamAssassin 3.3.2-bugs.debian.org_2005_01_02 (2011-06-06) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=4.0 tests=BAYES_00,DIGITS_LETTERS, FOURLA,FROMDEVELOPER,HAS_PACKAGE,MURPHY_DRUGS_REL8,RCVD_IN_SBLXBL, RCVD_IN_SBLXBL_CBL autolearn=ham version=3.3.2-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 8; hammy, 151; neutral, 49; spammy, 0. spammytokens: hammytokens:0.000-+--Severity, 0.000-+--H*u:4.14.2, 0.000-+--H*UA:4.14.2, 0.000-+--H*u:x86_64, 0.000-+--UD:git Return-path: Received: from sfa71.servidoraweb.net ([201.235.253.71] helo=nube.usla.org.ar) by buxtehude.debian.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1YMdmm-00030o-Ne for submit@bugs.debian.org; Sat, 14 Feb 2015 14:33:33 +0000 Received: by nube.usla.org.ar (Postfix, from userid 5001) id 1FD431AC9172; Sat, 14 Feb 2015 11:28:41 -0300 (ART) Received: from box.localnet (unknown [94.118.128.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by nube.usla.org.ar (Postfix) with ESMTPSA id AFAE51AC9170 for ; Sat, 14 Feb 2015 11:28:39 -0300 (ART) From: Luciano Bello To: submit@bugs.debian.org Subject: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability Date: Sat, 14 Feb 2015 15:33:21 +0100 Message-ID: <3441163.lIScrJmJKV@box> User-Agent: KMail/4.14.2 (Linux/3.14-2-amd64; KDE/4.14.2; x86_64; ; ) MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" ;X-Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAKlBMVEUNAgXu6OulW0diMy/Py8v38/zd4N716/KGSDy+eWLa2d0yHCEYCw7i5N7REpRjAAACTklEQVQ4jV3SMW/aQBQH8NvjxUPlqcvRwR7rYzBxFogjAZ1R5kgNUpEYOiD2c4eL2qVOBo5OqKhSiSpVcQY358Hq6luQqkod+C59d4cNzhsY/r977x4HaPp3V5Vtv7TrQjVsd1v7qNB4N606ftTpewXjsZGIe3fQ4+jYOcB3l2F3Cel87qjSAPKHYyiv68z3pWAMDQMOwvndvAHj3T+Pm+o2Ydrf5/yV04C3fgWLbQPedWr4bFouDXytIXltYGTgvh6Fn2yVz2YaJm8OdzwaMB2T82A/CLu6YzYysDl/aBFyw1thGjzBHZcH8EkofB6TODh1dMMe+gERqc9ZmNJTW+cVUEIY5SwO/A9XsGsNCXUZxQnGjNLVEawTD0IFGNOuU4+arBcaOE48fnMEm/UCq58jweqbdG0DG6j1gnLuEhZ4PKlHaehTOMqIhpUzOkDPx9hlsBjHNw0Y+JCrlbwmRIMHzAIGhXG7gkhDBqMoVk0NWEchzFfnMWt/a8AnEuBWABfFy7pjOByuoy8ypa6PW0QMDwB3RLfIIjFhYf57s5rvIRpGURT/RGWYkhw9bsyLKFAVnyFLlrI4yZowIHGBVGVkc3UEg5CQM5WfBOnwCHphSQjMlzKN0/bqRQU9AQlsJEISfEzFslq3D7ksy4DBuiyEf0vXwG2eA8iMqDdkuchFdqFBxahApaDwUrHIZSk61wogVmUJeGAclnBMiA7cD4dlgSz4TOHlhR5bivb1CM4iU1YWC1lI3SJ+XaBnVegOIbJmbCEp9/K8A8TSw/4D1hnU0kL6huUAAAAASUVORK5CYII= Delivered-To: submit@bugs.debian.org Package: z88dk Severity: important Tags: security patch The security team received a report from the CERT Coordination Center that the Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability. It looks like this package includes the affected code at that's the reason of this bug report. The patch is available here: http://gitweb.dragonflybsd.org/dragonfly.git/blobdiff/4d133046c59a851141519d03553a70e903b3eefc..2841837793bd095a82f477e9c370cfe6cfb3862c:/lib/libc/regex/regcomp.c Please, can you confirm if the binary packages are affected? Are stable and testing affected? More information, here: http://www.kb.cert.org/vuls/id/695940 https://guidovranken.wordpress.com/2015/02/04/full-disclosure-heap-overflow-in-h-spencers-regex-library-on-32-bit-systems/ A CVE id has been requested already and the report will be updated with it eventually. Cheers, luciano   Reply sent to Moritz Muehlenhoff <jmm@inutil.org>:
You have taken responsibility.   -t  MIME-Version: 1.0 X-Mailer: MIME-tools 5.503 (Entity 5.503) X-Loop: owner@bugs.debian.org From: owner@bugs.debian.org (Debian Bug Tracking System) To: Moritz Muehlenhoff Subject: Bug#778399: marked as done (Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability) Message-ID: References: <20150219152859.GA32745@inutil.org> <3441163.lIScrJmJKV@box> X-Debian-PR-Message: closed 778399 X-Debian-PR-Package: z88dk X-Debian-PR-Keywords: patch security X-Debian-PR-Source: z88dk Date: Thu, 19 Feb 2015 15:39:14 +0000 Content-Type: multipart/mixed; boundary="----------=_1424360354-3079-0" This is a multi-part message in MIME format... ------------=_1424360354-3079-0 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Your message dated Thu, 19 Feb 2015 16:28:59 +0100 with message-id <20150219152859.GA32745@inutil.org> and subject line Re: Henry Spencer regular expressions (regex) library cont= ains a heap overflow vulnerability has caused the Debian Bug report #778399, regarding Henry Spencer regular expressions (regex) library contains a heap= overflow vulnerability to be marked as done. This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact owner@bugs.debian.org immediately.) --=20 778399: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D778399 Debian Bug Tracking System Contact owner@bugs.debian.org with problems ------------=_1424360354-3079-0 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by bugs.debian.org; 14 Feb 2015 14:33:33 +0000 X-Spam-Checker-Version: SpamAssassin 3.3.2-bugs.debian.org_2005_01_02 (2011-06-06) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=4.0 tests=BAYES_00,DIGITS_LETTERS, FOURLA,FROMDEVELOPER,HAS_PACKAGE,MURPHY_DRUGS_REL8,RCVD_IN_SBLXBL, RCVD_IN_SBLXBL_CBL autolearn=ham version=3.3.2-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 8; hammy, 151; neutral, 49; spammy, 0. spammytokens: hammytokens:0.000-+--Severity, 0.000-+--H*u:4.14.2, 0.000-+--H*UA:4.14.2, 0.000-+--H*u:x86_64, 0.000-+--UD:git Return-path: Received: from sfa71.servidoraweb.net ([201.235.253.71] helo=nube.usla.org.ar) by buxtehude.debian.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1YMdmm-00030o-Ne for submit@bugs.debian.org; Sat, 14 Feb 2015 14:33:33 +0000 Received: by nube.usla.org.ar (Postfix, from userid 5001) id 1FD431AC9172; Sat, 14 Feb 2015 11:28:41 -0300 (ART) Received: from box.localnet (unknown [94.118.128.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by nube.usla.org.ar (Postfix) with ESMTPSA id AFAE51AC9170 for ; Sat, 14 Feb 2015 11:28:39 -0300 (ART) From: Luciano Bello To: submit@bugs.debian.org Subject: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability Date: Sat, 14 Feb 2015 15:33:21 +0100 Message-ID: <3441163.lIScrJmJKV@box> User-Agent: KMail/4.14.2 (Linux/3.14-2-amd64; KDE/4.14.2; x86_64; ; ) MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" ;X-Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAKlBMVEUNAgXu6OulW0diMy/Py8v38/zd4N716/KGSDy+eWLa2d0yHCEYCw7i5N7REpRjAAACTklEQVQ4jV3SMW/aQBQH8NvjxUPlqcvRwR7rYzBxFogjAZ1R5kgNUpEYOiD2c4eL2qVOBo5OqKhSiSpVcQY358Hq6luQqkod+C59d4cNzhsY/r977x4HaPp3V5Vtv7TrQjVsd1v7qNB4N606ftTpewXjsZGIe3fQ4+jYOcB3l2F3Cel87qjSAPKHYyiv68z3pWAMDQMOwvndvAHj3T+Pm+o2Ydrf5/yV04C3fgWLbQPedWr4bFouDXytIXltYGTgvh6Fn2yVz2YaJm8OdzwaMB2T82A/CLu6YzYysDl/aBFyw1thGjzBHZcH8EkofB6TODh1dMMe+gERqc9ZmNJTW+cVUEIY5SwO/A9XsGsNCXUZxQnGjNLVEawTD0IFGNOuU4+arBcaOE48fnMEm/UCq58jweqbdG0DG6j1gnLuEhZ4PKlHaehTOMqIhpUzOkDPx9hlsBjHNw0Y+JCrlbwmRIMHzAIGhXG7gkhDBqMoVk0NWEchzFfnMWt/a8AnEuBWABfFy7pjOByuoy8ypa6PW0QMDwB3RLfIIjFhYf57s5rvIRpGURT/RGWYkhw9bsyLKFAVnyFLlrI4yZowIHGBVGVkc3UEg5CQM5WfBOnwCHphSQjMlzKN0/bqRQU9AQlsJEISfEzFslq3D7ksy4DBuiyEf0vXwG2eA8iMqDdkuchFdqFBxahApaDwUrHIZSk61wogVmUJeGAclnBMiA7cD4dlgSz4TOHlhR5bivb1CM4iU1YWC1lI3SJ+XaBnVegOIbJmbCEp9/K8A8TSw/4D1hnU0kL6huUAAAAASUVORK5CYII= Delivered-To: submit@bugs.debian.org Package: z88dk Severity: important Tags: security patch The security team received a report from the CERT Coordination Center that the Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability. It looks like this package includes the affected code at that's the reason of this bug report. The patch is available here: http://gitweb.dragonflybsd.org/dragonfly.git/blobdiff/4d133046c59a851141519d03553a70e903b3eefc..2841837793bd095a82f477e9c370cfe6cfb3862c:/lib/libc/regex/regcomp.c Please, can you confirm if the binary packages are affected? Are stable and testing affected? More information, here: http://www.kb.cert.org/vuls/id/695940 https://guidovranken.wordpress.com/2015/02/04/full-disclosure-heap-overflow-in-h-spencers-regex-library-on-32-bit-systems/ A CVE id has been requested already and the report will be updated with it eventually. Cheers, luciano ------------=_1424360354-3079-0 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 778399-done) by bugs.debian.org; 19 Feb 2015 15:34:25 +0000 X-Spam-Checker-Version: SpamAssassin 3.4.0-bugs.debian.org_2005_01_02 (2014-02-07) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=4.0 tests=BAYES_00,FOURLA, MURPHY_DRUGS_REL8,RCVD_IN_DNSWL_MED,T_RP_MATCHES_RCVD autolearn=no autolearn_force=no version=3.4.0-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 10; hammy, 93; neutral, 35; spammy, 2. spammytokens:0.915-+--Center, 0.900-+--center hammytokens:0.000-+--H*r:jmm, 0.000-+--HX-SA-Exim-Scanned:inutil.org, 0.000-+--H*RU:inutil.org, 0.000-+--H*r:inutil.org, 0.000-+--H*RU:83.151.30.8 Return-path: Received: from inutil.org ([83.151.30.8]) by buxtehude.debian.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1YOT7Q-00007y-Os for 778399-done@bugs.debian.org; Thu, 19 Feb 2015 15:34:24 +0000 Received: from jmm by inutil.org with local (Exim 4.72) (envelope-from ) id 1YOT2B-00004u-1F for 778399-done@bugs.debian.org; Thu, 19 Feb 2015 16:28:59 +0100 Date: Thu, 19 Feb 2015 16:28:59 +0100 From: Moritz Muehlenhoff To: 778399-done@bugs.debian.org Subject: Re: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability Message-ID: <20150219152859.GA32745@inutil.org> References: <3441163.lIScrJmJKV@box> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3441163.lIScrJmJKV@box> User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: jmm@inutil.org X-SA-Exim-Scanned: No (on inutil.org); SAEximRunCond expanded to false On Sat, Feb 14, 2015 at 03:33:21PM +0100, Luciano Bello wrote: > Package: z88dk > Severity: important > Tags: security patch > > The security team received a report from the CERT Coordination Center that the > Henry Spencer regular expressions (regex) library contains a heap overflow > vulnerability. It looks like this package includes the affected code at that's > the reason of this bug report. This is only used when building on Windows. Cheers, Moritz ------------=_1424360354-3079-0--   Notification sent to Luciano Bello <luciano@debian.org>:
Bug acknowledged by developer.   -t  MIME-Version: 1.0 X-Mailer: MIME-tools 5.503 (Entity 5.503) X-Loop: owner@bugs.debian.org From: owner@bugs.debian.org (Debian Bug Tracking System) To: Luciano Bello Subject: Bug#778399 closed by Moritz Muehlenhoff (Re: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability) Message-ID: References: <20150219152859.GA32745@inutil.org> <3441163.lIScrJmJKV@box> X-Debian-PR-Message: they-closed 778399 X-Debian-PR-Package: z88dk X-Debian-PR-Keywords: patch security X-Debian-PR-Source: z88dk Reply-To: 778399@bugs.debian.org Date: Thu, 19 Feb 2015 15:39:14 +0000 Content-Type: multipart/mixed; boundary="----------=_1424360354-3079-1" This is a multi-part message in MIME format... ------------=_1424360354-3079-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This is an automatic notification regarding your Bug report which was filed against the z88dk package: #778399: Henry Spencer regular expressions (regex) library contains a heap = overflow vulnerability It has been closed by Moritz Muehlenhoff . Their explanation is attached below along with your original report. If this explanation is unsatisfactory and you have not received a better one in a separate message then please contact Moritz Muehlenhoff by replying to this email. --=20 778399: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D778399 Debian Bug Tracking System Contact owner@bugs.debian.org with problems ------------=_1424360354-3079-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 778399-done) by bugs.debian.org; 19 Feb 2015 15:34:25 +0000 X-Spam-Checker-Version: SpamAssassin 3.4.0-bugs.debian.org_2005_01_02 (2014-02-07) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=4.0 tests=BAYES_00,FOURLA, MURPHY_DRUGS_REL8,RCVD_IN_DNSWL_MED,T_RP_MATCHES_RCVD autolearn=no autolearn_force=no version=3.4.0-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 10; hammy, 93; neutral, 35; spammy, 2. spammytokens:0.915-+--Center, 0.900-+--center hammytokens:0.000-+--H*r:jmm, 0.000-+--HX-SA-Exim-Scanned:inutil.org, 0.000-+--H*RU:inutil.org, 0.000-+--H*r:inutil.org, 0.000-+--H*RU:83.151.30.8 Return-path: Received: from inutil.org ([83.151.30.8]) by buxtehude.debian.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1YOT7Q-00007y-Os for 778399-done@bugs.debian.org; Thu, 19 Feb 2015 15:34:24 +0000 Received: from jmm by inutil.org with local (Exim 4.72) (envelope-from ) id 1YOT2B-00004u-1F for 778399-done@bugs.debian.org; Thu, 19 Feb 2015 16:28:59 +0100 Date: Thu, 19 Feb 2015 16:28:59 +0100 From: Moritz Muehlenhoff To: 778399-done@bugs.debian.org Subject: Re: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability Message-ID: <20150219152859.GA32745@inutil.org> References: <3441163.lIScrJmJKV@box> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3441163.lIScrJmJKV@box> User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: jmm@inutil.org X-SA-Exim-Scanned: No (on inutil.org); SAEximRunCond expanded to false On Sat, Feb 14, 2015 at 03:33:21PM +0100, Luciano Bello wrote: > Package: z88dk > Severity: important > Tags: security patch > > The security team received a report from the CERT Coordination Center that the > Henry Spencer regular expressions (regex) library contains a heap overflow > vulnerability. It looks like this package includes the affected code at that's > the reason of this bug report. This is only used when building on Windows. Cheers, Moritz ------------=_1424360354-3079-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by bugs.debian.org; 14 Feb 2015 14:33:33 +0000 X-Spam-Checker-Version: SpamAssassin 3.3.2-bugs.debian.org_2005_01_02 (2011-06-06) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=4.0 tests=BAYES_00,DIGITS_LETTERS, FOURLA,FROMDEVELOPER,HAS_PACKAGE,MURPHY_DRUGS_REL8,RCVD_IN_SBLXBL, RCVD_IN_SBLXBL_CBL autolearn=ham version=3.3.2-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 8; hammy, 151; neutral, 49; spammy, 0. spammytokens: hammytokens:0.000-+--Severity, 0.000-+--H*u:4.14.2, 0.000-+--H*UA:4.14.2, 0.000-+--H*u:x86_64, 0.000-+--UD:git Return-path: Received: from sfa71.servidoraweb.net ([201.235.253.71] helo=nube.usla.org.ar) by buxtehude.debian.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1YMdmm-00030o-Ne for submit@bugs.debian.org; Sat, 14 Feb 2015 14:33:33 +0000 Received: by nube.usla.org.ar (Postfix, from userid 5001) id 1FD431AC9172; Sat, 14 Feb 2015 11:28:41 -0300 (ART) Received: from box.localnet (unknown [94.118.128.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by nube.usla.org.ar (Postfix) with ESMTPSA id AFAE51AC9170 for ; Sat, 14 Feb 2015 11:28:39 -0300 (ART) From: Luciano Bello To: submit@bugs.debian.org Subject: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability Date: Sat, 14 Feb 2015 15:33:21 +0100 Message-ID: <3441163.lIScrJmJKV@box> User-Agent: KMail/4.14.2 (Linux/3.14-2-amd64; KDE/4.14.2; x86_64; ; ) MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" ;X-Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAKlBMVEUNAgXu6OulW0diMy/Py8v38/zd4N716/KGSDy+eWLa2d0yHCEYCw7i5N7REpRjAAACTklEQVQ4jV3SMW/aQBQH8NvjxUPlqcvRwR7rYzBxFogjAZ1R5kgNUpEYOiD2c4eL2qVOBo5OqKhSiSpVcQY358Hq6luQqkod+C59d4cNzhsY/r977x4HaPp3V5Vtv7TrQjVsd1v7qNB4N606ftTpewXjsZGIe3fQ4+jYOcB3l2F3Cel87qjSAPKHYyiv68z3pWAMDQMOwvndvAHj3T+Pm+o2Ydrf5/yV04C3fgWLbQPedWr4bFouDXytIXltYGTgvh6Fn2yVz2YaJm8OdzwaMB2T82A/CLu6YzYysDl/aBFyw1thGjzBHZcH8EkofB6TODh1dMMe+gERqc9ZmNJTW+cVUEIY5SwO/A9XsGsNCXUZxQnGjNLVEawTD0IFGNOuU4+arBcaOE48fnMEm/UCq58jweqbdG0DG6j1gnLuEhZ4PKlHaehTOMqIhpUzOkDPx9hlsBjHNw0Y+JCrlbwmRIMHzAIGhXG7gkhDBqMoVk0NWEchzFfnMWt/a8AnEuBWABfFy7pjOByuoy8ypa6PW0QMDwB3RLfIIjFhYf57s5rvIRpGURT/RGWYkhw9bsyLKFAVnyFLlrI4yZowIHGBVGVkc3UEg5CQM5WfBOnwCHphSQjMlzKN0/bqRQU9AQlsJEISfEzFslq3D7ksy4DBuiyEf0vXwG2eA8iMqDdkuchFdqFBxahApaDwUrHIZSk61wogVmUJeGAclnBMiA7cD4dlgSz4TOHlhR5bivb1CM4iU1YWC1lI3SJ+XaBnVegOIbJmbCEp9/K8A8TSw/4D1hnU0kL6huUAAAAASUVORK5CYII= Delivered-To: submit@bugs.debian.org Package: z88dk Severity: important Tags: security patch The security team received a report from the CERT Coordination Center that the Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability. It looks like this package includes the affected code at that's the reason of this bug report. The patch is available here: http://gitweb.dragonflybsd.org/dragonfly.git/blobdiff/4d133046c59a851141519d03553a70e903b3eefc..2841837793bd095a82f477e9c370cfe6cfb3862c:/lib/libc/regex/regcomp.c Please, can you confirm if the binary packages are affected? Are stable and testing affected? More information, here: http://www.kb.cert.org/vuls/id/695940 https://guidovranken.wordpress.com/2015/02/04/full-disclosure-heap-overflow-in-h-spencers-regex-library-on-32-bit-systems/ A CVE id has been requested already and the report will be updated with it eventually. Cheers, luciano ------------=_1424360354-3079-1--   Received: (at 778399-done) by bugs.debian.org; 19 Feb 2015 15:34:25 +0000 From jmm@inutil.org Thu Feb 19 15:34:24 2015 X-Spam-Checker-Version: SpamAssassin 3.4.0-bugs.debian.org_2005_01_02 (2014-02-07) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=4.0 tests=BAYES_00,FOURLA, MURPHY_DRUGS_REL8,RCVD_IN_DNSWL_MED,T_RP_MATCHES_RCVD autolearn=no autolearn_force=no version=3.4.0-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 10; hammy, 93; neutral, 35; spammy, 2. spammytokens:0.915-+--Center, 0.900-+--center hammytokens:0.000-+--H*r:jmm, 0.000-+--HX-SA-Exim-Scanned:inutil.org, 0.000-+--H*RU:inutil.org, 0.000-+--H*r:inutil.org, 0.000-+--H*RU:83.151.30.8 Return-path: Received: from inutil.org ([83.151.30.8]) by buxtehude.debian.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1YOT7Q-00007y-Os for 778399-done@bugs.debian.org; Thu, 19 Feb 2015 15:34:24 +0000 Received: from jmm by inutil.org with local (Exim 4.72) (envelope-from ) id 1YOT2B-00004u-1F for 778399-done@bugs.debian.org; Thu, 19 Feb 2015 16:28:59 +0100 Date: Thu, 19 Feb 2015 16:28:59 +0100 From: Moritz Muehlenhoff To: 778399-done@bugs.debian.org Subject: Re: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability Message-ID: <20150219152859.GA32745@inutil.org> References: <3441163.lIScrJmJKV@box> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3441163.lIScrJmJKV@box> User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: jmm@inutil.org X-SA-Exim-Scanned: No (on inutil.org); SAEximRunCond expanded to false On Sat, Feb 14, 2015 at 03:33:21PM +0100, Luciano Bello wrote: > Package: z88dk > Severity: important > Tags: security patch > > The security team received a report from the CERT Coordination Center that the > Henry Spencer regular expressions (regex) library contains a heap overflow > vulnerability. It looks like this package includes the affected code at that's > the reason of this bug report. This is only used when building on Windows. Cheers, Moritz   Changed Bug title to 'z88dk: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability' from 'Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability' Request was from Salvatore Bonaccorso <carnil@debian.org> to control@bugs.debian.org.   Received: (at control) by bugs.debian.org; 16 Mar 2015 11:53:45 +0000 From salvatore.bonaccorso@gmail.com Mon Mar 16 11:53:45 2015 X-Spam-Checker-Version: SpamAssassin 3.4.0-bugs.debian.org_2005_01_02 (2014-02-07) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-5.6 required=4.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,FROMDEVELOPER, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 13; hammy, 129; neutral, 29; spammy, 0. spammytokens: hammytokens:0.000-+--H*u:2014-03-12, 0.000-+--H*UA:2014-03-12, 0.000-+--H*u:1.5.23, 0.000-+--H*UA:1.5.23, 0.000-+--H*F:U*carnil Return-path: Received: from mail-we0-f174.google.com ([74.125.82.174]) by buxtehude.debian.org with esmtps (TLS1.2:RSA_ARCFOUR_SHA1:128) (Exim 4.80) (envelope-from ) id 1YXTaa-0006xd-Vx for control@bugs.debian.org; Mon, 16 Mar 2015 11:53:45 +0000 Received: by wegp1 with SMTP id p1so36330794weg.1 for ; Mon, 16 Mar 2015 04:53:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:subject:message-id:mime-version:content-type :content-disposition:user-agent; bh=A2OAMwYpvvl5y/ufYfPkZK448fsF5ugOVNWiAegurgs=; b=jKprnZ0vfRkeAVTnHR8FJAmzA/u0DTdME3YF1WhXWqK4vhsUIlmz28bN57mkBwYaQ0 R1rh4pyS6PhAiMpNkapIPcJQtMah6EnqOTOwj4YwlojIaKWvw0xZa+YweREJSZv376BX fzQY9gVo2qdeSmMjXAINKZmknsEvHe35qbDPHIIlOrBGiTqn5ZsRqFPqJnI6QNT8xNuk xM0IUFlOupER2H17IJzJ2HtEONe6g0L8L99xN/rOoQzg8IR+V9IORerpVDR3xztmT1CR dKyVlM/sF/Z73ksU1MHsw2lH2Ga7j3LLnBqkQz2J65Woz5kj3cZ7sTTCLEx81jVt2sjg o5ig== X-Received: by 10.180.106.197 with SMTP id gw5mr72266957wib.58.1426506817633; Mon, 16 Mar 2015 04:53:37 -0700 (PDT) Received: from eldamar ([2001:1620:f00:82fe:863a:4bff:feda:174c]) by mx.google.com with ESMTPSA id dz6sm15046562wib.0.2015.03.16.04.53.36 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 16 Mar 2015 04:53:36 -0700 (PDT) Sender: Salvatore Bonaccorso Date: Mon, 16 Mar 2015 12:53:36 +0100 From: Salvatore Bonaccorso To: control@bugs.debian.org Subject: Update for CVE-2015-2305 Message-ID: <20150316115336.GA1387@eldamar.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) Delivered-To: control@bugs.debian.org retitle 778389 php5: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778390 olsrd: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778391 llvm-toolchain-3.4: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778392 llvm-toolchain-3.5: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778393 llvm-toolchain-3.6: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778394 llvm-toolchain-snapshot: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778395 haskell-regex-posix: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778396 cups: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778397 librcsb-core-wrapper: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle openrpt#778398: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778399 z88dk: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778408 newlib: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778410 yap: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778403 vnc4: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778411 sma: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778406 clamav: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778401 knews: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778402 radare2: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778414 efl: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778404 ptlib: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778413 alpine: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778409 vigor: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability retitle 778412 nvi: CVE-2015-2305: Henry Spencer regular expressions (regex) library contains a heap overflow vulnerability   Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org.   Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@bugs.debian.org From: Debbugs Internal Request Subject: Internal Control Message-Id: Bug archived. Date: Tue, 14 Apr 2015 07:35:22 +0000 User-Agent: Fakemail v42.6.9 # A New Hope # A long time ago, in a galaxy far, far away # something happened. # # Magically this resulted in the following # action being taken, but this fake control # message doesn't tell you why it happened # # The action: # Bug archived. thanks # This fakemail brought to you by your local debbugs # administrator  debiancontributors-0.7.8/test/bts_spool/db-h/000077500000000000000000000000001345521122600212245ustar00rootroot00000000000000debiancontributors-0.7.8/test/bts_spool/db-h/00/000077500000000000000000000000001345521122600214435ustar00rootroot00000000000000debiancontributors-0.7.8/test/bts_spool/db-h/00/794000.log000066400000000000000000000416011345521122600227130ustar00rootroot00000000000000 Report forwarded to debian-bugs-dist@lists.debian.org, Debian Edu Developers <debian-edu@lists.debian.org>:
Bug#794000; Package debian-edu-config.   debian-bugs-dist@lists.debian.orgDebian Edu Developers  X-Loop: owner@bugs.debian.org Subject: Bug#794000: =?UTF-8?Q?GOsa=C2=B2's?= post-pwchange Hook (gosa-sync) cannot handle ";" (semikolon) in passwords Reply-To: Mike Gabriel , 794000@bugs.debian.org Resent-From: Mike Gabriel Resent-To: debian-bugs-dist@lists.debian.org Resent-CC: Debian Edu Developers X-Loop: owner@bugs.debian.org Resent-Date: Wed, 29 Jul 2015 17:27:15 +0000 Resent-Message-ID: Resent-Sender: owner@bugs.debian.org X-Debian-PR-Message: report 794000 X-Debian-PR-Package: debian-edu-config X-Debian-PR-Keywords: X-Debian-PR-Source: debian-edu-config Received: via spool by submit@bugs.debian.org id=B.143819040415460 (code B); Wed, 29 Jul 2015 17:27:15 +0000 Received: (at submit) by bugs.debian.org; 29 Jul 2015 17:20:04 +0000 X-Spam-Checker-Version: SpamAssassin 3.4.0-bugs.debian.org_2005_01_02 (2014-02-07) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-16.2 required=4.0 tests=BAYES_00,HAS_PACKAGE, PGPSIGNATURE,RCVD_IN_DNSWL_LOW,RP_MATCHES_RCVD autolearn=ham autolearn_force=no version=3.4.0-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 29; hammy, 150; neutral, 93; spammy, 0. spammytokens: hammytokens:0.000-+--H*c:pgp-signature, 0.000-+--H*c:protocol, 0.000-+--H*c:micalg, 0.000-+--H*c:signed, 0.000-+--H*RU:sk:grimnir Received: from freya.das-netzwerkteam.de ([88.198.48.199]) by buxtehude.debian.org with esmtps (TLS1.1:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84) (envelope-from ) id 1ZKV1Q-00040C-6k for submit@bugs.debian.org; Wed, 29 Jul 2015 17:20:04 +0000 Received: from grimnir.das-netzwerkteam.de (grimnir.das-netzwerkteam.de [78.46.204.98]) by freya.das-netzwerkteam.de (Postfix) with ESMTPS id 12B09A2 for ; Wed, 29 Jul 2015 19:20:00 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by grimnir.das-netzwerkteam.de (Postfix) with ESMTP id C4B263BB1E for ; Wed, 29 Jul 2015 19:19:59 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at grimnir.das-netzwerkteam.de Received: from grimnir.das-netzwerkteam.de ([127.0.0.1]) by localhost (grimnir.das-netzwerkteam.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4HgHSyEGsIcz for ; Wed, 29 Jul 2015 19:19:59 +0200 (CEST) Received: from grimnir.das-netzwerkteam.de (localhost [127.0.0.1]) by grimnir.das-netzwerkteam.de (Postfix) with ESMTPS id 663EB3BB12 for ; Wed, 29 Jul 2015 19:19:59 +0200 (CEST) Received: from bifrost.das-netzwerkteam.de (bifrost.das-netzwerkteam.de [178.62.101.154]) by mail.das-netzwerkteam.de (Horde Framework) with HTTP; Wed, 29 Jul 2015 17:19:59 +0000 Date: Wed, 29 Jul 2015 17:19:59 +0000 Message-ID: <20150729171959.Horde.bQEEflFT-BQBdZYCe6NtYw9@mail.das-netzwerkteam.de> From: Mike Gabriel To: submit@bugs.debian.org User-Agent: Internet Messaging Program (IMP) H5 (6.2.2) Accept-Language: de,en Organization: DAS-NETZWERKTEAM X-Originating-IP: 178.62.101.154 X-Remote-Browser: Mozilla/5.0 (X11; Linux x86_64; rv:32.0) Gecko/20100101 Firefox/32.0 Iceweasel/32.0 Content-Type: multipart/signed; boundary="=_Fcwbytmba-UVPu2D0tcaqg1"; protocol="application/pgp-signature"; micalg=pgp-sha1 MIME-Version: 1.0 Delivered-To: submit@bugs.debian.org This message is in MIME format and has been PGP signed. --=_Fcwbytmba-UVPu2D0tcaqg1 Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Package: debian-edu-config Severity: important Version: 1.818 We encountered an important issue with GOsa=C2=B2 in Debian Edu: a user=20= =20 tried=20to change his password (via the adminstrative change password=20=20 dialog). The=20favourized password contains ";" (semicolon). Trying to use that=20= =20 password,=20results in """ Post-even Hook reported problem. Could not verify password for=20=20 ''.=20Nothing done... The password change has been cancelled. """ (Translated from a German error message) """ (German) Post-even-Hook meldete ein Problem: Could not verify password for=20=20 ''.=20Nothing done... Die Passwort=C3=A4nderungen wurde abgebrochen. """ As the above message is produced by the debian-edu-config script=20=20 gosa-sync=20(same error message is found in that script), it needs to be=20= =20 fixed=20there. light+love, Mike --=20 DAS-NETZWERKTEAM mike=20gabriel, herweg 7, 24357 fleckeby fon: +49 (1520) 1976 148 GnuPG Key ID 0x25771B31 mail: mike.gabriel@das-netzwerkteam.de, http://das-netzwerkteam.de freeBusy: https://mail.das-netzwerkteam.de/freebusy/m.gabriel%40das-netzwerkteam.de.x= fb --=_Fcwbytmba-UVPu2D0tcaqg1 Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAABAgAGBQJVuQs/AAoJEJr0azAldxsxESUP/RxdAP8Ut0J59VsSfyFRMnC7 PP51ti2W0W83fpiMu+LZ/eEomLeOZkpF5rAW3My7fGGDK1fHZn5oyfIwtm/1oJgS ie8wEvlyhS44owt2KmPg6dTjPYOzP+lZnRShOdYFQNr/IYhYtRGT4ImPoj/YAWuX gcpOWv2jmrPkae51ESqv9CpCz6JzkOSOSds29RoVla9ORApYhP034oKu0kdwj7/G ysFDyI4SeL0jaVK8B+VimBBZE9QS5WVs7StRoPNOcgwknPk2Hq1WCUcp2Uk8wkrR +sKnntZd1i7O15DRMCPKjt0sKmCtr/v9Kns9LztvQgh1lI+psgNsondworK+Pjaz VTfxRm9BbmLn+3c7wv+pkvAXKCgV3ba4lkFR85pP3E5HHEp772Gm6Euw1IgZLmLu VZKiumN37nKbFcLmAM6zOS95kHidn6mih2JG58xgfNgUBLqfzMcmn+lqCEM+SbiK O0PMgKpb0CSVrT2PSQvcqPRdgdHqIJccssreWsSffVKC2hrWA/t41RUaujHQKygw Nth04Uld/CzwXNzHfoCZDkKqhTAyij56rqU9savgFsrXwfeHS3/5BP4U7CLqlX3i HYY9K4x1hbUIwQKWin+xHP3UAdaIMhEgRP+o5CxJdAcmOHjIXV2hoxtlO4Yz4bb/ IWjkyrgjeOzoLtT7nOP1 =lntl -----END PGP SIGNATURE----- --=_Fcwbytmba-UVPu2D0tcaqg1--   Acknowledgement sent to Mike Gabriel <mike.gabriel@das-netzwerkteam.de>:
New Bug report received and forwarded. Copy sent to Debian Edu Developers <debian-edu@lists.debian.org>.   -t  Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) Content-Type: text/plain; charset=utf-8 X-Loop: owner@bugs.debian.org From: owner@bugs.debian.org (Debian Bug Tracking System) To: Mike Gabriel Subject: Bug#794000: Acknowledgement (=?UTF-8?Q?GOsa=C2=B2's?= post-pwchange Hook (gosa-sync) cannot handle ";" (semikolon) in passwords) Message-ID: References: <20150729171959.Horde.bQEEflFT-BQBdZYCe6NtYw9@mail.das-netzwerkteam.de> X-Debian-PR-Message: ack 794000 X-Debian-PR-Package: debian-edu-config X-Debian-PR-Source: debian-edu-config Reply-To: 794000@bugs.debian.org Date: Wed, 29 Jul 2015 17:27:18 +0000 Thank you for filing a new Bug report with Debian. This is an automatically generated reply to let you know your message has been received. Your message is being forwarded to the package maintainers and other interested parties for their attention; they will reply in due course. Your message has been sent to the package maintainer(s): Debian Edu Developers If you wish to submit further information on this problem, please send it to 794000@bugs.debian.org. Please do not send mail to owner@bugs.debian.org unless you wish to report a problem with the Bug-tracking system. --=20 794000: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D794000 Debian Bug Tracking System Contact owner@bugs.debian.org with problems   Received: (at submit) by bugs.debian.org; 29 Jul 2015 17:20:04 +0000 From mike.gabriel@das-netzwerkteam.de Wed Jul 29 17:20:04 2015 X-Spam-Checker-Version: SpamAssassin 3.4.0-bugs.debian.org_2005_01_02 (2014-02-07) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-16.2 required=4.0 tests=BAYES_00,HAS_PACKAGE, PGPSIGNATURE,RCVD_IN_DNSWL_LOW,RP_MATCHES_RCVD autolearn=ham autolearn_force=no version=3.4.0-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 29; hammy, 150; neutral, 93; spammy, 0. spammytokens: hammytokens:0.000-+--H*c:pgp-signature, 0.000-+--H*c:protocol, 0.000-+--H*c:micalg, 0.000-+--H*c:signed, 0.000-+--H*RU:sk:grimnir Return-path: Received: from freya.das-netzwerkteam.de ([88.198.48.199]) by buxtehude.debian.org with esmtps (TLS1.1:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84) (envelope-from ) id 1ZKV1Q-00040C-6k for submit@bugs.debian.org; Wed, 29 Jul 2015 17:20:04 +0000 Received: from grimnir.das-netzwerkteam.de (grimnir.das-netzwerkteam.de [78.46.204.98]) by freya.das-netzwerkteam.de (Postfix) with ESMTPS id 12B09A2 for ; Wed, 29 Jul 2015 19:20:00 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by grimnir.das-netzwerkteam.de (Postfix) with ESMTP id C4B263BB1E for ; Wed, 29 Jul 2015 19:19:59 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at grimnir.das-netzwerkteam.de Received: from grimnir.das-netzwerkteam.de ([127.0.0.1]) by localhost (grimnir.das-netzwerkteam.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4HgHSyEGsIcz for ; Wed, 29 Jul 2015 19:19:59 +0200 (CEST) Received: from grimnir.das-netzwerkteam.de (localhost [127.0.0.1]) by grimnir.das-netzwerkteam.de (Postfix) with ESMTPS id 663EB3BB12 for ; Wed, 29 Jul 2015 19:19:59 +0200 (CEST) Received: from bifrost.das-netzwerkteam.de (bifrost.das-netzwerkteam.de [178.62.101.154]) by mail.das-netzwerkteam.de (Horde Framework) with HTTP; Wed, 29 Jul 2015 17:19:59 +0000 Date: Wed, 29 Jul 2015 17:19:59 +0000 Message-ID: <20150729171959.Horde.bQEEflFT-BQBdZYCe6NtYw9@mail.das-netzwerkteam.de> From: Mike Gabriel To: submit@bugs.debian.org Subject: =?utf-8?b?R09zYcKyJ3M=?= post-pwchange Hook (gosa-sync) cannot handle ";" (semikolon) in passwords User-Agent: Internet Messaging Program (IMP) H5 (6.2.2) Accept-Language: de,en Organization: DAS-NETZWERKTEAM X-Originating-IP: 178.62.101.154 X-Remote-Browser: Mozilla/5.0 (X11; Linux x86_64; rv:32.0) Gecko/20100101 Firefox/32.0 Iceweasel/32.0 Content-Type: multipart/signed; boundary="=_Fcwbytmba-UVPu2D0tcaqg1"; protocol="application/pgp-signature"; micalg=pgp-sha1 MIME-Version: 1.0 Delivered-To: submit@bugs.debian.org This message is in MIME format and has been PGP signed. --=_Fcwbytmba-UVPu2D0tcaqg1 Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Package: debian-edu-config Severity: important Version: 1.818 We encountered an important issue with GOsa=C2=B2 in Debian Edu: a user=20= =20 tried=20to change his password (via the adminstrative change password=20=20 dialog). The=20favourized password contains ";" (semicolon). Trying to use that=20= =20 password,=20results in """ Post-even Hook reported problem. Could not verify password for=20=20 ''.=20Nothing done... The password change has been cancelled. """ (Translated from a German error message) """ (German) Post-even-Hook meldete ein Problem: Could not verify password for=20=20 ''.=20Nothing done... Die Passwort=C3=A4nderungen wurde abgebrochen. """ As the above message is produced by the debian-edu-config script=20=20 gosa-sync=20(same error message is found in that script), it needs to be=20= =20 fixed=20there. light+love, Mike --=20 DAS-NETZWERKTEAM mike=20gabriel, herweg 7, 24357 fleckeby fon: +49 (1520) 1976 148 GnuPG Key ID 0x25771B31 mail: mike.gabriel@das-netzwerkteam.de, http://das-netzwerkteam.de freeBusy: https://mail.das-netzwerkteam.de/freebusy/m.gabriel%40das-netzwerkteam.de.x= fb --=_Fcwbytmba-UVPu2D0tcaqg1 Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAABAgAGBQJVuQs/AAoJEJr0azAldxsxESUP/RxdAP8Ut0J59VsSfyFRMnC7 PP51ti2W0W83fpiMu+LZ/eEomLeOZkpF5rAW3My7fGGDK1fHZn5oyfIwtm/1oJgS ie8wEvlyhS44owt2KmPg6dTjPYOzP+lZnRShOdYFQNr/IYhYtRGT4ImPoj/YAWuX gcpOWv2jmrPkae51ESqv9CpCz6JzkOSOSds29RoVla9ORApYhP034oKu0kdwj7/G ysFDyI4SeL0jaVK8B+VimBBZE9QS5WVs7StRoPNOcgwknPk2Hq1WCUcp2Uk8wkrR +sKnntZd1i7O15DRMCPKjt0sKmCtr/v9Kns9LztvQgh1lI+psgNsondworK+Pjaz VTfxRm9BbmLn+3c7wv+pkvAXKCgV3ba4lkFR85pP3E5HHEp772Gm6Euw1IgZLmLu VZKiumN37nKbFcLmAM6zOS95kHidn6mih2JG58xgfNgUBLqfzMcmn+lqCEM+SbiK O0PMgKpb0CSVrT2PSQvcqPRdgdHqIJccssreWsSffVKC2hrWA/t41RUaujHQKygw Nth04Uld/CzwXNzHfoCZDkKqhTAyij56rqU9savgFsrXwfeHS3/5BP4U7CLqlX3i HYY9K4x1hbUIwQKWin+xHP3UAdaIMhEgRP+o5CxJdAcmOHjIXV2hoxtlO4Yz4bb/ IWjkyrgjeOzoLtT7nOP1 =lntl -----END PGP SIGNATURE----- --=_Fcwbytmba-UVPu2D0tcaqg1--   Severity set to 'normal' from 'important' Request was from Holger Levsen <holger@layer-acht.org> to control@bugs.debian.org.   Received: (at control) by bugs.debian.org; 30 Jul 2015 09:44:10 +0000 From holger@layer-acht.org Thu Jul 30 09:44:10 2015 X-Spam-Checker-Version: SpamAssassin 3.4.0-bugs.debian.org_2005_01_02 (2014-02-07) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=4.0 tests=BAYES_00,HAS_BUG_NUMBER, PGPSIGNATURE,RCVD_IN_DNSWL_LOW,VALID_BTS_CONTROL autolearn=ham autolearn_force=no version=3.4.0-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 12; hammy, 93; neutral, 18; spammy, 0. spammytokens: hammytokens:0.000-+--H*F:U*holger, 0.000-+--H*RU:sk:holger@, 0.000-+--H*rp:U*holger, 0.000-+--H*F:D*layer-acht.org, 0.000-+--H*RU:62.201.164.66 Return-path: Received: from mail.holgerlevsen.de ([62.201.164.66] helo=alpha.holgerlevsen.de) by buxtehude.debian.org with esmtp (Exim 4.84) (envelope-from ) id 1ZKkNm-0006Ds-NC for control@bugs.debian.org; Thu, 30 Jul 2015 09:44:10 +0000 Received: from localhost (alpha.holgerlevsen.de [62.201.164.66]) by alpha.holgerlevsen.de (Postfix) with ESMTP id 93EA7CAD651 for ; Thu, 30 Jul 2015 11:44:06 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at alpha.holgerlevsen.de Received: from alpha.holgerlevsen.de ([62.201.164.66]) by localhost (mail.holgerlevsen.de [62.201.164.66]) (amavisd-new, port 10024) with ESMTP id Onj0Q48c6aaZ for ; Thu, 30 Jul 2015 11:44:06 +0200 (CEST) Received: from matrix.localnet (epsilon.holgerlevsen.de [62.201.164.82]) by alpha.holgerlevsen.de (Postfix) with ESMTP id D2B1ECAD64F for ; Thu, 30 Jul 2015 11:44:05 +0200 (CEST) From: Holger Levsen To: control@bugs.debian.org Subject: Re: Bug#794000: =?utf-8?q?GOsa=C2=B2=27s_post-pwchange_Hook?= (gosa-sync) cannot handle ";" (semikolon) in passwords Date: Thu, 30 Jul 2015 11:43:04 +0200 User-Agent: KMail/1.13.7 (Linux/3.16.0-0.bpo.4-amd64; KDE/4.8.4; x86_64; ; ) References: <20150729171959.Horde.bQEEflFT-BQBdZYCe6NtYw9@mail.das-netzwerkteam.de> In-Reply-To: <20150729171959.Horde.bQEEflFT-BQBdZYCe6NtYw9@mail.das-netzwerkteam.de> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart46400080.9SdTSyY0jj"; protocol="application/pgp-signature"; micalg=pgp-sha512 Content-Transfer-Encoding: 7bit Message-Id: <201507301143.11996.holger@layer-acht.org> Delivered-To: control@bugs.debian.org --nextPart46400080.9SdTSyY0jj Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: 7bit severity 794000 normal --nextPart46400080.9SdTSyY0jj Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIVAwUAVbnxqgkauFYGmqocAQoMOxAAuxuNQJmpo39I8rEGjtWpiuEPmbtQyroQ HhVcSx7PDX4WRmfSfEsxJz5y3pyOGrss7oVnWtGSuPXyCeZ8xKQNUsyXhB2olZO/ GABPWK/Lofyj49PClxz97Omlayc8MGaP0SuekyQn6ZSW2abWa4+TkMaL5wEBTQFs IIeUultmL67LlEdInpKzyuOxMWTkSnqGcctp8jFI2cyQt718Ba5yPTI+Oxij+hAI ioSCiOtpgBfOjuKjhqUuxB7IID4Torvp+dPERKyGK5SI+Gq18u4u7/d3S1gARgMj p80CutDYwAtEpobD/j6SdWjJbVfaZof9XacJT0JWMKZ3fRjDB4D8iPOsMoNk9pwd Yh2aDYDPnXsRghu4+RDKrCgDwZg5HuMl6BEv2WNo9SPn57P8n0l41H+CdcY9yUwo mSR7RuEhoMyg8XkyZBvQV0QECiz/MhM4LC41okIDldlEZKRA8EHSjc5Ev6XrS9sc adOk46S00r+WRnE78VM9B4frIQ+HpaVwj5IKORrHVRHkvmZznKHvyRRYKrzlEn+b vbAruPsOlA5VbGHC5AlLkFgvblSceKbd9JXALo18rWND+eaYICy+r7V8/INC5kWM GuErJukCYBvvNWYNwZngfHUo560GcyRuKeAD4Q2GPjInqHfrXbU1Tbt08TmVJiLK OOTcQeTqeH0= =JHuE -----END PGP SIGNATURE----- --nextPart46400080.9SdTSyY0jj--  debiancontributors-0.7.8/test/email/000077500000000000000000000000001345521122600174755ustar00rootroot00000000000000debiancontributors-0.7.8/test/email/Bits from the release team_this critter needs help!.mbox000066400000000000000000000165221345521122600320050ustar00rootroot00000000000000From jmw@debian.org Sun Jun 18 06:49:19 2017 Return-Path: Delivered-To: eriol@mornie.org Received: from bendel.debian.org (bendel.debian.org [82.195.75.100]) by sure.mornie.org (Postfix) with ESMTPS id AAA01A0581 for ; Sun, 18 Jun 2017 08:49:53 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by bendel.debian.org (Postfix) with QMQP id 4D8B5178; Sun, 18 Jun 2017 06:49:41 +0000 (UTC) X-Mailbox-Line: From debian-devel-announce-request@lists.debian.org Sun Jun 18 06:49:41 2017 Old-Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on sure.mornie.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,T_RP_MATCHES_RCVD autolearn=ham autolearn_force=no version=3.4.0 X-Original-To: lists-debian-devel-announce@bendel.debian.org Delivered-To: lists-debian-devel-announce@bendel.debian.org Received: from localhost (localhost [127.0.0.1]) by bendel.debian.org (Postfix) with ESMTP id 7E556140 for ; Sun, 18 Jun 2017 06:49:31 +0000 (UTC) X-Virus-Scanned: at lists.debian.org with policy bank moderated X-Amavis-Spam-Status: No, score=-12 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, LDO_WHITELIST=-5, PGPSIGNATURE=-5, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=ham autolearn_force=no Received: from bendel.debian.org ([127.0.0.1]) by localhost (lists.debian.org [127.0.0.1]) (amavisd-new, port 2525) with ESMTP id 3Lo0afp9Zmet for ; Sun, 18 Jun 2017 06:49:24 +0000 (UTC) X-policyd-weight: using cached result; rate: -2.75 Received: from hermione.home.powdarrmonkey.net (hogwarts.powdarrmonkey.net [IPv6:2001:8b0:caea:eda7:3ed9:2bff:fe02:9128]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by bendel.debian.org (Postfix) with ESMTPS id 588FE178 for ; Sun, 18 Jun 2017 06:49:24 +0000 (UTC) Received: from [2001:8b0:caea:eda7:35c7:78d:c809:23f2] (helo=lupin) by hermione.home.powdarrmonkey.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1dMU1R-0005tj-1R for debian-devel-announce@lists.debian.org; Sun, 18 Jun 2017 07:49:21 +0100 Received: from jona by lupin with local (Exim 4.89) (envelope-from ) id 1dMU1P-0003EK-Oq for debian-devel-announce@lists.debian.org; Sun, 18 Jun 2017 07:49:19 +0100 Date: Sun, 18 Jun 2017 07:49:19 +0100 From: Jonathan Wiltshire To: debian-devel-announce@lists.debian.org Subject: Bits from the release team: this critter needs help! Message-ID: <20170618064919.zthnb6wxbfo4aada@powdarrmonkey.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="hddkwfq57sonhc6n" Content-Disposition: inline User-Agent: NeoMutt/20170113 (1.7.2) X-Debian-Message: Signature check passed for Debian member Mail-Followup-To: debian-devel@lists.debian.org X-Rc-Virus: 2007-09-13_01 X-Rc-Spam: 2008-11-04_01 Resent-Message-ID: Resent-From: debian-devel-announce@lists.debian.org X-Mailing-List: archive/latest/2037 X-Loop: debian-devel-announce@lists.debian.org List-Id: List-URL: List-Post: List-Help: List-Subscribe: List-Unsubscribe: Precedence: list Resent-Sender: debian-devel-announce-request@lists.debian.org List-Archive: https://lists.debian.org/msgid-search/20170618064919.zthnb6wxbfo4aada@powdarrmonkey.net Resent-Date: Sun, 18 Jun 2017 06:49:41 +0000 (UTC) --hddkwfq57sonhc6n Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, As announced officially, we have now released Debian 9, "Stretch". There are too many people who should be thanked for their work on getting u= s to this point to list them all individually, and we would be sure to miss some. Nevertheless, we would like to particularly thank the installer team, the buildd and ftp teams, the CD team, the publicity team, the webmasters, the Release Notes editors, porters and all the package maintainers and translat= ors who have contributed to making Stretch a great release of which we should a= ll be proud. First point release =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D As was the case for Jessie, we anticipate that the first point release for Stretch will likely occur in approximately one month's time. Please co-ordinate fixes which you would like to see included in the point release with the Stable Release Team via a "pu" bug against the release.debian.org pseudopackage, including a debdiff of the current and proposed source packages. Remember to use reportbug unless you enjoy crafti= ng the metadata by hand. Buster =3D=3D=3D=3D=3D=3D The release of Stretch also means that you can now upload to unstable those changes you've been holding off during the freeze. Please do not rush to upload everything all at once, in order to reduce load on the buildds etc. Automatic testing migration is not yet re-enabled, but that will happen dur= ing the next day or so. As with stretch, we would ask that you co-ordinate particularly large transitions or changes; if your plans involve major toolchain changes or otherwise have the potential to cause problems in unstable for a long time (e.g. due to FTBFS issues), please talk to us. We know that there are a la= rge number of changes which have been waiting for the release to happen and we'= re keen not to stand in the way of those but would also like to avoid a number= of larger transitions becoming entangled. That's it for now; it is time for the celebrations to begin, whether at a Release Party[1] or otherwise. :-) 1: https://wiki.debian.org/ReleasePartyStretch For the release team: --=20 Jonathan Wiltshire jmw@debian.org Debian Developer http://people.debian.org/~jmw 4096R: 0xD3524C51 / 0A55 B7C5 1223 3942 86EC 74C3 5394 479D D352 4C51 --hddkwfq57sonhc6n Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEADLdyLGMneGYn8dtRNMqtfom+MkFAllGImkACgkQRNMqtfom +Mmb/w/+NGRuWgIQ/vvI05KKtB7EwKNpr2qVFCdqePTo35KwheYcKiqS7WenHvwE u7LUB3PcWb/5B2w1uAokAC8KF1YN2KnZrq3s38XgpYL/GZQdJmlxNswIVxnBRzFw B/3giWQ3J9ylDezE/fpBsitcUYnMXRmPbJ1jnAK4BlDuerddz2yZkJus0g3TtUiR vJwSyoJeXinguG56qZAjoPbjhRXClPLTNs3WQmKgWgS3nalLBbooHm2Urwkzt7mP UW77lQ6os4hJm2B/q5s92KvfHwjinkpsFVylOzOa4ACGVbMPNEwmBYBKySDimtwY JcD/JtfjhtGEx0nJUjezSfL4U+M9wKQg0h3WMrihrZtNKk8HeG0XOGfjO11wQtNK laJTYSWoaHhaEu8ZHi25diMgIew/BotI2GSHzutU99TMrMkA6LPBOh4gmygz/Kns igk7XSt597AaNrPXrHjlDvm5QP7yygsSSjxnM+b5OpcIZsHzPuTD/k+UZEbEGhFm DztwEWQhQrTPU/KxQeowTtn+j4tdAfDGq5LH/OlKeS0i7xm/NqNfVhusfYLCNKHa HIdqbSpyn2ipP8o4oM9bYmiSXXN5wldFCGKWUaI1xP9jh5NypKtEnkkjPeQKbBAW 9uonxQQShUWn4HACHdouBP51zERfe6u1nbVqP8mPITLAx9x7fo0= =6PF6 -----END PGP SIGNATURE----- --hddkwfq57sonhc6n-- debiancontributors-0.7.8/test/test_mine_bts.py000066400000000000000000000025731345521122600216260ustar00rootroot00000000000000# coding: utf8 # # Copyright (C) 2013 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from debiancontributors.datamine import DataMine import unittest import sys class TestMineBts(unittest.TestCase): @unittest.skipIf(sys.version_info[0] < 3, "only supported in python3") def test_gitdirs(self): """ Test gitdirs scanner """ mine = DataMine(configstr= """ source: test contribution: correspondant method: bts dirs: test/bts_spool/archive/ test/bts_spool/db-h/ url: "http://www.example.com/{name}" """) mine.scan() if __name__ == '__main__': unittest.main() debiancontributors-0.7.8/test/test_mine_email.py000066400000000000000000000026171345521122600221240ustar00rootroot00000000000000# coding: utf8 # # Copyright (C) 2018 Daniele Tricoli # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import ( absolute_import, division, print_function, unicode_literals ) import unittest from debiancontributors.datamine import DataMine class TestMineEmail(unittest.TestCase): def test_email_scanner(self): """ Test email scanner """ mine = DataMine(configstr=""" source: test contribution: developer method: mailfrom folders: test/email/* url: http://www.example.com/{email} """) mine.scan() emails = set(x.id for x in mine.submission.entries.keys()) self.assertNotIn('eriol@mornie.org', emails) self.assertIn('jmw@debian.org', emails) if __name__ == '__main__': unittest.main() debiancontributors-0.7.8/test/test_mine_git.py000066400000000000000000000046411345521122600216170ustar00rootroot00000000000000# coding: utf8 # # Copyright (C) 2013 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from debiancontributors.datamine import DataMine import unittest import os.path class TestMineGit(unittest.TestCase): def test_gitdirs(self): """ Test gitdirs scanner """ mine = DataMine(configstr= """ source: test contribution: committer method: gitdirs dirs: "." url: "http://www.example.com/{name}" """) mine.scan() def test_gitlogs(self): """ Test gitlogs scanner """ if not os.path.isdir(".git"): raise unittest.SkipTest("no idea where to find my git repo when not run from the git checkout") mine = DataMine(configstr= """ source: test contribution: committer method: gitlogs dirs: ".git" url: "http://www.example.com/{{email}}" """) mine.scan() emails = set(x.id for x in mine.submission.entries.keys()) self.assertIn("enrico@enricozini.org", emails) self.assertNotIn("enrico", emails) def test_gitlogs_authormap(self): """ Test gitlogs scanner """ if not os.path.isdir(".git"): raise unittest.SkipTest("no idea where to find my git repo when not run from the git checkout") mine = DataMine(configstr= """ source: test contribution: committer method: gitlogs dirs: ".git" author_map: enrico@.+ enrico i url: "http://www.example.com/{{email}}" """) mine.scan() emails = set(x.id for x in mine.submission.entries.keys()) self.assertNotIn("enrico@enricozini.org", emails) self.assertIn("enrico", emails) if __name__ == '__main__': unittest.main() debiancontributors-0.7.8/test/test_parser.py000066400000000000000000000101641345521122600213150ustar00rootroot00000000000000# coding: utf8 # # Copyright (C) 2013 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals import debiancontributors.parser as parser import unittest class TestParser(unittest.TestCase): def testGetKey(self): self.assertEquals(parser.get_key({"foo": "bar"}, "foo"), "bar") self.assertRaises(parser.Fail, parser.get_key, {}, "foo") def testGetKeyInt(self): self.assertEquals(parser.get_key_int({"foo": "7"}, "foo"), 7) self.assertRaises(parser.Fail, parser.get_key_int, {}, "foo") self.assertRaises(parser.Fail, parser.get_key_int, {"foo": ""}, "foo") self.assertRaises(parser.Fail, parser.get_key_int, {"foo": "seven"}, "foo") def testGetKeyString(self): self.assertEquals(parser.get_key_string({"foo": "7"}, "foo"), "7") self.assertEquals(parser.get_key_string({"foo": ""}, "foo", True), "") self.assertEquals(parser.get_key_string({"foo": None}, "foo", True), "") self.assertEquals(parser.get_key_string({}, "foo", True), "") self.assertRaises(parser.Fail, parser.get_key_string, {}, "foo") self.assertRaises(parser.Fail, parser.get_key_string, {"foo": ""}, "foo") def testGetKeyUnicode(self): self.assertEquals(parser.get_key_unicode({"foo": "7"}, "foo"), "7") self.assertEquals(parser.get_key_unicode({"foo": b"\xe2\x99\xa5"}, "foo"), "♥") self.assertEquals(parser.get_key_unicode({"foo": ""}, "foo", True), "") self.assertEquals(parser.get_key_unicode({"foo": None}, "foo", True), "") self.assertEquals(parser.get_key_unicode({}, "foo", True), "") self.assertRaises(parser.Fail, parser.get_key_unicode, {}, "foo") self.assertRaises(parser.Fail, parser.get_key_unicode, {"foo": ""}, "foo") self.assertRaises(parser.Fail, parser.get_key_unicode, {"foo": b'\xe0'}, "foo") def testGetKeySequence(self): self.assertEquals(parser.get_key_sequence({"foo": []}, "foo"), []) self.assertEquals(parser.get_key_sequence({"foo": [1, 2, "three"]}, "foo"), [1, 2, "three"]) self.assertEquals(parser.get_key_sequence({"foo": ()}, "foo"), ()) self.assertRaises(parser.Fail, parser.get_key_sequence, {}, "foo") self.assertRaises(parser.Fail, parser.get_key_sequence, {"foo": "bar"}, "foo") self.assertRaises(parser.Fail, parser.get_key_sequence, {"foo": {}}, "foo") def testGetKeySequenceOrObject(self): self.assertEquals(parser.get_key_sequence_or_object({"foo": []}, "foo"), []) self.assertEquals(parser.get_key_sequence_or_object({"foo": {}}, "foo"), [{}]) self.assertEquals(parser.get_key_sequence_or_object({"foo": [{}]}, "foo"), [{}]) self.assertRaises(parser.Fail, parser.get_key_sequence_or_object, {}, "foo") self.assertRaises(parser.Fail, parser.get_key_sequence_or_object, {"foo": "bar"}, "foo") def testGetKeyDateOrNone(self): from datetime import date self.assertEquals(parser.get_key_date_or_none({"foo": "2013-11-16"}, "foo"), date(2013, 11, 16)) self.assertEquals(parser.get_key_date_or_none({"foo": ""}, "foo"), None) self.assertEquals(parser.get_key_date_or_none({"foo": None}, "foo"), None) self.assertEquals(parser.get_key_date_or_none({}, "foo"), None) self.assertRaises(parser.Fail, parser.get_key_date_or_none, {"foo": "2013"}, "foo") if __name__ == '__main__': unittest.main() debiancontributors-0.7.8/test/test_submission.py000066400000000000000000000061741345521122600222220ustar00rootroot00000000000000# coding: utf8 # # Copyright (C) 2013 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals import debiancontributors as dc from datetime import date import unittest import json import os.path class TestSubmission(unittest.TestCase): def testIdentifier(self): i = dc.Identifier("login", "enrico") self.assertEquals(i.type, "login") self.assertEquals(i.id, "enrico") self.assertIsNone(i.desc) i = dc.Identifier("login", "enrico", "Enrico Zini") self.assertEquals(i.type, "login") self.assertEquals(i.id, "enrico") self.assertEquals(i.desc, "Enrico Zini") def testMinimalData(self): s = dc.Submission("test") s.add_contribution_data(dc.Identifier("login", "enrico"), "upload") js = s.to_json(indent=1) res = json.loads(js) self.assertEquals(res, [{ "id": [ { "type": "login", "id": "enrico" } ], "contributions": [ { "type": "upload" } ], }]) def testFullData(self): s = dc.Submission("test") s.add_contribution_data(dc.Identifier("login", "enrico"), "upload", begin=date(2013, 5, 1), end=date(2013, 11, 30), url="http://www.example.com") js = s.to_json(indent=1) res = json.loads(js) self.assertEquals(res, [{ "id": [ { "type": "login", "id": "enrico" } ], "contributions": [ { "type": "upload", "begin": "2013-05-01", "end": "2013-11-30", "url": "http://www.example.com" } ], }]) def test_auth_token(self): if not os.path.isdir("setup.py"): raise unittest.SkipTest("no idea where to find my setup.py when not run from the git checkout") s = dc.Submission("test") self.assertIsNone(s.auth_token) s.set_auth_token("foo") self.assertEquals(s.auth_token, "foo") mock_auth_token_file = os.path.abspath("setup.py") s.set_auth_token("@" + mock_auth_token_file) self.assertRegexpMatches(s.auth_token, "setup\(") s = dc.Submission("test", auth_token="foo") self.assertEquals(s.auth_token, "foo") s = dc.Submission("test", auth_token="@" + mock_auth_token_file) self.assertRegexpMatches(s.auth_token, "setup\(") if __name__ == '__main__': unittest.main() debiancontributors-0.7.8/test/test_types.py000066400000000000000000000124661345521122600211740ustar00rootroot00000000000000# coding: utf8 # # Copyright (C) 2013 Enrico Zini # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Lesser General Public License as # published by the Free Software Foundation, either version 3 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program. If not, see . from __future__ import print_function from __future__ import absolute_import from __future__ import division from __future__ import unicode_literals from debiancontributors.types import * import unittest class TestTypes(unittest.TestCase): def test_identifier(self): """ Test Identifier operations """ iem = Identifier("email", "enrico@enricozini.org") iem.validate() self.assertEquals(iem.type, "email") self.assertEquals(iem.id, "enrico@enricozini.org") self.assertIsNone(iem.desc) ied = Identifier("email", "enrico@enricozini.org", "Enrico Zini") ied.validate() self.assertEquals(ied.type, "email") self.assertEquals(ied.id, "enrico@enricozini.org") self.assertEquals(ied.desc, "Enrico Zini") ilo = Identifier("login", "enrico") ilo.validate() self.assertEquals(ilo.type, "login") self.assertEquals(ilo.id, "enrico") self.assertIsNone(ilo.desc) ild = Identifier("login", "enrico", "Enrico Zini") ild.validate() self.assertEquals(ild.type, "login") self.assertEquals(ild.id, "enrico") self.assertEquals(ild.desc, "Enrico Zini") ifp = Identifier("fpr", "1793D6AB75663E6BF104953A634F4BD1E7AD5568") ifp.validate() self.assertEquals(ifp.type, "fpr") self.assertEquals(ifp.id, "1793D6AB75663E6BF104953A634F4BD1E7AD5568") self.assertIsNone(ifp.desc) ifd = Identifier("fpr", "1793D6AB75663E6BF104953A634F4BD1E7AD5568", "Enrico Zini") ifd.validate() self.assertEquals(ifp.type, "fpr") self.assertEquals(ifd.type, "fpr") self.assertEquals(ifd.id, "1793D6AB75663E6BF104953A634F4BD1E7AD5568") self.assertEquals(ifd.desc, "Enrico Zini") self.assertEquals(iem, ied) self.assertEquals(ilo, ild) self.assertEquals(ifp, ifd) self.assertNotEquals(iem, ilo) self.assertNotEquals(iem, ifp) self.assertNotEquals(iem, ild) self.assertNotEquals(iem, ifd) self.assertNotEquals(ied, ilo) self.assertNotEquals(ied, ifp) self.assertNotEquals(ied, ild) self.assertNotEquals(ied, ifd) def test_bad_identifier(self): from debiancontributors.parser import Fail self.assertRaises(Fail, Identifier("foo", "").validate) self.assertRaises(Fail, Identifier(3, "").validate) self.assertRaises(Fail, Identifier("login", None).validate) self.assertRaises(Fail, Identifier("login", "").validate) self.assertRaises(Fail, Identifier("login", "enrico", 3).validate) self.assertRaises(Fail, Identifier("email", "enrico").validate) self.assertRaises(Fail, Identifier("fpr", "zzz").validate) def test_bad_contribution(self): from debiancontributors.parser import Fail self.assertRaises(Fail, Contribution(None).validate) self.assertRaises(Fail, Contribution("foo", 3).validate) self.assertRaises(Fail, Contribution("foo", None, 3).validate) self.assertRaises(Fail, Contribution("foo", url=3).validate) def test_auto(self): i = Identifier.create_auto("enrico") self.assertEquals(i.type, "login") self.assertEquals(i.id, "enrico") self.assertIsNone(i.desc) i = Identifier.create_auto("Enrico Zini ") self.assertEquals(i.type, "login") self.assertEquals(i.id, "enrico") self.assertEquals(i.desc, "Enrico Zini") i = Identifier.create_auto("enrico@debian.org") self.assertEquals(i.type, "email") self.assertEquals(i.id, "enrico@debian.org") self.assertIsNone(i.desc) i = Identifier.create_auto("Enrico Zini ") self.assertEquals(i.type, "email") self.assertEquals(i.id, "enrico@debian.org") self.assertEquals(i.desc, "Enrico Zini") i = Identifier.create_auto("1793 D6AB 7566 3E6B F104 953A 634F 4BD1 E7AD 5568") self.assertEquals(i.type, "fpr") self.assertEquals(i.id, "1793D6AB75663E6BF104953A634F4BD1E7AD5568") self.assertIsNone(i.desc) i = Identifier.create_auto("Enrico Zini <1793 D6AB 7566 3E6B F104 953A 634F 4BD1 E7AD 5568>") self.assertEquals(i.type, "fpr") self.assertEquals(i.id, "1793D6AB75663E6BF104953A634F4BD1E7AD5568") self.assertEquals(i.desc, "Enrico Zini") i = Identifier.create_auto("Enrico Zini, the Mad ") self.assertEquals(i.type, "login") self.assertEquals(i.id, "enrico") self.assertEquals(i.desc, "Enrico Zini, the Mad") if __name__ == '__main__': unittest.main()