bley-2.0.0/0000755000175000017500000000000012421233663013222 5ustar evgenievgeni00000000000000bley-2.0.0/PKG-INFO0000644000175000017500000000034512421233663014321 0ustar evgenievgeni00000000000000Metadata-Version: 1.0 Name: bley Version: 2.0.0 Summary: intelligent greylisting daemon for postfix Home-page: http://bley.mx Author: Evgeni Golov Author-email: evgeni@golov.de License: BSD Description: UNKNOWN Platform: UNKNOWN bley-2.0.0/README.md0000644000175000017500000001621512407263667014521 0ustar evgenievgeni00000000000000ABOUT ===== `bley` is an intelligent greylisting daemon for Postfix (and Exim). It uses various test (incl. RBL and SPF) to decide whether a sender should be greylisted or not, thus mostly eliminating the usual greylisting delay while still filtering most of the spam. PACKAGES ======== * Arch: [releases](https://aur.archlinux.org/packages/bley/) and [git](https://aur.archlinux.org/packages/bley-git) * [Debian](http://packages.debian.org/bley) * [Ubuntu](http://packages.ubuntu.com/bley) DEPENDENCIES ============ `bley` is written in [Python](http://python.org) using the [Twisted](http://twistedmatrix.com/) framework. It uses [pyspf](http://pypi.python.org/pypi/pyspf) for SPF validation and [publicsuffix](https://pypi.python.org/pypi/publicsuffix) for checking of domains against the [PublicSuffix.org](http://publicsuffix.org) database. Database interaction is implemented via [sqlite3](http://docs.python.org/2/library/sqlite3.html) for SQLite, [psycopg2](http://initd.org/psycopg/) for PostgreSQL and [MySQL-Python](http://mysql-python.sourceforge.net/) for MySQL. INSTALLATION ============ Quick and dirty --------------- Unpack the tarball (or clone the git tree), adjust `bley.conf.example`, rename it to `bley.conf` and run `./bley`. Still quick, but not dirty -------------------------- Unpack the tarball (or clone the git tree), run `python setup.py build` followed by `python setup.py install`, copy `/etc/bley/bley.conf.example` to `/etc/bley/bley.conf`, adjust it to your needs (see CONFIGURATION below) and run `/usr/bin/bley`. CONFIGURATION ============= Basically you just have to configure the database: dbtype = pgsql for PostgreSQL, mysql for MySQL or sqlite3 for SQLite dbhost = the host where the database runs on (usually localhost) dbport = the port where the database runs on (can be left unset for the default 5432 for PostgreSQL and 3306 for MySQL) dbuser = the name of the database user dbpass = the password of the database user dbname = the name (or path in case of SQLite) of the database dbpath = you can also set the path separately and load ${dbpath}/${dbname} After that you can point your Postfix to `bley` as a [policy server](http://www.postfix.org/SMTPD_POLICY_README.html) by adding `check_policy_service inet:127.0.0.1:1337` to your `smtpd_recipient_restrictions` in `main.cf`. `bley` will be working now, but you probably would like to tune it more for your needs (esp. the used DNWLs and DNSBLs, the greylisting times and the hit thresholds). Additional Configuration ------------------------ Sometimes, you want to bind `bley` to something else than `127.0.0.1:1337`, this can be achieved with the following two parameters. listen_addr = 127.0.0.1 listen_port = 1337 As `bley` is designed to be a deamon, it will write a pid file and a log file. The locations of the two can be configured with the following parameters. pid_file = bley.pid log_file = bley.log Setting `log_file` to the special word `syslog` will make `bley` log to `syslog` instead of a file, using the `mail` facility. If you want to inform the sender about the greylisting, you can adjust the message sent via reject_msg = greylisted, try again later The DNSWLs and DNSBLs `bley` uses for its tests can be set via dnsbls = ix.dnsbl.manitu.net, dnsbl.ahbl.org, dnsbl.sorbs.net dnswls = list.dnswl.org Thresholds define how many sub-checks have to hit, to trigger a feature (whitelisting in case of dnswl, greylisting in case of dnsbl and rfc). dnswl_threshold = 1 dnsbl_threshold = 1 rfc_threshold = 2 How long should a sender be greylisted, when should we allow him in at the very last and how long should he have to wait more, if he retries to early (all in minutes)? greylist_period = 29 greylist_max = 720 greylist_penalty= 10 After how many days should old entries be deleted from the database? Entries of senders who have not verified to be "good" should be purged earlier. purge_days = 40 purge_bad_days = 10 SPF ([Sender Policy Framework](http://www.openspf.org)) checks can be turned off. [SPF Best Guess](http://www.openspf.net/Best_Practices/No_Best_Guess) should always be turned off. use_spf = 1 use_spf_guess = 0 If you use Exim instead of Postfix, set this to 1. It will automatically close connections after the decision is sent. While Postfix supports checking multiple senders over the same connections, Exim does not. In fact it even closes the sending part of the socket as soon all details have been transmitted. exim_workaround = 0 Whitelisting ------------ In some situations, it is useful to be able to whitelist senders or recipients. This can be done by providing lists as files (syntax is [postgrey](http://postgrey.schweikert.ch/) compatible). whitelist_recipients_file = ./whitelist_recipients whitelist_clients_file = ./whitelist_clients ### whitelist_recipients_file This file contains a list of recipients who are excluded from greylisting. One entry per line. An entry can be either a full email address, the local part, a domain name or a regular expression: user@example.com postmaster@ example.com /app.*example/ ### whitelist_clients_file This file contains a list of clients who are excluded from greylisting. One entry per line. An entry can be either an IP adress, a subnet, a domain name or a regular expression. 192.0.2.200/30 example.net /sender.*example/ CHECKS ====== Known sender ------------ The first check is, of course, whether our database already knows about the `(ip, sender, recipient)` tuple. If it does, act accordignly, otherways continue with the other checks. DNSWL / DNSBL ------------- Check whether the sender IP address is listed in the configured DNSWLs and DNSBLs. If either one reaches the configured threshold, the mail is considered good or bad, depending on which threshold was reached. RFC --- While the following checks are not all about stricktly implementing the RFC, all of them try to identify suboptimal behaviour of the sending MTA, which often indicates a spammer. ### HELO Check whether the name used in `HELO/EHLO` matches the reverse DNS entry. ### DynIP Check whether the hostname looks like a dynamic one. ### sender equals recipient People usually do not send mail themself "over the Internet" (and local mail should not be checked by a policy daemon). Spammers on the other hand, often try to bypass address-checks by using the same address as sender and receiver. ### SPF The Sender Policy Framework allows domain owners to define which servers are allowed to send mail using their domain and which are not. bleygraph ========= `bley` includes a small graphing utility called `bleygraph`. It will analyze the `bley_log` table of the database, and plot a few graphs using [matplotlib](http://matplotlib.org/). There is not much configuration possible for `bleygraph`: the database settings are taken from the `bley` section of `bley.conf` and the path for the graph output (`destdir`) is the only setting in the `bleygraph` section of the configuration file. BUILD STATUS ============ [![Build Status](https://travis-ci.org/evgeni/bley.png?branch=master)](https://travis-ci.org/evgeni/bley) bley-2.0.0/bleygraph.10000644000175000017500000000075112417767474015305 0ustar evgenievgeni00000000000000.TH BLEYGRAPH "1" "May 2014" "bleygraph 2.0.0" "bley" .SH NAME bleygraph \- stats plotter for bley .SH SYNOPSIS .B bleygraph [\fIoptions\fR] .SH OPTIONS .TP \fB\-\-version\fR show program's version number and exit .TP \fB\-h\fR, \fB\-\-help\fR show this help message and exit .TP \fB\-d\fR DESTDIR, \fB\-\-destdir\fR=\fIDESTDIR\fR write to DESTDIR .TP \fB\-c\fR CONFFILE, \fB\-\-config\fR=\fICONFFILE\fR load configuration from CONFFILE .TP \fB\-q\fR, \fB\-\-quiet\fR be quiet (no output) bley-2.0.0/MANIFEST.in0000644000175000017500000000035312407263667014774 0ustar evgenievgeni00000000000000include *.conf include *.example include *.1 include bley.service include bley.logcheck include CHANGELOG.md include README.md include requirements.txt include Makefile include test/*.py include test/*.conf.in include test/whitelist_* bley-2.0.0/Makefile0000644000175000017500000000375612407263667014710 0ustar evgenievgeni00000000000000TRIAL_VERSION := $(shell trial --version |sed "s/[^0-9]//g") #TRIAL_FLAGS ?= $(shell test $(TRIAL_VERSION) -ge 1230 && echo "-j2") TRIAL ?= trial sdist: python setup.py sdist test: test-psql test-mysql test-sqlite pep8 --ignore=E501 ./bley . make test-clean test-sqlite: test-setup-sqlite $(TRIAL) $(TRIAL_FLAGS) test -[ ! -f ./test/bley_test.pid ] || kill $$(cat ./test/bley_test.pid) ./bleygraph -c ./test/bley_sqlite.conf -d ./test/bleygraphout make test-clean test-setup-sqlite: test-clean sed "s#.*DBPORT##;s#DBTYPE#sqlite3#;s#DBNAME#./test/bley_sqlite.db#" ./test/bley_test.conf.in > ./test/bley_sqlite.conf ./bley -c ./test/bley_sqlite.conf -p ./test/bley_test.pid test-psql: pg_virtualenv make test-psql-real test-psql-real: test-setup-psql $(TRIAL) $(TRIAL_FLAGS) test -[ ! -f ./test/bley_test.pid ] || kill $$(cat ./test/bley_test.pid) ./bleygraph -c ./test/bley_psql.conf -d ./test/bleygraphout make test-clean test-setup-psql: test-clean sed "s#DBHOST#$$PGHOST#;s#DBPORT#$$PGPORT#;s#DBUSER#$$PGUSER#;s#DBPASS#$$PGPASSWORD#;s#DBTYPE#pgsql#;s#DBNAME#bley_test#" ./test/bley_test.conf.in > ./test/bley_psql.conf createdb bley_test ./bley -c ./test/bley_psql.conf -p ./test/bley_test.pid test-mysql: my_virtualenv make test-mysql-real test-mysql-real: test-setup-mysql $(TRIAL) $(TRIAL_FLAGS) test -[ ! -f ./test/bley_test.pid ] || kill $$(cat ./test/bley_test.pid) ./bleygraph -c ./test/bley_mysql.conf -d ./test/bleygraphout make test-clean test-setup-mysql: test-clean sed "s#DBHOST#$$MYSQL_HOST#;s#DBPORT#$$MYSQL_TCP_PORT#;s#DBUSER#$$MYSQL_USER#;s#DBPASS#$$MYSQL_PWD#;s#DBTYPE#mysql#;s#DBNAME#bley_test#" ./test/bley_test.conf.in > ./test/bley_mysql.conf echo "CREATE DATABASE bley_test;" | mysql ./bley -c ./test/bley_mysql.conf -p ./test/bley_test.pid test-clean: -[ ! -f ./test/bley_test.pid ] || kill $$(cat ./test/bley_test.pid) rm -f ./test/bley_sqlite.db ./test/bley_sqlite.conf ./test/bley_psql.conf ./test/bley_mysql.conf rm -rf ./test/bleygraphout/ .PHONY: sdist test bley-2.0.0/postfix.py0000644000175000017500000000745712407263667015320 0ustar evgenievgeni00000000000000# Copyright (c) 2009-2014 Evgeni Golov # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # 3. Neither the name of the University nor the names of its contributors # may be used to endorse or promote products derived from this software # without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. from twisted.protocols.basic import LineOnlyReceiver from twisted.internet.protocol import Factory from twisted.internet.interfaces import IHalfCloseableProtocol from zope.interface import implements import ipaddr class PostfixPolicy(LineOnlyReceiver): '''Basic implementation of a Postfix policy service.''' implements(IHalfCloseableProtocol) required_params = [] def __init__(self): self.params = {} self.delimiter = '\n' def lineReceived(self, line): '''Parse stuff from Postfix and call check_policy() afterwards.''' line = line.strip().lower() if line == '': if len(self.params) > 0: valid_request = True for p in self.required_params: if p not in self.params: valid_request = False break if valid_request: self.check_policy() else: self.send_action('DUNNO') self.params = {} else: try: (pkey, pval) = line.split('=', 1) try: pval = pval.decode('utf-8', 'ignore') pval = pval.encode('us-ascii', 'ignore') except: pass if pkey == 'client_address': pval = ipaddr.IPAddress(pval).exploded self.params[pkey] = pval except: print 'Could not parse "%s"' % line def check_policy(self): '''Check the incoming mail based on our policy and tell Postfix about our decision. You probably want to override this one with a function that does something useful. ''' self.send_action('DUNNO') def send_action(self, action='DUNNO'): '''Send action back to Postfix. @type action: string @param action: the action to be sent to Postfix (default: 'DUNNO') ''' self.sendLine('action=%s\n' % action) if self.factory.exim_workaround: self.transport.loseConnection() def readConnectionLost(self): pass def writeConnectionLost(self): self.transport.loseConnection() class PostfixPolicyFactory(Factory): protocol = PostfixPolicy exim_workaround = False bley-2.0.0/bley.10000644000175000017500000000122012417767474014253 0ustar evgenievgeni00000000000000.TH BLEY "1" "May 2014" "bley 2.0.0" "bley" .SH NAME bley \- intelligent greylisting daemon for Postfix .SH SYNOPSIS .B bley [\fIoptions\fR] .SH OPTIONS .TP \fB\-\-version\fR show program's version number and exit .TP \fB\-h\fR, \fB\-\-help\fR show this help message and exit .TP \fB\-p\fR \fIPID_FILE\fR, \fB\-\-pidfile\fR=\fIPID_FILE\fR use PID_FILE for storing the PID .TP \fB\-c\fR \fICONFFILE\fR, \fB\-\-config\fR=\fICONFFILE\fR load configuration from CONFFILE .TP \fB\-v\fR, \fB\-\-verbose\fR use verbose output .TP \fB\-d\fR, \fB\-\-debug\fR don't daemonize the process and log to stdout \fB\-f\fR, \fB\-\-foreground\fR don't daemonize the process bley-2.0.0/CHANGELOG.md0000644000175000017500000000173112417767534015053 0ustar evgenievgeni000000000000002.0.0 ===== * No changes since 2.0.0-beta.2 2.0.0-beta.2 ============ * Lots of PEP8 and PyFlakes fixes * Python 2.6 support in setup.py 2.0.0-beta.1 ============ * New tool (bleygraph) for creating graphs * Rename bley.conf to bley.conf.example in the tarball * Rename README to README.md * Disable SPF Best Guess by default * Use Semantic Versioning (http://semver.org/) * Support for SQLite * Support for Exim * Support for Twisted 12, 13 and 14 * Support for publicsuffix.org * Support for systemd * Support for postgrey comptatible whitelist_recipients and whitelist_clients lists * Support for configuring the cache age * Improve documentation * Add tests * Add example logcheck filter rules 0.1.5 ===== * MySQL fixes * Add a manpage * Parse addresses case insensitive 0.1.4 ===== * Reconnect to dead databases 0.1.3 ===== * Strip non-ASCII chars from Postfix input 0.1.2 ===== * RBL fixes 0.1.1 ===== * MySQL fixes 0.1 === * Initial release bley-2.0.0/test/0000755000175000017500000000000012421233663014201 5ustar evgenievgeni00000000000000bley-2.0.0/test/test_bleyhelpers.py0000644000175000017500000000736712407263667020160 0ustar evgenievgeni00000000000000from twisted.trial import unittest import bleyhelpers import ipaddr class BleyHelpersTestCase(unittest.TestCase): ips = [ ("127.0.0.1", "1.0.0.127"), ("127.0.2.1", "1.2.0.127"), ("192.0.2.23", "23.2.0.192"), ("10.11.12.13", "13.12.11.10"), ("2001:DB8::1", "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2"), ("2001:0db8:1000:0100:0010:0001:0000:0000", "0.0.0.0.0.0.0.0.1.0.0.0.0.1.0.0.0.0.1.0.0.0.0.1.8.b.d.0.1.0.0.2"), ] dynamic_hosts = [ 'ip-178-231-86-123.unitymediagroup.de', 'p4FCA123E.dip0.t-ipconnect.de', 'muedsl-82-207-123-222.citykom.de', 'dslb-178-010-012-123.pools.arcor-ip.net', '108.30.123.23.dynamic.mundo-r.com', 'AMontsouris-159-1-80-123.w92-321.abo.wanadoo.fr', 'dyndsl-085-032-023-123.ewe-ip-backbone.de', 'HSI-KBW-078-043-123-011.hsi4.kabel-badenwuerttemberg.de', 'p578ABBAA.dip.t-dialin.net', 'e179123404.adsl.alicedsl.de', ] static_hosts = [ 'mail.example.com', 'mail.dynamicpower.example.com', 'chi.die-welt.net', 'v1234207132567817.yourvserver.net', 'vps-1018888-4321.united-hoster.de', 'static.88-198-123-123.clients.your-server.de', 'port-83-236-123-123.static.qsc.de', ] def test_reverse_ip(self): for ip in self.ips: self.assertEquals(bleyhelpers.reverse_ip(ip[0]), ip[1]) def test_domain_from_host(self): if not bleyhelpers.publicsuffix: raise unittest.SkipTest("publicsuffix module not available, " "domain tests skipped") domains = [ ("example.com", "example.com"), ("example.co.uk", "example.co.uk"), ("example.museum", "example.museum"), ("some.weird.host.example.com", "example.com"), ("ip-123-123-123-123.dyn.example.co.uk", "example.co.uk"), ("A.really.BIG.example.museum", "example.museum"), ] for domain in domains: self.assertEquals(bleyhelpers.domain_from_host(domain[0]), domain[1]) def test_check_dyn_host_dynamic(self): for host in self.dynamic_hosts: self.assertEquals(bleyhelpers.check_dyn_host(host), 1) def test_check_dyn_host_static(self): for host in self.static_hosts: self.assertEquals(bleyhelpers.check_dyn_host(host), 0) def test_check_helo_good(self): for host in self.dynamic_hosts + self.static_hosts: params = { 'client_name': host, 'helo_name': host, } self.assertEquals(bleyhelpers.check_helo(params), 0) def test_check_helo_domain(self): for host in self.dynamic_hosts + self.static_hosts: params = { 'client_name': host, 'helo_name': 'mail.%s' % host, } self.assertEquals(bleyhelpers.check_helo(params), 1) def test_check_helo_ip(self): for ip in self.ips: params = { 'client_name': '%s.dyn.example.com' % ip[1], 'client_address': ipaddr.IPAddress(ip[0]).exploded, 'helo_name': '[%s]' % ip[0], } self.assertEquals(bleyhelpers.check_helo(params), 1) def test_check_helo_bad(self): for ip in self.ips: params = { 'client_name': '%s.dyn.example.com' % ip[1], 'client_address': ipaddr.IPAddress(ip[0]).exploded, 'helo_name': 'windowsxp.local', } self.assertEquals(bleyhelpers.check_helo(params), 2) def test_check_spf(self): raise unittest.SkipTest("SPF checks need a working network") bley-2.0.0/test/whitelist_clients0000644000175000017500000000010412407262723017657 0ustar evgenievgeni00000000000000 wlclient.test /import.*mer/ 192.0.2.200/30 2001:DB8::123:0/112 bley-2.0.0/test/test_postfix.py0000644000175000017500000000116312407262723017312 0ustar evgenievgeni00000000000000from postfix import PostfixPolicyFactory from twisted.trial import unittest from twisted.test import proto_helpers class PostfixPolicyTestCase(unittest.TestCase): def setUp(self): factory = PostfixPolicyFactory() self.proto = factory.buildProtocol(('127.0.0.1', 0)) self.tr = proto_helpers.StringTransport() self.proto.makeConnection(self.tr) def test_DUNNO(self): self.proto.lineReceived("sender=root@example.com") self.proto.lineReceived("recipient=user@example.com") self.proto.lineReceived("") self.assertEqual(self.tr.value(), "action=DUNNO\n\n") bley-2.0.0/test/__init__.py0000644000175000017500000000002412407262723016311 0ustar evgenievgeni00000000000000# empty __init__.py bley-2.0.0/test/whitelist_recipients0000644000175000017500000000006512407262723020371 0ustar evgenievgeni00000000000000 postmaster@ abuse@ dontgreylist.test /app.*test/ bley-2.0.0/test/test_bley.py0000644000175000017500000003162112407263667016563 0ustar evgenievgeni00000000000000from postfix import PostfixPolicy from twisted.trial import unittest from twisted.internet.protocol import ClientFactory from twisted.internet.defer import Deferred, DeferredList from twisted.internet import task from twisted.internet import reactor import ipaddr class PostfixPolicyClient(PostfixPolicy): action = "" reason = "" def connectionMade(self): for x in self.factory.data: line = "%s=%s" % (x, self.factory.data[x]) self.sendLine(line) self.sendLine("") def lineReceived(self, line): line = line.strip() if line.startswith("action"): actionline = line.split(None, 1) self.action = actionline[0] if len(actionline) == 2: self.reason = actionline[1] if line == "": self.factory.action_received(self.action) self.transport.loseConnection() class PostfixPolicyClientFactory(ClientFactory): protocol = PostfixPolicyClient def __init__(self, data): self.deferred = Deferred() self.data = data def action_received(self, action, text=""): if self.deferred is not None: d, self.deferred = self.deferred, None d.callback(action) def clientConnectionFailed(self, connector, reason): if self.deferred is not None: d, self.deferred = self.deferred, None d.errback(reason) class BleyTestCase(unittest.TestCase): ipv4net = ipaddr.IPNetwork('192.0.2.0/24') ipv4generator = ipv4net.iterhosts() ipv6net = ipaddr.IPNetwork('2001:DB8::/32') ipv6generator = ipv6net.iterhosts() def _get_next_ipv4(self): try: return self.ipv4generator.next() except StopIteration: self.ipv4generator = self.ipv4net.iterhosts() return self.ipv4generator.next() def _get_next_ipv6(self): try: return self.ipv6generator.next() except StopIteration: self.ipv6generator = self.ipv6net.iterhosts() return self.ipv6generator.next() def _assert_dunno_action(self, action): self.assertEquals(action, "action=DUNNO") def _assert_defer_action(self, action): self.assertEquals(action, "action=DEFER_IF_PERMIT") def test_incomplete_request(self): data = { 'sender': 'root@example.com', 'recipient': 'user@example.com', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_dunno_action) return d def _test_good_client(self, ip): data = { 'sender': 'root@example.com', 'recipient': 'user@example.com', 'client_address': ip, 'client_name': 'localhost', 'helo_name': 'localhost', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_dunno_action) return d def test_good_client_v4(self): ip = self._get_next_ipv4() return self._test_good_client(ip) def test_good_client_v6(self): ip = self._get_next_ipv6() return self._test_good_client(ip) def _test_ip_help_and_dyn_host(self, ip): data = { 'sender': 'nothinguseful@example.com', 'recipient': 'nothinguseful@example.com', 'client_address': ip, 'client_name': 'client123.dyn.example.com', 'helo_name': '[%s]' % ip, } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_defer_action) return d def test_ip_help_and_dyn_host_v4(self): ip = self._get_next_ipv4() return self._test_ip_help_and_dyn_host(ip) def test_ip_help_and_dyn_host_v6(self): ip = self._get_next_ipv6() return self._test_ip_help_and_dyn_host(ip) def _test_same_sender_recipient_and_dyn_host(self, ip): data = { 'sender': 'nothinguseful@example.com', 'recipient': 'nothinguseful@example.com', 'client_address': ip, 'client_name': 'client123.dyn.example.com', 'helo_name': 'client123.dyn.example.com', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_defer_action) return d def test_same_sender_recipient_and_dyn_host_v4(self): ip = self._get_next_ipv4() return self._test_same_sender_recipient_and_dyn_host(ip) def test_same_sender_recipient_and_dyn_host_v6(self): ip = self._get_next_ipv6() return self._test_same_sender_recipient_and_dyn_host(ip) def _test_same_sender_recipient_and_ip_helo(self, ip): data = { 'sender': 'nothinguseful@example.com', 'recipient': 'nothinguseful@example.com', 'client_address': ip, 'client_name': 'localhost', 'helo_name': '[%s]' % ip, } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_defer_action) return d def test_same_sender_recipient_and_ip_helo_v4(self): ip = self._get_next_ipv4() return self._test_same_sender_recipient_and_ip_helo(ip) def test_same_sender_recipient_and_ip_helo_v6(self): ip = self._get_next_ipv6() return self._test_same_sender_recipient_and_ip_helo(ip) def test_zzz_greylisting(self): ip = self._get_next_ipv4() data = { 'sender': 'root@example.com', 'recipient': 'user@gl.example.com', 'client_address': ip, 'client_name': 'localhost', 'helo_name': 'invalid.local', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_defer_action) d2 = task.deferLater(reactor, 5, get_action, "127.0.0.1", 1337, data) d2.addCallback(self._assert_defer_action) d3 = task.deferLater(reactor, 65, get_action, "127.0.0.1", 1337, data) d3.addCallback(self._assert_dunno_action) return DeferredList([d, d2, d3]) def _test_bad_helo(self, ip): data = { 'sender': 'root@example.com', 'recipient': 'user@example.com', 'client_address': ip, 'client_name': 'localhost', 'helo_name': 'invalid.local', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_defer_action) return d def test_bad_helo_v4(self): ip = self._get_next_ipv4() return self._test_bad_helo(ip) def test_bad_helo_v6(self): ip = self._get_next_ipv6() return self._test_bad_helo(ip) def _test_postmaster(self, ip): data = { 'sender': 'angryuser@different.example.com', 'recipient': 'postmaster@example.com', 'client_address': ip, 'client_name': 'localhost', 'helo_name': 'invalid.local', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_dunno_action) return d def test_postmaster_v4(self): ip = self._get_next_ipv4() return self._test_postmaster(ip) def test_postmaster_v6(self): ip = self._get_next_ipv6() return self._test_postmaster(ip) def _test_whitelist_recipient_domain(self, ip): data = { 'sender': 'someone@example.com', 'recipient': 'user@dontgreylist.test', 'client_address': ip, 'client_name': 'localhost', 'helo_name': 'invalid.local', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_dunno_action) return d def test_whitelist_recipient_domain_v4(self): ip = self._get_next_ipv4() return self._test_whitelist_recipient_domain(ip) def test_whitelist_recipient_domain_v6(self): ip = self._get_next_ipv6() return self._test_whitelist_recipient_domain(ip) def _test_whitelist_recipient_subdomain(self, ip): data = { 'sender': 'someone@example.com', 'recipient': 'user@subdomain.dontgreylist.test', 'client_address': ip, 'client_name': 'localhost', 'helo_name': 'invalid.local', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_dunno_action) return d def test_whitelist_recipient_subdomain_v4(self): ip = self._get_next_ipv4() return self._test_whitelist_recipient_subdomain(ip) def test_whitelist_recipient_subdomain_v6(self): ip = self._get_next_ipv6() return self._test_whitelist_recipient_subdomain(ip) # domain name is a substring of whitelisted domain - should greylist def _test_whitelist_recipient_negative_test1(self, ip): data = { 'sender': 'someone@example.com', 'recipient': 'user@greylist.test', 'client_address': ip, 'client_name': 'localhost', 'helo_name': 'invalid.local', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_defer_action) return d def test_whitelist_recipient_negative_test1_v4(self): ip = self._get_next_ipv4() return self._test_whitelist_recipient_negative_test1(ip) def test_whitelist_recipient_negative_test1_v6(self): ip = self._get_next_ipv6() return self._test_whitelist_recipient_negative_test1(ip) # domain name contains whitelisted domainname, # but is not a subdomain - should greylist def _test_whitelist_recipient_negative_test2(self, ip): data = { 'sender': 'someone@example.com', 'recipient': 'user@xxxxdontgreylist.test', 'client_address': ip, 'client_name': 'localhost', 'helo_name': 'invalid.local', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_defer_action) return d def test_whitelist_recipient_negative_test2_v4(self): ip = self._get_next_ipv4() return self._test_whitelist_recipient_negative_test2(ip) def test_whitelist_recipient_negative_test2_v6(self): ip = self._get_next_ipv6() return self._test_whitelist_recipient_negative_test2(ip) def _test_whitelist_recipient_regex(self, ip): data = { 'sender': 'someone@example.com', 'recipient': 'user@application.test', 'client_address': ip, 'client_name': 'localhost', 'helo_name': 'invalid.local', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_dunno_action) return d def test_whitelist_recipient_regex_v4(self): ip = self._get_next_ipv4() return self._test_whitelist_recipient_regex(ip) def test_whitelist_recipient_regex_v6(self): ip = self._get_next_ipv6() return self._test_whitelist_recipient_regex(ip) def _test_whitelist_clients_domain(self, ip): data = { 'sender': 'someone@example.com', 'recipient': 'user@example.com', 'client_address': ip, 'client_name': 'mail.wlclient.test', 'helo_name': 'invalid.external', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_dunno_action) return d def test_whitelist_clients_domain_v4(self): ip = self._get_next_ipv4() return self._test_whitelist_clients_domain(ip) def test_whitelist_clients_domain_v6(self): ip = self._get_next_ipv6() return self._test_whitelist_clients_domain(ip) def _test_whitelist_clients_regex(self, ip): data = { 'sender': 'someone@example.com', 'recipient': 'user@example.com', 'client_address': ip, 'client_name': 'important.customer.test', 'helo_name': 'invalid.local', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_dunno_action) return d def test_whitelist_clients_regex_v4(self): ip = self._get_next_ipv4() return self._test_whitelist_clients_regex(ip) def test_whitelist_clients_regex_v6(self): ip = self._get_next_ipv6() return self._test_whitelist_clients_regex(ip) def _test_whitelist_client_ip(self, ip): data = { 'sender': 'someone@example.com', 'recipient': 'user@example.com', 'client_address': ip, 'client_name': 'localhost', 'helo_name': 'invalid.local', } d = get_action("127.0.0.1", 1337, data) d.addCallback(self._assert_dunno_action) return d def test_whitelist_client_ip_v4(self): ip = '192.0.2.202' return self._test_whitelist_client_ip(ip) def test_whitelist_client_ip_v6(self): ip = '2001:DB8::123:456' return self._test_whitelist_client_ip(ip) def get_action(host, port, data): factory = PostfixPolicyClientFactory(data) reactor.connectTCP(host, port, factory) return factory.deferred bley-2.0.0/test/bley_test.conf.in0000644000175000017500000000275312407262723017461 0ustar evgenievgeni00000000000000[bley] # On which IP and port should the daemon listen? listen_addr = 127.0.0.1 listen_port = 1337 # Where to save the PID file? pid_file = bley.pid # Where to save the log file? log_file = bley.log reject_msg = greylisted, try again later # Which database to use? [pgsql|mysql|sqlite3] # note: for sqlite3, just set dbname to the path to the database dbtype = DBTYPE dbhost = DBHOST dbport = DBPORT dbuser = DBUSER dbpass = DBPASS dbname = DBNAME whitelist_recipients_file = ./whitelist_recipients whitelist_clients_file = ./whitelist_clients # Which DNSBLs and DNSWLs to use? dnsbls = dnswls = # Whitelist after dnswl_threshold hits. dnswl_threshold = 1 # Greylist after dnsbl_threshold hits. dnsbl_threshold = 1 # Greylist after rfc_threshold hits in the RFC checks. rfc_threshold = 2 # Wait greylist_period minutes before accepting greyed sender. greylist_period = 1 # Max wait greylist_max minutes before accepting greyed sender. # (Accept all senders after 12h.) greylist_max = 720 # Add greylist_penalty minutes for every connection before greylist_period. greylist_penalty= 0 # Purge good entries from the database after purge_days inactivities. purge_days = 40 # Purge bad entries from the database after purge_bad_days inactivities. purge_bad_days = 10 # Use SPF? use_spf = 0 use_spf_guess = 0 # Use Exim workaround (close the socket after an action has been sent)? exim_workaround = 0 # How long should the cache entries be valid (in minutes)? cache_valid = 0 [bleygraph] destdir = stats bley-2.0.0/bley0000755000175000017500000003766312417767474014143 0ustar evgenievgeni00000000000000#!/usr/bin/env python # # Copyright (c) 2009-2014 Evgeni Golov # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # 3. Neither the name of the University nor the names of its contributors # may be used to endorse or promote products derived from this software # without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. import os import sys import datetime import logging from optparse import OptionParser from ConfigParser import SafeConfigParser import ipaddr import re from twisted.internet import reactor try: from twisted.scripts._twistd_unix import UnixApplicationRunner except: print "Could not import _twistd_unix. Exiting..." sys.exit(1) from twisted.application import internet, service from bley import BleyPolicyFactory logger = logging.getLogger('bley') __CREATE_DB_QUERY = ''' CREATE TABLE IF NOT EXISTS bley_status ( ip VARCHAR(39) NOT NULL, status SMALLINT NOT NULL DEFAULT 1, last_action TIMESTAMP NOT NULL default CURRENT_TIMESTAMP, sender VARCHAR(254), recipient VARCHAR(254), fail_count INT DEFAULT 0, INDEX bley_status_index USING btree (ip, sender, recipient), INDEX bley_status_action_index USING btree (last_action) ) CHARACTER SET 'ascii' ''' __UPDATE_DB_QUERY = ''' ALTER TABLE bley_status CHANGE last_action last_action TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP; ''' __CREATE_DB_QUERY_PG = ''' CREATE TABLE bley_status ( ip VARCHAR(39) NOT NULL, status SMALLINT NOT NULL DEFAULT 1, last_action TIMESTAMP NOT NULL, sender VARCHAR(254), recipient VARCHAR(254), fail_count INT DEFAULT 0 ); CREATE INDEX bley_status_index ON bley_status USING btree (ip ASC NULLS LAST, sender ASC NULLS LAST, recipient ASC NULLS LAST); CREATE INDEX bley_status_action_index ON bley_status USING btree (last_action ASC NULLS LAST); ''' __CHECK_DB_QUERY_PG = ''' SELECT tablename FROM pg_catalog.pg_tables WHERE tablename = 'bley_status' ''' __CREATE_DB_QUERY_SL = ''' CREATE TABLE IF NOT EXISTS bley_status ( ip VARCHAR(39) NOT NULL, status SMALLINT NOT NULL DEFAULT 1, last_action TIMESTAMP NOT NULL, sender VARCHAR(254), recipient VARCHAR(254), fail_count INT DEFAULT 0 ); CREATE INDEX IF NOT EXISTS bley_status_index ON bley_status (ip ASC, sender ASC, recipient ASC); CREATE INDEX IF NOT EXISTS bley_status_action_index ON bley_status (last_action); ''' __CLEAN_DB_QUERY = ''' DELETE FROM bley_status WHERE last_action<%(old)s OR (last_action<%(old_bad)s AND status>=2) ''' __CLEAN_DB_QUERY_SL = ''' DELETE FROM bley_status WHERE last_action<:old OR (last_action<:old_bad AND status>=2) ''' __CREATE_LOGDB_QUERY = ''' CREATE TABLE IF NOT EXISTS bley_log ( logtime TIMESTAMP NOT NULL default CURRENT_TIMESTAMP, ip VARCHAR(39) NOT NULL, sender VARCHAR(254), recipient VARCHAR(254), action VARCHAR(254), check_dnswl INT DEFAULT 0, check_dnsbl INT DEFAULT 0, check_helo INT DEFAULT 0, check_dyn INT DEFAULT 0, check_db INT DEFAULT 0, check_spf INT DEFAULT 0, check_s_eq_r INT DEFAULT 0, check_postmaster INT DEFAULT 0, check_cache INT DEFAULT 0, INDEX bley_log_index USING btree (logtime, action) ) CHARACTER SET 'ascii'; ''' __CREATE_LOGDB_QUERY_PG = ''' CREATE TABLE bley_log ( logtime TIMESTAMP NOT NULL, ip VARCHAR(39) NOT NULL, sender VARCHAR(254), recipient VARCHAR(254), action VARCHAR(254), check_dnswl INT DEFAULT 0, check_dnsbl INT DEFAULT 0, check_helo INT DEFAULT 0, check_dyn INT DEFAULT 0, check_db INT DEFAULT 0, check_spf INT DEFAULT 0, check_s_eq_r INT DEFAULT 0, check_postmaster INT DEFAULT 0, check_cache INT DEFAULT 0 ); CREATE INDEX bley_log_index ON bley_log USING btree (logtime DESC NULLS FIRST, action ASC NULLS LAST); ''' __CHECK_LOGDB_QUERY_PG = ''' SELECT tablename FROM pg_catalog.pg_tables WHERE tablename = 'bley_log' ''' __CREATE_LOGDB_QUERY_SL = ''' CREATE TABLE IF NOT EXISTS bley_log ( logtime TIMESTAMP NOT NULL, ip VARCHAR(39) NOT NULL, sender VARCHAR(254), recipient VARCHAR(254), action VARCHAR(254), check_dnswl INT DEFAULT 0, check_dnsbl INT DEFAULT 0, check_helo INT DEFAULT 0, check_dyn INT DEFAULT 0, check_db INT DEFAULT 0, check_spf INT DEFAULT 0, check_s_eq_r INT DEFAULT 0, check_postmaster INT DEFAULT 0, check_cache INT DEFAULT 0 ); CREATE INDEX IF NOT EXISTS bley_log_index ON bley_log (logtime DESC, action ASC); ''' def bley_start(): parser = OptionParser(version='2.0.0') parser.add_option("-p", "--pidfile", dest="pid_file", help="use PID_FILE for storing the PID") parser.add_option("-c", "--config", dest="conffile", action="append", help="load configuration from CONFFILE") parser.add_option("-v", "--verbose", action="store_true", dest="verbose", help="use verbose output") parser.add_option("-d", "--debug", action="store_true", dest="debug", help="don't daemonize the process and log to stdout") parser.add_option("-f", "--foreground", action="store_true", dest="foreground", help="don't daemonize the process") (settings, args) = parser.parse_args() if not settings.conffile: if os.path.isfile('/etc/bley/bley.conf'): settings.conffile = ['/etc/bley/bley.conf'] elif os.path.isfile('bley.conf'): settings.conffile = ['bley.conf'] else: print "Could not find a configuration file, exiting." sys.exit(1) settings.confdir = os.path.dirname(settings.conffile[0]) defaults = { 'listen_addr': '127.0.0.1', 'log_file': None, 'pid_file': None, 'exim_workaround': False, 'dbhost': None, 'dbname': None, 'dbpath': '', 'dbuser': None, 'dbpass': None, 'dbport': '0', 'cache_valid': '60', } config = SafeConfigParser(defaults) config.read(settings.conffile) settings.listen_addr = config.get('bley', 'listen_addr') settings.listen_port = config.getint('bley', 'listen_port') settings.pid_file = settings.pid_file or config.get('bley', 'pid_file') settings.log_file = config.get('bley', 'log_file') settings.cache_valid = config.getint('bley', 'cache_valid') settings.dbtype = config.get('bley', 'dbtype') if settings.dbtype == 'pgsql': database = 'psycopg2' settings.dbsettings = {'host': config.get('bley', 'dbhost'), 'database': config.get('bley', 'dbname'), 'user': config.get('bley', 'dbuser'), 'password': config.get('bley', 'dbpass')} elif settings.dbtype == 'mysql': database = 'MySQLdb' settings.dbsettings = {'host': config.get('bley', 'dbhost'), 'db': config.get('bley', 'dbname'), 'user': config.get('bley', 'dbuser'), 'passwd': config.get('bley', 'dbpass')} elif settings.dbtype == 'sqlite3': global __CLEAN_DB_QUERY database = 'sqlite3' settings.dbsettings = {'database': os.path.join(config.get('bley', 'dbpath'), config.get('bley', 'dbname')), 'detect_types': 1} __CLEAN_DB_QUERY = __CLEAN_DB_QUERY_SL else: print "No supported database configured." sys.exit(1) if (config.has_option('bley', 'dbport') and config.getint('bley', 'dbport') != 0): settings.dbsettings['port'] = config.getint('bley', 'dbport') exec "import %s as databasemodule" % database in globals(), locals() settings.database = databasemodule settings.reject_msg = config.get('bley', 'reject_msg') settings.dnswls = [d.strip() for d in config.get('bley', 'dnswls').split(',') if d.strip() != ""] settings.dnsbls = [d.strip() for d in config.get('bley', 'dnsbls').split(',') if d.strip() != ""] settings.dnswl_threshold = config.getint('bley', 'dnswl_threshold') settings.dnsbl_threshold = config.getint('bley', 'dnsbl_threshold') settings.rfc_threshold = config.getint('bley', 'rfc_threshold') settings.greylist_period = datetime.timedelta(0, config.getint('bley', 'greylist_period') * 60, 0) settings.greylist_max = datetime.timedelta(0, config.getint('bley', 'greylist_max') * 60, 0) settings.greylist_penalty = datetime.timedelta(0, config.getint('bley', 'greylist_penalty') * 60, 0) settings.purge_days = config.getint('bley', 'purge_days') settings.purge_bad_days = config.getint('bley', 'purge_bad_days') settings.use_spf = config.getint('bley', 'use_spf') settings.use_spf_guess = config.getint('bley', 'use_spf_guess') settings.exim_workaround = config.getboolean('bley', 'exim_workaround') if settings.debug: settings.foreground = True settings.log_file = None logger.setLevel(logging.INFO) if settings.log_file == 'syslog': from logging.handlers import SysLogHandler import platform system = platform.system() addr = None if system == 'Linux': addr = '/dev/log' elif system == 'Darwin': addr = '/var/run/syslog' elif 'BSD' in system: addr = '/var/run/log' lh = SysLogHandler(address=addr, facility=SysLogHandler.LOG_MAIL) formatter = logging.Formatter('%(name)s: %(message)s') lh.setFormatter(formatter) logger.addHandler(lh) elif settings.log_file in ['-', '', None]: from logging import StreamHandler lh = StreamHandler(sys.stdout) formatter = logging.Formatter('%(asctime)s %(name)s: %(message)s') lh.setFormatter(formatter) logger.addHandler(lh) else: from logging.handlers import WatchedFileHandler lh = WatchedFileHandler(settings.log_file) formatter = logging.Formatter('%(asctime)s %(name)s: %(message)s') lh.setFormatter(formatter) logger.addHandler(lh) if config.has_option('bley', 'whitelist_recipients_file'): settings.whitelist_recipients_file = config.get('bley', 'whitelist_recipients_file') if not os.path.isabs(settings.whitelist_recipients_file): settings.whitelist_recipients_file = os.path.join(settings.confdir, settings.whitelist_recipients_file) settings.whitelist_recipients = read_whitelist(settings.whitelist_recipients_file)[0] else: settings.whitelist_recipients = [] if config.has_option('bley', 'whitelist_clients_file'): settings.whitelist_clients_file = config.get('bley', 'whitelist_clients_file') if not os.path.isabs(settings.whitelist_clients_file): settings.whitelist_clients_file = os.path.join(settings.confdir, settings.whitelist_clients_file) (settings.whitelist_clients, settings.whitelist_clients_ip) = read_whitelist(settings.whitelist_clients_file) else: settings.whitelist_clients = [] settings.whitelist_clients_ip = [] logger.info("Starting up") db = settings.database.connect(**settings.dbsettings) dbc = db.cursor() if settings.dbtype == 'pgsql': dbc.execute(__CHECK_DB_QUERY_PG) if not dbc.fetchall(): dbc.execute(__CREATE_DB_QUERY_PG) dbc.execute(__CHECK_LOGDB_QUERY_PG) if not dbc.fetchall(): dbc.execute(__CREATE_LOGDB_QUERY_PG) elif settings.dbtype == 'sqlite3': dbc.executescript(__CREATE_DB_QUERY_SL) dbc.executescript(__CREATE_LOGDB_QUERY_SL) else: dbc.execute("set sql_notes = 0") dbc.execute(__CREATE_DB_QUERY) dbc.execute(__CREATE_LOGDB_QUERY) dbc.execute("set sql_notes = 1") dbc.execute(__UPDATE_DB_QUERY) db.commit() dbc.close() db.close() settings.db = settings.database.connect(**settings.dbsettings) class NoLogObserver(object): def emit(self, eventDict): return class NoAppLogger(object): def __init__(self, options): return def start(self, application): return def _getLogObserver(self): return NoLogObserver().emit def stop(self): return class BleyRunner(UnixApplicationRunner): loggerFactory = NoAppLogger def createOrGetApplication(self): bley_app = service.Application("bley") bley_service = internet.TCPServer(settings.listen_port, BleyPolicyFactory(settings), interface=settings.listen_addr) bley_service.setServiceParent(bley_app) return bley_app bley_config = {'originalname': None, 'euid': None, 'profile': None, 'no_save': True, 'debug': False, 'uid': None, 'gid': None, 'chroot': None, 'rundir': '.', 'nodaemon': settings.foreground, 'umask': None, 'pidfile': settings.pid_file, 'syslog': settings.log_file == 'syslog', 'prefix': 'bley', 'logfile': settings.log_file} runner = BleyRunner(bley_config) reactor.addSystemEventTrigger('before', 'shutdown', bley_stop, settings) reactor.callWhenRunning(clean_db, settings) runner.run() def bley_stop(settings): logger.info("Shutting down") def clean_db(settings): if settings.verbose: logger.info("cleaning database") db = settings.database.connect(**settings.dbsettings) dbc = db.cursor() now = datetime.datetime.now() old = now - datetime.timedelta(settings.purge_days, 0, 0) old_bad = now - datetime.timedelta(settings.purge_bad_days, 0, 0) p = {'old': str(old), 'old_bad': str(old_bad)} dbc.execute(__CLEAN_DB_QUERY, p) db.commit() dbc.close() db.close() reactor.callLater(30 * 60, clean_db, settings) def read_whitelist(whitelist_filename): global logger try: whitelist_fh = open(whitelist_filename, 'r') except: logger.warning('Could not open file: %s' % (whitelist_filename)) return (['postmaster@'], ()) whitelist = list() whitelist_ip = list() for line in whitelist_fh: line = line.split('#', 1)[0] line = line.split(';', 1)[0] line = line.strip() if line == "": continue if line.startswith('/') and line.endswith('/'): # line is regex whitelist.append(re.compile(line[1:-1])) continue try: line_ipaddr = ipaddr.IPNetwork(line) # line is IP address whitelist_ip.append(line_ipaddr) except (ValueError): # Ordinary string (domain name or username) whitelist.append(line) return (whitelist, whitelist_ip) bley_start() bley-2.0.0/bleygraph0000755000175000017500000002564012417767474015155 0ustar evgenievgeni00000000000000#!/usr/bin/env python # # Copyright (c) 2009-2014 Evgeni Golov # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # 3. Neither the name of the University nor the names of its contributors # may be used to endorse or promote products derived from this software # without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. import datetime import sys import os from optparse import OptionParser from ConfigParser import SafeConfigParser parser = OptionParser(version='2.0.0') parser.add_option("-d", "--destdir", dest="destdir", help="write to DESTDIR") parser.add_option("-c", "--config", dest="conffile", help="load configuration from CONFFILE") parser.add_option("-q", "--quiet", action="store_true", dest="quiet", help="be quiet (no output)") (settings, args) = parser.parse_args() if not settings.conffile: if os.path.isfile('/etc/bley/bley.conf'): settings.conffile = '/etc/bley/bley.conf' elif os.path.isfile('bley.conf'): settings.conffile = 'bley.conf' else: print "Could not find a configuration file, exiting." sys.exit(1) defaults = { 'dbhost': None, 'dbname': None, 'dbuser': None, 'dbpass': None, 'dbport': '0', } config = SafeConfigParser(defaults) config.read(settings.conffile) settings.destdir = settings.destdir or config.get('bleygraph', 'destdir') if not os.path.exists(settings.destdir): print 'creating destination dir: %s' % settings.destdir os.mkdir(settings.destdir) dbtype = config.get('bley', 'dbtype') if dbtype == 'pgsql': database = 'psycopg2' dbsettings = {'host': config.get('bley', 'dbhost'), 'database': config.get('bley', 'dbname'), 'user': config.get('bley', 'dbuser'), 'password': config.get('bley', 'dbpass')} elif dbtype == 'mysql': database = 'MySQLdb' dbsettings = {'host': config.get('bley', 'dbhost'), 'db': config.get('bley', 'dbname'), 'user': config.get('bley', 'dbuser'), 'passwd': config.get('bley', 'dbpass')} elif dbtype == 'sqlite3': database = 'sqlite3' dbsettings = {'database': config.get('bley', 'dbname'), 'detect_types': 1} else: print "No supported database configured." sys.exit(1) if config.has_option('bley', 'dbport') and config.getint('bley', 'dbport') != 0: dbsettings['port'] = config.getint('bley', 'dbport') exec("import %s as database" % database) import matplotlib matplotlib.use('Agg') import matplotlib.pyplot as plt import matplotlib.dates as mdates dnswl_threshold = config.getint('bley', 'dnswl_threshold') dnsbl_threshold = config.getint('bley', 'dnsbl_threshold') rfc_threshold = config.getint('bley', 'rfc_threshold') now = datetime.datetime.now() db = database.connect(**dbsettings) dbc = db.cursor() years = mdates.YearLocator() # every year months = mdates.MonthLocator() # every month hours = mdates.HourLocator() twohours = mdates.HourLocator(interval=2) fourhours = mdates.HourLocator(interval=4) halfday = mdates.HourLocator(interval=12) oneday = mdates.DayLocator() twoday = mdates.DayLocator(interval=2) fourday = mdates.DayLocator(interval=4) teday = mdates.DayLocator(interval=28) hoursFmt = mdates.DateFormatter('%Y/%m/%d %H:%M') daysFmt = mdates.DateFormatter('%Y/%m/%d') __TIMESLOTS = [ {'title': '12h', 'slot': 60, 'major_locator': twohours, 'minor_locator': hours, 'formatter': hoursFmt, 'slotname': '1h'}, {'title': '24h', 'slot': 2*60, 'major_locator': fourhours, 'minor_locator': twohours, 'formatter': hoursFmt, 'slotname': '2h'}, {'title': '7d', 'slot': 12*60, 'major_locator': oneday, 'minor_locator': halfday, 'formatter': daysFmt, 'slotname': '12h'}, {'title': '28d', 'slot': 2*24*60, 'major_locator': fourday, 'minor_locator': twoday, 'formatter': daysFmt, 'slotname': '2d'}, {'title': '365d', 'slot': 28*24*60, 'major_locator': teday, 'minor_locator': teday, 'formatter': daysFmt, 'slotname': '28d'}, ] __QUERY_BASE = "SELECT COUNT(action) FROM bley_log WHERE action=%s AND logtime<'%s' AND logtime>='%s'" __badcheck_queries = { 'in DNSBL': {'query': 'check_db=-1 and check_cache=0 and check_dnsbl>=%i' % (dnsbl_threshold), 'color': 'black'}, 'bad HELO': {'query': 'check_db=-1 and check_cache=0 and check_helo>=%i and check_dnsbl<%i' % (rfc_threshold, dnsbl_threshold), 'color': 'red'}, 'bad grey': {'query': 'check_db=2 and check_cache=0 and check_dnsbl<%i' % (dnsbl_threshold), 'color': 'blue'}, 'from DynIP': {'query': 'check_db=-1 and check_cache=0 and (check_helo+check_dyn)>=%i and check_helo<%i and check_dnsbl<%i' % (rfc_threshold, rfc_threshold, dnsbl_threshold), 'color': 'orange'}, 'bad SPF': {'query': 'check_db=-1 and check_cache=0 and (check_helo+check_dyn+check_spf)>=%i and (check_helo+check_dyn)<%i and check_dnsbl<%i' % (rfc_threshold, rfc_threshold, dnsbl_threshold), 'color': 'yellow'}, 'sender==recipient': {'query': 'check_db=-1 and check_cache=0 and (check_helo+check_dyn+check_spf+check_s_eq_r)>=%i and (check_helo+check_dyn+check_spf)<%i and check_dnsbl<%i' % (rfc_threshold, rfc_threshold, dnsbl_threshold), 'color': 'orange'}, 'bad cache': {'query': 'check_db=-1 and check_cache=1 and check_dnsbl<%i' % (dnsbl_threshold), 'color': 'pink'}, } __goodcheck_queries = { 'known good': {'query': '(check_db=0 or check_db=1) and check_cache=0', 'color': 'green'}, 'good grey': {'query': 'check_db=2 and check_cache=0', 'color': 'lightblue'}, 'new good': {'query': 'check_db=-1 and check_cache=0 and check_dnswl<%i' % (dnswl_threshold), 'color': 'lightgreen'}, 'in DNSWL': {'query': 'check_db=-1 and check_cache=0 and check_dnswl>=%i' % (dnswl_threshold), 'color': 'lightgrey'}, 'good cache': {'query': 'check_db=-1 and check_cache=1', 'color': 'darkgreen'}, } __HTML_TEMPLATE = ''' bley stats

%(ar)s

%(ch)s

''' __ar_files = [] __ch_files = [] for s in __TIMESLOTS: d = datetime.timedelta(0, s['slot']*60, 0) n = now-datetime.timedelta(0,30*60,0) i = 12 a = {'dates': [], 'mails': []} r = {'dates': [], 'mails': []} checks = {} for check in __badcheck_queries: checks[check] = {'dates': [], 'mails': [], 'color': __badcheck_queries[check]['color']} for check in __goodcheck_queries: checks[check] = {'dates': [], 'mails': [], 'color': __goodcheck_queries[check]['color']} if not settings.quiet: print "plotting %s:" % (s['title']) fig = plt.figure() ax = fig.add_subplot(111) fig2 = plt.figure() ax2 = fig2.add_subplot(111) while i: q1 = __QUERY_BASE % ("'DUNNO'", str(n), str(n-d)) q2 = __QUERY_BASE % ("'DEFER_IF_PERMIT'", str(n), str(n-d)) if dbtype == 'sqlite3': q1 = q1.replace('%s', '?') q2 = q2.replace('%s', '?') dbc.execute(q1) r1 = dbc.fetchone() dbc.execute(q2) r2 = dbc.fetchone() numdate = mdates.date2num(n) a['dates'].append(numdate) r['dates'].append(numdate) a['mails'].append(r1[0]) r['mails'].append(r2[0]) for check in __badcheck_queries: q = "%s AND %s" % (q2, __badcheck_queries[check]['query']) if dbtype == 'sqlite3': q = q.replace('%s', '?') dbc.execute(q) res = dbc.fetchone() checks[check]['dates'].append(numdate) checks[check]['mails'].append(res[0]) for check in __goodcheck_queries: q = "%s AND %s" % (q1, __goodcheck_queries[check]['query']) if dbtype == 'sqlite3': q = q.replace('%s', '?') dbc.execute(q) res = dbc.fetchone() checks[check]['dates'].append(numdate) checks[check]['mails'].append(res[0]) n = n-d i -= 1 ax.plot(a['dates'], a['mails'], label="ham", color='green') ax.plot(r['dates'], r['mails'], label="spam", color='red') for check in checks: ax2.plot(checks[check]['dates'], checks[check]['mails'], label=check, color=checks[check]['color']) ax.legend(loc=2) ax2.legend(loc=2) fig.text(0.125, 0, "ham [ max: %s, avg: %s, min: %s ]\nspam [ max: %s, avg: %s, min: %s ]" % (max(a['mails']), round(float(sum(a['mails']))/len(a['mails']),2), min(a['mails']), max(r['mails']), round(float(sum(r['mails']))/len(r['mails']),2), min(r['mails']))) ax.xaxis.set_major_formatter(s['formatter']) ax.xaxis.set_major_locator(s['major_locator']) ax.xaxis.set_minor_locator(s['minor_locator']) ax2.xaxis.set_major_formatter(s['formatter']) ax2.xaxis.set_major_locator(s['major_locator']) ax2.xaxis.set_minor_locator(s['minor_locator']) fig.suptitle('bley ACCEPT/REJECT stats for the last %s (slot=%s)' % (s['title'], s['slotname'])) fig2.suptitle('bley check stats for the last %s (slot=%s)' % (s['title'], s['slotname'])) fig.autofmt_xdate() fig2.autofmt_xdate() if not settings.quiet: print " - %s" % (os.path.join(settings.destdir, 'ar-%s.png' % s['title'])) print " - %s" % (os.path.join(settings.destdir, 'ch-%s.png' % s['title'])) fig.savefig(os.path.join(settings.destdir, 'ar-%s.png' % s['title'])) fig2.savefig(os.path.join(settings.destdir, 'ch-%s.png' % s['title'])) __ar_files.append('bley ACCEPT/REJECT stats for the last %s' % (s['title'], s['title'])) __ch_files.append('bley check stats for the last %s' % (s['title'], s['title'])) dbc.close() db.close() html = { 'ar': '
'.join(__ar_files), 'ch': '
'.join(__ch_files), } f = open(os.path.join(settings.destdir, 'index.html'), 'w') f.write(__HTML_TEMPLATE % html) f.close() bley-2.0.0/bley.conf.example0000644000175000017500000000311212407262723016456 0ustar evgenievgeni00000000000000[bley] # On which IP and port should the daemon listen? listen_addr = 127.0.0.1 listen_port = 1337 # Where to save the PID file? pid_file = bley.pid # Where to save the log file? log_file = bley.log reject_msg = greylisted, try again later # Which database to use? [pgsql|mysql|sqlite3] # note: for sqlite3, just set dbname to the path to the database dbtype = sqlite3 dbhost = localhost dbuser = bley dbpass = bley dbname = bley.db #dbport = 5432 # Static whitelist files whitelist_recipients_file = ./whitelist_recipients whitelist_clients_file = ./whitelist_clients # Which DNSBLs and DNSWLs to use? dnsbls = ix.dnsbl.manitu.net, dnsbl.ahbl.org, dnsbl.sorbs.net dnswls = list.dnswl.org # Whitelist after dnswl_threshold hits. dnswl_threshold = 1 # Greylist after dnsbl_threshold hits. dnsbl_threshold = 1 # Greylist after rfc_threshold hits in the RFC checks. rfc_threshold = 2 # Wait greylist_period minutes before accepting greyed sender. greylist_period = 29 # Max wait greylist_max minutes before accepting greyed sender. # (Accept all senders after 12h.) greylist_max = 720 # Add greylist_penalty minutes for every connection before greylist_period. greylist_penalty= 10 # Purge good entries from the database after purge_days inactivities. purge_days = 40 # Purge bad entries from the database after purge_bad_days inactivities. purge_bad_days = 10 # Use SPF? use_spf = 1 use_spf_guess = 0 # Use Exim workaround (close the socket after an action has been sent)? exim_workaround = 0 # How long should the cache entries be valid (in minutes)? cache_valid = 60 [bleygraph] destdir = stats bley-2.0.0/whitelist_recipients.example0000644000175000017500000000032712407262723021045 0ustar evgenievgeni00000000000000# Static list of recipients that do NOT want greylisting # Entries are compatible with postgrey, ie # user@ # example.com # /regexp/ # NB example.com includes all subdomains of example.com, too postmaster@ abuse@ bley-2.0.0/bleyhelpers.py0000644000175000017500000001261712407263667016134 0ustar evgenievgeni00000000000000# Copyright (c) 2009-2014 Evgeni Golov # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # 3. Neither the name of the University nor the names of its contributors # may be used to endorse or promote products derived from this software # without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. import spf import re import ipaddr try: import publicsuffix except ImportError: publicsuffix = None publicsuffixlist = None __dyn_host = re.compile('(\.bb\.|broadband|cable|dial|dip|dsl|dyn|gprs|pool|ppp|umts|wimax|wwan|[0-9]{1,3}[.-][0-9]{1,3}[.-][0-9]{1,3}[.-][0-9]{1,3})', re.I) __static_host = re.compile('(colo|dedi|hosting|mail|mx[^$]|smtp|static)', re.I) def reverse_ip(ip): '''Returns the IP address in reversed notation (A.B.C.D -> D.C.B.A). @type ip: string @param ip: the IP address to reverse @rtype: string @return: the reversed IP address ''' ip = ipaddr.IPAddress(ip) if ip.version == 4: a = str(ip.exploded).split('.') a.reverse() return '.'.join(a) else: a = list(str(ip.exploded).replace(':', '')) a.reverse() return '.'.join(a) def domain_from_host(host): '''Return the domain part of a host. @type host: string @param host: the host to extract the domain from @rtype: string @return: the extracted domain ''' if publicsuffix: global publicsuffixlist if publicsuffixlist is None: publicsuffixlist = publicsuffix.PublicSuffixList() domain = publicsuffixlist.get_public_suffix(host) else: d = host.split('.') if len(d) > 1: domain = '%s.%s' % (d[-2], d[-1]) else: domain = host return domain def check_dyn_host(host): '''Check the host for being a dynamic/dialup one. @type host: string @param host: the host to check @rtype: int @return: 0 if host is not dynamic, 1 if it is ''' if __static_host.search(host): return 0 if __dyn_host.search(host): return 1 return 0 def check_helo(params): '''Check the HELO for being RFC 5321 complaint. Returns 0 when the HELO match the reverse DNS. Returns 1 when the domain in the HELO match the domain of the reverse DNS or when the HELO is the IP address. Returns 2 else. @type params: dict @param params: the params from Postfix @rtype: int @return: the score of the HELO ''' if (params['helo_name'].startswith('[') and params['helo_name'].endswith(']')): try: params['helo_name'] = '[%s]' % ipaddr.IPAddress(params['helo_name'].strip('[]')).exploded except: pass if (params['client_name'] != 'unknown' and params['client_name'] == params['helo_name']): score = 0 elif (domain_from_host(params['helo_name']) == domain_from_host(params['client_name']) or params['helo_name'] == '[%s]' % params['client_address']): score = 1 else: score = 2 return score def check_spf(params, guess): '''Check the SPF record of the sending address. Try Best Guess when the domain has no SPF record. Returns 1 when the SPF result is in ['fail', 'softfail'], returns 0 else. @type params: dict @param params: the params from Postfix @type guess: int @param guess: 1 if use 'best guess', 0 if not @rtype: int @return: 1 if bad SPF, 0 else ''' score = 0 try: s = spf.query(params['client_address'], params['sender'], params['helo_name']) r = s.check() if r[0] in ['fail', 'softfail']: score = 1 elif r[0] in ['pass']: score = 0 elif guess > 0 and r[0] in ['none']: r = s.best_guess() if r[0] in ['fail', 'softfail']: score = 1 elif r[0] in ['pass']: score = 0 except: # DNS Errors, yay... print 'something went wrong in check_spf()' return score def adapt_query_for_sqlite3(query): # WARNING: This is a hack to convert the usual pyformat strings # to named ones used by sqlite3 return query.replace('%(', ':').replace(')s', '') bley-2.0.0/requirements.txt0000644000175000017500000000027412407262723016514 0ustar evgenievgeni00000000000000# We actually just need Twisted-Core>=8.1.0 and Twisted-Names>=8.1.0, # but pip is stupid and fails to do so. Let's install Twisted instead. Twisted>=8.1.0 pydns pyspf publicsuffix ipaddr bley-2.0.0/bley.py0000644000175000017500000004656512407263667014562 0ustar evgenievgeni00000000000000# Copyright (c) 2009-2014 Evgeni Golov # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # 3. Neither the name of the University nor the names of its contributors # may be used to endorse or promote products derived from this software # without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. from twisted.internet.protocol import Factory from twisted.names import client from twisted.internet import defer from twisted.internet import reactor import datetime import logging import re import bleyhelpers from postfix import PostfixPolicy from time import sleep import ipaddr logger = logging.getLogger('bley') regexp_type = type(re.compile('')) class BleyPolicy(PostfixPolicy): '''Implementation of intelligent greylisting based on `PostfixPolicy`''' db = None dbc = None required_params = ['sender', 'recipient', 'client_address', 'client_name', 'helo_name'] @defer.inlineCallbacks def check_policy(self): '''Check the incoming mail based on our policy and tell Postfix about our decision. The policy works as follows: 1. Accept if recipient=(postmaster|abuse) 2. Check local DB for an existing entry 3. When not found, check 1. DNSWLs (accept if found) 2. DNSBLs (reject if found) 3. HELO/dyn_host/sender_eq_recipient (reject if over threshold) 4. SPF (reject if over threshold) 5. Accept if not yet rejected 4. When found 1. Whitelisted: accept 2. Greylisted and waited: accept 3. Greylisted and not waited: reject @type postfix_params: dict @param postfix_params: parameters we got from Postfix ''' if not self.db: self.db = self.factory.settings.db try: self.dbc = self.db.cursor() except: self.safe_reconnect() check_results = {'DNSWL': 0, 'DNSBL': 0, 'HELO': 0, 'DYN': 0, 'DB': -1, 'SPF': 0, 'S_EQ_R': 0, 'WHITELISTED': 0, 'CACHE': 0} action = 'DUNNO' self.params['now'] = datetime.datetime.now() postfix_params = self.params # Strip everything after a + in the localpart, # usefull for mailinglists etc if postfix_params['sender'].find('+') != -1: postfix_params['sender'] = postfix_params['sender'][:postfix_params['sender'].find('+')] + postfix_params['sender'][postfix_params['sender'].find('@'):] if postfix_params['recipient'].find('+') != -1: postfix_params['recipient'] = postfix_params['recipient'][:postfix_params['recipient'].find('+')] + postfix_params['recipient'][postfix_params['recipient'].find('@'):] if postfix_params['client_address'] in self.factory.bad_cache.keys(): delta = datetime.datetime.now() - self.factory.bad_cache[postfix_params['client_address']] if delta < datetime.timedelta(0, self.factory.settings.cache_valid, 0): action = 'DEFER_IF_PERMIT %s (cached result)' % self.factory.settings.reject_msg check_results['CACHE'] = 1 if self.factory.settings.verbose: logger.info('decided CACHED action=%s, checks: %s, postfix: %s' % (action, check_results, postfix_params)) else: logger.info('decided CACHED action=%s, from=%s, to=%s' % (action, postfix_params['sender'], postfix_params['recipient'])) self.send_action(action) self.factory.log_action(postfix_params, action, check_results) return else: del self.factory.bad_cache[postfix_params['client_address']] if postfix_params['client_address'] in self.factory.good_cache.keys(): delta = datetime.datetime.now() - self.factory.good_cache[postfix_params['client_address']] if delta < datetime.timedelta(0, self.factory.settings.cache_valid, 0): action = 'DUNNO' check_results['CACHE'] = 1 if self.factory.settings.verbose: logger.info('decided CACHED action=%s, checks: %s, postfix: %s' % (action, check_results, postfix_params)) else: logger.info('decided CACHED action=%s, from=%s, to=%s' % (action, postfix_params['sender'], postfix_params['recipient'])) self.send_action(action) self.factory.log_action(postfix_params, action, check_results) return else: del self.factory.good_cache[postfix_params['client_address']] status = self.check_local_db(postfix_params) # -1 : not found # 0 : regular host, not in black, not in white, let it go # 1 : regular host, but in white, let it go, dont check EHLO # 2 : regular host, but in black, lets grey for now if self.check_whitelist(postfix_params['recipient'].lower(), self.factory.settings.whitelist_recipients): action = 'DUNNO' check_results['WHITELISTED'] = 1 elif self.check_whitelist(postfix_params['client_name'].lower(), self.factory.settings.whitelist_clients): action = 'DUNNO' check_results['WHITELISTED'] = 1 elif self.check_whitelist_ip(postfix_params['client_address'].lower(), self.factory.settings.whitelist_clients_ip): action = 'DUNNO' check_results['WHITELISTED'] = 1 elif status == -1: # not found in local db... check_results['DNSWL'] = yield self.check_dnswls(postfix_params['client_address'], self.factory.settings.dnswl_threshold) if check_results['DNSWL'] >= self.factory.settings.dnswl_threshold: new_status = 1 else: check_results['DNSBL'] = yield self.check_dnsbls(postfix_params['client_address'], self.factory.settings.dnsbl_threshold) check_results['HELO'] = bleyhelpers.check_helo(postfix_params) check_results['DYN'] = bleyhelpers.check_dyn_host(postfix_params['client_name']) # check_sender_eq_recipient: if postfix_params['sender'] == postfix_params['recipient']: check_results['S_EQ_R'] = 1 if self.factory.settings.use_spf and check_results['DNSBL'] < self.factory.settings.dnsbl_threshold and check_results['HELO'] + check_results['DYN'] + check_results['S_EQ_R'] < self.factory.settings.rfc_threshold: check_results['SPF'] = bleyhelpers.check_spf(postfix_params, self.factory.settings.use_spf_guess) else: check_results['SPF'] = 0 if check_results['DNSBL'] >= self.factory.settings.dnsbl_threshold or check_results['HELO'] + check_results['DYN'] + check_results['SPF'] + check_results['S_EQ_R'] >= self.factory.settings.rfc_threshold: new_status = 2 action = 'DEFER_IF_PERMIT %s' % self.factory.settings.reject_msg self.factory.bad_cache[postfix_params['client_address']] = datetime.datetime.now() else: new_status = 0 self.factory.good_cache[postfix_params['client_address']] = datetime.datetime.now() query = "INSERT INTO bley_status (ip, status, last_action, sender, recipient) VALUES(%(client_address)s, %(new_status)s, %(now)s, %(sender)s, %(recipient)s)" postfix_params['new_status'] = new_status try: self.safe_execute(query, postfix_params) except: # the other thread already commited while we checked, ignore pass elif status[0] >= 2: # found to be greyed check_results['DB'] = status[0] delta = datetime.datetime.now() - status[1] if delta > self.factory.settings.greylist_period + status[2] * self.factory.settings.greylist_penalty or delta > self.factory.settings.greylist_max: action = 'DUNNO' query = "UPDATE bley_status SET status=0, last_action=%(now)s WHERE ip=%(client_address)s AND sender=%(sender)s AND recipient=%(recipient)s" self.factory.good_cache[postfix_params['client_address']] = datetime.datetime.now() else: action = 'DEFER_IF_PERMIT %s' % self.factory.settings.reject_msg query = "UPDATE bley_status SET fail_count=fail_count+1 WHERE ip=%(client_address)s AND sender=%(sender)s AND recipient=%(recipient)s" self.factory.bad_cache[postfix_params['client_address']] = datetime.datetime.now() self.safe_execute(query, postfix_params) else: # found to be clean check_results['DB'] = status[0] action = 'DUNNO' query = "UPDATE bley_status SET last_action=%(now)s WHERE ip=%(client_address)s AND sender=%(sender)s AND recipient=%(recipient)s" self.safe_execute(query, postfix_params) self.factory.good_cache[postfix_params['client_address']] = datetime.datetime.now() if self.factory.settings.verbose: logger.info('decided action=%s, checks: %s, postfix: %s' % (action, check_results, postfix_params)) else: logger.info('decided action=%s, from=%s, to=%s' % (action, postfix_params['sender'], postfix_params['recipient'])) self.factory.log_action(postfix_params, action, check_results) self.send_action(action) def check_whitelist(self, email, whitelist): '''Check the arg email against a whitelist The whitelist consists of strings or compiled regular expressions from the whitelist file Return 1 if any of email matches the entire entry email matches the regular expression email domain (or subdomain) matches the entry (which is a domain name) email user@ part matches an entry of the form user@ ''' for entry_wl in whitelist: if isinstance(entry_wl, regexp_type): if entry_wl.search(email) is not None: logger.info('whitelisted %s due to rule %s' % (email, entry_wl.pattern)) return 1 else: # It's a regex, but it doesn't match # next whitelist item please continue # whitelist item is a string if entry_wl.endswith('@'): # user@ (any domain) match if email.startswith(entry_wl): logger.info('whitelisted %s due to rule %s' % (email, entry_wl)) return 1 else: continue if len(email) > len(entry_wl): entry_wl_length = len(entry_wl) + 1 if ('.' + entry_wl) == email[-entry_wl_length:]: # subdomain match logger.info('whitelisted %s due to rule %s' % (email, entry_wl)) return 1 elif ('@' + entry_wl) == email[-entry_wl_length:]: # @domain match logger.info('whitelisted %s due to rule %s' % (email, entry_wl)) return 1 if entry_wl == email: # Whole email match (for whitelist_recipients) # Or domain match (for whitelist_clients) logger.info('whitelisted %s due to rule %s' % (email, entry_wl)) return 1 return 0 def check_whitelist_ip(self, ipstr, whitelist_ip): '''Check the arg ipstr against a whitelist The whitelist consists of ip objects generated from the whitelist file Return 1 if ipstr is a valie ip address -- AND -- ipstr matches one of the entries in the whitelist_ip list ''' try: ip = ipaddr.IPAddress(ipstr) except (ValueError): return 0 for net in whitelist_ip: if ip in net: logger.info('whitelisted %s because it is in subnet %s' % (str(ip), str(net))) return 1 return 0 def check_local_db(self, postfix_params): '''Check the sender for being in the local database. Queries the local SQL database for the (ip,sender,recipient) tuple. @type postfix_params: dict @param postfix_params: parameters we got from Postfix @rtype: list @return: the result from SQL if any ''' query = """SELECT status,last_action,fail_count,sender,recipient FROM bley_status WHERE ip=%(client_address)s AND sender=%(sender)s AND recipient=%(recipient)s ORDER BY status ASC LIMIT 1""" try: self.safe_execute(query, postfix_params) result = self.dbc.fetchone() except: result = None logger.info('check_local_db failed. sending unknown.') if not result: return -1 else: return result @defer.inlineCallbacks def check_dnswls(self, ip, max): '''Check the IP address in DNSWLs. @type ip: string @param ip: the IP to check @type max: int @param max: stop after max hits @rtype: int @return: in how many DNSWLs did we find ip? ''' result = 0 for l in self.factory.settings.dnswls: try: d = yield self.check_dnsl(l, ip) result += 1 except Exception: pass if result >= max: break defer.returnValue(result) @defer.inlineCallbacks def check_dnsbls(self, ip, max): '''Check the IP address in DNSBLs. @type ip: string @param ip: the IP to check @type max: int @param max: stop after max hits @rtype: int @return: in how many DNSBLs did we find ip? ''' result = 0 for l in self.factory.settings.dnsbls: try: d = yield self.check_dnsl(l, ip) result += 1 except Exception: pass if result >= max: break defer.returnValue(result) def check_dnsl(self, lst, ip): '''Check the IP address in a DNS list. @type ip: string @param ip: the IP to check @type lst: sting @param lst: the DNS list to check in @rtype: C{Deferred} @return: twisted.names.client resolver ''' rip = bleyhelpers.reverse_ip(ip) lookup = '%s.%s' % (rip, lst) d = client.lookupAddress(lookup) return d def safe_execute(self, query, params=None): if self.factory.settings.dbtype == 'sqlite3': query = bleyhelpers.adapt_query_for_sqlite3(query) try: self.dbc.execute(query, params) self.db.commit() except self.factory.settings.database.OperationalError: self.safe_reconnect() if self.db: self.dbc.execute(query, params) self.db.commit() else: logger.info('Could not reconnect to the database, exiting.') reactor.stop() def safe_reconnect(self): logger.info('Reconnecting to the database after an error.') try: self.db.close() except: pass self.db = None retries = 0 while not self.db and retries < 30: try: self.factory.settings.db = self.db = self.factory.settings.database.connect(**self.factory.settings.dbsettings) self.dbc = self.db.cursor() except self.factory.settings.database.OperationalError: self.db = None retries += 1 sleep(1) class BleyPolicyFactory(Factory): protocol = BleyPolicy def __init__(self, settings): self.settings = settings self.good_cache = {} self.bad_cache = {} self.actionlog = [] self.exim_workaround = settings.exim_workaround reactor.callLater(30 * 60, self.dump_log) reactor.addSystemEventTrigger('before', 'shutdown', self.dump_log) def log_action(self, postfix_params, action, check_results): now = datetime.datetime.now() action = action.split(' ')[0] logline = {'time': str(now), 'ip': postfix_params['client_address'], 'from': postfix_params['sender'], 'to': postfix_params['recipient'], 'action': action} logline.update(check_results) self.actionlog.append(logline) def dump_log(self): query = '''INSERT INTO bley_log (logtime, ip, sender, recipient, action, check_dnswl, check_dnsbl, check_helo, check_dyn, check_db, check_spf, check_s_eq_r, check_postmaster, check_cache) VALUES(%(time)s, %(ip)s, %(from)s, %(to)s, %(action)s, %(DNSWL)s, %(DNSBL)s, %(HELO)s, %(DYN)s, %(DB)s, %(SPF)s, %(S_EQ_R)s, %(WHITELISTED)s, %(CACHE)s)''' if self.settings.dbtype == 'sqlite3': query = bleyhelpers.adapt_query_for_sqlite3(query) try: db = self.settings.database.connect(**self.settings.dbsettings) dbc = db.cursor() i = len(self.actionlog) while i: logline = self.actionlog.pop(0) dbc.execute(query, logline) i -= 1 db.commit() dbc.close() db.close() except self.settings.database.DatabaseError, e: logger.warn('SQL error: %s' % e) reactor.callLater(30 * 60, self.dump_log) bley-2.0.0/epydoc.conf0000644000175000017500000000102512407262723015355 0ustar evgenievgeni00000000000000[epydoc] # Epydoc section marker (required by ConfigParser) # Information about the project. name: bley url: http://bley.mx # The list of modules to document. Modules can be named using # dotted names, module filenames, or package directory names. # This option may be repeated. modules: bleyhelpers.py, bley.py, postfix.py # Write html output to the directory "apidocs" output: html target: docs/ # Include all automatically generated graphs. These graphs are # generated using Graphviz dot. graph: all dotpath: /usr/bin/dot bley-2.0.0/setup.cfg0000644000175000017500000000007312421233663015043 0ustar evgenievgeni00000000000000[egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 bley-2.0.0/bley.service0000644000175000017500000000055112407262723015543 0ustar evgenievgeni00000000000000[Unit] Description=Bley greylisting daemon Wants=postgresql.service mysql.service After=network.target remote-fs.target nss-lookup.target postgresql.service mysql.service [Service] Type=forking User=bley RuntimeDirectory=bley ExecStart=/usr/bin/bley -c /etc/bley/bley.conf -p /run/bley/bley.pid PIDFile=/run/bley/bley.pid [Install] WantedBy=multi-user.target bley-2.0.0/bley.egg-info/0000755000175000017500000000000012421233663015647 5ustar evgenievgeni00000000000000bley-2.0.0/bley.egg-info/PKG-INFO0000644000175000017500000000034512421233663016746 0ustar evgenievgeni00000000000000Metadata-Version: 1.0 Name: bley Version: 2.0.0 Summary: intelligent greylisting daemon for postfix Home-page: http://bley.mx Author: Evgeni Golov Author-email: evgeni@golov.de License: BSD Description: UNKNOWN Platform: UNKNOWN bley-2.0.0/bley.egg-info/not-zip-safe0000644000175000017500000000000112421233663020075 0ustar evgenievgeni00000000000000 bley-2.0.0/bley.egg-info/SOURCES.txt0000644000175000017500000000107012421233663017531 0ustar evgenievgeni00000000000000CHANGELOG.md MANIFEST.in Makefile README.md bley bley.1 bley.conf.example bley.logcheck bley.py bley.service bleygraph bleygraph.1 bleyhelpers.py epydoc.conf postfix.py requirements.txt setup.py whitelist_clients.example whitelist_recipients.example bley.egg-info/PKG-INFO bley.egg-info/SOURCES.txt bley.egg-info/dependency_links.txt bley.egg-info/not-zip-safe bley.egg-info/requires.txt bley.egg-info/top_level.txt test/__init__.py test/bley_test.conf.in test/test_bley.py test/test_bleyhelpers.py test/test_postfix.py test/whitelist_clients test/whitelist_recipientsbley-2.0.0/bley.egg-info/dependency_links.txt0000644000175000017500000000000112421233663021715 0ustar evgenievgeni00000000000000 bley-2.0.0/bley.egg-info/top_level.txt0000644000175000017500000000003112421233663020373 0ustar evgenievgeni00000000000000bleyhelpers postfix bley bley-2.0.0/bley.egg-info/requires.txt0000644000175000017500000000020212421233663020241 0ustar evgenievgeni00000000000000Twisted>=8.1.0 pyspf ipaddr [MySQL backend] MySQL-python [PostgreSQL backend] psycopg2 [publicsuffix.org support] publicsuffix bley-2.0.0/setup.py0000755000175000017500000000274412417767474014767 0ustar evgenievgeni00000000000000from setuptools import setup import subprocess try: from subprocess import check_output except: from subprocess import Popen, PIPE def check_output(*popenargs, **kwargs): return Popen(*popenargs, stdout=PIPE).communicate()[0] subprocess.check_output = check_output def systemd_unit_path(): try: command = ["pkg-config", "--variable=systemdsystemunitdir", "systemd"] path = subprocess.check_output(command, stderr=subprocess.STDOUT) return path.replace('\n', '') except (subprocess.CalledProcessError, OSError): return "/lib/systemd/system" setup( name="bley", version="2.0.0", description="intelligent greylisting daemon for postfix", author="Evgeni Golov", author_email="evgeni@golov.de", url="http://bley.mx", license="BSD", py_modules=['bley', 'bleyhelpers', 'postfix'], scripts=['bley', 'bleygraph'], zip_safe=False, install_requires=['Twisted>=8.1.0', 'pyspf', 'ipaddr'], extras_require={ 'PostgreSQL backend': ['psycopg2'], 'MySQL backend': ['MySQL-python'], 'publicsuffix.org support': ['publicsuffix'], }, data_files=[ ('/etc/bley', ['bley.conf.example', 'whitelist_recipients.example', 'whitelist_clients.example']), ('/usr/share/man/man1', ['bley.1', 'bleygraph.1']), ('/etc/logcheck/ignore.d.server/', ['bley.logcheck']), (systemd_unit_path(), ['bley.service']) ] ) bley-2.0.0/whitelist_clients.example0000644000175000017500000000055412407262723020343 0ustar evgenievgeni00000000000000# Static list of recipients that do NOT want greylisting # Entries are compatible with postgrey, ie # 192.0.2.0/24 # example.com # /regexp/ # NB example.com includes all subdomains of example.com, too # regexp is applied to the client name, not the IP address # client = remote mailserver, not sender's email address # 10.0.0.0/8 # 172.16.0.0/12 # 192.168.0.0/16 bley-2.0.0/bley.logcheck0000644000175000017500000000032312407262723015657 0ustar evgenievgeni00000000000000^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ bley\[[[:digit:]]+\]: decided action=(DUNNO|DEFER_IF_PERMIT) ^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ bley\[[[:digit:]]+\]: decided CACHED action=(DUNNO|DEFER_IF_PERMIT)