mythnettv-release-7/0000755000175000017500000000000011135202305013646 5ustar mikalmikalmythnettv-release-7/ChangeLog0000644000175000017500000002736011135202764015441 0ustar mikalmikal ------ Unsorted ChangeLog entries above this line ------ Release 6 -> Release 7: Better testing: - r232: Simple script to create the unit testing database New features: - r210: Allow users to override the default bittorrent upload rate - r212: Provide more sync information when running in verbose mode - r216: Add the markunread command - r218: Make only one download attempt for a given show per 24 hour period - r220: Change download interval to one hour, update last attempt time more frequently - r222: Add --oldestfirst and --newestfirst flags, which alter NextDownloads behaviour - r224: Add flag which forces downloads to run, even if they've failed recently - r226: Don't mark a bittorrent download as complete just because the torrent file has been completely downloaded - r234: Use reasonable defaults for mysql configuration if no config file can be found - r236: Use reasonable defaults for mysql configuration if no config file can be found - r238: Use reasonable defaults for mysql configuration if no config file can be found Bug fixes: - r206: Make the output of "video.py length" more friendly - r208: Release man pages as well - r214: Fix a bug introduced in r210 which broke bittorrent downloads - r228: Fix a typo in the recordings_tool help text - r230: Fix a typo in the recordings_tool help text - r240: Reduce size of proxy name in the usage table to deal with key length limitations in MySQL - r242: The schema for the channel table changed in MythTV 0.21 - r244: Fix typo - r246: Suppress repeated warnings of DB default use - r248: Bug fixes for linux.conf.au 2009 talk - r250: Include test runner script in release, prepare ChangeLog for release Release 5 -> Release 6: Better testing: - r114: Unit tests for video.py - r148: Updated unit test now that we create more tables - r151: Start work on syndication unit tests - r153: Unit test for bad syndication dates, as well as more flexiblity in db connections needed for this unit test - r157: Nicer unit test failure output, and refactor GetVideoDir() into the utility module - r181: Slightly improve syndication unit tests - r183: Add flag parsing to unit tests, and fix some more flag name changes Better documentation: - r117: Added a man page for video.py -- I'm not 100% happy with its name though Bug fixes: - r114: Fixed a bug where the new filename for video transcode could be the same as the input filename, resulting in video corruption. This was found with one of the new video.py unit tests - r116: The logic for the --prompt flag was the wrong way around. Fixed. - r119: Nicer download status messages - r121: Handle 404s in feed updates better - r129: Slight tweak to SVN submit script - r131: More accurate tracking of proxy usage (update during download, instead of just at the end) - r137: Proxy budget being exceeded doesn't count as a failed download attempt - r143: Subscribe now renables inactive subscriptions - r146: Add support to decimals to utility byte printer, fix a bug in the check for video directories - r155: Have users send problems to the mailing list, instead of me personally - r161: Don't throw exceptions for the videodir command line - r167, 169: Display friendly sizes in records_tool output - r171: Move verbose update arg into a flag - r173: Add "-vo null" to mplayer invocation per Ryan Lutz. This improves support on machines without X, and speeds up the identify operation - r175: Import patch from Thomas Mashos which fixes subscription removal, started work on syndication unit test improvements - r177: Fix character escaping bug in show subtitles during import - r179: Renamed --datadirdefault to --datadir. If set this will now change your data directory, regardless of if there was a previous value set. - r190: Recording_tool now prompts for deletes - r192: Improved explainvideodir output - r194: Don't crash in explainvideodir if there is no video directory - r197: Order results by subtitle in recordings_tool output New features: - r115: Upgraded schema to version 15 to support http_proxies for subscriptions. Added http_proxy command line, which allows you to use HTTP proxies for specified URLs. Moved HTTP accesses to use the proxy code. - r127: Bump schema to v17, and add proxy use tracking including the "proxyusage" command - r133: Allow daily budgets for proxy usage - r115: Provide a user agent for HTTP requests, instead of just Python-urllib/1.17 - r117: Users will now be prompted to subscribe to an annoucements video feed. This will happen exactly once. This behaviour may be disabled with the --nopromptforannounce command line flag. - r125: Add a full info dump command to video.py's command line interface - r139: Bump schema to 19, and implement categories with the "category" command - r141: Implement recording group support, and clarify category support - r151: Implement a helper (recordings_tool) for handling the MythTV recordings table, this is useful for testing. - r159: Add videodir and explainvideodir debugging commands, and update man page - r163: Add title list feature to recordings_tool - r165: Include recording count in title list - r185: Add the resetattempts command Development changes: - r123: Added a submit script to automate putting the revision number into the ChangeLog - r135: Tweak to new ChangeLog auto logging formatting Release 4 -> Release 5: - There is now a users mailing list at http://lists.stillhq.com/listinfo.cgi/mythnettv-stillhq.com - Moved to a public SVN server at http://www.stillhq.com/mythtv/mythnettv/svn/ - Added the 'justone' syntax to the download command - Another try at using gflags. This means that all the command lines have changed slightly. - Moved the core of MythTV out of the user interface file. - Started writing unit tests - Changed user output code so that it doesn't insist on writing to stdout. You can now write to other file descriptors, which makes things like unit tests much easier to write. - Added video/msvideo to the enclosure whitelist - Added HTTP download progress information - Added a flag which turns off the prompts for markread (--noprompt) - Patches from Thomas Mashos - Search ~/.mythtv/mysql.txt, /usr/share/mythtv/mysql.txt and /etc/mythtv/mysql.txt in that order for MySQL connection information - A manpage - setup.py - video.py now has a simple command line interface to let you query it - Fix update of inactive programs bug per http://ubuntuforums.org/showpost.php?p=5580005&postcount=4 - Better DB error handling - Included a COPYING file with the right version of the GPL (it was missing before) - Fixed a bug where programs would be downloaded more than once (found with a unit test!) - Started raising exceptions instead of just sys.exit(1). This should make life easier for user interfaces in the future - Default to using storage groups for storing recordings before falling back to the RecordFilePrefix. This makes the behaviour: use a storage group named "MythNetTV" if it exists; use the default storage group if it exists; use the value of RecordFilePrefix. - Transcode avc1 videos, because some need it - Force ASCII encoding of title, subtitle, and all fields in the database to get around feeds which use unicode which python / MySQL can't store correctly - If there is only one attachment to an item, and its not in our whitelist of video formats, then warn the user that you're assuming its a video file and then add it to the todo list - Slight tweak to the signature of video.MythNetTvVideo.Transcode() - Fix buf in RepairMissingDates which caused it to consistently crash - Fix typo in date warning code - Better handling of videos where the length of the video cannot be determined by mplayer Release 3 -> Release 4: - Removed gflags because people found it hard to find - Bug fix patch from David Linville applied - Fixed broken imports caused by refactoring - Transcode not needed for avc1 and m4v - Another bug fix patch from David Linville applied - Store filesize in the db - Removed some namespace imports I am not a fan of - Made verbosity optional for --update - Small code cleanups - Let the user know of repeated attempts to download a program - Documented the need for bittornado for bittorrent to work - Abandon downloading after 3 failed attempts (you can configure the number) - Detect stuck bittorrent downloads Release 2 -> Release 3: - Started work on an RSS exporter for MythTV recordings - DX50 doesn't need transcode - Tweaked supported video mime types so "Tikibar TV" and "Ask a ninja" work - First cut of Bittorrent support - Schema upgrades - Archive recordings as well as importing them - Improved --list output - Subtitle restrictions on download as well - Make subscriptions inactive instead of deleting them (for unsubscribe) - Better filename safening - More markread options - Refactored code to be more sane - Don't archive things imported from the archive location - Bulk import (--importmanylocal) THINGS I NEED TO REIMPLEMENT BECAUSE OF MY ACCIDENTAL DELETE: - Give up on downloads after 5 failed attempts Release 1 -> Release 2: - Started working harder to ensure video filenames are unique once imported into MythTV, I now prepend video files with epoch seconds at time of import - DIVX (not divx) doesn't need transcode, added. Check for the existance of the video directory, and return an error if it needs creating. - Handle storage groups, check for the existance of the data directory - Updated docs - Complained about the poor state of SWF support - WMV support - FLV support Beta 4 -> Release 1: - Inital work on transcoding smaller files to something else - Better error checking for MySQL configuration and accessibility - Fixed bug where MythNetTV was unhappy if there were no channels configured in the MySQL database - Found, and hopefully fixed, a bug where program dates were not always being tracked correctly. Also added a check to make sure this doesn't happen again - Added --markread, which lets you mark old shows as already downloaded Beta 3 -> Beta 4: - Made --update more terse - Made --update more liberal about what it considers to be a video, specifically added video/quicktime and text/html (to work around the rather broken http://www.mobuzztv.com/uk/rss/quicktime - Added a filter option to --download to constrain it to only downloading shows with a specific title - Added transcoding for mov files - If there is a straggling import, and it causes as error, just skip it and mark it as imported - Don't reset transferred data statistic when we start downloading a show again - Started implementing moniker support for downloads - Unsupported monikers will result in an attempt to download the URL using mplayer (which works for RTSP and MMS at the least) Beta 2 -> Beta 3: - Renamed to mythnettv - todoremote bug fixed (bad column name) - statistics bug fixed (no transfer stats caused crash) - It is now possible to do a --todoremote, --importremote, or --importlocal and provide all the needed information on the command line Beta 1 -> Beta 2: - Now 50% of downloaded programs will be from the oldest queued, and 50% are the newest queued - Added --importremote, --todoremote, and --importlocal - Implemented transcoding via mencoder. It's quite possible it'll encounter a format it doesn't know about. Please let me know if that happens to you. - Added --subscribe, --unsubscribe, --list and --update - Added --statistics, --log, and --nextdownload mythnettv-release-7/database.py0000644000175000017500000005242611135202764016006 0ustar mikalmikal#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2006, 2007, 2008 # Released under the terms of the GNU GPL v2 import datetime import MySQLdb import os import sys import traceback import unicodedata import gflags FLAGS = gflags.FLAGS gflags.DEFINE_string('db_host', '', 'The name of the host the MySQL database is on, ' 'don\'t define if you want to parse mysql.txt ' 'instead') gflags.DEFINE_string('db_user', '', 'The name of the user to connect to the database with, ' 'don\'t define if you want to parse mysql.txt ' 'instead') gflags.DEFINE_string('db_password', '', 'The password for the database user, ' 'don\'t define if you want to parse mysql.txt ' 'instead') gflags.DEFINE_string('db_name', '', 'The name of the database which MythNetTV uses, ' 'don\'t define if you want to parse mysql.txt ' 'instead') gflags.DEFINE_boolean('db_debugging', False, 'Output debugging messages for the database') CURRENT_SCHEMA='21' HAVE_WARNED_OF_DEFAULTS = False class FormatException(Exception): """ FormatException -- Used for reporting failures for format DB values """ def Normalize(value): normalized = unicodedata.normalize('NFKD', unicode(value)) normalized = normalized.encode('ascii', 'ignore') return normalized class MythNetTvDatabase: """MythNetTvDatabase -- handle all MySQL details""" def __init__(self, dbname=None, dbuser=None, dbpassword=None, dbhost=None): self.OpenConnection(dbname=dbname, dbuser=dbuser, dbpassword=dbpassword, dbhost=dbhost) self.CheckSchema() self.CleanLog() self.RepairMissingDates() def OpenConnection(self, dbname=None, dbuser=None, dbpassword=None, dbhost=None): """OpenConnection -- parse the MythTV config file and open a connection to the MySQL database""" global HAVE_WARNED_OF_DEFAULTS if dbname: # This override makes writing unit tests simpler db_name = dbname else: db_name = FLAGS.db_name if dbuser: user = dbuser else: user = FLAGS.db_user if dbpassword: password = dbpassword else: password = FLAGS.db_password if dbhost: host = dbhost else: host = FLAGS.db_host if not host or not user or not password or not db_name: # Load the text configuration file self.config_values = {} home = os.environ.get('HOME') if os.path.exists(home + '/.mythtv/mysql.txt'): dbinfo = home + '/.mythtv/mysql.txt' elif os.path.exists('/usr/share/mythtv/mysql.txt'): dbinfo = '/usr/share/mythtv/mysql.txt' else: dbinfo = '/etc/mythtv/mysql.txt' try: config = open(dbinfo) for line in config.readlines(): if not line.startswith('#') and len(line) > 5: (key, value) = line.rstrip('\n').split('=') self.config_values[key] = value except: if not HAVE_WARNED_OF_DEFAULTS and FLAGS.db_debugging: print 'Could not parse the MySQL configuration for MythTV from', print 'any mysql.txt in the search path. Using defaults instead.' HAVE_WARNED_OF_DEFAULTS = True self.config_values['DBName'] = 'mythconverg' self.config_values['DBUserName'] = 'mythtv' self.config_values['DBPassword'] = 'mythtv' self.config_values['DBHostName'] = 'localhost' # Fill in the blanks if not host: host = self.config_values['DBHostName'] if not user: user = self.config_values['DBUserName'] if not password: password = self.config_values['DBPassword'] if not db_name: db_name = self.config_values['DBName'] # Open the DB connection try: self.db_connection = MySQLdb.connect(host = host, user = user, passwd = password, db = db_name) except Exception, e: print 'Could not connect to the MySQL server: %s' % e sys.exit(1) def TableExists(self, table): """TableExists -- check if a table exists""" cursor = self.db_connection.cursor(MySQLdb.cursors.DictCursor) try: cursor.execute('describe %s;' % table) except MySQLdb.Error, (errno, errstr): if errno == 1146: return False else: print 'Error %d: %s' %(errno, errstr) sys.exit(1) cursor.close() return True def CheckSchema(self): """CheckSchema -- ensure we're running the latest schema version""" # Check if we even have an NetTv set of tables for table in ['log', 'settings', 'programs', 'subscriptions']: if not self.TableExists('mythnettv_%s' % table): if self.TableExists('mythiptv_%s' % table): self.Log('Renaming table %s to %s;' \ %('mythiptv_%s' % table, 'mythnettv_%s' % table)) self.ExecuteSql('rename table %s to %s;' \ %('mythiptv_%s' % table, 'mythnettv_%s' % table)) else: self.CreateTable(table) # Check the schema version self.version = self.GetSetting('schema') if int(self.version) < int(CURRENT_SCHEMA): print 'Updating tables' self.UpdateTables() elif int(self.version) > int(CURRENT_SCHEMA): print 'The database schema is newer than this version of the code, ' print 'it seems like you might need to upgrade?' print print 'Current = %s' % CURRENT_SCHEMA print 'Database = %s' % self.version sys.exit(1) # Make sure we have a chanid chanid = self.GetSetting('chanid') if chanid == None: channels_row = None try: # There is none cached in the settings table channels_row = self.GetOneRow('select chanid from channel where ' 'name = "MythNetTV" or ' 'name = "MythIPTV";') except: print 'There was a MySQL error when trying to read the channels ', print 'this probably indicates an error with your MySQL installation' sys.exit(1) if channels_row: # There is one in the MythTV Channels table though chanid = self.GetSettingWithDefault('chanid', channels_row['chanid']) else: # There isn't one in the MythTV Channels table chanid_row = self.GetOneRow('select max(chanid) + 1 from channel') if chanid_row.has_key('max(chanid) + 1'): chanid = chanid_row['max(chanid) + 1'] else: chanid = 1 self.db_connection.query('insert into channel (chanid, callsign, ' 'name) values (%d, "MythNetTV", ' '"MythNetTV")' % chanid) self.Log('Created MythNetTV channel with chanid %d' % chanid) # Redo the selecting to make sure it worked channels_row = self.GetOneRow('select chanid from channel where ' 'name = "MythNetTV";') chanid = self.GetSettingWithDefault('chanid', channels_row['chanid']) # Make sure that we're using the new name for the channel, and that we # use an @ to make it display properly in the UI self.db_connection.query('update channel set callsign = "MythNetTV", ' 'name = "MythNetTV" where name = "MythIPTV";') self.db_connection.query('update channel set channum = "@" ' 'where name = "MythNetTV" and channum is null;') def RepairMissingDates(self): """RepairMissingDates -- repair programs which are missing a date""" # At some point there was a bug which resulted in there being programs # in the MythNetTV TODO list which didn't have dates associated with them. # This doesn't have any nasty side effects, but will result in incorrect # ordering in the MythTV recordings interface, and downloads happening # out of order. We try to clean the problem up here, and report to the # the user if we need to. touched_count = 0 # Try using parsed date for row in self.GetRows('select guid, parsed_date, unparsed_date from ' 'mythnettv_programs where date is null and ' 'parsed_date like "(%)";'): if row.has_key('parsed_date') and row['parsed_date'] != None: parsed = row['parsed_date'][1:-1].split(', ') parsed_ints = [] for item in parsed: parsed_ints.append(int(item)) date = datetime.datetime(*parsed_ints[0:5]) prog = program.MythNetTvProgram(self) prog.Load(row['guid']) prog.SetDate(date) prog.Store() touched_count += 1 # Otherwise, just set it to now and get on with our lives for row in self.GetRows('select guid from mythnettv_programs ' 'where date is null;'): prog = program.MythNetTvProgram(self) prog.Load(row['guid']) prog.SetDate(datetime.datetime.now()) prog.Store() touched_count += 1 if touched_count > 0: print 'During startup, I found %d programs with invalid dates. This' \ % touched_count print 'indicates a bug in MythNetTV. I think its corrected now. If' print 'this message keeps appearing, please email mythnettv@stillhq.com' print 'and us know.' print def GetSetting(self, name): """GetSetting -- get the current value of a setting""" row = self.GetOneRow('select value from mythnettv_settings where ' 'name="%s";' % name) if row == None: return None return row['value'] def GetSettingWithDefault(self, name, default): """GetSettingWithDefault -- get a setting with a default value""" cursor = self.db_connection.cursor(MySQLdb.cursors.DictCursor) cursor.execute('select value from mythnettv_settings where ' 'name="%s";' % name) if cursor.rowcount != 0: retval = cursor.fetchone() cursor.close() return retval['value'] else: self.db_connection.query('insert into mythnettv_settings (name, value) ' 'values("%s", "%s");' %(name, default)) self.Log('Settings value %s defaulted to %s' %(name, default)) return default def WriteSetting(self, name, value): """WriteSetting -- write a setting to the database""" self.WriteOneRow('mythnettv_settings', 'name', {'name': name, 'value': value}) def GetOneRow(self, sql): """GetOneRow -- get one row which matches a query""" try: cursor = self.db_connection.cursor(MySQLdb.cursors.DictCursor) cursor.execute(sql) retval = cursor.fetchone() cursor.close() if retval == None: if FLAGS.db_debugging: print 'Database: no result for %s' % sql return retval for key in retval.keys(): if retval[key] == None: del retval[key] return retval except Exception, e: print 'Database error:' traceback.print_exc() sys.exit(1) def GetRows(self, sql): """GetRows -- return a bunch of rows as an array of dictionaries""" retval = [] cursor = self.db_connection.cursor(MySQLdb.cursors.DictCursor) cursor.execute(sql) for i in range(cursor.rowcount): row = cursor.fetchone() retval.append(row) return retval def GetWaitingForImport(self): """GetWaitingForImport -- return a list of the guids waiting for import""" cursor = self.db_connection.cursor(MySQLdb.cursors.DictCursor) cursor.execute('select guid from mythnettv_programs where ' 'download_finished = "1" and imported is NULL') guids = [] while True: program = cursor.fetchone() if program == None: break guids.append(program['guid']) return guids def FormatSqlValue(self, name, value): """FormatSqlValue -- some values get escaped for SQL use""" if type(value) == datetime.datetime: return 'STR_TO_DATE("%s", "%s")' \ %(value.strftime('%a, %d %b %Y %H:%M:%S'), '''%a, %d %b %Y %H:%i:%s''') if name == 'date': return 'STR_TO_DATE("%s", "%s")' %(value, '''%a, %d %b %Y %H:%i:%s''') if type(value) == long or type(value) == int: return value if value == None: return 'NULL' try: return '"%s"' % Normalize(value).replace('"', '""') except Exception, e: raise FormatException('Could not format string value %s = %s (%s)' %(name, value, e)) def WriteOneRow(self, table, key_col, dict): """WriteOneRow -- use a dictionary to write a row to the specified table""" cursor = self.db_connection.cursor(MySQLdb.cursors.DictCursor) cursor.execute('select %s from %s where %s = "%s"' \ %(key_col, table, key_col, dict[key_col])) if cursor.rowcount > 0: self.Log('Updating %s row with %s of %s' %(table, key_col, dict[key_col])) vals = [] for col in dict: val = '%s=%s' %(col, self.FormatSqlValue(col, dict[col])) vals.append(val) sql = 'update %s set %s where %s="%s";' %(table, ','.join(vals), key_col, dict[key_col]) else: self.Log('Creating %s row with %s of %s' %(table, key_col, dict[key_col])) vals = [] for col in dict: val = self.FormatSqlValue(col, dict[col]) vals.append(val) sql = 'insert into %s (%s) values(%s);' \ %(table, ','.join(dict.keys()), ','.join(vals)) cursor.close() self.db_connection.query(sql) def GetNextLogSequenceNumber(self): """GetNextLogSequenceNumber -- ghetto lookup of the highest sequence number""" try: cursor = self.db_connection.cursor(MySQLdb.cursors.DictCursor) cursor.execute('select max(sequence) + 1 from mythnettv_log;') retval = cursor.fetchone() cursor.close() if retval['max(sequence) + 1'] == None: return 1 return retval['max(sequence) + 1'] except: return 1 def Log(self, message): """Log -- write a log message to the database""" try: new_sequence = self.GetNextLogSequenceNumber() self.db_connection.query('insert into mythnettv_log (sequence, ' 'timestamp, message) values(%d, NOW(), "%s");' \ %(new_sequence, message)) except: print 'Failed to log: %s' % message def CleanLog(self): """CleanLog -- remove all but the newest xxx log messages""" min_sequence = self.GetNextLogSequenceNumber() - \ int(self.GetSettingWithDefault('loglines', '1000')) - 1 cursor = self.db_connection.cursor(MySQLdb.cursors.DictCursor) cursor.execute('delete from mythnettv_log where sequence < %d' \ % min_sequence) if cursor.rowcount > 0: self.Log('Deleted %d log lines before sequence number %d' \ %(cursor.rowcount, min_sequence)) cursor.close() def CreateTable(self, tablename): """CreateTable -- a table has been found to be missing, create it with the current schema""" print 'Info: Creating %s table' % tablename if tablename == 'log': self.db_connection.query('create table mythnettv_log (sequence int, ' 'timestamp datetime, message text);') self.db_connection.query('insert into mythnettv_log (sequence) ' 'values(0);') elif tablename == 'settings': self.db_connection.query('create table mythnettv_settings (name text, ' 'value text);') self.db_connection.query('insert into mythnettv_settings (name, value) ' 'values("schema", 7);') elif tablename == 'programs': self.db_connection.query('create table mythnettv_programs (guid text, ' 'url text, title text, subtitle text, ' 'description text unicode, date datetime, ' 'unparsed_date text, parsed_date text, ' 'download_started int, ' 'download_finished int, ' 'imported int, transfered int, size int, ' 'filename text);') elif tablename == 'subscriptions': self.db_connection.query('create table mythnettv_subscriptions (' 'url text, title text);') else: self.Log('Error: Don\'t know how to create %s' % tablename) print 'Error: Don\'t know how to create %s' % tablename sys.exit(1) self.Log('Creating %s table' % tablename) def UpdateTables(self): """UpdateTables -- handle schema upgrades""" if self.version == '4': self.Log('Upgrading schema from 4 to 5') self.db_connection.query('alter table mythnettv_programs ' 'add column parsed_date text;') self.version = '5' if self.version == '5': # This is a deliberate noop because the new table was created during the # startup checks self.Log('Upgrading schema from 5 to 6') self.version = '6' if self.version == '6': # Another noop, because we're renaming tables self.Log('Upgrading schema from 6 to 7') self.version = '7' if self.version == '7': # Start tracking the MIME type of videos, this helps with bittorrent self.Log('Upgrading schema from 7 to 8') self.db_connection.query('alter table mythnettv_programs ' 'add column mime_type text, ' 'add column tmp_name varchar(255);') self.version = '8' if self.version == '8': self.Log('Upgrading schema from 8 to 10') self.db_connection.query('alter table mythnettv_programs ' 'add column inactive tinyint, ' 'add column attempts tinyint;') self.version = '10' if self.version == '10': self.Log('Upgrading schema from 10 to 12') self.db_connection.query('alter table mythnettv_subscriptions ' 'add column inactive tinyint, ' 'add column archive_to text;') self.version = '12' if self.version == '12': self.Log('Upgrading schema from 12 to 13') self.db_connection.query('alter table mythnettv_subscriptions ' 'drop column archive_to;') self.db_connection.query('create table mythnettv_archive ' '(title text, path text);') self.version = '13' if self.version == '13': self.Log('Upgrading schema from 13 to 14') self.db_connection.query('alter table mythnettv_programs ' 'add column failed tinyint;') self.version = '14' if self.version == '14': self.Log('Upgrading schema from 14 to 15') self.db_connection.query('create table mythnettv_proxies ' '(url text, http_proxy text);') self.version = '15' if self.version == '15': self.Log('Upgrading schema from 15 to 16') self.db_connection.query('create table mythnettv_proxy_usage ' '(day date, http_proxy text, bytes int);') self.version = '16' if self.version == '16': self.Log('Upgrading schema from 16 to 17') self.db_connection.query('alter table mythnettv_proxy_usage ' 'modify column http_proxy varchar(256);') self.db_connection.query('alter table mythnettv_proxy_usage ' 'add primary key(day, http_proxy);') self.version = '17' if self.version == '17': self.Log('Upgrading schema from 17 to 18') self.db_connection.query('alter table mythnettv_proxies ' 'add column daily_budget int;') self.version = '18' if self.version == '18': self.Log('Upgrading schema from 18 to 19') self.db_connection.query('create table mythnettv_category ' '(title text, category varchar(64));') self.version = '19' if self.version == '19': self.Log('Upgrading schema from 19 to 20') self.db_connection.query('create table mythnettv_group ' '(title text, recgroup varchar(32));') self.version = '20' if self.version == '20': self.Log('Upgrading schema from 20 to 21') self.db_connection.query('alter table mythnettv_programs ' 'add column last_attempt datetime;') self.version = '21' if self.version != CURRENT_SCHEMA: print 'Unknown schema version. This is a bug. Please report it to' print 'mythnettv@stillhq.com' sys.exit(1) self.db_connection.query('update mythnettv_settings set value = "%s" ' 'where name = "schema";' % self.version) def ExecuteSql(self, sql): """ ExecuteSql -- execute some SQL and return the number of rows affected """ cursor = self.db_connection.cursor(MySQLdb.cursors.DictCursor) try: cursor.execute(sql) except Exception, e: print 'Database error: %s' % e print ' sql = %s' % sql raise e changed = cursor.rowcount cursor.close() return changed mythnettv-release-7/gflags.py0000644000175000017500000013732311135202764015505 0ustar mikalmikal#!/usr/bin/env python # Copyright (c) 2007, Google Inc. # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are # met: # # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # * Redistributions in binary form must reproduce the above # copyright notice, this list of conditions and the following disclaimer # in the documentation and/or other materials provided with the # distribution. # * Neither the name of Google Inc. nor the names of its # contributors may be used to endorse or promote products derived from # this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # --- # Author: Chad Lester # Design and style contributions by: # Amit Patel, Bogdan Cocosel, Daniel Dulitz, Eric Tiedemann, # Eric Veach, Laurence Gonsalves, Matthew Springer # Code reorganized a bit by Craig Silverstein """ This module is used to define and parse command line flags. This module defines a *distributed* flag-definition policy: rather than an application having to define all flags in or near main(), each python module defines flags that are useful to it. When one python module imports another, it gains access to the other's flags. (This is implemented by having all modules share a common, global registry object containing all the flag information.) Flags are defined through the use of one of the DEFINE_xxx functions. The specific function used determines how the flag is parsed, checked, and optionally type-converted, when it's seen on the command line. IMPLEMENTATION: DEFINE_* creates a 'Flag' object and registers it with a 'FlagValues' object (typically the global FlagValues FLAGS, defined here). The 'FlagValues' object can scan the command line arguments and pass flag arguments to the corresponding 'Flag' objects for value-checking and type conversion. The converted flag values are available as members of the 'FlagValues' object. Code can access the flag through a FlagValues object, for instancee gflags.FLAGS.myflag. Typically, the __main__ module passes the command line arguments to gflags.FLAGS for parsing. At bottom, this module calls getopt(), so getopt functionality is supported, including short- and long-style flags, and the use of -- to terminate flags. Methods defined by the flag module will throw 'FlagsError' exceptions. The exception argument will be a human-readable string. FLAG TYPES: This is a list of the DEFINE_*'s that you can do. All flags take a name, default value, help-string, and optional 'short' name (one-letter name). Some flags have other arguments, which are described with the flag. DEFINE_string: takes any input, and interprets it as a string. DEFINE_boolean: typically does not take an argument: say --myflag to set FLAGS.myflag to true, or --nomyflag to set FLAGS.myflag to false. Alternately, you can say --myflag=true or --myflag=t or --myflag=1 or --myflag=false or --myflag=f or --myflag=0 DEFINE_float: takes an input and interprets it as a floating point number. Takes optional args lower_bound and upper_bound; if the number specified on the command line is out of range, it will raise a FlagError. DEFINE_integer: takes an input and interprets it as an integer. Takes optional args lower_bound and upper_bound as for floats. DEFINE_enum: takes a list of strings which represents legal values. If the command-line value is not in this list, raise a flag error. Otherwise, assign to FLAGS.flag as a string. DEFINE_list: Takes a comma-separated list of strings on the commandline. Stores them in a python list object. DEFINE_spaceseplist: Takes a space-separated list of strings on the commandline. Stores them in a python list object. DEFINE_multistring: The same as DEFINE_string, except the flag can be specified more than once on the commandline. The result is a python list object (list of strings), even if the flag is only on the command line once. DEFINE_multi_int: The same as DEFINE_integer, except the flag can be specified more than once on the commandline. The result is a python list object (list of ints), even if the flag is only on the command line once. SPECIAL FLAGS: There are a few flags that have special meaning: --help (or -?) prints a list of all the flags in a human-readable fashion --helpshort prints a list of all the flags in the 'main' .py file only --flagfile=foo read flags from foo. -- as in getopt(), terminates flag-processing Note on --flagfile: Flags may be loaded from text files in addition to being specified on the commandline. Any flags you don't feel like typing, throw them in a file, one flag per line, for instance: --myflag=myvalue --nomyboolean_flag You then specify your file with the special flag '--flagfile=somefile'. You CAN recursively nest flagfile= tokens OR use multiple files on the command line. Lines beginning with a single hash '#' or a double slash '//' are comments in your flagfile. Any flagfile= will be interpreted as having a relative path from the current working directory rather than from the place the file was included from: myPythonScript.py --flagfile=config/somefile.cfg If somefile.cfg includes further --flagfile= directives, these will be referenced relative to the original CWD, not from the directory the including flagfile was found in! The caveat applies to people who are including a series of nested files in a different dir than they are executing out of. Relative path names are always from CWD, not from the directory of the parent include flagfile. We do now support '~' expanded directory names. Absolute path names ALWAYS work! EXAMPLE USAGE: import gflags FLAGS = gflags.FLAGS # Flag names are globally defined! So in general, we need to be # careful to pick names that are unlikely to be used by other libraries. # If there is a conflict, we'll get an error at import time. gflags.DEFINE_string("name", "Mr. President" "NAME: your name") gflags.DEFINE_integer("age", None, "AGE: your age in years", lower_bound=0) gflags.DEFINE_boolean("debug", 0, "produces debugging output") gflags.DEFINE_enum("gender", "male", ["male", "female"], "GENDER: your gender") def main(argv): try: argv = FLAGS(argv) # parse flags except gflags.FlagsError, e: print '%s\\nUsage: %s ARGS\\n%s' % (e, sys.argv[0], FLAGS) sys.exit(1) if FLAGS.debug: print 'non-flag arguments:', argv print 'Happy Birthday', FLAGS.name if FLAGS.age is not None: print "You are a %s, who is %d years old" % (FLAGS.gender, FLAGS.age) if __name__ == '__main__': main(sys.argv) """ import getopt import os import sys # Are we running at least python 2.2? try: if tuple(sys.version_info[:3]) < (2,2,0): raise NotImplementedError("requires python 2.2.0 or later") except AttributeError: # a very old python, that lacks sys.version_info raise NotImplementedError("requires python 2.2.0 or later") # Are we running under pychecker? _RUNNING_PYCHECKER = 'pychecker.python' in sys.modules # module exceptions: class FlagsError(Exception): "The base class for all flags errors" class DuplicateFlag(FlagsError): "Thrown if there is a flag naming conflict" class IllegalFlagValue(FlagsError): "The flag command line argument is illegal" # Global variable used by expvar _exported_flags = {} def __GetModuleName(globals_dict): """Given a globals dict, find the module in which it's defined.""" for name, module in sys.modules.iteritems(): if getattr(module, '__dict__', None) is globals_dict: if name == '__main__': return sys.argv[0] return name raise AssertionError, "No module was found" def __GetCallingModule(): """Get the name of the module that's calling into this module; e.g., the module calling a DEFINE_foo... function. """ # Walk down the stack to find the first globals dict that's not ours. for depth in range(1, sys.getrecursionlimit()): if not sys._getframe(depth).f_globals is globals(): return __GetModuleName(sys._getframe(depth).f_globals) raise AssertionError, "No module was found" def _GetMainModule(): """Get the module name from which execution started.""" for depth in range(1, sys.getrecursionlimit()): try: globals_of_main = sys._getframe(depth).f_globals except ValueError: return __GetModuleName(globals_of_main) raise AssertionError, "No module was found" class FlagValues: """ Used as a registry for 'Flag' objects. A 'FlagValues' can then scan command line arguments, passing flag arguments through to the 'Flag' objects that it owns. It also provides easy access to the flag values. Typically only one 'FlagValues' object is needed by an application: gflags.FLAGS This class is heavily overloaded: 'Flag' objects are registered via __setitem__: FLAGS['longname'] = x # register a new flag The .value member of the registered 'Flag' objects can be accessed as members of this 'FlagValues' object, through __getattr__. Both the long and short name of the original 'Flag' objects can be used to access its value: FLAGS.longname # parsed flag value FLAGS.x # parsed flag value (short name) Command line arguments are scanned and passed to the registered 'Flag' objects through the __call__ method. Unparsed arguments, including argv[0] (e.g. the program name) are returned. argv = FLAGS(sys.argv) # scan command line arguments The original registered Flag objects can be retrieved through the use of the dictionary-like operator, __getitem__: x = FLAGS['longname'] # access the registered Flag object The str() operator of a 'FlagValues' object provides help for all of the registered 'Flag' objects. """ def __init__(self): # Since everything in this class is so heavily overloaded, # the only way of defining and using fields is to access __dict__ # directly. self.__dict__['__flags'] = {} self.__dict__['__flags_by_module'] = {} # A dict module -> list of flag def FlagDict(self): return self.__dict__['__flags'] def _RegisterFlagByModule(self, module_name, flag): """We keep track of which flag is defined by which module so that we can later sort the flags by module. """ flags_by_module = self.__dict__['__flags_by_module'] flags_by_module.setdefault(module_name, []).append(flag) def __setitem__(self, name, flag): """ Register a new flag variable. """ fl = self.FlagDict() if not isinstance(flag, Flag): raise IllegalFlagValue, flag if not isinstance(name, type("")): raise FlagsError, "Flag name must be a string" if len(name) == 0: raise FlagsError, "Flag name cannot be empty" # If running under pychecker, duplicate keys are likely to be defined. # Disable check for duplicate keys when pycheck'ing. if (fl.has_key(name) and not flag.allow_override and not fl[name].allow_override and not _RUNNING_PYCHECKER): raise DuplicateFlag, name short_name = flag.short_name if short_name is not None: if (fl.has_key(short_name) and not flag.allow_override and not fl[short_name].allow_override and not _RUNNING_PYCHECKER): raise DuplicateFlag, short_name fl[short_name] = flag fl[name] = flag global _exported_flags _exported_flags[name] = flag def __getitem__(self, name): """ Retrieve the flag object. """ return self.FlagDict()[name] def __getattr__(self, name): """ Retrieve the .value member of a flag object. """ fl = self.FlagDict() if not fl.has_key(name): raise AttributeError, name return fl[name].value def __setattr__(self, name, value): """ Set the .value member of a flag object. """ fl = self.FlagDict() fl[name].value = value return value def __delattr__(self, name): """ Delete a previously-defined flag from a flag object. """ fl = self.FlagDict() if not fl.has_key(name): raise AttributeError, name del fl[name] def SetDefault(self, name, value): """ Change the default value of the named flag object. """ fl = self.FlagDict() if not fl.has_key(name): raise AttributeError, name fl[name].SetDefault(value) def __contains__(self, name): """ Return True if name is a value (flag) in the dict. """ return name in self.FlagDict() has_key = __contains__ # a synonym for __contains__() def __iter__(self): return self.FlagDict().iterkeys() def __call__(self, argv): """ Searches argv for flag arguments, parses them and then sets the flag values as attributes of this FlagValues object. All unparsed arguments are returned. Flags are parsed using the GNU Program Argument Syntax Conventions, using getopt: http://www.gnu.org/software/libc/manual/html_mono/libc.html#Getopt """ # Support any sequence type that can be converted to a list argv = list(argv) shortopts = "" longopts = [] fl = self.FlagDict() # This pre parses the argv list for --flagfile=<> options. argv = self.ReadFlagsFromFiles(argv) # Correct the argv to support the google style of passing boolean # parameters. Boolean parameters may be passed by using --mybool, # --nomybool, --mybool=(true|false|1|0). getopt does not support # having options that may or may not have a parameter. We replace # instances of the short form --mybool and --nomybool with their # full forms: --mybool=(true|false). original_argv = list(argv) shortest_matches = None for name, flag in fl.items(): if not flag.boolean: continue if shortest_matches is None: # Determine the smallest allowable prefix for all flag names shortest_matches = self.ShortestUniquePrefixes(fl) no_name = 'no' + name prefix = shortest_matches[name] no_prefix = shortest_matches[no_name] # Replace all occurences of this boolean with extended forms for arg_idx in range(1, len(argv)): arg = argv[arg_idx] if arg.find('=') >= 0: continue if arg.startswith('--'+prefix) and ('--'+name).startswith(arg): argv[arg_idx] = ('--%s=true' % name) elif arg.startswith('--'+no_prefix) and ('--'+no_name).startswith(arg): argv[arg_idx] = ('--%s=false' % name) # Loop over all of the flags, building up the lists of short options and # long options that will be passed to getopt. Short options are # specified as a string of letters, each letter followed by a colon if it # takes an argument. Long options are stored in an array of strings. # Each string ends with an '=' if it takes an argument. for name, flag in fl.items(): longopts.append(name + "=") if len(name) == 1: # one-letter option: allow short flag type also shortopts += name if not flag.boolean: shortopts += ":" try: optlist, unparsed_args = getopt.getopt(argv[1:], shortopts, longopts) except getopt.GetoptError, e: raise FlagsError, e for name, arg in optlist: if name.startswith('--'): # long option name = name[2:] short_option = 0 else: # short option name = name[1:] short_option = 1 if fl.has_key(name): flag = fl[name] if flag.boolean and short_option: arg = 1 flag.Parse(arg) if unparsed_args: # unparsed_args becomes the first non-flag detected by getopt to # the end of argv. Because argv may have been modified above, # return original_argv for this region. return argv[:1] + original_argv[-len(unparsed_args):] else: return argv[:1] def Reset(self): """ Reset the values to the point before FLAGS(argv) was called. """ for f in self.FlagDict().values(): f.Unparse() def RegisteredFlags(self): """ Return a list of all registered flags. """ return self.FlagDict().keys() def FlagValuesDict(self): """ Return a dictionary with flag names as keys and flag values as values. """ flag_values = {} for flag_name in self.RegisteredFlags(): flag = self.FlagDict()[flag_name] flag_values[flag_name] = flag.value return flag_values def __str__(self): """ Generate a help string for all known flags. """ helplist = [] flags_by_module = self.__dict__['__flags_by_module'] if flags_by_module: modules = flags_by_module.keys() modules.sort() # Print the help for the main module first, if possible. main_module = _GetMainModule() if main_module in modules: modules.remove(main_module) modules = [ main_module ] + modules for module in modules: self.__RenderModuleFlags(module, helplist) else: # Just print one long list of flags. self.__RenderFlagList(self.FlagDict().values(), helplist) return '\n'.join(helplist) def __RenderModuleFlags(self, module, output_lines): """ Generate a help string for a given module. """ flags_by_module = self.__dict__['__flags_by_module'] if module in flags_by_module: output_lines.append('\n%s:' % module) self.__RenderFlagList(flags_by_module[module], output_lines) def MainModuleHelp(self): """ Generate a help string for all known flags of the main module. """ helplist = [] self.__RenderModuleFlags(_GetMainModule(), helplist) return '\n'.join(helplist) def __RenderFlagList(self, flaglist, output_lines): fl = self.FlagDict() flaglist = [(flag.name, flag) for flag in flaglist] flaglist.sort() flagset = {} for (name, flag) in flaglist: # It's possible this flag got deleted or overridden since being # registered in the per-module flaglist. Check now against the # canonical source of current flag information, the FlagDict. if fl.get(name, None) != flag: # a different flag is using this name now continue # only print help once if flagset.has_key(flag): continue flagset[flag] = 1 flaghelp = " " if flag.short_name: flaghelp += "-%s," % flag.short_name if flag.boolean: flaghelp += "--[no]%s" % flag.name + ":" else: flaghelp += "--%s" % flag.name + ":" flaghelp += " " if flag.help: flaghelp += flag.help if flag.default_as_str: flaghelp += "\n (default: %s)" % flag.default_as_str if flag.parser.syntactic_help: flaghelp += "\n (%s)" % flag.parser.syntactic_help output_lines.append(flaghelp) def get(self, name, default): """ Retrieve the .value member of a flag object, or default if .value is None """ value = self.__getattr__(name) if value is not None: # Can't do if not value, b/c value might be '0' or "" return value else: return default def ShortestUniquePrefixes(self, fl): """ Returns a dictionary mapping flag names to their shortest unique prefix. """ # Sort the list of flag names sorted_flags = [] for name, flag in fl.items(): sorted_flags.append(name) if flag.boolean: sorted_flags.append('no%s' % name) sorted_flags.sort() # For each name in the sorted list, determine the shortest unique prefix # by comparing itself to the next name and to the previous name (the latter # check uses cached info from the previous loop). shortest_matches = {} prev_idx = 0 for flag_idx in range(len(sorted_flags)): curr = sorted_flags[flag_idx] if flag_idx == (len(sorted_flags) - 1): next = None else: next = sorted_flags[flag_idx+1] next_len = len(next) for curr_idx in range(len(curr)): if (next is None or curr_idx >= next_len or curr[curr_idx] != next[curr_idx]): # curr longer than next or no more chars in common shortest_matches[curr] = curr[:max(prev_idx, curr_idx) + 1] prev_idx = curr_idx break else: # curr shorter than (or equal to) next shortest_matches[curr] = curr prev_idx = curr_idx + 1 # next will need at least one more char return shortest_matches def __IsFlagFileDirective(self, flag_string): """ Detects the --flagfile= token. Takes a string which might contain a '--flagfile=' directive. Returns a Boolean. """ if isinstance(flag_string, type("")): if flag_string.startswith('--flagfile='): return 1 elif flag_string == '--flagfile': return 1 elif flag_string.startswith('-flagfile='): return 1 elif flag_string == '-flagfile': return 1 else: return 0 return 0 def ExtractFilename(self, flagfile_str): """Function to remove the --flagfile= (or variant) and return just the filename part. We can get strings that look like: --flagfile=foo, -flagfile=foo. The case of --flagfile foo and -flagfile foo shouldn't be hitting this function, as they are dealt with in the level above this funciton. """ if flagfile_str.startswith('--flagfile='): return os.path.expanduser((flagfile_str[(len('--flagfile=')):]).strip()) elif flagfile_str.startswith('-flagfile='): return os.path.expanduser((flagfile_str[(len('-flagfile=')):]).strip()) else: raise FlagsError('Hit illegal --flagfile type: %s' % flagfile_str) return '' def __GetFlagFileLines(self, filename, parsed_file_list): """Function to open a flag file, return its useful (!=comments,etc) lines. Takes: A filename to open and read A list of files we have already read THAT WILL BE CHANGED Returns: List of strings. See the note below. NOTE(springer): This function checks for a nested --flagfile= tag and handles the lower file recursively. It returns a list off all the lines that _could_ contain command flags. This is EVERYTHING except whitespace lines and comments (lines starting with '#' or '//'). """ line_list = [] # All line from flagfile. flag_line_list = [] # Subset of lines w/o comments, blanks, flagfile= tags. try: file_obj = open(filename, 'r') except IOError, e_msg: print e_msg print 'ERROR:: Unable to open flagfile: %s' % (filename) return flag_line_list line_list = file_obj.readlines() file_obj.close() parsed_file_list.append(filename) # This is where we check each line in the file we just read. for line in line_list: if line.isspace(): pass # Checks for comment (a line that starts with '#'). elif (line.startswith('#') or line.startswith('//')): pass # Checks for a nested "--flagfile=" flag in the current file. # If we find one, recursively parse down into that file. elif self.__IsFlagFileDirective(line): sub_filename = self.ExtractFilename(line) # We do a little safety check for reparsing a file we've already done. if not sub_filename in parsed_file_list: included_flags = self.__GetFlagFileLines(sub_filename, parsed_file_list) flag_line_list.extend(included_flags) else: # Case of hitting a circularly included file. print >>sys.stderr, ('Warning: Hit circular flagfile dependency: %s' % sub_filename) else: # Any line that's not a comment or a nested flagfile should # get copied into 2nd position, this leaves earlier arguements # further back in the list, which makes them have higher priority. flag_line_list.append(line.strip()) return flag_line_list def ReadFlagsFromFiles(self, argv): """Process command line args, but also allow args to be read from file Usage: Takes: a list of strings, usually sys.argv, which may contain one or more flagfile directives of the form --flagfile="./filename" References: Global gflags.FLAG class instance Returns: a new list which has the original list combined with what we read from any flagfile(s). This function should be called before the normal FLAGS(argv) call. This function simply scans the input list for a flag that looks like: --flagfile= Then it opens , reads all valid key and value pairs and inserts them into the input list between the first item of the list and any subsequent items in the list. Note that your application's flags are still defined the usual way using gflags DEFINE_flag() type functions. Notes (assuming we're getting a commandline of some sort as our input): --> Any flags on the command line we were passed in _should_ always take precedence!!! --> a further "--flagfile=" CAN be nested in a flagfile. It will be processed after the parent flag file is done. --> For duplicate flags, first one we hit should "win". --> In a flagfile, a line beginning with # or // is a comment --> Entirely blank lines _should_ be ignored """ parsed_file_list = [] rest_of_args = argv new_argv = [] while rest_of_args: current_arg = rest_of_args[0] rest_of_args = rest_of_args[1:] if self.__IsFlagFileDirective(current_arg): # This handles the case of -(-)flagfile foo. Inthis case the next arg # really is part of this one. if current_arg == '--flagfile' or current_arg =='-flagfile': if not rest_of_args: raise IllegalFlagValue, '--flagfile with no argument' flag_filename = os.path.expanduser(rest_of_args[0]) rest_of_args = rest_of_args[1:] else: # This handles the case of (-)-flagfile=foo. flag_filename = self.ExtractFilename(current_arg) new_argv = (new_argv[:1] + self.__GetFlagFileLines(flag_filename, parsed_file_list) + new_argv[1:]) else: new_argv.append(current_arg) return new_argv def FlagsIntoString(self): """ Retreive a string version of all the flags with assignments stored in this FlagValues object. Should mirror the behavior of the c++ version of FlagsIntoString. Each flag assignment is seperated by a newline. """ s = '' for flag in self.FlagDict().values(): if flag.value is not None: s += flag.Serialize() + '\n' return s def AppendFlagsIntoFile(self, filename): """ Appends all flags found in this FlagInfo object to the file specified. Output will be in the format of a flagfile. This should mirror the behavior of the c++ version of AppendFlagsIntoFile. """ out_file = open(filename, 'a') out_file.write(self.FlagsIntoString()) out_file.close() #end of the FLAGS registry class # The global FlagValues instance FLAGS = FlagValues() class Flag: """ 'Flag' objects define the following fields: .name - the name for this flag .default - the default value for this flag .default_as_str - default value as repr'd string, e.g., "'true'" (or None) .value - the most recent parsed value of this flag; set by Parse() .help - a help string or None if no help is available .short_name - the single letter alias for this flag (or None) .boolean - if 'true', this flag does not accept arguments .present - true if this flag was parsed from command line flags. .parser - an ArgumentParser object .serializer - an ArgumentSerializer object .allow_override - the flag may be redefined without raising an error The only public method of a 'Flag' object is Parse(), but it is typically only called by a 'FlagValues' object. The Parse() method is a thin wrapper around the 'ArgumentParser' Parse() method. The parsed value is saved in .value, and the .present member is updated. If this flag was already present, a FlagsError is raised. Parse() is also called during __init__ to parse the default value and initialize the .value member. This enables other python modules to safely use flags even if the __main__ module neglects to parse the command line arguments. The .present member is cleared after __init__ parsing. If the default value is set to None, then the __init__ parsing step is skipped and the .value member is initialized to None. Note: The default value is also presented to the user in the help string, so it is important that it be a legal value for this flag. """ def __init__(self, parser, serializer, name, default, help_string, short_name=None, boolean=0, allow_override=0): self.name = name self.default = default if not help_string: help_string = '(no help available)' self.help = help_string self.short_name = short_name self.boolean = boolean self.present = 0 self.parser = parser self.serializer = serializer self.allow_override = allow_override self.value = None # We can't allow a None override because it may end up not being # passed to C++ code when we're overriding C++ flags. So we # cowardly bail out until someone fixes the semantics of trying to # pass None to a C++ flag. See swig_flags.Init() for details on # this behavior. if default is None and allow_override: raise DuplicateFlag, name self.Unparse() self.default_as_str = self.__GetParsedValueAsString(self.value) def __GetParsedValueAsString(self, value): if value is None: return None if self.serializer: return repr(self.serializer.Serialize(value)) if self.boolean: if value: return repr('true') else: return repr('false') return repr(str(value)) def Parse(self, argument): try: self.value = self.parser.Parse(argument) except ValueError, e: # recast ValueError as IllegalFlagValue raise IllegalFlagValue, ("flag --%s: " % self.name) + str(e) self.present += 1 def Unparse(self): if self.default is None: self.value = None else: self.Parse(self.default) self.present = 0 def Serialize(self): if self.value is None: return '' if self.boolean: if self.value: return "--%s" % self.name else: return "--no%s" % self.name else: if not self.serializer: raise FlagsError, "Serializer not present for flag %s" % self.name return "--%s=%s" % (self.name, self.serializer.Serialize(self.value)) def SetDefault(self, value): """ Change the default value, and current value, of this flag object """ if value is not None: # See __init__ for logic details self.Parse(value) self.present -= 1 # reset .present after parsing new default value else: self.value = None self.default = value self.default_as_str = self.__GetParsedValueAsString(value) class ArgumentParser: """ This is a base class used to parse and convert arguments. The Parse() method checks to make sure that the string argument is a legal value and convert it to a native type. If the value cannot be converted, it should throw a 'ValueError' exception with a human readable explanation of why the value is illegal. Subclasses should also define a syntactic_help string which may be presented to the user to describe the form of the legal values. """ syntactic_help = "" def Parse(self, argument): """ The default implementation of Parse() accepts any value of argument, simply returning it unmodified. """ return argument class ArgumentSerializer: """ This is the base class for generating string representations of a flag value """ def Serialize(self, value): return str(value) class ListSerializer(ArgumentSerializer): def __init__(self, list_sep): self.list_sep = list_sep def Serialize(self, value): return self.list_sep.join([str(x) for x in value]) # The DEFINE functions are explained in the module doc string. def DEFINE(parser, name, default, help, flag_values=FLAGS, serializer=None, **args): """ This creates a generic 'Flag' object that parses its arguments with a 'Parser' and registers it with a 'FlagValues' object. Developers who need to create their own 'Parser' classes should call this module function. to register their flags. For example: DEFINE(DatabaseSpec(), "dbspec", "mysql:db0:readonly:hr", "The primary database") """ DEFINE_flag(Flag(parser, serializer, name, default, help, **args), flag_values) def DEFINE_flag(flag, flag_values=FLAGS): """ This registers a 'Flag' object with a 'FlagValues' object. By default, the global FLAGS 'FlagValue' object is used. Typical users will use one of the more specialized DEFINE_xxx functions, such as DEFINE_string or DEFINEE_integer. But developers who need to create Flag objects themselves should use this function to register their flags. """ # copying the reference to flag_values prevents pychecker warnings fv = flag_values fv[flag.name] = flag if flag_values == FLAGS: # We are using the global flags dictionary, so we'll want to sort the # usage output by calling module in FlagValues.__str__ (FLAGS is an # instance of FlagValues). This requires us to keep track # of which module is creating the flags. # Tell FLAGS who's defining flag. FLAGS._RegisterFlagByModule(__GetCallingModule(), flag) ############################### ################# STRING FLAGS ############################### def DEFINE_string(name, default, help, flag_values=FLAGS, **args): """ This registers a flag whose value can be any string. """ parser = ArgumentParser() serializer = ArgumentSerializer() DEFINE(parser, name, default, help, flag_values, serializer, **args) ############################### ################ BOOLEAN FLAGS ############################### #### and the special HELP flag ############################### class BooleanParser(ArgumentParser): """ A boolean value """ def Convert(self, argument): """ convert the argument to a boolean (integer); raise ValueError on errors """ if type(argument) == str: if argument.lower() in ['true', 't', '1']: return 1 elif argument.lower() in ['false', 'f', '0']: return 0 return int(argument) def Parse(self, argument): val = self.Convert(argument) return val class BooleanFlag(Flag): """ A basic boolean flag. Boolean flags do not take any arguments, and their value is either 0 (false) or 1 (true). The false value is specified on the command line by prepending the word 'no' to either the long or short flag name. For example, if a Boolean flag was created whose long name was 'update' and whose short name was 'x', then this flag could be explicitly unset through either --noupdate or --nox. """ def __init__(self, name, default, help, short_name=None, **args): p = BooleanParser() Flag.__init__(self, p, None, name, default, help, short_name, 1, **args) if not self.help: self.help = "a boolean value" def DEFINE_boolean(name, default, help, flag_values=FLAGS, **args): """ This registers a boolean flag - one that does not take an argument. If a user wants to specify a false value explicitly, the long option beginning with 'no' must be used: i.e. --noflag This flag will have a value of None, 0 or 1. None is possible if default=None and the user does not specify the flag on the command line. """ DEFINE_flag(BooleanFlag(name, default, help, **args), flag_values) class HelpFlag(BooleanFlag): """ HelpFlag is a special boolean flag that prints usage information and raises a SystemExit exception if it is ever found in the command line arguments. Note this is called with allow_override=1, so other apps can define their own --help flag, replacing this one, if they want. """ def __init__(self): BooleanFlag.__init__(self, "help", 0, "show this help", short_name="?", allow_override=1) def Parse(self, arg): if arg: doc = sys.modules["__main__"].__doc__ flags = str(FLAGS) print doc or ("\nUSAGE: %s [flags]\n" % sys.argv[0]) if flags: print "flags:" print flags sys.exit(1) class HelpshortFlag(BooleanFlag): """ HelpshortFlag is a special boolean flag that prints usage information for the "main" module, and rasies a SystemExit exception if it is ever found in the command line arguments. Note this is called with allow_override=1, so other apps can define their own --helpshort flag, replacing this one, if they want. """ def __init__(self): BooleanFlag.__init__(self, "helpshort", 0, "show usage only for this module", allow_override=1) def Parse(self, arg): if arg: doc = sys.modules["__main__"].__doc__ flags = FLAGS.MainModuleHelp() print doc or ("\nUSAGE: %s [flags]\n" % sys.argv[0]) if flags: print "flags:" print flags sys.exit(1) ############################### ################## FLOAT FLAGS ############################### class FloatParser(ArgumentParser): """ A floating point value; optionally bounded to a given upper and lower bound. """ number_article = "a" number_name = "number" syntactic_help = " ".join((number_article, number_name)) def __init__(self, lower_bound=None, upper_bound=None): self.lower_bound = lower_bound self.upper_bound = upper_bound sh = self.syntactic_help if lower_bound != None and upper_bound != None: sh = ("%s in the range [%s, %s]" % (sh, lower_bound, upper_bound)) elif lower_bound == 1: sh = "a positive %s" % self.number_name elif upper_bound == -1: sh = "a negative %s" % self.number_name elif lower_bound == 0: sh = "a non-negative %s" % self.number_name elif upper_bound != None: sh = "%s <= %s" % (self.number_name, upper_bound) elif lower_bound != None: sh = "%s >= %s" % (self.number_name, lower_bound) self.syntactic_help = sh def Convert(self, argument): """ convert the argument to a float; raise ValueError on errors """ return float(argument) def Parse(self, argument): val = self.Convert(argument) if ((self.lower_bound != None and val < self.lower_bound) or (self.upper_bound != None and val > self.upper_bound)): raise ValueError, "%s is not %s" % (val, self.syntactic_help) return val def DEFINE_float(name, default, help, lower_bound=None, upper_bound=None, flag_values = FLAGS, **args): """ This registers a flag whose value must be a float. If lower_bound, or upper_bound are set, then this flag must be within the given range. """ parser = FloatParser(lower_bound, upper_bound) serializer = ArgumentSerializer() DEFINE(parser, name, default, help, flag_values, serializer, **args) ############################### ################ INTEGER FLAGS ############################### class IntegerParser(FloatParser): """ An integer value; optionally bounded to a given upper or lower bound. """ number_article = "an" number_name = "integer" syntactic_help = " ".join((number_article, number_name)) def Convert(self, argument): __pychecker__ = 'no-returnvalues' if type(argument) == str: base = 10 if len(argument) > 2 and argument[0] == "0" and argument[1] == "x": base=16 try: return int(argument, base) # ValueError is thrown when argument is a string, and overflows an int. except ValueError: return long(argument, base) else: try: return int(argument) # OverflowError is thrown when argument is numeric, and overflows an int. except OverflowError: return long(argument) def DEFINE_integer(name, default, help, lower_bound=None, upper_bound=None, flag_values = FLAGS, **args): """ This registers a flag whose value must be an integer. If lower_bound, or upper_bound are set, then this flag must be within the given range. """ parser = IntegerParser(lower_bound, upper_bound) serializer = ArgumentSerializer() DEFINE(parser, name, default, help, flag_values, serializer, **args) ############################### ################### ENUM FLAGS ############################### class EnumParser(ArgumentParser): """ A string enum value """ def __init__(self, enum_values=None): self.enum_values = enum_values def Parse(self, argument): """ If enum_values is not specified, any string is allowed """ if self.enum_values and argument not in self.enum_values: raise ValueError, ("value should be one of <%s>" % "|".join(self.enum_values)) return argument class EnumFlag(Flag): """ A basic enum flag. The flag's value can be any string from the list of enum_values. """ def __init__(self, name, default, help, enum_values=[], short_name=None, **args): p = EnumParser(enum_values) g = ArgumentSerializer() Flag.__init__(self, p, g, name, default, help, short_name, **args) if not self.help: self.help = "an enum string" self.help = "<%s>: %s" % ("|".join(enum_values), self.help) def DEFINE_enum(name, default, enum_values, help, flag_values=FLAGS, **args): """ This registers a flag whose value can be a string from a set of specified values. """ DEFINE_flag(EnumFlag(name, default, help, enum_values, ** args), flag_values) ############################### ################### LIST FLAGS ############################### class BaseListParser(ArgumentParser): """ A base class for a string list parser. To extend, inherit from this class, and call BaseListParser.__init__(self, token, name) where token is a character used to tokenize, and name is a description of the separator """ def __init__(self, token=None, name=None): assert name self._token = token self._name = name self.syntactic_help = "a %s separated list" % self._name def Parse(self, argument): if argument == '': return [] else: return [s.strip() for s in argument.split(self._token)] class ListParser(BaseListParser): """ A string list parser (comma-separated) """ def __init__(self): BaseListParser.__init__(self, ',', 'comma') class WhitespaceSeparatedListParser(BaseListParser): """ A string list parser (whitespace-separated) """ def __init__(self): BaseListParser.__init__(self, None, 'whitespace') def DEFINE_list(name, default, help, flag_values=FLAGS, **args): """ This registers a flag whose value is a list of strings, separated by commas """ parser = ListParser() serializer = ListSerializer(',') DEFINE(parser, name, default, help, flag_values, serializer, **args) def DEFINE_spaceseplist(name, default, help, flag_values=FLAGS, **args): """ This registers a flag whose value is a list of strings, separated by any whitespace """ parser = WhitespaceSeparatedListParser() serializer = ListSerializer(' ') DEFINE(parser, name, default, help, flag_values, serializer, **args) ############################### ################## MULTI FLAGS ############################### class MultiFlag(Flag): """ MultiFlag is a specialized subclass of Flag that accumulates multiple values in a list when a command-line option appears multiple times. See the __doc__ for Flag for most behavior of this class. Only differences in behavior are described here: * the default value may be a single value -OR- a list of values * the value of the flag is always a list, even if the option was only supplied once, and even if the default value is a single value """ def __init__(self, *args, **kwargs): Flag.__init__(self, *args, **kwargs) self.help = (self.help + ';\n repeat this option to specify a list of values') def Parse(self, arguments): """Parse one or more arguments with the installed parser. Arguments: arguments: a single argument or a list of arguments (typically a list of default values); single arguments will be converted internally into a list containing one item """ if not isinstance(arguments, list): # Default value may be a list of values. Most other arguments will not # be, so convert them into a single-item list to make processing simpler # below. arguments = [ arguments ] if self.present: # keep a backup reference to list of previously supplied option values values = self.value else: # "erase" the defaults with an empty list values = [] for item in arguments: # have Flag superclass parse argument, overwriting self.value reference Flag.Parse(self, item) # also increments self.present values.append(self.value) # put list of option values back in member variable self.value = values def Serialize(self): if not self.serializer: raise FlagsError, "Serializer not present for flag %s" % self.name if self.value is None: return '' s = '' multi_value = self.value for self.value in multi_value: if s: s += ' ' s += Flag.Serialize(self) self.value = multi_value return s def DEFINE_multi(parser, serializer, name, default, help, flag_values=FLAGS, **args): """ This creates a generic 'MultiFlag' object that parses its arguments with a 'Parser' and registers it with a 'FlagValues' object. Developers who need to create their own 'Parser' classes for options which can appear multiple times can call this module function to register their flags. """ DEFINE_flag(MultiFlag(parser, serializer, name, default, help, **args), flag_values) def DEFINE_multistring(name, default, help, flag_values=FLAGS, **args): """ This registers a flag whose value can be a list of any strings. Use the flag on the command line multiple times to place multiple string values into the list. The 'default' may be a single string (which will be converted into a single-element list) or a list of strings. """ parser = ArgumentParser() serializer = ArgumentSerializer() DEFINE_multi(parser, serializer, name, default, help, flag_values, **args) def DEFINE_multi_int(name, default, help, lower_bound=None, upper_bound=None, flag_values=FLAGS, **args): """ This registers a flag whose value can be a list of any integers. Use the flag on the command line multiple times to place multiple integer values into the list. The 'default' may be a single integer (which will be converted into a single-element list) or a list of integers. """ parser = IntegerParser(lower_bound, upper_bound) serializer = ArgumentSerializer() DEFINE_multi(parser, serializer, name, default, help, flag_values, **args) # Now register the flags that we want to exist in all applications. # These are all defined with allow_override=1, so user-apps can use # these flagnames for their own purposes, if they want. DEFINE_flag(HelpFlag()) DEFINE_flag(HelpshortFlag()) mythnettv-release-7/mythnettvcore.py0000644000175000017500000001217311135202764017150 0ustar mikalmikal#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2006, 2007, 2008 # Released under the terms of the GNU GPL v2 # This file is the core of MythNetTV. It is intended that user interfaces call # into this module to get things done, and then present their own interface in # whatever manner makes sense to them import gflags import sys import database import program import proxyhandler import syndication import video gflags.DEFINE_boolean('oldestfirst', False, 'Download the oldest programs first') gflags.DEFINE_boolean('newestfirst', False, 'Download the newest programs first') gflags.DEFINE_boolean('verbose', False, 'Output verbose debugging information') FLAGS = gflags.FLAGS def NextDownloads(count, filter, subtitle, out=sys.stdout): """NextDownloads -- return a list of the GUIDs to download next. Optionally filter based on an exact match of title string. """ db = database.MythNetTvDatabase() remaining = int(count) if FLAGS.oldestfirst and FLAGS.newestfirst: out.write('Cannot download both oldest and newest first!\n') return [] old_target = remaining / 2 if FLAGS.oldestfirst: old_target = remaining elif FLAGS.newestfirst: old_target = 0 guids = [] if filter == None: title = 'is not NULL' else: title = '= "%s"' % filter out.write('Download constrained to "%s"\n' % filter) if subtitle == None: subtitle_filter = '.*' else: subtitle_filter = subtitle subscription_titles = [] inactive_titles = [] for row in db.GetRows('select * from mythnettv_subscriptions'): if not row['inactive']: subscription_titles.append(row['title']) if FLAGS.verbose: out.write(' %s is active\n' % row['title']) else: inactive_titles.append(row['title']) if FLAGS.verbose: out.write(' %s is inactive\n' % row['title']) for row in db.GetRows('select * from mythnettv_programs where ' 'download_finished is NULL and title %s ' 'and subtitle rlike "%s" and inactive is null ' 'order by date asc limit %d;' %(title, subtitle_filter, old_target)): # We should only download shows we are _currently_ subscribed to if not row['title'] in inactive_titles: guids.append(row['guid']) remaining -= 1 if FLAGS.verbose: print ' Adding %s : %s to the list of downloads' %(row['title'], row['subtitle']) if FLAGS.verbose: out.write('\n') for row in db.GetRows('select * from mythnettv_programs where ' 'download_finished is NULL and title %s ' 'and subtitle rlike "%s" ' 'and inactive is null ' 'and guid not in ("%s") ' 'order by date desc limit %d;' %(title, subtitle_filter, '", "'.join(guids), remaining)): # We should only download shows we are _currently_ subscribed to if not row['title'] in inactive_titles: guids.append(row['guid']) if FLAGS.verbose: print ' Adding %s : %s to the list of downloads' %(row['title'], row['subtitle']) if FLAGS.verbose: out.write('\n') out.write('%d matches\n' % len(guids)) return guids def DownloadAndImport(db, guid, out=sys.stdout): """DownloadAndImport -- perform all the steps to download and import a given guid. """ prog = program.MythNetTvProgram(db) try: out.write('\nDownloading %s\n' % guid) prog.Load(guid) if prog.Download(db.GetSettingWithDefault('datadir', FLAGS.datadir), out=out) == True: out.write('Download OK\n') prog.Import(out=out) return True except program.DownloadBudgetExceededException, e: out.write('Download Error: %s\n' % e) if 'attempts' in prog.persistant and prog.persistant['attempts']: prog.persistant['attempts'] -= 1 prog.Store() except Exception, e: out.write('Download Error: %s\n' % e) return False def Subscribe(url, title): """Subscribe -- subscribe to a new RSS or ATOM feed""" db = database.MythNetTvDatabase() db.WriteOneRow('mythnettv_subscriptions', 'url', {'url':url, 'title':title, 'inactive':None}) def Update(out, title=None): """Update -- download updates for all feeds""" db = database.MythNetTvDatabase() title_sql = '' if title: title_sql = 'and title = "%s"' % title for row in db.GetRows('select * from mythnettv_subscriptions ' 'where inactive is null %s' % title_sql): if FLAGS.verbose: out.write('Updating: %s\n' % row['url']) try: proxy = proxyhandler.HttpHandler(db) xmlfile = proxy.Open(row['url'], out=out) syndication.Sync(db, xmlfile, row['title'], out=out) except Exception, e: out.write('Failed to update %s: %s\n' %(row['url'], e)) mythnettv-release-7/mythnettv_test.py0000711000175000017500000001764511135202764017342 0ustar mikalmikal#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2006, 2007, 2008 # Released under the terms of the GNU GPL v2 # Unit tests for mythnettv's primary user interface # This hackery is to get around "mythnettv" not having a .py at the end of its # filename import imp mythnettv = imp.load_module('mythnettv', open('mythnettv'), 'mythnettv', ('.py', 'U', 1)) import cStringIO import gflags import os import sys import unittest import MySQLdb import testsetup FLAGS = gflags.FLAGS class GetPossibleTest(unittest.TestCase): def testEntryPresent(self): self.assertEquals(mythnettv.GetPossible([1, 2, 3, 4], 1), 2) def testEntryMissing(self): self.assertEquals(mythnettv.GetPossible([], 1), None) class UserInterfaceTests(unittest.TestCase): def Cursor(self): """ Return a cursor to the test database """ db_connection = MySQLdb.connect(host = 'localhost', user = 'test', passwd = 'test', db = 'mythnettv_tests') return db_connection.cursor(MySQLdb.cursors.DictCursor) def setUp(self): # Make sure we have a data directory on disk if not os.path.isdir('/tmp/testdata'): os.makedirs('/tmp/testdata') # Make sure we have a fake MythTV recordings directory if not os.path.isdir('/tmp/testmyth'): os.makedirs('/tmp/testmyth') testsetup.SetupTestingDatabase(self.Cursor()) # This magic fake command line is needed to trick mythnettv into using # the test database instance, and a random data directory REQUIRED_INVOKATION = ['mythnettv', '--db_host=localhost', '--db_user=test', '--db_password=test', '--db_name=mythnettv_tests', '--datadir=/tmp/testdata', '--nocommflag', '--nopromptforannounce'] def AssertOutputContains(self, output, substring): """ A simple wrapper to ensure that some output contains an expected string. """ self.assertNotEqual(output.find(substring), -1, """********* I expected to find the string "%s" in this output: %s *********""" %(substring, output)) def AssertOutputDoesntContain(self, output, substring): """ A simple wrapper to ensure that some output does not contain an expected string. """ self.assertEqual(output.find(substring), -1, """********* I expected to not find the string "%s" in this output: %s *********""" %(substring, output)) def testEverything(self): """ This is lots of different tests, but they need to be run in this order, so they are all in here. I am sure there is a better way I haven't thought of. """ # Just test if we can start up output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['list'], out=output) # Subscribe to something output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['subscribe', 'http://www.stillhq.com/mythtv/mythnettv/testdata/' 'gruen_mp4.xml', 'Gruen Transfer'], out=output) self.AssertOutputContains(output.getvalue(), 'Subscribed to') # The update the program list output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['update'], out=output) self.AssertOutputContains(output.getvalue(), 'Creating program for') # A second update should find nothing new output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['update'], out=output) self.AssertOutputDoesntContain(output.getvalue(), 'Creating program for') # Can we determine what to download next? output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['nextdownload', '100', 'Gruen Transfer'], out=output) self.AssertOutputContains(output.getvalue(), '1 matches') # Download the program output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['download', '100', 'Gruen Transfer'], out=output) self.AssertOutputContains(output.getvalue(), 'Done') self.AssertOutputContains(output.getvalue(), 'I don\'t know if we need to ' 'transcode videos in ' 'DVSD format') # There should be nothing left to download output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['nextdownload', '100', 'Gruen Transfer'], out=output) self.AssertOutputContains(output.getvalue(), '0 matches') # Import a local file and make sure that works too output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['importlocal', '/data/video/testdata/foo.avi', 'Local Import Test', '--', '--'], out=output) self.AssertOutputContains(output.getvalue(), 'Done') # Try a direct fetch of a URL to a video output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['importremote', 'http://www.stillhq.com/mythtv/mythnettv/testdata/' 'gruen_2008_ep10.mp4', 'Remote Import Test', '--', '--'], out=output) self.AssertOutputContains(output.getvalue(), 'Done') # Try a direct fetch of a URL to a video output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['url', 'http://www.stillhq.com/mythtv/mythnettv/testdata/' 'vimeo.rss', 'Sk8Columbia'], out=output) # Try an apple feed which used to cause unicode problems # TODO(mikal): move this into a more generic set of feed unit # tests output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['url', 'http://www.stillhq.com/mythtv/mythnettv/testdata/' 'apple-quick-tip-of-the-week.xml', 'Apple Quick Tips'], out=output) mythnettv.main(self.REQUIRED_INVOKATION + ['update'], out=output) # Try a feed which has had date problems in the past output = cStringIO.StringIO() mythnettv.main(self.REQUIRED_INVOKATION + ['url', 'http://www.stillhq.com/mythtv/mythnettv/testdata/' 'hak5.xml', 'hak5'], out=output) print output.getvalue() ################ # Check the state of the database at the end of the tests ################ cursor = self.Cursor() # There should be twelve tables now cursor.execute('show tables') self.assertEqual(cursor.rowcount, 12) # The channel table should have a MythNetTV channel cursor.execute('select * from channel where name="MythNetTV";') self.assertEqual(cursor.rowcount, 1) # The recordings table should have one Gruen in it cursor.execute('select * from recorded ' 'where subtitle="The Gruen Transfer Episode 10";') self.assertEqual(cursor.rowcount, 1) # The recordings table should have one local transfer in it cursor.execute('select * from recorded ' 'where title="Local Import Test";') self.assertEqual(cursor.rowcount, 1) # The recordings table should have one local transfer in it cursor.execute('select * from recorded ' 'where title="Remote Import Test";') self.assertEqual(cursor.rowcount, 1) if __name__ == "__main__": # Parse flags try: argv = FLAGS(sys.argv) except gflags.FlagsError, e: out.write('%s\n' % e) Usage(out) unittest.main() mythnettv-release-7/program.py0000644000175000017500000005441411135202764015710 0ustar mikalmikal#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2006, 2007, 2008 # Released under the terms of the GNU GPL v2\ import commands import datetime import MySQLdb import re import os import shutil import socket import subprocess import sys import tempfile import time import unicodedata import database import gflags import mythnettvcore import proxyhandler import utility import video from stat import * FLAGS = gflags.FLAGS gflags.DEFINE_boolean('commflag', True, 'Run the mythcommflag command on new videos') gflags.DEFINE_boolean('force', False, 'Force downloads to run, even if they failed recently') gflags.DEFINE_string('uploadrate', '', 'Override the default upload rate for bittorrent for ' 'just this one download') # Exceptions returned by this module class LoggingException(Exception): """ Log exceptions to the database as well as returning them as exceptions """ def __init__(self, db, error): if db: try: db.Log(error) except: pass Exception.__init__(self, error) class StorageException(LoggingException): """ Errors with storage of programs """ class DownloadException(LoggingException): """ Errors in the download process """ class DownloadBudgetExceededException(LoggingException): """ A proxy budget error """ def SafeForFilename(s): """SafeForFilename -- convert s into something which can be used for a filename. """ for c in [' ', '(', ')', '{', '}', '[', ']', ':', '\'', '"']: s = s.replace(c, '_') return s def Prompt(prompt): """Prompt -- prompt for input from the user""" sys.stdout.write('%s >> ' % prompt) return sys.stdin.readline().rstrip('\n') class MythNetTvProgram: """MythNetTvProgram -- a downloadable program. This class embodies everything we can do with a program. The existance of this class does not mean that the show has been fully downloaded and made available in MythTV yet. Instances of this class persist to the MySQL database. """ def __init__(self, db): self.persistant = {} self.db = db def FromUrl(self, url, guid): """FromUrl -- start a program based on its URL""" new_video = True # Some URLs have the ampersand escaped url = url.replace('&', '&') # Persist what we know now self.persistant['url'] = url self.persistant['filename'] = SafeForFilename(self.GetFilename(url)) self.persistant['guid'] = guid try: if self.db.GetOneRow('select * from mythnettv_programs ' 'where guid="%s";' % guid).keys() != []: new_video = False except: dummy = 'blah' self.Store() self.db.Log('Updated show from %s with guid %s' %(url, guid)) return new_video def FromInteractive(self, url, title, subtitle, description): """FromInteractive -- create a program by prompting the user for input for all the bits we need. We check if we have the data first, so that we're not too annoying. """ if url: self.persistant['url'] = url if title: self.persistant['title'] = title if subtitle: self.persistant['subtitle'] = subtitle if description: self.persistant['description'] = description for key in ['url', 'title', 'subtitle', 'description']: if not self.persistant.has_key(key): self.persistant[key] = Prompt(key) # TODO(mikal): Should I generate a more unique GUID? self.persistant['guid'] = self.persistant['url'] self.persistant['filename'] = SafeForFilename(self.GetFilename( self.persistant['url'])) self.Store() def GetFilename(self, url, out=sys.stdout): """GetFilename -- return the filename portion of a URL""" # Some URLs have the ampersand escaped re_filename = re.compile('.*/([^/\?]*).*') m = re_filename.match(url) if m: return m.group(1) if not '/' in url: return url raise(self.db, 'Could not determine local filename for %s\n' % url) def GetTitle(self): """GetTitle -- return the title of the program""" return self.persistant['title'] def GetSubtitle(self): """GetSubtitle -- return the subtitle of the program""" return self.persistant['subtitle'] def GetDate(self): """GetDate -- return the date of the program""" return self.persistant['unparsed_date'] def SetDate(self, date): """SetDate -- set the date of the program""" self.persistant['date'] = date.strftime('%a, %d %b %Y %H:%M:%S') self.persistant['unparsed_date'] = date.strftime('%a, %d %b %Y %H:%M:%S') self.persistant['parsed_date'] = date def GetMime(self): """GetMime -- return the program's mime type""" return self.persistant['mime_type'] def SetMime(self, mime): """SetMime -- set the program's mime type""" self.persistant['mime_type'] = mime def Load(self, guid): """Load -- load information based on a GUID from the DB""" self.persistant = self.db.GetOneRow('select * from mythnettv_programs ' 'where guid="%s";' % guid) def Store(self): """Store -- persist to MySQL""" # We store the date of the entry a lot of different ways if not self.persistant.has_key('date'): self.SetDate(datetime.datetime.now()) try: self.db.WriteOneRow('mythnettv_programs', 'guid', self.persistant) except MySQLdb.Error, (errno, errstr): if errno != 1064: raise StorageException(self.db, 'Could not store program %s: %s "%s"' %(self.persistant['guid'], errno, errstr)) except database.FormatException, e: raise e except Exception, e: raise StorageException(self.db, 'Could not store program: %s: "%s" (%s)' %(self.persistant['guid'], e, type(e))) def SetUrl(self, url): """SetUrl -- set just the URL for the program""" self.persistant['url'] = url def SetShowInfo(self, title, subtitle, description, date, date_parsed): """SetShowInfo -- set show meta data""" self.persistant['title'] = title self.persistant['subtitle'] = subtitle self.persistant['description'] = description self.persistant['date'] = date self.persistant['unparsed_date'] = date self.persistant['parsed_date'] = repr(date_parsed) self.Store() self.db.Log('Set show info for guid %s' % self.persistant['guid']) def TemporaryFilename(self, datadir, out=sys.stdout): """TemporaryFilename -- calculate the filename to use in the temporary directory """ filename = '%s/%s' %(datadir, self.persistant['filename']) out.write('Destination will be %s\n' % filename) self.db.Log('Downloading %s to %s' %(self.persistant['guid'], filename)) return filename def DownloadMPlayer(self, filename): """DownloadRTSP -- download a show using mplayer""" datadir = self.db.GetSettingWithDefault('datadir', FLAGS.datadir) (status, out) = commands.getstatusoutput('cd %s; ' 'mplayer -dumpstream "%s"' %(datadir, self.persistant['url'])) if status != 0: raise DownloadException('MPlayer download failed') shutil.move(datadir + '/stream.dump', filename) return os.stat(filename)[ST_SIZE] def DownloadHTTP(self, filename, force_proxy=None, force_budget=-1, out=sys.stdout): """DownloadHTTP -- download a show, using HTTP""" this_attempt_total = 0 out.write('Download URL is "%s"\n' % self.persistant['url']) done = self.persistant.get('download_finished', '0') if done != '1': proxy = proxyhandler.HttpHandler(self.db) try: remote = proxy.Open(self.persistant['url'], force_proxy=force_proxy, force_budget=force_budget, out=out) out.write('Downloading %s\n' % self.persistant['url']) except Exception, e: raise DownloadException(self.db, 'HTTP download failed: %s' % e) local = open(filename, 'w') total = int(self.persistant.get('transfered', 0)) count = 0 while done != '1': data = remote.read(1024) length = len(data) if length < 1024: done = '1' local.write(data) total += length this_attempt_total += length proxy.LogProxyUsage(length) if count > 3000: self.persistant['transfered'] = repr(total) # TODO(mikal): Size should be determined beforehand if possible self.persistant['size'] = repr(total) self.Store() count = 0 out.write('%s: downloaded %s\n' %(datetime.datetime.now(), utility.DisplayFriendlySize(this_attempt_total))) self.persistant['last_attempt'] = datetime.datetime.now() self.Store() count += 1 if not proxy.BudgetAllowsDownload(1024): budget = utility.DisplayFriendlySize(proxy.GetBudget()) raise DownloadBudgetExceededException(self.db, '%s budget of %s exceeded' %(proxy.UsedProxy(), budget)) remote.close() local.close() return total def Download(self, datadir, force_proxy=None, force_budget=-1, out=sys.stdout): """Download -- download the show""" one_hour = datetime.timedelta(hours=1) one_hour_ago = datetime.datetime.now() - one_hour out.write('Considering %s: %s\n' %(self.persistant['title'], self.persistant['subtitle'])) if 'last_attempt' in self.persistant and \ self.persistant['last_attempt'] > one_hour_ago: out.write('Last attempt was too recent. It was at %s\n' % self.persistant['last_attempt']) if not FLAGS.force: return False else: out.write('Download forced\n') self.persistant['last_attempt'] = datetime.datetime.now() filename = self.TemporaryFilename(datadir, out=out) self.persistant['download_started'] = '1' self.Store() out.write('Downloading %s: %s\n\n' %(self.persistant['title'], self.persistant['subtitle'])) if 'attempts' in self.persistant and self.persistant['attempts']: max_attempts = int(self.db.GetSettingWithDefault('attempts', 3)) print ('This is a repeat attempt (%d attempts so far, max is %d)' %(self.persistant['attempts'], max_attempts)) if self.persistant['attempts'] > max_attempts: out.write('Too many failed attempts, giving up on this program\n') self.persistant['download_finished'] = 0 self.persistant['imported'] = 0 self.persistant['failed'] = 1 self.Store() return False self.persistant.setdefault('attempts', 0) self.persistant['attempts'] += 1 self.Store() total = 0 if self.persistant['url'].endswith('torrent') \ or self.persistant.get('mime_type', '').endswith('torrent'): total = self.DownloadHTTP(filename, out=out) if total == 0: self.Store() return False # DownloadHTTP thinks everything is complete because the HTTP download # finished OK. That's wrong. self.persistant['download_finished'] = None self.persistant['imported'] = None self.Store() out.write('Now fetching the bittorrent data\n') download_ok = False try: if self.persistant.get('tmp_name', '') == '': (tmpfd, tmpname) = tempfile.mkstemp(dir=datadir) os.close(tmpfd) os.unlink(tmpname) self.persistant['tmp_name'] = tmpname self.Store() else: tmpname = self.persistant['tmp_name'] # Upload rate can either be the shipped default, a new default from # the settings tables, or a temporary override if FLAGS.uploadrate: upload_rate = FLAGS.uploadrate else: upload_rate = self.db.GetSettingWithDefault('uploadrate', 100) cmd = '/usr/bin/btdownloadheadless.bittornado ' \ '--max_upload_rate %s ' \ '--display_interval 5 --spew 1 --saveas %s %s ' \ %(upload_rate, tmpname, filename) po = subprocess.Popen(cmd, shell=True, bufsize=1, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) out.write('Executing: %s (pid %d)\n' %(cmd, po.pid)) start_time = datetime.datetime.now() line = po.stdout.readline() while line: line = line.rstrip('\n') line = line.rstrip(' ') if line.startswith('time left'): out.write('%s: %s --> %s\n' %(self.persistant['title'], self.persistant['subtitle'], line)) self.persistant['last_attempt'] = datetime.datetime.now() self.Store() if line.endswith('Download Failed!'): out.write('Detected failed download\n') break elif line.endswith(':'): wait_time = datetime.datetime.now() - start_time out.write('Have waited %s (%d seconds)\n' %(wait_time, wait_time.seconds)) if wait_time.seconds > 300: out.write('Waited %s for download to start. Giving up.\n' % wait_time) break elif line.endswith('Download Succeeded!'): out.write('Done!\n') download_ok = True break elif line.endswith(''): out.write('Download got stuck. Giving up.\n') break line = po.stdout.readline() except IOError, e: raise DownloadError('Error downloading bittorrent data: %s' % e) out.write('Bittorrent download finished, kill download processes\n') commands.getoutput('for pid in `ps -ef | grep %s | grep -v grep | ' 'tr -s " " | cut -f 2 -d " "`; do kill -9 $pid; done' % po.pid) if download_ok: os.unlink(filename) self.persistant['filename'] = tmpname.split('/')[-1] torrent_size = self.persistant.get('transfered', 0) try: video_size = os.stat(tmpname).st_size self.persistant['transfered'] = torrent_size + video_size out.write('Total transfered is %d bytes\n' % (torrent_size + video_size)) except Exception, e: raise DownloadException('Error: %s\n' % e) else: return False elif self.persistant['url'].startswith('http://'): total = self.DownloadHTTP(filename, force_proxy=force_proxy, force_budget=force_budget) else: total = self.DownloadMPlayer(filename) if total == 0: return False self.persistant['last_attempt'] = datetime.datetime.now() self.persistant['download_finished'] = '1' self.persistant['transfered'] = repr(total) self.persistant['size'] = repr(total) self.Store() out.write('Done\n') self.db.Log('Download of %s done' % self.persistant['guid']) return True def CopyLocalFile(self, datadir, out=sys.stdout): """CopyLocalFile -- copy a local file to the temporary directory, and treat it as if it was a download""" filename = self.TemporaryFilename(datadir, out=out) self.persistant['download_started'] = '1' self.Store() if self.persistant['url'] != filename: shutil.copyfile(self.persistant['url'], filename) self.persistant['download_finished'] = '1' size = os.stat(filename)[ST_SIZE] self.persistant['transfered'] = repr(size) self.persistant['size'] = repr(size) self.Store() self.db.Log('Download of %s done' % self.persistant['guid']) def Import(self, out=sys.stdout): """Import -- import a downloaded show into the MythTV user interface""" # Determine meta data self.db.Log('Importing %s' % self.persistant['guid']) datadir = self.db.GetSettingWithDefault('datadir', FLAGS.datadir) chanid = self.db.GetSetting('chanid') filename = '%s/%s' %(datadir, self.persistant['filename']) out.write('Importing %s\n' % filename) videodir = utility.GetVideoDir(self.db) vid = video.MythNetTvVideo(self.db, filename) # Try to use the publish time of the RSS entry as the start time... # The tuple will be in the format: 2003, 8, 6, 20, 43, 20 try: tuple = eval(self.persistant['parsed_date']) start = datetime.datetime(tuple[0], tuple[1], tuple[2], tuple[3], tuple[4], tuple[5]) except: start = datetime.datetime.now() # Ensure uniqueness for the start time interval = datetime.timedelta(seconds = 1) while not self.db.GetOneRow('select basename from recorded where ' 'starttime = %s and chanid = %s and ' 'basename != "%s"' \ %(self.db.FormatSqlValue('', start), chanid, filename)) == None: start += interval # Determine the duration of the video duration = datetime.timedelta(seconds = 60) try: duration = datetime.timedelta(seconds = vid.Length()) except video.LengthException, e: out.write('Could not determine the real length of the video.\n' 'Instead we will just pretend its only one minute long.\n\n' '%s\n' % e) finish = start + duration # Archive the original version of the video archiverow = self.db.GetOneRow('select * from mythnettv_archive ' 'where title="%s"' % self.persistant['title']) if archiverow: archive_location = ('%s/%s_%s' %(archiverow['path'], SafeForFilename(self.persistant['title']), SafeForFilename(self.persistant['subtitle']))) out.write('Possible archive location: %s\n' % archive_location) if self.persistant['url'].startswith(archiverow['path']): out.write('File was imported from the archive location, ' 'not archiving\n') elif not os.path.exists(archive_location): out.write('Archiving the original\n') shutil.copyfile(filename, archive_location) else: out.write('Archive destination already exists\n') # Transcode file to a better format if needed. transcoded is the filename # without the data directory portion if vid.NeedsTranscode(out=out): transcoded = vid.Transcode(datadir, out=out) os.remove(filename) else: re_justfilename = re.compile('^(.*)/(.+?)$') m = re_justfilename.match(filename) if m: transcoded = m.group(2) else: transcoded = filename out.write('Importing video %s...\n' % self.persistant['guid']) epoch = time.mktime(datetime.datetime.now().timetuple()) shutil.move('%s/%s' %(datadir, transcoded), '%s/%d_%s' %(videodir, epoch, transcoded)) # The quotes are missing around the description, because they are added # by the FormatSqlValue() call self.db.ExecuteSql('insert into recorded (chanid, starttime, endtime, ' 'title, subtitle, description, hostname, basename, ' 'progstart, progend, filesize) values ' '(%s, %s, %s, %s, ' '%s, %s, "%s", "%d_%s", %s, %s, %s)' %(chanid, self.db.FormatSqlValue('', start), self.db.FormatSqlValue('', finish), self.db.FormatSqlValue('', self.persistant['title']), self.db.FormatSqlValue('', self.persistant['subtitle']), self.db.FormatSqlValue('', self.persistant['description']), socket.gethostname(), epoch, transcoded, self.db.FormatSqlValue('', start), self.db.FormatSqlValue('', finish), self.db.FormatSqlValue('', self.persistant['size']))) # If there is a category set for this subscription, then set that as well row = self.db.GetOneRow('select * from mythnettv_category where ' 'title="%s";' % self.persistant['title']) if row: out.write('Setting category to %s\n' % row['category']) self.db.ExecuteSql('update recorded set category="%s" where ' 'basename="%d_%s";' %(row['category'], epoch, transcoded)) # Ditto the group row = self.db.GetOneRow('select * from mythnettv_group where ' 'title="%s";' % self.persistant['title']) if row: out.write('Setting recording group to %s\n' % row['recgroup']) self.db.ExecuteSql('update recorded set recgroup="%s" where ' 'basename="%d_%s";' %(row['recgroup'], epoch, transcoded)) if FLAGS.commflag: out.write('Rebuilding seek table\n') commands.getoutput('mythcommflag --rebuild --file "%s"' \ % videodir + '/' + transcoded) print 'Adding commercial flag job to backend queue' commands.getoutput('mythcommflag --queue --file "%s"' \ % videodir + '/' + transcoded) self.SetImported() out.write('Done\n\n') # And now mark the video as imported return def SetImported(self): """SetImported -- flag this program as having been imported""" self.persistant['download_finished'] = 1 self.persistant['imported'] = 1 self.Store() def SetNew(self): """SetNew -- make a program look like its new""" for field in ['download_started', 'download_finished', 'imported', 'filename', 'inactive', 'attempts', 'failed']: self.persistant[field] = None self.Store() def SetAttempts(self, count): """SetAttempts -- set the attempt count""" self.persistant['attempts'] = count self.Store() mythnettv-release-7/proxyhandler.py0000644000175000017500000000765111135202764016761 0ustar mikalmikal#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2008 # Released under the terms of the GNU GPL v2 # A simple wrapper around urllib2 which is used for all HTTP access. This # provides a central place for implementing the HTTP proxy functionality. import re import sys import urllib2 import database import utility class HttpHandler(object): """ Download HTTP content, possibly using a HTTP proxy as defined in the database. """ def __init__(self, db): self.db = db self.http_proxy = None self.budget = 0 def UsedProxy(self): """ Return the proxy which was used for the most recent download. """ return self.http_proxy def LookupProxy(self, url): """ Determine if a proxy should be used for a given URL. """ # Strip off the http:// if present if url.startswith('http://'): url = url[7:] for row in self.db.GetRows('select * from mythnettv_proxies;'): url_re = re.compile(row['url']) m = url_re.match(url) if m: return (row['http_proxy'], row['daily_budget']) return (None, None) def Open(self, url, force_proxy=None, force_budget=-1, out=sys.stdout): """ Return a file like object for the HTTP access, possibly using a proxy in the process. """ request = urllib2.Request(url) request.add_header('User-Agent', 'MythNetTV http://www.stillhq.com/mythtv/mythnettv/') (self.http_proxy, self.budget) = self.LookupProxy(url) if force_proxy: self.http_proxy = force_proxy if force_budget != -1: self.budget = force_budget opener = None if self.http_proxy: out.write('Using proxy %s for %s\n' %(self.http_proxy, url)) proxy_support = urllib2.ProxyHandler({'http': 'http://%s' % self.http_proxy}) opener = urllib2.build_opener(proxy_support) else: opener = urllib2.build_opener() return opener.open(request) def LogProxyUsage(self, bytes): """ Log how much we transferred through the proxy. """ # TODO(mikal): there is a potential bug here with the date rolling # over between these two statements, but it is very unlikely if self.http_proxy: self.db.ExecuteSql('insert ignore into mythnettv_proxy_usage ' '(day, http_proxy, bytes) values ' '(date(now()), "%s", 0);' % self.http_proxy) self.db.ExecuteSql('update mythnettv_proxy_usage ' 'set bytes = bytes + %d where ' 'day = date(now()) and http_proxy="%s";' %(bytes, self.http_proxy)) self.db.ExecuteSql('commit;') def ReportRecentProxyUsage(self, out=sys.stdout): """ Report on recent proxy usage """ for row in self.db.GetRows('select distinct(http_proxy) from ' 'mythnettv_proxy_usage;'): out.write('%s:\n' % row['http_proxy']) for subrow in self.db.GetRows('select * from mythnettv_proxy_usage ' 'where http_proxy="%s" ' 'order by day desc limit 7;' % row['http_proxy']): out.write(' %s = %s\n' %(subrow['day'], utility.DisplayFriendlySize(subrow['bytes']))) out.write('\n') def GetBudget(self): """ Return the current budget """ return self.budget def BudgetAllowsDownload(self, proposed_read_size): """ If I was to read proposed_read_size bytes, would I exceed the budget for this proxy? """ if not self.http_proxy: return True row = self.db.GetOneRow('select * from mythnettv_proxy_usage ' 'where http_proxy="%s" and day=date(now());' % self.http_proxy) if self.budget > row['bytes'] + proposed_read_size: return True return False mythnettv-release-7/proxyhandler_test.py0000711000175000017500000000503611135202764020006 0ustar mikalmikal#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2008 # Released under the terms of the GNU GPL v2 # Unit tests for mythnettv's proxyhandler module import gflags import os import sys import unittest import MySQLdb import database import testsetup import proxyhandler FLAGS = gflags.FLAGS class ProxyHandlerTest(unittest.TestCase): """ Test the proxy handling functionality. Note it is assumed in these tests that the testing database has already been setup by the mythnettv_test tests. """ def Cursor(self): """ Return a cursor to the test database """ db_connection = MySQLdb.connect(host = 'localhost', user = 'test', passwd = 'test', db = 'mythnettv_tests') return db_connection.cursor(MySQLdb.cursors.DictCursor) def setUp(self): # Clear out the proxies table so we start from a clean slate cursor = self.Cursor() cursor.execute('delete from mythnettv_proxies;') cursor.execute('commit;') # Now add one proxy cursor.execute('insert into mythnettv_proxies(url, http_proxy) values ' '("proxy\.stillhq\.com", "proxyhost.stillhq.com:3128");') cursor.execute('commit;') def testNoProxy(self): db = database.MythNetTvDatabase(dbname='mythnettv_tests', dbuser='test', dbpassword='test', dbhost='localhost') proxy = proxyhandler.HttpHandler(db) self.assertEquals(proxy.LookupProxy('www.stillhq.com/index.html'), (None, None)) self.assertEquals(proxy.LookupProxy('http://www.stillhq.com/index.html'), (None, None)) def testYesProxy(self): db = database.MythNetTvDatabase(dbname='mythnettv_tests', dbuser='test', dbpassword='test', dbhost='localhost') proxy = proxyhandler.HttpHandler(db) self.assertEquals(proxy.LookupProxy('proxy.stillhq.com/index.html'), ('proxyhost.stillhq.com:3128', None)) self.assertEquals(proxy.LookupProxy('http://proxy.stillhq.com/' 'index.html'), ('proxyhost.stillhq.com:3128', None)) if __name__ == "__main__": # Parse flags try: argv = FLAGS(sys.argv) except gflags.FlagsError, e: out.write('%s\n' % e) Usage(out) unittest.main() mythnettv-release-7/recordings_tool.py0000711000175000017500000001032511135202764017421 0ustar mikalmikal#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2008 # Released under the terms of the GNU GPL v2 # Helpers for manipulating the MythTV recordings table. Useful for testing # MythNetTV import datetime import os import sys import database import gflags import utility # Define command line flags FLAGS = gflags.FLAGS gflags.DEFINE_boolean('prompt', True, 'Prompt before deleting') def Usage(): print """Unknown command line. Try one of:' This is a helper script for the MythTV recordings table. Its useful for cleanup and inspection when testing MythNetTV. delete : delete all of the recordings with this title list : list all titles list <title> : list all the recordings with this title summary : print a summary of all recordings previously <title> : list previous recordings for this title rerecord <title> : allow re-records of this title rerecord <title> <subtitle> : allow re-records of this title / subtitle combination """ print '\n\nAdditionally, you can use these global flags:%s' % FLAGS sys.exit(1) if __name__ == '__main__': # Parse flags try: argv = FLAGS(sys.argv) except gflags.FlagsError, e: Usage() if len(argv) < 2: Usage() db = database.MythNetTvDatabase() if argv[1] == 'list': if len(argv) > 2: for row in db.GetRows('select * from recorded where title="%s" ' 'order by subtitle;' % argv[2]): start = row['starttime'] end = row['endtime'] length = end - start print row['subtitle'] print ' Recorded: %s' % start print ' Length: %.02f minutes' % (length.seconds / 60) print ' Video filename: %s' % row['basename'] print (' Video size: %s' % utility.DisplayFriendlySize(row['filesize'])) print else: for row in db.GetRows('select distinct(title), count(*), sum(filesize) ' 'from recorded group by title;'): print ('%s (%d recordings taking %s)' %(row['title'], row['count(*)'], utility.DisplayFriendlySize(row['sum(filesize)']))) elif argv[1] == 'delete': for row in db.GetRows('select * from recorded where title="%s" ' 'order by subtitle;' % argv[2]): print 'Deleting %s : %s' %(row['title'], row['subtitle']) if FLAGS.prompt: print 'Are you sure you want to delete this show?\n' confirm = raw_input('Type yes to do this: ') else: confirm = 'yes' if confirm == 'yes': db.ExecuteSql('delete from recorded where chanid=%s and starttime=%s;' %(row['chanid'], db.FormatSqlValue('starttime', row['starttime']))) os.unlink('%s/%s' %(utility.GetVideoDir(db), row['basename'])) print elif argv[1] == 'summary': for row in db.GetRows('select distinct(title), count(*), sum(filesize) ' 'from recorded group by title;'): print row['title'] size = utility.DisplayFriendlySize(row['sum(filesize)']) print ' %d recordings, %s' %(row['count(*)'], size) print elif argv[1] == 'previously': for row in db.GetRows('select distinct(subtitle) from oldrecorded ' 'where title="%s" order by subtitle;' % argv[2]): print row['subtitle'] allow_record = True for subrow in db.GetRows('select * from oldrecorded where title="%s" ' 'and subtitle="%s" order by starttime;' %(argv[2], row['subtitle'])): if subrow['duplicate'] != 0: allow_record = False print ' %s' % subrow['starttime'] if allow_record: print ' Re-record allowed' print elif argv[1] == 'rerecord': if len(argv) > 4: db.ExecuteSql('update oldrecorded set duplicate=0 where title="%s" ' 'and subtitle="%s";' %(argv[2], argv[3])) else: db.ExecuteSql('update oldrecorded set duplicate=0 where title="%s";' % argv[2]) else: print 'Unknown command' Usage() �����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/setup.py������������������������������������������������������������������������0000644�0001750�0001750�00000002435�11135202764�015375� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/usr/bin/python # mythnettv install script # Copyright (C) 2008, Thomas Mashos <thomas@weilandhomes.com> # # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA from distutils.core import setup import subprocess, glob, os.path setup( name="mythnettv", author="Michael Still", author_email="mikal@stillhq.com", url="http://www.stillhq.com/mythtv/mythnettv/", license="gpl", description="Plugin to download RSS video feeds for MythTV", data_files=[("share/mythnettv", glob.glob("*.py")), ("share/mythnettv", glob.glob("README*")), ("share/mythnettv", glob.glob("mythnettv*")), ("share/mythnettv", glob.glob("COPYING"))], ) �����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/syndication.py������������������������������������������������������������������0000644�0001750�0001750�00000014713�11135202764�016563� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2006, 2007, 2008 # Released under the terms of the GNU GPL v2 # Subscription functionality import datetime import feedparser import re import sys import database import gflags import mythnettvcore import program import utility import video # Define command line flags FLAGS = gflags.FLAGS complained_about_swf = False re_attributeparser = re.compile('([^=]*)="([^"]*)" *(.*)') def ParseAttributes(inputline): """ParseAttributes -- used to unmangle XML entity attributes""" line = inputline result = {} m = re_attributeparser.match(line) while m: result[m.group(1)] = m.group(2) line = m.group(3) m = re_attributeparser.match(line) return result def Download(db, url, guid, mime, title, subtitle, description, date, date_parsed, out=sys.stdout): """Download -- add a program to the list of waiting downloads""" prog = program.MythNetTvProgram(db) if prog.FromUrl(url, guid): out.write(' Creating program for %s: %s from %s\n\n' %(database.Normalize(title), database.Normalize(subtitle), guid)) elif FLAGS.verbose: out.write(' Already have %s: %s from %s\n' %(database.Normalize(title), database.Normalize(subtitle), guid)) # TODO(mikal): Consider using accessor methods here for info in ['download_started', 'download_finished', 'imported', 'inactive', 'attempts']: out.write(' %s: %s\n' %(info, prog.persistant.get(info, None))) out.write(' Bytes transferred: %s\n' % utility.DisplayFriendlySize(prog.persistant.get('transferred', 0))) for row in db.GetRows('select guid from mythnettv_programs ' 'where date is null;'): bad_program = program.MythNetTvProgram(db) bad_program.Load(row['guid']) bad_program.SetDate(datetime.datetime.now()) bad_program.Store() out.write('Program with guid = %s has invalid date, using now\n' % row['guid']) # Update program details prog.SetMime(mime) prog.SetShowInfo(title, subtitle, description, date, date_parsed) def GuessMimeType(url): """GuessMimeType -- guess a mime type based on a URL""" if url.endswith('m4v'): return 'video/x-m4v' return 'video/x-unguessable' def Sync(db, xmlfile, title, out=sys.stdout): """Sync -- sync up with an RSS feed""" global complained_about_swf # Grab the XML xmllines = xmlfile.readlines() # Modify the XML to work around namespace handling bugs in FeedParser lines = [] re_mediacontent = re.compile('(.*)<media:content([^>]*)/ *>(.*)') for line in xmllines: m = re_mediacontent.match(line) count = 1 while m: line = '%s<media:wannabe%d>%s</media:wannabe%d>%s' %(m.group(1), count, m.group(2), count, m.group(3)) m = re_mediacontent.match(line) count = count + 1 lines.append(line) # Parse the modified XML xml = ''.join(lines) parser = feedparser.parse(xml) # Find the media:content entries for entry in parser.entries: videos = {} description = entry.description subtitle = entry.title if entry.has_key('media_description'): description = entry['media_description'] # Enclosures if entry.has_key('enclosures'): for enclosure in entry.enclosures: try: videos[enclosure.type] = enclosure except: videos[GuessMimeType(enclosure.href)] = enclosure # Media:RSS for key in entry.keys(): if key.startswith('media_wannabe'): attrs = ParseAttributes(entry[key]) if attrs.has_key('type'): videos[attrs['type']] = attrs if attrs.has_key('title'): subtitle = attrs['title'] done = False if FLAGS.verbose: out.write(' Considering: %s: %s\n' %(title, subtitle)) for preferred in ['video/x-msvideo', 'video/mp4', 'video/x-xvid', 'video/wmv', 'video/x-ms-wmv', 'video/quicktime', 'video/x-m4v', 'video/x-flv', 'video/m4v', 'application/x-bittorrent', 'video/msvideo', 'video/vnd.objectvideo', 'video/ms-wmv', 'video/mpeg']: if not done and videos.has_key(preferred): Download(db, videos[preferred]['url'], entry.guid, preferred, title, subtitle, description, entry.date, entry.date_parsed, out=out) done = True if not done and videos.has_key('text/html'): db.Log('Warning: Treating text/html as an video enclosure type for ' '%s' % entry.guid) out.write('Warning: Treating text/html as an video enclosure from %s for' ' %s pointing to %s\n' %(repr(videos.keys()), subtitle, videos['text/html']['url'])) Download(db, videos['text/html']['url'], entry.guid, 'text/html', title, subtitle, description, entry.date, entry.date_parsed, out=out) done = True if not done and videos.has_key('application/x-shockwave-flash'): if not complained_about_swf: out.write('%s\n' % repr(videos)) out.write('Error: SWF is currently unsupported due to ffmpeg and mencoder not supporting compressed SWF files as input. Let mythnettv@stillhq.com know if you are aware of an open source way of transcoding these files.\n\n') complained_about_swf = True done = True if not done and len(videos.keys()) == 1: # If there is only one attachment, make the rather remarkable # assumption that it is a video out.write('Assuming that %s is a video format\n' % videos.keys()[0]) Download(db, videos[videos.keys()[0]]['url'], entry.guid, videos.keys()[0], title, subtitle, description, entry.date, entry.date_parsed, out=out) done = True if not done and videos: out.write('Error: Unsure which to prefer from: %s for %s\n [%s]\n\n' %(repr(videos.keys()), subtitle.encode('utf-8'), repr(videos))) if not done and FLAGS.verbose: out.write(' No downloadable content\n') �����������������������������������������������������mythnettv-release-7/syndication_test.py�������������������������������������������������������������0000711�0001750�0001750�00000010302�11135202764�017603� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2008 # Released under the terms of the GNU GPL v2 # Unit tests for mythnettv's syndication module import cStringIO import socket import sys import unittest import MySQLdb # This hackery is to get around "mythnettv" not having a .py at the end of its # filename import imp mythnettv = imp.load_module('mythnettv', open('mythnettv'), 'mythnettv', ('.py', 'U', 1)) import database import gflags import program import proxyhandler import mythnettvcore import syndication import testsetup FLAGS = gflags.FLAGS class SyndicationTests(unittest.TestCase): def Cursor(self): """ Return a cursor to the test database """ db_connection = MySQLdb.connect(host = 'localhost', user = 'test', passwd = 'test', db = 'mythnettv_tests') return db_connection.cursor(MySQLdb.cursors.DictCursor) def setUp(self): testsetup.SetupTestingDatabase(self.Cursor()) def testBadDates(self): """ testBadDates -- test a feed which has had date parsing problems in the past """ db = database.MythNetTvDatabase(dbname='mythnettv_tests', dbuser='test', dbpassword='test', dbhost='localhost') proxy = proxyhandler.HttpHandler(db) xmlfile = proxy.Open('http://www.stillhq.com/mythtv/mythnettv/testdata/' 'baddates.xml') output = cStringIO.StringIO() syndication.Sync(db, xmlfile, 'Bad Dates', out=output) xmlfile.close() class SyndicationTests(unittest.TestCase): """ The BoingBoing TV syndication file caused a bunch of problems, so it gets its own set of tests. """ def Cursor(self): """ Return a cursor to the test database """ db_connection = MySQLdb.connect(host = 'localhost', user = 'test', passwd = 'test', db = 'mythnettv_tests') return db_connection.cursor(MySQLdb.cursors.DictCursor) def setUp(self): testsetup.SetupTestingDatabase(self.Cursor()) def testImport(self): """ testImport -- make sure an import gives the expected results """ db = database.MythNetTvDatabase(dbname='mythnettv_tests', dbuser='test', dbpassword='test', dbhost='localhost') proxy = proxyhandler.HttpHandler(db) xmlfile = proxy.Open('http://www.stillhq.com/mythtv/mythnettv/testdata/' 'boingboing.xml') output = cStringIO.StringIO() syndication.Sync(db, xmlfile, 'Boing Boing', out=output) xmlfile.close() # 30 programs should have been created row = db.GetOneRow('select count(*) from mythnettv_programs where ' 'title="Boing Boing";') self.assertEqual(row['count(*)'], 30, 'There is the right number of programs') # Ok, so let's try to grab one of the troublesome programs prog = program.MythNetTvProgram(db) prog.Load('http://tv.boingboing.net/2008/09/03/' 'best-of-bbtv-david-b.html') self.assertEqual(prog.GetSubtitle(), 'Best of BBtv - David Byrne "Playing the Building."', 'Program has the wrong title') # Download it datadir = db.GetSettingWithDefault('datadir', '/tmp/testdata') # If we're running this test from Mikal's house, use the cache to speed it # up if socket.gethostname() != 'molokai': prog.Download(datadir) else: prog.Download(datadir, force_proxy='molokai.stillhq.com:3128', force_budget=1000000000) prog.Import() # Now make sure we have it cursor = self.Cursor() cursor.execute('select * from recorded where subtitle like "%Playing%";') self.assertEqual(cursor.rowcount, 1, "Wrong number of shows imported") if __name__ == "__main__": # Parse flags try: argv = FLAGS(sys.argv) except gflags.FlagsError, e: out.write('%s\n' % e) Usage(out) unittest.main() ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/testsetup.py��������������������������������������������������������������������0000644�0001750�0001750�00000013111�11135202764�016266� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2006, 2007, 2008 # Released under the terms of the GNU GPL v2 # Helpers for unit tests import datetime import os import sys import urllib import MySQLdb from socket import gethostname class TestSetupException(Exception): """ Test setup failed for some reason """ def SetupTestingDatabase(cursor): """ Create an empty database for testing """ tables = [] cursor.execute('show tables;') for row in cursor: tables.append(row[row.keys()[0]]) for table in tables: print 'Dropping old testing table: %s' % table cursor.execute('drop table %s;' % table) cursor.execute('commit;') # Create the needed MythTV channels cursor.execute("""CREATE TABLE `channel` ( `chanid` int(10) unsigned NOT NULL default '0', `channum` varchar(10) NOT NULL default '', `freqid` varchar(10) default NULL, `sourceid` int(10) unsigned default NULL, `callsign` varchar(20) NOT NULL default '', `name` varchar(64) NOT NULL default '', `icon` varchar(255) NOT NULL default 'none', `finetune` int(11) default NULL, `videofilters` varchar(255) NOT NULL default '', `xmltvid` varchar(64) NOT NULL default '', `recpriority` int(10) NOT NULL default '0', `contrast` int(11) default '32768', `brightness` int(11) default '32768', `colour` int(11) default '32768', `hue` int(11) default '32768', `tvformat` varchar(10) NOT NULL default 'Default', `commfree` tinyint(4) NOT NULL default '0', `visible` tinyint(1) NOT NULL default '1', `outputfilters` varchar(255) NOT NULL default '', `useonairguide` tinyint(1) default '0', `mplexid` smallint(6) default NULL, `serviceid` mediumint(8) unsigned default NULL, `atscsrcid` int(11) default NULL, `tmoffset` int(11) NOT NULL default '0', `atsc_major_chan` int(10) unsigned NOT NULL default '0', `atsc_minor_chan` int(10) unsigned NOT NULL default '0', PRIMARY KEY (`chanid`), KEY `channel_src` (`channum`,`sourceid`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1""") cursor.execute('commit;') # Create a fake MythTV settings table cursor.execute("""CREATE TABLE `settings` ( `value` varchar(128) NOT NULL default '', `data` text, `hostname` varchar(255) default NULL, KEY `value` (`value`,`hostname`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1""") cursor.execute('commit;') cursor.execute('insert into settings (value, data, hostname) ' 'values("RecordFilePrefix", "/tmp/testmyth", "%s");' % gethostname()) cursor.execute('commit;') # Create a fake recordings table cursor.execute("""CREATE TABLE `recorded` ( `chanid` int(10) unsigned NOT NULL default '0', `starttime` datetime NOT NULL default '0000-00-00 00:00:00', `endtime` datetime NOT NULL default '0000-00-00 00:00:00', `title` varchar(128) NOT NULL default '', `subtitle` varchar(128) NOT NULL default '', `description` text NOT NULL, `category` varchar(64) NOT NULL default '', `hostname` varchar(255) NOT NULL default '', `bookmark` tinyint(1) NOT NULL default '0', `editing` int(10) unsigned NOT NULL default '0', `cutlist` tinyint(1) NOT NULL default '0', `autoexpire` int(11) NOT NULL default '0', `commflagged` int(10) unsigned NOT NULL default '0', `recgroup` varchar(32) NOT NULL default 'Default', `recordid` int(11) default NULL, `seriesid` varchar(12) NOT NULL default '', `programid` varchar(20) NOT NULL default '', `lastmodified` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, `filesize` bigint(20) NOT NULL default '0', `stars` float NOT NULL default '0', `previouslyshown` tinyint(1) default '0', `originalairdate` date default NULL, `preserve` tinyint(1) NOT NULL default '0', `findid` int(11) NOT NULL default '0', `deletepending` tinyint(1) NOT NULL default '0', `transcoder` int(11) NOT NULL default '0', `timestretch` float NOT NULL default '1', `recpriority` int(11) NOT NULL default '0', `basename` varchar(128) NOT NULL default '', `progstart` datetime NOT NULL default '0000-00-00 00:00:00', `progend` datetime NOT NULL default '0000-00-00 00:00:00', `playgroup` varchar(32) NOT NULL default 'Default', `profile` varchar(32) NOT NULL default '', `duplicate` tinyint(1) NOT NULL default '0', `transcoded` tinyint(1) NOT NULL default '0', `watched` tinyint(4) NOT NULL default '0', PRIMARY KEY (`chanid`,`starttime`), KEY `endtime` (`endtime`), KEY `seriesid` (`seriesid`), KEY `programid` (`programid`), KEY `title` (`title`), KEY `recordid` (`recordid`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1""") cursor.execute('commit;') def DownloadTestData(filename): """ Download a file needed for a unit test, and store it in the testdata directory. """ if not os.path.exists('/tmp/testdata'): os.makedirs('/tmp/testdata') if not os.path.isdir('/tmp/testdata'): raise TestSetupException('/tmp/testdata is not a directory') if not os.path.exists('/tmp/testdata/%s' % filename): sys.stderr.write('Fetching %s\n' % filename) remote = urllib.urlopen('http://www.stillhq.com/mythtv/mythnettv/' 'testdata/%s' % filename) local = open('/tmp/testdata/%s' % filename, 'w') data = remote.read(1024 * 100) total = len(data) while data: sys.stderr.write(' %s ... %d bytes fetched\n' %(datetime.datetime.now(), total)) local.write(data) data = remote.read(1024 * 100) total += len(data) remote.close() local.close() sys.stderr.write(' %s ... %d total bytes fetched\n\n' %(datetime.datetime.now(), total)) �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/utility.py����������������������������������������������������������������������0000644�0001750�0001750�00000007703�11135202764�015743� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2008 # Released under the terms of the GNU GPL v2 # Simple utility methods import decimal import os import socket import types def DisplayFriendlySize(bytes): """DisplayFriendlySize -- turn a number of bytes into a nice string""" t = type(bytes) if t != types.LongType and t != types.IntType and t != decimal.Decimal: return 'NotANumber(%s=%s)' %(t, bytes) if bytes < 1024: return '%d bytes' % bytes if bytes < 1024 * 1024: return '%d kb (%d bytes)' %((bytes / 1024), bytes) if bytes < 1024 * 1024 * 1024: return '%d mb (%d bytes)' %((bytes / (1024 * 1024)), bytes) return '%d gb (%d bytes)' %((bytes / (1024 * 1024 * 1024)), bytes) def DisplayFriendlyTime(seconds): """DisplayFriendlyTime -- turn a number of seconds into a nice string""" if seconds < 60: return '%d seconds' % seconds if seconds < 60 * 60: minutes = seconds / 60 return '%d minutes, %d seconds' %(minutes, seconds - (minutes * 60)) hours = seconds / (60 * 60) minutes = (seconds - (hours * 60 * 60)) / 60 seconds = seconds - (hours * 60 * 60) - (minutes * 60) return '%d hours, %d minutes, %d seconds' %(hours, minutes, seconds) class FilenameException(Exception): """ Errors with filenames """ # The following is a way of simplifying the lookup code for video directoies. # The tuple format is (tablename, where, column) _VIDEODIRPATH = [('storagegroup', 'groupname="MythNetTV"', 'dirname'), ('storagegroup', 'groupname="Default"', 'dirname'), ('settings', 'value="RecordFilePrefix"', 'data')] def GetVideoDir(db): """GetVideoDir -- return the directory to store video in""" # Video directory lookup changes for the introduction of storage groups # in MythTV 0.21 # TODO(mikal): I wonder if this works with 0.21 properly? I should test. videodir = None for (table, where, column) in _VIDEODIRPATH: if db.TableExists(table): try: videodir = db.GetOneRow('select * from %s where ' '%s and hostname = "%s";' %(table, where, socket.gethostname()))[column] except: pass if videodir: break # Check we ended up with a video directory if videodir == None: raise FilenameException(db, 'Could not determine the video ' 'directory for this machine. Please report ' 'this to mythnettv@stillhq.com') # Check that it exists as well if not os.path.exists(videodir): raise FilenameException(db, 'MythTV is misconfigured. The video ' 'directory "%s" does not exist. Please create ' 'it, and then rerun MythNetTV.' % videodir) return videodir def ExplainVideoDir(db): """ExplainVideoDir -- return the directory to store video in""" videodir = None for (table, where, column) in _VIDEODIRPATH: if db.TableExists(table): try: print 'Checking %s for an entry where %s' %(table, where) for row in db.GetRows('select * from %s where %s;' %(table, where)): print 'Found %s for %s' %(row[column], row['hostname']) if row['hostname'] == socket.gethostname(): videodir = row[column] print 'Using this value' except Exception, e: print ' DB error: %s' % e else: print 'Skipped using table %s as it doesn\'t exist' % table if videodir: print 'Will use: %s' % videodir print print '** All entries below here are informational only **' print # Check we ended up with a video directory if videodir == None: print 'Found no video directory' return # Check that it exists as well if not os.path.exists(videodir): print 'Video directory does not exist on disk!' print 'End of checks' �������������������������������������������������������������mythnettv-release-7/utility_test.py�����������������������������������������������������������������0000711�0001750�0001750�00000004230�11135202764�016765� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2008 # Released under the terms of the GNU GPL v2 # Unit tests for mythnettv's primary user interface # Tests for the utility methods import unittest import MySQLdb import utility class DisplayFriendlySizeTest(unittest.TestCase): def testSmallUnchanged(self): """ Small values should be unchanged """ self.assertEqual(utility.DisplayFriendlySize(42), '42 bytes') def testKbWorks(self): """ Display values in kilobytes correctly """ fourtytwo_kb = 42 * 1024 self.assertEqual(utility.DisplayFriendlySize(fourtytwo_kb), '42 kb (%d bytes)' % fourtytwo_kb) def testMbWorks(self): """ Display values in megabytes correctly """ fourtytwo_mb = 42 * 1024 * 1024 self.assertEqual(utility.DisplayFriendlySize(fourtytwo_mb), '42 mb (%d bytes)' % fourtytwo_mb) def testGbWorks(self): """ Display values in gigabytes correctly """ fourtytwo_gb = 42 * 1024 * 1024 * 1024 self.assertEqual(utility.DisplayFriendlySize(fourtytwo_gb), '42 gb (%d bytes)' % fourtytwo_gb) def testTbWorks(self): """ Very large values just end up in gigabytes """ fourtytwo_tb = 42 * 1024 * 1024 * 1024 * 1024 self.assertEqual(utility.DisplayFriendlySize(fourtytwo_tb), '%d gb (%d bytes)' %(42 * 1024, fourtytwo_tb)) def testBadInput(self): """ What happens if you hand in something which isn't a number? """ self.assertEqual(utility.DisplayFriendlySize('banana'), 'NotANumber(<type \'str\'>=banana)') class DisplayFriendlyTimeTest(unittest.TestCase): def testSmallUnchanged(self): """ Small values should be unchanged """ self.assertEqual(utility.DisplayFriendlyTime(42), '42 seconds') def testMinutes(self): """ Minutes should work too """ self.assertEqual(utility.DisplayFriendlyTime(70), '1 minutes, 10 seconds') def testHours(self): """ Hours as well """ self.assertEquals(utility.DisplayFriendlyTime(3674), '1 hours, 1 minutes, 14 seconds') if __name__ == "__main__": unittest.main() ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/video.py������������������������������������������������������������������������0000644�0001750�0001750�00000016137�11135202764�015347� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������ #!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2006, 2007, 2008 # Released under the terms of the GNU GPL v2 import commands import datetime import os import re import sys from stat import * import gflags import utility # Define command line flags FLAGS = gflags.FLAGS # Exceptions returned by this module class LoggingException(Exception): """ Log exceptions to the database as well as returning them as exceptions """ def __init__(self, db, error): if db: try: db.Log(error) except: pass Exception.__init__(self, error) class TranscodeException(LoggingException): """ Transcoding problems """ class LengthException(Exception): """ Exception while determining the length of a video """ class ParseException(Exception): """ Exception while trying to parse video characteristics """ class MythNetTvVideo: """MythNetTvVideo -- video handling methods This uses mplayer to determine the length of the video. Specifically, this command line does the trick: mplayer -frames 0 -identify <filename> 2>&1 | grep ID_LENGTH """ def __init__(self, db, filename): """__init__ -- prime the pump to answer questions about the video""" self.db = db self.filename = filename self.values = {} if not os.path.exists(self.filename): raise ParseException('Video file missing') out = commands.getoutput('mplayer -vo null -frames 0 -identify %s 2>&1 | ' 'grep "="' \ % self.filename) for line in out.split('\n'): try: (key, value) = line.split('=') self.values[key] = value except: pass if len(self.values) == 0: raise ParseException('Could not parse video characteristics') def Length(self): """Length -- return the length of the video in seconds""" if 'ID_LENGTH' in self.values: return float(self.values['ID_LENGTH']) raise LengthException('Could not determine length of %s. ' 'Attributes found = %s' %(self.filename, self.values)) def NeedsTranscode(self, out=sys.stdout): """NeedsTranscode -- decide if a video needs transcoding before import""" # Doesn't need transcoding return_false = ['mp4v', '0x10000002', 'divx', 'DIVX', 'XVID', 'DX50'] # Does need transcoding # Note that some avc1 videos work, and some don't -- so all get transcoded return_true = ['avc1', 'theo', 'WMV2', 'FLV1'] if self.values['ID_VIDEO_FORMAT'] in return_false: out.write('Files in format %s don\'t need transcoding\n' % self.values['ID_VIDEO_FORMAT']) return False if self.values['ID_VIDEO_FORMAT'] in return_true: out.write('Files in format %s do need transcoding\n' % self.values['ID_VIDEO_FORMAT']) return True else: out.write(""" **************************************************************** I don't know if we need to transcode videos in %s format I'm going to give it a go without, and it if doesn't work please report it to mikal@stillhq.com **************************************************************** """ % self.values['ID_VIDEO_FORMAT']) return False def NewFilename(self, datadir, extn, out=sys.stdout): """NewFilename -- determine what filename to use after transcoding""" # We only return the filename portion, not the path re_filename = re.compile('^(.+)/(.+?)$') m = re_filename.match(self.filename) if m: file = m.group(2) else: file = self.filename # Try changing the extension to extn re_extension = re.compile('(.*)\.(.*?)') m = re_extension.match(file) if m: count = 1 proposed_filename = '%s.%s' % (m.group(1), extn) while os.path.exists('%s/%s' %(datadir, proposed_filename)): out.write('Rejected %s as it already exists\n' % proposed_filename) count += 1 proposed_filename = '%s_%d.%s' %(m.group(1), count, extn) return proposed_filename return 'new-%s' % file def Transcode(self, datadir, out=sys.stdout): """Transcode -- transcode the video to a better format. Returns the new filename. """ # If the file is small, go for a format which will hopefully look nicer out.write('Transcoding\n') format = '-ovc lavc -oac lavc -lavcopts abitrate=128 -ffourcc DX50' newfilename = self.NewFilename(datadir, 'avi', out=out) start_size = os.stat(self.filename)[ST_SIZE] command = 'mencoder %s %s -o %s/%s' %(self.filename, format, datadir, newfilename) (status, out) = commands.getstatusoutput(command) if status != 0: raise TranscodeException(self.db, 'Transcode failed: %s\n%s\nCommand: %s' %(status, out, command)) # Log the file growth if self.db: self.db.Log('Transcoding changed size of file from %d to %d' \ %(start_size, os.stat('%s/%s' %(datadir, newfilename))[ST_SIZE])) return newfilename def Usage(): print """Unknown command line. Try one of:' The MythNetTV video subsystem may be queried directly. This can be useful if you want to perform simple operations such as asking if a given video file needs to be transcoded before being imported into MythTV, or actually performing the transcode. length <path> : return the length of a video in seconds info <path> : return all available information for a given video needstranscode <path> : determine if a given file needs to be transcoded before being imported into MythTV transcode <path> <output path> : transcode the file into a format suitable for MythTV """ print '\n\nAdditionally, you can use these global flags:%s' % FLAGS sys.exit(1) if __name__ == '__main__': # Parse flags try: argv = FLAGS(sys.argv) except gflags.FlagsError, e: out.write('%s\n' % e) Usage(out) # Present a simple user interface to query the video subsystem filename = None try: filename = argv[2] if not os.path.exists(filename): print '%s: file not found' % filename Usage() except: print 'Could not find a file argument' Usage() # Construct a video object vid = None try: vid = MythNetTvVideo(None, filename) except Exception, e: print 'Video processing error: %s' % e sys.exit(1) if argv[1] == 'length': try: print 'Length of %s: %s' %(sys.argv[2], utility.DisplayFriendlyTime(vid.Length())) except Exception, e: print 'Length error: %s' % e sys.exit(1) elif argv[1] == 'info': for key in vid.values: print '%s: %s' %(key, vid.values[key]) elif argv[1] == 'needstranscode': print 'Needs transcoding: %s' % vid.NeedsTranscode() elif argv[1] == 'transcode': if vid.NeedsTranscode(): print ('Created output file: %s/%s' %(argv[3], vid.Transcode(argv[3]))) else: print 'Unknown command' Usage() ���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/video_test.py�������������������������������������������������������������������0000711�0001750�0001750�00000005031�11135202764�016370� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/usr/bin/python # Copyright (C) Michael Still (mikal@stillhq.com) 2006, 2007, 2008 # Released under the terms of the GNU GPL v2 # Unit tests for mythnettv's video module import gflags import os import sys import unittest import testsetup import video FLAGS = gflags.FLAGS class VideoTest(unittest.TestCase): def setUp(self): """ Download required testing files """ for filename in ['video_length_fail.mov', 'video_length_works.avi', 'video_notavideo.mp4']: testsetup.DownloadTestData(filename) def testVideoInit(self): vid = video.MythNetTvVideo(None, '/tmp/testdata/video_length_fail.mov') def testVideoLengthError(self): """ Make sure the right thing happens when we fail to get the length of a video """ try: vid = video.MythNetTvVideo(None, '/tmp/testdata/video_length_fail.mov') vid.Length() except video.LengthException, e: # This is the exception we want return self.fail() def testVideoLengthNotAVideo(self): """ Some sites don't return a video file, for example when your IP isn't in the country they allow downloads from. Make sure the right thing. """ try: vid = video.MythNetTvVideo(None, '/tmp/testdata/video_notavideo.mp4') vid.Length() except video.ParseException, e: # This is the exception we want return self.fail() def testVideoLengthWorks(self): """ This one should work """ vid = video.MythNetTvVideo(None, '/tmp/testdata/video_length_works.avi') len = vid.Length() self.assertEquals(int(len), 161) def testNeedsTranscode(self): """ A very simple test of whether to transcode a file """ vid = video.MythNetTvVideo(None, '/tmp/testdata/video_length_works.avi') needs = vid.NeedsTranscode() self.assertEquals(needs, False) def testEnsureNewFilenameUnique(self): """ When we generate a new filename, it should not be the same as an existing filename """ vid = video.MythNetTvVideo(None, '/tmp/testdata/video_length_works.avi') new_name = vid.NewFilename('/tmp/testdata', 'avi') if os.path.exists('/tmp/testdata/%s' % new_name): self.fail() def testTranscode(self): """ Do a sample transcode """ vid = video.MythNetTvVideo(None, '/tmp/testdata/video_length_works.avi') vid.Transcode('/tmp/testdata') if __name__ == "__main__": # Parse flags try: argv = FLAGS(sys.argv) except gflags.FlagsError, e: out.write('%s\n' % e) Usage(out) unittest.main() �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/mythnettv�����������������������������������������������������������������������0000755�0001750�0001750�00000057360�11135202764�015662� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/usr/bin/python # This script requires that mplayer be installed # Copyright (C) Michael Still (mikal@stillhq.com) 2006, 2007, 2008 # Released under the terms of the GNU GPL v2 # Latest source is always at http://www.stillhq.com/mythtv/mythnettv/ import commands import datetime import feedparser import os import re import shutil import subprocess import sys import tempfile import time import types import database import gflags import program import proxyhandler import syndication import mythnettvcore import utility import video __author__ = 'Michael Still (mikal@stillhq.com)' __version__ = 'Release 6' # Define command line flags FLAGS = gflags.FLAGS gflags.DEFINE_string('datadir', '', 'The location of the data directory. Will change the ' 'previous value') gflags.DEFINE_string('defaultuploadrate', '', 'The default bittorrent upload rate. If set will ' 'change the previous value') gflags.DEFINE_boolean('prompt', True, 'Prompt for user input when required') gflags.DEFINE_boolean('promptforannounce', True, 'Should the user be prompted about subscribing to the ' 'announce video feed?') def GetPossible(array, index): """GetPossible -- get a value from an array, handling its absence nicely""" try: return array[index] except: return None def Usage(out): out.write("""Unknown command line. Try one of: (manual usage) url <url> <title> : to download an RSS feed and load the shows from it into the TODO list. The title is as the show title in the MythTV user interface file <url> <title>: to do the same, but from a file, with a show title like url above download <num> : to download that number of shows from the TODO list. We download some of the oldest first, and then grab some of the newest as well. download <num> <title filter> : the same as above, but filter to only download shows with a title exactly matching the specified filter download <num> <title filter> <subtitle filter> : the same as above, but with a regexp title filter as well download <num> <title filter> <subtitle filter> justone : the same as above, but download just one and then mark all other matches as read cleartodo : permanently remove all items from the TODO list markread <num> : interactively mark some of the oldest <num> shows as already downloaded and imported markread <num> <title filter> : the same as above, but filter to only mark shows with a title exactly matching the specified filter markread <num> <title filter> <subtitle filter> : the same as above, but with a regexp title filter as well markunread <num> : interactively mark some of the youngest <num> shows as not already downloaded and imported resetattempts : interactively reset the number of attempts for matching programs to zero. This will cause previously failed programs to be retried resetattempts <title filter> : as above, but only for shows with this title (handy stuff) todoremote : add a remote URL to the TODO list. This will prompt for needed information about the video, and set the date of the program to now todoremote <url> <title> <subtitle> <description> : the same as above, but don't prompt for anything importremote : download and immediately import the named URL. Will prompt for needed information importremote <url> <title> <subtitle> <description> : the same as above, but don't prompt for anything importtorrent <url> <title> <subtitle> <description> : the same as above, but force the URL to be treated as a torrent. This is useful when MythNetTV doesn't automatically detect that the URL is to a torrent file. importlocal <file>: import the named file, using the title, subtitle and description from the command line. The file will be left on disk. importlocal <file>: import the named file. The file will be left on disk. Will prompt for needed information importmanylocal <path> <regexp> <title>: import all the files from path matching regexp. title is use as the title for the program, and the filename is used as the subtitle (subscription management) subscribe <url> <title> : subscribe to a URL, and specify the show title list : list subscriptions unsubscribe <url> <title> : unsubscribe from a feed, and remove feed programs from TODO list update : add new programs from subscribed URLs to the TODO list update <title> : as above, but just for this program (things you can do to subscriptions) archive <title> <path> : archive all programs with this title to the specified path. This is useful for shows you download and import, but want to build a non-MythTV archive of as well category <title> <category> : set the category for a given program. The category is used for parental filtering within MythTV. group <title> <group> : set the group for a given program. The group is used as a recording group within MythTV. http_proxy <url regexp> <proxy> : you can choose to use a HTTP proxy for URL requests matching a given regular expression. Use this command to define such an entry. This might be handy if some of the programs you wish to subscribe to are only accessible over a VPN. http_proxy <url regexp> <proxy> <budget mb> : the same a s above, but you can specify the maximum number of megabytes to download via the proxy in a given day. To see proxy usage information, use the proxyusage command. (reporting) statistics : show some simple statistics about MythNetTV log : dump the current internal log entries nextdownload <num> : if you executed download <num>, what would be downloaded? nextdownload <num> <title filter> : as above, but only for the specified title proxyusage : print a simple report on HTTP proxy usage over the last seven days (debugging) videodir : show where MythNetTV thinks it should be placing video files explainvideodir : verbosely explain why that video directory was selected. This can help debug when the wrong video directory is being used, or no video directory at all is found """) out.write('\n\nAdditionally, you can use these global flags:%s\n' % FLAGS) sys.exit(1) def main(argv, out=sys.stdout): # Parse flags try: argv = FLAGS(argv) except gflags.FlagsError, e: out.write('%s\n' % e) Usage(out) db = database.MythNetTvDatabase() # Update the data directory is set if FLAGS.datadir: db.WriteSetting('datadir', FLAGS.datadir) print 'Updated the data directory to %s' % FLAGS.datadir # Update the default upload rate for bittorrent if set if FLAGS.defaultuploadrate: db.WriteSetting('uploadrate', FLAGS.defaultuploadrate) print ('Updated default bittorrent upload rate to %s' % FLAGS.defaultuploadrate) # Make sure the data directory exists datadir = db.GetSettingWithDefault('datadir', FLAGS.datadir) if not os.path.exists(datadir): out.write('You need to create the configured data directory at "%s"\n' % datadir) sys.exit(1) # Ask if we can subscribe the user to the announcement video feed announce_prompted = db.GetSettingWithDefault('promptedforannounce', not FLAGS.promptforannounce) if not announce_prompted: row = db.GetOneRow('select * from mythnettv_subscriptions where ' 'title="MythNetTV announcements";') if not row and FLAGS.promptforannounce: out.write('You are not currently subscribed to the MythNetTV\n' 'announcements video feed. This feed is very low volume\n' 'and is used to announce important things like new releases\n' 'and major bug fixes.\n\n' 'Would you like to be subscribed? You will only be asked\n' 'this once.\n\n') confirm = raw_input('Type yes to subscribe: ') if confirm == 'yes': mythnettvcore.Subscribe('http://www.stillhq.com/mythtv/mythnettv/' 'announce/feed.xml', 'MythNetTV announcements') out.write('Subscribed to MythNetTV announcements\n') db.WriteSetting('promptedforannounce', True) # TODO(mikal): This command line processing is ghetto if len(argv) == 1: out.write('Invalid command line: %s\n' % ' '.join(argv)) Usage(out) if argv[1] == 'url': # Go and grab the XML file from the remote HTTP server, and then parse it # as an RSS feed with enclosures. Populates a TODO list in # mythnettv_programs proxy = proxyhandler.HttpHandler(db) xmlfile = proxy.Open(argv[2], out=out) syndication.Sync(db, xmlfile, argv[3], out=out) elif argv[1] == 'file': # Treat the local file as an RSS feed. Populates a TODO list in # mythnettv_programs xmlfile = open(argv[2]) syndication.Sync(db, xmlfile, argv[3]) elif argv[1] == 'download': # Download the specified number of programs and import them into MythTV filter = GetPossible(argv, 3) subtitle_filter = GetPossible(argv, 4) just_one = GetPossible(argv, 5) == 'justone' success = False previous = [] for guid in mythnettvcore.NextDownloads(argv[2], filter, subtitle_filter, out=out): if not just_one: mythnettvcore.DownloadAndImport(db, guid, out=out) elif just_one and not success: success = mythnettvcore.DownloadAndImport(db, guid, out=out) if not success: previous.append(guid) else: prog = program.MythNetTvProgram(db) prog.Load(guid) out.write('Skipping because we already have one\n') out.write(' %s: %s\n' %(prog.GetTitle(), prog.GetSubtitle())) out.write(' (%s)\n\n' % prog.GetDate()) prog.SetImported() prog.Store() # If some failed before we had one work, then we now need to mark them # read if we're fetching just one if just_one and previous: for guid in previous: prog = program.MythNetTvProgram(db) prog.Load(guid) out.write('Skipping because we already have one\n') out.write(' %s: %s\n' %(prog.GetTitle(), prog.GetSubtitle())) out.write(' (%s)\n\n' % prog.GetDate()) prog.SetImported() prog.Store() # And now make sure there aren't any stragglers for guid in db.GetWaitingForImport(): prog = program.MythNetTvProgram(db) prog.Load(guid) try: prog.Import(out=out) except: out.write('Couldn\'t import straggling %s, ' 'removing it from the queue\n' % guid) prog.SetImported() prog.Store() elif argv[1] == 'todoremote': # Add a remote URL to the TODO list. We have to prompt for a bunch of # stuff because we don't have a "real" RSS feed prog = program.MythNetTvProgram(db) url = GetPossible(argv, 2) title = GetPossible(argv, 3) subtitle = GetPossible(argv, 4) description = GetPossible(argv, 5) prog.FromInteractive(url, title, subtitle, description) elif argv[1] == 'importremote' or argv[1] == 'importtorrent': # Download a remote file and then import it as a program. We have to # prompt for details here, because this didn't come from a "real" RSS feed prog = program.MythNetTvProgram(db) url = GetPossible(argv, 2) title = GetPossible(argv, 3) subtitle = GetPossible(argv, 4) description = GetPossible(argv, 5) # TODO(mikal) # It is _possible_ that this show has already been created and # that this is an attempt to restart a download. Check. prog.FromInteractive(url, title, subtitle, description) prog.Store() if argv[1] == 'importtorrent': prog.SetMime('application/x-bittorrent') if prog.Download(db.GetSettingWithDefault('datadir', FLAGS.datadir), out=out) == True: prog.Import(out=out) elif argv[1] == 'importmanylocal': # Import files matching regexp from path with the title path = argv[2] regexp = argv[3] title = argv[4] rxp = re.compile(regexp) for ent in os.listdir(path): if os.path.isfile('%s/%s' %(path, ent)): out.write('\nConsidering %s\n' % ent) m = rxp.match(ent) if m: prog = program.MythNetTvProgram(db) prog.SetUrl(ent) prog.FromInteractive('%s/%s' %(path, ent), title, ent, '-') prog.CopyLocalFile(db.GetSettingWithDefault('datadir', FLAGS.datadir), out=out) prog.Import(out=out) elif argv[1] == 'importlocal': # Take a local file, copy it to the temporary directory, and then import # it as if we had downloaded it prog = program.MythNetTvProgram(db) prog.SetUrl(argv[2]) url = GetPossible(argv, 2) title = GetPossible(argv, 3) subtitle = GetPossible(argv, 4) description = GetPossible(argv, 5) prog.FromInteractive(url, title, subtitle, description) prog.CopyLocalFile(db.GetSettingWithDefault('datadir', FLAGS.datadir), out=out) prog.Import(out=out) elif argv[1] == 'subscribe': # Subscribe to an RSS feed mythnettvcore.Subscribe(argv[2], argv[3]) out.write('Subscribed to %s\n' % argv[3]) elif argv[1] == 'list': # List subscribed RSS feeds proxy = proxyhandler.HttpHandler(db) for row in db.GetRows('select distinct(title) from ' 'mythnettv_subscriptions order by title'): out.write('%s\n' % row['title']) for subrow in db.GetRows('select * from mythnettv_subscriptions where ' 'title="%s";' % row['title']): out.write(' - %s' % subrow['url']) (proxy_host, budget) = proxy.LookupProxy(subrow['url']) if proxy_host: out.write(' (proxy: %s' % proxy_host) if budget: out.write(' budget %s)' % utility.DisplayFriendlySize(budget)) else: out.write(')') out.write('\n') if subrow['inactive'] == 1: out.write(' (inactive)\n') subrow = db.GetOneRow('select * from mythnettv_archive ' 'where title="%s";' % row['title']) if subrow: out.write(' Archived to %s\n' % subrow['path']) subrow = db.GetOneRow('select * from mythnettv_category ' 'where title="%s";' % row['title']) if subrow: out.write(' Category is %s\n' % subrow['category']) subrow = db.GetOneRow('select * from mythnettv_group ' 'where title="%s";' % row['title']) if subrow: out.write(' Group is %s\n' % subrow['recgroup']) out.write('\n') elif argv[1] == 'unsubscribe': # Remove a subscription to an RSS feed if db.ExecuteSql('update mythnettv_subscriptions set inactive=1 ' 'where url = "%s";' % argv[2]) == 0: print 'No subscriptions with this URL found!' if db.ExecuteSql('update mythnettv_programs set inactive=1 ' 'where title = "%s";' % argv[3]) == 0: print 'No subscriptions with this title found!' elif argv[1] == 'update': # Update the TODO list based on subscriptions title = GetPossible(argv, 2) mythnettvcore.Update(out, title) elif argv[1] == 'archive': db.WriteOneRow('mythnettv_archive', 'title', {'title': argv[2], 'path': argv[3]}) elif argv[1] == 'category': db.WriteOneRow('mythnettv_category', 'title', {'title': argv[2], 'category': argv[3]}) elif argv[1] == 'group': db.WriteOneRow('mythnettv_group', 'title', {'title': argv[2], 'recgroup': argv[3]}) elif argv[1] == 'http_proxy': budget = GetPossible(argv, 4) if budget: budget = int(budget) * 1024 * 1024 out.write('Setting proxy budget to %d bytes\n' % budget) db.WriteOneRow('mythnettv_proxies', 'url', {'url': argv[2], 'http_proxy': argv[3], 'daily_budget': budget}) elif argv[1] == 'statistics': # Display some simple stats about the state of MythMYTHNETTV row = db.GetOneRow('select count(guid) from mythnettv_programs;') out.write('Programs tracked: %d\n' % row['count(guid)']) for show in db.GetRows('select distinct(title) from mythnettv_programs ' 'where title is not NULL;'): row = db.GetOneRow('select count(guid) from mythnettv_programs where ' 'title = "%s";' % show['title']) out.write(' %s: %d\n' %(show['title'], row['count(guid)'])) out.write('\n') row = db.GetOneRow('select count(guid) from mythnettv_programs where ' 'download_finished is NULL and title is not NULL ' 'and inactive is NULL;') out.write('Programs still to download: %d\n' % row['count(guid)']) for show in db.GetRows('select distinct(title) from mythnettv_programs ' 'where title is not NULL and ' 'inactive is NULL and ' 'download_finished is NULL;'): row = db.GetOneRow('select count(guid) from mythnettv_programs where ' 'title = "%s" and inactive is NULL ' 'and download_finished is NULL;' % show['title']) out.write(' %s: %d\n' %(show['title'], row['count(guid)'])) out.write('\n') try: row = db.GetOneRow('select sum(transfered) from mythnettv_programs;') out.write('Data transferred: %s\n' % (utility.DisplayFriendlySize(int(row['sum(transfered)'])))) except: # TODO(mikal): I am sure there is a better way of doing this dummy = 'blah' elif argv[1] == 'log': for logline in db.GetRows('select * from mythnettv_log order by ' 'sequence asc;'): out.write('%s %s\n' %(logline['timestamp'], logline['message'])) elif argv[1] == 'nextdownload': filter = GetPossible(argv, 3) subtitle_filter = GetPossible(argv, 4) for guid in mythnettvcore.NextDownloads(argv[2], filter, subtitle_filter, out=out): out.write('%s\n' % guid) prog = program.MythNetTvProgram(db) prog.Load(guid) out.write(' %s: %s\n\n' %(prog.GetTitle(), prog.GetSubtitle())) elif argv[1] == 'cleartodo': out.write("""The command you are executing will permanently remove all shows from the TODO list, as well as any record of shows which have already been downloaded. Basically you\'ll be back at the start again, although your preferences will remain set. Are you sure you want to do this?\n\n""") confirm = raw_input('Type yes to do this: ') if confirm == 'yes': db.ExecuteSql('delete from mythnettv_programs;') out.write('Deleted\n') elif argv[1] == 'markread': filter = GetPossible(argv, 3) subtitle_filter = GetPossible(argv, 4) for guid in mythnettvcore.NextDownloads(argv[2], filter, subtitle_filter): prog = program.MythNetTvProgram(db) prog.Load(guid) out.write(' %s: %s\n' %(prog.GetTitle(), prog.GetSubtitle())) out.write(' (%s)\n\n' % prog.GetDate()) if FLAGS.prompt: out.write('Are you sure you want to mark this show as downloaded?' '\n\n') confirm = raw_input('Type yes to do this: ') else: confirm = 'yes' if confirm == 'yes': prog.SetImported() prog.Store() out.write('Done\n') out.write('\n') elif argv[1] == 'markunread': for row in db.GetRows('select guid from mythnettv_programs where ' 'download_finished=1 or imported=1 or ' 'attempts is not null or failed=1 ' 'order by date desc limit %d;' % int(argv[2])): prog = program.MythNetTvProgram(db) prog.Load(row['guid']) out.write(' %s: %s\n' %(prog.GetTitle(), prog.GetSubtitle())) out.write(' (%s)\n\n' % prog.GetDate()) if FLAGS.prompt: out.write('Are you sure you want to mark this show as not downloaded?' '\n\n') confirm = raw_input('Type yes to do this: ') else: confirm = 'yes' if confirm == 'yes': prog.SetNew() prog.Store() out.write('Done\n') out.write('\n') elif argv[1] == 'resetattempts': filter = GetPossible(argv, 2) if filter: filter_sql = 'and title="%s"' % filter else: filter_sql = '' for row in db.GetRows('select * from mythnettv_programs where ' 'attempts > 0 and download_finished is null %s;' % filter_sql): prog = program.MythNetTvProgram(db) prog.Load(row['guid']) out.write(' %s: %s\n' %(prog.GetTitle(), prog.GetSubtitle())) out.write(' (%s)\n\n' % prog.GetDate()) if FLAGS.prompt: out.write('Are you sure you want to reset the attempt count for this ' 'show?\n\n') confirm = raw_input('Type yes to do this: ') else: confirm = 'yes' if confirm == 'yes': prog.SetAttempts(0) prog.Store() out.write('Done\n') out.write('\n') elif argv[1] == 'proxyusage': proxy = proxyhandler.HttpHandler(db) proxy.ReportRecentProxyUsage(out=out) elif argv[1] == 'videodir': try: print 'Video directory is: %s' % utility.GetVideoDir(db) except utility.FilenameException: print 'No video directory found! Use explainvideodir to debug' elif argv[1] == 'explainvideodir': utility.ExplainVideoDir(db) else: Usage(out) if __name__ == "__main__": main(sys.argv) ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/README��������������������������������������������������������������������������0000644�0001750�0001750�00000006062�11135202764�014543� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������I've been recently wanting a video blog aggregator which integrates with MythTV. Specifically, I wanted the videos to appear as recordings. I've had a horrible script for that for a few months, but ended up cleaning it up when it occurred to me that Akimbo (http://www.akimbo.com) is just a per user RSS feed, and something a lot like MythTV. So, I wrote MythNetTV. It takes RSS feeds, imports the entries in the feed and builds a TODO list. It then downloads shows and imports them into the MythTV recordings menu. This implementation has advantages over other RSS readers for MythTV in that it: - ignores the content of the post -- it's all about the video files - queues downloads, instead of making them when feeds are added, or new shows are posted (this stops you from running out of disk, or having a huge show backlog) The code has now been used by a variety of people, but its possible there are still bugs. If you want to give it a try, please take good backups. You can find the code here: http://www.stillhq.com/mythtv/mythnettv/ This file has several sections: - Getting started - Running from a remote machine Getting started =============== Before use, you need to make a temporary data directory in your current working directory: $ mkdir data You also need to install Mark Pilgrim's Universal Feed Parser from http://www.feedparser.org. On Debian or Ubuntu you can install it like this: $ sudo apt-get install python-feedparser You Python ctypes support. On Debian / Ubuntu, its as simple as: $ sudo apt-get install python-ctypes You also need the Python MySQL module: $ sudo apt-get install python-mysqldb Finally, you need mplayer and mecoder: $ sudo apt-get install mplayer mencoder Using MythNetTv is simple. If you want MythNetTv to manage your subscriptions, then do something like: $ ./mythnettv subscribe \ "http://video.google.com/videofeed?type=search&q=engedu&so=1&num=20&output=rss" \ "Goole EngEdu" You can update the view of the feed (i.e. find new posts) with: $ ./mythnettv update And you can find out what you are subscribed to with: $ ./mythnettv list To download shows, just tell it how many: $ ./mythnettv download 10 Will download 10 shows and import them into the MythTV recordings menu. You can also manually download an RSS feed just once, import local or remote videos, and get interesting statistics. Checkout the usage message for more information about those. Running from a remote machine ============================= If you're not using MythNetTV on your master backend, then you'll need to tweak your MythTV database just a little. Let's assume I want to run on a machine named maui. First I need to tell MythNetTV where to store imported programs. Log into your MySQL database execute this command: insert into settings (value, data, hostname) values("RecordFilePrefix", "/data/mythtv", "maui"); The path will of course need to change to whereever your recordings are stored. At the moment MythNetTV assumes that the recordings directory is NFS mounted, so you'll need to set that up as well. Cheers, Mikal (mikal@stillhq.com) ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/README.BITTORRENT���������������������������������������������������������������0000644�0001750�0001750�00000001220�11135202764�016225� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Bittorrent support in MythNetTv is implemented using bittornado, as it had the most parsable output of the clients I tried. Parsing is needed because bittorrent clients don't exit at the end of the download -- they keep serving the file out. MythNetTv therefore needs to detect a finished download and kill the bittorrent process. To install bittornado on a Debian system, you do this: $ sudo apt-get install bittornado Note that if you intend to download programs using Bittorrent via a cron job, you need to fix this rather annoying bug first: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=327505 Hopefully this will be fixed in bittornado soon. ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/DEVELOPERS����������������������������������������������������������������������0000644�0001750�0001750�00000001350�11135202764�015251� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������This is a brief explaination of the MythNetTV code: mythnettv ========= This source file is the original user interface. Its a simple command line oriented user interface, which doesn't let you run more than one command at a time. mythnettvcore.py ================ The user interface just calls into here. This file calls all the other modules as required. If you're writing a new user interface for MythNetTV, then you should just have to import this file and off you go. database.py =========== This code handles the MySQL database. program.py ========== This code is an abstraction over the database used to handle each program. syndication.py ====== Everything needed for subscriptions. video.py ======== Video helper routines. ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/TODO����������������������������������������������������������������������������0000644�0001750�0001750�00000000747�11135202764�014357� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������- Check for uniqueness of destination video file before moving to video directory - Fix dummy = 'blah' - Unsubscribe doesn't remove outstanding shows from todo list - Support HTTP proxies per program, to handle VPN-like connections - Provide a way to remove archive entries - Provide a way to remove proxy entries - Convert proxyhandler.HttpHandler to a file like object which logs usage - Add verbose output to all commands - Update man pages automatically - Include flags in man pages �������������������������mythnettv-release-7/COPYING�������������������������������������������������������������������������0000644�0001750�0001750�00000000346�11135202764�014715� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������This release is licensed under two versions: - the GNU GPL (COPYING.GPL) - or the GNU LGPL (COPYING.LGPL) You may use either license, at your discretion. For more information, please contact Michael Still at mikal@stillhq.com ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/TESTS-MISSING�������������������������������������������������������������������0000644�0001750�0001750�00000005140�11135202764�015613� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������The following commands have no unit testing coverage at all. Other commands are at least partially tested, even if that testing isn't complete yet. (manual usage) file <url> <title>: to do the same, but from a file, with a show title like url above cleartodo : permanently remove all items from the TODO list markread <num> : interactively mark some of the oldest <num> shows as already downloaded and imported markread <num> <title filter> : the same as above, but filter to only mark shows with a title exactly matching the specified filter markread <num> <title filter> <subtitle filter> : the same as above, but with a regexp title filter as well (handy stuff) todoremote : add a remote URL to the TODO list. This will prompt for needed information about the video, and set the date of the program to now todoremote <url> <title> <subtitle> <description> : the same as above, but don't prompt for anything importtorrent <url> <title> <subtitle> <description> : the same as above, but force the URL to be treated as a torrent. This is useful when MythNetTV doesn't automatically detect that the URL is to a torrent file. importlocal <file>: import the named file, using the title, subtitle and description from the command line. The file will be left on disk. importlocal <file>: import the named file. The file will be left on disk. Will prompt for needed information importmanylocal <path> <regexp> <title>: import all the files from path matching regexp. title is use as the title for the program, and the filename is used as the subtitle (subscription management) list : list subscriptions unsubscribe <url> : unsubscribe from a URL archive <title> <path> : archive all programs with this title to the specified path. This is useful for shows you download and import, but want to build a non-MythTV archive of as well (reporting) statistics : show some simple statistics about MythNetTV log : dump the current internal log entries��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/mythnettv.1���������������������������������������������������������������������0000644�0001750�0001750�00000014153�11135202764�016007� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������.TH "MYTHNETTV" "1" "24 August 2008" "" "" .SH "NAME" mythnettv \- RSS video aggregator for MythTV .SH "DESCRIPTION" \fBmythnettv\fP is a RSS video aggregtor for MythTV. It take downloads video feeds and places them in the recording directory in MythTV, then makes them accessible with the rest of your recordings. This implementation has advantages over other RSS readers for MythTV in that it: - ignores the content of the post -- it's all about the video files - queues downloads, instead of making them when feeds are added, or new shows are posted (this stops you from running out of disk, or having a huge show backlog) .SH "USAGE" (manual usage) url <url> <title> : to download an RSS feed and load the shows from it into the TODO list. The title is as the show title in the MythTV user interface file <url> <title>: to do the same, but from a file, with a show title like url above download <num> : to download that number of shows from the TODO list. We download some of the oldest first, and then grab some of the newest as well. download <num> <title filter> : the same as above, but filter to only download shows with a title exactly matching the specified filter download <num> <title filter> <subtitle filter> : the same as above, but with a regexp title filter as well download <num> <title filter> <subtitle filter> justone : the same as above, but download just one and then mark all other matches as read cleartodo : permanently remove all items from the TODO list markread <num> : interactively mark some of the oldest <num> shows as already downloaded and imported markread <num> <title filter> : the same as above, but filter to only mark shows with a title exactly matching the specified filter markread <num> <title filter> <subtitle filter> : the same as above, but with a regexp title filter as well (handy stuff) todoremote : add a remote URL to the TODO list. This will prompt for needed information about the video, and set the date of the program to now todoremote <url> <title> <subtitle> <description> : the same as above, but don't prompt for anything importremote : download and immediately import the named URL. Will prompt for needed information importremote <url> <title> <subtitle> <description> : the same as above, but don't prompt for anything importtorrent <url> <title> <subtitle> <description> : the same as above, but force the URL to be treated as a torrent. This is useful when MythNetTV doesn't automatically detect that the URL is to a torrent file. importlocal <file>: import the named file, using the title, subtitle and description from the command line. The file will be left on disk. importlocal <file>: import the named file. The file will be left on disk. Will prompt for needed information importmanylocal <path> <regexp> <title>: import all the files from path matching regexp. title is use as the title for the program, and the filename is used as the subtitle (subscription management) subscribe <url> <title> : subscribe to a URL, and specify the show title list : list subscriptions unsubscribe <url> : unsubscribe from a URL update : add new programs from subscribed URLs to the TODO list archive <title> <path> : archive all programs with this title to the specified path. This is useful for shows you download and import, but want to build a non-MythTV archive of as well http_proxy <url regexp> <proxy> : you can choose to use a HTTP proxy for URL requests matching a given regular expression. Use this command to define such an entry. This might be handy if some of the programs you wish to subscribe to are only accessible over a VPN. http_proxy <url regexp> <proxy> <budget mb> : the same as above, but you can specify the maximum number of megabytes to download via the proxy in a given day. To see proxy usage information, use the proxyusage command. (reporting) statistics : show some simple statistics about MythNetTV log : dump the current internal log entries nextdownload <num> : if you executed download <num>, what would be downloaded? proxyusage : print a simple report on HTTP proxy usage over the last seven days (debugging) videodir : show where MythNetTV thinks it should be placing video files explainvideodir : verbosely explain why that video directory was selected. This can help debug when the wrong video directory is being used, or no video directory at all is found .SH "CONTACT DETAILS" Information about \fBmythnettv\fP is posted to \fBhttp://www.stillhq.com/mythtv/mythnettv/\fP regularly. There is a users mailing list at \fBhttp://lists.stillhq.com/listinfo.cgi/mythnettv-stillhq.com\fP .SH "AUTHOR" \fBmythnettv\fP is by Michael Still <mikal@stillhq.com> ���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/video.1�������������������������������������������������������������������������0000644�0001750�0001750�00000002504�11135202764�015050� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������.TH "VIDEO" "1" "24 August 2008" "" "" .SH "NAME" video \- MythNetTV video processing module .SH "DESCRIPTION" \fBmythnettv\fP is a RSS video aggregtor for MythTV. It take downloads video feeds and places them in the recording directory in MythTV, then makes them accessible with the rest of your recordings. The MythNetTV video subsystem may be queried directly. This can be useful if you want to perform simple operations such as asking if a given video file needs to be transcoded before being imported into MythTV, or actually performing the transcode. .SH "USAGE" length <path> : return the length of a video in seconds info <path> : return all available information for a given video needstranscode <path> : determine if a given file needs to be transcoded before being imported into MythTV transcode <path> <output path> : transcode the file into a format suitable for MythTV .SH "CONTACT DETAILS" Information about \fBmythnettv\fP is posted to \fBhttp://www.stillhq.com/mythtv/mythnettv/\fP regularly. There is a users mailing list at \fBhttp://lists.stillhq.com/listinfo.cgi/mythnettv-stillhq.com\fP .SH "AUTHOR" \fBmythnettv\fP is by Michael Still <mikal@stillhq.com> ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/runtests.sh���������������������������������������������������������������������0000711�0001750�0001750�00000000167�11135202764�016101� 0����������������������������������������������������������������������������������������������������ustar �mikal���������������������������mikal������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������#!/bin/bash -e for item in *_test.py do echo echo "Running $item" echo ./$item done echo "All tests passed" ���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/��������������������������������������������������������������������������������0000755�0001750�0001750�00000000000�11151533543�014253� 5����������������������������������������������������������������������������������������������������ustar �thomas��������������������������thomas�����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������mythnettv-release-7/COPYING.BSD���������������������������������������������������������������������0000644�0001750�0001750�00000002733�11151533543�015722� 0����������������������������������������������������������������������������������������������������ustar �thomas��������������������������thomas�����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Copyright (c) The Regents of the University of California. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. �������������������������������������mythnettv-release-7/COPYING.GPL���������������������������������������������������������������������0000644�0001750�0001750�00000043103�10641012126�015720� 0����������������������������������������������������������������������������������������������������ustar �thomas��������������������������thomas����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������� GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. <one line to give the program's name and a brief idea of what it does.> Copyright (C) <year> <name of author> This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. <signature of Ty Coon>, 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. �����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������