pymetrics-0.8.1/BUGLIST0000664000076400007640000000303710665421742012733 0ustar regregOutstanding Bugs as of Version 0.7.9 ==================================== The NewFeatures24 metric does not yet check for the following 2.4 features: * Unified integers (PEP 237) * Simplified string substitution Template subclass (PEP 292) * Reversd iterators (PEP 322) * Subprocess module (PEP 324) * Decimal module data type and context (PEP 327) * Multi-line imports (PEP 328) * Locale-Independent Float/String Conversions (PEP 331) * dict.update() method now accepts the same argument forms as the dict constructor * ljust(), rjust(), and center() now take an optional argument for specifying a fill character other than a space. * rsplit() method that works like the split() method but splits from the end of the string * Three keyword parameters, cmp, key, and reverse, were added to the sort() method of lists. * Built-in function sorted(iterable) * Integer operations will no longer trigger an OverflowWarning. * New interpreter switch, -m, that takes a name, searches for the corresponding module on sys.path, and runs the module as a script. * The eval(expr, globals, locals) and execfile(filename, globals, locals) functions and the exec statement now accept any mapping type for the locals parameter. * None is now a constant. * zip() built-in function and itertools.izip() now return an empty list if called with no arguments. There is no warning that the NewFeatures24 metric is unable to check for runtime features or syntax changes that would raise an error. Also, it is unable to check for updated library modules. ~ pymetrics-0.8.1/CHANGE.LOG0000664000076400007640000002724611042150667013172 0ustar regregChange Log from 0.8.0 to 0.8.1 ============================== 1) Bug fixes: a) Version 0.8.0 was Python version sensitive. It behaved differently when run under versions before 2.5 and those at or after 2.5 due to changes in the way __import__ worked. b) BUGLIST was not being update which each new release. Change Log from 0.7.9 to 0.8.0 ============================== Converted distribution to use the standard Python setup.py method. Thanks to Carlos Garcia Campos (carlosgc-at-gnome.org). Change Log from 0.7.6 to 0.7.9 ============================== 1) Bug fixes: a) ProcessRun was not executed under all circumstances. b) Code blocks in __main__ were ignored. c) Missing input files raised an exception and terminate after printing help. d) Class doc strings were counted as function doc strings. e) Metric module with missing metric class raised a 'key' exception. f) File with trailing whitespace on last indented line caused exception. 2) Output sent to stderr also sent to stdout. This allows user to see error in content since the output from stderr and stdout are not in sync. 3) Changed default metric class name so that for metric module wxYz, the default metric class name is WxYzMetric (note capitalized name) when the metric class name is omitted in the --includeMetrics parameter. 4) Added Decorators to newFeatures24 metric. Change Log from 0.7.1 to 0.7.6 ============================== 1) Added Features a) New option -B to suppress Basic Metrics output b) Added the new -e option. When the option is used, the SQL command file does not try to recreate the SQL tables. In other words, only INSERT commands are generated. Use this option if you are sure the SQL data and token tables already exist. This option implies the -N option (it deletes the old SQL command file, if it exists). c) Added computation of COCOMO 2's Source Lines Of Code (SLOC) metric. In COCOMO 2, SLOC is defined as: * Only Source lines that are delivered as part of the product are included. * Test drivers and other support software is excluded * Source lines are created by the project staff. Code created by applications generators is excluded * One SLOC is one logical line of code * Declarations are counted as SLOC * Comments are not counted as SLOC * Blank and empty lines are not counted as SLOC The COCOMO 1 model's SLOC is now called DSI (Delivered Source Instructions) in COCOMO 2. For example, in COCOMO 2, a Python "if-else" statement is counted as one SLOC, but might be counted as several DSIs. Given all that, PyMetrics will count test code if requested. I believe that this is correct to count test code because it is the basis of good Extreme Programming methodology. This SLOC metric also introduced raw text metrics. This has lead to a new member function of MetricBase: processSrcLines( self, srcLines, *a, **kw) where srcLines is a string containing the raw text of the current file. This member function is called once per file. The SLOC metric is computed by default. The following definitions are used to compute SLOC: 1) All literals are treated as occupying one logical line, regardless of the number of physical lines that are spanned. 2) All parenthesized lists, expressions, and argument lists are treated as occupying one logical line, regardless of the number of physical lines spanned. 3) All documentation strings are ignored. 2) Bug Fixes a) In Help, the default value displayed for --quiet was reversed. It now reads more logically. 3) Miscellaneous a) Now clearly identify generated code with a comment in the first line starting with "-- Automatically generated ..." 4) Added testSLOC.py to sample/ subdirectory. It tests most of weird cases that may occur in a Python source file due to continuations, literals of various types, and code spanning multiple lines. The source code shows what is counted with comments on the end of each counted source line. As of thie release, the SLOC metric for this file is 28. Change Log from 0.6.0 to 0.7.1 ============================== Quite a lot has changed from the last version. The changes fall into two categories: 1) Added Features a) More command line options to add flexibility to what is computed and displayed. PLEASE RUN PyMetrics TO DISPLAY ALL THE OPTIONS. b) Easier creation and inclusion of your own metrics c) Creation of a SQL table, called metricData, in the the metrics database that contains all the metrics that have been calculated. This table also tracks metrics over time, so you can compare metrics from one run to the next. d) Added library name field to SQL output. This is an user-defined name to identify a collection of modules. For example, all the Zope modules might have a 'zope' library name while all your modules might have your initials as a library name. In this way, you could compare the metrics in your library against the metrics in the Zope library. You could also use this feature to define a standard set of metrics with a library name of 'std' and then you could measure all other modules against the 'std' modules. 2) Refactoring and Reworking a) Due to the major enhancements introduced in this version, a number of modules have been refactored to simplify them and make them more testable. b) Unit tests have been written for a number of modules. The tests are contained in the "test" subdirectory. The "test" subdirectory has been added to this distribution. 3) Added missing LICENSE and "examples" subdirectory. 4) Fixed various bugs, none critical. Change Log from 0.5.3 to 0.6 ============================ 1) Modified PyMetrics to handle IOErrors more gracefully and continue executing, if possible. 2) Check for lines ending in '\r' and convert character to blank since Python normally converts '\r' to '\n' on Windows and Mac systems. 3) Changed metric format so each metric number appears before its name. 4) Added the -f/--files option to allow the user to specify a file containing a list of modules to analyse. 5) Fixed Halstead metric code to handle degenerate programs better. 6) Changed SQL and CSV output to have symbolic names for types of names in fully-qualified function and class names. 7) Changed semantic type of 'self' from VARNAME to KEYWORD. 8) Now count any token of semantic significance, like keywords, function names, etc. 9) Fixed issue where tarball dir was not created at a higher level directory with the name of the version in the tarball. 10) Counts of Token Types now listed in alphabetic order. 11) Added -K option to suppress display of token types. 12) Introduced MetricBase class for use in designing your own metrics. Member functions include: processToken( self, fcnName, className, tok, *args, **kwds ) processStmt( self, fcnName, className, stmt, *args, **kwds ) processBlock( self, fcnName, className, block, *args, **kwds ) processFunction( self, fcnName, className, fcn, *args, **kwds ) processClass( self, fcnName, className, cls, *args, **kwds ) processModule( self, moduleName, module, *args, **kwds ) processRun( self, run, *args, **kwds ) where: context is context in which last token occurred. metrics is metrics up to the point where function called tok is last completed token stmt is last completed statement block is last completed block fcn is last completed function cls is last completed class mod is last completed module run is last completed run (i.e., everything) Each of the stmt, block, fcn, cls, mod, and run is a list of tokens. The context is a dictionary related to the last token processed before invoking one of the processX member function. The context contains: inFile current file name fcnDepth current nested function depth classDepth current nested class depth parenDepth current parenthesis depth bracketDepth current depth of brackets ([]) braceDepth current depth of braces ({}) blockNum current unique block number blockDepth current nested block depth 13) By default, the Halstead metric is not displayed. It is still available, as explained by the Help. This makes the -H option redundant at this point. I have removed this metric because I have not found it to be generally useful. You may differ and include it in any of you runs. 14) By default, the token and keyword counts are not displayed. They are still available using the -k option. The -K option is still present, but redundant at this point. Again, I did not find this metric of particular use, so I made it optional. 15) By far, the most time spend in running PyMetrics is in writing out the SQL and CSV files. Suppressing one or both, if you do not need them, will speed up processing. However, I did speed up the processing of these files by 20-50%. Change Log from 0.5.2 to 0.5.3 ============================== 1) PyMetrics is now available under CVS. 2) Fixed a bug where the tokenize library module allowed the position of a comment on a line by itself to affect the indentation of subsequent blocks. This would cause a crash if the affected code included a function return statement. 3) Fixed bug where the McCabe metric routine was being called twice for most tokens. 4) Changed tokenize module to yield 5 parameters, as it originally did. At an early stage, I erroneously thought that I needed to pass back an extra value. 5) Sped up processing slightly by skipping redundant tests. 6) Changed subdirectory name from tests to examples. Change Log from 0.5.1 to 0.5.2 ============================== This is a bug fix release. 1) Fixed bug where program failed if first line of file started with #!. Change Log from 0.5.0 to 0.5.1 ============================== This is mainly a bug fix release. The changes made also include: 1) Fixed several divide by zero bugs that could occur for degenerate Python programs. 2) Corrected version.py to reflect the correct version and name of the program. 3) Removed %DocStrings metric since it was meaningless. 4) Fixed bug where nested classes and nested functions were not correctly reported. 5) Changed format of report on classes/functions having or missing Doc strings. 6) Report on classes/functions now uses fully qualified names. 7) Refactored some code to common up handling of certain types of tokens. 8) Added the -z/--zero command line option. Normally, zero and empty metrics are not shown. This option forces the values to be displayed. Change Log from 0.4.0 to 0.5.0 ============================== 1) Simplified processing of command line arguents and parameter handling by objectifying parameter parsing. 2) Eliminated need for global variables when handling command line parameters and options. 3) Rearranged basic metrics to appear before computed metrics. 4) Moved code out of the main module, PyMetrics.py, into separate files. 5) Clarified some options in the Help output. Also added the -S option to suppress output of the SQL command file. 6) Defined and implemented endOfToken, endOfStmt, endOfFcn, endOfClass, endOfModule, and endOfRun functions for main program. 7) Implemented a very simple subscribe feature to allow various metric classes to be registered for the events in point 6). pymetrics-0.8.1/FILELIST0000664000076400007640000000011611042011043013003 0ustar regregBUGLIST CHANGE.LOG FILELIST INSTALL LICENSE README RELEASE setup.py pymetrics pymetrics-0.8.1/INSTALL0000664000076400007640000000101311042003647012706 0ustar regregThere are two steps to install PyMetrics: Step 1: Untar the compressed pymetrics-xx.xx.xx.tgz tarball. It creates a subdirectory called pymetrics-xx.xx.xx where xx.xx.xx is the version number. For example: $ cd someplace $ tar -xvzf pymetrics-xx.xx.xx.tar.gz This creates the subdirectory "pymetrics-xx.xx.xx" in directory "someplace". Step 2: Use Python's standard setup.py method to install PyMetrics in the normal Python library. See README for details on running PyMetrics. That's it. Enjoy. pymetrics-0.8.1/LICENSE0000664000076400007640000003543310664101026012676 0ustar regreg GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc. 675 Mass Ave, Cambridge, MA 02139, USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Library General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS pymetrics-0.8.1/pymetrics0000775000076400007640000000013411041771745013637 0ustar regreg#!/usr/bin/env python import PyMetrics.PyMetrics as pm import sys pm.main () sys.exit (0) pymetrics-0.8.1/PyMetrics/0000775000076400007640000000000011042151000013564 5ustar regregpymetrics-0.8.1/PyMetrics/sqltemplate.pyc0000664000076400007640000000377611042143023016657 0ustar regregmò ‚ÐFc@s5dZddd!ZdZdZdZdZdZdS( s9 sqltemplate - template for generating sql token output. s$Revision: 1.2 $i iþÿÿÿs'Reg. Charney sç-- -- Automatically generated table structure for token table `%s` -- DROP TABLE IF EXISTS %s; CREATE TABLE %s ( IDDateTime datetime NOT NULL default '0000-00-00 00:00:00', ID int(11) unsigned NOT NULL auto_increment, libraryName varchar(32) default '', fileName varchar(255) default NULL, lineNum int(11) NOT NULL, colNum int(11) NOT NULL, type varchar(16) NOT NULL default 'ERRORTOKEN', semtype varchar(16) default NULL, textLen int(11) NOT NULL default 1, text varchar(255) NOT NULL default '', fqnFunction varchar(255) default NULL, fqnClass varchar(255) default NULL, blockNum int(11) NOT NULL default 1, blockDepth int(11) NOT NULL default 0, fcnDepth int(11) NOT NULL default 0, classDepth int(11) NOT NULL default 0, parenDepth int(11) NOT NULL default 0, bracketDepth int(11) NOT NULL default 0, braceDepth int(11) NOT NULL default 0, PRIMARY KEY (IDDateTime,ID), FULLTEXT KEY FULLTEXTIDX (text) ) TYPE=MyISAM; -- -- Load data for table `%s` -- sINSERT INTO %s VALUES (%s); s -- Automatically generated table structure for metric data table `%s` -- DROP TABLE IF EXISTS %s; CREATE TABLE %s ( IDDateTime datetime NOT NULL default '0000-00-00 00:00:00', ID int(10) unsigned NOT NULL auto_increment, libraryName varchar(32) default '', metricName varchar(32) NOT NULL default '', srcFileName varchar(255) NOT NULL default '', varName varchar(255) NOT NULL default '', value decimal(15,5) NOT NULL default '0', PRIMARY KEY (IDDateTime,ID) ) TYPE=MyISAM; -- -- Load metric data for table `%s` -- N(t__doc__t __revision__t __author__ttokenHdrt tokenInserttdataHdrt dataInsert(RRRRRR((t;/home/reg/pymetrics/branches/0.8.0/PyMetrics/sqltemplate.pyt?s  "pymetrics-0.8.1/PyMetrics/sqltemplate.py0000664000076400007640000000345310664101030016505 0ustar regreg""" sqltemplate - template for generating sql token output. """ __revision__ = "$Revision: 1.2 $"[11:-2] __author__ = 'Reg. Charney ' tokenHdr = """-- -- Automatically generated table structure for token table `%s` -- DROP TABLE IF EXISTS %s; CREATE TABLE %s ( IDDateTime datetime NOT NULL default '0000-00-00 00:00:00', ID int(11) unsigned NOT NULL auto_increment, libraryName varchar(32) default '', fileName varchar(255) default NULL, lineNum int(11) NOT NULL, colNum int(11) NOT NULL, type varchar(16) NOT NULL default 'ERRORTOKEN', semtype varchar(16) default NULL, textLen int(11) NOT NULL default 1, text varchar(255) NOT NULL default '', fqnFunction varchar(255) default NULL, fqnClass varchar(255) default NULL, blockNum int(11) NOT NULL default 1, blockDepth int(11) NOT NULL default 0, fcnDepth int(11) NOT NULL default 0, classDepth int(11) NOT NULL default 0, parenDepth int(11) NOT NULL default 0, bracketDepth int(11) NOT NULL default 0, braceDepth int(11) NOT NULL default 0, PRIMARY KEY (IDDateTime,ID), FULLTEXT KEY FULLTEXTIDX (text) ) TYPE=MyISAM; -- -- Load data for table `%s` -- """ tokenInsert = """INSERT INTO %s VALUES (%s);\n""" dataHdr = """ -- Automatically generated table structure for metric data table `%s` -- DROP TABLE IF EXISTS %s; CREATE TABLE %s ( IDDateTime datetime NOT NULL default '0000-00-00 00:00:00', ID int(10) unsigned NOT NULL auto_increment, libraryName varchar(32) default '', metricName varchar(32) NOT NULL default '', srcFileName varchar(255) NOT NULL default '', varName varchar(255) NOT NULL default '', value decimal(15,5) NOT NULL default '0', PRIMARY KEY (IDDateTime,ID) ) TYPE=MyISAM; -- -- Load metric data for table `%s` -- """ dataInsert = """INSERT INTO %s VALUES (%s);\n""" pymetrics-0.8.1/PyMetrics/compute.pyc0000664000076400007640000004150311042143023015766 0ustar regregmò åó‡Hc@sPdZddd!ZdZdkZdkTdkTdZdefd „ƒYZdS( si Main computational modules for PyMetrics. $Id: compute.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ s$Revision: 1.2 $i iþÿÿÿs*Reg. Charney N(t*i tComputeMetricscBstZdZd„Zdd„Zd„Zd„Zd„Zd„Zd„Z d „Z d „Z d „Z d „Z d „Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„ZRS(s6 Class used to compute basic metrics for given module.cCsò||_||_||_||_||_||_||_g|_g|_ g|_ d|_ g|_ g|_g|_g|_g|_g|_g|_g|_g|_g|_g|_g|_g|_g|_|i|ƒdS(s) Initialize general computational object.N(tmetricInstancetselftcontextt runMetricstmetricstpatsotcot fqnFunctiontfqnClasstfcnExitstNonettokentstmttblocktfcntclstmodtruntprocessSrcLineSubscriberstprocessTokenSubscriberstprocessStmtSubscriberstprocessBlockSubscriberstprocessFunctionSubscriberstprocessClassSubscriberstprocessModuleSubscriberstprocessRunSubscriberst_ComputeMetrics__initMetrics(RRRRRRRR ((t7/home/reg/pymetrics/branches/0.8.0/PyMetrics/compute.pyt__init__s6                         t__main__cCs)|}t|ƒo|dd}n|S(s( Extract fully qualified name from list.iÿÿÿÿiN(tdefaulttresulttlentfqn(RR$R!R"((Rt __extractFQN2s  cCs$|ii|dƒd|i|s  cCsÚ||_|ii|ƒ|ii|ƒ|io|ii|ƒn|io|i i|ƒn|i i|ƒ|i i|ƒ|i |iƒ}|i |idƒ}x'|iD]}|i|||iƒq¶WdS(s. Handle processing after each token processed.N(ttokRRRtappendRR RR RRRR)R*R R+RR,t processToken(RR/R,R+R*((RR1Js    cCs_|i|iƒ}|i|idƒ}x'|iD]}|i |||i ƒq1Wg|i (dS(s' Handle processing at end of statement.N( RR)R R*R R R+RR,t processStmtR(RR+R,R*((RR2[s cCs_|i|iƒ}|i|idƒ}x'|iD]}|i |||i ƒq1Wg|i (dS(s# Handle processing at end of block.N( RR)R R*R R R+RR,t processBlockR(RR+R,R*((RR3ds cCsk|iƒ}|i|iƒ}|i|idƒ}x'|i D]}|i |||i ƒq=Wg|i (|S(s' Handle processing at end of function. N( Rt#_ComputeMetrics__checkNumberOfExitstmsgR)R R*R R R+RR,tprocessFunctionR(RR,R+R5R*((RR6ms   cCsêd }t|iƒ}|djoÄdig}|iD]}|t |ƒq6~ƒ}|djodpd}d||f}d|i d|i|iƒ|||f}x'tt|iƒƒD]}|id=qÁW|id ƒn|S( s2" Generate warning message if more than one exit. is, itstsexit%s at line%ss*In file %s, function %s has %d extra %s %stinFiletnumMultipleExitFcnsN(R R5R#RR tntjoint_[1]titstrtexitstpluraltexitStrRR)R tranget_ComputeMetrics__incr(RR@R>RBR;R=R5RA((Rt__checkNumberOfExitsys 3,cCs_|i|iƒ}|i|idƒ}x'|iD]}|i |||i ƒq1Wg|i (dS(s$ Handle processing at end of class. N( RR)R R*R R R+RR,t processClassR(RR+R,R*((RRFs cCsE|id}|}x!|iD]}|i||ƒqWg|i(dS(s$ Handle processing at end of class. R9N(RRt moduleNameRRR,t processModule(RR,RGR((RRH–s  cCs2x!|iD]}|i|iƒq Wg|i(dS(s$ Handle processing at end of class. N(RRt subsribert processRunR(RRI((RRJŸs  cCsx.|iiƒD]}|i|i|iƒqW|i}d}d}t i ƒ}x©|D]¡}|o‚|i|jo d}qè|itjo+|d}||id<|i|ƒqYqè|itjo|i|i7_qYqèn|i||ƒ}qYW|iS(sr This function is the start of the heavy lifting for computing the various metrics produced by PyMetrics.iit numTokensN(RRtkeystmR-tlextsrcLinest tokenlistt tokenListR t skipUntilttokCounttmytokentMyTokent invalidTokenR/ttextttypetWSRt_ComputeMetrics__postTokent ERRORTOKENt handleToken(RRNRVRMRRR/RQRS((Rt__call__¦s.       cCsog}x4|iƒD]&}||o|i||ƒqqW|i2|i2|i2|i 2|i 2|i 2|i 2|i 2|ii|ƒ|ii|ƒ|ii|ƒ|i i|ƒ|i i|ƒ|i i|ƒ|i i|ƒ|i i|ƒd|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_g|_t |_!t |_"t#|_$t |_%t |_&t |_'t |_(d|i)d All rights reserved, see LICENSE for details. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. $Id: __init__.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ s$Revision: 1.2 $i iþÿÿÿs'Reg. Charney N(t__doc__t __version__t __author__(RR((tPyMetrics/__init__.pyt?s pymetrics-0.8.1/PyMetrics/mccabe.py0000664000076400007640000000455411041771745015407 0ustar regreg""" Compute McCabe's Cyclomatic Metric. This routine computes McCabe's Cyclomatic metric for each function in a module/file. $Id: mccabe.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.3 $"[11:-2] __author__ = 'Reg. Charney ' from metricbase import MetricBase McCabeKeywords = { 'assert':0, 'break':1, 'continue':1, 'elif':1, 'else':1, 'for':1, 'if':1, 'while':1 } class McCabeMetric( MetricBase ): """ Compute McCabe's Cyclomatic McCabeMetric by function.""" def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): self.context = context self.runMetrics = runMetrics self.metrics = metrics self.pa = pa self.inFile = context['inFile'] self.fcnNames = {} def processToken( self, fcnName, className, tok, *args, **kwds ): """ Increment number of decision points in function.""" if tok and tok.text in McCabeKeywords: self.fcnNames[fcnName] = self.fcnNames.get(fcnName,0) + 1 def processFunction( self, fcnName, className, fcn, *args, **kwds ): """ Increment number of decision points in function.""" self.fcnNames[fcnName] = self.fcnNames.get(fcnName,0) + 1 def display( self ): """ Display McCabe Cyclomatic metric for each function """ result = {} # the next three lines ensure that fcnNames[None] is treated # like fcnNames['__main__'] and are sorted into alphabetical # order. if self.fcnNames.has_key(None): self.fcnNames['__main__'] = self.fcnNames.get(None,0) del self.fcnNames[None] if self.pa.quietSw: return result hdr = "\nMcCabe Complexity Metric for file %s" % self.inFile print hdr print "-"*len(hdr) + "\n" keyList = self.fcnNames.keys() if len( keyList ) > 0: keyList.sort() for k in keyList: if k: name = k else: name = "__main__" print "%11d %s" % (self.fcnNames[k],name) result[k] = self.fcnNames[k] else: print "%11d %s" % (1,'__main__') result['__main__'] = 1 print return result pymetrics-0.8.1/PyMetrics/mccabe.pyc0000664000076400007640000000503211042143522015525 0ustar regregmò åó‡Hc@sŽdZddd!ZdZdklZhdd<dd <d d <d d <d d <d d <dd <dd (s MetricBasetassertitbreakitcontinueteliftelsetfortiftwhilet McCabeMetriccBs2tZdZd„Zd„Zd„Zd„ZRS(s6 Compute McCabe's Cyclomatic McCabeMetric by function.cOs>||_||_||_||_|d|_h|_dS(NtinFile(tcontexttselft runMetricstmetricstpaR tfcnNames(R R R R Rtargstkwds((t6/home/reg/pymetrics/branches/0.8.0/PyMetrics/mccabe.pyt__init__s      cOs?|o4|itjo$|ii|dƒd|i| Copyright (c) 2007 by Reg. Charney All rights reserved, see LICENSE for details. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. $Id$ """ __revision__ = "$Id$" __author__ = 'Reg. Charney ' # Imports import sys import string from processargs import ProcessArgs, ProcessArgsError import token import sqltokenout import sqldataout import csvout from lexer import Lexer from compute import ComputeMetrics from globals import * PYTHON_VERSION = sys.version[:3] ############################################################################# ### Main script for PyMetrics utility. ############################################################################# def __importMetricModules( includeMetrics ): """ Import the modules specified in the parameter list. includeMetrics is a list of (metricModuleName, metricClassName) pairs. This function defines a dictionary containing only valid module/class names. When an error is found, the invalid module/class pair is removed from the included list of metrics. """ i = 0 metricModules = {} if PYTHON_VERSION < '2.5': pfx = '' # this fix is for Python 2.4 else: pfx = 'PyMetrics.' for m,n in includeMetrics: try: mm = pfx + m if PYTHON_VERSION < '2.5': mod = __import__( mm, globals(), locals(), [m] ) else: mod = __import__( mm, fromlist=[m] ) metricModules[m] = mod i += 1 except ImportError: sys.stderr.write( "Unable to import metric module %s -- ignored.\n\n" % mm ) # remove the erroneous metric module/class tuple del includeMetrics[i] return metricModules def __instantiateMetric( metricModules, context, runMetrics, metrics, pa ): """ Instantiate all user specified metric classes. The code works by finding the desired metric class in a metric module and instantiating the class. It does this by assuming that the metric class is in the dictionary of the metric module. """ metricInstance = {} inclIndx = -1 for m,n in pa.includeMetrics: inclIndx += 1 try: metricInstance[m] = None # default value if metric class does not exist. metricInstance[m] = metricModules[m].__dict__[n]( context, runMetrics, metrics, pa ) except KeyError: sys.stderr.write( "Module %s does not contain metric class %s -- metric %s ignored.\n\n" % (m,n,m) ) del( metricInstance[m] ) del( pa.includeMetrics[inclIndx] ) return metricInstance def __stats( so, m, inFileName, label, *args ): """ Print line of statistics.""" result = string.join(map(str, args), '') print "%11s %s" % (result, label) so and so.write( m, inFileName, label, result ) def __printSummary( so, context, runMetrics, metrics, pa ): """ Print basic summary information.""" # the following loop is a very, very ugly hack to distinguish between # tokens as they appear in the source; semantically generated tokens, # like DOCSTRING; and NONTOKENs, like numComments keys = [] for k in metrics.keys(): if str( k ).isdigit(): keys.append( (token.tok_name[k],k,metrics[k]) ) elif len( str( k ).split() ) > 1: keys.append( (k,SEMTOKEN,metrics[k]) ) else: keys.append( (k,NONTOKEN,metrics[k]) ) keys.sort() inFileName = context['inFile'] if pa.genKwCntSw: hdr = "Counts of Token Types in module %s" % context['inFile'] print print hdr print "-"*len(hdr) print for k,t,v in keys: if (pa.zeroSw or v): if t != NONTOKEN: __stats( so, 'basic', inFileName, k, v ) print if pa.genBasicSw: __displayBasicMetrics( keys, pa, so, inFileName ) def __displayBasicMetrics( keys, pa, so, inFileName ): """ Display the Basic metrics that PyMetrics computes.""" hdr = "Basic Metrics for module %s" % inFileName print print hdr print "-"*len( hdr ) print for k,t,v in keys: if t==NONTOKEN: if pa.zeroSw or not v in ([],{},(),0,0.00): __stats( so, 'basic', inFileName, k, v ) print def main(): """ Main routine for PyMetrics.""" # process command line args try: pa = ProcessArgs() except ProcessArgsError, e: sys.stderr.writelines( str(e) ) return if pa.genNewSw: __deleteOldOutputFiles( pa ) so, od = __genNewSqlCmdFiles( pa ) co = __genNewCsvFile( pa ) # import all the needed metric modules metricModules = __importMetricModules( pa.includeMetrics ) runMetrics = {} # metrics for whole run metrics = {} # metrics for this module context = {} # context in which token was used # main loop - where all the work is done for inFileName in pa.inFileNames: metrics.clear() context.clear() context['inFile'] = inFileName # instantiate all the desired metric classes metricInstance = __instantiateMetric( metricModules, context, runMetrics, metrics, pa ) cm = ComputeMetrics( metricInstance, context, runMetrics, metrics, pa, so, co ) # define lexographical scanner to use for this run # later, this may vary with file and language. lex = Lexer() if not pa.quietSw: print "=== File: %s ===" % inFileName try: lex.parse( inFileName ) # parse input file metrics["numCharacters"] = len(lex.srcLines) metrics["numLines"] = lex.lineCount # lines of code metrics = cm( lex ) # if printing desired, output summary and desired metrics # also, note that this preserves the order of the metrics desired if not pa.quietSw: __printSummary( od, context, runMetrics, metrics, pa ) for m,n in pa.includeMetrics: if metricInstance[m]: result = metricInstance[m].display() if metrics.has_key(m): metrics[m].append( result ) else: metrics[m] = result for r in result.keys(): od and od.write( m, inFileName, r, result[r] ) except IOError, e: sys.stderr.writelines( str(e) + " -- Skipping input file.\n\n") co and co.close() result = {} if len( pa.inFileNames ) > 0: for m,n in pa.includeMetrics: if metricInstance[m]: result = metricInstance[m].processRun( None ) if result: for r in result.keys(): od and od.write( m, None, r, result[r] ) od and od.close() if not pa.quietSw: n = len( pa.inFileNames ) print print "*** Processed %s module%s in run ***" % (n,(n>1) and 's' or '') def __genNewCsvFile( pa ): """ Determine output CSV data file, if any, and check it can be created.""" co = None try: if pa.genCsvSw: co = csvout.CsvOut( pa.csvFileName, genHdrSw=pa.genHdrSw, genNewSw=pa.genNewSw ) except StandardError, e: # this should not occur - it should be handled in processArgs. sys.stderr.writelines( str(e) + " -- No CSV file will be generated\n\n" ) pa.genCsvSw = False return co def __genNewSqlCmdFiles( pa ): """ determine output SQL tokens command file, if any, and check it can be created .""" so = None co = None od = None fd = None try: if pa.genSqlSw: if pa.sqlFileName: fd = open( pa.sqlFileName, 'a' ) else: fd = sys.stdout so = sqltokenout.SqlTokenOut( fd, pa.libName, pa.sqlFileName, pa.sqlTokenTableName, pa.genNewSw, pa.genExistsSw ) od = sqldataout.SqlDataOut( fd, pa.libName, pa.sqlFileName, pa.sqlMetricsTableName, pa.genNewSw, pa.genExistsSw ) except StandardError, e: # this should not occur - it should be handled in processArgs. sys.stderr.writelines( str(e) + " -- No SQL command file will be generated\n\n" ) pa.genSqlSw = False so and so.close() od and od.close() so = None od = None return so, od def __deleteOldOutputFiles( pa ): """ Generate new output files by ensuring old files deleted.""" import os try: if pa.genSqlSw and os.path.exists( pa.sqlFileName ): os.remove( pa.sqlFileName ) except IOError, e: sys.stderr.writelines( str(e) ) pa.genSqlSw = False try: if pa.genCsvSw and os.path.exists( pa.csvFileName ): os.remove( pa.csvFileName ) except IOError, e: sys.stderr.writelines( str(e) ) pa.genCsvSw = False return if __name__ == "__main__": main() sys.exit(0) pymetrics-0.8.1/PyMetrics/utils.pyc0000664000076400007640000000277711042143023015464 0ustar regregmò ‚ÐFc@s@dZdkZdkZdkZd„Zd„Zd„ZdS(su Utility functions used throughout the PyMetrics system. $Id: utils.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ NcCsB|iddƒ}|iddƒ}|iddƒ}d|dS(sH Place single quotes around strings and escaping existing single quotes.s\s\\t's\'t"s\"N(tstreplaceta(RR((t5/home/reg/pymetrics/branches/0.8.0/PyMetrics/utils.pytsqlQ s cCsT|iddƒ}|iddƒ}|iddƒ}|iddƒ}d|dS( s) Quote a string using rules for CSV data.s\s\\Rs\'s s\nRs""N(RRRtbtctd(RRRRR ((RtcsvQs cCsg}xt|D]l\}}}y'ti|}|i|||fƒWq t j o'}t dt |ƒd|ƒ‚q Xq W|S(s% Convert token type numbers to names.sUnknown value 's(' for token/semantic type in context %s N( tlstOuttlsttnamet blockDepthtsemtypettokenttok_nametsemNametappendtKeyErrortetstrtcontext(RR RR RR RR((Rt toTypeNames '(t__doc__tsysRtreRR R(RRRRRR ((Rt?s      pymetrics-0.8.1/PyMetrics/globals.py0000664000076400007640000000351511041771745015614 0ustar regreg""" Global variables for PyMetrics. $Id: globals.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.3 $"[11:-2] __author__ = 'Reg. Charney ' import token import tokenize # our token types KEYWORD = token.NT_OFFSET + 1 TEXT = token.NT_OFFSET + 2 WS = token.NT_OFFSET + 3 DOCSTRING = token.NT_OFFSET + 4 VARNAME = token.NT_OFFSET + 5 CLASSNAME = token.NT_OFFSET + 6 FCNNAME = token.NT_OFFSET + 7 INLINE = token.NT_OFFSET + 8 UNKNOWN = token.NT_OFFSET + 9 SEMTOKEN = token.NT_OFFSET + 10 # to distinguish semantic tokens NONTOKEN = token.NT_OFFSET + 11 # to distinguish non-tokens DECORATOR = token.NT_OFFSET + 12 # to indicate decorator token NUMBER = token.NUMBER OP = token.OP STRING = token.STRING COMMENT = tokenize.COMMENT NAME = token.NAME ERRORTOKEN = token.ERRORTOKEN ENDMARKER = token.ENDMARKER INDENT = token.INDENT DEDENT = token.DEDENT NEWLINE = token.NEWLINE EMPTY = tokenize.NL # new token types added to allow for character representation of new codes token.tok_name[KEYWORD] = "KEYWORD" # one of Python's reserved words token.tok_name[TEXT] = "TEXT" # obsolete - but kept for compatibility token.tok_name[WS] = "WS" # some form of whitespace token.tok_name[DOCSTRING] = "DOCSTRING" # literal that is also doc string token.tok_name[VARNAME] = "VARNAME" # name that is not keyword token.tok_name[CLASSNAME] = "CLASSNAME" # name defined in class statment token.tok_name[FCNNAME] = "FCNNAME" # name defined in def statement token.tok_name[INLINE] = "INLINE" # comment that follows other text on same line token.tok_name[UNKNOWN] = "UNKNOWN" # Unknown semantic type - this should not occur token.tok_name[DECORATOR] = 'DECORATOR' # Decorator marker pymetrics-0.8.1/PyMetrics/PyMetrics.pyc0000664000076400007640000002107711042143473016246 0ustar regregmò 9LjHc@södZdZdZdkZdkZdklZlZdkZdk Z dk Z dk Z dk l Z dklZdkTd„Zd „Zd „Zd „Zd „Zd „Zd„Zd„Zd„ZedjoeƒeidƒndS(sc PyMetrics - Complexity Measurements for Python code. Orignally based on grop.py by Jurgen Hermann. Modified by Reg. Charney to do Python complexity measurements. Copyright (c) 2001 by Jurgen Hermann Copyright (c) 2007 by Reg. Charney All rights reserved, see LICENSE for details. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. $Id$ s$Id$s(Reg. Charney N(s ProcessArgssProcessArgsError(sLexer(sComputeMetrics(t*cCsžd}h}d}x…|D]}\}}y@||}t|tƒt ƒdgƒ}|||<|d7}Wqt j o"t i id|ƒ||=qXqW|S(sI Import the modules specified in the parameter list. includeMetrics is a list of (metricModuleName, metricClassName) pairs. This function defines a dictionary containing only valid module/class names. When an error is found, the invalid module/class pair is removed from the included list of metrics. is PyMetrics.tanythingis/Unable to import metric module %s -- ignored. N(tit metricModulestpfxtincludeMetricstmtntmmt __import__tglobalstlocalstmodt ImportErrortsyststderrtwrite(RRRRRRRR ((t9/home/reg/pymetrics/branches/0.8.0/PyMetrics/PyMetrics.pyt__importMetricModules+s   c Csªh}d}x—|iD]Œ\}}|d7}y2d||<||i|||||ƒ|| All rights reserved, see LICENSE for details. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. $Id: __init__.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.2 $"[11:-2] __author__ = 'Reg. Charney ' pymetrics-0.8.1/PyMetrics/globals.pyc0000664000076400007640000000254711042140621015743 0ustar regregmò åó‡Hc@s°dZddd!ZdZdkZdkZeidZeidZeidZeid Z eid Z eid Z eid Z eid Z eidZeidZeidZeidZeiZeiZeiZeiZeiZeiZeiZeiZeiZeiZeiZdeieNiiiiiiiii i i tKEYWORDtTEXTtWSt DOCSTRINGtVARNAMEt CLASSNAMEtFCNNAMEtINLINEtUNKNOWNt DECORATOR(t__doc__t __version__t __author__ttokenttokenizet NT_OFFSETRRRRRRRRRtSEMTOKENtNONTOKENR tNUMBERtOPtSTRINGtCOMMENTtNAMEt ERRORTOKENt ENDMARKERtINDENTtDEDENTtNEWLINEtNLtEMPTYttok_name(RRRRRRRRRRRRRRRRRRR R RRRR RR R((tPyMetrics/globals.pyt?sJ                                   pymetrics-0.8.1/PyMetrics/lexer.py0000664000076400007640000001123611041771745015307 0ustar regreg""" Parsing classes. $Id: lexer.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.2 $"[11:-2] __author__ = 'Reg. Charney ' import sys import string import cStringIO import mytoken import keyword from globals import * class ParseError(Exception): pass class Lexer: """ Parse python source.""" def __init__( self ): self.prevToktype = None self.prevSemtype = None self.prevToktext = None def parse( self, inFileName ): """ Read and parse the source. """ fd = open( inFileName ) try: srcLines = fd.read() self.srcLines = string.expandtabs( srcLines ) finally: fd.close() self.tokenlist = [] self.__computeOffsets() self.__parseSource() def __parseSource(self): """ Parse the source in file.""" self.pos = 0 text = cStringIO.StringIO( self.srcLines ) try: tokenize.tokenize( text.readline, self ) except tokenize.TokenError, ex: msg = ex[0] line = ex[1][0] print line, self.srcLines[self.offset[line]:] raise ParseError("ERROR %s\nLine %d:%s" % ( msg, line, self.srcLines[self.offset[line]:])) def __computeOffsets(self): """ Compute and store line offsets in self.offset. """ self.offset = [0, 0] self.lineCount = 0 pos = 0 while pos < len( self.srcLines ): self.lineCount += 1 pos = string.find( self.srcLines, '\n', pos ) + 1 if not pos: break self.offset.append( pos ) self.offset.append( len( self.srcLines ) ) def __push(self, toktype, semtype, toktext, srow, scol, line): "Append given token to final list of tokens." self.tokenlist.append(mytoken.MyToken(type=toktype, semtype=semtype, text=toktext, row=srow, col=scol, line=line)) if toktype in [NEWLINE,INDENT,DEDENT,EMPTY,ENDMARKER]: self.prevToktype = None self.prevSemtype = None self.prevToktext = None elif toktype != WS: self.prevToktype = toktype self.prevSemtype = semtype self.prevToktext = toktext def __call__(self, toktype, toktext, (srow,scol), (erow,ecol), line): """ MyToken handler.""" semtype = None # calculate new positions oldpos = self.pos newpos = self.offset[srow] + scol self.pos = newpos + len(toktext) # check for extraneous '\r', usually produced in Windows and Mac systems if toktype == ERRORTOKEN: # Python treats a '\r' as an error if toktext in ['\r']: toktext = ' ' toktype = WS else: msg = "Invalid character %s in line %d column %d\n" % (str.__repr__(toktext), srow, scol+1) sys.stderr.writelines( msg ) sys.stdout.writelines( msg ) # next line is commented out so that invalid tokens are not output # self.__push(toktype, None, toktext, srow, scol, line) return # handle newlines if toktype in [NEWLINE, EMPTY]: self.__push(toktype, None, '\n', srow, scol, line) return # send the original whitespace, if needed # this is really a reconstruction based on last # and current token positions and lengths. if newpos > oldpos: # srow scol is the starting position for the current # token that follows the whitespace. # srow sws is the computed starting position of the # whitespace sws = scol - ( newpos - oldpos ) self.__push(WS, None, self.srcLines[oldpos:newpos], srow, sws, line) # skip tokens that indent/dedent if toktype in [INDENT, DEDENT]: self.pos = newpos self.__push(toktype, None, '', srow, scol, line) return # map token type to one of ours and set semantic type, if possible if token.LPAR <= toktype and toktype <= OP: toktype = OP if toktext == '@': semtype = DECORATOR elif toktype == NAME: if keyword.iskeyword(toktext) or toktext == "self": semtype = KEYWORD else: semtype = VARNAME if self.prevToktext == "def": semtype = FCNNAME elif self.prevToktext == "class": semtype = CLASSNAME # add token self.__push(toktype, semtype, toktext, srow, scol, line) pymetrics-0.8.1/PyMetrics/tokenize.pyc0000664000076400007640000002430111042140621016140 0ustar regregmò "6ÖFc@s›dZddd!ZdZdZdkZdkZdkTdkZgZeeƒD]"Z e dd jo ee qRqR[d d d d gZ [ [e Z d e e e3<d?e4<d@e3<dAe4<dBe3<dCe4<dDe3<dEe4<dFe3<dGe4<dHe3<dIe4<dJd<dKd<dLd<dMde>dd„Z?de„Z@df„ZAeBdgjoQdkCZCeDeCiEƒdjoe?eFeCiEdƒiGƒq—e?eCiHiGƒndS(jsTokenization help for Python programs. generate_tokens(readline) is a generator that breaks a stream of text into Python tokens. It accepts a readline-like method which is called repeatedly to get the next line of input (or "" for EOF). It generates 5-tuples with these members: the token type (see token.py) the token (a string) the starting (row, column) indices of the token (a 2-tuple of ints) the ending (row, column) indices of the token (a 2-tuple of ints) the original line (string) It is designed to match the working of the Python tokenizer exactly, except that it produces COMMENT tokens for comments and gives type OP for all operators Older entry points tokenize_loop(readline, tokeneater) tokenize(readline, tokeneater=printtoken) are the same, except instead of generating tokens, tokeneater is a callback function to which the 5 fields described above are passed as 5 arguments, each time a new token is found. $Id: tokenize.py,v 1.3 2005/02/15 07:08:58 rcharney Exp $ s$Revision: 1.3 $i iþÿÿÿsKa-Ping Yee s@GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip MontanaroN(t*it_tCOMMENTttokenizetgenerate_tokenstNLiicGsddi|ƒdS(Nt(t|t)(tjointchoices(R ((tPyMetrics/tokenize.pytgroup/scGst|ŒdS(NR(R R (R ((R tany0scGst|ŒdS(Nt?(R R (R ((R tmaybe1ss[ \f\t]*s #[^\r\n]*s\\\r?\ns [a-zA-Z_]\w*s0[xX][\da-fA-F]*[lL]?s 0[0-7]*[lL]?s [1-9]\d*[lL]?s [eE][-+]?\d+s\d+\.\d*s\.\d+s\d+s\d+[jJ]s[jJ]s[^'\\]*(?:\\.[^'\\]*)*'s[^"\\]*(?:\\.[^"\\]*)*"s%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''s%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""s [uU]?[rR]?'''s [uU]?[rR]?"""s&[uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'s&[uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"s\*\*=?s>>=?s<<=?s<>s!=s//=?s[+\-*/%&|^=<>]=?t~s[][(){}]s\r?\ns[:;.,`]t@s%[uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*t's%[uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*t"s'''s"""sr'''sr"""su'''su"""sur'''sur"""sR'''sR"""sU'''sU"""suR'''suR"""sUr'''sUr"""sUR'''sUR"""trtRtutUsr'sr"sR'sR"su'su"sU'sU"sur'sur"sUr'sUr"suR'suR"sUR'sUR"it TokenErrorcBstZRS(N(t__name__t __module__(((R R†stStopTokenizingcBstZRS(N(RR(((R Rˆsc CsA|\}}|\}}d||||t|t|ƒfGHdS(Ns%d,%d-%d,%d: %s %s(tsrowtscolterowtecolttok_namettypetreprttoken( R!R#t.4t.6tlineRRRR((R t printtokenŠscCs+yt||ƒWntj onXdS(s: The tokenize() function accepts two parameters: one representing the input stream, and one providing an output mechanism for tokenize(). The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. The second parameter, tokeneater, must also be a callable object. It is called once for each token, with five arguments, corresponding to the tuples generated by generate_tokens(). N(t tokenize_looptreadlinet tokeneaterR(R)R*((R RŽs cCs%xt|ƒD]}||Œq WdS(N(RR)t token_infoR*(R)R*R+((R R(¡s ccsÔd}}}tidd}}d\}} d}dg}d}t idƒ}x(|ƒ}|d}dt|ƒ}}|oø|ptd|f‚n|i|ƒ} | oN| idƒ}}t||| |||f||fVd\}} d}q8| oX|ddjoG|d d jo6t||||t|ƒf|fVd}d}qUq8||}||}qUn¿|djo‹| oƒ|pPnd}x~||jop||d jo|d}nD||d jo|tdt}n||d jo d}nP|d}q¤W||joPn||djo-t||||f|t|ƒf|fVq8||djoq8||djo4|i|ƒt|| |df||f|fVnxi||djo-|d }t d||f||f|fVqÏWn'|ptd|dff‚nd}x=||jo/t!i||ƒ}|o |i#dƒ\} }|| f||f|}}}|| |!|| } } | |jp| djo$| djot)| |||fVqs| djo+|djotpt*| |||fVqs| djot+| |||fVqs| t,jo}t-| }|i||ƒ} | o9| idƒ}|| |!} t| |||f|fVqý|| f}|| }|}Pqs| t.jp"| d t.jp| d t.jox| ddjoP|| f}t-| pt-| dp t-| d}|| d}} |}Pqýt| |||fVqs| |jot/| |||fVqs| djo d}qs| djo|d}n| djo|d}nt0| |||fVq;x-||jo||djo|d7}qW||jo5t||||f||df|fV|d7}q;q;WqUWx1|dD]%}t d|df|dfdfVqˆWt2d|df|dfdfVdS(sµ The generate_tokens() generator requires one argment, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed is the logical line; continuation lines are included. iRt 0123456789tisEOF in multi-line stringiþÿÿÿs\ iýÿÿÿs\ t s s s t#iÿÿÿÿsEOF in multi-line statementt.iis s\s([{s)]}s N(R-i(ii(R-i(3tlnumtparenlevt continuedtstringt ascii_letterst namecharstnumcharstcontstrtneedconttNonetcontlinetindentststrstarttretcompiletendprogR)R&tlentpostmaxLenRtmatchtendmatchtendtSTRINGt ERRORTOKENtcolumnttabsizeRtappendtINDENTtDEDENTt pseudoprogt pseudomatchtspantstarttsposteposR#tinitialtNUMBERtNEWLINERt triple_quotedtendprogst single_quotedtNAMEtOPtindentt ENDMARKER(R)R1RBR6R\R&R=RSRFRTR9R#RQRERRRCR<R;R3ROR@RIR8R7R2((R R¥sÞ     %  )#     - ' (  ' +      / (      ' #t__main__(s'''s"""sr'''sr"""sR'''sR"""su'''su"""sU'''sU"""sur'''sur"""sUr'''sUr"""suR'''suR"""sUR'''sUR"""(RRsr'sr"sR'sR"su'su"sU'sU"sur'sur"sUr'sUr"suR'suR"sUR'sUR"(It__doc__t __version__t __author__t __credits__R4R>R#t_[1]tdirtxt__all__tN_TOKENSRR RR R Rt WhitespacetCommenttIgnoretNamet Hexnumbert Octnumbert Decnumbert IntnumbertExponentt PointfloattExpfloatt Floatnumbert ImagnumbertNumbertSingletDoubletSingle3tDouble3tTripletStringtOperatortBrackettSpecialtFunnyt PlainTokentTokentContStrt PseudoExtrast PseudoTokentmapR?t tokenprogRNt single3progt double3progR:RXRWttRYRJt ExceptionRRR'RR(RRtsysRAtargvtopenR)tstdin(<RR„RhRsR(RƒRgR€R‹RXRyR RkRRbR R{RfRRvR'RoR>RmRJR†RRR`RxR‰R4RtRwRnRaRjRR}R‚RNRRYRpRqRiRlRRˆR‡RuRcRrR#RR~RzRWR|Re((R Rs’  L           *ð     ˆ  pymetrics-0.8.1/PyMetrics/simple.pyc0000664000076400007640000001036211042140621015603 0ustar regregmò ‚ÐFc@sGdZddd!ZdZdklZdkTdefd„ƒYZd S( sn Simple Metrics for each function within a file. $Id: simple.py,v 1.4 2005/09/17 04:28:12 rcharney Exp $ s$Revision: 1.4 $i iþÿÿÿs'Reg. Charney (s MetricBase(t*t SimpleMetriccBs>tZdZd„Zd„Zd„Zd„Zed„ZRS(s$ Compute simple metrics by function.cOs…||_||_||_||_|d|_h|_h|_h|_h|_ d|_ d|_ g|id Pretty print list of functions/classes that have doc strings.s%%s DocString present(+) or missing(-)t-s t+s%c %sN( tlentnamesttypeNamethdrtresulttkeystsorttktpfxRR(R3R2R4R6R8R5R9(R(Rt __printNames[s"    it%s %14.2f %ss %11d %sRt FunctionsRtClassesN(t_SimpleMetric__printNamesRR.R R6tkeyListR7R8R tzeroSwtfmtRtappendR R (RRR8RAR>R?((RRtdisplayXs     ( t__name__t __module__t__doc__RR"R%R.RRC(((RR s     !N(RFt __version__t __author__t metricbaset MetricBasetglobalsR(RJRGRRH((Rt?s   pymetrics-0.8.1/PyMetrics/doctestsw.py0000664000076400007640000000145711041771745016213 0ustar regreg""" doctestsw - used to globally set/unset doctest testing. This switch is needed because the standard doctest can not handle raise statements with arguments. Basically, if gDoctest is True, no arguments are passed when an exception is raised. Normally, gDoctest is False and we do raise exceptions with arguments. You must modify the doctestSw value to True/False to activate/deactivate use of the doctest module tests. To used this module, use the 'from' form of the import statement and write: from doctestsw import * ... if __name__ == "__main__": if doctestSw: import doctest doctest.testmod( sys.modules[__name__] ) $Id: doctestsw.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.2 $"[11:-2] __author__ = 'Reg. Charney ' doctestSw = True pymetrics-0.8.1/PyMetrics/pymetrics-setup.diff0000644000076400007640000102244411041767514017626 0ustar regregdiff -uNr a/pymetrics-0.7.9/compute.py b/pymetrics-0.7.9/compute.py --- a/pymetrics-0.7.9/compute.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/compute.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,595 +0,0 @@ -""" Main computational modules for PyMetrics. - - $Id: compute.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ -""" - -__version__ = "$Revision: 1.2 $"[11:-2] -__author__ = 'Reg. Charney ' - -import mytoken -from globals import * -from utils import * -MAXDISPLAY = 32 - -class ComputeMetrics( object ): - """ Class used to compute basic metrics for given module.""" - def __init__( self, metricInstance, context, runMetrics, metrics, pa, so, co ): - """ Initialize general computational object.""" - self.metricInstance = metricInstance - self.context = context - self.runMetrics = runMetrics - self.metrics = metrics - self.pa = pa - self.so = so - self.co = co - - self.fqnFunction = [] - self.fqnClass = [] - self.fcnExits = [] - - self.token = None - self.stmt = [] - self.block = [] - self.fcn = [] - self.cls = [] - self.mod = [] - self.run = [] - - self.processSrcLineSubscribers = [] - self.processTokenSubscribers = [] - self.processStmtSubscribers = [] - self.processBlockSubscribers = [] - self.processFunctionSubscribers = [] - self.processClassSubscribers = [] - self.processModuleSubscribers = [] - self.processRunSubscribers = [] - - self.__initMetrics( metricInstance ) - - # this is same as function in TokenHandler !!!!!!!! - def __extractFQN( self, fqn, default='__main__' ): - """ Extract fully qualified name from list.""" - result = default - if len( fqn ): - result = fqn[-1][0] - return result - - # this is same as function in TokenHandler !!!!!!!! - def __incr( self, name ): - "Increment count in metrics dictionary based on name as key." - self.metrics[name] = self.metrics.get(name,0) + 1 - - def processSrcLines( self, srcLine ): - """ Handle processing of each physical source line. - - The fcnName and className are meaningless until the - tokens are evaluated. - """ - fcnName = self.__extractFQN( self.fqnFunction ) - className = self.__extractFQN( self.fqnClass, None ) - - for subscriber in self.processSrcLineSubscribers: - subscriber.processSrcLines( srcLine ) - - def processToken( self, tok ): - """ Handle processing after each token processed.""" - self.token = tok - self.stmt.append( tok ) # we are always in a statememt - self.block.append( tok ) # we are always in a block - if self.fqnFunction: # we are inside some function - self.fcn.append( tok ) - if self.fqnClass: # we are inside some class - self.cls.append( tok ) - self.mod.append( tok ) # we are always in some module - self.run.append( tok ) # we are always in some run - - fcnName = self.__extractFQN( self.fqnFunction ) - className = self.__extractFQN( self.fqnClass, None ) - for subscriber in self.processTokenSubscribers: - subscriber.processToken( fcnName, className, self.token ) - - def processStmt( self ): - """ Handle processing at end of statement.""" - fcnName = self.__extractFQN( self.fqnFunction ) - className = self.__extractFQN( self.fqnClass, None ) - for subscriber in self.processStmtSubscribers: - subscriber.processStmt( fcnName, className, self.stmt ) - - self.stmt[:] = [] # clear out statement list - - def processBlock( self ): - """ Handle processing at end of block.""" - fcnName = self.__extractFQN( self.fqnFunction ) - className = self.__extractFQN( self.fqnClass, None ) - for subscriber in self.processBlockSubscribers: - subscriber.processBlock( fcnName, className, self.block ) - - self.block[:] = [] # clear out block list - - def processFunction( self ): - """ Handle processing at end of function. """ - msg = self.__checkNumberOfExits() - fcnName = self.__extractFQN( self.fqnFunction ) - className = self.__extractFQN( self.fqnClass, None ) - for subscriber in self.processFunctionSubscribers: - subscriber.processFunction( fcnName, className, self.fcn ) - - self.fcn[:] = [] # clear out function list - - return msg - - def __checkNumberOfExits( self ): - """" Generate warning message if more than one exit. """ - msg = None - n = len( self.fcnExits ) - if n > 0: - exits = ', '.join( [str(i) for i in self.fcnExits] ) - plural = ((n > 1) and 's') or '' - exitStr = "exit%s at line%s" % (plural,plural) - msg = ("In file %s, function %s has %d extra %s %s" % - (self.context['inFile'], - self.__extractFQN( self.fqnFunction ), - n, - exitStr, - exits)) - for i in range( len( self.fcnExits ) ): - del self.fcnExits[0] - self.__incr( 'numMultipleExitFcns') - - return msg - - def processClass( self ): - """ Handle processing at end of class. """ - fcnName = self.__extractFQN( self.fqnFunction ) - className = self.__extractFQN( self.fqnClass, None ) - for subscriber in self.processClassSubscribers: - subscriber.processClass( fcnName, className, self.cls ) - - self.cls[:] = [] - - def processModule( self ): - """ Handle processing at end of class. """ - moduleName = self.context['inFile'] - mod = self - for subscriber in self.processModuleSubscribers: - subscriber.processModule( moduleName, mod ) - - self.mod[:] = [] - - def processRun( self ): - """ Handle processing at end of class. """ - for subsriber in self.processRunSubscribers: - subsriber.processRun( self.run ) - - self.run[:] = [] - - def __call__( self, lex ): - """ This function is the start of the heavy lifting - for computing the various metrics produced by PyMetrics.""" - - for m in self.metricInstance.keys(): - self.metricInstance[m].processSrcLines( lex.srcLines ) - - # Loop through list of tokens and count types as needed. - - # skipUntil is set after an error is detected. It is an attempt to - # find a point to restart the analysis so that we do not get - # cascading errors. This problem often occurs in analysing - # foreign character sets when normal ascii was expected. - tokenList = lex.tokenlist - - skipUntil = None - tokCount = 0 - invalidToken = mytoken.MyToken() - - for tok in tokenList: - if skipUntil: - if tok.text in skipUntil: # found where to restart analysis - skipUntil = None - elif tok.type == WS: # count, but otherwise ignore, whitespace - tokCount = tokCount+1 - self.metrics['numTokens'] = tokCount - self.__postToken( tok ) - continue - elif tok.type == ERRORTOKEN: - invalidToken.text += tok.text - continue - - tokCount = self.handleToken( tokCount, tok ) - - return self.metrics - - def __initMetrics( self, metricInstance ): - """ Initialize all the local variables that will be - needed for analysing tokens.:""" - metricList = [] - for m in metricInstance.keys(): - if metricInstance[m]: # only append valid instances - metricList.append( metricInstance[m] ) - # clear out any old data while leaving reference to same - # thing (ie., pointers to these lists are always valid - del self.processSrcLineSubscribers[:] - del self.processTokenSubscribers[:] - del self.processStmtSubscribers[:] - del self.processBlockSubscribers[:] - del self.processFunctionSubscribers[:] - del self.processClassSubscribers[:] - del self.processModuleSubscribers[:] - del self.processRunSubscribers[:] - # since all metric classes are derived from MetricBase, - # we can assign all the processX functions to all the - # metrics - self.processSrcLineSubscribers.extend( metricList ) - self.processTokenSubscribers.extend( metricList ) - self.processStmtSubscribers.extend( metricList ) - self.processBlockSubscribers.extend( metricList ) - self.processFunctionSubscribers.extend( metricList ) - self.processClassSubscribers.extend( metricList ) - self.processModuleSubscribers.extend( metricList ) - self.processRunSubscribers.extend( metricList ) - - self.numSrcLines = 0 - self.blockDepth = 0 - self.numBlocks = 0 - self.parenDepth = 0 - self.bracketDepth = 0 - self.braceDepth = 0 - self.numNestedClasses = 0 - self.numKeywords = 0 - self.numComments = 0 - self.numEmpty = 0 - self.classDepth = 0 - self.fcnDepth = 0 - self.classDepthIncr = 0 - self.fcnDepthIncr = 0 - self.maxBlockDepth = -1 - self.maxClassDepth = -1 - self.fqnName = [] - self.defFunction = False - self.defClass = False - self.docString = True - self.findFcnHdrEnd = False - self.findClassHdrEnd = False - self.inClass = False - self.inFunction = False - self.metrics['numSrcLines'] = 0 - self.metrics['numTokens'] = 0 - self.metrics['numComments'] = 0 - self.metrics['numCommentsInline'] = 0 - self.metrics['numModuleDocStrings'] = 0 - self.metrics['numBlocks'] = 0 - self.metrics['numFunctions'] = 0 - self.metrics['numClasses'] = 0 - self.className = None - self.fcnName = None - self.saveTok = None - self.skipUntil = None # used to skip invalid chars until valid char found - self.invalidToken = None - self.checkForModuleDocString = True - - return metricList - - def __fitIn( self, tok ): - """ Truncate long tokens to MAXDISPLAY length. - Also, newlines are replace with '\\n' so text fits on a line.""" - #tmpText = tok.text[:].replace( '\n', '\\n' ) - tmpText = tok.text[:] - if len( tmpText ) > MAXDISPLAY: - tmpText = tmpText[:10].strip() + ' ... ' + \ - tmpText[-10:].strip() - return tmpText - - def __incrEach( self, tok ): - """ Increment count for each unique semantic type.""" - pfx = self.__genFQN( self.fqnName ) - key = self.__fitIn( tok ) - sep = tok.semtype == KEYWORD and ' ' or '.' - if tok.semtype in [FCNNAME,CLASSNAME]: - key = pfx - else: - if pfx: - key = pfx + sep + key - else: - key = '__main__' + sep + key - key = "%-10s %s" % (token.tok_name[tok.semtype],key) - - self.metrics[key] = self.metrics.get(key,0) + 1 - - def __postToken( self, tok ): - """ Post token processing for common tasks.""" - self.__incr( tok.type ) - if tok.semtype: # then some semantic value here - self.__incr( tok.semtype ) # count semantic type - self.__incrEach( tok ) - self.processToken( tok ) - if self.pa.verbose > 1: - print self.context, tok - self.so and self.so.write( self.context, tok, self.fqnFunction, self.fqnClass ) - self.co and self.co.write( self.context, tok, self.fqnFunction, self.fqnClass ) - - def __genFQN( self, fqnName ): - """ Generate a fully qualified name. """ - result = '.'.join( fqnName ) - return result - - def handleToken( self, tokCount, tok ): - """ Common code for handling tokens.""" - if tokCount == 0: # this is the first token of the module - self.prevTok = None - tokCount += 1 - self.metrics['numTokens'] = tokCount - - # don't treat whitespace as significant when looking at previous token - if not tok.type in [WS,INDENT,DEDENT,EMPTY,ENDMARKER]: - self.prevTok = self.saveTok - self.saveTok = tok - - # set up context for current token - self.context['blockNum'] = self.metrics.get('numBlocks',0) - self.context['blockDepth'] = self.metrics.get('blockDepth',0) - self.context['parenDepth'] = self.parenDepth - self.context['bracketDepth'] = self.bracketDepth - self.context['braceDepth'] = self.braceDepth - - # self.classDepthIncr is 1 if the class definition header - # has a newline after colon, else it is equal to zero - # meaning the class definition block is on same line as its header - self.classDepth += self.classDepthIncr - self.context['classDepth'] = self.classDepth - self.classDepthIncr = 0 - - # self.classFcnIncr is 1 if the function definition header - # has a newline after colon, else it is equal to zero - # meaning the function definition block is on same line - # as its header - self.fcnDepth += self.fcnDepthIncr # only incr at start of fcn body - self.context['fcnDepth'] = self.fcnDepth - self.fcnDepthIncr = 0 - - # start testing for types that change in context - - if self.doDocString(tok): return tokCount - if self.doInlineComment(tok, self.prevTok): return tokCount - if self.doHeaders(tok): return tokCount - - # return with types that don't change in context - - self.__postToken( tok ) - if tok.type == WS: - return tokCount - - # treat end of file as end of statement - if tok.type == EMPTY or tok.type == NEWLINE: - self.processStmt() - return tokCount - - # End of file forces closure of everything, but run - if tok.type == ENDMARKER: - self.processStmt() - self.processBlock() - self.processFunction() - self.processClass() - self.processModule() - return tokCount - - # at this point, we have encountered a non-white space token - # if a module doc string has not been found yet, - # it never will be. - numModDocStrings = self.metrics['numModuleDocStrings'] - if self.checkForModuleDocString and numModDocStrings == 0: - self.checkForModuleDocString = False - msg = (("Module %s is missing a module doc string. "+ - "Detected at line %d\n") % - (self.context['inFile'],tok.row) - ) - if msg and not self.pa.quietSw: - print msg - - if self.doOperators(tok): return tokCount - if self.doIndentDedent(tok): return tokCount - - self.docString = False - self.doKeywords(tok, self.prevTok) - - return tokCount - - def doKeywords(self, tok, prevTok): - """ Count keywords and check if keyword 'return' used more than - once in a given function/method.""" - if tok.semtype == KEYWORD: - self.__incr( 'numKeywords') - if tok.text == 'def': - self.defFunction = True - elif tok.text == 'class': - self.defClass = True - elif tok.text == 'return': - assert self.fcnDepth == len( self.fqnFunction ) - if self.fcnDepth == 0: # not in any function - if not self.pa.quietSw: # report on these types of errors - print (("Module %s contains the return statement at "+ - "line %d that is outside any function") % - (self.context['inFile'],tok.row) - ) - if prevTok.text == ':': - # this return on same line as conditional, - # so it must be an extra return - self.fcnExits.append( tok.row ) - elif self.blockDepth > 1: - # Let fcnBlockDepth be the block depth of the function body. - # We are trying to count the number of return statements - # in this function. Only one is allowed at the fcnBlockDepth - # for the function. If the self.blockDepth is greater than - # fcnBlockDepth, then this is a conditional return - i.e., - # an additional return - fcnBlockDepth = self.fqnFunction[-1][1] + 1 - if self.blockDepth > fcnBlockDepth: - self.fcnExits.append( tok.row ) - - def doIndentDedent(self, tok): - """ Handle indents and dedents by keeping track of block depth. - Also, handle cases where dedents close off blocks, functions, - and classes.""" - result = False - if tok.type == INDENT: - result = self.__doIndent() - elif tok.type == DEDENT: - result = self.__doDedent() - return result - - def __doDedent(self): - """ Handle dedents and remove function/class info as needed.""" - self._doDedentFcn() - - self.fcnName = None - if len( self.fqnFunction ) > 0: - self.fcnName = self.fqnFunction[-1][0] - - while len(self.fqnClass) and self.blockDepth <= self.fqnClass[-1][1]: - self.processClass() - self.classDepth -= 1 - if self.pa.verbose > 0: - print "Removing class %s" % self.__extractFQN( self.fqnClass, None ) - del self.fqnClass[-1] - del self.fqnName[-1] - - self.className = None - if len( self.fqnClass ) > 0: - self.className = self.fqnClass[-1][0] - - return True - - def _doDedentFcn(self): - """ Remove function whose scope is ended.""" - assert self.fcnDepth == len( self.fqnFunction ) - self.blockDepth -= 1 - self.metrics['blockDepth'] = self.blockDepth - while len(self.fqnFunction) and self.blockDepth<=self.fqnFunction[-1][1]: - self.__doDedentRemoveMsg() - del self.fqnFunction[-1] - del self.fqnName[-1] - self.fcnDepth -= 1 - assert self.fcnDepth == len( self.fqnFunction ) - - def __doDedentRemoveMsg(self): - """ Output message if debugging or user asks for info.""" - msg = self.processFunction() - if msg and not self.pa.quietSw: - print msg - if self.pa.verbose > 0: - print "Removing function %s" % self.__extractFQN( self.fqnFunction ) - - def __doIndent(self): - """ Increment indent count and record if maximum depth.""" - self.__incr( 'numBlocks' ) - self.blockDepth += 1 - self.metrics['blockDepth'] = self.blockDepth - if self.metrics.get('maxBlockDepth',0) < self.blockDepth: - self.metrics['maxBlockDepth'] = self.blockDepth - return True - - def doOperators(self, tok): - """ Keep track of the number of each operator. Also, handle the - case of the colon (:) terminating a class or def header.""" - result = False - if tok.type == OP: - if tok.text == ':': - if self.findFcnHdrEnd: - self.findFcnHdrEnd = False - self.docString = True - self.fcnDepthIncr = 1 - elif self.findClassHdrEnd: - self.findClassHdrEnd = False - self.docString = True - self.classDepthIncr = 1 - result = True - elif tok.text == '(': - self.parenDepth += 1 - elif tok.text == ')': - self.parenDepth -= 1 - elif tok.text == '[': - self.bracketDepth += 1 - elif tok.text == ']': - self.bracketDepth -= 1 - elif tok.text == '{': - self.braceDepth += 1 - elif tok.text == '}': - self.braceDepth -= 1 - result = True - return result - - def doHeaders(self, tok): - """ Process both class and function headers. - Create the fully qualified names and deal - with possibly erroneous headers.""" - result = False - if self.defFunction: - if tok.type == NAME: - self.__incr( 'numFunctions') - self.fqnName.append( tok.text ) - self.fcnName = self.__genFQN( self.fqnName ) - self.fqnFunction.append( (self.fcnName,self.blockDepth,FCNNAME) ) - self.defFunction = False - self.findFcnHdrEnd = True - if self.pa.verbose > 0: - print ("fqnFunction=%s" % - toTypeName( self.context, self.fqnFunction ) - ) - self.__postToken( tok ) - result = True - elif tok.type == ERRORTOKEN: - # we must compensate for the simple scanner mishandling errors - self.findFcnHdrEnd = True - self.invalidToken = mytoken.MyToken(type=NAME, - semtype=FCNNAME, - text=tok.text, - row=tok.row, - col=tok.col, - line=tok.line) - self.skipUntil = '(:\n' - result = True - elif self.defClass and tok.type == NAME: - self.__incr( 'numClasses' ) - self.fqnName.append( tok.text ) - self.className = self.__genFQN( self.fqnName ) - self.fqnClass.append( (self.className,self.blockDepth,CLASSNAME) ) - self.defClass = False - self.findClassHdrEnd = True - if self.pa.verbose > 0: - print "fqnClass=%s" % toTypeName( self.context, self.fqnClass ) - self.__postToken( tok ) - result = True - return result - - def doInlineComment(self, tok, prevTok): - """ Check for comments and distingish inline comments from - normal comments.""" - result = False - if tok.type == COMMENT: - self.metrics['numComments'] += 1 - # compensate for older tokenize including newline - # symbols in token when only thing on line is comment - # this patch makes all comments consistent - if tok.text[-1] == '\n': - tok.text = tok.text[:-1] - - if prevTok and prevTok.type != NEWLINE and prevTok.type != EMPTY: - tok.semtype = INLINE - self.metrics['numCommentsInline'] += 1 - self.__postToken( tok ) - result = True - - return result - - def doDocString(self, tok): - """ Determine if a given string is also a docstring. - Also, detect if docstring is also the module's docsting.""" - result = False - if self.docString and tok.type == token.STRING: - tok.semtype = DOCSTRING - if self.checkForModuleDocString: # found the module's doc string - self.metrics['numModuleDocStrings'] += 1 - self.checkForModuleDocString = False - self.__postToken( tok ) - self.docString = False - result = True - return result diff -uNr a/pymetrics-0.7.9/csvout.py b/pymetrics-0.7.9/csvout.py --- a/pymetrics-0.7.9/csvout.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/csvout.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,107 +0,0 @@ -""" CvsOut - class to produce CVS data output. - - $Id: csvout.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.3 $"[11:-2] -__author__ = 'Reg. Charney ' - -import sys -import time -import token -import tokenize -from utils import * - -class CsvOut( object ): - """ Class used to generate a CSV data file suitable for input to spreadsheet program.""" - def __init__( self, fileName, genHdrSw=True, genNewSw=False ): - """ Open output file and generate header line, if desired.""" - self.fileName = fileName - self.quotedFileName = '"'+self.fileName+'"' - self.IDDateTime = '"'+time.strftime("%Y-%m-%d %H:%M:%S",time.localtime())+'"' - self.toknum = 0 - - mode = "a" - if genNewSw: - mode = "w" - try: - if self.fileName: - self.fd = open( fileName, mode ) - else: - self.fd = sys.stdout - except IOError: - raise - - self.writeHdr( genNewSw, genHdrSw ) - - def writeHdr( self, genNewSw, genHdrSw ): - """ Write header information for CSV file.""" - if genNewSw and genHdrSw: - fldNames = [ - '"IDDateTime"', - '"tokNum"', - '"inFile"', - '"line"', - '"col"', - '"tokType"', - '"semType"', - '"tokLen"', - '"token"', - '"fqnFunction"', - '"fqnClass"', - '"blockNum"', - '"blockDepth"', - '"fcnDepth"', - '"classDepth"', - '"parenDepth"', - '"bracketDepth"', - '"braceDepth"' - ] - self.fd.write( ','.join( fldNames ) ) - self.fd.write( '\n' ) - - def close( self ): - """ Close output file if it is not stdout. """ - if self.fileName: - self.fd.flush() - self.fd.close() - - def write( self, context, tok, fqnFunction, fqnClass ): - """ Generate the CSV data line.""" - self.toknum += 1 - txt = tok.text - tt = tok.type - tn = token.tok_name[tt] - - sn = '' - if tok.semtype: - sn = token.tok_name[tok.semtype] - if tt == token.NEWLINE or tt == tokenize.NL: - txt = r'\n' - - sArgs = ','.join( ( - self.IDDateTime, - str( self.toknum ), - '"'+str( context['inFile'] )+'"', - str( tok.row ), - str( tok.col ), - '"'+tn+'"', - '"'+sn+'"', - str( len( txt ) ), - csvQ( txt ), - '"'+str( toTypeName( context, fqnFunction ) )+'"', - '"'+str( toTypeName( context, fqnClass ) )+'"', - str( context['blockNum'] ), - str( context['blockDepth'] ), - str( context['fcnDepth'] ), - str( context['classDepth'] ), - str( context['parenDepth'] ), - str( context['bracketDepth'] ), - str( context['braceDepth'] ) - ) ) - self.fd.write( sArgs ) - self.fd.write( '\n' ) - - def close( self ): - """ Close file, if it is opened.""" - self.fd and self.fd.close() - self.fd = None diff -uNr a/pymetrics-0.7.9/doctestsw.py b/pymetrics-0.7.9/doctestsw.py --- a/pymetrics-0.7.9/doctestsw.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/doctestsw.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,26 +0,0 @@ -""" doctestsw - used to globally set/unset doctest testing. - -This switch is needed because the standard doctest can not handle -raise statements with arguments. Basically, if gDoctest is True, -no arguments are passed when an exception is raised. Normally, -gDoctest is False and we do raise exceptions with arguments. - -You must modify the doctestSw value to True/False to -activate/deactivate use of the doctest module tests. - -To used this module, use the 'from' form of the import -statement and write: - -from doctestsw import * -... -if __name__ == "__main__": - if doctestSw: - import doctest - doctest.testmod( sys.modules[__name__] ) - -$Id: doctestsw.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.2 $"[11:-2] -__author__ = 'Reg. Charney ' - -doctestSw = True diff -uNr a/pymetrics-0.7.9/globals.py b/pymetrics-0.7.9/globals.py --- a/pymetrics-0.7.9/globals.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/globals.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,47 +0,0 @@ -""" Global variables for PyMetrics. - - $Id: globals.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.3 $"[11:-2] -__author__ = 'Reg. Charney ' - -import token -import tokenize - -# our token types -KEYWORD = token.NT_OFFSET + 1 -TEXT = token.NT_OFFSET + 2 -WS = token.NT_OFFSET + 3 -DOCSTRING = token.NT_OFFSET + 4 -VARNAME = token.NT_OFFSET + 5 -CLASSNAME = token.NT_OFFSET + 6 -FCNNAME = token.NT_OFFSET + 7 -INLINE = token.NT_OFFSET + 8 -UNKNOWN = token.NT_OFFSET + 9 -SEMTOKEN = token.NT_OFFSET + 10 # to distinguish semantic tokens -NONTOKEN = token.NT_OFFSET + 11 # to distinguish non-tokens -DECORATOR = token.NT_OFFSET + 12 # to indicate decorator token -NUMBER = token.NUMBER -OP = token.OP -STRING = token.STRING -COMMENT = tokenize.COMMENT -NAME = token.NAME -ERRORTOKEN = token.ERRORTOKEN -ENDMARKER = token.ENDMARKER -INDENT = token.INDENT -DEDENT = token.DEDENT -NEWLINE = token.NEWLINE -EMPTY = tokenize.NL - -# new token types added to allow for character representation of new codes - -token.tok_name[KEYWORD] = "KEYWORD" # one of Python's reserved words -token.tok_name[TEXT] = "TEXT" # obsolete - but kept for compatibility -token.tok_name[WS] = "WS" # some form of whitespace -token.tok_name[DOCSTRING] = "DOCSTRING" # literal that is also doc string -token.tok_name[VARNAME] = "VARNAME" # name that is not keyword -token.tok_name[CLASSNAME] = "CLASSNAME" # name defined in class statment -token.tok_name[FCNNAME] = "FCNNAME" # name defined in def statement -token.tok_name[INLINE] = "INLINE" # comment that follows other text on same line -token.tok_name[UNKNOWN] = "UNKNOWN" # Unknown semantic type - this should not occur -token.tok_name[DECORATOR] = 'DECORATOR' # Decorator marker diff -uNr a/pymetrics-0.7.9/halstead.py b/pymetrics-0.7.9/halstead.py --- a/pymetrics-0.7.9/halstead.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/halstead.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,266 +0,0 @@ -""" Compute HalsteadMetric Metrics. - -HalsteadMetric metrics, created by Maurice H. HalsteadMetric in 1977, consist -of a number of measures, including: - -Program length (N): N = N1 + N2 -Program vocabulary (n): n = n1 + n2 -Volume (V): V = N * LOG2(n) -Difficulty (D): D = (n1/2) * (N2/n2) -Effort (E): E = D * V -Average Volume (avgV) avgV = sum(V)/m -Average Effort (avgE) avgE = sum(E)/m - -where: - -n1 = number of distinct operands -n2 = number of distinct operators -N1 = total number of operands -N2 = total number of operators -m = number of modules - -What constitues an operand or operator is often open to -interpretation. In this implementation for the Python language: - - operators are of type OP, INDENT, DEDENT, or NEWLINE since these - serve the same purpose as braces and semicolon in C/C++, etc. - operands are not operators or whitespace or comments - (this means operands include keywords) - - $Id: halstead.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.3 $"[11:-2] -__author__ = 'Reg. Charney ' - -import math -import time -from metricbase import MetricBase -from globals import * - -class HalsteadMetric( MetricBase ): - """ Compute various HalsteadMetric metrics. """ - totalV = 0 - totalE = 0 - numModules = 0 - def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): - """ Initialization for the HalsteadMetric metrics.""" - self.inFile = context['inFile'] - self.context = context - self.runMetrics = runMetrics - self.metrics = metrics - self.pa = pa - self.inFile = context['inFile'] - self.numOperators = 0 - self.numOperands = 0 - self.uniqueOperators = {} - self.uniqueOperands = {} - HalsteadMetric.numModules += 1 - - # initialize category accummulators as dictionaries - self.hsDict = {} - for t in ['token','stmt','block','function','class','module','run']: - self.uniqueOperators[t] = {} - self.uniqueOperands[t] = {} - #for v in ['N','N1','N2','n','n1','n2','V','D','E','avgV','avgE']: - # self.hsDict[(t,v)] = 0 - - def processToken( self, currentFcn, currentClass, tok, *args, **kwds ): - """ Collect token data for Halstead metrics.""" - if tok.type in [WS, EMPTY, ENDMARKER, NEWLINE, EMPTY, COMMENT]: - pass - elif tok.type in [OP, INDENT, DEDENT]: - self.numOperators += 1 - self.uniqueOperators['token'][tok.text] = self.uniqueOperators['token'].get(tok.text, 0) + 1 - else: - self.numOperands += 1 - sDict = self.context.__repr__() - k = (sDict,tok.text) - self.uniqueOperands['token'][k] = self.uniqueOperands['token'].get(tok.text, 0) + 1 - - def processStmt( self, currentFcn, currentClass, stmt, *args, **kwds ): - """ Collect statement data for Halstead metrics.""" - - result = None - - # the two lines following this comment would compute the Halstead - # metrics for each statement in the run, However, it is - # normally overkill, so these lines are commented out. - - #lineNum = stmt[0].row - #result = self.computeCategory( 'stmt', lineNum, stmt ) - - return result - - def processBlock( self, currentFcn, currentClass, block, *args, **kwds ): - """ Collect block data for Halstead metrics.""" - - result = None - - # the two lines following this comment would compute the Halstead - # metrics for each statement in the run, However, it is - # normally overkill, so the two lines are commented out. - - #blockNum = self.context['blockNum'] - #result = self.computeCategory( 'block', blockNum, block ) - - return result - - def processFunction( self, currentFcn, currentClass, fcn, *args, **kwds ): - """ Collect function data for Halstead metrics.""" - result = self.computeCategory( 'function', currentFcn, fcn ) - return result - - def processClass( self, currentFcn, currentClass, cls, *args, **kwds ): - """ Collect class data for Halstead metrics.""" - result = self.computeCategory( 'class', currentClass, cls ) - return result - - def processModule( self, moduleName, mod, *args, **kwds ): - """ Collect module data for Halstead metrics.""" - result = self.computeCategory( 'module', moduleName, mod ) - return result - - def processRun( self, run, *args, **kwds ): - """ Collect run data for Halstead metrics.""" - datestamp = time.strftime("%Y-%m-%d.%H:%m%Z",time.localtime()) - result = self.computeCategory( 'run', datestamp, run ) - return result - - def __LOGb( self, x, b ): - """ convert to LOGb(x) from natural logs.""" - try: - result = math.log( x ) / math.log ( b ) - except OverflowError: - result = 1.0 - return result - - def computeIncr( self, cat, tok, uniqueOperators, uniqueOperands ): - """ Compute increment for token depending on which category it falls into.""" - operatorIncr = operandIncr = 0 - if tok.type in [WS, EMPTY, ENDMARKER, NEWLINE, EMPTY, COMMENT]: - return (operatorIncr,operandIncr) - - if tok.type in [OP, INDENT, DEDENT]: - operatorIncr = 1 - uniqueOperators[tok.text] = uniqueOperators.get(tok.text, 0) + 1 - else: - operandIncr = 1 - uniqueOperands[tok.text] = uniqueOperands.get(tok.text,0) + 1 - - return (operatorIncr,operandIncr) - - def computeCategory( self, cat, mod, lst ): - """ Collection data for cat of code.""" - modID= id( mod ) - numOperators = numOperands = 0 - for tok in lst: - result = self.computeIncr( cat, tok, self.uniqueOperators[cat], self.uniqueOperands[cat] ) - numOperators += result[0] - numOperands += result[1] - result = self.compute( cat, modID, numOperators, numOperands, self.uniqueOperators[cat], self.uniqueOperands[cat] ) - return result - - def compute( self, cat, modID, numOperators, numOperands, uniqueOperators, uniqueOperands, *args, **kwds ): - """ Do actual calculations here.""" - - n1 = len( uniqueOperands ) - n2 = len( uniqueOperators ) - N1 = numOperands - N2 = numOperators - N = N1 + N2 - n = n1 + n2 - V = float(N) * self.__LOGb( n, 2 ) - try: - D = (float(n1)/2.0) * (float(N2)/float(n2)) - except ZeroDivisionError: - D = 0.0 - E = D * V - HalsteadMetric.totalV += V - HalsteadMetric.totalE += E - avgV = HalsteadMetric.totalV / HalsteadMetric.numModules - avgE = HalsteadMetric.totalE / HalsteadMetric.numModules - - self.hsDict[(cat,modID,'n1')] = n1 - self.hsDict[(cat,modID,'n2')] = n2 - self.hsDict[(cat,modID,'N1')] = N1 - self.hsDict[(cat,modID,'N2')] = N2 - self.hsDict[(cat,modID,'N')] = N - self.hsDict[(cat,modID,'n')] = n - self.hsDict[(cat,modID,'V')] = V - self.hsDict[(cat,modID,'D')] = D - self.hsDict[(cat,modID,'E')] = E - self.hsDict[(cat,modID,'numModules')] = HalsteadMetric.numModules - self.hsDict[(cat,modID,'avgV')] = avgV - self.hsDict[(cat,modID,'avgE')] = avgE - - return self.hsDict - - def display( self, cat=None ): - """ Display the computed Halstead Metrics.""" - if self.pa.quietSw: - return self.hsDict - - hdr = "\nHalstead Metrics for %s" % self.inFile - print hdr - print "-"*len(hdr) + '\n' - - if len( self.hsDict ) == 0: - print "%-8s %-30s " % ('**N/A**','All Halstead metrics are zero') - return self.hsDict - - keyList = self.hsDict.keys() - keyList.sort() - if 0: - for k,i,v in keyList: - if cat: - if k!=cat: - continue - print "%14.2f %s %s %s" % (self.hsDict[(k,i,v)],k,i,v) - print - hdr1 = "Category Identifier D E N N1 N2 V avgE avgV n n1 n2" - hdr2 = "-------- ---------------------------------- -------- -------- ----- ---- ---- -------- -------- -------- ----- ---- ----" - # 12345678 123456789012345678901234567890 12345678 12345678 12345 1234 1234 12345678 12345678 12345678 12345 1234 1234 - fmt1 = "%-8s %-33s " - fmt2 = "%8.2e %8.2e %5d %4d %4d %8.2e %8.2e %8.2e %5d %4d %4d" - - # this loop uses the Main Line Standards break logic. It does this to convert the - # normal vertical output to a horizontal format. The control variables are the - # category name and the identifier value. - - oldK = oldI = None - vDict = {} - vList = [] - hdrSw = True # output header for first time thru - for k,i,v in keyList: - # only print data for the category we want - if cat: - if k != cat: - continue - - if v == "numModules": # ignore this value for now - continue - - if (oldK,oldI) != (k,i): # change in category/id - if oldK and oldI: # this is not first time thru - #t = tuple([self.hsDict[(k,i,v)] for v in vList]) - t = tuple([vDict[v] for v in vList]) - print fmt1 % (k,i), - print fmt2 % t - # initialize for next set of category/id - vDict = {} - vDict[v] = self.hsDict[(k,i,v)] - vList = [] - vList.append( v ) - oldK = k - oldI = i - if hdrSw: - print hdr1 - print hdr2 - hdrSw = False - else: # we are still in the same category/id - vDict[v] = self.hsDict[(k,i,v)] - vList.append( v ) - - print - - return self.hsDict diff -uNr a/pymetrics-0.7.9/__init__.py b/pymetrics-0.7.9/__init__.py --- a/pymetrics-0.7.9/__init__.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/__init__.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,21 +0,0 @@ -""" Used to indicate that this is a Python module directory - - PyMetrics - produce a variety of metrics for Python programs - - Copyright (c) 2005 by Reg. Charney - All rights reserved, see LICENSE for details. - - This program is free software; you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - $Id: __init__.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.2 $"[11:-2] -__author__ = 'Reg. Charney ' diff -uNr a/pymetrics-0.7.9/lexer.py b/pymetrics-0.7.9/lexer.py --- a/pymetrics-0.7.9/lexer.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/lexer.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,139 +0,0 @@ -""" Parsing classes. - - $Id: lexer.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.2 $"[11:-2] -__author__ = 'Reg. Charney ' - -import sys -import string -import cStringIO -import mytoken -import keyword -from globals import * - -class ParseError(Exception): - pass - -class Lexer: - """ Parse python source.""" - def __init__( self ): - self.prevToktype = None - self.prevSemtype = None - self.prevToktext = None - - def parse( self, inFileName ): - """ Read and parse the source. """ - fd = open( inFileName ) - try: - srcLines = fd.read() - self.srcLines = string.expandtabs( srcLines ) - finally: - fd.close() - - self.tokenlist = [] - - self.__computeOffsets() - self.__parseSource() - - def __parseSource(self): - """ Parse the source in file.""" - self.pos = 0 - text = cStringIO.StringIO( self.srcLines ) - try: - tokenize.tokenize( text.readline, self ) - except tokenize.TokenError, ex: - msg = ex[0] - line = ex[1][0] - print line, self.srcLines[self.offset[line]:] - raise ParseError("ERROR %s\nLine %d:%s" % ( - msg, line, self.srcLines[self.offset[line]:])) - - def __computeOffsets(self): - """ Compute and store line offsets in self.offset. """ - self.offset = [0, 0] - self.lineCount = 0 - pos = 0 - while pos < len( self.srcLines ): - self.lineCount += 1 - pos = string.find( self.srcLines, '\n', pos ) + 1 - if not pos: break - self.offset.append( pos ) - self.offset.append( len( self.srcLines ) ) - - - def __push(self, toktype, semtype, toktext, srow, scol, line): - "Append given token to final list of tokens." - - self.tokenlist.append(mytoken.MyToken(type=toktype, semtype=semtype, text=toktext, row=srow, col=scol, line=line)) - if toktype in [NEWLINE,INDENT,DEDENT,EMPTY,ENDMARKER]: - self.prevToktype = None - self.prevSemtype = None - self.prevToktext = None - elif toktype != WS: - self.prevToktype = toktype - self.prevSemtype = semtype - self.prevToktext = toktext - - - def __call__(self, toktype, toktext, (srow,scol), (erow,ecol), line): - """ MyToken handler.""" - semtype = None - - # calculate new positions - oldpos = self.pos - newpos = self.offset[srow] + scol - self.pos = newpos + len(toktext) - - # check for extraneous '\r', usually produced in Windows and Mac systems - if toktype == ERRORTOKEN: # Python treats a '\r' as an error - if toktext in ['\r']: - toktext = ' ' - toktype = WS - else: - msg = "Invalid character %s in line %d column %d\n" % (str.__repr__(toktext), srow, scol+1) - sys.stderr.writelines( msg ) - sys.stdout.writelines( msg ) - # next line is commented out so that invalid tokens are not output - # self.__push(toktype, None, toktext, srow, scol, line) - return - - # handle newlines - if toktype in [NEWLINE, EMPTY]: - self.__push(toktype, None, '\n', srow, scol, line) - return - - # send the original whitespace, if needed - # this is really a reconstruction based on last - # and current token positions and lengths. - if newpos > oldpos: - # srow scol is the starting position for the current - # token that follows the whitespace. - # srow sws is the computed starting position of the - # whitespace - sws = scol - ( newpos - oldpos ) - self.__push(WS, None, self.srcLines[oldpos:newpos], srow, sws, line) - - # skip tokens that indent/dedent - if toktype in [INDENT, DEDENT]: - self.pos = newpos - self.__push(toktype, None, '', srow, scol, line) - return - - # map token type to one of ours and set semantic type, if possible - if token.LPAR <= toktype and toktype <= OP: - toktype = OP - if toktext == '@': - semtype = DECORATOR - elif toktype == NAME: - if keyword.iskeyword(toktext) or toktext == "self": - semtype = KEYWORD - else: - semtype = VARNAME - if self.prevToktext == "def": - semtype = FCNNAME - elif self.prevToktext == "class": - semtype = CLASSNAME - - # add token - self.__push(toktype, semtype, toktext, srow, scol, line) diff -uNr a/pymetrics-0.7.9/mccabe.py b/pymetrics-0.7.9/mccabe.py --- a/pymetrics-0.7.9/mccabe.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/mccabe.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,75 +0,0 @@ -""" Compute McCabe's Cyclomatic Metric. - - This routine computes McCabe's Cyclomatic metric for each function - in a module/file. - - $Id: mccabe.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.3 $"[11:-2] -__author__ = 'Reg. Charney ' - -from metricbase import MetricBase - -McCabeKeywords = { - 'assert':0, - 'break':1, - 'continue':1, - 'elif':1, - 'else':1, - 'for':1, - 'if':1, - 'while':1 - } - -class McCabeMetric( MetricBase ): - """ Compute McCabe's Cyclomatic McCabeMetric by function.""" - def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): - self.context = context - self.runMetrics = runMetrics - self.metrics = metrics - self.pa = pa - self.inFile = context['inFile'] - self.fcnNames = {} - - def processToken( self, fcnName, className, tok, *args, **kwds ): - """ Increment number of decision points in function.""" - if tok and tok.text in McCabeKeywords: - self.fcnNames[fcnName] = self.fcnNames.get(fcnName,0) + 1 - - def processFunction( self, fcnName, className, fcn, *args, **kwds ): - """ Increment number of decision points in function.""" - self.fcnNames[fcnName] = self.fcnNames.get(fcnName,0) + 1 - - def display( self ): - """ Display McCabe Cyclomatic metric for each function """ - result = {} - # the next three lines ensure that fcnNames[None] is treated - # like fcnNames['__main__'] and are sorted into alphabetical - # order. - if self.fcnNames.has_key(None): - self.fcnNames['__main__'] = self.fcnNames.get(None,0) - del self.fcnNames[None] - - if self.pa.quietSw: - return result - - hdr = "\nMcCabe Complexity Metric for file %s" % self.inFile - print hdr - print "-"*len(hdr) + "\n" - keyList = self.fcnNames.keys() - if len( keyList ) > 0: - keyList.sort() - for k in keyList: - if k: - name = k - else: - name = "__main__" - print "%11d %s" % (self.fcnNames[k],name) - result[k] = self.fcnNames[k] - else: - print "%11d %s" % (1,'__main__') - result['__main__'] = 1 - - print - return result - diff -uNr a/pymetrics-0.7.9/metricbase.py b/pymetrics-0.7.9/metricbase.py --- a/pymetrics-0.7.9/metricbase.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/metricbase.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,48 +0,0 @@ -""" Metric base class for new user-defined metrics. - - $Id: metricbase.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.2 $"[11:-2] -__author__ = 'Reg. Charney ' - -class MetricBase( object ): - """ Metric template class.""" - def __init__( self, *args, **kwds ): - pass - - def processSrcLines( self, srcLines, *args, **kwds ): - """ Handle physical line after tab expansion.""" - pass - - def processToken( self, fcnName, className, tok, *args, **kwds ): - """ Handle processing after each token processed.""" - pass - - def processStmt( self, fcnName, className, stmt, *args, **kwds ): - """ Handle processing at end of statement.""" - pass - - def processBlock( self, fcnName, className, block, *args, **kwds ): - """ Handle processing at end of block.""" - pass - - def processFunction( self, fcnName, className, fcn, *args, **kwds ): - """ Handle processing at end of function. """ - pass - - def processClass( self, fcnName, className, cls, *args, **kwds ): - """ Handle processing at end of class. """ - pass - - def processModule( self, moduleName, module, *args, **kwds ): - """ Handle processing at end of module. """ - pass - - def processRun( self, run, *args, **kwds ): - """ Handle processing at end of run. """ - pass - - def compute( self, *args, **kwds ): - """ Compute the metric given all needed info known.""" - pass - diff -uNr a/pymetrics-0.7.9/mytoken.py b/pymetrics-0.7.9/mytoken.py --- a/pymetrics-0.7.9/mytoken.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/mytoken.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,33 +0,0 @@ -""" MyToken - PyMetrics' version of Token. - - $Id: mytoken.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.3 $"[11:-2] -__author__ = 'Reg. Charney ' - -import token -from globals import * - -class MyToken: - def __init__(self, **kwds ): - """ Initialize class with user-defined keywords.""" - self.__dict__.update(kwds) - - def __repr__( self ): - """ Pretty print token. - - Don't print text for special token types since they do not have - useful visible representation, other than blank. - """ - tn = token.tok_name[self.type] - sn = self.semtype - if sn: - sn = token.tok_name[self.semtype] - if self.type in [WS,NEWLINE,INDENT,DEDENT,EMPTY,ENDMARKER]: - s = "[type=%s semtype=%s row=%s col=%s len=%d]" % (tn,sn,self.row,self.col,len(self.text)) - else: - if self.type == COMMENT and self.text[-1] == '\n': - self.text = self.text[:-1] - s = "[type=%s semtype=%s row=%s col=%s len=%d text=<%s>]" % (tn,sn,self.row,self.col,len(self.text),self.text) - return s - diff -uNr a/pymetrics-0.7.9/newFeatures24.py b/pymetrics-0.7.9/newFeatures24.py --- a/pymetrics-0.7.9/newFeatures24.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/newFeatures24.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,204 +0,0 @@ -""" Metrics for New Features introduced in Python 2.4. - - $Id: newfeatures24.py,v 1.4 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.1 $"[11:-2] -__author__ = 'Reg. Charney ' - -from metricbase import MetricBase -from globals import * - -class NewFeatures24Metric( MetricBase ): - """ Compute simple metrics by function.""" - firstPass = True - def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): - """ Count features introduced after v2.3.""" - self.context = context - self.runMetrics = runMetrics - self.metrics = metrics - self.pa = pa - self.inFile = context['inFile'] - - self.prevTokText = '' - self.userDefinedSet = False - self.featureMetrics = {} - self.numGeneratorFunctions = 0 - self.numGeneratorExpressions = 0 - self.numListComprehension = 0 - self.numClassProperties = 0 - self.numClassVariables = 0 - self.numClassFunctions = 0 - self.numDecorators = 0 - self.isGeneratorFunction = False - self.isGeneratorExpression = False - self.isListComprehension = False - self.usedInbuiltSets = False - self.usedDecorators = False - self.maybeSetKeyword = False - self.stackParenOrBracket = []; - self.featureMetrics['numGeneratorFunctions'] = 0 - self.featureMetrics['numGeneratorExpressions'] = 0 - self.featureMetrics['numListComprehensions'] = 0 - self.featureMetrics['numModulesUsingSets'] = 0 - self.featureMetrics['numDecorators'] = 0 - - if NewFeatures24Metric.firstPass: - self.runMetrics['modulesUsingGeneratorFunctions'] = [] - self.runMetrics['modulesUsingGeneratorExpressions'] = [] - self.runMetrics['modulesUsingListComprehension'] = [] - self.runMetrics['modulesUsingSets'] = [] - self.runMetrics['modulesUsingDecorators'] = [] - NewFeatures24Metric.firstPass = False - - def processToken( self, currentFcn, currentClass, tok, *args, **kwds ): - """ Collect token and context sensitive data for simple metrics.""" - if tok.semtype == KEYWORD: - self.__handleKeywords( currentFcn, currentClass, tok, *args, **kwds ) - elif tok.type == OP: - if self.maybeSetKeyword and tok.text == '(': - if not self.userDefinedSet: - self.usedInbuiltSets = True - self.maybeSetKeyword = False - elif tok.text == '(' or tok.text == '[': - self.stackParenOrBracket.append( tok.text ) - elif tok.text == ')' or tok.text == ']': - if len( self.stackParenOrBracket ) > 0: - del self.stackParenOrBracket[-1] - elif tok.text == '@': - self.usedDecorators = True - self.featureMetrics['numDecorators'] += 1 - elif tok.semtype == VARNAME: - if tok.text in ['set', 'frozenset']: - if not self.prevTokText in ['.', 'def', 'class']: - self.maybeSetKeyword = True - elif tok.semtype in [FCNNAME,CLASSNAME]: - # We need to ignore user-defined global set - # functions and classes. - if tok.text in ['set', 'frozenset']: - self.userDefinedSet = True - if tok.type != WS: - self.prevTokText = tok.text - return - - def __handleKeywords( self, currentFcn, currentClass, tok, *args, **kwds ): - """ Check for generator functions or expressions and list comprehension.""" - if tok.text == "yield": - self.isGeneratorFunction = True - elif tok.text == "for": - if len( self.stackParenOrBracket ) > 0: - punct = self.stackParenOrBracket[-1] - if punct == '(': - self.featureMetrics['numGeneratorExpressions'] += 1 - elif punct == '[': - self.featureMetrics['numListComprehensions'] += 1 - return - - def processStmt( self, fcnName, className, stmt, *args, **kwds ): - """ Handle processing at end of statement.""" - self.stackParenOrBracket = [] - - def processFunction( self, fcnName, className, block, *args, **kwds ): - """ Output stats at end of each function.""" - if self.isGeneratorFunction: - self.featureMetrics['numGeneratorFunctions'] += 1 - self.isGenerator = False - if self.usedInbuiltSets: - self.featureMetrics['numModulesUsingSets'] += 1 - self.usedInbuiltSets = False - - def processModule( self, moduleName, mod, *args, **kwds ): - """ Output stats at end of each module.""" - self.moduleName = moduleName - if self.featureMetrics['numModulesUsingSets'] > 0: - self.runMetrics['modulesUsingSets'].append( moduleName ) - if self.featureMetrics['numGeneratorFunctions'] > 0: - self.runMetrics['modulesUsingGeneratorFunctions'].append( moduleName ) - if self.featureMetrics['numGeneratorExpressions'] > 0: - self.runMetrics['modulesUsingGeneratorExpressions'].append( moduleName ) - if self.featureMetrics['numListComprehensions'] > 0: - self.runMetrics['modulesUsingListComprehension'].append( moduleName ) - if self.featureMetrics['numDecorators'] > 0: - self.runMetrics['modulesUsingDecorators'].append( moduleName ) - return - - def processRun( self, run, *args, **kwds ): - """ Output stats at end of run.""" - def __printHeader( printHeader ): - """ Only print heading if something in body of report.""" - if printHeader: - print """Python 2.4 Features Used During Run""" - print """-----------------------------------""" - return False - - def __printSubHeader( printHeader, key, desc ): - if len( self.runMetrics[key] ) > 0: - printHeader = __printHeader( printHeader ) - h1 = "Modules using %s" % desc - h2 = '.'*len( h1 ) - print - print h1 - print h2 - print - for modName in self.runMetrics[key]: - print modName - return False - - printHeader = True - printHeader = __printSubHeader( printHeader, - 'modulesUsingSets', - 'builtin set/frozenset (PEP 218)' ) - printHeader = __printSubHeader( printHeader, - 'modulesUsingGeneratorFunctions', - 'Generator Functions' ) - printHeader = __printSubHeader( printHeader, - 'modulesUsingGeneratorExpressions (PEP 289)', - 'Generator Expressions' ) - printHeader = __printSubHeader( printHeader, - 'modulesUsingListComprehension', - 'List Comprehension' ) - printHeader = __printSubHeader( printHeader, - 'modulesUsingDecorators', - 'Decorators (PEP 318)' ) - - return None - - def compute( self, *args, **kwds ): - """ Compute any values needed.""" - try: - self.featureMetrics['%FunctionsThatAreGenerators'] = 100.0 * self.featureMetrics['numGeneratorFunctions']/self.metrics['numFunctions'] - except (KeyError, ZeroDivisionError): - self.featureMetrics['%FunctionsThatAreGenerators'] = 0.0 - - return self.featureMetrics - - def display( self, currentFcn=None ): - """ Display and return new features in 2.4 metrics for given function.""" - def __printDisplayHdr(): - """ Only print heading if something in body of report.""" - h1 = """Python 2.4 Features Used in %s""" % self.inFile - h2 = '-'*len( h1 ) - print h1 - print h2 - print - return False - - printHeader = True - self.compute() - keyList = self.featureMetrics.keys() - keyList.sort() - for k in keyList: - if self.pa.zeroSw or self.featureMetrics[k]: - fmt = ( k[0] == '%' and "%14.2f %s" ) or "%11d %s" - if printHeader: printHeader = __printDisplayHdr() - print fmt % (self.featureMetrics[k],k) - print - - return self.featureMetrics - -if __name__=="__main__": - def Incr( init ): - for i in range(10): - yield i - s = set(i for i in range(0,10,2)) - fs = frozenset(i for i in range(5)) - print s, fs diff -uNr a/pymetrics-0.7.9/processargs.py b/pymetrics-0.7.9/processargs.py --- a/pymetrics-0.7.9/processargs.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/processargs.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,483 +0,0 @@ -""" Process command line arguments. - -Usage: - ->>> pa = ProcessArgs( 'file1.py', 'file2.py', '/home/files/file3.py' ) ->>> pa = ProcessArgs( "inFile1.py", sqlFileName='sqlF1.sql', genCsvSw=False,genKwCntSw=True ) ->>> pa = ProcessArgs() #doctest +NORMALIZE_WHITESPACE +ELLIPSIS -python PyMetrics [ options ] pgm1.py [ pgm2.py ... ] - -Complexity metrics are computed for the Python input files -pgm1.py, pgm2.py, etc. At least one file name is required, -else this message appears. - -Three types of output can be produced: - -* Standard output for a quick summary of the main metrics. -* A text file containing SQL commands necessary to build a SQL table - in the database of your choice. -* A text file containing Comma-Separated Values (CSV) formatted data - necessary to load into most spreadsheet programs. - PLEASE NOTE: multi-line literals have the new-line character - replaced with the character "\\n" in the output text. - -Capitalized options negate the default option. - -options: - --version show program's version number and exit - -h, --help show this help message and exit - -s SQLFILENAME, --sql=SQLFILENAME - name of output SQL command file. (Default is - metricData.sql) - -t SQLTOKENTABLENAME, --tokentable=SQLTOKENTABLENAME - name of output SQL token table. (Default is - metricTokens) - -m SQLMETRICSTABLENAME, --metricstable=SQLMETRICSTABLENAME - name of output SQL metrics table. (Default is - metricData) - -c CSVFILENAME, --csv=CSVFILENAME - name of output CSV data file. (Default is - metricData.csv) - -f INFILELIST, --files=INFILELIST - File containing list of path names to modules for - analysis. - -i INCLUDEMETRICSSTR, --include=INCLUDEMETRICSSTR - list of metrics to include in run. This is a comma - separated list of metric module names with no - whitespace. Optionally, you can specify the class name - of the metric by following the module name with a - colon (:) and the metric class name. (Default metrics - are 'simple:SimpleMetric,mccabe:McCabeMetric, - sloc:SLOCMetric'. Default metric class name for metric - module 'wxYz' is 'WxYzMetric' when only module name - given -- note capitalized metric class name.) - -l LIBNAME, --library=LIBNAME - user-defined name applied to collection of modules - (Default is '') - -e, --exists assume SQL tables exist and does not generate creation - code. Using this option sets option -N. (Default is - False) - -N, --noold create new command output files and tables after - deleting old results, if any. Ignored if -e is set. - (Default is False) - -B, --nobasic suppress production of Basic metrics (Default is - False) - -S, --nosql suppress production of output SQL command text file. - (Default is False) - -C, --nocsv suppress production of CSV output text file. (Default - is False) - -H, --noheadings suppress heading line in csv file. (Default is False) - -k, --kwcnt generate keyword counts. (Default is False) - -K, --nokwcnt suppress keyword counts. (Default is True) - -q, --quiet suppress normal summary output to stdout. (Default is - False) - -z, --zero display zero or empty values in output to stdout. - (Default is to suppress zero/empty output) - -v, --verbose Produce verbose output - more -v's produce more - output. (Default is no verbose output to stdout) - -d, --debug Provide debug output, not usually generated - internal - use only - -No program file names given. - - - $Id: processargs.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.3 $"[11:-2] -__author__ = 'Reg. Charney ' - -import sys -from optparse import OptionParser, BadOptionError -from doctestsw import * - -usageStr = """python PyMetrics [ options ] pgm1.py [ pgm2.py ... ] - -Complexity metrics are computed for the Python input files -pgm1.py, pgm2.py, etc. At least one file name is required, -else this message appears. - -Three types of output can be produced: - -* Standard output for a quick summary of the main metrics. -* A text file containing SQL commands necessary to build a SQL table - in the database of your choice. -* A text file containing Comma-Separated Values (CSV) formatted data - necessary to load into most spreadsheet programs. - PLEASE NOTE: multi-line literals have the new-line character - replaced with the character "\\n" in the output text. - -Capitalized options negate the default option. -""" - -class PyMetricsOptionParser( OptionParser ): - """ Subclass OptionParser so I can override default error handler.""" - def __init__( self, *args, **kwds ): - """ Just call super class's __init__ since we aren't making changes here.""" - OptionParser.__init__( self, *args, **kwds ) - - def error( self, msg ): - """ Explicitly raise BadOptionError so calling program can handle it.""" - raise BadOptionError( msg ) - -class ProcessArgsError( Exception ): pass - -class ProcessArgs( object ): - """ Process command line arguments.""" - def __init__( self, - *pArgs, - **pKwds - ): - """ Initial processing of arguments.""" - - # precedence for defaults, parameters, and command line arguments/keywords - # is: - # - # command line parameters/keywords take precedence over - # parameters used to instantiate ProcessArgs that takes precedence over - # default values for keywords. - # - # This implies we set completed options with defaults first - # then override completed options with parameters - # then override completed options with command line args/keywords - # - - # default values for possible parameters - libName = '' - sqlFileName = "metricData.sql" - sqlTokenTableName = "metricTokens" - sqlMetricsTableName = "metricData" - csvFileName = "metricData.csv" - inFileList = None - includeMetricsStr = 'simple:SimpleMetric,mccabe:McCabeMetric,sloc:SLOCMetric' - excludeMetricsStr = None - debugSw = False - debugSw = False - quietSw = False - zeroSw = False - genBasicSw = True - genKwCntSw = False - genSqlSw = True - genCsvSw = True - genHdrSw = True - genAppendSw = True - genNewSw = False - genExistsSw = False - verbose = 0 - - self.__dict__.update( locals() ) - del( self.__dict__['self'] ) # remove recursive self from self.__dict__ - self.__dict__.update( pKwds ) - del( self.__dict__['pKwds'] ) # remove redundant pKwds in self.__dict__ - - # set up option parser - parser = PyMetricsOptionParser( '', version="%prog 0.7.9" ) - parser.add_option("-s", "--sql", - dest="sqlFileName", - default=self.sqlFileName, - help="name of output SQL command file. (Default is %s)" % self.sqlFileName ) - parser.add_option("-t", "--tokentable", - dest="sqlTokenTableName", - default=self.sqlTokenTableName, - help="name of output SQL token table. (Default is %s)" % self.sqlTokenTableName ) - parser.add_option("-m", "--metricstable", - dest="sqlMetricsTableName", - default=self.sqlMetricsTableName, - help="name of output SQL metrics table. (Default is %s)" % self.sqlMetricsTableName ) - parser.add_option("-c", "--csv", - dest="csvFileName", - default=self.csvFileName, - help="name of output CSV data file. (Default is %s)" % self.csvFileName ) - parser.add_option("-f", "--files", - dest="inFileList", - default=self.inFileList, - help="File containing list of path names to modules for analysis." ) - parser.add_option("-i", "--include", - dest="includeMetricsStr", - default=self.includeMetricsStr, - help="list of metrics to include in run. This is a comma separated list of metric module names with no whitespace. Optionally, you can specify the class name of the metric by following the module name with a colon (:) and the metric class name. (Default metrics are 'simple:SimpleMetric,mccabe:McCabeMetric,sloc:SLOCMetric'. Default metric class name for metric module 'wxYz' is 'WxYzMetric' when only module name given -- note capitalized metric class name.)" ) - parser.add_option("-l", "--library", - dest="libName", - default=self.libName, - help="user-defined name applied to collection of modules (Default is '')" ) - parser.add_option("-e", "--exists", - action="store_true", - dest="genExistsSw", - default=self.genExistsSw, - help="assume SQL tables exist and does not generate creation code. Using this option sets option -N. (Default is %s)" % (self.genExistsSw) ) - parser.add_option("-N", "--noold", - action="store_true", - dest="genNewSw", - default=self.genNewSw, - help="create new command output files and tables after deleting old results, if any. Ignored if -e is set. (Default is %s)" % (self.genNewSw) ) - parser.add_option("-B", "--nobasic", - action="store_false", - dest="genBasicSw", - default=self.genBasicSw, - help="suppress production of Basic metrics (Default is %s)" % (not self.genBasicSw) ) - parser.add_option("-S", "--nosql", - action="store_false", - dest="genSqlSw", - default=self.genSqlSw, - help="suppress production of output SQL command text file. (Default is %s)" % (not self.genSqlSw) ) - parser.add_option("-C", "--nocsv", - action="store_false", - dest="genCsvSw", - default=self.genCsvSw, - help="suppress production of CSV output text file. (Default is %s)" % (not self.genCsvSw) ) - parser.add_option("-H", "--noheadings", - action="store_false", - dest="genHdrSw", - default=self.genHdrSw, - help="suppress heading line in csv file. (Default is %s)" % (not self.genHdrSw) ) - parser.add_option("-k", "--kwcnt", - action="store_true", - dest="genKwCntSw", - default=self.genKwCntSw, - help="generate keyword counts. (Default is %s)" % (self.genKwCntSw,) ) - parser.add_option("-K", "--nokwcnt", - action="store_false", - dest="genKwCntSw", - default=self.genKwCntSw, - help="suppress keyword counts. (Default is %s)" % (not self.genKwCntSw) ) - parser.add_option("-q", "--quiet", - action="store_true", - dest="quietSw", - default=self.quietSw, - help="suppress normal summary output to stdout. (Default is %s)" % (self.quietSw) ) - parser.add_option("-z", "--zero", - action="store_true", - dest="zeroSw", - default=self.zeroSw, - help="display zero or empty values in output to stdout. (Default is to suppress zero/empty output)" ) - parser.add_option("-v", "--verbose", - action="count", - dest="verbose", - default=self.verbose, - help="Produce verbose output - more -v's produce more output. (Default is no verbose output to stdout)") - parser.add_option("-d", "--debug", - action="store_true", - dest="debugSw", - default=self.debugSw, - help="Provide debug output, not usually generated - internal use only") - - # parse the command line/arguments for this instance - try: - (options, args) = parser.parse_args() - except BadOptionError, e: - sys.stderr.writelines( "\nBadOptionError: %s\n" % str( e ) ) - sys.stderr.writelines( "\nThe valid options are:\n\n" ) - sys.stderr.writelines( parser.format_help() ) - sys.exit( 1 ) - - # augment parameter values from instantiation with - # command line values. - # the command line parameter values take precidence - # over values in program. - - args.extend( pArgs ) - - # convert command line arguments into instance values - self.__dict__.update( options.__dict__ ) - - if self.inFileList: - try: - inF = open( self.inFileList ) - files = inF.read().split() - inF.close() - args.extend( files ) - except IOError, e: - raise ProcessArgsError( e ) - - self.inFileNames = args - - self.includeMetrics = self.processIncludeMetrics( self.includeMetricsStr ) - - if len( args ) < 1: - print usageStr - print parser.format_help() - e = "No program file names given.\n" - # because of what I believe to be a bug in the doctest module, - # which makes it mishandle exceptions, I have 'faked' the handling - # of raising an exception and just return - if doctestSw: - print e - return - else: - raise ProcessArgsError( e ) - - so = None - if self.genSqlSw and self.sqlFileName: - # Try opening command file for input. If OK, then file exists. - # Else, the commamd file does not exist to create SQL table and - # this portion of the command file must be generated as if new - # table was to be created. - # - # NOTE: This assumption can be dangerous. If the table does exist - # but the command file is out of sync with the table, then a valid - # table can be deleted and recreated, thus losing the old data. - # The workaround for this is to examine the SQL database for the - # table and only create the table, if needed. - try: - if self.sqlFileName: - self.so = open( self.sqlFileName, "r" ) - self.so.close() - except IOError: - self.genNewSw = True - - try: - mode = "a" - if self.genNewSw: - mode = "w" - so = open( self.sqlFileName, mode ) # check to see that we can write file - so.close() - except IOError, e: - sys.stderr.writelines( str(e) + " -- No SQL command file will be generated\n" ) - self.genSqlSw = False - so = None - - co = None - if self.genCsvSw and self.csvFileName: - try: - mode = "a" - if self.genNewSw: - mode = "w" - co = open( self.csvFileName, mode ) # check we will be able to write file - co.close() - except IOError, e: - sys.stderr.writelines( str(e) + " -- No CSV output file will be generated\n" ) - self.genCsvSw = False - co = None - - if self.genExistsSw: - # Assuming that table already exists, means concatenating - # data is not needed - we only want new data in SQL command file. - self.genNewSw = True - - def conflictHandler( self, *args, **kwds ): - print "args=%s" % args - print "kwds=%s" % kwds - - def processIncludeMetrics( self, includeMetricsStr ): - includeMetrics = [] - try: - metricList = includeMetricsStr.split( ',' ) - for a in metricList: - s = a.split( ':' ) - if len( s ) == 2: # both metric class and module name given - includeMetrics.append( s ) - elif len( s ) == 1: - # only the module name given. Generate default metric - # class name by capitalizing first letter of module - # name and appending "Metric" so the default metric - # class name for module wxYz is WxYzMetric. - if s[0]: - defName = s[0][0].upper() + s[0][1:] + 'Metric' - includeMetrics.append( (s[0], defName) ) - else: - raise ProcessArgsError("Missing metric module name") - else: - raise ProcessArgsError("Malformed items in includeMetric string") - except AttributeError, e: - e = ( "Invalid list of metric names: %s" % - includeMetricsStr ) - raise ProcessArgsError( e ) - return includeMetrics - -def testpa( pa ): - """ Test of ProcessArgs. - - Usage: - - >>> pa=ProcessArgs('inFile.py') - >>> testpa(pa) #doctest: +NORMALIZE_WHITESPACE +ELLIPSIS - Arguments processed: - sqlFileName=metricData.sql - sqlTokenTableName=metricTokens - sqlMetricsTableName=metricData - csvFileName=metricData.csv - Include Metric Modules=simple:SimpleMetric,mccabe:McCabeMetric - quietSw=False - genCsvSw=True - genHdrSw=True - genKwCntSw=False - verbose=... - Metrics to be used are: - Module simple contains metric class SimpleMetric - Module mccabe contains metric class McCabeMetric - Input files: - inFile.py - >>> - """ - print """ -Arguments processed: -\tsqlFileName=%s -\tsqlTokenTableName=%s -\tsqlMetricsTableName=%s -\tcsvFileName=%s -\tinclude Metric Modules=%s -\tquietSw=%s -\tgenNewSw=%s -\tgenExistsSw=%s -\tgenSqlSw=%s -\tgenCsvSw=%s -\tgenHdrSw=%s -\tgenKwCntSw=%s -\tverbose=%s""" % ( - pa.sqlFileName, - pa.sqlTokenTableName, - pa.sqlMetricsTableName, - pa.csvFileName, - pa.includeMetricsStr, - pa.quietSw, - pa.genNewSw, - pa.genExistsSw, - pa.genSqlSw, - pa.genCsvSw, - pa.genHdrSw, - pa.genKwCntSw, - pa.verbose) - print "Metrics to be used are:" - for m,n in pa.includeMetrics: - print "\tModule %s contains metric class %s" % (m,n) - if pa.inFileNames: - print "Input files:" - for f in pa.inFileNames: - print "\t%s" % f - -if __name__ == "__main__": - - # ignore doctestsw.doctestSw if this is run as a standalone - # module. Instead, look at the first argument on the command - # line. If it is "doctest", set doctestSw = True and delete - # the parameter "doctest" from sys.argv, thus leaving things - # as _normal_. - if len(sys.argv) > 1 and sys.argv[1] == 'doctest': - doctestSw = True - sys.argv[1:] = sys.argv[2:] - import doctest - doctest.testmod( sys.modules[__name__] ) - sys.exit( 0 ) - - print "=== ProcessArgs: %s ===" % ("'file1.py', 'file2.py', '/home/files/file3.py'",) - try: - pa = ProcessArgs( 'file1.py', 'file2.py', '/home/files/file3.py' ) - testpa( pa ) - except ProcessArgsError, e: - sys.stderr.writelines( str( e ) ) - print - print "=== ProcessArgs: %s ===" % ("inFile1.py, sqlFileName='sqlF1.sql', genCsvSw=False,genKwCntSw=True",) - try: - pa = ProcessArgs( "inFile1.py", sqlFileName='sqlF1.sql', genCsvSw=False,genKwCntSw=True ) - testpa( pa ) - except ProcessArgsError, e: - sys.stderr.writelines( str( e ) ) - print - print "=== ProcessArgs: %s ===" % ("No arguments",) - try: - pa = ProcessArgs() - testpa( pa ) - except ProcessArgsError, e: - sys.stderr.writelines( str( e ) ) - print - sys.exit( 0 ) - diff -uNr a/pymetrics-0.7.9/pymetrics b/pymetrics-0.7.9/pymetrics --- a/pymetrics-0.7.9/pymetrics 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/pymetrics 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,7 @@ +#!/usr/bin/env python + +import PyMetrics.PyMetrics as pm +import sys + +pm.main () +sys.exit (0) diff -uNr a/pymetrics-0.7.9/PyMetrics/compute.py b/pymetrics-0.7.9/PyMetrics/compute.py --- a/pymetrics-0.7.9/PyMetrics/compute.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/compute.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,595 @@ +""" Main computational modules for PyMetrics. + + $Id: compute.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ +""" + +__version__ = "$Revision: 1.2 $"[11:-2] +__author__ = 'Reg. Charney ' + +import mytoken +from globals import * +from utils import * +MAXDISPLAY = 32 + +class ComputeMetrics( object ): + """ Class used to compute basic metrics for given module.""" + def __init__( self, metricInstance, context, runMetrics, metrics, pa, so, co ): + """ Initialize general computational object.""" + self.metricInstance = metricInstance + self.context = context + self.runMetrics = runMetrics + self.metrics = metrics + self.pa = pa + self.so = so + self.co = co + + self.fqnFunction = [] + self.fqnClass = [] + self.fcnExits = [] + + self.token = None + self.stmt = [] + self.block = [] + self.fcn = [] + self.cls = [] + self.mod = [] + self.run = [] + + self.processSrcLineSubscribers = [] + self.processTokenSubscribers = [] + self.processStmtSubscribers = [] + self.processBlockSubscribers = [] + self.processFunctionSubscribers = [] + self.processClassSubscribers = [] + self.processModuleSubscribers = [] + self.processRunSubscribers = [] + + self.__initMetrics( metricInstance ) + + # this is same as function in TokenHandler !!!!!!!! + def __extractFQN( self, fqn, default='__main__' ): + """ Extract fully qualified name from list.""" + result = default + if len( fqn ): + result = fqn[-1][0] + return result + + # this is same as function in TokenHandler !!!!!!!! + def __incr( self, name ): + "Increment count in metrics dictionary based on name as key." + self.metrics[name] = self.metrics.get(name,0) + 1 + + def processSrcLines( self, srcLine ): + """ Handle processing of each physical source line. + + The fcnName and className are meaningless until the + tokens are evaluated. + """ + fcnName = self.__extractFQN( self.fqnFunction ) + className = self.__extractFQN( self.fqnClass, None ) + + for subscriber in self.processSrcLineSubscribers: + subscriber.processSrcLines( srcLine ) + + def processToken( self, tok ): + """ Handle processing after each token processed.""" + self.token = tok + self.stmt.append( tok ) # we are always in a statememt + self.block.append( tok ) # we are always in a block + if self.fqnFunction: # we are inside some function + self.fcn.append( tok ) + if self.fqnClass: # we are inside some class + self.cls.append( tok ) + self.mod.append( tok ) # we are always in some module + self.run.append( tok ) # we are always in some run + + fcnName = self.__extractFQN( self.fqnFunction ) + className = self.__extractFQN( self.fqnClass, None ) + for subscriber in self.processTokenSubscribers: + subscriber.processToken( fcnName, className, self.token ) + + def processStmt( self ): + """ Handle processing at end of statement.""" + fcnName = self.__extractFQN( self.fqnFunction ) + className = self.__extractFQN( self.fqnClass, None ) + for subscriber in self.processStmtSubscribers: + subscriber.processStmt( fcnName, className, self.stmt ) + + self.stmt[:] = [] # clear out statement list + + def processBlock( self ): + """ Handle processing at end of block.""" + fcnName = self.__extractFQN( self.fqnFunction ) + className = self.__extractFQN( self.fqnClass, None ) + for subscriber in self.processBlockSubscribers: + subscriber.processBlock( fcnName, className, self.block ) + + self.block[:] = [] # clear out block list + + def processFunction( self ): + """ Handle processing at end of function. """ + msg = self.__checkNumberOfExits() + fcnName = self.__extractFQN( self.fqnFunction ) + className = self.__extractFQN( self.fqnClass, None ) + for subscriber in self.processFunctionSubscribers: + subscriber.processFunction( fcnName, className, self.fcn ) + + self.fcn[:] = [] # clear out function list + + return msg + + def __checkNumberOfExits( self ): + """" Generate warning message if more than one exit. """ + msg = None + n = len( self.fcnExits ) + if n > 0: + exits = ', '.join( [str(i) for i in self.fcnExits] ) + plural = ((n > 1) and 's') or '' + exitStr = "exit%s at line%s" % (plural,plural) + msg = ("In file %s, function %s has %d extra %s %s" % + (self.context['inFile'], + self.__extractFQN( self.fqnFunction ), + n, + exitStr, + exits)) + for i in range( len( self.fcnExits ) ): + del self.fcnExits[0] + self.__incr( 'numMultipleExitFcns') + + return msg + + def processClass( self ): + """ Handle processing at end of class. """ + fcnName = self.__extractFQN( self.fqnFunction ) + className = self.__extractFQN( self.fqnClass, None ) + for subscriber in self.processClassSubscribers: + subscriber.processClass( fcnName, className, self.cls ) + + self.cls[:] = [] + + def processModule( self ): + """ Handle processing at end of class. """ + moduleName = self.context['inFile'] + mod = self + for subscriber in self.processModuleSubscribers: + subscriber.processModule( moduleName, mod ) + + self.mod[:] = [] + + def processRun( self ): + """ Handle processing at end of class. """ + for subsriber in self.processRunSubscribers: + subsriber.processRun( self.run ) + + self.run[:] = [] + + def __call__( self, lex ): + """ This function is the start of the heavy lifting + for computing the various metrics produced by PyMetrics.""" + + for m in self.metricInstance.keys(): + self.metricInstance[m].processSrcLines( lex.srcLines ) + + # Loop through list of tokens and count types as needed. + + # skipUntil is set after an error is detected. It is an attempt to + # find a point to restart the analysis so that we do not get + # cascading errors. This problem often occurs in analysing + # foreign character sets when normal ascii was expected. + tokenList = lex.tokenlist + + skipUntil = None + tokCount = 0 + invalidToken = mytoken.MyToken() + + for tok in tokenList: + if skipUntil: + if tok.text in skipUntil: # found where to restart analysis + skipUntil = None + elif tok.type == WS: # count, but otherwise ignore, whitespace + tokCount = tokCount+1 + self.metrics['numTokens'] = tokCount + self.__postToken( tok ) + continue + elif tok.type == ERRORTOKEN: + invalidToken.text += tok.text + continue + + tokCount = self.handleToken( tokCount, tok ) + + return self.metrics + + def __initMetrics( self, metricInstance ): + """ Initialize all the local variables that will be + needed for analysing tokens.:""" + metricList = [] + for m in metricInstance.keys(): + if metricInstance[m]: # only append valid instances + metricList.append( metricInstance[m] ) + # clear out any old data while leaving reference to same + # thing (ie., pointers to these lists are always valid + del self.processSrcLineSubscribers[:] + del self.processTokenSubscribers[:] + del self.processStmtSubscribers[:] + del self.processBlockSubscribers[:] + del self.processFunctionSubscribers[:] + del self.processClassSubscribers[:] + del self.processModuleSubscribers[:] + del self.processRunSubscribers[:] + # since all metric classes are derived from MetricBase, + # we can assign all the processX functions to all the + # metrics + self.processSrcLineSubscribers.extend( metricList ) + self.processTokenSubscribers.extend( metricList ) + self.processStmtSubscribers.extend( metricList ) + self.processBlockSubscribers.extend( metricList ) + self.processFunctionSubscribers.extend( metricList ) + self.processClassSubscribers.extend( metricList ) + self.processModuleSubscribers.extend( metricList ) + self.processRunSubscribers.extend( metricList ) + + self.numSrcLines = 0 + self.blockDepth = 0 + self.numBlocks = 0 + self.parenDepth = 0 + self.bracketDepth = 0 + self.braceDepth = 0 + self.numNestedClasses = 0 + self.numKeywords = 0 + self.numComments = 0 + self.numEmpty = 0 + self.classDepth = 0 + self.fcnDepth = 0 + self.classDepthIncr = 0 + self.fcnDepthIncr = 0 + self.maxBlockDepth = -1 + self.maxClassDepth = -1 + self.fqnName = [] + self.defFunction = False + self.defClass = False + self.docString = True + self.findFcnHdrEnd = False + self.findClassHdrEnd = False + self.inClass = False + self.inFunction = False + self.metrics['numSrcLines'] = 0 + self.metrics['numTokens'] = 0 + self.metrics['numComments'] = 0 + self.metrics['numCommentsInline'] = 0 + self.metrics['numModuleDocStrings'] = 0 + self.metrics['numBlocks'] = 0 + self.metrics['numFunctions'] = 0 + self.metrics['numClasses'] = 0 + self.className = None + self.fcnName = None + self.saveTok = None + self.skipUntil = None # used to skip invalid chars until valid char found + self.invalidToken = None + self.checkForModuleDocString = True + + return metricList + + def __fitIn( self, tok ): + """ Truncate long tokens to MAXDISPLAY length. + Also, newlines are replace with '\\n' so text fits on a line.""" + #tmpText = tok.text[:].replace( '\n', '\\n' ) + tmpText = tok.text[:] + if len( tmpText ) > MAXDISPLAY: + tmpText = tmpText[:10].strip() + ' ... ' + \ + tmpText[-10:].strip() + return tmpText + + def __incrEach( self, tok ): + """ Increment count for each unique semantic type.""" + pfx = self.__genFQN( self.fqnName ) + key = self.__fitIn( tok ) + sep = tok.semtype == KEYWORD and ' ' or '.' + if tok.semtype in [FCNNAME,CLASSNAME]: + key = pfx + else: + if pfx: + key = pfx + sep + key + else: + key = '__main__' + sep + key + key = "%-10s %s" % (token.tok_name[tok.semtype],key) + + self.metrics[key] = self.metrics.get(key,0) + 1 + + def __postToken( self, tok ): + """ Post token processing for common tasks.""" + self.__incr( tok.type ) + if tok.semtype: # then some semantic value here + self.__incr( tok.semtype ) # count semantic type + self.__incrEach( tok ) + self.processToken( tok ) + if self.pa.verbose > 1: + print self.context, tok + self.so and self.so.write( self.context, tok, self.fqnFunction, self.fqnClass ) + self.co and self.co.write( self.context, tok, self.fqnFunction, self.fqnClass ) + + def __genFQN( self, fqnName ): + """ Generate a fully qualified name. """ + result = '.'.join( fqnName ) + return result + + def handleToken( self, tokCount, tok ): + """ Common code for handling tokens.""" + if tokCount == 0: # this is the first token of the module + self.prevTok = None + tokCount += 1 + self.metrics['numTokens'] = tokCount + + # don't treat whitespace as significant when looking at previous token + if not tok.type in [WS,INDENT,DEDENT,EMPTY,ENDMARKER]: + self.prevTok = self.saveTok + self.saveTok = tok + + # set up context for current token + self.context['blockNum'] = self.metrics.get('numBlocks',0) + self.context['blockDepth'] = self.metrics.get('blockDepth',0) + self.context['parenDepth'] = self.parenDepth + self.context['bracketDepth'] = self.bracketDepth + self.context['braceDepth'] = self.braceDepth + + # self.classDepthIncr is 1 if the class definition header + # has a newline after colon, else it is equal to zero + # meaning the class definition block is on same line as its header + self.classDepth += self.classDepthIncr + self.context['classDepth'] = self.classDepth + self.classDepthIncr = 0 + + # self.classFcnIncr is 1 if the function definition header + # has a newline after colon, else it is equal to zero + # meaning the function definition block is on same line + # as its header + self.fcnDepth += self.fcnDepthIncr # only incr at start of fcn body + self.context['fcnDepth'] = self.fcnDepth + self.fcnDepthIncr = 0 + + # start testing for types that change in context + + if self.doDocString(tok): return tokCount + if self.doInlineComment(tok, self.prevTok): return tokCount + if self.doHeaders(tok): return tokCount + + # return with types that don't change in context + + self.__postToken( tok ) + if tok.type == WS: + return tokCount + + # treat end of file as end of statement + if tok.type == EMPTY or tok.type == NEWLINE: + self.processStmt() + return tokCount + + # End of file forces closure of everything, but run + if tok.type == ENDMARKER: + self.processStmt() + self.processBlock() + self.processFunction() + self.processClass() + self.processModule() + return tokCount + + # at this point, we have encountered a non-white space token + # if a module doc string has not been found yet, + # it never will be. + numModDocStrings = self.metrics['numModuleDocStrings'] + if self.checkForModuleDocString and numModDocStrings == 0: + self.checkForModuleDocString = False + msg = (("Module %s is missing a module doc string. "+ + "Detected at line %d\n") % + (self.context['inFile'],tok.row) + ) + if msg and not self.pa.quietSw: + print msg + + if self.doOperators(tok): return tokCount + if self.doIndentDedent(tok): return tokCount + + self.docString = False + self.doKeywords(tok, self.prevTok) + + return tokCount + + def doKeywords(self, tok, prevTok): + """ Count keywords and check if keyword 'return' used more than + once in a given function/method.""" + if tok.semtype == KEYWORD: + self.__incr( 'numKeywords') + if tok.text == 'def': + self.defFunction = True + elif tok.text == 'class': + self.defClass = True + elif tok.text == 'return': + assert self.fcnDepth == len( self.fqnFunction ) + if self.fcnDepth == 0: # not in any function + if not self.pa.quietSw: # report on these types of errors + print (("Module %s contains the return statement at "+ + "line %d that is outside any function") % + (self.context['inFile'],tok.row) + ) + if prevTok.text == ':': + # this return on same line as conditional, + # so it must be an extra return + self.fcnExits.append( tok.row ) + elif self.blockDepth > 1: + # Let fcnBlockDepth be the block depth of the function body. + # We are trying to count the number of return statements + # in this function. Only one is allowed at the fcnBlockDepth + # for the function. If the self.blockDepth is greater than + # fcnBlockDepth, then this is a conditional return - i.e., + # an additional return + fcnBlockDepth = self.fqnFunction[-1][1] + 1 + if self.blockDepth > fcnBlockDepth: + self.fcnExits.append( tok.row ) + + def doIndentDedent(self, tok): + """ Handle indents and dedents by keeping track of block depth. + Also, handle cases where dedents close off blocks, functions, + and classes.""" + result = False + if tok.type == INDENT: + result = self.__doIndent() + elif tok.type == DEDENT: + result = self.__doDedent() + return result + + def __doDedent(self): + """ Handle dedents and remove function/class info as needed.""" + self._doDedentFcn() + + self.fcnName = None + if len( self.fqnFunction ) > 0: + self.fcnName = self.fqnFunction[-1][0] + + while len(self.fqnClass) and self.blockDepth <= self.fqnClass[-1][1]: + self.processClass() + self.classDepth -= 1 + if self.pa.verbose > 0: + print "Removing class %s" % self.__extractFQN( self.fqnClass, None ) + del self.fqnClass[-1] + del self.fqnName[-1] + + self.className = None + if len( self.fqnClass ) > 0: + self.className = self.fqnClass[-1][0] + + return True + + def _doDedentFcn(self): + """ Remove function whose scope is ended.""" + assert self.fcnDepth == len( self.fqnFunction ) + self.blockDepth -= 1 + self.metrics['blockDepth'] = self.blockDepth + while len(self.fqnFunction) and self.blockDepth<=self.fqnFunction[-1][1]: + self.__doDedentRemoveMsg() + del self.fqnFunction[-1] + del self.fqnName[-1] + self.fcnDepth -= 1 + assert self.fcnDepth == len( self.fqnFunction ) + + def __doDedentRemoveMsg(self): + """ Output message if debugging or user asks for info.""" + msg = self.processFunction() + if msg and not self.pa.quietSw: + print msg + if self.pa.verbose > 0: + print "Removing function %s" % self.__extractFQN( self.fqnFunction ) + + def __doIndent(self): + """ Increment indent count and record if maximum depth.""" + self.__incr( 'numBlocks' ) + self.blockDepth += 1 + self.metrics['blockDepth'] = self.blockDepth + if self.metrics.get('maxBlockDepth',0) < self.blockDepth: + self.metrics['maxBlockDepth'] = self.blockDepth + return True + + def doOperators(self, tok): + """ Keep track of the number of each operator. Also, handle the + case of the colon (:) terminating a class or def header.""" + result = False + if tok.type == OP: + if tok.text == ':': + if self.findFcnHdrEnd: + self.findFcnHdrEnd = False + self.docString = True + self.fcnDepthIncr = 1 + elif self.findClassHdrEnd: + self.findClassHdrEnd = False + self.docString = True + self.classDepthIncr = 1 + result = True + elif tok.text == '(': + self.parenDepth += 1 + elif tok.text == ')': + self.parenDepth -= 1 + elif tok.text == '[': + self.bracketDepth += 1 + elif tok.text == ']': + self.bracketDepth -= 1 + elif tok.text == '{': + self.braceDepth += 1 + elif tok.text == '}': + self.braceDepth -= 1 + result = True + return result + + def doHeaders(self, tok): + """ Process both class and function headers. + Create the fully qualified names and deal + with possibly erroneous headers.""" + result = False + if self.defFunction: + if tok.type == NAME: + self.__incr( 'numFunctions') + self.fqnName.append( tok.text ) + self.fcnName = self.__genFQN( self.fqnName ) + self.fqnFunction.append( (self.fcnName,self.blockDepth,FCNNAME) ) + self.defFunction = False + self.findFcnHdrEnd = True + if self.pa.verbose > 0: + print ("fqnFunction=%s" % + toTypeName( self.context, self.fqnFunction ) + ) + self.__postToken( tok ) + result = True + elif tok.type == ERRORTOKEN: + # we must compensate for the simple scanner mishandling errors + self.findFcnHdrEnd = True + self.invalidToken = mytoken.MyToken(type=NAME, + semtype=FCNNAME, + text=tok.text, + row=tok.row, + col=tok.col, + line=tok.line) + self.skipUntil = '(:\n' + result = True + elif self.defClass and tok.type == NAME: + self.__incr( 'numClasses' ) + self.fqnName.append( tok.text ) + self.className = self.__genFQN( self.fqnName ) + self.fqnClass.append( (self.className,self.blockDepth,CLASSNAME) ) + self.defClass = False + self.findClassHdrEnd = True + if self.pa.verbose > 0: + print "fqnClass=%s" % toTypeName( self.context, self.fqnClass ) + self.__postToken( tok ) + result = True + return result + + def doInlineComment(self, tok, prevTok): + """ Check for comments and distingish inline comments from + normal comments.""" + result = False + if tok.type == COMMENT: + self.metrics['numComments'] += 1 + # compensate for older tokenize including newline + # symbols in token when only thing on line is comment + # this patch makes all comments consistent + if tok.text[-1] == '\n': + tok.text = tok.text[:-1] + + if prevTok and prevTok.type != NEWLINE and prevTok.type != EMPTY: + tok.semtype = INLINE + self.metrics['numCommentsInline'] += 1 + self.__postToken( tok ) + result = True + + return result + + def doDocString(self, tok): + """ Determine if a given string is also a docstring. + Also, detect if docstring is also the module's docsting.""" + result = False + if self.docString and tok.type == token.STRING: + tok.semtype = DOCSTRING + if self.checkForModuleDocString: # found the module's doc string + self.metrics['numModuleDocStrings'] += 1 + self.checkForModuleDocString = False + self.__postToken( tok ) + self.docString = False + result = True + return result diff -uNr a/pymetrics-0.7.9/PyMetrics/csvout.py b/pymetrics-0.7.9/PyMetrics/csvout.py --- a/pymetrics-0.7.9/PyMetrics/csvout.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/csvout.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,107 @@ +""" CvsOut - class to produce CVS data output. + + $Id: csvout.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.3 $"[11:-2] +__author__ = 'Reg. Charney ' + +import sys +import time +import token +import tokenize +from utils import * + +class CsvOut( object ): + """ Class used to generate a CSV data file suitable for input to spreadsheet program.""" + def __init__( self, fileName, genHdrSw=True, genNewSw=False ): + """ Open output file and generate header line, if desired.""" + self.fileName = fileName + self.quotedFileName = '"'+self.fileName+'"' + self.IDDateTime = '"'+time.strftime("%Y-%m-%d %H:%M:%S",time.localtime())+'"' + self.toknum = 0 + + mode = "a" + if genNewSw: + mode = "w" + try: + if self.fileName: + self.fd = open( fileName, mode ) + else: + self.fd = sys.stdout + except IOError: + raise + + self.writeHdr( genNewSw, genHdrSw ) + + def writeHdr( self, genNewSw, genHdrSw ): + """ Write header information for CSV file.""" + if genNewSw and genHdrSw: + fldNames = [ + '"IDDateTime"', + '"tokNum"', + '"inFile"', + '"line"', + '"col"', + '"tokType"', + '"semType"', + '"tokLen"', + '"token"', + '"fqnFunction"', + '"fqnClass"', + '"blockNum"', + '"blockDepth"', + '"fcnDepth"', + '"classDepth"', + '"parenDepth"', + '"bracketDepth"', + '"braceDepth"' + ] + self.fd.write( ','.join( fldNames ) ) + self.fd.write( '\n' ) + + def close( self ): + """ Close output file if it is not stdout. """ + if self.fileName: + self.fd.flush() + self.fd.close() + + def write( self, context, tok, fqnFunction, fqnClass ): + """ Generate the CSV data line.""" + self.toknum += 1 + txt = tok.text + tt = tok.type + tn = token.tok_name[tt] + + sn = '' + if tok.semtype: + sn = token.tok_name[tok.semtype] + if tt == token.NEWLINE or tt == tokenize.NL: + txt = r'\n' + + sArgs = ','.join( ( + self.IDDateTime, + str( self.toknum ), + '"'+str( context['inFile'] )+'"', + str( tok.row ), + str( tok.col ), + '"'+tn+'"', + '"'+sn+'"', + str( len( txt ) ), + csvQ( txt ), + '"'+str( toTypeName( context, fqnFunction ) )+'"', + '"'+str( toTypeName( context, fqnClass ) )+'"', + str( context['blockNum'] ), + str( context['blockDepth'] ), + str( context['fcnDepth'] ), + str( context['classDepth'] ), + str( context['parenDepth'] ), + str( context['bracketDepth'] ), + str( context['braceDepth'] ) + ) ) + self.fd.write( sArgs ) + self.fd.write( '\n' ) + + def close( self ): + """ Close file, if it is opened.""" + self.fd and self.fd.close() + self.fd = None diff -uNr a/pymetrics-0.7.9/PyMetrics/doctestsw.py b/pymetrics-0.7.9/PyMetrics/doctestsw.py --- a/pymetrics-0.7.9/PyMetrics/doctestsw.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/doctestsw.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,26 @@ +""" doctestsw - used to globally set/unset doctest testing. + +This switch is needed because the standard doctest can not handle +raise statements with arguments. Basically, if gDoctest is True, +no arguments are passed when an exception is raised. Normally, +gDoctest is False and we do raise exceptions with arguments. + +You must modify the doctestSw value to True/False to +activate/deactivate use of the doctest module tests. + +To used this module, use the 'from' form of the import +statement and write: + +from doctestsw import * +... +if __name__ == "__main__": + if doctestSw: + import doctest + doctest.testmod( sys.modules[__name__] ) + +$Id: doctestsw.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.2 $"[11:-2] +__author__ = 'Reg. Charney ' + +doctestSw = True diff -uNr a/pymetrics-0.7.9/PyMetrics/globals.py b/pymetrics-0.7.9/PyMetrics/globals.py --- a/pymetrics-0.7.9/PyMetrics/globals.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/globals.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,47 @@ +""" Global variables for PyMetrics. + + $Id: globals.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.3 $"[11:-2] +__author__ = 'Reg. Charney ' + +import token +import tokenize + +# our token types +KEYWORD = token.NT_OFFSET + 1 +TEXT = token.NT_OFFSET + 2 +WS = token.NT_OFFSET + 3 +DOCSTRING = token.NT_OFFSET + 4 +VARNAME = token.NT_OFFSET + 5 +CLASSNAME = token.NT_OFFSET + 6 +FCNNAME = token.NT_OFFSET + 7 +INLINE = token.NT_OFFSET + 8 +UNKNOWN = token.NT_OFFSET + 9 +SEMTOKEN = token.NT_OFFSET + 10 # to distinguish semantic tokens +NONTOKEN = token.NT_OFFSET + 11 # to distinguish non-tokens +DECORATOR = token.NT_OFFSET + 12 # to indicate decorator token +NUMBER = token.NUMBER +OP = token.OP +STRING = token.STRING +COMMENT = tokenize.COMMENT +NAME = token.NAME +ERRORTOKEN = token.ERRORTOKEN +ENDMARKER = token.ENDMARKER +INDENT = token.INDENT +DEDENT = token.DEDENT +NEWLINE = token.NEWLINE +EMPTY = tokenize.NL + +# new token types added to allow for character representation of new codes + +token.tok_name[KEYWORD] = "KEYWORD" # one of Python's reserved words +token.tok_name[TEXT] = "TEXT" # obsolete - but kept for compatibility +token.tok_name[WS] = "WS" # some form of whitespace +token.tok_name[DOCSTRING] = "DOCSTRING" # literal that is also doc string +token.tok_name[VARNAME] = "VARNAME" # name that is not keyword +token.tok_name[CLASSNAME] = "CLASSNAME" # name defined in class statment +token.tok_name[FCNNAME] = "FCNNAME" # name defined in def statement +token.tok_name[INLINE] = "INLINE" # comment that follows other text on same line +token.tok_name[UNKNOWN] = "UNKNOWN" # Unknown semantic type - this should not occur +token.tok_name[DECORATOR] = 'DECORATOR' # Decorator marker diff -uNr a/pymetrics-0.7.9/PyMetrics/halstead.py b/pymetrics-0.7.9/PyMetrics/halstead.py --- a/pymetrics-0.7.9/PyMetrics/halstead.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/halstead.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,266 @@ +""" Compute HalsteadMetric Metrics. + +HalsteadMetric metrics, created by Maurice H. HalsteadMetric in 1977, consist +of a number of measures, including: + +Program length (N): N = N1 + N2 +Program vocabulary (n): n = n1 + n2 +Volume (V): V = N * LOG2(n) +Difficulty (D): D = (n1/2) * (N2/n2) +Effort (E): E = D * V +Average Volume (avgV) avgV = sum(V)/m +Average Effort (avgE) avgE = sum(E)/m + +where: + +n1 = number of distinct operands +n2 = number of distinct operators +N1 = total number of operands +N2 = total number of operators +m = number of modules + +What constitues an operand or operator is often open to +interpretation. In this implementation for the Python language: + + operators are of type OP, INDENT, DEDENT, or NEWLINE since these + serve the same purpose as braces and semicolon in C/C++, etc. + operands are not operators or whitespace or comments + (this means operands include keywords) + + $Id: halstead.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.3 $"[11:-2] +__author__ = 'Reg. Charney ' + +import math +import time +from metricbase import MetricBase +from globals import * + +class HalsteadMetric( MetricBase ): + """ Compute various HalsteadMetric metrics. """ + totalV = 0 + totalE = 0 + numModules = 0 + def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): + """ Initialization for the HalsteadMetric metrics.""" + self.inFile = context['inFile'] + self.context = context + self.runMetrics = runMetrics + self.metrics = metrics + self.pa = pa + self.inFile = context['inFile'] + self.numOperators = 0 + self.numOperands = 0 + self.uniqueOperators = {} + self.uniqueOperands = {} + HalsteadMetric.numModules += 1 + + # initialize category accummulators as dictionaries + self.hsDict = {} + for t in ['token','stmt','block','function','class','module','run']: + self.uniqueOperators[t] = {} + self.uniqueOperands[t] = {} + #for v in ['N','N1','N2','n','n1','n2','V','D','E','avgV','avgE']: + # self.hsDict[(t,v)] = 0 + + def processToken( self, currentFcn, currentClass, tok, *args, **kwds ): + """ Collect token data for Halstead metrics.""" + if tok.type in [WS, EMPTY, ENDMARKER, NEWLINE, EMPTY, COMMENT]: + pass + elif tok.type in [OP, INDENT, DEDENT]: + self.numOperators += 1 + self.uniqueOperators['token'][tok.text] = self.uniqueOperators['token'].get(tok.text, 0) + 1 + else: + self.numOperands += 1 + sDict = self.context.__repr__() + k = (sDict,tok.text) + self.uniqueOperands['token'][k] = self.uniqueOperands['token'].get(tok.text, 0) + 1 + + def processStmt( self, currentFcn, currentClass, stmt, *args, **kwds ): + """ Collect statement data for Halstead metrics.""" + + result = None + + # the two lines following this comment would compute the Halstead + # metrics for each statement in the run, However, it is + # normally overkill, so these lines are commented out. + + #lineNum = stmt[0].row + #result = self.computeCategory( 'stmt', lineNum, stmt ) + + return result + + def processBlock( self, currentFcn, currentClass, block, *args, **kwds ): + """ Collect block data for Halstead metrics.""" + + result = None + + # the two lines following this comment would compute the Halstead + # metrics for each statement in the run, However, it is + # normally overkill, so the two lines are commented out. + + #blockNum = self.context['blockNum'] + #result = self.computeCategory( 'block', blockNum, block ) + + return result + + def processFunction( self, currentFcn, currentClass, fcn, *args, **kwds ): + """ Collect function data for Halstead metrics.""" + result = self.computeCategory( 'function', currentFcn, fcn ) + return result + + def processClass( self, currentFcn, currentClass, cls, *args, **kwds ): + """ Collect class data for Halstead metrics.""" + result = self.computeCategory( 'class', currentClass, cls ) + return result + + def processModule( self, moduleName, mod, *args, **kwds ): + """ Collect module data for Halstead metrics.""" + result = self.computeCategory( 'module', moduleName, mod ) + return result + + def processRun( self, run, *args, **kwds ): + """ Collect run data for Halstead metrics.""" + datestamp = time.strftime("%Y-%m-%d.%H:%m%Z",time.localtime()) + result = self.computeCategory( 'run', datestamp, run ) + return result + + def __LOGb( self, x, b ): + """ convert to LOGb(x) from natural logs.""" + try: + result = math.log( x ) / math.log ( b ) + except OverflowError: + result = 1.0 + return result + + def computeIncr( self, cat, tok, uniqueOperators, uniqueOperands ): + """ Compute increment for token depending on which category it falls into.""" + operatorIncr = operandIncr = 0 + if tok.type in [WS, EMPTY, ENDMARKER, NEWLINE, EMPTY, COMMENT]: + return (operatorIncr,operandIncr) + + if tok.type in [OP, INDENT, DEDENT]: + operatorIncr = 1 + uniqueOperators[tok.text] = uniqueOperators.get(tok.text, 0) + 1 + else: + operandIncr = 1 + uniqueOperands[tok.text] = uniqueOperands.get(tok.text,0) + 1 + + return (operatorIncr,operandIncr) + + def computeCategory( self, cat, mod, lst ): + """ Collection data for cat of code.""" + modID= id( mod ) + numOperators = numOperands = 0 + for tok in lst: + result = self.computeIncr( cat, tok, self.uniqueOperators[cat], self.uniqueOperands[cat] ) + numOperators += result[0] + numOperands += result[1] + result = self.compute( cat, modID, numOperators, numOperands, self.uniqueOperators[cat], self.uniqueOperands[cat] ) + return result + + def compute( self, cat, modID, numOperators, numOperands, uniqueOperators, uniqueOperands, *args, **kwds ): + """ Do actual calculations here.""" + + n1 = len( uniqueOperands ) + n2 = len( uniqueOperators ) + N1 = numOperands + N2 = numOperators + N = N1 + N2 + n = n1 + n2 + V = float(N) * self.__LOGb( n, 2 ) + try: + D = (float(n1)/2.0) * (float(N2)/float(n2)) + except ZeroDivisionError: + D = 0.0 + E = D * V + HalsteadMetric.totalV += V + HalsteadMetric.totalE += E + avgV = HalsteadMetric.totalV / HalsteadMetric.numModules + avgE = HalsteadMetric.totalE / HalsteadMetric.numModules + + self.hsDict[(cat,modID,'n1')] = n1 + self.hsDict[(cat,modID,'n2')] = n2 + self.hsDict[(cat,modID,'N1')] = N1 + self.hsDict[(cat,modID,'N2')] = N2 + self.hsDict[(cat,modID,'N')] = N + self.hsDict[(cat,modID,'n')] = n + self.hsDict[(cat,modID,'V')] = V + self.hsDict[(cat,modID,'D')] = D + self.hsDict[(cat,modID,'E')] = E + self.hsDict[(cat,modID,'numModules')] = HalsteadMetric.numModules + self.hsDict[(cat,modID,'avgV')] = avgV + self.hsDict[(cat,modID,'avgE')] = avgE + + return self.hsDict + + def display( self, cat=None ): + """ Display the computed Halstead Metrics.""" + if self.pa.quietSw: + return self.hsDict + + hdr = "\nHalstead Metrics for %s" % self.inFile + print hdr + print "-"*len(hdr) + '\n' + + if len( self.hsDict ) == 0: + print "%-8s %-30s " % ('**N/A**','All Halstead metrics are zero') + return self.hsDict + + keyList = self.hsDict.keys() + keyList.sort() + if 0: + for k,i,v in keyList: + if cat: + if k!=cat: + continue + print "%14.2f %s %s %s" % (self.hsDict[(k,i,v)],k,i,v) + print + hdr1 = "Category Identifier D E N N1 N2 V avgE avgV n n1 n2" + hdr2 = "-------- ---------------------------------- -------- -------- ----- ---- ---- -------- -------- -------- ----- ---- ----" + # 12345678 123456789012345678901234567890 12345678 12345678 12345 1234 1234 12345678 12345678 12345678 12345 1234 1234 + fmt1 = "%-8s %-33s " + fmt2 = "%8.2e %8.2e %5d %4d %4d %8.2e %8.2e %8.2e %5d %4d %4d" + + # this loop uses the Main Line Standards break logic. It does this to convert the + # normal vertical output to a horizontal format. The control variables are the + # category name and the identifier value. + + oldK = oldI = None + vDict = {} + vList = [] + hdrSw = True # output header for first time thru + for k,i,v in keyList: + # only print data for the category we want + if cat: + if k != cat: + continue + + if v == "numModules": # ignore this value for now + continue + + if (oldK,oldI) != (k,i): # change in category/id + if oldK and oldI: # this is not first time thru + #t = tuple([self.hsDict[(k,i,v)] for v in vList]) + t = tuple([vDict[v] for v in vList]) + print fmt1 % (k,i), + print fmt2 % t + # initialize for next set of category/id + vDict = {} + vDict[v] = self.hsDict[(k,i,v)] + vList = [] + vList.append( v ) + oldK = k + oldI = i + if hdrSw: + print hdr1 + print hdr2 + hdrSw = False + else: # we are still in the same category/id + vDict[v] = self.hsDict[(k,i,v)] + vList.append( v ) + + print + + return self.hsDict diff -uNr a/pymetrics-0.7.9/PyMetrics/__init__.py b/pymetrics-0.7.9/PyMetrics/__init__.py --- a/pymetrics-0.7.9/PyMetrics/__init__.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/__init__.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,21 @@ +""" Used to indicate that this is a Python module directory + + PyMetrics - produce a variety of metrics for Python programs + + Copyright (c) 2005 by Reg. Charney + All rights reserved, see LICENSE for details. + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + $Id: __init__.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.2 $"[11:-2] +__author__ = 'Reg. Charney ' diff -uNr a/pymetrics-0.7.9/PyMetrics/lexer.py b/pymetrics-0.7.9/PyMetrics/lexer.py --- a/pymetrics-0.7.9/PyMetrics/lexer.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/lexer.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,139 @@ +""" Parsing classes. + + $Id: lexer.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.2 $"[11:-2] +__author__ = 'Reg. Charney ' + +import sys +import string +import cStringIO +import mytoken +import keyword +from globals import * + +class ParseError(Exception): + pass + +class Lexer: + """ Parse python source.""" + def __init__( self ): + self.prevToktype = None + self.prevSemtype = None + self.prevToktext = None + + def parse( self, inFileName ): + """ Read and parse the source. """ + fd = open( inFileName ) + try: + srcLines = fd.read() + self.srcLines = string.expandtabs( srcLines ) + finally: + fd.close() + + self.tokenlist = [] + + self.__computeOffsets() + self.__parseSource() + + def __parseSource(self): + """ Parse the source in file.""" + self.pos = 0 + text = cStringIO.StringIO( self.srcLines ) + try: + tokenize.tokenize( text.readline, self ) + except tokenize.TokenError, ex: + msg = ex[0] + line = ex[1][0] + print line, self.srcLines[self.offset[line]:] + raise ParseError("ERROR %s\nLine %d:%s" % ( + msg, line, self.srcLines[self.offset[line]:])) + + def __computeOffsets(self): + """ Compute and store line offsets in self.offset. """ + self.offset = [0, 0] + self.lineCount = 0 + pos = 0 + while pos < len( self.srcLines ): + self.lineCount += 1 + pos = string.find( self.srcLines, '\n', pos ) + 1 + if not pos: break + self.offset.append( pos ) + self.offset.append( len( self.srcLines ) ) + + + def __push(self, toktype, semtype, toktext, srow, scol, line): + "Append given token to final list of tokens." + + self.tokenlist.append(mytoken.MyToken(type=toktype, semtype=semtype, text=toktext, row=srow, col=scol, line=line)) + if toktype in [NEWLINE,INDENT,DEDENT,EMPTY,ENDMARKER]: + self.prevToktype = None + self.prevSemtype = None + self.prevToktext = None + elif toktype != WS: + self.prevToktype = toktype + self.prevSemtype = semtype + self.prevToktext = toktext + + + def __call__(self, toktype, toktext, (srow,scol), (erow,ecol), line): + """ MyToken handler.""" + semtype = None + + # calculate new positions + oldpos = self.pos + newpos = self.offset[srow] + scol + self.pos = newpos + len(toktext) + + # check for extraneous '\r', usually produced in Windows and Mac systems + if toktype == ERRORTOKEN: # Python treats a '\r' as an error + if toktext in ['\r']: + toktext = ' ' + toktype = WS + else: + msg = "Invalid character %s in line %d column %d\n" % (str.__repr__(toktext), srow, scol+1) + sys.stderr.writelines( msg ) + sys.stdout.writelines( msg ) + # next line is commented out so that invalid tokens are not output + # self.__push(toktype, None, toktext, srow, scol, line) + return + + # handle newlines + if toktype in [NEWLINE, EMPTY]: + self.__push(toktype, None, '\n', srow, scol, line) + return + + # send the original whitespace, if needed + # this is really a reconstruction based on last + # and current token positions and lengths. + if newpos > oldpos: + # srow scol is the starting position for the current + # token that follows the whitespace. + # srow sws is the computed starting position of the + # whitespace + sws = scol - ( newpos - oldpos ) + self.__push(WS, None, self.srcLines[oldpos:newpos], srow, sws, line) + + # skip tokens that indent/dedent + if toktype in [INDENT, DEDENT]: + self.pos = newpos + self.__push(toktype, None, '', srow, scol, line) + return + + # map token type to one of ours and set semantic type, if possible + if token.LPAR <= toktype and toktype <= OP: + toktype = OP + if toktext == '@': + semtype = DECORATOR + elif toktype == NAME: + if keyword.iskeyword(toktext) or toktext == "self": + semtype = KEYWORD + else: + semtype = VARNAME + if self.prevToktext == "def": + semtype = FCNNAME + elif self.prevToktext == "class": + semtype = CLASSNAME + + # add token + self.__push(toktype, semtype, toktext, srow, scol, line) diff -uNr a/pymetrics-0.7.9/PyMetrics/mccabe.py b/pymetrics-0.7.9/PyMetrics/mccabe.py --- a/pymetrics-0.7.9/PyMetrics/mccabe.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/mccabe.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,75 @@ +""" Compute McCabe's Cyclomatic Metric. + + This routine computes McCabe's Cyclomatic metric for each function + in a module/file. + + $Id: mccabe.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.3 $"[11:-2] +__author__ = 'Reg. Charney ' + +from metricbase import MetricBase + +McCabeKeywords = { + 'assert':0, + 'break':1, + 'continue':1, + 'elif':1, + 'else':1, + 'for':1, + 'if':1, + 'while':1 + } + +class McCabeMetric( MetricBase ): + """ Compute McCabe's Cyclomatic McCabeMetric by function.""" + def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): + self.context = context + self.runMetrics = runMetrics + self.metrics = metrics + self.pa = pa + self.inFile = context['inFile'] + self.fcnNames = {} + + def processToken( self, fcnName, className, tok, *args, **kwds ): + """ Increment number of decision points in function.""" + if tok and tok.text in McCabeKeywords: + self.fcnNames[fcnName] = self.fcnNames.get(fcnName,0) + 1 + + def processFunction( self, fcnName, className, fcn, *args, **kwds ): + """ Increment number of decision points in function.""" + self.fcnNames[fcnName] = self.fcnNames.get(fcnName,0) + 1 + + def display( self ): + """ Display McCabe Cyclomatic metric for each function """ + result = {} + # the next three lines ensure that fcnNames[None] is treated + # like fcnNames['__main__'] and are sorted into alphabetical + # order. + if self.fcnNames.has_key(None): + self.fcnNames['__main__'] = self.fcnNames.get(None,0) + del self.fcnNames[None] + + if self.pa.quietSw: + return result + + hdr = "\nMcCabe Complexity Metric for file %s" % self.inFile + print hdr + print "-"*len(hdr) + "\n" + keyList = self.fcnNames.keys() + if len( keyList ) > 0: + keyList.sort() + for k in keyList: + if k: + name = k + else: + name = "__main__" + print "%11d %s" % (self.fcnNames[k],name) + result[k] = self.fcnNames[k] + else: + print "%11d %s" % (1,'__main__') + result['__main__'] = 1 + + print + return result + diff -uNr a/pymetrics-0.7.9/PyMetrics/metricbase.py b/pymetrics-0.7.9/PyMetrics/metricbase.py --- a/pymetrics-0.7.9/PyMetrics/metricbase.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/metricbase.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,48 @@ +""" Metric base class for new user-defined metrics. + + $Id: metricbase.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.2 $"[11:-2] +__author__ = 'Reg. Charney ' + +class MetricBase( object ): + """ Metric template class.""" + def __init__( self, *args, **kwds ): + pass + + def processSrcLines( self, srcLines, *args, **kwds ): + """ Handle physical line after tab expansion.""" + pass + + def processToken( self, fcnName, className, tok, *args, **kwds ): + """ Handle processing after each token processed.""" + pass + + def processStmt( self, fcnName, className, stmt, *args, **kwds ): + """ Handle processing at end of statement.""" + pass + + def processBlock( self, fcnName, className, block, *args, **kwds ): + """ Handle processing at end of block.""" + pass + + def processFunction( self, fcnName, className, fcn, *args, **kwds ): + """ Handle processing at end of function. """ + pass + + def processClass( self, fcnName, className, cls, *args, **kwds ): + """ Handle processing at end of class. """ + pass + + def processModule( self, moduleName, module, *args, **kwds ): + """ Handle processing at end of module. """ + pass + + def processRun( self, run, *args, **kwds ): + """ Handle processing at end of run. """ + pass + + def compute( self, *args, **kwds ): + """ Compute the metric given all needed info known.""" + pass + diff -uNr a/pymetrics-0.7.9/PyMetrics/mytoken.py b/pymetrics-0.7.9/PyMetrics/mytoken.py --- a/pymetrics-0.7.9/PyMetrics/mytoken.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/mytoken.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,33 @@ +""" MyToken - PyMetrics' version of Token. + + $Id: mytoken.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.3 $"[11:-2] +__author__ = 'Reg. Charney ' + +import token +from globals import * + +class MyToken: + def __init__(self, **kwds ): + """ Initialize class with user-defined keywords.""" + self.__dict__.update(kwds) + + def __repr__( self ): + """ Pretty print token. + + Don't print text for special token types since they do not have + useful visible representation, other than blank. + """ + tn = token.tok_name[self.type] + sn = self.semtype + if sn: + sn = token.tok_name[self.semtype] + if self.type in [WS,NEWLINE,INDENT,DEDENT,EMPTY,ENDMARKER]: + s = "[type=%s semtype=%s row=%s col=%s len=%d]" % (tn,sn,self.row,self.col,len(self.text)) + else: + if self.type == COMMENT and self.text[-1] == '\n': + self.text = self.text[:-1] + s = "[type=%s semtype=%s row=%s col=%s len=%d text=<%s>]" % (tn,sn,self.row,self.col,len(self.text),self.text) + return s + diff -uNr a/pymetrics-0.7.9/PyMetrics/newFeatures24.py b/pymetrics-0.7.9/PyMetrics/newFeatures24.py --- a/pymetrics-0.7.9/PyMetrics/newFeatures24.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/newFeatures24.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,204 @@ +""" Metrics for New Features introduced in Python 2.4. + + $Id: newfeatures24.py,v 1.4 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.1 $"[11:-2] +__author__ = 'Reg. Charney ' + +from metricbase import MetricBase +from globals import * + +class NewFeatures24Metric( MetricBase ): + """ Compute simple metrics by function.""" + firstPass = True + def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): + """ Count features introduced after v2.3.""" + self.context = context + self.runMetrics = runMetrics + self.metrics = metrics + self.pa = pa + self.inFile = context['inFile'] + + self.prevTokText = '' + self.userDefinedSet = False + self.featureMetrics = {} + self.numGeneratorFunctions = 0 + self.numGeneratorExpressions = 0 + self.numListComprehension = 0 + self.numClassProperties = 0 + self.numClassVariables = 0 + self.numClassFunctions = 0 + self.numDecorators = 0 + self.isGeneratorFunction = False + self.isGeneratorExpression = False + self.isListComprehension = False + self.usedInbuiltSets = False + self.usedDecorators = False + self.maybeSetKeyword = False + self.stackParenOrBracket = []; + self.featureMetrics['numGeneratorFunctions'] = 0 + self.featureMetrics['numGeneratorExpressions'] = 0 + self.featureMetrics['numListComprehensions'] = 0 + self.featureMetrics['numModulesUsingSets'] = 0 + self.featureMetrics['numDecorators'] = 0 + + if NewFeatures24Metric.firstPass: + self.runMetrics['modulesUsingGeneratorFunctions'] = [] + self.runMetrics['modulesUsingGeneratorExpressions'] = [] + self.runMetrics['modulesUsingListComprehension'] = [] + self.runMetrics['modulesUsingSets'] = [] + self.runMetrics['modulesUsingDecorators'] = [] + NewFeatures24Metric.firstPass = False + + def processToken( self, currentFcn, currentClass, tok, *args, **kwds ): + """ Collect token and context sensitive data for simple metrics.""" + if tok.semtype == KEYWORD: + self.__handleKeywords( currentFcn, currentClass, tok, *args, **kwds ) + elif tok.type == OP: + if self.maybeSetKeyword and tok.text == '(': + if not self.userDefinedSet: + self.usedInbuiltSets = True + self.maybeSetKeyword = False + elif tok.text == '(' or tok.text == '[': + self.stackParenOrBracket.append( tok.text ) + elif tok.text == ')' or tok.text == ']': + if len( self.stackParenOrBracket ) > 0: + del self.stackParenOrBracket[-1] + elif tok.text == '@': + self.usedDecorators = True + self.featureMetrics['numDecorators'] += 1 + elif tok.semtype == VARNAME: + if tok.text in ['set', 'frozenset']: + if not self.prevTokText in ['.', 'def', 'class']: + self.maybeSetKeyword = True + elif tok.semtype in [FCNNAME,CLASSNAME]: + # We need to ignore user-defined global set + # functions and classes. + if tok.text in ['set', 'frozenset']: + self.userDefinedSet = True + if tok.type != WS: + self.prevTokText = tok.text + return + + def __handleKeywords( self, currentFcn, currentClass, tok, *args, **kwds ): + """ Check for generator functions or expressions and list comprehension.""" + if tok.text == "yield": + self.isGeneratorFunction = True + elif tok.text == "for": + if len( self.stackParenOrBracket ) > 0: + punct = self.stackParenOrBracket[-1] + if punct == '(': + self.featureMetrics['numGeneratorExpressions'] += 1 + elif punct == '[': + self.featureMetrics['numListComprehensions'] += 1 + return + + def processStmt( self, fcnName, className, stmt, *args, **kwds ): + """ Handle processing at end of statement.""" + self.stackParenOrBracket = [] + + def processFunction( self, fcnName, className, block, *args, **kwds ): + """ Output stats at end of each function.""" + if self.isGeneratorFunction: + self.featureMetrics['numGeneratorFunctions'] += 1 + self.isGenerator = False + if self.usedInbuiltSets: + self.featureMetrics['numModulesUsingSets'] += 1 + self.usedInbuiltSets = False + + def processModule( self, moduleName, mod, *args, **kwds ): + """ Output stats at end of each module.""" + self.moduleName = moduleName + if self.featureMetrics['numModulesUsingSets'] > 0: + self.runMetrics['modulesUsingSets'].append( moduleName ) + if self.featureMetrics['numGeneratorFunctions'] > 0: + self.runMetrics['modulesUsingGeneratorFunctions'].append( moduleName ) + if self.featureMetrics['numGeneratorExpressions'] > 0: + self.runMetrics['modulesUsingGeneratorExpressions'].append( moduleName ) + if self.featureMetrics['numListComprehensions'] > 0: + self.runMetrics['modulesUsingListComprehension'].append( moduleName ) + if self.featureMetrics['numDecorators'] > 0: + self.runMetrics['modulesUsingDecorators'].append( moduleName ) + return + + def processRun( self, run, *args, **kwds ): + """ Output stats at end of run.""" + def __printHeader( printHeader ): + """ Only print heading if something in body of report.""" + if printHeader: + print """Python 2.4 Features Used During Run""" + print """-----------------------------------""" + return False + + def __printSubHeader( printHeader, key, desc ): + if len( self.runMetrics[key] ) > 0: + printHeader = __printHeader( printHeader ) + h1 = "Modules using %s" % desc + h2 = '.'*len( h1 ) + print + print h1 + print h2 + print + for modName in self.runMetrics[key]: + print modName + return False + + printHeader = True + printHeader = __printSubHeader( printHeader, + 'modulesUsingSets', + 'builtin set/frozenset (PEP 218)' ) + printHeader = __printSubHeader( printHeader, + 'modulesUsingGeneratorFunctions', + 'Generator Functions' ) + printHeader = __printSubHeader( printHeader, + 'modulesUsingGeneratorExpressions (PEP 289)', + 'Generator Expressions' ) + printHeader = __printSubHeader( printHeader, + 'modulesUsingListComprehension', + 'List Comprehension' ) + printHeader = __printSubHeader( printHeader, + 'modulesUsingDecorators', + 'Decorators (PEP 318)' ) + + return None + + def compute( self, *args, **kwds ): + """ Compute any values needed.""" + try: + self.featureMetrics['%FunctionsThatAreGenerators'] = 100.0 * self.featureMetrics['numGeneratorFunctions']/self.metrics['numFunctions'] + except (KeyError, ZeroDivisionError): + self.featureMetrics['%FunctionsThatAreGenerators'] = 0.0 + + return self.featureMetrics + + def display( self, currentFcn=None ): + """ Display and return new features in 2.4 metrics for given function.""" + def __printDisplayHdr(): + """ Only print heading if something in body of report.""" + h1 = """Python 2.4 Features Used in %s""" % self.inFile + h2 = '-'*len( h1 ) + print h1 + print h2 + print + return False + + printHeader = True + self.compute() + keyList = self.featureMetrics.keys() + keyList.sort() + for k in keyList: + if self.pa.zeroSw or self.featureMetrics[k]: + fmt = ( k[0] == '%' and "%14.2f %s" ) or "%11d %s" + if printHeader: printHeader = __printDisplayHdr() + print fmt % (self.featureMetrics[k],k) + print + + return self.featureMetrics + +if __name__=="__main__": + def Incr( init ): + for i in range(10): + yield i + s = set(i for i in range(0,10,2)) + fs = frozenset(i for i in range(5)) + print s, fs diff -uNr a/pymetrics-0.7.9/PyMetrics/processargs.py b/pymetrics-0.7.9/PyMetrics/processargs.py --- a/pymetrics-0.7.9/PyMetrics/processargs.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/processargs.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,483 @@ +""" Process command line arguments. + +Usage: + +>>> pa = ProcessArgs( 'file1.py', 'file2.py', '/home/files/file3.py' ) +>>> pa = ProcessArgs( "inFile1.py", sqlFileName='sqlF1.sql', genCsvSw=False,genKwCntSw=True ) +>>> pa = ProcessArgs() #doctest +NORMALIZE_WHITESPACE +ELLIPSIS +python PyMetrics [ options ] pgm1.py [ pgm2.py ... ] + +Complexity metrics are computed for the Python input files +pgm1.py, pgm2.py, etc. At least one file name is required, +else this message appears. + +Three types of output can be produced: + +* Standard output for a quick summary of the main metrics. +* A text file containing SQL commands necessary to build a SQL table + in the database of your choice. +* A text file containing Comma-Separated Values (CSV) formatted data + necessary to load into most spreadsheet programs. + PLEASE NOTE: multi-line literals have the new-line character + replaced with the character "\\n" in the output text. + +Capitalized options negate the default option. + +options: + --version show program's version number and exit + -h, --help show this help message and exit + -s SQLFILENAME, --sql=SQLFILENAME + name of output SQL command file. (Default is + metricData.sql) + -t SQLTOKENTABLENAME, --tokentable=SQLTOKENTABLENAME + name of output SQL token table. (Default is + metricTokens) + -m SQLMETRICSTABLENAME, --metricstable=SQLMETRICSTABLENAME + name of output SQL metrics table. (Default is + metricData) + -c CSVFILENAME, --csv=CSVFILENAME + name of output CSV data file. (Default is + metricData.csv) + -f INFILELIST, --files=INFILELIST + File containing list of path names to modules for + analysis. + -i INCLUDEMETRICSSTR, --include=INCLUDEMETRICSSTR + list of metrics to include in run. This is a comma + separated list of metric module names with no + whitespace. Optionally, you can specify the class name + of the metric by following the module name with a + colon (:) and the metric class name. (Default metrics + are 'simple:SimpleMetric,mccabe:McCabeMetric, + sloc:SLOCMetric'. Default metric class name for metric + module 'wxYz' is 'WxYzMetric' when only module name + given -- note capitalized metric class name.) + -l LIBNAME, --library=LIBNAME + user-defined name applied to collection of modules + (Default is '') + -e, --exists assume SQL tables exist and does not generate creation + code. Using this option sets option -N. (Default is + False) + -N, --noold create new command output files and tables after + deleting old results, if any. Ignored if -e is set. + (Default is False) + -B, --nobasic suppress production of Basic metrics (Default is + False) + -S, --nosql suppress production of output SQL command text file. + (Default is False) + -C, --nocsv suppress production of CSV output text file. (Default + is False) + -H, --noheadings suppress heading line in csv file. (Default is False) + -k, --kwcnt generate keyword counts. (Default is False) + -K, --nokwcnt suppress keyword counts. (Default is True) + -q, --quiet suppress normal summary output to stdout. (Default is + False) + -z, --zero display zero or empty values in output to stdout. + (Default is to suppress zero/empty output) + -v, --verbose Produce verbose output - more -v's produce more + output. (Default is no verbose output to stdout) + -d, --debug Provide debug output, not usually generated - internal + use only + +No program file names given. + + + $Id: processargs.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.3 $"[11:-2] +__author__ = 'Reg. Charney ' + +import sys +from optparse import OptionParser, BadOptionError +from doctestsw import * + +usageStr = """python PyMetrics [ options ] pgm1.py [ pgm2.py ... ] + +Complexity metrics are computed for the Python input files +pgm1.py, pgm2.py, etc. At least one file name is required, +else this message appears. + +Three types of output can be produced: + +* Standard output for a quick summary of the main metrics. +* A text file containing SQL commands necessary to build a SQL table + in the database of your choice. +* A text file containing Comma-Separated Values (CSV) formatted data + necessary to load into most spreadsheet programs. + PLEASE NOTE: multi-line literals have the new-line character + replaced with the character "\\n" in the output text. + +Capitalized options negate the default option. +""" + +class PyMetricsOptionParser( OptionParser ): + """ Subclass OptionParser so I can override default error handler.""" + def __init__( self, *args, **kwds ): + """ Just call super class's __init__ since we aren't making changes here.""" + OptionParser.__init__( self, *args, **kwds ) + + def error( self, msg ): + """ Explicitly raise BadOptionError so calling program can handle it.""" + raise BadOptionError( msg ) + +class ProcessArgsError( Exception ): pass + +class ProcessArgs( object ): + """ Process command line arguments.""" + def __init__( self, + *pArgs, + **pKwds + ): + """ Initial processing of arguments.""" + + # precedence for defaults, parameters, and command line arguments/keywords + # is: + # + # command line parameters/keywords take precedence over + # parameters used to instantiate ProcessArgs that takes precedence over + # default values for keywords. + # + # This implies we set completed options with defaults first + # then override completed options with parameters + # then override completed options with command line args/keywords + # + + # default values for possible parameters + libName = '' + sqlFileName = "metricData.sql" + sqlTokenTableName = "metricTokens" + sqlMetricsTableName = "metricData" + csvFileName = "metricData.csv" + inFileList = None + includeMetricsStr = 'simple:SimpleMetric,mccabe:McCabeMetric,sloc:SLOCMetric' + excludeMetricsStr = None + debugSw = False + debugSw = False + quietSw = False + zeroSw = False + genBasicSw = True + genKwCntSw = False + genSqlSw = True + genCsvSw = True + genHdrSw = True + genAppendSw = True + genNewSw = False + genExistsSw = False + verbose = 0 + + self.__dict__.update( locals() ) + del( self.__dict__['self'] ) # remove recursive self from self.__dict__ + self.__dict__.update( pKwds ) + del( self.__dict__['pKwds'] ) # remove redundant pKwds in self.__dict__ + + # set up option parser + parser = PyMetricsOptionParser( '', version="%prog 0.7.9" ) + parser.add_option("-s", "--sql", + dest="sqlFileName", + default=self.sqlFileName, + help="name of output SQL command file. (Default is %s)" % self.sqlFileName ) + parser.add_option("-t", "--tokentable", + dest="sqlTokenTableName", + default=self.sqlTokenTableName, + help="name of output SQL token table. (Default is %s)" % self.sqlTokenTableName ) + parser.add_option("-m", "--metricstable", + dest="sqlMetricsTableName", + default=self.sqlMetricsTableName, + help="name of output SQL metrics table. (Default is %s)" % self.sqlMetricsTableName ) + parser.add_option("-c", "--csv", + dest="csvFileName", + default=self.csvFileName, + help="name of output CSV data file. (Default is %s)" % self.csvFileName ) + parser.add_option("-f", "--files", + dest="inFileList", + default=self.inFileList, + help="File containing list of path names to modules for analysis." ) + parser.add_option("-i", "--include", + dest="includeMetricsStr", + default=self.includeMetricsStr, + help="list of metrics to include in run. This is a comma separated list of metric module names with no whitespace. Optionally, you can specify the class name of the metric by following the module name with a colon (:) and the metric class name. (Default metrics are 'simple:SimpleMetric,mccabe:McCabeMetric,sloc:SLOCMetric'. Default metric class name for metric module 'wxYz' is 'WxYzMetric' when only module name given -- note capitalized metric class name.)" ) + parser.add_option("-l", "--library", + dest="libName", + default=self.libName, + help="user-defined name applied to collection of modules (Default is '')" ) + parser.add_option("-e", "--exists", + action="store_true", + dest="genExistsSw", + default=self.genExistsSw, + help="assume SQL tables exist and does not generate creation code. Using this option sets option -N. (Default is %s)" % (self.genExistsSw) ) + parser.add_option("-N", "--noold", + action="store_true", + dest="genNewSw", + default=self.genNewSw, + help="create new command output files and tables after deleting old results, if any. Ignored if -e is set. (Default is %s)" % (self.genNewSw) ) + parser.add_option("-B", "--nobasic", + action="store_false", + dest="genBasicSw", + default=self.genBasicSw, + help="suppress production of Basic metrics (Default is %s)" % (not self.genBasicSw) ) + parser.add_option("-S", "--nosql", + action="store_false", + dest="genSqlSw", + default=self.genSqlSw, + help="suppress production of output SQL command text file. (Default is %s)" % (not self.genSqlSw) ) + parser.add_option("-C", "--nocsv", + action="store_false", + dest="genCsvSw", + default=self.genCsvSw, + help="suppress production of CSV output text file. (Default is %s)" % (not self.genCsvSw) ) + parser.add_option("-H", "--noheadings", + action="store_false", + dest="genHdrSw", + default=self.genHdrSw, + help="suppress heading line in csv file. (Default is %s)" % (not self.genHdrSw) ) + parser.add_option("-k", "--kwcnt", + action="store_true", + dest="genKwCntSw", + default=self.genKwCntSw, + help="generate keyword counts. (Default is %s)" % (self.genKwCntSw,) ) + parser.add_option("-K", "--nokwcnt", + action="store_false", + dest="genKwCntSw", + default=self.genKwCntSw, + help="suppress keyword counts. (Default is %s)" % (not self.genKwCntSw) ) + parser.add_option("-q", "--quiet", + action="store_true", + dest="quietSw", + default=self.quietSw, + help="suppress normal summary output to stdout. (Default is %s)" % (self.quietSw) ) + parser.add_option("-z", "--zero", + action="store_true", + dest="zeroSw", + default=self.zeroSw, + help="display zero or empty values in output to stdout. (Default is to suppress zero/empty output)" ) + parser.add_option("-v", "--verbose", + action="count", + dest="verbose", + default=self.verbose, + help="Produce verbose output - more -v's produce more output. (Default is no verbose output to stdout)") + parser.add_option("-d", "--debug", + action="store_true", + dest="debugSw", + default=self.debugSw, + help="Provide debug output, not usually generated - internal use only") + + # parse the command line/arguments for this instance + try: + (options, args) = parser.parse_args() + except BadOptionError, e: + sys.stderr.writelines( "\nBadOptionError: %s\n" % str( e ) ) + sys.stderr.writelines( "\nThe valid options are:\n\n" ) + sys.stderr.writelines( parser.format_help() ) + sys.exit( 1 ) + + # augment parameter values from instantiation with + # command line values. + # the command line parameter values take precidence + # over values in program. + + args.extend( pArgs ) + + # convert command line arguments into instance values + self.__dict__.update( options.__dict__ ) + + if self.inFileList: + try: + inF = open( self.inFileList ) + files = inF.read().split() + inF.close() + args.extend( files ) + except IOError, e: + raise ProcessArgsError( e ) + + self.inFileNames = args + + self.includeMetrics = self.processIncludeMetrics( self.includeMetricsStr ) + + if len( args ) < 1: + print usageStr + print parser.format_help() + e = "No program file names given.\n" + # because of what I believe to be a bug in the doctest module, + # which makes it mishandle exceptions, I have 'faked' the handling + # of raising an exception and just return + if doctestSw: + print e + return + else: + raise ProcessArgsError( e ) + + so = None + if self.genSqlSw and self.sqlFileName: + # Try opening command file for input. If OK, then file exists. + # Else, the commamd file does not exist to create SQL table and + # this portion of the command file must be generated as if new + # table was to be created. + # + # NOTE: This assumption can be dangerous. If the table does exist + # but the command file is out of sync with the table, then a valid + # table can be deleted and recreated, thus losing the old data. + # The workaround for this is to examine the SQL database for the + # table and only create the table, if needed. + try: + if self.sqlFileName: + self.so = open( self.sqlFileName, "r" ) + self.so.close() + except IOError: + self.genNewSw = True + + try: + mode = "a" + if self.genNewSw: + mode = "w" + so = open( self.sqlFileName, mode ) # check to see that we can write file + so.close() + except IOError, e: + sys.stderr.writelines( str(e) + " -- No SQL command file will be generated\n" ) + self.genSqlSw = False + so = None + + co = None + if self.genCsvSw and self.csvFileName: + try: + mode = "a" + if self.genNewSw: + mode = "w" + co = open( self.csvFileName, mode ) # check we will be able to write file + co.close() + except IOError, e: + sys.stderr.writelines( str(e) + " -- No CSV output file will be generated\n" ) + self.genCsvSw = False + co = None + + if self.genExistsSw: + # Assuming that table already exists, means concatenating + # data is not needed - we only want new data in SQL command file. + self.genNewSw = True + + def conflictHandler( self, *args, **kwds ): + print "args=%s" % args + print "kwds=%s" % kwds + + def processIncludeMetrics( self, includeMetricsStr ): + includeMetrics = [] + try: + metricList = includeMetricsStr.split( ',' ) + for a in metricList: + s = a.split( ':' ) + if len( s ) == 2: # both metric class and module name given + includeMetrics.append( s ) + elif len( s ) == 1: + # only the module name given. Generate default metric + # class name by capitalizing first letter of module + # name and appending "Metric" so the default metric + # class name for module wxYz is WxYzMetric. + if s[0]: + defName = s[0][0].upper() + s[0][1:] + 'Metric' + includeMetrics.append( (s[0], defName) ) + else: + raise ProcessArgsError("Missing metric module name") + else: + raise ProcessArgsError("Malformed items in includeMetric string") + except AttributeError, e: + e = ( "Invalid list of metric names: %s" % + includeMetricsStr ) + raise ProcessArgsError( e ) + return includeMetrics + +def testpa( pa ): + """ Test of ProcessArgs. + + Usage: + + >>> pa=ProcessArgs('inFile.py') + >>> testpa(pa) #doctest: +NORMALIZE_WHITESPACE +ELLIPSIS + Arguments processed: + sqlFileName=metricData.sql + sqlTokenTableName=metricTokens + sqlMetricsTableName=metricData + csvFileName=metricData.csv + Include Metric Modules=simple:SimpleMetric,mccabe:McCabeMetric + quietSw=False + genCsvSw=True + genHdrSw=True + genKwCntSw=False + verbose=... + Metrics to be used are: + Module simple contains metric class SimpleMetric + Module mccabe contains metric class McCabeMetric + Input files: + inFile.py + >>> + """ + print """ +Arguments processed: +\tsqlFileName=%s +\tsqlTokenTableName=%s +\tsqlMetricsTableName=%s +\tcsvFileName=%s +\tinclude Metric Modules=%s +\tquietSw=%s +\tgenNewSw=%s +\tgenExistsSw=%s +\tgenSqlSw=%s +\tgenCsvSw=%s +\tgenHdrSw=%s +\tgenKwCntSw=%s +\tverbose=%s""" % ( + pa.sqlFileName, + pa.sqlTokenTableName, + pa.sqlMetricsTableName, + pa.csvFileName, + pa.includeMetricsStr, + pa.quietSw, + pa.genNewSw, + pa.genExistsSw, + pa.genSqlSw, + pa.genCsvSw, + pa.genHdrSw, + pa.genKwCntSw, + pa.verbose) + print "Metrics to be used are:" + for m,n in pa.includeMetrics: + print "\tModule %s contains metric class %s" % (m,n) + if pa.inFileNames: + print "Input files:" + for f in pa.inFileNames: + print "\t%s" % f + +if __name__ == "__main__": + + # ignore doctestsw.doctestSw if this is run as a standalone + # module. Instead, look at the first argument on the command + # line. If it is "doctest", set doctestSw = True and delete + # the parameter "doctest" from sys.argv, thus leaving things + # as _normal_. + if len(sys.argv) > 1 and sys.argv[1] == 'doctest': + doctestSw = True + sys.argv[1:] = sys.argv[2:] + import doctest + doctest.testmod( sys.modules[__name__] ) + sys.exit( 0 ) + + print "=== ProcessArgs: %s ===" % ("'file1.py', 'file2.py', '/home/files/file3.py'",) + try: + pa = ProcessArgs( 'file1.py', 'file2.py', '/home/files/file3.py' ) + testpa( pa ) + except ProcessArgsError, e: + sys.stderr.writelines( str( e ) ) + print + print "=== ProcessArgs: %s ===" % ("inFile1.py, sqlFileName='sqlF1.sql', genCsvSw=False,genKwCntSw=True",) + try: + pa = ProcessArgs( "inFile1.py", sqlFileName='sqlF1.sql', genCsvSw=False,genKwCntSw=True ) + testpa( pa ) + except ProcessArgsError, e: + sys.stderr.writelines( str( e ) ) + print + print "=== ProcessArgs: %s ===" % ("No arguments",) + try: + pa = ProcessArgs() + testpa( pa ) + except ProcessArgsError, e: + sys.stderr.writelines( str( e ) ) + print + sys.exit( 0 ) + diff -uNr a/pymetrics-0.7.9/PyMetrics/PyMetrics.py b/pymetrics-0.7.9/PyMetrics/PyMetrics.py --- a/pymetrics-0.7.9/PyMetrics/PyMetrics.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/PyMetrics.py 2008-07-23 13:21:52.000000000 +0200 @@ -0,0 +1,281 @@ +#! /usr/bin/env python +""" PyMetrics - Complexity Measurements for Python code. + + Orignally based on grop.py by Jurgen Hermann. + Modified by Reg. Charney to do Python complexity measurements. + + Copyright (c) 2001 by Jurgen Hermann + Copyright (c) 2007 by Reg. Charney + + All rights reserved, see LICENSE for details. + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + $Id$ +""" +__revision__ = "$Id$" +__author__ = 'Reg. Charney ' + +# Imports +import sys +import string +from processargs import ProcessArgs, ProcessArgsError +import token +import sqltokenout +import sqldataout +import csvout +from lexer import Lexer +from compute import ComputeMetrics +from globals import * + +############################################################################# +### Main script for PyMetrics utility. +############################################################################# + +def __importMetricModules( includeMetrics ): + """ Import the modules specified in the parameter list. + + includeMetrics is a list of (metricModuleName, metricClassName) + pairs. This function defines a dictionary containing only valid + module/class names. When an error is found, the invalid + module/class pair is removed from the included list of metrics. + """ + i = 0 + metricModules = {} + for m,n in includeMetrics: + try: + mm = "PyMetrics.%s" % (m) + mod = __import__( mm, fromlist = [m] ) + metricModules[m] = mod + i += 1 + except ImportError: + sys.stderr.write( "Unable to import metric module %s -- ignored.\n\n" % m ) + # remove the erroneous metric module/class tuple + del includeMetrics[i] + + return metricModules + +def __instantiateMetric( metricModules, context, runMetrics, metrics, pa ): + """ Instantiate all user specified metric classes. + + The code works by finding the desired metric class in a metric module and + instantiating the class. It does this by assuming that the metric + class is in the dictionary of the metric module. + """ + metricInstance = {} + inclIndx = -1 + for m,n in pa.includeMetrics: + inclIndx += 1 + try: + metricInstance[m] = None # default value if metric class does not exist. + metricInstance[m] = metricModules[m].__dict__[n]( context, runMetrics, metrics, pa ) + except KeyError: + sys.stderr.write( "Module %s does not contain metric class %s -- metric %s ignored.\n\n" % (m,n,m) ) + del( metricInstance[m] ) + del( pa.includeMetrics[inclIndx] ) + + return metricInstance + +def __stats( so, m, inFileName, label, *args ): + """ Print line of statistics.""" + result = string.join(map(str, args), '') + print "%11s %s" % (result, label) + so and so.write( m, inFileName, label, result ) + +def __printSummary( so, context, runMetrics, metrics, pa ): + """ Print basic summary information.""" + # the following loop is a very, very ugly hack to distinguish between + # tokens as they appear in the source; semantically generated tokens, + # like DOCSTRING; and NONTOKENs, like numComments + + keys = [] + for k in metrics.keys(): + if str( k ).isdigit(): + keys.append( (token.tok_name[k],k,metrics[k]) ) + elif len( str( k ).split() ) > 1: + keys.append( (k,SEMTOKEN,metrics[k]) ) + else: + keys.append( (k,NONTOKEN,metrics[k]) ) + keys.sort() + + inFileName = context['inFile'] + if pa.genKwCntSw: + hdr = "Counts of Token Types in module %s" % context['inFile'] + print + print hdr + print "-"*len(hdr) + print + for k,t,v in keys: + if (pa.zeroSw or v): + if t != NONTOKEN: + __stats( so, 'basic', inFileName, k, v ) + print + + if pa.genBasicSw: + __displayBasicMetrics( keys, pa, so, inFileName ) + +def __displayBasicMetrics( keys, pa, so, inFileName ): + """ Display the Basic metrics that PyMetrics computes.""" + hdr = "Basic Metrics for module %s" % inFileName + print + print hdr + print "-"*len( hdr ) + print + for k,t,v in keys: + if t==NONTOKEN: + if pa.zeroSw or not v in ([],{},(),0,0.00): + __stats( so, 'basic', inFileName, k, v ) + print + +def main(): + """ Main routine for PyMetrics.""" + # process command line args + try: + pa = ProcessArgs() + except ProcessArgsError, e: + sys.stderr.writelines( str(e) ) + return + + if pa.genNewSw: + __deleteOldOutputFiles( pa ) + + so, od = __genNewSqlCmdFiles( pa ) + co = __genNewCsvFile( pa ) + + # import all the needed metric modules + metricModules = __importMetricModules( pa.includeMetrics ) + + runMetrics = {} # metrics for whole run + metrics = {} # metrics for this module + context = {} # context in which token was used + + # main loop - where all the work is done + for inFileName in pa.inFileNames: + + metrics.clear() + context.clear() + context['inFile'] = inFileName + + # instantiate all the desired metric classes + metricInstance = __instantiateMetric( metricModules, context, runMetrics, metrics, pa ) + + cm = ComputeMetrics( metricInstance, context, runMetrics, metrics, pa, so, co ) + + # define lexographical scanner to use for this run + # later, this may vary with file and language. + lex = Lexer() + + if not pa.quietSw: + print "=== File: %s ===" % inFileName + + try: + lex.parse( inFileName ) # parse input file + + metrics["numCharacters"] = len(lex.srcLines) + metrics["numLines"] = lex.lineCount # lines of code + + metrics = cm( lex ) + + # if printing desired, output summary and desired metrics + # also, note that this preserves the order of the metrics desired + if not pa.quietSw: + __printSummary( od, context, runMetrics, metrics, pa ) + for m,n in pa.includeMetrics: + if metricInstance[m]: + result = metricInstance[m].display() + if metrics.has_key(m): + metrics[m].append( result ) + else: + metrics[m] = result + for r in result.keys(): + od and od.write( m, inFileName, r, result[r] ) + except IOError, e: + sys.stderr.writelines( str(e) + " -- Skipping input file.\n\n") + + co and co.close() + + result = {} + if len( pa.inFileNames ) > 0: + for m,n in pa.includeMetrics: + if metricInstance[m]: + result = metricInstance[m].processRun( None ) + if result: + for r in result.keys(): + od and od.write( m, None, r, result[r] ) + od and od.close() + + if not pa.quietSw: + n = len( pa.inFileNames ) + print + print "*** Processed %s module%s in run ***" % (n,(n>1) and 's' or '') + +def __genNewCsvFile( pa ): + """ Determine output CSV data file, if any, + and check it can be created.""" + co = None + try: + if pa.genCsvSw: + co = csvout.CsvOut( pa.csvFileName, genHdrSw=pa.genHdrSw, genNewSw=pa.genNewSw ) + except StandardError, e: + # this should not occur - it should be handled in processArgs. + sys.stderr.writelines( str(e) + " -- No CSV file will be generated\n\n" ) + pa.genCsvSw = False + + return co + +def __genNewSqlCmdFiles( pa ): + """ determine output SQL tokens command file, if any, + and check it can be created .""" + so = None + co = None + od = None + fd = None + + try: + if pa.genSqlSw: + if pa.sqlFileName: + fd = open( pa.sqlFileName, 'a' ) + else: + fd = sys.stdout + so = sqltokenout.SqlTokenOut( fd, pa.libName, pa.sqlFileName, pa.sqlTokenTableName, pa.genNewSw, pa.genExistsSw ) + od = sqldataout.SqlDataOut( fd, pa.libName, pa.sqlFileName, pa.sqlMetricsTableName, pa.genNewSw, pa.genExistsSw ) + except StandardError, e: + # this should not occur - it should be handled in processArgs. + sys.stderr.writelines( str(e) + " -- No SQL command file will be generated\n\n" ) + pa.genSqlSw = False + so and so.close() + od and od.close() + so = None + od = None + + return so, od + +def __deleteOldOutputFiles( pa ): + """ Generate new output files by ensuring old files deleted.""" + import os + try: + if pa.genSqlSw and os.path.exists( pa.sqlFileName ): + os.remove( pa.sqlFileName ) + except IOError, e: + sys.stderr.writelines( str(e) ) + pa.genSqlSw = False + + try: + if pa.genCsvSw and os.path.exists( pa.csvFileName ): + os.remove( pa.csvFileName ) + except IOError, e: + sys.stderr.writelines( str(e) ) + pa.genCsvSw = False + return + +if __name__ == "__main__": + main() + sys.exit(0) diff -uNr a/pymetrics-0.7.9/PyMetrics/simple.py b/pymetrics-0.7.9/PyMetrics/simple.py --- a/pymetrics-0.7.9/PyMetrics/simple.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/simple.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,121 @@ +""" Simple Metrics for each function within a file. + + $Id: simple.py,v 1.4 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.4 $"[11:-2] +__author__ = 'Reg. Charney ' + +from metricbase import MetricBase +from globals import * + +class SimpleMetric( MetricBase ): + """ Compute simple metrics by function.""" + def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): + self.context = context + self.runMetrics = runMetrics + self.metrics = metrics + self.pa = pa + self.inFile = context['inFile'] + self.simpleMetrics = {} + self.mcc = {} + self.fcnNames = {} + self.classNames = {} + self.lastFcnName = None + self.lastClassName = None + self.metrics['DocFunctions'] = [] + self.metrics['DocClasses'] = [] + + def processToken( self, currentFcn, currentClass, tok, *args, **kwds ): + """ Collect token and context sensitive data for simple metrics.""" + if tok.semtype == DOCSTRING: + self.metrics['numDocStrings'] = self.metrics.get('numDocStrings',0) + 1 + if currentFcn and currentFcn != "__main__": + if self.lastFcnName != currentFcn: + self.lastFcnName = currentFcn + self.metrics['numFcnDocStrings'] = self.metrics.get('numFcnDocStrings',0) + 1 + self.fcnNames[currentFcn] = tok.row + elif currentClass == None: # this doc string must be module doc string + self.lastFcnName = "__main__" + self.metrics['numFcnDocStrings'] = self.metrics.get('numFcnDocStrings',0) + 1 + self.fcnNames[self.lastFcnName] = tok.row + if currentClass: + if self.lastClassName != currentClass: + self.lastClassName = currentClass + self.metrics['numClassDocStrings'] = self.metrics.get('numClassDocStrings',0) + 1 + self.classNames[currentClass] = tok.row + elif tok.semtype == FCNNAME: + self.fcnNames[currentFcn] = 0 + elif tok.semtype == CLASSNAME: + self.classNames[currentClass] = 0 + + def processBlock( self, currentFcn, currentClass, block, *args, **kwds ): + """ Collect token and context sensitive data for simple metrics.""" + self.metrics['numBlocks'] = self.metrics.get('numBlocks',0)+1 + + def compute( self, *args, **kwds ): + """ Compute any values needed.""" + if self.metrics.get('numModuleDocStrings', 0) > 0: + self.metrics['numFunctions'] = self.metrics.get('numFunctions', 0) + 1 + + try: + self.simpleMetrics['%Comments'] = 100.0 * self.metrics['numComments']/self.metrics['numLines'] + except (KeyError, ZeroDivisionError): + self.simpleMetrics['%Comments'] = 0.0 + + try: + self.simpleMetrics['%CommentsInline'] = 100.0 * self.metrics['numCommentsInline']/self.metrics['numLines'] + except (KeyError, ZeroDivisionError): + self.simpleMetrics['%CommentsInline'] = 0.0 + + if 0: + try: + self.simpleMetrics['%DocStrings'] = 100.0 * self.metrics['numDocStrings']/(self.metrics['numModuleDocStrings']+self.metrics['numClasses']+self.metrics['numFunctions']) + except (KeyError, ZeroDivisionError): + self.simpleMetrics['%DocStrings'] = 0.0 + + try: + self.simpleMetrics['%FunctionsHavingDocStrings'] = 100.0 * self.metrics['numFcnDocStrings']/self.metrics['numFunctions'] + except (KeyError, ZeroDivisionError): + self.simpleMetrics['%FunctionsHavingDocStrings'] = 0.0 + + try: + self.simpleMetrics['%ClassesHavingDocStrings'] = 100.0 * self.metrics['numClassDocStrings']/self.metrics['numClasses'] + except (KeyError, ZeroDivisionError): + self.simpleMetrics['%ClassesHavingDocStrings'] = 0.0 + + return self.simpleMetrics + + def display( self, currentFcn=None ): + """ Display and return simple metrics for given function.""" + + def __printNames( typeName, names ): + """ Pretty print list of functions/classes that have doc strings.""" + if len( names ): # only output something if it exists + hdr = "%s DocString present(+) or missing(-)" % typeName + print + print hdr + print "-"*len(hdr) + "\n" + result = [] + keys = names.keys() + keys.sort() + for k in keys: + if k: + pfx = (names[k] and '+') or '-' + print "%c %s" % (pfx,k) + print + + result = (self.inFile, names) + return result + + self.compute() + keyList = self.simpleMetrics.keys() + keyList.sort() + for k in keyList: + if self.pa.zeroSw or self.simpleMetrics[k]: + fmt = ( k[0] == '%' and "%14.2f %s" ) or "%11d %s" + print fmt % (self.simpleMetrics[k],k) + + self.metrics['DocFunctions'].append( __printNames( 'Functions', self.fcnNames ) ) + self.metrics['DocClasses'].append( __printNames( 'Classes', self.classNames ) ) + + return self.simpleMetrics diff -uNr a/pymetrics-0.7.9/PyMetrics/sloc.py b/pymetrics-0.7.9/PyMetrics/sloc.py --- a/pymetrics-0.7.9/PyMetrics/sloc.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/sloc.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,151 @@ +""" Compute COCOMO 2's SLOC (Source Lines Of Code) metric. + + Compute Source Lines Of Code for each function/class/file. + This is based on COCOMO 2's definition of constitutes a + line of code. + + Algorithm: + Delete all non-literal blank lines + Delete all non-literal comments + Delete all doc strings + Combine parenthesised expressions into one logical line + Combine continued lines into one logical line + Return count of resulting lines + + Conventions: + Continued lines are those ending in \) are treated as one + physical line. + Paremeter lists and expressions enclosed in parens (), + braces {}, or brackets [] are treated as being part + of one physical line. + All literals are treated as being part of one physical + line + + $Id: sloc.py,v 1.2 2005/06/26 07:08:58 rcharney Exp $ +""" +__revision__ = "$Revision: 1.1 $"[11:-2] +__author__ = 'Reg. Charney ' + +import re + +from metricbase import MetricBase + +class SLOCMetric( MetricBase ): + """ Compute Source Lines Of Code metric.""" + def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): + # pattern definitions + patBackslashQuote = r'\\\'|\\\"' + patTripleQuote1 = r"r?'''.*?'''" + patTripleQuote2 = r'r?""".*?"""' + patQuote1 = r"'.*?'" + patQuote2 = r'".*?"' + self.patLiteral = patTripleQuote1 + '|' + patQuote1 + '|' + patTripleQuote2 + '|' + patQuote2 + self.patComment = '#.*\n' + self.patParen = '\(.*?\)' + self.patDefOrClass = '^\s*(def|class)\s+\w.*:' + self.patAllBlank = '^\s*\n|^\s*@\n|^\s*~\n' + # compile patterns + self.reBackslashQuote = re.compile( patBackslashQuote ) + self.reLiteral = re.compile( self.patLiteral, re.M + re.S ) + self.reComment = re.compile( self.patComment ) + self.reParen = re.compile( self.patParen, re.M + re.S ) + self.reDefOrClass = re.compile( self.patDefOrClass, re.M + re.S ) + self.reAllBlank = re.compile( self.patAllBlank, re.M + re.S ) + + self.numSLOC = 0 + self.context = context + self.runMetrics = runMetrics + self.metrics = metrics + self.pa = pa + self.inFile = context['inFile'] + self.fcnNames = {} + + def processSrcLines( self, rawLines, *args, **kwds ): + """ Process all raw source lines in one fell swoop. + If a given line is not blank and contains something + besides a comment, then increment the numSLOC.""" + if len( rawLines ) == 0: + # ignore file + self.numSLOC = 0 + return + + noBackslashQuotes = re.sub( self.reBackslashQuote, '*', rawLines ) + noLiteralLines = re.sub( self.reLiteral, '@', noBackslashQuotes ) + noCommentLines = re.sub( self.reComment, '~\n', noLiteralLines ) + noBlankLines = re.sub( self.reAllBlank, '%', noCommentLines ) + noParenLines = re.sub( self.reParen, '(...)', noBlankLines ) + self.numSLOC = noParenLines.count( '\n' ) + return + + def display( self ): + """ Display SLOC metric for each file """ + result = {} + result[self.inFile] = self.numSLOC + + if self.pa.quietSw: + return result + + hdr = "\nCOCOMO 2's SLOC Metric for %s" % self.inFile + print hdr + print "-"*len(hdr) + "\n" + print "%11d %s" % (self.numSLOC,self.inFile) + print + + return result + +def __main( fn, debugSw ): + """ Process lines from input file.""" + fd = open( fn ) + rawLines = fd.read() + fd.close() + + class PA: pass + pa = PA() + pa.quietSw = True + pa.debugSw = debugSw + + __processFile(fn, pa, rawLines) + +def __processFile(fn, pa, rawLines): + """ Invoke metric function after setting up scaffolding.""" + sloc = SLOCMetric( {'inFile':fn},[],pa ) + sloc.processSrcLines( rawLines ) + print sloc.display() + +def __getFNList(): + """ Return list of input file names, + regardless of whether the list came + from a file or the command line.""" + import sys + debugSw = False + fnList = [] + for arg in sys.argv[1:]: + if arg == '-d': + debugSw = True + continue + if arg == '-f': + fd = open(sys.argv[2]) + lines = fd.read() + fd.close() + fnList = __normalizeLines( lines ) + continue + fnList.append( arg ) + return fnList,debugSw + +def __normalizeLines( lines ): + """ Remove trailing newlines, if needed, + and return list of input file names.""" + fnList = [] + if len( lines ) > 0: + if lines[-1] == '\n': + lines = lines[:-1] + fnList = lines.split( '\n' ) + return fnList + +if __name__ == "__main__": + fnList,debugSw = __getFNList() + for fn in fnList: + try: + fn!='' and __main( fn, debugSw ) + except: + pass diff -uNr a/pymetrics-0.7.9/PyMetrics/sqldataout.py b/pymetrics-0.7.9/PyMetrics/sqldataout.py --- a/pymetrics-0.7.9/PyMetrics/sqldataout.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/sqldataout.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,78 @@ +""" SqlDataOut - class to produce SQL data command file output. + + $Id: SqlDataOut.py,v 1.2 2005/02/15 07:08:58 rcharney Exp $ +""" +__revision__ = "$Revision: 1.2 $"[11:-2] +__author__ = 'Reg. Charney ' + +import sys +import time +import token +import tokenize +import utils + +import sqltemplate + +class InvalidTableNameError( Exception ): + """ Used to indicate that the SQL table name is invalid.""" + pass + +class SqlDataOut( object ): + """ Class used to generate a command file suitable for runnning against + any SQL dbms.""" + def __init__( self, + fd, + libName, + fileName, + tableName, + genNewSw=False, + genExistsSw=False ): + """ Initialize instance of SqlDataOut.""" + if tableName == '': + raise InvalidTableNameError( tableName ) + if not fd: + raise IOError( "Output file does not yet exist" ) + + timestamp = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()) + self.libName = libName + self.fileName = fileName + self.tableName = tableName + self.quotedFileName = '"'+self.fileName+'"' + self.IDDateTime = '"'+timestamp+'"' + self.toknum = 0 + self.fd = fd + + if not genExistsSw: + self.writeHdr( genNewSw, tableName ) + + def writeHdr( self, genNewSw, tableName ): + """ Write header information for creating SQL command file.""" + if genNewSw: + import re + r = re.compile( '\w+' ) + if r.match( tableName ): + self.fd.write( + sqltemplate.dataHdr % + (tableName, tableName, tableName, tableName) + ) + else: + raise AttributeError( 'Invalid table name' ) + + def write( self, metricName, srcFileName, varName, value ): + """ Generate the Sql INSERT line into the sql command file.""" + sArgs = ','.join( ( + self.IDDateTime, + '0', + '"'+str( self.libName )+'"', + '"'+str( metricName )+'"', + '"'+str( srcFileName )+'"', + '"'+str( varName )+'"', + '"'+str( value )+'"' + ) ) + sOut = sqltemplate.dataInsert % (self.tableName, sArgs) + self.fd and self.fd.write( sOut ) + + def close( self ): + """ Close file, if it is opened.""" + self.fd and self.fd.close() + self.fd = None diff -uNr a/pymetrics-0.7.9/PyMetrics/sqltemplate.py b/pymetrics-0.7.9/PyMetrics/sqltemplate.py --- a/pymetrics-0.7.9/PyMetrics/sqltemplate.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/sqltemplate.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,66 @@ +""" sqltemplate - template for generating sql token output. """ + +__revision__ = "$Revision: 1.2 $"[11:-2] +__author__ = 'Reg. Charney ' + +tokenHdr = """-- +-- Automatically generated table structure for token table `%s` +-- + +DROP TABLE IF EXISTS %s; + +CREATE TABLE %s ( + IDDateTime datetime NOT NULL default '0000-00-00 00:00:00', + ID int(11) unsigned NOT NULL auto_increment, + libraryName varchar(32) default '', + fileName varchar(255) default NULL, + lineNum int(11) NOT NULL, + colNum int(11) NOT NULL, + type varchar(16) NOT NULL default 'ERRORTOKEN', + semtype varchar(16) default NULL, + textLen int(11) NOT NULL default 1, + text varchar(255) NOT NULL default '', + fqnFunction varchar(255) default NULL, + fqnClass varchar(255) default NULL, + blockNum int(11) NOT NULL default 1, + blockDepth int(11) NOT NULL default 0, + fcnDepth int(11) NOT NULL default 0, + classDepth int(11) NOT NULL default 0, + parenDepth int(11) NOT NULL default 0, + bracketDepth int(11) NOT NULL default 0, + braceDepth int(11) NOT NULL default 0, + PRIMARY KEY (IDDateTime,ID), + FULLTEXT KEY FULLTEXTIDX (text) +) TYPE=MyISAM; + +-- +-- Load data for table `%s` +-- +""" +tokenInsert = """INSERT INTO %s VALUES (%s);\n""" + +dataHdr = """ +-- Automatically generated table structure for metric data table `%s` +-- + +DROP TABLE IF EXISTS %s; + +CREATE TABLE %s ( + IDDateTime datetime NOT NULL default '0000-00-00 00:00:00', + ID int(10) unsigned NOT NULL auto_increment, + libraryName varchar(32) default '', + metricName varchar(32) NOT NULL default '', + srcFileName varchar(255) NOT NULL default '', + varName varchar(255) NOT NULL default '', + value decimal(15,5) NOT NULL default '0', + PRIMARY KEY (IDDateTime,ID) +) TYPE=MyISAM; + + +-- +-- Load metric data for table `%s` +-- +""" + +dataInsert = """INSERT INTO %s VALUES (%s);\n""" + diff -uNr a/pymetrics-0.7.9/PyMetrics/sqltokenout.py b/pymetrics-0.7.9/PyMetrics/sqltokenout.py --- a/pymetrics-0.7.9/PyMetrics/sqltokenout.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/sqltokenout.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,92 @@ +""" SqlTokenOut - class to produce SQL Token command file output. + + $Id: SqlTokenOut.py,v 1.2 2005/02/15 07:08:58 rcharney Exp $ +""" +__revision__ = "$Revision: 1.2 $"[11:-2] +__author__ = 'Reg. Charney ' + +import sys +import time +import token +import tokenize +from utils import * + +import string +import sqltemplate + +class InvalidTableNameError( Exception ): pass + +class SqlTokenOut( object ): + """ Class used to generate a command file suitable for runnning against + any SQL dbms.""" + def __init__( self, fd, libName, fileName, tableName, genNewSw=False, genExistsSw=False ): + if tableName == '': + raise InvalidTableNameError( tableName ) + + self.libName = libName + self.fileName = fileName + self.tableName = tableName + self.quotedFileName = '"'+self.fileName+'"' + self.IDDateTime = '"'+time.strftime("%Y-%m-%d %H:%M:%S",time.localtime())+'"' + self.toknum = 0 + self.fd = fd + + if not genExistsSw: + self.writeHdr( genNewSw, tableName ) + + def writeHdr( self, genNewSw, tableName ): + """ Write header information for creating SQL command file.""" + if genNewSw: + self.fd.write( sqltemplate.tokenHdr % (tableName,tableName,tableName,tableName) ) + + def write( self, context, tok, fqnFunction, fqnClass ): + """ Generate the Sql INSERT line into the sql command file.""" + self.toknum += 1 + txt = tok.text + tt = tok.type + tn = token.tok_name[tt] + + if tt == token.NEWLINE or tt == tokenize.NL: + txt = r'\n' + sn = self.__formSemanticName(tok) + sArgs = self.__formArgString(context, tok, tn, sn, txt, fqnFunction, fqnClass) + sOut = sqltemplate.tokenInsert % (self.tableName, sArgs) + self.fd.write( sOut ) + + def __formSemanticName(self, tok): + """ Form semantic name by decoding semtype.""" + sn = '' + if tok.semtype: + sn = token.tok_name[tok.semtype] + return sn + + def __formArgString(self, context, tok, tn, sn, txt, fqnFunction, fqnClass): + """ Generate arguments string for use in write method.""" + sArgs = ','.join( ( + self.IDDateTime, + str( self.toknum ), + '"'+str( self.libName )+'"', + '"'+str( context['inFile'] )+'"', + str( tok.row ), + str( tok.col ), + '"'+tn+'"', + '"'+sn+'"', + str( len( txt ) ), + sqlQ( txt ), + '"'+str( toTypeName( context, fqnFunction ) )+'"', + '"'+str( toTypeName( context, fqnClass ) )+'"', + str( context['blockNum'] ), + str( context['blockDepth'] ), + str( context['fcnDepth'] ), + str( context['classDepth'] ), + str( context['parenDepth'] ), + str( context['bracketDepth'] ), + str( context['braceDepth'] ) + ) ) + return sArgs + + def close( self ): + """ Close file, if it is opened.""" + self.fd and self.fd.close() + self.fd = None + diff -uNr a/pymetrics-0.7.9/PyMetrics/tokenize.py b/pymetrics-0.7.9/PyMetrics/tokenize.py --- a/pymetrics-0.7.9/PyMetrics/tokenize.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/tokenize.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,304 @@ +"""Tokenization help for Python programs. + +generate_tokens(readline) is a generator that breaks a stream of +text into Python tokens. It accepts a readline-like method which is called +repeatedly to get the next line of input (or "" for EOF). It generates +5-tuples with these members: + + the token type (see token.py) + the token (a string) + the starting (row, column) indices of the token (a 2-tuple of ints) + the ending (row, column) indices of the token (a 2-tuple of ints) + the original line (string) + +It is designed to match the working of the Python tokenizer exactly, except +that it produces COMMENT tokens for comments and gives type OP for all +operators + +Older entry points + tokenize_loop(readline, tokeneater) + tokenize(readline, tokeneater=printtoken) +are the same, except instead of generating tokens, tokeneater is a callback +function to which the 5 fields described above are passed as 5 arguments, +each time a new token is found. + + $Id: tokenize.py,v 1.3 2005/02/15 07:08:58 rcharney Exp $ +""" +__version__ = "$Revision: 1.3 $"[11:-2] +__author__ = 'Ka-Ping Yee ' +__credits__ = \ + 'GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip Montanaro' + +import string, re +from token import * + +import token +__all__ = [x for x in dir(token) if x[0] != '_'] + ["COMMENT", "tokenize", + "generate_tokens", "NL"] +del x +del token + +COMMENT = N_TOKENS +tok_name[COMMENT] = 'COMMENT' +NL = N_TOKENS + 1 +tok_name[NL] = 'NL' +N_TOKENS += 2 + +def group(*choices): return '(' + '|'.join(choices) + ')' +def any(*choices): return group(*choices) + '*' +def maybe(*choices): return group(*choices) + '?' + +Whitespace = r'[ \f\t]*' +Comment = r'#[^\r\n]*' +Ignore = Whitespace + any(r'\\\r?\n' + Whitespace) + maybe(Comment) +Name = r'[a-zA-Z_]\w*' + +Hexnumber = r'0[xX][\da-fA-F]*[lL]?' +Octnumber = r'0[0-7]*[lL]?' +Decnumber = r'[1-9]\d*[lL]?' +Intnumber = group(Hexnumber, Octnumber, Decnumber) +Exponent = r'[eE][-+]?\d+' +Pointfloat = group(r'\d+\.\d*', r'\.\d+') + maybe(Exponent) +Expfloat = r'\d+' + Exponent +Floatnumber = group(Pointfloat, Expfloat) +Imagnumber = group(r'\d+[jJ]', Floatnumber + r'[jJ]') +Number = group(Imagnumber, Floatnumber, Intnumber) + +# Tail end of ' string. +Single = r"[^'\\]*(?:\\.[^'\\]*)*'" +# Tail end of " string. +Double = r'[^"\\]*(?:\\.[^"\\]*)*"' +# Tail end of ''' string. +Single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''" +# Tail end of """ string. +Double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""' +Triple = group("[uU]?[rR]?'''", '[uU]?[rR]?"""') +# Single-line ' or " string. +String = group(r"[uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'", + r'[uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"') + +# Because of leftmost-then-longest match semantics, be sure to put the +# longest operators first (e.g., if = came before ==, == would get +# recognized as two instances of =). +Operator = group(r"\*\*=?", r">>=?", r"<<=?", r"<>", r"!=", + r"//=?", + r"[+\-*/%&|^=<>]=?", + r"~") + +Bracket = '[][(){}]' +Special = group(r'\r?\n', r'[:;.,`]','@') +Funny = group(Operator, Bracket, Special) + +PlainToken = group(Number, Funny, String, Name) +Token = Ignore + PlainToken + +# First (or only) line of ' or " string. +ContStr = group(r"[uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" + + group("'", r'\\\r?\n'), + r'[uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' + + group('"', r'\\\r?\n')) +PseudoExtras = group(r'\\\r?\n', Comment, Triple) +PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name) + +tokenprog, pseudoprog, single3prog, double3prog = map( + re.compile, (Token, PseudoToken, Single3, Double3)) +endprogs = {"'": re.compile(Single), '"': re.compile(Double), + "'''": single3prog, '"""': double3prog, + "r'''": single3prog, 'r"""': double3prog, + "u'''": single3prog, 'u"""': double3prog, + "ur'''": single3prog, 'ur"""': double3prog, + "R'''": single3prog, 'R"""': double3prog, + "U'''": single3prog, 'U"""': double3prog, + "uR'''": single3prog, 'uR"""': double3prog, + "Ur'''": single3prog, 'Ur"""': double3prog, + "UR'''": single3prog, 'UR"""': double3prog, + 'r': None, 'R': None, 'u': None, 'U': None} + +triple_quoted = {} +for t in ("'''", '"""', + "r'''", 'r"""', "R'''", 'R"""', + "u'''", 'u"""', "U'''", 'U"""', + "ur'''", 'ur"""', "Ur'''", 'Ur"""', + "uR'''", 'uR"""', "UR'''", 'UR"""'): + triple_quoted[t] = t +single_quoted = {} +for t in ("'", '"', + "r'", 'r"', "R'", 'R"', + "u'", 'u"', "U'", 'U"', + "ur'", 'ur"', "Ur'", 'Ur"', + "uR'", 'uR"', "UR'", 'UR"' ): + single_quoted[t] = t + +tabsize = 8 + +class TokenError(Exception): pass + +class StopTokenizing(Exception): pass + +def printtoken(type, token, (srow, scol), (erow, ecol), line): # for testing + print "%d,%d-%d,%d:\t%s\t%s" % \ + (srow, scol, erow, ecol, tok_name[type], repr(token)) + +def tokenize(readline, tokeneater=printtoken): + """ + The tokenize() function accepts two parameters: one representing the + input stream, and one providing an output mechanism for tokenize(). + + The first parameter, readline, must be a callable object which provides + the same interface as the readline() method of built-in file objects. + Each call to the function should return one line of input as a string. + + The second parameter, tokeneater, must also be a callable object. It is + called once for each token, with five arguments, corresponding to the + tuples generated by generate_tokens(). + """ + try: + tokenize_loop(readline, tokeneater) + except StopTokenizing: + pass + +# backwards compatible interface +def tokenize_loop(readline, tokeneater): + for token_info in generate_tokens(readline): + tokeneater(*token_info) + +def generate_tokens(readline): + """ + The generate_tokens() generator requires one argment, readline, which + must be a callable object which provides the same interface as the + readline() method of built-in file objects. Each call to the function + should return one line of input as a string. + + The generator produces 5-tuples with these members: the token type; the + token string; a 2-tuple (srow, scol) of ints specifying the row and + column where the token begins in the source; a 2-tuple (erow, ecol) of + ints specifying the row and column where the token ends in the source; + and the line on which the token was found. The line passed is the + logical line; continuation lines are included. + """ + lnum = parenlev = continued = 0 + namechars, numchars = string.ascii_letters + '_', '0123456789' + contstr, needcont = '', 0 + contline = None + indents = [0] + strstart = (0,0) + endprog = re.compile('') + + while 1: # loop over lines in stream + line = readline() + lnum = lnum + 1 + pos, maxLen = 0, len(line) + + if contstr: # continued string + if not line: + raise TokenError, ("EOF in multi-line string", strstart) + endmatch = endprog.match(line) + if endmatch: + pos = end = endmatch.end(0) + yield (STRING, contstr + line[:end], + strstart, (lnum, end), contline + line) + contstr, needcont = '', 0 + contline = None + elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n': + yield (ERRORTOKEN, contstr + line, + strstart, (lnum, len(line)), contline) + contstr = '' + contline = None + continue + else: + contstr = contstr + line + contline = contline + line + continue + + elif parenlev == 0 and not continued: # new statement + if not line: break + column = 0 + while pos < maxLen: # measure leading whitespace + if line[pos] == ' ': column = column + 1 + elif line[pos] == '\t': column = (column/tabsize + 1)*tabsize + elif line[pos] == '\f': column = 0 + else: break + pos = pos + 1 + if pos == maxLen: break + + if line[pos] in '\r\n': # this is empty line + yield (NL, line[pos:], + (lnum, pos), (lnum, len(line)), line) + elif line[pos] == '#': # this is start of comment + pass # skip any effect on indentation + else: + if column > indents[-1]: # count indents or dedents + indents.append(column) + yield (INDENT, line[:pos], (lnum, 0), (lnum, pos), line) + while column < indents[-1]: + indents = indents[:-1] + yield (DEDENT, '', (lnum, pos), (lnum, pos), line) + + else: # continued statement + if not line: + raise TokenError, ("EOF in multi-line statement", (lnum, 0)) + continued = 0 + + while pos < maxLen: + pseudomatch = pseudoprog.match(line, pos) + if pseudomatch: # scan for tokens + start, end = pseudomatch.span(1) + spos, epos, pos = (lnum, start), (lnum, end), end + token, initial = line[start:end], line[start] + + if initial in numchars or \ + (initial == '.' and token != '.'): # ordinary number + yield (NUMBER, token, spos, epos, line) + elif initial in '\r\n': + yield (parenlev > 0 and NL or NEWLINE, + token, spos, epos, line) + elif initial == '#': + yield (COMMENT, token, spos, epos, line) + elif token in triple_quoted: + endprog = endprogs[token] + endmatch = endprog.match(line, pos) + if endmatch: # all on one line + pos = endmatch.end(0) + token = line[start:pos] + yield (STRING, token, spos, (lnum, pos), line) + else: + strstart = (lnum, start) # multiple lines + contstr = line[start:] + contline = line + break + elif initial in single_quoted or \ + token[:2] in single_quoted or \ + token[:3] in single_quoted: + if token[-1] == '\n': # continued string + strstart = (lnum, start) + endprog = (endprogs[initial] or endprogs[token[1]] or + endprogs[token[2]]) + contstr, needcont = line[start:], 1 + contline = line + break + else: # ordinary string + yield (STRING, token, spos, epos, line) + elif initial in namechars: # ordinary name + yield (NAME, token, spos, epos, line) + elif initial == '\\': # continued stmt + continued = 1 + else: + if initial in '([{': parenlev = parenlev + 1 + elif initial in ')]}': parenlev = parenlev - 1 + yield (OP, token, spos, epos, line) + else: + #yield (WS, line[pos], (lnum, pos), (lnum, pos+1), line) + while (pos < maxLen) and (line[pos] in ' \t'): + pos += 1 + if pos < maxLen: + yield (ERRORTOKEN, line[pos], (lnum, pos), (lnum, pos+1), line) + pos += 1 + + for indent in indents[1:]: # pop remaining indent levels + yield (DEDENT, '', (lnum, 0), (lnum, 0), '') + yield (ENDMARKER, '', (lnum, 0), (lnum, 0), '') + +if __name__ == '__main__': # testing + import sys + if len(sys.argv) > 1: tokenize(open(sys.argv[1]).readline) + else: tokenize(sys.stdin.readline) diff -uNr a/pymetrics-0.7.9/PyMetrics/utils.py b/pymetrics-0.7.9/PyMetrics/utils.py --- a/pymetrics-0.7.9/PyMetrics/utils.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/utils.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,62 @@ +""" Utility functions used throughout the PyMetrics system. + + $Id: utils.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ +""" + +import sys +import token +import re + +def sqlQ( s ): + """ Place single quotes around strings and escaping existing single quotes.""" + a = s.replace( "\\","\\\\" ) + a = a.replace( "'", "\\'" ) + a = a.replace( '"', '\\"' ) + return '"'+a+'"' + +def csvQ( s ): + """ Quote a string using rules for CSV data.""" + a = s.replace("\\","\\\\") + b = a.replace( "'", "\\'" ) + c = b.replace( "\n", "\\n" ) + d = c.replace( '"', '""' ) + return '"'+d+'"' + +def toTypeName( context, lst ): + """ Convert token type numbers to names.""" + lstOut = [] + for name,blockDepth,semtype in lst: + try: + semName = token.tok_name[semtype] + lstOut.append( (name,blockDepth,semName) ) + except KeyError, e: + raise KeyError( "Unknown value '"+str( e )+"' for token/semantic type in context %s\n" % context ) + + return lstOut + +if 0: + def mainTest(): + """ Built-in tests """ + def check( qs, s ): + print "<%s>==<%s>" % (s.__repr__(),qs.__repr__()) + print "[%s]==[%s]" % (s,qs) + try: + assert( s.__repr__() == qs.__repr__() ) + assert( s, qs ) + except AssertionError: + print "Failed" + + s0 = ''; qs0 = sqlQ(s0) + check( qs0, '""' ) + s1 = 'aName'; qs1 = sqlQ(s1) + check( qs1, '"aName"' ) + s2 = 'A literal with a double quote (\") in it'; qs2 = sqlQ( s2 ) + check( qs2, '"A literal with a double quote (\\\") in it"' ) + s3 = '\'A literal with a single quote (\') in it\''; qs3 = sqlQ( s3 ) + check( qs3, '"\\\'A literal with a single quote (\\\') in it\\\'"' ) + s4 = """A multi- + line literal."""; qs4 = sqlQ( s4 ) + check( qs4, '"A multi-\nline literal."' ) + + if __name__ == "__main__": + mainTest() diff -uNr a/pymetrics-0.7.9/PyMetrics/version.py b/pymetrics-0.7.9/PyMetrics/version.py --- a/pymetrics-0.7.9/PyMetrics/version.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/PyMetrics/version.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,24 @@ +""" PyMetrics version information. + + Copyright (c) 2005 by Reg. Charney + All rights reserved, see LICENSE for details. + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + $Id: version.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ +""" +__version__ = "$Revision: 1.3 $"[11:-2] +__author__ = 'Reg. Charney ' + +project = "pymetrics" +revision = '$Revision: 1.3 $'[11:-2] +release = '0.0' + diff -uNr a/pymetrics-0.7.9/PyMetrics.py b/pymetrics-0.7.9/PyMetrics.py --- a/pymetrics-0.7.9/PyMetrics.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/PyMetrics.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,280 +0,0 @@ -#! /usr/bin/env python -""" PyMetrics - Complexity Measurements for Python code. - - Orignally based on grop.py by Jurgen Hermann. - Modified by Reg. Charney to do Python complexity measurements. - - Copyright (c) 2001 by Jurgen Hermann - Copyright (c) 2007 by Reg. Charney - - All rights reserved, see LICENSE for details. - - This program is free software; you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - $Id$ -""" -__revision__ = "$Id$" -__author__ = 'Reg. Charney ' - -# Imports -import sys -import string -from processargs import ProcessArgs, ProcessArgsError -import token -import sqltokenout -import sqldataout -import csvout -from lexer import Lexer -from compute import ComputeMetrics -from globals import * - -############################################################################# -### Main script for PyMetrics utility. -############################################################################# - -def __importMetricModules( includeMetrics ): - """ Import the modules specified in the parameter list. - - includeMetrics is a list of (metricModuleName, metricClassName) - pairs. This function defines a dictionary containing only valid - module/class names. When an error is found, the invalid - module/class pair is removed from the included list of metrics. - """ - i = 0 - metricModules = {} - for m,n in includeMetrics: - try: - mod = __import__( m ) - metricModules[m] = mod - i += 1 - except ImportError: - sys.stderr.write( "Unable to import metric module %s -- ignored.\n\n" % m ) - # remove the erroneous metric module/class tuple - del includeMetrics[i] - - return metricModules - -def __instantiateMetric( metricModules, context, runMetrics, metrics, pa ): - """ Instantiate all user specified metric classes. - - The code works by finding the desired metric class in a metric module and - instantiating the class. It does this by assuming that the metric - class is in the dictionary of the metric module. - """ - metricInstance = {} - inclIndx = -1 - for m,n in pa.includeMetrics: - inclIndx += 1 - try: - metricInstance[m] = None # default value if metric class does not exist. - metricInstance[m] = metricModules[m].__dict__[n]( context, runMetrics, metrics, pa ) - except KeyError: - sys.stderr.write( "Module %s does not contain metric class %s -- metric %s ignored.\n\n" % (m,n,m) ) - del( metricInstance[m] ) - del( pa.includeMetrics[inclIndx] ) - - return metricInstance - -def __stats( so, m, inFileName, label, *args ): - """ Print line of statistics.""" - result = string.join(map(str, args), '') - print "%11s %s" % (result, label) - so and so.write( m, inFileName, label, result ) - -def __printSummary( so, context, runMetrics, metrics, pa ): - """ Print basic summary information.""" - # the following loop is a very, very ugly hack to distinguish between - # tokens as they appear in the source; semantically generated tokens, - # like DOCSTRING; and NONTOKENs, like numComments - - keys = [] - for k in metrics.keys(): - if str( k ).isdigit(): - keys.append( (token.tok_name[k],k,metrics[k]) ) - elif len( str( k ).split() ) > 1: - keys.append( (k,SEMTOKEN,metrics[k]) ) - else: - keys.append( (k,NONTOKEN,metrics[k]) ) - keys.sort() - - inFileName = context['inFile'] - if pa.genKwCntSw: - hdr = "Counts of Token Types in module %s" % context['inFile'] - print - print hdr - print "-"*len(hdr) - print - for k,t,v in keys: - if (pa.zeroSw or v): - if t != NONTOKEN: - __stats( so, 'basic', inFileName, k, v ) - print - - if pa.genBasicSw: - __displayBasicMetrics( keys, pa, so, inFileName ) - -def __displayBasicMetrics( keys, pa, so, inFileName ): - """ Display the Basic metrics that PyMetrics computes.""" - hdr = "Basic Metrics for module %s" % inFileName - print - print hdr - print "-"*len( hdr ) - print - for k,t,v in keys: - if t==NONTOKEN: - if pa.zeroSw or not v in ([],{},(),0,0.00): - __stats( so, 'basic', inFileName, k, v ) - print - -def main(): - """ Main routine for PyMetrics.""" - # process command line args - try: - pa = ProcessArgs() - except ProcessArgsError, e: - sys.stderr.writelines( str(e) ) - return - - if pa.genNewSw: - __deleteOldOutputFiles( pa ) - - so, od = __genNewSqlCmdFiles( pa ) - co = __genNewCsvFile( pa ) - - # import all the needed metric modules - metricModules = __importMetricModules( pa.includeMetrics ) - - runMetrics = {} # metrics for whole run - metrics = {} # metrics for this module - context = {} # context in which token was used - - # main loop - where all the work is done - for inFileName in pa.inFileNames: - - metrics.clear() - context.clear() - context['inFile'] = inFileName - - # instantiate all the desired metric classes - metricInstance = __instantiateMetric( metricModules, context, runMetrics, metrics, pa ) - - cm = ComputeMetrics( metricInstance, context, runMetrics, metrics, pa, so, co ) - - # define lexographical scanner to use for this run - # later, this may vary with file and language. - lex = Lexer() - - if not pa.quietSw: - print "=== File: %s ===" % inFileName - - try: - lex.parse( inFileName ) # parse input file - - metrics["numCharacters"] = len(lex.srcLines) - metrics["numLines"] = lex.lineCount # lines of code - - metrics = cm( lex ) - - # if printing desired, output summary and desired metrics - # also, note that this preserves the order of the metrics desired - if not pa.quietSw: - __printSummary( od, context, runMetrics, metrics, pa ) - for m,n in pa.includeMetrics: - if metricInstance[m]: - result = metricInstance[m].display() - if metrics.has_key(m): - metrics[m].append( result ) - else: - metrics[m] = result - for r in result.keys(): - od and od.write( m, inFileName, r, result[r] ) - except IOError, e: - sys.stderr.writelines( str(e) + " -- Skipping input file.\n\n") - - co and co.close() - - result = {} - if len( pa.inFileNames ) > 0: - for m,n in pa.includeMetrics: - if metricInstance[m]: - result = metricInstance[m].processRun( None ) - if result: - for r in result.keys(): - od and od.write( m, None, r, result[r] ) - od and od.close() - - if not pa.quietSw: - n = len( pa.inFileNames ) - print - print "*** Processed %s module%s in run ***" % (n,(n>1) and 's' or '') - -def __genNewCsvFile( pa ): - """ Determine output CSV data file, if any, - and check it can be created.""" - co = None - try: - if pa.genCsvSw: - co = csvout.CsvOut( pa.csvFileName, genHdrSw=pa.genHdrSw, genNewSw=pa.genNewSw ) - except StandardError, e: - # this should not occur - it should be handled in processArgs. - sys.stderr.writelines( str(e) + " -- No CSV file will be generated\n\n" ) - pa.genCsvSw = False - - return co - -def __genNewSqlCmdFiles( pa ): - """ determine output SQL tokens command file, if any, - and check it can be created .""" - so = None - co = None - od = None - fd = None - - try: - if pa.genSqlSw: - if pa.sqlFileName: - fd = open( pa.sqlFileName, 'a' ) - else: - fd = sys.stdout - so = sqltokenout.SqlTokenOut( fd, pa.libName, pa.sqlFileName, pa.sqlTokenTableName, pa.genNewSw, pa.genExistsSw ) - od = sqldataout.SqlDataOut( fd, pa.libName, pa.sqlFileName, pa.sqlMetricsTableName, pa.genNewSw, pa.genExistsSw ) - except StandardError, e: - # this should not occur - it should be handled in processArgs. - sys.stderr.writelines( str(e) + " -- No SQL command file will be generated\n\n" ) - pa.genSqlSw = False - so and so.close() - od and od.close() - so = None - od = None - - return so, od - -def __deleteOldOutputFiles( pa ): - """ Generate new output files by ensuring old files deleted.""" - import os - try: - if pa.genSqlSw and os.path.exists( pa.sqlFileName ): - os.remove( pa.sqlFileName ) - except IOError, e: - sys.stderr.writelines( str(e) ) - pa.genSqlSw = False - - try: - if pa.genCsvSw and os.path.exists( pa.csvFileName ): - os.remove( pa.csvFileName ) - except IOError, e: - sys.stderr.writelines( str(e) ) - pa.genCsvSw = False - return - -if __name__ == "__main__": - main() - sys.exit(0) diff -uNr a/pymetrics-0.7.9/setup.py b/pymetrics-0.7.9/setup.py --- a/pymetrics-0.7.9/setup.py 1970-01-01 01:00:00.000000000 +0100 +++ b/pymetrics-0.7.9/setup.py 2008-07-23 12:33:13.000000000 +0200 @@ -0,0 +1,13 @@ +#!/usr/bin/python + +from distutils.core import setup + +setup (name = "PyMetrics", + version = "0.7.9", + author = "Reg. Charney", + author_email = "pymetrics@charneyday.com", + description = "PyMetrics produces metrics for Python programs", + url = "http://sourceforge.net/projects/pymetrics/", + packages = ['PyMetrics'], + scripts = ['pymetrics'] +) diff -uNr a/pymetrics-0.7.9/simple.py b/pymetrics-0.7.9/simple.py --- a/pymetrics-0.7.9/simple.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/simple.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,121 +0,0 @@ -""" Simple Metrics for each function within a file. - - $Id: simple.py,v 1.4 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.4 $"[11:-2] -__author__ = 'Reg. Charney ' - -from metricbase import MetricBase -from globals import * - -class SimpleMetric( MetricBase ): - """ Compute simple metrics by function.""" - def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): - self.context = context - self.runMetrics = runMetrics - self.metrics = metrics - self.pa = pa - self.inFile = context['inFile'] - self.simpleMetrics = {} - self.mcc = {} - self.fcnNames = {} - self.classNames = {} - self.lastFcnName = None - self.lastClassName = None - self.metrics['DocFunctions'] = [] - self.metrics['DocClasses'] = [] - - def processToken( self, currentFcn, currentClass, tok, *args, **kwds ): - """ Collect token and context sensitive data for simple metrics.""" - if tok.semtype == DOCSTRING: - self.metrics['numDocStrings'] = self.metrics.get('numDocStrings',0) + 1 - if currentFcn and currentFcn != "__main__": - if self.lastFcnName != currentFcn: - self.lastFcnName = currentFcn - self.metrics['numFcnDocStrings'] = self.metrics.get('numFcnDocStrings',0) + 1 - self.fcnNames[currentFcn] = tok.row - elif currentClass == None: # this doc string must be module doc string - self.lastFcnName = "__main__" - self.metrics['numFcnDocStrings'] = self.metrics.get('numFcnDocStrings',0) + 1 - self.fcnNames[self.lastFcnName] = tok.row - if currentClass: - if self.lastClassName != currentClass: - self.lastClassName = currentClass - self.metrics['numClassDocStrings'] = self.metrics.get('numClassDocStrings',0) + 1 - self.classNames[currentClass] = tok.row - elif tok.semtype == FCNNAME: - self.fcnNames[currentFcn] = 0 - elif tok.semtype == CLASSNAME: - self.classNames[currentClass] = 0 - - def processBlock( self, currentFcn, currentClass, block, *args, **kwds ): - """ Collect token and context sensitive data for simple metrics.""" - self.metrics['numBlocks'] = self.metrics.get('numBlocks',0)+1 - - def compute( self, *args, **kwds ): - """ Compute any values needed.""" - if self.metrics.get('numModuleDocStrings', 0) > 0: - self.metrics['numFunctions'] = self.metrics.get('numFunctions', 0) + 1 - - try: - self.simpleMetrics['%Comments'] = 100.0 * self.metrics['numComments']/self.metrics['numLines'] - except (KeyError, ZeroDivisionError): - self.simpleMetrics['%Comments'] = 0.0 - - try: - self.simpleMetrics['%CommentsInline'] = 100.0 * self.metrics['numCommentsInline']/self.metrics['numLines'] - except (KeyError, ZeroDivisionError): - self.simpleMetrics['%CommentsInline'] = 0.0 - - if 0: - try: - self.simpleMetrics['%DocStrings'] = 100.0 * self.metrics['numDocStrings']/(self.metrics['numModuleDocStrings']+self.metrics['numClasses']+self.metrics['numFunctions']) - except (KeyError, ZeroDivisionError): - self.simpleMetrics['%DocStrings'] = 0.0 - - try: - self.simpleMetrics['%FunctionsHavingDocStrings'] = 100.0 * self.metrics['numFcnDocStrings']/self.metrics['numFunctions'] - except (KeyError, ZeroDivisionError): - self.simpleMetrics['%FunctionsHavingDocStrings'] = 0.0 - - try: - self.simpleMetrics['%ClassesHavingDocStrings'] = 100.0 * self.metrics['numClassDocStrings']/self.metrics['numClasses'] - except (KeyError, ZeroDivisionError): - self.simpleMetrics['%ClassesHavingDocStrings'] = 0.0 - - return self.simpleMetrics - - def display( self, currentFcn=None ): - """ Display and return simple metrics for given function.""" - - def __printNames( typeName, names ): - """ Pretty print list of functions/classes that have doc strings.""" - if len( names ): # only output something if it exists - hdr = "%s DocString present(+) or missing(-)" % typeName - print - print hdr - print "-"*len(hdr) + "\n" - result = [] - keys = names.keys() - keys.sort() - for k in keys: - if k: - pfx = (names[k] and '+') or '-' - print "%c %s" % (pfx,k) - print - - result = (self.inFile, names) - return result - - self.compute() - keyList = self.simpleMetrics.keys() - keyList.sort() - for k in keyList: - if self.pa.zeroSw or self.simpleMetrics[k]: - fmt = ( k[0] == '%' and "%14.2f %s" ) or "%11d %s" - print fmt % (self.simpleMetrics[k],k) - - self.metrics['DocFunctions'].append( __printNames( 'Functions', self.fcnNames ) ) - self.metrics['DocClasses'].append( __printNames( 'Classes', self.classNames ) ) - - return self.simpleMetrics diff -uNr a/pymetrics-0.7.9/sloc.py b/pymetrics-0.7.9/sloc.py --- a/pymetrics-0.7.9/sloc.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/sloc.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,151 +0,0 @@ -""" Compute COCOMO 2's SLOC (Source Lines Of Code) metric. - - Compute Source Lines Of Code for each function/class/file. - This is based on COCOMO 2's definition of constitutes a - line of code. - - Algorithm: - Delete all non-literal blank lines - Delete all non-literal comments - Delete all doc strings - Combine parenthesised expressions into one logical line - Combine continued lines into one logical line - Return count of resulting lines - - Conventions: - Continued lines are those ending in \) are treated as one - physical line. - Paremeter lists and expressions enclosed in parens (), - braces {}, or brackets [] are treated as being part - of one physical line. - All literals are treated as being part of one physical - line - - $Id: sloc.py,v 1.2 2005/06/26 07:08:58 rcharney Exp $ -""" -__revision__ = "$Revision: 1.1 $"[11:-2] -__author__ = 'Reg. Charney ' - -import re - -from metricbase import MetricBase - -class SLOCMetric( MetricBase ): - """ Compute Source Lines Of Code metric.""" - def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): - # pattern definitions - patBackslashQuote = r'\\\'|\\\"' - patTripleQuote1 = r"r?'''.*?'''" - patTripleQuote2 = r'r?""".*?"""' - patQuote1 = r"'.*?'" - patQuote2 = r'".*?"' - self.patLiteral = patTripleQuote1 + '|' + patQuote1 + '|' + patTripleQuote2 + '|' + patQuote2 - self.patComment = '#.*\n' - self.patParen = '\(.*?\)' - self.patDefOrClass = '^\s*(def|class)\s+\w.*:' - self.patAllBlank = '^\s*\n|^\s*@\n|^\s*~\n' - # compile patterns - self.reBackslashQuote = re.compile( patBackslashQuote ) - self.reLiteral = re.compile( self.patLiteral, re.M + re.S ) - self.reComment = re.compile( self.patComment ) - self.reParen = re.compile( self.patParen, re.M + re.S ) - self.reDefOrClass = re.compile( self.patDefOrClass, re.M + re.S ) - self.reAllBlank = re.compile( self.patAllBlank, re.M + re.S ) - - self.numSLOC = 0 - self.context = context - self.runMetrics = runMetrics - self.metrics = metrics - self.pa = pa - self.inFile = context['inFile'] - self.fcnNames = {} - - def processSrcLines( self, rawLines, *args, **kwds ): - """ Process all raw source lines in one fell swoop. - If a given line is not blank and contains something - besides a comment, then increment the numSLOC.""" - if len( rawLines ) == 0: - # ignore file - self.numSLOC = 0 - return - - noBackslashQuotes = re.sub( self.reBackslashQuote, '*', rawLines ) - noLiteralLines = re.sub( self.reLiteral, '@', noBackslashQuotes ) - noCommentLines = re.sub( self.reComment, '~\n', noLiteralLines ) - noBlankLines = re.sub( self.reAllBlank, '%', noCommentLines ) - noParenLines = re.sub( self.reParen, '(...)', noBlankLines ) - self.numSLOC = noParenLines.count( '\n' ) - return - - def display( self ): - """ Display SLOC metric for each file """ - result = {} - result[self.inFile] = self.numSLOC - - if self.pa.quietSw: - return result - - hdr = "\nCOCOMO 2's SLOC Metric for %s" % self.inFile - print hdr - print "-"*len(hdr) + "\n" - print "%11d %s" % (self.numSLOC,self.inFile) - print - - return result - -def __main( fn, debugSw ): - """ Process lines from input file.""" - fd = open( fn ) - rawLines = fd.read() - fd.close() - - class PA: pass - pa = PA() - pa.quietSw = True - pa.debugSw = debugSw - - __processFile(fn, pa, rawLines) - -def __processFile(fn, pa, rawLines): - """ Invoke metric function after setting up scaffolding.""" - sloc = SLOCMetric( {'inFile':fn},[],pa ) - sloc.processSrcLines( rawLines ) - print sloc.display() - -def __getFNList(): - """ Return list of input file names, - regardless of whether the list came - from a file or the command line.""" - import sys - debugSw = False - fnList = [] - for arg in sys.argv[1:]: - if arg == '-d': - debugSw = True - continue - if arg == '-f': - fd = open(sys.argv[2]) - lines = fd.read() - fd.close() - fnList = __normalizeLines( lines ) - continue - fnList.append( arg ) - return fnList,debugSw - -def __normalizeLines( lines ): - """ Remove trailing newlines, if needed, - and return list of input file names.""" - fnList = [] - if len( lines ) > 0: - if lines[-1] == '\n': - lines = lines[:-1] - fnList = lines.split( '\n' ) - return fnList - -if __name__ == "__main__": - fnList,debugSw = __getFNList() - for fn in fnList: - try: - fn!='' and __main( fn, debugSw ) - except: - pass diff -uNr a/pymetrics-0.7.9/sqldataout.py b/pymetrics-0.7.9/sqldataout.py --- a/pymetrics-0.7.9/sqldataout.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/sqldataout.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,78 +0,0 @@ -""" SqlDataOut - class to produce SQL data command file output. - - $Id: SqlDataOut.py,v 1.2 2005/02/15 07:08:58 rcharney Exp $ -""" -__revision__ = "$Revision: 1.2 $"[11:-2] -__author__ = 'Reg. Charney ' - -import sys -import time -import token -import tokenize -import utils - -import sqltemplate - -class InvalidTableNameError( Exception ): - """ Used to indicate that the SQL table name is invalid.""" - pass - -class SqlDataOut( object ): - """ Class used to generate a command file suitable for runnning against - any SQL dbms.""" - def __init__( self, - fd, - libName, - fileName, - tableName, - genNewSw=False, - genExistsSw=False ): - """ Initialize instance of SqlDataOut.""" - if tableName == '': - raise InvalidTableNameError( tableName ) - if not fd: - raise IOError( "Output file does not yet exist" ) - - timestamp = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()) - self.libName = libName - self.fileName = fileName - self.tableName = tableName - self.quotedFileName = '"'+self.fileName+'"' - self.IDDateTime = '"'+timestamp+'"' - self.toknum = 0 - self.fd = fd - - if not genExistsSw: - self.writeHdr( genNewSw, tableName ) - - def writeHdr( self, genNewSw, tableName ): - """ Write header information for creating SQL command file.""" - if genNewSw: - import re - r = re.compile( '\w+' ) - if r.match( tableName ): - self.fd.write( - sqltemplate.dataHdr % - (tableName, tableName, tableName, tableName) - ) - else: - raise AttributeError( 'Invalid table name' ) - - def write( self, metricName, srcFileName, varName, value ): - """ Generate the Sql INSERT line into the sql command file.""" - sArgs = ','.join( ( - self.IDDateTime, - '0', - '"'+str( self.libName )+'"', - '"'+str( metricName )+'"', - '"'+str( srcFileName )+'"', - '"'+str( varName )+'"', - '"'+str( value )+'"' - ) ) - sOut = sqltemplate.dataInsert % (self.tableName, sArgs) - self.fd and self.fd.write( sOut ) - - def close( self ): - """ Close file, if it is opened.""" - self.fd and self.fd.close() - self.fd = None diff -uNr a/pymetrics-0.7.9/sqltemplate.py b/pymetrics-0.7.9/sqltemplate.py --- a/pymetrics-0.7.9/sqltemplate.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/sqltemplate.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,66 +0,0 @@ -""" sqltemplate - template for generating sql token output. """ - -__revision__ = "$Revision: 1.2 $"[11:-2] -__author__ = 'Reg. Charney ' - -tokenHdr = """-- --- Automatically generated table structure for token table `%s` --- - -DROP TABLE IF EXISTS %s; - -CREATE TABLE %s ( - IDDateTime datetime NOT NULL default '0000-00-00 00:00:00', - ID int(11) unsigned NOT NULL auto_increment, - libraryName varchar(32) default '', - fileName varchar(255) default NULL, - lineNum int(11) NOT NULL, - colNum int(11) NOT NULL, - type varchar(16) NOT NULL default 'ERRORTOKEN', - semtype varchar(16) default NULL, - textLen int(11) NOT NULL default 1, - text varchar(255) NOT NULL default '', - fqnFunction varchar(255) default NULL, - fqnClass varchar(255) default NULL, - blockNum int(11) NOT NULL default 1, - blockDepth int(11) NOT NULL default 0, - fcnDepth int(11) NOT NULL default 0, - classDepth int(11) NOT NULL default 0, - parenDepth int(11) NOT NULL default 0, - bracketDepth int(11) NOT NULL default 0, - braceDepth int(11) NOT NULL default 0, - PRIMARY KEY (IDDateTime,ID), - FULLTEXT KEY FULLTEXTIDX (text) -) TYPE=MyISAM; - --- --- Load data for table `%s` --- -""" -tokenInsert = """INSERT INTO %s VALUES (%s);\n""" - -dataHdr = """ --- Automatically generated table structure for metric data table `%s` --- - -DROP TABLE IF EXISTS %s; - -CREATE TABLE %s ( - IDDateTime datetime NOT NULL default '0000-00-00 00:00:00', - ID int(10) unsigned NOT NULL auto_increment, - libraryName varchar(32) default '', - metricName varchar(32) NOT NULL default '', - srcFileName varchar(255) NOT NULL default '', - varName varchar(255) NOT NULL default '', - value decimal(15,5) NOT NULL default '0', - PRIMARY KEY (IDDateTime,ID) -) TYPE=MyISAM; - - --- --- Load metric data for table `%s` --- -""" - -dataInsert = """INSERT INTO %s VALUES (%s);\n""" - diff -uNr a/pymetrics-0.7.9/sqltokenout.py b/pymetrics-0.7.9/sqltokenout.py --- a/pymetrics-0.7.9/sqltokenout.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/sqltokenout.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,92 +0,0 @@ -""" SqlTokenOut - class to produce SQL Token command file output. - - $Id: SqlTokenOut.py,v 1.2 2005/02/15 07:08:58 rcharney Exp $ -""" -__revision__ = "$Revision: 1.2 $"[11:-2] -__author__ = 'Reg. Charney ' - -import sys -import time -import token -import tokenize -from utils import * - -import string -import sqltemplate - -class InvalidTableNameError( Exception ): pass - -class SqlTokenOut( object ): - """ Class used to generate a command file suitable for runnning against - any SQL dbms.""" - def __init__( self, fd, libName, fileName, tableName, genNewSw=False, genExistsSw=False ): - if tableName == '': - raise InvalidTableNameError( tableName ) - - self.libName = libName - self.fileName = fileName - self.tableName = tableName - self.quotedFileName = '"'+self.fileName+'"' - self.IDDateTime = '"'+time.strftime("%Y-%m-%d %H:%M:%S",time.localtime())+'"' - self.toknum = 0 - self.fd = fd - - if not genExistsSw: - self.writeHdr( genNewSw, tableName ) - - def writeHdr( self, genNewSw, tableName ): - """ Write header information for creating SQL command file.""" - if genNewSw: - self.fd.write( sqltemplate.tokenHdr % (tableName,tableName,tableName,tableName) ) - - def write( self, context, tok, fqnFunction, fqnClass ): - """ Generate the Sql INSERT line into the sql command file.""" - self.toknum += 1 - txt = tok.text - tt = tok.type - tn = token.tok_name[tt] - - if tt == token.NEWLINE or tt == tokenize.NL: - txt = r'\n' - sn = self.__formSemanticName(tok) - sArgs = self.__formArgString(context, tok, tn, sn, txt, fqnFunction, fqnClass) - sOut = sqltemplate.tokenInsert % (self.tableName, sArgs) - self.fd.write( sOut ) - - def __formSemanticName(self, tok): - """ Form semantic name by decoding semtype.""" - sn = '' - if tok.semtype: - sn = token.tok_name[tok.semtype] - return sn - - def __formArgString(self, context, tok, tn, sn, txt, fqnFunction, fqnClass): - """ Generate arguments string for use in write method.""" - sArgs = ','.join( ( - self.IDDateTime, - str( self.toknum ), - '"'+str( self.libName )+'"', - '"'+str( context['inFile'] )+'"', - str( tok.row ), - str( tok.col ), - '"'+tn+'"', - '"'+sn+'"', - str( len( txt ) ), - sqlQ( txt ), - '"'+str( toTypeName( context, fqnFunction ) )+'"', - '"'+str( toTypeName( context, fqnClass ) )+'"', - str( context['blockNum'] ), - str( context['blockDepth'] ), - str( context['fcnDepth'] ), - str( context['classDepth'] ), - str( context['parenDepth'] ), - str( context['bracketDepth'] ), - str( context['braceDepth'] ) - ) ) - return sArgs - - def close( self ): - """ Close file, if it is opened.""" - self.fd and self.fd.close() - self.fd = None - diff -uNr a/pymetrics-0.7.9/tokenize.py b/pymetrics-0.7.9/tokenize.py --- a/pymetrics-0.7.9/tokenize.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/tokenize.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,304 +0,0 @@ -"""Tokenization help for Python programs. - -generate_tokens(readline) is a generator that breaks a stream of -text into Python tokens. It accepts a readline-like method which is called -repeatedly to get the next line of input (or "" for EOF). It generates -5-tuples with these members: - - the token type (see token.py) - the token (a string) - the starting (row, column) indices of the token (a 2-tuple of ints) - the ending (row, column) indices of the token (a 2-tuple of ints) - the original line (string) - -It is designed to match the working of the Python tokenizer exactly, except -that it produces COMMENT tokens for comments and gives type OP for all -operators - -Older entry points - tokenize_loop(readline, tokeneater) - tokenize(readline, tokeneater=printtoken) -are the same, except instead of generating tokens, tokeneater is a callback -function to which the 5 fields described above are passed as 5 arguments, -each time a new token is found. - - $Id: tokenize.py,v 1.3 2005/02/15 07:08:58 rcharney Exp $ -""" -__version__ = "$Revision: 1.3 $"[11:-2] -__author__ = 'Ka-Ping Yee ' -__credits__ = \ - 'GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip Montanaro' - -import string, re -from token import * - -import token -__all__ = [x for x in dir(token) if x[0] != '_'] + ["COMMENT", "tokenize", - "generate_tokens", "NL"] -del x -del token - -COMMENT = N_TOKENS -tok_name[COMMENT] = 'COMMENT' -NL = N_TOKENS + 1 -tok_name[NL] = 'NL' -N_TOKENS += 2 - -def group(*choices): return '(' + '|'.join(choices) + ')' -def any(*choices): return group(*choices) + '*' -def maybe(*choices): return group(*choices) + '?' - -Whitespace = r'[ \f\t]*' -Comment = r'#[^\r\n]*' -Ignore = Whitespace + any(r'\\\r?\n' + Whitespace) + maybe(Comment) -Name = r'[a-zA-Z_]\w*' - -Hexnumber = r'0[xX][\da-fA-F]*[lL]?' -Octnumber = r'0[0-7]*[lL]?' -Decnumber = r'[1-9]\d*[lL]?' -Intnumber = group(Hexnumber, Octnumber, Decnumber) -Exponent = r'[eE][-+]?\d+' -Pointfloat = group(r'\d+\.\d*', r'\.\d+') + maybe(Exponent) -Expfloat = r'\d+' + Exponent -Floatnumber = group(Pointfloat, Expfloat) -Imagnumber = group(r'\d+[jJ]', Floatnumber + r'[jJ]') -Number = group(Imagnumber, Floatnumber, Intnumber) - -# Tail end of ' string. -Single = r"[^'\\]*(?:\\.[^'\\]*)*'" -# Tail end of " string. -Double = r'[^"\\]*(?:\\.[^"\\]*)*"' -# Tail end of ''' string. -Single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''" -# Tail end of """ string. -Double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""' -Triple = group("[uU]?[rR]?'''", '[uU]?[rR]?"""') -# Single-line ' or " string. -String = group(r"[uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'", - r'[uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"') - -# Because of leftmost-then-longest match semantics, be sure to put the -# longest operators first (e.g., if = came before ==, == would get -# recognized as two instances of =). -Operator = group(r"\*\*=?", r">>=?", r"<<=?", r"<>", r"!=", - r"//=?", - r"[+\-*/%&|^=<>]=?", - r"~") - -Bracket = '[][(){}]' -Special = group(r'\r?\n', r'[:;.,`]','@') -Funny = group(Operator, Bracket, Special) - -PlainToken = group(Number, Funny, String, Name) -Token = Ignore + PlainToken - -# First (or only) line of ' or " string. -ContStr = group(r"[uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" + - group("'", r'\\\r?\n'), - r'[uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' + - group('"', r'\\\r?\n')) -PseudoExtras = group(r'\\\r?\n', Comment, Triple) -PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name) - -tokenprog, pseudoprog, single3prog, double3prog = map( - re.compile, (Token, PseudoToken, Single3, Double3)) -endprogs = {"'": re.compile(Single), '"': re.compile(Double), - "'''": single3prog, '"""': double3prog, - "r'''": single3prog, 'r"""': double3prog, - "u'''": single3prog, 'u"""': double3prog, - "ur'''": single3prog, 'ur"""': double3prog, - "R'''": single3prog, 'R"""': double3prog, - "U'''": single3prog, 'U"""': double3prog, - "uR'''": single3prog, 'uR"""': double3prog, - "Ur'''": single3prog, 'Ur"""': double3prog, - "UR'''": single3prog, 'UR"""': double3prog, - 'r': None, 'R': None, 'u': None, 'U': None} - -triple_quoted = {} -for t in ("'''", '"""', - "r'''", 'r"""', "R'''", 'R"""', - "u'''", 'u"""', "U'''", 'U"""', - "ur'''", 'ur"""', "Ur'''", 'Ur"""', - "uR'''", 'uR"""', "UR'''", 'UR"""'): - triple_quoted[t] = t -single_quoted = {} -for t in ("'", '"', - "r'", 'r"', "R'", 'R"', - "u'", 'u"', "U'", 'U"', - "ur'", 'ur"', "Ur'", 'Ur"', - "uR'", 'uR"', "UR'", 'UR"' ): - single_quoted[t] = t - -tabsize = 8 - -class TokenError(Exception): pass - -class StopTokenizing(Exception): pass - -def printtoken(type, token, (srow, scol), (erow, ecol), line): # for testing - print "%d,%d-%d,%d:\t%s\t%s" % \ - (srow, scol, erow, ecol, tok_name[type], repr(token)) - -def tokenize(readline, tokeneater=printtoken): - """ - The tokenize() function accepts two parameters: one representing the - input stream, and one providing an output mechanism for tokenize(). - - The first parameter, readline, must be a callable object which provides - the same interface as the readline() method of built-in file objects. - Each call to the function should return one line of input as a string. - - The second parameter, tokeneater, must also be a callable object. It is - called once for each token, with five arguments, corresponding to the - tuples generated by generate_tokens(). - """ - try: - tokenize_loop(readline, tokeneater) - except StopTokenizing: - pass - -# backwards compatible interface -def tokenize_loop(readline, tokeneater): - for token_info in generate_tokens(readline): - tokeneater(*token_info) - -def generate_tokens(readline): - """ - The generate_tokens() generator requires one argment, readline, which - must be a callable object which provides the same interface as the - readline() method of built-in file objects. Each call to the function - should return one line of input as a string. - - The generator produces 5-tuples with these members: the token type; the - token string; a 2-tuple (srow, scol) of ints specifying the row and - column where the token begins in the source; a 2-tuple (erow, ecol) of - ints specifying the row and column where the token ends in the source; - and the line on which the token was found. The line passed is the - logical line; continuation lines are included. - """ - lnum = parenlev = continued = 0 - namechars, numchars = string.ascii_letters + '_', '0123456789' - contstr, needcont = '', 0 - contline = None - indents = [0] - strstart = (0,0) - endprog = re.compile('') - - while 1: # loop over lines in stream - line = readline() - lnum = lnum + 1 - pos, maxLen = 0, len(line) - - if contstr: # continued string - if not line: - raise TokenError, ("EOF in multi-line string", strstart) - endmatch = endprog.match(line) - if endmatch: - pos = end = endmatch.end(0) - yield (STRING, contstr + line[:end], - strstart, (lnum, end), contline + line) - contstr, needcont = '', 0 - contline = None - elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n': - yield (ERRORTOKEN, contstr + line, - strstart, (lnum, len(line)), contline) - contstr = '' - contline = None - continue - else: - contstr = contstr + line - contline = contline + line - continue - - elif parenlev == 0 and not continued: # new statement - if not line: break - column = 0 - while pos < maxLen: # measure leading whitespace - if line[pos] == ' ': column = column + 1 - elif line[pos] == '\t': column = (column/tabsize + 1)*tabsize - elif line[pos] == '\f': column = 0 - else: break - pos = pos + 1 - if pos == maxLen: break - - if line[pos] in '\r\n': # this is empty line - yield (NL, line[pos:], - (lnum, pos), (lnum, len(line)), line) - elif line[pos] == '#': # this is start of comment - pass # skip any effect on indentation - else: - if column > indents[-1]: # count indents or dedents - indents.append(column) - yield (INDENT, line[:pos], (lnum, 0), (lnum, pos), line) - while column < indents[-1]: - indents = indents[:-1] - yield (DEDENT, '', (lnum, pos), (lnum, pos), line) - - else: # continued statement - if not line: - raise TokenError, ("EOF in multi-line statement", (lnum, 0)) - continued = 0 - - while pos < maxLen: - pseudomatch = pseudoprog.match(line, pos) - if pseudomatch: # scan for tokens - start, end = pseudomatch.span(1) - spos, epos, pos = (lnum, start), (lnum, end), end - token, initial = line[start:end], line[start] - - if initial in numchars or \ - (initial == '.' and token != '.'): # ordinary number - yield (NUMBER, token, spos, epos, line) - elif initial in '\r\n': - yield (parenlev > 0 and NL or NEWLINE, - token, spos, epos, line) - elif initial == '#': - yield (COMMENT, token, spos, epos, line) - elif token in triple_quoted: - endprog = endprogs[token] - endmatch = endprog.match(line, pos) - if endmatch: # all on one line - pos = endmatch.end(0) - token = line[start:pos] - yield (STRING, token, spos, (lnum, pos), line) - else: - strstart = (lnum, start) # multiple lines - contstr = line[start:] - contline = line - break - elif initial in single_quoted or \ - token[:2] in single_quoted or \ - token[:3] in single_quoted: - if token[-1] == '\n': # continued string - strstart = (lnum, start) - endprog = (endprogs[initial] or endprogs[token[1]] or - endprogs[token[2]]) - contstr, needcont = line[start:], 1 - contline = line - break - else: # ordinary string - yield (STRING, token, spos, epos, line) - elif initial in namechars: # ordinary name - yield (NAME, token, spos, epos, line) - elif initial == '\\': # continued stmt - continued = 1 - else: - if initial in '([{': parenlev = parenlev + 1 - elif initial in ')]}': parenlev = parenlev - 1 - yield (OP, token, spos, epos, line) - else: - #yield (WS, line[pos], (lnum, pos), (lnum, pos+1), line) - while (pos < maxLen) and (line[pos] in ' \t'): - pos += 1 - if pos < maxLen: - yield (ERRORTOKEN, line[pos], (lnum, pos), (lnum, pos+1), line) - pos += 1 - - for indent in indents[1:]: # pop remaining indent levels - yield (DEDENT, '', (lnum, 0), (lnum, 0), '') - yield (ENDMARKER, '', (lnum, 0), (lnum, 0), '') - -if __name__ == '__main__': # testing - import sys - if len(sys.argv) > 1: tokenize(open(sys.argv[1]).readline) - else: tokenize(sys.stdin.readline) diff -uNr a/pymetrics-0.7.9/utils.py b/pymetrics-0.7.9/utils.py --- a/pymetrics-0.7.9/utils.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/utils.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,62 +0,0 @@ -""" Utility functions used throughout the PyMetrics system. - - $Id: utils.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ -""" - -import sys -import token -import re - -def sqlQ( s ): - """ Place single quotes around strings and escaping existing single quotes.""" - a = s.replace( "\\","\\\\" ) - a = a.replace( "'", "\\'" ) - a = a.replace( '"', '\\"' ) - return '"'+a+'"' - -def csvQ( s ): - """ Quote a string using rules for CSV data.""" - a = s.replace("\\","\\\\") - b = a.replace( "'", "\\'" ) - c = b.replace( "\n", "\\n" ) - d = c.replace( '"', '""' ) - return '"'+d+'"' - -def toTypeName( context, lst ): - """ Convert token type numbers to names.""" - lstOut = [] - for name,blockDepth,semtype in lst: - try: - semName = token.tok_name[semtype] - lstOut.append( (name,blockDepth,semName) ) - except KeyError, e: - raise KeyError( "Unknown value '"+str( e )+"' for token/semantic type in context %s\n" % context ) - - return lstOut - -if 0: - def mainTest(): - """ Built-in tests """ - def check( qs, s ): - print "<%s>==<%s>" % (s.__repr__(),qs.__repr__()) - print "[%s]==[%s]" % (s,qs) - try: - assert( s.__repr__() == qs.__repr__() ) - assert( s, qs ) - except AssertionError: - print "Failed" - - s0 = ''; qs0 = sqlQ(s0) - check( qs0, '""' ) - s1 = 'aName'; qs1 = sqlQ(s1) - check( qs1, '"aName"' ) - s2 = 'A literal with a double quote (\") in it'; qs2 = sqlQ( s2 ) - check( qs2, '"A literal with a double quote (\\\") in it"' ) - s3 = '\'A literal with a single quote (\') in it\''; qs3 = sqlQ( s3 ) - check( qs3, '"\\\'A literal with a single quote (\\\') in it\\\'"' ) - s4 = """A multi- - line literal."""; qs4 = sqlQ( s4 ) - check( qs4, '"A multi-\nline literal."' ) - - if __name__ == "__main__": - mainTest() diff -uNr a/pymetrics-0.7.9/version.py b/pymetrics-0.7.9/version.py --- a/pymetrics-0.7.9/version.py 2008-07-23 12:38:28.000000000 +0200 +++ b/pymetrics-0.7.9/version.py 1970-01-01 01:00:00.000000000 +0100 @@ -1,24 +0,0 @@ -""" PyMetrics version information. - - Copyright (c) 2005 by Reg. Charney - All rights reserved, see LICENSE for details. - - This program is free software; you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - $Id: version.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ -""" -__version__ = "$Revision: 1.3 $"[11:-2] -__author__ = 'Reg. Charney ' - -project = "pymetrics" -revision = '$Revision: 1.3 $'[11:-2] -release = '0.0' - pymetrics-0.8.1/PyMetrics/lexer.pyc0000664000076400007640000001030511042143023015425 0ustar regregmò åó‡Hc@szdZddd!ZdZdkZdkZdkZdkZdkZdkTde fd„ƒYZ d fd „ƒYZ dS( sN Parsing classes. $Id: lexer.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ s$Revision: 1.2 $i iþÿÿÿs'Reg. Charney N(t*t ParseErrorcBstZRS(N(t__name__t __module__(((t5/home/reg/pymetrics/branches/0.8.0/PyMetrics/lexer.pyRstLexercBsDtZdZd„Zd„Zd„Zd„Zd„Zd„ZRS(s Parse python source.cCsd|_d|_d|_dS(N(tNonetselft prevToktypet prevSemtypet prevToktext(R((Rt__init__s  cCs]t|ƒ}z"|iƒ}ti|ƒ|_Wd|iƒXg|_ |i ƒ|i ƒdS(s Read and parse the source. N( topent inFileNametfdtreadtsrcLineststringt expandtabsRtcloset tokenlistt_Lexer__computeOffsetst_Lexer__parseSource(RR RR((Rtparses     cCs§d|_ti|iƒ}yti|i|ƒWnntij o_}|d}|dd}|G|i|i |GHt d|||i|i |fƒ‚nXdS(s Parse the source in file.iisERROR %s Line %d:%sN(Rtpost cStringIOtStringIORttextttokenizetreadlinet TokenErrortextmsgtlinetoffsetR(RRRR R!((Rt __parseSource's  cCs ddg|_d|_d}xb|t|iƒjoK|id7_ti|id|ƒd}|pPn|ii|ƒq!W|iit|iƒƒdS(s0 Compute and store line offsets in self.offset. iis N( RR"t lineCountRtlenRRtfindtappend(RR((Rt__computeOffsets4s cCs¥|iitid|d|d|d|d|d|ƒƒ|t t t ttgjod|_d|_d|_n-|tjo||_||_||_ndS(s+Append given token to final list of tokens.ttypetsemtypeRtrowtcolR!N(RRR'tmytokentMyTokenttoktypeR*ttoktexttsrowtscolR!tNEWLINEtINDENTtDEDENTtEMPTYt ENDMARKERRRR R tWS(RR/R*R0R1R2R!((Rt__pushAs:      cCsL|\}}|\}} d} |i} |i ||}|t |ƒ|_|tjoh|dgjod}t}qÀdti|ƒ||df} tii| ƒtii| ƒdSn|ttgjo$|i|dd|||ƒdSn|| jo8||| } |itd|i| |!|| |ƒn|ttgjo-||_|i|dd|||ƒdSnti |jo.|t!jo!t!}|djo t"} q,ns|t#joet$i%|ƒp |d jo t&} q,t'} |i(d jo t)} q,|i(d jo t*} q,n|i|| ||||ƒdS( s MyToken handler.s t s*Invalid character %s in line %d column %d iNs tt@Rtdeftclass(+R1R2terowtecolRR*RRtoldposR"tnewposR%R0R/t ERRORTOKENR8tstrt__repr__R tsyststderrt writelineststdoutR3R6t _Lexer__pushR!tswsRR4R5ttokentLPARtOPt DECORATORtNAMEtkeywordt iskeywordtKEYWORDtVARNAMER tFCNNAMEt CLASSNAME(RR/R0t.6t.8R!R1R2R?R@RAR RKR*RB((Rt__call__OsH     *     ( RRt__doc__R RRRRJRY(((RRs    ( RZt __version__t __author__RFRRR-RQtglobalst ExceptionRR( RRRQRRR\RFR-R[((Rt?s      pymetrics-0.8.1/PyMetrics/simple.py0000664000076400007640000001247210664101027015452 0ustar regreg""" Simple Metrics for each function within a file. $Id: simple.py,v 1.4 2005/09/17 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.4 $"[11:-2] __author__ = 'Reg. Charney ' from metricbase import MetricBase from globals import * class SimpleMetric( MetricBase ): """ Compute simple metrics by function.""" def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): self.context = context self.runMetrics = runMetrics self.metrics = metrics self.pa = pa self.inFile = context['inFile'] self.simpleMetrics = {} self.mcc = {} self.fcnNames = {} self.classNames = {} self.lastFcnName = None self.lastClassName = None self.metrics['DocFunctions'] = [] self.metrics['DocClasses'] = [] def processToken( self, currentFcn, currentClass, tok, *args, **kwds ): """ Collect token and context sensitive data for simple metrics.""" if tok.semtype == DOCSTRING: self.metrics['numDocStrings'] = self.metrics.get('numDocStrings',0) + 1 if currentFcn and currentFcn != "__main__": if self.lastFcnName != currentFcn: self.lastFcnName = currentFcn self.metrics['numFcnDocStrings'] = self.metrics.get('numFcnDocStrings',0) + 1 self.fcnNames[currentFcn] = tok.row elif currentClass == None: # this doc string must be module doc string self.lastFcnName = "__main__" self.metrics['numFcnDocStrings'] = self.metrics.get('numFcnDocStrings',0) + 1 self.fcnNames[self.lastFcnName] = tok.row if currentClass: if self.lastClassName != currentClass: self.lastClassName = currentClass self.metrics['numClassDocStrings'] = self.metrics.get('numClassDocStrings',0) + 1 self.classNames[currentClass] = tok.row elif tok.semtype == FCNNAME: self.fcnNames[currentFcn] = 0 elif tok.semtype == CLASSNAME: self.classNames[currentClass] = 0 def processBlock( self, currentFcn, currentClass, block, *args, **kwds ): """ Collect token and context sensitive data for simple metrics.""" self.metrics['numBlocks'] = self.metrics.get('numBlocks',0)+1 def compute( self, *args, **kwds ): """ Compute any values needed.""" if self.metrics.get('numModuleDocStrings', 0) > 0: self.metrics['numFunctions'] = self.metrics.get('numFunctions', 0) + 1 try: self.simpleMetrics['%Comments'] = 100.0 * self.metrics['numComments']/self.metrics['numLines'] except (KeyError, ZeroDivisionError): self.simpleMetrics['%Comments'] = 0.0 try: self.simpleMetrics['%CommentsInline'] = 100.0 * self.metrics['numCommentsInline']/self.metrics['numLines'] except (KeyError, ZeroDivisionError): self.simpleMetrics['%CommentsInline'] = 0.0 if 0: try: self.simpleMetrics['%DocStrings'] = 100.0 * self.metrics['numDocStrings']/(self.metrics['numModuleDocStrings']+self.metrics['numClasses']+self.metrics['numFunctions']) except (KeyError, ZeroDivisionError): self.simpleMetrics['%DocStrings'] = 0.0 try: self.simpleMetrics['%FunctionsHavingDocStrings'] = 100.0 * self.metrics['numFcnDocStrings']/self.metrics['numFunctions'] except (KeyError, ZeroDivisionError): self.simpleMetrics['%FunctionsHavingDocStrings'] = 0.0 try: self.simpleMetrics['%ClassesHavingDocStrings'] = 100.0 * self.metrics['numClassDocStrings']/self.metrics['numClasses'] except (KeyError, ZeroDivisionError): self.simpleMetrics['%ClassesHavingDocStrings'] = 0.0 return self.simpleMetrics def display( self, currentFcn=None ): """ Display and return simple metrics for given function.""" def __printNames( typeName, names ): """ Pretty print list of functions/classes that have doc strings.""" if len( names ): # only output something if it exists hdr = "%s DocString present(+) or missing(-)" % typeName print print hdr print "-"*len(hdr) + "\n" result = [] keys = names.keys() keys.sort() for k in keys: if k: pfx = (names[k] and '+') or '-' print "%c %s" % (pfx,k) print result = (self.inFile, names) return result self.compute() keyList = self.simpleMetrics.keys() keyList.sort() for k in keyList: if self.pa.zeroSw or self.simpleMetrics[k]: fmt = ( k[0] == '%' and "%14.2f %s" ) or "%11d %s" print fmt % (self.simpleMetrics[k],k) self.metrics['DocFunctions'].append( __printNames( 'Functions', self.fcnNames ) ) self.metrics['DocClasses'].append( __printNames( 'Classes', self.classNames ) ) return self.simpleMetrics pymetrics-0.8.1/PyMetrics/doctestsw.pyc0000664000076400007640000000200611042143023016324 0ustar regregmò åó‡Hc@s#dZddd!ZdZeZdS(s· doctestsw - used to globally set/unset doctest testing. This switch is needed because the standard doctest can not handle raise statements with arguments. Basically, if gDoctest is True, no arguments are passed when an exception is raised. Normally, gDoctest is False and we do raise exceptions with arguments. You must modify the doctestSw value to True/False to activate/deactivate use of the doctest module tests. To used this module, use the 'from' form of the import statement and write: from doctestsw import * ... if __name__ == "__main__": if doctestSw: import doctest doctest.testmod( sys.modules[__name__] ) $Id: doctestsw.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ s$Revision: 1.2 $i iþÿÿÿs'Reg. Charney N(t__doc__t __version__t __author__tTruet doctestSw(RRR((t9/home/reg/pymetrics/branches/0.8.0/PyMetrics/doctestsw.pyt?s pymetrics-0.8.1/PyMetrics/version.py0000664000076400007640000000160711042005323015635 0ustar regreg""" PyMetrics version information. Copyright (c) 2005-2008 by Reg. Charney All rights reserved, see LICENSE for details. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. $Id: version.py,v 1.4 2008/07/24 01:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.4 $"[11:-2] __author__ = 'Reg. Charney ' project = "pymetrics" revision = 'PYMETRICS_VERSION'[5:-2] release = '0' pymetrics-0.8.1/PyMetrics/compute.py0000664000076400007640000005646311041771745015657 0ustar regreg""" Main computational modules for PyMetrics. $Id: compute.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.2 $"[11:-2] __author__ = 'Reg. Charney ' import mytoken from globals import * from utils import * MAXDISPLAY = 32 class ComputeMetrics( object ): """ Class used to compute basic metrics for given module.""" def __init__( self, metricInstance, context, runMetrics, metrics, pa, so, co ): """ Initialize general computational object.""" self.metricInstance = metricInstance self.context = context self.runMetrics = runMetrics self.metrics = metrics self.pa = pa self.so = so self.co = co self.fqnFunction = [] self.fqnClass = [] self.fcnExits = [] self.token = None self.stmt = [] self.block = [] self.fcn = [] self.cls = [] self.mod = [] self.run = [] self.processSrcLineSubscribers = [] self.processTokenSubscribers = [] self.processStmtSubscribers = [] self.processBlockSubscribers = [] self.processFunctionSubscribers = [] self.processClassSubscribers = [] self.processModuleSubscribers = [] self.processRunSubscribers = [] self.__initMetrics( metricInstance ) # this is same as function in TokenHandler !!!!!!!! def __extractFQN( self, fqn, default='__main__' ): """ Extract fully qualified name from list.""" result = default if len( fqn ): result = fqn[-1][0] return result # this is same as function in TokenHandler !!!!!!!! def __incr( self, name ): "Increment count in metrics dictionary based on name as key." self.metrics[name] = self.metrics.get(name,0) + 1 def processSrcLines( self, srcLine ): """ Handle processing of each physical source line. The fcnName and className are meaningless until the tokens are evaluated. """ fcnName = self.__extractFQN( self.fqnFunction ) className = self.__extractFQN( self.fqnClass, None ) for subscriber in self.processSrcLineSubscribers: subscriber.processSrcLines( srcLine ) def processToken( self, tok ): """ Handle processing after each token processed.""" self.token = tok self.stmt.append( tok ) # we are always in a statememt self.block.append( tok ) # we are always in a block if self.fqnFunction: # we are inside some function self.fcn.append( tok ) if self.fqnClass: # we are inside some class self.cls.append( tok ) self.mod.append( tok ) # we are always in some module self.run.append( tok ) # we are always in some run fcnName = self.__extractFQN( self.fqnFunction ) className = self.__extractFQN( self.fqnClass, None ) for subscriber in self.processTokenSubscribers: subscriber.processToken( fcnName, className, self.token ) def processStmt( self ): """ Handle processing at end of statement.""" fcnName = self.__extractFQN( self.fqnFunction ) className = self.__extractFQN( self.fqnClass, None ) for subscriber in self.processStmtSubscribers: subscriber.processStmt( fcnName, className, self.stmt ) self.stmt[:] = [] # clear out statement list def processBlock( self ): """ Handle processing at end of block.""" fcnName = self.__extractFQN( self.fqnFunction ) className = self.__extractFQN( self.fqnClass, None ) for subscriber in self.processBlockSubscribers: subscriber.processBlock( fcnName, className, self.block ) self.block[:] = [] # clear out block list def processFunction( self ): """ Handle processing at end of function. """ msg = self.__checkNumberOfExits() fcnName = self.__extractFQN( self.fqnFunction ) className = self.__extractFQN( self.fqnClass, None ) for subscriber in self.processFunctionSubscribers: subscriber.processFunction( fcnName, className, self.fcn ) self.fcn[:] = [] # clear out function list return msg def __checkNumberOfExits( self ): """" Generate warning message if more than one exit. """ msg = None n = len( self.fcnExits ) if n > 0: exits = ', '.join( [str(i) for i in self.fcnExits] ) plural = ((n > 1) and 's') or '' exitStr = "exit%s at line%s" % (plural,plural) msg = ("In file %s, function %s has %d extra %s %s" % (self.context['inFile'], self.__extractFQN( self.fqnFunction ), n, exitStr, exits)) for i in range( len( self.fcnExits ) ): del self.fcnExits[0] self.__incr( 'numMultipleExitFcns') return msg def processClass( self ): """ Handle processing at end of class. """ fcnName = self.__extractFQN( self.fqnFunction ) className = self.__extractFQN( self.fqnClass, None ) for subscriber in self.processClassSubscribers: subscriber.processClass( fcnName, className, self.cls ) self.cls[:] = [] def processModule( self ): """ Handle processing at end of class. """ moduleName = self.context['inFile'] mod = self for subscriber in self.processModuleSubscribers: subscriber.processModule( moduleName, mod ) self.mod[:] = [] def processRun( self ): """ Handle processing at end of class. """ for subsriber in self.processRunSubscribers: subsriber.processRun( self.run ) self.run[:] = [] def __call__( self, lex ): """ This function is the start of the heavy lifting for computing the various metrics produced by PyMetrics.""" for m in self.metricInstance.keys(): self.metricInstance[m].processSrcLines( lex.srcLines ) # Loop through list of tokens and count types as needed. # skipUntil is set after an error is detected. It is an attempt to # find a point to restart the analysis so that we do not get # cascading errors. This problem often occurs in analysing # foreign character sets when normal ascii was expected. tokenList = lex.tokenlist skipUntil = None tokCount = 0 invalidToken = mytoken.MyToken() for tok in tokenList: if skipUntil: if tok.text in skipUntil: # found where to restart analysis skipUntil = None elif tok.type == WS: # count, but otherwise ignore, whitespace tokCount = tokCount+1 self.metrics['numTokens'] = tokCount self.__postToken( tok ) continue elif tok.type == ERRORTOKEN: invalidToken.text += tok.text continue tokCount = self.handleToken( tokCount, tok ) return self.metrics def __initMetrics( self, metricInstance ): """ Initialize all the local variables that will be needed for analysing tokens.:""" metricList = [] for m in metricInstance.keys(): if metricInstance[m]: # only append valid instances metricList.append( metricInstance[m] ) # clear out any old data while leaving reference to same # thing (ie., pointers to these lists are always valid del self.processSrcLineSubscribers[:] del self.processTokenSubscribers[:] del self.processStmtSubscribers[:] del self.processBlockSubscribers[:] del self.processFunctionSubscribers[:] del self.processClassSubscribers[:] del self.processModuleSubscribers[:] del self.processRunSubscribers[:] # since all metric classes are derived from MetricBase, # we can assign all the processX functions to all the # metrics self.processSrcLineSubscribers.extend( metricList ) self.processTokenSubscribers.extend( metricList ) self.processStmtSubscribers.extend( metricList ) self.processBlockSubscribers.extend( metricList ) self.processFunctionSubscribers.extend( metricList ) self.processClassSubscribers.extend( metricList ) self.processModuleSubscribers.extend( metricList ) self.processRunSubscribers.extend( metricList ) self.numSrcLines = 0 self.blockDepth = 0 self.numBlocks = 0 self.parenDepth = 0 self.bracketDepth = 0 self.braceDepth = 0 self.numNestedClasses = 0 self.numKeywords = 0 self.numComments = 0 self.numEmpty = 0 self.classDepth = 0 self.fcnDepth = 0 self.classDepthIncr = 0 self.fcnDepthIncr = 0 self.maxBlockDepth = -1 self.maxClassDepth = -1 self.fqnName = [] self.defFunction = False self.defClass = False self.docString = True self.findFcnHdrEnd = False self.findClassHdrEnd = False self.inClass = False self.inFunction = False self.metrics['numSrcLines'] = 0 self.metrics['numTokens'] = 0 self.metrics['numComments'] = 0 self.metrics['numCommentsInline'] = 0 self.metrics['numModuleDocStrings'] = 0 self.metrics['numBlocks'] = 0 self.metrics['numFunctions'] = 0 self.metrics['numClasses'] = 0 self.className = None self.fcnName = None self.saveTok = None self.skipUntil = None # used to skip invalid chars until valid char found self.invalidToken = None self.checkForModuleDocString = True return metricList def __fitIn( self, tok ): """ Truncate long tokens to MAXDISPLAY length. Also, newlines are replace with '\\n' so text fits on a line.""" #tmpText = tok.text[:].replace( '\n', '\\n' ) tmpText = tok.text[:] if len( tmpText ) > MAXDISPLAY: tmpText = tmpText[:10].strip() + ' ... ' + \ tmpText[-10:].strip() return tmpText def __incrEach( self, tok ): """ Increment count for each unique semantic type.""" pfx = self.__genFQN( self.fqnName ) key = self.__fitIn( tok ) sep = tok.semtype == KEYWORD and ' ' or '.' if tok.semtype in [FCNNAME,CLASSNAME]: key = pfx else: if pfx: key = pfx + sep + key else: key = '__main__' + sep + key key = "%-10s %s" % (token.tok_name[tok.semtype],key) self.metrics[key] = self.metrics.get(key,0) + 1 def __postToken( self, tok ): """ Post token processing for common tasks.""" self.__incr( tok.type ) if tok.semtype: # then some semantic value here self.__incr( tok.semtype ) # count semantic type self.__incrEach( tok ) self.processToken( tok ) if self.pa.verbose > 1: print self.context, tok self.so and self.so.write( self.context, tok, self.fqnFunction, self.fqnClass ) self.co and self.co.write( self.context, tok, self.fqnFunction, self.fqnClass ) def __genFQN( self, fqnName ): """ Generate a fully qualified name. """ result = '.'.join( fqnName ) return result def handleToken( self, tokCount, tok ): """ Common code for handling tokens.""" if tokCount == 0: # this is the first token of the module self.prevTok = None tokCount += 1 self.metrics['numTokens'] = tokCount # don't treat whitespace as significant when looking at previous token if not tok.type in [WS,INDENT,DEDENT,EMPTY,ENDMARKER]: self.prevTok = self.saveTok self.saveTok = tok # set up context for current token self.context['blockNum'] = self.metrics.get('numBlocks',0) self.context['blockDepth'] = self.metrics.get('blockDepth',0) self.context['parenDepth'] = self.parenDepth self.context['bracketDepth'] = self.bracketDepth self.context['braceDepth'] = self.braceDepth # self.classDepthIncr is 1 if the class definition header # has a newline after colon, else it is equal to zero # meaning the class definition block is on same line as its header self.classDepth += self.classDepthIncr self.context['classDepth'] = self.classDepth self.classDepthIncr = 0 # self.classFcnIncr is 1 if the function definition header # has a newline after colon, else it is equal to zero # meaning the function definition block is on same line # as its header self.fcnDepth += self.fcnDepthIncr # only incr at start of fcn body self.context['fcnDepth'] = self.fcnDepth self.fcnDepthIncr = 0 # start testing for types that change in context if self.doDocString(tok): return tokCount if self.doInlineComment(tok, self.prevTok): return tokCount if self.doHeaders(tok): return tokCount # return with types that don't change in context self.__postToken( tok ) if tok.type == WS: return tokCount # treat end of file as end of statement if tok.type == EMPTY or tok.type == NEWLINE: self.processStmt() return tokCount # End of file forces closure of everything, but run if tok.type == ENDMARKER: self.processStmt() self.processBlock() self.processFunction() self.processClass() self.processModule() return tokCount # at this point, we have encountered a non-white space token # if a module doc string has not been found yet, # it never will be. numModDocStrings = self.metrics['numModuleDocStrings'] if self.checkForModuleDocString and numModDocStrings == 0: self.checkForModuleDocString = False msg = (("Module %s is missing a module doc string. "+ "Detected at line %d\n") % (self.context['inFile'],tok.row) ) if msg and not self.pa.quietSw: print msg if self.doOperators(tok): return tokCount if self.doIndentDedent(tok): return tokCount self.docString = False self.doKeywords(tok, self.prevTok) return tokCount def doKeywords(self, tok, prevTok): """ Count keywords and check if keyword 'return' used more than once in a given function/method.""" if tok.semtype == KEYWORD: self.__incr( 'numKeywords') if tok.text == 'def': self.defFunction = True elif tok.text == 'class': self.defClass = True elif tok.text == 'return': assert self.fcnDepth == len( self.fqnFunction ) if self.fcnDepth == 0: # not in any function if not self.pa.quietSw: # report on these types of errors print (("Module %s contains the return statement at "+ "line %d that is outside any function") % (self.context['inFile'],tok.row) ) if prevTok.text == ':': # this return on same line as conditional, # so it must be an extra return self.fcnExits.append( tok.row ) elif self.blockDepth > 1: # Let fcnBlockDepth be the block depth of the function body. # We are trying to count the number of return statements # in this function. Only one is allowed at the fcnBlockDepth # for the function. If the self.blockDepth is greater than # fcnBlockDepth, then this is a conditional return - i.e., # an additional return fcnBlockDepth = self.fqnFunction[-1][1] + 1 if self.blockDepth > fcnBlockDepth: self.fcnExits.append( tok.row ) def doIndentDedent(self, tok): """ Handle indents and dedents by keeping track of block depth. Also, handle cases where dedents close off blocks, functions, and classes.""" result = False if tok.type == INDENT: result = self.__doIndent() elif tok.type == DEDENT: result = self.__doDedent() return result def __doDedent(self): """ Handle dedents and remove function/class info as needed.""" self._doDedentFcn() self.fcnName = None if len( self.fqnFunction ) > 0: self.fcnName = self.fqnFunction[-1][0] while len(self.fqnClass) and self.blockDepth <= self.fqnClass[-1][1]: self.processClass() self.classDepth -= 1 if self.pa.verbose > 0: print "Removing class %s" % self.__extractFQN( self.fqnClass, None ) del self.fqnClass[-1] del self.fqnName[-1] self.className = None if len( self.fqnClass ) > 0: self.className = self.fqnClass[-1][0] return True def _doDedentFcn(self): """ Remove function whose scope is ended.""" assert self.fcnDepth == len( self.fqnFunction ) self.blockDepth -= 1 self.metrics['blockDepth'] = self.blockDepth while len(self.fqnFunction) and self.blockDepth<=self.fqnFunction[-1][1]: self.__doDedentRemoveMsg() del self.fqnFunction[-1] del self.fqnName[-1] self.fcnDepth -= 1 assert self.fcnDepth == len( self.fqnFunction ) def __doDedentRemoveMsg(self): """ Output message if debugging or user asks for info.""" msg = self.processFunction() if msg and not self.pa.quietSw: print msg if self.pa.verbose > 0: print "Removing function %s" % self.__extractFQN( self.fqnFunction ) def __doIndent(self): """ Increment indent count and record if maximum depth.""" self.__incr( 'numBlocks' ) self.blockDepth += 1 self.metrics['blockDepth'] = self.blockDepth if self.metrics.get('maxBlockDepth',0) < self.blockDepth: self.metrics['maxBlockDepth'] = self.blockDepth return True def doOperators(self, tok): """ Keep track of the number of each operator. Also, handle the case of the colon (:) terminating a class or def header.""" result = False if tok.type == OP: if tok.text == ':': if self.findFcnHdrEnd: self.findFcnHdrEnd = False self.docString = True self.fcnDepthIncr = 1 elif self.findClassHdrEnd: self.findClassHdrEnd = False self.docString = True self.classDepthIncr = 1 result = True elif tok.text == '(': self.parenDepth += 1 elif tok.text == ')': self.parenDepth -= 1 elif tok.text == '[': self.bracketDepth += 1 elif tok.text == ']': self.bracketDepth -= 1 elif tok.text == '{': self.braceDepth += 1 elif tok.text == '}': self.braceDepth -= 1 result = True return result def doHeaders(self, tok): """ Process both class and function headers. Create the fully qualified names and deal with possibly erroneous headers.""" result = False if self.defFunction: if tok.type == NAME: self.__incr( 'numFunctions') self.fqnName.append( tok.text ) self.fcnName = self.__genFQN( self.fqnName ) self.fqnFunction.append( (self.fcnName,self.blockDepth,FCNNAME) ) self.defFunction = False self.findFcnHdrEnd = True if self.pa.verbose > 0: print ("fqnFunction=%s" % toTypeName( self.context, self.fqnFunction ) ) self.__postToken( tok ) result = True elif tok.type == ERRORTOKEN: # we must compensate for the simple scanner mishandling errors self.findFcnHdrEnd = True self.invalidToken = mytoken.MyToken(type=NAME, semtype=FCNNAME, text=tok.text, row=tok.row, col=tok.col, line=tok.line) self.skipUntil = '(:\n' result = True elif self.defClass and tok.type == NAME: self.__incr( 'numClasses' ) self.fqnName.append( tok.text ) self.className = self.__genFQN( self.fqnName ) self.fqnClass.append( (self.className,self.blockDepth,CLASSNAME) ) self.defClass = False self.findClassHdrEnd = True if self.pa.verbose > 0: print "fqnClass=%s" % toTypeName( self.context, self.fqnClass ) self.__postToken( tok ) result = True return result def doInlineComment(self, tok, prevTok): """ Check for comments and distingish inline comments from normal comments.""" result = False if tok.type == COMMENT: self.metrics['numComments'] += 1 # compensate for older tokenize including newline # symbols in token when only thing on line is comment # this patch makes all comments consistent if tok.text[-1] == '\n': tok.text = tok.text[:-1] if prevTok and prevTok.type != NEWLINE and prevTok.type != EMPTY: tok.semtype = INLINE self.metrics['numCommentsInline'] += 1 self.__postToken( tok ) result = True return result def doDocString(self, tok): """ Determine if a given string is also a docstring. Also, detect if docstring is also the module's docsting.""" result = False if self.docString and tok.type == token.STRING: tok.semtype = DOCSTRING if self.checkForModuleDocString: # found the module's doc string self.metrics['numModuleDocStrings'] += 1 self.checkForModuleDocString = False self.__postToken( tok ) self.docString = False result = True return result pymetrics-0.8.1/PyMetrics/mytoken.pyc0000664000076400007640000000316411042143023016001 0ustar regregmò åó‡Hc@s@dZddd!ZdZdkZdkTdfd„ƒYZdS( sg MyToken - PyMetrics' version of Token. $Id: mytoken.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ s$Revision: 1.3 $i iþÿÿÿs'Reg. Charney N(t*tMyTokencBstZd„Zd„ZRS(NcKs|ii|ƒdS(s- Initialize class with user-defined keywords.N(tselft__dict__tupdatetkwds(RR((t7/home/reg/pymetrics/branches/0.8.0/PyMetrics/mytoken.pyt__init__ scCsìti|i}|i}|oti|i}n|ittt t t t gjo,d|||i |it|iƒf}ng|itjo(|iddjo|id |_nd|||i |it|iƒ|if}|S(sª Pretty print token. Don't print text for special token types since they do not have useful visible representation, other than blank. s)[type=%s semtype=%s row=%s col=%s len=%d]iÿÿÿÿs s3[type=%s semtype=%s row=%s col=%s len=%d text=<%s>]N(ttokenttok_nameRttypettntsemtypetsntWStNEWLINEtINDENTtDEDENTtEMPTYt ENDMARKERtrowtcoltlenttexttstCOMMENT(RRR R ((Rt__repr__s ",$.(t__name__t __module__RR(((RR s (t__doc__t __version__t __author__RtglobalsR(RRRR((Rt?s   pymetrics-0.8.1/PyMetrics/processargs.pyc0000664000076400007640000003644511042143023016656 0ustar regregmò : ˆHc@s5dZddd!ZdZdkZdklZlZdkTdZd efd „ƒYZ d e fd „ƒYZ d e fd„ƒYZ d„Zedjo—eeiƒdjoeiddjoHeZeideid)dkZeieieƒeidƒnddfGHy e dddƒZeeƒWn+e j oZeiieeƒƒnXHddfGHy,e ddddedeƒZeeƒWn+e j oZeiieeƒƒnXHdd fGHye ƒZeeƒWn+e j oZeiieeƒƒnXHeidƒndS(!s Process command line arguments. Usage: >>> pa = ProcessArgs( 'file1.py', 'file2.py', '/home/files/file3.py' ) >>> pa = ProcessArgs( "inFile1.py", sqlFileName='sqlF1.sql', genCsvSw=False,genKwCntSw=True ) >>> pa = ProcessArgs() #doctest +NORMALIZE_WHITESPACE +ELLIPSIS python PyMetrics [ options ] pgm1.py [ pgm2.py ... ] Complexity metrics are computed for the Python input files pgm1.py, pgm2.py, etc. At least one file name is required, else this message appears. Three types of output can be produced: * Standard output for a quick summary of the main metrics. * A text file containing SQL commands necessary to build a SQL table in the database of your choice. * A text file containing Comma-Separated Values (CSV) formatted data necessary to load into most spreadsheet programs. PLEASE NOTE: multi-line literals have the new-line character replaced with the character "\n" in the output text. Capitalized options negate the default option. options: --version show program's version number and exit -h, --help show this help message and exit -s SQLFILENAME, --sql=SQLFILENAME name of output SQL command file. (Default is metricData.sql) -t SQLTOKENTABLENAME, --tokentable=SQLTOKENTABLENAME name of output SQL token table. (Default is metricTokens) -m SQLMETRICSTABLENAME, --metricstable=SQLMETRICSTABLENAME name of output SQL metrics table. (Default is metricData) -c CSVFILENAME, --csv=CSVFILENAME name of output CSV data file. (Default is metricData.csv) -f INFILELIST, --files=INFILELIST File containing list of path names to modules for analysis. -i INCLUDEMETRICSSTR, --include=INCLUDEMETRICSSTR list of metrics to include in run. This is a comma separated list of metric module names with no whitespace. Optionally, you can specify the class name of the metric by following the module name with a colon (:) and the metric class name. (Default metrics are 'simple:SimpleMetric,mccabe:McCabeMetric, sloc:SLOCMetric'. Default metric class name for metric module 'wxYz' is 'WxYzMetric' when only module name given -- note capitalized metric class name.) -l LIBNAME, --library=LIBNAME user-defined name applied to collection of modules (Default is '') -e, --exists assume SQL tables exist and does not generate creation code. Using this option sets option -N. (Default is False) -N, --noold create new command output files and tables after deleting old results, if any. Ignored if -e is set. (Default is False) -B, --nobasic suppress production of Basic metrics (Default is False) -S, --nosql suppress production of output SQL command text file. (Default is False) -C, --nocsv suppress production of CSV output text file. (Default is False) -H, --noheadings suppress heading line in csv file. (Default is False) -k, --kwcnt generate keyword counts. (Default is False) -K, --nokwcnt suppress keyword counts. (Default is True) -q, --quiet suppress normal summary output to stdout. (Default is False) -z, --zero display zero or empty values in output to stdout. (Default is to suppress zero/empty output) -v, --verbose Produce verbose output - more -v's produce more output. (Default is no verbose output to stdout) -d, --debug Provide debug output, not usually generated - internal use only No program file names given. $Id: processargs.py,v 1.4 2008/07/24 04:28:12 rcharney Exp $ s$Revision: 1.3 $i iþÿÿÿs'Reg. Charney N(s OptionParsersBadOptionError(t*sÉpython PyMetrics [ options ] pgm1.py [ pgm2.py ... ] Complexity metrics are computed for the Python input files pgm1.py, pgm2.py, etc. At least one file name is required, else this message appears. Three types of output can be produced: * Standard output for a quick summary of the main metrics. * A text file containing SQL commands necessary to build a SQL table in the database of your choice. * A text file containing Comma-Separated Values (CSV) formatted data necessary to load into most spreadsheet programs. PLEASE NOTE: multi-line literals have the new-line character replaced with the character "\n" in the output text. Capitalized options negate the default option. tPyMetricsOptionParsercBs tZdZd„Zd„ZRS(s? Subclass OptionParser so I can override default error handler.cOsti|||ŽdS(sF Just call super class's __init__ since we aren't making changes here.N(t OptionParsert__init__tselftargstkwds(RRR((t;/home/reg/pymetrics/branches/0.8.0/PyMetrics/processargs.pyRrscCst|ƒ‚dS(sB Explicitly raise BadOptionError so calling program can handle it.N(tBadOptionErrortmsg(RR ((Rterrorvs(t__name__t __module__t__doc__RR (((RRps  tProcessArgsErrorcBstZRS(N(R R (((RRzst ProcessArgscBs)tZdZd„Zd„Zd„ZRS(s Process command line arguments.c OsEd}d}d}d} d}db} d}db} t }t }t }t }t } t }t }t }t }t }t }t }d}|iitƒƒ|id=|ii|ƒ|id =tdd d ƒ}|id d ddd|idd|iƒ|iddddd|idd|iƒ|iddddd|idd|iƒ|iddddd|idd|iƒ|idd dd!d|idd"ƒ|id#d$dd%d|idd&ƒ|id'd(dd)d|idd*ƒ|id+d,d-d.dd/d|idd0|iƒ|id1d2d-d.dd3d|idd4|iƒ|id5d6d-d7dd8d|idd9|i ƒ|id:d;d-d7dd<d|idd=|i ƒ|id>d?d-d7dd@d|iddA|i ƒ|idBdCd-d7ddDd|iddE|i ƒ|idFdGd-d.ddHd|iddI|ifƒ|idJdKd-d7ddHd|iddL|i ƒ|idMdNd-d.ddOd|i ddP|i ƒ|idQdRd-d.ddSd|i ddTƒ|idUdVd-dWddXd|iddYƒ|idZd[d-d.dd\d|i dd]ƒy|iƒ\}}Wnbt"j oV}t$i%i&d^t'|ƒƒt$i%i&d_ƒt$i%i&|i(ƒƒt$i)d`ƒnX|i*|ƒ|ii|iƒ|iody<t,|iƒ} | i.ƒi/ƒ}| i1ƒ|i*|ƒWq:t2j o}t3|ƒ‚q:Xn||_4|i5|iƒ|_6t7|ƒd`jo:t8GH|i(ƒGHda}t9o |GHdbSq¥t3|ƒ‚ndb}|ioÜ|ioÒy4|io&t,|idcƒ|_:|i:i1ƒnWnt2j ot |_nXy:dd}|io de}nt,|i|ƒ}|i1ƒWq‘t2j o2}t$i%i&t'|ƒdfƒt |_db}q‘Xndb}|io‰|ioy:dd}|io de}nt,|i|ƒ}|i1ƒWq*t2j o2}t$i%i&t'|ƒdgƒt |_db}q*Xn|io t |_ndbS(hs! Initial processing of arguments.tsmetricData.sqlt metricTokenst metricDatasmetricData.csvs7simple:SimpleMetric,mccabe:McCabeMetric,sloc:SLOCMetriciRtpKwdstversions%prog PYMETRICS_VERSIONs-ss--sqltdestt sqlFileNametdefaultthelps0name of output SQL command file. (Default is %s)s-ts --tokentabletsqlTokenTableNames/name of output SQL token table. (Default is %s)s-ms--metricstabletsqlMetricsTableNames1name of output SQL metrics table. (Default is %s)s-cs--csvt csvFileNames-name of output CSV data file. (Default is %s)s-fs--filest inFileLists;File containing list of path names to modules for analysis.s-is --includetincludeMetricsStrsÅlist of metrics to include in run. This is a comma separated list of metric module names with no whitespace. Optionally, you can specify the class name of the metric by following the module name with a colon (:) and the metric class name. (Default metrics are 'simple:SimpleMetric,mccabe:McCabeMetric,sloc:SLOCMetric'. Default metric class name for metric module 'wxYz' is 'WxYzMetric' when only module name given -- note capitalized metric class name.)s-ls --librarytlibNamesBuser-defined name applied to collection of modules (Default is '')s-es--existstactiont store_truet genExistsSwsnassume SQL tables exist and does not generate creation code. Using this option sets option -N. (Default is %s)s-Ns--nooldtgenNewSwstcreate new command output files and tables after deleting old results, if any. Ignored if -e is set. (Default is %s)s-Bs --nobasict store_falset genBasicSws4suppress production of Basic metrics (Default is %s)s-Ss--nosqltgenSqlSwsDsuppress production of output SQL command text file. (Default is %s)s-Cs--nocsvtgenCsvSws<suppress production of CSV output text file. (Default is %s)s-Hs --noheadingstgenHdrSws2suppress heading line in csv file. (Default is %s)s-ks--kwcntt genKwCntSws(generate keyword counts. (Default is %s)s-Ks --nokwcnts(suppress keyword counts. (Default is %s)s-qs--quiettquietSws9suppress normal summary output to stdout. (Default is %s)s-zs--zerotzeroSws\display zero or empty values in output to stdout. (Default is to suppress zero/empty output)s-vs --verbosetcounttverboses`Produce verbose output - more -v's produce more output. (Default is no verbose output to stdout)s-ds--debugtdebugSws?Provide debug output, not usually generated - internal use onlys BadOptionError: %s s The valid options are: isNo program file names given. Ntrtatws* -- No SQL command file will be generated s) -- No CSV output file will be generated (=RRRRRtNoneRRtexcludeMetricsStrtFalseR-R)R*tTrueR$R(R%R&R't genAppendSwR"R!R,Rt__dict__tupdatetlocalsRRtparsert add_optiont parse_argstoptionsRRtetsyststderrt writelineststrt format_helptexittextendtpArgstopentinFtreadtsplittfilestclosetIOErrorRt inFileNamestprocessIncludeMetricstincludeMetricstlentusageStrt doctestSwtsotmodetco( RRERR*RUR,R9RR(R2RRR$RGRJRR"RR%RR&R5R-R'RRSR!RR)RTR<R=((RR~s\                                                            cOsd|GHd|GHdS(Nsargs=%sskwds=%s(RR(RRR((RtconflictHandlerds cCsg}yÐ|idƒ}xº|D]²}|idƒ}t|ƒdjo|i|ƒqt|ƒdjoZ|do?|ddiƒ|ddd}|i|d|fƒqÑt dƒ‚qt dƒ‚qWWn+t j o}d |}t |ƒ‚nX|S( Nt,t:iiitMetricsMissing metric module names'Malformed items in includeMetric strings Invalid list of metric names: %s( RORRIt metricListR/tsRPtappendtuppertdefNameRtAttributeErrorR=(RRR/R=ROR[R^RZ((RRNhs$ $ (R R R RRVRN(((RR|s  æ cCs¹d|i|i|i|i|i|i|i|i|i |i |i |i |i f GHdGHx&|iD]\}}d||fGHqfW|io&dGHx|iD]}d|GHqžWndS(s‹ Test of ProcessArgs. Usage: >>> pa=ProcessArgs('inFile.py') >>> testpa(pa) #doctest: +NORMALIZE_WHITESPACE +ELLIPSIS Arguments processed: sqlFileName=metricData.sql sqlTokenTableName=metricTokens sqlMetricsTableName=metricData csvFileName=metricData.csv Include Metric Modules=simple:SimpleMetric,mccabe:McCabeMetric quietSw=False genCsvSw=True genHdrSw=True genKwCntSw=False verbose=... Metrics to be used are: Module simple contains metric class SimpleMetric Module mccabe contains metric class McCabeMetric Input files: inFile.py >>> sé Arguments processed: sqlFileName=%s sqlTokenTableName=%s sqlMetricsTableName=%s csvFileName=%s include Metric Modules=%s quietSw=%s genNewSw=%s genExistsSw=%s genSqlSw=%s genCsvSw=%s genHdrSw=%s genKwCntSw=%s verbose=%ssMetrics to be used are:s# Module %s contains metric class %ss Input files:s %sN(tpaRRRRRR)R"R!R%R&R'R(R,ROtmtnRMtf(R`RaRcRb((Rttestpa‚sW   t__main__itdoctestiis=== ProcessArgs: %s ===s.'file1.py', 'file2.py', '/home/files/file3.py'sfile1.pysfile2.pys/home/files/file3.pysCinFile1.py, sqlFileName='sqlF1.sql', genCsvSw=False,genKwCntSw=Trues inFile1.pyRs sqlF1.sqlR&R(s No arguments(R t __version__t __author__R>toptparseRRt doctestswRQRt ExceptionRtobjectRRdR RPtargvR4RRRfttestmodtmodulesRCR`R=R?R@RAR3(RR=RRdRhR>RRRQR`RfRRRgR((Rt?UsP   ÿ = *     pymetrics-0.8.1/PyMetrics/processargs.py0000664000076400007640000005130411042151000016474 0ustar regreg""" Process command line arguments. Usage: >>> pa = ProcessArgs( 'file1.py', 'file2.py', '/home/files/file3.py' ) >>> pa = ProcessArgs( "inFile1.py", sqlFileName='sqlF1.sql', genCsvSw=False,genKwCntSw=True ) >>> pa = ProcessArgs() #doctest +NORMALIZE_WHITESPACE +ELLIPSIS python PyMetrics [ options ] pgm1.py [ pgm2.py ... ] Complexity metrics are computed for the Python input files pgm1.py, pgm2.py, etc. At least one file name is required, else this message appears. Three types of output can be produced: * Standard output for a quick summary of the main metrics. * A text file containing SQL commands necessary to build a SQL table in the database of your choice. * A text file containing Comma-Separated Values (CSV) formatted data necessary to load into most spreadsheet programs. PLEASE NOTE: multi-line literals have the new-line character replaced with the character "\\n" in the output text. Capitalized options negate the default option. options: --version show program's version number and exit -h, --help show this help message and exit -s SQLFILENAME, --sql=SQLFILENAME name of output SQL command file. (Default is metricData.sql) -t SQLTOKENTABLENAME, --tokentable=SQLTOKENTABLENAME name of output SQL token table. (Default is metricTokens) -m SQLMETRICSTABLENAME, --metricstable=SQLMETRICSTABLENAME name of output SQL metrics table. (Default is metricData) -c CSVFILENAME, --csv=CSVFILENAME name of output CSV data file. (Default is metricData.csv) -f INFILELIST, --files=INFILELIST File containing list of path names to modules for analysis. -i INCLUDEMETRICSSTR, --include=INCLUDEMETRICSSTR list of metrics to include in run. This is a comma separated list of metric module names with no whitespace. Optionally, you can specify the class name of the metric by following the module name with a colon (:) and the metric class name. (Default metrics are 'simple:SimpleMetric,mccabe:McCabeMetric, sloc:SLOCMetric'. Default metric class name for metric module 'wxYz' is 'WxYzMetric' when only module name given -- note capitalized metric class name.) -l LIBNAME, --library=LIBNAME user-defined name applied to collection of modules (Default is '') -e, --exists assume SQL tables exist and does not generate creation code. Using this option sets option -N. (Default is False) -N, --noold create new command output files and tables after deleting old results, if any. Ignored if -e is set. (Default is False) -B, --nobasic suppress production of Basic metrics (Default is False) -S, --nosql suppress production of output SQL command text file. (Default is False) -C, --nocsv suppress production of CSV output text file. (Default is False) -H, --noheadings suppress heading line in csv file. (Default is False) -k, --kwcnt generate keyword counts. (Default is False) -K, --nokwcnt suppress keyword counts. (Default is True) -q, --quiet suppress normal summary output to stdout. (Default is False) -z, --zero display zero or empty values in output to stdout. (Default is to suppress zero/empty output) -v, --verbose Produce verbose output - more -v's produce more output. (Default is no verbose output to stdout) -d, --debug Provide debug output, not usually generated - internal use only No program file names given. $Id: processargs.py,v 1.4 2008/07/24 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.3 $"[11:-2] __author__ = 'Reg. Charney ' import sys from optparse import OptionParser, BadOptionError from doctestsw import * usageStr = """python PyMetrics [ options ] pgm1.py [ pgm2.py ... ] Complexity metrics are computed for the Python input files pgm1.py, pgm2.py, etc. At least one file name is required, else this message appears. Three types of output can be produced: * Standard output for a quick summary of the main metrics. * A text file containing SQL commands necessary to build a SQL table in the database of your choice. * A text file containing Comma-Separated Values (CSV) formatted data necessary to load into most spreadsheet programs. PLEASE NOTE: multi-line literals have the new-line character replaced with the character "\\n" in the output text. Capitalized options negate the default option. """ class PyMetricsOptionParser( OptionParser ): """ Subclass OptionParser so I can override default error handler.""" def __init__( self, *args, **kwds ): """ Just call super class's __init__ since we aren't making changes here.""" OptionParser.__init__( self, *args, **kwds ) def error( self, msg ): """ Explicitly raise BadOptionError so calling program can handle it.""" raise BadOptionError( msg ) class ProcessArgsError( Exception ): pass class ProcessArgs( object ): """ Process command line arguments.""" def __init__( self, *pArgs, **pKwds ): """ Initial processing of arguments.""" # precedence for defaults, parameters, and command line arguments/keywords # is: # # command line parameters/keywords take precedence over # parameters used to instantiate ProcessArgs that takes precedence over # default values for keywords. # # This implies we set completed options with defaults first # then override completed options with parameters # then override completed options with command line args/keywords # # default values for possible parameters libName = '' sqlFileName = "metricData.sql" sqlTokenTableName = "metricTokens" sqlMetricsTableName = "metricData" csvFileName = "metricData.csv" inFileList = None includeMetricsStr = 'simple:SimpleMetric,mccabe:McCabeMetric,sloc:SLOCMetric' excludeMetricsStr = None debugSw = False debugSw = False quietSw = False zeroSw = False genBasicSw = True genKwCntSw = False genSqlSw = True genCsvSw = True genHdrSw = True genAppendSw = True genNewSw = False genExistsSw = False verbose = 0 self.__dict__.update( locals() ) del( self.__dict__['self'] ) # remove recursive self from self.__dict__ self.__dict__.update( pKwds ) del( self.__dict__['pKwds'] ) # remove redundant pKwds in self.__dict__ # set up option parser parser = PyMetricsOptionParser( '', version="%prog 0.8.1" ) parser.add_option("-s", "--sql", dest="sqlFileName", default=self.sqlFileName, help="name of output SQL command file. (Default is %s)" % self.sqlFileName ) parser.add_option("-t", "--tokentable", dest="sqlTokenTableName", default=self.sqlTokenTableName, help="name of output SQL token table. (Default is %s)" % self.sqlTokenTableName ) parser.add_option("-m", "--metricstable", dest="sqlMetricsTableName", default=self.sqlMetricsTableName, help="name of output SQL metrics table. (Default is %s)" % self.sqlMetricsTableName ) parser.add_option("-c", "--csv", dest="csvFileName", default=self.csvFileName, help="name of output CSV data file. (Default is %s)" % self.csvFileName ) parser.add_option("-f", "--files", dest="inFileList", default=self.inFileList, help="File containing list of path names to modules for analysis." ) parser.add_option("-i", "--include", dest="includeMetricsStr", default=self.includeMetricsStr, help="list of metrics to include in run. This is a comma separated list of metric module names with no whitespace. Optionally, you can specify the class name of the metric by following the module name with a colon (:) and the metric class name. (Default metrics are 'simple:SimpleMetric,mccabe:McCabeMetric,sloc:SLOCMetric'. Default metric class name for metric module 'wxYz' is 'WxYzMetric' when only module name given -- note capitalized metric class name.)" ) parser.add_option("-l", "--library", dest="libName", default=self.libName, help="user-defined name applied to collection of modules (Default is '')" ) parser.add_option("-e", "--exists", action="store_true", dest="genExistsSw", default=self.genExistsSw, help="assume SQL tables exist and does not generate creation code. Using this option sets option -N. (Default is %s)" % (self.genExistsSw) ) parser.add_option("-N", "--noold", action="store_true", dest="genNewSw", default=self.genNewSw, help="create new command output files and tables after deleting old results, if any. Ignored if -e is set. (Default is %s)" % (self.genNewSw) ) parser.add_option("-B", "--nobasic", action="store_false", dest="genBasicSw", default=self.genBasicSw, help="suppress production of Basic metrics (Default is %s)" % (not self.genBasicSw) ) parser.add_option("-S", "--nosql", action="store_false", dest="genSqlSw", default=self.genSqlSw, help="suppress production of output SQL command text file. (Default is %s)" % (not self.genSqlSw) ) parser.add_option("-C", "--nocsv", action="store_false", dest="genCsvSw", default=self.genCsvSw, help="suppress production of CSV output text file. (Default is %s)" % (not self.genCsvSw) ) parser.add_option("-H", "--noheadings", action="store_false", dest="genHdrSw", default=self.genHdrSw, help="suppress heading line in csv file. (Default is %s)" % (not self.genHdrSw) ) parser.add_option("-k", "--kwcnt", action="store_true", dest="genKwCntSw", default=self.genKwCntSw, help="generate keyword counts. (Default is %s)" % (self.genKwCntSw,) ) parser.add_option("-K", "--nokwcnt", action="store_false", dest="genKwCntSw", default=self.genKwCntSw, help="suppress keyword counts. (Default is %s)" % (not self.genKwCntSw) ) parser.add_option("-q", "--quiet", action="store_true", dest="quietSw", default=self.quietSw, help="suppress normal summary output to stdout. (Default is %s)" % (self.quietSw) ) parser.add_option("-z", "--zero", action="store_true", dest="zeroSw", default=self.zeroSw, help="display zero or empty values in output to stdout. (Default is to suppress zero/empty output)" ) parser.add_option("-v", "--verbose", action="count", dest="verbose", default=self.verbose, help="Produce verbose output - more -v's produce more output. (Default is no verbose output to stdout)") parser.add_option("-d", "--debug", action="store_true", dest="debugSw", default=self.debugSw, help="Provide debug output, not usually generated - internal use only") # parse the command line/arguments for this instance try: (options, args) = parser.parse_args() except BadOptionError, e: sys.stderr.writelines( "\nBadOptionError: %s\n" % str( e ) ) sys.stderr.writelines( "\nThe valid options are:\n\n" ) sys.stderr.writelines( parser.format_help() ) sys.exit( 1 ) # augment parameter values from instantiation with # command line values. # the command line parameter values take precidence # over values in program. args.extend( pArgs ) # convert command line arguments into instance values self.__dict__.update( options.__dict__ ) if self.inFileList: try: inF = open( self.inFileList ) files = inF.read().split() inF.close() args.extend( files ) except IOError, e: raise ProcessArgsError( e ) self.inFileNames = args self.includeMetrics = self.processIncludeMetrics( self.includeMetricsStr ) if len( args ) < 1: print usageStr print parser.format_help() e = "No program file names given.\n" # because of what I believe to be a bug in the doctest module, # which makes it mishandle exceptions, I have 'faked' the handling # of raising an exception and just return if doctestSw: print e return else: raise ProcessArgsError( e ) so = None if self.genSqlSw and self.sqlFileName: # Try opening command file for input. If OK, then file exists. # Else, the commamd file does not exist to create SQL table and # this portion of the command file must be generated as if new # table was to be created. # # NOTE: This assumption can be dangerous. If the table does exist # but the command file is out of sync with the table, then a valid # table can be deleted and recreated, thus losing the old data. # The workaround for this is to examine the SQL database for the # table and only create the table, if needed. try: if self.sqlFileName: self.so = open( self.sqlFileName, "r" ) self.so.close() except IOError: self.genNewSw = True try: mode = "a" if self.genNewSw: mode = "w" so = open( self.sqlFileName, mode ) # check to see that we can write file so.close() except IOError, e: sys.stderr.writelines( str(e) + " -- No SQL command file will be generated\n" ) self.genSqlSw = False so = None co = None if self.genCsvSw and self.csvFileName: try: mode = "a" if self.genNewSw: mode = "w" co = open( self.csvFileName, mode ) # check we will be able to write file co.close() except IOError, e: sys.stderr.writelines( str(e) + " -- No CSV output file will be generated\n" ) self.genCsvSw = False co = None if self.genExistsSw: # Assuming that table already exists, means concatenating # data is not needed - we only want new data in SQL command file. self.genNewSw = True def conflictHandler( self, *args, **kwds ): print "args=%s" % args print "kwds=%s" % kwds def processIncludeMetrics( self, includeMetricsStr ): includeMetrics = [] try: metricList = includeMetricsStr.split( ',' ) for a in metricList: s = a.split( ':' ) if len( s ) == 2: # both metric class and module name given includeMetrics.append( s ) elif len( s ) == 1: # only the module name given. Generate default metric # class name by capitalizing first letter of module # name and appending "Metric" so the default metric # class name for module wxYz is WxYzMetric. if s[0]: defName = s[0][0].upper() + s[0][1:] + 'Metric' includeMetrics.append( (s[0], defName) ) else: raise ProcessArgsError("Missing metric module name") else: raise ProcessArgsError("Malformed items in includeMetric string") except AttributeError, e: e = ( "Invalid list of metric names: %s" % includeMetricsStr ) raise ProcessArgsError( e ) return includeMetrics def testpa( pa ): """ Test of ProcessArgs. Usage: >>> pa=ProcessArgs('inFile.py') >>> testpa(pa) #doctest: +NORMALIZE_WHITESPACE +ELLIPSIS Arguments processed: sqlFileName=metricData.sql sqlTokenTableName=metricTokens sqlMetricsTableName=metricData csvFileName=metricData.csv Include Metric Modules=simple:SimpleMetric,mccabe:McCabeMetric quietSw=False genCsvSw=True genHdrSw=True genKwCntSw=False verbose=... Metrics to be used are: Module simple contains metric class SimpleMetric Module mccabe contains metric class McCabeMetric Input files: inFile.py >>> """ print """ Arguments processed: \tsqlFileName=%s \tsqlTokenTableName=%s \tsqlMetricsTableName=%s \tcsvFileName=%s \tinclude Metric Modules=%s \tquietSw=%s \tgenNewSw=%s \tgenExistsSw=%s \tgenSqlSw=%s \tgenCsvSw=%s \tgenHdrSw=%s \tgenKwCntSw=%s \tverbose=%s""" % ( pa.sqlFileName, pa.sqlTokenTableName, pa.sqlMetricsTableName, pa.csvFileName, pa.includeMetricsStr, pa.quietSw, pa.genNewSw, pa.genExistsSw, pa.genSqlSw, pa.genCsvSw, pa.genHdrSw, pa.genKwCntSw, pa.verbose) print "Metrics to be used are:" for m,n in pa.includeMetrics: print "\tModule %s contains metric class %s" % (m,n) if pa.inFileNames: print "Input files:" for f in pa.inFileNames: print "\t%s" % f if __name__ == "__main__": # ignore doctestsw.doctestSw if this is run as a standalone # module. Instead, look at the first argument on the command # line. If it is "doctest", set doctestSw = True and delete # the parameter "doctest" from sys.argv, thus leaving things # as _normal_. if len(sys.argv) > 1 and sys.argv[1] == 'doctest': doctestSw = True sys.argv[1:] = sys.argv[2:] import doctest doctest.testmod( sys.modules[__name__] ) sys.exit( 0 ) print "=== ProcessArgs: %s ===" % ("'file1.py', 'file2.py', '/home/files/file3.py'",) try: pa = ProcessArgs( 'file1.py', 'file2.py', '/home/files/file3.py' ) testpa( pa ) except ProcessArgsError, e: sys.stderr.writelines( str( e ) ) print print "=== ProcessArgs: %s ===" % ("inFile1.py, sqlFileName='sqlF1.sql', genCsvSw=False,genKwCntSw=True",) try: pa = ProcessArgs( "inFile1.py", sqlFileName='sqlF1.sql', genCsvSw=False,genKwCntSw=True ) testpa( pa ) except ProcessArgsError, e: sys.stderr.writelines( str( e ) ) print print "=== ProcessArgs: %s ===" % ("No arguments",) try: pa = ProcessArgs() testpa( pa ) except ProcessArgsError, e: sys.stderr.writelines( str( e ) ) print sys.exit( 0 ) pymetrics-0.8.1/PyMetrics/newFeatures24.py0000664000076400007640000002023111041771745016621 0ustar regreg""" Metrics for New Features introduced in Python 2.4. $Id: newfeatures24.py,v 1.4 2005/09/17 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.1 $"[11:-2] __author__ = 'Reg. Charney ' from metricbase import MetricBase from globals import * class NewFeatures24Metric( MetricBase ): """ Compute simple metrics by function.""" firstPass = True def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): """ Count features introduced after v2.3.""" self.context = context self.runMetrics = runMetrics self.metrics = metrics self.pa = pa self.inFile = context['inFile'] self.prevTokText = '' self.userDefinedSet = False self.featureMetrics = {} self.numGeneratorFunctions = 0 self.numGeneratorExpressions = 0 self.numListComprehension = 0 self.numClassProperties = 0 self.numClassVariables = 0 self.numClassFunctions = 0 self.numDecorators = 0 self.isGeneratorFunction = False self.isGeneratorExpression = False self.isListComprehension = False self.usedInbuiltSets = False self.usedDecorators = False self.maybeSetKeyword = False self.stackParenOrBracket = []; self.featureMetrics['numGeneratorFunctions'] = 0 self.featureMetrics['numGeneratorExpressions'] = 0 self.featureMetrics['numListComprehensions'] = 0 self.featureMetrics['numModulesUsingSets'] = 0 self.featureMetrics['numDecorators'] = 0 if NewFeatures24Metric.firstPass: self.runMetrics['modulesUsingGeneratorFunctions'] = [] self.runMetrics['modulesUsingGeneratorExpressions'] = [] self.runMetrics['modulesUsingListComprehension'] = [] self.runMetrics['modulesUsingSets'] = [] self.runMetrics['modulesUsingDecorators'] = [] NewFeatures24Metric.firstPass = False def processToken( self, currentFcn, currentClass, tok, *args, **kwds ): """ Collect token and context sensitive data for simple metrics.""" if tok.semtype == KEYWORD: self.__handleKeywords( currentFcn, currentClass, tok, *args, **kwds ) elif tok.type == OP: if self.maybeSetKeyword and tok.text == '(': if not self.userDefinedSet: self.usedInbuiltSets = True self.maybeSetKeyword = False elif tok.text == '(' or tok.text == '[': self.stackParenOrBracket.append( tok.text ) elif tok.text == ')' or tok.text == ']': if len( self.stackParenOrBracket ) > 0: del self.stackParenOrBracket[-1] elif tok.text == '@': self.usedDecorators = True self.featureMetrics['numDecorators'] += 1 elif tok.semtype == VARNAME: if tok.text in ['set', 'frozenset']: if not self.prevTokText in ['.', 'def', 'class']: self.maybeSetKeyword = True elif tok.semtype in [FCNNAME,CLASSNAME]: # We need to ignore user-defined global set # functions and classes. if tok.text in ['set', 'frozenset']: self.userDefinedSet = True if tok.type != WS: self.prevTokText = tok.text return def __handleKeywords( self, currentFcn, currentClass, tok, *args, **kwds ): """ Check for generator functions or expressions and list comprehension.""" if tok.text == "yield": self.isGeneratorFunction = True elif tok.text == "for": if len( self.stackParenOrBracket ) > 0: punct = self.stackParenOrBracket[-1] if punct == '(': self.featureMetrics['numGeneratorExpressions'] += 1 elif punct == '[': self.featureMetrics['numListComprehensions'] += 1 return def processStmt( self, fcnName, className, stmt, *args, **kwds ): """ Handle processing at end of statement.""" self.stackParenOrBracket = [] def processFunction( self, fcnName, className, block, *args, **kwds ): """ Output stats at end of each function.""" if self.isGeneratorFunction: self.featureMetrics['numGeneratorFunctions'] += 1 self.isGenerator = False if self.usedInbuiltSets: self.featureMetrics['numModulesUsingSets'] += 1 self.usedInbuiltSets = False def processModule( self, moduleName, mod, *args, **kwds ): """ Output stats at end of each module.""" self.moduleName = moduleName if self.featureMetrics['numModulesUsingSets'] > 0: self.runMetrics['modulesUsingSets'].append( moduleName ) if self.featureMetrics['numGeneratorFunctions'] > 0: self.runMetrics['modulesUsingGeneratorFunctions'].append( moduleName ) if self.featureMetrics['numGeneratorExpressions'] > 0: self.runMetrics['modulesUsingGeneratorExpressions'].append( moduleName ) if self.featureMetrics['numListComprehensions'] > 0: self.runMetrics['modulesUsingListComprehension'].append( moduleName ) if self.featureMetrics['numDecorators'] > 0: self.runMetrics['modulesUsingDecorators'].append( moduleName ) return def processRun( self, run, *args, **kwds ): """ Output stats at end of run.""" def __printHeader( printHeader ): """ Only print heading if something in body of report.""" if printHeader: print """Python 2.4 Features Used During Run""" print """-----------------------------------""" return False def __printSubHeader( printHeader, key, desc ): if len( self.runMetrics[key] ) > 0: printHeader = __printHeader( printHeader ) h1 = "Modules using %s" % desc h2 = '.'*len( h1 ) print print h1 print h2 print for modName in self.runMetrics[key]: print modName return False printHeader = True printHeader = __printSubHeader( printHeader, 'modulesUsingSets', 'builtin set/frozenset (PEP 218)' ) printHeader = __printSubHeader( printHeader, 'modulesUsingGeneratorFunctions', 'Generator Functions' ) printHeader = __printSubHeader( printHeader, 'modulesUsingGeneratorExpressions (PEP 289)', 'Generator Expressions' ) printHeader = __printSubHeader( printHeader, 'modulesUsingListComprehension', 'List Comprehension' ) printHeader = __printSubHeader( printHeader, 'modulesUsingDecorators', 'Decorators (PEP 318)' ) return None def compute( self, *args, **kwds ): """ Compute any values needed.""" try: self.featureMetrics['%FunctionsThatAreGenerators'] = 100.0 * self.featureMetrics['numGeneratorFunctions']/self.metrics['numFunctions'] except (KeyError, ZeroDivisionError): self.featureMetrics['%FunctionsThatAreGenerators'] = 0.0 return self.featureMetrics def display( self, currentFcn=None ): """ Display and return new features in 2.4 metrics for given function.""" def __printDisplayHdr(): """ Only print heading if something in body of report.""" h1 = """Python 2.4 Features Used in %s""" % self.inFile h2 = '-'*len( h1 ) print h1 print h2 print return False printHeader = True self.compute() keyList = self.featureMetrics.keys() keyList.sort() for k in keyList: if self.pa.zeroSw or self.featureMetrics[k]: fmt = ( k[0] == '%' and "%14.2f %s" ) or "%11d %s" if printHeader: printHeader = __printDisplayHdr() print fmt % (self.featureMetrics[k],k) print return self.featureMetrics if __name__=="__main__": def Incr( init ): for i in range(10): yield i s = set(i for i in range(0,10,2)) fs = frozenset(i for i in range(5)) print s, fs pymetrics-0.8.1/PyMetrics/sqldataout.py0000664000076400007640000000473610664101030016340 0ustar regreg""" SqlDataOut - class to produce SQL data command file output. $Id: SqlDataOut.py,v 1.2 2005/02/15 07:08:58 rcharney Exp $ """ __revision__ = "$Revision: 1.2 $"[11:-2] __author__ = 'Reg. Charney ' import sys import time import token import tokenize import utils import sqltemplate class InvalidTableNameError( Exception ): """ Used to indicate that the SQL table name is invalid.""" pass class SqlDataOut( object ): """ Class used to generate a command file suitable for runnning against any SQL dbms.""" def __init__( self, fd, libName, fileName, tableName, genNewSw=False, genExistsSw=False ): """ Initialize instance of SqlDataOut.""" if tableName == '': raise InvalidTableNameError( tableName ) if not fd: raise IOError( "Output file does not yet exist" ) timestamp = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()) self.libName = libName self.fileName = fileName self.tableName = tableName self.quotedFileName = '"'+self.fileName+'"' self.IDDateTime = '"'+timestamp+'"' self.toknum = 0 self.fd = fd if not genExistsSw: self.writeHdr( genNewSw, tableName ) def writeHdr( self, genNewSw, tableName ): """ Write header information for creating SQL command file.""" if genNewSw: import re r = re.compile( '\w+' ) if r.match( tableName ): self.fd.write( sqltemplate.dataHdr % (tableName, tableName, tableName, tableName) ) else: raise AttributeError( 'Invalid table name' ) def write( self, metricName, srcFileName, varName, value ): """ Generate the Sql INSERT line into the sql command file.""" sArgs = ','.join( ( self.IDDateTime, '0', '"'+str( self.libName )+'"', '"'+str( metricName )+'"', '"'+str( srcFileName )+'"', '"'+str( varName )+'"', '"'+str( value )+'"' ) ) sOut = sqltemplate.dataInsert % (self.tableName, sArgs) self.fd and self.fd.write( sOut ) def close( self ): """ Close file, if it is opened.""" self.fd and self.fd.close() self.fd = None pymetrics-0.8.1/PyMetrics/mytoken.py0000664000076400007640000000221311041771745015651 0ustar regreg""" MyToken - PyMetrics' version of Token. $Id: mytoken.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.3 $"[11:-2] __author__ = 'Reg. Charney ' import token from globals import * class MyToken: def __init__(self, **kwds ): """ Initialize class with user-defined keywords.""" self.__dict__.update(kwds) def __repr__( self ): """ Pretty print token. Don't print text for special token types since they do not have useful visible representation, other than blank. """ tn = token.tok_name[self.type] sn = self.semtype if sn: sn = token.tok_name[self.semtype] if self.type in [WS,NEWLINE,INDENT,DEDENT,EMPTY,ENDMARKER]: s = "[type=%s semtype=%s row=%s col=%s len=%d]" % (tn,sn,self.row,self.col,len(self.text)) else: if self.type == COMMENT and self.text[-1] == '\n': self.text = self.text[:-1] s = "[type=%s semtype=%s row=%s col=%s len=%d text=<%s>]" % (tn,sn,self.row,self.col,len(self.text),self.text) return s pymetrics-0.8.1/PyMetrics/sloc.pyc0000664000076400007640000001256111042140435015260 0ustar regregmò ‚ÐFc@sÆdZddd!ZdZdkZdklZdefd„ƒYZd „Zd „Zd „Z d „Z e d joLe ƒ\Z Z x:e D].Zyedjo eee ƒWqŒqŒXqŒWndS(s“ Compute COCOMO 2's SLOC (Source Lines Of Code) metric. Compute Source Lines Of Code for each function/class/file. This is based on COCOMO 2's definition of constitutes a line of code. Algorithm: Delete all non-literal blank lines Delete all non-literal comments Delete all doc strings Combine parenthesised expressions into one logical line Combine continued lines into one logical line Return count of resulting lines Conventions: Continued lines are those ending in \) are treated as one physical line. Paremeter lists and expressions enclosed in parens (), braces {}, or brackets [] are treated as being part of one physical line. All literals are treated as being part of one physical line $Id: sloc.py,v 1.2 2005/06/26 07:08:58 rcharney Exp $ s$Revision: 1.1 $i iþÿÿÿs'Reg. Charney N(s MetricBaset SLOCMetriccBs)tZdZd„Zd„Zd„ZRS(s% Compute Source Lines Of Code metric.c OsYd} d}d}d} d} |d| d|d| |_d|_d|_d |_ d |_ t i | ƒ|_ t i |it it iƒ|_t i |iƒ|_t i |it it iƒ|_t i |i t it iƒ|_t i |i t it iƒ|_d |_||_||_||_||_|d |_h|_dS( Ns \\\'|\\\"s r?'''.*?'''s r?""".*?"""s'.*?'s".*?"t|s#.* s\(.*?\)s^\s*(def|class)\s+\w.*:s^\s* |^\s*@ |^\s*~ itinFile(tpatBackslashQuotetpatTripleQuote1tpatTripleQuote2t patQuote1t patQuote2tselft patLiteralt patCommenttpatParent patDefOrClasst patAllBlanktretcompiletreBackslashQuotetMtSt reLiteralt reCommenttreParent reDefOrClasst reAllBlanktnumSLOCtcontextt runMetricstmetricstpaRtfcnNames( RRRRRtargstkwdsRRRRR((tPyMetrics/sloc.pyt__init__#s.!    """"      c Os²t|ƒdjod|_dSnti|id|ƒ}ti|id|ƒ}ti|i d|ƒ}ti|i d|ƒ}ti|id|ƒ}|idƒ|_dS( s£ Process all raw source lines in one fell swoop. If a given line is not blank and contains something besides a comment, then increment the numSLOC.iNt*t@s~ t%s(...)s (tlentrawLinesRRRtsubRtnoBackslashQuotesRtnoLiteralLinesRtnoCommentLinesRt noBlankLinesRt noParenLinestcount( RR&RRR)R,R+R(R*((R tprocessSrcLines?s cCsjh}|i||i<|iio|Snd|i}|GHdt|ƒdGHd|i|ifGHH|S(s# Display SLOC metric for each file s COCOMO 2's SLOC Metric for %st-s s %11d %sN(tresultRRRRtquietSwthdrR%(RR0R2((R tdisplayPs  (t__name__t __module__t__doc__R!R.R3(((R R!s   cCsdt|ƒ}|iƒ}|iƒdfd„ƒY}|ƒ}t|_ ||_ t |||ƒdS(s Process lines from input file.tPAcBstZRS(N(R4R5(((R R7fsN( topentfntfdtreadR&tcloseR7RtTrueR1tdebugSwt __processFile(R9R>R&RR7R:((R t__main`s      cCs7thd|tfnListtargvtargR=R8R:R;tlinesR<t__normalizeLinestappend(RDR>RGRBR:RF((R t __getFNListss"      cCsOg}t|ƒdjo2|ddjo|d }n|idƒ}n|S(sO Remove trailing newlines, if needed, and return list of input file names.iiÿÿÿÿs N(RDR%RGtsplit(RGRD((R RH‡st__main__t(R6t __revision__t __author__Rt metricbaset MetricBaseRR@R?RJRHR4RDR>R9( RDRNRRQR>RJRORR@R?RHR9((R t?s"   ?   pymetrics-0.8.1/PyMetrics/examples/0000775000076400007640000000000010665706437015440 5ustar regregpymetrics-0.8.1/PyMetrics/examples/sample.py0000664000076400007640000000361010665706437017273 0ustar regreg""" This is script doc string. """ g = 1 # block 1 at level 0 # this is a comment that starts a line after indent # this comment starts in column 1 h=2 # another inline comment # see if this comment is preceded by whitespace def f(a,b,c): # function with multiple returns """ This is multi-line function doc string. """ if a > b: # block 2 at level 1 if b > c: # block 3 at level 2 a = c # block 4 at level 3 else: # block 3 at level 2 return # block 4 at level 3 - explicit return print a # block 2 at level 1 # implicit return f(1,2,3) # block 1 at level 0 [f(i,3,5) for i in range(5)] def g(a,b): a += 1 if a > 'c': print "a is larger" def h(b): return b*b return h(b) g(3,2) v = 123 # this function definition shows that parameter names are not # defined until after the parameter list is completed. That is, # it is invalid to define: def h(a,b=a): ... def h(a, b=v,c=(v,),d={'a':(2,4,6)},e=[3,5,7]): print "b: a=%s b=%s" % (a,b) def k( a , b ) : c = a d = b if c > d: return d def kk( c ): return c*c return kk(c+d) class C: """ This is class doc string.""" def __init__( self ): """ This is a member function doc string""" if 1: return elif 0: return class D (object): def dd( self, *args, **kwds ): return args, kwds def main(): "This is single line doc string" h(3) c = C() g(3,2) f(7,5,3) # next, test if whitespace affects token sequences h ( 'a' , 'z' ) [ 1 : ] # next, test tokenization for statement that crosses lines h ( 'b' , 'y' ) [ 1 : ] if __name__ == "__main__": main() pymetrics-0.8.1/PyMetrics/examples/commentDedent.py0000664000076400007640000000171210665706437020601 0ustar regreg""" Simple test of effect of exdented comment which changed block depth incorrectly.""" def foo1(): a = "Test 1 OK" # Comment 1 should NOT affect indentation if a: # should work, but ends function if comment affects indentation print a def foo2(): if True: # Comment 2 should NOT affect indentation return "Test 2 OK" def foo3(): a = "Test 3 OK" # Comment 2 should NOT affect indentation if a: # next line in error if comment affect indentation return a def foo4(): # this comment at start of function should no effect compilation if True: return "Test 4 OK" # this command should also not effect compilation return "Test 4 Failed" foo1() print foo2() print foo3() print foo4() if True: # this comment should not effect the block structure of the program print "Test 5 OK" else: # this comment should not effect program compilation or execution print "Test 5 failed!!!" pymetrics-0.8.1/PyMetrics/utils.py0000664000076400007640000000367310664101030015316 0ustar regreg""" Utility functions used throughout the PyMetrics system. $Id: utils.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ """ import sys import token import re def sqlQ( s ): """ Place single quotes around strings and escaping existing single quotes.""" a = s.replace( "\\","\\\\" ) a = a.replace( "'", "\\'" ) a = a.replace( '"', '\\"' ) return '"'+a+'"' def csvQ( s ): """ Quote a string using rules for CSV data.""" a = s.replace("\\","\\\\") b = a.replace( "'", "\\'" ) c = b.replace( "\n", "\\n" ) d = c.replace( '"', '""' ) return '"'+d+'"' def toTypeName( context, lst ): """ Convert token type numbers to names.""" lstOut = [] for name,blockDepth,semtype in lst: try: semName = token.tok_name[semtype] lstOut.append( (name,blockDepth,semName) ) except KeyError, e: raise KeyError( "Unknown value '"+str( e )+"' for token/semantic type in context %s\n" % context ) return lstOut if 0: def mainTest(): """ Built-in tests """ def check( qs, s ): print "<%s>==<%s>" % (s.__repr__(),qs.__repr__()) print "[%s]==[%s]" % (s,qs) try: assert( s.__repr__() == qs.__repr__() ) assert( s, qs ) except AssertionError: print "Failed" s0 = ''; qs0 = sqlQ(s0) check( qs0, '""' ) s1 = 'aName'; qs1 = sqlQ(s1) check( qs1, '"aName"' ) s2 = 'A literal with a double quote (\") in it'; qs2 = sqlQ( s2 ) check( qs2, '"A literal with a double quote (\\\") in it"' ) s3 = '\'A literal with a single quote (\') in it\''; qs3 = sqlQ( s3 ) check( qs3, '"\\\'A literal with a single quote (\\\') in it\\\'"' ) s4 = """A multi- line literal."""; qs4 = sqlQ( s4 ) check( qs4, '"A multi-\nline literal."' ) if __name__ == "__main__": mainTest() pymetrics-0.8.1/PyMetrics/sqltokenout.py0000664000076400007640000000607410664101030016544 0ustar regreg""" SqlTokenOut - class to produce SQL Token command file output. $Id: SqlTokenOut.py,v 1.2 2005/02/15 07:08:58 rcharney Exp $ """ __revision__ = "$Revision: 1.2 $"[11:-2] __author__ = 'Reg. Charney ' import sys import time import token import tokenize from utils import * import string import sqltemplate class InvalidTableNameError( Exception ): pass class SqlTokenOut( object ): """ Class used to generate a command file suitable for runnning against any SQL dbms.""" def __init__( self, fd, libName, fileName, tableName, genNewSw=False, genExistsSw=False ): if tableName == '': raise InvalidTableNameError( tableName ) self.libName = libName self.fileName = fileName self.tableName = tableName self.quotedFileName = '"'+self.fileName+'"' self.IDDateTime = '"'+time.strftime("%Y-%m-%d %H:%M:%S",time.localtime())+'"' self.toknum = 0 self.fd = fd if not genExistsSw: self.writeHdr( genNewSw, tableName ) def writeHdr( self, genNewSw, tableName ): """ Write header information for creating SQL command file.""" if genNewSw: self.fd.write( sqltemplate.tokenHdr % (tableName,tableName,tableName,tableName) ) def write( self, context, tok, fqnFunction, fqnClass ): """ Generate the Sql INSERT line into the sql command file.""" self.toknum += 1 txt = tok.text tt = tok.type tn = token.tok_name[tt] if tt == token.NEWLINE or tt == tokenize.NL: txt = r'\n' sn = self.__formSemanticName(tok) sArgs = self.__formArgString(context, tok, tn, sn, txt, fqnFunction, fqnClass) sOut = sqltemplate.tokenInsert % (self.tableName, sArgs) self.fd.write( sOut ) def __formSemanticName(self, tok): """ Form semantic name by decoding semtype.""" sn = '' if tok.semtype: sn = token.tok_name[tok.semtype] return sn def __formArgString(self, context, tok, tn, sn, txt, fqnFunction, fqnClass): """ Generate arguments string for use in write method.""" sArgs = ','.join( ( self.IDDateTime, str( self.toknum ), '"'+str( self.libName )+'"', '"'+str( context['inFile'] )+'"', str( tok.row ), str( tok.col ), '"'+tn+'"', '"'+sn+'"', str( len( txt ) ), sqlQ( txt ), '"'+str( toTypeName( context, fqnFunction ) )+'"', '"'+str( toTypeName( context, fqnClass ) )+'"', str( context['blockNum'] ), str( context['blockDepth'] ), str( context['fcnDepth'] ), str( context['classDepth'] ), str( context['parenDepth'] ), str( context['bracketDepth'] ), str( context['braceDepth'] ) ) ) return sArgs def close( self ): """ Close file, if it is opened.""" self.fd and self.fd.close() self.fd = None pymetrics-0.8.1/PyMetrics/csvout.py0000664000076400007640000000636611041771745015523 0ustar regreg""" CvsOut - class to produce CVS data output. $Id: csvout.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.3 $"[11:-2] __author__ = 'Reg. Charney ' import sys import time import token import tokenize from utils import * class CsvOut( object ): """ Class used to generate a CSV data file suitable for input to spreadsheet program.""" def __init__( self, fileName, genHdrSw=True, genNewSw=False ): """ Open output file and generate header line, if desired.""" self.fileName = fileName self.quotedFileName = '"'+self.fileName+'"' self.IDDateTime = '"'+time.strftime("%Y-%m-%d %H:%M:%S",time.localtime())+'"' self.toknum = 0 mode = "a" if genNewSw: mode = "w" try: if self.fileName: self.fd = open( fileName, mode ) else: self.fd = sys.stdout except IOError: raise self.writeHdr( genNewSw, genHdrSw ) def writeHdr( self, genNewSw, genHdrSw ): """ Write header information for CSV file.""" if genNewSw and genHdrSw: fldNames = [ '"IDDateTime"', '"tokNum"', '"inFile"', '"line"', '"col"', '"tokType"', '"semType"', '"tokLen"', '"token"', '"fqnFunction"', '"fqnClass"', '"blockNum"', '"blockDepth"', '"fcnDepth"', '"classDepth"', '"parenDepth"', '"bracketDepth"', '"braceDepth"' ] self.fd.write( ','.join( fldNames ) ) self.fd.write( '\n' ) def close( self ): """ Close output file if it is not stdout. """ if self.fileName: self.fd.flush() self.fd.close() def write( self, context, tok, fqnFunction, fqnClass ): """ Generate the CSV data line.""" self.toknum += 1 txt = tok.text tt = tok.type tn = token.tok_name[tt] sn = '' if tok.semtype: sn = token.tok_name[tok.semtype] if tt == token.NEWLINE or tt == tokenize.NL: txt = r'\n' sArgs = ','.join( ( self.IDDateTime, str( self.toknum ), '"'+str( context['inFile'] )+'"', str( tok.row ), str( tok.col ), '"'+tn+'"', '"'+sn+'"', str( len( txt ) ), csvQ( txt ), '"'+str( toTypeName( context, fqnFunction ) )+'"', '"'+str( toTypeName( context, fqnClass ) )+'"', str( context['blockNum'] ), str( context['blockDepth'] ), str( context['fcnDepth'] ), str( context['classDepth'] ), str( context['parenDepth'] ), str( context['bracketDepth'] ), str( context['braceDepth'] ) ) ) self.fd.write( sArgs ) self.fd.write( '\n' ) def close( self ): """ Close file, if it is opened.""" self.fd and self.fd.close() self.fd = None pymetrics-0.8.1/PyMetrics/sqltokenout.pyc0000664000076400007640000000722211042143023016702 0ustar regregmò ‚ÐFc@s†dZddd!ZdZdkZdkZdkZdkZdkTdkZdk Z de fd„ƒYZ d e fd „ƒYZ dS( s SqlTokenOut - class to produce SQL Token command file output. $Id: SqlTokenOut.py,v 1.2 2005/02/15 07:08:58 rcharney Exp $ s$Revision: 1.2 $i iþÿÿÿs'Reg. Charney N(t*tInvalidTableNameErrorcBstZRS(N(t__name__t __module__(((t;/home/reg/pymetrics/branches/0.8.0/PyMetrics/sqltokenout.pyRst SqlTokenOutcBsJtZdZeed„Zd„Zd„Zd„Zd„Zd„Z RS(sV Class used to generate a command file suitable for runnning against any SQL dbms.cCs |djot|ƒ‚n||_||_||_d|id|_dtidtiƒƒd|_ d|_ ||_ |p|i ||ƒndS(Ntt"s%Y-%m-%d %H:%M:%Si(t tableNameRtlibNametselftfileNametquotedFileNamettimetstrftimet localtimet IDDateTimettoknumtfdt genExistsSwtwriteHdrtgenNewSw(R RR R RRR((Rt__init__s    #  cCs2|o'|iiti||||fƒndS(s8 Write header information for creating SQL command file.N(RR Rtwritet sqltemplatettokenHdrR(R RR((RR%sc Cs²|id7_|i}|i} ti| }| ti jp| t i jo d}n|i |ƒ}|i|||||||ƒ}ti|i|f} |ii| ƒdS(s8 Generate the Sql INSERT line into the sql command file.is\nN(R Rttokttextttxtttypetttttokenttok_namettntNEWLINEttokenizetNLt_SqlTokenOut__formSemanticNametsnt_SqlTokenOut__formArgStringtcontextt fqnFunctiontfqnClasstsArgsRt tokenInsertRtsOutRR( R R(RR)R*R+R!R&RRR-((RR*s     !cCs(d}|ioti|i}n|S(s( Form semantic name by decoding semtype.RN(R&RtsemtypeRR (R RR&((Rt__formSemanticName8s  c Cs'di|it|iƒdt|iƒddt|dƒdt|iƒt|i ƒd|dd|dtt |ƒƒt|ƒdtt||ƒƒddtt||ƒƒdt|dƒt|dƒt|dƒt|dƒt|dƒt|d ƒt|d ƒfƒ}|Sd S( s3 Generate arguments string for use in write method.t,RtinFiletblockNumt blockDepthtfcnDeptht classDeptht parenDeptht bracketDeptht braceDepthN(tjoinR RtstrRR R(RtrowtcolR!R&tlenRtsqlQt toTypeNameR)R*R+( R R(RR!R&RR)R*R+((Rt__formArgString?sÿ cCs$|io |iiƒd|_dS(s Close file, if it is opened.N(R RtclosetNone(R ((RRAXs( RRt__doc__tFalseRRRR%R'RA(((RRs     (RCt __revision__t __author__tsysR RR#tutilststringRt ExceptionRtobjectR( RERIRR RFRGRR#RR((Rt?s       pymetrics-0.8.1/PyMetrics/metricbase.py0000664000076400007640000000306611041771745016310 0ustar regreg""" Metric base class for new user-defined metrics. $Id: metricbase.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ """ __version__ = "$Revision: 1.2 $"[11:-2] __author__ = 'Reg. Charney ' class MetricBase( object ): """ Metric template class.""" def __init__( self, *args, **kwds ): pass def processSrcLines( self, srcLines, *args, **kwds ): """ Handle physical line after tab expansion.""" pass def processToken( self, fcnName, className, tok, *args, **kwds ): """ Handle processing after each token processed.""" pass def processStmt( self, fcnName, className, stmt, *args, **kwds ): """ Handle processing at end of statement.""" pass def processBlock( self, fcnName, className, block, *args, **kwds ): """ Handle processing at end of block.""" pass def processFunction( self, fcnName, className, fcn, *args, **kwds ): """ Handle processing at end of function. """ pass def processClass( self, fcnName, className, cls, *args, **kwds ): """ Handle processing at end of class. """ pass def processModule( self, moduleName, module, *args, **kwds ): """ Handle processing at end of module. """ pass def processRun( self, run, *args, **kwds ): """ Handle processing at end of run. """ pass def compute( self, *args, **kwds ): """ Compute the metric given all needed info known.""" pass pymetrics-0.8.1/PyMetrics/sqldataout.pyc0000664000076400007640000000551111042143023016472 0ustar regregmò ‚ÐFc@sdZddd!ZdZdkZdkZdkZdkZdkZdkZde fd„ƒYZ de fd „ƒYZ dS( s~ SqlDataOut - class to produce SQL data command file output. $Id: SqlDataOut.py,v 1.2 2005/02/15 07:08:58 rcharney Exp $ s$Revision: 1.2 $i iþÿÿÿs'Reg. Charney NtInvalidTableNameErrorcBstZdZRS(s5 Used to indicate that the SQL table name is invalid.(t__name__t __module__t__doc__(((t:/home/reg/pymetrics/branches/0.8.0/PyMetrics/sqldataout.pyRs t SqlDataOutcBs8tZdZeed„Zd„Zd„Zd„ZRS(sV Class used to generate a command file suitable for runnning against any SQL dbms.cCs½|djot|ƒ‚n|ptdƒ‚ntidtiƒƒ}||_||_ ||_d|i d|_ d|d|_ d|_ ||_|p|i||ƒndS(s# Initialize instance of SqlDataOut.tsOutput file does not yet exists%Y-%m-%d %H:%M:%St"iN(t tableNameRtfdtIOErrorttimetstrftimet localtimet timestamptlibNametselftfileNametquotedFileNamet IDDateTimettoknumt genExistsSwtwriteHdrtgenNewSw(RR RRRRRR((Rt__init__s      cCsj|o_dk}|idƒ}|i|ƒo'|iit i ||||fƒqft dƒ‚ndS(s8 Write header information for creating SQL command file.Ns\w+sInvalid table name( RtretcompiletrtmatchRRR twritet sqltemplatetdataHdrtAttributeError(RRRRR((RR0s 'c Cs¤di|iddt|iƒddt|ƒddt|ƒddt|ƒddt|ƒdfƒ}t i |i |f}|io|ii|ƒdS(s8 Generate the Sql INSERT line into the sql command file.t,t0RN(tjoinRRtstrRt metricNamet srcFileNametvarNametvaluetsArgsRt dataInsertRtsOutR R(RR%R&R'R(R)R+((RR=sp cCs$|io |iiƒd|_dS(s Close file, if it is opened.N(RR tclosetNone(R((RR,Ks(RRRtFalseRRRR,(((RRs  ( Rt __revision__t __author__tsysR ttokenttokenizetutilsRt ExceptionRtobjectR( R/R4R RR0R1R2R3RR((Rt?s       pymetrics-0.8.1/PyMetrics/tokenize.py0000664000076400007640000003021010665433042016005 0ustar regreg"""Tokenization help for Python programs. generate_tokens(readline) is a generator that breaks a stream of text into Python tokens. It accepts a readline-like method which is called repeatedly to get the next line of input (or "" for EOF). It generates 5-tuples with these members: the token type (see token.py) the token (a string) the starting (row, column) indices of the token (a 2-tuple of ints) the ending (row, column) indices of the token (a 2-tuple of ints) the original line (string) It is designed to match the working of the Python tokenizer exactly, except that it produces COMMENT tokens for comments and gives type OP for all operators Older entry points tokenize_loop(readline, tokeneater) tokenize(readline, tokeneater=printtoken) are the same, except instead of generating tokens, tokeneater is a callback function to which the 5 fields described above are passed as 5 arguments, each time a new token is found. $Id: tokenize.py,v 1.3 2005/02/15 07:08:58 rcharney Exp $ """ __version__ = "$Revision: 1.3 $"[11:-2] __author__ = 'Ka-Ping Yee ' __credits__ = \ 'GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip Montanaro' import string, re from token import * import token __all__ = [x for x in dir(token) if x[0] != '_'] + ["COMMENT", "tokenize", "generate_tokens", "NL"] del x del token COMMENT = N_TOKENS tok_name[COMMENT] = 'COMMENT' NL = N_TOKENS + 1 tok_name[NL] = 'NL' N_TOKENS += 2 def group(*choices): return '(' + '|'.join(choices) + ')' def any(*choices): return group(*choices) + '*' def maybe(*choices): return group(*choices) + '?' Whitespace = r'[ \f\t]*' Comment = r'#[^\r\n]*' Ignore = Whitespace + any(r'\\\r?\n' + Whitespace) + maybe(Comment) Name = r'[a-zA-Z_]\w*' Hexnumber = r'0[xX][\da-fA-F]*[lL]?' Octnumber = r'0[0-7]*[lL]?' Decnumber = r'[1-9]\d*[lL]?' Intnumber = group(Hexnumber, Octnumber, Decnumber) Exponent = r'[eE][-+]?\d+' Pointfloat = group(r'\d+\.\d*', r'\.\d+') + maybe(Exponent) Expfloat = r'\d+' + Exponent Floatnumber = group(Pointfloat, Expfloat) Imagnumber = group(r'\d+[jJ]', Floatnumber + r'[jJ]') Number = group(Imagnumber, Floatnumber, Intnumber) # Tail end of ' string. Single = r"[^'\\]*(?:\\.[^'\\]*)*'" # Tail end of " string. Double = r'[^"\\]*(?:\\.[^"\\]*)*"' # Tail end of ''' string. Single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''" # Tail end of """ string. Double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""' Triple = group("[uU]?[rR]?'''", '[uU]?[rR]?"""') # Single-line ' or " string. String = group(r"[uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'", r'[uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"') # Because of leftmost-then-longest match semantics, be sure to put the # longest operators first (e.g., if = came before ==, == would get # recognized as two instances of =). Operator = group(r"\*\*=?", r">>=?", r"<<=?", r"<>", r"!=", r"//=?", r"[+\-*/%&|^=<>]=?", r"~") Bracket = '[][(){}]' Special = group(r'\r?\n', r'[:;.,`]','@') Funny = group(Operator, Bracket, Special) PlainToken = group(Number, Funny, String, Name) Token = Ignore + PlainToken # First (or only) line of ' or " string. ContStr = group(r"[uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" + group("'", r'\\\r?\n'), r'[uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' + group('"', r'\\\r?\n')) PseudoExtras = group(r'\\\r?\n', Comment, Triple) PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name) tokenprog, pseudoprog, single3prog, double3prog = map( re.compile, (Token, PseudoToken, Single3, Double3)) endprogs = {"'": re.compile(Single), '"': re.compile(Double), "'''": single3prog, '"""': double3prog, "r'''": single3prog, 'r"""': double3prog, "u'''": single3prog, 'u"""': double3prog, "ur'''": single3prog, 'ur"""': double3prog, "R'''": single3prog, 'R"""': double3prog, "U'''": single3prog, 'U"""': double3prog, "uR'''": single3prog, 'uR"""': double3prog, "Ur'''": single3prog, 'Ur"""': double3prog, "UR'''": single3prog, 'UR"""': double3prog, 'r': None, 'R': None, 'u': None, 'U': None} triple_quoted = {} for t in ("'''", '"""', "r'''", 'r"""', "R'''", 'R"""', "u'''", 'u"""', "U'''", 'U"""', "ur'''", 'ur"""', "Ur'''", 'Ur"""', "uR'''", 'uR"""', "UR'''", 'UR"""'): triple_quoted[t] = t single_quoted = {} for t in ("'", '"', "r'", 'r"', "R'", 'R"', "u'", 'u"', "U'", 'U"', "ur'", 'ur"', "Ur'", 'Ur"', "uR'", 'uR"', "UR'", 'UR"' ): single_quoted[t] = t tabsize = 8 class TokenError(Exception): pass class StopTokenizing(Exception): pass def printtoken(type, token, (srow, scol), (erow, ecol), line): # for testing print "%d,%d-%d,%d:\t%s\t%s" % \ (srow, scol, erow, ecol, tok_name[type], repr(token)) def tokenize(readline, tokeneater=printtoken): """ The tokenize() function accepts two parameters: one representing the input stream, and one providing an output mechanism for tokenize(). The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. The second parameter, tokeneater, must also be a callable object. It is called once for each token, with five arguments, corresponding to the tuples generated by generate_tokens(). """ try: tokenize_loop(readline, tokeneater) except StopTokenizing: pass # backwards compatible interface def tokenize_loop(readline, tokeneater): for token_info in generate_tokens(readline): tokeneater(*token_info) def generate_tokens(readline): """ The generate_tokens() generator requires one argment, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed is the logical line; continuation lines are included. """ lnum = parenlev = continued = 0 namechars, numchars = string.ascii_letters + '_', '0123456789' contstr, needcont = '', 0 contline = None indents = [0] strstart = (0,0) endprog = re.compile('') while 1: # loop over lines in stream line = readline() lnum = lnum + 1 pos, maxLen = 0, len(line) if contstr: # continued string if not line: raise TokenError, ("EOF in multi-line string", strstart) endmatch = endprog.match(line) if endmatch: pos = end = endmatch.end(0) yield (STRING, contstr + line[:end], strstart, (lnum, end), contline + line) contstr, needcont = '', 0 contline = None elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n': yield (ERRORTOKEN, contstr + line, strstart, (lnum, len(line)), contline) contstr = '' contline = None continue else: contstr = contstr + line contline = contline + line continue elif parenlev == 0 and not continued: # new statement if not line: break column = 0 while pos < maxLen: # measure leading whitespace if line[pos] == ' ': column = column + 1 elif line[pos] == '\t': column = (column/tabsize + 1)*tabsize elif line[pos] == '\f': column = 0 else: break pos = pos + 1 if pos == maxLen: break if line[pos] in '\r\n': # this is empty line yield (NL, line[pos:], (lnum, pos), (lnum, len(line)), line) elif line[pos] == '#': # this is start of comment pass # skip any effect on indentation else: if column > indents[-1]: # count indents or dedents indents.append(column) yield (INDENT, line[:pos], (lnum, 0), (lnum, pos), line) while column < indents[-1]: indents = indents[:-1] yield (DEDENT, '', (lnum, pos), (lnum, pos), line) else: # continued statement if not line: raise TokenError, ("EOF in multi-line statement", (lnum, 0)) continued = 0 while pos < maxLen: pseudomatch = pseudoprog.match(line, pos) if pseudomatch: # scan for tokens start, end = pseudomatch.span(1) spos, epos, pos = (lnum, start), (lnum, end), end token, initial = line[start:end], line[start] if initial in numchars or \ (initial == '.' and token != '.'): # ordinary number yield (NUMBER, token, spos, epos, line) elif initial in '\r\n': yield (parenlev > 0 and NL or NEWLINE, token, spos, epos, line) elif initial == '#': yield (COMMENT, token, spos, epos, line) elif token in triple_quoted: endprog = endprogs[token] endmatch = endprog.match(line, pos) if endmatch: # all on one line pos = endmatch.end(0) token = line[start:pos] yield (STRING, token, spos, (lnum, pos), line) else: strstart = (lnum, start) # multiple lines contstr = line[start:] contline = line break elif initial in single_quoted or \ token[:2] in single_quoted or \ token[:3] in single_quoted: if token[-1] == '\n': # continued string strstart = (lnum, start) endprog = (endprogs[initial] or endprogs[token[1]] or endprogs[token[2]]) contstr, needcont = line[start:], 1 contline = line break else: # ordinary string yield (STRING, token, spos, epos, line) elif initial in namechars: # ordinary name yield (NAME, token, spos, epos, line) elif initial == '\\': # continued stmt continued = 1 else: if initial in '([{': parenlev = parenlev + 1 elif initial in ')]}': parenlev = parenlev - 1 yield (OP, token, spos, epos, line) else: #yield (WS, line[pos], (lnum, pos), (lnum, pos+1), line) while (pos < maxLen) and (line[pos] in ' \t'): pos += 1 if pos < maxLen: yield (ERRORTOKEN, line[pos], (lnum, pos), (lnum, pos+1), line) pos += 1 for indent in indents[1:]: # pop remaining indent levels yield (DEDENT, '', (lnum, 0), (lnum, 0), '') yield (ENDMARKER, '', (lnum, 0), (lnum, 0), '') if __name__ == '__main__': # testing import sys if len(sys.argv) > 1: tokenize(open(sys.argv[1]).readline) else: tokenize(sys.stdin.readline) pymetrics-0.8.1/PyMetrics/csvout.pyc0000664000076400007640000000662511042143023015643 0ustar regregmò åó‡Hc@s^dZddd!ZdZdkZdkZdkZdkZdkTdefd„ƒYZ dS( si CvsOut - class to produce CVS data output. $Id: csvout.py,v 1.3 2005/09/17 04:28:12 rcharney Exp $ s$Revision: 1.3 $i iþÿÿÿs'Reg. Charney N(t*tCsvOutcBsAtZdZeed„Zd„Zd„Zd„Zd„ZRS(sR Class used to generate a CSV data file suitable for input to spreadsheet program.cCs½||_d|id|_dtidtiƒƒd|_d|_d}|o d}ny0|iot ||ƒ|_ n t i |_ Wntj o ‚nX|i||ƒdS(s7 Open output file and generate header line, if desired.t"s%Y-%m-%d %H:%M:%SitatwN(tfileNametselftquotedFileNamettimetstrftimet localtimet IDDateTimettoknumtmodetgenNewSwtopentfdtsyststdouttIOErrortwriteHdrtgenHdrSw(RRRRR ((t6/home/reg/pymetrics/branches/0.8.0/PyMetrics/csvout.pyt__init__s #   cCs{|op|oiddddddddd d d d d dddddg}|iidi|ƒƒ|iidƒndS(s' Write header information for CSV file.s "IDDateTime"s"tokNum"s"inFile"s"line"s"col"s "tokType"s "semType"s"tokLen"s"token"s "fqnFunction"s "fqnClass"s "blockNum"s "blockDepth"s "fcnDepth"s "classDepth"s "parenDepth"s"bracketDepth"s "braceDepth"t,s N(RRtfldNamesRRtwritetjoin(RRRR((RR$s <cCs,|io|iiƒ|iiƒndS(s( Close output file if it is not stdout. N(RRRtflushtclose(R((RR>s  c Cs«|id7_|i}|i} ti| }d}|i oti|i }n| ti jp | t ijo d}ndi|it|iƒdt|dƒdt|iƒt|iƒd|dd|dtt|ƒƒt|ƒdtt||ƒƒddtt||ƒƒdt|dƒt|dƒt|d ƒt|d ƒt|d ƒt|d ƒt|d ƒfƒ}|ii|ƒ|iidƒdS(s Generate the CSV data line.its\nRRtinFiletblockNumt blockDepthtfcnDeptht classDeptht parenDeptht bracketDeptht braceDepths N(RR ttokttextttxtttypetttttokenttok_namettntsntsemtypetNEWLINEttokenizetNLRR tstrtcontexttrowtcoltlentcsvQt toTypeNamet fqnFunctiontfqnClasstsArgsRR( RR5R'R;R<R=R.R/R)R+((RRDs      ÿ cCs$|io |iiƒd|_dS(s Close file, if it is opened.N(RRRtNone(R((RRhs( t__name__t __module__t__doc__tTruetFalseRRRR(((RRs    $( RAt __version__t __author__RRR,R2tutilstobjectR(RRRERR,R2RD((Rt?s     pymetrics-0.8.1/PyMetrics/sloc.py0000664000076400007640000001167310664101030015115 0ustar regreg""" Compute COCOMO 2's SLOC (Source Lines Of Code) metric. Compute Source Lines Of Code for each function/class/file. This is based on COCOMO 2's definition of constitutes a line of code. Algorithm: Delete all non-literal blank lines Delete all non-literal comments Delete all doc strings Combine parenthesised expressions into one logical line Combine continued lines into one logical line Return count of resulting lines Conventions: Continued lines are those ending in \) are treated as one physical line. Paremeter lists and expressions enclosed in parens (), braces {}, or brackets [] are treated as being part of one physical line. All literals are treated as being part of one physical line $Id: sloc.py,v 1.2 2005/06/26 07:08:58 rcharney Exp $ """ __revision__ = "$Revision: 1.1 $"[11:-2] __author__ = 'Reg. Charney ' import re from metricbase import MetricBase class SLOCMetric( MetricBase ): """ Compute Source Lines Of Code metric.""" def __init__( self, context, runMetrics, metrics, pa, *args, **kwds ): # pattern definitions patBackslashQuote = r'\\\'|\\\"' patTripleQuote1 = r"r?'''.*?'''" patTripleQuote2 = r'r?""".*?"""' patQuote1 = r"'.*?'" patQuote2 = r'".*?"' self.patLiteral = patTripleQuote1 + '|' + patQuote1 + '|' + patTripleQuote2 + '|' + patQuote2 self.patComment = '#.*\n' self.patParen = '\(.*?\)' self.patDefOrClass = '^\s*(def|class)\s+\w.*:' self.patAllBlank = '^\s*\n|^\s*@\n|^\s*~\n' # compile patterns self.reBackslashQuote = re.compile( patBackslashQuote ) self.reLiteral = re.compile( self.patLiteral, re.M + re.S ) self.reComment = re.compile( self.patComment ) self.reParen = re.compile( self.patParen, re.M + re.S ) self.reDefOrClass = re.compile( self.patDefOrClass, re.M + re.S ) self.reAllBlank = re.compile( self.patAllBlank, re.M + re.S ) self.numSLOC = 0 self.context = context self.runMetrics = runMetrics self.metrics = metrics self.pa = pa self.inFile = context['inFile'] self.fcnNames = {} def processSrcLines( self, rawLines, *args, **kwds ): """ Process all raw source lines in one fell swoop. If a given line is not blank and contains something besides a comment, then increment the numSLOC.""" if len( rawLines ) == 0: # ignore file self.numSLOC = 0 return noBackslashQuotes = re.sub( self.reBackslashQuote, '*', rawLines ) noLiteralLines = re.sub( self.reLiteral, '@', noBackslashQuotes ) noCommentLines = re.sub( self.reComment, '~\n', noLiteralLines ) noBlankLines = re.sub( self.reAllBlank, '%', noCommentLines ) noParenLines = re.sub( self.reParen, '(...)', noBlankLines ) self.numSLOC = noParenLines.count( '\n' ) return def display( self ): """ Display SLOC metric for each file """ result = {} result[self.inFile] = self.numSLOC if self.pa.quietSw: return result hdr = "\nCOCOMO 2's SLOC Metric for %s" % self.inFile print hdr print "-"*len(hdr) + "\n" print "%11d %s" % (self.numSLOC,self.inFile) print return result def __main( fn, debugSw ): """ Process lines from input file.""" fd = open( fn ) rawLines = fd.read() fd.close() class PA: pass pa = PA() pa.quietSw = True pa.debugSw = debugSw __processFile(fn, pa, rawLines) def __processFile(fn, pa, rawLines): """ Invoke metric function after setting up scaffolding.""" sloc = SLOCMetric( {'inFile':fn},[],pa ) sloc.processSrcLines( rawLines ) print sloc.display() def __getFNList(): """ Return list of input file names, regardless of whether the list came from a file or the command line.""" import sys debugSw = False fnList = [] for arg in sys.argv[1:]: if arg == '-d': debugSw = True continue if arg == '-f': fd = open(sys.argv[2]) lines = fd.read() fd.close() fnList = __normalizeLines( lines ) continue fnList.append( arg ) return fnList,debugSw def __normalizeLines( lines ): """ Remove trailing newlines, if needed, and return list of input file names.""" fnList = [] if len( lines ) > 0: if lines[-1] == '\n': lines = lines[:-1] fnList = lines.split( '\n' ) return fnList if __name__ == "__main__": fnList,debugSw = __getFNList() for fn in fnList: try: fn!='' and __main( fn, debugSw ) except: pass pymetrics-0.8.1/PyMetrics/metricbase.pyc0000664000076400007640000000440211042140435016431 0ustar regregmò åó‡Hc@s3dZddd!ZdZdefd„ƒYZdS(sr Metric base class for new user-defined metrics. $Id: metricbase.py,v 1.2 2005/09/17 04:28:12 rcharney Exp $ s$Revision: 1.2 $i iþÿÿÿs'Reg. Charney t MetricBasecBshtZdZd„Zd„Zd„Zd„Zd„Zd„Zd„Z d„Z d „Z d „Z RS( s Metric template class.cOsdS(N((tselftargstkwds((tPyMetrics/metricbase.pyt__init__ scOsdS(s* Handle physical line after tab expansion.N((RtsrcLinesRR((RtprocessSrcLines scOsdS(s. Handle processing after each token processed.N((RtfcnNamet classNamettokRR((Rt processTokenscOsdS(s' Handle processing at end of statement.N((RRR tstmtRR((Rt processStmtscOsdS(s# Handle processing at end of block.N((RRR tblockRR((Rt processBlockscOsdS(s' Handle processing at end of function. N((RRR tfcnRR((RtprocessFunctionscOsdS(s$ Handle processing at end of class. N((RRR tclsRR((Rt processClass!scOsdS(s% Handle processing at end of module. N((Rt moduleNametmoduleRR((Rt processModule%scOsdS(s" Handle processing at end of run. N((RtrunRR((Rt processRun)scOsdS(s0 Compute the metric given all needed info known.N((RRR((Rtcompute-s( t__name__t __module__t__doc__RRR R RRRRRR(((RRs          N(Rt __version__t __author__tobjectR(RRR((Rt?s pymetrics-0.8.1/README0000664000076400007640000000067010665637020012555 0ustar regreg******************** README file *********************** For installation instructions, see the INSTALL file. To run PyMetrics, enter either python PyMetrics.py or ./PyMetrics.py It will give you instructions on how to use the program and list its options. The subdirectory "./examples" contains two dreadful test programs. While you can use them, you can also use any of the Python source modules in this directory as examples. pymetrics-0.8.1/RELEASE0000664000076400007640000000031011042002466012655 0ustar regregRelease Notes for PyMetrics 0.7.9 ================================= * Changed distribution method to use Python's standard setup.py method. Thanks to Carlos Garcia Campos (carlosgc-at-gnome.org). pymetrics-0.8.1/setup.py0000664000076400007640000000061411042005141013364 0ustar regreg#!/usr/bin/python from distutils.core import setup setup (name = "PyMetrics", version = "PYMETRICS_VERSION", author = "Reg. Charney", author_email = "pymetrics@charneyday.com", description = "PyMetrics produces metrics for Python programs", url = "http://sourceforge.net/projects/pymetrics/", packages = ['PyMetrics'], scripts = ['pymetrics'] )