././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1733227525.9063692 python-hostlist-2.2.1/0000755000175000017500000000000014723572006013436 5ustar00kentkent././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1461251211.0 python-hostlist-2.2.1/CHANGES0000664000175000017500000000361112706166213014432 0ustar00kentkentThe CHANGES file does not seem to be updated after 1.11. We take the opportunity to start recording changes in the spec file %changelog instead. Version 1.11 (2011-02-04) Add --collect-similar mode to pshbak. Version 1.10 (2010-10-11) Add rewritten version of dbuck. Add some pshbak improvements. Version 1.9 (2010-09-22) Add pshbak, a dshbak replacement. Version 1.8 (2010-09-08) Add dbuck man page. Add --substitute option to hostlist. Version 1.7 (2010-07-08) Add dbuck utility. Version 1.6 (2009-10-02) Add --non-empty and --quiet. Optimize numerically_sorted. Version 1.5 (2009-02-22) Make each "-" on the command line count as one hostlist argument. If multiple hostlists are given on stdin they are combined to a union hostlist before being used in the way requested by the options. Add hostgrep utility to search for lines matching a hostlist. Make the build system (used when building tar.gz and RPMs from the source code held in git) smarter. Version 1.4 (2008-12-28) Support Python 3. Import reduce from functools if possible. Use Python 2/3 installation trick from .../Demo/distutils/test2to3 Version 1.3 (2008-09-30) Add -s/--separator, -p/--prepend, -a/--append and --version options contributed by Pär Andersson at NSC. Let -e be the short form of the --expand option (-w is now deprecated). Add a manual page for hostlist(1). Version 1.2 (2008-09-18) Add "--prefix /usr" in the installation script of the spec file (needed on SUSE Linux where the default is /usr/local). Version 1.1 (2008-09-17) Move the command line utility to a separate 'hostlist' command. Provide a python-hostlist.spec file to build RPM packages. Inspired by a contribution by Dr. Holger Obermaier at Rechenzentrum, Universität Karlsruhe. Version 1.0 (2008-07-25) Initial version. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1222436975.0 python-hostlist-2.2.1/COPYING0000644000175000017500000004310311067164157014474 0ustar00kentkent GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1286805412.0 python-hostlist-2.2.1/MANIFEST.in0000644000175000017500000000024711454613644015201 0ustar00kentkentinclude COPYING include README include CHANGES include python-hostlist.spec include hostlist.1 include hostgrep.1 include pshbak.1 include dbuck.1 include MANIFEST.in ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1733227525.9063692 python-hostlist-2.2.1/PKG-INFO0000644000175000017500000000130714723572006014534 0ustar00kentkentMetadata-Version: 2.1 Name: python-hostlist Version: 2.2.1 Summary: Python module for hostlist handling Home-page: http://www.nsc.liu.se/~kent/python-hostlist/ Author: Kent Engström Author-email: kent@nsc.liu.se License: GPL2+ Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Science/Research Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: GNU General Public License (GPL) Classifier: Topic :: System :: Clustering Classifier: Topic :: System :: Systems Administration Classifier: Programming Language :: Python :: 3 License-File: COPYING The hostlist.py module knows how to expand and collect Slurm hostlist expressions. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/README0000644000175000017500000000357614723572005014330 0ustar00kentkentINTRODUCTION ============ The Python module hostlist.py knows how to expand and collect hostlist expressions, such as those used by Slurm and related programs. Example: % python Python 2.5.1 (r251:54863, Jul 10 2008, 17:24:48) [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import hostlist >>> hosts = hostlist.expand_hostlist("n[1-10,17]") >>> hosts ['n1', 'n2', 'n3', 'n4', 'n5', 'n6', 'n7', 'n8', 'n9', 'n10', 'n17'] >>> hostlist.collect_hostlist(hosts) 'n[1-10,17]' >>> hostlist.collect_hostlist(["x1y1","x2y2", "x1y2", "x2y1"]) 'x[1-2]y[1-2]' Bad hostlists or hostnames will result in the hostlist.BadHostlist exception being raised. The 'hostlist' command is provided to expand/collect hostlists and perform set operations on them. Example: % hostlist n[1-10] n[5-20] n[1-20] % hostlist --difference n[1-10] n[5-20] n[1-4] % hostlist --expand --intersection n[1-10] n[5-20] n5 n6 n7 n8 n9 n10 BUILDING ======== Build RPM packages from the tar.gz archive by running: rpmbuild -ta python-hostlist-2.2.1.tar.gz If you do not have the tar archive, create it first: python setup.py sdist rpmbuild -ta dist/python-hostlist-2.2.1.tar.gz If you have cloned the git repository, you should also be able to build RPM packages using "make rpms". You may also install directly by running the following commands if you have unpacked a tarball: python setup.py build (as yourself) python setup.py install (as root) If you have cloned the git repository, you will have to do make prepare; cd versioned before building/installing via setup.py. RELEASES AND FEEDBACK ===================== You will find new releases at: http://www.nsc.liu.se/~kent/python-hostlist/ If you have questions, suggestions, bug reports or patches, please send them to: kent@nsc.liu.se. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/dbuck0000644000175000017500000004507014723572005014456 0ustar00kentkent#!/usr/bin/env python3 # data aggregation tool - hostlist collected histograms on numerical data __version__ = "2.2.1" # Copyright (C) 2010 Peter Kjellström # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA # 02110-1301, USA. import sys import optparse import math import os import stat import time from pprint import pprint from hostlist import collect_hostlist from hostlist import expand_hostlist from subprocess import getoutput def dprint(dstr): if opts.debug: print("Debug: %s" % dstr) def eprint(estr, terminate=True): sys.stderr.write("Error: %s\n" % estr) if terminate: sys.exit(1) def wprint(wstr): sys.stderr.write("Warning: %s\n" % wstr) def iprint(istr): if opts.verbose: print("Info: %s" % istr) def gettermwidth(): try: cols = int(getoutput("/bin/stty -F /dev/tty size").split()[1]) except Exception: cols = 80 return cols def redboldstr(instr): if (opts.color == 'never' or instr == ""): return instr return "\033[1m\033[31m%s\033[0m" % instr # Opens cache file in read or write mode. Do lots of checks for write case. def opencachefile(fmode): tmpdir = os.path.join(os.environ.get("TMPDIR", "/tmp"), "dbuck-%i" % os.geteuid()) dprint("trying to open cache file in dir %s in mode %s" % (tmpdir, fmode)) if fmode == "w": oldmask = os.umask(0o77) if not os.path.isdir(tmpdir): iprint("creating cache directory: %s" % tmpdir) wprint("this version automatically saves a copy of the last data") wprint("for more information see man page (--previous, --no-cache)") os.mkdir(tmpdir) statstrct = os.stat(tmpdir) if ((stat.S_IWOTH & statstrct.st_mode) or (stat.S_IWGRP & statstrct.st_mode) or (statstrct.st_uid != os.geteuid())): eprint("incorrect permissions on tmpdir: %s" % tmpdir, terminate=False) raise BaseException f = open(tmpdir + "/cache-%i" % os.getsid(os.getpid()), fmode) if fmode == "w": os.umask(oldmask) return f def readdata(cachefile): if opts.previous: if opts.verbose: cachestat = os.stat(cachefile.name) iprint("reading data from cache file created: %s" % time.ctime(cachestat.st_mtime)) try: rawdata = cachefile.readlines() except: eprint("unable to read from cache file") else: try: rawdata = sys.stdin.readlines() except KeyboardInterrupt: iprint("caught keyboard interrupt, exiting...") sys.exit(1) if not opts.no_cache: try: cachefile.writelines(rawdata) cachefile.close() except: wprint("unable to write to cache file") return rawdata ## Statistical functions def mean(list): return sum(list)/len(list) def median(list): tmp = sorted(list) if (len(tmp) % 2) == 0: return (tmp[(len(tmp)//2)-1] + tmp[(len(tmp)//2)]) / 2 else: return tmp[int(len(tmp)//2)] def stdev(list): m = mean(list) return math.sqrt(mean( [ (m - x)**2 for x in list ] )) # clean and refine indata: list of STRING -> list of [ "hostname", float vaule ] def refine_data(rawlist): dprint("read in %i lines of data" % len(rawlist)) # Transform additional separators -> " " for char in opts.field_separators: dprint("adding additional field separator: \"%s\"" % char) for i in range(len(rawlist)): tmp = rawlist[i].replace(char, " ") rawlist[i] = tmp if opts.key == None: if len(rawlist) < 3: eprint("not enough data for auto-detect, please use -k") # list to hold candidates for KEY key = [] # Lets have a look at the last three lines for lnum in range(len(rawlist)-3,len(rawlist)): sline = rawlist[lnum].strip().split() if len(sline) < 2: key.append(0) continue # For anonymous mode start the search at column 0 if opts.anonymous: coloffset = 0 else: coloffset = 1 # The first column that can be converted to a float will be our candidate for i in [ x + coloffset for x in range(len(sline) - coloffset) ]: tmp = None try: tmp = float(sline[i]) except ValueError: pass if tmp != None: dprint("auto-detect row=%i found data at column %i" % (lnum, i)) key.append(i) break dprint("key list after auto-detect: %s" % str(key)) # If more than half of the investigated lines have the same candidate... for candidate in key: if key.count(candidate) > (len(key) // 2): opts.key = candidate iprint("auto-detected data at column: %i" % candidate) break # No winner found (or winner was 'bad line') if opts.key == None or ((opts.key == 0) and not opts.anonymous): eprint("Unable to auto-detect KEY from data") nreject = 0 cleandata = [] for line in rawlist: sline = line.strip().split() tmp = None try: tmp = float(sline[opts.key]) except (ValueError, IndexError): pass if tmp != None: if opts.anonymous: sline[0] = "na" cleandata.append([ sline[0].strip(":"), tmp ]) else: nreject += 1 iprint("rejected line: \"%s\"" % line.strip()) return (cleandata, nreject) def addoverflowbuckets(blist, rmin, vmax): blist.insert(0, {'ub': rmin, 'special': 'underflow'}) blist.append({'ub': vmax, 'special':'overflow'}) return # New bucket creation function def create_buckets_new(valuelist, num, vmin, vmax): dprint("range is %f to %f split into %i buckets" % (vmin, vmax, num)) blist = [] ub = vmin for b in range(num): ub += (vmax-vmin) / num blist.append({'ub': ub}) blist[-1]['ub'] = vmax return blist def parse_range(rstr): rlist = rstr.split("-") dprint("list representation of raw range argument: %s" % str(rlist)) try: while True: rlist[rlist.index('')+1] = '-' + rlist[rlist.index('')+1] rlist.remove('') dprint("processed one \"-\" character in range argument") except ValueError: pass except IndexError: eprint("invalid range specified") dprint("list representation of processed range argument: %s" % str(rlist)) try: rlist = list(map(float, rlist)) except ValueError: eprint("invalid range specified") if (len(rlist) != 2 or rlist[0] >= rlist[1]): eprint("inverted, incomplete or null range") return rlist ## ### Main program ## optp = optparse.OptionParser(usage=("usage: %prog [options] < DATA" + "\n %prog [options] --previous" + "\n\nNote: long options can be abbreviated as long as unambiguous")) optp.add_option("-a", "--anonymous", action="store_true", default=False, help="anonymous data, only handle data (implies --bars and allows -k0)") optp.add_option("-b", "--bars", action="store_true", default=False, help="draw histogram style bars instead of list of keys") optp.add_option("--color", action="store", type="string", metavar="WHEN", default="auto", help="allow colors in output; WHEN can be 'always', 'never', 'auto' (default: 'auto')") optp.add_option("--no-cache", action="store_true", default=False, help="do not save a cached copy of the data for later use (see also --previous)") optp.add_option("--highlight-hostlist", action="store", type="string", metavar="HOSTLIST", help="highlight the specified HOSTLIST in the output table") optp.add_option("-r", "--range", action="store", type="string", metavar="LOW-HI", help="explicitly specify a value range") optp.add_option("-z", "--zero", action="store_true", default=False, help="first bucket starts at zero not at lowest value") optp.add_option("-o", "--show-overflow", action="store_true", default=False, help="include two extra buckets (over- and under-flow)") optp.add_option("-k", "--key", action="store", type="int", default=None, help="use data at position KEY (default: auto)") optp.add_option("-n", "--nbuckets", action="store", type="int", default=5, help="number of buckets to use (default: %default)") optp.add_option("-p", "--previous", action="store_true", default=False, help="read cached data from previous run instead of normal stdin") optp.add_option("-s", "--statistics", action="store_true", default=False, help="include a statistical summary") optp.add_option("-S", "--chop-long-lines", action="store_true", default=False, help="chop too long lines / enforce one output line per bucket") optp.add_option("-t", "--field-separators", action="store", type="string", default="", help="_additional_ field separators (default: \"\")") optp.add_option("-v", "--verbose", action="store_true", default=False) optp.add_option("--debug", action="store_true", default=False) (opts, args) = optp.parse_args(sys.argv[1:]) if opts.debug: opts.verbose = True if args != []: optp.print_help() sys.exit(1) if opts.nbuckets < 1: eprint("number of buckets must be a positive integer") if (opts.show_overflow and not (opts.range or opts.zero)): wprint("--show-overflow only has meaning with --range or --zero") if opts.range: if opts.zero: wprint("ignoring --zero because of --range use") opts.zero = False (rmin, rmax) = parse_range(opts.range) if opts.color == 'auto': if not sys.stdout.isatty(): opts.color='never' if opts.anonymous: opts.bars = True elif opts.key == 0: eprint("-k0 only possible with --anonymous") if opts.previous: try: cachefile = opencachefile("r") except: eprint("unable to open cachefile") elif not opts.no_cache: try: cachefile = opencachefile("w") except: eprint("unable to open cachefile") else: cachefile = None rawdata = readdata(cachefile) termwidth = gettermwidth() dprint("termwidth: %i" % termwidth) # do list of str -> list of [ hostname, value ] and discard bad lines if (len(rawdata) == 0): eprint("no data found") (data, nbadlines) = refine_data(rawdata) if (len(data) == 0): eprint("no data found") # sort it data.sort(key=lambda x: x[1]) # put the values in a simple list valuelist = [x[1] for x in data] if opts.range: vmin = min(valuelist) vmax = max(valuelist) elif opts.zero: rmin = 0.0 vmin = min(valuelist) rmax = vmax = max(valuelist) else: rmin = vmin = min(valuelist) rmax = vmax = max(valuelist) if (opts.range or opts.zero): nunder = len([ 1 for x in valuelist if x < rmin ]) nover = len([ 1 for x in valuelist if x > rmax ]) #dprint("cleaned up data: %s" % str(data)) # Create the bucket list newbuckets = create_buckets_new(valuelist, opts.nbuckets, rmin, rmax) if (opts.range or opts.zero): addoverflowbuckets(newbuckets, rmin, vmax) for nbucket in range(len(newbuckets)): newbuckets[nbucket]['nodelist'] = [] if opts.debug: dprint("bucketlist created:") for nbucket in range(len(newbuckets)): dprint(" bucket[%i] %f" % (nbucket, newbuckets[nbucket]['ub'])) # Dump out some statistics if opts.statistics: print("Statistical summary") print("-" * (termwidth - 1)) print(" %-26s: %i" % ("Number of values", len(valuelist))) print(" %-26s: %i" % ("Number of rejected lines", nbadlines)) if (opts.range or opts.zero): print(" %-26s: %i" % ("Number of overflow values", nover)) print(" %-26s: %i" % ("Number of underflow values", nunder)) print(" %-26s: %f" % ("Min value", valuelist[0])) print(" %-26s: %f" % ("Max value", valuelist[-1])) print(" %-26s: %f" % ("Mean", mean(valuelist))) print(" %-26s: %f" % ("Median", median(valuelist))) print(" %-26s: %f" % ("Standard deviation", stdev(valuelist))) print(" %-26s: %f" % ("Sum", sum(valuelist))) print() if opts.debug: pprint(["final newbuckets", newbuckets]) # Populate the buckets with data currentbucket = 0 for (node, value) in data: while ((newbuckets[currentbucket]['ub'] < value) or (value == rmin and currentbucket == 0 and 'special' in newbuckets[0])): currentbucket += 1 if opts.debug: print() dprint("filling next bucket") if opts.debug: sys.stdout.write(".") newbuckets[currentbucket]['nodelist'].append(node) if opts.debug: print() # Compute number of characters needed for printing values and number of nodes ncharvalue = len(str("%.2f" % max(valuelist[-1], rmax))) ncharnodecnt = max(len(str(max( [ len(bucket['nodelist']) for bucket in newbuckets ] ))), 3) # debug print of full data structure with data #pprint(["final newbuckets", newbuckets]) dprint("value pad: %i" % ncharvalue) dprint("node count pad: %i" % ncharnodecnt) # Print out a header if --verbose if opts.verbose: header = "%sLOW-%sHI: %sCNT" % (" " * (ncharvalue - 3), " " * (ncharvalue - 2), " " * (ncharnodecnt - 3)) if not opts.bars: header += " HOSTLIST" print(header) print("-" * termwidth) if opts.highlight_hostlist: highlight_set = set(expand_hostlist(opts.highlight_hostlist)) # To later be able to scale the bars find the largest bucket if opts.bars: maxbucketsize = 0 for bucket in newbuckets: if not opts.show_overflow and 'special' in bucket: continue maxbucketsize = max(maxbucketsize, len(bucket['nodelist'])) dprint("largest number of values in any bucket: %i" % maxbucketsize) barscale = max(1.0, maxbucketsize / (termwidth - (ncharvalue * 2 + ncharnodecnt + 5.0))) dprint("available space for bars %i" % (termwidth - (ncharvalue * 2 + ncharnodecnt + 5))) dprint("barscale %f" % barscale) # Main output print lower = valuelist[0] for bucket in newbuckets: # figure out the padding for each column pad1 = (ncharvalue - len(str("%.2f" % lower))) * " " pad2 = (ncharvalue - len(str("%.2f" % bucket['ub']))) * " " pad3 = (ncharnodecnt - len(str(len(bucket['nodelist'])))) * " " availablespace = termwidth - (ncharvalue * 2 + ncharnodecnt + 5) dprint("available space for nodelist: %i" % availablespace) # Three output modes for keys: bars, highlight and normal if opts.bars: if opts.highlight_hostlist: tnum = len(bucket['nodelist']) hnum = 0 for i in highlight_set: hnum += bucket['nodelist'].count(i) nnum = tnum - hnum # Always show highlighted data even if it rounds down to zero characters if ((hnum > 0) and (int(hnum/barscale) == 0)): roundup = 1 else: roundup = 0 if opts.chop_long_lines: nodeliststr = ( redboldstr("#" * (int(hnum/barscale) + roundup)) + "#" * int(nnum/barscale) ) else: nodeliststr = redboldstr("#" * hnum) + "#" * nnum else: nodeliststr = "#" * len(bucket['nodelist']) if opts.chop_long_lines: nodeliststr = nodeliststr[:int(len(nodeliststr)/barscale)] elif opts.highlight_hostlist: hnodeliststr = collect_hostlist(highlight_set & set(bucket['nodelist'])) nnodeliststr = collect_hostlist(set(bucket['nodelist']) - highlight_set) dprint("len(hnodeliststr) is: %i" % len(hnodeliststr)) dprint("len(nnodeliststr) is: %i" % len(nnodeliststr)) if hnodeliststr and nnodeliststr: sep = "," availablespace -= 1 else: sep = "" dprint("sep is %s" % str(not (sep == ""))) if (not opts.chop_long_lines or len(hnodeliststr + nnodeliststr) <= availablespace): nodeliststr = redboldstr(hnodeliststr) + sep + nnodeliststr elif ((len(hnodeliststr) + 3) <= availablespace): availablespace = availablespace + len(sep) - len(hnodeliststr) nodeliststr = redboldstr(hnodeliststr) nnodeliststr = sep + nnodeliststr nodeliststr += nnodeliststr[:(availablespace - 3)] + "..." else: availablespace += len(sep) nodeliststr = redboldstr(hnodeliststr[:(availablespace - 3)] + "...") else: nodeliststr = collect_hostlist(bucket['nodelist']) if (opts.chop_long_lines and len(nodeliststr) > availablespace): nodeliststr = nodeliststr[:(availablespace - 3)] + "..." pad4 = '' if len(bucket['nodelist']) == 0 else ' ' if 'special' in bucket: if opts.show_overflow: specpad = " " * max(len("%s%.2f-%s%.2f" % (pad1, lower ,pad2, bucket['ub'])) - 9, 0) print("%-9s%s: %s%i%s%s" % (bucket['special'], specpad, pad3, len(bucket['nodelist']), pad4, nodeliststr)) else: print("%s%.2f-%s%.2f: %s%i%s%s" % (pad1, lower, pad2, bucket['ub'], pad3, len(bucket['nodelist']), pad4, nodeliststr)) lower = bucket['ub'] ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/dbuck.10000644000175000017500000001202314723572005014605 0ustar00kentkent.TH dbuck 1 "Version 2.2.1" .SH NAME dbuck \- format output from pdsh command or similar .SH SYNOPSIS .B dbuck .RI [ OPTIONS ] .BI < DATA .br .B dbuck .RI [ OPTIONS ] .B --previous .SH DESCRIPTION Reads data from stdin (or cache file if --previous is given) and outputs a human readable report on stdout. dbuck is similar to dshbak but is targeted at numerical data such as temperatures, power consumption, loadavg etc. Output is a bucket sorted table, a sort of histogram. Data is assumed to be N lines of column wise space separated integers or decimal numbers. By default dbuck will autodetect the column to use (picking the first valid one) and sort everything into five (linear) buckets. Any line missing numerical data at the targeted column will be ignored. A complete copy of the data will also be saved for future use (see --previous). .SH OPTIONS .TP .B -h, --help Print help message .TP .B -a, --anonymous Anonymous data, only handle data (implies --bars and allows -k to be any value including zero). See example 2 below. .TP .B -b, --bars Draw histogram style bars instead of list of keys. Can be scaled down to fit terminal with -S,--chop-long-lines. .TP .BI "--color=" WHEN Allow colors in output; WHEN can be 'always', 'never', 'auto' (default: 'auto') .TP .B --no-cache Do not save a copy of the data in a cache file. By default dbuck will save data in a per user per session cache file for later use with --previous. .TP .BI "--highlight=" HOSTLIST ",--highlight-hostlist=" HOSTLIST Highlight the specified HOSTLIST in the output table using colors. .TP .BI "-r " LOW-HI ", --range=" LOW-HI Explicitly specify a range from minimum value of lowest bucket to maximum value of highest bucket (default: minimum value to maximum value seen in input data). Both LOW and HI can be negative numbers (integers or decimal). .TP .BI "-k " KEY ", --key=" KEY Use data at position KEY (default: auto). dbuck counts from 0 but field 0 is normally reserved for index/hostname. This means that in normal mode -k can range from 1 to the number of fields while in anonymous mode (-a/--anonymous) -k can also be 0. .TP .B -z, --zero Make dbuck generate buckets from zero (0.0) instead of lowest value seen in indata. .TP .B -o, --show-overflow Include two extra buckets for overflow and underflow. This option is only valid with a custom range (--range or --zero). Default behavior is to otherwise only count the over- and underflows and present them in the statistical summary. .TP .BI "-n " NBUCKETS ", --nbuckets=" NBUCKETS Number of buckets to use (default: 5) .TP .B -p, --previous Read data from cache file instead of from stdin. Cache files are saved by default per user per session unless disabled by --no-cache. .TP .B -s, --statistics Output a statistical summary (min, max, mean, sum, ...) .TP .B -S, --chop-long-lines Chop too long lines / enforce one output line per bucket .TP .BI "-t " FIELD_SEPARATORS ", --field-separators=" FIELD_SEPARATORS Additional field separators, space not optional (default: "") .TP .B -v, --verbose Be verbose .TP .B --debug Output debugging information .SH EXAMPLE 1 $ cat test/dbuck.testdata n1: 139 W n11: 128 W n13: 127 W n9: 127 W ... .TP $ cat test/dbuck.testdata | ./dbuck -s -n 4 --verbose Info: auto-detect unanimously selected key: 1 Info: Creating simple linear bucket set Statistical summary -------------------------------------- Number of values : 30 Number of rejected lines : 0 Min value : 115.000000 Max value : 209.000000 Mean : 135.466667 Median : 127.005617 Standard deviation : 25.807956 Sum : 4064.000000 LOW- HI: CNT HOSTLIST -------------------------------------- 115.00-138.50: 24 n[7-30] 138.50-162.00: 2 n[1,3] 162.00-185.50: 1 n6 185.50-209.00: 3 n[2,4-5] .SH EXAMPLE 2 Process resident size and total vm size from ps using the --anonymous option. Unlike the example above there's no hostname associated with each data point so this implies --bars. .TP $ ps -eo rss,vsize | ./dbuck --anonymous --chop-long-lines --verbose Info: auto-detect unanimously selected key: 0 Info: rejected line: "RSS VSZ" LOW- HI: CNT HOSTLIST -------------------------------------------------------------- 0.00-116425.60: 271 #################################### 116425.60-232851.20: 16 ## 232851.20-349276.80: 3 349276.80-465702.40: 3 465702.40-582128.00: 1 .TP Allowing dbuck to automatically find data, it picked up the RSS value (selected key: 0). Now we'll specify "-k 1" to select the vsize data in column 1. .TP $ ps -eo rss,vsize | ./dbuck --anonymous --chop-long-lines -k 1 0.00- 53763877.60: 293 ############################## 53763877.60-107527755.20: 0 107527755.20-161291632.80: 0 161291632.80-215055510.40: 0 215055510.40-268819388.00: 1 .SH AUTHOR Written by Peter Kjellström . The program is published part of python-hostlist at http://www.nsc.liu.se/~kent/python-hostlist/ .SH SEE ALSO .I hostlist (1) .I pdsh (1) .I dshbak (1) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/hostgrep0000644000175000017500000001235514723572005015221 0ustar00kentkent#!/usr/bin/env python3 # Grep-like utility understanding hostlists __version__ = "2.2.1" # Copyright (C) 2009 Kent Engström # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA # 02110-1301, USA. import string import sys import optparse import re from hostlist import expand_hostlist, collect_hostlist, numerically_sorted, BadHostlist, __version__ as library_version def die(s, exit_code = 1): sys.stderr.write(s + "\n") sys.exit(1) def hosts_from_line(line): # --restrict option if restrict_regexp: try: m = restrict_regexp.search(line) line = m.group(1) except Exception: line = "" words = set() for m in finder.finditer(line): word = m.group(0) if not '[' in word: words.add(word) else: try: hosts = expand_hostlist(word) for host in hosts: words.add(host) except BadHostlist: words.add(word) return words def emit(line, filename, line_no): if opts.line_number: line = str(line_no) + ":" + line if do_print_filename: line = filename + ":" + line sys.stdout.write(line) def search(f, filename): line_no = 0 for line in f: line_no += 1 words = hosts_from_line(line) if opts.all: does_match = search_set <= words else: does_match = search_set & words if bool(does_match) ^ bool(opts.invert_match): emit(line, filename, line_no) def compute_finder_regexp(search_set): # We have always considered letters and digits to be part of potential hostnames, # so we start with them and add any other characters seen search_chars_set = set(string.ascii_uppercase + string.ascii_lowercase + string.digits) for host in search_set: search_chars_set |= set(host) # We rely on re.escape being able to escape characters for the # inside of a [...] character set, which seems to be the case. search_chars = re.escape("".join(sorted(search_chars_set))) return re.compile(r"[" + search_chars + r"]+(\[[0-9,-]+\])*") # MAIN op = optparse.OptionParser(usage="usage: %prog [OPTION]... HOSTLIST [FILES]...", add_help_option = False) op.add_option("--all", action="store_true", help="Require all hosts in the hostlist to be found in the line.") op.add_option("--any", action="store_false", dest = "all", help="Require some host in the hostlist to be found in the line (default).") op.add_option("-h", "--no-filename", action="store_false", dest = "print_filename", help="Do not show filename before match (default for <= 1 file)") op.add_option("-H", "--with-filename", action="store_true", dest = "print_filename", help="Show filename before match (default for > 1 file)") op.add_option("-v", "--invert-match", action="store_true", help="Invert the sense of matching, to select non-matching lines.") op.add_option("-n", "--line-number", action="store_true", help="Show line number before match") op.add_option("--restrict", action="store", type="string", help="Restrict host matching to the part of the line that is matched as group 1 by this regexp") op.add_option("--help", action="help", help="Show help") op.add_option("--version", action="store_true", help="Show version") (opts, args) = op.parse_args() if opts.version: print("Version %s (library version %s)" % (__version__, library_version)) sys.exit() if len(args) < 1: die("You must specify a hostlist") else: search_list = args[0] search_set = set(expand_hostlist(search_list)) finder = compute_finder_regexp(search_set) if opts.print_filename in (True, False): do_print_filename = opts.print_filename else: if len(args) > 2: do_print_filename = True else: do_print_filename = False if opts.restrict: try: restrict_regexp = re.compile(opts.restrict) except Exception: die("Bad --restrict regexp %s" % opts.restrict) else: restrict_regexp = None try: if len(args) == 1: # Seach stdin search(sys.stdin, "(standard input)") else: for filename in args[1:]: f = open(filename) search(f, filename) f.close() except KeyboardInterrupt: sys.exit() except IOError as e: sys.stderr.write(e.strerror + " '" + e.filename + "'\n") sys.exit(1) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/hostgrep.10000644000175000017500000000551014723572005015353 0ustar00kentkent.TH hostgrep 1 "Version 2.2.1" .SH NAME hostgrep \- print lines matching hostlist .SH SYNOPSIS .B hostgrep .RI [ OPTION "]... " HOSTLIST " [" FILE ]... .SH DESCRIPTION Search the files for lines matching the given hostlist. If no files are given, search the standard input. For each line, the program tries to find all hostnames and hostlists. Hostlists are then expanded to hostnames. The set of hostnames in a line is then compared to the set of hostnames given by the .I HOSTLIST on the command line. By default, the line is printed if any hostname from the .I HOSTLIST is found in the line. .SH OPTIONS .TP .B --all Print a line if all hostnames in the given .I HOSTLIST is found in the line. .TP .B --any Print a line if any hostname in the given .I HOSTLIST is found in the line (the default). .TP .B -h, --no-filename Do not show the filename in front of matching line. This is the default if at most one .I FILE is given. .TP .B -H, --with-filename Show the filename in front of matching lines. This is the default if two or more .I FILES are given. .TP .B -v, --invert-match Invert the sense of matching, to select non-matching lines. .TP .B -n, --line-number Show line number before match. .TP .BI --restrict= REGEXP Restrict host matching to the part of the line that is matched as group 1 by this regexp. .SH EXAMPLES .TP Search for lines in an accounting file where any of the three hosts \ is present (as a hostname or part of a hostlist): .B hostgrep n[100-102] /var/log/slurm/accounting/2009-02-22 .TP Search for lines in the accounting files where all of the three hosts \ are present (as a hostname or part of a hostlist). Do not show filenames: .B hostgrep -h --all n[100-102] /var/log/slurm/accounting/* .TP Search for lines on stdin where nodes in n[1-8] are mentioned \ in the part of the line that comes before the first colon: .B hostgrep --restrict='^([^:]*):' n[1-8] .SH BUGS The program has a rather naive notion of what constitutes hostnames and hostlist. For version 1.21 and earlier, hostnames were allowed to contain exactly upper and lower case A-Z and the digits 0-9. This meant that you could not grep for names like "rack02-pos13". From version 1.22, the code dynamically adds additional characters from the hostlist provided on the command line, so that it will work to grep for hostnames containing those characters. .SH AUTHOR Written by Kent Engström . The program is published at http://www.nsc.liu.se/~kent/python-hostlist/ .SH SEE ALSO .I hostlist (1) The hostlist expression syntax is used by several programs developed at .B LLNL (https://computing.llnl.gov/linux/), for example .B SLURM (https://computing.llnl.gov/linux/slurm/) and .B Pdsh (https://computing.llnl.gov/linux/pdsh.html). See the .B HOSTLIST EXPRESSIONS section of the .BR pdsh (1) manual page for a short introduction to the hostlist syntax. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/hostlist0000644000175000017500000002153714723572005015241 0ustar00kentkent#!/usr/bin/env python3 # Hostlist utility __version__ = "2.2.1" # Copyright (C) 2008-2018 # Kent Engström , # Thomas Bellman , # Pär Lindfors and # Torbjörn Lönnemark , # National Supercomputer Centre # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA # 02110-1301, USA. import sys import os import optparse import operator import re from hostlist import expand_hostlist, collect_hostlist, numerically_sorted, parse_slurm_tasks_per_node, BadHostlist, __version__ as library_version # Python 3 compatibility try: from functools import reduce except ImportError: pass # earlier Python versions have this in __builtin__ # Helper functions def flatten(list_of_lists): res = [] for l in list_of_lists: res.extend(l) return res def complain(msg): sys.stderr.write(f"{sys.argv[0]}: {msg}\n") def complain_about_whitespace(s): fixed = re.sub(r'(\s|\N{ZERO WIDTH SPACE})', '', s) if fixed != s: complain(f"Spurious whitespace seen in argument {s!r}.") # Operators def func_union(args): return reduce(operator.or_, args) def func_intersection(args): return reduce(operator.and_, args) def func_difference(args): return reduce(operator.sub, args) def func_xor(args): return reduce(operator.xor, args) op = optparse.OptionParser(usage="usage: %prog [OPTION]... [HOSTLIST]...") op.add_option("-u", "--union", action="store_const", dest="func", const=func_union, default=func_union, help="compute the union of the hostlist arguments (default)") op.add_option("-i", "--intersection", action="store_const", dest="func", const=func_intersection, help="compute the intersection of the hostlist arguments") op.add_option("-d", "--difference", action="store_const", dest="func", const=func_difference, help="compute the difference between the first hostlist argument and the rest") op.add_option("-x", "--symmetric-difference", action="store_const", dest="func", const=func_xor, help="compute the symmetric difference between the first hostlist argument and the rest") op.add_option("-c", "--collapse", action="store_false", dest="expand", help="output the result as a hostlist expression (default)") op.add_option("-o", "--offset", action="store", type="int", help="skip OFFSET hosts from the beginning (default: skip nothing)") op.add_option("-l", "--limit", action="store", type="int", help="limit result to the LIMIT first hosts (default: no limit)") op.add_option("-n", "--count", action="store_true", help="output the number of hosts instead of a hostlist") op.add_option("-e", "--expand", action="store_true", help="output the result as an expanded list of hostnames") op.add_option("-w", action="store_true", dest="expand_deprecated", help="DEPRECATED version of -e/--expand") op.add_option("-q", "--quiet", action="store_true", help="output nothing (useful with --non-empty)") op.add_option("-0", "--non-empty", action="store_true", help="return success only if the resulting hostlist is non-empty") op.add_option("-s", "--separator", action="store", type="string", default="\n", help="separator to use between hostnames when outputting an expanded list (default is newline)") op.add_option("-p", "--prepend", action="store", type="string", default="", help="string to prepend to each hostname when outputting an expanded list") op.add_option("-a", "--append", action="store", type="string", default="", help="string to append to each hostname when outputting an expanded list") op.add_option("-S", "--substitute", action="store", type="string", help="regular expression substitution ('from,to') to apply to each hostname") op.add_option("--append-slurm-tasks", action="store", type="string", help="append task count based on the passed SLURM_TASKS_PER_NODE string") op.add_option("--repeat-slurm-tasks", action="store", type="string", help="repeat hostnames based on the passed SLURM_TASKS_PER_NODE string") op.add_option("--chop", action="store", type="int", help="chop into chunks of this size when doing --collapse (default: all in one chunk)") op.add_option("--version", action="store_true", help="show version") (opts, args) = op.parse_args() if opts.version: print("Version %s (library version %s)" % (__version__, library_version)) sys.exit() func_args = [] try: for a in args: input_string = a if a == "-": input_string = sys.stdin.read() func_args.append(set()) for e in input_string.split(): complain_about_whitespace(e) func_args[-1] |= set(expand_hostlist(e)) except BadHostlist as e: sys.stderr.write("Bad hostlist ``%s'' encountered: %s\n" % ((a,) + e.args)) sys.exit(os.EX_DATAERR) if not func_args: op.print_help() sys.exit(os.EX_USAGE) if opts.expand_deprecated: sys.stderr.write("WARNING: Option -w is deprecated. Use -e or --expand instead!\n") # Set up initial hostlist from the arguments and the function res = opts.func(func_args) # Handle --substitute if opts.substitute: try: from_re, to = opts.substitute.split(",", 1) res = [re.sub(from_re, to, host) for host in res] res = set(res) # remove duplicates that may have been created except (ValueError, re.error): sys.stderr.write("Bad --substitute option: '%s'\n" % opts.substitute) sys.exit(os.EX_DATAERR) # Sort numerically res = numerically_sorted(res) # res can be list or set before this line # Handle --offset if opts.offset is not None: res = res[opts.offset:] # Handle --limit if opts.limit is not None: res = res[:opts.limit] # Handle options using SLURM task lists if opts.append_slurm_tasks and opts.repeat_slurm_tasks: sys.stderr.write("You cannot use --append-slurm-tasks and --repeat--slurm-tasks at the same time.\n") sys.exit(os.EX_DATAERR) elif opts.append_slurm_tasks or opts.repeat_slurm_tasks: if opts.append_slurm_tasks: task_list = opts.append_slurm_tasks else: task_list = opts.repeat_slurm_tasks try: task_list = parse_slurm_tasks_per_node(task_list) except BadHostlist as e: sys.stderr.write("Bad task list encountered: %s\n" % e.args) sys.exit(os.EX_DATAERR) if len(task_list) != len(res): sys.stderr.write("Length of tasks list != number of hostnames\n") sys.exit(os.EX_DATAERR) # Output in the right way if opts.quiet: pass elif opts.count: print(len(res)) elif opts.expand or opts.expand_deprecated: if opts.append_slurm_tasks: print(opts.separator.join([opts.prepend + host + opts.append + str(tasks) for host, tasks in zip(res, task_list)])) elif opts.repeat_slurm_tasks: repeated_hosts = flatten([[host]*tasks for host, tasks in zip(res, task_list)]) print(opts.separator.join([opts.prepend + host + opts.append for host in repeated_hosts])) else: print(opts.separator.join([opts.prepend + host + opts.append for host in res])) else: # --collapse if opts.chop and opts.chop > 0: chunk_size = opts.chop else: chunk_size = len(res) i=0 while i < len(res): try: if i > 0: sys.stdout.write(opts.separator) sys.stdout.write(opts.prepend + collect_hostlist(res[i:i+chunk_size]) + opts.append) i += chunk_size except BadHostlist as e: sys.stderr.write("Bad hostname encountered: %s\n" % e.args) sys.exit(os.EX_DATAERR) sys.stdout.write("\n") # Exit if opts.non_empty and len(res) == 0: sys.exit(os.EX_NOINPUT) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/hostlist.10000644000175000017500000001361514723572005015376 0ustar00kentkent.TH hostlist 1 "Version 2.2.1" .SH NAME hostlist \- handle hostlist expressions .SH SYNOPSIS .B hostlist .RI [ OPTION "]... [" HOSTLIST ]... .SH DESCRIPTION Perform a set operation (union, intersection, difference or symmetric difference) on the given hostlists. Output the result as a hostlist, a count of hostnames or an expanded list of hostnames. If "-" is used instead of a hostlist argument, an arbitrary number of hostlists are read from stdin. The union of them is used as if it had been present on the command line as a single hostlist argument. .SH OPTIONS .TP .B -u, --union Compute the union of the hostlists. A hostname is present in the output if it is included in at least one of the hostlists. This is the default operation. .TP .B -i, --intersection Compute the intersection of the hostlists. A hostname is present in the output if it is included in all the hostlists. .TP .B -d, --difference Compute the difference of the hostlists. A hostname is present in the output if it is included in the first hostlist but not any of the following. .TP .B -x, --symmetric-difference Compute the symmetric difference of the hostlists. A hostname is present in the output if it is included in an odd number of the hostlists. .TP .BI "-o " OFFSET ", --offset=" OFFSET Filter the result to skip .I OFFSET hosts from the beginning. If .I OFFSET is not less than the number of hosts in the result, the result will become empty. If .I OFFSET is negative, keep .I -OFFSET hosts from the end of the result. The default is to skip nothing. .TP .BI "-l " LIMIT ", --limit=" LIMIT Filter the result to limit it to the first .I LIMIT hosts. If .I LIMIT is not less than the number of hosts in the result, this option does nothing. This filter is applied after the .B --offset filter (see above). The default is to have no limit. .TP .B -c, --collapse Output the result as a hostlist expression. This is the default. See the .B --chop option for splitting into multiple hostlists of a certain size. .TP .B -n, --count Output the number of hostnames in the result. .TP .B -e, --expand Output the result as an expanded list of hostnames. .TP .B -w Output the result as an expanded list of hostnames. This option is deprecated. Use -e or --expand instead. .TP .B -q, --quiet Output nothing. This option is useful together with .BR --non-empty . .TP .B -0, --non-empty Return success if the resulting hostlist is non-empty and failure if it is empty. .TP .BI "-s " SEPARATOR ", --separator=" SEPARATOR Use .I SEPARATOR as the separator between the hostnames in the expanded list (and between (chopped) hostlists in collapsed mode). The default is a newline. .TP .BI "-p " PREPEND ", --prepend=" PREPEND Output .I PREPEND before each hostname in the expanded list (and before each hostlist in collapsed mode). The default is to prepend nothing. .TP .BI "-a " APPEND ", --append=" APPEND Output .I APPEND after each hostname in the expanded list (and after each hostlist in collapsed mode). The default is to append nothing. .TP .BI "-S " FROM,TO ", --substitute=" FROM,TO Apply a regular expression substitution to each hostname, replacing all .I FROM with .IR TO . The default is to do no substitution. .TP .BI "--append-slurm-tasks=" SLURM_TASKS_PER_NODE Append the number of tasks parsed from the .I SLURM_TASKS_PER_NODE string. You need to use .B -e/--expand and you probably need to specify .B -a/--append and .B -s/--separator too. NOTE: The hostlist is always sorted internally by this program. The task counts from .I SLURM_TASKS_PER_NODE is then applied to each hostname in the sorted list. .TP .BI "--repeat-slurm-tasks=" SLURM_TASKS_PER_NODE Repeat each hostname so it is listed multiple times as specified by the .I SLURM_TASKS_PER_NODE string. You need to use .B -e/--expand. NOTE: The hostlist is always sorted internally by this program. The task counts from .I SLURM_TASKS_PER_NODE is then applied to each hostname in the sorted list. .TP .BI "--chop=" CHUNKSIZE When outputting as collapsed hostlist (the default mode .BR --collapse ) split into hostlists of size .I CHUNKSIZE and output each hostlist separately. The last hostlist may be smaller than the rest. .TP .B --version Show the version of the utility and the underlying Python library. .SH EXAMPLES .TP Output the union of n[10-19] and n[15-24] (which is n[10-24]): .B hostlist n[10-19] n[15-24] .TP Output the result of removing n15 and n[17-18] from n[1-20] \ (which is n[1-14,16,19-20]): .B hostlist -d n[1-20] n15 n[17-18] .TP Output the result as an expanded list of hostnames (one hostname per line): .B hostlist -d -e n[1-20] n15 n[17-18] .TP Output the result as an expanded list of hostnames separated by commas: .B hostlist -d -e -s, n[1-20] n15 n[17-18] .TP Output the result as an expanded list of hostnames (followed by space and \ the digit "8") separated by spaces: .B hostlist -d -e -s " " -a " 8" n[1-20] n15 n[17-18] .TP Expand a hostlist, replacing "n" with "ni": .B hostlist -e -S n,ni n[1-20] .TP Print INCLUDED as n11 is in n[10-20]: if .B hostlist -q0 -i n11 n[10-20]; then echo INCLUDED; else echo NOT INCLUDED; fi .SH AUTHOR Written by Kent Engström . The program is published at http://www.nsc.liu.se/~kent/python-hostlist/ .SH NOTES The square brackets used in the hostlist expression syntax are also used in shell glob patterns. This may cause unwanted surprises if you have files in your current directory named after hosts present in the hostlist. Always quote the hostlist expression to avoid this problem: % hostlist n[1-10] n[1-10] % touch n1 % hostlist n[1-10] n1 % echo n[1-10] n1 % hostlist "n[1-10]" n[1-10] .SH SEE ALSO The hostlist expression syntax is used by several programs developed at .B LLNL (https://computing.llnl.gov/linux/), for example .B SLURM (https://computing.llnl.gov/linux/slurm/) and .B Pdsh (https://computing.llnl.gov/linux/pdsh.html). See the .B HOSTLIST EXPRESSIONS section of the .BR pdsh (1) manual page for a short introduction to the hostlist syntax. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/hostlist.py0000644000175000017500000003654114723572005015671 0ustar00kentkent#!/usr/bin/env python # # Hostlist library # # Copyright (C) 2008-2018 # Kent Engström , # Thomas Bellman , # Pär Lindfors and # Torbjörn Lönnemark , # National Supercomputer Centre # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA # 02110-1301, USA. """Handle hostlist expressions. This module provides operations to expand and collect hostlist expressions. The hostlist expression syntax is the same as in several programs developed at LLNL (https://computing.llnl.gov/linux/). However in corner cases the behaviour of this module have not been compared for compatibility with pdsh/dshbak/SLURM et al. """ __version__ = "2.2.1" import re import itertools # Replace range with xrange on Python 2, do nothing on Python 3 (where xrange # does not exist, and range returns an iterator) try: range = xrange except: pass # Exception used for error reporting to the caller class BadHostlist(Exception): pass # Configuration to guard against ridiculously long expanded lists MAX_SIZE = 100000 # Hostlist expansion def expand_hostlist(hostlist, allow_duplicates=False, sort=False): """Expand a hostlist expression string to a Python list. Example: expand_hostlist("n[9-11],d[01-02]") ==> ['n9', 'n10', 'n11', 'd01', 'd02'] Unless allow_duplicates is true, duplicates will be purged from the results. If sort is true, the output will be sorted. """ results = [] bracket_level = 0 part = "" for c in hostlist + ",": if c == "," and bracket_level == 0: # Comma at top level, split! if part: results.extend(expand_part(part)) part = "" bad_part = False else: part += c if c == "[": bracket_level += 1 elif c == "]": bracket_level -= 1 if bracket_level > 1: raise BadHostlist("nested brackets") elif bracket_level < 0: raise BadHostlist("unbalanced brackets") if bracket_level > 0: raise BadHostlist("unbalanced brackets") if not allow_duplicates: results = remove_duplicates(results) if sort: results = numerically_sorted(results) return results def expand_part(s): """Expand a part (e.g. "x[1-2]y[1-3][1-3]") (no outer level commas).""" # Base case: the empty part expand to the singleton list of "" if s == "": return [""] # Split into: # 1) prefix string (may be empty) # 2) rangelist in brackets (may be missing) # 3) the rest m = re.match(r'([^,\[]*)(\[[^\]]*\])?(.*)', s) (prefix, rangelist, rest) = m.group(1,2,3) # Expand the rest first (here is where we recurse!) rest_expanded = expand_part(rest) # Expand our own part if not rangelist: # If there is no rangelist, our own contribution is the prefix only us_expanded = [prefix] else: # Otherwise expand the rangelist (adding the prefix before) us_expanded = expand_rangelist(prefix, rangelist[1:-1]) # Combine our list with the list from the expansion of the rest # (but guard against too large results first) if len(us_expanded) * len(rest_expanded) > MAX_SIZE: raise BadHostlist("results too large") return [us_part + rest_part for us_part in us_expanded for rest_part in rest_expanded] def expand_rangelist(prefix, rangelist): """ Expand a rangelist (e.g. "1-10,14"), putting a prefix before.""" # Split at commas and expand each range separately results = [] for range_ in rangelist.split(","): results.extend(expand_range(prefix, range_)) return results def expand_range(prefix, range_): """ Expand a range (e.g. 1-10 or 14), putting a prefix before.""" # Check for a single number first m = re.match(r'^[0-9]+$', range_) if m: return ["%s%s" % (prefix, range_)] # Otherwise split low-high m = re.match(r'^([0-9]+)-([0-9]+)$', range_) if not m: raise BadHostlist("bad range") (s_low, s_high) = m.group(1,2) low = int(s_low) high = int(s_high) width = len(s_low) if high < low: raise BadHostlist("start > stop") elif high - low > MAX_SIZE: raise BadHostlist("range too large") results = [] for i in range(low, high+1): results.append("%s%0*d" % (prefix, width, i)) return results def remove_duplicates(l): """Remove duplicates from a list (but keep the order).""" seen = set() results = [] for e in l: if e not in seen: results.append(e) seen.add(e) return results # Hostlist collection def collect_hostlist(hosts, silently_discard_bad = False): """Collect a hostlist string from a Python list of hosts. We start grouping from the rightmost numerical part. Duplicates are removed. A bad hostname raises an exception (unless silently_discard_bad is true causing the bad hostname to be silently discarded instead). """ # Split hostlist into a list of (host, "") for the iterative part. # (Also check for bad node names now) # The idea is to move already collected numerical parts from the # left side (seen by each loop) to the right side (just copied). left_right = [] for host in hosts: # We remove leading and trailing whitespace first, and skip empty lines host = host.strip() if host == "": continue # We cannot accept a host containing any of the three special # characters in the hostlist syntax (comma and flat brackets) if re.search(r'[][,]', host): if silently_discard_bad: continue else: raise BadHostlist("forbidden character") left_right.append((host, "")) # Call the iterative function until it says it's done looping = True while looping: left_right, looping = collect_hostlist_1(left_right) return ",".join([left + right for left, right in left_right]) def collect_hostlist_1(left_right): """Collect a hostlist string from a list of hosts (left+right). The input is a list of tuples (left, right). The left part is analyzed, while the right part is just passed along (it can contain already collected range expressions). """ # Scan the list of hosts (left+right) and build two things: # *) a set of all hosts seen (used later) # *) a list where each host entry is preprocessed for correct sorting sortlist = [] remaining = set() for left, right in left_right: host = left + right remaining.add(host) # Match the left part into parts m = re.match(r'^(.*?)([0-9]+)?([^0-9]*)$', left) (prefix, num_str, suffix) = m.group(1,2,3) # Add the right part unprocessed to the suffix. # This ensures than an already computed range expression # in the right part is not analyzed again. suffix = suffix + right if num_str is None: # A left part with no numeric part at all gets special treatment! # The regexp matches with the whole string as the suffix, # with nothing in the prefix or numeric parts. # We do not want that, so we move it to the prefix and put # None as a special marker where the suffix should be. assert prefix == "" sortlist.append(((host, None), None, None, host)) else: # A left part with at least an numeric part # (we care about the rightmost numeric part) num_int = int(num_str) num_width = len(num_str) # This width includes leading zeroes sortlist.append(((prefix, suffix), num_int, num_width, host)) # Sort lexicographically, first on prefix, then on suffix, then on # num_int (numerically), then... # This determines the order of the final result. def sort_key(entry): """ Sort sortlist entries without causing TypeError in Python 3. Prefix is always present and a string. If suffix is None, use the empty string. The empty string will sort before all other strings (as None did in Python 2). If num_int is None, use -1 instead. -1 sorts before all *possible* numbers (as None did in Python 2). Negative numbers cannot be present in a hostlist. If num_width is None, use -1 instead. As before, -1 sorts before all possible widths. An actual num_width of 0 should not be possible, so we could technically use 0, but using -1 doesn't hurt, so we might as well use that instead. """ ((prefix, suffix), num_int, num_width, host) = entry return ( (prefix, '' if suffix is None else suffix), -1 if num_int is None else num_int, -1 if num_width is None else num_width, host ) sortlist.sort(key=sort_key) # We are ready to collect the result parts as a list of new (left, # right) tuples. results = [] needs_another_loop = False # Now group entries with the same prefix+suffix combination (the # key is the first element in the sortlist) to loop over them and # then to loop over the list of hosts sharing the same # prefix+suffix combination. for ((prefix, suffix), group) in itertools.groupby(sortlist, key=lambda x:x[0]): if suffix is None: # Special case: a host with no numeric part results.append(("", prefix)) # Move everything to the right part remaining.remove(prefix) else: # The general case. We prepare to collect a list of # ranges expressed as (low, high, width) for later # formatting. range_list = [] for ((prefix2, suffix2), num_int, num_width, host) in group: if host not in remaining: # Below, we will loop internally to enumate a whole range # at a time. We then remove the covered hosts from the set. # Therefore, skip the host here if it is gone from the set. continue assert num_int is not None # Scan for a range starting at the current host low = num_int while True: host = "%s%0*d%s" % (prefix, num_width, num_int, suffix) if host in remaining: remaining.remove(host) num_int += 1 else: break high = num_int - 1 assert high >= low range_list.append((low, high, num_width)) # We have a list of ranges to format. We make sure # we move our handled numerical part to the right to # stop it from being processed again. needs_another_loop = True if len(range_list) == 1 and range_list[0][0] == range_list[0][1]: # Special case to make sure that n1 is not shown as n[1] etc results.append((prefix, "%0*d%s" % (range_list[0][2], range_list[0][0], suffix))) else: # General case where high > low results.append((prefix, "[" + \ ",".join([format_range(l, h, w) for l, h, w in range_list]) + \ "]" + suffix)) # At this point, the set of remaining hosts should be empty and we # are ready to return the result, together with the flag that says # if we need to loop again (we do if we have added something to a # left part). assert not remaining return results, needs_another_loop def format_range(low, high, width): """Format a range from low to high inclusively, with a certain width.""" if low == high: return "%0*d" % (width, low) else: return "%0*d-%0*d" % (width, low, width, high) # Sort a list of hosts numerically def numerically_sorted(l): """Sort a list of hosts numerically. E.g. sorted order should be n1, n2, n10; not n1, n10, n2. """ return sorted(l, key=numeric_sort_key) numeric_sort_key_regexp = re.compile("([0-9]+)|([^0-9]+)") def numeric_sort_key(x): """Compose a sorting key to compare strings "numerically": We split numerical (integer) and non-numerical parts into a list, making sure that the numerical parts are converted to Python ints, and then sort on the lists. Thus, if we sort x10y and x9z8, we will compare ["x", 10, "y"] with ["x", 9, "x", "8"] and return x9z8 before x10y". Python 3 complication: We cannot compare int and str, so while we can compare x10y and x9z8, we cannot compare x10y and 9z8. Kludge: insert a blank string first if the list would otherwise start with an integer. This will give the same ordering as before, as integers seem to compare smaller than strings in Python 2. """ keylist = [int(i_ni[0]) if i_ni[0] else i_ni[1] for i_ni in numeric_sort_key_regexp.findall(x)] if keylist and isinstance(keylist[0], int): keylist.insert(0, "") return keylist # Parse SLURM_TASKS_PER_NODE into a list of task numbers # # Description from the SLURM sbatch man page: # Number of tasks to be initiated on each node. Values # are comma separated and in the same order as # SLURM_NODELIST. If two or more consecutive nodes are # to have the same task count, that count is followed by # "(x#)" where "#" is the repetition count. For example, # "SLURM_TASKS_PER_NODE=2(x3),1" indicates that the first # three nodes will each execute three tasks and the # fourth node will execute one task. def parse_slurm_tasks_per_node(s): res = [] for part in s.split(","): m = re.match(r'^([0-9]+)(\(x([0-9]+)\))?$', part) if m: tasks = int(m.group(1)) repetitions = m.group(3) if repetitions is None: repetitions = 1 else: repetitions = int(repetitions) if repetitions > MAX_SIZE: raise BadHostlist("task list repetitions too large") for i in range(repetitions): res.append(tasks) else: raise BadHostlist("bad task list syntax") return res # # Keep this part to tell users where the command line interface went # if __name__ == '__main__': import os, sys sys.stderr.write("The command line utility has been moved to a separate 'hostlist' program.\n") sys.exit(os.EX_USAGE) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/pshbak0000644000175000017500000004473414723572005014644 0ustar00kentkent#!/usr/bin/env python3 # rewrite of dshbak using python-hostlist __version__ = "2.2.1" # Copyright (C) 2010 Mattias Slabanja # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA # 02110-1301, USA. import sys import optparse import re from hostlist import collect_hostlist, expand_hostlist, __version__ as library_version from difflib import unified_diff, SequenceMatcher from itertools import count def scan(): """Scan stdin, store lines by host, and return it all in a dictionary indexed by host. Input lines are expected to be on the format ":". Lines not matching that format are added to the dictionary using None as key. The linesplit-re is designed to match the original dshbak behavior. """ linesplitter = re.compile(r'^ *([A-Za-z0-9.-]+) *: ?(.*)$') text_dict = {} for line in sys.stdin: match = linesplitter.match(line) if match: host, hostline = match.groups() else: # The linesplitter regexp did NOT match. # This line will be added to text_dict[None] host = None hostline = line.rstrip('\n') if host in text_dict: # The groups in the linesplitter regexp does not include the trailing '\n' text_dict[host] += "\n" + hostline else: text_dict[host] = hostline return text_dict def collect(text_dict): """Collect hosts having identical output. Return a list of (host set, text) tuples.""" reverse_dict = {} for host, text in text_dict.items(): if text in reverse_dict: reverse_dict[text].add(host) else: reverse_dict[text] = set((host,)) return [(host_set, text) for text, host_set in reverse_dict.items()] # Colors enabled by default def in_red(s): return '\033[91m' + s + '\033[0m' def in_blue(s): return '\033[94m' + s + '\033[0m' def in_gray(s): return '\033[90m' + s + '\033[0m' in_plain = lambda x:x def output(host_set_or_str, text, count_hosts=False): """Prepend the output with a hostname framed with horizontal lines.""" hline = '-' * 16 if isinstance(host_set_or_str, str): header = host_set_or_str # for the benefit of the garbage header else: header = collect_hostlist(host_set_or_str) if count_hosts: header = "%d: %s" % (len(host_set_or_str), header) print(in_gray(hline)) print(in_gray(header)) print(in_gray(hline)) print(text) class Collector: """Collect relatively matching texts, find similarities and differences Depends on SequenceMatcher. Texts being similar enough when doing a line-per-line comparison to a reference text can be added with the try_add_text method. When texts of interest has been added, calling the process method will sort out which parts are identical (per line intersection of all matches between reference and rest of the texts) and will store the individual differences. The heavy part of the processing is done by SequenceMatcher.get_matching_blocks called from try_add_text. """ def __init__(self, ref_text, label=None, match_limit=0.5): ref_lines = ref_text.split('\n') self._N = len(ref_lines) self._matchers = [SequenceMatcher(None, L, L) for L in ref_lines] assert match_limit <= 1.0 and match_limit >= 0 self._match_limit = match_limit self._lmb_pairs = [[] for i in range(self._N)] self._labels = [] self._label_counter = count(1) self._reset_processing_result() self._add_lines(ref_lines) self._push_label_name(label) def try_add_text(self, text, label=None): """Test if text is similar enough, if it is, add it """ lines = text.split('\n') if len(lines) != self._N: return False self._update_seq1(lines) # The quick_ratio method is somewhat expensive ratio = (1.0/self._N) * sum([mat.quick_ratio() for mat in self._matchers]) if ratio < self._match_limit: return False if not self.text is None: self._reset_processing_result() self._add_lines(lines) self._push_label_name(label) return True def render_line(self, template, label=None, col_t=in_plain, col_f=in_plain): """Render a line-template into a text line. %(h)s, $(1)s, %(2)s, .. is substituted to hostname (label), diff1, diff2, ... percent-sign needs to be escaped by replacing them with %(p)s. If template is not a string, it is expected to be list of (bool, string) tuples, where the strings will be processed with col_t and col_f for True and False, respectively. This is used to control the colors of the rendered line. """ if type(template) is str: return template % self._diffs[label] line = [] for matching, ttext in template: if matching: line.append(col_t(ttext)) else: line.append(col_f(ttext)) return (''.join(line)) % self._diffs[label] def process(self, spliced_diff=False, collected_diff=False): """Process the gathered texts spliced_diff - splice in the lines that are differing collected_diff - try to collect the differing lines """ if not self.text is None: self._reset_processing_result() for label in self._labels: self._diffs[label] = {} self._diffs[label]['h'] = label self._diffs[label]['p'] = '%' lines = [] # For each set of lines, sort out differences for lmbs in self._lmb_pairs: line_template, N_diffs = self._process_line(lmbs) lines.append(self.render_line(line_template, col_f=in_red)) if N_diffs > 0 and spliced_diff: d = self._diffs text_dict = dict([(l, self.render_line(line_template, l, col_f=in_red, col_t=in_blue)) for l in self._labels]) if collected_diff: hosts_text_list = [(collect_hostlist(hs), t) for hs, t in collect(text_dict)] else: hosts_text_list = text_dict.items() for host, text in sorted(hosts_text_list, key=lambda x:x[0]): lines.append(in_gray(host) + ": " + text) self.labels = self._labels self.N_diffs = next(self._diff_counter)-1 self.diffs = self._diffs self.text = '\n'.join(lines) def _reset_processing_result(self): self.text = None self.labels = None self.N_diffs = None self.diffs = None self._diff_counter = count(1) # Initialize dict for dummy diff-labels self._diffs = {} self._diffs[None] = {} self._diffs[None].update([('h', '[HOSTNAME]'), ('p', '%')]) def _update_seq1(self, lines): # Simply set seq1 in all SequenceMatchers for L, mat in zip(lines, self._matchers): mat.set_seq1(L) def _add_lines(self, lines): # Add a new set of lines, matchers seq1 must already be updated for lmbs, L, mat in zip(self._lmb_pairs, lines, self._matchers): lmbs.append((L, mat.get_matching_blocks())) #This is what's taking time def _push_label_name(self, label=None): if label is None: self._labels.append('text%i' % next(self._label_counter)) else: self._labels.append(label) def _process_line(self, lmbs): # Get non- and minimal matching blocks for a set of lines. # lmbs is a list of (line, matching_blocks)-pairs, for # all lines to consider. # # * Using a mask, construct a "minimum match" # * Sort out "supremum non-match"/"minumum match" into # matching-blocks-like format and # * create a line template with escaped %-signs ref_L, _ = lmbs[0] ref_L_N = len(ref_L) part_mask = [1] * (ref_L_N+1) #Extra element to indicate trailing diff for _, mb in lmbs[1:]: #Next matching-block ipn = jpn = 0 for i, j, n in mb: if ((i != 0 or j != 0) and (jpn != j or ipn != i)): #There is a diff, update the partion mask for jind in range(jpn, j): part_mask[jind] = False for jind in range(j, ref_L_N+1): if part_mask[jind]: part_mask[jind] += 1 jpn = j + n ipn = i + n part_mask, end_part = part_mask[:-1], part_mask[-1] # collect combined non and minimal matching blocks (nm_nb) # between the reference line and all other lines. # False = non-matching, True = matching. # non-matching blocks can be 0-length (n==0). # Also, when we are at it, create an affiliated line_template. nm_mb = [] diff_idxs = [] line_template = [] def push_m(j, n): nm_mb.append((True, j, n)) line_template.append((True, ref_L[j:j+n].replace('%', '%(p)s'))) def push_n(j, n): nm_mb.append((False, j, n)) di = next(self._diff_counter) line_template.append((False, '%%(%i)s' % di)) diff_idxs.append(di) opj = 0 op = 1 for j in range(0, ref_L_N): if part_mask[j] is False: if op is False: # Continue along non-matching block continue elif j-opj > 0: # from matching to non-matching block push_m(opj, j-opj) else: if part_mask[j] == op: # Continue along matching block continue elif op is False: # from non-matching to matching block push_n(opj, j-opj) else: # go between matching (via 0-length non-matching) if j-opj > 0: push_m(opj, j-opj) push_n(j, 0) op = part_mask[j] opj = j if op is False: push_n(opj, ref_L_N-opj) else: push_m(opj, ref_L_N-opj) if op != end_part: push_n(ref_L_N, 0) if len(diff_idxs) > 0: # This line contains differences, # find them and store them. # Create dummy diffs d = self._diffs[None] d.update([(str(k), '[DIFF%i]'%k) for k in diff_idxs]) index = [-1]*(ref_L_N+2) padded_mb = [(True, 0, 0)] + nm_mb + [(True, ref_L_N, 0)] diff_i0 = diff_idxs[0] for label, lmb in zip(self._labels, lmbs): L, mb = lmb index[ref_L_N] = len(L) for i, j, n in mb: index[j:j+n] = range(i, i+n) dc = count(diff_i0) diffs = self._diffs[label] for mb_i in range(1, len(padded_mb)-1): if padded_mb[mb_i][0] is False: # False == non-matching block _, jA, nA = padded_mb[mb_i-1] _, jB, _ = padded_mb[mb_i+1] iA = index[jA+nA-1]+1 iB = index[jB] diffs[str(next(dc))] = L[iA:iB] return line_template, len(diff_idxs) # MAIN op = optparse.OptionParser(usage="usage: %prog [OPTION]...", add_help_option = False) op.add_option("-c", "--collect", action="store_true", help="Collect identical output.") op.add_option("-C", "--collect-similar", action="store_true", help="Collect similar output. Collects output that is " "in relatively close proximity to each other. ") C_group = optparse.OptionGroup(op, 'Collect-similar mode options', 'Options that can be used together with ' 'the --collect-similar mode. These options ' 'control if/how the "collected-part" and ' 'differences should be displayed.') C_group.add_option("", "--spliced-diff", action="store_true", help="Print differing lines, spliced into the indentical part.") C_group.add_option("", "--no-collected", action="store_true", help="Don't print the collected part.") C_group.add_option("", "--format-diff", metavar="FORMAT", help="After the collected part, print the individual " "differences for each host as specified by FORMAT. " "FORMAT is a string where $h and $1, $2, ... will be substituted " "with hostname and difference 1, 2, etc. " "E.g. --format-diff='$h: $3 $1'") op.add_option_group(C_group) op.add_option("-d", "--unified-diff", action="store_true", help="Print the most frequent output in its full form, " "and all other outputs as unified diffs " "relative the most frequent output. This option implies --collect.") op.add_option("-n", "--count", action="store_true", help="Show the number of hosts in the header.") op.add_option("-g", "--with-garbage", action="store_true", help="Also collect and print input not conforming to the " ' "host : output"-format. ' "Garbage output will be presented separated from host output.") op.add_option("", "--color", metavar='Y/N', help="Force ANSI color usage; either y[es] or n[o]. " "ANSI colors will be used by default for tty output.") op.add_option("-h", "--help", action="help", help="Show help") op.add_option("--version", action="store_true", help="Show version") (opts, args) = op.parse_args() if opts.version: print("Version %s (library version %s)" % (__version__, library_version)) sys.exit() # Check to see if we should override use of colors (default = yes for tty) if opts.color: if (opts.color.lower() in ["n", "no", "0"]): in_red = in_blue = in_gray = in_plain elif not (opts.color.lower() in ["y", "yes", "1"]): sys.stderr.write('Error: expected --color y[es] or --color n[o]\n') sys.exit(1) elif not sys.stdout.isatty(): in_red = in_blue = in_gray = in_plain try: text_dict = scan() if opts.with_garbage and None in text_dict: # The user wants to see garbage-output, and it is non-empty. # Print it before the host output regardless of diff/normal mode. output('NON-FORMATTED OUTPUT', text_dict[None]) # Remove garbage lines from text_dict text_dict.pop(None,None) if opts.unified_diff: # "Unified diff mode", print the most abundant output, O, in full, and all # other outputs as a unified diff relative O. hosts_text_list = collect(text_dict) # Sort in descending order of the number of hosts hosts_text_list.sort(key = lambda x: -len(x[0])) if len(hosts_text_list) == 0: sys.exit() # The most abundant output ref_host_set, ref_text = hosts_text_list.pop(0) ref_hostlist = collect_hostlist(ref_host_set) # Split into lines for use with difflib. ref_lines = ref_text.split('\n') output(ref_host_set, ref_text, opts.count) for host_set, text in hosts_text_list: output(host_set, '\n'.join(unified_diff(ref_lines, text.split('\n'), fromfile=ref_hostlist, tofile=collect_hostlist(host_set), lineterm='')), opts.count) elif opts.collect_similar: collects = [] for host, text in text_dict.items(): picked = False for c in collects: picked = c.try_add_text(text, host) if picked: break if not picked: collects.append(Collector(text, host)) if opts.format_diff: fmt_re = re.compile(r'\$(h|[1-9][0-9]*)|\${(h|[1-9][0-9]*)}') for c in collects: c.process(opts.spliced_diff, opts.collect) if not opts.no_collected: output(set(c._labels), c.text, opts.count) if opts.format_diff and c.N_diffs > 0: def fmt_fun(m): v = m.group(1) or m.group(2) if v.isdigit() and int(v) > c.N_diffs: return '' return '%%(%s)s' % v fmt = fmt_re.sub(fmt_fun, opts.format_diff.replace('%', '%(p)s')) lines = [(h, c.render_line(fmt, h)) for h in c.labels] for _, line in sorted(lines, key=lambda x:x[0]): print(line) else: # "Normal" mode, just print the output if opts.collect: hosts_text_list = collect(text_dict) else: hosts_text_list = [(set((n,)), l) for n,l in text_dict.items()] # Sort in descending order of the number of hosts hosts_text_list.sort(key = lambda x: -len(x[0])) for host_set, text in hosts_text_list: output(host_set, text, opts.count) except KeyboardInterrupt: sys.exit() except IOError as e: if e.errno == 32: # IOError Broken Pipe, ignore this sys.exit() raise ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/pshbak.10000644000175000017500000000671214723572005014775 0ustar00kentkent.TH pshbak 1 "Version 2.2.1" .SH NAME pshbak \- format output from pdsh command .SH SYNOPSIS .B pshbak .RI [ OPTION "]... " .SH DESCRIPTION .B pshbak formats output from parallel shells such as .B pdsh and .BR dsh , into a more readable output. It can be used as a drop-in replacement for .BR dshbak . Output from .BR pdsh , where each line is prefixed by a hostname followed by a colon, is read via stdin, collected per hostname, and printed in a per hostname fashion. With the often used .B -c option, hosts having identical output are collected and the output is printed only once. .SH OPTIONS .TP .B -c, --collect Collect hosts having identical output and make the identical output appear to only once. The collected hostlist representing the nodes is shown in the header before the output. .TP .B -d, --unified-diff Print only the most frequent output in full. Then print any diverging output as a unified diff relative the most frequent output. This mode of operation implies the .B --collect option .TP .B -C, --collect-similar Collect hosts having relatively similar output, and print the identical part of the output only once. The parts of the output which differs among the hosts are replaced with enumerated tags ([DIFF1], [DIFF2], ...). For two outputs to be considered relatively similar, they must contain the same number of lines, and in total, the lines must not be too different. The differences can be printed using either the .BR --format-diff or the .BR --spliced-diff option. .TP .B --spliced-diff When using .BR --collect-similar "," lines containing differences are prefixed by its hostname and printed together with the collected part of the output. .TP .B --no-collected When using .BR --collect-similar "," don't print the collected part. This is intended to be useful together with the .B --format-diff option. .TP \fB\--format-diff\fR \fIFORMAT\fR When using .BR --collect-similar "," after to collected part, format and print the differences for each host as specified by \fIFORMAT\fR. \fIFORMAT\fR is expected to be a text string containing zero or more occurrences of $h, $1, $2, $3..., which in the output will be replaced by hostname, DIFF1, DIFF2, DIFF3, etc. .TP .B -n, --count Show the number of hosts in the header (before the hostlist). .TP .B -g, --with-garbage Also print any output not conforming to the "host:output" format in a special .I NON-FORMATTED OUTPUT section before the correctly formatted data. The default mode of operation is to silently ignore non-conforming lines. .TP \fB\--color\fR [yes|no] Control whether ANSI color codes should be used. The default behavior is to enable ANSI color codes when writing to a normal tty, and disable it otherwise. .TP .B -h, --help Display a brief help message. .SH EXAMPLES .TP Collect information about the running kernel version in a \ compute cluster having node names n1, n2, ..., n400 using pdsh and pshbak: .B pdsh -w n[1-400] uname -r | .B pshbak -c .SH AUTHOR Written by Mattias Slabanja . The program is published at http://www.nsc.liu.se/~kent/python-hostlist/ .SH SEE ALSO .IR hostlist "(1), " pdsh "(1), " dshbak "(1)" The hostlist expression syntax is used by several programs developed at .B LLNL (https://computing.llnl.gov/linux/), for example .B SLURM (https://computing.llnl.gov/linux/slurm/) and .B Pdsh (http://code.google.com/p/pdsh/). See the .B HOSTLIST EXPRESSIONS section of the .BR pdsh (1) manual page for a short introduction to the hostlist syntax. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/python-hostlist.spec0000644000175000017500000000774214723572005017513 0ustar00kentkent%global srcname hostlist %define py3_shbang_opts -E %define extra_install_args --prefix /usr Name: python-%{srcname} Version: 2.2.1 Release: 1%{?dist} Summary: Python module for hostlist handling Vendor: NSC Group: Development/Languages License: GPL2+ URL: http://www.nsc.liu.se/~kent/python-hostlist/ Source0: http://www.nsc.liu.se/~kent/python-hostlist/%{name}-%{version}.tar.gz BuildArch: noarch %global _description %{expand: The hostlist.py module knows how to expand and collect Slurm hostlist expressions. The package also includes the 'hostlist' binary which can be used to collect/expand hostlists and perform set operations on them, 'pshbak' which collects output like 'dshbak' but using our hostlist library, 'hostgrep' which is a grep-like utility that understands hostlists, and 'dbuck' which summarizes numerical data from multiple hosts.} %description %_description %package -n python3-%{srcname} Summary: %{summary} BuildRequires: python%{python3_pkgversion}-devel python3-setuptools %description -n python3-%{srcname} %_description %prep %autosetup %build %py3_build %install %py3_install -- %{?extra_install_args} %files -n python3-%{srcname} %defattr(-,root,root,-) %{python3_sitelib}/* %{python3_sitelib}/__pycache__/* %doc README %doc COPYING %doc CHANGES /usr/bin/hostlist /usr/bin/hostgrep /usr/bin/pshbak /usr/bin/dbuck %{_mandir}/man1/hostlist.1.gz %{_mandir}/man1/hostgrep.1.gz %{_mandir}/man1/pshbak.1.gz %{_mandir}/man1/dbuck.1.gz %changelog * Mon Dec 2 2024 Kent Engström - 2.2.1-1 - Rework dist tag handling for mock build. * Fri Nov 29 2024 Kent Engström - 2.2.0-1 - Complain about spurious whitespace in "hostlist" tool arguments. - Remove dist tag for initial SRPM creation in a new way. * Fri Nov 22 2024 Kent Engström - 2.1.0-1 - Remove whitespace from hostlist arguments in the "hostlist" tool. * Fri Oct 25 2024 Kent Engström - 2.0.0-1 - Remove Python 2 support claim from package metadata - Bump major version number due to removal of Python 2 support * Tue Oct 15 2024 Torbjörn Lönnemark - 1.25.0-1 - Migrate from distutils to setuptools - Drop support for Python 2 * Tue Apr 16 2024 Torbjörn Lönnemark - 1.24.0-1 - Always install tools as part of the Python 3 package * Wed Nov 30 2022 Torbjörn Lönnemark - 1.23.0-1 - Fix TypeError in Python 3 when collecting 'n n1' - Build python2 packages on <= el8 by default * Tue Oct 11 2022 Torbjörn Lönnemark - 1.22-1 - hostgrep: dynamically add characters allowed in hostnames. - Make python2 support opt-in at build time * Mon Oct 19 2020 Torbjörn Lönnemark - 1.21-1 - Fixes for building on el8 * Tue Jan 14 2020 Kent Engström - 1.20-1 - Adapt to Python 3 stricter comparison rules - Fix Python 2+2 support for hostgrep, pshbak, dbuck * Mon Sep 30 2019 Torbjörn Lönnemark - 1.19-1 - dbuck: Don't print hostlist padding for empty buckets * Thu Jun 21 2018 Kent Engström - 1.18-1 - Accept whitespace in hostlists passed as arguments - Support both Python 2 and Python 3 natively * Mon Jan 23 2017 Kent Engström - 1.17-1 - New features in dbuck by cap@nsc.liu.se: - Add option -z, --zero - Add option -b, --bars - Add option --highligh-hostlist and --color - Add option -a, --anonymous - Add option -p, --previous and --no-cache - Also other fixes and cleanups in dbuck * Mon May 23 2016 Kent Engström - 1.16-1 - Ignore PYTHONPATH et al. in installed scripts * Thu Apr 21 2016 Kent Engström - 1.15-1 - Add missing options to the hostgrep(1) man page. - Add --restrict option to hostgrep. - Add --repeat-slurm-tasks option. - dbuck: major rewrite, add -r/-o, remove -b/-m - dbuck: add a check for sufficient input when not using -k - dbuck: Fix incorrect upper bound of underflow bucket ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1733227525.9063692 python-hostlist-2.2.1/python_hostlist.egg-info/0000755000175000017500000000000014723572006020402 5ustar00kentkent././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/python_hostlist.egg-info/PKG-INFO0000644000175000017500000000130714723572005021477 0ustar00kentkentMetadata-Version: 2.1 Name: python-hostlist Version: 2.2.1 Summary: Python module for hostlist handling Home-page: http://www.nsc.liu.se/~kent/python-hostlist/ Author: Kent Engström Author-email: kent@nsc.liu.se License: GPL2+ Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Science/Research Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: GNU General Public License (GPL) Classifier: Topic :: System :: Clustering Classifier: Topic :: System :: Systems Administration Classifier: Programming Language :: Python :: 3 License-File: COPYING The hostlist.py module knows how to expand and collect Slurm hostlist expressions. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/python_hostlist.egg-info/SOURCES.txt0000644000175000017500000000050414723572005022264 0ustar00kentkentCHANGES COPYING MANIFEST.in README dbuck dbuck.1 hostgrep hostgrep.1 hostlist hostlist.1 hostlist.py pshbak pshbak.1 python-hostlist.spec setup.py python_hostlist.egg-info/PKG-INFO python_hostlist.egg-info/SOURCES.txt python_hostlist.egg-info/dependency_links.txt python_hostlist.egg-info/top_level.txt test/test_hostlist.py././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/python_hostlist.egg-info/dependency_links.txt0000644000175000017500000000000114723572005024447 0ustar00kentkent ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/python_hostlist.egg-info/top_level.txt0000644000175000017500000000001114723572005023123 0ustar00kentkenthostlist ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1733227525.9063692 python-hostlist-2.2.1/setup.cfg0000644000175000017500000000004614723572006015257 0ustar00kentkent[egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1733227525.0 python-hostlist-2.2.1/setup.py0000644000175000017500000000257414723572005015157 0ustar00kentkentfrom setuptools import setup # Version VERSION = "2.2.1" if "#" in VERSION: import sys sys.stderr.write("Bad version %s\n" % VERSION) sys.exit(1) setup(name = "python-hostlist", version = VERSION, description = "Python module for hostlist handling", long_description = "The hostlist.py module knows how to expand and collect Slurm hostlist expressions.", author = "Kent Engström", author_email = "kent@nsc.liu.se", url = "http://www.nsc.liu.se/~kent/python-hostlist/", license = "GPL2+", classifiers = ['Development Status :: 5 - Production/Stable', 'Intended Audience :: Science/Research', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: GNU General Public License (GPL)', 'Topic :: System :: Clustering', 'Topic :: System :: Systems Administration', 'Programming Language :: Python :: 3', ], py_modules = ["hostlist"], scripts = ["hostlist", "hostgrep", "pshbak", "dbuck"], data_files = [("share/man/man1", ["hostlist.1", "hostgrep.1", "pshbak.1", "dbuck.1"])], ) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1733227525.9053693 python-hostlist-2.2.1/test/0000755000175000017500000000000014723572006014415 5ustar00kentkent././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1672739581.0 python-hostlist-2.2.1/test/test_hostlist.py0000644000175000017500000001216714354775375017724 0ustar00kentkentfrom hostlist import expand_hostlist, collect_hostlist, BadHostlist import unittest class TestExpand1(unittest.TestCase): def expand_eq(self, hostlist, expanded_list): self.assertEqual(expand_hostlist(hostlist), expanded_list) def expand_sort_eq(self, hostlist, expanded_list): self.assertEqual(expand_hostlist(hostlist, sort=True), expanded_list) def expand_length(self, hostlist, expanded_length): self.assertEqual(len(expand_hostlist(hostlist)), expanded_length) def expand_bad(self, hostlist): self.assertRaises(BadHostlist, expand_hostlist, hostlist) def test_expand(self): self.expand_eq("n[9-11]", ["n9", "n10", "n11"]) self.expand_sort_eq("n[9-11]", ["n9", "n10", "n11"]) self.expand_eq("n[09-11]", ["n09", "n10", "n11"]) self.expand_eq("n[009-11]", ["n009", "n010", "n011"]) self.expand_sort_eq("n[009-11]", ["n009", "n010", "n011"]) self.expand_eq("n[009-011]", ["n009", "n010", "n011"]) self.expand_eq("n[17-17]", ["n17"]) self.expand_eq("n1,n3", ["n1", "n3"]) self.expand_sort_eq("n1,n3", ["n1", "n3"]) self.expand_eq("n3,n1", ["n3", "n1"]) self.expand_sort_eq("n3,n1", ["n1", "n3"]) self.expand_eq("n1,n3,n1", ["n1", "n3"]) self.expand_sort_eq("n1,n3,n1", ["n1", "n3"]) self.expand_eq("n3,n1,n3", ["n3", "n1"]) self.expand_sort_eq("n3,n1,n3", ["n1", "n3"]) self.expand_eq("n[1],n3", ["n1", "n3"]) self.expand_eq("n[1,3]", ["n1", "n3"]) self.expand_eq("n[3,1]", ["n3", "n1"]) self.expand_sort_eq("n[3,1]", ["n1", "n3"]) self.expand_eq("n[1,3,1]", ["n1", "n3"]) self.expand_eq("n1,n2,n[9-11],n3", ["n1", "n2", "n9", "n10", "n11", "n3"]) self.expand_eq("n[1-3]m[4-6]", ["n1m4", "n1m5", "n1m6", "n2m4", "n2m5", "n2m6", "n3m4", "n3m5", "n3m6"]) self.expand_eq("n[1-2][4-5]m", ["n14m", "n15m", "n24m", "n25m"]) self.expand_eq("[1-2][4-5]", ["14", "15", "24", "25"]) self.expand_length("n[1-100]m[1-100]", 100*100) self.expand_length("[1-10][1-10][1-10]", 10*10*10) self.expand_eq("n[1-5,3-8]", ["n1", "n2", "n3", "n4", "n5", "n6", "n7", "n8"]) self.expand_eq("n[3-8,1-5]", ["n3", "n4", "n5", "n6", "n7", "n8", "n1", "n2"]) self.expand_sort_eq("n[3-8,1-5]", ["n1", "n2", "n3", "n4", "n5", "n6", "n7", "n8"]) self.expand_sort_eq("[3-4]n,n[1-2]", ["3n", "4n", "n1", "n2"]) self.expand_eq("", []) self.expand_bad("n[]") self.expand_bad("n[-]") self.expand_bad("n[1-]") self.expand_bad("n[-1]") self.expand_bad("n[1,]") self.expand_bad("n[,1]") self.expand_bad("n[1-3,]") self.expand_bad("n[,1-3]") self.expand_bad("n[3-1]") self.expand_bad("n[") self.expand_bad("n]") self.expand_bad("n[[]]") self.expand_bad("n[1,[]]") self.expand_bad("n[x]") self.expand_bad("n[1-10x]") self.expand_bad("n[1-1000000]") self.expand_bad("n[1-1000][1-1000]") def collect_eq(self, hostlist, expanded_list): # Note the order of the arguments! This makes it easier to # copy tests between the expand and collect parts! self.assertEqual(hostlist, collect_hostlist(expanded_list)) def test_collect(self): self.collect_eq("n[9-11]", ["n9", "n10", "n11"]) self.collect_eq("n[09-11]", ["n09", "n10", "n11"]) self.collect_eq("n[009-011]", ["n009", "n010", "n011"]) self.collect_eq("n[1-3,9-11]", ["n1", "n2", "n9", "n10", "n11", "n3"]) self.collect_eq("m1,n[9-11],p[7-8]", ["n9", "n10", "p7", "m1", "n11", "p8"]) self.collect_eq("x[1-2]y[4-5]", ["x1y4", "x1y5", "x2y4", "x2y5"]) self.collect_eq("[1-2]y[4-5]z", ["1y4z", "1y5z", "2y4z", "2y5z"]) self.collect_eq("x1y[4-5],x2y4", ["x1y4", "x1y5", "x2y4"]) self.collect_eq("x1y5,x2y[4-5]", ["x1y5", "x2y4", "x2y5"]) self.collect_eq("x1y5,x2y4", ["x1y5", "x2y4"]) self.collect_eq("", [""]) self.collect_eq("n[9,09]", ["n09","n9"]) self.collect_eq("n[9,09]", ["n9","n09"]) self.collect_eq("n[9-10]", ["n9","n10"]) self.collect_eq("n[09-10]", ["n09","n10"]) self.collect_eq("n[009,10]", ["n009","n10"]) self.collect_eq("x", ["x"]) self.collect_eq("x", ["x", "x"]) self.collect_eq("x,y", ["x", "y", "x"]) self.collect_eq("n1", ["n1"]) self.collect_eq("n1", ["n1", "n1"]) self.collect_eq("n[1-2]", ["n1", "n2", "n1"]) self.collect_eq("x,y[10-12],z", ["z","y10","y12", "x", "y11"]) self.collect_eq("[3-4]n,n[1-2]", ["n1","n2","3n","4n"]) # Same base host name with and without a number # Test for new regression introduced with Python 3 self.collect_eq("n,n1", ["n", "n1"]) self.collect_eq("n,n1", ["n1", "n"]) if __name__ == '__main__': unittest.main()