pax_global_header00006660000000000000000000000064124025541030014506gustar00rootroot0000000000000052 comment=1b6c926f462306dd2fb190b79de5fdb74894689b arriero-0.6/000077500000000000000000000000001240255410300130165ustar00rootroot00000000000000arriero-0.6/.gitignore000066400000000000000000000000321240255410300150010ustar00rootroot00000000000000/debian/ /debian/** *.pyc arriero-0.6/AUTHORS000066400000000000000000000001131240255410300140610ustar00rootroot00000000000000Maximiliano Curia Margarita Manterola arriero-0.6/COPYING000066400000000000000000000432541240255410300140610ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. arriero-0.6/Changelog000066400000000000000000000011151240255410300146260ustar00rootroot00000000000000Version 0.6: * Make filter-orig a pattern list. * Allow the user to specify extra git-buildpackage options when building (-o). * Add feature to uscan class that allows to download arbitrary versions. * Ignore upstream releases when they have no changes. * Use downloaded tarballs, if present. * Create debian branch on pull, if missing. * Use current directory if no package name is received. * Several bugfixes regarding handling of configuration file * Added 'urgency' property for changelog, 'source_name' for package * Improve status output Version 0.5: * Initial release. arriero-0.6/TODO000066400000000000000000000066761240255410300135250ustar00rootroot00000000000000This list doesn't pretend to be complete, it's sort of a roadmap, or a pensieve for some bothering issues. Fetch upstream and friends: - Fix Uscan uversion and version. upstream-version, latest upstream version debian-uversion, upstream version of current package debian-mangled-version, after removing the dfsg part - Fix Uscan, no tarball found - Fix Uscan, download current version - Fix Uscan, dfsg rule If there is not a new version the output is confusing. - Add support of multiple upstream tarballs: http://raphaelhertzog.com/2010/09/07/how-to-use-multiple-upstream-tarballs-in-debian-source-packages/ Might need to hack uscan, again Some packages don't have a watch file, and some might need a feature not present in uscan (xawtv, for example), or upstream might not have any releases (cinnamon-themes), another example is kde-l10n, which uses a multiwatch file, sort of supported by uscan and arriero. Gbp configuration: - Per package configuration: + Read gbp config to know: upstream-branch, debian-branch - Use the gbp configuration, when possible Package cloning: - Add the branches to sync in .git/config (in clone) +++ IMPORTANT +++ - Download the current version instead of new version (if needed). - Add autoconfigure to clone: + if there is an origin/upstream branch, upstream is pushed + if there is a origin/pristine-tar branch add pristine-tar + configure this in the .git/gbp.conf if not configured in the debian/gbp.conf - Optionally do not ignore out of scope packages - Check packages bugs - If there's no basedir anywhere, we should use the current dir - Add a configuration for a rule that generates the dfsg version + arriero users prune-nonfree (it should be configurable) + uscan supports copyright format 1.0 exclude-files (need to test this with arriero) + gbp uses filters (filter-orig in arriero) - Add a logcheck like functionality - Obtain a set of packages for a given module version - Make sure to apply the packaging changes (commits) always in the "unmerged" branch. - Make the "unmerged" branch to work with command line configurable, allow aliases (codename/releases) and make it configurable per package - Export build dir to work outside the git repo - Allow an upstream remote - Add a Log class, to handle lintian style reporting - Move the Graph usage to it's own class - Clone needs to fetch the upstream branch - Verify that the command used to clone exists - Allow to choose between pbuilder/cowbuilder/qemubuilder/debuild/sbuild - Document the default setup requirements - Parse build log and extract errors. - Clean up after overlay Addons ------ * Allow having extra files that define extra actions. These files should list the actions that they define, with a help string. Arriero would then read from the files according to the configuration. * Also add the possibility of running external scripts that receive some specified environment. Distro branches --------------- * Allow having different branches for different distributions (stable, unstable, experimental). * Could be achieved by having a file that defines different profiles (kde4.11, kde4.10, etc), with each profile including distro, branch, and maybe other stuff. * Unresolved: how to associate each package with the available profiles. Status ------ * Add a status that indicates if the package needs to be uploaded or not (if it's not UNRELEASED), using rmadison: e.g. rmadison -a amd64 -s unstable cinnamon -u debian arriero-0.6/arriero.1000066400000000000000000000152331240255410300145470ustar00rootroot00000000000000.TH "arriero" 1 "2014 Mar 11" "Debian" "arriero" .SH NAME arriero \- simplifies management of several Debian packages .SH SYNOPSIS .BI "arriero [" "--config FILE" "] [" --verbose "] [" --quiet "] command [" "options" "] [" "package names" "]" .SH "DESCRIPTION" Arriero is a tool that allows simplifying the management of Debian packages, particularly useful when having to make new upstream releases, builds and uploads of similar packages. . It relies heavily in the use of .B git-buildpackage and general .B git practices, so it's only useful for packages currently maintained through git. .SH "GENERAL OPTIONS" .TP .BR -c , "--config " \fIFILE\fR Specifies the location of the config file to use. The config file holds all information related to packages. It's recommended to have different config files in order to work with different groups of packages. If not specified, the default config file .I ~/.config/arriero.conf is read. .TP .BR -v , " --verbose " Show info and debug messages. By default, only warnings and errors are shown. .TP .BR -q , " --quiet " Only show critical errors. If both \fBquiet\fR and \fBverbose\fR are specified, \fBverbose\fR is honored. .TP .BR -a , " --all " Work with all packages. When this option is not specified, package names need to be specified following the command option, separated by spaces. .SH "COMMANDS" The main action that arriero will perform is determined by the command it receives. Each command may have its own specific options, that modify its behavior. .SS .B "build" Build each package in a pbuilder. This will call .B git-pbuilder which will read local configurations from .IR /etc/pbuilderrc " and " ~/.pbuilderrc . .TP .BR -D , " --distribution " , " --dist " \fIdist-name\fR Build the package for the specified distribution .TP .BR -A , " --architecture " , " --arch " \fIarch-name\fR Build the package for the specified architecture .TP .BR -U , " --local-upload " After a successful build is finished, the package is uploaded, using the \fIupload-command\fR, using \fIlocal\fR as the host to upload to. .SS .B "clone" Obtain the repository for each package. This command can either receive a list of package names or a git URL to clone from. When specifying a URL, it will create a new entry in the configuration file; if specifying a package name, it needs to already be present in the configuration. .TP .BR --basedir The base directory in which to create the clone. After making the clone successfully, the package will be located in \fIbasedir/package_name\fR .TP .BR --upstream-branch The branch where the upstream code is located. .TP .BR --debian-branch The branch where the Debian code is located. When performing a clone from a URL, if the branches are not manually specified, arriero will try to guess their names, and store the guessed names in the configuration file. .SS .B "exec" Execute one or more scripts for each package. The scripts invoked will receive the properties of the packages as environment variables, and will be executed inside the package directory. .TP .BR -x , " --script " \fIscript_name\fR The name of the script to be executed. This option can be present multiple times. In that case, each script will be called, in the same order as presented in the command line. If one of the scripts fails for a certain package, the following ones will not be executed for that package .SS .B "fetch-upstream" Fetch the current upstream tarball for each package. .SS .B "list" List packages matching some criteria, with a specific format. This command allows specifying the desired format with which each package is going to be displayed. .TP .BR -f , " --fields " \fIfield_list\fR Fields to include while generating the list. The list of fields should be comma separated. The fields available are: .RS .TP basedir .TP branch .TP build_file .TP changes_file .TP debian_branch .TP depends .TP distribution .TP export_dir .TP is_dfsg .TP is_native .TP is_merged .TP name .TP path .TP pristine_tar_branch .TP tarball_dir .TP upstream_branch .TP upstream_version .TP vcs_git .TP version .RE .TP .BR -F , " --format " \fIfield_format\fR The format to use may include fields by name or order, as specified in the \fI--fields\fR parameter. .TP .BR -e , " --include-empty-results" By default, results where nothing would be listed are skipped, if this option is specified, they will be shown even when there is no string to show. .SS .B "overlay" Combine upstream and debian branches into either the original debian branch, or a new branch. This command is intended to be used when the debian branch doesn't include the upstream code and the user needs to have them together in order to work on the package (for example, to create a quilt package). .B Important: this command does not handle cleaning up the branch after the work is done. This has to be done manually by the user. .TP .BR -b , " --branch " \fIbranch-name\fR The name of the new branch to create with the overlay. If specified and the branch already exists, the command will fail without modifying anything. If not specified, the debian branch for the package will be used. .SS .B "pull" Obtain any new changes from the packages' repositories. .SS .B "push" Push local changes to the packages' repositories. .SS .B "release" Change the distribution in the changelog, committing the change to the local git. This command only has effect if the distribution in the changelog is either UNRELEASED or different than the one passed here. .TP .BR -D , "--distribution " \fIdistribution-name\fR The distribution to make the release to. .TP .BR -P , --pre-release If this option is received, the release will contain a ~ after the debian version. The number after the ~ will get incremented each time the release command is called. This allows for maitainers to keep track of internal tests until it's time to actually release the package. If this option is not passed, but the version in the changelog was already a pre-release (i.e. it contained a ~), the it's modified to be a final release (without ~) .SS .B "status" Show the status of each package. This command checks both the repository state (by using git to query any local/remote changes) and the upstream state (by using uscan) .SS .B "update" Get the new upstream release for each package. This command not only downloads the new upstream tarball, but also updates the debian/changelog with a new entry for the new release, with distribution set to UNRELEASED. .SS .B "upload" Upload each package. This command uses the upload-command set in the config file to upload each built package (packages that have not been built are ignored). \" .SH "EXAMPLES" \" TODO .SH "AUTHORS" Maximiliano Curia , Margarita Manterola arriero-0.6/bin/000077500000000000000000000000001240255410300135665ustar00rootroot00000000000000arriero-0.6/bin/arriero000077500000000000000000000003341240255410300151570ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf8 -*- import sys from arriero import arriero if __name__ == '__main__': exit_code = arriero.main() sys.exit(exit_code) # vi:expandtab:softtabstop=4:shiftwidth=4:smarttab arriero-0.6/examples/000077500000000000000000000000001240255410300146345ustar00rootroot00000000000000arriero-0.6/examples/arriero.conf000066400000000000000000000121341240255410300171470ustar00rootroot00000000000000[DEFAULT] basedir=~/local/kde upload-command = reprepro -Vb ~/share/reprepro include %(distribution)s %(changes_file)s [kde-req] packages: akonadi soprano # packages: akonadi automoc eigen2 phonon polkit-qt-1 soprano path: %(basedir)s/kde-req/%(package)s [soprano] debian-branch: experimental #[polkit-kde] #packages: polkit-kde-1 #path: %(basedir)s/kde-req/%(package)s #depends: kde4libs [kde_libs] packages: kactivities kde4libs nepomuk-core nepomuk-widgets path: %(basedir)s/kde-sc/%(package)s depends: kde-req debian-branch: kde4.10 [kactivities] depends: kde4libs [nepomuk-core] depends: kde4libs [nepomuk-widgets] depends: nepomuk-core [kdepimlibs] packages: kdepimlibs path: %(basedir)s/kde-sc/%(package)s depends: kde_libs debian-branch: kde4.10 [kdegames] packages: bomber bovo granatier kajongg kapman katomic kblackbox kblocks kbounce kbreakout kdiamond kfourinline kgoldrunner kigo killbots kiriki kjumpingcube klickety klines kmahjongg kmines knavalbattle knetwalk kolf kollision konquest kpat kreversi kshisen ksirk ksnakeduel kspaceduel ksquares ksudoku ktuberling kubrick libkdegames libkmahjongg lskat palapeli picmi path: %(basedir)s/kde-sc/%(package)s depends: kde_libs libkdegames [kajongg] depends: libkdegames libkmahjongg [kmahjongg] depends: libkdegames libkmahjongg [kshisen] depends: libkdegames libkmahjongg [kdemultimedia] packages: audiocd-kio dragon ffmpegthumbs juk kmix kscd libkcddb libkcompactdisc mplayerthumbs path: %(basedir)s/kde-sc/%(package)s depends: kde_libs [audiocd-kio] depends: libkcddb libkcompactdisc [kdeaccessibility] packages: jovie kaccessible kmag kmouth kmousetool path: %(basedir)s/kde-sc/%(package)s depends: kde_libs debian-branch: kde4.10 [kde_base_artwork] packages: kde-base-artwork kde-wallpapers oxygen-icons path: %(basedir)s/kde-sc/%(package)s depends: kde_libs [kde-wallpapers] debian-branch: kde4.10 [oxygen-icons] debian-branch: kde4.10 [kde_baseapps] packages: kde-baseapps kate konsole path: %(basedir)s/kde-sc/%(package)s depends: kde_libs debian-branch: kde4.10 [konsole] depends: kde-baseapps [kdeedu] packages: analitza blinken cantor kalgebra kalzium kanagram kbruch kgeography khangman kig kiten klettres kmplot kstars ktouch kturtle kwordquiz libkdeedu marble pairs parley rocs step path: %(basedir)s/kde-sc/%(package)s depends: kde_libs libkdeedu debian-branch: kde4.10 [cantor] depends: cantor [kalgebra] depends: analitza [kmplot] debian-branch: master [pairs] debian-branch: master [kde-workspace] packages: kde-workspace path: %(basedir)s/kde-sc/%(package)s depends: kdepimlibs debian-branch: kde4.10 [kde-runtime] packages: kde-runtime path: %(basedir)s/kde-sc/%(package)s depends: kdepimlibs debian-branch: kde4.10 [kdegraphics] packages: gwenview kamera kcolorchooser kdegraphics-mobipocket kdegraphics-strigi-analyzer kdegraphics-thumbnailers kgamma kolourpaint kruler ksaneplugin ksnapshot libkdcraw libkexiv2 libkipi libksane okular svgpart path: %(basedir)s/kde-sc/%(package)s # gwenview build-depends on libkonq5-dev depends: kde-baseapps debian-branch: kde4.10 [gwenview] depends: libkipi [kdegraphics-mobipocket] depends: okular [kdegraphics-thumbnailers] depends: libkdcraw libkexiv2 [ksaneplugin] depends: libksane debian-branch: master [ksnapshot] depends: libkipi [kdesdk] packages: kdesdk path: %(basedir)s/kde-sc/%(package)s depends: kde-baseapps debian-branch: kde4.10 [kdeartwork] packages: kdeartwork path: %(basedir)s/kde-sc/%(package)s depends: kde-workspace kdegraphics debian-branch: kde4.10 [kdebindings] # perlqt and perlkde not packaged yet # qyoto -> uics is licensed under a qt preview license. :( # (is it really needed?) packages: korundum kross-interpreters pykde4 qtruby smokegen smokekde smokeqt path: %(basedir)s/kde-sc/%(package)s depends: kdegraphics kdepimlibs smokegen debian-branch: kde4.10 [korundum] depends: qtruby [qtruby] depends: smokekde [qyoto] debian-branch: master [smokekde] depends: smokeqt [kdepim] packages: kdepim kdepim-runtime path: %(basedir)s/kde-sc/%(package)s depends: kdepimlibs debian-branch: experimental [kdewebdev] packages: kdewebdev path: %(basedir)s/kde-sc/%(package)s depends: kdepimlibs debian-branch: kde4.10 [kdeadmin] packages: kdeadmin path: %(basedir)s/kde-sc/%(package)s depends: kdepimlibs debian-branch: kde4.10 [kdeutils] packages: ark filelight kcalc kcharselect kdf kfloppy kgpg kremotecontrol ktimer kwallet print-manager superkaramba sweeper path: %(basedir)s/kde-sc/%(package)s depends: kde-baseapps debian-branch: kde4.10 [kcharselect] debian-branch: master [print-manager] debian-branch: master [kdenetwork] packages: kdenetwork path: %(basedir)s/kde-sc/%(package)s depends: kde-baseapps kde-workspace debian-branch: kde4.10 [kdeplasma-addons] packages: kdeplasma-addons path: %(basedir)s/kde-sc/%(package)s depends: kdegraphics kde-workspace debian-branch: kde4.10 [kdetoys] packages: kdetoys path: %(basedir)s/kde-sc/%(package)s depends: kde-workspace debian-branch: kde4.10 [kde-extras] packages: amarok path: %(basedir)s/kde-extras/%(package)s depends: kde_libs arriero-0.6/examples/gbp.example.conf000066400000000000000000000011121240255410300177000ustar00rootroot00000000000000[git-buildpackage] postbuild = lintian -I --show-overrides --suppress-tags library-not-linked-against-libc $GBP_CHANGES_FILE export-dir = ../build-area/ tarball-dir = ../tarballs/ arch = amd64 [git-import-orig] merge = False filter = ['.svn', '.hg', '.bzr', 'CVS', 'debian/*'] [git-import-dsc] filter = [ 'CVS', '.cvsignore', '.hg', '.hgignore' '.bzr', '.bzrignore', '.gitignore' ] [remote-config pkg-libvirt] remote-url-pattern = ssh://git.debian.org/git/pkg-libvirt/%(pkg)s template-dir = /srv/alioth.debian.org/chroot/home/groups/pkg-libvirt/git-template arriero-0.6/scripts/000077500000000000000000000000001240255410300145055ustar00rootroot00000000000000arriero-0.6/scripts/add_myself_to_uploaders.sh000077500000000000000000000016121240255410300217330ustar00rootroot00000000000000#!/bin/bash for path in $(arriero list -f path "$@"); do echo $path ( cd $path changes=0 uploaders=$(sed -n -r '/^Uploaders:/,/^[^[:space:]]/{ /^(Uploaders:|[[:space:]])/p }' debian/control) if echo "$uploaders" | grep -q 'Maximiliano Curia'; then : else sed -i -r '/^Uploaders:/,/^[^[:space:]]/{ /^(Uploaders:|[[:space:]])/ s|([^,])\s*$|\1,| /^[[:space:]]/ s|^[[:space:]]+| | /^(Uploaders:|[[:space:]])/! i \ Maximiliano Curia }' debian/control changes=1 fi if [ $changes -gt 0 ]; then dch 'Add myself to uploaders.' git commit -a -m 'Add myself to uploaders.' fi ) done arriero-0.6/scripts/bump_debhelper.sh000077500000000000000000000012111240255410300200140ustar00rootroot00000000000000#!/bin/bash for path in $(arriero list -f path "$@"); do echo $path ( cd $path changes=0 if [ $(cat debian/compat) -lt 9 ]; then echo 9 > debian/compat changes=1 fi dhv=$(sed -n -r 's!.*\W(debhelper\s+\(>= ([^)]+)\)).*!\2!p' debian/control) if [ $dhv != "9" ]; then sed -i -r 's!(\Wdebhelper\s+\(>=) [^)]+\)!\1 9)!' debian/control changes=1 fi if [ $changes -gt 0 ]; then dch 'Bump debhelper build-dep and compat to 9.' git commit -a -m 'Bump debhelper build-dep and compat to 9.' fi ) done arriero-0.6/scripts/bump_kde-sc.sh000066400000000000000000000012201240255410300172250ustar00rootroot00000000000000#!/bin/bash MSG='Bump kde-sc-dev-latest build dependency.' for path in $(arriero -c ~/.config/arriero-kde4.12.conf list -f path \ $(sed -n '/^[^# ]/s/:.*//p' ~/tmp/kde4.12)); do echo $path; ( cd $path change=0 if grep -q 'kde-sc-dev-latest ([^)]*)' debian/control && \ ! grep -q 'kde-sc-dev-latest (>= 4:4\.12[^)]*)' debian/control; then sed -i 's/kde-sc-dev-latest ([^)][^)]*)/kde-sc-dev-latest (>= 4:4.12)/' debian/control change=1 fi if [ $change -gt 0 ]; then dch "$MSG"; git commit -a -m "$MSG" fi ) done arriero-0.6/scripts/bump_standards.sh000077500000000000000000000010241240255410300200470ustar00rootroot00000000000000#!/bin/bash for path in $(arriero list -f path "$@"); do echo $path ( cd $path changes=0 version=$(sed -n 's/^Standards-Version:\s*\(\S*\)/\1/p' debian/control) if [ "$version" != "3.9.4" ]; then sed -i 's/^Standards-Version:.*$/Standards-Version: 3.9.4/' debian/control changes=1 fi if [ $changes -gt 0 ]; then dch 'Bump Standards-Version to 3.9.4.' git commit -a -m 'Bump Standards-Version to 3.9.4.' fi ) done arriero-0.6/scripts/need_to_merge.sh000066400000000000000000000006731240255410300176430ustar00rootroot00000000000000#!/bin/bash for path in $(arriero list -f path "$@"); do # echo $path ( cd $path branch=$(git rev-parse --abbrev-ref HEAD) if [ "$branch" != "master" ]; then out=$(git rev-list --left-right --count master..."$branch") master=$(echo $out | sed 's/\s.*//') if [ "$master" -gt 0 ]; then echo $path echo $out fi fi ) done arriero-0.6/scripts/new_repos.sh000066400000000000000000000024311240255410300170420ustar00rootroot00000000000000#!/bin/sh path="$1" vcs_pull="git://git.debian.org/collab-maint/cinnamon" vcs_push="git.debian.org:/git/collab-maint/cinnamon" vcs_upstream="https://github.com/linuxmint" if [ -e "$path" ]; then echo "Directory already exists" > /dev/stderr exit 1 fi name=$(basename "$path") mkdir "$path" cd "$path" git init git remote add origin "${vcs_pull}/${name}.git" git remote set-url --push origin "${vcs_push}/${name}.git" git remote add "$name" "${vcs_upstream}/${name}.git" git checkout --orphan master git reset; git commit --allow-empty -m 'Initial debian branch' git checkout --orphan upstream git reset; git commit --allow-empty -m 'Initial upstream branch' git checkout --orphan pristine-tar git reset; git commit --allow-empty -m 'Initial pristine-tar branch' git push origin master upstream pristine-tar --tags git branch --set-upstream-to origin/master master git branch --set-upstream-to origin/upstream upstream git branch --set-upstream-to origin/pristine-tar pristine-tar git config remote.origin.push refs/heads/master git config --add remote.origin.push refs/heads/upstream git config --add remote.origin.push refs/heads/pristine-tar git config --add remote.origin.push refs/tags/debian/* git config --add remote.origin.push refs/tags/upstream/* git pull --all arriero-0.6/scripts/new_upstream_release.py000077500000000000000000000044261240255410300213010ustar00rootroot00000000000000#!/usr/bin/env python from collections import defaultdict import re import subprocess import sys COMMIT='New upstream release.' MSG=' * New upstream release.\n' changelog_trail = re.compile(r'^ --') changelog_multimaint = re.compile(r'^\s*\[\s*(\b[^]]*\b)\s*\]\s*$') changelog_empty = re.compile(r'^\s*$') entry = re.compile(r' \* New upstream release\.') TRAIL = ('', 0) for path in subprocess.check_output(['arriero', 'list', '-f', 'path'] + sys.argv[1:]).split('\n'): if not path: continue print path output = subprocess.check_output(['git', 'log', 'HEAD~1..HEAD', '--oneline'], cwd=path) last = re.sub('^[^\s]+\s+', '', output.rstrip('\n')) if last != COMMIT: continue filename = '%s/debian/changelog' % (path,) f = open(filename) lines = f.readlines() skip = 0 first_block = [] maintainers = defaultdict(set) maintainer = TRAIL entries = [] for i, line in enumerate(lines): first_block.append(line) # print line, m = changelog_multimaint.match(line) if m: maintainer = (m.group(1), i) elif changelog_empty.match(line): maintainer = TRAIL else: maintainers[maintainer].add(i) if entry.match(line): entries.append((i, maintainer)) if changelog_trail.match(line): break skip = len(first_block) delete = set() # print maintainers for e in entries: delete.add(e[0]) if e[1] == TRAIL: continue maintainers[e[1]].remove(e[0]) if not maintainers[e[1]]: i = e[1][1] delete.add(i) if i > 2 and changelog_empty.match(first_block[i - 1]): delete.add(i - 1) f = open(filename,'w') # f = sys.stdout for i, line in enumerate(first_block): if i == 2: f.write(MSG) if changelog_multimaint.match(line) and i not in delete: f.write('\n') if i in delete: continue f.write(line) for line in lines[skip:]: f.write(line) f.close() subprocess.call(['git', 'commit', '-a', '--amend', '-m', 'New upstream release.'], cwd=path) arriero-0.6/scripts/tag_uploaded.sh000077500000000000000000000002371240255410300174760ustar00rootroot00000000000000#!/bin/bash TAG_VERSION=$(echo $version | tr ':~' '%_') TAG="debian/$TAG_VERSION" DESC="$version $distribution; urgency=$urgency" git tag -s -m "$DESC" $TAG arriero-0.6/scripts/update_vcs_fields.sh000077500000000000000000000013461240255410300205330ustar00rootroot00000000000000#!/bin/bash MSG='Update vcs fields.' for path in $(arriero list -f path "$@"); do echo $path; ( cd $path change=0 if grep -q '^Vcs-Browser:\s*http:\/\/git\.debian\.org' debian/control; then sed -i 's/^Vcs-Browser:\s*http:\/\/git\.debian\.org/Vcs-Browser: http:\/\/anonscm.debian.org\/gitweb/' debian/control change=1 fi if grep -q '^Vcs-Git:\s*git:\/\/git\.debian\.org' debian/control; then sed -i 's/^Vcs-Git:\s*git:\/\/git\.debian\.org/Vcs-Git: git:\/\/anonscm.debian.org/' debian/control change=1 fi if [ $change -gt 0 ]; then dch "$MSG"; git commit -a -m "$MSG" fi ) done arriero-0.6/setup.py000077500000000000000000000007621240255410300145400ustar00rootroot00000000000000#!/usr/bin/env python from distutils.core import setup setup(name='Arriero', version='0.5', description='Arriero Package Helper', author='Maximiliano Cueria', author_email='maxy@debian.org', url='', requires=['git', 'deb822'], package_dir={'arriero': 'src'}, py_modules=[ 'arriero.arriero', 'arriero.actions', 'arriero.moo', 'arriero.uscan', 'arriero.util'], scripts=['bin/arriero'], ) arriero-0.6/src/000077500000000000000000000000001240255410300136055ustar00rootroot00000000000000arriero-0.6/src/__init__.py000066400000000000000000000000001240255410300157040ustar00rootroot00000000000000arriero-0.6/src/actions.py000066400000000000000000000611731240255410300156270ustar00rootroot00000000000000# -*- coding: utf8 -*- # Copyright: 2013-2014, Maximiliano Curia # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the Free # Software Foundation; either version 2 of the License, or (at your option) # any later version. # # This program is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for # more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., 51 # Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import argparse import collections import deb822 import git import logging import os import re import string import subprocess # Own imports import moo import util import uscan class ActionError(Exception): pass class Action(object): '''Abstract class that all actions should subclass.''' def __init__(self, arriero, argv): '''Basic constructor for the class that calls the necessary functions. It follows the Template Pattern. ''' self._arriero = arriero self._parser = argparse.ArgumentParser() self._set_parse_arguments() self._options = self._parser.parse_args(argv) self._process_options() def _get_packages(self, names): '''Expands the list of names and turns it into packages.''' packages = set() modules = self._arriero.list_modules() for name in names: if name in modules: module = self._arriero.get_module(name) packages.update(module.packages) else: packages.add(name) return packages def _set_parse_arguments(self): '''Add parsing arguments to self._parse.''' def _process_options(self): '''Process the options after they were parsed.''' def run(self): '''Execute this action's main goal. Returns True if successful.''' def print_status(self): '''Print a status report of the run.''' # Methods related to parsing arguments # --all def _add_argument_all(self): self._parser.add_argument('-a', '--all', action='store_true') def _process_argument_all(self): # If --all, we ignore whatever was passed as names if self._options.all: self._options.names = self._arriero.list_packages() # names def _add_argument_names(self, strict=True): if strict: names_choices = set(['']) | self._arriero.list_all() else: names_choices = None self._parser.add_argument('names', nargs='*', default='', choices=names_choices) def _process_argument_names(self): if not self._options.names: # Try to find out if we are standing on a current package cwd = os.getcwd() for pkg_name in self._arriero.list_packages(): pkg = self._arriero.get_package(pkg_name) if pkg.path == cwd: self._options.names = [pkg_name] return raise argparse.ArgumentError(None, 'No package names received.') class ActionPackages(Action): def _get_name(self): return self.__class__.__name__ def _set_parse_arguments(self): self._add_argument_all() self._add_argument_names(strict=True) def _process_options(self): self._process_argument_all() self._process_argument_names() def run(self): '''Method for iterating packages and applying a function.''' packages = self._get_packages(self._options.names) self._results = collections.defaultdict(set) for package_name in packages: logging.info('%s: executing %s action.', package_name, self._get_name()) package = self._arriero.get_package(package_name) status = self._package_action(package) self._results[status].add(package.name) if self._results[moo.ERROR]: return False return True def print_status(self): for package in self._results[moo.IGNORE]: logging.info('%s: ignored', package) for package in self._results[moo.ERROR]: logging.error('%s: failed', package) def _package_action(self, package): '''To override.''' class ActionFields(ActionPackages): '''Common class for List and Exec''' available_fields = set([ 'basedir', 'branch', 'build_file', 'changes_file', 'debian_branch', 'depends', 'distribution', 'export_dir', 'i', 'is_dfsg', 'is_native', 'is_merged', 'name', 'path', 'pristine_tar_branch', 'source_name', 'tarball_dir', 'upstream_branch', 'upstream_version', 'urgency', 'vcs_git', 'version', ]) available_sorts = set(('raw', 'alpha', 'build')) def _set_parse_arguments(self): super(ActionFields, self)._set_parse_arguments() self._parser.add_argument('-f', '--fields', default='name') self._parser.add_argument('-s', '--sort', choices=self.available_sorts, default='raw') def _process_options(self, *formats): super(ActionFields, self)._process_options() self.fields = util.split(self._options.fields) self.field_names = {} for field_name in self.fields: if field_name not in self.available_fields: raise ActionError( 'action %s: Field %s not known' % ( self._get_name(), field_name ) ) self.field_names[field_name] = None formatter = string.Formatter() for format_string in formats: if not format_string: continue for (_, field_name, _, _) in formatter.parse(format_string): if not field_name: continue if field_name not in self.available_fields: raise ActionError( 'action %s: Field %s not known' % ( self.__class__.__name__, field_name ) ) self.field_names[field_name] = None def _get_field(self, package, field): obj = getattr(package, field) if hasattr(obj, '__call__'): result = obj() else: result = obj return result if isinstance(result, (str, unicode)) else str(result) def _get_field_values(self, package, known=None): values = [] by_name = {} for field in self.field_names: if known and field in known: field_value = known[field] else: field_value = self._get_field(package, field) by_name[field] = field_value for field in self.fields: values.append(by_name[field]) return values, by_name def _get_packages(self, names): return self._sorted(super(ActionFields, self)._get_packages(names)) def run(self): '''Method for iterating packages and applying a function.''' packages = self._get_packages(self._options.names) self._results = collections.defaultdict(set) for i, package_name in enumerate(packages): logging.info('%s: executing %s action.', package_name, self._get_name()) package = self._arriero.get_package(package_name) status = self._package_action_i(package, i) self._results[status].add(package.name) if self._results[moo.ERROR]: return False return True # Sort methods def raw_sort(self, packages): return packages def alpha_sort(self, packages): return sorted(packages) def build_sort(self, packages): return self._arriero.sort_buildable(packages) def _sorted(self, packages): method_name = self._options.sort + '_sort' method = getattr(self, method_name) return method(packages) class List(ActionFields): '''Query the available packages with formatting.''' def _set_parse_arguments(self): super(List, self)._set_parse_arguments() self._parser.add_argument('-F', '--format', default=None) self._parser.add_argument('-e', '--include-empty-results', action='store_true', dest='empty') def _process_options(self): super(List, self)._process_options(self._options.format) def _is_empty(self, iterable): '''Returns True if the iterables has all empty elements.''' if not iterable: return True for value in iterable: if value: return False return True def _package_action_i(self, package, i): values, by_name = self._get_field_values(package, {'i': i}) by_name['i'] = i if self._options.empty or not self._is_empty(values): if self._options.format: print self._options.format.format(*values, **by_name) else: print '\t'.join(values) return moo.OK class Exec(ActionFields): '''Run a command on each package.''' def _set_parse_arguments(self): super(Exec, self)._set_parse_arguments() self._parser.add_argument('-x', '--script', action='append') self._parser.add_argument('--no-env', action='store_false', dest='env') self._parser.add_argument('--no-chdir', action='store_false', dest='chdir') def _process_options(self): super(Exec, self)._process_options(*self._options.script) def _package_action_i(self, package, i): status = moo.OK kwargs = {'interactive': True, 'shell': True} if self._options.chdir: kwargs['cwd'] = package.path values, by_name = self._get_field_values(package, {'i': i}) if self._options.env: kwargs['env'] = dict(os.environ, **by_name) for script in self._options.script: script_formatted = script.format(*values, **by_name) try: util.log_check_call(script_formatted, **kwargs) except subprocess.CalledProcessError as e: logging.error('%s: %s', package.name, e) status = moo.ERROR break return status class Clone(Action): '''Clone upstream repositories.''' def _set_parse_arguments(self): self._add_argument_all() self._add_argument_names(strict=False) self._parser.add_argument('--basedir', default=None) self._parser.add_argument('--upstream-branch', default=None) self._parser.add_argument('--debian-branch', default=None) def _process_options(self): self._process_argument_all() self._process_argument_names() # Split the names into URLs and packages. self._packages = set() self._urls = set() for name in self._options.names: if ':' in name: self._urls.add(name) else: self._packages.add(self._arriero.get_package(name)) def run(self): self._not_ok = set() for url in self._urls: if not self.url_clone(url): self._not_ok.add(url) for package in self._packages: if not self.package_clone(package): self._not_ok.add(package.name) # Run status if self._not_ok: return False return True def get_remote_heads(self, url): heads = set() cmd = ['git', 'ls-remote', '--heads', url] p = subprocess.Popen(cmd, stdout=subprocess.PIPE) for line in p.stdout: m = re.search('\srefs/heads/(.*)$', line) if m: heads.add(m.group(1)) return heads def guess_branches(self, url): '''Use git ls-remote to check which branches are there.''' heads = self.get_remote_heads(url) upstream_branch = 'upstream' debian_branch = 'master' # Review this: Lucky guess? if 'debian' in heads: debian_branch = 'debian' if 'upstream' not in heads: if 'master' in heads: upstream_branch = 'master' elif 'unstable' in heads: debian_branch = 'unstable' pristine_tar = False if 'pristine-tar' in heads: pristine_tar = True return debian_branch, upstream_branch, pristine_tar def url_clone(self, url): '''Clone a package from the provided url.''' # Check if this URL is already configured for package_name in self._arriero.list_packages(): package = self._arriero.get_package(package_name) if package.vcs_git == url: logging.warning( 'The URL %s is already configured by package %s.', package.vcs_git, package.name) logging.warning('Switching to cloning from configuration file.') self._packages.add(package) return True # Get basedir for this package if self._options.basedir: basedir = self._options.basedir else: basedir = self._arriero.get_config_option('DEFAULT', 'basedir') # Guess destdir for this package name = os.path.basename(url) if name.endswith('.git'): name = name[:-4] destdir = os.path.join(basedir, name) # Guess the branches debian_branch, upstream_branch, pristine_tar = self.guess_branches(url) self.clone(basedir, destdir, url, debian_branch, upstream_branch, pristine_tar) # Obtain package name from control file destdir = os.path.expanduser(destdir) control_filepath = os.path.join(destdir, 'debian', 'control') if not os.path.exists(control_filepath): logging.error('Unable to find debian/control while cloning %s', url) return False control_file = open(control_filepath) # Deb822 will parse just the first paragraph, which is ok. control = deb822.Deb822(control_file) package_name = control['Source'] if not self._arriero.add_new_package(package_name, url, destdir, debian_branch, upstream_branch, pristine_tar): logging.error('Clone successful for package not in configuration. ' 'You will not be able to use arriero with it.') return False package = self._arriero.get_package(package_name) return package.fetch_upstream() # TODO: this method should probably be in the package and not here def clone(self, basedir, destdir, url, debian_branch, upstream_branch, pristine_tar): '''Verify the directories for the clone, and clone.''' basedir = os.path.expanduser(basedir) destdir = os.path.expanduser(destdir) util.ensure_path(basedir) logging.debug('basedir: %s', basedir) logging.debug('destdir: %s', destdir) if os.path.exists(destdir): logging.error('Cloning %s, directory already exists: %s', url, destdir) return False dirname, basename = os.path.split(destdir) # The command line parameters override internal branches if self._options.debian_branch: debian_branch = self._options.debian_branch if self._options.upstream_branch: upstream_branch = self._options.upstream_branch cmd = ['gbp-clone'] if debian_branch: cmd.append('--debian-branch=%s' % debian_branch) if upstream_branch: cmd.append('--upstream-branch=%s' % upstream_branch) if pristine_tar: cmd.append('--pristine-tar') else: cmd.append('--no-pristine-tar') cmd.append(url) cmd.append(basename) util.log_check_call(cmd, cwd=dirname) logging.info('Successfully cloned %s', url) def package_clone(self, package): """Clone a package that is already in the config file.""" # TODO: shouldn't we check if this returned true or false? self.clone(package.basedir, package.path, package.vcs_git, package.debian_branch, package.upstream_branch, package.pristine_tar) return package.fetch_upstream() class Build(Action): '''Merge and compile the received packages.''' def _set_parse_arguments(self): self._add_argument_all() self._add_argument_names(strict=True) self._parser.add_argument('-D', '--distribution', '--dist', default=None) self._parser.add_argument('-A', '--architecture', '--arch', default=None) self._parser.add_argument('-U', '--local-upload', action='store_true', dest='local_upload') self._parser.add_argument('-o', '--builder-options', action='append') def _process_options(self): self._process_argument_all() self._process_argument_names() def _build_package(self, name): package = self._arriero.get_package(name) # TODO: why is build catching the exception? try: package.build( distribution=self._options.distribution, architecture=self._options.architecture, builder_options=self._options.builder_options) except (moo.GitBranchError, moo.GitDiverge, subprocess.CalledProcessError) as e: logging.error('%s: Build failed.', package.name) logging.error('%s: %s', package.name, str(e)) return False if self._options.local_upload: try: package.local_upload() except subprocess.CalledProcessError as e: logging.error('%s: Upload failed.', package.name) logging.error('%s: %s', package.name, str(e)) return False return True def run(self): packages = self._get_packages(self._options.names) self._error = set() self._ignored = set() for package in self._arriero.sort_buildable( packages, error=self._error, ignored=self._ignored): if not self._build_package(package): self._error.add(package) if self._error: return False return True def print_status(self): for package in self._ignored: logging.info('%s: ignored', package) for package in self._error: logging.error('%s: failed', package) class Upload(ActionPackages): def _set_parse_arguments(self): self._add_argument_all() self._add_argument_names(strict=True) self._parser.add_argument('--host', default='local') self._parser.add_argument('-f', '--force', action='store_true') def _package_action(self, package): return package.upload(host=self._options.host, force=self._options.force) class Pull(ActionPackages): def _package_action(self, package): status = moo.OK if package.debian_branch not in package.repo.branches: if not package.create_branch(package.debian_branch): status = moo.ERROR return status if not package.switch_branches(package.debian_branch): status = moo.ERROR else: try: package.pull() logging.info('%s: Successfully pulled.', package.name) except (moo.GitDiverge, git.exc.GitCommandError) as e: logging.error('%s: Failed to pull.', package.name) logging.error('%s: %s', package.name, e) status = moo.ERROR return status class Push(ActionPackages): def _package_action(self, package): status = moo.OK if not package.switch_branches(package.debian_branch): status = moo.ERROR else: if not package.push(): status = moo.ERROR return status class Release(ActionPackages): def _set_parse_arguments(self): self._add_argument_all() self._add_argument_names(strict=True) self._parser.add_argument('-D', '--distribution', default=None) self._parser.add_argument('-P', '--pre-release', action='store_true', dest='pre_release') def _package_action(self, package): if not package.switch_branches(package.debian_branch): logging.error('%s: can\'t switch to branch %s.', package.name, package.debian_branch) return moo.ERROR package.release(distribution=self._options.distribution, pre_release=self._options.pre_release) return moo.OK class Update(ActionPackages): def _package_action(self, package): status = moo.OK try: package.new_upstream_release() except (moo.GitDirty, moo.GitDiverge, uscan.UscanError, subprocess.CalledProcessError) as e: logging.error('%s: Failed to get new upstream release.', package.name) logging.error('%s: %s', package.name, e) status = moo.ERROR return status class Status(ActionPackages): def _package_action(self, package): print '\n'.join(package.get_status()) class PrepareOverlay(ActionPackages): def _set_parse_arguments(self): self._add_argument_all() self._add_argument_names(strict=True) self._parser.add_argument('-b', '--branch', default=None) def _package_action(self, package): status = moo.OK try: package.prepare_overlay(overlay_branch=self._options.branch) except moo.GitBranchError: logging.error('%s: failure while preparing overlay', package.name) status = moo.ERROR return status class FetchUpstream(ActionPackages): def _package_action(self, package): status = moo.OK try: package.fetch_upstream() except moo.GitDirty: logging.error('%s: unable to fetch upstream release', package.name) status = moo.ERROR return status class CheckIfChanged(ActionPackages): def _package_action(self, package): status = moo.OK if package.is_native(): return status sections = package.changelog.sections if len(sections) < 2: return status current = package.upstream_version previous = moo.Version(sections[1].version).upstream changes = False try: package.git.diff( package.tag_template('upstream', current), package.tag_template('upstream', previous), quiet=True) except git.exc.GitCommandError: changes = True print "%s: %s" % (package.name, "changed since %s" % previous if changes else "%s = %s" % (current, previous)) return status # Dictionary containing the available actions in this module and some help # about what they are for. AVAILABLE_ACTIONS = { 'status': (Status, 'Show the status of the package/s'), 'update': (Update, 'Get the new upstream release of the package/s'), 'build': (Build, 'Build the package/s in a pbuilder'), 'list': (List, 'List package/s with a specific format'), 'exec': (Exec, 'Run commands for each package/s'), 'clone': (Clone, 'Obtain the repository for the package/s'), 'upload': (Upload, 'Upload the package/s'), 'pull': (Pull, 'Pull new changes to the package/s repositories'), 'push': (Push, 'Push local changes to the package/s repositories'), 'release': (Release, 'Change the distribution in the changelog'), 'overlay': (PrepareOverlay, 'Combine upstream and debian branches'), 'fetch-upstream': (FetchUpstream, 'Fetch upstream tarball'), 'check-if-changed': (CheckIfChanged, 'Check changes in the upstream tarball'), } # vi:expandtab:softtabstop=4:shiftwidth=4:smarttab arriero-0.6/src/arriero.py000077500000000000000000000362011240255410300156270ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf8 -*- # Copyright: 2013-2014, Maximiliano Curia # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the Free # Software Foundation; either version 2 of the License, or (at your option) # any later version. # # This program is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for # more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., 51 # Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. ''' Arriero is a tool for simplifying maintaining many debian packages. It allows to quickly update, build, push and upload many packages at the same time. ''' # System imports import ConfigParser import argparse import collections import logging import os import shutil import subprocess import sys # Own imports import actions import moo import util class Arriero(object): '''Main class to interface with the user. This class handles the CLI, the config values. The individual actions are delegated to the actions module. ''' defaults = { 'config_files': ['/etc/arriero.conf', '~/.config/arriero.conf'], 'upstream-branch': 'upstream', 'debian-branch': 'master', 'pristine-tar': 'False', 'pristine-tar-branch': 'pristine-tar', 'is-merged': 'False', 'filter-orig': '', 'basedir': '~', 'path': '%(basedir)s/%(package)s', 'depends': '', 'upload-command': 'dput local %(changes_file)s', 'vcs-git': '', } def __init__(self): self.commands = {} # TODO: dynamically read other files and their actions self.commands.update(actions.AVAILABLE_ACTIONS) self.config = None self._cache = {} self.packages = {} self._architecture = None self._build_images = collections.defaultdict(bool) def list_commands(self): return self.commands.keys() def show_commands_help(self): result = ['\nAvailable commands:'] commands = sorted(actions.AVAILABLE_ACTIONS.items()) for command, (classname, helptext) in commands: result.append(' %-22s%s.' % (command, helptext)) return '\n'.join(result) @property def architecture(self): if not self._architecture: self._architecture = subprocess.check_output( ['dpkg-architecture', '-qDEB_BUILD_ARCH']).rstrip('\n') return self._architecture def update_options(self, options): '''Store internal options according to what the user wants.''' if options.config: self.update_config(options.config) self.verbose = options.verbose def update_config(self, config_files=None): '''Read the configuration file, update package sets.''' if config_files: self.config_files = util.split(config_files) else: self.config_files = self.defaults['config_files'] self.config_files = map(os.path.expanduser, self.config_files) self.config = ConfigParser.SafeConfigParser() read_files = self.config.read(self.config_files) # Verify that the config files were there for orig_file in self.config_files: if orig_file not in read_files: logging.warning('Could not parse config file: %s', orig_file) self._cache['_modules'] = set() self._cache['_packages'] = set() self._cache['_parents'] = collections.defaultdict(set) for section in self.config.sections(): if self.config.has_option(section, 'packages'): self._cache['_modules'].add(section) section_packages = util.split( self.config.get(section, 'packages')) for package in section_packages: self._cache['_parents'][package].add(section) if not self.config.has_option(package, 'packages'): self._cache['_packages'].add(package) else: self._cache['_packages'].add(section) def get_config_option(self, section, option, raw=False, inherit=True): '''Obtain the requested config option for section.''' if not self.config: self.update_config() if self.config.has_section(section) and \ self.config.has_option(section, option): return self.config.get(section, option, raw=raw) elif section == 'DEFAULT' and self.config.has_option(section, option): return self.config.get(section, option, raw=raw) if inherit: # Inherit configuration options from parent modules done = set([section]) queue = collections.deque(self._cache['_parents'][section]) while queue: parent = queue.popleft() if parent in done: continue done.add(parent) if self.config.has_option(parent, option): return self.config.get(parent, option, raw=raw) queue.extend(self._cache['_parents'][parent]) return self.defaults.get(option, None) def write_config(self): '''Writes any changes to the config file to disk.''' config_file = self.config_files[-1] if os.path.exists(config_file): shutil.copyfile(config_file, config_file + '.bak') config_open = open(config_file, 'w') self.config.write(config_open) config_open.close() def add_new_package(self, package_name, git_url, path, debian_branch, upstream_branch, pristine_tar): '''Adds a new package to the configuration.''' if package_name in self.list_all(): logging.error( 'Package %s definition already in the config file. ' 'Not adding it.', package_name) return False basedir = self.get_config_option('DEFAULT', 'basedir') basedir = os.path.expanduser(basedir) if path.startswith(basedir): path = '%(basedir)s' + path[len(basedir):] self.config.add_section(package_name) # TODO: Need to make this general, deb-src package have no git repo if git_url: self.config.set(package_name, 'vcs-git', git_url) self.config.set(package_name, 'path', path) self.config.set(package_name, 'debian-branch', debian_branch) self.config.set(package_name, 'pristine-tar', str(pristine_tar)) if (upstream_branch): self.config.set(package_name, 'upstream-branch', upstream_branch) self.write_config() self.update_config() return True def list_modules(self): if not self.config: self.update_config() return self._cache['_modules'] def list_packages(self): if not self.config: self.update_config() return self._cache['_packages'] def list_all(self): return self.list_modules() | self.list_packages() def call(self, cmd, argv): '''Execute the command the user requested.''' exit_code = 0 action = self.commands[cmd][0] try: instance = action(self, argv) success = instance.run() if success is True: exit_code = 0 elif success is False: exit_code = 1 else: exit_code = success instance.print_status() return exit_code except actions.ActionError as e: logging.critical(e.message) return 255 def _read_packages(self, name, expanded_modules=None): '''Returns the set of packages in a particular module.''' if name in self._cache: return self._cache[name].packages config_packages = self.get_config_option(name, 'packages', inherit=False) if not config_packages: return set([name]) # Expand package groups in the packages list if expanded_modules is None: expanded_modules = set() expanded_modules.add(name) packages = set() for element in util.split(config_packages): if element not in self.list_modules(): packages.add(element) elif element not in expanded_modules: packages.update(self._read_packages(element, expanded_modules)) return packages def _read_depends(self, name): '''Returns the dependencies for a particular module.''' if name in self._cache: return self._cache[name].depends raw_depends = self.get_config_option(name, 'depends') not_expanded = set() if raw_depends: not_expanded.update(util.split(raw_depends)) if name in not_expanded: not_expanded.remove(name) depends = set() for dependency in not_expanded: # Do not expand parents if name in self._cache['_parents'] and \ dependency in self._cache['_parents'][name]: # Add it as a package if dependency in self._cache['_packages']: depends.add(dependency) continue depends.update(self._read_packages(dependency)) if name in depends: depends.remove(name) return depends def get_module(self, name): '''Returns a Module object.''' if name in self._cache: return self._cache[name] packages = self._read_packages(name) depends = self._read_depends(name) self._cache[name] = moo.Module(packages, depends) return self._cache[name] def get_package(self, name): '''Returns a Package object.''' if name not in self.packages: self.packages[name] = moo.Package(name, self, self.get_module(name)) return self.packages[name] def build_depends_graph(self, pending, ignored): # TODO: this should be in moo.py # TODO: this should also be a class of its own # build the graph graph = collections.defaultdict(lambda: moo.Node(set(), set())) visited = set() to_visit = set(pending) ready = collections.deque() while to_visit: name = to_visit.pop() if name in visited: continue visited.add(name) package = self.get_package(name) depends = package.depends ignored |= depends - pending depends &= pending if not depends: ready.append(name) continue graph[name].input.update(depends) for input in depends: graph[input].output.add(name) to_visit.update(depends) return graph, ready def update_depends_graph(self, graph, package_name, ready): ''' Remove the built package from the needed dependencies of depending packages. ''' for child in graph[package_name].output: graph[child].input.remove(package_name) # if there are no more dependencies, the package can be built if not graph[child].input: ready.append(child) def sort_buildable(self, packages, done=None, ignored=None, error=None): if done is None: done = set() if ignored is None: ignored = set() if error is None: error = set() pending = set(packages) graph, ready = self.build_depends_graph(pending, ignored) # ready is a set that contains the packages ready to be built while ready: name = ready.popleft() yield name if name in error: continue done.add(name) self.update_depends_graph(graph, name, ready) if len(done) != len(packages) and not error: not_ready = set(name for name in graph if graph[name].input) error.update(not_ready) raise moo.GraphError('Not a DAG?') elif error: not_ready = set(name for name in graph if graph[name].input) error.update(not_ready) def ensure_builder_image(self, distribution, architecture): '''Once per run, checks that the build image is up to date.''' distribution = distribution.lower() if self._build_images[(distribution, architecture)]: return env = {'ARCH': architecture, 'DIST': distribution} # First tries to update, if it fails, tries to create. try: cmd = ['git-pbuilder', 'update'] logging.info('Updating build image for %s-%s', architecture, distribution) subprocess.check_call(cmd, env=dict(os.environ, **env)) except subprocess.CalledProcessError: # Distribution unreleased is usually intended to be unstable. # If the user wants something different, they can create the file or # symlink it manually to something else if distribution == 'unreleased': env['GIT_PBUILDER_OPTIONS'] = '--distribution=unstable' logging.warning('Build image for %s-%s not found. Creating...', architecture, distribution) cmd = ['git-pbuilder', 'create'] subprocess.check_call(cmd, env=dict(os.environ, **env)) self._build_images[(distribution, architecture)] = True def main(): arriero = Arriero() cmds = arriero.list_commands() parser = argparse.ArgumentParser( description=__doc__, epilog=arriero.show_commands_help(), formatter_class=argparse.RawTextHelpFormatter) parser.add_argument('-c', '--config', help='Specify a config file.', metavar='FILE') parser.add_argument('-v', '--verbose', action='store_true', help='Show more information.') parser.add_argument('-q', '--quiet', action='store_true', help='Show only critical errors.') parser.add_argument('command', choices=cmds, metavar='COMMAND', help='Command to execute. See options below.') options, argv = parser.parse_known_args() # Initialize logging logging.basicConfig(format='[%(levelname)s] %(message)s', level=logging.WARNING) logger = logging.getLogger() if options.verbose: logger.setLevel(logging.DEBUG) elif options.quiet: logger.setLevel(logging.CRITICAL) # Update the config and other options arriero.update_options(options) try: exit_code = arriero.call(options.command, argv) except argparse.ArgumentError as error: logging.error('Error while parsing arguments: %s', error) exit_code = 100 return exit_code if __name__ == '__main__': exit_code = main() sys.exit(exit_code) # vi:expandtab:softtabstop=4:shiftwidth=4:smarttab arriero-0.6/src/cementery.py000066400000000000000000000045021240255410300161530ustar00rootroot00000000000000 import sys import git ## # Misc functions ## def action_report(msg, action): ''' Executes action and prints the success state. ''' print '%s: ' % (msg,), if action(): print 'done.' return True else: print 'fail.' return False def check_report(msg, action): if not action(): print >>sys.stderr, '%s' % (msg,) return False return True def git_checkout(repo, branch): ''' Switch repo to branch. ''' try: repo.git.checkout(branch) except git.exc.GitCommandError: return False return True def list_module(self, name, order='build'): module = self.get_module(name) if order not in set(('raw', 'alpha', 'build')): raise ValueError('Invalid value for order') s = lambda ps: ps if order == 'alpha': s = sorted elif order == 'build': s = lambda packages: self.sort_buildable(packages) for package in s(module.packages): print package class Arriero_attic(object): # Old commands, no longer used. def set_debian_push(self, name): module = self.get_module(name) # TODO: fix, ugly ugly for package_name in module.packages: package = self.get_package(package_name) try: remote = package.git.config('branch.%s.remote' % ( package.debian_branch,)) except git.exc.GitCommandError: remote = 'origin' try: ref = package.git.config('branch.%s.merge' % ( package.debian_branch,)) except git.exc.GitCommandError: ref = 'refs/heads/%s' % (package.debian_branch,) try: package.git.config('--get', 'remote.%s.push' % (remote,), ref) except git.exc.GitCommandError: package.git.config('--add', 'remote.%s.push' % (remote,), ref) def checkout_debian(self, name): module = self.get_module(name) for package_name in module.packages: package = self.get_package(package_name) if not package.switch_branches(package.debian_branch): logging.error('Failure while switching branches for: %s', package_name) arriero-0.6/src/moo.py000066400000000000000000001204511240255410300147540ustar00rootroot00000000000000# -*- coding: utf8 -*- # Copyright: 2013-2014, Maximiliano Curia # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the Free # Software Foundation; either version 2 of the License, or (at your option) # any later version. # # This program is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for # more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., 51 # Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import apt_pkg import collections import deb822 import git import glob import logging import operator import os import re # Git buildpackage imports import gbp.deb.changelog as changelog import gbp.deb.control as control try: from gbp.deb.source import DebianSource except ImportError: DebianSource = None # Own imports import uscan import util ## # Constants ## OK = 'Ok' IGNORE = 'Ignored' ERROR = 'Error' MISS_DEP = 'Missing dependencies' ## # Data structures ## Module = collections.namedtuple('Module', ['packages', 'depends']) class Version(object): '''Represents the different parts of the version string. Attributes: version : Full version string epoch : Epoch if any, None otherwise upstream: Upstream version debian : Debian revision if any, None otherwise ''' known_vendors = set(['ubuntu']) vendor_re = re.compile(r'\d(?P[[:alpha:]]*)?[1-9][0-9.~]*$') def __init__(self, version_string): ''' Split version in epoch, upstream_version, debian_revision, returns a Version namedtuple. ''' self.version = version_string self.epoch = None self.debian = None upstream = version_string if ':' in upstream: self.epoch, upstream = upstream.split(':', 1) if '-' in upstream: upstream, self.debian = upstream.rsplit('-', 1) self.upstream = upstream @staticmethod def from_parts(epoch=None, upstream='0', debian=None): version_string = '' if epoch: version_string += '%s:' % epoch version_string += upstream if debian: version_string += '-%s' % debian return Version(version_string) def is_native(self): return not self.debian @property def vendor(self): if self.is_native(): version = self.upstream else: version = self.debian match = self.vendor_re.search(self.debian) if match and match.group('vendor') in self.known_vendors: return match.group('vendor') return '' def new_upstream_version(self, upstream): '''Returns a Version for the new upstream release.''' # Can be '' epoch = '%s:' % self.epoch if self.epoch else '' # Let's trust upstream # if dfsg_re.match(self.upstream) and dfsg_re.match(upstream): # upstream += '+dfsg' debian = '-1' if self.debian else '' new_version = epoch + upstream + debian return Version(new_version) def _bump(self, revision): '''Bump a revision so dpkg --compare-version considers it greater.''' match = re.match('(.*?)([0-9]*)$', revision) numeric_part = 0 if match.group(2): numeric_part = int(match.group(2)) numeric_part += 1 return match.group(1) + str(numeric_part) def is_pre_release(self): '''Check if its a pre release.''' return (self.debian and '~' in self.debian) or \ (not self.debian and '~' in self.upstream) def pre_release(self): '''Returns a pre release Version from the current one.''' def bump_pre(revision): if '~' not in revision: return revision + '~' base, pre = revision.split('~', 1) return base + '~' + self._bump(pre) if self.is_native(): new_upstream = bump_pre(self.upstream) new_debian = self.debian else: new_upstream = self.upstream new_debian = bump_pre(self.debian) return Version.from_parts(self.epoch, new_upstream, new_debian) def release(self): '''Returns a release Version from a pre release one.''' def remove_pre(revision): pos = revision.find('~') if pos == -1: return revision return revision[:pos] if self.is_native(): new_upstream = remove_pre(self.upstream) new_debian = self.debian else: new_upstream = self.upstream new_debian = remove_pre(self.debian) return Version.from_parts(self.epoch, new_upstream, new_debian) def __str__(self): return self.version def __repr__(self): return self.version def __cmp__(self, other): return apt_pkg.version_compare(self.version, other.version) # Graph node Node = collections.namedtuple('Node', ['input', 'output']) class GraphError(Exception): pass # Precompiled regexps dfsg_re = re.compile(r'(.*)[+.]dfsg(?:.\d+)?$') ## # Exceptions ## class GitBranchError(Exception): pass class GitDirty(Exception): pass class GitDiverge(Exception): pass class GitRemoteNotFound(Exception): pass def version_to_tag(version): return version.replace('~', '_').replace(':', '%') def tag_to_version(tag): return tag.replace('_', '~').replace('%', ':') class Package(object): def __init__(self, name, arriero, base): self._name = name self._arriero = arriero self._base = base self._path = None self._vcs_git = None self._repo = None self._changelog = None self._control = None self._distribution = None self._architecture = self._arriero.architecture apt_pkg.init() @property def name(self): return self._name @property def basedir(self): return self._arriero.get_config_option(self.name, 'basedir') @property def depends(self): return self._base.depends # TODO: read gbp config @property def upstream_branch(self): return self._arriero.get_config_option(self.name, 'upstream-branch') # TODO: read gbp config @property def debian_branch(self): return self._arriero.get_config_option(self.name, 'debian-branch') @property def filter_orig(self): patterns = self._arriero.get_config_option(self.name, 'filter-orig') return util.split(patterns) @property def pristine_tar(self): value = self._arriero.get_config_option(self.name, 'pristine-tar') return value.lower() in ('true', 'on', 'yes') # TODO: read gbp config @property def pristine_tar_branch(self): return self._arriero.get_config_option(self.name, 'pristine-tar-branch') @property def is_merged(self): value = self._arriero.get_config_option(self.name, 'is-merged') return value.lower() in ('true', 'on', 'yes') @property def path(self): if not self._path: value = self._arriero.get_config_option( self.name, 'path', raw=True) basedir = self.basedir package = self.name path = value % locals() self._path = os.path.expanduser(path) return self._path @property def vcs_git(self): if not self._vcs_git: value = self._arriero.get_config_option( self.name, 'vcs-git', raw=True) package = self.name self._vcs_git = value % locals() return self._vcs_git def check_path(self): if os.path.isdir(self.path): return self.path # TODO: read gbp config @property def tarball_dir(self): return os.path.normpath(os.path.join(self.path, '..', 'tarballs')) # TODO: read gbp config @property def export_dir(self): return os.path.normpath(os.path.join(self.path, '..', 'build-area')) @property def repo(self): if not self._repo: self._repo = git.Repo(self.path) return self._repo @property def git(self): return self.repo.git @property def branch(self): return self.repo.active_branch.name @property def changelog(self, force=False): if not self._changelog: self.update_changelog() return self._changelog def update_changelog(self): self._changelog = changelog.ChangeLog( filename=os.path.join(self.path, 'debian', 'changelog')) @property def control(self, force=False): if not self._control: self.update_control() return self._control @property def source_name(self): return self.control.name def update_control(self): self._control = control.Control( filename=os.path.join(self.path, 'debian', 'control')) @property def build_file(self, check=True): build_file = os.path.join( self.export_dir, '%s_%s_%s.build' % (self.name, self.noepoch, self._architecture)) if check and not os.path.exists(build_file): return '' return build_file @property def changes_file(self, check=True): changes_file = os.path.join( self.export_dir, '%s_%s_%s.changes' % (self.source_name, self.noepoch, self._architecture)) if check and not os.path.exists(changes_file): return '' return changes_file @property def version(self): return self.changelog.version @property def noepoch(self): return self.changelog.noepoch @property def epoch(self): return self.changelog.epoch @property def upstream_version(self): return self.changelog.upstream_version @property def debian_version(self): return self.changelog.debian_version @property def distribution(self): if self._distribution: return self._distribution return self.changelog['distribution'] @property def urgency(self): return self.changelog['urgency'] def tag_template(self, name, version): '''Returns a tag version correctly formatted according to the name.''' value = {'version': version_to_tag(version)} # TODO: read gbp config to get the templates if name == 'upstream': return 'upstream/%(version)s' % value if name == 'debian': return 'debian/%(version)s' % value def uscan(self): return uscan.Uscan(self.path, destdir=self.tarball_dir) def is_native(self): if DebianSource: return DebianSource(self.path).is_native() return self.changelog.is_native() def is_dfsg(self): if self.is_native(): return False if dfsg_re.match(self.upstream_version): return True return False def create_branch(self, branch, tracking=None): if not tracking: # Search a remote to track trackable = [] for remote in self.repo.remotes: remote.fetch() if self.vcs_git and self.vcs_git == remote.url: logging.debug('%s: remote %s configured as vcs-git', self.name, remote.name) tracking = remote break if branch in remote.refs: logging.debug('%s: found %s/%s', self.name, remote.name, branch) trackable.append(remote) if not tracking and trackable: tracking = trackable[0] if tracking and branch in tracking.refs: # Create a new branch with tracking head = self.repo.create_head(branch, commit=tracking.refs[branch]) head.set_tracking_branch(tracking.refs[branch]) return head else: return False def switch_branches(self, branch): if self.repo.is_dirty(untracked_files=True): logging.warn('%s: branch %s has uncommitted changes.', self.name, self.repo.active_branch.name) return False try: self.git.checkout(branch) except git.exc.GitCommandError: # Does the branch even exists? return False # TODO: Check if really needed self._changelog = None return True def branch_to_ref(self, branch): return 'refs/heads/%s' % (branch,) def ahead_behind_ref(self, ref1, ref2): out = self.git.rev_list('--left-right', '--count', '%s...%s' % (ref1, ref2)) return map(int, out.split()) def ahead_behind(self, local=None, remote=None): # Hacky check between two refs # returns two numbers, commits ahead and commits behind if not local: local = self.branch ref1 = self.branch_to_ref(local) if not remote: try: remote_name = self.git.config('branch.%s.remote' % local) except git.exc.GitCommandError: remote_name = 'origin' try: remote_branch = self.git.config('branch.%s.merge' % local) except git.exc.GitCommandError: remote_branch = local prefix = 'refs/heads/' if remote_branch.startswith(prefix): remote_branch = remote_branch[len(prefix):] remote = '%s/%s' % (remote_name, remote_branch) # Check if the remote actually exists git_remote = self.repo.remote(remote_name) if git_remote not in self.repo.remotes: raise(GitRemoteNotFound('Remote (%s) not found' % remote_name)) for ref in git_remote.refs: if ref.name == remote: break else: raise( GitRemoteNotFound( 'Could not find branch (%s) in remote (%s)' % (remote_branch, remote_name))) ref2 = 'refs/remotes/%s' % (remote,) return self.ahead_behind_ref(ref1, ref2) # TODO: DEADCODE def merge_if_not_diverged(self, branch): ahead, behind = self.ahead_behind( ref1=self.branch_to_ref(self.branch), ref2=self.branch_to_ref(branch)) if ahead and behind: raise( GitDiverge( 'Current branch (%s) has diverged with (%s)' % (self.branch, branch))) elif not ahead and behind: self.git.merge('-m', 'Merge with %s', (branch,)) return True return False def prepare_overlay(self, overlay_branch=None): '''Combines the upstream and debian branches into one working tree. If the debian_branch of the package already contains the upstream source (stored in self.merge attribute), then that branch is checked out, nothing else happens. If the package is a native package, the debian branch is checked out, nothing else happens. Args: overlay_branch: if received, the result is checked into a new branch instead of a dettached HEAD. ''' # Try to switch to debian branch. If this fails, there are probably # uncommitted changes. logging.debug('%s: switching to branch %s for overlay creation.', self.name, self.debian_branch) if not self.switch_branches(self.debian_branch): raise GitBranchError('Could not change to branch %s' % self.debian_branch) # Create working branch, if the user so requested if overlay_branch: logging.debug('%s: creating new overlay branch %s.', self.name, overlay_branch) try: self.git.checkout('-b', overlay_branch) except git.exc.GitCommandError as e: logging.error('%s: %s', self.name, str(e)) raise GitBranchError(str(e)) if self.is_merged: logging.info('%s: skipping overlay creation, already merged.', self.name) return if self.is_native(): logging.info('%s: native package, no overlay needed.', self.name) return # Checkout the upstream tree into the current branch, then reset the # index so that the files are there, but not scheduled to be committed. logging.info('%s: copying tree from latest upstream tag.', self.name) self.git.checkout(self.tag_template('upstream', self.upstream_version), '.', ours=True) # TODO: add :!.gitignore, only if the git version supports it. self.git.reset() logging.info('%s: overlay created, branch needs to be manually cleaned', self.name) def build(self, ignore_branch=False, distribution=None, architecture=None, builder_options=None): if not ignore_branch and self.debian_branch != self.branch: if not self.switch_branches(self.debian_branch): raise(GitBranchError('Could not change to branch %s' % ( self.debian_branch,))) if distribution: self._distribution = distribution if architecture: self._architecture = architecture self._arriero.ensure_builder_image(self.distribution, self._architecture) # TODO: add this as options. cmd = [ 'git-buildpackage', '--git-pbuilder', '--git-export-dir=%s' % self.export_dir, '--git-arch=%s' % self._architecture, '--git-dist=%s' % self.distribution.lower() ] if self._arriero.verbose: cmd.append('--git-verbose') if self.pristine_tar: cmd.append('--git-pristine-tar') # If already generated, use that cmd.append('--git-tarball-dir=%s' % self.export_dir) else: cmd.append('--git-tarball-dir=%s' % self.tarball_dir) if not ignore_branch: cmd.append('--git-debian-branch=%s' % self.debian_branch) if not self.is_merged and not self.is_native(): cmd.append('--git-overlay') if builder_options: cmd.extend(builder_options) util.log_check_call(cmd, cwd=self.path, interactive=True) lintian_cmd = [ 'lintian', '-I', '--pedantic', '--show-overrides', self.changes_file] util.log_check_call(lintian_cmd, cwd=self.path) # Switch to work branch if not ignore_branch: self.switch_branches(self.debian_branch) def release(self, distribution=None, pre_release=False, ignore_branch=False): if not ignore_branch and self.debian_branch != self.branch: if not self.switch_branches(self.debian_branch): raise(GitBranchError('Could not change to branch %s' % ( self.debian_branch,))) cmd = ['dch'] if self.distribution.lower() == 'unreleased': version = Version(self.version) if pre_release: if not version.is_pre_release(): cmd.append('-b') new_version = version.pre_release() cmd.extend(['-v', str(new_version)]) elif version.is_pre_release(): new_version = version.release() # First we need to remove ~ part util.log_check_call( ['dch', '--release-heuristic', 'changelog', '-v', str(new_version), ''], cwd=self.path) cmd.append('-r') else: cmd.append('-r') else: # TODO(marga): this shouldn't do anything if there are no changes cmd.append('-r') if distribution: cmd.append('-D') cmd.append(distribution) cmd.append('') util.log_check_call(cmd, cwd=self.path) self.update_changelog() # If there were any changes to the changelog, commit them if self.repo.index.diff(None, 'debian/changelog'): if pre_release: msg = 'Pre-release %s' % self.version else: msg = 'Release to %s' % self.distribution self.git.commit('debian/changelog', '-m', msg) def local_upload(self): self.upload('local') def upload(self, host, force=False): upload_command = self._arriero.get_config_option( self.name, 'upload-command', raw=True) if not upload_command: return ERROR if not self.changes_file: return IGNORE cmd_variables = { 'changes_file': self.changes_file, 'package': self.name, 'version': self.version, 'distribution': self.distribution, 'host': host, } try: full_command = upload_command % cmd_variables except (ValueError, KeyError) as e: logging.error('%s: unable to format upload-command: %s', self.name, upload_command) logging.error('%s: %s' % (e.__class__.__name__, e.message)) return ERROR util.log_check_call(full_command, shell=True) def push(self): def push_branch(branch): try: remote_name = self.git.config('branch.%s.remote' % branch) except git.exc.GitCommandError as e: logging.debug('%s: No remote associated.\n%s', self.name, e) return True head = self.branch_to_ref(branch) try: self.git.push(remote_name, head) except git.exc.GitCommandError as e: logging.error('%s: Failed to push.\n%s', self.name, e) return False return True def push_tags(branch, template): try: remote_name = self.git.config('branch.%s.remote' % branch) except git.exc.GitCommandError as e: logging.debug('%s: No remote associated.\n%s', self.name, e) return True tags = 'refs/tags/%s' % self.tag_template(template, '*') try: self.git.push(remote_name, tags) except git.exc.GitCommandError as e: logging.error('%s: Failed to push.\n%s', self.name, e) return False return True logging.debug('Pushing %s', self.name) if not push_branch(self.debian_branch): return False if not push_branch(self.upstream_branch): return False if self.pristine_tar: if not push_branch(self.pristine_tar_branch): return False return push_tags(self.debian_branch, 'debian') and \ push_tags(self.upstream_branch, 'upstream') def safe_pull(self): # update to check ahead/behind # TODO: fix ugly hack try: remote_name = self.git.config('branch.%s.remote' % (self.branch,)) except git.exc.GitCommandError: remote_name = 'origin' self.git.fetch(remote_name) try: ahead, behind = self.ahead_behind() if ahead > 0 and behind > 0: raise(GitDiverge('Needs to merge with head.')) except GitRemoteNotFound as e: logging.info('Branch not associated with a remote: %s' % e.message) return False # TODO: return True if there were changes self.git.pull() def pull(self): def pull_branch(branch): try: remote_name = self.git.config('branch.%s.remote' % branch) except git.exc.GitCommandError as e: logging.debug('%s: No remote associated.\n%s', self.name, e) return True head = self.branch_to_ref(branch) try: remote_branch = self.git.config('branch.%s.merge' % branch) except git.exc.GitCommandError: remote_branch = head try: logging.debug('%s %s %s:%s', 'pull' if branch == self.branch else 'fetch', remote_name, remote_branch, head) if branch == self.branch: self.git.pull(tags=True) else: self.git.fetch(remote_name, '%s:%s' % (remote_branch, head)) except git.exc.GitCommandError as e: logging.error('%s: Failed to pull.\n%s', self.name, e) return False return True logging.debug('Pulling %s', self.name) if not pull_branch(self.debian_branch): return False if not pull_branch(self.upstream_branch): return False if self.pristine_tar: if not pull_branch(self.pristine_tar_branch): return False return True def get_new_version(self, upstream): ''' Obtains a new version. ''' old = Version(self.version) new = old.new_upstream_version(upstream) return new.version def new_dfsg_version(self, upstream_version): branch = self.branch rules = os.path.join(self.path, 'debian', 'rules') dfsg_version = upstream_version + '+dfsg' dfsg_tag = self.tag_template('upstream', dfsg_version) if dfsg_tag in self.repo.tags: # Already there return dfsg_version self.git.checkout(self.tag_template('upstream', upstream_version)) self.git.checkout('heads/%s' % branch, '--', 'debian') self.git.reset() util.log_check_call( ['fakeroot', rules, 'prune-nonfree'], cwd=self.path) util.log_check_call(['rm', '-rf', 'debian'], cwd=self.path) if (not self.repo.is_dirty(untracked_files=True)): # No changes made by the prune-nonfree call, we are free \o/ self.switch_branches(branch) return upstream_version self.git.commit('-a', '-m', 'DFSG version %s' % (dfsg_version,)) self.repo.create_tag(dfsg_tag) self.switch_branches(branch) return dfsg_version def fetch_upstream(self): '''Fetch upstream tarball when there is no upstream_branch.''' if not self.upstream_branch: return True if self.is_native(): return True if (self.upstream_branch not in self.repo.branches): logging.debug('Creating upstream branch for %s.', self.name) original_branch = self.branch # Create upstream branch self.git.checkout('--orphan', self.upstream_branch) self.git.rm('-rf', '.') self.git.commit('--allow-empty', '-m', 'Upstream branch') self.switch_branches(original_branch) result = self.get_upstream_release(current=True) return result def get_upstream_release_uscan(self, current=False, version=None): '''Download the upstream tarball with uscan. Args: current: if True downloads the tarball even if it's the same as the one in the changelog file. ''' try: pkg_scan = self.uscan() pkg_scan.scan(download=True, force_download=current, version=version) except uscan.UscanError as e: logging.error( '%s: Could not download upstream tarball: %s', self.name, str(e)) return False # ignore uptodate status if we forced the download if not current and pkg_scan.uptodate: return False return (pkg_scan.uversion, pkg_scan.tarball) def _get_local(self, file_glob, version_re, requested_version): files = [] for filename in glob.iglob( os.path.join(self.tarball_dir, file_glob)): basename = os.path.basename(filename) m = re.search(version_re, basename) if not m: continue version = m.group(1) if requested_version: if apt_pkg.version_compare(version, requested_version) == 0: logging.debug('%s: %s found', self.name, filename) return (version, filename) elif apt_pkg.version_compare(version, self.upstream_version) > 0: logging.debug('%s: %s found', self.name, filename) files.append((version, filename)) if files: files.sort(cmp=apt_pkg.version_compare, key=operator.itemgetter(0), reverse=True) return files[0] return None def get_upstream_release_local(self, version=None): '''Check if new upstream release file is already downloaded.''' requested_version = version logging.debug('%s: looking for .orig files.', self.name) file_glob_orig = self.source_name + '_[0-9]*.orig.tar.*' version_orig_re = r'_([0-9.]+)\.' found = self._get_local(file_glob_orig, version_orig_re, requested_version) if found: return found logging.debug('%s: looking for non .orig files.', self.name) # not found, let's see if the file is downloaded without the .orig name copyright_file = open(os.path.join(self.path, 'debian', 'copyright')) copyright_deb822 = deb822.Deb822(copyright_file) if copyright_deb822 and 'Upstream-Name' in copyright_deb822: upstream_name = copyright_deb822['Upstream-Name'] else: upstream_name = self.source_name file_glob_other = upstream_name + '[-_][0-9]*.tar.*' version_re = r'[-_]([0-9.]+)\.tar' found = self._get_local(file_glob_other, version_re, requested_version) if found: return found if upstream_name != self.source_name: logging.debug('%s: looking for non .orig files using source_name.', self.name) file_glob_other = self.source_name + '[-_][0-9]*.tar.*' version_re = r'[-_]([0-9.]+)\.tar' found = self._get_local(file_glob_other, version_re, requested_version) if found: return found return False def get_upstream_release(self, ignore_branch=False, local_only=False, current=False, version=None): if self.is_native(): return False logging.debug('%s: Fetching upstream tarball.', self.name) if self.repo.is_dirty(untracked_files=True): raise(GitDirty('Uncommited changes')) if not ignore_branch and self.debian_branch != self.branch: if not self.switch_branches(self.debian_branch): raise(GitDirty('Could not switch branches')) if not local_only: self.safe_pull() # Need to reread changelog self.update_changelog() if version and dfsg_re.match(version): version = dfsg_re.sub(r'\1', version) # Let's check if it's already downloaded download = self.get_upstream_release_local(version=version) if not download: download = self.get_upstream_release_uscan(current=current, version=version) if not download: return False upstream_version, tarball = download tag = self.tag_template('upstream', upstream_version) if tag not in self.repo.tags: cmd = ['git-import-orig', '--upstream-version=%s' % upstream_version] # TODO: this should not be duplicated if self.debian_branch: cmd.append('--debian-branch=%s' % self.debian_branch) if self.upstream_branch: cmd.append('--upstream-branch=%s' % self.upstream_branch) if self.pristine_tar: cmd.append('--pristine-tar') if self.filter_orig: cmd.append('--filter-pristine-tar') for pattern in self.filter_orig: cmd.append('--filter=%s' % pattern) else: cmd.append('--no-pristine-tar') if self.is_merged: cmd.append('--merge') else: cmd.append('--no-merge') cmd.append(tarball) util.log_check_call(cmd, cwd=self.path) # TODO: if requested version is dfsg, it should apply the fixes # corresponding to that version if self.is_dfsg(): return self.new_dfsg_version(upstream_version) return upstream_version def upstream_changes(self, old_version=None, new_version=None): logging.debug('%s: Checking changes between upstream releases', self.name) if not old_version and not new_version: sections = self.changelog.sections new_version = self.upstream_version for section in sections: version = Version(section.version) if version.upstream != new_version: old_version = version.upstream if not old_version: return True elif not old_version: old_version = self.upstream_version elif not new_version: new_version = self.upstream_version old_tag = self.tag_template('upstream', old_version) new_tag = self.tag_template('upstream', new_version) # Are this versions imported and tagged? if old_tag not in self.repo.tags: self.get_upstream_release(version=old_version) if new_tag not in self.repo.tags: self.get_upstream_release(version=new_version) return self.repo.tags[old_tag].commit.tree.diff(new_tag) def new_upstream_release(self, ignore_branch=False, local_only=False): logging.debug('%s: Searching for a new upstream release', self.name) if not self.check_path(): logging.error('%s: %s doesn\'t exist', self.name, self.path) return False upstream_version = self.get_upstream_release(ignore_branch, local_only, current=False) if not upstream_version: logging.debug('%s: No upstream version found.', self.name) return False if not self.upstream_changes(new_version=upstream_version): return False if self.upstream_version == upstream_version: return False version = self.get_new_version(upstream_version) cmd = ['dch', '-p', '-v', version, 'New upstream release.'] util.log_check_call(cmd, cwd=self.path) # Just changed the changelog, but its probably not going to be used # anymore. self._changelog = None self.git.add(os.path.join('debian', 'changelog')) self.git.commit('-m', 'New upstream release.') return True def get_status(self): status = [] status.append('Package: %s' % self.name) status.append('+ Directory: %s' % self.path) if not self.check_path(): status.append('! Status: Error, %s doesn\'t exist' % self.path) return status status.extend(self.get_branch_status()) return status def get_branch_status(self): status = [] status.append('+ Branch: %s' % self.branch) error = False branches = {'debian': self.debian_branch} if not self.is_native(): branches['upstream'] = self.upstream_branch if self.pristine_tar: branches['pristine-tar'] = self.pristine_tar_branch for k, v in branches.iteritems(): if v not in self.repo.heads: status.append('! Status: Error, Missing %s branch: %s' % (k, v)) error = True if error: return status # Check if there are non commited changes dirty = self.repo.is_dirty(untracked_files=True) if dirty: status.append('! Status: Uncommited changes') status.append(self.git.status()) # Check current version (only in debian) if self.branch not in self.debian_branch: if dirty: status.append('! Status: Can\'t check version, dirty branch') return status # If it's not dirty we can change branches, right? if not self.switch_branches(self.debian_branch): status.append( '! Status: Error, change to branch %s failed' % self.debian_branch) return status else: status.append('* Switched to branch: %s' % self.branch) # Check upstream tag with current version # TODO: # status.append( self.tag_template('upstream') % { # 'version': self.upstream_version} # Check if released # released = (self.distribution.lower() != 'unreleased') status.append('+ Distribution: %s' % self.distribution) # if released, check if tagged in debian repo # TODO: # status.append( self.tag_template('debian') % { # 'version': self.noepoch} # TODO: it should be possible to obtain the state without actually # changing branches status.extend(self.get_uscan_status()) status.extend(self.get_repo_status()) status.extend(self.get_build_status()) return status def get_uscan_status(self): '''Check if the package is up to date with uscan.''' # Native packages don't have uscan status if self.is_native(): return [] try: uscan_status = self.uscan() uscan_status.scan(download=False) except uscan.UscanError as e: return ['! Status: Error while running uscan: %s' % str(e)] status = [] if not uscan_status.uptodate: status.append( '! Status: New upstream release available. Local version: %s. ' '! Upstream version: %s.\nStatus: Source URL: %s' % (uscan_status.version, uscan_status.uversion, uscan_status.url)) if uscan_status.tarball: status.append('- Source: Already downloaded in %s' % uscan_status.tarball) return status def get_repo_status(self): # Check if up to date with git repo status = [] remote_name = self.git.config('branch.%s.remote' % self.branch) self.git.fetch(remote_name) try: ahead, behind = self.ahead_behind() if behind > 0: status.append( '! Status: Remote changes commited (%d), pull them' % behind) if ahead > 0: status.append( '! Status: Local changes commited (%d)' % ahead) except GitRemoteNotFound: status.append('! Status: Branch not associated with a remote') return status def get_build_status(self): status = [] if not self.changes_file: status.append('- Status: The package has not been built') signed = False if self.changes_file: ret = util.quiet('gpg', '--verify', self.changes_file) signed = (ret == 0) unreleased = (self.distribution.lower() == 'unreleased') if signed and not unreleased: status.append('+ Status: Package signed for %s' % self.distribution) if self.changes_file and not signed: status.append('- Status: Package has been built but not signed') # Check lintian # Check for reported errors # Check errors reported upstream # tarball_dir # export_dir # self.changelog.version # self.changelog.noepoch return status # vi:expandtab:softtabstop=4:shiftwidth=4:smarttab arriero-0.6/src/uscan.py000066400000000000000000000165001240255410300152720ustar00rootroot00000000000000# -*- coding: utf8 -*- # Copyright: 2013-2014, Maximiliano Curia # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the Free # Software Foundation; either version 2 of the License, or (at your option) # any later version. # # This program is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for # more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., 51 # Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import glob import logging import os import re import subprocess import util # Taken from gbp.deb.uscan and modified to suit this program # Copyright: 2012, Guido Günther # License: GPL-2+ '''Interface to uscan''' class UscanError(Exception): pass class Uscan(object): cmd = '/usr/bin/uscan' def __init__(self, dir='.', destdir='..'): self._uptodate = False self._tarball = None self._version = None self._uversion = None self._url = None self._dir = os.path.abspath(dir) self._destdir = destdir @property def uptodate(self): return self._uptodate @property def tarball(self): return self._tarball @property def version(self): return self._version if self._version else self._uversion @property def uversion(self): return self._uversion @property def url(self): return self._url def _parse(self, out, destdir=None): r''' Parse the uscan output return and update the object's properties @param out: uscan output @type out: string >>> u = Uscan('http://example.com/') >>> u._parse('virt-viewer_0.4.0.orig.tar.gz') >>> u.tarball '../virt-viewer_0.4.0.orig.tar.gz' >>> u.uptodate False >>> u._parse('') Traceback (most recent call last): ... UscanError: Couldn't find 'upstream-url' in uscan output ''' ext = None source = None self._uptodate = False d = {} if not destdir: destdir = self._destdir # Check if uscan downloaded something for row in out.split("\n"): # uscan >= 2.10.70 has a target element: for n in ( 'package', 'debian-uversion', 'upstream-version', 'upstream-url', 'target', 'messages', 'status', 'warnings'): m = re.match("<%s>(.*)" % (n, n), row) if m: d[n] = m.group(1) if not d: raise UscanError('No watch file found.') if 'warnings' in d: raise UscanError(d['warnings']) if 'target' in d: source = d['target'] if not source and 'messages' in d: m = re.match(r'.*symlinked ([^\s]+) to it', d['messages']) if m: source = m.group(1) else: m = re.match(r'Successfully downloaded updated package ' '(.+)', d['messages']) if m: source = m.group(1) if 'status' in d: self._uptodate = (d['status'] == 'up to date') if 'upstream-version' in d: self._uversion = d['upstream-version'] if 'debian-uversion' in d: self._version = d['debian-uversion'] if 'upstream-url' in d: self._url = d['upstream-url'] ext = os.path.splitext(self._url)[1] if (not source) and ('package' in d) and self.version and ext: source = '%s_%s.orig.tar%s' % (d['package'], self.version, ext) if source: source = os.path.join(destdir, source) if os.path.exists(source): self._tarball = source if (not self._tarball) and ('package' in d) and self.version: source = '%s_%s.orig.tar.*' % (d['package'], self.version) wild = os.path.join(destdir, source) files = glob.glob(wild) if len(files): self._tarball = files[0] if (not self._tarball) and self._url: source = self._url.rsplit('/', 1)[1] source = os.path.join(destdir, source) if os.path.exists(source): self._tarball = source def _raise_error(self, out): r''' Parse the uscan output for errors and warnings and raise a L{UscanError} exception based on this. If no error detail is found a generic error message is used. @param out: uscan output @type out: string @raises UscanError: exception raised >>> u = Uscan('http://example.com/') >>> u._raise_error("uscan warning: " ... "In watchfile debian/watch, reading webpage\n" ... "http://a.b/ failed: 500 Cant connect " ... "to example.com:80 (Bad hostname)") Traceback (most recent call last): ... UscanError: Uscan failed: uscan warning: In watchfile debian/watch, reading webpage http://a.b/ failed: 500 Cant connect to example.com:80 (Bad hostname) >>> u._raise_error("uscan: Can't use --verbose if " ... "you're using --dehs!") Traceback (most recent call last): ... UscanError: Uscan failed: uscan: Can't use --verbose if you're using --dehs! >>> u = u._raise_error('') Traceback (most recent call last): ... UscanError: Uscan failed - debug by running 'uscan --verbose' ''' msg = None for n in ('errors', 'warnings'): m = re.search("<%s>(.*)" % (n, n), out, re.DOTALL) if m: msg = "Uscan failed: %s" % m.group(1) break if not msg: msg = "Uscan failed - debug by running 'uscan --verbose'" raise UscanError(msg) def scan(self, destdir=None, download=True, force_download=False, version=None): '''Invoke uscan to fetch a new upstream version''' if not destdir: destdir = self._destdir util.ensure_path(destdir) cmd = [self.cmd, '--symlink', '--destdir=%s' % destdir, '--dehs'] if not download: cmd.append('--report') if download and force_download or download and version: cmd.append('--force-download') if version: cmd.append('--download-version=%s' % version) logging.debug('Calling uscan: %s', cmd) p = subprocess.Popen(cmd, cwd=self._dir, stdout=subprocess.PIPE) out = p.communicate()[0] # uscan exits with 1 in case of uptodate and when an error occured. # Don't fail in the uptodate case: self._parse(out, destdir) if not self.uptodate and p.returncode: self._raise_error(out) if download and not self._tarball: raise UscanError("Couldn't find tarball") # vi:expandtab:softtabstop=4:shiftwidth=4:smarttab arriero-0.6/src/util.py000066400000000000000000000056541240255410300151460ustar00rootroot00000000000000#!/usr/bin/env python # -*- coding: utf8 -*- # Copyright: 2013-2014, Maximiliano Curia # # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the Free # Software Foundation; either version 2 of the License, or (at your option) # any later version. # # This program is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for # more details. # # You should have received a copy of the GNU General Public License along with # this program; if not, write to the Free Software Foundation, Inc., 51 # Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import logging import os import re import select import subprocess def quiet(*argv, **kwargs): '''Make an OS call without generating any output.''' devnull = open(os.devnull, 'r+') return subprocess.call(argv, stdin=devnull, stdout=devnull, stderr=devnull, **kwargs) def split(values): '''Split a comma separated string of values into a list.''' return filter(lambda x: x is not '', re.split(r'[\s,]+', values)) def ensure_path(path): '''Create path if it doesn't exist.''' if not os.path.exists(path): logging.info('Creating path: %s', path) os.makedirs(path) def log_check_call(cmd, interactive=False, **kwargs): '''Equivalent to check_call, but logging before and after.''' str_cmd = cmd if isinstance(cmd, (str, unicode)) else ' '.join(cmd) logging.debug('Executing: %s with %s', str_cmd, kwargs) subprocess.check_call(cmd, **kwargs) return # TODO: make this work # Current problems: # * Pressing Ctrl-C interrupts arriero, but not the running process # * Some processes are interactive but their output should still be logged # (git-build-package). And for some we only want errors. if not interactive: call = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) input_fds = [call.stdout, call.stderr] levels = {call.stdout: logging.info, call.stderr: logging.error} else: call = subprocess.Popen(cmd, stderr=subprocess.PIPE, **kwargs) input_fds = [call.stderr] levels = {call.stderr: logging.error} def check_io(): ready_to_read = select.select(input_fds, [], [], 1000)[0] for io in ready_to_read: line = io.readline() if line: levels[io](line[:1]) # keep checking stdout/stderr until the child exits while call.poll() is None: check_io() check_io() # check again to catch anything after the process exits if call.returncode: raise subprocess.CalledProcessError(call.returncode, cmd) # vi:expandtab:softtabstop=4:shiftwidth=4:smarttab