pax_global_header00006660000000000000000000000064140353211110014501gustar00rootroot0000000000000052 comment=b7c19a4898dcdaac9c7f47b660243ad568609371 apt-transport-s3-2.1.0/000077500000000000000000000000001403532111100146425ustar00rootroot00000000000000apt-transport-s3-2.1.0/CHANGELOG000066400000000000000000000024431403532111100160570ustar00rootroot000000000000002019-08-23 Marcin Kulisz * Working Python3 version 2019-06-14 Marcin Kulisz * Switching to VHost style urls * PEP8 compliance * Add support for special AWS regions (cn-north-1 and cn-northwest-1) * Add support for alternate services with 'Endpoint' * Switching to VHost style urls * Adjusting metadata naming convention * Request region from metadata when not configured * Token doesn't have to be an empty string anymore * Updating info about breaking TLS if bucket name * Removing logging to stderr info about from where * Copyright and repo url updates * Higher detail in error handling and information from APT Message 2015-09-23 Marcin Kulisz * Fixing new line issue in Authorization header caused by more restrictive urllib in python 2.7.10, thx to jensschroer (https://github.com/jensschroer) * Added sha512 handling for checksums, thx to Frank Zwart for supplied patch 2015-01-09 Marcin Kulisz * Fixing '+~ ' handling and making http errors more readable, thx for the patches to Shyamal Prasad 2014-08-06 Marcin Kulisz * Initial release * Operational s3 transport with IAM Roles present on the server apt-transport-s3-2.1.0/COPYING000066400000000000000000000432541403532111100157050ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. apt-transport-s3-2.1.0/README.md000066400000000000000000000071221403532111100161230ustar00rootroot00000000000000[License and Copyright]: #license--copyright [Requirements]: #requirements [Configuration]: #configuration [Usage]: #usage [Contribution]: #contribution # apt-transport-s3 ### Table of Contents 1. [License & Copyright][License and Copyright] 2. [Requirements][Requirements] 3. [Configuration][Contribution] 4. [Usage][Usage] 5. [Contribution][Contribution] ## apt-transport-s3 Allow to have a privately hosted apt repository on S3. Access keys are read from `/etc/apt/s3auth.conf` file or IAM role if machine is hosted on AWS or has access to AWS metadata server on 169.254.169.254. They are also taken from the usual environment variables. ## License & Copyright # Copyright (C) 2018- Mayara Cloud Ltd. # Copyright (C) 2016-2018 Claranet Ltd. # Copyright (C) 2014-2016 Bashton Ltd. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA. ## Requirements ### Additional package dependencies (except installed by default in Debian) 1. python-configobj ## Configuration /etc/apt/s3auth.conf or IAM role can provide credentials required for using private apt repositories. NOTE: Region MUST match the region the buckets are stored in and if not defined it will try to fetch it from the metadata service. Setting Endpoint allows for using providers other than Amazon AWS. If set, Endpoint disregards Region. Example of s3auth.conf file: ``` AccessKeyId = myaccesskey SecretAccessKey = mysecretaccesskey Region = 'us-east-1' Endpoint = 'nyc3.digitaloceanspaces.com' ``` ## Usage Install the .deb package from the releases page. The bucket repo should be specified using an s3:// prefix, for example: `deb s3://aptbucketname/repo/ trusty main contrib non-free` if you need to use a proxy to connect to the internet you can specify this as an APT configuration directive (for example in /etc/apt/apt.conf.d/90apt-transport-s3) `Acquire::http::Proxy "http://myproxy:3128/";` Bucket name hosting repo can not contain dots in it's name as this (according to [AWS S3 naming convention](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html)) will invalidate virtual host style paths TLS certificates. ## Testing The module will run in interactive mode. It accepts on `stdin` and outputs on `stdout`. The messages it accepts on stdin are in the following format and [documented here](http://www.fifi.org/doc/libapt-pkg-doc/method.html/index.html#abstract). ``` 600 URI Acquire URI:s3://my-s3-repository/project-a/dists/trusty/main/binary-amd64/Packages Filename:Packages.downloaded Fail-Ignore:true Index-File:true ``` This message will trigger an s3 get from the above bucket and key and save it to Filename. It needs a blank line after the message to trigger the processing by the s3 method. ## Contribution If you want to contribute a patch via PR please create it against development branch. Patches via email are welcome as well. apt-transport-s3-2.1.0/debian/000077500000000000000000000000001403532111100160645ustar00rootroot00000000000000apt-transport-s3-2.1.0/debian/README.source000066400000000000000000000037361403532111100202540ustar00rootroot00000000000000apt-transport-s3 for Debian ----------------------- The apt-transport-s3 source is obtained from by the release tag. The Debian packaging tracks the latest tarball from the above url. The apt-transport-s3 package source is maintained within above github repository on Debian branches named accordingly to DEP-14 (https://dep-team.pages.debian.net/deps/dep14/). * Starting from upload of version 2.0.0-1 uploads to Debian are made with **dgit** tool ** after upstream release on Github following set off commands should allow for source only Debian upload - origtargz (for fetching orig tarball for the new release it should use uscan and for new Debian version it should fetch code from Debian repos) - dgit --gbp sbuild (or build depends on local setup) - dgit --gbp push-source * 'master' branch is where upstream release branch is located * 'development' branch is where upstream development is happening, all pull|merge requests or patches send should be done against this branch * The 'debian/*' branchs are for keeping Debian package specific files (per suit) and merging in (with --no-ff) from 'master' branch to build packages. ** There are no Debian specific patches as package maintainers are also the upstream * We carry upstream tags on Debian branch prefixed with 'upstream/' applied on master merge into debian baranch (with --no-ff) ex.: upstream/1.0.1 * We prefix our tags with 'debian/' e.g. debian/1.0.1-1 ** experimental releases are prefixed 'debian/experimental' e.g. debian/experimental/4.0-1 ** Debian tagging should be done after package had been successfully uploaded to Debian repository (to avoid having tags for debs which doesn't exist in the archive) Information how to use gbp work flow following liks may be useful: https://honk.sigxcpu.org/piki/development/debian_packages_in_git/ https://wiki.debian.org/PackagingWithGit apt-transport-s3-2.1.0/debian/apt-transport-s3.8000066400000000000000000000025421403532111100213210ustar00rootroot00000000000000.\" (C) Copyright 2014 Marcin Kulisz (kuLa) , .TH APT-TRANSPORT-S3 8 "August 14, 2014" .SH NAME .SH DESCRIPTION apt-transport-s3 \- it's a package which contains apt transport ('plugin') allowing apt to fetch files stored in private repositories held on AWS S3. .SH Bucket name According to the S3 bucket naming convention to make it working with this transport over TLS bucket name can NOT contain dots in it. More information about it can be found at https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html .SH AWS Credentials To access repositories held on S3 it is compulsory to have correct permissions into the bucket where repository is located and those permissions have to be reflected in at least one of below two places. .B IAM Roles default place where S3 transport is looking for credentials allowing for fetching files from S3 bucket .B /etc/apt/s3auth.conf credentials configuration file in ini format without section headers .SH URI format URI format is in standard apt format only difference with S3 transport is that instead of HTTP(s), cdrom or FTP protocol .B s3 should be used .br ex. .IR "deb [trusted=yes] s3://bucket_name/ testing main contrib non-free" .PP .SH EXAMPLE /etc/apt/s3auth.conf AccessKeyId = myaccesskey SecretAccessKey = mysecrectkey Region = eu-west-1 Endpoint = 'nyc3.digitaloceanspaces.com' apt-transport-s3-2.1.0/debian/changelog000066400000000000000000000055671403532111100177530ustar00rootroot00000000000000apt-transport-s3 (2.0.0-1) unstable; urgency=medium [ Hayden Myers ] * Python3 Fixes. [ Marcin Kulisz ] * New upstream - Initial py3 implementation - Removing unused code - Added comments into the code - Changelog update * Migrating to debhelper 12 specific debhelper-compat mode * Std-ver bump to 4.4.0 * Repointed d/watch to new location * Adjusting package to py3 -- Marcin Kulisz Tue, 27 Aug 2019 14:22:40 +0100 apt-transport-s3 (1.4.0-1) unstable; urgency=medium [ agx ] * Making proper defaults for the token [ George Alton ] * Adds more detailed error information to APT Message [ Steven Aerts ] * Request region from metadata when not configured * Escape S3 paths correctly [ Andrew James ] * Add support for alternate services with 'Endpoint' [ Difan Zhang ] * Add support for special AWS regions (cn-north-1 and cn-northwest-1). [ Marcin Kulisz ] * Small adjustment to d/gbp * Update to repo location and copyrights * Man page update * New upstream - Copyright and repo url updates in readme and else where - Removing logging to stderr info about from where - Updating info about breaking TLS if bucket name - Removing code duplication - PEP8 compliance - Adjusting meta data naming convention - Minor err msg layout fixes - Switching to VHost style urls - Fixing url quotation and removing quote method - Changing Region & Endpoint approach * std-ver bump and copyright url fix * New (stripped) upstream key * Added d/upstream/metadata * DH compat bump to 12 * Removal of the python version field from the source package -- Marcin Kulisz Mon, 24 Jun 2019 15:42:39 +0100 apt-transport-s3 (1.3.0-1) unstable; urgency=medium * Added build-dep on dh-python and bumping up std-ver to 4.1.1 * d/copyright update * new upstream - support for AWS AUTHv4, thx to Stephen Grier - support for path based buckets - support for proxy -- Marcin Kulisz Thu, 23 Nov 2017 19:54:42 +0000 apt-transport-s3 (1.2.1-1) unstable; urgency=medium * fixing signature handling in d/wach * fixing Vcs-Git url * new upstream v1.2.1 -- Marcin Kulisz (kuLa) Thu, 08 Oct 2015 18:19:59 +0100 apt-transport-s3 (1.2.0-1) unstable; urgency=medium * added missing (from binary) manpage * bumping up standards-verion to 3.9.6 * pointing VCS fields in d/control to the packaging branch for d/sid * new upstream with patches for handling '+~ ' chars and modified error handling for problematic URLs * fixing dch * fixed d/watch and added gpg sign handling -- Marcin Kulisz (kuLa) Sun, 30 Aug 2015 13:44:09 +0100 apt-transport-s3 (1.1.0-2) unstable; urgency=low * Initial release (Closes: #756022) -- Marcin Kulisz (kuLa) Thu, 14 Aug 2014 08:43:59 +0100 apt-transport-s3-2.1.0/debian/control000066400000000000000000000021321403532111100174650ustar00rootroot00000000000000Source: apt-transport-s3 Section: admin Priority: optional Maintainer: Debian Cloud Team Uploaders: Marcin Kulisz , David Watson Build-Depends: python3, debhelper-compat (= 12), dh-python Standards-Version: 4.4.0 Homepage: https://github.com/MayaraCloud/apt-transport-s3 Vcs-Git: https://salsa.debian.org/cloud-team/apt-transport-s3.git Vcs-Browser: https://salsa.debian.org/cloud-team/apt-transport-s3/tree/debian/sid Package: apt-transport-s3 Architecture: all Depends: ${misc:Depends}, ${python3:Depends}, apt, python3, python3-configobj Description: APT transport for privately held AWS S3 repositories This package contains the APT AWS S3 transport. It makes possible to fetch files from repositories privately held on AWS S3. . To start using S3 based repo it's enough to have IAM role on the server and to add line similar to the below to apt sources.list (more information in 'man apt-transport-s3'): deb s3://s3.amazonaws.com/BUCKETNAME wheezy main apt-transport-s3-2.1.0/debian/copyright000066400000000000000000000021301403532111100200130ustar00rootroot00000000000000Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Upstream-Name: apt-transport-s3 Source: https://github.com/MayaraCloud/apt-transport-s3 Files: * Copyright: 2017- Marcin Kulisz 2014-2017 David Watson 2014-2017 Marcin Kulisz License: GPL-2+ This package is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. . This package is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. . You should have received a copy of the GNU General Public License along with this program. If not, see . On Debian systems, the complete text of the GNU General Public License version 2 can be found in "/usr/share/common-licenses/GPL-2". apt-transport-s3-2.1.0/debian/gbp.conf000066400000000000000000000001351403532111100175020ustar00rootroot00000000000000[DEFAULT] upstream-branch = master debian-branch = debian/sid [buildpackage] dist = DEP14 apt-transport-s3-2.1.0/debian/gitlab-ci.yml000066400000000000000000000004221403532111100204400ustar00rootroot00000000000000# Creation of this file is documented at: # https://salsa.debian.org/salsa-ci-team/pipeline#basic-use --- include: - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml apt-transport-s3-2.1.0/debian/install000066400000000000000000000000311403532111100174470ustar00rootroot00000000000000s3 /usr/lib/apt/methods/ apt-transport-s3-2.1.0/debian/manpages000066400000000000000000000000321403532111100175750ustar00rootroot00000000000000debian/apt-transport-s3.8 apt-transport-s3-2.1.0/debian/rules000077500000000000000000000002741403532111100171470ustar00rootroot00000000000000#!/usr/bin/make -f #DH_VERBOSE = 1 DPKG_EXPORT_BUILDFLAGS = 1 include /usr/share/dpkg/default.mk %: dh $@ --with python3 override_dh_installchangelogs: dh_installchangelogs CHANGELOG apt-transport-s3-2.1.0/debian/source/000077500000000000000000000000001403532111100173645ustar00rootroot00000000000000apt-transport-s3-2.1.0/debian/source/format000066400000000000000000000000141403532111100205720ustar00rootroot000000000000003.0 (quilt) apt-transport-s3-2.1.0/debian/upstream/000077500000000000000000000000001403532111100177245ustar00rootroot00000000000000apt-transport-s3-2.1.0/debian/upstream/metadata000066400000000000000000000001231403532111100214230ustar00rootroot00000000000000Name: apt-transport-s3 Repository: https://github.com/MayaraCloud/apt-transport-s3 apt-transport-s3-2.1.0/debian/upstream/signing-key.asc000066400000000000000000000126371403532111100226510ustar00rootroot00000000000000-----BEGIN PGP PUBLIC KEY BLOCK----- mQINBE5x98YBEADTTQ+N8ub+Q7DnK9A+n8gsRWOguTVt1ldDY/JEH8/bh70jhb4J FwtXNSKNGos+Yyxd9OzKQCaLHwO581YpceP/nebfW4mfz2eEZ3QGWVRWHukiXpK+ SxWeBYMhnZNFjvR8p8mpxnFAyACUYGEKS0qoBQQiOCMrPOSe/SerJjFu8jtGLzkT V089N95jB3w4VH1457EtPdWxV01KYyMBI4SRQfmkwLd6YowW2JEP1hSzbtpYRnhg uIvPCxHLQoj/qEQrIAv+IXcZMoafQNRS5Jkhya2Sq/0Cr0Dh9oiFqJl+Ri79a1d6 gh0WurZw4mBOhfJI6eNX3qndLcJnBGQvquDx1bQT1XJlxDBjTwHLINN4QzOE8/p0 S0e0DUf1tvUntEfT3TCduGWj1nxx6DJo/JbaRXV32+H4lnmmMIfONTcjMF9OvHsd wF25lqbxVUJVIugUZjXrkA2avMK3hMvJ23ufElAAr8FFglbyU1zHOCeMnE/vYpjY RDEFLMHrdwCA4Q/Cg+3n5oci+VJcTHDzbUMgxaYhOF3ESX4oKnAq+CqW8+CWuXsi boWa9WvAq9sZfVDyvfl4xBJEeKpSkKX0p4u7yW2r+MnJ+Y/Nn5fcFIdTJ/o+qT85 XKxkTypCg1f/Z8i46F+pHjXlOPhxgGQKJG/mjYIK6Ib+xwOXFfh1j8nAhQARAQAB tCFNYXJjaW4gS3VsaXN6IDxkZWJpYW5Aa3VsaXN6Lm5ldD6JAjoEEwEIACQCGwMC HgECF4AFCwkIBwMFFQoJCAsFFgIDAQAFAlgkg6QCGQEACgkQaGkw3VjDOLOOZg/+ LCcgFsu2izuTgwk5fsOJxcSHAh6zXUbixO7TxNrRjuX7psn83mXvt5BPu/iNVPq9 zR/IAnUWh29OgV0R7p6/iHL9Qcapr85gC9xGHrpK/7enSEDTcPyligquhzvlV12g d8sdbGg6XwuqKVBQUN8fJCC80SIToRA11uxnaiLj3u/orpD6XKLWzco0o6jp119j RwEsS/IbcfIr4DCCmAyC6oaH5uOOJq1wyx7cleMcjpGx96ra4nADORpJMFz6OvgY YZS5nvAFPOp49XOmw9Bli/FmDvwuykz54cTlnWmsrjJHijbAdiTgLZ3FT8vDc+xL HC7DjYoILWOrTBWBljmm7MpQOFLywZhqCb9B0v5P20Il9UXxhzdM4HS0DmRLGxt7 h8VWmKAvOxu8ndiPhzYFhp7e/4hKNL8GBCniUg1ObZGlciUKA+54h7GopTHF5/w1 guYJmguGxfzAam1eoJiEoxNeOvAom8uDukc7GiLbh9jrgS0ODjqbjWe4E8JMZ9G+ rKQP3bzmerEWXivm7VJQHaQgR+VLujBGGO5key3pG1EnTGubNsBZ9yVnmsZUS6XX Upi23DaT3x1dSICJLJdqiZelzHgZR2b2NwBpYf9OKMBYw5YaQ5d7kQT5yQmqcxyD NQevr7xxk6DUTI/tAdUVAnL3DQiJD49KIB7p2jnY7km0H01hcmNpbiBLdWxpc3og PGt1bGFAa3VsaXN6Lm5ldD6JAjcEEwEIACECGwMCHgECF4AFCwkIBwMFFQoJCAsF FgIDAQAFAlgkg5oACgkQaGkw3VjDOLNsyg//ft4aErycE7U2lbnHdW3Z0uycI8cu jX+KzqO1AFD2NAMS+haqru+jZpQlUpx8JZQuKKrawDMM5Mx03DRXAkCQn1gCLu5a WBpw9IbwiFL1X2tgO3hqFLdf0do9aRQO63mYd6qITPigEQ1iXX5o/XDhcNgiEY4J cKEg+CbdwTY7pyrf+6k57Ul6G9+MtNeblG2B0FOWlSakug2RJ9mH7VzvZGCA4h6k abhoJ5qAkyi/2IXkDf8hSLMrWmrsSv1aGd+vjxwmn2vfp/aXbvACb/UjKa1BHtWa ulW49bgxW2gB0NRa9p/U3QVc/c5ZhK+mxsmvhF1hPgc4TF8iMAtXLBUHqvIaEzym pfDQcGHvmsrYNFabffneEv1mV+txWWBJQIkUsF5vm+mCjJyRGoPC4n6+sd64O8tI LwcC6ujMr/M5dY9Z69+Ul2LfgKuTdJozNj4KIPLD5jjzpsC5rTCE6nc9hlGrUJQb L8Q9pJbpWrqkBHbfNu13Tw2PN7F1WZj24phUS6aVF24K1Xea9VbJASzYqp7iVFzj UPYKQlSgg+udDohFTvJt7GB+BHaTNxHOj+2mzfyZI6JsuU4eHCLpLHILSbdbebsS PfiWgh0/X8heSxHTtFLuX3gK+pOiR1VvFrBUrnEwpaJZYwq0pnmb/iF7rp6Fc2oJ tP6Q+qv6OGl1FOO0KE1hcmNpbiBLdWxpc3ogKGt1TGEpIDxkZWJpYW5Aa3VsaXN6 Lm5ldD6JAjcEEwEIACECGwMCHgECF4AFAlIbeCkFCwkIBwMFFQoJCAsFFgIDAQAA CgkQaGkw3VjDOLNBIw/9E8be+9SlzBKpxzSb1DXpUv4/uPtHk7mzqf/OUmTu3yhB RyZZ9+FY88l6/+qDXU4sc7L5nvRR0CDsYFuTmpeQWSwoNVFEA9FH6suDVrpP+Yyl 0Vx4hCQkVnrM/14nCzD4nHONg3LsaDjbUc+jeM5y61MBH8sNeosQQf0WyFKWfzv8 BpR4ndlWWBwckh3IpACd6rE68Q189exJ6rdZc1LLjLEKFAfSK7Gq8Qtcd4MWWYMl oRoLEOW7UClqduFru5Bd9epD1Zyse7Zl8cUJiDXfSZfvfpJTf/q1fB9dDFHaOzFP z9v0t8swpuZK+0aMb13TCX71wTkZvcO6j5dWMsAOpvVL6/LCvo0N9brdjULEP4Yy rL8W+ET4Sm0JFn1+L0TCX1C0YBLTu3wimypPQi1UDBH/maWudxtUMwe/l4CCE6j0 9/UIrUb3gF1H+ei8qgoA+Nr9GVpwqsJJfZyVIBaQHuY/DMAV9IT7/uz4hr7CXzWu 8XtbvhTb+yfp7MmXHQQPhIfMildOHjzEg1P7qXMW723fNBNF+U7p5o6HLeKdHv51 WpxkkdP9wXoDN88Q8hC3n66GchbWOt2FBgnDHsZ+wRzyLt59DknlBCnltUIEbRh8 sX6hkY41+dUTSxRxALYC6w7TgxyYyo6k4nTcI9dbALxNWC8IJvMzEtJc/qtKqzC0 H01hcmNpbiBLdWxpc3ogPGt1bGFAZGViaWFuLm9yZz6JAjcEEwEIACEFAlds8e0C GwMFCwkIBwMFFQoJCAsFFgMCAQACHgECF4AACgkQaGkw3VjDOLPypQ/8DmJAfPUa ApIPl79JW1nJe+VwpCACpgfMAV06lY8rX0lWLGawO8AY9QQpn9bfk9PRrszLaqJA LOFd7xrRDX5wugGIETBe15g819Cu7kBI9IkSmyPBWLVsM8vB7W2XovIKU0aGReqw RVdMTx2QuBmT9M9+U6e7/Jvmx1ZhSbhOQCG5XQ5ATESqxXudJKNlY6AP7qBmGoEv JvRxCFtr7kx9U4RHIICOxNCO7EFiVtxCN8tQqSkf+ZMex7kS2I6XIzj78g1+wxDn Vgvv1TNn82NiM5BP8vcuGF/3YpALHseHqACFsPOa8gCepdAnudDJ9n9eSTGRWRzl Kee6+QlU5a4cARkTTEDx9pnoCqe7WlI9l6FHOE/QDqpVO1JOwZNHrh4sN8cXbqYj YBrgqTvlG1m6g1iJfktY8orYyo8w7rSE489ZTsnHkWi8lv6uuZcvW9i4e8XrQulq DBR9oktVHECM8CBNme2LiV7q/L+v1kc9IxIJdsjpVwj/c9Xh4ngpTCgdUjzdlcAz YuQEesBnUui8I63eZUtpqw4S5iaq4xfsWRKPgla8pi4d8MwEy2ShcmkLXs8wQaGT s1p19M/iebwNR4dHjUPjpUZzZyagfoKkmuAAt9Ul4VuT7PeR4YVYWgcoLhBHU93L 8WPDuRpTN6f+rm1mkRaKmNt9XywPz3iW5Z65Ag0ETnH3xgEQANUVIvC6KwvXs7eD V2PJdp5Qk8MgZXpkqCqt9XR3wdfJAfdvJZQA/yul1RWeMT/1HV3EmObg4OIk6DUk BewyUF1qsCj6Q2hJeMG4w0pb9ooth1mTyZaKElo1Ws6rSEKZEN9FQhzxB2gAZMxw QBzwVdsnI4dNGQ0GroDmakcXl9FJlIYett66jDTj2BvNNsau8THcrn8EesnMklc5 1LBSjNSFqSA79KVfWsAnbSFZlk75a6mQwm0ELIjRcIEFGPhGIsbHQ7yfFi1xRfwb rMKFblbgXGfgFkj8ptG7lahccggzJ8nTZzr+X22k3DGh2PCgDvzykpE/8MOo1cQ8 3dYfT1s4I1gm3Rd2q4EEgjvEGKjKRwhyRGySJDqCHXFKyv5wvYo35Dx97Aimc+jY unjuYi9YadqpbNSFtORjNq43/LXeQZbTi3dz5t7JzhemsFnMTJ4A7uTI71OzL2/7 F6PpvP0M+f8DYvgXBMCOyr80BHGDlYnZwzezh0mNWK/8J/g3+2gnSgvS4s4TH1vf oVlpK0LwI87pqLeF5q9VOuYMcXfNNxI/LCGCmrTiEqROTgL5FauawpJkd0Mh5jG+ 1exPB7ru2DItY7Jp8GS900xsM9y2P5FafHmbaDGSiDo6EEQmNR/dS1nzRyXBiV7I Z1js8eu03704UfXxUUZdp0vldAKtABEBAAGJAh8EGAECAAkFAk5x98YCGwwACgkQ aGkw3VjDOLNBGRAAvYCQeI6UKyvlvcyg490mCsktXKJjUplrgHpaRZpgoroIjIuM XOnXaWmyGLSflQiS8f8+bjTqJUN1pM0Lcbtp6KQ/qyCtjOyXFVjytD1+ff2GiJEF LhGCGNd3fBcYabPBk2URo1XsoIH1nyegq2JZvrteGQRRa8DY/aYtF1MHsJW/U9X5 8ws72Npw3eT+LaUBzVrwIkQNpuiA3WueyM2bILuIcWUhv0tZhuAVwT5U3SivwTq/ 65GzmCsOui1925OkRNwM0GDj/SOIxoBNfyhlDyMpG7VJszwuWIbftPtGkYKiEZiU 2sARLgs4Ri9YBwBaLZyT81BSzisSVkjh0rtVZb/Rd2IR1cHWX8h8fJ0a4kuZAaHi juScWlcGUIXW/RPhMgO0u3euuSeAKWTL7j9JZ8lbd9nJbJZyuKmv44g/iqQ9pqWI MBP7odX6FuJz6+ORSe/w6+N1F8inDsKhHN9WzFQgd/jraCiYL8Gu7CPZAEp9N1Vm b+R47/SmJDcAsSadeshaDTgt8+ytLSukNxTuuC45vR5wKewU0MgB2cWKzsFMpRGE WKk74wZk7tTisIGDpgCcOIOLL84PIqDCjc5+y8QPZDkbOMwCfAVFl7S+G+qfIcFZ mZ/FZdCzktUxFZaN+O+OrKeE7FkTtWnDdzfmxdQ7rDpS/3N1g/2UsQPnONQ= =DDp7 -----END PGP PUBLIC KEY BLOCK----- apt-transport-s3-2.1.0/debian/watch000066400000000000000000000003451403532111100171170ustar00rootroot00000000000000version=3 opts="filenamemangle=s/(?:.*)?v?(\d[\d\.]*)\.tar\.gz/apt-transport-s3-$1.tar.gz/,pgpsigurlmangle=s/$/.asc/" \ https://github.com/MayaraCloud/apt-transport-s3/releases .*/v\d\S*/apt-transport-s3-?(\d[\d\.]*)\.tar\.gz apt-transport-s3-2.1.0/s3000077500000000000000000000546721403532111100151330ustar00rootroot00000000000000#!/usr/bin/python3 -u # Copyright (C) 2018- Mayara Cloud Ltd. # Copyright (C) 2016-2018 Claranet Ltd. # Copyright (C) 2014-2016 Bashton Ltd. # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # # https://github.com/MayaraCloud/apt-transport-s3 import urllib.request import urllib.error import urllib.parse import time import hashlib import hmac import json import sys import os import datetime import xml.etree.ElementTree as ET import socket import ssl from configobj import ConfigObj RETRIES = 5 SPECIAL_REGION_ENDPOINTS = { 'cn-north-1': 's3.cn-north-1.amazonaws.com.cn', 'cn-northwest-1': 's3.cn-northwest-1.amazonaws.com.cn' } TOKEN_TTL_SECONDS = 21600 TOKEN_HEADER = "X-aws-ec2-metadata-token" TOKEN_HEADER_TTL = "X-aws-ec2-metadata-token-ttl-seconds" def wait_time(c): return pow(2, c) - 1 class AWSCredentials(object): """ Class for dealing with IAM role credentials from meta-data server and later on to deal with boto/aws config provided keys """ def __init__(self, config_file=None): self.conf_file = config_file host = 'http://169.254.169.254' path_session_token = 'latest/api/token' path_sec_cred = '/latest/meta-data/iam/security-credentials/' path_instance_metadata = '/latest/dynamic/instance-identity/document' self.session_token_metadata = urllib.parse.urljoin(host, path_session_token) self.credentials_metadata = urllib.parse.urljoin(host, path_sec_cred) self.instance_metadata = urllib.parse.urljoin( host, path_instance_metadata) self.session_token = None def __imdsv2_ensure_token(self): # Get IMDSv2 session token request = urllib.request.Request(self.session_token_metadata, method='PUT', headers={TOKEN_HEADER_TTL: TOKEN_TTL_SECONDS}) response = None for i in range(0, RETRIES): try: response = urllib.request.urlopen(request, None, 10) self.session_token = response.read() break except ssl.SSLError as e: if 'IMDSv2 timed out' in e.message: time.sleep(wait_time(i + 1)) else: raise e except socket.timeout: time.sleep(wait_time(i + 1)) except urllib.error.URLError as e: if hasattr(e, 'reason'): raise Exception("IMDSv2 URL error reason: {}, probable cause is \ that you don't have IAM role on this machine".format(e.reason)) elif hasattr(e, 'code'): raise Exception("Server error code: {}".format(e.code)) finally: if response: response.close() else: raise Exception("IMDSv2 session token request timed out") def __get_role(self): if not self.session_token: self.__imdsv2_ensure_token() # Read IAM role from AWS metadata store request = urllib.request.Request(self.credentials_metadata, headers={TOKEN_HEADER: self.session_token}) response = None for i in range(0, RETRIES): try: response = urllib.request.urlopen(request, None, 10) self.iamrole = str(response.read(), "utf-8") break except ssl.SSLError as e: if 'timed out' in e.message: time.sleep(wait_time(i + 1)) else: raise e except socket.timeout: time.sleep(wait_time(i + 1)) except urllib.error.URLError as e: if hasattr(e, 'reason'): raise Exception("URL error reason: {}, probable cause is \ that you don't have IAM role on this machine".format(e.reason)) elif hasattr(e, 'code'): raise Exception("Server error code: {}".format(e.code)) finally: if response: response.close() else: raise Exception("GetRole request timed out") def __load_config(self): """ Loading config file from predefined location. Example config file content: AccessKeyId = mykey SecretAccessKey = mysecretkey Region = eu-west-2 # this is obligatory if instance is in different region then apt repo """ # Checking if 'file' exists, if it does read it if os.path.isfile(os.path.expanduser(self.conf_file)): config = ConfigObj(os.path.expanduser(self.conf_file)) return config else: raise Exception("Config file: {} doesn't exist".format(self.conf_file)) def __request_json(self, url): if not self.session_token: self.__imdsv2_ensure_token() request = urllib.request.Request(url, headers={TOKEN_HEADER: self.session_token}) response = None for i in range(0, RETRIES): try: response = urllib.request.urlopen(request, None, 30) return json.loads(response.read().decode('utf-8')) except ssl.SSLError as e: if 'timed out' in e.message: time.sleep(wait_time(i + 1)) else: raise e except socket.timeout: time.sleep(wait_time(i + 1)) except urllib.error.URLError as e: if hasattr(e, 'reason'): raise Exception("URL error reason: {}".format(e.reason)) elif hasattr(e, 'code'): raise Exception("Server error code: {}".format(e.code)) finally: if response: response.close() raise Exception("request for {} timed out".format(url)) def __get_region(self, config): # try to load region from config file region = config.get('Region') # if region is not defied load it from instance metadata if region is None or region == '': region = self.__request_json(self.instance_metadata)['region'] return region def __get_endpoint(self, config): """ We assume that if endpoint attribute exists in the config this takes priority over Region and bucket URL construction. If it doesn't exist checking Region and comparing it to the dict containing special regions (slightly different fqdn's). If it's there use it, if not construct host for bucket url using Region from the config or from the instance metadata (if bucket and host are in the same region it will work, if not error will be thrown) """ if config.get('Endpoint') is not None: host = config['Endpoint'] else: if self.region in list(SPECIAL_REGION_ENDPOINTS.keys()): host = SPECIAL_REGION_ENDPOINTS[self.region] else: host = 's3.{}.amazonaws.com'.format(self.region) return host def get_credentials(self): """ Read IAM credentials from AWS metadata store. Note: This method should be explicitly called after constructing new object, as in 'explicit is better than implicit'. """ data = {} try: data = self.__load_config() except: pass self.region = self.__get_region(data) self.host = self.__get_endpoint(data) # Setting up AWS creds based on environment variables if env vars # doesn't exist setting var value to None if data.get("AccessKeyId") is None: data['AccessKeyId'] = os.environ.get("AWS_ACCESS_KEY_ID", None) data['SecretAccessKey'] = os.environ.get( "AWS_SECRET_ACCESS_KEY", None) data['Token'] = os.environ.get("AWS_SESSION_TOKEN", None) if data.get("AccessKeyId") is None: self.__get_role() data = self.__request_json(urllib.parse.urljoin(self.credentials_metadata, str(self.iamrole, 'utf-8'))) self.access_key = data['AccessKeyId'] if self.access_key is None or self.access_key == '': raise Exception("AccessKeyId required") self.secret_key = data['SecretAccessKey'] if self.secret_key is None or self.secret_key == '': raise Exception("SecretAccessKey required") self.token = data.get('Token') def v4Sign(self, key, msg): return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest() def getSignatureKey(self, dateStamp, serviceName): kDate = self.v4Sign( ('AWS4' + self.secret_key).encode('utf-8'), dateStamp) kRegion = self.v4Sign(kDate, self.region) kService = self.v4Sign(kRegion, serviceName) kSigning = self.v4Sign(kService, 'aws4_request') return kSigning def uriopen(self, uri): """uriopen(uri) open the remote file and return a file object.""" try: return urllib.request.urlopen(self._request(uri), None, 30) except urllib.error.HTTPError as e: # HTTPError is a "file like object" similar to what # urllib.request.urlopen returns, so return it and let caller # deal with the error code if e.code == 400: # token errors are buried in 400 messages so expose xmlResponse = ET.fromstring(e.read()) if xmlResponse is not None: e.msg = "{} - {}".format(e, xmlResponse.find("Message").text) if e.code == 301: e.msg = "{} - Set s3auth.conf region to match bucket 'Region':\ bucket may not be in {}".format(e, self.region) return e # For other errors, throw an exception directly except urllib.error.URLError as e: if hasattr(e, 'reason'): raise Exception("URL error reason: gc" % e.reason) elif hasattr(e, 'code'): raise Exception("Server error code: gc" % e.code) except socket.timeout: raise Exception("Socket timeout") def _request(self, uri): uri_parsed = urllib.parse.urlparse(uri) # Below if statement is to accommodate AWS guidance in regards buckets # naming convention presented in # https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html # S3 bucket names for apt repo can NOT contain '.' in the name # otherwise TLS certs are broken if '.' in uri_parsed.netloc: raise Exception("uri should not include fully qualified domain \ name for bucket") scheme = 'https' bucket = uri_parsed.netloc host = '{}.{}'.format(bucket, self.host) path = '{}'.format(urllib.parse.quote(uri_parsed.path, safe='-._~/')) s3url = urllib.parse.urlunparse( ( scheme, host, path, '', '', '' ) ) request = urllib.request.Request(s3url) request.add_header('x-amz-content-sha256', self._payload_hash(request)) # Create a date for headers and the credential string amzdate = datetime.datetime.utcnow().strftime('%Y%m%dT%H%M%SZ') request.add_header('x-amz-date', amzdate) if self.token is not None and self.token != '': request.add_header('x-amz-security-token', self.token) canonical_request = self._canonical_request(request, host, amzdate) authorization_header = self._authorization_header( canonical_request, amzdate) request.add_header('Authorization', authorization_header) return request def _authorization_header(self, canonical_request, amzdate): """ AWS documentation for signing requests with v4Sign: https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html """ datestamp = amzdate.split('T')[0] algorithm = 'AWS4-HMAC-SHA256' credential_scope = datestamp + '/' + self.region + '/s3/aws4_request' string_to_sign = algorithm + '\n' \ + amzdate + '\n' \ + credential_scope + '\n' \ + canonical_request signing_key = self.getSignatureKey(datestamp, 's3') signature = hmac.new(signing_key, string_to_sign.encode( 'utf-8'), hashlib.sha256).hexdigest() authorization_header = "{} Credential={}/{}, SignedHeaders={}, Signature={}".format( algorithm, self.access_key, credential_scope, self._signed_headers(), signature ) return authorization_header def _canonical_request(self, request, host, amzdate): canonical_uri = request.selector canonical_querystring = '' canonical_headers = 'host:' + host + '\n' \ + 'x-amz-content-sha256:' \ + self._payload_hash(request) + '\n' \ + 'x-amz-date:' + amzdate + '\n' if self.token is not None and self.token != '': canonical_headers += 'x-amz-security-token:' + self.token + '\n' canonical_request = request.get_method() + '\n' \ + canonical_uri + '\n' \ + canonical_querystring + '\n' \ + canonical_headers + '\n' \ + self._signed_headers() + '\n' \ + self._payload_hash(request) return hashlib.sha256(canonical_request.encode('utf-8')).hexdigest() def _signed_headers(self): signed_headers = 'host;x-amz-content-sha256;x-amz-date' if self.token is not None and self.token != '': signed_headers += ';x-amz-security-token' return signed_headers def _payload_hash(self, request): payload = request.data if payload is None: payload = ''.encode('utf-8') return hashlib.sha256(payload).hexdigest() class APTMessage(object): MESSAGE_CODES = { 100: 'Capabilities', 102: 'Status', 200: 'URI Start', 201: 'URI Done', 400: 'URI Failure', 600: 'URI Acquire', 601: 'Configuration', } def __init__(self, code, headers): self.code = code self.headers = headers def encode(self): result = '{0} {1}\n'.format(self.code, self.MESSAGE_CODES[self.code]) for item in list(self.headers.keys()): if self.headers[item] is not None: result += '{0}: {1}\n'.format(item, self.headers[item]) return result + '\n' class S3_method(object): __eof = False def __init__(self, config_file='/etc/apt/s3auth.conf'): self.iam = AWSCredentials(config_file) self.iam.get_credentials() self.send_capabilities() def fail(self, message='Failed'): self.send_uri_failure({'URI': self.uri, 'Message': message}) def _read_message(self): """ Apt uses for communication with its methods the text protocol similar to http. This function parses the protocol messages from stdin. """ if self.__eof: return None result = {} line = sys.stdin.readline() while line == '\n': line = sys.stdin.readline() if not line: self.__eof = True return None s = line.split(" ", 1) result['_number'] = int(s[0]) result['_text'] = s[1].strip() while not self.__eof: line = sys.stdin.readline() if not line: self.__eof = True return result if line == '\n': return result (item, value) = line.split(":", 1) if not result.get(item): result[item] = [] result[item].append(value.strip()) return result def send(self, code, headers): message = APTMessage(code, headers) sys.stdout.write(message.encode()) def send_capabilities(self): self.send(100, { 'Version': '1.1', 'Single-Instance': 'true', 'Send-Config': 'true'}) def send_status(self, headers): self.send(102, headers) def send_uri_start(self, headers): self.send(200, headers) def send_uri_done(self, headers): self.send(201, headers) def send_uri_failure(self, headers): self.send(400, headers) def run(self): """Loop through requests on stdin""" while True: message = self._read_message() if message is None: return 0 if message['_number'] == 601: try: self.configure(message) except Exception as e: self.fail(e.__class__.__name__ + ": " + str(e)) elif message['_number'] == 600: try: self.fetch(message) except Exception as e: self.fail(e.__class__.__name__ + ": " + str(e)) else: self.fail(str(message['_text'])) def configure(self, message): """ Reads APT configuration (dict) and checks for proxy settings by reading all available options with in 'Config-Item' and lookinup 'Acquire::http::Proxy' """ for item in message['Config-Item']: if item.startswith('Acquire::http::Proxy'): (key, value) = item.split('=', 1) if key == 'Acquire::http::Proxy': os.environ['http_proxy'] = value os.environ['https_proxy'] = value def format_error_response(self, response): """Extracts Info from AWS Error Repsonse and returns a formatted string When the S3 API returns an error the body of the response also contains information about the error. This function reads the Response object for the body and returns a string of the form "[{Code}] {Message}" will return an empty string when the response can not be parsed https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#RESTErrorResponses """ try: error = ET.fromstring(response.read()) except ET.ParseError: return '' code = error.find('Code') message = error.find('Message') if code is not None and message is not None: return '[{}] {}'.format(code.text, message.text) # fallback to returning an empty string return '' def fetch(self, msg): self.uri = msg['URI'][0] self.filename = msg['Filename'][0] self.send_status({'URI': self.uri, 'Message': 'Waiting for headers'}) for i in range(0, RETRIES): try: response = self.iam.uriopen(self.uri) except ssl.SSLError as e: if 'timed out' in e.message: time.sleep(wait_time(i + 1)) continue else: raise e except socket.timeout: time.sleep(wait_time(i + 1)) continue self.send_status({ 'URI': self.uri, 'Message': 'Waiting for headers'}) if response.code != 200: error_detail = self.format_error_response(response) self.send_uri_failure({ 'URI': self.uri, 'Message': '{} {} {}'.format(response.code, response.msg, error_detail), 'FailReason': 'HttpError' + str(response.code)}) try: while True: data = response.read(4096) if not len(data): break except ssl.SSLError as e: if 'timed out' in e.message: pass else: raise e except socket.timeout: pass finally: response.close() return self.send_uri_start({ 'URI': self.uri, 'Size': response.headers.get('content-length'), 'Last-Modified': response.headers.get('last-modified')}) f = open(self.filename, "wb") hash_sha256 = hashlib.sha256() hash_sha512 = hashlib.sha512() hash_md5 = hashlib.md5() try: while True: data = response.read(4096) if not len(data): break hash_sha256.update(data) hash_sha512.update(data) hash_md5.update(data) f.write(data) break except ssl.SSLError as e: if 'timed out' in e.message: time.sleep(wait_time(i + 1)) else: raise e except socket.timeout: time.sleep(wait_time(i + 1)) finally: response.close() f.close() else: raise Exception("Fetch request timed out") self.send_uri_done({ 'URI': self.uri, 'Filename': self.filename, 'Size': response.headers.get('content-length'), 'Last-Modified': response.headers.get('last-modified'), 'MD5-Hash': hash_md5.hexdigest(), 'MD5Sum-Hash': hash_md5.hexdigest(), 'SHA256-Hash': hash_sha256.hexdigest(), 'SHA512-Hash': hash_sha512.hexdigest()}) if __name__ == '__main__': try: config = '/etc/apt/s3auth.conf' if len(sys.argv) == 2 and os.path.isfile(sys.argv[1]): config = sys.argv[1] method = S3_method(config) ret = method.run() sys.exit(ret) except KeyboardInterrupt: pass apt-transport-s3-2.1.0/s3auth.conf000066400000000000000000000000221403532111100167120ustar00rootroot00000000000000Region='eu-west-2'