slapos.core-0.35.1/0000755000076600007660000000000012110425554022115 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/CHANGES.txt0000644000076600007660000003665512110424627023745 0ustar cedricdesaintmartincedricdesaintmartin00000000000000Changes ======= 0.35.1 (2012-02-18) ------------------- New features: * Add ComputerPartition._instance_guid getter in SLAP library. [Cedric de Saint Martin] * Add ComputerPartition._instance_guid support in slapproxy. [Cedric de Saint Martin] Fixes: * Fix link existence check when deploying instance if SR is not correctly installed. This fixes a misleading error. [Cedric de Saint Martin] * Improve message shown to user when requesting. [Cedric de Saint Martin] * Raise NotReady when _requested_state doesn't exist when trying to fetch it from getter. [Cedric de Saint Martin] 0.35 (2012-02-08) ----------------- * slapos: display version number with help. [Marco Mariani] * slapformat: backup slapos.xml to a zip archive at every change. [Marco Mariani] * slapformat: Don't check validity of ipv4 when trying to add address that already exists. [Cedric de Saint Martin] * slapgrid: create and run $MD5/buildout.cfg for eaiser debugging. [Marco Mariani] * slapgrid: keep running if cp.error() or sr.error() have issues (fixes 20130119-744D94). [Marco Mariani] * slapgrid does not crash when there are no certificates (fixes #20130121-136C24). [Marco Mariani] * Add slapproxy-query command. [Marco Mariani] * Other minor typo / output fixes. 0.34 (2013-01-23) ----------------- * networkcache: only match major release number in Debian, fixed platform detection for Ubuntu. [Marco Mariani] * symlink to software_release in each partition. [Marco Mariani] * slapos client: Properly expand "~" when giving configuration file location. [Cedric de Saint Martin] * slapgrid: stop instances that should be stopped even if buildout and/or reporting failed. [Cedric de Saint Martin] * slapgrid: Don't periodically force-process a stopped instance. [Cedric de Saint Martin] * slapgrid: Handle pid files of slapgrid launched through different entry points. [Cedric de Saint Martin] * Watchdog: Bang is called with correct instance certificates. [Cedric Le Ninivin] * Watchdog: Fix watchdog call. [Cedric le Ninivin] * Add a symlink of the used software release in each partitions. [Marco Mariani] * slapformat is verbose by default. [Cedric de Saint Martin] * slapproxy: Filter by instance_guid, allow computer partition renames and change of software_type and requested_state. [Marco Mariani] * slapproxy: Stop instance even if buildout/reporting is wrong. [Cedric de Saint Martin] * slapproxy: implement softwareInstanceRename method. [Marco Mariani] * slapproxy: alllow requests to software_type. [Marco Mariani] * Many other minor fixes. See git diff for details. 0.33.1 (2012-11-05) ------------------- * Fix "slapos console" argument parsing. [Cedric de Saint Martin] 0.33 (2012-11-02) ----------------- * Continue to improve new entry points. The following are now functional: - slapos node format - slapos node start/stop/restart/tail - slapos node supervisord/supervisorctl - slapos node supply and add basic usage. [Cedric de Saint Martin] * Add support for "SLAPOS_CONFIGURATION" and SLAPOS_CLIENT_CONFIGURATION environment variables. (commit c72a53b1) [Cédric de Saint Martin] * --only_sr also accepts plain text URIs. [Marco Mariani] 0.32.3 (2012-10-15) ------------------- * slapgrid: Adopt new return value strategy (0=OK, 1=failed, 2=promise failed) (commit 5d4e1522). [Cedric de Saint Martin] * slaplib: add requestComputer (commits 6cbe82e0, aafb86eb). [Łukasz Nowak] * slapgrid: Add stopasgroup and killasgroup to supervisor (commit 36e0ccc0). [Cedric de Saint Martin] * slapproxy: don't start in debug mode by default (commit e32259c8). [Cédric Le Ninivin * SlapObject: ALWAYS remove tmpdir (commit a652a610). [Cedric de Saint Martin] 0.32.2 (2012-10-11) ------------------- * slapgrid: Remove default delay, now that SlapOS Master is Fast as Light (tm). (commit 03a85d6b8) [Cedric de Saint Martin] * Fix watchdog entry point name, introduced in v0.31. (commit a8651ba12) [Cedric de Saint Martin] * slapgrid: Better filter of instances, won't process false positives anymore (hopefully). (commit ce0a73b41) [Cedric de Saint Martin] * Various output improvements. [Cedric de Saint Martin] 0.32.1 (2012-10-09) ------------------- * slapgrid: Make sure error logs are sent to SlapOS master. Finish implementation began in 0.32. [Cedric de Saint Martin] * slapgrid: Fix Usage Report in case of not empty partition with no SR. [Cedric de Saint Martin] 0.32 (2012-10-04) ----------------- * Introduce new, simpler "slapos" entry point. See documentation for more informations. Note: some functionnalities of this new entry point don't work yet or is not as simple as it should be. [Cedric de Saint Martin, Cedric Le Ninivin] * Revamped "slapos request" to work like described in documentation. [Cédric Le Ninivin, Cédric de Saint Martin] * Rewrote slapgrid logger to always log into stdout. (commits a4d277c881, 5440626dea)[Cédric de Saint Martin] 0.31.2 (2012-10-02) ------------------- * Update slapproxy behavior: when instance already exist, only update partition_parameter_kw. (commit 317d5c8e0aee) [Cedric de Saint Martin] 0.31.1 (2012-10-02) ------------------- * Fixed Watchdog call in slapgrid. [Cédric Le Ninivin] 0.31 (2012-10-02) ------------------- * Added slapos-watchdog to bang exited and failing serices in instance in supervisord. (commits 16b2e8b8, 1dade5cd7) [Cédric Le Ninivin] * Add safety checks before calling SlapOS Master if mandatory instance members of SLAP classes are not properly set. Will result in less calls to SlapOS Master in dirty cases. (commits 5097e87c9763, 5fad6316a0f6d, f2cd014ea8aa) [Cedric de Saint Martin] * Add "periodicty" functionnality support for instances: if an instance has not been processed by slapgrid after defined time, process it. (commits 7609fc7a3d, 56e1c7bfbd) [Cedric Le Ninivin] * slapproxy: Various improvements in slave support (commits 96c6b78b67, bcac5a397d, fbb680f53b)[Cedric Le Ninivin] * slapgrid: bulletproof slapgrid-cp: in case one instance is bad, still processes all other ones. (commits bac94cdb56, 77bc6c75b3d, bd68b88cc3) [Cedric de Saint Martin] * Add support for "upload to binary cache" URL blacklist [Cedric de Saint Martin] * Request on proxy are identified by requester and name (commit 0c739c3) [Cedric Le Ninivin] 0.30 (2012-09-19) ----------------- * Add initial "slave instances" support in slapproxy. [Cedric Le Ninivin] * slapgrid-ur fix: check for partition informations only if we have to destroy it. [Cedric de Saint Martin] 0.29 (2012-09-18) ----------------- * buildout: Migrate slap_connection magic instance profile part to slap-connection, and use variables names separated with '-'. [Cedric de Saint Martin] * slapgrid: Add support for instance.cfg instance profiles [Cedric de Saint Martin] * slapgrid-ur: much less calls to master. [Cedric de Saint Martin] 0.28.9 (2012-09-18) ------------------- * slapgrid: Don't process not updated partitions (regression introduced in 0.28.7). [Cedric de Saint Martin] 0.28.8 (2012-09-18) ------------------- * slapgrid: Don't process free partitions (regression introduced in 0.28.7). [Cedric de Saint Martin] 0.28.7 (2012-09-14) ------------------- * slapgrid: --maximal_delay reappeared to be used in special cases. [Cedric de Saint Martin] 0.28.6 (2012-09-10) ------------------- * register now use slapos.cfg.example from master. [Cédric Le Ninivin] 0.28.5 (2012-08-23) ------------------- * Updated slapos.cfg for register [Cédric Le Ninivin] 0.28.4 (2012-08-22) ------------------- * Fixed egg building. 0.28.3 (2012-08-22) ------------------- * Avoid artificial tap creation on system check. [Łukasz Nowak] 0.28.2 (2012-08-17) ------------------- * Resolved path problem in register [Cédric Le Ninivin] 0.28.1 (2012-08-17) ------------------- * Resolved critical naming conflict 0.28 (2012-08-17) ----------------- * Introduce "slapos node register" command, that will register computer to SlapOS Master (vifib.net by default) for you. [Cédric Le Ninivin] * Set .timestamp in partitions ONLY after slapgrid thinks it's okay (promises, ...). [Cedric de Saint Martin] * slapgrid-ur: when destroying (not reporting), only care about instances to destroy, completely ignore others. [Cedric de Saint Martin] 0.27 (2012-08-08) ----------------- * slapformat: Raise correct error when no IPv6 is available on selected interface. [Cedric de Saint Martin] * slapgrid: Introduce --only_sr and --only_cp. - only_sr filter and force the run of a single SR, and uses url_md5 (folder_id) - only_cp filter which computer patition, will be runned. it can be a list, splited by comman (slappartX,slappartY ...) [Rafael Monnerat] * slapgrid: Cleanup unused option (--usage-report-periodicity). [Cedric de Saint Martin] * slapgrid: --develop will work also for Computer Partitions. [Cedric de Saint Martin] * slaplib: setConnectionDict won't call Master if parameters haven't changed. [Cedric de Saint Martin] 0.26.2 (2012-07-09) ------------------- * Define UTF-8 encoding in SlapOS Node codebase, as defined in PEP-263. 0.26.1 (2012-07-06) ------------------- * slapgrid-sr: Add --develop option to make it ignore .completed files. * SLAP library: it is now possible to fetch whole dict of connection parameters. * SLAP library: it is now possible to fetch single instance parameter. * SLAP library: change Computer and ComputerPartition behavior to have proper caching of computer partition parameters. 0.26 (2012-07-05) ----------------- * slapformat: no_bridge option becomes 'not create_tap'. create_tap is true by default. So a bridge is used and tap will be created by default. [Cedric de Saint Martin] * Add delay for slapformat. [Cedric Le Ninivin] * If no software_type is given, use default one (i.e fix "error 500" when requesting new instance). [Cedric de Saint Martin] * slapgrid: promise based software release, new api to fetch full computer information from server. [Yingjie Xu] * slapproxy: new api to mock full computer information [Yingjie Xu] * slapgrid: minor fix randomise delay feature. [Yingjie Xu] * slapgrid: optimise slapgrid-cp, run buildout only if there is an update on server side. [Yingjie Xu] * libslap: Allow accessing ServerError. [Vincent Pelletier] 0.25 (2012-05-16) ----------------- * Fix support for no_bridge option in configuration files for some values: no_bridge = false was stated as true. [Cedric de Saint Martin] * Delay a randomized period of time before calling slapgrid. [Yingjie Xu] * slapformat: Don't require tunctl if no_bridge is set [Leonardo Rochael] * slapformat: remove monkey patching when creating address so that it doesn't return false positive. [Cedric de Saint Martin] * Various: clearer error messages. 0.24 (2012-03-29) ----------------- * Handles different errors in a user friendly way [Cedric de Saint Martin] * slapgrid: Supports software destruction. [Łukasz Nowak] * slap: added support to Supply.supply state parameter (available, destroyed) [Łukasz Nowak] 0.23 (2012-02-29) ----------------- * slapgrid : Don't create tarball of sofwtare release when shacache is not configured. [Yingjie Xu] 0.22 (2012-02-09) ----------------- * slapformat : Add no-bridge feature. [Cedric de Saint Martin] * slapgrid : Add binary cache support. [Yingjie Xu] 0.21 (2011-12-23) ----------------- * slap: Add renaming API. [Antoine Catton] 0.20 (2011-11-24) ----------------- * slapgrid: Support service-less parttions. [Antoine Catton] * slapgrid: Avoid gid collision while dropping privileges. [Antoine Catton] * slapgrid: Drop down network usage during usage reporting. [Łukasz Nowak] * general: Add sphinx documentation. [Romain Courteaud] 0.19 (2011-11-07) ----------------- * bang: Executable to be called by being banged computer. [Łukasz Nowak] 0.18 (2011-10-18) ----------------- * Fix 0.17 release: missing change for slap library. [Łukasz Nowak] 0.17 (2011-10-18) ----------------- * slap: Avoid request under the hood. [Łukasz Nowak] * slap: ComputerPartition.bang provided. It allows to update all instances in tree. [Łukasz Nowak] * slap: Computer.bang provided. It allows to bang all instances on computer. [Łukasz Nowak] 0.16 (2011-10-03) ----------------- * slapgrid: Bugfix for slapgrid introduced in 0.15. [Łukasz Nowak] 0.15 (2011-09-27) ----------------- * slapgrid: Sanitize environment variables as early as possible. [Arnaud Fontaine] * slap: Docstring bugfix. [Sebastien Robin] * slap: Make request asynchronous call. [Łukasz Nowak] 0.14 (2011-08-31) ----------------- * slapgrid: Implement SSL based authentication to shadir and shacache. [Łukasz Nowak] * slapgrid, slap: Fix usage report packing list generation. [Nicolas Godbert] 0.13 (2011-08-25) ----------------- * slapgrid: Implement software signing and shacache upload. [Lucas Carvalho] * slap: Support slave instances [Gabriel Monnerat] * slapformat: Generate always address for computer [Łukasz Nowak] * slapgrid: Support promises scripts [Antoine Catton] * general: slapos.core gets tests. [many contributors] 0.12 (2011-07-15) ----------------- * Include modifications that should have been included in 0.11. 0.11 (2011-07-15) ----------------- * Bug fix : slapconsole : shorthand methods request and supply now correctly return an object. [Cedric de Saint Martin] 0.10 (2011-07-13) ----------------- * Fix a bug in slapconsole where request and supply shorthand methods don't accept all needed parameters. [Cedric de Saint Martin] 0.9 (2011-07-11) ---------------- * slapconsole: Simplify usage and use configuration file. You can now just run slapconsole and type things like "request(kvm, 'mykvm')". [Cedric de Saint Martin] * slapformat: Fix issue of bridge not connected with real interface on Linux >= 2.6.39 [Arnaud Fontaine] * slapformat: Allow to have IPv6 only interface, with bridge still supporting local IPv4 stack. [Łukasz Nowak] 0.8 (2011-06-27) ---------------- * slapgrid: Bugfix for temporary extends cache permissions. [Łukasz Nowak] 0.7 (2011-06-27) ---------------- * slapgrid: Fallback to buildout in own search path. [Łukasz Nowak] 0.6 (2011-06-27) ---------------- * slap: Fix bug: state shall be XML encapsulated. [Łukasz Nowak] 0.5 (2011-06-24) ---------------- * slapgrid: Use temporary extends-cache directory in order to make faster remote profile refresh. [Łukasz Nowak] 0.4 (2011-06-24) ---------------- * general: Polish requirement versions. [Arnaud Fontaine] * general: Remove libnetworkcache. [Lucas Carvalho] * slap: Remove not needed method from interface. [Romain Courteaud] * slap: state parameter is accepted and transmitted to SlapOS master [Łukasz Nowak] * slapformat: Implement dry run. [Vincent Pelletier] * slapgrid: Allow to select any buildout binary used to bootstrap environment. [Łukasz Nowak] 0.3 (2011-06-14) ---------------- * slap: Implement SLA by filter_kw in OpenOrder.request. [Łukasz Nowak] * slap: Timeout network operations. [Łukasz Nowak] * slapformat: Make slapsoft and slapuser* system users. [Kazuhiko Shiozaki] * slapgrid: Add more tolerance with supervisord. [Łukasz Nowak] 0.2 (2011-06-01) ---------------- * Include required files in distribution [Łukasz Nowak] 0.1 (2011-05-27) ---------------- * Merged slapos.slap, slapos.tool.console, slapos.tool.format, slapos.tool.grid, slapos.tool.libnetworkcache and slapos.tool.proxy into one package: slapos.core slapos.core-0.35.1/MANIFEST.in0000644000076600007660000000013612035516222023652 0ustar cedricdesaintmartincedricdesaintmartin00000000000000include CHANGES.txt include slapos/proxy/schema.sql recursive-include slapos *.in *.txt *.xsd slapos.core-0.35.1/PKG-INFO0000644000076600007660000007304612110425554023224 0ustar cedricdesaintmartincedricdesaintmartin00000000000000Metadata-Version: 1.0 Name: slapos.core Version: 0.35.1 Summary: SlapOS core. Home-page: http://www.slapos.org Author: VIFIB Author-email: UNKNOWN License: GPLv3 Description: slapos.core =========== The core of SlapOS. Contains the SLAP library, and the slapgrid, slapformat, slapproxy tools. For more information, see http://www.slapos.org. Changes ======= 0.35.1 (2012-02-18) ------------------- New features: * Add ComputerPartition._instance_guid getter in SLAP library. [Cedric de Saint Martin] * Add ComputerPartition._instance_guid support in slapproxy. [Cedric de Saint Martin] Fixes: * Fix link existence check when deploying instance if SR is not correctly installed. This fixes a misleading error. [Cedric de Saint Martin] * Improve message shown to user when requesting. [Cedric de Saint Martin] * Raise NotReady when _requested_state doesn't exist when trying to fetch it from getter. [Cedric de Saint Martin] 0.35 (2012-02-08) ----------------- * slapos: display version number with help. [Marco Mariani] * slapformat: backup slapos.xml to a zip archive at every change. [Marco Mariani] * slapformat: Don't check validity of ipv4 when trying to add address that already exists. [Cedric de Saint Martin] * slapgrid: create and run $MD5/buildout.cfg for eaiser debugging. [Marco Mariani] * slapgrid: keep running if cp.error() or sr.error() have issues (fixes 20130119-744D94). [Marco Mariani] * slapgrid does not crash when there are no certificates (fixes #20130121-136C24). [Marco Mariani] * Add slapproxy-query command. [Marco Mariani] * Other minor typo / output fixes. 0.34 (2013-01-23) ----------------- * networkcache: only match major release number in Debian, fixed platform detection for Ubuntu. [Marco Mariani] * symlink to software_release in each partition. [Marco Mariani] * slapos client: Properly expand "~" when giving configuration file location. [Cedric de Saint Martin] * slapgrid: stop instances that should be stopped even if buildout and/or reporting failed. [Cedric de Saint Martin] * slapgrid: Don't periodically force-process a stopped instance. [Cedric de Saint Martin] * slapgrid: Handle pid files of slapgrid launched through different entry points. [Cedric de Saint Martin] * Watchdog: Bang is called with correct instance certificates. [Cedric Le Ninivin] * Watchdog: Fix watchdog call. [Cedric le Ninivin] * Add a symlink of the used software release in each partitions. [Marco Mariani] * slapformat is verbose by default. [Cedric de Saint Martin] * slapproxy: Filter by instance_guid, allow computer partition renames and change of software_type and requested_state. [Marco Mariani] * slapproxy: Stop instance even if buildout/reporting is wrong. [Cedric de Saint Martin] * slapproxy: implement softwareInstanceRename method. [Marco Mariani] * slapproxy: alllow requests to software_type. [Marco Mariani] * Many other minor fixes. See git diff for details. 0.33.1 (2012-11-05) ------------------- * Fix "slapos console" argument parsing. [Cedric de Saint Martin] 0.33 (2012-11-02) ----------------- * Continue to improve new entry points. The following are now functional: - slapos node format - slapos node start/stop/restart/tail - slapos node supervisord/supervisorctl - slapos node supply and add basic usage. [Cedric de Saint Martin] * Add support for "SLAPOS_CONFIGURATION" and SLAPOS_CLIENT_CONFIGURATION environment variables. (commit c72a53b1) [Cédric de Saint Martin] * --only_sr also accepts plain text URIs. [Marco Mariani] 0.32.3 (2012-10-15) ------------------- * slapgrid: Adopt new return value strategy (0=OK, 1=failed, 2=promise failed) (commit 5d4e1522). [Cedric de Saint Martin] * slaplib: add requestComputer (commits 6cbe82e0, aafb86eb). [Łukasz Nowak] * slapgrid: Add stopasgroup and killasgroup to supervisor (commit 36e0ccc0). [Cedric de Saint Martin] * slapproxy: don't start in debug mode by default (commit e32259c8). [Cédric Le Ninivin * SlapObject: ALWAYS remove tmpdir (commit a652a610). [Cedric de Saint Martin] 0.32.2 (2012-10-11) ------------------- * slapgrid: Remove default delay, now that SlapOS Master is Fast as Light (tm). (commit 03a85d6b8) [Cedric de Saint Martin] * Fix watchdog entry point name, introduced in v0.31. (commit a8651ba12) [Cedric de Saint Martin] * slapgrid: Better filter of instances, won't process false positives anymore (hopefully). (commit ce0a73b41) [Cedric de Saint Martin] * Various output improvements. [Cedric de Saint Martin] 0.32.1 (2012-10-09) ------------------- * slapgrid: Make sure error logs are sent to SlapOS master. Finish implementation began in 0.32. [Cedric de Saint Martin] * slapgrid: Fix Usage Report in case of not empty partition with no SR. [Cedric de Saint Martin] 0.32 (2012-10-04) ----------------- * Introduce new, simpler "slapos" entry point. See documentation for more informations. Note: some functionnalities of this new entry point don't work yet or is not as simple as it should be. [Cedric de Saint Martin, Cedric Le Ninivin] * Revamped "slapos request" to work like described in documentation. [Cédric Le Ninivin, Cédric de Saint Martin] * Rewrote slapgrid logger to always log into stdout. (commits a4d277c881, 5440626dea)[Cédric de Saint Martin] 0.31.2 (2012-10-02) ------------------- * Update slapproxy behavior: when instance already exist, only update partition_parameter_kw. (commit 317d5c8e0aee) [Cedric de Saint Martin] 0.31.1 (2012-10-02) ------------------- * Fixed Watchdog call in slapgrid. [Cédric Le Ninivin] 0.31 (2012-10-02) ------------------- * Added slapos-watchdog to bang exited and failing serices in instance in supervisord. (commits 16b2e8b8, 1dade5cd7) [Cédric Le Ninivin] * Add safety checks before calling SlapOS Master if mandatory instance members of SLAP classes are not properly set. Will result in less calls to SlapOS Master in dirty cases. (commits 5097e87c9763, 5fad6316a0f6d, f2cd014ea8aa) [Cedric de Saint Martin] * Add "periodicty" functionnality support for instances: if an instance has not been processed by slapgrid after defined time, process it. (commits 7609fc7a3d, 56e1c7bfbd) [Cedric Le Ninivin] * slapproxy: Various improvements in slave support (commits 96c6b78b67, bcac5a397d, fbb680f53b)[Cedric Le Ninivin] * slapgrid: bulletproof slapgrid-cp: in case one instance is bad, still processes all other ones. (commits bac94cdb56, 77bc6c75b3d, bd68b88cc3) [Cedric de Saint Martin] * Add support for "upload to binary cache" URL blacklist [Cedric de Saint Martin] * Request on proxy are identified by requester and name (commit 0c739c3) [Cedric Le Ninivin] 0.30 (2012-09-19) ----------------- * Add initial "slave instances" support in slapproxy. [Cedric Le Ninivin] * slapgrid-ur fix: check for partition informations only if we have to destroy it. [Cedric de Saint Martin] 0.29 (2012-09-18) ----------------- * buildout: Migrate slap_connection magic instance profile part to slap-connection, and use variables names separated with '-'. [Cedric de Saint Martin] * slapgrid: Add support for instance.cfg instance profiles [Cedric de Saint Martin] * slapgrid-ur: much less calls to master. [Cedric de Saint Martin] 0.28.9 (2012-09-18) ------------------- * slapgrid: Don't process not updated partitions (regression introduced in 0.28.7). [Cedric de Saint Martin] 0.28.8 (2012-09-18) ------------------- * slapgrid: Don't process free partitions (regression introduced in 0.28.7). [Cedric de Saint Martin] 0.28.7 (2012-09-14) ------------------- * slapgrid: --maximal_delay reappeared to be used in special cases. [Cedric de Saint Martin] 0.28.6 (2012-09-10) ------------------- * register now use slapos.cfg.example from master. [Cédric Le Ninivin] 0.28.5 (2012-08-23) ------------------- * Updated slapos.cfg for register [Cédric Le Ninivin] 0.28.4 (2012-08-22) ------------------- * Fixed egg building. 0.28.3 (2012-08-22) ------------------- * Avoid artificial tap creation on system check. [Łukasz Nowak] 0.28.2 (2012-08-17) ------------------- * Resolved path problem in register [Cédric Le Ninivin] 0.28.1 (2012-08-17) ------------------- * Resolved critical naming conflict 0.28 (2012-08-17) ----------------- * Introduce "slapos node register" command, that will register computer to SlapOS Master (vifib.net by default) for you. [Cédric Le Ninivin] * Set .timestamp in partitions ONLY after slapgrid thinks it's okay (promises, ...). [Cedric de Saint Martin] * slapgrid-ur: when destroying (not reporting), only care about instances to destroy, completely ignore others. [Cedric de Saint Martin] 0.27 (2012-08-08) ----------------- * slapformat: Raise correct error when no IPv6 is available on selected interface. [Cedric de Saint Martin] * slapgrid: Introduce --only_sr and --only_cp. - only_sr filter and force the run of a single SR, and uses url_md5 (folder_id) - only_cp filter which computer patition, will be runned. it can be a list, splited by comman (slappartX,slappartY ...) [Rafael Monnerat] * slapgrid: Cleanup unused option (--usage-report-periodicity). [Cedric de Saint Martin] * slapgrid: --develop will work also for Computer Partitions. [Cedric de Saint Martin] * slaplib: setConnectionDict won't call Master if parameters haven't changed. [Cedric de Saint Martin] 0.26.2 (2012-07-09) ------------------- * Define UTF-8 encoding in SlapOS Node codebase, as defined in PEP-263. 0.26.1 (2012-07-06) ------------------- * slapgrid-sr: Add --develop option to make it ignore .completed files. * SLAP library: it is now possible to fetch whole dict of connection parameters. * SLAP library: it is now possible to fetch single instance parameter. * SLAP library: change Computer and ComputerPartition behavior to have proper caching of computer partition parameters. 0.26 (2012-07-05) ----------------- * slapformat: no_bridge option becomes 'not create_tap'. create_tap is true by default. So a bridge is used and tap will be created by default. [Cedric de Saint Martin] * Add delay for slapformat. [Cedric Le Ninivin] * If no software_type is given, use default one (i.e fix "error 500" when requesting new instance). [Cedric de Saint Martin] * slapgrid: promise based software release, new api to fetch full computer information from server. [Yingjie Xu] * slapproxy: new api to mock full computer information [Yingjie Xu] * slapgrid: minor fix randomise delay feature. [Yingjie Xu] * slapgrid: optimise slapgrid-cp, run buildout only if there is an update on server side. [Yingjie Xu] * libslap: Allow accessing ServerError. [Vincent Pelletier] 0.25 (2012-05-16) ----------------- * Fix support for no_bridge option in configuration files for some values: no_bridge = false was stated as true. [Cedric de Saint Martin] * Delay a randomized period of time before calling slapgrid. [Yingjie Xu] * slapformat: Don't require tunctl if no_bridge is set [Leonardo Rochael] * slapformat: remove monkey patching when creating address so that it doesn't return false positive. [Cedric de Saint Martin] * Various: clearer error messages. 0.24 (2012-03-29) ----------------- * Handles different errors in a user friendly way [Cedric de Saint Martin] * slapgrid: Supports software destruction. [Łukasz Nowak] * slap: added support to Supply.supply state parameter (available, destroyed) [Łukasz Nowak] 0.23 (2012-02-29) ----------------- * slapgrid : Don't create tarball of sofwtare release when shacache is not configured. [Yingjie Xu] 0.22 (2012-02-09) ----------------- * slapformat : Add no-bridge feature. [Cedric de Saint Martin] * slapgrid : Add binary cache support. [Yingjie Xu] 0.21 (2011-12-23) ----------------- * slap: Add renaming API. [Antoine Catton] 0.20 (2011-11-24) ----------------- * slapgrid: Support service-less parttions. [Antoine Catton] * slapgrid: Avoid gid collision while dropping privileges. [Antoine Catton] * slapgrid: Drop down network usage during usage reporting. [Łukasz Nowak] * general: Add sphinx documentation. [Romain Courteaud] 0.19 (2011-11-07) ----------------- * bang: Executable to be called by being banged computer. [Łukasz Nowak] 0.18 (2011-10-18) ----------------- * Fix 0.17 release: missing change for slap library. [Łukasz Nowak] 0.17 (2011-10-18) ----------------- * slap: Avoid request under the hood. [Łukasz Nowak] * slap: ComputerPartition.bang provided. It allows to update all instances in tree. [Łukasz Nowak] * slap: Computer.bang provided. It allows to bang all instances on computer. [Łukasz Nowak] 0.16 (2011-10-03) ----------------- * slapgrid: Bugfix for slapgrid introduced in 0.15. [Łukasz Nowak] 0.15 (2011-09-27) ----------------- * slapgrid: Sanitize environment variables as early as possible. [Arnaud Fontaine] * slap: Docstring bugfix. [Sebastien Robin] * slap: Make request asynchronous call. [Łukasz Nowak] 0.14 (2011-08-31) ----------------- * slapgrid: Implement SSL based authentication to shadir and shacache. [Łukasz Nowak] * slapgrid, slap: Fix usage report packing list generation. [Nicolas Godbert] 0.13 (2011-08-25) ----------------- * slapgrid: Implement software signing and shacache upload. [Lucas Carvalho] * slap: Support slave instances [Gabriel Monnerat] * slapformat: Generate always address for computer [Łukasz Nowak] * slapgrid: Support promises scripts [Antoine Catton] * general: slapos.core gets tests. [many contributors] 0.12 (2011-07-15) ----------------- * Include modifications that should have been included in 0.11. 0.11 (2011-07-15) ----------------- * Bug fix : slapconsole : shorthand methods request and supply now correctly return an object. [Cedric de Saint Martin] 0.10 (2011-07-13) ----------------- * Fix a bug in slapconsole where request and supply shorthand methods don't accept all needed parameters. [Cedric de Saint Martin] 0.9 (2011-07-11) ---------------- * slapconsole: Simplify usage and use configuration file. You can now just run slapconsole and type things like "request(kvm, 'mykvm')". [Cedric de Saint Martin] * slapformat: Fix issue of bridge not connected with real interface on Linux >= 2.6.39 [Arnaud Fontaine] * slapformat: Allow to have IPv6 only interface, with bridge still supporting local IPv4 stack. [Łukasz Nowak] 0.8 (2011-06-27) ---------------- * slapgrid: Bugfix for temporary extends cache permissions. [Łukasz Nowak] 0.7 (2011-06-27) ---------------- * slapgrid: Fallback to buildout in own search path. [Łukasz Nowak] 0.6 (2011-06-27) ---------------- * slap: Fix bug: state shall be XML encapsulated. [Łukasz Nowak] 0.5 (2011-06-24) ---------------- * slapgrid: Use temporary extends-cache directory in order to make faster remote profile refresh. [Łukasz Nowak] 0.4 (2011-06-24) ---------------- * general: Polish requirement versions. [Arnaud Fontaine] * general: Remove libnetworkcache. [Lucas Carvalho] * slap: Remove not needed method from interface. [Romain Courteaud] * slap: state parameter is accepted and transmitted to SlapOS master [Łukasz Nowak] * slapformat: Implement dry run. [Vincent Pelletier] * slapgrid: Allow to select any buildout binary used to bootstrap environment. [Łukasz Nowak] 0.3 (2011-06-14) ---------------- * slap: Implement SLA by filter_kw in OpenOrder.request. [Łukasz Nowak] * slap: Timeout network operations. [Łukasz Nowak] * slapformat: Make slapsoft and slapuser* system users. [Kazuhiko Shiozaki] * slapgrid: Add more tolerance with supervisord. [Łukasz Nowak] 0.2 (2011-06-01) ---------------- * Include required files in distribution [Łukasz Nowak] 0.1 (2011-05-27) ---------------- * Merged slapos.slap, slapos.tool.console, slapos.tool.format, slapos.tool.grid, slapos.tool.libnetworkcache and slapos.tool.proxy into one package: slapos.core console ------- The slapconsole tool allows to interact with a SlapOS Master throught the SLAP library. For more information about SlapOS or slapconsole usages, please go to http://community.slapos.org. The slapconsole tool is only a bare Python console with several global variables defined and initialized. Initialization and configuration file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Slapconsole allows to automatically connect to a Master using URL and SSL certificate from given slapos.cfg. Certificate has to be *USER* certificate, manually obtained from SlapOS master web interface. Slapconsole tools reads the given slapos.cfg configuration file and use the following informations : * Master URL is read from [slapos] in the "master_url" parameter. * SSL Certificate is read from [slapconsole] in the "cert_file" parameter. * SSL Key is read from [slapconsole] in the "key_file" parameter. See slapos.cfg.example for examples. Global functions ~~~~~~~~~~~~~~~~ * request() is a shorthand for slap.registerOpenOrder().request() allowing to request instances. * supply() is a shorthand for slap.registerSupply().supply() allowing to request software installation. For more information about those methods, please read the SLAP library documentation. Global aliases ~~~~~~~~~~~~~~ "software_list" is a list containing all the Software Release URLs defined in client slapos.cfg configuration file. Also, each Software Release defined in this configuration file is translated into a global variable to ease the request of those Sofware Releases. This allows to request instances in a few words, i.e:: request("http://www.url.com/path/to/kvm/software.cfg", "mykvm") can be simplified into :: request(kvm, "mykvm") If the slapos.cfg file contains :: alias = kvm http://www.url.com/path/to/kvm/software.cfg Global objects ~~~~~~~~~~~~~~ "slap" is an instance of the SLAP library. It is only used for advanced usages. "slap" instance is obtained by doing :: slap = slapos.slap.slap() slap.initializeConnection(config.master_url, key_file=config.key_file, cert_file=config.cert_file) Examples ~~~~~~~~ :: >>> # Request instance >>> request(kvm, "myuniquekvm") >>> # Request instance on specific computer >>> request(kvm, "myotheruniquekvm", filter_kw={ "computer_guid": "COMP-12345" }) >>> # Request instance, specifying parameters (here nbd_ip and nbd_port) >>> request(kvm, "mythirduniquekvm", partition_parameter_kw={"nbd_ip":"2a01:e35:2e27:460:e2cb:4eff:fed9:48dc", "nbd_port":"1024"}) >>> # Request software installation on owned computer >>> supply(kvm, "mycomputer") >>> # Fetch existing instance status >>> request(kvm, "myuniquekvm").getState() >>> # Fetch instance information on already launched instance >>> request(kvm, "myuniquekvm").getConnectionParameter("url") format ====== slapformat is an application to prepare SlapOS ready node (machine). It "formats" the machine by: - creating users and groups - creating bridge interface - creating needed tap interfaces - creating needed directories with proper ownership and permissions In the end special report is generated and information are posted to configured SlapOS server. This program shall be only run by root. Requirements ------------ Linux with IPv6, bridging and tap interface support. Binaries: * brctl * groupadd * ip * tunctl * useradd grid ==== slapgrid is a client of SLAPos. SLAPos provides support for deploying a SaaS system in a minute. Slapgrid allows you to easily deploy instances of softwares based on buildout profiles. For more informations about SLAP and SLAPos, please see the SLAP documentation. Requirements ------------ A working SLAP server with informations about your computer, in order to retrieve them. As Vifib servers use IPv6 only, we strongly recommend an IPv6 enabled UNIX box. For the same reasons, Python >= 2.6 with development headers is also strongly recommended (IPv6 support is not complete in previous releases). For now, gcc and glibc development headers are required to build most software releases. Concepts -------- Here are the fundamental concepts of slapgrid : A Software Release (SR) is just a software. A Computer Partition (CP) is an instance of a Software Release. Imagine you want to install with slapgrid some software and run it. You will have to install the software as a Software Release, and then instantiate it, i.e configuring it for your needs, as a Computer Partition. How it works ------------ When run, slapgrid will authenticate to the SLAP library with a computer_id and fetch the list of Software Releases to install or remove and Computer Partitions to start or stop. Then, it will process each Software Release, and each Computer Partition. It will also periodically send to SLAP the usage report of each Computer Partition. Installation ------------ With easy_install:: $ easy_install slapgrid slapgrid needs several directories to be created and configured before being able to run : a software releases directory, and an instances directory with configured computer partition directory(ies). You should create for each Computer Partition directory created a specific user and associate it with its Computer Partition directory. Each Computer Partition directory should belongs to this specific user, with permissions of 0750. Usage ----- slapgrid needs several informations in order to run. You can specify them by adding arguments to the slapgrid command line, or by putting then in a configuration file. Beware : you need a valid computer resource on server side. Examples -------- simple example : Just run slapgrid: $ slapgrid --instance-root /path/to/instance/root --software-root /path/to/software_root --master-url https://some.server/some.resource --computer-id my.computer.id configuration file example:: [slapgrid] instance_root = /path/to/instance/root software_root = /path/to/software/root master_url = https://slapos.server/slap_service computer_id = my.computer.id then run slapgrid:: $ slapgrid --configuration-file = path/to/configuration/file proxy ===== Implement minimalist SlapOS Master server without any security, designed to work only from localhost with one SlapOS Node (a.k.a Computer). It implements (or should implement) the SLAP API, as currently implemented in the SlapOS Master (see slaptool.py in Master). The only behavioral difference from the SlapOS Master is: When the proxy doesn't find any free partition (and/or in case of slave instance, any compatible master instance), it will throw a NotFoundError (404). slap ==== Simple Language for Accounting and Provisioning python library. Developer note - python version ------------------------------- This library is used on client (slapgrid) and server side. Server is using python2.4 and client is using python2.6 Having this in mind, code of this library *have* to work on python2.4 How it works ------------ The SLAP main server which is in charge of service coordination receives from participating servers the number of computer paritions which are available, the type of resource which a party is ready provide, and request from parties for resources which are needed. Each participating server is identified by a unique ID and runs a slap-server daemon. This daemon collects from the main server the installation tasks and does the installation of resources, then notifies the main server of completion whenever a resource is configured, installed and available. The data structure on the main server is the following: A - Action: an action which can happen to provide a resource or account its usage CP - Computer Partition: provides a URL to Access a Cloud Resource RI - Resource Item: describes a resource CI - Contract Item: describes the contract to attach the DL to (This is unclear still) R - Resource: describes a type of cloud resource (ex. MySQL Table) is published on slapgrid.org DL - Delivery Line: Describes an action happening on a resource item on a computer partition D - Delivery: groups multiple Delivery Lines Keywords: slapos core Platform: UNKNOWN Classifier: Programming Language :: Python slapos.core-0.35.1/README.txt0000644000076600007660000000025012033041765023612 0ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core =========== The core of SlapOS. Contains the SLAP library, and the slapgrid, slapformat, slapproxy tools. For more information, see http://www.slapos.org. slapos.core-0.35.1/setup.cfg0000644000076600007660000000032212110425554023733 0ustar cedricdesaintmartincedricdesaintmartin00000000000000[build_sphinx] source-dir = documentation/source build-dir = documentation/build all_files = 1 [upload_sphinx] upload-dir = documentation/build/html [egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 slapos.core-0.35.1/setup.py0000644000076600007660000000522012105143232023620 0ustar cedricdesaintmartincedricdesaintmartin00000000000000from setuptools import setup, find_packages import glob import os from slapos.version import version name = 'slapos.core' long_description = open("README.txt").read() + "\n" + \ open("CHANGES.txt").read() + "\n" for f in sorted(glob.glob(os.path.join('slapos', 'README.*.txt'))): long_description += '\n' + open(f).read() + '\n' additional_install_requires = [] # Even if argparse is available in python2.7, some python2.7 installations # do not have it, so checking python version is dangerous try: import argparse except ImportError: additional_install_requires.append('argparse') setup(name=name, version=version, description="SlapOS core.", long_description=long_description, classifiers=[ "Programming Language :: Python", ], keywords='slapos core', license='GPLv3', url='http://www.slapos.org', author='VIFIB', namespace_packages=['slapos'], packages=find_packages(), include_package_data=True, install_requires=[ 'Flask', # used by proxy 'lxml', # needed to play with XML trees 'netaddr>=0.7.5', # to play safely with IPv6 prefixes 'netifaces', # to fetch information about network devices 'pyflakes', # For testing purposes 'setuptools', # namespaces 'supervisor', # slapgrid uses supervisor to manage processes 'unittest2', 'xml_marshaller>=0.9.3', # to unmarshall/marshall python objects to/from # XML 'zope.interface', # slap library implementes interfaces 'zc.buildout', ] + additional_install_requires, zip_safe=False, # proxy depends on Flask, which has issues with # accessing templates entry_points={ 'console_scripts': [ # One entry point to control them all 'slapos = slapos.entry:main', 'slapos-watchdog = slapos.grid.watchdog:main', 'slapproxy = slapos.proxy:main', 'slapproxy-query = slapos.proxy.query:main', # Deprecated entry points 'slapconsole = slapos.client:slapconsole', 'slapformat = slapos.format:main', 'slapgrid = slapos.grid.slapgrid:run', 'slapgrid-sr = slapos.grid.slapgrid:runSoftwareRelease', 'slapgrid-cp = slapos.grid.slapgrid:runComputerPartition', 'slapgrid-ur = slapos.grid.slapgrid:runUsageReport', 'slapgrid-supervisorctl = slapos.grid.svcbackend:supervisorctl', 'slapgrid-supervisord = slapos.grid.svcbackend:supervisord', 'bang = slapos.bang:main', ] }, test_suite="slapos.tests", ) slapos.core-0.35.1/slapos/0000755000076600007660000000000012110425554023416 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/__init__.py0000644000076600007660000000036412030324717025532 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# See http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages try: __import__('pkg_resources').declare_namespace(__name__) except ImportError: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) slapos.core-0.35.1/slapos/bang.py0000644000076600007660000000511412035516223024700 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import slapos.slap.slap import argparse import ConfigParser def main(*args): parser = argparse.ArgumentParser() parser.add_argument("-m", "--message", default='', help="Message for bang.") parser.add_argument("configuration_file", nargs=1, type=argparse.FileType(), help="SlapOS configuration file.") if len(args) == 0: argument = parser.parse_args() else: argument = parser.parse_args(list(args)) configuration_file = argument.configuration_file[0] message = argument.message # Loads config (if config specified) configuration = ConfigParser.SafeConfigParser() configuration.readfp(configuration_file) computer_id = configuration.get('slapos', 'computer_id') master_url = configuration.get('slapos', 'master_url') if configuration.has_option('slapos', 'key_file'): key_file = configuration.get('slapos', 'key_file') else: key_file = None if configuration.has_option('slapos', 'cert_file'): cert_file = configuration.get('slapos', 'cert_file') else: cert_file = None slap = slapos.slap.slap() slap.initializeConnection(master_url, key_file=key_file, cert_file=cert_file) computer = slap.registerComputer(computer_id) print 'Banging to %r' % master_url computer.bang(message) print 'Bang with message %r' % message slapos.core-0.35.1/slapos/cache.py0000644000076600007660000000413112100006516025022 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- import ast import argparse import ConfigParser import hashlib import json import re import sys import urllib2 from slapos.grid import networkcache from slapos.grid.distribution import patched_linux_distribution def maybe_md5(s): return re.match('[0-9a-f]{32}', s) def cache_lookup(): parser = argparse.ArgumentParser() parser.add_argument("configuration_file", help="SlapOS configuration file") parser.add_argument("software_url", help="Your software url or MD5 hash") args = parser.parse_args() configuration_parser = ConfigParser.SafeConfigParser() configuration_parser.read(args.configuration_file) configuration_parser.items('networkcache') cache_dir = configuration_parser.get('networkcache', 'download-binary-dir-url') if maybe_md5(args.software_url): md5 = args.software_url else: md5 = hashlib.md5(args.software_url).hexdigest() try: response = urllib2.urlopen('%s/%s' % (cache_dir, md5)) except urllib2.HTTPError as e: if e.code == 404: print 'Object not in cache: %s' % args.software_url else: print 'Error during cache lookup: %s (%s)' % (e.code, e.reason) sys.exit(10) entries = json.loads(response.read()) linux_distribution = patched_linux_distribution() header_printed = False ostable = [] for entry in entries: meta = json.loads(entry[0]) os = ast.literal_eval(meta['os']) if not header_printed: print 'Software URL: %s' % meta['software_url'] print 'MD5: %s' % md5 print '-------------' print 'Available for: ' print 'distribution | version | id | compatible?' print '-----------------+--------------+----------------+-------------' header_printed = True ostable.append(os) ostable.sort() for os in ostable: compatible = 'yes' if networkcache.os_matches(os, linux_distribution) else 'no' print '%-16s | %12s | %s | %s' % (os[0], os[1], os[2].center(14), compatible) slapos.core-0.35.1/slapos/client.py0000644000076600007660000002463412107454171025262 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import argparse import ConfigParser import pprint from optparse import OptionParser, Option import os from slapos.slap import ResourceNotReady import slapos.slap.slap import sys class Parser(OptionParser): """ Parse all arguments. """ def __init__(self, usage=None, version=None): """ Initialize all options possibles. """ OptionParser.__init__(self, usage=usage, version=version, option_list=[ Option("-u", "--master_url", default=None, action="store", help="Url of SlapOS Master to use."), Option("-k", "--key_file", action="store", help="SSL Authorisation key file."), Option("-c", "--cert_file", action="store", help="SSL Authorisation certificate file.") ]) def check_args(self): """ Check arguments """ (options, args) = self.parse_args() if len(args) == 0: self.error("Incorrect number of arguments") elif not os.path.isfile(args[0]): self.error("%s: Not found or not a regular file." % args[0]) # Return options and only first element of args since there is only one. return options, args[0] def argToDict(element): """ convert a table of string 'key=value' to dict """ if element is not None: element_dict = dict([arg.split('=') for arg in element]) return element_dict def check_request_args(): """ Parser for request """ parser = argparse.ArgumentParser() parser.add_argument("configuration_file", help="SlapOS configuration file.") parser.add_argument("reference", help="Your instance reference") parser.add_argument("software_url", help="Your software url") parser.add_argument("--node", nargs = '*', help = "Node request option " "'option1=value1 option2=value2'") parser.add_argument("--type", type = str, help = "Define software type to be requested") parser.add_argument("--slave", action = "store_true", default=False, help = "Ask for a slave instance") parser.add_argument("--configuration", nargs = '*', help = "Give your configuration " "'option1=value1 option2=value2'") args = parser.parse_args() # Convert to dict if args.configuration is not None: args.configuration = argToDict(args.configuration) if args.node is not None: args.node = argToDict(args.node) return args class Config: def __init__(self, option_dict, configuration_file_path=None): """ Set options given by parameters. """ # Set options parameters for option, value in option_dict.__dict__.items(): setattr(self, option, value) # Load configuration file configuration_parser = ConfigParser.SafeConfigParser() if configuration_file_path: configuration_file_path = os.path.expanduser(configuration_file_path) if not os.path.isfile(configuration_file_path): raise OSError('Specified configuration file %s does not exist.' ' Exiting.' % configuration_file_path) configuration_parser.read(configuration_file_path) # Merges the arguments and configuration try: configuration_dict = dict(configuration_parser.items('slapconsole')) except ConfigParser.NoSectionError: pass else: for key in configuration_dict: if not getattr(self, key, None): setattr(self, key, configuration_dict[key]) configuration_dict = dict(configuration_parser.items('slapos')) master_url = configuration_dict.get('master_url', None) # Backward compatibility, if no key and certificate given in option # take one from slapos configuration if not getattr(self, 'key_file', None) and \ not getattr(self, 'cert_file', None): self.key_file = configuration_dict.get('key_file') self.cert_file = configuration_dict.get('cert_file') if not master_url: raise ValueError("No option 'master_url'") elif master_url.startswith('https') and \ self.key_file is None and \ self.cert_file is None: raise ValueError("No option 'key_file' and/or 'cert_file'") else: setattr(self, 'master_url', master_url) def init(config): """Initialize Slap instance, connect to server and create aliases to common software releases""" slap = slapos.slap.slap() slap.initializeConnection(config.master_url, key_file=config.key_file, cert_file=config.cert_file) local = globals().copy() local['slap'] = slap # Create aliases as global variables try: alias = config.alias.split('\n') except AttributeError: alias = [] software_list = [] for software in alias: if software is not '': name, url = software.split(' ') software_list.append(name) local[name] = url # Create global variable too see available aliases local['software_list'] = software_list # Create global shortcut functions to request instance and software def shorthandRequest(*args, **kwargs): return slap.registerOpenOrder().request(*args, **kwargs) def shorthandSupply(*args, **kwargs): return slap.registerSupply().supply(*args, **kwargs) local['request'] = shorthandRequest local['supply'] = shorthandSupply return local def request(): """Run when invoking slapos request. Request an instance.""" # Parse arguments and inititate needed parameters # XXX-Cedric: move argument parsing to main entry point options = check_request_args() config = Config(options, options.configuration_file) local = init(config) # Request instance print("Requesting %s..." % config.reference) if config.software_url in local: config.software_url = local[config.software_url] try: partition = local['slap'].registerOpenOrder().request( software_release = config.software_url, partition_reference = config.reference, partition_parameter_kw = config.configuration, software_type = config.type, filter_kw = config.node, shared = config.slave ) print "Instance requested.\nState is : %s." % partition.getState() print "Connection parameters of instance are:" pprint.pprint(partition.getConnectionParameterDict()) print "You can rerun command to get up-to-date informations." except ResourceNotReady: print("Instance requested. Master is provisionning it. Please rerun in a " "couple of minutes to get connection informations.") exit(2) def _supply(software_url, computer_id, local, remove=False): """ Request installation of Software Release 'software_url' on computer 'computer_id'. if destroy argument is True, request deletion of Software Release. """ # XXX-Cedric Implement software_group support # XXX-Cedric Implement computer_group support if not remove: state = 'available' print 'Requesting installation of %s Software Release...' % software_url else: state = 'destroyed' print 'Requesting deletion of %s Software Release...' % software_url if software_url in local: software_url = local[software_url] local['slap'].registerSupply().supply( software_release=software_url, computer_guid=computer_id, state=state, ) print 'Done.' def supply(): """ Run when invoking slapos supply. Mostly argument parsing. """ # XXX-Cedric: move argument parsing to main entry point parser = argparse.ArgumentParser() parser.add_argument("configuration_file", help="SlapOS configuration file") parser.add_argument("software_url", help="Your software url") parser.add_argument("node", help="Target node") args = parser.parse_args() config = Config(args, args.configuration_file) _supply(args.software_url, args.node, init(config)) def remove(): """ Run when invoking slapos remove. Mostly argument parsing. """ # XXX-Cedric: move argument parsing to main entry point parser = argparse.ArgumentParser() parser.add_argument("configuration_file", help="SlapOS configuration file.") parser.add_argument("software_url", help="Your software url") parser.add_argument("node", help="Target node") args = parser.parse_args() config = Config(args, args.configuration_file) _supply(args.software_url, args.node, init(config), remove=True) def slapconsole(): """Ran when invoking slapconsole""" # Parse arguments usage = """usage: %s [options] CONFIGURATION_FILE slapconsole allows you interact with slap API. You can play with the global "slap" object and with the global "request" method. examples : >>> # Request instance >>> request(kvm, "myuniquekvm") >>> # Request software installation on owned computer >>> supply(kvm, "mycomputer") >>> # Fetch instance informations on already launched instance >>> request(kvm, "myuniquekvm").getConnectionParameter("url")""" % sys.argv[0] config = Config(*Parser(usage=usage).check_args()) local = init(config) __import__("code").interact(banner="", local=local) slapos.core-0.35.1/slapos/entry.py0000644000076600007660000001772312105143232025135 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2012 Vifib SARL and Contributors. All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import argparse import ConfigParser import os import sys from slapos.bang import main as bang from slapos.client import slapconsole as console from slapos.client import request as request from slapos.client import remove as remove from slapos.client import supply as supply from slapos.format import main as format from slapos.cache import cache_lookup from slapos.grid.slapgrid import runComputerPartition as instance from slapos.grid.slapgrid import runSoftwareRelease as software from slapos.grid.slapgrid import runUsageReport as report from slapos.grid.svcbackend import supervisord from slapos.grid.svcbackend import supervisorctl from slapos.register.register import main as register from slapos.version import version # Note: this whole file is a hack. We should better try dedicated library # like https://github.com/dhellmann/cliff or https://github.com/docopt/docopt. GLOBAL_SLAPOS_CONFIGURATION = os.environ.get( 'SLAPOS_CONFIGURATION', '/etc/opt/slapos/slapos.cfg') USER_SLAPOS_CONFIGURATION = os.environ.get( 'SLAPOS_CLIENT_CONFIGURATION', os.environ.get('SLAPOS_CONFIGURATION', '~/.slapos/slapos.cfg')) class EntryPointNotImplementedError(NotImplementedError): def __init__(self, *args, **kw_args): NotImplementedError.__init__(self, *args, **kw_args) def checkSlaposCfg(): """ Check if a slapos configuration file was given as a argument. If a slapos configuration file is given it return True else False """ # XXX-Cedric: dangerous but quick way to achieve way to not provide # configuration file for each command without changing underlying code. # It the long term, it should be done in a better way (no guessing). for element in sys.argv: if '.cfg' in element: if os.path.exists(element): configuration = ConfigParser.SafeConfigParser() configuration.read(element) if configuration.has_section('slapos'): return True return False def checkOption(option): """ Check if a given option is already in call line Add it and its values if missing """ option = option.split() key = option[0] for element in sys.argv: if key in element: return True sys.argv.append(key) if len(option) > 1 : sys.argv = sys.argv + option[1:] return True def call(fun, config=False, option=None): """ Add missing options to sys.argv Add config if asked and it is missing Call function fun """ if option is None: option = [] for element in option: checkOption(element) if config: if not checkSlaposCfg(): sys.argv = [sys.argv[0]] + [os.path.expanduser(config)] + sys.argv[1:] fun() sys.exit(0) def dispatch(command, is_node_command): """ Dispatch to correct SlapOS module. Here we could use introspection to get rid of the big "if" statements, but we want to control every input. Here we give default option and configuration file if they are needed, i.e If configuration file is not given: define it arbitrarily, and so on. """ if is_node_command: # XXX-Cedric: should we check if we are root? if command == 'register': call(register) elif command == 'software': call(software, config=GLOBAL_SLAPOS_CONFIGURATION, option=['--pidfile /opt/slapos/slapgrid-sr.pid']) elif command == 'instance': call(instance, config=GLOBAL_SLAPOS_CONFIGURATION, option=['--pidfile /opt/slapos/slapgrid-cp.pid']) elif command == 'report': call(report, config=GLOBAL_SLAPOS_CONFIGURATION, option=['--pidfile /opt/slapos/slapgrid-ur.pid']) elif command == 'bang': call(bang, config=True) elif command == 'format': call(format, config=GLOBAL_SLAPOS_CONFIGURATION, option=['-c', '-v']) elif command == 'supervisord': call(supervisord, config=GLOBAL_SLAPOS_CONFIGURATION) elif command == 'supervisorctl': call(supervisorctl, config=GLOBAL_SLAPOS_CONFIGURATION) elif command in ['start', 'stop', 'restart', 'status', 'tail']: # Again, too hackish sys.argv[-2:-2] = [command] call(supervisorctl, config=GLOBAL_SLAPOS_CONFIGURATION) else: return False elif command == 'request': call(request, config=USER_SLAPOS_CONFIGURATION) elif command == 'supply': call(supply, config=USER_SLAPOS_CONFIGURATION) elif command == 'remove': call(remove, config=USER_SLAPOS_CONFIGURATION) elif command == 'start': raise EntryPointNotImplementedError(command) elif command == 'stop': raise EntryPointNotImplementedError(command) elif command == 'destroy': raise EntryPointNotImplementedError(command) elif command == 'console': call(console, config=USER_SLAPOS_CONFIGURATION) elif command == 'cache-lookup': call(cache_lookup, config=GLOBAL_SLAPOS_CONFIGURATION) else: return False def main(): """ Main entry point of SlapOS Node. Used to dispatch commands to python module responsible of the operation. """ # If "node" arg is the first: we strip it and set a switch if len(sys.argv) > 1 and sys.argv[1] == "node": sys.argv.pop(1) # Hackish way to show status if no argument is specified if len(sys.argv) is 1: sys.argv.append('status') is_node = True else: is_node = False usage = """SlapOS %s command line interface. For more informations, refer to SlapOS documentation. Client subcommands usage: slapos request [--configuration arg1=value1 arg2=value2 ... argN=valueN] slapos supply slapos console slapos cache-lookup Node subcommands usage: slapos node slapos node register slapos node software slapos node instance slapos node report slapos node format slapos node start slapos node stop slapos node restart slapos node tail [process] slapos node status slapos node supervisorctl slapos node supervisord """ % version # Parse arguments # XXX remove the "positional arguments" from help message parser = argparse.ArgumentParser(usage=usage) parser.add_argument('command') parser.add_argument('argument_list', nargs=argparse.REMAINDER) namespace = parser.parse_args() # Set sys.argv for the sub-entry point that we will call command_line = [namespace.command] command_line.extend(namespace.argument_list) sys.argv = command_line try: if not dispatch(namespace.command, is_node): parser.print_help() sys.exit(1) except EntryPointNotImplementedError, exception: print ('The command %s does not exist or is not yet implemented. Please ' 'have a look at http://community.slapos.org to read documentation or ' 'forum. Please also make sure that SlapOS Node is up to ' 'date.' % exception) sys.exit(1) slapos.core-0.35.1/slapos/format.py0000644000076600007660000013213212105151770025262 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- # vim: set et sts=2: ############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## from optparse import OptionParser, Option from xml_marshaller import xml_marshaller import ConfigParser import errno import fcntl import grp import json import logging import netaddr import netifaces import os import pwd import random import slapos.slap as slap import socket import struct import subprocess import sys import threading import time import zipfile import lxml.etree from slapos.version import version # set up logging logger = logging.getLogger("slapformat") logger.setLevel(logging.INFO) def prettify_xml(xml): root = lxml.etree.fromstring(xml) return lxml.etree.tostring(root, pretty_print=True) from slapos.util import mkdir_p class OS(object): """Wrap parts of the 'os' module to provide logging of performed actions.""" _os = os def __init__(self, config): self._dry_run = config.dry_run self._verbose = config.verbose self._logger = config.logger add = self._addWrapper add('chown') add('chmod') add('makedirs') add('mkdir') def _addWrapper(self, name): def wrapper(*args, **kw): if self._verbose: arg_list = [repr(x) for x in args] + [ '%s=%r' % (x, y) for x, y in kw.iteritems() ] self._logger.debug('%s(%s)' % (name, ', '.join(arg_list))) if not self._dry_run: getattr(self._os, name)(*args, **kw) setattr(self, name, wrapper) def __getattr__(self, name): return getattr(self._os, name) class UsageError(Exception): pass class NoAddressOnInterface(Exception): """ Exception raised if there is no address on the interface to construct IPv6 address with. Attributes: brige: String, the name of the interface. """ def __init__(self, interface): super(NoAddressOnInterface, self).__init__( 'No IPv6 found on interface %s to construct IPv6 with.' % interface ) class AddressGenerationError(Exception): """ Exception raised if the generation of an IPv6 based on the prefix obtained from the interface failed. Attributes: addr: String, the invalid address the exception is raised for. """ def __init__(self, addr): super(AddressGenerationError, self).__init__( 'Generated IPv6 %s seems not to be a valid IP.' % addr ) def callAndRead(argument_list, raise_on_error=True): popen = subprocess.Popen(argument_list, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) result = popen.communicate()[0] if raise_on_error and popen.returncode != 0: raise ValueError('Issue while invoking %r, result was:\n%s' % ( argument_list, result)) return popen.returncode, result def isGlobalScopeAddress(a): """Returns True if a is global scope IP v4/6 address""" ip = netaddr.IPAddress(a) return not ip.is_link_local() and not ip.is_loopback() and \ not ip.is_reserved() and ip.is_unicast() def netmaskToPrefixIPv4(netmask): """Convert string represented netmask to its integer prefix""" return netaddr.strategy.ipv4.netmask_to_prefix[ netaddr.strategy.ipv4.str_to_int(netmask)] def netmaskToPrefixIPv6(netmask): """Convert string represented netmask to its integer prefix""" return netaddr.strategy.ipv6.netmask_to_prefix[ netaddr.strategy.ipv6.str_to_int(netmask)] def _getDict(instance): """ Serialize an object instance into dictionaries. List and dict will remains the same, basic type too. But encapsulated object will be returned as dict. Set, collections and other aren't handle for now. Args: instance: an object of any type. Returns: A dictionary if the given object wasn't a list, a list otherwise. """ if isinstance(instance, list): return [_getDict(item) for item in instance] elif isinstance(instance, dict): result = {} for key in instance: result[key] = _getDict(instance[key]) return result else: try: dikt = instance.__dict__ except AttributeError: return instance result = {} for key, value in dikt.iteritems(): result[key] = _getDict(value) return result class Computer(object): "Object representing the computer" instance_root = None software_root = None def __init__(self, reference, interface=None, addr=None, netmask=None, ipv6_interface=None, software_user='slapsoft'): """ Attributes: reference: String, the reference of the computer. interface: String, the name of the computer's used interface. """ self.reference = str(reference) self.interface = interface self.partition_list = [] self.address = addr self.netmask = netmask self.ipv6_interface = ipv6_interface self.software_user = software_user def __getinitargs__(self): return (self.reference, self.interface) def getAddress(self, allow_tap=False): """ Return a list of the interface address not attributed to any partition (which are therefore free for the computer itself). Returns: False if the interface isn't available, else the list of the free addresses. """ if self.interface is None: return {'addr': self.address, 'netmask': self.netmask} computer_partition_address_list = [] for partition in self.partition_list: for address in partition.address_list: if netaddr.valid_ipv6(address['addr']): computer_partition_address_list.append(address['addr']) # Going through addresses of the computer's interface for address_dict in self.interface.getGlobalScopeAddressList(): # Comparing with computer's partition addresses if address_dict['addr'] not in computer_partition_address_list: return address_dict if allow_tap: # all addresses on interface are for partition, so let's add new one computer_tap = Tap('compdummy') computer_tap.createWithOwner(User('root'), attach_to_tap=True) self.interface.addTap(computer_tap) return self.interface.addAddr() # Can't find address raise NoAddressOnInterface('No valid IPv6 found on %s.' % self.interface.name) def send(self, config): """ Send a marshalled dictionary of the computer object serialized via_getDict. """ slap_instance = slap.slap() connection_dict = {} if config.key_file and config.cert_file: connection_dict['key_file'] = config.key_file connection_dict['cert_file'] = config.cert_file slap_instance.initializeConnection(config.master_url, **connection_dict) slap_computer = slap_instance.registerComputer(self.reference) if config.dry_run: return try: slap_computer.updateConfiguration(xml_marshaller.dumps(_getDict(self))) except slap.NotFoundError as error: raise slap.NotFoundError("%s\nERROR : This SlapOS node is not recognised by " "SlapOS Master. Please make sure computer_id of slapos.cfg looks " "like 'COMP-123' and is correct.\nError is : 404 Not Found." % error) def dump(self, path_to_xml, path_to_json): """ Dump the computer object to an xml file via xml_marshaller. Args: path_to_xml: String, path to the file to load. path_to_json: String, path to the JSON version to save. """ computer_dict = _getDict(self) if path_to_json: with open(path_to_json, 'wb') as fout: fout.write(json.dumps(computer_dict, sort_keys=True, indent=2)) new_xml = xml_marshaller.dumps(computer_dict) new_pretty_xml = prettify_xml(new_xml) path_to_archive = path_to_xml + '.zip' if os.path.exists(path_to_archive): # the archive file exists, we only backup if something has changed with open(path_to_xml, 'rb') as fin: if fin.read() == new_pretty_xml: # computer configuration did not change, nothing to write return if os.path.exists(path_to_xml): self.backup_xml(path_to_archive, path_to_xml) with open(path_to_xml, 'wb') as fout: fout.write(new_pretty_xml) def backup_xml(self, path_to_archive, path_to_xml): """ Stores a copy of the current xml file to an historical archive. """ xml_content = open(path_to_xml).read() saved_filename = os.path.basename(path_to_xml) + time.strftime('.%Y%m%d-%H:%M') with zipfile.ZipFile(path_to_archive, 'a') as archive: archive.writestr(saved_filename, xml_content, zipfile.ZIP_DEFLATED) @classmethod def load(cls, path_to_xml, reference, ipv6_interface): """ Create a computer object from a valid xml file. Arg: path_to_xml: String, a path to a valid file containing a valid configuration. Return: A Computer object. """ dumped_dict = xml_marshaller.loads(open(path_to_xml).read()) # Reconstructing the computer object from the xml computer = Computer( reference = reference, addr = dumped_dict['address'], netmask = dumped_dict['netmask'], ipv6_interface = ipv6_interface, software_user = dumped_dict.get('software_user', 'slapsoft'), ) for partition_dict in dumped_dict['partition_list']: if partition_dict['user']: user = User(partition_dict['user']['name']) else: user = User('root') if partition_dict['tap']: tap = Tap(partition_dict['tap']['name']) else: tap = Tap(partition_dict['reference']) address_list = partition_dict['address_list'] partition = Partition( reference = partition_dict['reference'], path = partition_dict['path'], user = user, address_list = address_list, tap = tap, ) computer.partition_list.append(partition) return computer def construct(self, alter_user=True, alter_network=True, create_tap=True): """ Construct the computer object as it is. """ if alter_network and self.address is not None: self.interface.addAddr(self.address, self.netmask) for path in self.instance_root, self.software_root: if not os.path.exists(path): os.makedirs(path, 0755) else: os.chmod(path, 0755) # own self.software_root by software user slapsoft = User(self.software_user) slapsoft.path = self.software_root if alter_user: slapsoft.create() slapsoft_pw = pwd.getpwnam(slapsoft.name) os.chown(self.software_root, slapsoft_pw.pw_uid, slapsoft_pw.pw_gid) os.chmod(self.software_root, 0755) try: for partition_index, partition in enumerate(self.partition_list): # Reconstructing User's partition.path = os.path.join(self.instance_root, partition.reference) partition.user.setPath(partition.path) partition.user.additional_group_list = [slapsoft.name] if alter_user: partition.user.create() # Reconstructing Tap if partition.user and partition.user.isAvailable(): owner = partition.user else: owner = User('root') if alter_network and create_tap: # In case it has to be attached to the TAP network device, only one # is necessary for the interface to assert carrier if self.interface.attach_to_tap and partition_index == 0: partition.tap.createWithOwner(owner, attach_to_tap=True) else: partition.tap.createWithOwner(owner) self.interface.addTap(partition.tap) # Reconstructing partition's directory partition.createPath(alter_user) # Reconstructing partition's address # There should be two addresses on each Computer Partition: # * global IPv6 # * local IPv4, took from slapformat:ipv4_local_network if len(partition.address_list) == 0: # regenerate partition.address_list.append(self.interface.addIPv4LocalAddress()) partition.address_list.append(self.interface.addAddr()) elif alter_network: # regenerate list of addresses old_partition_address_list = partition.address_list partition.address_list = [] if len(old_partition_address_list) != 2: raise ValueError( 'There should be exactly 2 stored addresses. Got: %r' % (old_partition_address_list,)) if not any([netaddr.valid_ipv6(q['addr']) for q in old_partition_address_list]): raise ValueError('Not valid ipv6 addresses loaded') if not any([netaddr.valid_ipv4(q['addr']) for q in old_partition_address_list]): raise ValueError('Not valid ipv6 addresses loaded') for address in old_partition_address_list: if netaddr.valid_ipv6(address['addr']): partition.address_list.append(self.interface.addAddr( address['addr'], address['netmask'])) elif netaddr.valid_ipv4(address['addr']): partition.address_list.append(self.interface.addIPv4LocalAddress( address['addr'])) else: raise ValueError('Address %r is incorrect' % address['addr']) finally: if alter_network and create_tap and self.interface.attach_to_tap: try: self.partition_list[0].tap.detach() except IndexError: pass class Partition(object): "Represent a computer partition" def __init__(self, reference, path, user, address_list, tap): """ Attributes: reference: String, the name of the partition. path: String, the path to the partition folder. user: User, the user linked to this partition. address_list: List of associated IP addresses. tap: Tap, the tap interface linked to this partition. """ self.reference = str(reference) self.path = str(path) self.user = user self.address_list = address_list or [] self.tap = tap def __getinitargs__(self): return (self.reference, self.path, self.user, self.address_list, self.tap) def createPath(self, alter_user=True): """ Create the directory of the partition, assign to the partition user and give it the 750 permission. In case if path exists just modifies it. """ self.path = os.path.abspath(self.path) owner = self.user if self.user else User('root') if not os.path.exists(self.path): os.mkdir(self.path, 0750) if alter_user: owner_pw = pwd.getpwnam(owner.name) os.chown(self.path, owner_pw.pw_uid, owner_pw.pw_gid) os.chmod(self.path, 0750) class User(object): """User: represent and manipulate a user on the system.""" path = None def __init__(self, user_name, additional_group_list=None): """ Attributes: user_name: string, the name of the user, who will have is home in """ self.name = str(user_name) self.additional_group_list = additional_group_list def __getinitargs__(self): return (self.name,) def setPath(self, path): self.path = path def create(self): """ Create a user on the system who will be named after the self.name with its own group and directory. Returns: True: if the user creation went right """ # XXX: This method shall be no-op in case if all is correctly setup # This method shall check if all is correctly done # This method shall not reset groups, just add them try: grp.getgrnam(self.name) except KeyError: callAndRead(['groupadd', self.name]) user_parameter_list = ['-d', self.path, '-g', self.name, '-s', '/bin/false'] if self.additional_group_list is not None: user_parameter_list.extend(['-G', ','.join(self.additional_group_list)]) user_parameter_list.append(self.name) try: pwd.getpwnam(self.name) except KeyError: user_parameter_list.append('-r') callAndRead(['useradd'] + user_parameter_list) else: callAndRead(['usermod'] + user_parameter_list) return True def isAvailable(self): """ Determine the availability of a user on the system Return: True: if available False: otherwise """ try: pwd.getpwnam(self.name) return True except KeyError: return False class Tap(object): "Tap represent a tap interface on the system" IFF_TAP = 0x0002 TUNSETIFF = 0x400454ca KEEP_TAP_ATTACHED_EVENT = threading.Event() def __init__(self, tap_name): """ Attributes: tap_name: String, the name of the tap interface. """ self.name = str(tap_name) def __getinitargs__(self): return (self.name,) def attach(self): """ Attach to the TAP interface, meaning that it just opens the TAP interface and waits for the caller to notify that it can be safely detached. Linux distinguishes administrative and operational state of an network interface. The former can be set manually by running ``ip link set dev up|down'', whereas the latter states that the interface can actually transmit data (for a wired network interface, it basically means that there is carrier, e.g. the network cable is plugged into a switch for example). In case of bridge: In order to be able to check the uniqueness of IPv6 address assigned to the bridge, the network interface must be up from an administrative *and* operational point of view. However, from Linux 2.6.39, the bridge reflects the state of the underlying device (e.g. the bridge asserts carrier if at least one of its ports has carrier) whereas it always asserted carrier before. This should work fine for "real" network interface, but will not work properly if the bridge only binds TAP interfaces, which, from 2.6.36, reports carrier only and only if an userspace program is attached. """ tap_fd = os.open("/dev/net/tun", os.O_RDWR) try: # Attach to the TAP interface which has previously been created fcntl.ioctl(tap_fd, self.TUNSETIFF, struct.pack("16sI", self.name, self.IFF_TAP)) except IOError as error: # If EBUSY, it means another program is already attached, thus just # ignore it... if error.errno != errno.EBUSY: os.close(tap_fd) raise else: # Block until the caller send an event stating that the program can be # now detached safely, thus bringing down the TAP device (from 2.6.36) # and the bridge at the same time (from 2.6.39) self.KEEP_TAP_ATTACHED_EVENT.wait() finally: os.close(tap_fd) def detach(self): """ Detach to the TAP network interface by notifying the thread which attach to the TAP and closing the TAP file descriptor """ self.KEEP_TAP_ATTACHED_EVENT.set() def createWithOwner(self, owner, attach_to_tap=False): """ Create a tap interface on the system. """ # some systems does not have -p switch for tunctl #callAndRead(['tunctl', '-p', '-t', self.name, '-u', owner.name]) check_file = '/sys/devices/virtual/net/%s/owner' % self.name owner_id = None if os.path.exists(check_file): owner_id = open(check_file).read().strip() try: owner_id = int(owner_id) except ValueError: pass if owner_id != pwd.getpwnam(owner.name).pw_uid: callAndRead(['tunctl', '-t', self.name, '-u', owner.name]) callAndRead(['ip', 'link', 'set', self.name, 'up']) if attach_to_tap: threading.Thread(target=self.attach).start() class Interface(object): """Represent a network interface on the system""" def __init__(self, name, ipv4_local_network, ipv6_interface=None): """ Attributes: name: String, the name of the interface """ self.name = str(name) self.ipv4_local_network = ipv4_local_network self.ipv6_interface = ipv6_interface # Attach to TAP network interface, only if the interface interface does not # report carrier _, result = callAndRead(['ip', 'addr', 'list', self.name]) self.attach_to_tap = 'DOWN' in result.split('\n', 1)[0] def __getinitargs__(self): return (self.name,) def getIPv4LocalAddressList(self): """ Returns currently configured local IPv4 addresses which are in ipv4_local_network """ if not socket.AF_INET in netifaces.ifaddresses(self.name): return [] return [dict(addr=q['addr'], netmask=q['netmask']) for q in netifaces.ifaddresses(self.name)[socket.AF_INET] if netaddr.IPAddress( q['addr'], 4) in netaddr.glob_to_iprange( netaddr.cidr_to_glob(self.ipv4_local_network))] def getGlobalScopeAddressList(self): """Returns currently configured global scope IPv6 addresses""" if self.ipv6_interface: interface_name = self.ipv6_interface else: interface_name = self.name try: address_list = [q for q in netifaces.ifaddresses(interface_name)[socket.AF_INET6] if isGlobalScopeAddress(q['addr'].split('%')[0])] except KeyError: raise ValueError("%s must have at least one IPv6 address assigned" % \ interface_name) # XXX: Missing implementation of Unique Local IPv6 Unicast Addresses as # defined in http://www.rfc-editor.org/rfc/rfc4193.txt # XXX: XXX: XXX: IT IS DISALLOWED TO IMPLEMENT link-local addresses as # Linux and BSD are possibly wrongly implementing it -- it is "too local" # it is impossible to listen or access it on same node # XXX: IT IS DISALLOWED to implement ad hoc solution like inventing node # local addresses or anything which does not exists in RFC! return address_list def getInterfaceList(self): """Returns list of interfaces already present on bridge""" interface_list = [] _, result = callAndRead(['brctl', 'show']) in_interface = False for line in result.split('\n'): if len(line.split()) > 1: if self.name in line: interface_list.append(line.split()[-1]) in_interface = True continue if in_interface: break elif in_interface: if line.strip(): interface_list.append(line.strip()) return interface_list def addTap(self, tap): """ Add the tap interface tap to the bridge. Args: tap: Tap, the tap interface. """ if tap.name not in self.getInterfaceList(): callAndRead(['brctl', 'addif', self.name, tap.name]) def _addSystemAddress(self, address, netmask, ipv6=True): """Adds system address to interface Returns True if address was added successfully. Returns False if there was issue. """ if ipv6: address_string = '%s/%s' % (address, netmaskToPrefixIPv6(netmask)) af = socket.AF_INET6 if self.ipv6_interface: interface_name = self.ipv6_interface else: interface_name = self.name else: af = socket.AF_INET address_string = '%s/%s' % (address, netmaskToPrefixIPv4(netmask)) interface_name = self.name # check if address is already took by any other interface for interface in netifaces.interfaces(): if interface != interface_name: address_dict = netifaces.ifaddresses(interface) if af in address_dict: if address in [q['addr'].split('%')[0] for q in address_dict[af]]: return False if not af in netifaces.ifaddresses(interface_name) \ or not address in [q['addr'].split('%')[0] for q in netifaces.ifaddresses(interface_name)[af] ]: # add an address callAndRead(['ip', 'addr', 'add', address_string, 'dev', interface_name]) # Fake success for local ipv4 if not ipv6: return True # wait few moments time.sleep(2) # Fake success for local ipv4 if not ipv6: return True # check existence on interface for ipv6 _, result = callAndRead(['ip', 'addr', 'list', interface_name]) for l in result.split('\n'): if address in l: if 'tentative' in l: # duplicate, remove callAndRead(['ip', 'addr', 'del', address_string, 'dev', interface_name]) return False # found and clean return True # even when added not found, this is bad... return False def _generateRandomIPv4Address(self, netmask): # no addresses found, generate new one # Try 10 times to add address, raise in case if not possible try_num = 10 while try_num > 0: addr = random.choice([q for q in netaddr.glob_to_iprange( netaddr.cidr_to_glob(self.ipv4_local_network))]).format() if dict(addr=addr, netmask=netmask) not in \ self.getIPv4LocalAddressList(): # Checking the validity of the IPv6 address if self._addSystemAddress(addr, netmask, False): return dict(addr=addr, netmask=netmask) try_num -= 1 raise AddressGenerationError(addr) def addIPv4LocalAddress(self, addr=None): """Adds local IPv4 address in ipv4_local_network""" netmask = '255.255.255.255' local_address_list = self.getIPv4LocalAddressList() if addr is None: return self._generateRandomIPv4Address(netmask) elif dict(addr=addr, netmask=netmask) not in local_address_list: if self._addSystemAddress(addr, netmask, False): return dict(addr=addr, netmask=netmask) else: logger.warning('Impossible to add old local IPv4 %s. Generating ' 'new IPv4 address.' % addr) return self._generateRandomIPv4Address(netmask) else: # confirmed to be configured return dict(addr=addr, netmask=netmask) def addAddr(self, addr = None, netmask = None): """ Adds IP address to interface. If addr is specified and exists already on interface does nothing. If addr is specified and does not exists on interface, tries to add given address. If it is not possible (ex. because network changed) calculates new address. Args: addr: Wished address to be added to interface. netmask: Wished netmask to be used. Returns: Tuple of (address, netmask). Raises: AddressGenerationError: Couldn't construct valid address with existing one's on the interface. NoAddressOnInterface: There's no address on the interface to construct an address with. """ # Getting one address of the interface as base of the next addresses if self.ipv6_interface: interface_name = self.ipv6_interface else: interface_name = self.name interface_addr_list = self.getGlobalScopeAddressList() # No address found if len(interface_addr_list) == 0: raise NoAddressOnInterface(interface_name) address_dict = interface_addr_list[0] if addr is not None: if dict(addr=addr, netmask=netmask) in interface_addr_list: # confirmed to be configured return dict(addr=addr, netmask=netmask) if netmask == address_dict['netmask']: # same netmask, so there is a chance to add good one interface_network = netaddr.ip.IPNetwork('%s/%s' % (address_dict['addr'], netmaskToPrefixIPv6(address_dict['netmask']))) requested_network = netaddr.ip.IPNetwork('%s/%s' % (addr, netmaskToPrefixIPv6(netmask))) if interface_network.network == requested_network.network: # same network, try to add if self._addSystemAddress(addr, netmask): # succeed, return it return dict(addr=addr, netmask=netmask) else: logger.warning('Impossible to add old public IPv6 %s. ' 'Generating new IPv6 address.' % addr) # Try 10 times to add address, raise in case if not possible try_num = 10 netmask = address_dict['netmask'] while try_num > 0: addr = ':'.join(address_dict['addr'].split(':')[:-1] + ['%x' % ( random.randint(1, 65000), )]) socket.inet_pton(socket.AF_INET6, addr) if dict(addr=addr, netmask=netmask) not in \ self.getGlobalScopeAddressList(): # Checking the validity of the IPv6 address if self._addSystemAddress(addr, netmask): return dict(addr=addr, netmask=netmask) try_num -= 1 raise AddressGenerationError(addr) class Parser(OptionParser): """ Parse all arguments. """ def __init__(self, usage=None, version=version): """ Initialize all options possibles. """ OptionParser.__init__(self, usage=usage, version=version, option_list=[ Option("-x", "--computer_xml", help="Path to file with computer's XML. If does not exists, " "will be created", default=None, type=str), Option("--computer_json", help="Path to a JSON version of the computer's XML (for development only).", default=None, type=str), Option("-l", "--log_file", help="The path to the log file used by the script.", type=str), Option("-i", "--input_definition_file", help="Path to file to read definition of computer instead of " "declaration. Using definition file allows to disable " "'discovery' of machine services and allows to define computer " "configuration in fully controlled manner.", type=str), Option("-o", "--output_definition_file", help="Path to file to write definition of computer from " "declaration.", type=str), Option("-n", "--dry_run", help="Don't actually do anything.", default=False, action="store_true"), Option("-v", "--verbose", default=False, action="store_true", help="Verbose output."), Option("-c", "--console", default=False, action="store_true", help="Console output."), Option('--alter_user', choices=['True', 'False'], help="Shall slapformat alter user database [default: True]"), Option('--alter_network', choices=['True', 'False'], help="Shall slapformat alter network configuration [default: True]"), Option('--now', help="Launch slapformat without delay", default=False, action="store_true"), ]) def check_args(self, args): """ Check arguments """ if args: (options, args) = self.parse_args(list(args)) else: (options, args) = self.parse_args() if len(args) != 1: self.error("Incorrect number of arguments") return options, args[0] def parse_computer_definition(config, definition_path): config.logger.info('Using definition file %r' % definition_path) computer_definition = ConfigParser.RawConfigParser({ 'software_user': 'slapsoft', }) computer_definition.read(definition_path) interface = None address = None netmask = None if computer_definition.has_option('computer', 'address'): address, netmask = computer_definition.get('computer', 'address').split('/') if config.alter_network and config.interface_name is not None \ and config.ipv4_local_network is not None: interface = Interface(config.interface_name, config.ipv4_local_network, config.ipv6_interface) computer = Computer( reference=config.computer_id, interface=interface, addr=address, netmask=netmask, ipv6_interface=config.ipv6_interface, software_user=computer_definition.get('computer', 'software_user'), ) partition_list = [] for partition_number in range(int(config.partition_amount)): section = 'partition_%s' % partition_number user = User(computer_definition.get(section, 'user')) address_list = [] for a in computer_definition.get(section, 'address').split(): address, netmask = a.split('/') address_list.append(dict(addr=address, netmask=netmask)) tap = Tap(computer_definition.get(section, 'network_interface')) partition = Partition(reference=computer_definition.get(section, 'pathname'), path=os.path.join(config.instance_root, computer_definition.get(section, 'pathname')), user=user, address_list=address_list, tap=tap) partition_list.append(partition) computer.partition_list = partition_list return computer def parse_computer_xml(config, xml_path): if os.path.exists(xml_path): config.logger.info('Loading previous computer data from %r' % xml_path) computer = Computer.load(xml_path, reference=config.computer_id, ipv6_interface=config.ipv6_interface) # Connect to the interface defined by the configuration computer.interface = Interface(config.interface_name, config.ipv4_local_network, config.ipv6_interface) else: # If no pre-existent configuration found, create a new computer object config.logger.warning('Creating new data computer with id %r' % config.computer_id) computer = Computer( reference=config.computer_id, interface=Interface(config.interface_name, config.ipv4_local_network, config.ipv6_interface), addr=None, netmask=None, ipv6_interface=config.ipv6_interface, software_user=config.software_user, ) partition_amount = int(config.partition_amount) existing_partition_amount = len(computer.partition_list) if existing_partition_amount > partition_amount: raise ValueError('Requested amount of computer partitions (%s) is lower ' 'then already configured (%s), cannot continue' % (partition_amount, len(computer.partition_list))) config.logger.info('Adding %s new partitions' % (partition_amount-existing_partition_amount)) for nb_iter in range(existing_partition_amount, partition_amount): # add new ones user = User("%s%s" % (config.user_base_name, nb_iter)) tap = Tap("%s%s" % (config.tap_base_name, nb_iter)) path = os.path.join(config.instance_root, "%s%s" % ( config.partition_base_name, nb_iter)) computer.partition_list.append( Partition( reference="%s%s" % (config.partition_base_name, nb_iter), path=path, user=user, address_list=None, tap=tap, )) return computer def write_computer_definition(config, computer): computer_definition = ConfigParser.RawConfigParser() computer_definition.add_section('computer') if computer.address is not None and computer.netmask is not None: computer_definition.set('computer', 'address', '/'.join( [computer.address, computer.netmask])) for partition_number, partition in enumerate(computer.partition_list): section = 'partition_%s' % partition_number computer_definition.add_section(section) address_list = [] for address in partition.address_list: address_list.append('/'.join([address['addr'], address['netmask']])) computer_definition.set(section, 'address', ' '.join(address_list)) computer_definition.set(section, 'user', partition.user.name) computer_definition.set(section, 'network_interface', partition.tap.name) computer_definition.set(section, 'pathname', partition.reference) computer_definition.write(open(config.output_definition_file, 'w')) config.logger.info('Stored computer definition in %r' % config.output_definition_file) def run(config): if config.input_definition_file: computer = parse_computer_definition(config, config.input_definition_file) else: # no definition file, figure out computer computer = parse_computer_xml(config, config.computer_xml) computer.instance_root = config.instance_root computer.software_root = config.software_root config.logger.info('Updating computer') address = computer.getAddress(config.create_tap) computer.address = address['addr'] computer.netmask = address['netmask'] if config.output_definition_file: write_computer_definition(config, computer) computer.construct(alter_user=config.alter_user, alter_network=config.alter_network, create_tap=config.create_tap) if getattr(config, 'certificate_repository_path'): mkdir_p(config.certificate_repository_path, mode=0o700) # Dumping and sending to the erp5 the current configuration if not config.dry_run: computer.dump(path_to_xml=config.computer_xml, path_to_json=config.computer_json) config.logger.info('Posting information to %r' % config.master_url) computer.send(config) config.logger.info('slapformat successfully prepared computer.') class Config(object): key_file = None cert_file = None alter_network = None alter_user = None create_tap = None computer_xml = None computer_json = None input_definition_file = None logger = None log_file = None output_definition_file = None verbose = None dry_run = None console = None software_user = None @staticmethod def checkRequiredBinary(binary_list): missing_binary_list = [] for b in binary_list: if type(b) != type([]): b = [b] try: callAndRead(b) except ValueError: pass except OSError: missing_binary_list.append(b) if missing_binary_list: raise UsageError('Some required binaries are missing or not ' 'functional: %s' % (','.join(missing_binary_list), )) def setConfig(self, option_dict, configuration_file_path): """ Set options given by parameters. """ self.key_file = None self.cert_file = None # set up logging # XXX-Cedric: change code to use global logger self.logger = logger # Set options parameters for option, value in option_dict.__dict__.items(): setattr(self, option, value) # Load configuration file configuration_parser = ConfigParser.SafeConfigParser() if configuration_parser.read(configuration_file_path) != [configuration_file_path]: raise UsageError('Cannot find or parse configuration file: %s' % configuration_file_path) # Merges the arguments and configuration for section in ("slapformat", "slapos"): configuration_dict = dict(configuration_parser.items(section)) for key in configuration_dict: if not getattr(self, key, None): setattr(self, key, configuration_dict[key]) # setup some nones for parameter in ['interface_name', 'partition_base_name', 'user_base_name', 'tap_base_name', 'ipv4_local_network', 'ipv6_interface']: if getattr(self, parameter, None) is None: setattr(self, parameter, None) # Backward compatibility if not getattr(self, "interface_name", None) \ and getattr(self, "bridge_name", None): setattr(self, "interface_name", self.bridge_name) self.logger.warning('bridge_name option is deprecated and should be ' 'replaced by interface_name.') if not getattr(self, "create_tap", None) \ and getattr(self, "no_bridge", None): setattr(self, "create_tap", not self.no_bridge) self.logger.warning('no_bridge option is deprecated and should be ' 'replaced by create_tap.') # Set defaults lately if self.alter_network is None: self.alter_network = 'True' if self.alter_user is None: self.alter_user = 'True' if self.software_user is None: self.software_user = 'slapsoft' if self.create_tap is None: self.create_tap = True # Configure logging if self.console: self.logger.addHandler(logging.StreamHandler()) # Convert strings to booleans for o in ['alter_network', 'alter_user', 'create_tap']: attr = getattr(self, o) if isinstance(attr, str): if attr.lower() == 'true': root_needed = True setattr(self, o, True) elif attr.lower() == 'false': setattr(self, o, False) else: message = 'Option %r needs to be "True" or "False", wrong value: ' \ '%r' % (o, getattr(self, o)) self.logger.error(message) raise UsageError(message) if not self.dry_run: if self.alter_user: self.checkRequiredBinary(['groupadd', 'useradd', 'usermod']) if self.create_tap: self.checkRequiredBinary([['tunctl', '-d']]) if self.alter_network: self.checkRequiredBinary(['ip']) # Required, even for dry run if self.alter_network and self.create_tap: self.checkRequiredBinary(['brctl']) # Check if root is needed if (self.alter_network or self.alter_user) and not self.dry_run: root_needed = True else: root_needed = False # check root if root_needed and os.getuid() != 0: message = "Root rights are needed" self.logger.error(message) raise UsageError(message) if self.log_file: if not os.path.isdir(os.path.dirname(self.log_file)): # fallback to console only if directory for logs does not exists and # continue to run raise ValueError('Please create directory %r to store %r log file' % ( os.path.dirname(self.log_file), self.log_file)) else: file_handler = logging.FileHandler(self.log_file) file_handler.setFormatter(logging.Formatter("%(asctime)s - " "%(name)s - %(levelname)s - %(message)s")) self.logger.addHandler(file_handler) self.logger.info('Configured logging to file %r' % self.log_file) # Check mandatory options for parameter in ('computer_id', 'instance_root', 'master_url', 'software_root', 'computer_xml'): if not getattr(self, parameter, None): raise UsageError("Parameter '%s' is not defined." % parameter) # Check existence of SSL certificate files, if defined for attribute in ['key_file', 'cert_file', 'master_ca_file']: file_location = getattr(self, attribute, None) if file_location is not None: if not os.path.exists(file_location): self.logger.fatal('File %r does not exist or is not readable.' % file_location) sys.exit(1) self.logger.info("Started.") if self.verbose: self.logger.setLevel(logging.DEBUG) self.logger.debug("Verbose mode enabled.") if self.dry_run: self.logger.info("Dry-run mode enabled.") if self.create_tap: self.logger.info("Tap creation mode enabled.") # Calculate path once self.computer_xml = os.path.abspath(self.computer_xml) if self.input_definition_file: self.input_definition_file = os.path.abspath(self.input_definition_file) if self.output_definition_file: self.output_definition_file = os.path.abspath(self.output_definition_file) def tracing_monkeypatch(config): """Substitute os module and callAndRead function with tracing wrappers.""" global os global callAndRead real_callAndRead = callAndRead os = OS(config) if config.dry_run: def dry_callAndRead(argument_list, raise_on_error=True): if argument_list == ['brctl', 'show']: return real_callAndRead(argument_list, raise_on_error) else: return 0, '' callAndRead = dry_callAndRead def fake_getpwnam(user): class result(object): pw_uid = 12345 pw_gid = 54321 return result pwd.getpwnam = fake_getpwnam else: dry_callAndRead = real_callAndRead if config.verbose: def logging_callAndRead(argument_list, raise_on_error=True): config.logger.debug(' '.join(argument_list)) return dry_callAndRead(argument_list, raise_on_error) callAndRead = logging_callAndRead def main(*args): "Run default configuration." # Parse arguments usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0] options, configuration_file_path = Parser(usage=usage).check_args(args) config = Config() try: config.setConfig(options, configuration_file_path) except UsageError as err: sys.stderr.write(err.message + '\n') sys.stderr.write("For help use --help\n") sys.exit(1) tracing_monkeypatch(config) # Add delay between 0 and 1 hour # XXX should be the contrary: now by default, and cron should have # --maximal-delay=3600 if not config.now: duration = float(60*60) * random.random() print("Sleeping for %s seconds. To disable this feature, " \ "use with --now parameter in manual." % duration) time.sleep(duration) try: run(config) except: config.logger.exception('Uncaught exception:') raise slapos.core-0.35.1/slapos/grid/0000755000076600007660000000000012110425554024343 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/grid/__init__.py0000644000076600007660000000245212035516223026457 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## slapos.core-0.35.1/slapos/grid/distribution.py0000644000076600007660000000417712100006516027435 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- """ Provides helper functions to check if two binary caches are compatible. os_matches(...): returns True if the arguments reference compatible platforms. patched_linux_distribution(...): a patched version of platform.linux_distribution() this is the same function provided with the python package in Debian and Ubuntu: see http://bugs.python.org/issue9514 otherwise, Ubuntu will always be reported as an unstable Debian, regardless of the version. """ import platform import re def _debianize(os): """ keep only the major release number in case of debian, otherwise minor releases would be seen as not compatible to each other. """ distname, version, id_ = os if distname == 'debian' and '.' in version: version = version.split('.')[0] return distname, version, id_ def os_matches(os1, os2): return _debianize(os1) == _debianize(os2) _distributor_id_file_re = re.compile("(?:DISTRIB_ID\s*=)\s*(.*)", re.I) _release_file_re = re.compile("(?:DISTRIB_RELEASE\s*=)\s*(.*)", re.I) _codename_file_re = re.compile("(?:DISTRIB_CODENAME\s*=)\s*(.*)", re.I) def patched_linux_distribution(distname='', version='', id='', supported_dists=platform._supported_dists, full_distribution_name=1): # check for the Debian/Ubuntu /etc/lsb-release file first, needed so # that the distribution doesn't get identified as Debian. try: etclsbrel = open("/etc/lsb-release", "rU") for line in etclsbrel: m = _distributor_id_file_re.search(line) if m: _u_distname = m.group(1).strip() m = _release_file_re.search(line) if m: _u_version = m.group(1).strip() m = _codename_file_re.search(line) if m: _u_id = m.group(1).strip() if _u_distname and _u_version: return (_u_distname, _u_version, _u_id) except (EnvironmentError, UnboundLocalError): pass return platform.linux_distribution(distname, version, id, supported_dists, full_distribution_name) slapos.core-0.35.1/slapos/grid/exception.py0000644000076600007660000000273112035516223026716 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## """Exposed exceptions""" class PathDoesNotExistError(Exception): pass class WrongPermissionError(Exception): pass class BuildoutFailedError(Exception): pass slapos.core-0.35.1/slapos/grid/networkcache.py0000644000076600007660000001346312101523453027376 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2010, 2011, 2012 ViFiB SARL and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import ast import json import platform import shutil import traceback from slapos.grid.distribution import os_matches, patched_linux_distribution try: try: from slapos.libnetworkcache import NetworkcacheClient, UploadError, \ DirectoryNotFound except ImportError: LIBNETWORKCACHE_ENABLED = False else: LIBNETWORKCACHE_ENABLED = True except: print 'There was problem while trying to import slapos.libnetworkcache:'\ '\n%s' % traceback.format_exc() LIBNETWORKCACHE_ENABLED = False print 'Networkcache forced to be disabled.' def fallback_call(function): """Decorator which disallow to have any problem while calling method""" def wrapper(self, *args, **kwd): """ Log the call, and the result of the call """ try: return function(self, *args, **kwd) except: # indeed, *any* exception is swallowed print 'There was problem while calling method %r:\n%s' % ( function.__name__, traceback.format_exc()) return False wrapper.__doc__ = function.__doc__ return wrapper @fallback_call def download_network_cached(cache_url, dir_url, software_url, software_root, key, path, logger, signature_certificate_list, download_from_binary_cache_url_blacklist=None): """Downloads from a network cache provider return True if download succeeded. """ if not LIBNETWORKCACHE_ENABLED: return False if not(cache_url and dir_url and software_url and software_root): return False for url in download_from_binary_cache_url_blacklist: if software_url.startswith(url): return False try: nc = NetworkcacheClient(cache_url, dir_url, signature_certificate_list=signature_certificate_list or None) except TypeError: logger.warning('Incompatible version of networkcache, not using it.') return False logger.info('Downloading %s binary from network cache.' % software_url) try: file_descriptor = None json_entry_list = nc.select_generic(key) for entry in json_entry_list: json_information, _ = entry try: tags = json.loads(json_information) if tags.get('machine') != platform.machine(): continue if not os_matches(ast.literal_eval(tags.get('os')), patched_linux_distribution()): continue if tags.get('software_url') != software_url: continue if tags.get('software_root') != software_root: continue sha512 = tags.get('sha512') file_descriptor = nc.download(sha512) break except Exception: continue if file_descriptor is not None: f = open(path, 'w+b') try: shutil.copyfileobj(file_descriptor, f) finally: f.close() file_descriptor.close() return True except (IOError, DirectoryNotFound), e: logger.info('Failed to download from network cache %s: %s' % \ (software_url, str(e))) return False @fallback_call def upload_network_cached(software_root, software_url, cached_key, cache_url, dir_url, path, logger, signature_private_key_file, shacache_cert_file, shacache_key_file, shadir_cert_file, shadir_key_file): """Upload file to a network cache server""" if not LIBNETWORKCACHE_ENABLED: return False if not (software_root and software_url and cached_key \ and cache_url and dir_url): return False logger.info('Uploading %s binary into network cache.' % software_url) # YXU: "file" and "urlmd5" should be removed when server side is ready kw = dict( file="file", urlmd5="urlmd5", software_url=software_url, software_root=software_root, machine=platform.machine(), os=str(patched_linux_distribution()) ) f = open(path, 'r') # convert '' into None in order to call nc nicely if not signature_private_key_file: signature_private_key_file = None if not shacache_cert_file: shacache_cert_file = None if not shacache_key_file: shacache_key_file = None if not shadir_cert_file: shadir_cert_file = None if not shadir_key_file: shadir_key_file = None try: nc = NetworkcacheClient(cache_url, dir_url, signature_private_key_file=signature_private_key_file, shacache_cert_file=shacache_cert_file, shacache_key_file=shacache_key_file, shadir_cert_file=shadir_cert_file, shadir_key_file=shadir_key_file) except TypeError: logger.warning('Incompatible version of networkcache, not using it.') return False try: return nc.upload_generic(f, cached_key, **kw) except (IOError, UploadError), e: logger.info('Fail to upload file. %s' % (str(e))) return False finally: f.close() return True slapos.core-0.35.1/slapos/grid/slapgrid.py0000644000076600007660000014635012100006516026523 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- # vim: set et sts=2: ############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import argparse import ConfigParser from exception import BuildoutFailedError from hashlib import md5 from lxml import etree import logging import os import pkg_resources import random import socket import StringIO import subprocess import sys import tempfile import time import traceback import warnings if sys.version_info < (2, 6): warnings.warn('Used python version (%s) is old and have problems with' ' IPv6 connections' % sys.version.split('\n')[0]) from slapos.slap.slap import NotFoundError from slapos.slap.slap import ServerError from SlapObject import Software, Partition, WrongPermissionError, \ PathDoesNotExistError from svcbackend import launchSupervisord from utils import createPrivateDirectory from utils import dropPrivileges from utils import getSoftwareUrlHash from utils import setRunning from utils import setFinished from utils import SlapPopen from utils import updateFile from slapos import slap MANDATORY_PARAMETER_LIST = [ 'computer_id', 'instance_root', 'master_url', 'software_root', ] # XXX: should be moved to SLAP library COMPUTER_PARTITION_DESTROYED_STATE = 'destroyed' COMPUTER_PARTITION_STARTED_STATE = 'started' COMPUTER_PARTITION_STOPPED_STATE = 'stopped' # Global variables about return state of slapgrid SLAPGRID_SUCCESS = 0 SLAPGRID_FAIL = 1 SLAPGRID_PROMISE_FAIL = 2 # XXX hardcoded watchdog_path WATCHDOG_PATH = '/opt/slapos/bin/slapos-watchdog' class _formatXMLError(Exception): pass def parseArgumentTupleAndReturnSlapgridObject(*argument_tuple): """Parses arguments either from command line, from method parameters or from config file. Then returns a new instance of slapgrid.Slapgrid with those parameters. Also returns the options dict and unused variable list, and configures logger. """ parser = argparse.ArgumentParser() parser.add_argument("--instance-root", help="The instance root directory location.") parser.add_argument("--software-root", help="The software_root directory location.") parser.add_argument("--master-url", help="The master server URL. Mandatory.") parser.add_argument("--computer-id", help="The computer id defined in the server.") parser.add_argument("--supervisord-socket", help="The socket supervisor will use.") parser.add_argument("--supervisord-configuration-path", help="The location where supervisord configuration will be stored.") parser.add_argument("--buildout", default=None, help="Location of buildout binary.") parser.add_argument("--pidfile", help="The location where pidfile will be created.") parser.add_argument("--logfile", help="The location where slapgrid logfile will be created.") parser.add_argument("--key_file", help="SSL Authorisation key file.") parser.add_argument("--cert_file", help="SSL Authorisation certificate file.") parser.add_argument("--signature_private_key_file", help="Signature private key file.") parser.add_argument("--master_ca_file", help="Root certificate of SlapOS master key.") parser.add_argument("--certificate_repository_path", help="Path to directory where downloaded certificates would be stored.") parser.add_argument("-c", "--console", action="store_true", default=False, help="Deprecated, doesn't do anything.") parser.add_argument("-v", "--verbose", action="store_true", default=False, help="Be verbose.") parser.add_argument("--maximum-periodicity", type=int, default=None, help="Periodicity at which buildout should be run in instance.") parser.add_argument("--promise-timeout", type=int, default=3, help="Promise timeout in seconds.") parser.add_argument("--now", action="store_true", default=False, help="Launch slapgrid without delay. Default behavior.") parser.add_argument("--all", action="store_true", default=False, help="Launch slapgrid to process all Softare Releases" "and/or Computer Partitions.") parser.add_argument("--only-sr", help="Force the update of a single software release (use url hash)," "event if is already installed. This option will make all others " "sofware releases be ignored.") parser.add_argument("--only-cp", help="Update a single or a list of computer partitions " "(ie.:slappartX, slappartY)," "this option will make all others computer partitions be ignored.") parser.add_argument("configuration_file", nargs=1, type=argparse.FileType(), help="SlapOS configuration file.") # Deprecated options parser.add_argument("--develop", action="store_true", default=False, help="Deprecated, same as --all.") parser.add_argument("--only_sr", help="Deprecated, same as --only-sr.") parser.add_argument("--only_cp", help="Deprecated, same as --only-cp.") parser.add_argument("--maximal_delay", help="Deprecated. Will only work from configuration file in the future.") # Parses arguments if argument_tuple == (): # No arguments given to entry point : we parse sys.argv. argument_option_instance = parser.parse_args() else: argument_option_instance = \ parser.parse_args(list(argument_tuple)) # Parses arguments from config file, if needed, then merge previous arguments option_dict = {} configuration_file = argument_option_instance.configuration_file[0] # Loads config (if config specified) slapgrid_configuration = ConfigParser.SafeConfigParser() slapgrid_configuration.readfp(configuration_file) # Merges the two dictionnaries option_dict = dict(slapgrid_configuration.items("slapos")) if slapgrid_configuration.has_section("networkcache"): option_dict.update(dict(slapgrid_configuration.items("networkcache"))) for argument_key, argument_value in vars(argument_option_instance ).iteritems(): if argument_value is not None: option_dict.update({argument_key: argument_value}) # Configures logger. if option_dict['verbose']: level = logging.DEBUG else: level = logging.INFO logging.basicConfig(format='%(asctime)s %(levelname)-8s %(message)s', level=level, datefmt='%Y-%m-%dT%H:%M:%S') if option_dict.get('logfile'): console = logging.FileHandler(option_dict['logfile']) console.setLevel(level) console.setFormatter(logging.Formatter( '%(asctime)s %(name)-18s: %(levelname)-8s %(message)s')) logging.getLogger('').addHandler(console) missing_mandatory_parameter_list = [] for mandatory_parameter in MANDATORY_PARAMETER_LIST: if not mandatory_parameter in option_dict: missing_mandatory_parameter_list.append(mandatory_parameter) if option_dict.get('all') is True: option_dict['develop'] = True if option_dict.get('maximum_periodicity') is not None: option_dict['force_periodicity'] = True repository_required = False if 'key_file' in option_dict: repository_required = True if not 'cert_file' in option_dict: missing_mandatory_parameter_list.append('cert_file') if 'cert_file' in option_dict: repository_required = True if not 'key_file' in option_dict: missing_mandatory_parameter_list.append('key_file') if repository_required: if 'certificate_repository_path' not in option_dict: missing_mandatory_parameter_list.append('certificate_repository_path') if len(missing_mandatory_parameter_list) > 0: parser.error('Missing mandatory parameters:\n%s' % '\n'.join( missing_mandatory_parameter_list)) key_file = option_dict.get('key_file') cert_file = option_dict.get('cert_file') master_ca_file = option_dict.get('master_ca_file') signature_private_key_file = option_dict.get('signature_private_key_file') mandatory_file_list = [key_file, cert_file, master_ca_file] # signature_private_key_file is not mandatory, we must be able to run # slapgrid scripts without this parameter. if signature_private_key_file: mandatory_file_list.append(signature_private_key_file) for k in ['shacache-cert-file', 'shacache-key-file', 'shadir-cert-file', 'shadir-key-file']: mandatory_file_list.append(option_dict.get(k, None)) for f in mandatory_file_list: if f is not None: if not os.path.exists(f): parser.error('File %r does not exist.' % f) certificate_repository_path = option_dict.get('certificate_repository_path') if certificate_repository_path is not None: if not os.path.isdir(certificate_repository_path): parser.error('Directory %r does not exist' % certificate_repository_path) # Supervisord configuration location if not option_dict.get('supervisord_configuration_path'): option_dict['supervisord_configuration_path'] = \ os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf') # Supervisord socket if not option_dict.get('supervisord_socket'): option_dict['supervisord_socket'] = \ os.path.join(option_dict['instance_root'], 'supervisord.socket') signature_certificate_list_string = \ option_dict.get('signature-certificate-list', None) if signature_certificate_list_string is not None: cert_marker = "-----BEGIN CERTIFICATE-----" signature_certificate_list = [cert_marker + '\n' + q.strip() \ for q in signature_certificate_list_string.split(cert_marker) \ if q.strip()] else: signature_certificate_list = None # Parse cache / binary cache options # Backward compatibility about "binary-cache-url-blacklist" deprecated option if option_dict.get("binary-cache-url-blacklist") and not \ option_dict.get("download-from-binary-cache-url-blacklist"): option_dict["download-from-binary-cache-url-blacklist"] = \ option_dict["binary-cache-url-blacklist"] option_dict["download-from-binary-cache-url-blacklist"] = [ url.strip() for url in option_dict.get( "download-from-binary-cache-url-blacklist", "").split('\n') if url] option_dict["upload-to-binary-cache-url-blacklist"] = [ url.strip() for url in option_dict.get( "upload-to-binary-cache-url-blacklist", "").split('\n') if url] # Sleep for a random time to avoid SlapOS Master being DDOSed by an army of # SlapOS Nodes configured with cron. if option_dict["now"]: # XXX-Cedric: deprecate "--now" maximal_delay = 0 else: maximal_delay = int(option_dict.get("maximal_delay", "0")) if maximal_delay > 0: duration = random.randint(1, maximal_delay) logging.info("Sleeping for %s seconds. To disable this feature, " \ "check --now parameter in slapgrid help." % duration) time.sleep(duration) # Return new Slapgrid instance and options return ([Slapgrid(software_root=option_dict['software_root'], instance_root=option_dict['instance_root'], master_url=option_dict['master_url'], computer_id=option_dict['computer_id'], supervisord_socket=option_dict['supervisord_socket'], supervisord_configuration_path=option_dict[ 'supervisord_configuration_path'], key_file=key_file, cert_file=cert_file, master_ca_file=master_ca_file, certificate_repository_path=certificate_repository_path, signature_private_key_file=signature_private_key_file, signature_certificate_list=signature_certificate_list, download_binary_cache_url=\ option_dict.get('download-binary-cache-url', None), upload_binary_cache_url=\ option_dict.get('upload-binary-cache-url', None), download_from_binary_cache_url_blacklist=\ option_dict.get('download-from-binary-cache-url-blacklist', []), upload_to_binary_cache_url_blacklist=\ option_dict.get('upload-to-binary-cache-url-blacklist', []), upload_cache_url=option_dict.get('upload-cache-url', None), download_binary_dir_url=\ option_dict.get('download-binary-dir-url', None), upload_binary_dir_url=\ option_dict.get('upload-binary-dir-url', None), upload_dir_url=option_dict.get('upload-dir-url', None), buildout=option_dict.get('buildout'), promise_timeout=option_dict['promise_timeout'], shacache_cert_file=option_dict.get('shacache-cert-file', None), shacache_key_file=option_dict.get('shacache-key-file', None), shadir_cert_file=option_dict.get('shadir-cert-file', None), shadir_key_file=option_dict.get('shadir-key-file', None), develop=option_dict.get('develop', False), software_release_filter_list=option_dict.get('only-sr', # Try to fetch from deprecated argument option_dict.get('only_sr', None)), computer_partition_filter_list=option_dict.get('only-cp', # Try to fetch from deprecated argument option_dict.get('only_cp', None)), force_periodicity = option_dict.get('force_periodicity', False), maximum_periodicity = option_dict.get('maximum_periodicity', 86400), ), option_dict]) def realRun(argument_tuple, method_list): slapgrid_object, option_dict = \ parseArgumentTupleAndReturnSlapgridObject(*argument_tuple) pidfile = option_dict.get('pidfile') if pidfile: setRunning(pidfile) try: failed = False failed_promise = False for method in method_list: # Quite complicated way to figure out if everything went fine return_value = getattr(slapgrid_object, method)() if return_value == SLAPGRID_FAIL: failed = True if return_value == SLAPGRID_PROMISE_FAIL: failed_promise = True finally: if pidfile: setFinished(pidfile) if failed: sys.exit(SLAPGRID_FAIL) if failed_promise: sys.exit(SLAPGRID_PROMISE_FAIL) sys.exit(SLAPGRID_SUCCESS) def run(*argument_tuple): """Hooks for generic entry point to proces Software Releases (sr), Computer Partitions (cp) and Usage Reports (ur) Will run one by one each task (sr, cp, ur). If specified, will run in the user wanted order. """ realRun(argument_tuple, ['processSoftwareReleaseList', 'processComputerPartitionList', 'agregateAndSendUsage']) def runSoftwareRelease(*argument_tuple): """Hook for entry point to process Software Releases only """ realRun(argument_tuple, ['processSoftwareReleaseList']) def runComputerPartition(*argument_tuple): """Hook for entry point to process Computer Partitions only """ realRun(argument_tuple, ['processComputerPartitionList']) def runUsageReport(*argument_tuple): """Hook for entry point to process Usage Reports only """ realRun(argument_tuple, ['agregateAndSendUsage']) class Slapgrid(object): """ Main class for SlapGrid. Fetches and processes informations from master server and pushes usage information to master server. """ class PromiseError(Exception): pass def __init__(self, software_root, instance_root, master_url, computer_id, supervisord_socket, supervisord_configuration_path, buildout, force_periodicity=False, maximum_periodicity=86400, key_file=None, cert_file=None, signature_private_key_file=None, signature_certificate_list=None, download_binary_cache_url=None, upload_binary_cache_url=None, download_from_binary_cache_url_blacklist=None, upload_to_binary_cache_url_blacklist=None, upload_cache_url=None, download_binary_dir_url=None, upload_binary_dir_url=None, upload_dir_url=None, master_ca_file=None, certificate_repository_path=None, promise_timeout=3, shacache_cert_file=None, shacache_key_file=None, shadir_cert_file=None, shadir_key_file=None, develop=False, software_release_filter_list=None, computer_partition_filter_list=None, ): """Makes easy initialisation of class parameters""" # Parses arguments self.software_root = os.path.abspath(software_root) self.instance_root = os.path.abspath(instance_root) self.master_url = master_url self.computer_id = computer_id self.supervisord_socket = supervisord_socket self.supervisord_configuration_path = supervisord_configuration_path self.key_file = key_file self.cert_file = cert_file self.master_ca_file = master_ca_file self.certificate_repository_path = certificate_repository_path self.signature_private_key_file = signature_private_key_file self.signature_certificate_list = signature_certificate_list self.download_binary_cache_url = download_binary_cache_url self.upload_binary_cache_url = upload_binary_cache_url self.download_from_binary_cache_url_blacklist = \ download_from_binary_cache_url_blacklist self.upload_to_binary_cache_url_blacklist = \ upload_to_binary_cache_url_blacklist self.upload_cache_url = upload_cache_url self.download_binary_dir_url = download_binary_dir_url self.upload_binary_dir_url = upload_binary_dir_url self.upload_dir_url = upload_dir_url self.shacache_cert_file = shacache_cert_file self.shacache_key_file = shacache_key_file self.shadir_cert_file = shadir_cert_file self.shadir_key_file = shadir_key_file # Configures logger self.logger = logging.getLogger('Slapgrid') # Creates objects from slap module self.slap = slap.slap() self.slap.initializeConnection(self.master_url, key_file=self.key_file, cert_file=self.cert_file, master_ca_file=self.master_ca_file) self.computer = self.slap.registerComputer(self.computer_id) # Defines all needed paths self.instance_etc_directory = os.path.join(self.instance_root, 'etc') self.supervisord_configuration_directory = \ os.path.join(self.instance_etc_directory, 'supervisord.conf.d') self.buildout = buildout self.promise_timeout = promise_timeout self.develop = develop if software_release_filter_list is not None: self.software_release_filter_list = \ software_release_filter_list.split(",") else: self.software_release_filter_list = [] self.computer_partition_filter_list = [] if computer_partition_filter_list is not None: self.computer_partition_filter_list = \ computer_partition_filter_list.split(",") self.maximum_periodicity = maximum_periodicity self.force_periodicity = force_periodicity def getWatchdogLine(self): invocation_list = [WATCHDOG_PATH] invocation_list.append("--master-url '%s' " % self.master_url) if self.certificate_repository_path is not None: invocation_list.append("--certificate-repository-path '%s'" \ % self.certificate_repository_path) invocation_list.append("--computer-id '%s'" % self.computer_id) return ' '.join(invocation_list) def checkEnvironmentAndCreateStructure(self): """Checks for software_root and instance_root existence, then creates needed files and directories. """ # Checks for software_root and instance_root existence if not os.path.isdir(self.software_root): error = "%s does not exist." % self.software_root raise OSError(error) if not os.path.isdir(self.instance_root): error = "%s does not exist." % self.instance_root raise OSError(error) # Creates everything needed try: # Creates instance_root structure createPrivateDirectory(self.instance_etc_directory) createPrivateDirectory(os.path.join(self.instance_root, 'var')) createPrivateDirectory(os.path.join(self.instance_root, 'var', 'log')) createPrivateDirectory(os.path.join(self.instance_root, 'var', 'run')) createPrivateDirectory(self.supervisord_configuration_directory) # Creates supervisord configuration updateFile(self.supervisord_configuration_path, pkg_resources.resource_stream(__name__, 'templates/supervisord.conf.in').read() % dict( supervisord_configuration_directory=\ self.supervisord_configuration_directory, supervisord_socket=os.path.abspath(self.supervisord_socket), supervisord_loglevel='info', supervisord_logfile=os.path.abspath(os.path.join( self.instance_root, 'var', 'log', 'supervisord.log')), supervisord_logfile_maxbytes='50MB', supervisord_nodaemon='false', supervisord_pidfile=os.path.abspath(os.path.join( self.instance_root, 'var', 'run', 'supervisord.pid')), supervisord_logfile_backups='10', watchdog_command = self.getWatchdogLine(), )) except (WrongPermissionError, PathDoesNotExistError) as error: raise error def getComputerPartitionList(self): try: computer_partition_list = self.computer.getComputerPartitionList() except socket.error as error: self.logger.fatal(error) raise return computer_partition_list def processSoftwareReleaseList(self): """Will process each Software Release. """ self.checkEnvironmentAndCreateStructure() logger = logging.getLogger('SoftwareReleases') logger.info("Processing software releases...") # Boolean to know if every instance has correctly been deployed clean_run = True for software_release in self.computer.getSoftwareReleaseList(): state = software_release.getState() try: software_release_uri = software_release.getURI() url_hash = md5(software_release_uri).hexdigest() software_path = os.path.join(self.software_root, url_hash) software = Software(url=software_release_uri, software_root=self.software_root, buildout=self.buildout, signature_private_key_file=self.signature_private_key_file, signature_certificate_list=self.signature_certificate_list, download_binary_cache_url=self.download_binary_cache_url, upload_binary_cache_url=self.upload_binary_cache_url, download_from_binary_cache_url_blacklist=\ self.download_from_binary_cache_url_blacklist, upload_to_binary_cache_url_blacklist=\ self.upload_to_binary_cache_url_blacklist, upload_cache_url=self.upload_cache_url, download_binary_dir_url=self.download_binary_dir_url, upload_binary_dir_url=self.upload_binary_dir_url, upload_dir_url=self.upload_dir_url, shacache_cert_file=self.shacache_cert_file, shacache_key_file=self.shacache_key_file, shadir_cert_file=self.shadir_cert_file, shadir_key_file=self.shadir_key_file) if state == 'available': completed_tag = os.path.join(software_path, '.completed') if self.develop or (not os.path.exists(completed_tag) and \ len(self.software_release_filter_list) == 0) or \ url_hash in self.software_release_filter_list or \ url_hash in (md5(uri).hexdigest() for uri in self.software_release_filter_list): try: software_release.building() except NotFoundError: pass software.install() file_descriptor = open(completed_tag, 'w') file_descriptor.write(time.asctime()) file_descriptor.close() elif state == 'destroyed': if os.path.exists(software_path): logger.info('Destroying %r...' % software_release_uri) software.destroy() logger.info('Destroyed %r.' % software_release_uri) # Send log before exiting except (SystemExit, KeyboardInterrupt): exception = traceback.format_exc() software_release.error(exception) raise # Buildout failed: send log but don't print it to output (already done) except BuildoutFailedError as exception: clean_run = False try: software_release.error(exception) except (SystemExit, KeyboardInterrupt): raise except Exception: exception = traceback.format_exc() logger.error('Problem during reporting error, continuing:\n' + exception) # For everything else: log it, send it, continue. except Exception: exception = traceback.format_exc() logger.error(exception) software_release.error(exception) clean_run = False else: if state == 'available': try: software_release.available() except NotFoundError: pass elif state == 'destroyed': try: software_release.destroyed() except (NotFoundError, ServerError): print traceback.format_exc() logger.info("Finished software releases.") # Return success value if not clean_run: return SLAPGRID_FAIL return SLAPGRID_SUCCESS def _launchSupervisord(self): launchSupervisord(self.supervisord_socket, self.supervisord_configuration_path) def _checkPromises(self, computer_partition): self.logger.info("Checking promises...") instance_path = os.path.join(self.instance_root, computer_partition.getId()) uid, gid = None, None stat_info = os.stat(instance_path) #stat sys call to get statistics informations uid = stat_info.st_uid gid = stat_info.st_gid promise_present = False # Get the list of promises promise_dir = os.path.join(instance_path, 'etc', 'promise') if os.path.exists(promise_dir) and os.path.isdir(promise_dir): cwd = instance_path promises_list = os.listdir(promise_dir) # Check whether every promise is kept for promise in promises_list: promise_present = True command = [os.path.join(promise_dir, promise)] promise = os.path.basename(command[0]) self.logger.info("Checking promise %r.", promise) kw = dict(stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE) process_handler = subprocess.Popen(command, preexec_fn=lambda: dropPrivileges(uid, gid), cwd=cwd, env={}, **kw) process_handler.stdin.flush() process_handler.stdin.close() process_handler.stdin = None time.sleep(self.promise_timeout) if process_handler.poll() is None: process_handler.terminate() raise Slapgrid.PromiseError("The promise %r timed out" % promise) elif process_handler.poll() != 0: stderr = process_handler.communicate()[1] if stderr is None: stderr = 'No error output from %r.' % promise else: stderr = 'Promise %r:' % promise + stderr raise Slapgrid.PromiseError(stderr) if not promise_present: self.logger.info("No promise.") def processComputerPartition(self, computer_partition): """ Process a Computer Partition, depending on its state """ logger = logging.getLogger('ComputerPartitionProcessing') computer_partition_id = computer_partition.getId() # Sanity checks before processing # Those values should not be None or empty string or any falsy value if not computer_partition_id: raise ValueError('Computer Partition id is empty.') # Check if we defined explicit list of partitions to process. # If so, if current partition not in this list, skip. if len(self.computer_partition_filter_list) > 0 and \ (computer_partition_id not in self.computer_partition_filter_list): return logger.info('Processing Computer Partition %s...' % computer_partition_id) instance_path = os.path.join(self.instance_root, computer_partition_id) # Try to get partition timestamp (last modification date) timestamp_path = os.path.join(instance_path, '.timestamp') parameter_dict = computer_partition.getInstanceParameterDict() if 'timestamp' in parameter_dict: timestamp = parameter_dict['timestamp'] else: timestamp = None try: software_url = computer_partition.getSoftwareRelease().getURI() except NotFoundError: # Problem with instance: SR URI not set. # Try to process it anyway, it may need to be deleted. software_url = None try: software_path = os.path.join(self.software_root, getSoftwareUrlHash(software_url)) except TypeError: # Problem with instance: SR URI not set. # Try to process it anyway, it may need to be deleted. software_path = None periodicity = self.maximum_periodicity if software_path: # Get periodicity from periodicity file if not forced if not self.force_periodicity: periodicity_path = os.path.join(software_path, 'periodicity') if os.path.exists(periodicity_path): try: periodicity = int(open(periodicity_path).read()) except ValueError: os.remove(periodicity_path) exception = traceback.format_exc() logger.error(exception) # Check if timestamp from server is more recent than local one. # If not: it's not worth processing this partition (nothing has # changed). if computer_partition_id not in self.computer_partition_filter_list and \ (not self.develop) and os.path.exists(timestamp_path): old_timestamp = open(timestamp_path).read() last_runtime = int(os.path.getmtime(timestamp_path)) if timestamp: try: if int(timestamp) <= int(old_timestamp): # Check periodicity, i.e if periodicity is one day, partition # should be processed at least every day. # Only do it for "started" instances if computer_partition.getState() != COMPUTER_PARTITION_STARTED_STATE: return if int(time.time()) <= (last_runtime + periodicity): self.logger.info('Partition already up-to-date, skipping.') return except ValueError: os.remove(timestamp_path) exception = traceback.format_exc() logger.error(exception) local_partition = Partition( software_path=software_path, instance_path=instance_path, supervisord_partition_configuration_path=os.path.join( self.supervisord_configuration_directory, '%s.conf' % computer_partition_id), supervisord_socket=self.supervisord_socket, computer_partition=computer_partition, computer_id=self.computer_id, partition_id=computer_partition_id, server_url=self.master_url, software_release_url=software_url, certificate_repository_path=self.certificate_repository_path, buildout=self.buildout) computer_partition_state = computer_partition.getState() if computer_partition_state == COMPUTER_PARTITION_STARTED_STATE: local_partition.install() computer_partition.available() local_partition.start() self._checkPromises(computer_partition) computer_partition.started() elif computer_partition_state == COMPUTER_PARTITION_STOPPED_STATE: try: local_partition.install() computer_partition.available() except Exception: raise finally: # Instance has to be stopped even if buildout/reporting is wrong. local_partition.stop() computer_partition.stopped() elif computer_partition_state == COMPUTER_PARTITION_DESTROYED_STATE: local_partition.stop() try: computer_partition.stopped() except (SystemExit, KeyboardInterrupt): exception = traceback.format_exc() computer_partition.error(exception) raise except Exception: pass else: error_string = "Computer Partition %r has unsupported state: %s" % \ (computer_partition_id, computer_partition_state) computer_partition.error(error_string) raise NotImplementedError(error_string) # If partition has been successfully processed, write timestamp if timestamp: timestamp_path = os.path.join(instance_path, '.timestamp') open(timestamp_path, 'w').write(timestamp) def FilterComputerPartitionList(self, computer_partition_list): """ Try to filter valid partitions to be processed from free partitions. """ logger = logging.getLogger('ComputerPartitionProcessing') filtered_computer_partition_list = [] for computer_partition in computer_partition_list: try: computer_partition_path = os.path.join(self.instance_root, computer_partition.getId()) if not os.path.exists(computer_partition_path): raise NotFoundError('Partition directory %s does not exist.' % computer_partition_path) # Check state of partition. If it is in "destroyed" state, check if it # partition is actually installed in the Computer or if it is "free" # partition # XXX-Cedric: Temporary AND ugly solution to check if an instance # is in the partition. Dangerous because not 100% sure it is empty computer_partition_state = computer_partition.getState() if computer_partition_state == COMPUTER_PARTITION_DESTROYED_STATE and \ os.listdir(computer_partition_path) == []: continue # If partition has no SR: skip it. try: os.path.join(self.software_root, getSoftwareUrlHash( computer_partition.getSoftwareRelease().getURI())) except (NotFoundError, TypeError): # This is surely free partition. Check it... if os.listdir(computer_partition_path) == []: continue # Everything seems fine filtered_computer_partition_list.append(computer_partition) # XXX-Cedric: factor all this error handling # Send log before exiting except (SystemExit, KeyboardInterrupt): exception = traceback.format_exc() computer_partition.error(exception) raise # Buildout failed: send log but don't print it to output (already done) except BuildoutFailedError, exception: try: computer_partition.error(exception) except (SystemExit, KeyboardInterrupt): raise except Exception: exception = traceback.format_exc() logger.error('Problem during reporting error, continuing:\n' + exception) # For everything else: log it, send it, continue. except Exception as exception: logger.error(traceback.format_exc()) try: computer_partition.error(exception) except (SystemExit, KeyboardInterrupt): raise except Exception: exception = traceback.format_exc() logger.error('Problem during reporting error, continuing:\n' + exception) return filtered_computer_partition_list def processComputerPartitionList(self): """ Will start supervisord and process each Computer Partition. """ logger = logging.getLogger('ComputerPartitionProcessing') logger.info('Processing computer partitions...') # Prepares environment self.checkEnvironmentAndCreateStructure() self._launchSupervisord() # Boolean to know if every instance has correctly been deployed clean_run = True # Boolean to know if every promises correctly passed clean_run_promise = True # Filter all dummy / empty partitions computer_partition_list = self.FilterComputerPartitionList( self.getComputerPartitionList()) for computer_partition in computer_partition_list: # Nothing should raise outside of the current loop iteration, so that # even if something is terribly wrong while processing an instance, it # won't prevent processing other ones. try: # Process the partition itself self.processComputerPartition(computer_partition) # Send log before exiting except (SystemExit, KeyboardInterrupt): exception = traceback.format_exc() computer_partition.error(exception) raise except Slapgrid.PromiseError as exception: clean_run_promise = False try: logger.error(exception) computer_partition.error(exception) except (SystemExit, KeyboardInterrupt): raise except Exception: exception = traceback.format_exc() logger.error('Problem during reporting error, continuing:\n' + exception) # Buildout failed: send log but don't print it to output (already done) except BuildoutFailedError, exception: clean_run = False try: computer_partition.error(exception) except (SystemExit, KeyboardInterrupt): raise except Exception: exception = traceback.format_exc() logger.error('Problem during reporting error, continuing:\n' + exception) # For everything else: log it, send it, continue. except Exception as exception: clean_run = False logger.error(traceback.format_exc()) try: computer_partition.error(exception) except (SystemExit, KeyboardInterrupt): raise except Exception: exception = traceback.format_exc() logger.error('Problem during reporting error, continuing:\n' + exception) logger.info("Finished computer partitions.") # Return success value if not clean_run: return SLAPGRID_FAIL if not clean_run_promise: return SLAPGRID_PROMISE_FAIL return SLAPGRID_SUCCESS def validateXML(self, to_be_validated, xsd_model): """Validates a given xml file""" logger = logging.getLogger('XMLValidating') #We retrieve the xsd model xsd_model = StringIO.StringIO(xsd_model) xmlschema_doc = etree.parse(xsd_model) xmlschema = etree.XMLSchema(xmlschema_doc) try: document = etree.fromstring(to_be_validated) except (etree.XMLSyntaxError, etree.DocumentInvalid) as e: logger.info('Failed to parse this XML report : %s\n%s' % \ (to_be_validated, _formatXMLError(e))) logger.error(_formatXMLError(e)) return False if xmlschema.validate(document): return True return False def asXML(self, computer_partition_usage_list): """Generates a XML report from computer partition usage list """ xml = ['', '', '', 'Resource consumptions', '', '%s' % time.strftime("%Y-%m-%d at %H:%M:%S"), '%s' % self.computer_id, '', '', '', '', '', '', ''] for computer_partition_usage in computer_partition_usage_list: try: root = etree.fromstring(computer_partition_usage.usage) except UnicodeError: self.logger.info("Failed to read %s." % ( computer_partition_usage.usage)) self.logger.error(UnicodeError) raise "Failed to read %s." % (computer_partition_usage.usage) except (etree.XMLSyntaxError, etree.DocumentInvalid) as e: self.logger.info("Failed to parse %s." % (computer_partition_usage.usage)) self.logger.error(e) raise _formatXMLError(e) except Exception: raise "Failed to generate XML report." for movement in root.findall('movement'): xml.append('') for child in movement.getchildren(): if child.tag == "reference": xml.append('<%s>%s' % (child.tag, computer_partition_usage.getId(), child.tag)) else: xml.append('<%s>%s' % (child.tag, child.text, child.tag)) xml.append('') xml.append('') return ''.join(xml) def agregateAndSendUsage(self): """Will agregate usage from each Computer Partition. """ # Prepares environment self.checkEnvironmentAndCreateStructure() self._launchSupervisord() slap_computer_usage = self.slap.registerComputer(self.computer_id) computer_partition_usage_list = [] logger = logging.getLogger('UsageReporting') logger.info("Aggregating and sending usage reports...") #We retrieve XSD models try: computer_consumption_model = \ pkg_resources.resource_string( 'slapos.slap', 'doc/computer_consumption.xsd') except IOError: computer_consumption_model = \ pkg_resources.resource_string( __name__, '../../../../slapos/slap/doc/computer_consumption.xsd') try: partition_consumption_model = \ pkg_resources.resource_string( 'slapos.slap', 'doc/partition_consumption.xsd') except IOError: partition_consumption_model = \ pkg_resources.resource_string( __name__, '../../../../slapos/slap/doc/partition_consumption.xsd') clean_run = True # Loop on the different computer partitions computer_partition_list = self.FilterComputerPartitionList( slap_computer_usage.getComputerPartitionList()) for computer_partition in computer_partition_list: try: computer_partition_id = computer_partition.getId() #We want execute all the script in the report folder instance_path = os.path.join(self.instance_root, computer_partition.getId()) report_path = os.path.join(instance_path, 'etc', 'report') if os.path.isdir(report_path): script_list_to_run = os.listdir(report_path) else: script_list_to_run = [] #We now generate the pseudorandom name for the xml file # and we add it in the invocation_list f = tempfile.NamedTemporaryFile() name_xml = '%s.%s' % ('slapreport', os.path.basename(f.name)) path_to_slapreport = os.path.join(instance_path, 'var', 'xml_report', name_xml) failed_script_list = [] for script in script_list_to_run: invocation_list = [] invocation_list.append(os.path.join(instance_path, 'etc', 'report', script)) #We add the xml_file name in the invocation_list #f = tempfile.NamedTemporaryFile() #name_xml = '%s.%s' % ('slapreport', os.path.basename(f.name)) #path_to_slapreport = os.path.join(instance_path, 'var', name_xml) invocation_list.append(path_to_slapreport) #Dropping privileges uid, gid = None, None stat_info = os.stat(instance_path) #stat sys call to get statistics informations uid = stat_info.st_uid gid = stat_info.st_gid kw = dict(stdout=subprocess.PIPE, stderr=subprocess.STDOUT) process_handler = SlapPopen(invocation_list, preexec_fn=lambda: dropPrivileges(uid, gid), cwd=os.path.join(instance_path, 'etc', 'report'), env=None, **kw) if process_handler.returncode is None: process_handler.kill() if process_handler.returncode != 0: clean_run = False failed_script_list.append("Script %r failed." % script) logger.warning("Failed to run %r" % invocation_list) if len(failed_script_list): computer_partition.error('\n'.join(failed_script_list)) # Whatever happens, don't stop processing other instances except Exception: computer_partition_id = computer_partition.getId() exception = traceback.format_exc() issue = "Cannot run usage script(s) for %r: %s" % ( computer_partition_id, exception) logger.info(issue) #Now we loop through the different computer partitions to report report_usage_issue_cp_list = [] for computer_partition in computer_partition_list: try: filename_delete_list = [] computer_partition_id = computer_partition.getId() instance_path = os.path.join(self.instance_root, computer_partition_id) dir_reports = os.path.join(instance_path, 'var', 'xml_report') #The directory xml_report contain a number of files equal #to the number of software instance running inside the same partition if os.path.isdir(dir_reports): filename_list = os.listdir(dir_reports) else: filename_list = [] #logger.debug('name List %s' % filename_list) usage = '' for filename in filename_list: file_path = os.path.join(dir_reports, filename) if os.path.exists(file_path): usage_file = open(file_path, 'r') usage = usage_file.read() usage_file.close() #We check the validity of xml content of each reports if not self.validateXML(usage, partition_consumption_model): logger.info('WARNING: The XML file %s generated by slapreport is ' 'not valid - This report is left as is at %s where you can ' 'inspect what went wrong ' % (filename, dir_reports)) # Warn the SlapOS Master that a partition generates corrupted xml # report else: computer_partition_usage = self.slap.registerComputerPartition( self.computer_id, computer_partition_id) computer_partition_usage.setUsage(usage) computer_partition_usage_list.append(computer_partition_usage) filename_delete_list.append(filename) else: logger.debug("Usage report %r not found, ignored" % file_path) #After sending the aggregated file we remove all the valid xml reports for filename in filename_delete_list: os.remove(os.path.join(dir_reports, filename)) # Whatever happens, don't stop processing other instances except Exception: computer_partition_id = computer_partition.getId() exception = traceback.format_exc() issue = "Cannot run usage script(s) for %r: %s" % ( computer_partition_id, exception) logger.info(issue) for computer_partition_usage in computer_partition_usage_list: logger.info('computer_partition_usage_list : %s - %s' % \ (computer_partition_usage.usage, computer_partition_usage.getId())) #If there is, at least, one report if computer_partition_usage_list != []: try: #We generate the final XML report with asXML method computer_consumption = self.asXML(computer_partition_usage_list) logger.info('Final xml report : %s' % computer_consumption) #We test the XML report before sending it if self.validateXML(computer_consumption, computer_consumption_model): logger.info('XML file generated by asXML is valid') slap_computer_usage.reportUsage(computer_consumption) else: logger.info('XML file generated by asXML is not valid !') raise 'XML file generated by asXML is not valid !' except Exception: computer_partition_id = computer_partition.getId() exception = traceback.format_exc() issue = "Cannot report usage for %r: %s" % (computer_partition_id, exception) logger.info(issue) computer_partition.error(issue) report_usage_issue_cp_list.append(computer_partition_id) for computer_partition in computer_partition_list: if computer_partition.getState() == COMPUTER_PARTITION_DESTROYED_STATE: try: computer_partition_id = computer_partition.getId() try: software_url = computer_partition.getSoftwareRelease().getURI() software_path = os.path.join(self.software_root, getSoftwareUrlHash(software_url)) except (NotFoundError, TypeError): software_url = None software_path = None local_partition = Partition( software_path=software_path, instance_path=os.path.join(self.instance_root, computer_partition.getId()), supervisord_partition_configuration_path=os.path.join( self.supervisord_configuration_directory, '%s.conf' % computer_partition_id), supervisord_socket=self.supervisord_socket, computer_partition=computer_partition, computer_id=self.computer_id, partition_id=computer_partition_id, server_url=self.master_url, software_release_url=software_url, certificate_repository_path=self.certificate_repository_path, buildout=self.buildout, ) local_partition.stop() try: computer_partition.stopped() except (SystemExit, KeyboardInterrupt): exception = traceback.format_exc() computer_partition.error(exception) raise except Exception: pass if computer_partition.getId() in report_usage_issue_cp_list: logger.info('Ignoring destruction of %r, as not report usage was ' 'sent' % computer_partition.getId()) continue local_partition.destroy() except (SystemExit, KeyboardInterrupt): exception = traceback.format_exc() computer_partition.error(exception) raise except Exception: clean_run = False exception = traceback.format_exc() computer_partition.error(exception) logger.error(exception) try: computer_partition.destroyed() except slap.NotFoundError: logger.debug('Ignored slap error while trying to inform about ' 'destroying not fully configured Computer Partition %r' % computer_partition.getId()) except ServerError as server_error: logger.debug('Ignored server error while trying to inform about ' 'destroying Computer Partition %r. Error is :\n%r' % (computer_partition.getId(), server_error.args[0])) logger.info("Finished usage reports.") # Return success value if not clean_run: return SLAPGRID_FAIL return SLAPGRID_SUCCESS slapos.core-0.35.1/slapos/grid/SlapObject.py0000644000076600007660000006152512106730157026757 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- # vim: set et sts=2: ############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import logging import os import shutil import subprocess import pkg_resources import pwd import stat import tarfile import tempfile import textwrap import utils import xmlrpclib from supervisor import xmlrpc from slapos.slap.slap import NotFoundError from svcbackend import getSupervisorRPC from exception import BuildoutFailedError, WrongPermissionError, \ PathDoesNotExistError from networkcache import download_network_cached, upload_network_cached from watchdog import getWatchdogID REQUIRED_COMPUTER_PARTITION_PERMISSION = '0750' class Software(object): """This class is responsible for installing a software release""" def __init__(self, url, software_root, buildout, signature_private_key_file=None, signature_certificate_list=None, upload_cache_url=None, upload_dir_url=None, shacache_cert_file=None, shacache_key_file=None, shadir_cert_file=None, shadir_key_file=None, download_binary_cache_url=None, upload_binary_cache_url=None, download_binary_dir_url=None, upload_binary_dir_url=None, download_from_binary_cache_url_blacklist = [], upload_to_binary_cache_url_blacklist = []): """Initialisation of class parameters """ self.url = url self.software_root = software_root self.software_url_hash = utils.getSoftwareUrlHash(self.url) self.software_path = os.path.join(self.software_root, self.software_url_hash) self.buildout = buildout self.logger = logging.getLogger('BuildoutManager') self.signature_private_key_file = signature_private_key_file self.signature_certificate_list = signature_certificate_list self.upload_cache_url = upload_cache_url self.upload_dir_url = upload_dir_url self.shacache_cert_file = shacache_cert_file self.shacache_key_file = shacache_key_file self.shadir_cert_file = shadir_cert_file self.shadir_key_file = shadir_key_file self.download_binary_cache_url = download_binary_cache_url self.upload_binary_cache_url = upload_binary_cache_url self.download_binary_dir_url = download_binary_dir_url self.upload_binary_dir_url = upload_binary_dir_url self.download_from_binary_cache_url_blacklist = \ download_from_binary_cache_url_blacklist self.upload_to_binary_cache_url_blacklist = \ upload_to_binary_cache_url_blacklist def install(self): """ Fetches binary cache if possible. Installs from buildout otherwise. """ self.logger.info("Installing software release %s..." % self.url) cache_dir = tempfile.mkdtemp() try: tarpath = os.path.join(cache_dir, self.software_url_hash) # Check if we can download from cache if (not os.path.exists(self.software_path)) \ and download_network_cached( self.download_binary_cache_url, self.download_binary_dir_url, self.url, self.software_root, self.software_url_hash, tarpath, self.logger, self.signature_certificate_list, self.download_from_binary_cache_url_blacklist): tar = tarfile.open(tarpath) try: self.logger.info("Extracting archive of cached software release...") tar.extractall(path=self.software_root) finally: tar.close() else: self._install_from_buildout() # Upload to binary cache if possible blacklisted = False for url in self.upload_to_binary_cache_url_blacklist: if self.url.startswith(url): blacklisted = True self.logger.debug("Can't upload from binary cache: " "Software Release URL is blacklisted.") if (self.software_root and self.url and self.software_url_hash \ and self.upload_binary_cache_url \ and self.upload_binary_dir_url \ and not blacklisted): self.uploadSoftwareRelease(tarpath) finally: shutil.rmtree(cache_dir) def _install_from_buildout(self): """ Fetches buildout configuration from the server, run buildout with it. If it fails, we notify the server. """ root_stat_info = os.stat(self.software_root) os.environ = utils.getCleanEnvironment(pwd.getpwuid(root_stat_info.st_uid ).pw_dir) if not os.path.isdir(self.software_path): os.mkdir(self.software_path) extends_cache = tempfile.mkdtemp() if os.getuid() == 0: # In case when running as root copy ownership, to simplify logic for path in [self.software_path, extends_cache]: path_stat_info = os.stat(path) if root_stat_info.st_uid != path_stat_info.st_uid or\ root_stat_info.st_gid != path_stat_info.st_gid: os.chown(path, root_stat_info.st_uid, root_stat_info.st_gid) try: buildout_parameter_list = [ 'buildout:extends-cache=%s' % extends_cache, 'buildout:directory=%s' % self.software_path,] if self.signature_private_key_file or \ self.upload_cache_url or \ self.upload_dir_url is not None: buildout_parameter_list.append('buildout:networkcache-section=networkcache') for buildout_option, value in ( ('%ssignature-private-key-file=%s', self.signature_private_key_file), ('%supload-cache-url=%s', self.upload_cache_url), ('%supload-dir-url=%s', self.upload_dir_url), ('%sshacache-cert-file=%s', self.shacache_cert_file), ('%sshacache-key-file=%s', self.shacache_key_file), ('%sshadir-cert-file=%s', self.shadir_cert_file), ('%sshadir-key-file=%s', self.shadir_key_file), ): if value: buildout_parameter_list.append( \ buildout_option % ('networkcache:', value)) buildout_cfg = os.path.join(self.software_path, 'buildout.cfg') self.createProfileIfMissing(buildout_cfg, self.url) buildout_parameter_list.extend(['-c', buildout_cfg]) utils.bootstrapBuildout(self.software_path, self.buildout, additional_buildout_parametr_list=buildout_parameter_list) utils.launchBuildout(self.software_path, os.path.join(self.software_path, 'bin', 'buildout'), additional_buildout_parametr_list=buildout_parameter_list) finally: shutil.rmtree(extends_cache) def createProfileIfMissing(self, buildout_cfg, url): root_stat_info = os.stat(self.software_root) if not os.path.exists(buildout_cfg): with open(buildout_cfg, 'wb') as fout: fout.write(textwrap.dedent("""\ # Created by slapgrid. extends {url} # but you can change it for development purposes. [buildout] extends = {url} """.format(url=url))) os.chown(buildout_cfg, root_stat_info.st_uid, root_stat_info.st_gid) def uploadSoftwareRelease(self, tarpath): """ Try to tar and upload an installed Software Release. """ self.logger.info("Creating archive of software release...") tar = tarfile.open(tarpath, "w:gz") try: tar.add(self.software_path, arcname=self.software_url_hash) finally: tar.close() self.logger.info("Trying to upload archive of software release...") upload_network_cached( self.software_root, self.url, self.software_url_hash, self.upload_binary_cache_url, self.upload_binary_dir_url, tarpath, self.logger, self.signature_private_key_file, self.shacache_cert_file, self.shacache_key_file, self.shadir_cert_file, self.shadir_key_file) def destroy(self): """Removes software release.""" def retry(func, path, exc): # inspired by slapos.buildout hard remover if func == os.path.islink: os.unlink(path) else: os.chmod(path, 0600) func(path) try: if os.path.exists(self.software_path): self.logger.info('Removing path %r' % self.software_path) shutil.rmtree(self.software_path, onerror=retry) else: self.logger.info('Path %r does not exists, no need to remove.' % self.software_path) except IOError as error: error_string = "I/O error while removing software (%s): %s" % (self.url, error) raise IOError(error_string) class Partition(object): """This class is responsible of the installation of an instance """ def __init__(self, software_path, instance_path, supervisord_partition_configuration_path, supervisord_socket, computer_partition, computer_id, partition_id, server_url, software_release_url, buildout, certificate_repository_path=None, ): """Initialisation of class parameters""" self.buildout = buildout self.software_path = software_path self.instance_path = instance_path self.run_path = os.path.join(self.instance_path, 'etc', 'run') self.service_path = os.path.join(self.instance_path, 'etc', 'service') self.supervisord_partition_configuration_path = \ supervisord_partition_configuration_path self.supervisord_socket = supervisord_socket self.computer_partition = computer_partition self.logger = logging.getLogger('Partition') self.computer_id = computer_id self.partition_id = partition_id self.server_url = server_url self.software_release_url = software_release_url self.key_file = '' self.cert_file = '' if certificate_repository_path is not None: self.key_file = os.path.join(certificate_repository_path, self.partition_id + '.key') self.cert_file = os.path.join(certificate_repository_path, self.partition_id + '.crt') self._updateCertificate() def _updateCertificate(self): if not os.path.exists(self.key_file) or \ not os.path.exists(self.cert_file): self.logger.info('Certificate and key not found, downloading to %r and ' '%r' % (self.cert_file, self.key_file)) try: partition_certificate = self.computer_partition.getCertificate() except NotFoundError: raise NotFoundError('Partition %s is not known from SlapOS Master.' % \ self.partition_id) open(self.key_file, 'w').write(partition_certificate['key']) open(self.cert_file, 'w').write(partition_certificate['certificate']) for f in [self.key_file, self.cert_file]: os.chmod(f, 0400) os.chown(f, *self.getUserGroupId()) def getUserGroupId(self): """Returns tuple of (uid, gid) of partition""" stat_info = os.stat(self.instance_path) uid = stat_info.st_uid gid = stat_info.st_gid return (uid, gid) def addServiceToGroup(self, partition_id, runner_list, path, extension = ''): uid, gid = self.getUserGroupId() program_partition_template = pkg_resources.resource_stream(__name__, 'templates/program_partition_supervisord.conf.in').read() for runner in runner_list: self.partition_supervisor_configuration += '\n' + \ program_partition_template % dict( program_id='_'.join([partition_id, runner]), program_directory=self.instance_path, program_command=os.path.join(path, runner), program_name=runner+extension, instance_path=self.instance_path, user_id=uid, group_id=gid, # As supervisord has no environment to inherit, setup a minimalistic one HOME=pwd.getpwuid(uid).pw_dir, USER=pwd.getpwuid(uid).pw_name, ) def updateSymlink(self, sr_symlink, software_path): if os.path.lexists(sr_symlink): if not os.path.islink(sr_symlink): self.logger.debug('Not a symlink: %s, has been ignored' % (sr_symlink)) return os.unlink(sr_symlink) os.symlink(software_path, sr_symlink) os.lchown(sr_symlink, *self.getUserGroupId()) def install(self): """ Creates configuration file from template in software_path, then installs the software partition with the help of buildout """ self.logger.info("Installing Computer Partition %s..." \ % self.computer_partition.getId()) # Checks existence and permissions of Partition directory # Note : Partitions have to be created and configured before running slapgrid if not os.path.isdir(self.instance_path): raise PathDoesNotExistError('Please create partition directory %s' % self.instance_path) sr_symlink = os.path.join(self.instance_path, 'software_release') self.updateSymlink(sr_symlink, self.software_path) instance_stat_info = os.stat(self.instance_path) permission = oct(stat.S_IMODE(instance_stat_info.st_mode)) if permission != REQUIRED_COMPUTER_PARTITION_PERMISSION: raise WrongPermissionError('Wrong permissions in %s : actual ' \ 'permissions are : %s, wanted ' \ 'are %s' % (self.instance_path, permission, REQUIRED_COMPUTER_PARTITION_PERMISSION)) os.environ = utils.getCleanEnvironment(pwd.getpwuid( instance_stat_info.st_uid).pw_dir) # Generates buildout part from template template_location = os.path.join(self.software_path, 'instance.cfg') # Backward compatibility: "instance.cfg" file was named "template.cfg". if not os.path.exists(template_location): template_location = os.path.join(self.software_path, 'template.cfg') config_location = os.path.join(self.instance_path, 'buildout.cfg') self.logger.debug("Copying %r to %r" % (template_location, config_location)) try: shutil.copy(template_location, config_location) except IOError, e: # Template not found on SR, we notify user. raise IOError('Software Release %s is not correctly installed.\n' '%s' % (self.software_release_url, e)) # fill generated buildout with additional information buildout_text = open(config_location).read() buildout_text += '\n\n' + pkg_resources.resource_string(__name__, 'templates/buildout-tail.cfg.in') % dict( computer_id=self.computer_id, partition_id=self.partition_id, server_url=self.server_url, software_release_url=self.software_release_url, key_file=self.key_file, cert_file=self.cert_file ) open(config_location, 'w').write(buildout_text) os.chmod(config_location, 0640) # Try to find the best possible buildout: # *) if software_root/bin/bootstrap exists use this one to bootstrap # locally # *) as last resort fallback to buildout binary from software_path bootstrap_candidate_dir = os.path.abspath(os.path.join(self.software_path, 'bin')) if os.path.isdir(bootstrap_candidate_dir): bootstrap_candidate_list = [q for q in os.listdir(bootstrap_candidate_dir) if q.startswith('bootstrap')] else: bootstrap_candidate_list = [] uid, gid = self.getUserGroupId() os.chown(config_location, -1, int(gid)) if len(bootstrap_candidate_list) == 0: buildout_binary = os.path.join(self.software_path, 'bin', 'buildout') self.logger.warning("Falling back to default buildout %r" % buildout_binary) else: if len(bootstrap_candidate_list) != 1: raise ValueError('More than one bootstrap candidate found.') # Reads uid/gid of path, launches buildout with thoses privileges bootstrap_file = os.path.abspath(os.path.join(bootstrap_candidate_dir, bootstrap_candidate_list[0])) first_line = open(bootstrap_file, 'r').readline() invocation_list = [] if first_line.startswith('#!'): invocation_list = first_line[2:].split() invocation_list.append(bootstrap_file) self.logger.debug('Invoking %r in %r' % (' '.join(invocation_list), self.instance_path)) process_handler = utils.SlapPopen(invocation_list, preexec_fn=lambda: utils.dropPrivileges(uid, gid), cwd=self.instance_path, env=utils.getCleanEnvironment(pwd.getpwuid(uid).pw_dir), stdout=subprocess.PIPE, stderr=subprocess.STDOUT) if process_handler.returncode is None or process_handler.returncode != 0: message = 'Failed to bootstrap buildout in %r.' % (self.instance_path) self.logger.error(message) raise BuildoutFailedError('%s:\n%s\n' % (message, process_handler.output)) buildout_binary = os.path.join(self.instance_path, 'sbin', 'buildout') if not os.path.exists(buildout_binary): # use own buildout generation utils.bootstrapBuildout(self.instance_path, self.buildout, ['buildout:bin-directory=%s'% os.path.join(self.instance_path, 'sbin')]) buildout_binary = os.path.join(self.instance_path, 'sbin', 'buildout') # Launches buildout utils.launchBuildout(self.instance_path, buildout_binary) # Generates supervisord configuration file from template self.logger.info("Generating supervisord config file from template...") # check if CP/etc/run exists and it is a directory # iterate over each file in CP/etc/run # iterate over each file in CP/etc/service adding WatchdogID to their name # if at least one is not 0750 raise -- partition has something funny runner_list = [] service_list = [] if os.path.exists(self.run_path): if os.path.isdir(self.run_path): runner_list = os.listdir(self.run_path) if os.path.exists(self.service_path): if os.path.isdir(self.service_path): service_list = os.listdir(self.service_path) if len(runner_list) == 0 and len(service_list) == 0: self.logger.warning('No runners nor services found for partition %r' % self.partition_id) if os.path.exists(self.supervisord_partition_configuration_path): os.unlink(self.supervisord_partition_configuration_path) else: partition_id = self.computer_partition.getId() group_partition_template = pkg_resources.resource_stream(__name__, 'templates/group_partition_supervisord.conf.in').read() self.partition_supervisor_configuration = group_partition_template % dict( instance_id=partition_id, program_list=','.join(['_'.join([partition_id, runner]) for runner in runner_list+service_list])) # Same method to add to service and run self.addServiceToGroup(partition_id, runner_list,self.run_path) self.addServiceToGroup(partition_id, service_list,self.service_path, extension=getWatchdogID()) utils.updateFile(self.supervisord_partition_configuration_path, self.partition_supervisor_configuration) self.updateSupervisor() def start(self): """Asks supervisord to start the instance. If this instance is not installed, we install it. """ supervisor = self.getSupervisorRPC() partition_id = self.computer_partition.getId() try: supervisor.startProcessGroup(partition_id, False) except xmlrpclib.Fault, e: if e.faultString.startswith('BAD_NAME:'): self.logger.info("Nothing to start on %s..." % \ self.computer_partition.getId()) else: self.logger.info("Requested start of %s..." % self.computer_partition.getId()) def stop(self): """Asks supervisord to stop the instance.""" supervisor = self.getSupervisorRPC() partition_id = self.computer_partition.getId() try: supervisor.stopProcessGroup(partition_id, False) except xmlrpclib.Fault, e: if e.faultString.startswith('BAD_NAME:'): self.logger.info('Partition %s not known in supervisord, ignoring' % partition_id) else: self.logger.info("Requested stop of %s..." % self.computer_partition.getId()) def destroy(self): """Destroys the partition and makes it available for subsequent use." """ self.logger.info("Destroying Computer Partition %s..." \ % self.computer_partition.getId()) # Launches "destroy" binary if exists destroy_executable_location = os.path.join(self.instance_path, 'sbin', 'destroy') if os.path.exists(destroy_executable_location): uid, gid = self.getUserGroupId() self.logger.debug('Invoking %r' % destroy_executable_location) process_handler = utils.SlapPopen([destroy_executable_location], preexec_fn=lambda: utils.dropPrivileges(uid, gid), cwd=self.instance_path, env=utils.getCleanEnvironment(pwd.getpwuid(uid).pw_dir), stdout=subprocess.PIPE, stderr=subprocess.STDOUT) if process_handler.returncode is None or process_handler.returncode != 0: message = 'Failed to destroy Computer Partition in %r.' % \ self.instance_path self.logger.error(message) raise subprocess.CalledProcessError(message, process_handler.output) # Manually cleans what remains try: for f in [self.key_file, self.cert_file]: if f: if os.path.exists(f): os.unlink(f) # better to manually remove symlinks because rmtree might choke on them sr_symlink = os.path.join(self.instance_path, 'software_release') if os.path.islink(sr_symlink): os.unlink(sr_symlink) for root, dirs, file_list in os.walk(self.instance_path): for directory in dirs: shutil.rmtree(os.path.join(self.instance_path, directory)) for file in file_list: os.remove(os.path.join(self.instance_path, file)) if os.path.exists(self.supervisord_partition_configuration_path): os.remove(self.supervisord_partition_configuration_path) self.updateSupervisor() except IOError as error: error_string = "I/O error while freeing partition (%s): %s" \ % (self.instance_path, error) raise IOError(error_string) def fetchInformations(self): """Fetch usage informations with buildout, returns it. """ raise NotImplementedError def getSupervisorRPC(self): return getSupervisorRPC(self.supervisord_socket) def updateSupervisor(self): """Forces supervisord to reload its configuration""" # Note: This method shall wait for results from supervisord # In future it will be not needed, as update command # is going to be implemented on server side. self.logger.debug('Updating supervisord') supervisor = self.getSupervisorRPC() # took from supervisord.supervisorctl.do_update result = supervisor.reloadConfig() added, changed, removed = result[0] for gname in removed: results = supervisor.stopProcessGroup(gname) fails = [res for res in results if res['status'] == xmlrpc.Faults.FAILED] if fails: self.logger.warning('Problem while stopping process %r, will try later' % gname) else: self.logger.info('Stopped %r' % gname) supervisor.removeProcessGroup(gname) self.logger.info('Removed %r' % gname) for gname in changed: results = supervisor.stopProcessGroup(gname) self.logger.info('Stopped %r' % gname) supervisor.removeProcessGroup(gname) supervisor.addProcessGroup(gname) self.logger.info('Updated %r' % gname) for gname in added: supervisor.addProcessGroup(gname) self.logger.info('Updated %r' % gname) self.logger.debug('Supervisord updated') slapos.core-0.35.1/slapos/grid/svcbackend.py0000644000076600007660000001466512074051043027031 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- # vim: set et sts=2: ############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## from supervisor import xmlrpc import time from utils import SlapPopen import logging import os import sys import xmlrpclib from optparse import OptionParser import ConfigParser import socket as socketlib import subprocess def getSupervisorRPC(socket): supervisor_transport = xmlrpc.SupervisorTransport('', '', 'unix://' + socket) server_proxy = xmlrpclib.ServerProxy('http://127.0.0.1', supervisor_transport) return getattr(server_proxy, 'supervisor') def launchSupervisord(socket, configuration_file): logger = logging.getLogger('SVCBackend') supervisor = getSupervisorRPC(socket) if os.path.exists(socket): trynum = 1 while trynum < 6: try: status = supervisor.getState() except xmlrpclib.Fault as e: if e.faultCode == 6 and e.faultString == 'SHUTDOWN_STATE': logger.info('Supervisor in shutdown procedure, will check again later.') trynum += 1 time.sleep(2 * trynum) except Exception: # In case if there is problem with connection, assume that supervisord # is not running and try to run it break else: if status['statename'] == 'RUNNING' and status['statecode'] == 1: logger.debug('Supervisord already running.') return elif status['statename'] == 'SHUTDOWN_STATE' and status['statecode'] == 6: logger.info('Supervisor in shutdown procedure, will check again later.') trynum += 1 time.sleep(2 * trynum) else: log_message = 'Unknown supervisord state %r. Will try to start.' % status logger.warning(log_message) break logger.info("Launching supervisord with clean environment.") # Extract python binary to prevent shebang size limit invocation_list = ["supervisord", '-c'] invocation_list.append("import sys ; sys.path=" + str(sys.path) + " ; import " "supervisor.supervisord ; sys.argv[1:1]=['-c','" + configuration_file + "'] ; supervisor.supervisord.main()") supervisord_popen = SlapPopen(invocation_list, env={}, executable=sys.executable, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) result = supervisord_popen.communicate()[0] if supervisord_popen.returncode == 0: logger.info('Supervisord command invoked with: %s' % result) try: default_timeout = socketlib.getdefaulttimeout() current_timeout = 1 trynum = 1 while trynum < 6: try: socketlib.setdefaulttimeout(current_timeout) status = supervisor.getState() if status['statename'] == 'RUNNING' and status['statecode'] == 1: return logger.warning('Wrong status name %(statename)r and code ' '%(statecode)r, trying again' % status) trynum += 1 except Exception: current_timeout = 5 * trynum trynum += 1 else: logger.info('Supervisord started correctly in try %s.' % trynum) return logger.warning('Issue while checking supervisord.') finally: socketlib.setdefaulttimeout(default_timeout) else: log_message = 'Supervisord unknown problem: %s' % result logger.warning(log_message) def getOptionDict(*argument_tuple): usage = """ Typical usage: * %prog CONFIGURATION_FILE [arguments passed to supervisor] """.strip() parser = OptionParser(usage=usage) # Parses arguments if argument_tuple: (argument_option_instance, argument_list) = parser.parse_args(list(argument_tuple)) else: # No arguments given to entry point : we parse sys.argv. (argument_option_instance, argument_list) = parser.parse_args() if not argument_list: parser.error("Configuration file is obligatory. Consult documentation by " "calling with -h.") configuration_file = argument_list[0] if not os.path.exists(configuration_file): parser.error("Could not read configuration file : %s" % configuration_file) slapgrid_configuration = ConfigParser.SafeConfigParser() slapgrid_configuration.read(configuration_file) # Merges the two dictionnaries option_dict = dict(slapgrid_configuration.items("slapos")) # Supervisord configuration location option_dict.setdefault('supervisord_configuration_path', os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf')) # Supervisord socket option_dict.setdefault('supervisord_socket', os.path.join(option_dict['instance_root'], 'supervisord.socket')) return option_dict, argument_list[1:] def supervisorctl(*argument_tuple): option_dict, args = getOptionDict(*argument_tuple) import supervisor.supervisorctl launchSupervisord(option_dict['supervisord_socket'], option_dict['supervisord_configuration_path']) supervisor.supervisorctl.main(args=['-c', option_dict['supervisord_configuration_path']] + args) def supervisord(*argument_tuple): option_dict, _ = getOptionDict(*argument_tuple) launchSupervisord(option_dict['supervisord_socket'], option_dict['supervisord_configuration_path']) slapos.core-0.35.1/slapos/grid/templates/0000755000076600007660000000000012110425554026341 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/grid/templates/buildout-tail.cfg.in0000644000076600007660000000135212035516223032206 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# This is beginning of zc.builodout profile's tail added by slapgrid [buildout] # put buildout generated binaries in specific directory bin-directory = ${buildout:directory}/sbin # protect software and run parts offline offline = true [slap-connection] computer-id = %(computer_id)s partition-id = %(partition_id)s server-url = %(server_url)s software-release-url = %(software_release_url)s key-file = %(key_file)s cert-file = %(cert_file)s [slap_connection] # Kept for backward compatiblity computer_id = %(computer_id)s partition_id = %(partition_id)s server_url = %(server_url)s software_release_url = %(software_release_url)s key_file = %(key_file)s cert_file = %(cert_file)s # This is end of zc.builodout profile's tail added by slapgrid slapos.core-0.35.1/slapos/grid/templates/group_partition_supervisord.conf.in0000644000076600007660000000006212030324717035505 0ustar cedricdesaintmartincedricdesaintmartin00000000000000[group:%(instance_id)s] programs=%(program_list)s slapos.core-0.35.1/slapos/grid/templates/program_partition_supervisord.conf.in0000644000076600007660000000113512037271015036021 0ustar cedricdesaintmartincedricdesaintmartin00000000000000[program:%(program_id)s] directory=%(program_directory)s command=%(program_command)s process_name=%(program_name)s autostart=false autorestart=false startsecs=0 startretries=0 exitcodes=0 stopsignal=TERM stopwaitsecs=60 stopasgroup=true killasgroup=true user=%(user_id)s group=%(group_id)s serverurl=AUTO redirect_stderr=true stdout_logfile=%(instance_path)s/.%(program_id)s.log stdout_logfile_maxbytes=100KB stdout_logfile_backups=1 stderr_logfile=%(instance_path)s/.%(program_id)s.log stderr_logfile_maxbytes=100KB stderr_logfile_backups=1 environment=USER="%(USER)s",LOGNAME="%(USER)s",HOME="%(HOME)s" slapos.core-0.35.1/slapos/grid/templates/supervisord.conf.in0000644000076600007660000000121412073021643032177 0ustar cedricdesaintmartincedricdesaintmartin00000000000000[rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [include] files = %(supervisord_configuration_directory)s/*.conf [supervisorctl] serverurl = unix://%(supervisord_socket)s [supervisord] loglevel = %(supervisord_loglevel)s logfile = %(supervisord_logfile)s logfile_maxbytes = %(supervisord_logfile_maxbytes)s nodaemon = %(supervisord_nodaemon)s pidfile = %(supervisord_pidfile)s logfile-backups = %(supervisord_logfile_backups)s [unix_http_server] file=%(supervisord_socket)s chmod=0700 [eventlistener:watchdog] command=%(watchdog_command)s events=PROCESS_STATE_EXITED, PROCESS_STATE_FATAL slapos.core-0.35.1/slapos/grid/utils.py0000644000076600007660000003012412073300773026060 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import logging import hashlib import os import pkg_resources import stat import subprocess import sys import pwd import grp from exception import BuildoutFailedError, WrongPermissionError from hashlib import md5 # Such umask by default will create paths with full permission # for user, non writable by group and not accessible by others SAFE_UMASK = 027 PYTHON_ENVIRONMENT_REMOVE_LIST = [ 'PYTHONHOME', 'PYTHONPATH', 'PYTHONSTARTUP', 'PYTHONY2K', 'PYTHONOPTIMIZE', 'PYTHONDEBUG', 'PYTHONDONTWRITEBYTECODE', 'PYTHONINSPECT', 'PYTHONNOUSERSITE', 'PYTHONNOUSERSITE', 'PYTHONUNBUFFERED', 'PYTHONVERBOSE', ] SYSTEM_ENVIRONMENT_REMOVE_LIST = [ 'ENV', 'LOGNAME', 'TEMP', 'TMP', 'TMPDIR', 'USER', ] LOCALE_ENVIRONMENT_REMOVE_LIST = [ 'LANG', 'LANGUAGE', 'LC_ADDRESS', 'LC_COLLATE', 'LC_CTYPE', 'LC_IDENTIFICATION', 'LC_MEASUREMENT', 'LC_MESSAGES', 'LC_MONETARY', 'LC_NAME', 'LC_NUMERIC', 'LC_PAPER', 'LC_SOURCED', 'LC_TELEPHONE', 'LC_TIME', ] class SlapPopen(subprocess.Popen): """ Almost normal subprocess with greedish features and logging. Each line is logged "live", and self.output is a string containing the whole log. """ def __init__(self, *args, **kwargs): kwargs.update(stdin=subprocess.PIPE) subprocess.Popen.__init__(self, *args, **kwargs) self.stdin.flush() self.stdin.close() self.stdin = None logger = logging.getLogger('SlapProcessManager') # XXX-Cedric: this algorithm looks overkill for simple logging. output_lines = [] while True: line = self.stdout.readline() if line == '' and self.poll() != None: break output_lines.append(line) if line[-1:] == '\n': line = line[:-1] logger.info(line) self.output = ''.join(output_lines) def getSoftwareUrlHash(url): return md5(url).hexdigest() def getCleanEnvironment(home_path='/tmp'): logger = logging.getLogger('CleanEnvironment') changed_env = {} removed_env = [] env = os.environ.copy() # Clean python related environment variables for k in PYTHON_ENVIRONMENT_REMOVE_LIST + SYSTEM_ENVIRONMENT_REMOVE_LIST \ + LOCALE_ENVIRONMENT_REMOVE_LIST: old = env.pop(k, None) if old is not None: removed_env.append(k) changed_env['HOME'] = env['HOME'] = home_path for k in sorted(changed_env.iterkeys()): logger.debug('Overriden %s = %r' % (k, changed_env[k])) logger.debug('Removed from environment: %s' % ', '.join(sorted(removed_env))) return env def setRunning(pid_file): """Creates a pidfile. If a pidfile already exists, we exit""" logger = logging.getLogger('Slapgrid') if os.path.exists(pid_file): # Pid file is present try: pid = int(open(pid_file, 'r').readline()) except ValueError: pid = None # XXX This could use psutil library. if pid is not None and os.path.exists("/proc/%s" % pid): # In case process is present, ignore. logger.warning('New slapos process started, but another slapos ' 'process is aleady running with pid %s, exiting.' % pid) sys.exit(10) logger.info('Existing pid file %r was stale one, overwritten' % pid_file) # Start new process write_pid(pid_file) def setFinished(pid_file): try: os.remove(pid_file) except OSError: pass def write_pid(pid_file): logger = logging.getLogger('Slapgrid') pid = os.getpid() try: f = open(pid_file, 'w') f.write('%s' % pid) f.close() except (IOError, OSError): logger.critical('slapgrid could not write pidfile %s' % pid_file) raise def dropPrivileges(uid, gid): """Drop privileges to uid, gid if current uid is 0 Do tests to check if dropping was successful and that no system call is able to re-raise dropped privileges Does nothing in case if uid and gid are not 0 """ logger = logging.getLogger('dropPrivileges') # XXX-Cedric: remove format / just do a print, otherwise formatting is done # twice current_uid, current_gid = os.getuid(), os.getgid() if uid == 0 or gid == 0: raise OSError('Dropping privileges to uid = %r or ' \ 'gid = %r is too dangerous' % (uid, gid)) if not(current_uid == 0 and current_gid == 0): logger.debug('Running as uid = %r, gid = %r, dropping not needed and not ' 'possible' % (current_uid, current_gid)) return # drop privileges user_name = pwd.getpwuid(uid)[0] group_list = set([x.gr_gid for x in grp.getgrall() if user_name in x.gr_mem]) group_list.add(gid) os.initgroups(pwd.getpwuid(uid)[0], gid) os.setgid(gid) os.setuid(uid) # assert that privileges are dropped message_pre = 'After dropping to uid = %r and gid = %r ' \ 'and group_list = %s' % ( uid, gid, group_list) new_uid, new_gid, new_group_list = os.getuid(), os.getgid(), os.getgroups() if not (new_uid == uid and new_gid == gid and set(new_group_list) == group_list): raise OSError('%s new_uid = %r and new_gid = %r and ' \ 'new_group_list = %r which is fatal.' % (message_pre, new_uid, new_gid, new_group_list)) # assert that it is not possible to go back to running one try: try: os.setuid(current_uid) except OSError: try: os.setgid(current_gid) except OSError: try: os.setgroups([current_gid]) except OSError: raise except OSError: pass else: raise ValueError('%s it was possible to go back to uid = %r and gid = ' '%r which is fatal.' % (message_pre, current_uid, current_gid)) logger.debug('Succesfully dropped privileges to uid=%r gid=%r' % (uid, gid)) def bootstrapBuildout(path, buildout=None, additional_buildout_parametr_list=None): if additional_buildout_parametr_list is None: additional_buildout_parametr_list = [] logger = logging.getLogger('BuildoutManager') # Reads uid/gid of path, launches buildout with thoses privileges stat_info = os.stat(path) uid = stat_info.st_uid gid = stat_info.st_gid invocation_list = [sys.executable, '-S'] kw = dict() if buildout is not None: invocation_list.append(buildout) invocation_list.extend(additional_buildout_parametr_list) else: try: __import__('zc.buildout') except ImportError: logger.warning('Using old style bootstrap of included bootstrap file. ' 'Consider having zc.buildout available in search path.') invocation_list.append(pkg_resources.resource_filename(__name__, 'zc.buildout-bootstap.py')) invocation_list.extend(additional_buildout_parametr_list) else: # buildout is importable, so use this one invocation_list.extend(["-c", "import sys ; sys.path=" + str(sys.path) + " ; import zc.buildout.buildout ; sys.argv[1:1]=" + \ repr(additional_buildout_parametr_list + ['bootstrap']) + " ; " "zc.buildout.buildout.main()"]) if buildout is not None: invocation_list.append('bootstrap') try: umask = os.umask(SAFE_UMASK) logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK)) logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list), path)) kw.update(stdout=subprocess.PIPE, stderr=subprocess.STDOUT) process_handler = SlapPopen(invocation_list, preexec_fn=lambda: dropPrivileges(uid, gid), cwd=path, **kw) if process_handler.returncode is None or process_handler.returncode != 0: message = 'Failed to run buildout profile in directory %r' % (path) logger.error(message) raise BuildoutFailedError('%s:\n%s\n' % (message, process_handler.output)) except OSError as error: raise BuildoutFailedError(error) finally: old_umask = os.umask(umask) logger.debug('Restore umask from %03o to %03o' % (old_umask, umask)) def launchBuildout(path, buildout_binary, additional_buildout_parametr_list=None): """ Launches buildout.""" logger = logging.getLogger('BuildoutManager') if additional_buildout_parametr_list is None: additional_buildout_parametr_list = [] # Reads uid/gid of path, launches buildout with thoses privileges stat_info = os.stat(path) uid = stat_info.st_uid gid = stat_info.st_gid # Extract python binary to prevent shebang size limit file = open(buildout_binary, 'r') line = file.readline() file.close() invocation_list = [] if line.startswith('#!'): line = line[2:] # Prepares parameters for buildout invocation_list = line.split() + [buildout_binary] # Run buildout without reading user defaults invocation_list.append('-U') invocation_list.extend(additional_buildout_parametr_list) try: umask = os.umask(SAFE_UMASK) logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK)) logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list), path)) kw = dict(stdout=subprocess.PIPE, stderr=subprocess.STDOUT) process_handler = SlapPopen(invocation_list, preexec_fn=lambda: dropPrivileges(uid, gid), cwd=path, env=getCleanEnvironment(pwd.getpwuid(uid).pw_dir), **kw) if process_handler.returncode is None or process_handler.returncode != 0: message = 'Failed to run buildout profile in directory %r' % (path) logger.error(message) raise BuildoutFailedError('%s:\n%s\n' % (message, process_handler.output)) except OSError as error: raise BuildoutFailedError(error) finally: old_umask = os.umask(umask) logger.debug('Restore umask from %03o to %03o' % (old_umask, umask)) def updateFile(file_path, content, mode='0600'): """Creates an executable with "content" as content.""" altered = False if not (os.path.isfile(file_path)) or \ not(hashlib.md5(open(file_path).read()).digest() ==\ hashlib.md5(content).digest()): altered = True file_file = open(file_path, 'w') file_file.write(content) file_file.flush() file_file.close() os.chmod(file_path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC) if oct(stat.S_IMODE(os.stat(file_path).st_mode)) != mode: os.chmod(file_path, int(mode, 8)) altered = True return altered def updateExecutable(executable_path, content): """Creates an executable with "content" as content.""" return updateFile(executable_path, content, '0700') def createPrivateDirectory(path): """Creates directory belonging to root with umask 077""" if not os.path.isdir(path): os.mkdir(path) os.chmod(path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC) permission = oct(stat.S_IMODE(os.stat(path).st_mode)) if permission not in ('0700'): raise WrongPermissionError('Wrong permissions in %s ' \ ': is %s, should be 0700' % (path, permission)) slapos.core-0.35.1/slapos/grid/watchdog.py0000644000076600007660000001110112063614456026517 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import argparse import os.path import slapos.slap.slap import sys def getWatchdogID(): return "-on-watch" def parseArgumentTuple(): """Parses arguments either from command line, from method parameters or from config file. Then returns a new instance of slapgrid.Slapgrid with those parameters. Also returns the options dict and unused variable list, and configures logger. """ parser = argparse.ArgumentParser() parser.add_argument("--master-url", help="The master server URL. Mandatory.", required=True) parser.add_argument("--computer-id", help="The computer id defined in the server.", required=True) parser.add_argument("--certificate-repository-path", help="Path to partition certificates.", default=None) option = parser.parse_args() # Build option_dict option_dict = {} for argument_key, argument_value in vars(option).iteritems(): option_dict.update({argument_key: argument_value}) return option_dict class Watchdog(): process_state_events = ['PROCESS_STATE_EXITED', 'PROCESS_STATE_FATAL'] def __init__(self, option_dict): for option, value in option_dict.items(): setattr(self, option, value) self.stdin = sys.stdin self.stdout = sys.stdout self.stderr = sys.stderr self.slap = slapos.slap.slap() def initialize_connection(self, partition_id): cert_file = None key_file = None if self.certificate_repository_path is not None: cert_file = os.path.join(self.certificate_repository_path, "%s.crt" % partition_id) key_file = os.path.join(self.certificate_repository_path, "%s.key" % partition_id) self.slap.initializeConnection( slapgrid_uri=self.master_url, key_file=key_file, cert_file=cert_file) def write_stdout(self, s): self.stdout.write(s) self.stdout.flush() def write_stderr(self, s): self.stderr.write(s) self.stderr.flush() def run(self): while 1: self.write_stdout('READY\n') line = self.stdin.readline() # read header line from stdin headers = dict([x.split(':') for x in line.split()]) data = sys.stdin.read(int(headers['len'])) # read the event payload self.handle_event(headers, data) self.write_stdout('RESULT 2\nOK') # transition from READY to ACKNOWLEDGED def handle_event(self, headers, payload): if headers['eventname'] in self.process_state_events: payload_dict = dict([x.split(':') for x in payload.split()]) if getWatchdogID() in payload_dict['processname']: self.handle_process_state_change_event(headers, payload_dict) def handle_process_state_change_event(self, headers, payload_dict): partition_id = payload_dict['groupname'] self.initialize_connection(partition_id) partition = slapos.slap.ComputerPartition( computer_id=self.computer_id, connection_helper=self.slap._connection_helper, partition_id=partition_id) partition.bang("%s process in partition %s encountered a problem" % (payload_dict['processname'], partition_id)) def main(): watchdog = Watchdog(parseArgumentTuple()) watchdog.run() if __name__ == '__main__': main() slapos.core-0.35.1/slapos/grid/zc.buildout-bootstap.py0000644000076600007660000002347412073300773031025 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2006 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Bootstrap a buildout-based project Simply run this script in a directory containing a buildout.cfg. The script accepts buildout command-line options, so you can use the -c option to specify an alternate configuration file. """ import os, shutil, sys, tempfile, urllib, urllib2, subprocess from optparse import OptionParser if sys.platform == 'win32': def quote(c): if ' ' in c: return '"%s"' % c # work around spawn lamosity on windows else: return c else: quote = str # See zc.buildout.easy_install._has_broken_dash_S for motivation and comments. stdout, stderr = subprocess.Popen( [sys.executable, '-Sc', 'try:\n' ' import ConfigParser\n' 'except ImportError:\n' ' print 1\n' 'else:\n' ' print 0\n'], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() has_broken_dash_S = bool(int(stdout.strip())) # In order to be more robust in the face of system Pythons, we want to # run without site-packages loaded. This is somewhat tricky, in # particular because Python 2.6's distutils imports site, so starting # with the -S flag is not sufficient. However, we'll start with that: if not has_broken_dash_S and 'site' in sys.modules: # We will restart with python -S. args = sys.argv[:] args[0:0] = [sys.executable, '-S'] args = map(quote, args) os.execv(sys.executable, args) # Now we are running with -S. We'll get the clean sys.path, import site # because distutils will do it later, and then reset the path and clean # out any namespace packages from site-packages that might have been # loaded by .pth files. clean_path = sys.path[:] sys.path[:] = clean_path for k, v in sys.modules.items(): if k in ('setuptools', 'pkg_resources') or ( hasattr(v, '__path__') and len(v.__path__)==1 and not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))): # This is a namespace package. Remove it. sys.modules.pop(k) is_jython = sys.platform.startswith('java') setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py' distribute_source = 'http://python-distribute.org/distribute_setup.py' # parsing arguments def normalize_to_url(option, opt_str, value, parser): if value: if '://' not in value: # It doesn't smell like a URL. value = 'file://%s' % ( urllib.pathname2url( os.path.abspath(os.path.expanduser(value))),) if opt_str == '--download-base' and not value.endswith('/'): # Download base needs a trailing slash to make the world happy. value += '/' else: value = None name = opt_str[2:].replace('-', '_') setattr(parser.values, name, value) usage = '''\ [DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options] Bootstraps a buildout-based project. Simply run this script in a directory containing a buildout.cfg, using the Python that you want bin/buildout to use. Note that by using --setup-source and --download-base to point to local resources, you can keep this script from going over the network. ''' parser = OptionParser(usage=usage) parser.add_option("-v", "--version", dest="version", help="use a specific zc.buildout version") parser.add_option("-d", "--distribute", action="store_true", dest="use_distribute", default=False, help="Use Distribute rather than Setuptools.") parser.add_option("--setup-source", action="callback", dest="setup_source", callback=normalize_to_url, nargs=1, type="string", help=("Specify a URL or file location for the setup file. " "If you use Setuptools, this will default to " + setuptools_source + "; if you use Distribute, this " "will default to " + distribute_source +".")) parser.add_option("--download-base", action="callback", dest="download_base", callback=normalize_to_url, nargs=1, type="string", help=("Specify a URL or directory for downloading " "zc.buildout and either Setuptools or Distribute. " "Defaults to PyPI.")) parser.add_option("--eggs", help=("Specify a directory for storing eggs. Defaults to " "a temporary directory that is deleted when the " "bootstrap script completes.")) parser.add_option("-t", "--accept-buildout-test-releases", dest='accept_buildout_test_releases', action="store_true", default=False, help=("Normally, if you do not specify a --version, the " "bootstrap script and buildout gets the newest " "*final* versions of zc.buildout and its recipes and " "extensions for you. If you use this flag, " "bootstrap and buildout will get the newest releases " "even if they are alphas or betas.")) parser.add_option("-c", None, action="store", dest="config_file", help=("Specify the path to the buildout configuration " "file to be used.")) options, args = parser.parse_args() # if -c was provided, we push it back into args for buildout's main function if options.config_file is not None: args += ['-c', options.config_file] if options.eggs: eggs_dir = os.path.abspath(os.path.expanduser(options.eggs)) else: eggs_dir = tempfile.mkdtemp() if options.setup_source is None: if options.use_distribute: options.setup_source = distribute_source else: options.setup_source = setuptools_source if options.accept_buildout_test_releases: args.append('buildout:accept-buildout-test-releases=true') args.append('bootstrap') try: import pkg_resources import setuptools # A flag. Sometimes pkg_resources is installed alone. if not hasattr(pkg_resources, '_distribute'): raise ImportError except ImportError: ez_code = urllib2.urlopen( options.setup_source).read().replace('\r\n', '\n') ez = {} exec ez_code in ez setup_args = dict(to_dir=eggs_dir, download_delay=0) if options.download_base: setup_args['download_base'] = options.download_base if options.use_distribute: setup_args['no_fake'] = True ez['use_setuptools'](**setup_args) if 'pkg_resources' in sys.modules: reload(sys.modules['pkg_resources']) import pkg_resources # This does not (always?) update the default working set. We will # do it. for path in sys.path: if path not in pkg_resources.working_set.entries: pkg_resources.working_set.add_entry(path) cmd = [quote(sys.executable), '-c', quote('from setuptools.command.easy_install import main; main()'), '-mqNxd', quote(eggs_dir)] if not has_broken_dash_S: cmd.insert(1, '-S') find_links = options.download_base if not find_links: find_links = os.environ.get('bootstrap-testing-find-links') if find_links: cmd.extend(['-f', quote(find_links)]) if options.use_distribute: setup_requirement = 'distribute' else: setup_requirement = 'setuptools' ws = pkg_resources.working_set setup_requirement_path = ws.find( pkg_resources.Requirement.parse(setup_requirement)).location env = dict( os.environ, PYTHONPATH=setup_requirement_path) requirement = 'zc.buildout' version = options.version if version is None and not options.accept_buildout_test_releases: # Figure out the most recent final version of zc.buildout. import setuptools.package_index _final_parts = '*final-', '*final' def _final_version(parsed_version): for part in parsed_version: if (part[:1] == '*') and (part not in _final_parts): return False return True index = setuptools.package_index.PackageIndex( search_path=[setup_requirement_path]) if find_links: index.add_find_links((find_links,)) req = pkg_resources.Requirement.parse(requirement) if index.obtain(req) is not None: best = [] bestv = None for dist in index[req.project_name]: distv = dist.parsed_version if _final_version(distv): if bestv is None or distv > bestv: best = [dist] bestv = distv elif distv == bestv: best.append(dist) if best: best.sort() version = best[-1].version if version: requirement = '=='.join((requirement, version)) cmd.append(requirement) if is_jython: import subprocess exitcode = subprocess.Popen(cmd, env=env).wait() else: # Windows prefers this, apparently; otherwise we would prefer subprocess exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env])) if exitcode != 0: sys.stdout.flush() sys.stderr.flush() print ("An error occurred when trying to install zc.buildout. " "Look above this message for any errors that " "were output by easy_install.") sys.exit(exitcode) ws.add_entry(eggs_dir) ws.require(requirement) import zc.buildout.buildout zc.buildout.buildout.main(args) if not options.eggs: # clean up temporary egg directory shutil.rmtree(eggs_dir) slapos.core-0.35.1/slapos/proxy/0000755000076600007660000000000012110425554024577 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/proxy/__init__.py0000644000076600007660000001116412035766413026723 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import os import sys from optparse import OptionParser, Option import logging import logging.handlers import ConfigParser class Parser(OptionParser): """ Parse all arguments. """ def __init__(self, usage=None, version=None): """ Initialize all options possibles. """ OptionParser.__init__(self, usage=usage, version=version, option_list=[ Option("-l", "--log_file", help="The path to the log file used by the script.", type=str), Option("-v", "--verbose", default=False, action="store_true", help="Verbose output."), Option("-c", "--console", default=False, action="store_true", help="Console output."), Option("-u", "--database-uri", type=str, help="URI for sqlite database"), ]) def check_args(self): """ Check arguments """ (options, args) = self.parse_args() if len(args) != 1: self.error("Incorrect number of arguments") return options, args[0] class Config: def setConfig(self, option_dict, configuration_file_path): """ Set options given by parameters. """ # Set options parameters for option, value in option_dict.__dict__.items(): setattr(self, option, value) # Load configuration file configuration_parser = ConfigParser.SafeConfigParser() configuration_parser.read(configuration_file_path) # Merges the arguments and configuration for section in ("slapproxy", "slapos"): configuration_dict = dict(configuration_parser.items(section)) for key in configuration_dict: if not getattr(self, key, None): setattr(self, key, configuration_dict[key]) # set up logging self.logger = logging.getLogger("slapproxy") self.logger.setLevel(logging.INFO) if self.console: self.logger.addHandler(logging.StreamHandler()) if not self.database_uri: raise ValueError('database-uri is required.') if self.log_file: if not os.path.isdir(os.path.dirname(self.log_file)): # fallback to console only if directory for logs does not exists and # continue to run raise ValueError('Please create directory %r to store %r log file' % ( os.path.dirname(self.log_file), self.log_file)) else: file_handler = logging.FileHandler(self.log_file) file_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")) self.logger.addHandler(file_handler) self.logger.info('Configured logging to file %r' % self.log_file) self.logger.info("Started.") if self.verbose: self.logger.setLevel(logging.DEBUG) self.logger.debug("Verbose mode enabled.") def run(config): from views import app app.config['computer_id'] = config.computer_id app.config['DATABASE_URI'] = config.database_uri app.run(host=config.host, port=int(config.port)) def main(): "Run default configuration." usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0] try: # Parse arguments config = Config() config.setConfig(*Parser(usage=usage).check_args()) run(config) return_code = 0 except SystemExit, err: # Catch exception raise by optparse return_code = err sys.exit(return_code) slapos.core-0.35.1/slapos/proxy/db_version.py0000644000076600007660000000023212105142232027271 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- import pkg_resources DB_VERSION = pkg_resources.resource_stream('slapos.proxy', 'schema.sql').readline().strip().split(':')[1] slapos.core-0.35.1/slapos/proxy/query.py0000644000076600007660000001536312105142232026317 0ustar cedricdesaintmartincedricdesaintmartin00000000000000#!/opt/slapos/bin/py # -*- coding: utf-8 -*- # vim: set et sts=4: import collections import ConfigParser from optparse import OptionParser, Option import sys import lxml.etree import sqlite3 from slapos.proxy.db_version import DB_VERSION class Parser(OptionParser): """ Parse all arguments. """ def __init__(self, usage=None, version=None): """ Initialize all options possibles. """ OptionParser.__init__(self, usage=usage, version=version, option_list=[ Option("-u", "--database-uri", type=str, help="URI for sqlite database"), Option('--show-instances', help="View instance information", default=False, action="store_true"), Option('--show-params', help="View published parameters", default=False, action="store_true"), Option('--show-network', help="View network information", default=False, action="store_true"), Option('--show-all', help="View all information", default=False, action="store_true"), ]) def check_args(self): """ Check arguments """ (options, args) = self.parse_args() if len(args) < 1: self.error("Incorrect number of arguments") return options, args[0] class Config: def setConfig(self, option_dict, configuration_file_path): """ Set options given by parameters. """ # Set options parameters for option, value in option_dict.__dict__.items(): setattr(self, option, value) # Load configuration file configuration_parser = ConfigParser.SafeConfigParser() configuration_parser.read(configuration_file_path) # Merges the arguments and configuration for section in ("slapproxy", "slapos"): configuration_dict = dict(configuration_parser.items(section)) for key in configuration_dict: if not getattr(self, key, None): setattr(self, key, configuration_dict[key]) if not self.database_uri: raise ValueError('database-uri is required.') tbl_computer = 'computer' + DB_VERSION tbl_software = 'software' + DB_VERSION tbl_partition = 'partition' + DB_VERSION tbl_partition_network = 'partition_network' + DB_VERSION tbl_slave = 'slave' + DB_VERSION null_str = u"-" def print_table(qry, tablename, skip=None): if skip is None: skip = set() columns = [c[0] for c in qry.description if c[0] not in skip] rows = [] for row in qry.fetchall(): line = {} for col in columns: val = row[col] if val is None: val = null_str line[col] = val.strip() rows.append(line) max_width = {col: len(col) for col in columns} for row in rows: for col in columns: val = row[col] max_width[col] = max(max_width[col], len(val) if val else 0) hdr = [col.center(max_width[col]) for col in columns] print if rows: print 'table %s:' % tablename, else: print 'table %s: empty' % tablename return if skip: print 'skipping %s' % ', '.join(skip) else: print print ' | '.join(hdr) print '-+-'.join('-'*len(h) for h in hdr) for row in rows: cells = [row[col].ljust(max_width[col]) for col in columns] print ' | '.join(cells) def print_params(conn): cur = conn.cursor() print qry = cur.execute("SELECT reference, partition_reference, software_type, connection_xml FROM %s" % tbl_partition) for row in qry.fetchall(): if not row['connection_xml']: continue xml = str(row['connection_xml']) print '%s: %s (type %s)' % (row['reference'], row['partition_reference'], row['software_type']) instance = lxml.etree.fromstring(xml) for parameter in list(instance): name = parameter.get('id') text = parameter.text if text and name in ('ssh-key', 'ssh-public-key'): text = text[:20] + '...' + text[-20:] print ' %s = %s' % (name, text) print def print_computer_table(conn): cur = conn.cursor() qry = cur.execute("SELECT * FROM %s" % tbl_computer) print_table(qry, tbl_computer) def print_software_table(conn): cur = conn.cursor() qry = cur.execute("SELECT * FROM %s" % tbl_software) print_table(qry, tbl_software) def print_partition_table(conn): cur = conn.cursor() qry = cur.execute("SELECT * FROM %s WHERE slap_state<>'free'" % tbl_partition) print_table(qry, tbl_partition, skip=['xml', 'connection_xml', 'slave_instance_list']) def print_slave_table(conn): cur = conn.cursor() qry = cur.execute("SELECT * FROM %s" % tbl_slave) print_table(qry, tbl_slave, skip=['connection_xml']) def print_tables(conn): print_computer_table(conn) print_software_table(conn) print_partition_table(conn) print_slave_table(conn) def print_network(conn): print cur = conn.cursor() addr = collections.defaultdict(list) qry = cur.execute(""" SELECT * FROM %s WHERE partition_reference NOT IN ( SELECT reference FROM %s WHERE slap_state='free') """ % (tbl_partition_network, tbl_partition)) for row in qry: addr[row['partition_reference']].append(row['address']) for partition_reference in sorted(addr.keys()): addresses = addr[partition_reference] print '%s: %s' % (partition_reference, ', '.join(addresses)) def run(config): conn = sqlite3.connect(config.database_uri) conn.row_factory = sqlite3.Row fn = [] if config.show_all or config.show_instances: fn.append(print_tables) if config.show_all or config.show_params: fn.append(print_params) if config.show_all or config.show_network: fn.append(print_network) if fn: for f in fn: f(conn) else: print 'usage: %s [ --show-params | --show-network | --show-instances | --show-all ]' % sys.argv[0] def main(): "Run default configuration." usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0] try: # Parse arguments config = Config() config.setConfig(*Parser(usage=usage).check_args()) run(config) return_code = 0 except SystemExit, err: # Catch exception raise by optparse return_code = err sys.exit(return_code) slapos.core-0.35.1/slapos/proxy/schema.sql0000644000076600007660000000215712073300773026570 0ustar cedricdesaintmartincedricdesaintmartin00000000000000--version:10 CREATE TABLE IF NOT EXISTS software%(version)s (url VARCHAR(255) UNIQUE); CREATE TABLE IF NOT EXISTS computer%(version)s ( address VARCHAR(255), netmask VARCHAR(255), CONSTRAINT uniq PRIMARY KEY (address, netmask)); CREATE TABLE IF NOT EXISTS partition%(version)s ( reference VARCHAR(255) UNIQUE, slap_state VARCHAR(255) DEFAULT 'free', software_release VARCHAR(255), xml TEXT, connection_xml TEXT, slave_instance_list TEXT, software_type VARCHAR(255), partition_reference VARCHAR(255), requested_by VARCHAR(255), -- only used for debugging, -- slapproxy does not support proper scope requested_state VARCHAR(255) NOT NULL DEFAULT 'started' ); CREATE TABLE IF NOT EXISTS slave%(version)s ( reference VARCHAR(255) UNIQUE, connection_xml TEXT, hosted_by VARCHAR(255), asked_by VARCHAR(255) -- only used for debugging, -- slapproxy does not support proper scope ); CREATE TABLE IF NOT EXISTS partition_network%(version)s ( partition_reference VARCHAR(255), reference VARCHAR(255), address VARCHAR(255), netmask VARCHAR(255) ); slapos.core-0.35.1/slapos/proxy/views.py0000644000076600007660000004435312110422211026302 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- # vim: set et sts=2: ############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## from lxml import etree import sqlite3 from slapos.slap.slap import Computer, ComputerPartition, \ SoftwareRelease, SoftwareInstance, NotFoundError from slapos.proxy.db_version import DB_VERSION from flask import g, Flask, request, abort import xml_marshaller app = Flask(__name__) class UnauthorizedError(Exception): pass def xml2dict(xml): result_dict = {} if xml is not None and xml != '': tree = etree.fromstring(xml.encode('utf-8')) for element in tree.iter(tag=etree.Element): if element.tag == 'parameter': key = element.get('id') value = result_dict.get(key, None) if value is not None: value = value + ' ' + element.text else: value = element.text result_dict[key] = value return result_dict def dict2xml(dictionnary): instance = etree.Element('instance') for parameter_id, parameter_value in dictionnary.iteritems(): # cast everything to string parameter_value = str(parameter_value) etree.SubElement(instance, "parameter", attrib={'id': parameter_id}).text = parameter_value return etree.tostring(instance, pretty_print=True, xml_declaration=True, encoding='utf-8') def partitiondict2partition(partition): slap_partition = ComputerPartition(app.config['computer_id'], partition['reference']) slap_partition._software_release_document = None slap_partition._requested_state = 'destroyed' slap_partition._need_modification = 0 slap_partition._instance_guid = partition['reference'] if partition['software_release']: slap_partition._need_modification = 1 slap_partition._requested_state = partition['requested_state'] slap_partition._parameter_dict = xml2dict(partition['xml']) address_list = [] for address in execute_db('partition_network', 'SELECT * FROM %s WHERE partition_reference=?', [partition['reference']]): address_list.append((address['reference'], address['address'])) slap_partition._parameter_dict['ip_list'] = address_list slap_partition._parameter_dict['slap_software_type'] = \ partition['software_type'] if not partition['slave_instance_list'] == None: slap_partition._parameter_dict['slave_instance_list'] = \ xml_marshaller.xml_marshaller.loads(partition['slave_instance_list']) slap_partition._connection_dict = xml2dict(partition['connection_xml']) slap_partition._software_release_document = SoftwareRelease( software_release=partition['software_release'], computer_guid=app.config['computer_id']) return slap_partition def execute_db(table, query, args=(), one=False): try: cur = g.db.execute(query % (table + DB_VERSION,), args) except: app.logger.error('There was some issue during processing query %r on table %r with args %r' % (query, table, args)) raise rv = [dict((cur.description[idx][0], value) for idx, value in enumerate(row)) for row in cur.fetchall()] return (rv[0] if rv else None) if one else rv def connect_db(): return sqlite3.connect(app.config['DATABASE_URI']) @app.before_request def before_request(): g.db = connect_db() schema = app.open_resource('schema.sql') schema = schema.read() % dict(version=DB_VERSION) g.db.cursor().executescript(schema) g.db.commit() @app.after_request def after_request(response): g.db.commit() g.db.close() return response @app.route('/getComputerInformation', methods=['GET']) def getComputerInformation(): # Kept only for backward compatiblity return getFullComputerInformation() @app.route('/getFullComputerInformation', methods=['GET']) def getFullComputerInformation(): computer_id = request.args['computer_id'] if app.config['computer_id'] == computer_id: slap_computer = Computer(computer_id) slap_computer._software_release_list = [] for sr in execute_db('software', 'select * from %s'): slap_computer._software_release_list.append(SoftwareRelease( software_release=sr['url'], computer_guid=computer_id)) slap_computer._computer_partition_list = [] for partition in execute_db('partition', 'SELECT * FROM %s'): slap_computer._computer_partition_list.append(partitiondict2partition( partition)) return xml_marshaller.xml_marshaller.dumps(slap_computer) else: raise NotFoundError, "Only accept request for: %s" % \ app.config['computer_id'] @app.route('/setComputerPartitionConnectionXml', methods=['POST']) def setComputerPartitionConnectionXml(): slave_reference = request.form['slave_reference'].encode() computer_partition_id = request.form['computer_partition_id'] connection_xml = request.form['connection_xml'] connection_dict = xml_marshaller.xml_marshaller.loads( connection_xml.encode()) connection_xml = dict2xml(connection_dict) if slave_reference == 'None': query = 'UPDATE %s SET connection_xml=? WHERE reference=?' argument_list = [connection_xml, computer_partition_id.encode()] execute_db('partition', query, argument_list) return 'done' else: query = 'UPDATE %s SET connection_xml=? , hosted_by=? WHERE reference=?' argument_list = [connection_xml, computer_partition_id.encode(), slave_reference] execute_db('slave', query, argument_list) return 'done' @app.route('/buildingSoftwareRelease', methods=['POST']) def buildingSoftwareRelease(): return 'Ignored' @app.route('/availableSoftwareRelease', methods=['POST']) def availableSoftwareRelease(): return 'Ignored' @app.route('/softwareReleaseError', methods=['POST']) def softwareReleaseError(): return 'Ignored' @app.route('/buildingComputerPartition', methods=['POST']) def buildingComputerPartition(): return 'Ignored' @app.route('/availableComputerPartition', methods=['POST']) def availableComputerPartition(): return 'Ignored' @app.route('/softwareInstanceError', methods=['POST']) def softwareInstanceError(): return 'Ignored' @app.route('/softwareInstanceBang', methods=['POST']) def softwareInstanceBang(): return 'Ignored' @app.route('/startedComputerPartition', methods=['POST']) def startedComputerPartition(): return 'Ignored' @app.route('/stoppedComputerPartition', methods=['POST']) def stoppedComputerPartition(): return 'Ignored' @app.route('/destroyedComputerPartition', methods=['POST']) def destroyedComputerPartition(): return 'Ignored' @app.route('/useComputer', methods=['POST']) def useComputer(): return 'Ignored' @app.route('/loadComputerConfigurationFromXML', methods=['POST']) def loadComputerConfigurationFromXML(): xml = request.form['xml'] computer_dict = xml_marshaller.xml_marshaller.loads(str(xml)) if app.config['computer_id'] == computer_dict['reference']: execute_db('computer', 'INSERT OR REPLACE INTO %s values(:address, :netmask)', computer_dict) for partition in computer_dict['partition_list']: execute_db('partition', 'INSERT OR IGNORE INTO %s (reference) values(:reference)', partition) execute_db('partition_network', 'DELETE FROM %s WHERE partition_reference = ?', [partition['reference']]) for address in partition['address_list']: address['reference'] = partition['tap']['name'] address['partition_reference'] = partition['reference'] execute_db('partition_network', 'INSERT OR REPLACE INTO %s (reference, partition_reference, address, netmask) values(:reference, :partition_reference, :addr, :netmask)', address) return 'done' else: raise UnauthorizedError, "Only accept request for: %s" % \ app.config['computer_id'] @app.route('/registerComputerPartition', methods=['GET']) def registerComputerPartition(): computer_reference = request.args['computer_reference'] computer_partition_reference = request.args['computer_partition_reference'] if app.config['computer_id'] == computer_reference: partition = execute_db('partition', 'SELECT * FROM %s WHERE reference=?', [computer_partition_reference.encode()], one=True) if partition is None: raise UnauthorizedError return xml_marshaller.xml_marshaller.dumps( partitiondict2partition(partition)) else: raise UnauthorizedError, "Only accept request for: %s" % \ app.config['computer_id'] @app.route('/supplySupply', methods=['POST']) def supplySupply(): url = request.form['url'] computer_id = request.form['computer_id'] if app.config['computer_id'] == computer_id: execute_db('software', 'INSERT OR REPLACE INTO %s VALUES(?)', [url]) else: raise UnauthorizedError, "Only accept request for: %s" % \ app.config['computer_id'] return '%r added' % url @app.route('/requestComputerPartition', methods=['POST']) def requestComputerPartition(): shared_xml = request.form.get('shared_xml') share = xml_marshaller.xml_marshaller.loads(shared_xml) if not share: return request_not_shared() else: return request_slave() @app.route('/softwareInstanceRename', methods=['POST']) def softwareInstanceRename(): new_name = request.form['new_name'].encode() computer_partition_id = request.form['computer_partition_id'].encode() q = 'UPDATE %s SET partition_reference = ? WHERE reference = ?' execute_db('partition', q, [new_name, computer_partition_id]) return 'done' def request_not_shared(): software_release = request.form['software_release'].encode() # some supported parameters software_type = request.form.get('software_type').encode() partition_reference = request.form.get('partition_reference', '').encode() partition_id = request.form.get('computer_partition_id', '').encode() partition_parameter_kw = request.form.get('partition_parameter_xml', None) requested_state = xml_marshaller.xml_marshaller.loads(request.form.get('state')) if partition_parameter_kw: partition_parameter_kw = xml_marshaller.xml_marshaller.loads( partition_parameter_kw.encode()) else: partition_parameter_kw = {} instance_xml = dict2xml(partition_parameter_kw) args = [] a = args.append q = 'SELECT * FROM %s WHERE partition_reference=?' a(partition_reference) partition = execute_db('partition', q, args, one=True) args = [] a = args.append q = 'UPDATE %s SET slap_state="busy"' if requested_state: q += ', requested_state=?' a(requested_state) # If partition doesn't exist: create it and insert parameters if partition is None: partition = execute_db('partition', 'SELECT * FROM %s WHERE slap_state="free"', (), one=True) if partition is None: app.logger.warning('No more free computer partition') abort(404) q += ' ,software_release=?' a(software_release) if partition_reference: q += ' ,partition_reference=?' a(partition_reference) if partition_id: q += ' ,requested_by=?' a(partition_id) if not software_type: software_type = 'RootSoftwareInstance' # # XXX change software_type when requested # if software_type: q += ' ,software_type=?' a(software_type) # Else: only update partition_parameter_kw if instance_xml: q += ' ,xml=?' a(instance_xml) q += ' WHERE reference=?' a(partition['reference'].encode()) execute_db('partition', q, args) args = [] partition = execute_db('partition', 'SELECT * FROM %s WHERE reference=?', [partition['reference'].encode()], one=True) address_list = [] for address in execute_db('partition_network', 'SELECT * FROM %s WHERE partition_reference=?', [partition['reference']]): address_list.append((address['reference'], address['address'])) # XXX it should be ComputerPartition, not a SoftwareInstance return xml_marshaller.xml_marshaller.dumps(SoftwareInstance( xml=partition['xml'], connection_xml=partition['connection_xml'], slap_computer_id=app.config['computer_id'], slap_computer_partition_id=partition['reference'], slap_software_release_url=partition['software_release'], slap_server_url='slap_server_url', slap_software_type=partition['software_type'], slave_instance_list=partition['slave_instance_list'], instance_guid=partition['reference'], ip_list=address_list )) def request_slave(): """ Function to organise link between slave and master. Slave information are stored in places: 1. slave table having information such as slave reference, connection information to slave (given by slave master), hosted_by and asked_by reference. 2. A dictionnary in slave_instance_list of selected slave master in which are stored slave_reference, software_type, slave_title and partition_parameter_kw stored as individual keys. """ software_release = request.form['software_release'].encode() # some supported parameters software_type = request.form.get('software_type').encode() partition_reference = request.form.get('partition_reference', '').encode() partition_id = request.form.get('computer_partition_id', '').encode() # Contain slave parameters to be given to slave master partition_parameter_kw = request.form.get('partition_parameter_xml', None) if partition_parameter_kw: partition_parameter_kw = xml_marshaller.xml_marshaller.loads( partition_parameter_kw.encode()) else: partition_parameter_kw = {} filter_kw = xml_marshaller.xml_marshaller.loads(request.form.get('filter_xml').encode()) instance_xml = dict2xml(partition_parameter_kw) # We will search for a master corresponding to request args = [] a = args.append q = 'SELECT * FROM %s WHERE software_release=?' a(software_release) if software_type: q += ' AND software_type=?' a(software_type) if 'instance_guid' in filter_kw: q += ' AND reference=?' a(filter_kw['instance_guid']) partition = execute_db('partition', q, args, one=True) if partition is None: app.logger.warning('No partition corresponding to slave request: %s' % \ args) abort(404) # We set slave dictionary as described in docstring new_slave = {} slave_reference = partition_id + '_' + partition_reference new_slave['slave_title'] = slave_reference new_slave['slap_software_type'] = software_type new_slave['slave_reference'] = slave_reference for key in partition_parameter_kw : if partition_parameter_kw[key] is not None : new_slave[key] = partition_parameter_kw[key] # Add slave to partition slave_list if not present else replace information slave_instance_list = partition['slave_instance_list'] if slave_instance_list == None: slave_instance_list = [] else: slave_instance_list = xml_marshaller.xml_marshaller.loads(slave_instance_list) for x in slave_instance_list: if x['slave_reference'] == slave_reference: slave_instance_list.remove(x) slave_instance_list.append(new_slave) # Update slave_instance_list in database args = [] a = args.append q = 'UPDATE %s SET slave_instance_list=?' a(xml_marshaller.xml_marshaller.dumps(slave_instance_list)) q += ' WHERE reference=?' a(partition['reference'].encode()) execute_db('partition', q, args) args = [] partition = execute_db('partition', 'SELECT * FROM %s WHERE reference=?', [partition['reference'].encode()], one=True) # Add slave to slave table if not there slave = execute_db('slave', 'SELECT * FROM %s WHERE reference=?', [slave_reference], one=True) if slave is None : execute_db('slave', 'INSERT OR IGNORE INTO %s (reference,asked_by,hosted_by) values(:reference,:asked_by,:hosted_by)', [slave_reference,partition_id,partition['reference']]) slave = execute_db('slave','SELECT * FROM %s WHERE reference=?', [slave_reference], one = True) address_list = [] for address in execute_db('partition_network', 'SELECT * FROM %s WHERE partition_reference=?', [partition['reference']]): address_list.append((address['reference'], address['address'])) # XXX it should be ComputerPartition, not a SoftwareInstance return xml_marshaller.xml_marshaller.dumps(SoftwareInstance( _connection_dict=xml2dict(slave['connection_xml']), xml = instance_xml, slap_computer_id=app.config['computer_id'], slap_computer_partition_id=slave['hosted_by'], slap_software_release_url=partition['software_release'], slap_server_url='slap_server_url', slap_software_type=partition['software_type'], ip_list=address_list )) slapos.core-0.35.1/slapos/README.console.txt0000644000076600007660000000550112037271015026555 0ustar cedricdesaintmartincedricdesaintmartin00000000000000console ------- The slapconsole tool allows to interact with a SlapOS Master throught the SLAP library. For more information about SlapOS or slapconsole usages, please go to http://community.slapos.org. The slapconsole tool is only a bare Python console with several global variables defined and initialized. Initialization and configuration file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Slapconsole allows to automatically connect to a Master using URL and SSL certificate from given slapos.cfg. Certificate has to be *USER* certificate, manually obtained from SlapOS master web interface. Slapconsole tools reads the given slapos.cfg configuration file and use the following informations : * Master URL is read from [slapos] in the "master_url" parameter. * SSL Certificate is read from [slapconsole] in the "cert_file" parameter. * SSL Key is read from [slapconsole] in the "key_file" parameter. See slapos.cfg.example for examples. Global functions ~~~~~~~~~~~~~~~~ * request() is a shorthand for slap.registerOpenOrder().request() allowing to request instances. * supply() is a shorthand for slap.registerSupply().supply() allowing to request software installation. For more information about those methods, please read the SLAP library documentation. Global aliases ~~~~~~~~~~~~~~ "software_list" is a list containing all the Software Release URLs defined in client slapos.cfg configuration file. Also, each Software Release defined in this configuration file is translated into a global variable to ease the request of those Sofware Releases. This allows to request instances in a few words, i.e:: request("http://www.url.com/path/to/kvm/software.cfg", "mykvm") can be simplified into :: request(kvm, "mykvm") If the slapos.cfg file contains :: alias = kvm http://www.url.com/path/to/kvm/software.cfg Global objects ~~~~~~~~~~~~~~ "slap" is an instance of the SLAP library. It is only used for advanced usages. "slap" instance is obtained by doing :: slap = slapos.slap.slap() slap.initializeConnection(config.master_url, key_file=config.key_file, cert_file=config.cert_file) Examples ~~~~~~~~ :: >>> # Request instance >>> request(kvm, "myuniquekvm") >>> # Request instance on specific computer >>> request(kvm, "myotheruniquekvm", filter_kw={ "computer_guid": "COMP-12345" }) >>> # Request instance, specifying parameters (here nbd_ip and nbd_port) >>> request(kvm, "mythirduniquekvm", partition_parameter_kw={"nbd_ip":"2a01:e35:2e27:460:e2cb:4eff:fed9:48dc", "nbd_port":"1024"}) >>> # Request software installation on owned computer >>> supply(kvm, "mycomputer") >>> # Fetch existing instance status >>> request(kvm, "myuniquekvm").getState() >>> # Fetch instance information on already launched instance >>> request(kvm, "myuniquekvm").getConnectionParameter("url") slapos.core-0.35.1/slapos/README.format.txt0000644000076600007660000000105212030324717026401 0ustar cedricdesaintmartincedricdesaintmartin00000000000000format ====== slapformat is an application to prepare SlapOS ready node (machine). It "formats" the machine by: - creating users and groups - creating bridge interface - creating needed tap interfaces - creating needed directories with proper ownership and permissions In the end special report is generated and information are posted to configured SlapOS server. This program shall be only run by root. Requirements ------------ Linux with IPv6, bridging and tap interface support. Binaries: * brctl * groupadd * ip * tunctl * useradd slapos.core-0.35.1/slapos/README.grid.txt0000644000076600007660000000527012030324717026044 0ustar cedricdesaintmartincedricdesaintmartin00000000000000grid ==== slapgrid is a client of SLAPos. SLAPos provides support for deploying a SaaS system in a minute. Slapgrid allows you to easily deploy instances of softwares based on buildout profiles. For more informations about SLAP and SLAPos, please see the SLAP documentation. Requirements ------------ A working SLAP server with informations about your computer, in order to retrieve them. As Vifib servers use IPv6 only, we strongly recommend an IPv6 enabled UNIX box. For the same reasons, Python >= 2.6 with development headers is also strongly recommended (IPv6 support is not complete in previous releases). For now, gcc and glibc development headers are required to build most software releases. Concepts -------- Here are the fundamental concepts of slapgrid : A Software Release (SR) is just a software. A Computer Partition (CP) is an instance of a Software Release. Imagine you want to install with slapgrid some software and run it. You will have to install the software as a Software Release, and then instantiate it, i.e configuring it for your needs, as a Computer Partition. How it works ------------ When run, slapgrid will authenticate to the SLAP library with a computer_id and fetch the list of Software Releases to install or remove and Computer Partitions to start or stop. Then, it will process each Software Release, and each Computer Partition. It will also periodically send to SLAP the usage report of each Computer Partition. Installation ------------ With easy_install:: $ easy_install slapgrid slapgrid needs several directories to be created and configured before being able to run : a software releases directory, and an instances directory with configured computer partition directory(ies). You should create for each Computer Partition directory created a specific user and associate it with its Computer Partition directory. Each Computer Partition directory should belongs to this specific user, with permissions of 0750. Usage ----- slapgrid needs several informations in order to run. You can specify them by adding arguments to the slapgrid command line, or by putting then in a configuration file. Beware : you need a valid computer resource on server side. Examples -------- simple example : Just run slapgrid: $ slapgrid --instance-root /path/to/instance/root --software-root /path/to/software_root --master-url https://some.server/some.resource --computer-id my.computer.id configuration file example:: [slapgrid] instance_root = /path/to/instance/root software_root = /path/to/software/root master_url = https://slapos.server/slap_service computer_id = my.computer.id then run slapgrid:: $ slapgrid --configuration-file = path/to/configuration/file slapos.core-0.35.1/slapos/README.proxy.txt0000644000076600007660000000075312060344637026307 0ustar cedricdesaintmartincedricdesaintmartin00000000000000proxy ===== Implement minimalist SlapOS Master server without any security, designed to work only from localhost with one SlapOS Node (a.k.a Computer). It implements (or should implement) the SLAP API, as currently implemented in the SlapOS Master (see slaptool.py in Master). The only behavioral difference from the SlapOS Master is: When the proxy doesn't find any free partition (and/or in case of slave instance, any compatible master instance), it will throw a NotFoundError (404). slapos.core-0.35.1/slapos/README.slap.txt0000644000076600007660000000271412037271015026055 0ustar cedricdesaintmartincedricdesaintmartin00000000000000slap ==== Simple Language for Accounting and Provisioning python library. Developer note - python version ------------------------------- This library is used on client (slapgrid) and server side. Server is using python2.4 and client is using python2.6 Having this in mind, code of this library *have* to work on python2.4 How it works ------------ The SLAP main server which is in charge of service coordination receives from participating servers the number of computer paritions which are available, the type of resource which a party is ready provide, and request from parties for resources which are needed. Each participating server is identified by a unique ID and runs a slap-server daemon. This daemon collects from the main server the installation tasks and does the installation of resources, then notifies the main server of completion whenever a resource is configured, installed and available. The data structure on the main server is the following: A - Action: an action which can happen to provide a resource or account its usage CP - Computer Partition: provides a URL to Access a Cloud Resource RI - Resource Item: describes a resource CI - Contract Item: describes the contract to attach the DL to (This is unclear still) R - Resource: describes a type of cloud resource (ex. MySQL Table) is published on slapgrid.org DL - Delivery Line: Describes an action happening on a resource item on a computer partition D - Delivery: groups multiple Delivery Lines slapos.core-0.35.1/slapos/register/0000755000076600007660000000000012110425554025242 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/register/__init__.py0000644000076600007660000000036412035516223027356 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# See http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages try: __import__('pkg_resources').declare_namespace(__name__) except ImportError: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) slapos.core-0.35.1/slapos/register/register.py0000644000076600007660000003350412101724464027447 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2012 Vifib SARL and Contributors. All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import base64 import ConfigParser from getpass import getpass import logging from optparse import OptionParser, Option import os import shutil import sys import tempfile import urllib2 class SlapError(Exception): """ Slap error """ def __init__(self, message): self.msg = message class UsageError(SlapError): pass class ExecError(SlapError): pass class Parser(OptionParser): """ Parse all arguments. """ def __init__(self, usage=None, version=None): """ Initialize all options possibles. """ OptionParser.__init__(self, usage=usage, version=version, option_list=[ Option("--interface-name", help="Interface name to access internet", default='eth0', type=str), Option("--master-url", help="URL of vifib master", default='https://slap.vifib.com', type=str), Option("--master-url-web", help="URL of vifib master webservice to register certificates", default='https://www.vifib.net', type=str), Option("--partition-number", help="Number of partition on computer", default='10', type=int), Option("--ipv4-local-network", help="Base of ipv4 local network", default='10.0.0.0/16', type=str), Option("--ipv6-interface", help="Interface name to get ipv6", default='', type=str), Option("--login", help="User login on Vifib master webservice", default=None, type=str), Option("--password", help="User password on Vifib master webservice", default=None, type=str), Option("-t", "--create-tap", help="""Will trigger creation of one virtual "tap" interface per \ Partition and attach it to primary interface. Requires primary interface to be \ a bridge. defaults to false. Needed to host virtual machines.""", default=False, action="store_true"), Option("-n", "--dry-run", help="Simulate the execution steps", default=False, action="store_true"), ]) def check_args(self): """ Check arguments """ (options, args) = self.parse_args() if len(args) != 1: self.error("Incorrect number of arguments") node_name = args[0] if options.password != None and options.login == None : self.error("Please enter your login with your password") return options, node_name def get_login(): """Get user id and encode it for basic identification""" login = raw_input("Vifib Login: ") password = getpass() identification = base64.encodestring('%s:%s' % (login, password))[:-1] return identification def check_login(identification, master_url_web): """Check if logged correctly on vifib""" request = urllib2.Request(master_url_web) # Prepare header for basic authentification authheader = "Basic %s" % identification request.add_header("Authorization", authheader) home_page_url = urllib2.urlopen(request).read() if 'Logout' in home_page_url: return 1 else : return 0 def get_certificates(identification, node_name, master_url_web): """Download certificates on vifib master""" register_server_url = '/'.join([master_url_web, ("add-a-server/WebSection_registerNewComputer?dialog_id=WebSection_viewServerInformationDialog&dialog_method=WebSection_registerNewComputer&title={}&object_path=/erp5/web_site_module/hosting/add-a-server&update_method=&cancel_url=https%3A//www.vifib.net/add-a-server/WebSection_viewServerInformationDialog&Base_callDialogMethod=&field_your_title=Essai1&dialog_category=None&form_id=view".format(node_name))]) request = urllib2.Request(register_server_url) # Prepare header for basic authentification authheader = "Basic %s" % identification request.add_header("Authorization", authheader) url = urllib2.urlopen(request) page = url.read() return page def parse_certificates(source): """Parse html gotten from vifib to make certificate and key files""" c_start = source.find("Certificate:") c_end = source.find("", c_start) k_start = source.find("-----BEGIN PRIVATE KEY-----") k_end = source.find("", k_start) return [source[c_start:c_end], source[k_start:k_end]] def get_computer_name(certificate): """Parse certificate to get computer name and return it""" k = certificate.find("COMP-") i = certificate.find("/email", k) return certificate[k:i] def save_former_config(config): """Save former configuration if found""" # Check for config file in /etc/opt/slapos/ if os.path.exists('/etc/opt/slapos/slapos.cfg'): former_slapos_configuration = '/etc/opt/slapos' else : former_slapos_configuration = 0 if former_slapos_configuration: saved_slapos_configuration = former_slapos_configuration + '.old' while True: if os.path.exists(saved_slapos_configuration): print "Slapos configuration detected in %s" % saved_slapos_configuration if saved_slapos_configuration[len(saved_slapos_configuration) - 1] != 'd' : saved_slapos_configuration = saved_slapos_configuration[:len(saved_slapos_configuration) - 1] \ + str(int(saved_slapos_configuration[len(saved_slapos_configuration) - 1]) + 1 ) else : saved_slapos_configuration += ".1" else: break config.logger.info("Former slapos configuration detected in %s moving to %s" % (former_slapos_configuration, saved_slapos_configuration)) shutil.move(former_slapos_configuration, saved_slapos_configuration) def get_slapos_conf_example(): """ Get slapos.cfg.example and return its path """ register_server_url = "http://git.erp5.org/gitweb/slapos.core.git/blob_plain/HEAD:/slapos.cfg.example" request = urllib2.Request(register_server_url) url = urllib2.urlopen(request) page = url.read() _, path = tempfile.mkstemp() slapos_cfg_example = open(path,'w') slapos_cfg_example.write(page) slapos_cfg_example.close() return path def slapconfig(config): """Base Function to configure slapos in /etc/opt/slapos""" dry_run = config.dry_run # Create slapos configuration directory if needed slap_configuration_directory = os.path.normpath(config.slapos_configuration) if not os.path.exists(slap_configuration_directory): config.logger.info ("Creating directory: %s" % slap_configuration_directory) if not dry_run: os.mkdir(slap_configuration_directory, 0711) user_certificate_repository_path = os.path.join(slap_configuration_directory,'ssl') if not os.path.exists(user_certificate_repository_path): config.logger.info ("Creating directory: %s" % user_certificate_repository_path) if not dry_run: os.mkdir(user_certificate_repository_path, 0711) key_file = os.path.join(user_certificate_repository_path, 'key') cert_file = os.path.join(user_certificate_repository_path, 'certificate') for (src, dst) in [(config.key, key_file), (config.certificate, cert_file)]: config.logger.info ("Copying to %r, and setting minimum privileges" % dst) if not dry_run: destination = open(dst,'w') destination.write(''.join(src)) destination.close() os.chmod(dst, 0600) os.chown(dst, 0, 0) certificate_repository_path = os.path.join(slap_configuration_directory, 'ssl', 'partition_pki') if not os.path.exists(certificate_repository_path): config.logger.info ("Creating directory: %s" % certificate_repository_path) if not dry_run: os.mkdir(certificate_repository_path, 0711) # Put slapos configuration file slap_configuration_file_location = os.path.join(slap_configuration_directory, 'slapos.cfg') config.logger.info ("Creating slap configuration: %s" % slap_configuration_file_location) # Get example configuration file slapos_cfg_example = get_slapos_conf_example() configuration_example_parser = ConfigParser.RawConfigParser() configuration_example_parser.read(slapos_cfg_example) os.remove(slapos_cfg_example) # prepare slapos section slaposconfig = dict( computer_id=config.computer_id, master_url=config.master_url, key_file=key_file, cert_file=cert_file, certificate_repository_path=certificate_repository_path) for key in slaposconfig: configuration_example_parser.set('slapos', key, slaposconfig[key]) # prepare slapformat slapformatconfig = dict( interface_name=config.interface_name, ipv4_local_network=config.ipv4_local_network, partition_amount=config.partition_number, create_tap=config.create_tap ) for key in slapformatconfig : configuration_example_parser.set('slapformat', key, slapformatconfig[key]) if not config.ipv6_interface == '': configuration_example_parser.set('slapformat', 'ipv6_interface', config.ipv6_interface) if not dry_run: slap_configuration_file = open(slap_configuration_file_location, "w") configuration_example_parser.write(slap_configuration_file) config.logger.info ("SlapOS configuration: DONE") # Class containing all parameters needed for configuration class Config: def setConfig(self, option_dict, node_name): """ Set options given by parameters. """ # Set options parameters for option, value in option_dict.__dict__.items(): setattr(self, option, value) self.node_name = node_name # Define logger for register self.logger = logging.getLogger('Register') self.logger.setLevel(logging.DEBUG) # create console handler and set level to debug self.ch = logging.StreamHandler() self.ch.setLevel(logging.INFO) # create formatter self.formatter = logging.Formatter('%(levelname)s - %(message)s') # add formatter to ch self.ch.setFormatter(self.formatter) # add ch to logger self.logger.addHandler(self.ch) def COMPConfig(self, slapos_configuration, computer_id, certificate, key): self.slapos_configuration = slapos_configuration self.computer_id = computer_id self.certificate = certificate self.key = key def displayUserConfig(self): self.logger.debug("Computer Name: %s" % self.node_name) self.logger.debug("Master URL: %s" % self.master_url) self.logger.debug("Number of partition: %s" % self.partition_number) self.logger.info("Using Interface %s" % self.interface_name) self.logger.debug("Ipv4 sub network: %s" % self.ipv4_local_network) self.logger.debug("Ipv6 Interface: %s" %self.ipv6_interface) def register(config): """Register new computer on VIFIB and generate slapos.cfg""" # Get User identification and check them if config.login == None : while True : print ("Please enter your Vifib login") user_id = get_login() if check_login(user_id, config.master_url_web): break config.logger.warning ("Wrong login/password") else: if config.password == None : config.password = getpass() user_id = base64.encodestring('%s:%s' % (config.login, config.password))[:-1] if not check_login(user_id, config.master_url_web): config.logger.error ("Wrong login/password") return 1 # Get source code of page having certificate and key certificate_key = get_certificates(user_id, config.node_name, config.master_url_web) # Parse certificate and key and get computer id certificate_key = parse_certificates(certificate_key) certificate = certificate_key[0] key = certificate_key[1] COMP = get_computer_name(certificate) # Getting configuration parameters slapos_configuration = '/etc/opt/slapos/' config.COMPConfig(slapos_configuration=slapos_configuration, computer_id=COMP, certificate = certificate, key = key ) # Save former configuration if not config.dry_run: save_former_config(config) # Prepare Slapos Configuration slapconfig(config) print "Node has successfully been configured as %s." % COMP return 0 def main(): "Run default configuration." usage = "usage: slapos node %s NODE_NAME [options] " % sys.argv[0] try: # Parse arguments config = Config() config.setConfig(*Parser(usage=usage).check_args()) return_code = register(config) except UsageError, err: print >> sys.stderr, err.msg print >> sys.stderr, "For help use --help" return_code = 16 except ExecError, err: print >> sys.stderr, err.msg return_code = 16 except SystemExit, err: # Catch exception raise by optparse return_code = err sys.exit(return_code) slapos.core-0.35.1/slapos/slap/0000755000076600007660000000000012110425554024355 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/slap/__init__.py0000644000076600007660000000304412061611713026466 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly adviced to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public License # as published by the Free Software Foundation; either version 2.1 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import sys if sys.version_info < (2, 6): import warnings warnings.warn('Used python version (%s) is old and has problems with' ' IPv6 connections' % '.'.join([str(q) for q in sys.version_info[:3]])) from slap import * slapos.core-0.35.1/slapos/slap/doc/0000755000076600007660000000000012110425554025122 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/slap/doc/computer_consumption.xsd0000644000076600007660000000422012030324717032134 0ustar cedricdesaintmartincedricdesaintmartin00000000000000 slapos.core-0.35.1/slapos/slap/doc/partition_consumption.xsd0000644000076600007660000000171612030324717032316 0ustar cedricdesaintmartincedricdesaintmartin00000000000000 slapos.core-0.35.1/slapos/slap/doc/software_instance.xsd0000644000076600007660000000123312030324717031357 0ustar cedricdesaintmartincedricdesaintmartin00000000000000 slapos.core-0.35.1/slapos/slap/interface/0000755000076600007660000000000012110425554026315 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/slap/interface/__init__.py0000644000076600007660000000250012035516223030423 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly adviced to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public License # as published by the Free Software Foundation; either version 2.1 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## slapos.core-0.35.1/slapos/slap/interface/slap.py0000644000076600007660000003154012110425212027620 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly adviced to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public License # as published by the Free Software Foundation; either version 2.1 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## from zope.interface import Interface """ Note: all strings accepted/returned by the slap library are encoded in UTF-8. """ class IException(Interface): """ Classes which implement IException are used to report errors. """ class IServerError(IException): """ Classes which implement IServerError are used to report unexpected error from the slap server. """ class INotFoundError(IException): """ Classes which implement INotFoundError are used to report missing informations on the slap server. """ class IResourceNotReady(IException): """ Classes which implement IResourceNotReady are used to report resource not ready on the slap server. """ class IUnauthorized(IException): """ Classes which implement IUnauthorized are used to report missing permissions on the slap server. """ class IRequester(Interface): """ Classes which implement IRequester can request software instance to the slapgrid server. """ def request(software_release, software_type, partition_reference, shared=False, partition_parameter_kw=None, filter_kw=None): """ Request software release instantiation to slapgrid server. Returns a new computer partition document, where this sofware release will be installed. software_release -- uri of the software release which has to be instanciated software_type -- type of component provided by software_release partition_reference -- local reference of the instance used by the recipe to identify the instances. shared -- boolean to use a shared service partition_parameter_kw -- dictionary of parameter used to fill the parameter dict of newly created partition. filter_kw -- dictionary of filtering parameter to select the requested computer partition. computer_guid - computer of the requested partition partition_type - virtio, slave, full, limited port - port provided by the requested partition Example: request('http://example.com/toto/titi', 'typeA', 'mysql_1') """ class IBuildoutController(Interface): """ Classes which implement IBuildoutController can report the buildout run status to the slapgrid server. """ def available(): """ Notify (to the slapgrid server) that the software instance is available. """ def building(): """ Notify (to the slapgrid server) that the buildout is not available and under creation. """ def error(error_log): """ Notify (to the slapgrid server) that the buildout is not available and reports an error. error_log -- a text describing the error It can be a traceback for example. """ class ISoftwareRelease(IBuildoutController): """ Software release interface specification """ def getURI(): """ Returns a string representing the uri of the software release. """ def getComputerId(): """ Returns a string representing the identifier of the computer where the SR is installed. """ def getState(): """ Returns a string representing the expected state of the software installation. The result can be: available, destroyed """ def destroyed(): """ Notify (to the slapgrid server) that the software installation has been correctly destroyed. """ class IComputerPartition(IBuildoutController, IRequester): """ Computer Partition interface specification Classes which implement IComputerPartition can propagate the computer partition state to the SLAPGRID server and request new computer partition creation. """ def stopped(): """ Notify (to the slapgrid server) that the software instance is available and stopped. """ def started(): """ Notify (to the slapgrid server) that the software instance is available and started. """ def destroyed(): """ Notify (to the slapgrid server) that the software instance has been correctly destroyed. """ def getId(): """ Returns a string representing the identifier of the computer partition inside the slapgrid server. """ def getInstanceGuid(): """ Returns a string representing the unique identifier of the instance inside the slapgrid server. """ def getState(): """ Returns a string representing the expected state of the computer partition. The result can be: started, stopped, destroyed """ def getSoftwareRelease(): """ Returns the software release associate to the computer partition. Raise an INotFoundError if no software release is associated. """ def getInstanceParameterDict(): """ Returns a dictionary of instance parameters. The contained values can be used to fill the software instanciation profile. """ def getConnectionParameterDict(): """ Returns a dictionary of connection parameters. The contained values are connection parameters of a compute partition. """ def setUsage(usage_log): """ Associate a usage log to the computer partition. This method does not report the usage to the slapgrid server. See IComputer.report. usage_log -- a text describing the computer partition usage. It can be an XML for example. """ def bang(log): """ Report a problem detected on a computer partition. This will trigger the reinstanciation of all partitions in the instance tree. log -- a text explaining why the method was called """ def getCertificate(): """ Returns a dictionnary containing the authentification certificates associated to the computer partition. The dictionnary keys are: key -- value is a SSL key certificate -- value is a SSL certificate Raise an INotFoundError if no software release is associated. """ def setConnectionDict(connection_dict, slave_reference=None): """ Store the connection parameters associated to a partition. connection_dict -- dictionary of parameter used to fill the connection dict of the partition. slave_reference -- current reference of the slave instance to modify """ def getInstanceParameter(key): """ Returns the instance parameter associated to the key. Raise an INotFoundError if no key is defined. key -- a string name of the parameter """ def getConnectionParameter(key): """ Return the connection parameter associate to the key. Raise an INotFoundError if no key is defined. key -- a string name of the parameter """ def rename(partition_reference, slave_reference=None): """ Change the partition reference of a partition partition_reference -- new local reference of the instance used by the recipe to identify the instances. slave_reference -- current reference of the slave instance to modify """ def getStatus(): """ Returns a dictionnary containing the latest status of the computer partition. The dictionnary keys are: user -- user who reported the latest status created_at -- date of the status text -- message log of the status """ class IComputer(Interface): """ Computer interface specification Classes which implement IComputer can fetch informations from the slapgrid server to know which Software Releases and Software Instances have to be installed. """ def getSoftwareReleaseList(): """ Returns the list of software release which has to be supplied by the computer. Raise an INotFoundError if computer_guid doesn't exist. """ def getComputerPartitionList(): """ Returns the list of configured computer partitions associated to this computer. Raise an INotFoundError if computer_guid doesn't exist. """ def reportUsage(computer_partition_list): """ Report the computer usage to the slapgrid server. IComputerPartition.setUsage has to be called on each computer partition to define each usage. computer_partition_list -- a list of computer partition for which the usage needs to be reported. """ def bang(log): """ Report a problem detected on a computer. This will trigger IComputerPartition.bang on all instances hosted by the Computer. log -- a text explaining why the method was called """ def updateConfiguration(configuration_xml): """ Report the current computer configuration. configuration_xml -- computer XML description generated by slapformat """ def getStatus(): """ Returns a dictionnary containing the latest status of the computer. The dictionnary keys are: user -- user who reported the latest status created_at -- date of the status text -- message log of the status """ def generateCertificate(): """ Returns a dictionnary containing the new certificate files for the computer. The dictionnary keys are: key -- key file certificate -- certificate file Raise ValueError is another certificate is already valid. """ def revokeCertificate(): """ Revoke current computer certificate. Raise ValueError is there is not valid certificate. """ class IOpenOrder(IRequester): """ Open Order interface specification Classes which implement Open Order describe which kind of software instances is requested by a given client. """ def requestComputer(computer_reference): """ Request a computer to slapgrid server. Returns a new computer document. computer_reference -- local reference of the computer """ class ISupply(Interface): """ Supply interface specification Classes which implement Supply describe which kind of software releases a given client is ready to supply. """ def supply(software_release, computer_guid=None): """ Tell that given client is ready to supply given sofware release software_release -- uri of the software release which has to be instanciated computer_guid -- the identifier of the computer inside the slapgrid server. """ class slap(Interface): """ Initialise slap connection to the slapgrid server Slapgrid server URL is defined during the slap library installation, as recipes should not use another server. """ def initializeConnection(slapgrid_uri, authentification_key=None): """ Initialize the connection parameters to the slapgrid servers. slapgrid_uri -- uri the slapgrid server connector authentification_key -- string the authentificate the agent. Example: https://slapos.server/slap_interface """ def registerComputer(computer_guid): """ Instanciate a computer in the slap library. computer_guid -- the identifier of the computer inside the slapgrid server. """ def registerComputerPartition(computer_guid, partition_id): """ Instanciate a computer partition in the slap library. computer_guid -- the identifier of the computer inside the slapgrid server. partition_id -- the identifier of the computer partition inside the slapgrid server. Raise an INotFoundError if computer_guid doesn't exist. """ def registerSoftwareRelease(software_release): """ Instanciate a software release in the slap library. software_release -- uri of the software release definition """ def registerOpenOrder(): """ Instanciate an open order in the slap library. """ def registerSupply(): """ Instanciate a supply in the slap library. """ slapos.core-0.35.1/slapos/slap/slap.py0000644000076600007660000006713312110424431025671 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2010, 2011, 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly adviced to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public License # as published by the Free Software Foundation; either version 2.1 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## """ Simple, easy to (un)marshall classes for slap client/server communication """ __all__ = ["slap", "ComputerPartition", "Computer", "SoftwareRelease", "Supply", "OpenOrder", "NotFoundError", "Unauthorized", "ResourceNotReady", "ServerError"] from interface import slap as interface from xml_marshaller import xml_marshaller import httplib import logging import socket import ssl import traceback import urllib import urlparse import zope.interface log = logging.getLogger(__name__) DEFAULT_SOFTWARE_TYPE = 'default' # httplib.HTTPSConnection with key verification class HTTPSConnectionCA(httplib.HTTPSConnection): """Patched version of HTTPSConnection which verifies server certificate""" def __init__(self, *args, **kwargs): self.ca_file = kwargs.pop('ca_file') if self.ca_file is None: raise ValueError('ca_file is required argument.') httplib.HTTPSConnection.__init__(self, *args, **kwargs) def connect(self): "Connect to a host on a given (SSL) port and verify its certificate." sock = socket.create_connection((self.host, self.port), self.timeout, self.source_address) if self._tunnel_host: self.sock = sock self._tunnel() self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ca_certs=self.ca_file, cert_reqs=ssl.CERT_REQUIRED) class SlapDocument: def __init__(self, connection_helper=None): if connection_helper is not None: # Do not require connection_helper to be provided, but when it's not, # cause failures when accessing _connection_helper property. self._connection_helper = connection_helper class SoftwareRelease(SlapDocument): """ Contains Software Release information """ zope.interface.implements(interface.ISoftwareRelease) def __init__(self, software_release=None, computer_guid=None, **kw): """ Makes easy initialisation of class parameters XXX **kw args only kept for compatibility """ SlapDocument.__init__(self, kw.pop('connection_helper', None)) self._software_instance_list = [] if software_release is not None: software_release = software_release.encode('UTF-8') self._software_release = software_release self._computer_guid = computer_guid def __getinitargs__(self): return (self._software_release, self._computer_guid, ) def getComputerId(self): if not self._computer_guid: raise NameError('computer_guid has not been defined.') else: return self._computer_guid def getURI(self): if not self._software_release: raise NameError('software_release has not been defined.') else: return self._software_release def error(self, error_log): try: # Does not follow interface self._connection_helper.POST('/softwareReleaseError', { 'url': self.getURI(), 'computer_id' : self.getComputerId(), 'error_log': error_log}) except Exception: exception = traceback.format_exc() log.error(exception) def available(self): self._connection_helper.POST('/availableSoftwareRelease', { 'url': self.getURI(), 'computer_id': self.getComputerId()}) def building(self): self._connection_helper.POST('/buildingSoftwareRelease', { 'url': self.getURI(), 'computer_id': self.getComputerId()}) def destroyed(self): self._connection_helper.POST('/destroyedSoftwareRelease', { 'url': self.getURI(), 'computer_id': self.getComputerId()}) def getState(self): return getattr(self, '_requested_state', 'available') # XXX What is this SoftwareInstance class? class SoftwareInstance(SlapDocument): """ Contains Software Instance information """ def __init__(self, **kwargs): """ Makes easy initialisation of class parameters """ for k, v in kwargs.iteritems(): setattr(self, k, v) """Exposed exceptions""" # XXX Why do we need to expose exceptions? class ResourceNotReady(Exception): zope.interface.implements(interface.IResourceNotReady) class ServerError(Exception): zope.interface.implements(interface.IServerError) class NotFoundError(Exception): zope.interface.implements(interface.INotFoundError) class Unauthorized(Exception): zope.interface.implements(interface.IUnauthorized) class Supply(SlapDocument): zope.interface.implements(interface.ISupply) def supply(self, software_release, computer_guid=None, state='available'): try: self._connection_helper.POST('/supplySupply', { 'url': software_release, 'computer_id': computer_guid, 'state': state}) except NotFoundError: raise NotFoundError("Computer %s has not been found by SlapOS Master." % computer_guid) class OpenOrder(SlapDocument): zope.interface.implements(interface.IOpenOrder) def request(self, software_release, partition_reference, partition_parameter_kw=None, software_type=None, filter_kw=None, state=None, shared=False): if partition_parameter_kw is None: partition_parameter_kw = {} if filter_kw is None: filter_kw = {} request_dict = { 'software_release': software_release, 'partition_reference': partition_reference, 'partition_parameter_xml': xml_marshaller.dumps(partition_parameter_kw), 'filter_xml': xml_marshaller.dumps(filter_kw), # XXX Cedric: Why state and shared are marshalled? First is a string # And second is a boolean. 'state': xml_marshaller.dumps(state), 'shared_xml': xml_marshaller.dumps(shared), } if software_type is not None: request_dict['software_type'] = software_type else: # Let's enforce a default software type request_dict['software_type'] = DEFAULT_SOFTWARE_TYPE try: self._connection_helper.POST('/requestComputerPartition', request_dict) except ResourceNotReady: return ComputerPartition( request_dict=request_dict, connection_helper=self._connection_helper, ) else: xml = self._connection_helper.response.read() software_instance = xml_marshaller.loads(xml) computer_partition = ComputerPartition( software_instance.slap_computer_id.encode('UTF-8'), software_instance.slap_computer_partition_id.encode('UTF-8'), connection_helper=self._connection_helper, ) if shared: computer_partition._synced = True computer_partition._connection_dict = software_instance._connection_dict computer_partition._parameter_dict = software_instance._parameter_dict return computer_partition def requestComputer(self, computer_reference): """ Requests a computer. """ self._connection_helper.POST('/requestComputer', {'computer_title': computer_reference}) xml = self._connection_helper.response.read() computer = xml_marshaller.loads(xml) computer._connection_helper = self._connection_helper return computer def _syncComputerInformation(func): """ Synchronize computer object with server information """ def decorated(self, *args, **kw): if getattr(self, '_synced', 0): return func(self, *args, **kw) computer = self._connection_helper.getFullComputerInformation(self._computer_id) for key, value in computer.__dict__.items(): if isinstance(value, unicode): # convert unicode to utf-8 setattr(self, key, value.encode('utf-8')) else: setattr(self, key, value) setattr(self, '_synced', True) for computer_partition in self.getComputerPartitionList(): setattr(computer_partition, '_synced', True) return func(self, *args, **kw) return decorated class Computer(SlapDocument): zope.interface.implements(interface.IComputer) def __init__(self, computer_id, connection_helper=None): SlapDocument.__init__(self, connection_helper) self._computer_id = computer_id def __getinitargs__(self): return (self._computer_id, ) @_syncComputerInformation def getSoftwareReleaseList(self): """ Returns the list of software release which has to be supplied by the computer. Raise an INotFoundError if computer_guid doesn't exist. """ for software_relase in self._software_release_list: software_relase._connection_helper = self._connection_helper return self._software_release_list @_syncComputerInformation def getComputerPartitionList(self): for computer_partition in self._computer_partition_list: computer_partition._connection_helper = self._connection_helper return [x for x in self._computer_partition_list ] def reportUsage(self, computer_usage): if computer_usage == "": return self._connection_helper.POST('/useComputer', { 'computer_id': self._computer_id, 'use_string': computer_usage}) def updateConfiguration(self, xml): self._connection_helper.POST( '/loadComputerConfigurationFromXML', { 'xml' : xml }) return self._connection_helper.response.read() def bang(self, message): self._connection_helper.POST('/computerBang', { 'computer_id': self._computer_id, 'message': message}) def getStatus(self): self._connection_helper.GET( '/getComputerStatus?computer_id=%s' % self._computer_id) return xml_marshaller.loads(self._connection_helper.response.read()) def revokeCertificate(self): self._connection_helper.POST('/revokeComputerCertificate', { 'computer_id': self._computer_id}) def generateCertificate(self): self._connection_helper.POST('/generateComputerCertificate', { 'computer_id': self._computer_id}) xml = self._connection_helper.response.read() return xml_marshaller.loads(xml) def _syncComputerPartitionInformation(func): """ Synchronize computer partition object with server information """ def decorated(self, *args, **kw): if getattr(self, '_synced', 0): return func(self, *args, **kw) if not self._computer_id: # XXX Is it only in case of requesting instance? raise ResourceNotReady("Instance is not ready yet.") # XXX: This is a ugly way to keep backward compatibility, # We should stablise slap library soon. computer = self._connection_helper.getFullComputerInformation(self._computer_id) found_computer_partition = None for computer_partition in computer._computer_partition_list: if computer_partition.getId() == self.getId(): found_computer_partition = computer_partition break if found_computer_partition is None: raise NotFoundError("No information for partition %s" % self.getId()) else: for key, value in found_computer_partition.__dict__.items(): if isinstance(value, unicode): # convert unicode to utf-8 setattr(self, key, value.encode('utf-8')) if isinstance(value, dict): new_dict = {} for ink, inv in value.iteritems(): if isinstance(inv, (list, tuple)): new_inv = [] for elt in inv: if isinstance(elt, (list, tuple)): new_inv.append([x.encode('utf-8') for x in elt]) elif isinstance(elt, dict): new_inv.append(dict([(x.encode('utf-8'), y and y.encode("utf-8")) for x,y in elt.iteritems()])) else: new_inv.append(elt.encode('utf-8')) new_dict[ink.encode('utf-8')] = new_inv elif inv is None: new_dict[ink.encode('utf-8')] = None else: new_dict[ink.encode('utf-8')] = inv.encode('utf-8') setattr(self, key, new_dict) else: setattr(self, key, value) setattr(self, '_synced', True) return func(self, *args, **kw) return decorated class ComputerPartition(SlapDocument): zope.interface.implements(interface.IComputerPartition) def __init__(self, computer_id=None, partition_id=None, request_dict=None, connection_helper=None): SlapDocument.__init__(self, connection_helper) if request_dict is not None and (computer_id is not None or partition_id is not None): raise TypeError('request_dict conflicts with computer_id and ' 'partition_id') if request_dict is None and (computer_id is None or partition_id is None): raise TypeError('computer_id and partition_id or request_dict are ' 'required') self._computer_id = computer_id self._partition_id = partition_id self._request_dict = request_dict def __getinitargs__(self): return (self._computer_id, self._partition_id, ) @_syncComputerPartitionInformation def request(self, software_release, software_type, partition_reference, shared=False, partition_parameter_kw=None, filter_kw=None, state=None): if partition_parameter_kw is None: partition_parameter_kw = {} elif not isinstance(partition_parameter_kw, dict): raise ValueError("Unexpected type of partition_parameter_kw '%s'" % \ partition_parameter_kw) if filter_kw is None: filter_kw = {} elif not isinstance(filter_kw, dict): raise ValueError("Unexpected type of filter_kw '%s'" % \ filter_kw) # Let enforce a default software type if software_type is None: software_type = DEFAULT_SOFTWARE_TYPE request_dict = { 'computer_id': self._computer_id, 'computer_partition_id': self._partition_id, 'software_release': software_release, 'software_type': software_type, 'partition_reference': partition_reference, 'shared_xml': xml_marshaller.dumps(shared), 'partition_parameter_xml': xml_marshaller.dumps( partition_parameter_kw), 'filter_xml': xml_marshaller.dumps(filter_kw), 'state': xml_marshaller.dumps(state), } try: self._connection_helper.POST('/requestComputerPartition', request_dict) except ResourceNotReady: return ComputerPartition( request_dict=request_dict, connection_helper=self._connection_helper, ) else: xml = self._connection_helper.response.read() software_instance = xml_marshaller.loads(xml) computer_partition = ComputerPartition( software_instance.slap_computer_id.encode('UTF-8'), software_instance.slap_computer_partition_id.encode('UTF-8'), connection_helper=self._connection_helper, ) if shared: computer_partition._synced = True computer_partition._connection_dict = getattr(software_instance, '_connection_dict', None) computer_partition._parameter_dict = getattr(software_instance, '_parameter_dict', None) return computer_partition def building(self): self._connection_helper.POST('/buildingComputerPartition', { 'computer_id': self._computer_id, 'computer_partition_id': self.getId()}) def available(self): self._connection_helper.POST('/availableComputerPartition', { 'computer_id': self._computer_id, 'computer_partition_id': self.getId()}) def destroyed(self): self._connection_helper.POST('/destroyedComputerPartition', { 'computer_id': self._computer_id, 'computer_partition_id': self.getId(), }) def started(self): self._connection_helper.POST('/startedComputerPartition', { 'computer_id': self._computer_id, 'computer_partition_id': self.getId(), }) def stopped(self): self._connection_helper.POST('/stoppedComputerPartition', { 'computer_id': self._computer_id, 'computer_partition_id': self.getId(), }) def error(self, error_log): try: self._connection_helper.POST('/softwareInstanceError', { 'computer_id': self._computer_id, 'computer_partition_id': self.getId(), 'error_log': error_log}) except Exception: exception = traceback.format_exc() log.error(exception) def bang(self, message): self._connection_helper.POST('/softwareInstanceBang', { 'computer_id': self._computer_id, 'computer_partition_id': self.getId(), 'message': message}) def rename(self, new_name, slave_reference=None): post_dict = dict( computer_id=self._computer_id, computer_partition_id=self.getId(), new_name=new_name, ) if slave_reference is not None: post_dict.update(slave_reference=slave_reference) self._connection_helper.POST('/softwareInstanceRename', post_dict) def getId(self): if not self._partition_id: raise ResourceNotReady() return self._partition_id @_syncComputerPartitionInformation def getInstanceGuid(self): """Sync if not synced, then returns instance_guid""" if not getattr(self, '_instance_guid', None): raise ResourceNotReady() return self._instance_guid @_syncComputerPartitionInformation def getState(self): """Sync if not synced, then returns _requested_state.""" if not getattr(self, '_requested_state', None): raise ResourceNotReady() return self._requested_state @_syncComputerPartitionInformation def getInstanceParameterDict(self): return getattr(self, '_parameter_dict', None) or {} @_syncComputerPartitionInformation def getConnectionParameterDict(self): return getattr(self, '_connection_dict', None) or {} @_syncComputerPartitionInformation def getSoftwareRelease(self): """ Returns the software release associate to the computer partition. """ if not getattr(self, '_software_release_document', None): raise NotFoundError("No software release information for partition %s" % self.getId()) else: return self._software_release_document def setConnectionDict(self, connection_dict, slave_reference=None): if self.getConnectionParameterDict() != connection_dict: self._connection_helper.POST('/setComputerPartitionConnectionXml', { 'computer_id': self._computer_id, 'computer_partition_id': self._partition_id, 'connection_xml': xml_marshaller.dumps(connection_dict), 'slave_reference': slave_reference}) @_syncComputerPartitionInformation def getInstanceParameter(self, key): parameter_dict = getattr(self, '_parameter_dict', None) or {} if key in parameter_dict: return parameter_dict[key] else: raise NotFoundError("%s not found" % key) @_syncComputerPartitionInformation def getConnectionParameter(self, key): connection_dict = getattr(self, '_connection_dict', None) or {} if key in connection_dict: return connection_dict[key] else: raise NotFoundError("%s not found" % key) def setUsage(self, usage_log): # XXX: this implementation has not been reviewed self.usage = usage_log def getCertificate(self): self._connection_helper.GET( '/getComputerPartitionCertificate?computer_id=%s&' 'computer_partition_id=%s' % (self._computer_id, self._partition_id)) return xml_marshaller.loads(self._connection_helper.response.read()) def getStatus(self): self._connection_helper.GET( '/getComputerPartitionStatus?computer_id=%s&' 'computer_partition_id=%s' % (self._computer_id, self._partition_id)) return xml_marshaller.loads(self._connection_helper.response.read()) class ConnectionHelper: error_message_timeout = "\nThe connection timed out. Please try again later." error_message_connect_fail = "Couldn't connect to the server. Please " \ "double check given master-url argument, and make sure that IPv6 is " \ "enabled on your machine and that the server is available. The " \ "original error was: " ssl_error_message_connect_fail = "\nCouldn't authenticate computer. Please "\ "check that certificate and key exist and are valid. " def __init__(self, connection_wrapper, host, path, key_file=None, cert_file=None, master_ca_file=None, timeout=None): self.connection_wrapper = connection_wrapper self.host = host self.path = path self.key_file = key_file self.cert_file = cert_file self.master_ca_file = master_ca_file self.timeout = timeout def getComputerInformation(self, computer_id): self.GET('/getComputerInformation?computer_id=%s' % computer_id) return xml_marshaller.loads(self.response.read()) def getFullComputerInformation(self, computer_id): """ Retrieve from SlapOS Master Computer instance containing all needed informations (Software Releases, Computer Partitions, ...). """ method = '/getFullComputerInformation?computer_id=%s' % computer_id if not computer_id: # XXX-Cedric: should raise something smarter than "NotFound". raise NotFoundError(method) try: self.GET(method) except NotFoundError: # XXX: This is a ugly way to keep backward compatibility, # We should stablise slap library soon. self.GET('/getComputerInformation?computer_id=%s' % computer_id) return xml_marshaller.loads(self.response.read()) def connect(self): connection_dict = dict( host=self.host) if self.key_file and self.cert_file: connection_dict.update( key_file=self.key_file, cert_file=self.cert_file) if self.master_ca_file is not None: connection_dict.update(ca_file=self.master_ca_file) self.connection = self.connection_wrapper(**connection_dict) def GET(self, path): try: default_timeout = socket.getdefaulttimeout() socket.setdefaulttimeout(self.timeout) try: self.connect() self.connection.request('GET', self.path + path) self.response = self.connection.getresponse() # If ssl error : may come from bad configuration except ssl.SSLError, e: if e.message == "The read operation timed out": raise socket.error(str(e) + self.error_message_timeout) raise ssl.SSLError(str(e) + self.ssl_error_message_connect_fail) except socket.error, e: if e.message == "timed out": raise socket.error(str(e) + self.error_message_timeout) raise socket.error(self.error_message_connect_fail + str(e)) # check self.response.status and raise exception early if self.response.status == httplib.REQUEST_TIMEOUT: # resource is not ready raise ResourceNotReady(path) elif self.response.status == httplib.NOT_FOUND: raise NotFoundError(path) elif self.response.status == httplib.FORBIDDEN: raise Unauthorized(path) elif self.response.status != httplib.OK: message = 'Server responded with wrong code %s with %s' % \ (self.response.status, path) raise ServerError(message) finally: socket.setdefaulttimeout(default_timeout) def POST(self, path, parameter_dict, content_type="application/x-www-form-urlencoded"): try: default_timeout = socket.getdefaulttimeout() socket.setdefaulttimeout(self.timeout) try: self.connect() header_dict = {'Content-type': content_type} self.connection.request("POST", self.path + path, urllib.urlencode(parameter_dict), header_dict) # If ssl error : must come from bad configuration except ssl.SSLError, e: raise ssl.SSLError(self.ssl_error_message_connect_fail + str(e)) except socket.error, e: raise socket.error(self.error_message_connect_fail + str(e)) self.response = self.connection.getresponse() # check self.response.status and raise exception early if self.response.status == httplib.REQUEST_TIMEOUT: # resource is not ready raise ResourceNotReady("%s - %s" % (path, parameter_dict)) elif self.response.status == httplib.NOT_FOUND: raise NotFoundError("%s - %s" % (path, parameter_dict)) elif self.response.status == httplib.FORBIDDEN: raise Unauthorized("%s - %s" % (path, parameter_dict)) elif self.response.status != httplib.OK: message = 'Server responded with wrong code %s with %s' % \ (self.response.status, path) raise ServerError(message) finally: socket.setdefaulttimeout(default_timeout) class slap: zope.interface.implements(interface.slap) def initializeConnection(self, slapgrid_uri, key_file=None, cert_file=None, master_ca_file=None, timeout=60): scheme, netloc, path, query, fragment = urlparse.urlsplit( slapgrid_uri) if not(query == '' and fragment == ''): raise AttributeError('Passed URL %r issue: not parseable'% slapgrid_uri) if scheme == 'http': connection_wrapper = httplib.HTTPConnection elif scheme == 'https': if master_ca_file is not None: connection_wrapper = HTTPSConnectionCA else: connection_wrapper = httplib.HTTPSConnection else: raise AttributeError('Passed URL %r issue: there is no support for %r p' 'rotocol' % (slapgrid_uri, scheme)) self._connection_helper = ConnectionHelper(connection_wrapper, netloc, path, key_file, cert_file, master_ca_file, timeout) # XXX-Cedric: this method is never used and thus should be removed. def registerSoftwareRelease(self, software_release): """ Registers connected representation of software release and returns SoftwareRelease class object """ return SoftwareRelease(software_release=software_release, connection_helper=self._connection_helper, ) def registerComputer(self, computer_guid): """ Registers connected representation of computer and returns Computer class object """ return Computer(computer_guid, connection_helper=self._connection_helper) def registerComputerPartition(self, computer_guid, partition_id): """ Registers connected representation of computer partition and returns Computer Partition class object """ if not computer_guid or not partition_id: # XXX-Cedric: should raise something smarter than NotFound raise NotFoundError self._connection_helper.GET('/registerComputerPartition?' \ 'computer_reference=%s&computer_partition_reference=%s' % ( computer_guid, partition_id)) result = xml_marshaller.loads(self._connection_helper.response.read()) # XXX: dirty hack to make computer partition usable. xml_marshaller is too # low-level for our needs here. result._connection_helper = self._connection_helper return result def registerOpenOrder(self): return OpenOrder(connection_helper=self._connection_helper) def registerSupply(self): return Supply(connection_helper=self._connection_helper) slapos.core-0.35.1/slapos/slap/slap_recipe_mysqldatabase.py.txt0000644000076600007660000000620612030324717032751 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly adviced to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## from slapos.slap import slap, ComputerPartition, ResourceNotReady import sys import MySQLDB class Recipe: def __init__(self, buildout, options): self.buildout, self.options = buildout, options def install(self): # register self on slap # online computer partition, slap gave connector computer_partition = slap(self.options['vifib_server_url']).register( ComputerPartition, coordinate_kw=dict( computer_id=self.options['computer_id'], # c09fba86-b690-493d-b2b5-03a0df580b28 partition_id=self.options['partition_id'], # 007 resource_url=self.options['resource_url'], # http://slapgrid.org/resource/mysqldatabase-5.4.5 ) ) # request mysql server instance in version 5.4.5 mysql_server_partition = computer_partition.request( self.options['parent_resource_url'], # http://slapgrid.org/resource/mysqlserver-5.4.5 self.options['parent_resource_id'], # main_server ) # invoke installation try: connect = MySQLDB(mysql_server_partition.getIP(), mysql_server_partition.getPort(), mysql_server_partition.getUser(), mysql_server_partition.getPassword()).connect() except ResourceNotReady: # accept asynchronous mode self.logger.info('Not yet available, postponing...') return [] if not connect.isDatabaseCreated(self.options['database_name']): computer_partition.building() try: connect.query('CREATE DATABASE IF NOT EXISTS %s' % self.options['database_name']) except: # issue during installation message = 'Issue during creation %s:%s' % sys.traceback_info_as_list() computer_partition.error(message) self.logger.error(message) if connect.isDatabaseCreated(self.options['database_name']): computer_partition.available() return [] update = install slapos.core-0.35.1/slapos/slapos.xsd0000644000076600007660000000167112030324717025444 0ustar cedricdesaintmartincedricdesaintmartin00000000000000 slapos.core-0.35.1/slapos/tests/0000755000076600007660000000000012110425554024560 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/tests/__init__.py0000644000076600007660000000000012030324717026657 0ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/tests/entry.py0000644000076600007660000001377512035516223026310 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2012 Vifib SARL and Contributors. All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import os import shutil import slapos.entry as entry import sys import tempfile import unittest class BasicMixin: def setUp(self): self._tempdir = tempfile.mkdtemp() self._original_sys_argv = sys.argv sys.argv = ['entry'] def tearDown(self): shutil.rmtree(self._tempdir, True) sys.argv = self._original_sys_argv class TestcheckSlaposCfg (BasicMixin, unittest.TestCase): """ Tests on checkSlaposCfg function """ def test_slapos_cfg_detection_when_no_configuration_given(self): """ If no configuration file is given then False is return """ sys.argv = ['entry', '--logfile', '/log/file'] self.assertFalse(entry.checkSlaposCfg()) def test_slapos_cfg_detection_when_file_does_not_exists(self): """ If given configuration file does not exists then False is return """ slapos_cfg = os.path.join(self._tempdir, 'slapos.cfg') sys.argv = ['entry', slapos_cfg] self.assertFalse(entry.checkSlaposCfg()) def test_slapos_cfg_detection_when_no_slapos_section(self): """ If given configuration file does not have slapos section then False is return """ slapos_cfg = os.path.join(self._tempdir, 'slapos.cfg') open(slapos_cfg, 'w').write('[slapformat]') sys.argv = ['entry', slapos_cfg] self.assertFalse(entry.checkSlaposCfg()) def test_slapos_cfg_detection_when_correct(self): """ If given configuration file have slapos section then True is return """ slapos_cfg = os.path.join(self._tempdir, 'slapos.cfg') open(slapos_cfg, 'w').write('[slapos]') sys.argv.append(slapos_cfg) self.assertTrue(entry.checkSlaposCfg()) class TestcheckOption (BasicMixin, unittest.TestCase): def test_present_option_is_not_added(self): """ If the option is already there do not add it """ sys.argv += ['-vc', '--logfile', '/opt/slapos/slapos.log'] original_sysargv = sys.argv entry.checkOption("--logfile /opt/slapos/format.log") self.assertEqual(original_sysargv, sys.argv) def test_missing_option_is_added(self): """ If the option is not there add it """ sys.argv += ['-vc', '--pidfile', '/opt/slapos/slapos.pid'] original_sysargv = sys.argv option = "--logfile /opt/slapgrid/slapformat.log" entry.checkOption(option) self.assertNotEqual(original_sysargv, sys.argv) self.assertTrue(option in " ".join(sys.argv)) class TestCall (BasicMixin, unittest.TestCase): """ Testing call function """ def test_config_and_option_are_added(self): """ Test missing options and config are added """ sys.argv += ['-vc'] original_sysargv = sys.argv def fun(): return 0 options = ["--logfile /opt/slapos/logfile", "--pidfile /opt/slapos/pidfile"] config = '/etc/opt/slapos/slapos.cfg' try: entry.call(fun, config=config, option=options) except SystemExit, e: self.assertEqual(e[0], 0) self.assertNotEqual(original_sysargv, sys.argv) for x in options: self.assertTrue(x in " ".join(sys.argv)) self.assertEqual(config, sys.argv[1]) def test_config_and_missing_option_are_added(self): """ Test missing options and config are added but do not replace already present option """ missing_option = "--logfile /opt/slapos/logfile" present_option = "--pidfile /opt/slapos/pidfile" default_present_option = "--pidfile /opt/slapos/pidfile.default" sys.argv += ['-vc', present_option] original_sysargv = sys.argv def fun(): return 0 options = [default_present_option, missing_option] config = '/etc/opt/slapos/slapos.cfg' try: entry.call(fun, config=config, option=options) except SystemExit, e: self.assertEqual(e[0], 0) self.assertNotEqual(original_sysargv, sys.argv) for x in (missing_option, present_option): self.assertTrue(x in " ".join(sys.argv)) self.assertFalse(default_present_option in " ".join(sys.argv)) self.assertEqual(config, sys.argv[1]) def test_present_config_and_option_are_not_added(self): """ Test already present options and config are not added """ present_option = "--pidfile /opt/slapos/pidfile" default_present_option = "--pidfile /opt/slapos/pidfile.default" slapos_cfg = os.path.join(self._tempdir, 'slapos.cfg') open(slapos_cfg, 'w').write('[slapos]') sys.argv += ['-vc', slapos_cfg, present_option.split()[0], present_option.split()[1]] original_sysargv = sys.argv def fun(): return 0 options = [default_present_option] config = '/etc/opt/slapos/slapos.cfg' try: entry.call(fun, config=config, option=options) except SystemExit, e: self.assertEqual(e[0], 0) self.assertEqual(original_sysargv, sys.argv) slapos.core-0.35.1/slapos/tests/interface.py0000644000076600007660000000775712030324717027112 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly adviced to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import unittest from zope.interface.verify import verifyClass import zope.interface import types from slapos import slap def getOnlyImplementationAssertionMethod(klass, method_list): """Returns method which verifies if a klass only implements its interfaces""" def testMethod(self): implemented_method_list = [x for x in dir(klass) \ if ((not x.startswith('_')) and callable(getattr(klass, x)))] for interface_method in method_list: if interface_method in implemented_method_list: implemented_method_list.remove(interface_method) if implemented_method_list: raise AssertionError("Unexpected methods %s" % implemented_method_list) return testMethod def getImplementationAssertionMethod(klass, interface): """Returns method which verifies if interface is properly implemented by klass""" def testMethod(self): verifyClass(interface, klass) return testMethod def getDeclarationAssertionMethod(klass): """Returns method which verifies if klass is declaring interface""" def testMethod(self): self.assertNotEqual(0, len(list(zope.interface.implementedBy(klass)))) return testMethod def generateTestMethodListOnClass(klass, module): """Generate test method on klass""" for class_id in dir(module): implementing_class = getattr(module, class_id) if type(implementing_class) not in (types.ClassType, types.TypeType): continue # add methods to assert that publicly available classes are defining # interfaces method_name = 'test_%s_declares_interface' % (class_id,) setattr(klass, method_name, getDeclarationAssertionMethod( implementing_class)) implemented_method_list = [] for interface in list(zope.interface.implementedBy(implementing_class)): # for each interface which class declares add a method which verify # implementation method_name = 'test_%s_implements_%s' % (class_id, interface.__identifier__) setattr(klass, method_name, getImplementationAssertionMethod( implementing_class, interface)) for interface_klass in interface.__iro__: implemented_method_list.extend(interface_klass.names()) # for each interface which class declares, check that no other method are # available method_name = 'test_%s_only_implements' % class_id setattr(klass, method_name, getOnlyImplementationAssertionMethod( implementing_class, implemented_method_list)) class TestInterface(unittest.TestCase): """Tests all publicly available classes of slap Classes are checked *if* they implement interface and if the implementation is correct. """ # add methods to test class generateTestMethodListOnClass(TestInterface, slap) if __name__ == '__main__': unittest.main() slapos.core-0.35.1/slapos/tests/mock/0000755000076600007660000000000012110425554025511 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/tests/mock/__init__.py0000644000076600007660000000000012030324717027610 0ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/tests/mock/httplib.py0000644000076600007660000000610712063614456027545 0ustar cedricdesaintmartincedricdesaintmartin00000000000000#!/usr/bin/env python2 # -*- coding: utf-8 -*- """Mocked httplib """ __all__ = [] def log(message): """Need to be overridden to get a proper logger """ pass class HTTPConnection(object): scheme = 'http' def _callback(self, path, method, body, headers): """To get it works properly, you need to override HTTPConnection._callback. This method received the instance, the path, method and request body as parameter, and it has to return a tuple with headers dictionary and body response string. @param self object instance reference @param URL the parsed URL @param method the http method @param body the request body @param headers the request headers @return tuple containing status integer, headers dictionary and body response""" return (0, {}, '', ) def __init__(self, host, port=None, strict=None, timeout=None, source_address=None): self.host = host self.port = port self.strict = strict self.timeout = timeout self.source_address = source_address self.__response = None def request(self, method, url, body=None, headers=None): status, headers, body = self._callback(url, method, body, headers) self.__response = HTTPResponse('HTTP/1.1', status, 'OK', body, headers) def getresponse(self): response = self.__response self.__response = None return response def set_debuglevel(self, level): pass def set_tunnel(self, host, port=None, headers=None): pass def connect(self): pass def close(self): pass def putrequest(self, request, selector, skip_host=None, skip_accept_encoding=None): pass def putheader(self, *args): pass def endheaders(self): pass def send(self, data): pass class HTTPSConnection(HTTPConnection): def __init__(self, host, port=None, key_file=None, cert_file=None, strict=None, timeout=None, source_address=None): super(HTTPSConnection, self).__init__(host, port, strict, timeout, source_address) self.certificate = open(cert_file, 'r').read() self.key = open(key_file, 'r').read() def request(self, method, url, body=None, headers=None): headers['certificate'] = self.certificate headers['key'] = self.key status, headers, body = self._callback(url, method, body, headers) self.__response = HTTPResponse('HTTP/1.1', status, 'OK', body, headers) def getresponse(self): response = self.__response self.__response = None return response class HTTPResponse(object): def __init__(self, version, status, reason, content, headers=()): self.version = version self.status = status self.reason = reason self.__headers = headers self.__content = content def read(self, amt=None): result = None if amt is None: result = self.__content self.__content = '' else: end = max(amt, len(self.__content)) result = self.__content[:end] del self.__content[:end] return result def getheader(self, name, default=None): pass def getheaders(self): pass slapos.core-0.35.1/slapos/tests/pyflakes/0000755000076600007660000000000012110425554026376 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos/tests/pyflakes/__init__.py0000644000076600007660000000424512073300773030517 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2013 Vifib SARL and Contributors. All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly adviced to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import os import pkg_resources import pyflakes.scripts.pyflakes import sys import unittest2 class CheckCodeConsistency(unittest2.TestCase): """Lints all SlapOS Node and SLAP library code base.""" def setUp(self): self._original_argv = sys.argv sys.argv = [sys.argv[0], os.path.join( pkg_resources.get_distribution('slapos.core').location, 'slapos', ) ] def tearDown(self): sys.argv = self._original_argv @unittest2.skip('pyflakes test is disabled') def testCodeConsistency(self): if pyflakes.scripts.pyflakes.main.func_code.co_argcount: pyflakes.scripts.pyflakes.main([ os.path.join( pkg_resources.get_distribution('slapos.core').location, 'slapos', )]) else: pyflakes.scripts.pyflakes.main() slapos.core-0.35.1/slapos/tests/slap.py0000644000076600007660000007723512061611713026106 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly adviced to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import xml_marshaller import unittest import urlparse import httplib import slapos.slap import os class UndefinedYetException(Exception): """To catch exceptions which are not yet defined""" class SlapMixin(unittest.TestCase): """ Usefull methods for slap tests """ def setUp(self): self._server_url = os.environ.get('TEST_SLAP_SERVER_URL', None) if self._server_url is None: self._patchHttplib() self.server_url = 'http://localhost/' else: self.server_url = self._server_url print 'Testing against SLAP server %r' % self.server_url self.slap = slapos.slap.slap() self.partition_id = 'PARTITION_01' def tearDown(self): self._unpatchHttplib() def _patchHttplib(self): """Overrides httplib""" import mock.httplib self.saved_httplib = dict() for fake in vars(mock.httplib): self.saved_httplib[fake] = getattr(httplib, fake, None) setattr(httplib, fake, getattr(mock.httplib, fake)) def _unpatchHttplib(self): """Restores httplib overriding""" import httplib for name, original_value in self.saved_httplib.items(): setattr(httplib, name, original_value) del self.saved_httplib def _getTestComputerId(self): """ Returns the computer id used by the test """ return self.id() return os.environ.get('TEST_SLAP_COMPUTER_ID', self.id()) class TestSlap(SlapMixin): """ Test slap against slap server """ def test_slap_initialisation(self): """ Asserts that slap initialisation works properly in case of passing correct url """ slap_instance = slapos.slap.slap() slap_instance.initializeConnection(self.server_url) self.assertTrue( slap_instance._connection_helper.host in self.server_url) self.assertTrue( slap_instance._connection_helper.path in self.server_url) def test_slap_initialisation_wrong_url(self): """ Asserts that slap initialisation raises exception when passed url is not correct """ server_url = 'https://user:pass@server/path/path?parameter=notAcceptable' slap_instance = slapos.slap.slap() self.assertRaises(AttributeError, slap_instance.initializeConnection, server_url) def test_registerComputer_with_new_guid(self): """ Asserts that calling slap.registerComputer with new guid returns Computer object """ computer_guid = self._getTestComputerId() self.slap = slapos.slap.slap() self.slap.initializeConnection(self.server_url) computer = self.slap.registerComputer(computer_guid) self.assertTrue(isinstance(computer, slapos.slap.Computer)) def test_registerComputer_with_existing_guid(self): """ Asserts that calling slap.registerComputer with already used guid returns Computer object """ computer_guid = self._getTestComputerId() self.slap = slapos.slap.slap() self.slap.initializeConnection(self.server_url) computer = self.slap.registerComputer(computer_guid) self.assertTrue(isinstance(computer, slapos.slap.Computer)) computer2 = self.slap.registerComputer(computer_guid) self.assertTrue(isinstance(computer2, slapos.slap.Computer)) # XXX: There is naming conflict in slap library. # SoftwareRelease is currently used as suboject of Slap transmission object def test_registerSoftwareRelease_with_new_uri(self): """ Asserts that calling slap.registerSoftwareRelease with new guid returns SoftwareRelease object """ software_release_uri = 'http://server/' + self._getTestComputerId() self.slap = slapos.slap.slap() self.slap.initializeConnection(self.server_url) software_release = self.slap.registerSoftwareRelease(software_release_uri) self.assertTrue(isinstance(software_release, slapos.slap.SoftwareRelease)) def test_registerSoftwareRelease_with_existing_uri(self): """ Asserts that calling slap.registerSoftwareRelease with already used guid returns SoftwareRelease object """ software_release_uri = 'http://server/' + self._getTestComputerId() self.slap = slapos.slap.slap() self.slap.initializeConnection(self.server_url) software_release = self.slap.registerSoftwareRelease(software_release_uri) self.assertTrue(isinstance(software_release, slapos.slap.SoftwareRelease)) software_release2 = self.slap.registerSoftwareRelease(software_release_uri) self.assertTrue(isinstance(software_release2, slapos.slap.SoftwareRelease)) def test_registerComputerPartition_new_partition_id_known_computer_guid(self): """ Asserts that calling slap.registerComputerPartition on known computer returns ComputerPartition object """ computer_guid = self._getTestComputerId() partition_id = self.partition_id self.slap.initializeConnection(self.server_url) self.slap.registerComputer(computer_guid) def server_response(self, path, method, body, header): parsed_url = urlparse.urlparse(path.lstrip('/')) parsed_qs = urlparse.parse_qs(parsed_url.query) if parsed_url.path == 'registerComputerPartition' and \ parsed_qs['computer_reference'][0] == computer_guid and \ parsed_qs['computer_partition_reference'][0] == partition_id: partition = slapos.slap.ComputerPartition( computer_guid, partition_id) return (200, {}, xml_marshaller.xml_marshaller.dumps(partition)) else: return (404, {}, '') httplib.HTTPConnection._callback = server_response partition = self.slap.registerComputerPartition(computer_guid, partition_id) self.assertTrue(isinstance(partition, slapos.slap.ComputerPartition)) def test_registerComputerPartition_existing_partition_id_known_computer_guid(self): """ Asserts that calling slap.registerComputerPartition on known computer returns ComputerPartition object """ self.test_registerComputerPartition_new_partition_id_known_computer_guid() partition = self.slap.registerComputerPartition(self._getTestComputerId(), self.partition_id) self.assertTrue(isinstance(partition, slapos.slap.ComputerPartition)) def test_registerComputerPartition_unknown_computer_guid(self): """ Asserts that calling slap.registerComputerPartition on unknown computer raises NotFoundError exception """ computer_guid = self._getTestComputerId() self.slap.initializeConnection(self.server_url) partition_id = 'PARTITION_01' def server_response(self, path, method, body, header): parsed_url = urlparse.urlparse(path.lstrip('/')) parsed_qs = urlparse.parse_qs(parsed_url.query) if parsed_url.path == 'registerComputerPartition' and \ parsed_qs['computer_reference'][0] == computer_guid and \ parsed_qs['computer_partition_reference'][0] == partition_id: slapos.slap.ComputerPartition(computer_guid, partition_id) return (404, {}, '') else: return (0, {}, '') httplib.HTTPConnection._callback = server_response self.assertRaises(slapos.slap.NotFoundError, self.slap.registerComputerPartition, computer_guid, partition_id) def test_getFullComputerInformation_empty_computer_guid(self): """ Asserts that calling getFullComputerInformation with empty computer_id raises early, before calling master. """ self.slap.initializeConnection(self.server_url) def server_response(self_httpconnection, path, method, body, header): # Shouldn't even be called self.assertFalse(True) httplib.HTTPConnection._callback = server_response self.assertRaises( slapos.slap.NotFoundError, self.slap._connection_helper.getFullComputerInformation, None) def test_registerComputerPartition_empty_computer_guid(self): """ Asserts that calling registerComputerPartition with empty computer_id raises early, before calling master. """ self.slap.initializeConnection(self.server_url) def server_response(self_httpconnection, path, method, body, header): # Shouldn't even be called self.assertFalse(True) httplib.HTTPConnection._callback = server_response self.assertRaises( slapos.slap.NotFoundError, self.slap.registerComputerPartition, None, 'PARTITION_01') def test_registerComputerPartition_empty_computer_partition_id(self): """ Asserts that calling registerComputerPartition with empty computer_partition_id raises early, before calling master. """ self.slap.initializeConnection(self.server_url) def server_response(self_httpconnection, path, method, body, header): # Shouldn't even be called self.assertFalse(True) httplib.HTTPConnection._callback = server_response self.assertRaises( slapos.slap.NotFoundError, self.slap.registerComputerPartition, self._getTestComputerId(), None) def test_registerComputerPartition_empty_computer_guid_empty_computer_partition_id(self): """ Asserts that calling registerComputerPartition with empty computer_partition_id raises early, before calling master. """ self.slap.initializeConnection(self.server_url) def server_response(self_httpconnection, path, method, body, header): # Shouldn't even be called self.assertFalse(True) httplib.HTTPConnection._callback = server_response self.assertRaises( slapos.slap.NotFoundError, self.slap.registerComputerPartition, None, None) class TestComputer(SlapMixin): """ Tests slapos.slap.slap.Computer class functionality """ def test_computer_getComputerPartitionList_no_partition(self): """ Asserts that calling Computer.getComputerPartitionList without Computer Partitions returns empty list """ computer_guid = self._getTestComputerId() slap = self.slap slap.initializeConnection(self.server_url) def server_response(self, path, method, body, header): parsed_url = urlparse.urlparse(path.lstrip('/')) parsed_qs = urlparse.parse_qs(parsed_url.query) if parsed_url.path == 'registerComputerPartition' and \ 'computer_reference' in parsed_qs and \ 'computer_partition_reference' in parsed_qs: slap_partition = slapos.slap.ComputerPartition( parsed_qs['computer_reference'][0], parsed_qs['computer_partition_reference'][0]) return (200, {}, xml_marshaller.xml_marshaller.dumps(slap_partition)) elif parsed_url.path == 'getFullComputerInformation' and \ 'computer_id' in parsed_qs: slap_computer = slapos.slap.Computer(parsed_qs['computer_id'][0]) slap_computer._software_release_list = [] slap_computer._computer_partition_list = [] return (200, {}, xml_marshaller.xml_marshaller.dumps(slap_computer)) elif parsed_url.path == 'requestComputerPartition': return (408, {}, '') else: return (404, {}, '') httplib.HTTPConnection._callback = server_response computer = self.slap.registerComputer(computer_guid) self.assertEqual(computer.getComputerPartitionList(), []) def _test_computer_empty_computer_guid(self, computer_method): """ Helper method checking if calling Computer method with empty id raises early. """ self.slap.initializeConnection(self.server_url) def server_response(self_httpconnection, path, method, body, header): # Shouldn't even be called self.assertFalse(True) httplib.HTTPConnection._callback = server_response computer = self.slap.registerComputer(None) self.assertRaises( slapos.slap.NotFoundError, getattr(computer, computer_method)) def test_computer_getComputerPartitionList_empty_computer_guid(self): """ Asserts that calling getComputerPartitionList with empty computer_guid raises early, before calling master. """ self._test_computer_empty_computer_guid('getComputerPartitionList') def test_computer_getSoftwareReleaseList_empty_computer_guid(self): """ Asserts that calling getSoftwareReleaseList with empty computer_guid raises early, before calling master. """ self._test_computer_empty_computer_guid('getSoftwareReleaseList') def test_computer_getComputerPartitionList_only_partition(self): """ Asserts that calling Computer.getComputerPartitionList with only Computer Partitions returns empty list """ self.computer_guid = self._getTestComputerId() partition_id = 'PARTITION_01' self.slap = slapos.slap.slap() self.slap.initializeConnection(self.server_url) self.computer = self.slap.registerComputer(self.computer_guid) self.partition = self.slap.registerComputerPartition(self.computer_guid, partition_id) self.assertEqual(self.computer.getComputerPartitionList(), []) @unittest.skip("Not implemented") def test_computer_reportUsage_non_valid_xml_raises(self): """ Asserts that calling Computer.reportUsage with non DTD (not defined yet) XML raises (not defined yet) exception """ self.computer_guid = self._getTestComputerId() self.slap = slapos.slap.slap() self.slap.initializeConnection(self.server_url) self.computer = self.slap.registerComputer(self.computer_guid) non_dtd_xml = """ value """ self.assertRaises(UndefinedYetException, self.computer.reportUsage, non_dtd_xml) @unittest.skip("Not implemented") def test_computer_reportUsage_valid_xml_invalid_partition_raises(self): """ Asserts that calling Computer.reportUsage with DTD (not defined yet) XML which refers to invalid partition raises (not defined yet) exception """ self.computer_guid = self._getTestComputerId() partition_id = 'PARTITION_01' self.slap = slapos.slap.slap() self.slap.initializeConnection(self.server_url) self.computer = self.slap.registerComputer(self.computer_guid) self.partition = self.slap.registerComputerPartition(self.computer_guid, partition_id) # XXX: As DTD is not defined currently proper XML is not known bad_partition_dtd_xml = """ etc/run/wrapper && echo "while :; do echo "Working\\nWorking\\n" ; sleep 0.1; done" >> etc/run/wrapper && chmod 755 etc/run/wrapper """ DAEMON_CONTENT = """#!/bin/sh mkdir -p etc/service && echo "#!/bin/sh" > etc/service/daemon && echo "touch launched if [ -f ./crashed ]; then while :; do echo "Working\\nWorking\\n" ; sleep 0.1; done else touch ./crashed; echo "Failing\\nFailing\\n"; sleep 1; exit 111; fi" >> etc/service/daemon && chmod 755 etc/service/daemon && touch worked """ class BasicMixin: def assertSortedListEqual(self, list1, list2, msg=None): self.assertListEqual(sorted(list1), sorted(list2), msg) def setUp(self): self._tempdir = tempfile.mkdtemp() self.software_root = os.path.join(self._tempdir, 'software') self.instance_root = os.path.join(self._tempdir, 'instance') logging.basicConfig(level=logging.DEBUG) self.setSlapgrid() def setSlapgrid(self, develop=False): if getattr(self, 'master_url', None) is None: self.master_url = 'http://127.0.0.1:80/' self.computer_id = 'computer' self.supervisord_socket = os.path.join(self._tempdir, 'supervisord.sock') self.supervisord_configuration_path = os.path.join(self._tempdir, 'supervisord') self.usage_report_periodicity = 1 self.buildout = None self.grid = slapgrid.Slapgrid(self.software_root, self.instance_root, self.master_url, self.computer_id, self.supervisord_socket, self.supervisord_configuration_path, self.buildout, develop=develop) # monkey patch buildout bootstrap def dummy(*args, **kw): pass slapos.grid.utils.bootstrapBuildout = dummy def launchSlapgrid(self,develop=False): self.setSlapgrid(develop=develop) return self.grid.processComputerPartitionList() def launchSlapgridSoftware(self,develop=False): self.setSlapgrid(develop=develop) return self.grid.processSoftwareReleaseList() def tearDown(self): # XXX: Hardcoded pid, as it is not configurable in slapos svc = os.path.join(self.instance_root, 'var', 'run', 'supervisord.pid') if os.path.exists(svc): try: pid = int(open(svc).read().strip()) except ValueError: pass else: os.kill(pid, signal.SIGTERM) shutil.rmtree(self._tempdir, True) class TestBasicSlapgridCP(BasicMixin, unittest.TestCase): def test_no_software_root(self): self.assertRaises(OSError, self.grid.processComputerPartitionList) def test_no_instance_root(self): os.mkdir(self.software_root) self.assertRaises(OSError, self.grid.processComputerPartitionList) def test_no_master(self): os.mkdir(self.software_root) os.mkdir(self.instance_root) self.assertRaises(socket.error, self.grid.processComputerPartitionList) class MasterMixin(BasicMixin): def _patchHttplib(self): """Overrides httplib""" import mock.httplib self.saved_httplib = dict() for fake in vars(mock.httplib): self.saved_httplib[fake] = getattr(httplib, fake, None) setattr(httplib, fake, getattr(mock.httplib, fake)) def _unpatchHttplib(self): """Restores httplib overriding""" import httplib for name, original_value in self.saved_httplib.items(): setattr(httplib, name, original_value) del self.saved_httplib def _mock_sleep(self): self.fake_waiting_time = None self.real_sleep = time.sleep def mocked_sleep(secs): if self.fake_waiting_time is not None: secs = self.fake_waiting_time self.real_sleep(secs) time.sleep = mocked_sleep def _unmock_sleep(self): time.sleep = self.real_sleep def setUp(self): self._patchHttplib() self._mock_sleep() BasicMixin.setUp(self) def tearDown(self): self._unpatchHttplib() self._unmock_sleep() BasicMixin.tearDown(self) class ComputerForTest: """ Class to set up environment for tests setting instance, software and server response """ def __init__(self, software_root, instance_root, instance_amount=1, software_amount=1): """ Will set up instances, software and sequence """ self.sequence = [] self.instance_amount = instance_amount self.software_amount = software_amount self.software_root = software_root self.instance_root = instance_root if not os.path.isdir(self.instance_root): os.mkdir(self.instance_root) if not os.path.isdir(self.software_root): os.mkdir(self.software_root) self.setSoftwares() self.setInstances() self.setServerResponse() def setSoftwares(self): """ Will set requested amount of software """ self.software_list = range(0,self.software_amount) for i in self.software_list: name = str(i) self.software_list[i] = SoftwareForTest(self.software_root, name=name) def setInstances(self): """ Will set requested amount of instance giving them by default first software """ self.instance_list = range(0, self.instance_amount) for i in self.instance_list: name = str(i) if len(self.software_list) is not 0: software = self.software_list[0] else: software = None self.instance_list[i] = InstanceForTest(self.instance_root, name=name, software=software) def getComputer (self, computer_id): """ Will return current requested state of computer """ slap_computer = slapos.slap.Computer(computer_id) slap_computer._software_release_list = [] slap_computer._computer_partition_list = [] for instance in self.instance_list: slap_computer._computer_partition_list.append( instance.getInstance(computer_id)) for software in self.software_list: slap_computer._software_release_list.append( software.getSoftware(computer_id)) return slap_computer def setServerResponse(self): httplib.HTTPConnection._callback = self.getServerResponse() httplib.HTTPSConnection._callback = self.getServerResponse() def getServerResponse(self): """ Define _callback. Will register global sequence of message, sequence by partition and error and error message by partition """ def server_response(self_httplib, path, method, body, header): parsed_url = urlparse.urlparse(path.lstrip('/')) self.sequence.append(parsed_url.path) if method == 'GET': parsed_qs = urlparse.parse_qs(parsed_url.query) else: parsed_qs = urlparse.parse_qs(body) if parsed_url.path == 'getFullComputerInformation' and \ 'computer_id' in parsed_qs: slap_computer = self.getComputer(parsed_qs['computer_id'][0]) return (200, {}, xml_marshaller.xml_marshaller.dumps(slap_computer)) if method == 'POST' and 'computer_partition_id' in parsed_qs: instance = self.instance_list[int(parsed_qs['computer_partition_id'][0])] instance.sequence.append(parsed_url.path) instance.header_list.append(header) if parsed_url.path == 'availableComputerPartition': return (200, {}, '') if parsed_url.path == 'startedComputerPartition': instance.state = 'started' return (200, {}, '') if parsed_url.path == 'stoppedComputerPartition': instance.state = 'stopped' return (200, {}, '') if parsed_url.path == 'destroyedComputerPartition': instance.state = 'destroyed' return (200, {}, '') if parsed_url.path == 'softwareInstanceBang': return (200, {}, '') if parsed_url.path == 'softwareInstanceError': instance.error_log = '\n'.join([line for line \ in parsed_qs['error_log'][0].splitlines() if 'dropPrivileges' not in line]) instance.error = True return (200, {}, '') elif method == 'POST' and 'url' in parsed_qs: # XXX hardcoded to first sofwtare release! software = self.software_list[0] software.sequence.append(parsed_url.path) if parsed_url.path == 'buildingSoftwareRelease': return (200, {}, '') if parsed_url.path == 'softwareReleaseError': software.error_log = '\n'.join([line for line \ in parsed_qs['error_log'][0].splitlines() if 'dropPrivileges' not in line]) software.error = True return (200, {}, '') else: return (500, {}, '') return server_response class InstanceForTest: """ Class containing all needed paramaters and function to simulate instances """ def __init__(self, instance_root, name, software): self.instance_root = instance_root self.software = software self.requested_state = 'stopped' self.state = None self.error = False self.error_log = None self.sequence = [] self.header_list = [] self.name = name self.partition_path = os.path.join(self.instance_root, self.name) os.mkdir(self.partition_path, 0750) self.timestamp = None def getInstance (self, computer_id): """ Will return current requested state of instance """ partition = slapos.slap.ComputerPartition(computer_id, self.name) partition._software_release_document = self.getSoftwareRelease() partition._requested_state = self.requested_state if self.software is not None: if self.timestamp is not None : partition._parameter_dict = {'timestamp': self.timestamp} return partition def getSoftwareRelease (self): """ Return software release for Instance """ if self.software is not None: sr = slapos.slap.SoftwareRelease() sr._software_release = self.software.name return sr else: return None def setPromise (self, promise_name, promise_content): """ This function will set promise and return its path """ promise_path = os.path.join(self.partition_path, 'etc', 'promise') if not os.path.isdir(promise_path): os.makedirs(promise_path) promise = os.path.join(promise_path, promise_name) open(promise, 'w').write(promise_content) os.chmod(promise, 0777) def setCertificate(self, certificate_repository_path): if not os.path.exists(certificate_repository_path): os.mkdir(certificate_repository_path) self.cert_file = os.path.join(certificate_repository_path, "%s.crt" % self.name) self.certificate = str(random()) open(self.cert_file, 'w').write(self.certificate) self.key_file = os.path.join(certificate_repository_path, "%s.key" % self.name) self.key = str(random()) open(self.key_file, 'w').write(self.key) class SoftwareForTest: """ Class to prepare and simulate software. each instance has a sotfware attributed """ def __init__(self, software_root, name=''): """ Will set file and variable for software """ self.software_root = software_root self.name = 'http://sr%s/' % name self.sequence = [] self.software_hash = \ slapos.grid.utils.getSoftwareUrlHash(self.name) self.srdir = os.path.join(self.software_root, self.software_hash) self.requested_state = 'available' os.mkdir(self.srdir) self.setTemplateCfg() self.srbindir = os.path.join(self.srdir, 'bin') os.mkdir(self.srbindir) self.setBuildout() def getSoftware (self, computer_id): """ Will return current requested state of software """ software = slapos.slap.SoftwareRelease(self.name, computer_id) software._requested_state = self.requested_state return software def setTemplateCfg (self,template="""[buildout]"""): """ Set template.cfg """ open(os.path.join(self.srdir, 'template.cfg'), 'w').write(template) def setBuildout (self, buildout="""#!/bin/sh touch worked"""): """ Set a buildout exec in bin """ open(os.path.join(self.srbindir, 'buildout'), 'w').write(buildout) os.chmod(os.path.join(self.srbindir, 'buildout'), 0755) def setPeriodicity(self,periodicity): """ Set a periodicity file """ open(os.path.join(self.srdir, 'periodicity'), 'w').write( """%s""" % (periodicity)) class TestSlapgridCPWithMaster(MasterMixin, unittest.TestCase): def test_nothing_to_do(self): ComputerForTest(self.software_root,self.instance_root, 0, 0) self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['etc', 'var']) self.assertSortedListEqual(os.listdir(self.software_root), []) def test_one_partition(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) partition = os.path.join(self.instance_root, '0') self.assertSortedListEqual(os.listdir(partition), ['buildout.cfg', 'software_release', 'worked']) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'stoppedComputerPartition']) def test_one_partition_instance_cfg(self): """ Check that slapgrid processes instance is profile is not named "template.cfg" but "instance.cfg". """ computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) partition = os.path.join(self.instance_root, '0') self.assertSortedListEqual(os.listdir(partition), ['buildout.cfg', 'software_release', 'worked']) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'stoppedComputerPartition']) def test_one_free_partition(self): """ Test if slapgrid cp don't process "free" partition """ computer = ComputerForTest(self.software_root,self.instance_root, software_amount = 0) partition = computer.instance_list[0] partition.requested_state = 'destroyed' self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0','etc', 'var']) self.assertSortedListEqual(os.listdir(partition.partition_path), []) self.assertSortedListEqual(os.listdir(self.software_root), []) self.assertEqual(partition.sequence, []) def test_one_partition_started(self): computer = ComputerForTest(self.software_root,self.instance_root) partition = computer.instance_list[0] partition.requested_state = 'started' partition.software.setBuildout(WRAPPER_CONTENT) self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(partition.partition_path), ['.0_wrapper.log', 'buildout.cfg', 'etc', 'software_release', 'worked']) tries = 50 wrapper_log = os.path.join(partition.partition_path, '.0_wrapper.log') while tries > 0: tries -= 1 if os.path.getsize(wrapper_log) > 0: break time.sleep(0.1) self.assertTrue('Working' in open(wrapper_log, 'r').read()) self.assertSortedListEqual(os.listdir(self.software_root), [partition.software.software_hash]) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'startedComputerPartition']) self.assertEqual(partition.state,'started') def test_one_partition_started_stopped(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'started' instance.software.setBuildout("""#!/bin/sh touch worked && mkdir -p etc/run && ( cat <<'HEREDOC' #!%(python)s import signal def handler(signum, frame): print 'Signal handler called with signal', signum raise SystemExit signal.signal(signal.SIGTERM, handler) while True: print "Working" HEREDOC )> etc/run/wrapper && chmod 755 etc/run/wrapper """ % dict(python = sys.executable)) self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), ['.0_wrapper.log', 'buildout.cfg', 'etc', 'software_release', 'worked']) wrapper_log = os.path.join(instance.partition_path, '.0_wrapper.log') tries = 50 while tries > 0: tries -= 1 if os.path.getsize(wrapper_log) > 0: break time.sleep(0.1) os.path.getsize(wrapper_log) self.assertTrue('Working' in open(wrapper_log, 'r').read()) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'startedComputerPartition']) self.assertEqual(instance.state,'started') computer.sequence = [] instance.requested_state = 'stopped' self.assertEqual(self.launchSlapgrid(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), ['.0_wrapper.log', '.0_wrapper.log.1', 'buildout.cfg', 'etc', 'software_release', 'worked']) tries = 50 expected_text = 'Signal handler called with signal 15' while tries > 0: tries -= 1 found = expected_text in open(wrapper_log, 'r').read() if found: break time.sleep(0.1) self.assertTrue(found) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'stoppedComputerPartition']) self.assertEqual(instance.state,'stopped') def test_one_broken_partition_stopped(self): """ Check that, for, an already started instance if stop is requested, processes will be stopped even if instance is broken (buildout fails to run) but status is still started. """ computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'started' instance.software.setBuildout("""#!/bin/sh touch worked && mkdir -p etc/run && ( cat <<'HEREDOC' #!%(python)s import signal def handler(signum, frame): print 'Signal handler called with signal', signum raise SystemExit signal.signal(signal.SIGTERM, handler) while True: print "Working" HEREDOC )> etc/run/wrapper && chmod 755 etc/run/wrapper """ % dict(python = sys.executable)) self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), ['.0_wrapper.log', 'buildout.cfg', 'etc', 'software_release', 'worked']) wrapper_log = os.path.join(instance.partition_path, '.0_wrapper.log') tries = 50 while tries > 0: tries -= 1 if os.path.getsize(wrapper_log) > 0: break time.sleep(0.1) os.path.getsize(wrapper_log) self.assertTrue('Working' in open(wrapper_log, 'r').read()) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'startedComputerPartition']) self.assertEqual(instance.state,'started') computer.sequence = [] instance.requested_state = 'stopped' instance.software.setBuildout("""#!/bin/sh exit 1 """) self.assertEqual(self.launchSlapgrid(), slapgrid.SLAPGRID_FAIL) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), ['.0_wrapper.log', '.0_wrapper.log.1', 'buildout.cfg', 'etc', 'software_release', 'worked']) tries = 50 expected_text = 'Signal handler called with signal 15' while tries > 0: tries -= 1 found = expected_text in open(wrapper_log, 'r').read() if found: break time.sleep(0.1) self.assertTrue(found) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'softwareInstanceError']) self.assertEqual(instance.state, 'started') def test_one_partition_stopped_started(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'stopped' instance.software.setBuildout(WRAPPER_CONTENT) self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) partition = os.path.join(self.instance_root, '0') self.assertSortedListEqual(os.listdir(partition), ['buildout.cfg', 'etc', 'software_release', 'worked']) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'stoppedComputerPartition']) self.assertEqual('stopped',instance.state) instance.requested_state = 'started' computer.sequence = [] self.assertEqual(self.launchSlapgrid(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) partition = os.path.join(self.instance_root, '0') self.assertSortedListEqual(os.listdir(partition), ['.0_wrapper.log', 'etc', 'buildout.cfg', 'software_release', 'worked']) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) tries = 50 wrapper_log = os.path.join(instance.partition_path, '.0_wrapper.log') while tries > 0: tries -= 1 if os.path.getsize(wrapper_log) > 0: break time.sleep(0.1) self.assertTrue('Working' in open(wrapper_log, 'r').read()) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'startedComputerPartition']) self.assertEqual('started',instance.state) def test_one_partition_destroyed(self): """ Test that an existing partition with "destroyed" status will only be stopped by slapgrid-cp, not processed """ computer = ComputerForTest(self.software_root, self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'destroyed' dummy_file_name = 'dummy_file' dummy_file = open( os.path.join(instance.partition_path, dummy_file_name), 'w') dummy_file.write('dummy') dummy_file.close() self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) partition = os.path.join(self.instance_root, '0') self.assertSortedListEqual(os.listdir(partition), [dummy_file_name]) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'stoppedComputerPartition']) self.assertEqual('stopped', instance.state) class TestSlapgridCPWithMasterWatchdog(MasterMixin, unittest.TestCase): def setUp(self): MasterMixin.setUp(self) # Prepare watchdog self.watchdog_banged = os.path.join(self._tempdir, 'watchdog_banged') watchdog_path = os.path.join(self._tempdir, 'watchdog') open(watchdog_path, 'w').write( WATCHDOG_TEMPLATE % dict(python_path=sys.executable, sys_path=sys.path, watchdog_banged=self.watchdog_banged)) os.chmod(watchdog_path, 0755) self.grid.watchdog_path = watchdog_path slapos.grid.slapgrid.WATCHDOG_PATH = watchdog_path def test_one_failing_daemon_in_service_will_bang_with_watchdog(self): """ Check that a failing service watched by watchdog trigger bang 1.Prepare computer and set a service named daemon in etc/service (to be watched by watchdog). This daemon will fail. 2.Prepare file for supervisord to call watchdog -Set sys.path -Monkeypatch computer partition bang 3.Check damemon is launched 4.Wait for it to fail 5.Wait for file generated by monkeypacthed bang to appear """ computer = ComputerForTest(self.software_root, self.instance_root) partition = computer.instance_list[0] partition.requested_state = 'started' partition.software.setBuildout(DAEMON_CONTENT) self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(partition.partition_path), ['.0_daemon.log', 'buildout.cfg', 'etc', 'software_release', 'worked']) tries = 200 daemon_log = os.path.join(partition.partition_path, '.0_daemon.log') while tries > 0: tries -= 1 if os.path.getsize(daemon_log) > 0: break time.sleep(0.1) self.assertTrue('Failing' in open(daemon_log, 'r').read()) tries = 200 while tries > 0: tries -= 1 if os.path.exists(self.watchdog_banged): break time.sleep(0.1) self.assertTrue(os.path.exists(self.watchdog_banged)) self.assertTrue('daemon' in open(self.watchdog_banged, 'r').read()) def test_one_failing_daemon_in_run_will_not_bang_with_watchdog(self): """ Check that a failing service watched by watchdog trigger bang 1.Prepare computer and set a service named daemon in etc/run (not watched by watchdog). This daemon will fail. 2.Prepare file for supervisord to call watchdog -Set sys.path -Monkeypatch computer partition bang 3.Check damemon is launched 4.Wait for it to fail 5.Check that file generated by monkeypacthed bang do not appear """ computer = ComputerForTest(self.software_root,self.instance_root) partition = computer.instance_list[0] partition.requested_state = 'started' RUN_CONTENT = textwrap.dedent("""\ #!/bin/sh mkdir -p etc/run && echo "#!/bin/sh" > etc/run/daemon && echo "touch launched touch ./crashed; echo "Failing\\nFailing\\n"; sleep 1; exit 111; " >> etc/run/daemon && chmod 755 etc/run/daemon && touch worked """) partition.software.setBuildout(RUN_CONTENT) self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(partition.partition_path), ['.0_daemon.log', 'buildout.cfg', 'etc', 'software_release', 'worked']) tries = 200 daemon_log = os.path.join(partition.partition_path, '.0_daemon.log') while tries > 0: tries -= 1 if os.path.getsize(daemon_log) > 0: break time.sleep(0.1) time.sleep(1) self.assertTrue('Failing' in open(daemon_log, 'r').read()) tries = 200 while tries > 0: tries -= 1 if os.path.exists(self.watchdog_banged): break time.sleep(0.1) self.assertFalse(os.path.exists(self.watchdog_banged)) def test_watched_by_watchdog_bang(self): """ Test that a process going to fatal or exited mode in supervisord is banged if watched by watchdog Certificates used for the bang are also checked (ie: watchdog id in process name) """ computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] certificate_repository_path = os.path.join(self._tempdir, 'partition_pki') instance.setCertificate(certificate_repository_path) watchdog = Watchdog(dict(master_url='https://127.0.0.1/', computer_id=self.computer_id, certificate_repository_path=certificate_repository_path)) for event in watchdog.process_state_events: instance.sequence = [] instance.header_list = [] headers = dict(eventname=event) payload = "processname:%s groupname:%s from_state:RUNNING"\ % ('daemon'+getWatchdogID(),instance.name) watchdog.handle_event(headers,payload) self.assertEqual(instance.sequence,['softwareInstanceBang']) self.assertEqual(instance.header_list[0]['key'], instance.key) self.assertEqual(instance.header_list[0]['certificate'], instance.certificate) def test_unwanted_events_will_not_bang(self): """ Test that a process going to a mode not watched by watchdog in supervisord is not banged if watched by watchdog """ computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] watchdog = Watchdog(dict(master_url=self.master_url, computer_id=self.computer_id, certificate_repository_path=None)) for event in ['EVENT', 'PROCESS_STATE', 'PROCESS_STATE_RUNNING', 'PROCESS_STATE_BACKOFF', 'PROCESS_STATE_STOPPED']: computer.sequence = [] headers = dict(eventname=event) payload = "processname:%s groupname:%s from_state:RUNNING"\ % ('daemon'+getWatchdogID(),instance.name) watchdog.handle_event(headers,payload) self.assertEqual(instance.sequence,[]) def test_not_watched_by_watchdog_do_not_bang(self): """ Test that a process going to fatal or exited mode in supervisord is not banged if not watched by watchdog (ie: no watchdog id in process name) """ computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] watchdog = Watchdog(dict(master_url=self.master_url, computer_id=self.computer_id, certificate_repository_path=None)) for event in watchdog.process_state_events: computer.sequence = [] headers = dict(eventname=event) payload = "processname:%s groupname:%s from_state:RUNNING"\ % ('daemon',instance.name) watchdog.handle_event(headers,payload) self.assertEqual(computer.sequence,[]) class TestSlapgridCPPartitionProcessing (MasterMixin, unittest.TestCase): def test_partition_timestamp(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] timestamp = str(int(time.time())) instance.timestamp = timestamp self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) partition = os.path.join(self.instance_root, '0') self.assertSortedListEqual(os.listdir(partition), ['.timestamp', 'buildout.cfg', 'software_release', 'worked']) self.assertSortedListEqual( os.listdir(self.software_root), [instance.software.software_hash]) timestamp_path = os.path.join(instance.partition_path, '.timestamp') self.setSlapgrid() self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertTrue(timestamp in open(timestamp_path,'r').read()) self.assertEqual(instance.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) def test_partition_timestamp_develop(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] timestamp = str(int(time.time())) instance.timestamp = timestamp self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) partition = os.path.join(self.instance_root, '0') self.assertSortedListEqual(os.listdir(partition), ['.timestamp', 'buildout.cfg', 'software_release', 'worked']) self.assertSortedListEqual( os.listdir(self.software_root), [instance.software.software_hash]) self.assertEqual(self.launchSlapgrid(develop=True), slapgrid.SLAPGRID_SUCCESS) self.assertEqual(self.launchSlapgrid(), slapgrid.SLAPGRID_SUCCESS) self.assertEqual(instance.sequence, ['availableComputerPartition', 'stoppedComputerPartition', 'availableComputerPartition','stoppedComputerPartition']) def test_partition_old_timestamp(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] timestamp = str(int(time.time())) instance.timestamp = timestamp self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) partition = os.path.join(self.instance_root, '0') self.assertSortedListEqual(os.listdir(partition), ['.timestamp', 'buildout.cfg', 'software_release', 'worked']) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) instance.timestamp = str(int(timestamp) - 1) self.assertEqual(self.launchSlapgrid(), slapgrid.SLAPGRID_SUCCESS) self.assertEqual(instance.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) def test_partition_timestamp_new_timestamp(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] timestamp = str(int(time.time())) instance.timestamp = timestamp self.assertEqual(self.launchSlapgrid(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) partition = os.path.join(self.instance_root, '0') self.assertSortedListEqual(os.listdir(partition), ['.timestamp', 'buildout.cfg', 'software_release', 'worked']) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) instance.timestamp = str(int(timestamp)+1) self.assertEqual(self.launchSlapgrid(), slapgrid.SLAPGRID_SUCCESS) self.assertEqual(self.launchSlapgrid(), slapgrid.SLAPGRID_SUCCESS) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'stoppedComputerPartition', 'getFullComputerInformation', 'availableComputerPartition','stoppedComputerPartition', 'getFullComputerInformation']) def test_partition_timestamp_no_timestamp(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] timestamp = str(int(time.time())) instance.timestamp = timestamp self.launchSlapgrid() self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) partition = os.path.join(self.instance_root, '0') self.assertSortedListEqual(os.listdir(partition), ['.timestamp', 'buildout.cfg', 'software_release', 'worked']) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) instance.timestamp = None self.launchSlapgrid() self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'stoppedComputerPartition', 'getFullComputerInformation', 'availableComputerPartition','stoppedComputerPartition',]) def test_partition_periodicity_is_not_overloaded_if_forced(self): """ If periodicity file in software directory but periodicity is forced periodicity will be the one given by parameter 1. We set force_periodicity parameter to True 2. We put a periodicity file in the software release directory with an unwanted periodicity 3. We process partition list and wait more than unwanted periodicity 4. We relaunch, partition should not be processed """ computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] timestamp = str(int(time.time())) instance.timestamp = timestamp instance.requested_state = 'started' unwanted_periodicity = 2 instance.software.setPeriodicity(unwanted_periodicity) self.grid.force_periodicity = True self.launchSlapgrid() time.sleep(unwanted_periodicity + 1) self.setSlapgrid() self.grid.force_periodicity = True self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertNotEqual(unwanted_periodicity,self.grid.maximum_periodicity) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'startedComputerPartition', 'getFullComputerInformation']) def test_one_partition_periodicity_from_file_does_not_disturb_others(self): """ If time between last processing of instance and now is superior to periodicity then instance should be proceed 1. We set a wanted maximum_periodicity in periodicity file in in one software release directory and not the other one 2. We process computer partition and check if wanted_periodicity was used as maximum_periodicty 3. We wait for a time superior to wanted_periodicty 4. We launch processComputerPartition and check that partition using software with periodicity was runned and not the other 5. We check that modification time of .timestamp was modified """ computer = ComputerForTest(self.software_root,self.instance_root,20,20) instance0 = computer.instance_list[0] timestamp = str(int(time.time()-5)) instance0.timestamp = timestamp instance0.requested_state = 'started' for instance in computer.instance_list[1:]: instance.software = \ computer.software_list[computer.instance_list.index(instance)] instance.timestamp = timestamp wanted_periodicity = 1 instance0.software.setPeriodicity(wanted_periodicity) self.launchSlapgrid() self.assertNotEqual(wanted_periodicity, self.grid.maximum_periodicity) last_runtime = os.path.getmtime( os.path.join(instance0.partition_path, '.timestamp')) time.sleep(wanted_periodicity + 1) for instance in computer.instance_list[1:]: self.assertEqual(instance.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) time.sleep(1) self.launchSlapgrid() self.assertEqual(instance0.sequence, ['availableComputerPartition', 'startedComputerPartition', 'availableComputerPartition', 'startedComputerPartition', ]) for instance in computer.instance_list[1:]: self.assertEqual(instance.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) self.assertGreater( os.path.getmtime(os.path.join(instance0.partition_path,'.timestamp')), last_runtime) self.assertNotEqual(wanted_periodicity,self.grid.maximum_periodicity) def test_one_partition_stopped_is_not_processed_after_periodicity(self): """ Check that periodicity doesn't force processing a partition if it is not started. """ computer = ComputerForTest(self.software_root,self.instance_root,20,20) instance0 = computer.instance_list[0] timestamp = str(int(time.time()-5)) instance0.timestamp = timestamp for instance in computer.instance_list[1:]: instance.software = \ computer.software_list[computer.instance_list.index(instance)] instance.timestamp = timestamp wanted_periodicity = 1 instance0.software.setPeriodicity(wanted_periodicity) self.launchSlapgrid() self.assertNotEqual(wanted_periodicity, self.grid.maximum_periodicity) last_runtime = os.path.getmtime( os.path.join(instance0.partition_path, '.timestamp')) time.sleep(wanted_periodicity + 1) for instance in computer.instance_list[1:]: self.assertEqual(instance.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) time.sleep(1) self.launchSlapgrid() self.assertEqual(instance0.sequence, ['availableComputerPartition', 'stoppedComputerPartition', ]) for instance in computer.instance_list[1:]: self.assertEqual(instance.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) self.assertEqual( os.path.getmtime(os.path.join(instance0.partition_path,'.timestamp')), last_runtime) self.assertNotEqual(wanted_periodicity,self.grid.maximum_periodicity) def test_one_partition_destroyed_is_not_processed_after_periodicity(self): """ Check that periodicity doesn't force processing a partition if it is not started. """ computer = ComputerForTest(self.software_root,self.instance_root,20,20) instance0 = computer.instance_list[0] timestamp = str(int(time.time()-5)) instance0.timestamp = timestamp instance0.requested_state = 'stopped' for instance in computer.instance_list[1:]: instance.software = \ computer.software_list[computer.instance_list.index(instance)] instance.timestamp = timestamp wanted_periodicity = 1 instance0.software.setPeriodicity(wanted_periodicity) self.launchSlapgrid() self.assertNotEqual(wanted_periodicity, self.grid.maximum_periodicity) last_runtime = os.path.getmtime( os.path.join(instance0.partition_path, '.timestamp')) time.sleep(wanted_periodicity + 1) for instance in computer.instance_list[1:]: self.assertEqual(instance.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) time.sleep(1) instance0.requested_state = 'destroyed' self.launchSlapgrid() self.assertEqual(instance0.sequence, ['availableComputerPartition', 'stoppedComputerPartition', ]) for instance in computer.instance_list[1:]: self.assertEqual(instance.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) self.assertEqual( os.path.getmtime(os.path.join(instance0.partition_path,'.timestamp')), last_runtime) self.assertNotEqual(wanted_periodicity,self.grid.maximum_periodicity) def test_one_partition_buildout_fail_does_not_disturb_others(self): """ 1. We set up two instance one using a corrupted buildout 2. One will fail but the other one will be processed correctly """ computer = ComputerForTest(self.software_root,self.instance_root,2,2) instance0 = computer.instance_list[0] instance1 = computer.instance_list[1] instance1.software = computer.software_list[1] instance0.software.setBuildout("""#!/bin/sh exit 42""") self.launchSlapgrid() self.assertEqual(instance0.sequence, ['softwareInstanceError']) self.assertEqual(instance1.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) def test_one_partition_lacking_software_path_does_not_disturb_others(self): """ 1. We set up two instance but remove software path of one 2. One will fail but the other one will be processed correctly """ computer = ComputerForTest(self.software_root,self.instance_root,2,2) instance0 = computer.instance_list[0] instance1 = computer.instance_list[1] instance1.software = computer.software_list[1] shutil.rmtree(instance0.software.srdir) self.launchSlapgrid() self.assertEqual(instance0.sequence, ['softwareInstanceError']) self.assertEqual(instance1.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) def test_one_partition_lacking_software_bin_path_does_not_disturb_others(self): """ 1. We set up two instance but remove software bin path of one 2. One will fail but the other one will be processed correctly """ computer = ComputerForTest(self.software_root,self.instance_root,2,2) instance0 = computer.instance_list[0] instance1 = computer.instance_list[1] instance1.software = computer.software_list[1] shutil.rmtree(instance0.software.srbindir) self.launchSlapgrid() self.assertEqual(instance0.sequence, ['softwareInstanceError']) self.assertEqual(instance1.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) def test_one_partition_lacking_path_does_not_disturb_others(self): """ 1. We set up two instances but remove path of one 2. One will fail but the other one will be processed correctly """ computer = ComputerForTest(self.software_root,self.instance_root, 2, 2) instance0 = computer.instance_list[0] instance1 = computer.instance_list[1] instance1.software = computer.software_list[1] shutil.rmtree(instance0.partition_path) self.launchSlapgrid() self.assertEqual(instance0.sequence, ['softwareInstanceError']) self.assertEqual(instance1.sequence, ['availableComputerPartition', 'stoppedComputerPartition']) def test_one_partition_buildout_fail_is_correctly_logged(self): """ 1. We set up an instance using a corrupted buildout 2. It will fail, make sure that whole log is sent to master """ computer = ComputerForTest(self.software_root, self.instance_root, 1, 1) instance = computer.instance_list[0] line1 = "Nerdy kitten: Can I has a process crash?" line2 = "Cedric: Sure, here it is." instance.software.setBuildout("""#!/bin/sh echo %s; echo %s; exit 42""" % (line1, line2)) self.launchSlapgrid() self.assertEqual(instance.sequence, ['softwareInstanceError']) # We don't care of actual formatting, we just want to have full log self.assertTrue(line1 in instance.error_log) self.assertTrue(line2 in instance.error_log) self.assertTrue("Failed to run buildout" in instance.error_log) class TestSlapgridUsageReport(MasterMixin, unittest.TestCase): """ Test suite about slapgrid-ur """ def test_slapgrid_destroys_instance_to_be_destroyed(self): """ Test than an instance in "destroyed" state is correctly destroyed """ computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'started' instance.software.setBuildout(WRAPPER_CONTENT) self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), ['.0_wrapper.log', 'buildout.cfg', 'etc', 'software_release', 'worked']) tries = 50 wrapper_log = os.path.join(instance.partition_path, '.0_wrapper.log') while tries > 0: tries -= 1 if os.path.getsize(wrapper_log) > 0: break time.sleep(0.1) self.assertTrue('Working' in open(wrapper_log, 'r').read()) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'startedComputerPartition']) self.assertEqual(instance.state,'started') # Then destroy the instance computer.sequence = [] instance.requested_state = 'destroyed' self.assertEqual(self.grid.agregateAndSendUsage(), slapgrid.SLAPGRID_SUCCESS) # Assert partition directory is empty self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), []) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) # Assert supervisor stopped process tries = 50 wrapper_log = os.path.join(instance.partition_path, '.0_wrapper.log') exists = False while tries > 0: tries -= 1 if os.path.exists(wrapper_log): exists = True break time.sleep(0.1) self.assertFalse(exists) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'stoppedComputerPartition', 'destroyedComputerPartition']) self.assertEqual(instance.state,'destroyed') def test_slapgrid_destroys_instance_to_be_destroyed_without_sr_uri(self): """ Test than an instance in "destroyed" state but without SR informations is correctly destroyed """ computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.software.name = None computer.sequence = [] instance.requested_state = 'destroyed' self.assertEqual(self.grid.agregateAndSendUsage(), slapgrid.SLAPGRID_SUCCESS) # Assert partition directory is empty self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), []) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) # Assert supervisor stopped process tries = 50 wrapper_log = os.path.join(instance.partition_path, '.0_wrapper.log') exists = False while tries > 0: tries -= 1 if os.path.exists(wrapper_log): exists = True break time.sleep(0.1) self.assertFalse(exists) self.assertEqual(computer.sequence, ['getFullComputerInformation']) def test_slapgrid_ignore_destroyed_instance_without_sr(self): """ Test than an instance in "destroyed" state but without SR at all is correctly ignored """ computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.software = None computer.sequence = [] instance.requested_state = 'destroyed' self.assertEqual(self.grid.agregateAndSendUsage(), slapgrid.SLAPGRID_SUCCESS) # Assert partition directory is empty self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), []) # Assert supervisor stopped process tries = 50 wrapper_log = os.path.join(instance.partition_path, '.0_wrapper.log') exists = False while tries > 0: tries -= 1 if os.path.exists(wrapper_log): exists = True break time.sleep(0.1) self.assertFalse(exists) self.assertEqual(computer.sequence, ['getFullComputerInformation']) def test_slapgrid_not_destroy_bad_instance(self): """ Checks that slapgrid-ur don't destroy instance not to be destroyed. """ computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'started' instance.software.setBuildout(WRAPPER_CONTENT) self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), ['.0_wrapper.log', 'buildout.cfg', 'etc', 'software_release', 'worked']) tries = 50 wrapper_log = os.path.join(instance.partition_path, '.0_wrapper.log') while tries > 0: tries -= 1 if os.path.getsize(wrapper_log) > 0: break time.sleep(0.1) self.assertTrue('Working' in open(wrapper_log, 'r').read()) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) self.assertEqual(computer.sequence, ['getFullComputerInformation', 'availableComputerPartition', 'startedComputerPartition']) self.assertEqual('started',instance.state) # Then run usage report and see if it is still working computer.sequence = [] self.assertEqual(self.grid.agregateAndSendUsage(), slapgrid.SLAPGRID_SUCCESS) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), ['.0_wrapper.log', 'buildout.cfg', 'etc', 'software_release', 'worked']) tries = 50 wrapper_log = os.path.join(instance.partition_path, '.0_wrapper.log') while tries > 0: tries -= 1 if os.path.getsize(wrapper_log) > 0: break time.sleep(0.1) self.assertTrue('Working' in open(wrapper_log, 'r').read()) self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), ['.0_wrapper.log', 'buildout.cfg', 'etc', 'software_release', 'worked']) tries = 50 wrapper_log = os.path.join(instance.partition_path, '.0_wrapper.log') while tries > 0: tries -= 1 if os.path.getsize(wrapper_log) > 0: break time.sleep(0.1) self.assertEqual(computer.sequence, ['getFullComputerInformation']) self.assertEqual('started',instance.state) def test_slapgrid_ignore_free_instance(self): """ Test than a free instance (so in "destroyed" state, but empty) is ignored. """ computer = ComputerForTest(self.software_root, self.instance_root) instance = computer.instance_list[0] computer.sequence = [] instance.requested_state = 'destroyed' self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) # Assert partition directory is empty self.assertSortedListEqual(os.listdir(self.instance_root), ['0', 'etc', 'var']) self.assertSortedListEqual(os.listdir(instance.partition_path), []) self.assertSortedListEqual(os.listdir(self.software_root), [instance.software.software_hash]) self.assertEqual(computer.sequence, ['getFullComputerInformation']) class TestSlapgridSoftwareRelease(MasterMixin, unittest.TestCase): def test_one_software_buildout_fail_is_correctly_logged(self): """ 1. We set up a software using a corrupted buildout 2. It will fail, make sure that whole log is sent to master """ computer = ComputerForTest(self.software_root, self.instance_root, 1, 1) software = computer.software_list[0] line1 = "Nerdy kitten: Can I has a process crash?" line2 = "Cedric: Sure, here it is." software.setBuildout("""#!/bin/sh echo %s; echo %s; exit 42""" % (line1, line2)) self.launchSlapgridSoftware() self.assertEqual(software.sequence, ['buildingSoftwareRelease', 'softwareReleaseError']) # We don't care of actual formatting, we just want to have full log self.assertTrue(line1 in software.error_log) self.assertTrue(line2 in software.error_log) self.assertTrue("Failed to run buildout" in software.error_log) class SlapgridInitialization(unittest.TestCase): """ "Abstract" class setting setup and teardown for TestSlapgridArgumentTuple and TestSlapgridConfigurationFile. """ def setUp(self): """ Create the minimun default argument and configuration. """ self.certificate_repository_path = tempfile.mkdtemp() self.fake_file_descriptor = tempfile.NamedTemporaryFile() self.slapos_config_descriptor = tempfile.NamedTemporaryFile() self.slapos_config_descriptor.write(""" [slapos] software_root = /opt/slapgrid instance_root = /srv/slapgrid master_url = https://slap.vifib.com/ computer_id = your computer id buildout = /path/to/buildout/binary """ % dict(fake_file=self.fake_file_descriptor.name)) self.slapos_config_descriptor.seek(0) self.default_arg_tuple = ( '--cert_file', self.fake_file_descriptor.name, '--key_file', self.fake_file_descriptor.name, '--master_ca_file', self.fake_file_descriptor.name, '--certificate_repository_path', self.certificate_repository_path, '-c', self.slapos_config_descriptor.name, '--now') self.signature_key_file_descriptor = tempfile.NamedTemporaryFile() self.signature_key_file_descriptor.seek(0) def tearDown(self): """ Removing the temp file. """ self.fake_file_descriptor.close() self.slapos_config_descriptor.close() self.signature_key_file_descriptor.close() shutil.rmtree(self.certificate_repository_path, True) class TestSlapgridArgumentTuple(SlapgridInitialization): """ Test suite about arguments given to slapgrid command. """ def test_empty_argument_tuple(self): """ Raises if the argument list if empty and without configuration file. """ parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject # XXX: SystemExit is too generic exception, it is only known that # something is wrong self.assertRaises(SystemExit, parser, *()) def test_default_argument_tuple(self): """ Check if we can have the slapgrid object returned with the minimum arguments. """ parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject return_list = parser(*self.default_arg_tuple) self.assertEquals(2, len(return_list)) def test_signature_private_key_file_non_exists(self): """ Raises if the signature_private_key_file does not exists. """ parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject argument_tuple = ("--signature_private_key_file", "/non/exists/path") + \ self.default_arg_tuple # XXX: SystemExit is too generic exception, it is only known that # something is wrong self.assertRaises(SystemExit, parser, *argument_tuple) def test_signature_private_key_file(self): """ Check if the signature private key argument value is available on slapgrid object. """ parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject argument_tuple = ("--signature_private_key_file", self.signature_key_file_descriptor.name) + \ self.default_arg_tuple slapgrid_object = parser(*argument_tuple)[0] self.assertEquals(self.signature_key_file_descriptor.name, slapgrid_object.signature_private_key_file) def test_backward_compatibility_all(self): """ Check if giving --all triggers "develop" option. """ parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject argument_tuple = ("--all",) + self.default_arg_tuple slapgrid_object = parser(*argument_tuple)[0] self.assertTrue(slapgrid_object.develop) def test_backward_compatibility_not_all(self): """ Check if not giving --all neither --develop triggers "develop" option to be False. """ parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject argument_tuple = self.default_arg_tuple slapgrid_object = parser(*argument_tuple)[0] self.assertFalse(slapgrid_object.develop) def test_force_periodicity_if_periodicity_not_given(self): """ Check if not giving --maximum-periodicity triggers "force_periodicity" option to be false. """ parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject argument_tuple = self.default_arg_tuple slapgrid_object = parser(*argument_tuple)[0] self.assertFalse(slapgrid_object.force_periodicity) def test_force_periodicity_if_periodicity_given(self): """ Check if giving --maximum-periodicity triggers "force_periodicity" option. """ parser = slapgrid.parseArgumentTupleAndReturnSlapgridObject argument_tuple = ("--maximum-periodicity","40") + self.default_arg_tuple slapgrid_object = parser(*argument_tuple)[0] self.assertTrue(slapgrid_object.force_periodicity) class TestSlapgridConfigurationFile(SlapgridInitialization): def test_upload_binary_cache_blacklist(self): """ Check if giving --upload-to-binary-cache-url-blacklist triggers option. """ self.slapos_config_descriptor.write(""" [slapos] software_root = /opt/slapgrid instance_root = /srv/slapgrid master_url = https://slap.vifib.com/ computer_id = your computer id buildout = /path/to/buildout/binary [networkcache] upload-to-binary-cache-url-blacklist = http://1 http://2/bla """ % dict(fake_file=self.fake_file_descriptor.name)) self.slapos_config_descriptor.seek(0) slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject( *self.default_arg_tuple)[0] self.assertEqual( slapgrid_object.upload_to_binary_cache_url_blacklist, ['http://1', 'http://2/bla'] ) self.assertEqual( slapgrid_object.download_from_binary_cache_url_blacklist, [] ) def test_download_binary_cache_blacklist(self): """ Check if giving --download-from-binary-cache-url-blacklist triggers option. """ self.slapos_config_descriptor.write(""" [slapos] software_root = /opt/slapgrid instance_root = /srv/slapgrid master_url = https://slap.vifib.com/ computer_id = your computer id buildout = /path/to/buildout/binary [networkcache] download-from-binary-cache-url-blacklist = http://1 http://2/bla """ % dict(fake_file=self.fake_file_descriptor.name)) self.slapos_config_descriptor.seek(0) slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject( *self.default_arg_tuple)[0] self.assertEqual( slapgrid_object.upload_to_binary_cache_url_blacklist, [] ) self.assertEqual( slapgrid_object.download_from_binary_cache_url_blacklist, ['http://1', 'http://2/bla'] ) def test_upload_download_binary_cache_blacklist(self): """ Check if giving both --download-from-binary-cache-url-blacklist and --upload-to-binary-cache-url-blacklist triggers options. """ self.slapos_config_descriptor.write(""" [slapos] software_root = /opt/slapgrid instance_root = /srv/slapgrid master_url = https://slap.vifib.com/ computer_id = your computer id buildout = /path/to/buildout/binary [networkcache] upload-to-binary-cache-url-blacklist = http://1 http://2/bla download-from-binary-cache-url-blacklist = http://3 http://4/bla """ % dict(fake_file=self.fake_file_descriptor.name)) self.slapos_config_descriptor.seek(0) slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject( *self.default_arg_tuple)[0] self.assertEqual( slapgrid_object.upload_to_binary_cache_url_blacklist, ['http://1', 'http://2/bla'] ) self.assertEqual( slapgrid_object.download_from_binary_cache_url_blacklist, ['http://3', 'http://4/bla'] ) def test_backward_compatibility_download_binary_cache_blacklist(self): """ Check if giving both --binary-cache-url-blacklist and --upload-to-binary-cache-blacklist triggers options. """ self.slapos_config_descriptor.write(""" [slapos] software_root = /opt/slapgrid instance_root = /srv/slapgrid master_url = https://slap.vifib.com/ computer_id = your computer id buildout = /path/to/buildout/binary [networkcache] binary-cache-url-blacklist = http://1 http://2/bla """ % dict(fake_file=self.fake_file_descriptor.name)) self.slapos_config_descriptor.seek(0) slapgrid_object = slapgrid.parseArgumentTupleAndReturnSlapgridObject( *self.default_arg_tuple)[0] self.assertEqual( slapgrid_object.upload_to_binary_cache_url_blacklist, [] ) self.assertEqual( slapgrid_object.download_from_binary_cache_url_blacklist, ['http://1', 'http://2/bla'] ) class TestSlapgridCPWithMasterPromise(MasterMixin, unittest.TestCase): def test_one_failing_promise(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'started' worked_file = os.path.join(instance.partition_path, 'fail_worked') fail = textwrap.dedent("""\ #!/usr/bin/env sh touch "%s" exit 127""" % worked_file) instance.setPromise('fail', fail) self.assertEqual(self.grid.processComputerPartitionList(), slapos.grid.slapgrid.SLAPGRID_PROMISE_FAIL) self.assertTrue(os.path.isfile(worked_file)) self.assertTrue(instance.error) self.assertNotEqual('started', instance.state) def test_one_succeeding_promise(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'started' self.fake_waiting_time = 0.1 worked_file = os.path.join(instance.partition_path, 'succeed_worked') succeed = ("""#!/usr/bin/env sh touch "%(worked_file)s" exit 0""" % {'worked_file': worked_file}) instance.setPromise('succeed', succeed) self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) self.assertTrue(os.path.isfile(worked_file)) self.assertFalse(instance.error) self.assertEqual(instance.state, 'started') def test_stderr_has_been_sent(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] httplib.HTTPConnection._callback = computer.getServerResponse() instance.requested_state = 'started' self.fake_waiting_time = 0.5 promise_path = os.path.join(instance.partition_path, 'etc', 'promise') os.makedirs(promise_path) succeed = os.path.join(promise_path, 'stderr_writer') worked_file = os.path.join(instance.partition_path, 'stderr_worked') with open(succeed, 'w') as f: f.write("""#!/usr/bin/env sh touch "%(worked_file)s" echo Error 1>&2 exit 127""" % {'worked_file': worked_file}) os.chmod(succeed, 0777) self.assertEqual(self.grid.processComputerPartitionList(), slapos.grid.slapgrid.SLAPGRID_PROMISE_FAIL) self.assertTrue(os.path.isfile(worked_file)) self.assertEqual(instance.error_log, 'Error') self.assertTrue(instance.error) self.assertIsNone(instance.state) def test_timeout_works(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'started' self.fake_waiting_time = 0.1 promise_path = os.path.join(instance.partition_path, 'etc', 'promise') os.makedirs(promise_path) succeed = os.path.join(promise_path, 'timed_out_promise') worked_file = os.path.join(instance.partition_path, 'timed_out_worked') with open(succeed, 'w') as f: f.write("""#!/usr/bin/env sh touch "%(worked_file)s" sleep 5 exit 0""" % {'worked_file': worked_file}) os.chmod(succeed, 0777) self.assertEqual(self.grid.processComputerPartitionList(), slapos.grid.slapgrid.SLAPGRID_PROMISE_FAIL) self.assertTrue(os.path.isfile(worked_file)) self.assertTrue(instance.error) self.assertIsNone(instance.state) def test_two_succeeding_promises(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'started' self.fake_waiting_time = 0.1 for i in range (0,2): worked_file = os.path.join(instance.partition_path, 'succeed_%s_worked' % i) succeed = ("""#!/usr/bin/env sh touch "%(worked_file)s" exit 0""" % {'worked_file': worked_file}) instance.setPromise('succeed_%s' % i, succeed) self.assertEqual(self.grid.processComputerPartitionList(), slapgrid.SLAPGRID_SUCCESS) for i in range(0,2): worked_file = os.path.join(instance.partition_path, 'succeed_%s_worked' % i) self.assertTrue(os.path.isfile(worked_file)) self.assertFalse(instance.error) self.assertEqual(instance.state, 'started') def test_one_succeeding_one_failing_promises(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'started' self.fake_waiting_time = 0.1 for i in range(2): worked_file = os.path.join(instance.partition_path, 'promise_worked_%d' % i) lockfile = os.path.join(instance.partition_path, 'lock') promise=("""#!/usr/bin/env sh touch "%(worked_file)s" if [ ! -f %(lockfile)s ] then touch "%(lockfile)s" exit 0 else exit 127 fi""" % {'worked_file': worked_file, 'lockfile': lockfile}) instance.setPromise('promise_%s'%i,promise) self.assertEqual(self.grid.processComputerPartitionList(), slapos.grid.slapgrid.SLAPGRID_PROMISE_FAIL) self.assertEquals(instance.error, 1) self.assertNotEqual('started',instance.state) def test_one_succeeding_one_timing_out_promises(self): computer = ComputerForTest(self.software_root,self.instance_root) instance = computer.instance_list[0] instance.requested_state = 'started' self.fake_waiting_time = 0.1 for i in range(2): worked_file = os.path.join(instance.partition_path, 'promise_worked_%d' % i) lockfile = os.path.join(instance.partition_path, 'lock') promise = ("""#!/usr/bin/env sh touch "%(worked_file)s" if [ ! -f %(lockfile)s ] then touch "%(lockfile)s" else sleep 5 fi exit 0""" % {'worked_file': worked_file, 'lockfile': lockfile}) instance.setPromise('promise_%d' % i, promise) self.assertEqual(self.grid.processComputerPartitionList(), slapos.grid.slapgrid.SLAPGRID_PROMISE_FAIL) self.assertEquals(instance.error, 1) self.assertNotEqual(instance.state,'started') slapos.core-0.35.1/slapos/tests/slapobject.py0000644000076600007660000002101212061607367027265 0ustar cedricdesaintmartincedricdesaintmartin00000000000000############################################################################## # # Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly adviced to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## from slapos.grid import SlapObject from slapos.grid import utils from slapos.grid import networkcache from slapos.tests.slapgrid import BasicMixin import os import unittest class FakeCallAndRead: def __init__(self): self.external_command_list = [] def __call__(self, *args, **kwargs): additional_buildout_parametr_list = \ kwargs.get('additional_buildout_parametr_list') self.external_command_list.extend(additional_buildout_parametr_list) FakeCallAndRead = FakeCallAndRead() # Backup modules original_install_from_buildout = SlapObject.Software._install_from_buildout original_upload_network_cached = networkcache.upload_network_cached originalBootstrapBuildout = utils.bootstrapBuildout originalLaunchBuildout = utils.launchBuildout originalUploadSoftwareRelease = SlapObject.Software.uploadSoftwareRelease class TestSoftwareSlapObject(BasicMixin, unittest.TestCase): """ Test for Software class. """ def setUp(self): BasicMixin.setUp(self) os.mkdir(self.software_root) self.signature_private_key_file = '/signature/private/key_file' self.upload_cache_url = 'http://example.com/uploadcache' self.upload_dir_url = 'http://example.com/uploaddir' self.shacache_cert_file = '/path/to/shacache/cert/file' self.shacache_key_file = '/path/to/shacache/key/file' self.shadir_cert_file = '/path/to/shadir/cert/file' self.shadir_key_file = '/path/to/shadir/key/file' # Monkey patch utils module utils.bootstrapBuildout = FakeCallAndRead utils.launchBuildout = FakeCallAndRead def tearDown(self): global originalBootstrapBuildout global originalLaunchBuildout BasicMixin.tearDown(self) FakeCallAndRead.external_command_list = [] # Un-monkey patch utils module utils.bootstrapBuildout = originalBootstrapBuildout utils.launchBuildout = originalLaunchBuildout SlapObject.Software._install_from_buildout = original_install_from_buildout networkcache.upload_network_cached = original_upload_network_cached SlapObject.Software.uploadSoftwareRelease = originalUploadSoftwareRelease # Test methods def test_software_install_with_networkcache(self): """ Check if the networkcache parameters are propagated. """ software = SlapObject.Software( url='http://example.com/software.cfg', software_root=self.software_root, buildout=self.buildout, signature_private_key_file='/signature/private/key_file', upload_cache_url='http://example.com/uploadcache', upload_dir_url='http://example.com/uploaddir', shacache_cert_file=self.shacache_cert_file, shacache_key_file=self.shacache_key_file, shadir_cert_file=self.shadir_cert_file, shadir_key_file=self.shadir_key_file) software.install() command_list = FakeCallAndRead.external_command_list self.assertTrue('buildout:networkcache-section=networkcache' in command_list) self.assertTrue('networkcache:signature-private-key-file=%s' % self.signature_private_key_file in command_list) self.assertTrue('networkcache:upload-cache-url=%s' % self.upload_cache_url in command_list) self.assertTrue('networkcache:upload-dir-url=%s' % self.upload_dir_url in command_list) self.assertTrue('networkcache:shacache-cert-file=%s' % self.shacache_cert_file in command_list) self.assertTrue('networkcache:shacache-key-file=%s' % self.shacache_key_file in command_list) self.assertTrue('networkcache:shadir-cert-file=%s' % self.shadir_cert_file in command_list) self.assertTrue('networkcache:shadir-key-file=%s' % self.shadir_key_file in command_list) def test_software_install_without_networkcache(self): """ Check if the networkcache parameters are not propagated if they are not available. """ software = SlapObject.Software( url='http://example.com/software.cfg', software_root=self.software_root, buildout=self.buildout) software.install() command_list = FakeCallAndRead.external_command_list self.assertFalse('buildout:networkcache-section=networkcache' in command_list) self.assertFalse('networkcache:signature-private-key-file=%s' % self.signature_private_key_file in command_list) self.assertFalse('networkcache:upload-cache-url=%s' % self.upload_cache_url in command_list) self.assertFalse('networkcache:upload-dir-url=%s' % self.upload_dir_url in command_list) # XXX-Cedric: do the same with upload def test_software_install_networkcache_upload_blacklist(self): """ Check if the networkcache upload blacklist parameters are propagated. """ def fakeBuildout(*args, **kw): pass SlapObject.Software._install_from_buildout = fakeBuildout def fake_upload_network_cached(*args, **kw): self.assertFalse(True) networkcache.upload_network_cached = fake_upload_network_cached upload_to_binary_cache_url_blacklist = ["http://example.com"] software = SlapObject.Software( url='http://example.com/software.cfg', software_root=self.software_root, buildout=self.buildout, signature_private_key_file='/signature/private/key_file', upload_cache_url='http://example.com/uploadcache', upload_dir_url='http://example.com/uploaddir', shacache_cert_file=self.shacache_cert_file, shacache_key_file=self.shacache_key_file, shadir_cert_file=self.shadir_cert_file, shadir_key_file=self.shadir_key_file, upload_to_binary_cache_url_blacklist=\ upload_to_binary_cache_url_blacklist, ) software.install() def test_software_install_networkcache_upload_blacklist_side_effect(self): """ Check if the networkcache upload blacklist parameters only prevent blacklisted Software Release to be uploaded. """ def fakeBuildout(*args, **kw): pass SlapObject.Software._install_from_buildout = fakeBuildout def fakeUploadSoftwareRelease(*args, **kw): self.uploaded = True SlapObject.Software.uploadSoftwareRelease = fakeUploadSoftwareRelease upload_to_binary_cache_url_blacklist = ["http://anotherexample.com"] software = SlapObject.Software( url='http://example.com/software.cfg', software_root=self.software_root, buildout=self.buildout, signature_private_key_file='/signature/private/key_file', upload_cache_url='http://example.com/uploadcache', upload_dir_url='http://example.com/uploaddir', upload_binary_cache_url='http://example.com/uploadcache', upload_binary_dir_url='http://example.com/uploaddir', shacache_cert_file=self.shacache_cert_file, shacache_key_file=self.shacache_key_file, shadir_cert_file=self.shadir_cert_file, shadir_key_file=self.shadir_key_file, upload_to_binary_cache_url_blacklist=\ upload_to_binary_cache_url_blacklist, ) software.install() self.assertTrue(getattr(self, 'uploaded', False)) slapos.core-0.35.1/slapos/tests/slapproxy.py0000644000076600007660000004252612073300773027207 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2012 Vifib SARL and Contributors. # All Rights Reserved. # # WARNING: This program as such is intended to be used by professional # programmers who take the whole responsibility of assessing all potential # consequences resulting from its eventual inadequacies and bugs # End users who are looking for a ready-to-use solution with commercial # guarantees and support are strongly advised to contract a Free Software # Service Company # # This program is Free Software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 3 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # ############################################################################## import os import logging import shutil import slapos.proxy import slapos.proxy.views as views import slapos.slap.slap import tempfile import unittest import xml_marshaller class WrongFormat(Exception): pass class ProxyOption: """ Will simulate options given to slapproxy """ def __init__(self, proxy_db): self.verbose=True self.database_uri=proxy_db self.console = False self.log_file = None class BasicMixin: def setUp(self): """ Will set files and start slapproxy """ self._tempdir = tempfile.mkdtemp() logging.basicConfig(level=logging.DEBUG) self.setFiles() self.startProxy() def setFiles(self): """ Set environment to run slapproxy """ self.slapos_cfg = os.path.join(self._tempdir, 'slapos.cfg') self.proxy_db = os.path.join(self._tempdir, 'lib', 'proxy.db') self.proxyaddr = 'http://127.0.0.1:8080/' self.computer_id = 'computer' open(self.slapos_cfg, 'w').write("""[slapos] software_root = %(tempdir)s/opt/slapgrid instance_root = %(tempdir)s/srv/slapgrid master_url = %(proxyaddr)s computer_id = computer [slapproxy] host = 127.0.0.1 port = 8080 database_uri = %(tempdir)s/lib/proxy.db """% dict (tempdir = self._tempdir, proxyaddr = self.proxyaddr)) for directory in ("opt", "srv", "lib"): path = os.path.join(self._tempdir, directory) os.mkdir(path) def startProxy(self): """ Set config for slapproxy and start it """ config = slapos.proxy.Config() config.setConfig(*(ProxyOption(self.proxy_db), self.slapos_cfg)) views.app.config['TESTING'] = True views.app.config['computer_id'] = self.computer_id views.app.config['DATABASE_URI'] = self.proxy_db views.app.config['HOST'] = config.host views.app.config['port'] = config.port self.app = views.app.test_client() def add_free_partition (self, partition_amount): """ Will simulate a slapformat first run and create "partition_amount" partitions """ computer_dict = {'reference':self.computer_id, 'address':'123.456.789', 'netmask':'fffffffff', 'partition_list':[]} for i in range(0, partition_amount): partition_example = {'reference':'slappart%s' %i, 'address_list':[]} computer_dict['partition_list'].append(partition_example) request_dict = { 'computer_id': self.computer_id, 'xml': xml_marshaller.xml_marshaller.dumps(computer_dict), } self.app.post('/loadComputerConfigurationFromXML', data = request_dict) def tearDown(self): """ Remove files generated for test """ shutil.rmtree(self._tempdir, True) class TestInformation(BasicMixin, unittest.TestCase): """ Test Basic response of slapproxy """ def test_getComputerInformation (self): """ Check that getComputerInformation return a Computer and database is generated """ rv = self.app.get('/getComputerInformation?computer_id=' +self.computer_id) self.assertIsInstance( xml_marshaller.xml_marshaller.loads(rv.data), slapos.slap.Computer) self.assertTrue(os.path.exists(self.proxy_db)) def test_getFullComputerInformation (self): """ Check that getFullComputerInformation return a Computer and database is generated """ rv = self.app.get('/getFullComputerInformation?computer_id=' + self.computer_id) self.assertIsInstance( xml_marshaller.xml_marshaller.loads(rv.data), slapos.slap.Computer) self.assertTrue(os.path.exists(self.proxy_db)) def test_getComputerInformation_wrong_computer (self): """ Test that computer information won't be given to a requester different from the one specified """ with self.assertRaises(slapos.slap.NotFoundError): self.app.get('/getComputerInformation?computer_id=' + self.computer_id + '42') def test_partition_are_empty (self): """ Test that empty partition are empty :) """ self.add_free_partition(10) rv = self.app.get('/getFullComputerInformation?computer_id=' + self.computer_id) computer = xml_marshaller.xml_marshaller.loads(rv.data) for slap_partition in computer._computer_partition_list: self.assertIsNone(slap_partition._software_release_document) self.assertEqual(slap_partition._requested_state, 'destroyed') self.assertEqual(slap_partition._need_modification, 0) class MasterMixin(BasicMixin): """ Define advanced tool for test proxy simulating behavior slap library tools """ def request(self, software_release, software_type, partition_reference, partition_id, shared=False, partition_parameter_kw=None, filter_kw=None, state=None): """ Simulate a request with above parameters Return response by server (a computer partition or an error) """ if partition_parameter_kw is None: partition_parameter_kw = {} if filter_kw is None: filter_kw = {} # Let enforce a default software type if software_type is None: software_type = 'default' request_dict = { 'computer_id': self.computer_id, 'computer_partition_id': partition_id, 'software_release': software_release, 'software_type': software_type, 'partition_reference': partition_reference, 'shared_xml': xml_marshaller.xml_marshaller.dumps(shared), 'partition_parameter_xml': xml_marshaller.xml_marshaller.dumps( partition_parameter_kw), 'filter_xml': xml_marshaller.xml_marshaller.dumps(filter_kw), 'state': xml_marshaller.xml_marshaller.dumps(state), } rv = self.app.post('/requestComputerPartition', data = request_dict) xml = rv.data try: software_instance = xml_marshaller.xml_marshaller.loads(xml) except: raise WrongFormat("Could not be parsed by xml_marshaller") computer_partition = slapos.slap.ComputerPartition( software_instance.slap_computer_id, software_instance.slap_computer_partition_id, ) if shared: computer_partition._synced = True computer_partition._connection_dict = getattr(software_instance, '_connection_dict', None) computer_partition._parameter_dict = getattr(software_instance, '_parameter_dict', None) return computer_partition def setConnectionDict(self, partition_id, connection_dict, slave_reference=None): self.app.post('/setComputerPartitionConnectionXml', data = { 'computer_id': self.computer_id, 'computer_partition_id': partition_id, 'connection_xml': xml_marshaller.xml_marshaller.dumps(connection_dict), 'slave_reference': slave_reference}) def getPartitionInformation(self,computer_partition_id): """ Return computer information as stored in proxy for corresponding id """ rv = self.app.get('/getFullComputerInformation?computer_id=' + self.computer_id) computer = xml_marshaller.xml_marshaller.loads(rv.data) for instance in computer._computer_partition_list: if instance._partition_id == computer_partition_id: return instance class TestRequest (MasterMixin, unittest.TestCase): """ Set of tests for requests """ def test_two_request_one_partition_free (self): """ Since slapproxy does not implement scope, providing two partition_id values will still succeed, even if only one partition is available. """ self.add_free_partition(1) self.assertIsInstance(self.request('http://sr//', None, 'Maria', 'slappart2'), slapos.slap.ComputerPartition) self.assertIsInstance(self.request('http://sr//', None, 'Maria', 'slappart3'), slapos.slap.ComputerPartition) def test_two_request_two_partition_free (self): """ If two requests are made with two available partition both will succeed """ self.add_free_partition(2) self.assertIsInstance(self.request('http://sr//', None, 'Maria', 'slappart2'), slapos.slap.ComputerPartition) self.assertIsInstance(self.request('http://sr//', None, 'Maria', 'slappart3'), slapos.slap.ComputerPartition) def test_two_same_request_from_one_partition (self): """ Request will return same partition for two equal requests """ self.add_free_partition(2) self.assertEqual( self.request('http://sr//', None, 'Maria', 'slappart2').__dict__, self.request('http://sr//', None, 'Maria', 'slappart2').__dict__) def test_two_requests_with_different_parameters_but_same_reference (self): """ Request will return same partition for two different requests but will only update parameters """ self.add_free_partition(2) wanted_domain1 = 'fou.org' wanted_domain2 = 'carzy.org' request1 = self.request('http://sr//', None, 'Maria', 'slappart2', partition_parameter_kw = {'domain':wanted_domain1}) request1_dict = request1.__dict__ requested_result1 = self.getPartitionInformation( request1_dict['_partition_id']) request2 = self.request('http://sr1//', 'Papa', 'Maria', 'slappart2', partition_parameter_kw = {'domain':wanted_domain2}) request2_dict = request2.__dict__ requested_result2 = self.getPartitionInformation( request2_dict['_partition_id']) # Test we received same partition for key in request1_dict: self.assertEqual(request1_dict[key],request2_dict[key]) # Test that only parameters changed for key in requested_result2.__dict__ : if not key in ('_parameter_dict', '_software_release_document'): self.assertEqual(requested_result2.__dict__[key], requested_result1.__dict__[key]) elif key in ('_software_release_document'): self.assertEqual(requested_result2.__dict__[key].__dict__, requested_result1.__dict__[key].__dict__) #Test parameters where set correctly self.assertEqual(wanted_domain1, requested_result1._parameter_dict['domain']) self.assertEqual(wanted_domain2, requested_result2._parameter_dict['domain']) def test_two_different_request_from_two_partition (self): """ Since slapproxy does not implement scope, two request with different partition_id will still return the same partition. """ self.add_free_partition(2) self.assertEqual( self.request('http://sr//', None, 'Maria', 'slappart2').__dict__, self.request('http://sr//', None, 'Maria', 'slappart3').__dict__) def test_two_different_request_from_one_partition (self): """ Two different request from same partition will return two different partitions """ self.add_free_partition(2) self.assertNotEqual( self.request('http://sr//', None, 'Maria', 'slappart2').__dict__, self.request('http://sr//', None, 'frontend', 'slappart2').__dict__) def test_slave_request_no_corresponding_partition (self): """ Slave instance request will fail if no corresponding are found """ self.add_free_partition(2) with self.assertRaises(WrongFormat): self.request('http://sr//', None, 'Maria', 'slappart2', shared=True) def test_slave_request_set_parameters (self): """ Parameters sent in slave request must be put in slave master slave instance list. 1. We request a slave instance we defined parameters 2. We check parameters are in the dictionnary defining slave in slave master slave_instance_list """ self.add_free_partition(6) # Provide partition master_partition_id = self.request('http://sr//', None, 'Maria', 'slappart4')._partition_id # First request of slave instance wanted_domain = 'fou.org' self.request('http://sr//', None, 'Maria', 'slappart2', shared=True, partition_parameter_kw = {'domain':wanted_domain}) # Get updated information for master partition master_partition = self.getPartitionInformation(master_partition_id) our_slave = master_partition._parameter_dict['slave_instance_list'][0] self.assertEqual(our_slave.get('domain'),wanted_domain) def test_slave_request_set_parameters_are_updated (self): """ Parameters sent in slave request must be put in slave master slave instance list and updated when they change. 1. We request a slave instance we defined parameters 2. We check parameters are in the dictionnary defining slave in slave master slave_instance_list 3. We request same slave instance with changed parameters 4. We check parameters are in the dictionnary defining slave in slave master slave_instance_list have changed """ self.add_free_partition(6) # Provide partition master_partition_id = self.request('http://sr//', None, 'Maria', 'slappart4')._partition_id # First request of slave instance wanted_domain_1 = 'crazy.org' self.request('http://sr//', None, 'Maria', 'slappart2', shared=True, partition_parameter_kw = {'domain':wanted_domain_1}) # Get updated information for master partition master_partition = self.getPartitionInformation(master_partition_id) our_slave = master_partition._parameter_dict['slave_instance_list'][0] self.assertEqual(our_slave.get('domain'),wanted_domain_1) # Second request of slave instance wanted_domain_2 = 'maluco.org' self.request('http://sr//', None, 'Maria', 'slappart2', shared=True, partition_parameter_kw = {'domain':wanted_domain_2}) # Get updated information for master partition master_partition = self.getPartitionInformation(master_partition_id) our_slave = master_partition._parameter_dict['slave_instance_list'][0] self.assertNotEqual(our_slave.get('domain'),wanted_domain_1) self.assertEqual(our_slave.get('domain'),wanted_domain_2) def test_slave_request_one_corresponding_partition (self): """ Successfull request slave instance follow these steps: 1. Provide one corresponding partition 2. Ask for Slave instance. But no connection parameters But slave is added to Master Instance slave list 3. Master Instance get updated information (including slave list) 4. Master instance post information about slave connection parameters 5. Ask for slave instance is successfull and return a computer instance with connection information """ self.add_free_partition(6) # Provide partition master_partition_id = self.request('http://sr//', None, 'Maria', 'slappart4')._partition_id # First request of slave instance name = 'Maria' requester = 'slappart2' our_slave = self.request('http://sr//', None, name, requester, shared=True) self.assertIsInstance(our_slave, slapos.slap.ComputerPartition) self.assertEqual(our_slave._connection_dict,{}) # Get updated information for master partition master_partition = self.getPartitionInformation(master_partition_id) slave_for_master = master_partition._parameter_dict['slave_instance_list'][0] # Send information about slave slave_address = {'url':'%s.master.com'} self.setConnectionDict(partition_id=master_partition._partition_id, connection_dict=slave_address, slave_reference=slave_for_master['slave_reference']) # Successfull slave request with connection parameters our_slave = self.request('http://sr//', None, name, requester, shared=True) self.assertIsInstance(our_slave, slapos.slap.ComputerPartition) self.assertEqual(slave_address, our_slave._connection_dict) slapos.core-0.35.1/slapos/util.py0000644000076600007660000000062112105142434024741 0ustar cedricdesaintmartincedricdesaintmartin00000000000000# -*- coding: utf-8 -*- import os, errno def mkdir_p(path, mode=0o777): """\ Creates a directory and its parents, if needed. NB: If the directory already exists, it does not change its permission. """ try: os.makedirs(path, mode) except OSError as exc: if exc.errno == errno.EEXIST and os.path.isdir(path): pass else: raise slapos.core-0.35.1/slapos/version.py0000644000076600007660000000002512110425541025446 0ustar cedricdesaintmartincedricdesaintmartin00000000000000 version = '0.35.1' slapos.core-0.35.1/slapos.core.egg-info/0000755000076600007660000000000012110425554026037 5ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos.core-0.35.1/slapos.core.egg-info/dependency_links.txt0000644000076600007660000000000112110425554032105 0ustar cedricdesaintmartincedricdesaintmartin00000000000000 slapos.core-0.35.1/slapos.core.egg-info/entry_points.txt0000644000076600007660000000107612110425554031341 0ustar cedricdesaintmartincedricdesaintmartin00000000000000[console_scripts] slapformat = slapos.format:main slapos = slapos.entry:main slapgrid = slapos.grid.slapgrid:run slapos-watchdog = slapos.grid.watchdog:main bang = slapos.bang:main slapproxy-query = slapos.proxy.query:main slapgrid-sr = slapos.grid.slapgrid:runSoftwareRelease slapgrid-supervisorctl = slapos.grid.svcbackend:supervisorctl slapgrid-supervisord = slapos.grid.svcbackend:supervisord slapconsole = slapos.client:slapconsole slapproxy = slapos.proxy:main slapgrid-cp = slapos.grid.slapgrid:runComputerPartition slapgrid-ur = slapos.grid.slapgrid:runUsageReport slapos.core-0.35.1/slapos.core.egg-info/namespace_packages.txt0000644000076600007660000000000712110425554032367 0ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos slapos.core-0.35.1/slapos.core.egg-info/not-zip-safe0000644000076600007660000000000112030325633030263 0ustar cedricdesaintmartincedricdesaintmartin00000000000000 slapos.core-0.35.1/slapos.core.egg-info/PKG-INFO0000644000076600007660000007304612110425554027146 0ustar cedricdesaintmartincedricdesaintmartin00000000000000Metadata-Version: 1.0 Name: slapos.core Version: 0.35.1 Summary: SlapOS core. Home-page: http://www.slapos.org Author: VIFIB Author-email: UNKNOWN License: GPLv3 Description: slapos.core =========== The core of SlapOS. Contains the SLAP library, and the slapgrid, slapformat, slapproxy tools. For more information, see http://www.slapos.org. Changes ======= 0.35.1 (2012-02-18) ------------------- New features: * Add ComputerPartition._instance_guid getter in SLAP library. [Cedric de Saint Martin] * Add ComputerPartition._instance_guid support in slapproxy. [Cedric de Saint Martin] Fixes: * Fix link existence check when deploying instance if SR is not correctly installed. This fixes a misleading error. [Cedric de Saint Martin] * Improve message shown to user when requesting. [Cedric de Saint Martin] * Raise NotReady when _requested_state doesn't exist when trying to fetch it from getter. [Cedric de Saint Martin] 0.35 (2012-02-08) ----------------- * slapos: display version number with help. [Marco Mariani] * slapformat: backup slapos.xml to a zip archive at every change. [Marco Mariani] * slapformat: Don't check validity of ipv4 when trying to add address that already exists. [Cedric de Saint Martin] * slapgrid: create and run $MD5/buildout.cfg for eaiser debugging. [Marco Mariani] * slapgrid: keep running if cp.error() or sr.error() have issues (fixes 20130119-744D94). [Marco Mariani] * slapgrid does not crash when there are no certificates (fixes #20130121-136C24). [Marco Mariani] * Add slapproxy-query command. [Marco Mariani] * Other minor typo / output fixes. 0.34 (2013-01-23) ----------------- * networkcache: only match major release number in Debian, fixed platform detection for Ubuntu. [Marco Mariani] * symlink to software_release in each partition. [Marco Mariani] * slapos client: Properly expand "~" when giving configuration file location. [Cedric de Saint Martin] * slapgrid: stop instances that should be stopped even if buildout and/or reporting failed. [Cedric de Saint Martin] * slapgrid: Don't periodically force-process a stopped instance. [Cedric de Saint Martin] * slapgrid: Handle pid files of slapgrid launched through different entry points. [Cedric de Saint Martin] * Watchdog: Bang is called with correct instance certificates. [Cedric Le Ninivin] * Watchdog: Fix watchdog call. [Cedric le Ninivin] * Add a symlink of the used software release in each partitions. [Marco Mariani] * slapformat is verbose by default. [Cedric de Saint Martin] * slapproxy: Filter by instance_guid, allow computer partition renames and change of software_type and requested_state. [Marco Mariani] * slapproxy: Stop instance even if buildout/reporting is wrong. [Cedric de Saint Martin] * slapproxy: implement softwareInstanceRename method. [Marco Mariani] * slapproxy: alllow requests to software_type. [Marco Mariani] * Many other minor fixes. See git diff for details. 0.33.1 (2012-11-05) ------------------- * Fix "slapos console" argument parsing. [Cedric de Saint Martin] 0.33 (2012-11-02) ----------------- * Continue to improve new entry points. The following are now functional: - slapos node format - slapos node start/stop/restart/tail - slapos node supervisord/supervisorctl - slapos node supply and add basic usage. [Cedric de Saint Martin] * Add support for "SLAPOS_CONFIGURATION" and SLAPOS_CLIENT_CONFIGURATION environment variables. (commit c72a53b1) [Cédric de Saint Martin] * --only_sr also accepts plain text URIs. [Marco Mariani] 0.32.3 (2012-10-15) ------------------- * slapgrid: Adopt new return value strategy (0=OK, 1=failed, 2=promise failed) (commit 5d4e1522). [Cedric de Saint Martin] * slaplib: add requestComputer (commits 6cbe82e0, aafb86eb). [Łukasz Nowak] * slapgrid: Add stopasgroup and killasgroup to supervisor (commit 36e0ccc0). [Cedric de Saint Martin] * slapproxy: don't start in debug mode by default (commit e32259c8). [Cédric Le Ninivin * SlapObject: ALWAYS remove tmpdir (commit a652a610). [Cedric de Saint Martin] 0.32.2 (2012-10-11) ------------------- * slapgrid: Remove default delay, now that SlapOS Master is Fast as Light (tm). (commit 03a85d6b8) [Cedric de Saint Martin] * Fix watchdog entry point name, introduced in v0.31. (commit a8651ba12) [Cedric de Saint Martin] * slapgrid: Better filter of instances, won't process false positives anymore (hopefully). (commit ce0a73b41) [Cedric de Saint Martin] * Various output improvements. [Cedric de Saint Martin] 0.32.1 (2012-10-09) ------------------- * slapgrid: Make sure error logs are sent to SlapOS master. Finish implementation began in 0.32. [Cedric de Saint Martin] * slapgrid: Fix Usage Report in case of not empty partition with no SR. [Cedric de Saint Martin] 0.32 (2012-10-04) ----------------- * Introduce new, simpler "slapos" entry point. See documentation for more informations. Note: some functionnalities of this new entry point don't work yet or is not as simple as it should be. [Cedric de Saint Martin, Cedric Le Ninivin] * Revamped "slapos request" to work like described in documentation. [Cédric Le Ninivin, Cédric de Saint Martin] * Rewrote slapgrid logger to always log into stdout. (commits a4d277c881, 5440626dea)[Cédric de Saint Martin] 0.31.2 (2012-10-02) ------------------- * Update slapproxy behavior: when instance already exist, only update partition_parameter_kw. (commit 317d5c8e0aee) [Cedric de Saint Martin] 0.31.1 (2012-10-02) ------------------- * Fixed Watchdog call in slapgrid. [Cédric Le Ninivin] 0.31 (2012-10-02) ------------------- * Added slapos-watchdog to bang exited and failing serices in instance in supervisord. (commits 16b2e8b8, 1dade5cd7) [Cédric Le Ninivin] * Add safety checks before calling SlapOS Master if mandatory instance members of SLAP classes are not properly set. Will result in less calls to SlapOS Master in dirty cases. (commits 5097e87c9763, 5fad6316a0f6d, f2cd014ea8aa) [Cedric de Saint Martin] * Add "periodicty" functionnality support for instances: if an instance has not been processed by slapgrid after defined time, process it. (commits 7609fc7a3d, 56e1c7bfbd) [Cedric Le Ninivin] * slapproxy: Various improvements in slave support (commits 96c6b78b67, bcac5a397d, fbb680f53b)[Cedric Le Ninivin] * slapgrid: bulletproof slapgrid-cp: in case one instance is bad, still processes all other ones. (commits bac94cdb56, 77bc6c75b3d, bd68b88cc3) [Cedric de Saint Martin] * Add support for "upload to binary cache" URL blacklist [Cedric de Saint Martin] * Request on proxy are identified by requester and name (commit 0c739c3) [Cedric Le Ninivin] 0.30 (2012-09-19) ----------------- * Add initial "slave instances" support in slapproxy. [Cedric Le Ninivin] * slapgrid-ur fix: check for partition informations only if we have to destroy it. [Cedric de Saint Martin] 0.29 (2012-09-18) ----------------- * buildout: Migrate slap_connection magic instance profile part to slap-connection, and use variables names separated with '-'. [Cedric de Saint Martin] * slapgrid: Add support for instance.cfg instance profiles [Cedric de Saint Martin] * slapgrid-ur: much less calls to master. [Cedric de Saint Martin] 0.28.9 (2012-09-18) ------------------- * slapgrid: Don't process not updated partitions (regression introduced in 0.28.7). [Cedric de Saint Martin] 0.28.8 (2012-09-18) ------------------- * slapgrid: Don't process free partitions (regression introduced in 0.28.7). [Cedric de Saint Martin] 0.28.7 (2012-09-14) ------------------- * slapgrid: --maximal_delay reappeared to be used in special cases. [Cedric de Saint Martin] 0.28.6 (2012-09-10) ------------------- * register now use slapos.cfg.example from master. [Cédric Le Ninivin] 0.28.5 (2012-08-23) ------------------- * Updated slapos.cfg for register [Cédric Le Ninivin] 0.28.4 (2012-08-22) ------------------- * Fixed egg building. 0.28.3 (2012-08-22) ------------------- * Avoid artificial tap creation on system check. [Łukasz Nowak] 0.28.2 (2012-08-17) ------------------- * Resolved path problem in register [Cédric Le Ninivin] 0.28.1 (2012-08-17) ------------------- * Resolved critical naming conflict 0.28 (2012-08-17) ----------------- * Introduce "slapos node register" command, that will register computer to SlapOS Master (vifib.net by default) for you. [Cédric Le Ninivin] * Set .timestamp in partitions ONLY after slapgrid thinks it's okay (promises, ...). [Cedric de Saint Martin] * slapgrid-ur: when destroying (not reporting), only care about instances to destroy, completely ignore others. [Cedric de Saint Martin] 0.27 (2012-08-08) ----------------- * slapformat: Raise correct error when no IPv6 is available on selected interface. [Cedric de Saint Martin] * slapgrid: Introduce --only_sr and --only_cp. - only_sr filter and force the run of a single SR, and uses url_md5 (folder_id) - only_cp filter which computer patition, will be runned. it can be a list, splited by comman (slappartX,slappartY ...) [Rafael Monnerat] * slapgrid: Cleanup unused option (--usage-report-periodicity). [Cedric de Saint Martin] * slapgrid: --develop will work also for Computer Partitions. [Cedric de Saint Martin] * slaplib: setConnectionDict won't call Master if parameters haven't changed. [Cedric de Saint Martin] 0.26.2 (2012-07-09) ------------------- * Define UTF-8 encoding in SlapOS Node codebase, as defined in PEP-263. 0.26.1 (2012-07-06) ------------------- * slapgrid-sr: Add --develop option to make it ignore .completed files. * SLAP library: it is now possible to fetch whole dict of connection parameters. * SLAP library: it is now possible to fetch single instance parameter. * SLAP library: change Computer and ComputerPartition behavior to have proper caching of computer partition parameters. 0.26 (2012-07-05) ----------------- * slapformat: no_bridge option becomes 'not create_tap'. create_tap is true by default. So a bridge is used and tap will be created by default. [Cedric de Saint Martin] * Add delay for slapformat. [Cedric Le Ninivin] * If no software_type is given, use default one (i.e fix "error 500" when requesting new instance). [Cedric de Saint Martin] * slapgrid: promise based software release, new api to fetch full computer information from server. [Yingjie Xu] * slapproxy: new api to mock full computer information [Yingjie Xu] * slapgrid: minor fix randomise delay feature. [Yingjie Xu] * slapgrid: optimise slapgrid-cp, run buildout only if there is an update on server side. [Yingjie Xu] * libslap: Allow accessing ServerError. [Vincent Pelletier] 0.25 (2012-05-16) ----------------- * Fix support for no_bridge option in configuration files for some values: no_bridge = false was stated as true. [Cedric de Saint Martin] * Delay a randomized period of time before calling slapgrid. [Yingjie Xu] * slapformat: Don't require tunctl if no_bridge is set [Leonardo Rochael] * slapformat: remove monkey patching when creating address so that it doesn't return false positive. [Cedric de Saint Martin] * Various: clearer error messages. 0.24 (2012-03-29) ----------------- * Handles different errors in a user friendly way [Cedric de Saint Martin] * slapgrid: Supports software destruction. [Łukasz Nowak] * slap: added support to Supply.supply state parameter (available, destroyed) [Łukasz Nowak] 0.23 (2012-02-29) ----------------- * slapgrid : Don't create tarball of sofwtare release when shacache is not configured. [Yingjie Xu] 0.22 (2012-02-09) ----------------- * slapformat : Add no-bridge feature. [Cedric de Saint Martin] * slapgrid : Add binary cache support. [Yingjie Xu] 0.21 (2011-12-23) ----------------- * slap: Add renaming API. [Antoine Catton] 0.20 (2011-11-24) ----------------- * slapgrid: Support service-less parttions. [Antoine Catton] * slapgrid: Avoid gid collision while dropping privileges. [Antoine Catton] * slapgrid: Drop down network usage during usage reporting. [Łukasz Nowak] * general: Add sphinx documentation. [Romain Courteaud] 0.19 (2011-11-07) ----------------- * bang: Executable to be called by being banged computer. [Łukasz Nowak] 0.18 (2011-10-18) ----------------- * Fix 0.17 release: missing change for slap library. [Łukasz Nowak] 0.17 (2011-10-18) ----------------- * slap: Avoid request under the hood. [Łukasz Nowak] * slap: ComputerPartition.bang provided. It allows to update all instances in tree. [Łukasz Nowak] * slap: Computer.bang provided. It allows to bang all instances on computer. [Łukasz Nowak] 0.16 (2011-10-03) ----------------- * slapgrid: Bugfix for slapgrid introduced in 0.15. [Łukasz Nowak] 0.15 (2011-09-27) ----------------- * slapgrid: Sanitize environment variables as early as possible. [Arnaud Fontaine] * slap: Docstring bugfix. [Sebastien Robin] * slap: Make request asynchronous call. [Łukasz Nowak] 0.14 (2011-08-31) ----------------- * slapgrid: Implement SSL based authentication to shadir and shacache. [Łukasz Nowak] * slapgrid, slap: Fix usage report packing list generation. [Nicolas Godbert] 0.13 (2011-08-25) ----------------- * slapgrid: Implement software signing and shacache upload. [Lucas Carvalho] * slap: Support slave instances [Gabriel Monnerat] * slapformat: Generate always address for computer [Łukasz Nowak] * slapgrid: Support promises scripts [Antoine Catton] * general: slapos.core gets tests. [many contributors] 0.12 (2011-07-15) ----------------- * Include modifications that should have been included in 0.11. 0.11 (2011-07-15) ----------------- * Bug fix : slapconsole : shorthand methods request and supply now correctly return an object. [Cedric de Saint Martin] 0.10 (2011-07-13) ----------------- * Fix a bug in slapconsole where request and supply shorthand methods don't accept all needed parameters. [Cedric de Saint Martin] 0.9 (2011-07-11) ---------------- * slapconsole: Simplify usage and use configuration file. You can now just run slapconsole and type things like "request(kvm, 'mykvm')". [Cedric de Saint Martin] * slapformat: Fix issue of bridge not connected with real interface on Linux >= 2.6.39 [Arnaud Fontaine] * slapformat: Allow to have IPv6 only interface, with bridge still supporting local IPv4 stack. [Łukasz Nowak] 0.8 (2011-06-27) ---------------- * slapgrid: Bugfix for temporary extends cache permissions. [Łukasz Nowak] 0.7 (2011-06-27) ---------------- * slapgrid: Fallback to buildout in own search path. [Łukasz Nowak] 0.6 (2011-06-27) ---------------- * slap: Fix bug: state shall be XML encapsulated. [Łukasz Nowak] 0.5 (2011-06-24) ---------------- * slapgrid: Use temporary extends-cache directory in order to make faster remote profile refresh. [Łukasz Nowak] 0.4 (2011-06-24) ---------------- * general: Polish requirement versions. [Arnaud Fontaine] * general: Remove libnetworkcache. [Lucas Carvalho] * slap: Remove not needed method from interface. [Romain Courteaud] * slap: state parameter is accepted and transmitted to SlapOS master [Łukasz Nowak] * slapformat: Implement dry run. [Vincent Pelletier] * slapgrid: Allow to select any buildout binary used to bootstrap environment. [Łukasz Nowak] 0.3 (2011-06-14) ---------------- * slap: Implement SLA by filter_kw in OpenOrder.request. [Łukasz Nowak] * slap: Timeout network operations. [Łukasz Nowak] * slapformat: Make slapsoft and slapuser* system users. [Kazuhiko Shiozaki] * slapgrid: Add more tolerance with supervisord. [Łukasz Nowak] 0.2 (2011-06-01) ---------------- * Include required files in distribution [Łukasz Nowak] 0.1 (2011-05-27) ---------------- * Merged slapos.slap, slapos.tool.console, slapos.tool.format, slapos.tool.grid, slapos.tool.libnetworkcache and slapos.tool.proxy into one package: slapos.core console ------- The slapconsole tool allows to interact with a SlapOS Master throught the SLAP library. For more information about SlapOS or slapconsole usages, please go to http://community.slapos.org. The slapconsole tool is only a bare Python console with several global variables defined and initialized. Initialization and configuration file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Slapconsole allows to automatically connect to a Master using URL and SSL certificate from given slapos.cfg. Certificate has to be *USER* certificate, manually obtained from SlapOS master web interface. Slapconsole tools reads the given slapos.cfg configuration file and use the following informations : * Master URL is read from [slapos] in the "master_url" parameter. * SSL Certificate is read from [slapconsole] in the "cert_file" parameter. * SSL Key is read from [slapconsole] in the "key_file" parameter. See slapos.cfg.example for examples. Global functions ~~~~~~~~~~~~~~~~ * request() is a shorthand for slap.registerOpenOrder().request() allowing to request instances. * supply() is a shorthand for slap.registerSupply().supply() allowing to request software installation. For more information about those methods, please read the SLAP library documentation. Global aliases ~~~~~~~~~~~~~~ "software_list" is a list containing all the Software Release URLs defined in client slapos.cfg configuration file. Also, each Software Release defined in this configuration file is translated into a global variable to ease the request of those Sofware Releases. This allows to request instances in a few words, i.e:: request("http://www.url.com/path/to/kvm/software.cfg", "mykvm") can be simplified into :: request(kvm, "mykvm") If the slapos.cfg file contains :: alias = kvm http://www.url.com/path/to/kvm/software.cfg Global objects ~~~~~~~~~~~~~~ "slap" is an instance of the SLAP library. It is only used for advanced usages. "slap" instance is obtained by doing :: slap = slapos.slap.slap() slap.initializeConnection(config.master_url, key_file=config.key_file, cert_file=config.cert_file) Examples ~~~~~~~~ :: >>> # Request instance >>> request(kvm, "myuniquekvm") >>> # Request instance on specific computer >>> request(kvm, "myotheruniquekvm", filter_kw={ "computer_guid": "COMP-12345" }) >>> # Request instance, specifying parameters (here nbd_ip and nbd_port) >>> request(kvm, "mythirduniquekvm", partition_parameter_kw={"nbd_ip":"2a01:e35:2e27:460:e2cb:4eff:fed9:48dc", "nbd_port":"1024"}) >>> # Request software installation on owned computer >>> supply(kvm, "mycomputer") >>> # Fetch existing instance status >>> request(kvm, "myuniquekvm").getState() >>> # Fetch instance information on already launched instance >>> request(kvm, "myuniquekvm").getConnectionParameter("url") format ====== slapformat is an application to prepare SlapOS ready node (machine). It "formats" the machine by: - creating users and groups - creating bridge interface - creating needed tap interfaces - creating needed directories with proper ownership and permissions In the end special report is generated and information are posted to configured SlapOS server. This program shall be only run by root. Requirements ------------ Linux with IPv6, bridging and tap interface support. Binaries: * brctl * groupadd * ip * tunctl * useradd grid ==== slapgrid is a client of SLAPos. SLAPos provides support for deploying a SaaS system in a minute. Slapgrid allows you to easily deploy instances of softwares based on buildout profiles. For more informations about SLAP and SLAPos, please see the SLAP documentation. Requirements ------------ A working SLAP server with informations about your computer, in order to retrieve them. As Vifib servers use IPv6 only, we strongly recommend an IPv6 enabled UNIX box. For the same reasons, Python >= 2.6 with development headers is also strongly recommended (IPv6 support is not complete in previous releases). For now, gcc and glibc development headers are required to build most software releases. Concepts -------- Here are the fundamental concepts of slapgrid : A Software Release (SR) is just a software. A Computer Partition (CP) is an instance of a Software Release. Imagine you want to install with slapgrid some software and run it. You will have to install the software as a Software Release, and then instantiate it, i.e configuring it for your needs, as a Computer Partition. How it works ------------ When run, slapgrid will authenticate to the SLAP library with a computer_id and fetch the list of Software Releases to install or remove and Computer Partitions to start or stop. Then, it will process each Software Release, and each Computer Partition. It will also periodically send to SLAP the usage report of each Computer Partition. Installation ------------ With easy_install:: $ easy_install slapgrid slapgrid needs several directories to be created and configured before being able to run : a software releases directory, and an instances directory with configured computer partition directory(ies). You should create for each Computer Partition directory created a specific user and associate it with its Computer Partition directory. Each Computer Partition directory should belongs to this specific user, with permissions of 0750. Usage ----- slapgrid needs several informations in order to run. You can specify them by adding arguments to the slapgrid command line, or by putting then in a configuration file. Beware : you need a valid computer resource on server side. Examples -------- simple example : Just run slapgrid: $ slapgrid --instance-root /path/to/instance/root --software-root /path/to/software_root --master-url https://some.server/some.resource --computer-id my.computer.id configuration file example:: [slapgrid] instance_root = /path/to/instance/root software_root = /path/to/software/root master_url = https://slapos.server/slap_service computer_id = my.computer.id then run slapgrid:: $ slapgrid --configuration-file = path/to/configuration/file proxy ===== Implement minimalist SlapOS Master server without any security, designed to work only from localhost with one SlapOS Node (a.k.a Computer). It implements (or should implement) the SLAP API, as currently implemented in the SlapOS Master (see slaptool.py in Master). The only behavioral difference from the SlapOS Master is: When the proxy doesn't find any free partition (and/or in case of slave instance, any compatible master instance), it will throw a NotFoundError (404). slap ==== Simple Language for Accounting and Provisioning python library. Developer note - python version ------------------------------- This library is used on client (slapgrid) and server side. Server is using python2.4 and client is using python2.6 Having this in mind, code of this library *have* to work on python2.4 How it works ------------ The SLAP main server which is in charge of service coordination receives from participating servers the number of computer paritions which are available, the type of resource which a party is ready provide, and request from parties for resources which are needed. Each participating server is identified by a unique ID and runs a slap-server daemon. This daemon collects from the main server the installation tasks and does the installation of resources, then notifies the main server of completion whenever a resource is configured, installed and available. The data structure on the main server is the following: A - Action: an action which can happen to provide a resource or account its usage CP - Computer Partition: provides a URL to Access a Cloud Resource RI - Resource Item: describes a resource CI - Contract Item: describes the contract to attach the DL to (This is unclear still) R - Resource: describes a type of cloud resource (ex. MySQL Table) is published on slapgrid.org DL - Delivery Line: Describes an action happening on a resource item on a computer partition D - Delivery: groups multiple Delivery Lines Keywords: slapos core Platform: UNKNOWN Classifier: Programming Language :: Python slapos.core-0.35.1/slapos.core.egg-info/requires.txt0000644000076600007660000000017512110425554030442 0ustar cedricdesaintmartincedricdesaintmartin00000000000000Flask lxml netaddr>=0.7.5 netifaces pyflakes setuptools supervisor unittest2 xml_marshaller>=0.9.3 zope.interface zc.buildoutslapos.core-0.35.1/slapos.core.egg-info/SOURCES.txt0000644000076600007660000000343612110425554027731 0ustar cedricdesaintmartincedricdesaintmartin00000000000000CHANGES.txt MANIFEST.in README.txt setup.cfg setup.py slapos/README.console.txt slapos/README.format.txt slapos/README.grid.txt slapos/README.proxy.txt slapos/README.slap.txt slapos/__init__.py slapos/bang.py slapos/cache.py slapos/client.py slapos/entry.py slapos/format.py slapos/slapos.xsd slapos/util.py slapos/version.py slapos.core.egg-info/PKG-INFO slapos.core.egg-info/SOURCES.txt slapos.core.egg-info/dependency_links.txt slapos.core.egg-info/entry_points.txt slapos.core.egg-info/namespace_packages.txt slapos.core.egg-info/not-zip-safe slapos.core.egg-info/requires.txt slapos.core.egg-info/top_level.txt slapos/grid/SlapObject.py slapos/grid/__init__.py slapos/grid/distribution.py slapos/grid/exception.py slapos/grid/networkcache.py slapos/grid/slapgrid.py slapos/grid/svcbackend.py slapos/grid/utils.py slapos/grid/watchdog.py slapos/grid/zc.buildout-bootstap.py slapos/grid/templates/buildout-tail.cfg.in slapos/grid/templates/group_partition_supervisord.conf.in slapos/grid/templates/program_partition_supervisord.conf.in slapos/grid/templates/supervisord.conf.in slapos/proxy/__init__.py slapos/proxy/db_version.py slapos/proxy/query.py slapos/proxy/schema.sql slapos/proxy/views.py slapos/register/__init__.py slapos/register/register.py slapos/slap/__init__.py slapos/slap/slap.py slapos/slap/slap_recipe_mysqldatabase.py.txt slapos/slap/doc/computer_consumption.xsd slapos/slap/doc/partition_consumption.xsd slapos/slap/doc/software_instance.xsd slapos/slap/interface/__init__.py slapos/slap/interface/slap.py slapos/tests/__init__.py slapos/tests/entry.py slapos/tests/interface.py slapos/tests/slap.py slapos/tests/slapformat.py slapos/tests/slapgrid.py slapos/tests/slapobject.py slapos/tests/slapproxy.py slapos/tests/mock/__init__.py slapos/tests/mock/httplib.py slapos/tests/pyflakes/__init__.pyslapos.core-0.35.1/slapos.core.egg-info/top_level.txt0000644000076600007660000000000712110425554030566 0ustar cedricdesaintmartincedricdesaintmartin00000000000000slapos