package-manager-1.3.3/0000755000175000017500000000000013241343432014066 5ustar rhaistrhaistpackage-manager-1.3.3/README.rst0000777000175000017500000000000013241343432020717 2doc/overview.rstustar rhaistrhaistpackage-manager-1.3.3/COPYING0000644000175000017500000000351613241343432015126 0ustar rhaistrhaistCopyright (c) 2016 The Board of Trustees of the University of Illinois. All rights reserved. Developed by: Cybersecurity Directorate National Center for Supercomputing Applications University of Illinois http://illinois.edu Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal with the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimers. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimers in the documentation and/or other materials provided with the distribution. * Neither the names of the National Center for Supercomputing Applications, the University of Illinois, nor the names of its contributors may be used to endorse or promote products derived from this Software without specific prior written permission. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE SOFTWARE. Note that some files in the distribution may carry their own copyright notices. package-manager-1.3.3/bro-pkg.config0000644000175000017500000000530613241343432016622 0ustar rhaistrhaist# This is an example config file for bro-pkg to explain what # settings are possible as well as their default values. # The order of precedence for how bro-pkg finds/reads config files: # # (1) bro-pkg --configfile=/path/to/custom/config # (2) the BRO_PKG_CONFIG_FILE environment variable # (3) a config file located at $HOME/.bro-pkg/config # (4) if none of the above exist, then bro-pkg uses builtin/default # values for all settings shown below [sources] # The default package source repository from which bro-pkg fetches # packages. The default source may be removed, changed, or # additional sources may be added as long as they use a unique key # and a value that is a valid git URL. bro = https://github.com/bro/packages [paths] # Directory where source repositories are cloned, packages are # installed, and other package manager state information is # maintained. If left blank, this defaults to $HOME/.bro-pkg state_dir = # The directory where package scripts are copied upon installation. # A subdirectory named "packages" is always created within the # specified path and the package manager will copy the directory # specified by the "script_dir" option of each package's bro-pkg.meta # file there. # If left blank, this defaults to /script_dir # A typical path to set here is /share/bro/site # If you decide to change this location after having already # installed packages, bro-pkg will automatically relocate them # the next time you run any bro-pkg command. script_dir = # The directory where package plugins are copied upon installation. # A subdirectory named "packages" is always created within the # specified path and the package manager will copy the directory # specified by the "plugin_dir" option of each package's bro-pkg.meta # file there. # If left blank, this defaults to /plugin_dir # A typical path to set here is /lib/bro/plugins # If you decide to change this location after having already # installed packages, bro-pkg will automatically relocate them # the next time you run any bro-pkg command. plugin_dir = # The directory containing Bro distribution source code. This is only # needed when installing packages that contain Bro plugins that are # not pre-built. bro_dist = [user_vars] # For any key in this section that is matched for value interpolation # in a package's bro-pkg.meta file, the corresponding value is # substituted during execution of the package's `build_command`. # This section is typically automatically populated with the # the answers supplied during package installation prompts # and, as a convenience feature, used to recall the last-used settings # during subsequent operations (e.g. upgrades) on the same package. package-manager-1.3.3/CHANGES0000644000175000017500000005123713241343432015071 0ustar rhaistrhaist 1.3.3 | 2018-02-15 11:04:09 -0600 * Release 1.3.3. 1.3.2-1 | 2018-02-15 11:02:20 -0600 * GH-26: fix error when config_file is not in script_dir or plugin_dir (Corelight) 1.3.2 | 2018-02-01 17:38:15 -0600 * Release 1.3.2. 1.3.1-5 | 2018-02-01 17:37:34 -0600 * Improve logging stream levels for a couple log messages (Corelight) * Change default confirmation prompt answer to 'no' on test failures (Corelight) 1.3.1-3 | 2018-02-01 17:17:54 -0600 * Minor doc improvements (Corelight) 1.3.1-2 | 2018-02-01 16:13:25 -0600 * Teach bro-pkg to emit an error when operating on dirty git clones When trying to use the install, test, or bundle commands on local git clones that have modifications or untracked files, bro-pkg now aborts with an error message. This is to help catch situations where one thinks their uncommitted local changes will be installed by bro-pkg, but actually will not be unless they are committed in the git repo. (Corelight) 1.3.1-1 | 2018-02-01 15:43:19 -0600 * Fix install command to be able to re-install an existing package Before, using 'install' on a previously-installed package would not correctly update the cached git repository so if there were any changes since the last installation they would not be seen after the operation completed. (Corelight) 1.3.1 | 2017-12-07 19:33:23 -0600 * Release 1.3.1. 1.3-1 | 2017-12-07 19:29:35 -0600 * GH-22: fix 'suggests' field in json output to use key-value pairs (Corelight) 1.3 | 2017-11-28 19:34:15 -0600 * Release 1.3. 1.2.6-2 | 2017-11-28 19:29:38 -0600 * GH-22: add a 'suggests' metadata field This new field can be used in the same way as 'depends', but is used for packages that are merely complementary and not strictly required for the suggesting package to function. A "(suggested)" indicator for install, bundle, and upgrade commands will now appear during their bro-pkg confirmation prompts. These commands also now have a --nosuggestions flag to ignore all suggested packages. This changes the bropkg.manager.validate_dependencies() API to return an additional boolean indicating whether a package was included in the list as a result of suggestions. (Corelight) 1.2.6-1 | 2017-11-27 09:46:02 -0600 * GH-24: improve script_dir metadata field documentation (Corelight) 1.2.6 | 2017-11-23 15:27:46 -0600 * Release 1.2.6. 1.2.5-1 | 2017-11-23 15:27:12 -0600 * GH-23: more "Edit on GitHub" breadcrumb customization. (Corelight) 1.2.5 | 2017-11-23 14:42:33 -0600 * Release 1.2.5. 1.2.4-1 | 2017-11-23 14:41:09 -0600 * GH-23: one more "Edit on GitHub" link fix attempt. (Corelight) 1.2.4 | 2017-11-23 13:57:00 -0600 * Release 1.2.4. 1.2.3-1 | 2017-11-23 13:54:11 -0600 * GH-23: another attempt to remove "Edit on GitHub" link. (Corelight) 1.2.3 | 2017-11-23 13:29:07 -0600 * Release 1.2.3. 1.2.2-2 | 2017-11-23 13:28:15 -0600 * GH-23: remove broken "Edit on GitHub" link from stable version of docs. (Corelight) 1.2.2-1 | 2017-11-23 13:27:06 -0600 * Add a missing dependency for building docs. (Corelight) 1.2.2 | 2017-10-16 16:14:02 -0500 * Release 1.2.2. 1.2.1-1 | 2017-10-16 16:11:37 -0500 * GH-21: again trying to give aggregate metadata consistent ordering (Jon Siwek) 1.2.1 | 2017-10-11 15:03:56 -0500 * Release 1.2.1. 1.2-1 | 2017-10-11 15:02:21 -0500 * Fix building PyPI distribution from Python 3+ It should now still include the configparser backport dependency. (Jon Siwek) 1.2 | 2017-09-29 13:15:37 -0500 * Release 1.2. 1.1-1 | 2017-09-29 13:15:00 -0500 * BIT-1852: allow 'bro-pkg' as a 'depends' field value This allows packages to depend on new features of bro-pkg not available in previous version. (Jon Siwek) 1.1 | 2017-09-28 18:46:30 -0500 * Release 1.1. 1.0.4-32 | 2017-09-28 18:45:54 -0500 * BIT-1852: allow plugin_dir to also accept tarfiles This allows packagers to specify a tarfile containing the Bro plugin instead of a directory. May be useful as the default Bro plugin CMake skeleton will end up producing a tarfile as part of the plugin build process that contains the minimal set of files comprising the plugin. (Jon Siwek) 1.0.4-31 | 2017-09-28 16:16:39 -0500 * GH-9: add new 'user_vars' bro-pkg.meta field and config file section This field allows for packages to solicit users for extra configuration settings which may then be referenced in that package's build_command. Useful for asking users for installation paths of external dependencies, which may differ between users/systems. (Jon Siwek) 1.0.4-30 | 2017-09-27 11:14:52 -0500 * GH-9: show 'external_depends' metadata during 'unbundle' command (Jon Siwek) * Add manager.bundle_info() to API. It retrieves information on all packages in a bundle without installing the bundle. (Jon Siwek) 1.0.4-28 | 2017-09-27 10:14:16 -0500 * Remove an error msg when upgrading packages that have no test_command (Jon Siwek) 1.0.4-27 | 2017-09-26 13:27:53 -0500 * GH-9: allow 'external_depends' field in bro-pkg.meta. The contents of this field get displayed to users in the install and upgrade commands to indicate extra pre-requisite software their OS should have installed. (Jon Siwek) 1.0.4-26 | 2017-09-26 11:41:37 -0500 * Put install's "unit test skipped" msg into log stream instead of stdout. (Jon Siwek) 1.0.4-25 | 2017-09-22 12:26:56 -0500 * GH-15: add --nodeps to install, bundle, and upgrade commands This option allows to bypasses dependency resolution/checking, accepting risk of causing broken/unusable package state. (Jon Siwek) 1.0.4-24 | 2017-09-20 13:11:48 -0500 * Code formatting. (Jon Siwek) 1.0.4-23 | 2017-09-20 13:07:57 -0500 * Add '--allvers' option to output metadata for ALL versions when outputting JSON. (Terry Fleury) * Add JSON output as option to 'info' command. (Terry Fleury) 1.0.4-18 | 2017-09-20 12:52:17 -0500 * Add 'info' option to force read metadata from remote GitHub. (Terry Fleury) 1.0.4-16 | 2017-09-20 12:39:21 -0500 * Version numbers should be sorted naturally, e.g., 1.10.0 is newer than 1.9.0. (Justin Azoff, Terry Fleury) 1.0.4-14 | 2017-09-20 12:31:59 -0500 * Add option for 'list' command '--nodesc' to suppress description output. (Terry Fleury) 1.0.4-12 | 2017-09-16 13:36:53 -0500 * GH-13: emit an error when a package uses a malformed 'depends' field (Jon Siwek) 1.0.4-11 | 2017-09-16 12:44:50 -0500 * Fix a typo in an error message. (Jon Siwek) 1.0.4-10 | 2017-09-15 13:04:46 -0500 * GH-12: use active branch as package version for local git repos (Jon Siwek) 1.0.4-9 | 2017-09-15 11:47:09 -0500 * GH-10: add a cross-reference for metadata value interpolation in docs (Jon Siwek) 1.0.4-8 | 2017-09-15 11:39:24 -0500 * GH-10: improve docs regarding metadata value interpolation (Jon Siwek) 1.0.4-6 | 2017-09-14 19:52:21 -0500 * Minor quickstart doc simplification. (Jon Siwek) 1.0.4-5 | 2017-09-14 18:56:16 -0500 * Require extra configparser package only for Python < 3.5 where it is not included (Hilko Bengen) * Fix imports for python3 (Robert Haist) 1.0.4-2 | 2017-09-14 15:05:36 -0500 * Add a docutils config file to prevent sphinx/smartypants "smart quotes" Without it, later versions of sphinx are turning quote characters into smart/curly quotes which breaks the generated man page. (Jon Siwek) 1.0.4-1 | 2017-09-14 15:03:04 -0500 * Quickstart doc improvements, add missing links to dependencies (Jon Siwek) 1.0.4 | 2017-07-18 11:41:56 -0500 * Release 1.0.4. 1.0.3-1 | 2017-07-18 11:39:38 -0500 * Give aggregate metadata file a consistent ordering Addresses https://github.com/bro/package-manager/issues/6 (Jon Siwek) 1.0.3 | 2017-06-23 10:17:26 -0500 * Release 1.0.3. 1.0.2-3 | 2017-06-23 10:15:41 -0500 * Install command now doesn't warn/prompt when package has no test_command. (Jon Siwek) * Fix a unit test. (Jon Siwek) * Improve usage documentation: add offline usage example. (Jon Siwek) 1.0.2 | 2017-06-07 22:11:20 -0500 * Release 1.0.2. 1.0.1-1 | 2017-06-07 22:04:55 -0500 * Improve confirmation prompt behavior. Just pressing enter defaults to "yes" and anything besides y/ye/yes (case insensitive) is treated as "no" and aborts. Addresses https://github.com/bro/package-manager/issues/5 (Jon Siwek) 1.0.1 | 2017-06-05 09:54:59 -0500 * Release 1.0.1. 1.0-3 | 2017-06-05 09:54:31 -0500 * Improved handling of packages without scripts. The 'load' and 'unload' commands now return more precise messages if used for packages without scripts. (Jan Grashoefer) 1.0-1 | 2017-02-06 21:02:31 -0600 * Return a proper "bro-config not found" message when running tests. (Jon Siwek) 1.0 | 2017-01-24 19:53:49 -0600 * Release 1.0. 0.9-24 | 2017-01-24 19:53:27 -0600 * Simply directions for uploading to PyPi. (Jon Siwek) 0.9-23 | 2017-01-24 19:41:22 -0600 * Add dependency on btest. (Jon Siwek) 0.9-22 | 2017-01-24 18:09:28 -0600 * Change manager.validate_dependencies API (Jon Siwek) 0.9-21 | 2017-01-21 12:40:09 -0600 * Doc fixes. (Jon Siwek) 0.9-20 | 2017-01-21 12:29:48 -0600 * Run package tests before install/upgrade. (Jon Siwek) * Add 'test' command and 'test_command' metadata field. (Jon Siwek) 0.9-16 | 2017-01-13 18:50:52 -0600 * Fix non-deterministic order of {installed,loaded}_packages() (Jon Siwek) * Add unit tests. (Jon Siwek) * Fix bundle/unbundle --help output. (Jon Siwek) * Don't copy a package's .git or bro-pkg.meta when installing them. (Jon Siwek) 0.9-8 | 2017-01-12 10:23:26 -0600 * Add 'depends' bro-pkg.meta field. (Jon Siwek) * Teach install/upgrade/bundle commands to operate on package dependencies. These commands now construct a dependency graph based on the packages the user has requested plus any already-installed packages and then validates the graph to make sure there are no dependency conflicts. If there are no conflicts, then the set of requested packages get installed/bundled along with any additional packages that were required to satisfy dependencies. For those additional packages, bro-pkg always tries to pick the latest release version that satisfies all dependency specifications. (Jon Siwek) 0.9-2 | 2017-01-09 11:16:43 -0600 * Improve 'info' output for packages w/ empty metadata files. (Jon Siwek) 0.9-1 | 2016-12-09 11:52:23 -0600 * Fix doc typo. (Jon Siwek) 0.9 | 2016-12-08 21:19:02 -0600 * Release 0.9. 0.8-6 | 2016-12-08 21:15:06 -0600 * Fix tracking package versions/changes via branches. A previous change to use shallow git clones broke the tracking of package versioning via git branches. (Jon Siwek) 0.8-5 | 2016-12-08 20:04:53 -0600 * Fix bundle/unbundle automatic lower-casing of urls in bundle manifest (Jon Siwek) 0.8-4 | 2016-12-08 19:50:49 -0600 * Add new bro-pkg.meta field: config_files This allows package authors to specify a list of files that users may directly modify. The upgrade, remove, install, and bundle operations all work in a way that attempts to either preserve a user's local changes or save a backup of their version. (Jon Siwek) 0.8-3 | 2016-12-07 22:54:49 -0600 * BIT-1768: improve handling of packages that have no Bro scripts (Jon Siwek) 0.8-2 | 2016-12-07 20:46:44 -0600 * BIT-1766: add tags() and short_description() to PackageInfo API (Jon Siwek) 0.8-1 | 2016-12-07 20:09:58 -0600 * BIT-1767: improve handling of non-ascii character encoding (Jon Siwek) 0.8 | 2016-11-29 11:21:29 -0600 * Release 0.8. 0.7-19 | 2016-11-29 11:19:17 -0600 * Improve output of info command and allow offline usage. (Jon Siwek) * Update documentation related to metadata aggregation. (Jon Siwek) * Metadata aggregation changes: - bro-pkg.index is now expected to be a simple list of package urls - add --aggregate flag to refresh command that will crawl packages and aggregate their metadata into an aggregate.meta file within local package source clones - add --sources flag to refresh command to explicitly specify operation on a specific set of package sources - add --push flag to refresh command to push local package source aggregate.meta changes to the remote repo (Jon Siwek) 0.7-10 | 2016-11-21 18:04:08 -0600 * Package versioning doc improvements. (Jon Siwek) 0.7-9 | 2016-11-21 12:15:14 -0600 * Update docs regarding package versioning. (Jon Siwek) 0.7-8 | 2016-10-21 10:13:16 -0500 * Using shutil.move instead of os.rename, which works across file systems. (Robin Sommer) 0.7-6 | 2016-10-11 18:54:50 -0500 * BIT-1725: allow bundling on offline machines When requesting to bundle the set of installed packages, they are directly copied into the bundle instead of trying to clone from the remote git repo. (Jon Siwek) 0.7-5 | 2016-10-04 14:05:50 -0500 * BIT-1713: in python3, show usage info when bro-pkg is given no args (Jon siwek) 0.7-4 | 2016-10-03 18:20:57 -0500 * BIT-1716: do shallow git clones of packages to use less space (Jon Siwek) 0.7-3 | 2016-10-03 17:08:31 -0500 * BIT-1715: improve message when failing to clone a package source And make that situation a warning instead of a fatal error. (Jon Siwek) 0.7-2 | 2016-10-03 16:49:24 -0500 * Allow empty values for paths in config file. (Jon Siwek) 0.7-1 | 2016-10-03 15:55:00 -0500 * BIT-1714: expand '~' and env. vars in config file values. And reject relative paths in [paths] section. (Jon Siwek) 0.7 | 2016-09-29 15:26:14 -0500 * Release 0.7. 0.6-14 | 2016-09-29 15:25:44 -0500 * Improve output of 'info' command. (Jon Siwek) 0.6-13 | 2016-09-29 14:59:31 -0500 * Tweak output of bundle/unbundle. (Jon Siwek) 0.6-12 | 2016-09-29 14:43:11 -0500 * Python 3 compatibility fixes. (Jon Siwek) 0.6-11 | 2016-09-28 18:11:44 -0500 * Add 'bundle' and 'unbundle' commands. (Jon Siwek) 0.6-10 | 2016-09-27 13:56:08 -0500 * Fix potential race condition in package installation. (Jon Siwek) 0.6-9 | 2016-09-26 15:31:50 -0500 * Improve formatting of package short descriptions. (Jon Siwek) 0.6-8 | 2016-09-26 15:15:18 -0500 * BIT-1699: improve package validation and format of output. (Jon Siwek) 0.6-7 | 2016-09-26 13:40:26 -0500 * Add a 'purge' command that removes all installed packages. (Jon Siwek) 0.6-6 | 2016-09-21 12:59:01 -0500 * BIT-1699: Add --force flag for install/remove/upgrade. (Jon Siwek) 0.6-5 | 2016-09-21 12:22:18 -0500 * BIT-1699: Add confirmation prompts for install/upgrade/remove. Removed the --all flag for upgrade command. (Jon Siwek) 0.6-4 | 2016-09-21 10:47:53 -0500 * BIT-1697: Improve output format of list/search. (Jon Siwek) 0.6-3 | 2016-09-19 13:29:27 -0500 * BIT-1698: autoconfig command now directly writes to config file. (Jon Siwek) 0.6-2 | 2016-09-19 12:09:52 -0500 * BIT-1697: Add description field to list/search output. (Jon Siwek) 0.6-1 | 2016-09-19 10:52:51 -0500 * BIT-1701: Improve error message when 'install' can't create symlink. (Jon Siwek) 0.6 | 2016-09-14 15:16:30 -0500 * Release 0.6. 0.5-2 | 2016-09-14 15:15:50 -0500 * Improve docs for script_dir/plugin_dir config file options. (Jon Siwek) 0.5-1 | 2016-09-14 15:09:12 -0500 * Improve automatic relocation of script_dir. (Jon Siwek) 0.5 | 2016-09-05 15:03:14 -0500 * Release 0.5. 0.4-2 | 2016-09-05 15:01:38 -0500 * Improve error message for insufficient permissions (Jon Siwek) 0.4-1 | 2016-09-05 14:09:55 -0500 * Improve output of 'info' command. (Jon Siwek) 0.4 | 2016-08-16 16:55:48 -0500 * Release 0.4. 0.3-4 | 2016-08-16 16:55:20 -0500 * Docs: add 'autoconfig' command (Jon Siwek) 0.3-3 | 2016-08-16 16:10:21 -0500 * Add 'autoconfig' command, remove 'bro_exe' option. bro-pkg can now use bro-config to automatically determine paths. (Jon Siwek) 0.3-2 | 2016-08-15 13:32:18 -0500 * Docs: clarifications (Jon Siwek) 0.3-1 | 2016-08-12 16:20:30 -0500 * Docs: add bro.org link in README (Jon Siwek) 0.3 | 2016-08-12 15:54:18 -0500 * Release 0.3. 0.2-47 | 2016-08-12 15:53:13 -0500 * Docs: add versioning/release guide for developers (Jon Siwek) 0.2-46 | 2016-08-11 19:45:21 -0500 * setup.py packaging improvements (Jon Siwek) 0.2-45 | 2016-08-11 14:35:06 -0500 * Docs: fix sphinx-argparse 'nosubcommands' option (Jon Siwek) 0.2-44 | 2016-08-11 14:16:22 -0500 * Docs: adjust admonitions in README (Jon Siwek) 0.2-43 | 2016-08-11 14:09:44 -0500 * Docs: organize external links (Jon Siwek) 0.2-42 | 2016-08-10 19:58:15 -0500 * Fix `list all` to show installed packages that don't come from a source (Jon Siwek) 0.2-41 | 2016-08-10 19:35:53 -0500 * Docs: add package creation walkthroughs (Jon Siwek) * Don't load packages unless they have a __load__.bro (Jon Siwek) * Improve handling of relative paths in package urls (Jon Siwek) 0.2-38 | 2016-08-10 14:29:30 -0500 * Teach search command to inspect package index tags (Jon Siwek) 0.2-37 | 2016-08-09 18:55:30 -0500 * Delay printing installation activity ticks. (Jon Siwek) 0.2-36 | 2016-08-09 18:41:54 -0500 * Change package sources to use index files instead of git submodules. (Jon Siwek) 0.2-35 | 2016-08-09 13:54:46 -0500 * Install packages in a thread. So it doesn't look like bro-pkg is hanging when installing a package that has to be built. (Jon Siwek) 0.2-34 | 2016-08-09 13:06:24 -0500 * Using ``install --version`` now also pins the package. (Jon Siwek) 0.2-33 | 2016-08-08 16:19:59 -0500 * info command can now take a --version flag. (Jon Siwek) 0.2-32 | 2016-08-08 15:16:54 -0500 * Fix install command to use the right version of bro-pkg.meta. (Jon Siwek) 0.2-31 | 2016-08-08 12:47:02 -0500 * Change config/metadata file option names. (Jon Siwek) bro-pkg.meta: scriptpath -> script_dir pluginpath -> plugin_dir buildcmd -> build_command bro-pkg config file: statedir -> state_dir scriptdir -> script_dir plugindir -> plugin_dir 0.2-30 | 2016-08-07 13:45:00 -0500 * Rename pkg.meta to bro-pkg.meta. (Jon Siwek) 0.2-29 | 2016-08-07 13:08:00 -0500 * upgrade command: require --all when omitting package list. (Jon Siwek) 0.2-28 | 2016-08-07 12:51:20 -0500 * Docs: adjust bro-pkg --help output. (Jon Siwek) 0.2-27 | 2016-08-07 12:35:40 -0500 * Add warnings to not directly modify certain files/dirs. (Jon Siwek) 0.2-26 | 2016-08-07 12:05:40 -0500 * Docs: show actual version in sidebar. (Jon Siwek) 0.2-25 | 2016-08-06 15:08:12 -0500 * Docs: don't restrict argparse-generated table widths. (Jon Siwek) * Docs: misc. style fixes and clarifications. (Jon Siwek) * Docs: switch theme (again) to Read the Docs theme. 0.2-22 | 2016-08-06 13:12:26 -0500 * Docs: clarifications, organization, style. (Jon Siwek) 0.2-21 | 2016-08-05 13:41:47 -0500 * Docs: change theme. (Jon Siwek) * Docs: GitPython and semantic_version build requirements (Jon Siwek) 0.2-19 | 2016-08-04 13:25:48 -0500 * Docs: add requirements file (Jon Siwek) 0.2-18 | 2016-08-03 16:10:18 -0500 * Docs: fix Sphinx argparse extension to check for outdated modules. Sphinx itself should now always be able to determine whether the command-line bro-pkg documentation needs to be rebuilt. (Jon Siwek) 0.2-17 | 2016-08-02 14:46:16 -0500 * Docs: rebuild docs on version string change. (Jon Siwek) 0.2-16 | 2016-08-02 14:34:49 -0500 * Docs: give each bro-pkg command its own section (Jon Siwek) 0.2-15 | 2016-08-01 14:34:06 -0500 * Documentation organization/layout/theming. (Jon Siwek) 0.2-14 | 2016-07-24 14:55:28 -0500 * Clarification to installation doc. (Jon Siwek) 0.2-13 | 2016-07-24 12:18:40 -0500 * Add a 'gh-pages' Makefile target. (Jon Siwek) * Add .nojekyll file to built sphinx docs. (Jon Siwek) 0.2-11 | 2016-07-23 15:32:24 -0500 * Add installation/setup docs. (Jon Siwek) 0.2-10 | 2016-07-23 14:19:09 -0500 * Add a prebuilt man page. (Jon Siwek) 0.2-9 | 2016-07-22 15:07:57 -0500 * Change bro-pkg to not require a config file and add setup.py. (Jon Siwek) 0.2-8 | 2016-07-21 12:14:53 -0500 * Improve speed of checking for changed remote package sources. (Jon Siwek) * Improve how 'info' command locates packages. (Jon Siwek) * Emit an error when package has non-existant 'scriptpath'. (Jon Siwek) * Documentation clarifications. (Jon Siwek) 0.2-4 | 2016-07-21 09:49:15 -0500 * Make output of 'info' command have a stable order. (Jon Siwek) 0.2-3 | 2016-07-21 09:44:42 -0500 * Python3 compatibility fixes. (Jon Siwek) 0.2-2 | 2016-07-20 14:35:38 -0500 * Documentation and other clean up. Sphinx is used docs: manual, api reference, man page. (Jon Siwek) 0.2-1 | 2016-07-14 12:26:56 -0500 * Fix a mistake in --help output. (Jon Siwek) 0.2 | 2016-07-14 10:51:11 -0500 * Release 0.2. package-manager-1.3.3/bro-pkg0000755000175000017500000021423213241343432015361 0ustar rhaistrhaist#! /usr/bin/env python from __future__ import print_function from bropkg._util import ( make_dir, find_program, read_bro_config_line, ) import os import io import sys import errno import argparse import logging import threading import tarfile import filecmp import shutil import subprocess import json try: from backports import configparser except ImportError as err: import configparser import git import bropkg def get_input(prompt): if sys.version_info[0] < 3: prompt_bytes = prompt.encode(sys.stdout.encoding) return raw_input(prompt_bytes).decode(sys.stdin.encoding) return input(prompt) def confirmation_prompt(prompt, default_to_yes=True): yes = {'y', 'ye', 'yes'} if default_to_yes: prompt += ' [Y/n] ' else: prompt += ' [N/y] ' choice = get_input(prompt).lower() if not choice: return default_to_yes if choice in yes: return True print('Abort.') return False def prompt_for_user_vars(manager, config, force, pkg_infos): answers = {} for info in pkg_infos: name = info.package.qualified_name() requested_user_vars = info.user_vars() if requested_user_vars is None: print_error(str.format('error: malformed user_vars in "{}"', name)) sys.exit(1) if not requested_user_vars: continue for key, value, desc in requested_user_vars: from_env = False default_value = os.environ.get(key) if default_value: from_env = True else: if config.has_section('user_vars'): v = config.get('user_vars', key) if v: default_value = v else: default_value = value else: default_value = value if force: answers[key] = default_value else: if from_env: print(str.format( '{} will use value of {} ({}) from environment: {}', name, key, desc, default_value)) answers[key] = default_value else: prompt = '{} asks for {} ({}) ? [{}] '.format( name, key, desc, default_value) response = get_input(prompt) if response: answers[key] = response else: answers[key] = default_value if not force and answers: for key, value in answers.items(): if not config.has_section('user_vars'): config.add_section('user_vars') config.set('user_vars', key, value) config_file = os.path.join(config_dir(), 'config') with io.open(config_file, 'w', encoding=sys.stdout.encoding) as f: config.write(f) print('Saved answers to config file: {}'.format(config_file)) manager.user_vars = answers def print_error(*args, **kwargs): print(*args, file=sys.stderr, **kwargs) def config_items(config, section): # Same as config.items(section), but exclude default keys. defaults = {key for key, _ in config.items('DEFAULT')} items = sorted(config.items(section)) return [(key, value) for (key, value) in items if key not in defaults] def file_is_not_empty(path): return os.path.isfile(path) and os.path.getsize(path) > 0 def find_configfile(): configfile = os.environ.get('BRO_PKG_CONFIG_FILE') if configfile and file_is_not_empty(configfile): return configfile configfile = os.path.join(config_dir(), 'config') if file_is_not_empty(configfile): return configfile return None def config_dir(): return os.path.join(os.path.expanduser('~'), '.bro-pkg') def create_config(configfile): config = configparser.SafeConfigParser() if configfile: if not os.path.isfile(configfile): print_error('error: invalid config file "{}"'.format(configfile)) sys.exit(1) config.read(configfile) if not config.has_section('sources'): config.add_section('sources') if not config.has_section('paths'): config.add_section('paths') if not configfile: config.set('sources', 'bro', 'https://github.com/bro/packages') def config_option_set(config, section, option): return config.has_option(section, option) and config.get(section, option) def get_option(config, section, option, default): if config_option_set(config, section, option): return config.get(section, option) return default state_dir = get_option(config, 'paths', 'state_dir', os.path.join(config_dir())) script_dir = get_option(config, 'paths', 'script_dir', os.path.join(state_dir, 'script_dir')) plugin_dir = get_option(config, 'paths', 'plugin_dir', os.path.join(state_dir, 'plugin_dir')) bro_dist = get_option(config, 'paths', 'bro_dist', '') config.set('paths', 'state_dir', state_dir) config.set('paths', 'script_dir', script_dir) config.set('paths', 'plugin_dir', plugin_dir) config.set('paths', 'bro_dist', bro_dist) def expand_config_values(config, section): for key, value in config.items(section): value = os.path.expandvars(os.path.expanduser(value)) config.set(section, key, value) expand_config_values(config, 'sources') expand_config_values(config, 'paths') for key, value in config.items('paths'): if value and not os.path.isabs(value): print_error(str.format('error: invalid config file value for key' ' "{}" in section [paths]: "{}" is not' ' an absolute path', key, value)) sys.exit(1) return config def active_git_branch(path): try: repo = git.Repo(path) except git.exc.NoSuchPathError as error: return None if not repo.working_tree_dir: return None rval = repo.active_branch if not rval: return None rval = str(rval) return rval def is_git_repo_dirty(git_url): if not git_url.startswith('.') and not git_url.startswith('/'): return False try: repo = git.Repo(git_url) except git.exc.NoSuchPathError as error: return False return repo.is_dirty(untracked_files=True) def create_manager(config): state_dir = config.get('paths', 'state_dir') script_dir = config.get('paths', 'script_dir') plugin_dir = config.get('paths', 'plugin_dir') bro_dist = config.get('paths', 'bro_dist') try: manager = bropkg.Manager(state_dir=state_dir, script_dir=script_dir, plugin_dir=plugin_dir, bro_dist=bro_dist) except (OSError, IOError) as error: if error.errno == errno.EACCES: print_error('{}: {}'.format(type(error).__name__, error)) def check_permission(d): if os.access(d, os.W_OK): return print_error( 'error: user does not have write access in {}'.format(d)) check_permission(state_dir) check_permission(script_dir) check_permission(plugin_dir) sys.exit(1) raise for key, value in config_items(config, 'sources'): error = manager.add_source(name=key, git_url=value) if error: print_error(str.format( 'warning: skipped using package source named "{}": {}', key, error)) return manager class InstallWorker(threading.Thread): def __init__(self, manager, package_name, package_version): super(InstallWorker, self).__init__() self.manager = manager self.package_name = package_name self.package_version = package_version self.error = '' def run(self): self.error = self.manager.install( self.package_name, self.package_version) def cmd_test(manager, args, config): if args.version and len(args.package) > 1: print_error( 'error: "install --version" may only be used for a single package') sys.exit(1) package_infos = [] for name in args.package: if is_git_repo_dirty(name): print_error('error: local git clone at {} is dirty'.format(name)) sys.exit(1) version = args.version if args.version else active_git_branch(name) package_info = manager.info(name, version=version, prefer_installed=False) if package_info.invalid_reason: print_error(str.format('error: invalid package "{}": {}', name, package_info.invalid_reason)) sys.exit(1) if not version: version = package_info.best_version() package_infos.append((package_info, version)) all_passed = True for info, version in package_infos: name = info.package.qualified_name() if 'test_command' not in info.metadata: print(str.format('{}: no test_command found in metadata, skipping', name)) continue error_msg, passed, test_dir = manager.test(name, version) if error_msg: all_passed = False print_error(str.format('error: failed to run tests for "{}": {}', name, error_msg)) continue if passed: print(str.format('{}: all tests passed', name)) else: all_passed = False print_error(str.format('error: package "{}" tests failed, inspect' ' contents of {} for details.', name, test_dir)) if not all_passed: sys.exit(1) def cmd_install(manager, args, config): if args.version and len(args.package) > 1: print_error( 'error: "install --version" may only be used for a single package') sys.exit(1) package_infos = [] for name in args.package: if is_git_repo_dirty(name): print_error('error: local git clone at {} is dirty'.format(name)) sys.exit(1) version = args.version if args.version else active_git_branch(name) package_info = manager.info(name, version=version, prefer_installed=False) if package_info.invalid_reason: print_error(str.format('error: invalid package "{}": {}', name, package_info.invalid_reason)) sys.exit(1) if not version: version = package_info.best_version() package_infos.append((package_info, version, False)) new_pkgs = [] if not args.nodeps: to_validate = [(info.package.qualified_name(), version) for info, version, _ in package_infos] invalid_reason, new_pkgs = manager.validate_dependencies( to_validate, ignore_suggestions=args.nosuggestions) if invalid_reason: print_error('error: failed to resolve dependencies:', invalid_reason) sys.exit(1) if not args.force: package_listing = '' for info, version, _ in package_infos: name = info.package.qualified_name() package_listing += ' {} ({})\n'.format(name, version) print('The following packages will be INSTALLED:') print(package_listing) if new_pkgs: dependency_listing = '' for info, version, suggested in new_pkgs: name = info.package.qualified_name() dependency_listing += ' {} ({})'.format( name, version) if suggested: dependency_listing += ' (suggested)' dependency_listing += '\n' print('The following dependencies will be INSTALLED:') print(dependency_listing) allpkgs = package_infos + new_pkgs extdep_listing = '' for info, version, _ in allpkgs: name = info.package.qualified_name() extdeps = info.dependencies(field='external_depends') if extdeps is None: extdep_listing += ' from {} ({}):\n \n'.format( name, version) continue if extdeps: extdep_listing += ' from {} ({}):\n'.format(name, version) for extdep, semver in sorted(extdeps.items()): extdep_listing += ' {} {}\n'.format(extdep, semver) if extdep_listing: print('Verify the following REQUIRED external dependencies:\n' '(Ensure their installation on all relevant systems before' ' proceeding):') print(extdep_listing) if not confirmation_prompt('Proceed?'): return package_infos += new_pkgs prompt_for_user_vars(manager, config, args.force, [info for info, _, _ in package_infos]) if not args.skiptests: for info, version, _ in package_infos: name = info.package.qualified_name() if 'test_command' not in info.metadata: bropkg.LOG.info( 'Skipping unit tests for "%s": no test_command in metadata', name) continue print('Running unit tests for "{}"'.format(name)) error, passed, test_dir = manager.test(name, version) error_msg = '' if error: error_msg = str.format( 'failed to run tests for {}: {}', name, error) elif not passed: error_msg = str.format( '{} tests failed, inspect contents of {} for details', name, test_dir) if error_msg: print_error('error: {}'.format(error_msg)) if args.force: continue if not confirmation_prompt('Proceed to install anyway?', default_to_yes=False): return join_timeout = 0.01 tick_interval = 1 for info, version, _ in package_infos: name = info.package.qualified_name() time_accumulator = 0 tick_count = 0 is_overwriting = False ipkg = manager.find_installed_package(name) if ipkg: is_overwriting = True modifications = manager.modified_config_files(ipkg) backup_files = manager.backup_modified_files(name, modifications) prev_upstream_config_files = manager.save_temporary_config_files( ipkg) worker = InstallWorker(manager, name, version) worker.start() while worker.isAlive(): worker.join(join_timeout) time_accumulator += join_timeout if time_accumulator >= tick_interval: if tick_count == 0: print('Installing "{}"'.format(name), end='') else: print('.', end='') sys.stdout.flush() tick_count += 1 time_accumulator -= tick_interval if tick_count != 0: print('') if worker.error: print('Failed installing "{}": {}'.format(name, worker.error)) continue ipkg = manager.find_installed_package(name) print('Installed "{}" ({})'.format(name, ipkg.status.current_version)) if is_overwriting: for i, mf in enumerate(modifications): next_upstream_config_file = mf[1] if not os.path.isfile(next_upstream_config_file): print("\tConfig file no longer exists:") print("\t\t" + next_upstream_config_file) print("\tPrevious, locally modified version backed up to:") print("\t\t" + backup_files[i]) continue prev_upstream_config_file = prev_upstream_config_files[i][1] if filecmp.cmp(prev_upstream_config_file, next_upstream_config_file): # Safe to restore user's version shutil.copy2(backup_files[i], next_upstream_config_file) continue print("\tConfig file has been overwritten with a different version:") print("\t\t" + next_upstream_config_file) print("\tPrevious, locally modified version backed up to:") print("\t\t" + backup_files[i]) if manager.has_scripts(ipkg): load_error = manager.load(name) if load_error: print('Failed loading "{}": {}'.format(name, load_error)) else: print('Loaded "{}"'.format(name)) def cmd_bundle(manager, args, config): packages_to_bundle = [] prefer_existing_clones = False if args.manifest: manifest_is_file = False if len(args.manifest) == 1: config = configparser.SafeConfigParser(delimiters='=') config.optionxform = str if config.read(args.manifest[0]) and config.has_section('bundle'): packages = config.items('bundle') manifest_is_file = True if not manifest_is_file: packages = [(name, '') for name in args.manifest] to_validate = [] new_pkgs = [] for name, version in packages: if is_git_repo_dirty(name): print_error('error: local git clone at {} is dirty'.format(name)) sys.exit(1) if not version: version = active_git_branch(name) info = manager.info(name, version=version, prefer_installed=False) if info.invalid_reason: print_error(str.format('error: invalid package "{}": {}', name, info.invalid_reason)) sys.exit(1) if not version: version = info.best_version() to_validate.append((info.package.qualified_name(), version)) packages_to_bundle.append((info.package.qualified_name(), info.package.git_url, version, False, False)) if not args.nodeps: invalid_reason, new_pkgs = manager.validate_dependencies( to_validate, True, ignore_suggestions=args.nosuggestions) if invalid_reason: print_error('error: failed to resolve dependencies:', invalid_reason) sys.exit(1) for info, version, suggested in new_pkgs: packages_to_bundle.append((info.package.qualified_name(), info.package.git_url, version, True, suggested)) else: prefer_existing_clones = True for ipkg in manager.installed_packages(): packages_to_bundle.append((ipkg.package.qualified_name(), ipkg.package.git_url, ipkg.status.current_version, False, False)) if not packages_to_bundle: print_error('error: no packages to put in bundle') sys.exit(1) if not args.force: package_listing = '' for name, _, version, is_dependency, is_suggestion in packages_to_bundle: package_listing += ' {} ({})'.format(name, version) if is_suggestion: package_listing += ' (suggested)' elif is_dependency: package_listing += ' (dependency)' package_listing += '\n' print('The following packages will be BUNDLED into {}:'.format( args.bundle_filename)) print(package_listing) if not confirmation_prompt('Proceed?'): return git_urls = [(git_url, version) for _, git_url, version, _, _ in packages_to_bundle] error = manager.bundle(args.bundle_filename, git_urls, prefer_existing_clones=prefer_existing_clones) if error: print_error('error: failed to create bundle: {}'.format(error)) sys.exit(1) print('Bundle successfully written: {}'.format(args.bundle_filename)) def cmd_unbundle(manager, args, config): prev_load_status = {} for ipkg in manager.installed_packages(): prev_load_status[ipkg.package.git_url] = ipkg.status.is_loaded if args.replace: cmd_purge(manager, args, config) error, bundle_info = manager.bundle_info(args.bundle_filename) if error: print_error('error: failed to unbundle {}: {}'.format( args.bundle_filename, error)) sys.exit(1) for git_url, version, pkg_info in bundle_info: if pkg_info.invalid_reason: name = pkg_info.package.qualified_name() print_error('error: bundle {} contains invalid package {}: {}'.format( args.bundle_filename, name, pkg_info.invalid_reason)) sys.exit(1) if not bundle_info: print('No packages in bundle.') return if not args.force: package_listing = '' for git_url, version, _ in bundle_info: name = git_url for pkg in manager.source_packages(): if pkg.git_url == git_url: name = pkg.qualified_name() break package_listing += ' {} ({})\n'.format(name, version) print('The following packages will be INSTALLED:') print(package_listing) extdep_listing = '' for git_url, version, info in bundle_info: name = git_url for pkg in manager.source_packages(): if pkg.git_url == git_url: name = pkg.qualified_name() break extdeps = info.dependencies(field='external_depends') if extdeps is None: extdep_listing += ' from {} ({}):\n \n'.format( name, version) continue if extdeps: extdep_listing += ' from {} ({}):\n'.format(name, version) for extdep, semver in sorted(extdeps.items()): extdep_listing += ' {} {}\n'.format(extdep, semver) if extdep_listing: print('Verify the following REQUIRED external dependencies:\n' '(Ensure their installation on all relevant systems before' ' proceeding):') print(extdep_listing) if not confirmation_prompt('Proceed?'): return prompt_for_user_vars(manager, config, args.force, [info for _, _, info in bundle_info]) error = manager.unbundle(args.bundle_filename) if error: print_error('error: failed to unbundle {}: {}'.format( args.bundle_filename, error)) sys.exit(1) for git_url, _, _ in bundle_info: if git_url in prev_load_status: need_load = prev_load_status[git_url] else: need_load = True ipkg = manager.find_installed_package(git_url) if not ipkg: print('Skipped loading "{}": failed to install'.format(git_url)) continue name = ipkg.package.qualified_name() if not need_load: print('Skipped loading "{}"'.format(name)) continue load_error = manager.load(name) if load_error: print('Failed loading "{}": {}'.format(name, load_error)) else: print('Loaded "{}"'.format(name)) print('Unbundling complete.') def cmd_remove(manager, args, config): packages_to_remove = [] for name in args.package: ipkg = manager.find_installed_package(name) if not ipkg: print_error( 'error: package "{}" is not installed'.format(name)) sys.exit(1) packages_to_remove.append(ipkg) if not args.force: package_listing = '' for ipkg in packages_to_remove: name = ipkg.package.qualified_name() package_listing += ' {}\n'.format(name) print('The following packages will be REMOVED:') print(package_listing) if not confirmation_prompt('Proceed?'): return for ipkg in packages_to_remove: name = ipkg.package.qualified_name() modifications = manager.modified_config_files(ipkg) backup_files = manager.backup_modified_files(name, modifications) if manager.remove(name): print('Removed "{}"'.format(name)) if backup_files: print('\tCreated backups of locally modified config files:') for backup_file in backup_files: print('\t' + backup_file) else: print('Failed removing "{}": no such package installed'.format(name)) def cmd_purge(manager, args, config): packages_to_remove = manager.installed_packages() if not packages_to_remove: print('No packages to remove.') return if not args.force: package_listing = '' names_to_remove = [ipkg.package.qualified_name() for ipkg in packages_to_remove] for name in names_to_remove: package_listing += ' {}\n'.format(name) print('The following packages will be REMOVED:') print(package_listing) if not confirmation_prompt('Proceed?'): return for ipkg in packages_to_remove: name = ipkg.package.qualified_name() modifications = manager.modified_config_files(ipkg) backup_files = manager.backup_modified_files(name, modifications) if manager.remove(name): print('Removed "{}"'.format(name)) if backup_files: print('\tCreated backups of locally modified config files:') for backup_file in backup_files: print('\t' + backup_file) else: print('Unknown error removing "{}"'.format(name)) def outdated(manager): return [ipkg.package.qualified_name() for ipkg in manager.installed_packages() if ipkg.status.is_outdated] def cmd_refresh(manager, args, config): if not args.sources: args.sources = list(manager.sources.keys()) for source in args.sources: print('Refresh package source: {}'.format(source)) src_pkgs_before = {i.qualified_name() for i in manager.source_packages()} error = manager.refresh_source( source, aggregate=args.aggregate, push=args.push) if error: print_error( 'error: failed to refresh "{}": {}'.format(source, error)) continue src_pkgs_after = {i.qualified_name() for i in manager.source_packages()} if src_pkgs_before == src_pkgs_after: print('\tNo changes') else: print('\tChanges:') diff = src_pkgs_before.symmetric_difference(src_pkgs_after) for name in diff: change = 'Added' if name in src_pkgs_after else 'Removed' print('\t\t{} {}'.format(change, name)) if args.aggregate: print('\tMetadata aggregated') if args.push: print('\tPushed aggregated metadata') outdated_before = {i for i in outdated(manager)} print('Refresh installed packages') manager.refresh_installed_packages() outdated_after = {i for i in outdated(manager)} if outdated_before == outdated_after: print('\tNo new outdated packages') else: print('\tNew outdated packages:') diff = outdated_before.symmetric_difference(outdated_after) for name in diff: ipkg = manager.find_installed_package(name) version_change = version_change_string(manager, ipkg) print('\t\t{} {}'.format(name, version_change)) def version_change_string(manager, installed_package): old_version = installed_package.status.current_version new_version = old_version version_change = '' if installed_package.status.tracking_method == 'version': versions = manager.package_versions(installed_package) if len(versions): new_version = versions[-1] version_change = '({} -> {})'.format(old_version, new_version) else: version_change = '({})'.format(new_version) return version_change def cmd_upgrade(manager, args, config): if args.package: pkg_list = args.package else: pkg_list = outdated(manager) outdated_packages = [] package_listing = '' for name in pkg_list: ipkg = manager.find_installed_package(name) if not ipkg: print_error('error: package "{}" is not installed'.format(name)) sys.exit(1) name = ipkg.package.qualified_name() if not ipkg.status.is_outdated: continue info = manager.info(name, version=ipkg.status.current_version, prefer_installed=False) if info.invalid_reason: print_error(str.format('error: invalid package "{}": {}', name, info.invalid_reason)) sys.exit(1) next_version = ipkg.status.current_version if ipkg.status.tracking_method == 'version' and info.versions: next_version = info.versions[-1] outdated_packages.append((info, next_version, False)) version_change = version_change_string(manager, ipkg) package_listing += ' {} {}\n'.format(name, version_change) if not outdated_packages: print('All packages already up-to-date.') return new_pkgs = [] if not args.nodeps: to_validate = [(info.package.qualified_name(), next_version) for info, next_version, _ in outdated_packages] invalid_reason, new_pkgs = manager.validate_dependencies( to_validate, ignore_suggestions=args.nosuggestions) if invalid_reason: print_error('error: failed to resolve dependencies:', invalid_reason) sys.exit(1) allpkgs = outdated_packages + new_pkgs if not args.force: print('The following packages will be UPGRADED:') print(package_listing) if new_pkgs: dependency_listing = '' for info, version, suggestion in new_pkgs: name = info.package.qualified_name() dependency_listing += ' {} ({})'.format(name, version) if suggestion: dependency_listing += ' (suggested)' dependency_listing += '\n' print('The following dependencies will be INSTALLED:') print(dependency_listing) extdep_listing = '' for info, version, _ in allpkgs: name = info.package.qualified_name() extdeps = info.dependencies(field='external_depends') if extdeps is None: extdep_listing += ' from {} ({}):\n \n'.format( name, version) continue if extdeps: extdep_listing += ' from {} ({}):\n'.format(name, version) for extdep, semver in sorted(extdeps.items()): extdep_listing += ' {} {}\n'.format(extdep, semver) if extdep_listing: print('Verify the following REQUIRED external dependencies:\n' '(Ensure their installation on all relevant systems before' ' proceeding):') print(extdep_listing) if not confirmation_prompt('Proceed?'): return prompt_for_user_vars(manager, config, args.force, [info for info, _, _ in allpkgs]) if not args.skiptests: to_test = [(info, next_version) for info, next_version, _ in outdated_packages] for info, version, _ in new_pkgs: to_test.append((info, version)) for info, version in to_test: name = info.package.qualified_name() if 'test_command' not in info.metadata: bropkg.LOG.info( 'Skipping unit tests for "%s": no test_command in metadata', name) continue print('Running unit tests for "{}"'.format(name)) error, passed, test_dir = manager.test(name, version) error_msg = '' if error: error_msg = str.format( 'failed to run tests for {}: {}', name, error) elif not passed: error_msg = str.format( '{} tests failed, inspect contents of {} for details', name, test_dir) if error_msg: print_error('error: {}'.format(error_msg)) if args.force: continue if not confirmation_prompt('Proceed to install anyway?', default_to_yes=False): return join_timeout = 0.01 tick_interval = 1 for info, version, _ in new_pkgs: name = info.package.qualified_name() time_accumulator = 0 tick_count = 0 worker = InstallWorker(manager, name, version) worker.start() while worker.isAlive(): worker.join(join_timeout) time_accumulator += join_timeout if time_accumulator >= tick_interval: if tick_count == 0: print('Installing "{}"'.format(name), end='') else: print('.', end='') sys.stdout.flush() tick_count += 1 time_accumulator -= tick_interval if tick_count != 0: print('') if worker.error: print('Failed installing "{}": {}'.format(name, worker.error)) continue ipkg = manager.find_installed_package(name) print('Installed "{}" ({})'.format(name, ipkg.status.current_version)) if manager.has_scripts(ipkg): load_error = manager.load(name) if load_error: print('Failed loading "{}": {}'.format(name, load_error)) else: print('Loaded "{}"'.format(name)) for info, next_version, _ in outdated_packages: name = info.package.qualified_name() ipkg = manager.find_installed_package(name) modifications = manager.modified_config_files(ipkg) backup_files = manager.backup_modified_files(name, modifications) prev_upstream_config_files = manager.save_temporary_config_files(ipkg) res = manager.upgrade(name) if res: print('Failed upgrading "{}": {}'.format(name, res)) else: ipkg = manager.find_installed_package(name) print('Upgraded "{}" ({})'.format( name, ipkg.status.current_version)) for i, mf in enumerate(modifications): next_upstream_config_file = mf[1] if not os.path.isfile(next_upstream_config_file): print("\tConfig file no longer exists:") print("\t\t" + next_upstream_config_file) print("\tPrevious, locally modified version backed up to:") print("\t\t" + backup_files[i]) continue prev_upstream_config_file = prev_upstream_config_files[i][1] if filecmp.cmp(prev_upstream_config_file, next_upstream_config_file): # Safe to restore user's version shutil.copy2(backup_files[i], next_upstream_config_file) continue print("\tConfig file has been updated to a newer version:") print("\t\t" + next_upstream_config_file) print("\tPrevious, locally modified version backed up to:") print("\t\t" + backup_files[i]) def cmd_load(manager, args, config): for name in args.package: ipkg = manager.find_installed_package(name) if not ipkg: print('Failed to load "{}": no such package installed'.format(name)) continue if not manager.has_scripts(ipkg): print('The package "{}" does not contain scripts to load.'.format(name)) continue name = ipkg.package.qualified_name() load_error = manager.load(name) if load_error: print('Failed to load "{}": {}'.format(name, load_error)) else: print('Loaded "{}"'.format(name)) def cmd_unload(manager, args, config): for name in args.package: ipkg = manager.find_installed_package(name) if not ipkg: print('Failed to unload "{}": no such package installed'.format(name)) continue if not manager.has_scripts(ipkg): print('The package "{}" does not contain scripts to unload.'.format(name)) continue name = ipkg.package.qualified_name() if manager.unload(name): print('Unloaded "{}"'.format(name)) else: print( 'Failed unloading "{}": no such package installed'.format(name)) def cmd_pin(manager, args, config): for name in args.package: ipkg = manager.find_installed_package(name) if not ipkg: print('Failed to pin "{}": no such package installed'.format(name)) continue name = ipkg.package.qualified_name() ipkg = manager.pin(name) if ipkg: print('Pinned "{}" at version: {} ({})'.format( name, ipkg.status.current_version, ipkg.status.current_hash)) else: print('Failed pinning "{}": no such package installed'.format(name)) def cmd_unpin(manager, args, config): for name in args.package: ipkg = manager.find_installed_package(name) if not ipkg: print('Failed to unpin "{}": no such package installed'.format(name)) continue name = ipkg.package.qualified_name() ipkg = manager.unpin(name) if ipkg: print('Unpinned "{}" from version: {} ({})'.format( name, ipkg.status.current_version, ipkg.status.current_hash)) else: print( 'Failed unpinning "{}": no such package installed'.format(name)) def cmd_list(manager, args, config): pkg_dict = dict() for ipkg in manager.installed_packages(): pkg_dict[ipkg.package.qualified_name()] = ipkg for pkg in manager.source_packages(): pkg_qn = pkg.qualified_name() if pkg_qn not in pkg_dict: pkg_dict[pkg_qn] = pkg if args.category == 'all': filtered_pkgs = pkg_dict elif args.category == 'installed': filtered_pkgs = {key: value for key, value in pkg_dict.items() if isinstance(value, bropkg.InstalledPackage)} elif args.category == 'not_installed': filtered_pkgs = {key: value for key, value in pkg_dict.items() if not isinstance(value, bropkg.InstalledPackage)} elif args.category == 'loaded': filtered_pkgs = {key: value for key, value in pkg_dict.items() if isinstance(value, bropkg.InstalledPackage) and value.status.is_loaded} elif args.category == 'unloaded': filtered_pkgs = {key: value for key, value in pkg_dict.items() if isinstance(value, bropkg.InstalledPackage) and not value.status.is_loaded} elif args.category == 'outdated': filtered_pkgs = {key: value for key, value in pkg_dict.items() if isinstance(value, bropkg.InstalledPackage) and value.status.is_outdated} else: raise NotImplementedError for pkg_name, val in sorted(filtered_pkgs.items()): if isinstance(val, bropkg.InstalledPackage): pkg = val.package out = '{} (installed: {})'.format( pkg_name, val.status.current_version) else: pkg = val out = pkg_name if not args.nodesc: desc = pkg.short_description() if desc: out += ' - ' + desc print(out) def cmd_search(manager, args, config): src_pkgs = manager.source_packages() matches = set() for search_text in args.search_text: if search_text[0] == '/' and search_text[-1] == '/': import re try: regex = re.compile(search_text[1:-1]) except re.error as error: print('invalid regex: {}'.format(error)) sys.exit(1) else: for pkg in src_pkgs: if regex.search(pkg.name_with_source_directory()): matches.add(pkg) for tag in pkg.tags(): if regex.search(tag): matches.add(pkg) else: for pkg in src_pkgs: if search_text in pkg.name_with_source_directory(): matches.add(pkg) for tag in pkg.tags(): if search_text in tag: matches.add(pkg) if matches: for match in sorted(matches): out = match.qualified_name() ipkg = manager.find_installed_package(match.qualified_name()) if ipkg: out += ' (installed: {})'.format(ipkg.status.current_version) desc = match.short_description() if desc: out += ' - ' + desc print(out) else: print("no matches") def cmd_info(manager, args, config): if args.version and len(args.package) > 1: print_error( 'error: "info --version" may only be used for a single package') sys.exit(1) # Dictionary for storing package info to output as JSON pkginfo = dict() for name in args.package: info = manager.info(name, version=args.version, prefer_installed=(args.nolocal != True)) if info.package: name = info.package.qualified_name() if args.json: pkginfo[name] = dict() pkginfo[name]['metadata'] = dict() else: print('"{}" info:'.format(name)) if info.invalid_reason: if args.json: pkginfo[name]['invalid'] = info.invalid_reason else: print('\tinvalid package: {}\n'.format(info.invalid_reason)) continue if args.json: pkginfo[name]['url'] = info.package.git_url pkginfo[name]['versions'] = info.versions else: print('\turl: {}'.format(info.package.git_url)) print('\tversions: {}'.format(info.versions)) if info.status: if args.json: pkginfo[name]['install_status'] = dict() for key, value in sorted(info.status.__dict__.items()): pkginfo[name]['install_status'][key] = value else: print('\tinstall status:') for key, value in sorted(info.status.__dict__.items()): print('\t\t{} = {}'.format(key, value)) if args.json: pkginfo[name]['metadata'][info.metadata_version] = dict() else: print('\tmetadata (from version "{}"):'.format( info.metadata_version)) if len(info.metadata) == 0: if args.json != True: print('\t\t') else: if args.json: _fill_metadata_version( pkginfo[name]['metadata'][info.metadata_version], info.metadata) else: for key, value in sorted(info.metadata.items()): value = value.replace('\n', '\n\t\t\t') print('\t\t{} = {}'.format(key, value)) # If --json and --allvers given, check for multiple versions and # add the metadata for each version to the pkginfo. if args.json and args.allvers: for vers in info.versions: # Skip the version that was already processed if vers != info.metadata_version: info2 = manager.info(name, vers, prefer_installed=(args.nolocal != True)) pkginfo[name]['metadata'][info2.metadata_version] = dict() _fill_metadata_version( pkginfo[name]['metadata'][info2.metadata_version], info2.metadata) if not args.json: print() if args.json: print(json.dumps(pkginfo, indent=args.jsonpretty, sort_keys=True)) def _fill_metadata_version(pkginfo_name_metadata_version, info_metadata): """ Fill a dict with metadata information. This helper function is called by cmd_info to fill metadata information for a specific package version. Args: pkginfo_name_metadata_version (dict of str -> dict): Corresponds to pkginfo[name]['metadata'][info.metadata_version] in cmd_info. info_metadata (dict of str->str): Corresponds to info.metadata in cmd_info. Side effect: New dict entries are added to pkginfo_name_metadata_version. """ for key, value in info_metadata.items(): if key == 'depends' or key == 'suggests': pkginfo_name_metadata_version[key] = dict() deps = value.split('\n') for i in range(1, len(deps)): deplist = deps[i].split(' ') pkginfo_name_metadata_version[key][deplist[0]] =\ deplist[1] else: pkginfo_name_metadata_version[key] = value def cmd_config(manager, args, config): if args.config_param == 'all': if sys.version_info[0] < 3: from StringIO import StringIO else: from io import StringIO out = StringIO() config.write(out) print(out.getvalue()) out.close() elif args.config_param == 'sources': for key, value in config_items(config, 'sources'): print('{} = {}'.format(key, value)) elif args.config_param == 'user_vars': if config.has_section('user_vars'): for key, value in config_items(config, 'user_vars'): print('{} = {}'.format(key, value)) else: print(config.get('paths', args.config_param)) def cmd_autoconfig(manager, args, config): bro_config = find_program('bro-config') if not bro_config: print_error('error: "bro-config" not found in PATH') sys.exit(1) cmd = subprocess.Popen([bro_config, '--site_dir', '--plugin_dir', '--bro_dist'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, universal_newlines=True) script_dir = read_bro_config_line(cmd.stdout) plugin_dir = read_bro_config_line(cmd.stdout) bro_dist = read_bro_config_line(cmd.stdout) make_dir(config_dir()) config_file = os.path.join(config_dir(), 'config') config_file_exists = os.path.isfile(config_file) def change_config_value(config, section, option, new_value, use_prompt): if not use_prompt: config.set(section, option, new_value) return old_value = config.get(section, option) if old_value == new_value: return prompt = u'Set "{}" config option to: {} ?'.format(option, new_value) if old_value: prompt += u'\n(previous value: {})'.format(old_value) if confirmation_prompt(prompt): config.set(section, option, new_value) change_config_value(config, 'paths', 'script_dir', script_dir, config_file_exists) change_config_value(config, 'paths', 'plugin_dir', plugin_dir, config_file_exists) change_config_value(config, 'paths', 'bro_dist', bro_dist, config_file_exists) with io.open(config_file, 'w', encoding=sys.stdout.encoding) as f: config.write(f) print('Successfully wrote config file to {}'.format(config_file)) def cmd_env(manager, args, config): bro_config = find_program('bro-config') bropath = os.environ.get('BROPATH') pluginpath = os.environ.get('BRO_PLUGIN_PATH') if bro_config: cmd = subprocess.Popen([bro_config, '--bropath', '--plugin_dir'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, universal_newlines=True) line1 = read_bro_config_line(cmd.stdout) line2 = read_bro_config_line(cmd.stdout) if not bropath: bropath = line1 if not pluginpath: pluginpath = line2 bropaths = [p for p in bropath.split(':')] if bropath else [] pluginpaths = [p for p in pluginpath.split(':')] if pluginpath else [] bropaths.append(manager.bropath()) pluginpaths.append(manager.bro_plugin_path()) if os.environ['SHELL'].endswith('csh'): print(u'setenv BROPATH {}'.format(':'.join(bropaths))) print(u'setenv BRO_PLUGIN_PATH {}'.format(':'.join(pluginpaths))) else: print(u'export BROPATH={}'.format(':'.join(bropaths))) print(u'export BRO_PLUGIN_PATH={}'.format(':'.join(pluginpaths))) def top_level_parser(): top_parser = argparse.ArgumentParser( formatter_class=argparse.RawDescriptionHelpFormatter, description='A command-line package manager for Bro.', epilog='Environment Variables:\n\n' ' ``BRO_PKG_CONFIG_FILE``:\t' 'Same as ``--configfile`` option, but has less precedence.' ) top_parser.add_argument('--version', action='version', version='%(prog)s ' + bropkg.__version__) top_parser.add_argument('--configfile', help='Path to Bro Package Manager config file.') top_parser.add_argument('--verbose', '-v', action='count', default=0, help='Increase program output for debugging.' ' Use multiple times for more output (e.g. -vvv).') return top_parser def argparser(): pkg_name_help = 'The name(s) of package(s) to operate on. The package' \ ' may be named in several ways. If the package is part' \ ' of a package source, it may be referred to by the' \ ' base name of the package (last component of git URL)' \ ' or its path within the package source.' \ ' If two packages in different package sources' \ ' have conflicting paths, then the package source' \ ' name may be prepended to the package path to resolve' \ ' the ambiguity. A full git URL may also be used to refer' \ ' to a package that does not belong to a source. E.g. for' \ ' a package source called "bro" that has a package named' \ ' "foo" located in "alice/bro-pkg.index" the following' \ ' names work: "foo", "alice/foo", "bro/alice/foo".' top_parser = top_level_parser() command_parser = top_parser.add_subparsers( title='commands', dest='command', help='See `%(prog)s -h` for per-command usage info.') command_parser.required = True # test sub_parser = command_parser.add_parser( 'test', help='Runs unit tests for Bro packages.', description='Runs the unit tests for the specified Bro packages.' ' In most cases, the "bro" and "bro-config" programs will' ' need to be in PATH before running this command.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_test) sub_parser.add_argument( 'package', nargs='+', help=pkg_name_help) sub_parser.add_argument( '--version', default=None, help='The version of the package to test. Only one package may be' ' specified at a time when using this flag. A version tag or branch' ' name may be specified here.' ' If the package name refers to a local git repo with a working tree,' ' then its currently active branch is used.' ' The default for other cases is to use' ' the latest version tag, or if a package has none, the "master"' ' branch.') # install sub_parser = command_parser.add_parser( 'install', help='Installs Bro packages.', description='Installs packages from a configured package source or' ' directly from a git URL. After installing, the package' ' is marked as being "loaded" (see the ``load`` command).', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_install) sub_parser.add_argument( 'package', nargs='+', help=pkg_name_help) sub_parser.add_argument( '--force', action='store_true', help='Skip the confirmation prompt.') sub_parser.add_argument( '--skiptests', action='store_true', help='Skip running unit tests for packages before installation.') sub_parser.add_argument( '--nodeps', action='store_true', help='Skip all dependency resolution/checks. Note that using this' ' option risks putting your installed package collection into a ' ' broken or unusable state.') sub_parser.add_argument( '--nosuggestions', action='store_true', help='Skip automatically installing suggested packages.') sub_parser.add_argument( '--version', default=None, help='The version of the package to install. Only one package may be' ' specified at a time when using this flag. A version tag or branch' ' name may be specified here.' ' If the package name refers to a local git repo with a working tree,' ' then its currently active branch is used.' ' The default for other cases is to use' ' the latest version tag, or if a package has none, the "master"' ' branch.') # bundle sub_parser = command_parser.add_parser( 'bundle', help='Creates a bundle file containing a collection of Bro packages.', description='This command creates a bundle file containing a collection' ' of Bro packages. If ``--manifest`` is used, the user' ' suplies the list of packages to put in the bundle, else' ' all currently installed packages are put in the bundle.' ' A bundle file can be unpacked on any target system,' ' resulting in a repeatable/specific set of packages' ' being installed on that target system (see the' ' ``unbundle`` command). This command may be useful for' ' those that want to manage packages on a system that' ' otherwise has limited network connectivity. E.g. one can' ' use a system with an internet connection to create a' ' bundle, transport that bundle to the target machine' ' using whatever means are appropriate, and finally' ' unbundle/install it on the target machine.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_bundle) sub_parser.add_argument( 'bundle_filename', help='The path of the bundle file to create.' ' It will be overwritten if it already exists.') sub_parser.add_argument( '--force', action='store_true', help='Skip the confirmation prompt.') sub_parser.add_argument( '--nodeps', action='store_true', help='Skip all dependency resolution/checks. Note that using this' ' option risks creating a bundle of packages that is in a ' ' broken or unusable state.') sub_parser.add_argument( '--nosuggestions', action='store_true', help='Skip automatically bundling suggested packages.') sub_parser.add_argument( '--manifest', nargs='+', help='This may either be a file name or a list of packages to include' ' in the bundle. If a file name is supplied, it should be in INI' ' format with a single ``[bundle]`` section. The keys in that section' ' correspond to package names and their values correspond to git' ' version tags or branch names. The values may be left blank to' ' indicate that the latest available version should be used.') # unbundle sub_parser = command_parser.add_parser( 'unbundle', help='Unpacks Bro packages from a bundle file and installs them.', description='This command unpacks a bundle file formerly created by the' ' ``bundle`` command and installs all the packages' ' contained within.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_unbundle) sub_parser.add_argument( 'bundle_filename', help='The path of the bundle file to install.') sub_parser.add_argument( '--force', action='store_true', help='Skip the confirmation prompt.') sub_parser.add_argument( '--replace', action='store_true', help='Using this flag first removes all installed packages before then' ' installing the packages from the bundle.') # remove sub_parser = command_parser.add_parser( 'remove', help='Uninstall a package.', description='Unloads (see the ``unload`` command) and uninstalls a' ' previously installed package.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_remove) sub_parser.add_argument('package', nargs='+', help=pkg_name_help) sub_parser.add_argument( '--force', action='store_true', help='Skip the confirmation prompt.') # purge sub_parser = command_parser.add_parser( 'purge', help='Uninstall all packages.', description='Unloads (see the ``unload`` command) and uninstalls all' ' previously installed packages.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_purge) sub_parser.add_argument( '--force', action='store_true', help='Skip the confirmation prompt.') # refresh sub_parser = command_parser.add_parser( 'refresh', help='Retrieve updated package metadata.', description='Retrieve latest package metadata from sources and checks' ' whether any installed packages have available upgrades.' ' Note that this does not actually upgrade any packages (see the' ' ``upgrade`` command for that).', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_refresh) sub_parser.add_argument( '--aggregate', action='store_true', help='Crawls the urls listed in package source bro-pkg.index files and' ' aggregates the metadata found in their bro-pkg.meta files. The' ' aggregated metadata is stored in the local clone of the package' ' source that bro-pkg uses internally locating package metadata.' ' For each package, the metadata is taken from the highest available' ' git version tag or the master branch if no version tags exist') sub_parser.add_argument( '--push', action='store_true', help='Push all local changes to package sources to upstream repos') sub_parser.add_argument('--sources', nargs='+', help='A list of package source names to operate on. If this argument' ' is not used, then the command will operate on all configured' ' sources.') # upgrade sub_parser = command_parser.add_parser( 'upgrade', help='Upgrade installed packages to latest versions.', description='Uprades the specified package(s) to latest available' ' version. If no specific packages are specified, then all installed' ' packages that are outdated and not pinned are upgraded. For packages' ' that are installed with ``--version`` using a git branch name, the' ' package is updated to the latest commit on that branch, else the' ' package is updated to the highest available git version tag.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_upgrade) sub_parser.add_argument( 'package', nargs='*', default=[], help=pkg_name_help) sub_parser.add_argument( '--force', action='store_true', help='Skip the confirmation prompt.') sub_parser.add_argument( '--skiptests', action='store_true', help='Skip running unit tests for packages before installation.') sub_parser.add_argument( '--nodeps', action='store_true', help='Skip all dependency resolution/checks. Note that using this' ' option risks putting your installed package collection into a ' ' broken or unusable state.') sub_parser.add_argument( '--nosuggestions', action='store_true', help='Skip automatically installing suggested packages.') # load sub_parser = command_parser.add_parser( 'load', help='Register packages to be be auto-loaded by Bro.', description='The Bro Package Manager keeps track of all packages that' ' are marked as "loaded" and maintains a single Bro script that, when' ' loaded by Bro (e.g. via ``@load packages``), will load the scripts' ' from all "loaded" packages at once.' ' This command adds a set of packages to the "loaded packages" list.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_load) sub_parser.add_argument( 'package', nargs='+', default=[], help='Name(s) of package(s) to load.') # unload sub_parser = command_parser.add_parser( 'unload', help='Unregister packages to be be auto-loaded by Bro.', description='The Bro Package Manager keeps track of all packages that' ' are marked as "loaded" and maintains a single Bro script that, when' ' loaded by Bro, will load the scripts from all "loaded" packages at' ' once. This command removes a set of packages from the "loaded' ' packages" list.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_unload) sub_parser.add_argument( 'package', nargs='+', default=[], help=pkg_name_help) # pin sub_parser = command_parser.add_parser( 'pin', help='Prevent packages from being automatically upgraded.', description='Pinned packages are ignored by the ``upgrade`` command.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_pin) sub_parser.add_argument( 'package', nargs='+', default=[], help=pkg_name_help) # unpin sub_parser = command_parser.add_parser( 'unpin', help='Allows packages to be automatically upgraded.', description='Packages that are not pinned are automatically upgraded' ' by the ``upgrade`` command', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_unpin) sub_parser.add_argument( 'package', nargs='+', default=[], help=pkg_name_help) # list sub_parser = command_parser.add_parser( 'list', help='Lists packages.', description='Outputs a list of packages that match a given category.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_list) sub_parser.add_argument('category', nargs='?', default='installed', choices=['all', 'installed', 'not_installed', 'loaded', 'unloaded', 'outdated'], help='Package category used to filter listing.') sub_parser.add_argument( '--nodesc', action='store_true', help='Do not display description text, just the package name(s).') # search sub_parser = command_parser.add_parser( 'search', help='Search packages for matching names.', description='Perform a substring search on package names and metadata' ' tags. Surround search text with slashes to indicate it is a regular' ' expression (e.g. ``/text/``).', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_search) sub_parser.add_argument( 'search_text', nargs='+', default=[], help='The text(s) or pattern(s) to look for.') # info sub_parser = command_parser.add_parser( 'info', help='Display package information.', description='Shows detailed information/metadata for given packages.' ' If the package is currently installed, additional information about' ' the status of it is displayed. E.g. the installed version or whether' ' it is currently marked as "pinned" or "loaded."', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_info) sub_parser.add_argument( 'package', nargs='+', default=[], help=pkg_name_help) sub_parser.add_argument( '--version', default=None, help='The version of the package metadata to inspect. A version tag,' ' branch name, or commit hash and only one package at a time may be' ' given when using this flag. If unspecified, the behavior depends' ' on whether the package is currently installed. If installed,' ' the metadata will be pulled from the installed version. If not' ' installed, the latest version tag is used, or if a package has no' ' version tags, the "master" branch is used.') sub_parser.add_argument( '--nolocal', action='store_true', help='Do not read information from locally installed packages.' ' Instead read info from remote GitHub.') sub_parser.add_argument( '--json', action='store_true', help='Output package information as JSON.') sub_parser.add_argument( '--jsonpretty', type=int, default=None, metavar='SPACES', help='Optional number of spaces to indent for pretty-printed' ' JSON output.') sub_parser.add_argument( '--allvers', action='store_true', help='When outputting package information as JSON, show metadata for' ' all versions. This option can be slow since remote repositories' ' may be cloned multiple times. Also, installed packages will show' ' metadata only for the installed version unless the --nolocal ' ' option is given.') # config sub_parser = command_parser.add_parser( 'config', help='Show Bro Package Manager configuration info.', description='The default output of this command is a valid package' ' manager config file that corresponds to the one currently being used,' ' but also with any defaulted field values filled in. This command' ' also allows for only the value of a specific field to be output if' ' the name of that field is given as an argument to the command.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_config) sub_parser.add_argument( 'config_param', nargs='?', default='all', choices=['all', 'sources', 'user_vars', 'state_dir', 'script_dir', 'plugin_dir', 'bro_dist'], help='Name of a specific config file field to output.') # autoconfig sub_parser = command_parser.add_parser( 'autoconfig', help='Generate a Bro Package Manager configuration file.', description='The output of this command is a valid package manager' ' config file that is generated by using the ``bro-config`` script' ' that is installed along with Bro. It is the suggested configuration' ' to use for most Bro installations. For this command to work, the' ' ``bro-config`` script must be in ``PATH``.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_autoconfig) # env sub_parser = command_parser.add_parser( 'env', help='Show the value of environment variables that need to be set for' ' Bro to be able to use installed packages.', description='This command returns shell commands that, when executed,' ' will correctly set ``BROPATH`` and ``BRO_PLUGIN_PATH`` to utilize the' ' scripts and plugins from packages installed by the package manager.' ' For this command to function properly, either have the ``bro-config``' ' script (installed by bro) in ``PATH``, or have the ``BROPATH`` and' ' ``BRO_PLUGIN_PATH`` environment variables already set so this command' ' can append package-specific paths to them.', formatter_class=argparse.ArgumentDefaultsHelpFormatter) sub_parser.set_defaults(run_cmd=cmd_env) return top_parser def main(): args = argparser().parse_args() if args.verbose > 0: formatter = logging.Formatter( '%(asctime)s %(levelname)-8s %(message)s', '%Y-%m-%d %H:%M:%S') handler = logging.StreamHandler() handler.setFormatter(formatter) if args.verbose == 1: bropkg.LOG.setLevel(logging.WARNING) elif args.verbose == 2: bropkg.LOG.setLevel(logging.INFO) elif args.verbose == 3: bropkg.LOG.setLevel(logging.DEBUG) bropkg.LOG.addHandler(handler) configfile = args.configfile if not configfile: configfile = find_configfile() config = create_config(configfile) manager = create_manager(config) args.run_cmd(manager, args, config) if __name__ == '__main__': main() sys.exit(0) package-manager-1.3.3/.update-changes.cfg0000644000175000017500000000065013241343432017516 0ustar rhaistrhaist# Automatically adapt version in files. function replace_python_package_version { file=$1 version=$2 cat $file | sed "s#^\\( *__version__ *= *\\)\"\\([0-9.-]\\{1,\\}\\)\"#\1\"$version\"#g" >$file.tmp cat $file.tmp >$file rm -f $file.tmp git add $file } function new_version_hook { version=$1 replace_python_package_version bropkg/__init__.py $version make man git add ./doc/man } package-manager-1.3.3/Makefile0000644000175000017500000000220313241343432015523 0ustar rhaistrhaistVERSION=`cat VERSION` .PHONY: all all: .PHONY: doc doc: man html .PHONY: man man: (cd doc && make man && mkdir -p man && cp _build/man/bro-pkg.1 man) .PHONY: html html: (cd doc && make html) .PHONY: livehtml livehtml: (cd doc && make livehtml) .PHONY: gh-pages gh-pages: (cd doc && make clean html) (cd doc/_build/html && tar -czf /tmp/bro-pkg-html.tar.gz .) git checkout gh-pages git rm -rf . && git clean -fdx tar -xzf /tmp/bro-pkg-html.tar.gz && rm /tmp/bro-pkg-html.tar.gz git add . @echo "You are now in the 'gh-pages' branch." @echo "Make sure 'git status' looks ok, push, then switch back to 'master'" .PHONY: test test: @( cd testing && make ) .PHONY: upload upload: twine-check BROPKG_PYPI_DIST=yes python setup.py bdist_wheel twine upload -u bro dist/bro_pkg-$(VERSION)-py2.py3-none-any.whl .PHONY: twine-check twine-check: @type twine > /dev/null 2>&1 || \ { \ echo "Uploading to PyPi requires 'twine' and it's not found in PATH."; \ echo "Install it and/or make sure it is in PATH."; \ echo "E.g. you could use the following command to install it:"; \ echo "\tpip install twine"; \ echo ; \ exit 1; \ } package-manager-1.3.3/VERSION0000644000175000017500000000000613241343432015132 0ustar rhaistrhaist1.3.3 package-manager-1.3.3/requirements.doc.txt0000644000175000017500000000012713241343432020116 0ustar rhaistrhaistSphinx sphinxcontrib-napoleon GitPython semantic_version configparser sphinx_rtd_theme package-manager-1.3.3/bropkg/0000755000175000017500000000000013241343432015352 5ustar rhaistrhaistpackage-manager-1.3.3/bropkg/_util.py0000644000175000017500000000563413241343432017050 0ustar rhaistrhaist""" These are meant to be private utility methods for internal use. """ import os import sys import errno import shutil import git def make_dir(path): """Create a directory or do nothing if it already exists. Raises: OSError: if directory cannot be created """ try: os.makedirs(path) except OSError as exception: if exception.errno != errno.EEXIST: raise elif os.path.isfile(path): raise def remove_trailing_slashes(path): if path.endswith('/'): return remove_trailing_slashes(path[:-1]) return path def delete_path(path): if os.path.islink(path): os.remove(path) return if not os.path.exists(path): return if os.path.isdir(path): shutil.rmtree(path) else: os.remove(path) def copy_over_path(src, dst, ignore=None): delete_path(dst) shutil.copytree(src, dst, symlinks=True, ignore=ignore) def make_symlink(target_path, link_path, force=True): try: os.symlink(target_path, link_path) except OSError as error: if error.errno == errno.EEXIST and force and os.path.islink(link_path): os.remove(link_path) os.symlink(target_path, link_path) else: raise error def find_sentence_end(s): beg = 0 while True: period_idx = s.find('.', beg) if period_idx == -1: return -1 if period_idx == len(s) - 1: return period_idx next_char = s[period_idx + 1] if next_char.isspace(): return period_idx beg = period_idx + 1 def git_clone_shallow(git_url, dst_path): git.Git().clone(git_url, dst_path, '--no-single-branch', depth=1) rval = git.Repo(dst_path) rval.git.fetch(tags=True) return rval def is_exe(path): return os.path.isfile(path) and os.access(path, os.X_OK) def find_program(prog_name): path, _ = os.path.split(prog_name) if path: return prog_name if is_exe(prog_name) else '' for path in os.environ["PATH"].split(os.pathsep): path = os.path.join(path.strip('"'), prog_name) if is_exe(path): return path return '' def stdout_encoding(): if sys.stdout.encoding: return sys.stdout.encoding import locale return locale.getpreferredencoding() def read_bro_config_line(stdout): rval = stdout.readline() # Python 2 returned bytes, Python 3 returned unicode if isinstance(rval, bytes): rval = rval.decode(stdout_encoding()) return rval.strip() def get_bro_version(): bro_config = find_program('bro-config') if not bro_config: return '' import subprocess cmd = subprocess.Popen([bro_config, '--version'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, universal_newlines=True) return read_bro_config_line(cmd.stdout) package-manager-1.3.3/bropkg/manager.py0000644000175000017500000025615113241343432017350 0ustar rhaistrhaist""" A module defining the main Bro Package Manager interface which supplies methods to interact with and operate on Bro packages. """ import os import copy import json import shutil import filecmp import tarfile import subprocess try: from backports import configparser except ImportError as err: import configparser import git import semantic_version as semver from ._util import ( make_dir, delete_path, make_symlink, copy_over_path, git_clone_shallow, get_bro_version, stdout_encoding, find_program, read_bro_config_line, ) from .source import ( AGGREGATE_DATA_FILE, Source ) from .package import ( METADATA_FILENAME, name_from_path, user_vars, canonical_url, Package, PackageInfo, PackageStatus, InstalledPackage ) from . import ( __version__, LOG, ) class Manager(object): """A package manager object performs various operations on packages. It uses a state directory and a manifest file within it to keep track of package sources, installed packages and their statuses. Attributes: sources (dict of str -> :class:`.source.Source`): dictionary package sources keyed by the name given to :meth:`add_source()` installed_pkgs (dict of str -> :class:`.package.InstalledPackage`): a dictionary of installed packaged keyed on package names (the last component of the package's git URL) bro_dist (str): path to the Bro source code distribution. This is needed for packages that contain Bro plugins that need to be built from source code. state_dir (str): the directory where the package manager will a maintain manifest file, package/source git clones, and other persistent state the manager needs in order to operate user_vars (dict of str -> str): dictionary of key-value pairs where the value will be substituted into package build commands in place of the key. backup_dir (str): a directory where the package manager will store backup files (e.g. locally modified package config files) log_dir (str): a directory where the package manager will store misc. logs files (e.g. package build logs) scratch_dir (str): a directory where the package manager performs miscellaneous/temporary file operations script_dir (str): the directory where the package manager will copy each installed package's `script_dir` (as given by its :file:`bro-pkg.meta` file). Each package gets a subdirectory within `script_dir` associated with its name. plugin_dir (str): the directory where the package manager will copy each installed package's `plugin_dir` (as given by its :file:`bro-pkg.meta` file). Each package gets a subdirectory within `plugin_dir` associated with its name. source_clonedir (str): the directory where the package manager will clone package sources. Each source gets a subdirectory associated with its name. package_clonedir (str): the directory where the package manager will clone installed packages. Each package gets a subdirectory associated with its name. package_testdir (str): the directory where the package manager will run tests. Each package gets a subdirectory associated with its name. manifest (str): the path to the package manager's manifest file. This file maintains a list of installed packages and their status. autoload_script (str): path to a Bro script named :file:`packages.bro` that the package manager maintains. It is a list of ``@load`` for each installed package that is marked as loaded (see :meth:`load()`). autoload_package (str): path to a Bro :file:`__load__.bro` script which is just a symlink to `autoload_script`. It's always located in a directory named :file:`packages`, so as long as :envvar:`BROPATH` is configured correctly, ``@load packages`` will load all installed packages that have been marked as loaded. """ def __init__(self, state_dir, script_dir, plugin_dir, bro_dist='', user_vars=None): """Creates a package manager instance. Args: state_dir (str): value to set the `state_dir` attribute to script_dir (str): value to set the `script_dir` attribute to plugin_dir (str): value to set the `plugin_dir` attribute to bro_dist (str): value to set the `bro_dist` attribute to user_vars (dict of str -> str): key-value pair substitutions for use in package build commands. Raises: OSError: when a package manager state directory can't be created IOError: when a package manager state file can't be created """ LOG.debug('init Manager version %s', __version__) self.sources = {} self.installed_pkgs = {} self.bro_dist = bro_dist self.state_dir = state_dir self.user_vars = {} if user_vars is None else user_vars self.backup_dir = os.path.join(self.state_dir, 'backups') self.log_dir = os.path.join(self.state_dir, 'logs') self.scratch_dir = os.path.join(self.state_dir, 'scratch') self._script_dir = script_dir self.script_dir = os.path.join(script_dir, 'packages') self._plugin_dir = plugin_dir self.plugin_dir = os.path.join(plugin_dir, 'packages') self.source_clonedir = os.path.join(self.state_dir, 'clones', 'source') self.package_clonedir = os.path.join( self.state_dir, 'clones', 'package') self.package_testdir = os.path.join(self.state_dir, 'testing') self.manifest = os.path.join(self.state_dir, 'manifest.json') self.autoload_script = os.path.join(self.script_dir, 'packages.bro') self.autoload_package = os.path.join(self.script_dir, '__load__.bro') make_dir(self.state_dir) make_dir(self.log_dir) make_dir(self.scratch_dir) make_dir(self.source_clonedir) make_dir(self.package_clonedir) make_dir(self.script_dir) make_dir(self.plugin_dir) _create_readme(os.path.join(self.script_dir, 'README')) _create_readme(os.path.join(self.plugin_dir, 'README')) if not os.path.exists(self.manifest): self._write_manifest() prev_script_dir, prev_plugin_dir = self._read_manifest() if prev_script_dir != self.script_dir: LOG.info('relocating script_dir %s -> %s', prev_script_dir, self.script_dir) if os.path.exists(prev_script_dir): delete_path(self.script_dir) shutil.move(prev_script_dir, self.script_dir) prev_bropath = os.path.dirname(prev_script_dir) for pkg_name in self.installed_pkgs: old_link = os.path.join(prev_bropath, pkg_name) new_link = os.path.join(self.bropath(), pkg_name) if os.path.lexists(old_link): LOG.info('moving package link %s -> %s', old_link, new_link) shutil.move(old_link, new_link) else: LOG.info('skip moving package link %s -> %s', old_link, new_link) self._write_manifest() if prev_plugin_dir != self.plugin_dir: LOG.info('relocating plugin_dir %s -> %s', prev_plugin_dir, self.plugin_dir) if os.path.exists(prev_plugin_dir): delete_path(self.plugin_dir) shutil.move(prev_plugin_dir, self.plugin_dir) self._write_manifest() self._write_autoloader() make_symlink('packages.bro', self.autoload_package) def _write_autoloader(self): """Write the :file:`__load__.bro` loader script. Raises: IOError: if :file:`__load__.bro` loader script cannot be written """ with open(self.autoload_script, 'w') as f: content = ('# WARNING: This file is managed by bro-pkg.\n' '# Do not make direct modifications here.\n') for ipkg in self.loaded_packages(): content += '@load ./{}\n'.format(ipkg.package.name) f.write(content) def _read_manifest(self): """Read the manifest file containing the list of installed packages. Returns: tuple: (previous script_dir, previous plugin_dir) Raises: IOError: when the manifest file can't be read """ with open(self.manifest, 'r') as f: data = json.load(f) version = data['manifest_version'] pkg_list = data['installed_packages'] self.installed_pkgs = {} for dicts in pkg_list: pkg_dict = dicts['package_dict'] status_dict = dicts['status_dict'] pkg_name = pkg_dict['name'] del pkg_dict['name'] if version == 0 and 'index_data' in pkg_dict: del pkg_dict['index_data'] pkg = Package(**pkg_dict) status = PackageStatus(**status_dict) self.installed_pkgs[pkg_name] = InstalledPackage(pkg, status) return data['script_dir'], data['plugin_dir'] def _write_manifest(self): """Writes the manifest file containing the list of installed packages. Raises: IOError: when the manifest file can't be written """ pkg_list = [] for _, installed_pkg in self.installed_pkgs.items(): pkg_list.append({'package_dict': installed_pkg.package.__dict__, 'status_dict': installed_pkg.status.__dict__}) data = {'manifest_version': 1, 'script_dir': self.script_dir, 'plugin_dir': self.plugin_dir, 'installed_packages': pkg_list} with open(self.manifest, 'w') as f: json.dump(data, f, indent=2, sort_keys=True) def bropath(self): """Return the path where installed package scripts are located. This path can be added to :envvar:`BROPATH` for interoperability with Bro. """ return os.path.dirname(self.script_dir) def bro_plugin_path(self): """Return the path where installed package plugins are located. This path can be added to :envvar:`BRO_PLUGIN_PATH` for interoperability with Bro. """ return os.path.dirname(self.plugin_dir) def add_source(self, name, git_url): """Add a git repository that acts as a source of packages. Args: name (str): a short name that will be used to reference the package source. git_url (str): the git URL of the package source Returns: str: empty string if the source is successfully added, else the reason why it failed. """ if name in self.sources: existing_source = self.sources[name] if existing_source.git_url == git_url: LOG.debug('duplicate source "%s"', name) return True return 'source already exists with different URL: {}'.format( existing_source.git_url) clone_path = os.path.join(self.source_clonedir, name) try: source = Source(name=name, clone_path=clone_path, git_url=git_url) except git.exc.GitCommandError as error: LOG.warning('failed to clone git repo: %s', error) return 'failed to clone git repo' else: self.sources[name] = source return '' def source_packages(self): """Return a list of :class:`.package.Package` within all sources.""" rval = [] for _, source in self.sources.items(): rval += source.packages() return rval def installed_packages(self): """Return list of :class:`.package.InstalledPackage`.""" return [ipkg for _, ipkg in sorted(self.installed_pkgs.items())] def loaded_packages(self): """Return list of loaded :class:`.package.InstalledPackage`.""" rval = [] for _, ipkg in sorted(self.installed_pkgs.items()): if ipkg.status.is_loaded: rval.append(ipkg) return rval def package_build_log(self, pkg_path): """Return the path to the package manager's build log for a package. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". """ name = name_from_path(pkg_path) return os.path.join(self.log_dir, '{}-build.log'.format(name)) def match_source_packages(self, pkg_path): """Return a list of :class:`.package.Package` that match a given path. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". """ rval = [] canon_url = canonical_url(pkg_path) for pkg in self.source_packages(): if pkg.matches_path(canon_url): rval.append(pkg) return rval def find_installed_package(self, pkg_path): """Return an :class:`.package.InstalledPackage` if one matches the name. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". A package's name is the last component of it's git URL. """ pkg_name = name_from_path(pkg_path) return self.installed_pkgs.get(pkg_name) def has_scripts(self, installed_pkg): """Return whether a :class:`.package.InstalledPackage` installed scripts. Args: installed_pkg(:class:`.package.InstalledPackage`): the installed package to check for whether it has installed any Bro scripts. Returns: bool: True if the package has installed Bro scripts. """ return os.path.exists(os.path.join(self.script_dir, installed_pkg.package.name)) def save_temporary_config_files(self, installed_pkg): """Return a list of temporary package config file backups. Args: installed_pkg(:class:`.package.InstalledPackage`): the installed package to save temporary config file backups for. Returns: list of (str, str): tuples that describe the config files backups. The first element is the config file as specified in the package metadata (a file path relative to the package's root directory). The second element is an absolute file system path to where that config file has been copied. It should be considered temporary, so make use of it before doing any further operations on packages. """ import re metadata = installed_pkg.package.metadata config_files = re.split(',\s*', metadata.get('config_files', '')) if not config_files: return [] pkg_name = installed_pkg.package.name clone_dir = os.path.join(self.package_clonedir, pkg_name) rval = [] for config_file in config_files: config_file_path = os.path.join(clone_dir, config_file) if not os.path.isfile(config_file_path): LOG.info("package '%s' claims config file at '%s'," " but it does not exist", pkg_name, config_file) continue backup_file = os.path.join(self.scratch_dir, 'tmpcfg', config_file) make_dir(os.path.dirname(backup_file)) shutil.copy2(config_file_path, backup_file) rval.append((config_file, backup_file)) return rval def modified_config_files(self, installed_pkg): """Return a list of package config files that the user has modified. Args: installed_pkg(:class:`.package.InstalledPackage`): the installed package to check for whether it has installed any Bro scripts. Returns: list of (str, str): tuples that describe the modified config files. The first element is the config file as specified in the package metadata (a file path relative to the package's root directory). The second element is an absolute file system path to where that config file is currently installed. """ import re metadata = installed_pkg.package.metadata config_files = re.split(',\s*', metadata.get('config_files', '')) if not config_files: return [] pkg_name = installed_pkg.package.name script_install_dir = os.path.join(self.script_dir, pkg_name) plugin_install_dir = os.path.join(self.plugin_dir, pkg_name) clone_dir = os.path.join(self.package_clonedir, pkg_name) script_dir = metadata.get('script_dir', '') plugin_dir = metadata.get('plugin_dir', 'build') rval = [] for config_file in config_files: their_config_file_path = os.path.join(clone_dir, config_file) if not os.path.isfile(their_config_file_path): LOG.info("package '%s' claims config file at '%s'," " but it does not exist", pkg_name, config_file) continue if config_file.startswith(plugin_dir): our_config_file_path = os.path.join( plugin_install_dir, config_file[len(plugin_dir):]) if not os.path.isfile(our_config_file_path): LOG.info("package '%s' config file '%s' not found" " in plugin_dir: %s", pkg_name, config_file, our_config_file_path) continue elif config_file.startswith(script_dir): our_config_file_path = os.path.join( script_install_dir, config_file[len(script_dir):]) if not os.path.isfile(our_config_file_path): LOG.info("package '%s' config file '%s' not found" " in script_dir: %s", pkg_name, config_file, our_config_file_path) continue else: # Their config file is outside script/plugin install dirs, # so no way user has it even installed, much less modified. LOG.warning("package '%s' config file '%s' not within" " plugin_dir or script_dir", pkg_name, config_file) continue if not filecmp.cmp(our_config_file_path, their_config_file_path): rval.append((config_file, our_config_file_path)) return rval def backup_modified_files(self, backup_subdir, modified_files): """Creates backups of modified config files Args: modified_files(list of (str, str)): the return value of :meth:`modified_config_files()`. backup_subdir(str): the subdir of `backup_dir` in which Returns: list of str: paths indicating the backup locations. The order of the returned list corresponds directly to the order of `modified_files`. """ import time rval = [] for modified_file in modified_files: config_file = modified_file[0] config_file_dir = os.path.dirname(config_file) install_path = modified_file[1] filename = os.path.basename(install_path) backup_dir = os.path.join( self.backup_dir, backup_subdir, config_file_dir) timestamp = time.strftime('.%Y-%m-%d-%H:%M:%S') backup_path = os.path.join(backup_dir, filename + timestamp) make_dir(backup_dir) shutil.copy2(install_path, backup_path) rval.append(backup_path) return rval def refresh_source(self, name, aggregate=False, push=False): """Pull latest git information from a package source. This makes the latest pre-aggregated package metadata available or performs the aggregation locally in order to push it to the actual package source. Locally aggregated data also takes precedence over the source's pre-aggregated data, so it can be useful in the case the operator of the source does not update their pre-aggregated data at a frequent enough interval. Args: name(str): the name of the package source. E.g. the same name used as a key to :meth:`add_source()`. aggregate (bool): whether to perform a local metadata aggregation by crawling all packages listed in the source's index files. push (bool): whether to push local changes to the aggregated metadata to the remote package source. If the `aggregate` flag is set, the data will be pushed after the aggregation is finished. Returns: str: an empty string if no errors occurred, else a description of what went wrong. """ if name not in self.sources: return 'source name does not exist' source = self.sources[name] LOG.debug('refresh "%s": pulling %s', name, source.git_url) aggregate_file = os.path.join(source.clone.working_dir, AGGREGATE_DATA_FILE) agg_file_ours = os.path.join( self.scratch_dir, AGGREGATE_DATA_FILE) agg_file_their_orig = os.path.join(self.scratch_dir, AGGREGATE_DATA_FILE + '.orig') delete_path(agg_file_ours) delete_path(agg_file_their_orig) if os.path.isfile(aggregate_file): shutil.copy2(aggregate_file, agg_file_ours) source.clone.git.reset(hard=True) source.clone.git.clean('-f', '-x', '-d') if os.path.isfile(aggregate_file): shutil.copy2(aggregate_file, agg_file_their_orig) try: source.clone.remote().pull() except git.exc.GitCommandError as error: LOG.error('failed to pull source %s: %s', name, error) return 'failed to pull from remote source' if os.path.isfile(agg_file_ours): if os.path.isfile(aggregate_file): # There's a tracked version of the file after pull. if os.path.isfile(agg_file_their_orig): # We had local modifications to the file. if filecmp.cmp(aggregate_file, agg_file_their_orig): # Their file hasn't changed, use ours. shutil.copy2(agg_file_ours, aggregate_file) LOG.debug( "aggegrate file in source unchanged, restore local one") else: # Their file changed, use theirs. LOG.debug( "aggegrate file in source changed, discard local one") else: # File was untracked before pull and tracked after, # use their version. LOG.debug("new aggegrate file in source, discard local one") else: # They don't have the file after pulling, so restore ours. shutil.copy2(agg_file_ours, aggregate_file) LOG.debug("no aggegrate file in source, restore local one") if aggregate: parser = configparser.SafeConfigParser() for index_file in source.package_index_files(): urls = [] with open(index_file) as f: urls = [line.rstrip('\n') for line in f] for url in urls: pkg_name = name_from_path(url) clonepath = os.path.join(self.scratch_dir, pkg_name) delete_path(clonepath) try: clone = git_clone_shallow(url, clonepath) except git.exc.GitCommandError as error: LOG.warn('failed to clone %s, skipping aggregation: %s', url, error) continue version_tags = _get_version_tags(clone) if len(version_tags): version = version_tags[-1] else: version = 'master' clone.git.checkout(version) metadata_file = os.path.join( clone.working_dir, METADATA_FILENAME) # Use raw parser so no value interpolation takes place. metadata_parser = configparser.RawConfigParser() invalid_reason = _parse_package_metadata( metadata_parser, metadata_file) if invalid_reason: LOG.warn('skipping aggregation of %s: bad metadata: %s', url, invalid_reason) continue metadata = _get_package_metadata(metadata_parser) index_dir = os.path.dirname(index_file)[len( self.source_clonedir) + len(name) + 2:] qualified_name = os.path.join(index_dir, pkg_name) parser.add_section(qualified_name) for key, value in sorted(metadata.items()): parser.set(qualified_name, key, value) parser.set(qualified_name, 'url', url) parser.set(qualified_name, 'version', version) with open(aggregate_file, 'w') as f: parser.write(f) if push: if os.path.isfile(os.path.join(source.clone.working_dir, AGGREGATE_DATA_FILE)): source.clone.git.add(AGGREGATE_DATA_FILE) if source.clone.is_dirty(): source.clone.git.commit( '--message', 'Update aggregated metadata.') source.clone.git.push() return '' def refresh_installed_packages(self): """Fetch latest git information for installed packages. This retrieves information about outdated packages, but does not actually upgrade their installations. Raises: IOError: if the package manifest file can't be written """ for ipkg in self.installed_packages(): clonepath = os.path.join(self.package_clonedir, ipkg.package.name) clone = git.Repo(clonepath) LOG.debug('fetch package %s', ipkg.package.qualified_name()) try: clone.remote().fetch() except git.exc.GitCommandError as error: LOG.warn('failed to fetch package %s: %s', ipkg.package.qualified_name(), error) ipkg.status.is_outdated = _is_clone_outdated( clone, ipkg.status.current_version, ipkg.status.tracking_method) self._write_manifest() def upgrade(self, pkg_path): """Upgrade a package to the latest available version. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". Returns: str: an empty string if package upgrade succeeded else an error string explaining why it failed. Raises: IOError: if the manifest can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('upgrading "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('upgrading "%s": no matching package', pkg_path) return "no such package installed" if ipkg.status.is_pinned: LOG.info('upgrading "%s": package is pinned', pkg_path) return "package is pinned" if not ipkg.status.is_outdated: LOG.info('upgrading "%s": package not outdated', pkg_path) return "package is not outdated" clonepath = os.path.join(self.package_clonedir, ipkg.package.name) clone = git.Repo(clonepath) if ipkg.status.tracking_method == 'version': version_tags = _get_version_tags(clone) return self._install(ipkg.package, version_tags[-1]) elif ipkg.status.tracking_method == 'branch': clone.remote().pull() return self._install(ipkg.package, ipkg.status.current_version) else: raise NotImplementedError def remove(self, pkg_path): """Remove an installed package. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". Returns: bool: True if an installed package was removed, else False. Raises: IOError: if the package manifest file can't be written OSError: if the installed package's directory can't be deleted """ pkg_path = canonical_url(pkg_path) LOG.debug('removing "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('removing "%s": could not find matching package', pkg_path) return False self.unload(pkg_path) pkg_to_remove = ipkg.package delete_path(os.path.join(self.package_clonedir, pkg_to_remove.name)) delete_path(os.path.join(self.script_dir, pkg_to_remove.name)) delete_path(os.path.join(self.plugin_dir, pkg_to_remove.name)) delete_path(os.path.join(self.bropath(), pkg_to_remove.name)) del self.installed_pkgs[pkg_to_remove.name] self._write_manifest() LOG.debug('removed "%s"', pkg_path) return True def pin(self, pkg_path): """Pin a currently installed package to the currently installed version. Pinned packages are never upgraded when calling :meth:`upgrade()`. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". Returns: :class:`.package.InstalledPackage`: None if no matching installed package could be found, else the installed package that was pinned. Raises: IOError: when the manifest file can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('pinning "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('pinning "%s": no matching package', pkg_path) return None if ipkg.status.is_pinned: LOG.debug('pinning "%s": already pinned', pkg_path) return ipkg ipkg.status.is_pinned = True self._write_manifest() LOG.debug('pinned "%s"', pkg_path) return ipkg def unpin(self, pkg_path): """Unpin a currently installed package and allow it to be upgraded. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". Returns: :class:`.package.InstalledPackage`: None if no matching installed package could be found, else the installed package that was unpinned. Raises: IOError: when the manifest file can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('unpinning "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('unpinning "%s": no matching package', pkg_path) return None if not ipkg.status.is_pinned: LOG.debug('unpinning "%s": already unpinned', pkg_path) return ipkg ipkg.status.is_pinned = False self._write_manifest() LOG.debug('unpinned "%s"', pkg_path) return ipkg def load(self, pkg_path): """Mark an installed package as being "loaded". The collection of "loaded" packages is a convenient way for Bro to more simply load a whole group of packages installed via the package manager. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". Returns: str: empty string if the package is successfully marked as loaded, else an explanation of why it failed. Raises: IOError: if the loader script or manifest can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('loading "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('loading "%s": no matching package', pkg_path) return 'no such package' if ipkg.status.is_loaded: LOG.debug('loading "%s": already loaded', pkg_path) return '' pkg_load_script = os.path.join(self.script_dir, ipkg.package.name, '__load__.bro') if not os.path.exists(pkg_load_script): LOG.debug('loading "%s": %s does not exist', pkg_path, pkg_load_script) return 'no __load__.bro within package script_dir' ipkg.status.is_loaded = True self._write_autoloader() self._write_manifest() LOG.debug('loaded "%s"', pkg_path) return '' def unload(self, pkg_path): """Unmark an installed package as being "loaded". The collection of "loaded" packages is a convenient way for Bro to more simply load a whole group of packages installed via the package manager. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". Returns: bool: True if a package is successfully unmarked as loaded. Raises: IOError: if the loader script or manifest can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('unloading "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('unloading "%s": no matching package', pkg_path) return False if not ipkg.status.is_loaded: LOG.debug('unloading "%s": already unloaded', pkg_path) return True ipkg.status.is_loaded = False self._write_autoloader() self._write_manifest() LOG.debug('unloaded "%s"', pkg_path) return True def bundle_info(self, bundle_file): """Retrieves information on all packages contained in a bundle. Args: bundle_file (str): the path to the bundle to inspect. Returns: (str, list of (str, str, :class:`.package.PackageInfo`)): a tuple with the the first element set to an empty string if the information successfully retrieved, else an error message explaining why the bundle file was invalid. The second element of the tuple is a list containing information on each package contained in the bundle: the exact git URL and version string from the bundle's manifest along with the package info object retrieved by inspecting git repo contained in the bundle. """ bundle_dir = os.path.join(self.scratch_dir, 'bundle') delete_path(bundle_dir) make_dir(bundle_dir) infos = [] try: with tarfile.open(bundle_file) as tf: tf.extractall(bundle_dir) except Exception as error: return (str(error), infos) manifest_file = os.path.join(bundle_dir, 'manifest.txt') config = configparser.SafeConfigParser(delimiters='=') config.optionxform = str if not config.read(manifest_file): return ('invalid bundle: no manifest file', infos) if not config.has_section('bundle'): return ('invalid bundle: no [bundle] section in manifest file', infos) manifest = config.items('bundle') for git_url, version in manifest: package = Package(git_url=git_url) pkg_path = os.path.join(bundle_dir, package.name) pkg_info = self.info(pkg_path, version=version, prefer_installed=False) infos.append((git_url, version, pkg_info)) return ('', infos) def info(self, pkg_path, version='', prefer_installed=True): """Retrieves information about a package. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". version (str): may be a git version tag, branch name, or commit hash from which metadata will be pulled. If an empty string is given, then the latest git version tag is used (or the "master" branch if no version tags exist). prefer_installed (bool): if this is set, then the information from any current installation of the package is returned instead of retrieving the latest information from the package's git repo. The `version` parameter is also ignored when this is set as it uses whatever version of the package is currently installed. Returns: A :class:`.package.PackageInfo` object. """ pkg_path = canonical_url(pkg_path) LOG.debug('getting info on "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if prefer_installed and ipkg: status = ipkg.status pkg_name = ipkg.package.name clonepath = os.path.join(self.package_clonedir, pkg_name) clone = git.Repo(clonepath) return _info_from_clone(clone, ipkg.package, status, status.current_version) else: status = None matches = self.match_source_packages(pkg_path) if not matches: package = Package(git_url=pkg_path) try: return self._info(package, status, version) except git.exc.GitCommandError as error: LOG.info('getting info on "%s": invalid git repo path: %s', pkg_path, error) LOG.info('getting info on "%s": matched no source package', pkg_path) reason = 'package name not found in sources and not a valid git URL' return PackageInfo(package=package, invalid_reason=reason, status=status) if len(matches) > 1: matches_string = [match.qualified_name() for match in matches] LOG.info('getting info on "%s": matched multiple packages: %s', pkg_path, matches_string) reason = str.format('"{}" matches multiple packages, try a more' ' specific name from: {}', pkg_path, matches_string) return PackageInfo(invalid_reason=reason, status=status) package = matches[0] try: return self._info(package, status, version) except git.exc.GitCommandError as error: LOG.info('getting info on "%s": invalid git repo path: %s', pkg_path, error) reason = 'git repository is either invalid or unreachable' return PackageInfo(package=package, invalid_reason=reason, status=status) def _info(self, package, status, version): """Retrieves information about a package. Returns: A :class:`.package.PackageInfo` object. Raises: git.exc.GitCommandError: when failing to clone the package repo """ clonepath = os.path.join(self.scratch_dir, package.name) clone = _clone_package(package, clonepath) versions = _get_version_tags(clone) if not version: if len(versions): version = versions[-1] else: if 'master' not in _get_branch_names(clone): reason = 'git repo has no "master" branch or version tags' return PackageInfo(package=package, status=status, invalid_reason=reason) version = 'master' try: clone.git.checkout(version) except git.exc.GitCommandError: reason = 'no such commit, branch, or version tag: "{}"'.format( version) return PackageInfo(package=package, status=status, invalid_reason=reason) LOG.debug('checked out "%s", branch/version "%s"', package, version) return _info_from_clone(clone, package, status, version) def package_versions(self, installed_package): """Returns a list of version number tags available for a package. Args: installed_package (:class:`.package.InstalledPackage`): the package for which version number tags will be retrieved. Returns: list of str: the version number tags. """ name = installed_package.package.name clonepath = os.path.join(self.package_clonedir, name) clone = git.Repo(clonepath) return _get_version_tags(clone) def validate_dependencies(self, requested_packages, ignore_installed_packages=False, ignore_suggestions=False): """Validates package dependencies. Args: requested_packages (list of (str, str)): a list of (package name or git URL, version) string tuples validate. If the version string is empty, the latest available version of the package is used. ignore_installed_packages (bool): whether the dependency analysis should consider installed packages as satisfying dependency requirements. ignore_suggestions (bool): whether the dependency analysis should consider installing dependencies that are marked in another package's 'suggests' metadata field. Returns: (str, list of (:class:`.package.PackageInfo`, str, bool)): the first element of the tuple is an empty string if dependency graph was successfully validated, else an error string explaining what is invalid. In the case it was validated, the second element is a list of tuples where the first elements are dependency packages that would need to be installed in order to satisfy the dependencies of the requested packages (it will not include any packages that are already installed or that are in the `requested_packages` argument). The second element of tuples in the list is a version string of the associated package that satisfies dependency requirements. The third element of the tuples in the list is a boolean value indicating whether the package is included in the list because it's merely suggested by another package. """ class Node(object): def __init__(self, name): self.name = name self.info = None self.requested_version = None # (tracking method, version) self.installed_version = None # (tracking method, version) self.dependers = dict() # name -> version self.is_suggestion = False def __str__(self): return str.format( '{}\n\trequested: {}\n\tinstalled: {}\n\tdependers: {}\n\tsuggestion: {}', self.name, self.requested_version, self.installed_version, self.dependers, self.is_suggestion) new_pkgs = [] graph = dict() # 1. Try to make nodes for everything in the dependency graph... # Add nodes for packages that are requested for installation for name, version in requested_packages: info = self.info(name, version=version, prefer_installed=False) if info.invalid_reason: return ('invalid package "{}": {}'.format(name, info.invalid_reason), new_pkgs) node = Node(info.package.qualified_name()) node.info = info method = 'version' if version in node.info.versions else 'branch' node.requested_version = (method, version) graph[node.name] = node # Recursively add nodes for all dependencies of requested packages to_process = copy.copy(graph) while to_process: (_, node) = to_process.popitem() dd = node.info.dependencies(field='depends') ds = node.info.dependencies(field='suggests') if dd is None: return (str.format('package "{}" has malformed "depends" field', node.name), new_pkgs) all_deps = dd.copy() if not ignore_suggestions: if ds is None: return (str.format('package "{}" has malformed "suggests" field', node.name), new_pkgs) all_deps.update(ds) for dep_name, _ in all_deps.items(): if dep_name == 'bro': # A bro node will get added later. continue if dep_name == 'bro-pkg': # A bro-pkg node will get added later. continue # Suggestion status propagates to 'depends' field of suggested packages. is_suggestion = node.is_suggestion or dep_name in ds and dep_name not in dd info = self.info(dep_name, prefer_installed=False) if info.invalid_reason: return (str.format( 'package "{}" has invalid dependency "{}": {}', node.name, dep_name, info.invalid_reason), new_pkgs) dep_name = info.package.qualified_name() if dep_name in graph: if graph[dep_name].is_suggestion and not is_suggestion: # Suggestion found to be required by another package. graph[dep_name].is_suggestion = False continue if dep_name in to_process: if to_process[dep_name].is_suggestion and not is_suggestion: # Suggestion found to be required by another package. to_process[dep_name].is_suggestion = False continue node = Node(dep_name) node.info = info node.is_suggestion = is_suggestion graph[node.name] = node to_process[node.name] = node # Add nodes for things that are already installed (including bro) if not ignore_installed_packages: bro_version = get_bro_version() if bro_version: node = Node('bro') node.installed_version = ('version', bro_version) graph['bro'] = node else: LOG.warning( 'could not get bro version: no bro_config in PATH?') node = Node('bro-pkg') node.installed_version = ('version', __version__) graph['bro-pkg'] = node for ipkg in self.installed_packages(): name = ipkg.package.qualified_name() status = ipkg.status if name not in graph: info = self.info(name, prefer_installed=True) node = Node(name) node.info = info graph[node.name] = node graph[name].installed_version = ( status.tracking_method, status.current_version) # 2. Fill in the edges of the graph with dependency information. for name, node in graph.items(): if name == 'bro': continue if name == 'bro-pkg': continue dd = node.info.dependencies(field='depends') ds = node.info.dependencies(field='suggests') if dd is None: return (str.format('package "{}" has malformed "depends" field', node.name), new_pkgs) all_deps = dd.copy() if not ignore_suggestions: if ds is None: return (str.format('package "{}" has malformed "suggests" field', node.name), new_pkgs) all_deps.update(ds) for dep_name, dep_version in all_deps.items(): if dep_name == 'bro': if 'bro' in graph: graph['bro'].dependers[name] = dep_version elif dep_name == 'bro-pkg': if 'bro-pkg' in graph: graph['bro-pkg'].dependers[name] = dep_version else: for _, dependency_node in graph.items(): if dependency_node.name == 'bro': continue if dependency_node.name == 'bro-pkg': continue if dependency_node.info.package.matches_path(dep_name): dependency_node.dependers[name] = dep_version break # 3. Try to solve for a connected graph with no edge conflicts. for name, node in graph.items(): if not node.dependers: if node.installed_version: continue if node.requested_version: continue new_pkgs.append((node.info, node.info.best_version(), node.is_suggestion)) continue if node.requested_version: # Check that requested version doesn't conflict with dependers. track_method, required_version = node.requested_version if track_method == 'branch': for depender_name, version_spec in node.dependers.items(): if version_spec == '*': continue if version_spec.startswith('branch='): version_spec = version_spec[len('branch='):] if version_spec == required_version: continue return (str.format( 'unsatisfiable dependency: requested "{}" ({}),' ' but "{}" requires {}', node.name, required_version, depender_name, version_spec), new_pkgs) else: req_semver = semver.Version.coerce(required_version) for depender_name, version_spec in node.dependers.items(): if version_spec.startswith('branch='): version_spec = version_spec[len('branch='):] return (str.format( 'unsatisfiable dependency: requested "{}" ({}),' ' but "{}" requires {}', node.name, required_version, depender_name, version_spec), new_pkgs) else: try: semver_spec = semver.Spec(version_spec) except ValueError: return (str.format( 'package "{}" has invalid semver spec: {}', depender_name, version_spec), new_pkgs) if req_semver not in semver_spec: return (str.format( 'unsatisfiable dependency: requested "{}" ({}),' ' but "{}" requires {}', node.name, required_version, depender_name, version_spec), new_pkgs) elif node.installed_version: # Check that installed version doesn't conflict with dependers. track_method, required_version = node.installed_version if track_method == 'branch': for depender_name, version_spec in node.dependers.items(): if version_spec == '*': continue if version_spec.startswith('branch='): version_spec = version_spec[len('branch='):] if version_spec == required_version: continue return (str.format( 'unsatisfiable dependency: "{}" ({}) is installed,' ' but "{}" requires {}', node.name, required_version, depender_name, version_spec), new_pkgs) else: req_semver = semver.Version.coerce(required_version) for depender_name, version_spec in node.dependers.items(): if version_spec.startswith('branch='): version_spec = version_spec[len('branch='):] return (str.format( 'unsatisfiable dependency: "{}" ({}) is installed,' ' but "{}" requires {}', node.name, required_version, depender_name, version_spec), new_pkgs) else: try: semver_spec = semver.Spec(version_spec) except ValueError: return (str.format( 'package "{}" has invalid semver spec: {}', depender_name, version_spec), new_pkgs) if req_semver not in semver_spec: return (str.format( 'unsatisfiable dependency: "{}" ({}) is installed,' ' but "{}" requires {}', node.name, required_version, depender_name, version_spec), new_pkgs) else: # Choose best version that satisfies constraints if not node.info.versions: best_version = 'master' for depender_name, version_spec in node.dependers.items(): if version_spec == '*': continue if version_spec.startswith('branch='): version_spec = version_spec[len('branch='):] if version_spec == best_version: continue return (str.format( 'unsatisfiable dependency "{}": "{}" requires {}', node.name, depender_name, version_spec), new_pkgs) else: best_version = None need_branch = False need_version = False def no_best_version_string(node): rval = str.format( '"{}" has no version satisfying dependencies:\n', node.name) for depender_name, version_spec in node.dependers.items(): rval += str.format('\t{} needs {}\n', depender_name, version_spec) return (rval, new_pkgs) for _, version_spec in node.dependers.items(): if version_spec.startswith('branch='): need_branch = True elif version_spec != '*': need_version = True if need_branch and need_version: return (no_best_version_string(node), new_pkgs) if need_branch: branch_name = None for depender_name, version_spec in node.dependers.items(): if version_spec == '*': continue if not branch_name: branch_name = version_spec[len('branch='):] continue if branch_name != version_spec[len('branch='):]: return (no_best_version_string(), new_pkgs) if branch_name: best_version = branch_name else: best_version = 'master' elif need_version: for version in node.info.versions[::-1]: req_semver = semver.Version.coerce(version) satisfied = True for depender_name, version_spec in node.dependers.items(): try: semver_spec = semver.Spec(version_spec) except ValueError: return (str.format( 'package "{}" has invalid semver spec: {}', depender_name, version_spec), new_pkgs) if req_semver not in semver_spec: satisfied = False break if satisfied: best_version = version break if not best_version: return (no_best_version_string(node), new_pkgs) else: # Must have been all '*' wildcards best_version = node.info.best_version() new_pkgs.append((node.info, best_version, node.is_suggestion)) return ('', new_pkgs) def bundle(self, bundle_file, package_list, prefer_existing_clones=False): """Creates a package bundle. Args: bundle_file (str): filesystem path of the zip file to create. package_list (list of (str, str)): a list of (git URL, version) string tuples to put in the bundle. If the version string is empty, the latest available version of the package is used. prefer_existing_clones (bool): if True and the package list contains a package at a version that is already installed, then the existing git clone of that package is put into the bundle instead of cloning from the remote repository. Returns: str: empty string if the bundle is successfully created, else an error string explaining what failed. """ bundle_dir = os.path.join(self.scratch_dir, 'bundle') delete_path(bundle_dir) make_dir(bundle_dir) manifest_file = os.path.join(bundle_dir, 'manifest.txt') config = configparser.SafeConfigParser(delimiters='=') config.optionxform = str config.add_section('bundle') def match_package_url_and_version(git_url, version): for ipkg in self.installed_packages(): if ipkg.package.git_url != git_url: continue if ipkg.status.current_version != version: continue return ipkg return None for git_url, version in package_list: name = name_from_path(git_url) clonepath = os.path.join(bundle_dir, name) config.set('bundle', git_url, version) if prefer_existing_clones: ipkg = match_package_url_and_version(git_url, version) if ipkg: src = os.path.join( self.package_clonedir, ipkg.package.name) shutil.copytree(src, clonepath, symlinks=True) clone = git.Repo(clonepath) clone.git.reset(hard=True) clone.git.clean('-f', '-x', '-d') for modified_config in self.modified_config_files(ipkg): dst = os.path.join(clonepath, modified_config[0]) shutil.copy2(modified_config[1], dst) continue try: git_clone_shallow(git_url, clonepath) except git.exc.GitCommandError as error: return 'failed to clone {}: {}'.format(git_url, error) with open(manifest_file, 'w') as f: config.write(f) archive = shutil.make_archive(bundle_dir, 'gztar', bundle_dir) delete_path(bundle_file) shutil.move(archive, bundle_file) return '' def unbundle(self, bundle_file): """Installs all packages contained within a bundle. Args: bundle_file (str): the path to the bundle to install. Returns: str: an empty string if the operation was successful, else an error message indicated what went wrong. """ bundle_dir = os.path.join(self.scratch_dir, 'bundle') delete_path(bundle_dir) make_dir(bundle_dir) try: with tarfile.open(bundle_file) as tf: tf.extractall(bundle_dir) except Exception as error: return str(error) manifest_file = os.path.join(bundle_dir, 'manifest.txt') config = configparser.SafeConfigParser(delimiters='=') config.optionxform = str if not config.read(manifest_file): return 'invalid bundle: no manifest file' if not config.has_section('bundle'): return 'invalid bundle: no [bundle] section in manifest file' manifest = config.items('bundle') for git_url, version in manifest: package = Package(git_url=git_url) clonepath = os.path.join(self.package_clonedir, package.name) delete_path(clonepath) shutil.move(os.path.join(bundle_dir, package.name), clonepath) error = self._install(package, version, use_existing_clone=True) if error: return error return '' def test(self, pkg_path, version=''): """Test a package. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". version (str): if not given, then the latest git version tag is used (or if no version tags exist, the "master" branch is used). If given, it may be either a git version tag or a git branch name. Returns: (str, bool, str): a tuple containing an error message string, a boolean indicating whether the tests passed, as well as a path to the directory in which the tests were run. In the case where tests failed, the directory can be inspected to figure out what went wrong. In the case where the error message string is not empty, the error message indicates the reason why tests could not be run. """ pkg_path = canonical_url(pkg_path) LOG.debug('testing "%s"', pkg_path) pkg_info = self.info(pkg_path, version=version, prefer_installed=False) if pkg_info.invalid_reason: return (pkg_info.invalid_reason, 'False', '') if 'test_command' not in pkg_info.metadata: return ('Package does not specify a test_command', False, '') if not version: version = pkg_info.metadata_version package = pkg_info.package test_dir = os.path.join(self.package_testdir, package.name) clone_dir = os.path.join(test_dir, 'clones') stage_script_dir = os.path.join(test_dir, 'scripts', 'packages') stage_plugin_dir = os.path.join(test_dir, 'plugins', 'packages') delete_path(test_dir) make_dir(clone_dir) make_dir(stage_script_dir) make_dir(stage_plugin_dir) request = [(package.qualified_name(), version)] invalid_deps, new_pkgs = self.validate_dependencies(request, True) if invalid_deps: return (invalid_deps, False, test_dir) pkgs = [] pkgs.append((pkg_info, version)) for info, version, _ in new_pkgs: pkgs.append((info, version)) # Clone all packages, checkout right version, and build/install to # staging area. for info, version in pkgs: clonepath = os.path.join(clone_dir, info.package.name) try: clone = _clone_package(info.package, clonepath) except git.exc.GitCommandError as error: LOG.warning('failed to clone git repo: %s', error) return ('failed to clone {}'.format(info.package.git_url), False, test_dir) try: clone.git.checkout(version) except git.exc.GitCommandError as error: LOG.warning('failed to checkout git repo version: %s', error) return (str.format('failed to checkout {} of {}', version, info.package.git_url), False, test_dir) fail_msg = self._stage(info.package, version, clone, stage_script_dir, stage_plugin_dir) if fail_msg: return (fail_msg, False, test_dir) # Finally, run tests (with correct environment set) test_command = pkg_info.metadata['test_command'] LOG.debug('running test_command for %s: %s', package.name, test_command) bro_config = find_program('bro-config') bropath = os.environ.get('BROPATH') pluginpath = os.environ.get('BRO_PLUGIN_PATH') if bro_config: cmd = subprocess.Popen([bro_config, '--bropath', '--plugin_dir'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, universal_newlines=True) line1 = read_bro_config_line(cmd.stdout) line2 = read_bro_config_line(cmd.stdout) if not bropath: bropath = line1 if not pluginpath: pluginpath = line2 else: LOG.warning('bro-config not found when running tests for %s', package.name) return ('bro-config not found in PATH', False, test_dir) bropath = os.path.dirname(stage_script_dir) + ':' + bropath pluginpath = os.path.dirname(stage_plugin_dir) + ':' + pluginpath env = os.environ.copy() env['BROPATH'] = bropath env['BRO_PLUGIN_PATH'] = pluginpath cwd = os.path.join(clone_dir, package.name) cmd = subprocess.Popen(test_command, shell=True, cwd=cwd, env=env) return ('', cmd.wait() == 0, test_dir) def _stage(self, package, version, clone, stage_script_dir, stage_plugin_dir): metadata_file = os.path.join(clone.working_dir, METADATA_FILENAME) # First use raw parser so no value interpolation takes place. raw_metadata_parser = configparser.RawConfigParser() invalid_reason = _parse_package_metadata( raw_metadata_parser, metadata_file) if invalid_reason: return invalid_reason raw_metadata = _get_package_metadata(raw_metadata_parser) requested_user_vars = user_vars(raw_metadata) if requested_user_vars is None: return "package has malformed 'user_vars' metadata field" substitutions = { 'bro_dist': self.bro_dist, } substitutions.update(self.user_vars) for k, v, _ in requested_user_vars: val_from_env = os.environ.get(k) if val_from_env: substitutions[k] = val_from_env if k not in substitutions: substitutions[k] = v metadata_parser = configparser.SafeConfigParser( defaults=substitutions) invalid_reason = _parse_package_metadata( metadata_parser, metadata_file) if invalid_reason: return invalid_reason metadata = _get_package_metadata(metadata_parser) LOG.debug('building "%s": version %s', package, version) build_command = metadata.get('build_command', '') if build_command: LOG.debug('building "%s": running build_command: %s', package, build_command) bufsize = 4096 build = subprocess.Popen(build_command, shell=True, cwd=clone.working_dir, bufsize=bufsize, stdout=subprocess.PIPE, stderr=subprocess.PIPE) try: buildlog = self.package_build_log(clone.working_dir) with open(buildlog, 'wb') as f: LOG.info('installing "%s": writing build log: %s', package, buildlog) f.write(u'=== STDERR ===\n'.encode(stdout_encoding())) while True: data = build.stderr.read(bufsize) if data: f.write(data) else: break f.write(u'=== STDOUT ===\n'.encode(stdout_encoding())) while True: data = build.stdout.read(bufsize) if data: f.write(data) else: break except EnvironmentError as error: LOG.warning( 'installing "%s": failed to write build log %s %s: %s', package, buildlog, error.errno, error.strerror) returncode = build.wait() if returncode != 0: return 'package build_command failed, see log in {}'.format( buildlog) pkg_script_dir = metadata.get('script_dir', '') script_dir_src = os.path.join(clone.working_dir, pkg_script_dir) script_dir_dst = os.path.join(stage_script_dir, package.name) if not os.path.exists(script_dir_src): return str.format("package's 'script_dir' does not exist: {}", pkg_script_dir) if os.path.isfile(os.path.join(script_dir_src, '__load__.bro')): symlink_path = os.path.join(os.path.dirname(stage_script_dir), package.name) try: make_symlink(os.path.join( 'packages', package.name), symlink_path) except OSError as exception: error = 'could not create symlink at {}'.format(symlink_path) error += ': {}: {}'.format(type(exception).__name__, exception) return error error = _copy_package_dir(package, 'script_dir', script_dir_src, script_dir_dst, self.scratch_dir) if error: return error else: if 'script_dir' in metadata: return str.format("no __load__.bro file found" " in package's 'script_dir' : {}", pkg_script_dir) else: LOG.warning('installing "%s": no __load__.bro in implicit' ' script_dir, skipped installing scripts', package) pkg_plugin_dir = metadata.get('plugin_dir', 'build') plugin_dir_src = os.path.join(clone.working_dir, pkg_plugin_dir) plugin_dir_dst = os.path.join(stage_plugin_dir, package.name) if not os.path.exists(plugin_dir_src): LOG.info('installing "%s": package "plugin_dir" does not exist: %s', package, pkg_plugin_dir) if pkg_plugin_dir != 'build': # It's common for a package to not have build directory for # for plugins, so don't error out in that case, just log it. return str.format("package's 'plugin_dir' does not exist: {}", pkg_plugin_dir) error = _copy_package_dir(package, 'plugin_dir', plugin_dir_src, plugin_dir_dst, self.scratch_dir) if error: return error def install(self, pkg_path, version=''): """Install a package. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "bro" with package named "foo" in :file:`alice/bro-pkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "bro/alice/foo". version (str): if not given, then the latest git version tag is installed (or if no version tags exist, the "master" branch is installed). If given, it may be either a git version tag or a git branch name. Returns: str: empty string if package installation succeeded else an error string explaining why it failed. Raises: IOError: if the manifest can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('installing "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if ipkg: conflict = ipkg.package if conflict.qualified_name().endswith(pkg_path): LOG.debug('installing "%s": re-install: %s', pkg_path, conflict) clonepath = os.path.join(self.package_clonedir, conflict.name) _clone_package(conflict, clonepath) return self._install(conflict, version) else: LOG.info( 'installing "%s": matched already installed package: %s', pkg_path, conflict) return str.format( 'package with name "{}" ({}) is already installed', conflict.name, conflict) matches = self.match_source_packages(pkg_path) if not matches: try: package = Package(git_url=pkg_path) return self._install(package, version) except git.exc.GitCommandError as error: LOG.info('installing "%s": invalid git repo path: %s', pkg_path, error) LOG.info('installing "%s": matched no source package', pkg_path) return 'package not found in sources and also not a valid git URL' if len(matches) > 1: matches_string = [match.qualified_name() for match in matches] LOG.info('installing "%s": matched multiple packages: %s', pkg_path, matches_string) return str.format('"{}" matches multiple packages, try a more' ' specific name from: {}', pkg_path, matches_string) try: return self._install(matches[0], version) except git.exc.GitCommandError as error: LOG.warning('installing "%s": source package git repo is invalid', pkg_path) return 'failed to clone package "{}": {}'.format(pkg_path, error) return '' def _install(self, package, version, use_existing_clone=False): """Install a :class:`.package.Package`. Returns: str: empty string if package installation succeeded else an error string explaining why it failed. Raises: git.exc.GitCommandError: if the git repo is invalid IOError: if the package manifest file can't be written """ clonepath = os.path.join(self.package_clonedir, package.name) ipkg = self.find_installed_package(package.name) if use_existing_clone or ipkg: clone = git.Repo(clonepath) else: clone = _clone_package(package, clonepath) status = PackageStatus() status.is_loaded = ipkg.status.is_loaded if ipkg else False status.is_pinned = ipkg.status.is_pinned if ipkg else False version_tags = _get_version_tags(clone) if version: if version in version_tags: status.tracking_method = 'version' else: branches = _get_branch_names(clone) if version in branches: status.tracking_method = 'branch' else: LOG.info( 'branch "%s" not in available branches: %s', version, branches) return 'no such branch or version tag: "{}"'.format(version) else: if len(version_tags): version = version_tags[-1] status.tracking_method = 'version' else: if 'master' not in _get_branch_names(clone): return 'git repo has no "master" branch or version tags' version = 'master' status.tracking_method = 'branch' status.current_version = version status.current_hash = _get_hash(clone, version) clone.git.checkout(version) status.is_outdated = _is_clone_outdated( clone, version, status.tracking_method) metadata_file = os.path.join(clone.working_dir, METADATA_FILENAME) # Use raw parser so no value interpolation takes place. raw_metadata_parser = configparser.RawConfigParser() invalid_reason = _parse_package_metadata( raw_metadata_parser, metadata_file) if invalid_reason: return invalid_reason raw_metadata = _get_package_metadata(raw_metadata_parser) fail_msg = self._stage(package, version, clone, self.script_dir, self.plugin_dir) if fail_msg: return fail_msg if not package.source: # If installing directly from git URL, see if it actually is found # in a package source and fill in those details. for pkg in self.source_packages(): if pkg.git_url == package.git_url: package.source = pkg.source package.directory = pkg.directory package.metadata = pkg.metadata break package.metadata = raw_metadata self.installed_pkgs[package.name] = InstalledPackage(package, status) self._write_manifest() LOG.debug('installed "%s"', package) return '' def _get_version_tags(clone): tags = [] for tagref in clone.tags: tag = str(tagref.name) try: semver.Version.coerce(tag) except ValueError: # Skip tags that aren't compatible semantic versions. continue else: tags.append(tag) return sorted(tags, key=semver.Version.coerce) def _get_branch_names(clone): rval = [] for ref in clone.references: branch_name = str(ref.name) if not branch_name.startswith('origin/'): continue rval.append(branch_name.split('/')[-1]) return rval def _get_ref(clone, ref_name): for ref in clone.refs: if ref.name.split('/')[-1] == ref_name: return ref def _is_version_outdated(clone, version): version_tags = _get_version_tags(clone) return version != version_tags[-1] def _is_branch_outdated(clone, branch): it = clone.iter_commits('{0}..origin/{0}'.format(branch)) num_commits_behind = sum(1 for c in it) return num_commits_behind > 0 def _is_clone_outdated(clone, ref_name, tracking_method): if tracking_method == 'version': return _is_version_outdated(clone, ref_name) elif tracking_method == 'branch': return _is_branch_outdated(clone, ref_name) else: raise NotImplementedError def _get_hash(clone, ref_name): return _get_ref(clone, ref_name).object.hexsha def _copy_package_dir(package, dirname, src, dst, scratch_dir): """Copy a directory from a package to its installation location. Returns: str: empty string if package dir copy succeeded else an error string explaining why it failed. """ if not os.path.exists(src): return '' if os.path.isfile(src) and tarfile.is_tarfile(src): tmp_dir = os.path.join(scratch_dir, 'untar') delete_path(tmp_dir) make_dir(tmp_dir) try: with tarfile.open(src) as tf: tf.extractall(tmp_dir) except Exception as error: return str(error) ld = os.listdir(tmp_dir) if len(ld) != 1: return 'failed to copy package {}: invalid tarfile'.format(dirname) src = os.path.join(tmp_dir, ld[0]) if not os.path.isdir(src): return 'failed to copy package {}: not a dir or tarfile'.format(dirname) def ignore(_, files): rval = [] for f in files: if f in {'.git', 'bro-pkg.meta'}: rval.append(f) return rval try: copy_over_path(src, dst, ignore=ignore) except shutil.Error as error: errors = error.args[0] reasons = "" for err in errors: src, dst, msg = err reason = 'failed to copy {}: {} -> {}: {}'.format( dirname, src, dst, msg) reasons += '\n' + reason LOG.warning('installing "%s": %s', package, reason) return 'failed to copy package {}: {}'.format(dirname, reasons) return '' def _create_readme(file_path): if os.path.exists(file_path): return with open(file_path, 'w') as f: f.write('WARNING: This directory is managed by bro-pkg.\n') f.write("Don't make direct modifications to anything within it.\n") def _clone_package(package, clonepath): """Clone a :class:`.package.Package` git repo. Returns: git.Repo: the cloned package Raises: git.exc.GitCommandError: if the git repo is invalid """ delete_path(clonepath) return git_clone_shallow(package.git_url, clonepath) def _get_package_metadata(parser): metadata = {item[0]: item[1] for item in parser.items('package')} return metadata def _parse_package_metadata(parser, metadata_file): """Return string explaining why metadata is invalid, or '' if valid. """ if not parser.read(metadata_file): LOG.warning('%s: missing metadata file', metadata_file) return 'missing {} metadata file'.format(METADATA_FILENAME) if not parser.has_section('package'): LOG.warning('%s: metadata missing [package]', metadata_file) return '{} is missing [package] section'.format(METADATA_FILENAME) return '' def _info_from_clone(clone, package, status, version): """Retrieves information about a package. Returns: A :class:`.package.PackageInfo` object. """ versions = _get_version_tags(clone) metadata_file = os.path.join(clone.working_dir, METADATA_FILENAME) # Use raw parser so no value interpolation takes place. metadata_parser = configparser.RawConfigParser() invalid_reason = _parse_package_metadata( metadata_parser, metadata_file) if invalid_reason: return PackageInfo(package=package, invalid_reason=invalid_reason, status=status, versions=versions, metadata_version=version) metadata = _get_package_metadata(metadata_parser) return PackageInfo(package=package, invalid_reason=invalid_reason, status=status, metadata=metadata, versions=versions, metadata_version=version) package-manager-1.3.3/bropkg/source.py0000644000175000017500000001137213241343432017230 0ustar rhaistrhaist""" A module containing the definition of a "package source": a git repository containing a collection of :file:`bro-pkg.index` files. These are simple INI files that can describe many Bro packages. Each section of the file names a Bro package along with the git URL where it is located and metadata tags that help classify/describe it. """ import os import shutil import git try: from backports import configparser except ImportError as err: import configparser from . import LOG from .package import ( name_from_path, Package ) from ._util import git_clone_shallow #: The name of package index files. INDEX_FILENAME = 'bro-pkg.index' #: The name of the package source file where package metadata gets aggregated. AGGREGATE_DATA_FILE = 'aggregate.meta' class Source(object): """A Bro package source. This class contains properties of a package source like its name, remote git URL, and local git clone. Attributes: name (str): The name of the source as given by a config file key in it's ``[sources]`` section. git_url (str): The git URL of the package source. clone (git.Repo): The local git clone of the package source. """ def __init__(self, name, clone_path, git_url): """Create a package source. Raises: git.exc.GitCommandError: if the git repo is invalid OSError: if the git repo is invalid and can't be re-initialized """ git_url = os.path.expanduser(git_url) self.name = name self.git_url = git_url self.clone = None try: self.clone = git.Repo(clone_path) except git.exc.NoSuchPathError: LOG.debug('creating source clone of "%s" at %s', name, clone_path) self.clone = git_clone_shallow(git_url, clone_path) except git.exc.InvalidGitRepositoryError: LOG.debug('deleting invalid source clone of "%s" at %s', name, clone_path) shutil.rmtree(clone_path) self.clone = git_clone_shallow(git_url, clone_path) else: LOG.debug('found source clone of "%s" at %s', name, clone_path) old_url = self.clone.git.config('--local', '--get', 'remote.origin.url') if git_url != old_url: LOG.debug( 'url of source "%s" changed from %s to %s, reclone at %s', name, old_url, git_url, clone_path) shutil.rmtree(clone_path) self.clone = git_clone_shallow(git_url, clone_path) def __str__(self): return self.git_url def __repr__(self): return self.git_url def package_index_files(self): """Return a list of paths to package index files in the source.""" rval = [] visited_dirs = set() for root, dirs, files in os.walk(self.clone.working_dir, followlinks=True): stat = os.stat(root) visited_dirs.add((stat.st_dev, stat.st_ino)) dirs_to_visit_next = [] for d in dirs: stat = os.stat(os.path.join(root, d)) if (stat.st_dev, stat.st_ino) not in visited_dirs: dirs_to_visit_next.append(d) dirs[:] = dirs_to_visit_next try: dirs.remove('.git') except ValueError: pass for filename in files: if filename == INDEX_FILENAME: rval.append(os.path.join(root, filename)) return sorted(rval) def packages(self): """Return a list of :class:`.package.Package` in the source.""" rval = [] # Use raw parser so no value interpolation takes place. parser = configparser.RawConfigParser() aggregate_file = os.path.join( self.clone.working_dir, AGGREGATE_DATA_FILE) parser.read(aggregate_file) for index_file in self.package_index_files(): relative_path = index_file[len(self.clone.working_dir) + 1:] directory = os.path.dirname(relative_path) lines = [] with open(index_file) as f: lines = [line.rstrip('\n') for line in f] for url in lines: pkg_name = name_from_path(url) agg_key = os.path.join(directory, pkg_name) metadata = {} if parser.has_section(agg_key): metadata = {key: value for key, value in parser.items(agg_key)} package = Package(git_url=url, source=self.name, directory=directory, metadata=metadata) rval.append(package) return rval package-manager-1.3.3/bropkg/__init__.py0000644000175000017500000000111113241343432017455 0ustar rhaistrhaist""" This package defines a Python interface for installing, managing, querying, and performing other operations on Bro Packages and Package Sources. The main entry point is the :class:`Manager ` class. This package provides a logger named `LOG` to which logging stream handlers may be added in order to help log/debug applications. """ import logging __version__ = "1.3.3" __all__ = ['manager', 'package', 'source'] LOG = logging.getLogger(__name__) LOG.addHandler(logging.NullHandler()) from .manager import * from .package import * from .source import * package-manager-1.3.3/bropkg/package.py0000644000175000017500000003114413241343432017322 0ustar rhaistrhaist""" A module with various data structures used for interacting with and querying the properties and status of Bro packages. """ import os import re from ._util import ( remove_trailing_slashes, find_sentence_end, ) #: The name of files used by packages to store their metadata. METADATA_FILENAME = 'bro-pkg.meta' def name_from_path(path): """Returns the name of a package given a path to its git repository.""" return canonical_url(path).split('/')[-1] def canonical_url(path): """Returns the url of a package given a path to its git repo.""" url = remove_trailing_slashes(path) if url.startswith('.') or url.startswith('/'): url = os.path.realpath(url) return url def tags(metadata_dict): """Return a list of tag strings found in the metadata's 'tags' field.""" if 'tags' not in metadata_dict: return [] import re return re.split(',\s*', metadata_dict['tags']) def short_description(metadata_dict): """Returns the first sentence of the metadata's 'desciption' field.""" if 'description' not in metadata_dict: return '' description = metadata_dict['description'] lines = description.split('\n') rval = '' for line in lines: line = line.lstrip() rval += ' ' period_idx = find_sentence_end(line) if period_idx == -1: rval += line else: rval += line[:period_idx + 1] break return rval.lstrip() def user_vars(metadata_dict): """Returns a list of (str, str, str) from metadata's 'user_vars' field. Each entry in the returned list is a the name of a variable, it's value, and its description. If the 'user_vars' field is not present, an empty list is returned. If it is malformed, then None is returned. """ text = metadata_dict.get('user_vars') if not text: return [] rval = [] text = text.strip() entries = re.split('(\w+\s+\\[.*\\]\s+".*")\s+', text) entries = list(filter(None, entries)) for entry in entries: m = re.match('(\w+)\s+\\[(.*)\\]\s+"(.*)"', entry) if not m: return None groups = m.groups() if len(groups) != 3: return None rval.append((groups[0], groups[1], groups[2])) return rval def dependencies(metadata_dict, field='depends'): """Returns a dictionary of (str, str) based on metadata's dependency field. The keys indicate the name of a package (shorthand name or full git URL) or just 'bro' to indicate a dependency on a particular bro version. The values indicate a semantic version requirement. If the dependency field is malformed (e.g. number of keys not equal to number of values), then None is returned. """ if field not in metadata_dict: return dict() rval = dict() depends = metadata_dict[field] parts = depends.split() keys = parts[::2] values = parts[1::2] if len(keys) != len(values): return None for i, k in enumerate(keys): if i < len(values): rval[k] = values[i] return rval class InstalledPackage(object): """An installed package and its current status. Attributes: package (:class:`Package`): the installed package status (:class:`PackageStatus`): the status of the installed package """ def __init__(self, package, status): self.package = package self.status = status def __lt__(self, other): return str(self.package) < str(other.package) class PackageStatus(object): """The status of an installed package. This class contains properties of a package related to how the package manager will operate on it. Attributes: is_loaded (bool): whether a package is marked as "loaded". is_pinned (bool): whether a package is allowed to be upgraded. is_outdated (bool): whether a newer version of the package exists. tracking_method (str): either "branch" or "version" to indicate whether package upgrades should stick to a git branch or use git version tags. current_version (str): the current version of the installed package, which is either a git branch name or a git version tag. current_hash (str): the git sha1 hash associated with installed package's current version/commit. """ def __init__(self, is_loaded=False, is_pinned=False, is_outdated=False, tracking_method=None, current_version=None, current_hash=None): self.is_loaded = is_loaded self.is_pinned = is_pinned self.is_outdated = is_outdated self.tracking_method = tracking_method self.current_version = current_version self.current_hash = current_hash class PackageInfo(object): """Contains information on an arbitrary package. If the package is installed, then its status is also available. Attributes: package (:class:`Package`): the relevant Bro package status (:class:`PackageStatus`): this attribute is set for installed packages metadata (dict of str -> str): the contents of the package's :file:`bro-pkg.meta` file versions (list of str): a list of the package's availabe git version tags metadata_version: the package version that the metadata is from invalid_reason (str): this attribute is set when there is a problem with gathering package information and explains what went wrong """ def __init__(self, package=None, status=None, metadata=None, versions=None, metadata_version='', invalid_reason=''): self.package = package self.status = status self.metadata = {} if metadata is None else metadata self.versions = [] if versions is None else versions self.metadata_version = metadata_version self.invalid_reason = invalid_reason def tags(self): """Return a list of keyword tags associated with the package. This will be the contents of the package's `tags` field.""" return tags(self.metadata) def short_description(self): """Return a short description of the package. This will be the first sentence of the package's 'description' field.""" return short_description(self.metadata) def dependencies(self, field='depends'): """Returns a dictionary of dependency -> version strings. The keys indicate the name of a package (shorthand name or full git URL) or just 'bro' to indicate a dependency on a particular bro version. The values indicate a semantic version requirement. If the dependency field is malformed (e.g. number of keys not equal to number of values), then None is returned. """ return dependencies(self.metadata, field) def user_vars(self): """Returns a list of (str, str, str) from metadata's 'user_vars' field. Each entry in the returned list is a the name of a variable, it's value, and its description. If the 'user_vars' field is not present, an empty list is returned. If it is malformed, then None is returned. """ return user_vars(self.metadata) def best_version(self): """Returns the best/latest version of the package that is available. If the package has any git release tags, this returns the highest one, else it returns the 'master' branch. """ if self.versions: return self.versions[-1] return 'master' class Package(object): """A Bro package. This class contains properties of a package that are defined by the package git repository itself and the package source it came from. Attributes: git_url (str): the git URL which uniquely identifies where the Bro package is located name (str): the canonical name of the package, which is always the last component of the git URL path source (str): the package source this package comes from, which may be empty if the package is not a part of a source (i.e. the user is referring directly to the package's git URL). directory (str): the directory within the package source where the :file:`bro-pkg.index` containing this package is located. E.g. if the package source has a package named "foo" declared in :file:`alice/bro-pkg.index`, then `dir` is equal to "alice". It may also be empty if the package is not part of a package source or if it's located in a top-level :file:`bro-pkg.index` file. metadata (dict of str -> str): the contents of the package's :file:`bro-pkg.meta` file. If the package has not been installed then this information may come from the last aggregation of the source's :file:`aggregate.meta` file (it may not be accurate/up-to-date). """ def __init__(self, git_url, source='', directory='', metadata=None): self.git_url = canonical_url(git_url) self.name = name_from_path(git_url) self.source = source self.directory = directory self.metadata = {} if metadata is None else metadata def __str__(self): return self.qualified_name() def __repr__(self): return self.git_url def __lt__(self, other): return str(self) < str(other) def tags(self): """Return a list of keyword tags associated with the package. This will be the contents of the package's `tags` field and may return results from the source's aggregated metadata if the package has not been installed yet.""" return tags(self.metadata) def short_description(self): """Return a short description of the package. This will be the first sentence of the package's 'description' field and may return results from the source's aggregated metadata if the package has not been installed yet.""" return short_description(self.metadata) def dependencies(self, field='depends'): """Returns a dictionary of dependency -> version strings. The keys indicate the name of a package (shorthand name or full git URL) or just 'bro' to indicate a dependency on a particular bro version. The values indicate a semantic version requirement. If the dependency field is malformed (e.g. number of keys not equal to number of values), then None is returned. """ return dependencies(self.metadata, field) def user_vars(self): """Returns a list of (str, str, str) from metadata's 'user_vars' field. Each entry in the returned list is a the name of a variable, it's value, and its description. If the 'user_vars' field is not present, an empty list is returned. If it is malformed, then None is returned. """ return user_vars(self.metadata) def name_with_source_directory(self): """Return the package's within its package source. E.g. for a package source with a package named "foo" in :file:`alice/bro-pkg.index`, this method returns "alice/foo". If the package has no source or sub-directory within the source, then just the package name is returned. """ if self.directory: return '{}/{}'.format(self.directory, self.name) return self.name def qualified_name(self): """Return the shortest name that qualifies/distinguishes the package. If the package is part of a source, then this returns "source_name/:meth:`name_with_source_directory()`", else the package's git URL is returned. """ if self.source: return '{}/{}'.format(self.source, self.name_with_source_directory()) return self.git_url def matches_path(self, path): """Return whether this package has a matching path/name. E.g for a package with :meth:`qualified_name()` of "bro/alice/foo", the following inputs will match: "foo", "alice/foo", "bro/alice/foo" """ path_parts = path.split('/') if self.source: pkg_path = self.qualified_name() pkg_path_parts = pkg_path.split('/') for i, part in reversed(list(enumerate(path_parts))): ri = i - len(path_parts) if part != pkg_path_parts[ri]: return False return True else: if len(path_parts) == 1 and path_parts[-1] == self.name: return True return path == self.git_url package-manager-1.3.3/setup.py0000644000175000017500000000237313241343432015605 0ustar rhaistrhaistfrom setuptools import setup from sys import version_info import os install_requires = ['gitpython', 'semantic_version', 'btest'] if version_info[0] + version_info[1] / 10 < 3.5 or os.environ.get('BROPKG_PYPI_DIST'): install_requires.append("configparser") setup( name='bro-pkg', version=open('VERSION').read().replace('-', '.dev', 1).strip(), description='The Bro Package Manager', long_description=open('README').read(), license='University of Illinois/NCSA Open Source License', keywords='bro broctl brocontrol package manager scripts plugins security', maintainer='The Bro Project', maintainer_email='info@bro.org', url='https://github.com/bro/package-manager', scripts=['bro-pkg'], packages=['bropkg'], install_requires=install_requires, classifiers=[ 'Development Status :: 5 - Production/Stable', 'Environment :: Console', 'License :: OSI Approved :: University of Illinois/NCSA Open Source License', 'Operating System :: POSIX :: Linux', 'Operating System :: MacOS :: MacOS X', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Topic :: System :: Networking :: Monitoring', 'Topic :: Utilities', ], ) package-manager-1.3.3/setup.cfg0000644000175000017500000000003413241343432015704 0ustar rhaistrhaist[bdist_wheel] universal = 1 package-manager-1.3.3/bro-pkg.py0000777000175000017500000000000013241343432017262 2bro-pkgustar rhaistrhaistpackage-manager-1.3.3/.gitignore0000644000175000017500000000005613241343432016057 0ustar rhaistrhaist*.pyc *.egg-info .state doc/_build build dist package-manager-1.3.3/testing/0000755000175000017500000000000013241343432015543 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/0000755000175000017500000000000013241343432017321 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/rot13/0000755000175000017500000000000013241343432020271 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/rot13/configure0000755000175000017500000000470213241343432022203 0ustar rhaistrhaist#!/bin/sh # # Wrapper for viewing/setting options that the plugin's CMake # scripts will recognize. # # Don't edit this. Edit configure.plugin to add plugin-specific options. # set -e command="$0 $*" if [ -e `dirname $0`/configure.plugin ]; then # Include custom additions. . `dirname $0`/configure.plugin fi # Check for `cmake` command. type cmake > /dev/null 2>&1 || { echo "\ This package requires CMake, please install it first, then you may use this configure script to access CMake equivalent functionality.\ " >&2; exit 1; } usage() { cat 1>&2 </dev/null 2>&1; then plugin_usage 1>&2 fi echo exit 1 } # Function to append a CMake cache entry definition to the # CMakeCacheEntries variable # $1 is the cache entry variable name # $2 is the cache entry variable type # $3 is the cache entry variable value append_cache_entry () { CMakeCacheEntries="$CMakeCacheEntries -D $1:$2=$3" } # set defaults builddir=build brodist=`cd ../../.. && pwd` installroot="default" CMakeCacheEntries="" while [ $# -ne 0 ]; do case "$1" in -*=*) optarg=`echo "$1" | sed 's/[-_a-zA-Z0-9]*=//'` ;; *) optarg= ;; esac case "$1" in --help|-h) usage ;; --bro-dist=*) brodist=`cd $optarg && pwd` ;; --install-root=*) installroot=$optarg ;; *) if type plugin_option >/dev/null 2>&1; then plugin_option $1 && shift && continue; fi echo "Invalid option '$1'. Try $0 --help to see available options." exit 1 ;; esac shift done if [ ! -e "$brodist/bro-path-dev.in" ]; then echo "Cannot determine Bro source directory, use --bro-dist=DIR." exit 1 fi append_cache_entry BRO_DIST PATH $brodist append_cache_entry CMAKE_MODULE_PATH PATH $brodist/cmake if [ "$installroot" != "default" ]; then mkdir -p $installroot append_cache_entry BRO_PLUGIN_INSTALL_ROOT PATH $installroot fi echo "Build Directory : $builddir" echo "Bro Source Directory : $brodist" mkdir -p $builddir cd $builddir cmake $CMakeCacheEntries .. echo "# This is the command used to configure this build" > config.status echo $command >> config.status chmod u+x config.status package-manager-1.3.3/testing/packages/rot13/COPYING.edit-me0000644000175000017500000000325313241343432022652 0ustar rhaistrhaist### ### This is a BSD-style license. If you're happy with it, just edit ### the XXX parts below and remove this comment. Otherwise, put in ### your own license instead. ### Copyright (c) 2015 by Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: (1) Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. (2) Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. (3) Neither the name of , nor the names of contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. package-manager-1.3.3/testing/packages/rot13/bro-pkg.meta0000644000175000017500000000022213241343432022476 0ustar rhaistrhaist[package] script_dir = scripts/Demo/Rot13 build_command = ./configure --bro-dist=%(bro_dist)s && make test_command = cd testing && btest -d tests package-manager-1.3.3/testing/packages/rot13/CHANGES0000644000175000017500000000000013241343432021252 0ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/rot13/configure.plugin0000644000175000017500000000057413241343432023500 0ustar rhaistrhaist#!/bin/sh # # Hooks to add custom options to the configure script. # plugin_usage() { : # Do nothing # cat <//__load__.bro instead. # @load ./init.bro event bro_init() { print "rot13 plugin is loaded"; } package-manager-1.3.3/testing/packages/rot13/scripts/types.bro0000644000175000017500000000000113241343432023617 0ustar rhaistrhaist package-manager-1.3.3/testing/packages/rot13/testing/0000755000175000017500000000000013241343432021746 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/rot13/testing/btest.cfg0000644000175000017500000000000013241343432023536 0ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/rot13/testing/tests/0000755000175000017500000000000013241343432023110 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/rot13/testing/tests/rot130000644000175000017500000000013713241343432024004 0ustar rhaistrhaist# @TEST-EXEC: bro %INPUT > output # @TEST-EXEC: btest-diff output print Demo::rot13("Hello"); package-manager-1.3.3/testing/packages/rot13/testing/tests/main0000644000175000017500000000010213241343432023750 0ustar rhaistrhaist# @TEST-EXEC: bro rot13 > output # @TEST-EXEC: btest-diff output package-manager-1.3.3/testing/packages/rot13/testing/Baseline/0000755000175000017500000000000013241343432023470 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/rot13/testing/Baseline/tests.rot13/0000755000175000017500000000000013241343432025601 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/rot13/testing/Baseline/tests.rot13/output0000644000175000017500000000003513241343432027062 0ustar rhaistrhaistUryyb rot13 plugin is loaded package-manager-1.3.3/testing/packages/rot13/testing/Baseline/tests.main/0000755000175000017500000000000013241343432025555 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/rot13/testing/Baseline/tests.main/output0000644000175000017500000000005613241343432027041 0ustar rhaistrhaistrot13 plugin is loaded rot13 script is loaded package-manager-1.3.3/testing/packages/rot13/src/0000755000175000017500000000000013241343432021060 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/rot13/src/rot13.bif0000644000175000017500000000062513241343432022515 0ustar rhaistrhaistmodule Demo; function rot13%(s: string%) : string %{ char* rot13 = copy_string(s->CheckString()); for ( char* p = rot13; *p; p++ ) { char b = islower(*p) ? 'a' : 'A'; *p = (*p - b + 13) % 26 + b; } BroString* bs = new BroString(1, reinterpret_cast(rot13), strlen(rot13)); return new StringVal(bs); %} package-manager-1.3.3/testing/packages/rot13/src/Plugin.cc0000644000175000017500000000053213241343432022625 0ustar rhaistrhaist #include "Plugin.h" namespace plugin { namespace Demo_Rot13 { Plugin plugin; } } using namespace plugin::Demo_Rot13; plugin::Configuration Plugin::Configure() { plugin::Configuration config; config.name = "Demo::Rot13"; config.description = ""; config.version.major = 0; config.version.minor = 1; return config; } package-manager-1.3.3/testing/packages/rot13/src/Plugin.h0000644000175000017500000000046013241343432022467 0ustar rhaistrhaist #ifndef BRO_PLUGIN_DEMO_ROT13 #define BRO_PLUGIN_DEMO_ROT13 #include namespace plugin { namespace Demo_Rot13 { class Plugin : public ::plugin::Plugin { protected: // Overridden from plugin::Plugin. virtual plugin::Configuration Configure(); }; extern Plugin plugin; } } #endif package-manager-1.3.3/testing/packages/rot13/README0000644000175000017500000000012413241343432021146 0ustar rhaistrhaist Demo::Rot13 ================================= package-manager-1.3.3/testing/packages/corge/0000755000175000017500000000000013241343432020420 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/corge/bro-pkg.meta0000644000175000017500000000001213241343432022622 0ustar rhaistrhaist[package] package-manager-1.3.3/testing/packages/corge/__load__.bro0000644000175000017500000000002613241343432022635 0ustar rhaistrhaistprint "corge loaded"; package-manager-1.3.3/testing/packages/qux/0000755000175000017500000000000013241343432020136 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/qux/bro-pkg.meta0000644000175000017500000000001213241343432022340 0ustar rhaistrhaist[package] package-manager-1.3.3/testing/packages/qux/__load__.bro0000644000175000017500000000002413241343432022351 0ustar rhaistrhaistprint "qux loaded"; package-manager-1.3.3/testing/packages/grault/0000755000175000017500000000000013241343432020617 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/grault/bro-pkg.meta0000644000175000017500000000001213241343432023021 0ustar rhaistrhaist[package] package-manager-1.3.3/testing/packages/grault/__load__.bro0000644000175000017500000000002713241343432023035 0ustar rhaistrhaistprint "grault loaded"; package-manager-1.3.3/testing/packages/foo/0000755000175000017500000000000013241343432020104 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/foo/bro-pkg.meta0000644000175000017500000000001213241343432022306 0ustar rhaistrhaist[package] package-manager-1.3.3/testing/packages/foo/__load__.bro0000644000175000017500000000002413241343432022317 0ustar rhaistrhaistprint "foo loaded"; package-manager-1.3.3/testing/packages/baz/0000755000175000017500000000000013241343432020075 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/baz/bro-pkg.meta0000644000175000017500000000001213241343432022277 0ustar rhaistrhaist[package] package-manager-1.3.3/testing/packages/baz/__load__.bro0000644000175000017500000000002413241343432022310 0ustar rhaistrhaistprint "baz loaded"; package-manager-1.3.3/testing/packages/bar/0000755000175000017500000000000013241343432020065 5ustar rhaistrhaistpackage-manager-1.3.3/testing/packages/bar/bro-pkg.meta0000644000175000017500000000001213241343432022267 0ustar rhaistrhaist[package] package-manager-1.3.3/testing/packages/bar/__load__.bro0000644000175000017500000000005113241343432022300 0ustar rhaistrhaistevent bro_init() { print "bar loaded"; } package-manager-1.3.3/testing/Makefile0000644000175000017500000000060713241343432017206 0ustar rhaistrhaist.PHONY: all all: btest-installation-check @btest -j -f diag.log .PHONY: btest-installation-check btest-installation-check: @type btest > /dev/null 2>&1 || \ { \ echo "btest was not located in PATH."; \ echo "Install it and/or make sure it is in PATH."; \ echo "E.g. you could use the following command to install it:"; \ echo "\tpip install btest"; \ echo ; \ exit 1; \ } package-manager-1.3.3/testing/scripts/0000755000175000017500000000000013241343432017232 5ustar rhaistrhaistpackage-manager-1.3.3/testing/scripts/bro-pkg0000755000175000017500000000055613241343432020527 0ustar rhaistrhaist#! /usr/bin/env bash force="" case "$1" in install) force="--force" ;; upgrade) force="--force" ;; bundle) force="--force" ;; unbundle) force="--force" ;; remove) force="--force" ;; purge) force="--force" ;; esac $TEST_BASE/../bro-pkg --configfile=config "$@" $force package-manager-1.3.3/testing/scripts/initializer0000755000175000017500000000107013241343432021501 0ustar rhaistrhaist#! /usr/bin/env bash cp -R $PACKAGES . for p in packages/*; do ( cd $p && git init && git add * && git commit -m 'init' ) done cp -R $SOURCES . find sources -name 'bro-pkg.index' -exec sed -i -e "s#^#$(pwd)/packages/#" {} \; for s in sources/*; do ( cd $s && git init && git add * && git commit -m 'init' ) done echo "\ [sources] one = $(pwd)/sources/one [paths] state_dir = $(pwd)/state script_dir = $(pwd)/scripts plugin_dir = $(pwd)/plugins " >> config type bro-config > /dev/null 2>&1 && echo "bro_dist = $(bro-config --bro_dist)" >> config || true package-manager-1.3.3/testing/btest.cfg0000644000175000017500000000054613241343432017352 0ustar rhaistrhaist[btest] TestDirs = tests TmpDir = %(testbase)s/.tmp IgnoreDirs = .svn CVS .tmp .git IgnoreFiles = *.tmp *.swp .DS_Store .gitignore BaselineDir = baselines Initializer = %(testbase)s/scripts/initializer [environment] PATH=%(testbase)s/scripts:%(default_path)s SCRIPTS=%(testbase)s/scripts SOURCES=%(testbase)s/sources PACKAGES=%(testbase)s/packages package-manager-1.3.3/testing/tests/0000755000175000017500000000000013241343432016705 5ustar rhaistrhaistpackage-manager-1.3.3/testing/tests/metadata-script_dir0000644000175000017500000000036713241343432022556 0ustar rhaistrhaist# @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg install foo # @TEST-EXEC: btest-diff scripts/foo/__load__.bro cd packages/foo mkdir scripts echo 'script_dir = scripts' >> bro-pkg.meta mv __load__.bro scripts/ git add * git commit -m 'new stuff' package-manager-1.3.3/testing/tests/info0000644000175000017500000000175513241343432017573 0ustar rhaistrhaist# @TEST-EXEC: bro-pkg install foo # @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg refresh # @TEST-EXEC: bro-pkg install bar # @TEST-EXEC: bro-pkg info foo | grep -v 'url:' | grep -v 'current_hash' > foo.info # @TEST-EXEC: btest-diff foo.info # @TEST-EXEC: bro-pkg info bar | grep -v 'url:' | grep -v 'current_hash' > bar.info # @TEST-EXEC: btest-diff bar.info cd packages/foo echo 'print "foo 1.0.0";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "foo 1.0.1";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "foo 1.0.2";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 cd ../bar echo 'print "bar+ loaded";' > __load__.bro git commit -am 'new stuff' echo 'print "bar 1.0.0 loaded";' > __load__.bro echo 'tags = sega sunset' >> bro-pkg.meta echo 'description = bar bar bar' >> bro-pkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "bar++ loaded";' > __load__.bro git commit -am 'new stuff' package-manager-1.3.3/testing/tests/user_vars0000644000175000017500000000101313241343432020634 0ustar rhaistrhaist# @TEST-EXEC: bash %INPUT # @TEST-EXEC: LAST_VAR=/home/jon/sandbox bro-pkg install foo # @TEST-EXEC: btest-diff state/logs/foo-build.log cd packages/foo echo 'user_vars =' >> bro-pkg.meta echo ' TEST_VAR [/usr] "First description is here"' >> bro-pkg.meta echo ' ANOTHER_VAR [/usr/local] "Second description is here"' >> bro-pkg.meta echo ' LAST_VAR [/opt] "Last description is here"' >> bro-pkg.meta echo 'build_command = echo "%(LAST_VAR)s\n%(ANOTHER_VAR)s\n%(TEST_VAR)s"' >> bro-pkg.meta git commit -am 'new stuff' package-manager-1.3.3/testing/tests/purge0000644000175000017500000000065513241343432017760 0ustar rhaistrhaist# @TEST-EXEC: bro-pkg install foo bar baz # @TEST-EXEC: test -f scripts/packages/foo/__load__.bro # @TEST-EXEC: test -f scripts/packages/bar/__load__.bro # @TEST-EXEC: test -f scripts/packages/baz/__load__.bro # @TEST-EXEC: bro-pkg purge # @TEST-EXEC: test ! -d scripts/packages/foo # @TEST-EXEC: test ! -d scripts/packages/bar # @TEST-EXEC: test ! -d scripts/packages/baz # @TEST-EXEC: btest-diff scripts/packages/packages.bro package-manager-1.3.3/testing/tests/metadata-config_files0000644000175000017500000000233713241343432023042 0ustar rhaistrhaist# @TEST-EXEC: bash setup # @TEST-EXEC: bro-pkg install foo bar # @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg refresh # @TEST-EXEC: bro-pkg upgrade foo # @TEST-EXEC: btest-diff scripts/foo/config.bro # @TEST-EXEC: btest-diff scripts/bar/config.bro # @TEST-EXEC: cp state/backups/one/alice/foo/config.bro* foo_backup.bro # @TEST-EXEC: btest-diff foo_backup.bro # @TEST-EXEC: bro-pkg install bar --version=2.0.0 # @TEST-EXEC: cp scripts/bar/config.bro barconfig2.bro # @TEST-EXEC: btest-diff barconfig2.bro # @TEST-EXEC: cp state/backups/one/alice/bar/config.bro* bar_backup.bro # @TEST-EXEC: btest-diff bar_backup.bro @TEST-START-FILE setup cd packages/foo echo 'print "foo orig";' > config.bro echo 'config_files = config.bro' >> bro-pkg.meta git add * git commit -am 'new stuff' cd ../bar echo 'print "bar 1.0.0";' > config.bro echo 'config_files = config.bro' >> bro-pkg.meta git add * git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 @TEST-END-FILE echo 'print "foo mod";' > scripts/foo/config.bro echo 'print "bar mod";' > scripts/bar/config.bro cd packages/foo echo 'print "foo new";' > config.bro git commit -am 'new stuff' cd ../bar echo 'print "bar 2.0.0";' > config.bro git commit -am 'new stuff' git tag -a 2.0.0 -m 2.0.0 package-manager-1.3.3/testing/tests/remove0000644000175000017500000000035013241343432020123 0ustar rhaistrhaist# @TEST-EXEC: bro-pkg install foo bar baz # @TEST-EXEC: test -f scripts/packages/foo/__load__.bro # @TEST-EXEC: bro-pkg remove foo # @TEST-EXEC: test ! -d scripts/packages/foo # @TEST-EXEC: btest-diff scripts/packages/packages.bro package-manager-1.3.3/testing/tests/metadata-suggests0000644000175000017500000000270313241343432022254 0ustar rhaistrhaist# @TEST-EXEC: bro-pkg install foo # @TEST-EXEC: bro-pkg list installed > no_suggests.out # @TEST-EXEC: btest-diff no_suggests.out # @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg refresh # @TEST-EXEC: bro-pkg upgrade foo # @TEST-EXEC: bro-pkg list installed > upgrade_suggests.out # @TEST-EXEC: btest-diff upgrade_suggests.out # @TEST-EXEC: bro-pkg purge # @TEST-EXEC: bro-pkg install foo # @TEST-EXEC: bro-pkg list installed > install_suggests.out # @TEST-EXEC: btest-diff install_suggests.out # @TEST-EXEC: bro-pkg purge # @TEST-EXEC: bro-pkg bundle test.bundle --manifest foo # @TEST-EXEC: bro-pkg unbundle test.bundle # @TEST-EXEC: bro-pkg list installed > bundle_suggests.out # @TEST-EXEC: btest-diff bundle_suggests.out cd packages/foo echo 'suggests = bar *' >> bro-pkg.meta git commit -am 'new stuff' cd ../bar echo 'suggests = baz >=1.0.0' >> bro-pkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 cd ../baz echo 'suggests = grault branch=master' >> bro-pkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "2.0.0";' > __load__.bro git commit -am 'new stuff' git tag -a 2.0.0 -m 2.0.0 cd ../grault echo 'suggests = corge ==1.0.1' >> bro-pkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 cd ../corge git tag -a 1.0.0 -m 1.0.0 echo 'suggests = foo * bar *' >> bro-pkg.meta git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "2.0.0";' > __load__.bro git commit -am 'new stuff' git tag -a 2.0.0 -m 2.0.0 package-manager-1.3.3/testing/tests/plugin_tarfile0000644000175000017500000000076713241343432021646 0ustar rhaistrhaist# @TEST-REQUIRES: type bro-config # @TEST-REQUIRES: test -n "$(bro-config --bro_dist)" # @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg install rot13 # @TEST-EXEC: btest-diff plugins/packages/rot13/__bro_plugin__ # @TEST-EXEC: btest-diff scripts/packages/rot13/__load__.bro cd packages/rot13 echo "plugin_dir = build/Demo_Rot13.tgz" >> bro-pkg.meta git commit -am 'new stuff' cd ../.. echo "$(pwd)/packages/rot13" >> sources/one/bob/bro-pkg.index cd sources/one git commit -am 'add rot13 package' package-manager-1.3.3/testing/tests/refresh0000644000175000017500000000220513241343432020265 0ustar rhaistrhaist# @TEST-EXEC: bro-pkg install foo # @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg refresh # @TEST-EXEC: bro-pkg list outdated > outdated.out # @TEST-EXEC: btest-diff outdated.out # @TEST-EXEC: bro-pkg list all > list.out # @TEST-EXEC: btest-diff list.out # @TEST-EXEC: bro-pkg refresh --aggregate # @TEST-EXEC: bro-pkg search lucky > search.out # @TEST-EXEC: btest-diff search.out # @TEST-EXEC: bro-pkg list all > list_after_agg.out # @TEST-EXEC: btest-diff list_after_agg.out # note: foo's description isn't in the list output since the installed # version's metadata has no 'description' field cd packages/foo echo 'tags = esoteric lucky land' >> bro-pkg.meta echo 'description = This is the foo package description' >> bro-pkg.meta git commit -am 'new stuff' cd .. mkdir new_pkg cd new_pkg git init echo '[package]' > bro-pkg.meta echo 'tags = esoteric lucky land' >> bro-pkg.meta echo 'description = This is the new_pkg package description' >> bro-pkg.meta echo 'print "hello";' >> __load__.bro git add * git commit -m 'init' cd ../.. echo "$(pwd)/packages/new_pkg" >> sources/one/alice/bro-pkg.index cd sources/one git commit -am 'add new_pkg' package-manager-1.3.3/testing/tests/load0000644000175000017500000000055713241343432017556 0ustar rhaistrhaist# @TEST-EXEC: bro-pkg install foo bar # @TEST-EXEC: btest-diff scripts/packages/packages.bro # @TEST-EXEC: bro-pkg unload foo # @TEST-EXEC: cp scripts/packages/packages.bro after_unload.out # @TEST-EXEC: btest-diff after_unload.out # @TEST-EXEC: bro-pkg load foo # @TEST-EXEC: cp scripts/packages/packages.bro after_load.out # @TEST-EXEC: btest-diff after_load.out package-manager-1.3.3/testing/tests/install0000644000175000017500000000263313241343432020302 0ustar rhaistrhaist# @TEST-EXEC: bro-pkg install foo # @TEST-EXEC: btest-diff scripts/packages/foo/__load__.bro # @TEST-EXEC: bro-pkg install alice/bar # @TEST-EXEC: btest-diff scripts/packages/bar/__load__.bro # @TEST-EXEC: bro-pkg install one/alice/baz # @TEST-EXEC: btest-diff scripts/packages/baz/__load__.bro # @TEST-EXEC: bro-pkg install $(pwd)/packages/qux # @TEST-EXEC: btest-diff scripts/packages/qux/__load__.bro # @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg install corge # @TEST-EXEC: btest-diff scripts/packages/corge/__load__.bro # @TEST-EXEC: bro-pkg remove corge # @TEST-EXEC: bro-pkg install corge --version=master # @TEST-EXEC: cp scripts/packages/corge/__load__.bro corge.master # @TEST-EXEC: btest-diff corge.master cd packages/corge git tag -a 1.0.0 -m 1.0.0 echo 'print "hello";' >> __load__.bro git commit -am 'new stuff' cd ../grault echo 'print "grault 1.0.0";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "grault 1.0.1";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "grault 1.0.2";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 # @TEST-EXEC: bro-pkg install grault # @TEST-EXEC: btest-diff scripts/packages/grault/__load__.bro # @TEST-EXEC: bro-pkg remove grault # @TEST-EXEC: bro-pkg install grault --version=1.0.1 # @TEST-EXEC: cp scripts/packages/grault/__load__.bro grault.1.0.1 # @TEST-EXEC: btest-diff grault.1.0.1 package-manager-1.3.3/testing/tests/list0000644000175000017500000000167313241343432017612 0ustar rhaistrhaist# @TEST-EXEC: bro-pkg list all > all.out # @TEST-EXEC: btest-diff all.out # @TEST-EXEC: bro-pkg install foo bar # @TEST-EXEC: bro-pkg unload bar # @TEST-EXEC: bro-pkg list installed > installed.out # @TEST-EXEC: btest-diff installed.out # @TEST-EXEC: bro-pkg list > list.out # @TEST-EXEC: cmp list.out installed.out # @TEST-EXEC: bro-pkg list loaded > loaded.out # @TEST-EXEC: btest-diff loaded.out # @TEST-EXEC: bro-pkg list unloaded > unloaded.out # @TEST-EXEC: btest-diff unloaded.out # @TEST-EXEC: bro-pkg list not_installed > not_installed.out # @TEST-EXEC: btest-diff not_installed.out # @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg refresh # @TEST-EXEC: bro-pkg list outdated > outdated.out # @TEST-EXEC: btest-diff outdated.out # @TEST-EXEC: bro-pkg upgrade # @TEST-EXEC: bro-pkg list outdated > after_upgrade.out # @TEST-EXEC: btest-diff after_upgrade.out cd packages/foo echo 'print "hello";' >> __load__.bro git commit -am 'new stuff' package-manager-1.3.3/testing/tests/test0000644000175000017500000000132713241343432017612 0ustar rhaistrhaist# @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg test rot13 # @TEST-EXEC: bash we_need_to_go_deeper # @TEST-EXEC-FAIL: bro-pkg test rot13 echo "$(pwd)/packages/rot13" >> sources/one/bob/bro-pkg.index cd sources/one git commit -am 'add rot13 package' cd ../../packages/rot13 echo 'depends = bar *' >> bro-pkg.meta echo -e "@load bar\n$(cat scripts/Demo/Rot13/__load__.bro)" > scripts/Demo/Rot13/__load__.bro cd testing/Baseline/tests.main echo "rot13 plugin is loaded" > output echo "bar loaded" >> output echo "rot13 script is loaded" >> output git commit -am 'new stuff' @TEST-START-FILE we_need_to_go_deeper cd packages/rot13 echo 'hello' > testing/Baseline/tests.rot13/output git commit -am 'new stuff' @TEST-END-FILE package-manager-1.3.3/testing/tests/pin0000644000175000017500000000270413241343432017421 0ustar rhaistrhaist# @TEST-EXEC: bash setup_foo # @TEST-EXEC: bro-pkg install foo bar # @TEST-EXEC: bro-pkg pin foo bar # @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg refresh # @TEST-EXEC: bro-pkg upgrade # @TEST-EXEC: btest-diff scripts/packages/foo/__load__.bro # @TEST-EXEC: btest-diff scripts/packages/bar/__load__.bro # @TEST-EXEC: bro-pkg unpin foo bar # @TEST-EXEC: bro-pkg upgrade # @TEST-EXEC: cp scripts/packages/foo/__load__.bro foo_after_unpin.out # @TEST-EXEC: cp scripts/packages/bar/__load__.bro bar_after_unpin.out # @TEST-EXEC: btest-diff foo_after_unpin.out # @TEST-EXEC: btest-diff bar_after_unpin.out @TEST-START-FILE setup_foo cd packages/foo echo 'print "foo 1.0.0";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "foo 1.0.1";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "foo 1.0.2";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 @TEST-END-FILE cd packages/foo echo 'print "foo 1.0.3";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.3 -m 1.0.3 echo 'print "foo 1.0.4";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.4 -m 1.0.4 echo 'print "foo master";' > __load__.bro git commit -am 'new stuff' cd ../bar echo 'print "bar+ loaded";' > __load__.bro git commit -am 'new stuff' echo 'print "bar 1.0.0 loaded";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "bar++ loaded";' > __load__.bro git commit -am 'new stuff' package-manager-1.3.3/testing/tests/metadata-depends0000644000175000017500000000266613241343432022042 0ustar rhaistrhaist# @TEST-EXEC: bro-pkg install foo # @TEST-EXEC: bro-pkg list installed > no_depends.out # @TEST-EXEC: btest-diff no_depends.out # @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg refresh # @TEST-EXEC: bro-pkg upgrade foo # @TEST-EXEC: bro-pkg list installed > upgrade_depends.out # @TEST-EXEC: btest-diff upgrade_depends.out # @TEST-EXEC: bro-pkg purge # @TEST-EXEC: bro-pkg install foo # @TEST-EXEC: bro-pkg list installed > install_depends.out # @TEST-EXEC: btest-diff install_depends.out # @TEST-EXEC: bro-pkg purge # @TEST-EXEC: bro-pkg bundle test.bundle --manifest foo # @TEST-EXEC: bro-pkg unbundle test.bundle # @TEST-EXEC: bro-pkg list installed > bundle_depends.out # @TEST-EXEC: btest-diff bundle_depends.out cd packages/foo echo 'depends = bar *' >> bro-pkg.meta git commit -am 'new stuff' cd ../bar echo 'depends = baz >=1.0.0' >> bro-pkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 cd ../baz echo 'depends = grault branch=master' >> bro-pkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "2.0.0";' > __load__.bro git commit -am 'new stuff' git tag -a 2.0.0 -m 2.0.0 cd ../grault echo 'depends = corge ==1.0.1' >> bro-pkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 cd ../corge git tag -a 1.0.0 -m 1.0.0 echo 'depends = foo * bar *' >> bro-pkg.meta git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "2.0.0";' > __load__.bro git commit -am 'new stuff' git tag -a 2.0.0 -m 2.0.0 package-manager-1.3.3/testing/tests/plugin0000644000175000017500000000060613241343432020130 0ustar rhaistrhaist# @TEST-REQUIRES: type bro-config # @TEST-REQUIRES: test -n "$(bro-config --bro_dist)" # @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg install rot13 # @TEST-EXEC: btest-diff plugins/packages/rot13/__bro_plugin__ # @TEST-EXEC: btest-diff scripts/packages/rot13/__load__.bro echo "$(pwd)/packages/rot13" >> sources/one/bob/bro-pkg.index cd sources/one git commit -am 'add rot13 package' package-manager-1.3.3/testing/tests/bundle0000644000175000017500000000264513241343432020110 0ustar rhaistrhaist# @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg install foo baz # @TEST-EXEC: bro-pkg install bar --version 1.0.0 # @TEST-EXEC: bro-pkg bundle test.bundle # @TEST-EXEC: bro-pkg purge # @TEST-EXEC: bro-pkg unbundle test.bundle # @TEST-EXEC: bro-pkg list installed > snapshot.out # @TEST-EXEC: btest-diff snapshot.out # @TEST-EXEC: bro-pkg purge # @TEST-EXEC: bro-pkg bundle test.bundle --manifest foo bar baz # @TEST-EXEC: bro-pkg unbundle test.bundle # @TEST-EXEC: bro-pkg list installed > args.out # @TEST-EXEC: btest-diff args.out # @TEST-EXEC: bro-pkg purge # @TEST-EXEC: bro-pkg bundle test.bundle --manifest manifest.txt # @TEST-EXEC: bro-pkg unbundle test.bundle # @TEST-EXEC: bro-pkg list installed > manifest.out # @TEST-EXEC: btest-diff manifest.out cd packages/foo echo 'print "foo 1.0.0";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "foo 1.0.1";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "foo 1.0.2";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 cd ../bar echo 'print "bar 1.0.0";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "bar 1.0.1";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "bar 1.0.2";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 @TEST-START-FILE manifest.txt [bundle] foo = 1.0.0 bar = 1.0.1 baz = master @TEST-END-FILE package-manager-1.3.3/testing/tests/upgrade0000644000175000017500000000216713241343432020265 0ustar rhaistrhaist# @TEST-EXEC: bash setup_foo # @TEST-EXEC: bro-pkg install foo bar # @TEST-EXEC: bash %INPUT # @TEST-EXEC: bro-pkg refresh # @TEST-EXEC: bro-pkg upgrade # @TEST-EXEC: btest-diff scripts/packages/foo/__load__.bro # @TEST-EXEC: btest-diff scripts/packages/bar/__load__.bro @TEST-START-FILE setup_foo cd packages/foo echo 'print "foo 1.0.0";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "foo 1.0.1";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "foo 1.0.2";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 @TEST-END-FILE cd packages/foo echo 'print "foo 1.0.3";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.3 -m 1.0.3 echo 'print "foo 1.0.4";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.4 -m 1.0.4 echo 'print "foo master";' > __load__.bro git commit -am 'new stuff' cd ../bar echo 'print "bar+ loaded";' > __load__.bro git commit -am 'new stuff' echo 'print "bar 1.0.0 loaded";' > __load__.bro git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "bar++ loaded";' > __load__.bro git commit -am 'new stuff' package-manager-1.3.3/testing/.gitignore0000644000175000017500000000004013241343432017525 0ustar rhaistrhaistdiag.log .btest.failed.dat .tmp package-manager-1.3.3/testing/baselines/0000755000175000017500000000000013241343432017510 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.refresh/0000755000175000017500000000000013241343432022307 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.refresh/outdated.out0000644000175000017500000000004213241343432024645 0ustar rhaistrhaistone/alice/foo (installed: master) package-manager-1.3.3/testing/baselines/tests.refresh/list_after_agg.out0000644000175000017500000000024513241343432026013 0ustar rhaistrhaistone/alice/bar one/alice/baz one/alice/foo (installed: master) one/alice/new_pkg - This is the new_pkg package description one/alice/qux one/bob/corge one/bob/grault package-manager-1.3.3/testing/baselines/tests.refresh/list.out0000644000175000017500000000017313241343432024014 0ustar rhaistrhaistone/alice/bar one/alice/baz one/alice/foo (installed: master) one/alice/new_pkg one/alice/qux one/bob/corge one/bob/grault package-manager-1.3.3/testing/baselines/tests.refresh/search.out0000644000175000017500000000020413241343432024301 0ustar rhaistrhaistone/alice/foo (installed: master) - This is the foo package description one/alice/new_pkg - This is the new_pkg package description package-manager-1.3.3/testing/baselines/tests.upgrade/0000755000175000017500000000000013241343432022300 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.upgrade/scripts.packages.foo.__load__.bro0000644000175000017500000000002313241343432030537 0ustar rhaistrhaistprint "foo 1.0.4"; package-manager-1.3.3/testing/baselines/tests.upgrade/scripts.packages.bar.__load__.bro0000644000175000017500000000002613241343432030523 0ustar rhaistrhaistprint "bar++ loaded"; package-manager-1.3.3/testing/baselines/tests.install/0000755000175000017500000000000013241343432022317 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.install/scripts.packages.baz.__load__.bro0000644000175000017500000000002413241343432030550 0ustar rhaistrhaistprint "baz loaded"; package-manager-1.3.3/testing/baselines/tests.install/scripts.packages.foo.__load__.bro0000644000175000017500000000002413241343432030557 0ustar rhaistrhaistprint "foo loaded"; package-manager-1.3.3/testing/baselines/tests.install/scripts.packages.grault.__load__.bro0000644000175000017500000000002613241343432031274 0ustar rhaistrhaistprint "grault 1.0.2"; package-manager-1.3.3/testing/baselines/tests.install/scripts.packages.bar.__load__.bro0000644000175000017500000000005113241343432030540 0ustar rhaistrhaistevent bro_init() { print "bar loaded"; } package-manager-1.3.3/testing/baselines/tests.install/corge.master0000644000175000017500000000004513241343432024632 0ustar rhaistrhaistprint "corge loaded"; print "hello"; package-manager-1.3.3/testing/baselines/tests.install/scripts.packages.corge.__load__.bro0000644000175000017500000000002613241343432031075 0ustar rhaistrhaistprint "corge loaded"; package-manager-1.3.3/testing/baselines/tests.install/scripts.packages.qux.__load__.bro0000644000175000017500000000002413241343432030611 0ustar rhaistrhaistprint "qux loaded"; package-manager-1.3.3/testing/baselines/tests.install/grault.1.0.10000644000175000017500000000002613241343432024172 0ustar rhaistrhaistprint "grault 1.0.1"; package-manager-1.3.3/testing/baselines/tests.metadata-script_dir/0000755000175000017500000000000013241343432024571 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.metadata-script_dir/scripts.foo.__load__.bro0000644000175000017500000000002413241343432031254 0ustar rhaistrhaistprint "foo loaded"; package-manager-1.3.3/testing/baselines/tests.load/0000755000175000017500000000000013241343432021570 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.load/after_unload.out0000644000175000017500000000014113241343432024760 0ustar rhaistrhaist# WARNING: This file is managed by bro-pkg. # Do not make direct modifications here. @load ./bar package-manager-1.3.3/testing/baselines/tests.load/scripts.packages.packages.bro0000644000175000017500000000015513241343432027316 0ustar rhaistrhaist# WARNING: This file is managed by bro-pkg. # Do not make direct modifications here. @load ./bar @load ./foo package-manager-1.3.3/testing/baselines/tests.load/after_load.out0000644000175000017500000000015513241343432024422 0ustar rhaistrhaist# WARNING: This file is managed by bro-pkg. # Do not make direct modifications here. @load ./bar @load ./foo package-manager-1.3.3/testing/baselines/tests.purge/0000755000175000017500000000000013241343432021773 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.purge/scripts.packages.packages.bro0000644000175000017500000000012513241343432027516 0ustar rhaistrhaist# WARNING: This file is managed by bro-pkg. # Do not make direct modifications here. package-manager-1.3.3/testing/baselines/tests.bundle/0000755000175000017500000000000013241343432022122 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.bundle/manifest.out0000644000175000017500000000014413241343432024460 0ustar rhaistrhaistone/alice/bar (installed: 1.0.1) one/alice/baz (installed: master) one/alice/foo (installed: 1.0.0) package-manager-1.3.3/testing/baselines/tests.bundle/snapshot.out0000644000175000017500000000014413241343432024511 0ustar rhaistrhaistone/alice/bar (installed: 1.0.0) one/alice/baz (installed: master) one/alice/foo (installed: 1.0.2) package-manager-1.3.3/testing/baselines/tests.bundle/args.out0000644000175000017500000000014413241343432023606 0ustar rhaistrhaistone/alice/bar (installed: 1.0.2) one/alice/baz (installed: master) one/alice/foo (installed: 1.0.2) package-manager-1.3.3/testing/baselines/tests.info/0000755000175000017500000000000013241343432021604 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.info/foo.info0000644000175000017500000000037413241343432023250 0ustar rhaistrhaist"one/alice/foo" info: versions: ['1.0.0', '1.0.1', '1.0.2'] install status: current_version = master is_loaded = True is_outdated = True is_pinned = False tracking_method = branch metadata (from version "master"): package-manager-1.3.3/testing/baselines/tests.info/bar.info0000644000175000017500000000040313241343432023222 0ustar rhaistrhaist"one/alice/bar" info: versions: ['1.0.0'] install status: current_version = 1.0.0 is_loaded = True is_outdated = False is_pinned = False tracking_method = version metadata (from version "1.0.0"): description = bar bar bar tags = sega sunset package-manager-1.3.3/testing/baselines/tests.list/0000755000175000017500000000000013241343432021624 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.list/outdated.out0000644000175000017500000000004213241343432024162 0ustar rhaistrhaistone/alice/foo (installed: master) package-manager-1.3.3/testing/baselines/tests.list/all.out0000644000175000017500000000012513241343432023123 0ustar rhaistrhaistone/alice/bar one/alice/baz one/alice/foo one/alice/qux one/bob/corge one/bob/grault package-manager-1.3.3/testing/baselines/tests.list/unloaded.out0000644000175000017500000000004213241343432024144 0ustar rhaistrhaistone/alice/bar (installed: master) package-manager-1.3.3/testing/baselines/tests.list/not_installed.out0000644000175000017500000000007113241343432025212 0ustar rhaistrhaistone/alice/baz one/alice/qux one/bob/corge one/bob/grault package-manager-1.3.3/testing/baselines/tests.list/installed.out0000644000175000017500000000010413241343432024327 0ustar rhaistrhaistone/alice/bar (installed: master) one/alice/foo (installed: master) package-manager-1.3.3/testing/baselines/tests.list/loaded.out0000644000175000017500000000004213241343432023601 0ustar rhaistrhaistone/alice/foo (installed: master) package-manager-1.3.3/testing/baselines/tests.list/after_upgrade.out0000644000175000017500000000000013241343432025153 0ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.user_vars/0000755000175000017500000000000013241343432022662 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.user_vars/state.logs.foo-build.log0000644000175000017500000000010013241343432027316 0ustar rhaistrhaist=== STDERR === === STDOUT === /home/jon/sandbox /usr/local /usr package-manager-1.3.3/testing/baselines/tests.remove/0000755000175000017500000000000013241343432022146 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.remove/scripts.packages.packages.bro0000644000175000017500000000015513241343432027674 0ustar rhaistrhaist# WARNING: This file is managed by bro-pkg. # Do not make direct modifications here. @load ./bar @load ./baz package-manager-1.3.3/testing/baselines/tests.metadata-suggests/0000755000175000017500000000000013241343432024273 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.metadata-suggests/bundle_suggests.out0000644000175000017500000000025013241343432030216 0ustar rhaistrhaistone/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: master) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-1.3.3/testing/baselines/tests.metadata-suggests/upgrade_suggests.out0000644000175000017500000000025013241343432030374 0ustar rhaistrhaistone/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: master) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-1.3.3/testing/baselines/tests.metadata-suggests/install_suggests.out0000644000175000017500000000025013241343432030413 0ustar rhaistrhaistone/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: master) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-1.3.3/testing/baselines/tests.metadata-suggests/no_suggests.out0000644000175000017500000000004213241343432027360 0ustar rhaistrhaistone/alice/foo (installed: master) package-manager-1.3.3/testing/baselines/tests.metadata-config_files/0000755000175000017500000000000013241343432025056 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.metadata-config_files/scripts.foo.config.bro0000644000175000017500000000002113241343432031270 0ustar rhaistrhaistprint "foo new"; package-manager-1.3.3/testing/baselines/tests.metadata-config_files/foo_backup.bro0000644000175000017500000000002113241343432027663 0ustar rhaistrhaistprint "foo mod"; package-manager-1.3.3/testing/baselines/tests.metadata-config_files/barconfig2.bro0000644000175000017500000000002313241343432027571 0ustar rhaistrhaistprint "bar 2.0.0"; package-manager-1.3.3/testing/baselines/tests.metadata-config_files/bar_backup.bro0000644000175000017500000000002113241343432027644 0ustar rhaistrhaistprint "bar mod"; package-manager-1.3.3/testing/baselines/tests.metadata-config_files/scripts.bar.config.bro0000644000175000017500000000002113241343432031251 0ustar rhaistrhaistprint "bar mod"; package-manager-1.3.3/testing/baselines/tests.metadata-depends/0000755000175000017500000000000013241343432024051 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.metadata-depends/upgrade_depends.out0000644000175000017500000000025013241343432027730 0ustar rhaistrhaistone/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: master) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-1.3.3/testing/baselines/tests.metadata-depends/install_depends.out0000644000175000017500000000025013241343432027747 0ustar rhaistrhaistone/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: master) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-1.3.3/testing/baselines/tests.metadata-depends/bundle_depends.out0000644000175000017500000000025013241343432027552 0ustar rhaistrhaistone/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: master) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-1.3.3/testing/baselines/tests.metadata-depends/no_depends.out0000644000175000017500000000004213241343432026714 0ustar rhaistrhaistone/alice/foo (installed: master) package-manager-1.3.3/testing/baselines/tests.pin/0000755000175000017500000000000013241343432021437 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.pin/scripts.packages.foo.__load__.bro0000644000175000017500000000002313241343432027676 0ustar rhaistrhaistprint "foo 1.0.2"; package-manager-1.3.3/testing/baselines/tests.pin/scripts.packages.bar.__load__.bro0000644000175000017500000000005113241343432027660 0ustar rhaistrhaistevent bro_init() { print "bar loaded"; } package-manager-1.3.3/testing/baselines/tests.pin/bar_after_unpin.out0000644000175000017500000000002613241343432025324 0ustar rhaistrhaistprint "bar++ loaded"; package-manager-1.3.3/testing/baselines/tests.pin/foo_after_unpin.out0000644000175000017500000000002313241343432025340 0ustar rhaistrhaistprint "foo 1.0.4"; package-manager-1.3.3/testing/baselines/tests.plugin_tarfile/0000755000175000017500000000000013241343432023655 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.plugin_tarfile/scripts.packages.rot13.__load__.bro0000644000175000017500000000031413241343432032304 0ustar rhaistrhaist# # This is loaded when a user activates the plugin. Include scripts here that should be # loaded automatically at that point. # # @load ./bar.bro event bro_init() { print "rot13 script is loaded"; } package-manager-1.3.3/testing/baselines/tests.plugin_tarfile/plugins.packages.rot13.__bro_plugin__0000644000175000017500000000001413241343432032733 0ustar rhaistrhaistDemo::Rot13 package-manager-1.3.3/testing/baselines/tests.plugin/0000755000175000017500000000000013241343432022147 5ustar rhaistrhaistpackage-manager-1.3.3/testing/baselines/tests.plugin/scripts.packages.rot13.__load__.bro0000644000175000017500000000031413241343432030576 0ustar rhaistrhaist# # This is loaded when a user activates the plugin. Include scripts here that should be # loaded automatically at that point. # # @load ./bar.bro event bro_init() { print "rot13 script is loaded"; } package-manager-1.3.3/testing/baselines/tests.plugin/plugins.packages.rot13.__bro_plugin__0000644000175000017500000000001413241343432031225 0ustar rhaistrhaistDemo::Rot13 package-manager-1.3.3/testing/sources/0000755000175000017500000000000013241343432017226 5ustar rhaistrhaistpackage-manager-1.3.3/testing/sources/one/0000755000175000017500000000000013241343432020007 5ustar rhaistrhaistpackage-manager-1.3.3/testing/sources/one/bob/0000755000175000017500000000000013241343432020551 5ustar rhaistrhaistpackage-manager-1.3.3/testing/sources/one/bob/bro-pkg.index0000644000175000017500000000001513241343432023137 0ustar rhaistrhaistcorge grault package-manager-1.3.3/testing/sources/one/alice/0000755000175000017500000000000013241343432021064 5ustar rhaistrhaistpackage-manager-1.3.3/testing/sources/one/alice/bro-pkg.index0000644000175000017500000000002013241343432023446 0ustar rhaistrhaistfoo bar baz qux package-manager-1.3.3/README0000777000175000017500000000000013241343432020110 2doc/overview.rstustar rhaistrhaistpackage-manager-1.3.3/doc/0000755000175000017500000000000013241343432014633 5ustar rhaistrhaistpackage-manager-1.3.3/doc/_static/0000755000175000017500000000000013241343432016261 5ustar rhaistrhaistpackage-manager-1.3.3/doc/_static/theme_overrides.css0000644000175000017500000000057613241343432022167 0ustar rhaistrhaist/* Prevent sphinx-argparse option output (e.g. --version) from wrapping. */ span.option { white-space: nowrap; } /* override table width restrictions */ html body div.wy-table-responsive table td, html body div.wy-table-responsive table th { white-space: normal; } html body div.wy-table-responsive { margin-bottom: 24px; max-width: 100%; overflow: visible; } package-manager-1.3.3/doc/developers.rst0000644000175000017500000000644313241343432017544 0ustar rhaistrhaist.. _Sphinx: http://www.sphinx-doc.org .. _Read the Docs: http://bro-package-manager.readthedocs.io/en/latest .. _GitHub: https://github.com/bro/package-manager .. _Google Style Docstrings: http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html .. _bro-aux: https://github.com/bro/bro-aux .. _PyPi: https://pypi.python.org/pypi Developer's Guide ================= This a guide for developers working on the Bro Package Manager itself. Versioning/Releases ------------------- After making a commit to the *master* branch, you can use the :program:`update-changes` script in the `bro-aux`_ repository to automatically adapt version numbers and regenerate the :program:`bro-pkg` man page. Make sure to install the `documentation dependencies`_ before using it. Releases are hosted at PyPi_. To build and upload a release: #. Finalize the git repo tag and version with ``update-changes -R `` if not done already. #. Upload the distribution (you will need the credentials for the 'bro' account on PyPi): .. code-block:: console $ make upload Documentation ------------- Documentation is written in reStructuredText (reST), which Sphinx_ uses to generate HTML documentation and a man page. .. _documentation dependencies: Dependencies ~~~~~~~~~~~~ To build documentation locally, find the minimal requirements in :file:`requirements.doc.txt`: .. literalinclude:: ../requirements.doc.txt They can be installed like: pip install -r requirements.doc.txt Local Build/Preview ~~~~~~~~~~~~~~~~~~~ Use the :file:`Makefile` targets ``make html`` and ``make man`` to build the HTML and man page, respectively. To view the generated HTML output, open :file:`doc/_build/index.html`. The generated man page is located in :file:`doc/man/bro-pkg.1`. If you have also installed :program:`sphinx-autobuild` (e.g. via :program:`pip`), there's a :file:`Makefile` target, ``make livehtml``, you can use to help preview documentation changes as you edit the reST files. Remote Hosting ~~~~~~~~~~~~~~ The GitHub_ repository has a webhook configured to automatically rebuild the HTML documentation hosted at `Read the Docs`_ whenever a commit is pushed. Style Conventions ~~~~~~~~~~~~~~~~~ The following style conventions are (generally) used. ========================== =============================== =========================== Documentation Subject reST Markup Preview ========================== =============================== =========================== File Path ``:file:`path``` :file:`path` File Path w/ Substitution ``:file:`{}/path``` :file:`{}/path` OS-Level Commands ``:command:`cmd``` :command:`cmd` Program Names ``:program:`prog``` :program:`prog` Environment Variables ``:envvar:`VAR``` :envvar:`VAR` Literal Text (e.g. code) ````code```` ``code`` Substituted Literal Text ``:samp:`code {}``` :samp:`code {}` Variable/Type Name ```x``` `x` INI File Option ```name``` `name` ========================== =============================== =========================== Python API docstrings roughly follow the `Google Style Docstrings`_ format. package-manager-1.3.3/doc/api/0000755000175000017500000000000013241343432015404 5ustar rhaistrhaistpackage-manager-1.3.3/doc/api/index.rst0000644000175000017500000000042513241343432017246 0ustar rhaistrhaistPython API Reference ==================== .. automodule:: bropkg :members: :undoc-members: :show-inheritance: The following Python modules are all provided as part of the ``bropkg`` public interface: .. toctree:: :maxdepth: 2 manager package source package-manager-1.3.3/doc/api/package.rst0000644000175000017500000000020513241343432017526 0ustar rhaistrhaistbropkg.package module ===================== .. automodule:: bropkg.package :members: :undoc-members: :show-inheritance: package-manager-1.3.3/doc/api/source.rst0000644000175000017500000000020213241343432017430 0ustar rhaistrhaistbropkg.source module ==================== .. automodule:: bropkg.source :members: :undoc-members: :show-inheritance: package-manager-1.3.3/doc/api/manager.rst0000644000175000017500000000020513241343432017545 0ustar rhaistrhaistbropkg.manager module ===================== .. automodule:: bropkg.manager :members: :undoc-members: :show-inheritance: package-manager-1.3.3/doc/man/0000755000175000017500000000000013241343432015406 5ustar rhaistrhaistpackage-manager-1.3.3/doc/man/bro-pkg.10000644000175000017500000005706113241343432017042 0ustar rhaistrhaist.\" Man page generated from reStructuredText. . .TH "BRO-PKG" "1" "Feb 15, 2018" "1.3.3" "Bro Package Manager" .SH NAME bro-pkg \- Bro Package Manager . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .sp A command\-line package manager for Bro. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg [\-h] [\-\-version] [\-\-configfile CONFIGFILE] [\-\-verbose] {test,install,bundle,unbundle,remove,purge,refresh,upgrade,load,unload,pin,unpin,list,search,info,config,autoconfig,env} ... .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Options: .INDENT 7.0 .TP .B \-\-version show program\(aqs version number and exit .TP .B \-\-configfile Path to Bro Package Manager config file. .sp See \fI\%Config File\fP\&. .TP .B \-\-verbose=0\fP,\fB \-v=0 Increase program output for debugging. Use multiple times for more output (e.g. \-vvv). .UNINDENT .UNINDENT .sp Environment Variables: .INDENT 0.0 .INDENT 3.5 \fBBRO_PKG_CONFIG_FILE\fP: Same as \fB\-\-configfile\fP option, but has less precedence. .UNINDENT .UNINDENT .SH COMMANDS .SS test .sp Runs the unit tests for the specified Bro packages. In most cases, the "bro" and "bro\-config" programs will need to be in PATH before running this command. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg test [\-h] [\-\-version VERSION] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bpackage The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "bro" that has a package named "foo" located in "alice/bro\-pkg.index" the following names work: "foo", "alice/foo", "bro/alice/foo". .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-version The version of the package to test. Only one package may be specified at a time when using this flag. A version tag or branch name may be specified here. If the package name refers to a local git repo with a working tree, then its currently active branch is used. The default for other cases is to use the latest version tag, or if a package has none, the "master" branch. .UNINDENT .UNINDENT .SS install .sp Installs packages from a configured package source or directly from a git URL. After installing, the package is marked as being "loaded" (see the \fBload\fP command). .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg install [\-h] [\-\-force] [\-\-skiptests] [\-\-nodeps] [\-\-nosuggestions] [\-\-version VERSION] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bpackage The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "bro" that has a package named "foo" located in "alice/bro\-pkg.index" the following names work: "foo", "alice/foo", "bro/alice/foo". .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-force=False Skip the confirmation prompt. .TP .B \-\-skiptests=False Skip running unit tests for packages before installation. .TP .B \-\-nodeps=False Skip all dependency resolution/checks. Note that using this option risks putting your installed package collection into a broken or unusable state. .TP .B \-\-nosuggestions=False Skip automatically installing suggested packages. .TP .B \-\-version The version of the package to install. Only one package may be specified at a time when using this flag. A version tag or branch name may be specified here. If the package name refers to a local git repo with a working tree, then its currently active branch is used. The default for other cases is to use the latest version tag, or if a package has none, the "master" branch. .UNINDENT .UNINDENT .SS remove .sp Unloads (see the \fBunload\fP command) and uninstalls a previously installed package. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg remove [\-h] [\-\-force] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bpackage The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "bro" that has a package named "foo" located in "alice/bro\-pkg.index" the following names work: "foo", "alice/foo", "bro/alice/foo". .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-force=False Skip the confirmation prompt. .UNINDENT .UNINDENT .SS purge .sp Unloads (see the \fBunload\fP command) and uninstalls all previously installed packages. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg purge [\-h] [\-\-force] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Options: .INDENT 7.0 .TP .B \-\-force=False Skip the confirmation prompt. .UNINDENT .UNINDENT .SS bundle .sp This command creates a bundle file containing a collection of Bro packages. If \fB\-\-manifest\fP is used, the user suplies the list of packages to put in the bundle, else all currently installed packages are put in the bundle. A bundle file can be unpacked on any target system, resulting in a repeatable/specific set of packages being installed on that target system (see the \fBunbundle\fP command). This command may be useful for those that want to manage packages on a system that otherwise has limited network connectivity. E.g. one can use a system with an internet connection to create a bundle, transport that bundle to the target machine using whatever means are appropriate, and finally unbundle/install it on the target machine. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg bundle [\-h] [\-\-force] [\-\-nodeps] [\-\-nosuggestions] [\-\-manifest MANIFEST [MANIFEST ...]] bundle_filename .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bbundle_filename The path of the bundle file to create. It will be overwritten if it already exists. .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-force=False Skip the confirmation prompt. .TP .B \-\-nodeps=False Skip all dependency resolution/checks. Note that using this option risks creating a bundle of packages that is in a broken or unusable state. .TP .B \-\-nosuggestions=False Skip automatically bundling suggested packages. .TP .B \-\-manifest This may either be a file name or a list of packages to include in the bundle. If a file name is supplied, it should be in INI format with a single \(ga\(ga[bundle]\(ga\(ga section. The keys in that section correspond to package names and their values correspond to git version tags or branch names. The values may be left blank to indicate that the latest available version should be used. .UNINDENT .UNINDENT .SS unbundle .sp This command unpacks a bundle file formerly created by the \fBbundle\fP command and installs all the packages contained within. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg unbundle [\-h] [\-\-force] [\-\-replace] bundle_filename .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bbundle_filename The path of the bundle file to install. .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-force=False Skip the confirmation prompt. .TP .B \-\-replace=False Using this flag first removes all installed packages before then installing the packages from the bundle. .UNINDENT .UNINDENT .SS refresh .sp Retrieve latest package metadata from sources and checks whether any installed packages have available upgrades. Note that this does not actually upgrade any packages (see the \fBupgrade\fP command for that). .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg refresh [\-h] [\-\-aggregate] [\-\-push] [\-\-sources SOURCES [SOURCES ...]] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Options: .INDENT 7.0 .TP .B \-\-aggregate=False Crawls the urls listed in package source bro\-pkg.index files and aggregates the metadata found in their bro\-pkg.meta files. The aggregated metadata is stored in the local clone of the package source that bro\-pkg uses internally locating package metadata. For each package, the metadata is taken from the highest available git version tag or the master branch if no version tags exist .TP .B \-\-push=False Push all local changes to package sources to upstream repos .TP .B \-\-sources A list of package source names to operate on. If this argument is not used, then the command will operate on all configured sources. .UNINDENT .UNINDENT .SS upgrade .sp Uprades the specified package(s) to latest available version. If no specific packages are specified, then all installed packages that are outdated and not pinned are upgraded. For packages that are installed with \fB\-\-version\fP using a git branch name, the package is updated to the latest commit on that branch, else the package is updated to the highest available git version tag. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg upgrade [\-h] [\-\-force] [\-\-skiptests] [\-\-nodeps] [\-\-nosuggestions] [package [package ...]] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bpackage The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "bro" that has a package named "foo" located in "alice/bro\-pkg.index" the following names work: "foo", "alice/foo", "bro/alice/foo". .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-force=False Skip the confirmation prompt. .TP .B \-\-skiptests=False Skip running unit tests for packages before installation. .TP .B \-\-nodeps=False Skip all dependency resolution/checks. Note that using this option risks putting your installed package collection into a broken or unusable state. .TP .B \-\-nosuggestions=False Skip automatically installing suggested packages. .UNINDENT .UNINDENT .SS load .sp The Bro Package Manager keeps track of all packages that are marked as "loaded" and maintains a single Bro script that, when loaded by Bro (e.g. via \fB@load packages\fP), will load the scripts from all "loaded" packages at once. This command adds a set of packages to the "loaded packages" list. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg load [\-h] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bpackage Name(s) of package(s) to load. .UNINDENT .UNINDENT .SS unload .sp The Bro Package Manager keeps track of all packages that are marked as "loaded" and maintains a single Bro script that, when loaded by Bro, will load the scripts from all "loaded" packages at once. This command removes a set of packages from the "loaded packages" list. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg unload [\-h] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bpackage The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "bro" that has a package named "foo" located in "alice/bro\-pkg.index" the following names work: "foo", "alice/foo", "bro/alice/foo". .UNINDENT .UNINDENT .SS pin .sp Pinned packages are ignored by the \fBupgrade\fP command. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg pin [\-h] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bpackage The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "bro" that has a package named "foo" located in "alice/bro\-pkg.index" the following names work: "foo", "alice/foo", "bro/alice/foo". .UNINDENT .UNINDENT .SS unpin .sp Packages that are not pinned are automatically upgraded by the \fBupgrade\fP command .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg unpin [\-h] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bpackage The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "bro" that has a package named "foo" located in "alice/bro\-pkg.index" the following names work: "foo", "alice/foo", "bro/alice/foo". .UNINDENT .UNINDENT .SS list .sp Outputs a list of packages that match a given category. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg list [\-h] [\-\-nodesc] [{all,installed,not_installed,loaded,unloaded,outdated}] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bcategory Package category used to filter listing. .sp Possible choices: all, installed, not_installed, loaded, unloaded, outdated .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-nodesc=False Do not display description text, just the package name(s). .UNINDENT .UNINDENT .SS search .sp Perform a substring search on package names and metadata tags. Surround search text with slashes to indicate it is a regular expression (e.g. \fB/text/\fP). .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg search [\-h] search_text [search_text ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bsearch_text The text(s) or pattern(s) to look for. .UNINDENT .UNINDENT .SS info .sp Shows detailed information/metadata for given packages. If the package is currently installed, additional information about the status of it is displayed. E.g. the installed version or whether it is currently marked as "pinned" or "loaded." .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg info [\-h] [\-\-version VERSION] [\-\-nolocal] [\-\-json] [\-\-jsonpretty SPACES] [\-\-allvers] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bpackage The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "bro" that has a package named "foo" located in "alice/bro\-pkg.index" the following names work: "foo", "alice/foo", "bro/alice/foo". .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-version The version of the package metadata to inspect. A version tag, branch name, or commit hash and only one package at a time may be given when using this flag. If unspecified, the behavior depends on whether the package is currently installed. If installed, the metadata will be pulled from the installed version. If not installed, the latest version tag is used, or if a package has no version tags, the "master" branch is used. .TP .B \-\-nolocal=False Do not read information from locally installed packages. Instead read info from remote GitHub. .TP .B \-\-json=False Output package information as JSON. .TP .B \-\-jsonpretty Optional number of spaces to indent for pretty\-printed JSON output. .TP .B \-\-allvers=False When outputting package information as JSON, show metadata for all versions. This option can be slow since remote repositories may be cloned multiple times. Also, installed packages will show metadata only for the installed version unless the \-\-nolocal option is given. .UNINDENT .UNINDENT .SS config .sp The default output of this command is a valid package manager config file that corresponds to the one currently being used, but also with any defaulted field values filled in. This command also allows for only the value of a specific field to be output if the name of that field is given as an argument to the command. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg config [\-h] [{all,sources,user_vars,state_dir,script_dir,plugin_dir,bro_dist}] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .Bconfig_param Name of a specific config file field to output. .sp Possible choices: all, sources, user_vars, state_dir, script_dir, plugin_dir, bro_dist .UNINDENT .UNINDENT .SS autoconfig .sp The output of this command is a valid package manager config file that is generated by using the \fBbro\-config\fP script that is installed along with Bro. It is the suggested configuration to use for most Bro installations. For this command to work, the \fBbro\-config\fP script must be in \fBPATH\fP\&. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg autoconfig [\-h] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .UNINDENT .SS env .sp This command returns shell commands that, when executed, will correctly set \fBBROPATH\fP and \fBBRO_PLUGIN_PATH\fP to utilize the scripts and plugins from packages installed by the package manager. For this command to function properly, either have the \fBbro\-config\fP script (installed by bro) in \fBPATH\fP, or have the \fBBROPATH\fP and \fBBRO_PLUGIN_PATH\fP environment variables already set so this command can append package\-specific paths to them. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: bro\-pkg env [\-h] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .UNINDENT .SH CONFIG FILE .sp The \fBbro\-pkg\fP command\-line tool uses an INI\-format config file to allow users to customize their Package Sources, Package installation paths, Bro executable/source paths, and other \fBbro\-pkg\fP options. .sp See the default/example config file below for explanations of the available options and how to customize them: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C # This is an example config file for bro\-pkg to explain what # settings are possible as well as their default values. # The order of precedence for how bro\-pkg finds/reads config files: # # (1) bro\-pkg \-\-configfile=/path/to/custom/config # (2) the BRO_PKG_CONFIG_FILE environment variable # (3) a config file located at $HOME/.bro\-pkg/config # (4) if none of the above exist, then bro\-pkg uses builtin/default # values for all settings shown below [sources] # The default package source repository from which bro\-pkg fetches # packages. The default source may be removed, changed, or # additional sources may be added as long as they use a unique key # and a value that is a valid git URL. bro = https://github.com/bro/packages [paths] # Directory where source repositories are cloned, packages are # installed, and other package manager state information is # maintained. If left blank, this defaults to $HOME/.bro\-pkg state_dir = # The directory where package scripts are copied upon installation. # A subdirectory named "packages" is always created within the # specified path and the package manager will copy the directory # specified by the "script_dir" option of each package\(aqs bro\-pkg.meta # file there. # If left blank, this defaults to /script_dir # A typical path to set here is /share/bro/site # If you decide to change this location after having already # installed packages, bro\-pkg will automatically relocate them # the next time you run any bro\-pkg command. script_dir = # The directory where package plugins are copied upon installation. # A subdirectory named "packages" is always created within the # specified path and the package manager will copy the directory # specified by the "plugin_dir" option of each package\(aqs bro\-pkg.meta # file there. # If left blank, this defaults to /plugin_dir # A typical path to set here is /lib/bro/plugins # If you decide to change this location after having already # installed packages, bro\-pkg will automatically relocate them # the next time you run any bro\-pkg command. plugin_dir = # The directory containing Bro distribution source code. This is only # needed when installing packages that contain Bro plugins that are # not pre\-built. bro_dist = [user_vars] # For any key in this section that is matched for value interpolation # in a package\(aqs bro\-pkg.meta file, the corresponding value is # substituted during execution of the package\(aqs \(gabuild_command\(ga. # This section is typically automatically populated with the # the answers supplied during package installation prompts # and, as a convenience feature, used to recall the last\-used settings # during subsequent operations (e.g. upgrades) on the same package. .ft P .fi .UNINDENT .UNINDENT .SH AUTHOR The Bro Project .SH COPYRIGHT 2016, The Bro Project .\" Generated by docutils manpage writer. . package-manager-1.3.3/doc/index.rst0000644000175000017500000000022413241343432016472 0ustar rhaistrhaist.. include:: overview.rst .. toctree:: :hidden: :numbered: quickstart bro-pkg package source api/index developers package-manager-1.3.3/doc/quickstart.rst0000644000175000017500000001517413241343432017567 0ustar rhaistrhaist.. _PyPI: https://pypi.python.org/pypi .. _BroControl: https://www.bro.org/sphinx/components/broctl/README.html Quickstart Guide ================ Dependencies ------------ * Python 2.7+ or 3.0+ * git: https://git-scm.com * GitPython: https://pypi.python.org/pypi/GitPython * semantic_version: https://pypi.python.org/pypi/semantic_version * btest: https://pypi.python.org/pypi/btest * configparser backport (not needed when using Python 3.5+): https://pypi.python.org/pypi/configparser Note that following the suggested `Installation`_ process via :program:`pip` will automatically install dependencies for you. Installation ------------ Using the latest stable release on PyPI_: .. code-block:: console $ pip install bro-pkg Using the latest git development version: .. code-block:: console $ pip install git+git://github.com/bro/package-manager@master Basic Configuration ------------------- After installing via :program:`pip`, additional configuration is required. First, make sure that the :program:`bro-config` script that gets installed with :program:`bro` is in your :envvar:`PATH`. Then, as the user you want to run :program:`bro-pkg` with, do: .. code-block:: console $ bro-pkg autoconfig This automatically generates a config file with the following suggested settings that should work for most Bro deployments: - `script_dir`: set to the location of Bro's :file:`site` scripts directory (e.g. :file:`{}/share/bro/site`) - `plugin_dir`: set to the location of Bro's default plugin directory (e.g. :file:`{}/lib/bro/plugins`) - `bro_dist`: set to the location of Bro's source code. If you didn't build/install Bro from source code, this field will not be set, but it's only needed if you plan on installing packages that have uncompiled Bro plugins. With those settings, the package manager will install Bro scripts, Bro plugins, and BroControl plugins into directories where :program:`bro` and :program:`broctl` will, by default, look for them. BroControl clusters will also automatically distribute installed package scripts/plugins to all nodes. .. note:: If your Bro installation is owned by "root" and you intend to run :program:`bro-pkg` as a different user, then you should grant "write" access to the directories specified by `script_dir` and `plugin_dir`. E.g. you could do something like: .. code-block:: console $ sudo chgrp $USER $(bro-config --site_dir) $(bro-config --plugin_dir) $ sudo chmod g+rwX $(bro-config --site_dir) $(bro-config --plugin_dir) The final step is to edit your :file:`site/local.bro`. If you want to have Bro automatically load the scripts from all :ref:`installed ` packages that are also marked as ":ref:`loaded `" add: .. code-block:: bro @load packages If you prefer to manually pick the package scripts to load, you may instead add lines like :samp:`@load {}`, where :samp:`{}` is the :ref:`shorthand name ` of the desired package. If you want to further customize your configuration, see the `Advanced Configuration`_ section and also check :ref:`here ` for a full explanation of config file options. Otherwise you're ready to use :ref:`bro-pkg `. Advanced Configuration ---------------------- If you prefer to not use the suggested `Basic Configuration`_ settings for `script_dir` and `plugin_dir`, the default configuration will install all package scripts/plugins within :file:`~/.bro-pkg` or you may change them to whatever location you prefer. These will be referred to as "non-standard" locations in the sense that vanilla configurations of either :program:`bro` or :program:`broctl` will not detect scripts/plugins in those locations without additional configuration. When using non-standard location, follow these steps to integrate with :program:`bro` and :program:`broctl`: - To get command-line :program:`bro` to be aware of Bro scripts/plugins in a non-standard location, make sure the :program:`bro-config` script (that gets installed along with :program:`bro`) is in your :envvar:`PATH` and run: .. code-block:: console $ `bro-pkg env` Note that this sets up the environment only for the current shell session. - To get :program:`broctl` to be aware of scripts/plugins in a non-standard location, run: .. code-block:: console $ bro-pkg config script_dir And set the `SitePolicyPath` option in :file:`broctl.cfg` based on the output you see. Similarly, run: .. code-block:: console $ bro-pkg config plugin_dir And set the `SitePluginPath` option in :file:`broctl.cfg` based on the output you see. Usage ----- Check the output of :ref:`bro-pkg --help ` for an explanation of all available functionality of the command-line tool. Package Upgrades/Versioning ~~~~~~~~~~~~~~~~~~~~~~~~~~~ When installing packages, note that the :ref:`install command `, has a ``--version`` flag that may be used to install specific package versions which may either be git release tags or branch names. The way that :program:`bro-pkg` receives updates for a package depends on whether the package is first installed to track stable releases or a specific git branch. See the :ref:`package upgrade process ` documentation to learn how :program:`bro-pkg` treats each situation. Offline Usage ~~~~~~~~~~~~~ It's common to have limited network/internet access on the systems where Bro is deployed. To accomodate those scenarios, :program:`bro-pkg` can be used as normally on a system that *does* have network access to create bundles of its package installation environment. Those bundles can then be transferred to the deployment systems via whatever means are appropriate (SSH, USB flash drive, etc). For example, on the package management system you can do typical package management tasks, like install and update packages: .. code-block:: console $ bro-pkg install Then, via the :ref:`bundle command `, create a bundle file which contains a snapshot of all currently installed packages: .. code-block:: console $ bro-pkg bundle bro-packages.bundle Then transfer :file:`bro-packages.bundle` to the Bro deployment management host. For Bro clusters using BroControl_, this will be the system acting as the "manager" node. Then on that system (assuming it already as :program:`bro-pkg` installed and configured): .. code-block:: console $ bro-pkg unbundle bro-packages.bundle Finally, if you're using BroControl_, and the unbundling process was successful, you need to deploy the changes to worker nodes: .. code-block:: console $ broctl deploy package-manager-1.3.3/doc/package.rst0000644000175000017500000005712213241343432016767 0ustar rhaistrhaist.. _Bro Scripting: https://www.bro.org/sphinx/scripting/index.html .. _Bro Plugins: https://www.bro.org/sphinx/devel/plugins.html .. _BroControl Plugins: https://www.bro.org/sphinx/components/broctl/README.html#plugins .. _Semantic Version Specification: https://python-semanticversion.readthedocs.io/en/latest/reference.html#version-specifications-the-spec-class .. _btest: https://github.com/bro/btest .. _configparser interpolation: https://docs.python.org/3/library/configparser.html#interpolation-of-values How-To: Create a Package ======================== A Bro package may contain Bro scripts, Bro plugins, or BroControl plugins. Any number or combination of those components may be included within a single package. The minimum requirement for a package is that it be in its own git repository and contain a metadata file named :file:`bro-pkg.meta` at its top-level that begins with the line:: [package] This is the package's metadata file in INI file format and may contain :ref:`additional fields ` that describe the package as well as how it inter-operates with Bro, the package manager, or other packages. .. _package-shorthand-name: Note that the shorthand name for your package that may be used by :ref:`bro-pkg ` and Bro script :samp:`@load {}` directives will be the last component of its git URL. E.g. a package at ``https://github.com/bro/foo`` may be referred to as **foo** when using :program:`bro-pkg` and a Bro script that wants to load all the scripts within that package can use: .. code-block:: bro @load foo Walkthroughs ------------ Pure Bro Script Package ~~~~~~~~~~~~~~~~~~~~~~~ #. Create a git repository: .. code-block:: console $ mkdir foo && cd foo && git init #. Create a package metadata file, :file:`bro-pkg.meta`: .. code-block:: console $ echo '[package]' > bro-pkg.meta #. Create a :file:`__load__.bro` script with example code in it: .. code-block:: console $ echo 'event bro_init() { print "foo is loaded"; }' > __load__.bro #. (Optional) Relocate your :file:`__load__.bro` script to any subdirectory: .. code-block:: console $ mkdir scripts && mv __load__.bro scripts $ echo 'script_dir = scripts' >> bro-pkg.meta #. Commit everything to git: .. code-block:: console $ git add * && git commit -m 'First commit' #. (Optional) Test that Bro correctly loads the script after installing the package with :program:`bro-pkg`: .. code-block:: console $ bro-pkg install . $ bro foo $ bro-pkg remove . #. (Optional) :ref:`Create a release version tag `. See `Bro Scripting`_ for more information on developing Bro scripts. Binary Bro Plugin Package ~~~~~~~~~~~~~~~~~~~~~~~~~ See `Bro Plugins`_ for more complete information on developing Bro plugins, though the following step are the essentials needed to create a package. #. Create a plugin skeleton using :file:`aux/bro-aux/plugin-support/init-plugin` from Bro's source distribution: .. code-block:: console $ init-plugin ./rot13 Demo Rot13 #. Create a git repository .. code-block:: console $ cd rot13 && git init #. Create a package metadata file, :file:`bro-pkg.meta`:: [package] script_dir = scripts/Demo/Rot13 build_command = ./configure --bro-dist=%(bro_dist)s && make See the :ref:`Value Interpolation ` section for more information on what the ``%(bro_dist)s`` string does. #. Add example script code: .. code-block:: console $ echo 'event bro_init() { print "rot13 plugin is loaded"; }' >> scripts/__load__.bro $ echo 'event bro_init() { print "rot13 script is loaded"; }' >> scripts/Demo/Rot13/__load__.bro #. Add an example builtin-function in :file:`src/rot13.bif`: .. code-block:: c++ module Demo; function rot13%(s: string%) : string %{ char* rot13 = copy_string(s->CheckString()); for ( char* p = rot13; *p; p++ ) { char b = islower(*p) ? 'a' : 'A'; *p = (*p - b + 13) % 26 + b; } BroString* bs = new BroString(1, reinterpret_cast(rot13), strlen(rot13)); return new StringVal(bs); %} #. Commit everything to git: .. code-block:: console $ git add * && git commit -m 'First commit' #. (Optional) Test that Bro correctly loads the plugin after installing the package with :program:`bro-pkg`: .. code-block:: console $ bro-pkg install . $ bro rot13 -e 'print Demo::rot13("Hello")' $ bro-pkg remove . #. (Optional) :ref:`Create a release version tag `. BroControl Plugin Package ~~~~~~~~~~~~~~~~~~~~~~~~~ #. Create a git repository: .. code-block:: console $ mkdir foo && cd foo && git init #. Create a package metadata file, :file:`bro-pkg.meta`: .. code-block:: console $ echo '[package]' > bro-pkg.meta #. Create an example BroControl plugin, :file:`foo.py`: .. code-block:: python import BroControl.plugin from BroControl import config class Foo(BroControl.plugin.Plugin): def __init__(self): super(Foo, self).__init__(apiversion=1) def name(self): return "foo" def pluginVersion(self): return 1 def init(self): self.message("foo plugin is initialized") return True #. Set the `plugin_dir` metadata field to directory where the plugin is located: .. code-block:: console $ echo 'plugin_dir = .' >> bro-pkg.meta #. Commit everything to git: .. code-block:: console $ git add * && git commit -m 'First commit' #. (Optional) Test that BroControl correctly loads the plugin after installing the package with :program:`bro-pkg`: .. code-block:: console $ bro-pkg install . $ broctl $ bro-pkg remove . #. (Optional) :ref:`Create a release version tag `. See `BroControl Plugins`_ for more information on developing BroControl plugins. If you want to distribute a BroControl plugin along with a Bro plugin in the same package, you may need to add the BroControl plugin's python script to the ``bro_plugin_dist_files()`` macro in the :file:`CMakeLists.txt` of the Bro plugin so that it gets copied into :file:`build/` along with the built Bro plugin. Or you could also modify your `build_command` to copy it there, but what ultimately matters is that the `plugin_dir` field points to a directory that contains both the Bro plugin and the BroControl plugin. Registering to a Package Source ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Registering a package to a package source is always the following basic steps: #) Create a :ref:`Package Index File ` for your package. #) Add the index file to the package source's git repository. The full process and conventions for submitting to the default package source can be found in the :file:`README` at: https://github.com/bro/packages .. _metadata-fields: Package Metadata ---------------- See the following sub-sections for a full list of available fields that may be used in :file:`bro-pkg.meta` files. `description` field ~~~~~~~~~~~~~~~~~~~ The description field may be used to give users a general overview of the package and its purpose. The :ref:`bro-pkg list ` will display the first sentence of description fields in the listings it displays. An example :file:`bro-pkg.meta` using a description field:: [package] description = Another example package. The description text may span multiple line: when adding line breaks, just indent the new lines so they are parsed as part of the 'description' value. `tags` field ~~~~~~~~~~~~ The `tags` field contains a comma-delimited set of metadata tags that further classify and describe the purpose of the package. This is used to help users better discover and search for packages. The :ref:`bro-pkg search ` command will inspect these tags. An example :file:`bro-pkg.meta` using tags:: [package] tags = bro plugin, broctl plugin, scan detection, intel Suggested Tags ^^^^^^^^^^^^^^ Some ideas for what to put in the `tags` field for packages: - bro scripting - conn - intel - geolocation - file analysis - sumstats, summary statistics - input - log, logging - notices - ** - ** - signatures - bro plugin - protocol analyzer - file analyzer - bifs - packet source - packet dumper - input reader - log writer - broctl plugin `script_dir` field ~~~~~~~~~~~~~~~~~~ The `script_dir` field is a path relative to the root of the package that contains a file named :file:`__load__.bro` and possibly other Bro scripts. The files located in this directory are copied into :file:`{}/packages/{}/`, where `` corresponds to the `script_dir` field of the user's :ref:`config file ` (typically :file:`{}/share/bro/site`). When the package is :ref:`loaded `, an :samp:`@load {}` directive is added to :file:`{}/packages/packages.bro`. You may place any valid Bro script code within :file:`__load__.bro`, but a package that contains many Bro scripts will typically have :file:`__load__.bro` just contain a list of ``@load`` directives to load other Bro scripts within the package. E.g. if you have a package named **foo** installed, then it's :file:`__load__.bro` will be what Bro loads when doing ``@load foo`` or running ``bro foo`` on the command-line. An example :file:`bro-pkg.meta`:: [package] script_dir = scripts For a :file:`bro-pkg.meta` that looks like the above, the package should have a file called :file:`scripts/__load__.bro`. If the `script_dir` field is not present in :file:`bro-pkg.meta`, it defaults to checking the top-level directory of the package for a :file:`__load__.bro` script. If it's found there, :program:`bro-pkg` use the top-level package directory as the value for `script_dir`. If it's not found, then :program:`bro-pkg` assumes the package contains no Bro scripts (which may be the case for some plugins). `plugin_dir` field ~~~~~~~~~~~~~~~~~~ The `plugin_dir` field is a path relative to the root of the package that contains either pre-built `Bro Plugins`_, `BroControl Plugins`_, or both. An example :file:`bro-pkg.meta`:: [package] script_dir = scripts plugin_dir = plugins For the above example, Bro and BroControl will load any plugins found in the installed package's :file:`plugins/` directory. If the `plugin_dir` field is not present in :file:`bro-pkg.meta`, it defaults to a directory named :file:`build/` at the top-level of the package. This is the default location where Bro binary plugins get placed when building them from source code (see the `build_command field`_). This field may also be set to the location of a tarfile that has a single top- level directory inside it containing the Bro plugin. The default CMake skeleton for Bro plugins produces such a tarfile located at :file:`build/_.tgz`. This is a good choice to use for packages that will be published to a wider audience as installing from this tarfile contains the minimal set of files needed for the plugin to work whereas some extra files will get installed to user systems if the `plugin_dir` uses the default :file:`build/` directory. `build_command` field ~~~~~~~~~~~~~~~~~~~~~ The `build_command` field is an arbitrary shell command that the package manager will run before installing the package. This is useful for distributing `Bro Plugins`_ as source code and having the package manager take care of building it on the user's machine before installing the package. An example :file:`bro-pkg.meta`:: [package] script_dir = scripts/Demo/Rot13 build_command = ./configure --bro-dist=%(bro_dist)s && make In the above example, the ``%(bro_dist)s`` string is :ref:`substituted ` for the path the user has set for the `bro_dist` field in the :ref:`package manager config file `. The default CMake skeleton for Bro plugins will use :file:`build/` as the directory for the final/built version of the plugin, which matches the defaulted value of the omitted `plugin_dir` metadata field. The `script_dir` field is set to the location where the author has placed custom scripts for their plugin. When a package has both a Bro plugin and Bro script components, the "plugin" part is always unconditionally loaded by Bro, but the "script" components must either be explicitly loaded (e.g. :samp:`@load {}`) or the package marked as :ref:`loaded `. .. _metadata-interpolation: Value Interpolation ^^^^^^^^^^^^^^^^^^^ The `build_command field`_ may reference the settings any given user has in their customized :ref:`package manager config file `. For example, if a metadata field's value contains the ``%(bro_dist)s`` string, then :program:`bro-pkg` operations that use that field will automatically substitute the actual value of `bro_dist` that the user has in their local config file. Note the trailing 's' character at the end of the interpolation string, ``%(bro_dist)s`` is intended/necessary for all such interpolation usages. Besides the `bro_dist` config key, any key inside the `user_vars` sections of their :ref:`package manager config file ` that matches the key of an entry in the package's `user_vars field`_ will be interpolated. Internally, the value substitution and metadata parsing is handled by Python's `configparser interpolation`_. See its documentation if you're interested in the details of how the interpolation works. `user_vars` field ~~~~~~~~~~~~~~~~~ The `user_vars` field is used to solicit feedback from users for use during execution of the `build_command field`_. An example :file:`bro-pkg.meta`:: [package] build_command = ./configure --bro-dist=%(bro_dist)s --with-librdkafka=%(LIBRDKAFKA_ROOT)s --with-libdub=%(LIBDBUS_ROOT)s && make user_vars = LIBRDKAFKA_ROOT [/usr] "Path to librdkafka installation" LIBDBUS_ROOT [/usr] "Path to libdbus installation" The format of the field is a sequence entries of the format:: key [value] "description" The `key` is the string that should match what you want to be interpolated within the `build_command field`_. The `value` is provided as a convenient default value that you'd typically expect to work for most users. The `description` is provided as an explanation for what the value will be used for. Here's what a typical user would see:: $ bro-pkg install bro-test-package The following packages will be INSTALLED: bro/jsiwek/bro-test-package (1.0.5) Proceed? [Y/n] y bro/jsiwek/bro-test-package asks for LIBRDKAFKA_ROOT (Path to librdkafka installation) ? [/usr] /usr/local Saved answers to config file: /Users/jon/.bro-pkg/config Installed "bro/jsiwek/bro-test-package" (master) Loaded "bro/jsiwek/bro-test-package" The :program:`bro-pkg` command will iterate over the `user_vars` field of all packages involved in the operation and prompt the user to provide a value that will work for their system. If a user is using the ``--force`` option to :program:`bro-pkg` commands or they are using the Python API directly, it will first look within the `user_vars` section of the user's :ref:`package manager config file ` and, if it can't find the key there, it will fallback to use the default value from the package's metadata. In any case, the user may choose to supply the value of a `user_vars` key via an environment variable, in which case, prompts are skipped for any keys located in the environment. The environment is also given priority over any values in the user's :ref:`package manager config file `. Available :program:`since bro-pkg v1.1`. `test_command` field ~~~~~~~~~~~~~~~~~~~~ The `test_command` field is an arbitrary shell command that the package manager will run when a user either manually runs the :ref:`test command ` or before the package is installed or upgraded. An example :file:`bro-pkg.meta`:: [package] test_command = cd testing && btest -d tests The recommended test framework for writing package unit tests is `btest`_. See its documentation for further explanation and examples. `config_files` field ~~~~~~~~~~~~~~~~~~~~ The `config_files` field may be used to specify a list of files that users are intended to directly modify after installation. Then, on operations that would otherwise destroy a user's local modifications to a config file, such as upgrading to a newer package version, :program:`bro-pkg` can instead save a backup and possibly prompt the user to review the differences. An example :file:`bro-pkg.meta`:: [package] script_dir = scripts config_files = scripts/foo_config.bro, scripts/bar_config.bro The value of `config_files` is a comma-delimited string of config file paths that are relative to the root directory of the package. Config files should either be located within the `script_dir` or `plugin_dir`. .. _package-dependencies: `depends` field ~~~~~~~~~~~~~~~ The `depends` field may be used to specify a list of dependencies that the package requires. An example :file:`bro-pkg.meta`:: [package] depends = bro >=2.5.0 foo * https://github.com/bro/bar >=2.0.0 package_source/path/bar branch=name_of_git_branch The field is a list of dependency names and their version requirement specifications. A dependency name may be either `bro`, `bro-pkg`, a full git URL of the package, or a :ref:`package shorthand name `. - The special `bro` dependency refers not to a package, but the version of Bro that the package requires in order to function. If the user has :program:`bro-config` in their :envvar:`PATH` when installing/upgrading a package that specifies a `bro` dependency, then :program:`bro-pkg` will enforce that the requirement is satisfied. - The special `bro-pkg` dependency refers to the version of the package manager that is required by the package. E.g. if a package takes advantage of new features that are not present in older versions of the package manager, then it should indicate that so users of those old version will see an error message an know to upgrade instead of seeing a cryptic error/exception, or worse, seeing no errors, but without the desired functionality being performed. Note that this feature itself is only available :program:`since bro-pkg v1.2`. - The full git URL may be directly specified in the `depends` metadata if you want to force the dependency to always resolve to a single, canonical git repository. Typically this is the safe approach to take when listing package dependencies and for publicly visible packages. - When using shorthand package dependency names, the user's :program:`bro-pkg` will try to resolve the name into a full git URL based on the package sources they have configured. Typically this approach may be most useful for internal or testing environments. A version requirement may be either a git branch name or a semantic version specification. When using a branch as a version requirement, prefix the branchname with ``branch=``, else see the `Semantic Version Specification`_ documentation for the complete rule set of acceptable version requirement strings. Here's a summary: - ``*``: any version (this will also satisfy/match on git branches) - ``<1.0.0``: versions less than 1.0.0 - ``<=1.0.0``: versions less than or equal to 1.0.0 - ``>1.0.0``: versions greater than 1.0.0 - ``>=1.0.0``: versions greater than or equal to 1.0.0 - ``==1.0.0``: exactly version 1.0.0 - ``!=1.0.0``: versions not equal to 1.0.0 - ``^1.3.4``: versions between 1.3.4 and 2.0.0 (not including 2.0.0) - ``~1.2.3``: versions between 1.2.3 and 1.3.0 (not including 1.3.0) - ``~=2.2``: versions between 2.2.0 and 3.0.0 (not included 3.0.0) - ``~=1.4.5``: versions between 1.4.5 and 1.5.0 (not including 3.0.0) - Any of the above may be combined by a separating comma to logically "and" the requirements together. E.g. ``>=1.0.0,<2.0.0`` means "greater or equal to 1.0.0 and less than 2.0.0". `external_depends` field ~~~~~~~~~~~~~~~~~~~~~~~~ The `external_depends` field follows the same format as the :ref:`depends field `, but the dependency names refer to external/third-party software packages. E.g. these would be set to typical package names you'd expect the package manager from any given operating system to use, like 'libpng-dev'. The version specification should also generally be given in terms of semantic versioning where possible. In any case, the name and version specification for an external dependency are only used for display purposes -- to help users understand extra pre-requisites that are needed for proceeding with package installation/upgrades. Available :program:`since bro-pkg v1.1`. `suggests` field ~~~~~~~~~~~~~~~~ The `suggests` field follows the same format as the :ref:`depends field `, but it's used for specifying optional packages that users may want to additionally install. This is helpful for suggesting complementary packages that aren't strictly required for the suggesting package to function properly. A package in `suggests` is functionaly equivalent to a package in `depends` except in the way it's presented to users in various prompts during :program:`bro-pkg` operations. Users also have the option to ignore suggestions by supplying an additional ``--nosuggestions`` flag to :program:`bro-pkg` commands. Available :program:`since bro-pkg v1.3`. .. _package-versioning: Package Versioning ------------------ Creating New Package Release Versions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Package's should use git tags for versioning their releases. Use the `Semantic Versioning `_ numbering scheme here. For example, to create a new tag for a package: .. code-block:: console $ git tag -a 1.0.0 -m 'Release 1.0.0' Then, assuming you've already set up a public/remote git repository (e.g. on GitHub) for your package, remember to push the tag to the remote repository: .. code-block:: console $ git push --tags Alternatively, if you expect to have a simple development process for your package, you may choose to not create any version tags and just always make commits directly to your package's *master* branch. Users will receive package updates differently depending on whether you decide to use release version tags or not. See the :ref:`package upgrade process ` documentation for more details on the differences. .. _package-upgrade-process: Package Upgrade Process ~~~~~~~~~~~~~~~~~~~~~~~ The :ref:`install command ` will either install a stable release version or the latest commit on a specific git branch of a package. The default installation behavior of :program:`bro-pkg` is to look for the latest release version tag and install that. If there are no such version tags, it will fall back to installing the latest commit of the package's *master* branch Upon installing a package via a git version tag, the :ref:`upgrade command ` will only upgrade the local installation of that package if a greater version tag is available. In other words, you only receive stable release upgrades for packages installed in this way. Upon installing a package via a git branch name, the :ref:`upgrade command ` will upgrade the local installation of the package whenever a new commit becomes available at the end of the branch. This method of tracking packages is suitable for testing out development/experimental versions of packages. package-manager-1.3.3/doc/docutils.conf0000644000175000017500000000003313241343432017324 0ustar rhaistrhaist[parsers] smart_quotes: no package-manager-1.3.3/doc/_templates/0000755000175000017500000000000013241343432016770 5ustar rhaistrhaistpackage-manager-1.3.3/doc/_templates/layout.html0000644000175000017500000000035613241343432021177 0ustar rhaistrhaist{% extends "!layout.html" %} {% if READTHEDOCS and current_version %} {% if current_version == "latest" or current_version == "stable" %} {% set current_version = current_version ~ " (" ~ version ~ ")" %} {% endif %} {% endif %} package-manager-1.3.3/doc/_templates/breadcrumbs.html0000644000175000017500000000120413241343432022144 0ustar rhaistrhaist{% extends "!breadcrumbs.html" %} {% block breadcrumbs_aside %}
  • {% if pagename != "search" %} {% if display_github %} {% if github_version == "master" %} {{ _('Edit on GitHub') }} {% endif %} {% elif show_source and has_source and sourcename %} {{ _('View page source') }} {% endif %} {% endif %}
    • {% endblock %} package-manager-1.3.3/doc/Makefile0000644000175000017500000001737313241343432016306 0ustar rhaistrhaist# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build PAPER = BUILDDIR = _build # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . .PHONY: help help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " applehelp to make an Apple Help Book" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " epub3 to make an epub3" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" @echo " text to make text files" @echo " man to make manual pages" @echo " texinfo to make Texinfo files" @echo " info to make Texinfo files and run them through makeinfo" @echo " gettext to make PO message catalogs" @echo " changes to make an overview of all changed/added/deprecated items" @echo " xml to make Docutils-native XML files" @echo " pseudoxml to make pseudoxml-XML files for display purposes" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" @echo " coverage to run coverage check of the documentation (if enabled)" @echo " dummy to check syntax errors of document sources" .PHONY: clean clean: rm -rf $(BUILDDIR)/* .PHONY: html html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html touch $(BUILDDIR)/html/.nojekyll @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." .PHONY: livehtml livehtml: sphinx-autobuild --ignore "../testing/*" --ignore "*.git/*" --ignore "*.lock" --ignore "*.pyc" --ignore "*.swp" --ignore "*.swpx" --ignore "*.swx" --watch .. --watch ../bropkg -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html .PHONY: dirhtml dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." .PHONY: singlehtml singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." .PHONY: pickle pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." .PHONY: json json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." .PHONY: htmlhelp htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." .PHONY: qthelp qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/BroPackageManager.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/BroPackageManager.qhc" .PHONY: applehelp applehelp: $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp @echo @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." @echo "N.B. You won't be able to view it unless you put it in" \ "~/Library/Documentation/Help or install it in your application" \ "bundle." .PHONY: devhelp devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/BroPackageManager" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/BroPackageManager" @echo "# devhelp" .PHONY: epub epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." .PHONY: epub3 epub3: $(SPHINXBUILD) -b epub3 $(ALLSPHINXOPTS) $(BUILDDIR)/epub3 @echo @echo "Build finished. The epub3 file is in $(BUILDDIR)/epub3." .PHONY: latex latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." .PHONY: latexpdf latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." .PHONY: latexpdfja latexpdfja: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through platex and dvipdfmx..." $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." .PHONY: text text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." .PHONY: man man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." .PHONY: texinfo texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." @echo "Run \`make' in that directory to run these through makeinfo" \ "(use \`make info' here to do that automatically)." .PHONY: info info: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo "Running Texinfo files through makeinfo..." make -C $(BUILDDIR)/texinfo info @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." .PHONY: gettext gettext: $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale @echo @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." .PHONY: changes changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." .PHONY: linkcheck linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." .PHONY: doctest doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." .PHONY: coverage coverage: $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage @echo "Testing of coverage in the sources finished, look at the " \ "results in $(BUILDDIR)/coverage/python.txt." .PHONY: xml xml: $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml @echo @echo "Build finished. The XML files are in $(BUILDDIR)/xml." .PHONY: pseudoxml pseudoxml: $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml @echo @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." .PHONY: dummy dummy: $(SPHINXBUILD) -b dummy $(ALLSPHINXOPTS) $(BUILDDIR)/dummy @echo @echo "Build finished. Dummy builder generates no files." package-manager-1.3.3/doc/bro-pkg.rst0000644000175000017500000000566413241343432016741 0ustar rhaistrhaist.. _bro-pkg: bro-pkg Command-Line Tool ========================= .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :nosubcommands: --configfile : @after See :ref:`bro-pkg-config-file`. Commands -------- .. _test-command: test ~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: test .. _install-command: install ~~~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: install .. _remove-command: remove ~~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: remove .. _purge-command: purge ~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: purge .. _bundle-command: bundle ~~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: bundle .. _unbundle-command: unbundle ~~~~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: unbundle .. _refresh-command: refresh ~~~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: refresh .. _upgrade-command: upgrade ~~~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: upgrade .. _load-command: load ~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: load .. _unload-command: unload ~~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: unload .. _pin-command: pin ~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: pin .. _unpin-command: unpin ~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: unpin .. _list-command: list ~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: list .. _search-command: search ~~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: search .. _info-command: info ~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: info .. _config-command: config ~~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: config .. _autoconfig-command: autoconfig ~~~~~~~~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: autoconfig .. _env-command: env ~~~ .. argparse:: :module: bro-pkg :func: argparser :prog: bro-pkg :path: env .. _bro-pkg-config-file: Config File ----------- The :program:`bro-pkg` command-line tool uses an INI-format config file to allow users to customize their :doc:`Package Sources `, :doc:`Package ` installation paths, Bro executable/source paths, and other :program:`bro-pkg` options. See the default/example config file below for explanations of the available options and how to customize them: .. literalinclude:: ../bro-pkg.config package-manager-1.3.3/doc/source.rst0000644000175000017500000000747113241343432016676 0ustar rhaistrhaist.. _Bro Packages Git Repository: https://github.com/bro/packages How-To: Create a Package Source =============================== :ref:`bro-pkg `, by default, is configured to obtain packages from a single "package source", the `Bro Packages Git Repository`_, which is hosted by and loosely curated by the Bro Team. However, users may :ref:`configure bro-pkg ` to use other package sources: either ones they've set up themselves for organization purposes or those hosted by other third parties. Package Source Setup -------------------- In order to set up such a package source, one simply has to create a git repository and then add :ref:`Package Index Files ` to it. These files may be created at any path in the package source's git repository. E.g. the `Bro Packages Git Repository`_ organizes package index files hierarchically based on package author names such as :file:`alice/bro-pkg.index` or :file:`bob/bro-pkg.index` where ``alice`` and ``bob`` are usually GitHub usernames or some unique way of identifying the organization/person that maintains Bro packages. However, a source is free to use a flat organization with a single, top-level :file:`bro-pkg.index`. After creating a git repo for the package source and adding package index files to it, it's ready to be used by :ref:`bro-pkg `. .. _package-index-file: Package Index Files ------------------- Files named :file:`bro-pkg.index` are used to describe the :doc:`Bro Packages ` found within the package source. They are simply a list of git URLs pointing to the git repositories of packages. For example:: https://github.com/bro/foo https://github.com/bro/bar https://github.com/bro/baz Local filesystem paths are also valid if the package source is only meant for your own private usage or testing. Adding Packages --------------- Adding packages is as simple as adding new :ref:`Package Index Files ` or extending existing ones with new URLs and then commiting/pushing those changes to the package source git repository. :ref:`bro-pkg ` will see new packages listed the next time it uses the :ref:`refresh command `. Removing Packages ----------------- Just remove the package's URL from the :ref:`Package Index File ` that it's contained within. After the next time :program:`bro-pkg` uses the :ref:`refresh command `, it will no longer see the now-removed package when viewing package listings via by the :ref:`list command `. Users that had previously installed the now-removed package may continue to use it and receive updates for it. Aggregating Metadata -------------------- The maintainer/operator of a package source may choose to periodically aggregate the metadata contained in its package's :file:`bro-pkg.meta` files. The :ref:`bro-pkg refresh ` is used to perform the task. For example: .. code-block:: console $ bro-pkg refresh --aggregate --push --sources my_source The optional ``--push`` flag is helpful for setting up cron jobs to automatically perform this task periodically, assuming you've set up your git configuration to push changesets without interactive prompts. E.g. to set up pushing to remote servers you could set up SSH public key authentication. Aggregated metadata gets written to a file named :file:`aggregate.meta` at the top-level of a package source and the :ref:`list `, :ref:`search `, and :ref:`info ` all may access this file. Having access to the aggregated metadata in this way is beneficial to all :program:`bro-pkg` users because they then will not have to crawl the set of packages listed in a source in order to obtain this metadata as it will have already been pre-aggregated by the operator of the package source. package-manager-1.3.3/doc/conf.py0000644000175000017500000002354113241343432016137 0ustar rhaistrhaist# -*- coding: utf-8 -*- # # Bro Package Manager documentation build configuration file, created by # sphinx-quickstart on Fri Jul 15 13:46:04 2016. # # This file is execfile()d with the current directory set to its # containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # import os import sys sys.path.insert(0, os.path.abspath('..')) sys.path.insert(0, os.path.abspath('./ext')) # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. # # needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = ['sphinxarg.ext', 'sphinx.ext.autodoc', 'sphinxcontrib.napoleon'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # # source_suffix = ['.rst', '.md'] source_suffix = '.rst' # The encoding of source files. # # source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = u'Bro Package Manager' copyright = u'2016, The Bro Project' author = u'The Bro Project' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. with open('../VERSION', 'r') as f: version = f.readline().strip() # The full version, including alpha/beta/rc tags. release = version # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: # # today = '' # # Else, today_fmt is used as the format for a strftime call. # # today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This patterns also effect to html_static_path and html_extra_path exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] # The reST default role (used for this markup: `text`) to use for all # documents. # # default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. # # add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). # # add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. # # show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. # modindex_common_prefix = [] # If true, keep warnings as "system message" paragraphs in the built documents. # keep_warnings = False # If true, `todo` and `todoList` produce output, else they produce nothing. todo_include_todos = False # -- Options for HTML output ---------------------------------------------- on_rtd = os.environ.get('READTHEDOCS', None) == 'True' if not on_rtd: # only import and set the theme if we're building docs locally import sphinx_rtd_theme html_theme = 'sphinx_rtd_theme' html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] html_theme_options = { 'collapse_navigation': False, 'display_version': True, } # The name for this set of Sphinx documents. # " v documentation" by default. # html_title = u'Bro Package Manager Documentation' # A shorter title for the navigation bar. Default is the same as html_title. # # html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. # # html_logo = None # The name of an image file (relative to this directory) to use as a favicon of # the docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. # # html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] def setup(app): #app.add_javascript("custom.js") app.add_stylesheet("theme_overrides.css") # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied # directly to the root of the documentation. # # html_extra_path = [] # If not None, a 'Last updated on:' timestamp is inserted at every page # bottom, using the given strftime format. # The empty string is equivalent to '%b %d, %Y'. # # html_last_updated_fmt = None # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. # # html_use_smartypants = True # Custom sidebar templates, maps document names to template names. # #html_sidebars = {'**': ['custom-sidebar.html']} # Additional templates that should be rendered to pages, maps page names to # template names. # # html_additional_pages = {} # If false, no module index is generated. # # html_domain_indices = True # If false, no index is generated. # # html_use_index = True # If true, the index is split into individual pages for each letter. # # html_split_index = False # If true, links to the reST sources are added to the pages. # # html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. # # html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. # # html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. # # html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). # html_file_suffix = None # Language to be used for generating the HTML full-text search index. # Sphinx supports the following languages: # 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' # 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr', 'zh' # # html_search_language = 'en' # A dictionary with options for the search language support, empty by default. # 'ja' uses this config value. # 'zh' user can custom change `jieba` dictionary path. # # html_search_options = {'type': 'default'} # The name of a javascript file (relative to the configuration directory) that # implements a search results scorer. If empty, the default will be used. # # html_search_scorer = 'scorer.js' # Output file base name for HTML help builder. htmlhelp_basename = 'BroPackageManagerdoc' # -- Options for LaTeX output --------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). # # 'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). # # 'pointsize': '10pt', # Additional stuff for the LaTeX preamble. # # 'preamble': '', # Latex figure (float) alignment # # 'figure_align': 'htbp', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ (master_doc, 'BroPackageManager.tex', u'Bro Package Manager Documentation', u'The Bro Project', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. # # latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. # # latex_use_parts = False # If true, show page references after internal links. # # latex_show_pagerefs = False # If true, show URL addresses after external links. # # latex_show_urls = False # Documents to append as an appendix to all manuals. # # latex_appendices = [] # It false, will not define \strong, \code, itleref, \crossref ... but only # \sphinxstrong, ..., \sphinxtitleref, ... To help avoid clash with user added # packages. # # latex_keep_old_macro_names = True # If false, no module index is generated. # # latex_domain_indices = True # -- Options for manual page output --------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('bro-pkg', 'bro-pkg', u'Bro Package Manager', [author], 1) ] # If true, show URL addresses after external links. # # man_show_urls = False # -- Options for Texinfo output ------------------------------------------- # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ (master_doc, 'BroPackageManager', u'Bro Package Manager Documentation', author, 'BroPackageManager', 'One line description of project.', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. # # texinfo_appendices = [] # If false, no module index is generated. # # texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. # # texinfo_show_urls = 'footnote' # If true, do not generate a @detailmenu in the "Top" node's menu. # # texinfo_no_detailmenu = False package-manager-1.3.3/doc/ext/0000755000175000017500000000000013241343432015433 5ustar rhaistrhaistpackage-manager-1.3.3/doc/ext/sphinxarg/0000755000175000017500000000000013241343432017436 5ustar rhaistrhaistpackage-manager-1.3.3/doc/ext/sphinxarg/parser.py0000644000175000017500000001052513241343432021307 0ustar rhaistrhaistfrom argparse import _HelpAction, _SubParsersAction import re class NavigationException(Exception): pass def parser_navigate(parser_result, path, current_path=None): if isinstance(path, str): if path == '': return parser_result path = re.split('\s+', path) current_path = current_path or [] if len(path) == 0: return parser_result if 'children' not in parser_result: raise NavigationException( 'Current parser have no children elements. (path: %s)' % ' '.join(current_path)) next_hop = path.pop(0) for child in parser_result['children']: if child['name'] == next_hop: current_path.append(next_hop) return parser_navigate(child, path, current_path) raise NavigationException( 'Current parser have no children element with name: %s (path: %s)' % ( next_hop, ' '.join(current_path))) def _try_add_parser_attribute(data, parser, attribname): attribval = getattr(parser, attribname, None) if attribval is None: return if not isinstance(attribval, str): return if len(attribval) > 0: data[attribname] = attribval def _format_usage_without_prefix(parser): """ Use private argparse APIs to get the usage string without the 'usage: ' prefix. """ fmt = parser._get_formatter() fmt.add_usage(parser.usage, parser._actions, parser._mutually_exclusive_groups, prefix='') return fmt.format_help().strip() def parse_parser(parser, data=None, **kwargs): if data is None: data = { 'name': '', 'usage': parser.format_usage().strip(), 'bare_usage': _format_usage_without_prefix(parser), 'prog': parser.prog, } _try_add_parser_attribute(data, parser, 'description') _try_add_parser_attribute(data, parser, 'epilog') for action in parser._get_positional_actions(): if isinstance(action, _HelpAction): continue if isinstance(action, _SubParsersAction): helps = {} for item in action._choices_actions: helps[item.dest] = item.help # commands which share an existing parser are an alias, # don't duplicate docs subsection_alias = {} subsection_alias_names = set() for name, subaction in action._name_parser_map.items(): if subaction not in subsection_alias: subsection_alias[subaction] = [] else: subsection_alias[subaction].append(name) subsection_alias_names.add(name) for name, subaction in action._name_parser_map.items(): if name in subsection_alias_names: continue subalias = subsection_alias[subaction] subaction.prog = '%s %s' % (parser.prog, name) subdata = { 'name': name if not subalias else '%s (%s)' % (name, ', '.join(subalias)), 'help': helps.get(name, ''), 'usage': subaction.format_usage().strip(), 'bare_usage': _format_usage_without_prefix(subaction), } parse_parser(subaction, subdata, **kwargs) data.setdefault('children', []).append(subdata) continue if 'args' not in data: data['args'] = [] arg = { 'name': action.dest, 'help': action.help or '', 'metavar': action.metavar } if action.choices: arg['choices'] = action.choices data['args'].append(arg) show_defaults = ( ('skip_default_values' not in kwargs) or (kwargs['skip_default_values'] is False)) for action in parser._get_optional_actions(): if isinstance(action, _HelpAction): continue if 'options' not in data: data['options'] = [] option = { 'name': action.option_strings, 'default': action.default if show_defaults else '==SUPPRESS==', 'help': action.help or '' } if action.choices: option['choices'] = action.choices if "==SUPPRESS==" not in option['help']: data['options'].append(option) return data package-manager-1.3.3/doc/ext/sphinxarg/__init__.py0000644000175000017500000000000013241343432021535 0ustar rhaistrhaistpackage-manager-1.3.3/doc/ext/sphinxarg/ext.py0000644000175000017500000004116113241343432020613 0ustar rhaistrhaistfrom argparse import ArgumentParser import os from docutils import nodes from docutils.statemachine import StringList from docutils.parsers.rst.directives import flag, unchanged from sphinx.util.compat import Directive from sphinx.util.nodes import nested_parse_with_titles from sphinxarg.parser import parse_parser, parser_navigate def map_nested_definitions(nested_content): if nested_content is None: raise Exception('Nested content should be iterable, not null') # build definition dictionary definitions = {} for item in nested_content: if not isinstance(item, nodes.definition_list): continue for subitem in item: if not isinstance(subitem, nodes.definition_list_item): continue if not len(subitem.children) > 0: continue classifier = '@after' idx = subitem.first_child_matching_class(nodes.classifier) if idx is not None: ci = subitem[idx] if len(ci.children) > 0: classifier = ci.children[0].astext() if classifier is not None and classifier not in ( '@replace', '@before', '@after'): raise Exception('Unknown classifier: %s' % classifier) idx = subitem.first_child_matching_class(nodes.term) if idx is not None: ch = subitem[idx] if len(ch.children) > 0: term = ch.children[0].astext() idx = subitem.first_child_matching_class(nodes.definition) if idx is not None: def_node = subitem[idx] def_node.attributes['classifier'] = classifier definitions[term] = def_node return definitions def print_arg_list(data, nested_content): definitions = map_nested_definitions(nested_content) items = [] if 'args' in data: for arg in data['args']: my_def = [nodes.paragraph(text=arg['help'])] if arg['help'] else [] name = arg['name'] my_def = apply_definition(definitions, my_def, name) if len(my_def) == 0: my_def.append(nodes.paragraph(text='Undocumented')) if 'choices' in arg: my_def.append(nodes.paragraph( text=('Possible choices: %s' % ', '.join([str(c) for c in arg['choices']])))) items.append( nodes.option_list_item( '', nodes.option_group('', nodes.option_string(text=name)), nodes.description('', *my_def))) return nodes.option_list('', *items) if items else None def print_opt_list(data, nested_content): definitions = map_nested_definitions(nested_content) items = [] if 'options' in data: for opt in data['options']: names = [] my_def = [nodes.paragraph(text=opt['help'])] if opt['help'] else [] for name in opt['name']: option_declaration = [nodes.option_string(text=name)] if opt['default'] is not None \ and opt['default'] != '==SUPPRESS==': option_declaration += nodes.option_argument( '', text='=' + str(opt['default'])) names.append(nodes.option('', *option_declaration)) my_def = apply_definition(definitions, my_def, name) if len(my_def) == 0: my_def.append(nodes.paragraph(text='Undocumented')) if 'choices' in opt: my_def.append(nodes.paragraph( text=('Possible choices: %s' % ', '.join([str(c) for c in opt['choices']])))) items.append( nodes.option_list_item( '', nodes.option_group('', *names), nodes.description('', *my_def))) return nodes.option_list('', *items) if items else None def print_command_args_and_opts(arg_list, opt_list, sub_list=None): items = [] if arg_list: items.append(nodes.definition_list_item( '', nodes.term(text='Positional arguments:'), nodes.definition('', arg_list))) if opt_list: items.append(nodes.definition_list_item( '', nodes.term(text='Options:'), nodes.definition('', opt_list))) if sub_list and len(sub_list): items.append(nodes.definition_list_item( '', nodes.term(text='Sub-commands:'), nodes.definition('', sub_list))) return nodes.definition_list('', *items) def apply_definition(definitions, my_def, name): if name in definitions: definition = definitions[name] classifier = definition['classifier'] if classifier == '@replace': return definition.children if classifier == '@after': return my_def + definition.children if classifier == '@before': return definition.children + my_def raise Exception('Unknown classifier: %s' % classifier) return my_def def print_subcommand_list(data, nested_content): definitions = map_nested_definitions(nested_content) items = [] if 'children' in data: for child in data['children']: my_def = [nodes.paragraph( text=child['help'])] if child['help'] else [] name = child['name'] my_def = apply_definition(definitions, my_def, name) if len(my_def) == 0: my_def.append(nodes.paragraph(text='Undocumented')) if 'description' in child: my_def.append(nodes.paragraph(text=child['description'])) my_def.append(nodes.literal_block(text=child['usage'])) my_def.append(print_command_args_and_opts( print_arg_list(child, nested_content), print_opt_list(child, nested_content), print_subcommand_list(child, nested_content) )) items.append( nodes.definition_list_item( '', nodes.term('', '', nodes.strong(text=name)), nodes.definition('', *my_def) ) ) return nodes.definition_list('', *items) class ArgParseDirective(Directive): has_content = True option_spec = dict(module=unchanged, func=unchanged, ref=unchanged, prog=unchanged, path=unchanged, nodefault=flag, manpage=unchanged, nosubcommands=unchanged, passparser=flag) def _construct_manpage_specific_structure(self, parser_info): """ Construct a typical man page consisting of the following elements: NAME (automatically generated, out of our control) SYNOPSIS DESCRIPTION OPTIONS FILES SEE ALSO BUGS """ # SYNOPSIS section synopsis_section = nodes.section( '', nodes.title(text='Synopsis'), nodes.literal_block(text=parser_info["bare_usage"]), ids=['synopsis-section']) # DESCRIPTION section description_section = nodes.section( '', nodes.title(text='Description'), nodes.paragraph(text=parser_info.get( 'description', parser_info.get( 'help', "undocumented").capitalize())), ids=['description-section']) nested_parse_with_titles( self.state, self.content, description_section) if parser_info.get('epilog'): # TODO: do whatever sphinx does to understand ReST inside # docstrings magically imported from other places. The nested # parse method invoked above seem to be able to do this but # I haven't found a way to do it for arbitrary text description_section += nodes.paragraph( text=parser_info['epilog']) # OPTIONS section options_section = nodes.section( '', nodes.title(text='Options'), ids=['options-section']) if 'args' in parser_info: options_section += nodes.paragraph() options_section += nodes.subtitle(text='Positional arguments:') options_section += self._format_positional_arguments(parser_info) if 'options' in parser_info: options_section += nodes.paragraph() options_section += nodes.subtitle(text='Optional arguments:') options_section += self._format_optional_arguments(parser_info) items = [ # NOTE: we cannot generate NAME ourselves. It is generated by # docutils.writers.manpage synopsis_section, description_section, # TODO: files # TODO: see also # TODO: bugs ] if len(options_section.children) > 1: items.append(options_section) if 'nosubcommands' not in self.options: # SUBCOMMANDS section (non-standard) subcommands_section = nodes.section( '', nodes.title(text='Sub-Commands'), ids=['subcommands-section']) if 'children' in parser_info: subcommands_section += self._format_subcommands(parser_info) if len(subcommands_section) > 1: items.append(subcommands_section) if os.getenv("INCLUDE_DEBUG_SECTION"): import json # DEBUG section (non-standard) debug_section = nodes.section( '', nodes.title(text="Argparse + Sphinx Debugging"), nodes.literal_block(text=json.dumps(parser_info, indent=' ')), ids=['debug-section']) items.append(debug_section) return items def _format_positional_arguments(self, parser_info): assert 'args' in parser_info items = [] for arg in parser_info['args']: arg_items = [] if arg['help']: arg_items.append(nodes.paragraph(text=arg['help'])) else: arg_items.append(nodes.paragraph(text='Undocumented')) if 'choices' in arg: arg_items.append( nodes.paragraph( text='Possible choices: ' + ', '.join(arg['choices']))) items.append( nodes.option_list_item( '', nodes.option_group( '', nodes.option( '', nodes.option_string(text=arg['metavar']) ) ), nodes.description('', *arg_items))) return nodes.option_list('', *items) def _format_optional_arguments(self, parser_info): assert 'options' in parser_info items = [] for opt in parser_info['options']: names = [] opt_items = [] for name in opt['name']: option_declaration = [nodes.option_string(text=name)] if opt['default'] is not None \ and opt['default'] != '==SUPPRESS==': option_declaration += nodes.option_argument( '', text='=' + str(opt['default'])) names.append(nodes.option('', *option_declaration)) if opt['help']: opt_items.append(nodes.paragraph(text=opt['help'])) else: opt_items.append(nodes.paragraph(text='Undocumented')) if 'choices' in opt: opt_items.append( nodes.paragraph( text='Possible choices: ' + ', '.join(opt['choices']))) items.append( nodes.option_list_item( '', nodes.option_group('', *names), nodes.description('', *opt_items))) return nodes.option_list('', *items) def _format_subcommands(self, parser_info): assert 'children' in parser_info items = [] for subcmd in parser_info['children']: subcmd_items = [] if subcmd['help']: subcmd_items.append(nodes.paragraph(text=subcmd['help'])) else: subcmd_items.append(nodes.paragraph(text='Undocumented')) items.append( nodes.definition_list_item( '', nodes.term('', '', nodes.strong( text=subcmd['bare_usage'])), nodes.definition('', *subcmd_items))) return nodes.definition_list('', *items) def _nested_parse_paragraph(self, text): content = nodes.paragraph() self.state.nested_parse(StringList(text.split("\n")), 0, content) return content def run(self): if 'module' in self.options and 'func' in self.options: module_name = self.options['module'] attr_name = self.options['func'] elif 'ref' in self.options: _parts = self.options['ref'].split('.') module_name = '.'.join(_parts[0:-1]) attr_name = _parts[-1] else: raise self.error( ':module: and :func: should be specified, or :ref:') mod = __import__(module_name, globals(), locals(), [attr_name]) file_dependency = mod.__file__ if file_dependency.endswith('.pyc'): file_dependency = file_dependency[:-1] env = self.state.document.settings.env if not hasattr(env, 'argparse_usages'): env.argparse_usages = [] env.argparse_usages.append({ 'docname': env.docname, 'lineno': self.lineno, 'dependency_file': file_dependency, 'dependency_mtime': os.stat(file_dependency).st_mtime, }) if not hasattr(mod, attr_name): raise self.error(( 'Module "%s" has no attribute "%s"\n' 'Incorrect argparse :module: or :func: values?' ) % (module_name, attr_name)) func = getattr(mod, attr_name) if isinstance(func, ArgumentParser): parser = func elif 'passparser' in self.options: parser = ArgumentParser() func(parser) else: parser = func() if 'path' not in self.options: self.options['path'] = '' path = str(self.options['path']) if 'prog' in self.options: parser.prog = self.options['prog'] result = parse_parser( parser, skip_default_values='nodefault' in self.options) result = parser_navigate(result, path) if 'manpage' in self.options: return self._construct_manpage_specific_structure(result) nested_content = nodes.paragraph() self.state.nested_parse( self.content, self.content_offset, nested_content) nested_content = nested_content.children items = [] # add common content between for item in nested_content: if not isinstance(item, nodes.definition_list): items.append(item) if 'description' in result: items.append(self._nested_parse_paragraph(result['description'])) items.append(nodes.literal_block(text=result['usage'])) if 'nosubcommands' in self.options: subcommands = None else: subcommands = print_subcommand_list(result, nested_content) items.append(print_command_args_and_opts( print_arg_list(result, nested_content), print_opt_list(result, nested_content), subcommands )) if 'epilog' in result: items.append(self._nested_parse_paragraph(result['epilog'])) return items def env_get_outdated_hook(app, env, added, changed, removed): rval = set() if not hasattr(env, 'argparse_usages'): return [] for usage in env.argparse_usages: docname = usage['docname'] dep_file = usage['dependency_file'] dep_mtime = usage['dependency_mtime'] current_mtime = os.stat(dep_file).st_mtime if current_mtime > dep_mtime and docname not in removed: rval.add(docname) for docname in rval: from sphinx.util.console import blue msg = blue('found outdated argparse doc: {0}'.format(docname)) app.builder.info(msg) return list(rval) def env_purge_doc_hook(app, env, docname): if not hasattr(env, 'argparse_usages'): return env.argparse_usages = [ usage for usage in env.argparse_usages if usage['docname'] != docname] def setup(app): app.add_directive('argparse', ArgParseDirective) app.connect('env-get-outdated', env_get_outdated_hook) app.connect('env-purge-doc', env_purge_doc_hook) package-manager-1.3.3/doc/ext/sphinxarg/LICENSE0000644000175000017500000000206713241343432020450 0ustar rhaistrhaistThe MIT License (MIT) Copyright (c) 2013 Alex Rudakov Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. package-manager-1.3.3/doc/ext/sphinxarg/README0000644000175000017500000000041213241343432020313 0ustar rhaistrhaistModified version of sphinx-argparse 0.1.15: https://github.com/ribozz/sphinx-argparse Added Sphinx extension hooks to check for whether the python module associated with an argparse directive is outdated and so the .rst file it is used in needs to be re-read. package-manager-1.3.3/doc/overview.rst0000644000175000017500000000214113241343432017231 0ustar rhaistrhaist.. _Bro: https://bro.org .. _Bro package source: https://github.com/bro/packages .. _documentation: http://bro-package-manager.readthedocs.io Bro Package Manager =================== The Bro Package Manager makes it easy for Bro_ users to install and manage third party scripts as well as plugins for Bro and BroControl. The command-line tool is preconfigured to download packages from the `Bro package source`_ , a GitHub repository that has been set up such that any developer can request their Bro package be included. See the ``README`` file of that repository for information regarding the package submission process. :note: It's left up to users to decide for themselves via code review, GitHub comments/stars, or other metrics whether any given package is trustworthy as there is no implied guarantees that it's secure just because it's been accepted into the default package source. See the package manager documentation_ for further usage information, how-to guides, and walkthroughs. For offline reading, it's also available in the ``docs/`` directory of the source code distribution.