octocatalog-diff-1.5.4/0000755000004100000410000000000013413744025014761 5ustar www-datawww-dataoctocatalog-diff-1.5.4/README.md0000644000004100000410000001361513413744025016246 0ustar www-datawww-data# octocatalog-diff #### Compile Puppet catalogs from 2 branches, versions, etc., and compare them `octocatalog-diff` is a tool that enables developers to be more efficient when testing changes to Puppet manifests. It is most commonly used to display differences in Puppet catalogs between stable and development branches. It does not require a working Puppet master (or puppetserver), so it is often run by developers on their workstations and in Continuous Integration environments. At GitHub, we manage thousands of nodes with a Puppet code base containing 500,000+ lines of code from over 200 contributors. We run `octocatalog-diff` thousands of times per day as part of Continuous Integration testing, and developers run it on their workstations as they are working with the code. `octocatalog-diff` is written in Ruby and is distributed as a gem. It runs on Mac OS and Unix/Linux platforms. We consider the 1.x release of `octocatalog-diff` to be stable and production-quality. We continue to maintain and enhance `octocatalog-diff` to meet GitHub's internal needs and to incorporate suggestions from the community. Please consult the [change log](/doc/CHANGELOG.md) for details. If you've been using version 0.6.1 or earlier, please read about [What's new in octocatalog-diff 1.0](/doc/versions/v1.md) for a summary of new capabilities and breaking changes. ## How? Traditional Puppet development generally takes one of two forms. Frequently, developers will test changes by running a Puppet agent (perhaps in `--noop` mode) to see if the desired change has resulted on an actual system. Others will use formal testing methodologies, such as `rspec-puppet` or the beaker framework to validate Puppet code. `octocatalog-diff` uses a different pattern. In its most common invocation, it compiles Puppet catalogs for both the stable branch (e.g. master) and the development branch, and then compares them. It filters out attributes or resources that have no effect on ultimate state of the target system (e.g. tags) and displays the remaining differences. Using this strategy, one can get feedback on changes without deploying Puppet code to a server and conducting a full Puppet run, and this tool works even if test coverage is incomplete. There are some [limitations](doc/limitations.md) to a catalog-based approach, meaning it will never completely replace unit, integration, or deployment testing. However, it does provide substantial time savings in both the development and testing cycle. In this repository, we provide example scripts for using `octocatalog-diff` in development and CI environments. `octocatalog-diff` is currently able to get catalogs by the following methods: - Compile catalog via the command line with a Puppet agent on your machine (as GitHub uses the tool internally) - Obtain catalog over the network from PuppetDB - Obtain catalog over the network using the API to query a Puppet Master / PuppetServer (Puppet 3.x and 4.x supported) - Read catalog from a JSON file ## Example Here is simulated output from running `octocatalog-diff` to compare the Puppet catalog changes between the master branch and the Puppet code in the current working directory: [octocatalog-diff screenshot] The example above reflects the changes in the Puppet catalog from switching an underlying device for a mounted file system. ## Documentation ### Installation and use in a development environment - [Installation](/doc/installation.md) - [Configuration](/doc/configuration.md) - [Basic command line usage](/doc/basic.md) - [Advanced command line usage](/doc/advanced.md) - [Troubleshooting](/doc/troubleshooting.md) ### Installation and use for CI - [Setting up octocatalog-diff in CI](/doc/advanced-ci.md) ### Technical details - [Requirements](/doc/requirements.md) - [Limitations](/doc/limitations.md) - [List of all command line options](/doc/optionsref.md) - [Environment variables](/doc/advanced-environment-variables.md) ### Project - [Roadmap](/doc/roadmap.md) - [Similar tools](/doc/similar.md) - [Contributing](/.github/CONTRIBUTING.md) - [Developer documentation](/doc/dev) - [API documentation](/doc/dev/api.md) ## What's in a name? During its original development at GitHub, this tool was simply called `catalog-diff`. However, there is already a [Puppet module with that name](https://forge.puppet.com/zack/catalog_diff) and we didn't want to create any confusion (in fact, a case could be made to use both approaches). So, we named the tool `octocatalog-diff` because who doesn't like the [octocat](https://octodex.github.com/)? Then one day in chat, someone referred to the tool as ":octocat:alog-diff", and that moniker caught on for electronic communication. ## Contributing Please see our [contributing document](/.github/CONTRIBUTING.md) if you would like to participate! ## Getting help If you have a problem or suggestion, please [open an issue](https://github.com/github/octocatalog-diff/issues/new) in this repository, and we will do our best to help. Please note that this project adheres to the [Open Code of Conduct](http://todogroup.org/opencodeofconduct/#GitHub%20Octocatalog-Diff/opensource@github.com). ## License `octocatalog-diff` is licensed under the [MIT license](LICENSE). It requires 3rd party ruby gems found [here](/vendor/cache). It also includes portions of other open source projects [here](/lib/octocatalog-diff/external/pson), [here](/spec/octocatalog-diff/fixtures/repos/default/modules/stdlib), [here](/spec/octocatalog-diff/support/httparty) and [here](/spec/octocatalog-diff/tests/external/pson). All 3rd party code and required gems are licensed either as MIT or Apache 2.0. ## Authors `octocatalog-diff` was designed and authored by [Kevin Paulisse](https://github.com/kpaulisse) and is now maintained, reviewed, and tested by Kevin and the rest of the Site Reliability Engineering team at GitHub. octocatalog-diff-1.5.4/bin/0000755000004100000410000000000013413744025015531 5ustar www-datawww-dataoctocatalog-diff-1.5.4/bin/octocatalog-diff0000755000004100000410000000250513413744025020666 0ustar www-datawww-data#!/usr/bin/env ruby if ENV['OCTOCATALOG_DIFF_DEVELOPER_PATH'] # Mostly used for internal testing, to allow pointing of a "real" octocatalog-diff install # at code being developed live. rel = File.expand_path(ENV['OCTOCATALOG_DIFF_DEVELOPER_PATH'], File.dirname(__FILE__)) abs = File.expand_path(ENV['OCTOCATALOG_DIFF_DEVELOPER_PATH']) if File.directory?(abs) && File.file?(File.join(abs, 'lib', 'octocatalog-diff.rb')) require File.join(abs, 'lib', 'octocatalog-diff') elsif File.directory?(rel) && File.file?(File.join(rel, 'lib', 'octocatalog-diff.rb')) require File.join(rel, 'lib', 'octocatalog-diff') else raise Errno::ENOENT, "Unable to resolve #{ENV['OCTOCATALOG_DIFF_DEVELOPER_PATH']} (tried #{abs} then #{rel})" end elsif File.file?(File.expand_path('../lib/octocatalog-diff.rb', File.dirname(__FILE__))) require_relative '../lib/octocatalog-diff' else require 'octocatalog-diff' end config_test = ARGV.include?('--config-test') logger = Logger.new(STDERR) logger.level = Logger::INFO logger.level = Logger::DEBUG if config_test options = OctocatalogDiff::API::V1.config(logger: logger, test: config_test) if config_test logger.info 'Exiting now because --config-test was specified' exit(0) end argv = ARGV.dup exit_code = OctocatalogDiff::Cli.cli(argv, Logger.new(STDERR), options) exit(exit_code) octocatalog-diff-1.5.4/LICENSE0000644000004100000410000000203713413744025015770 0ustar www-datawww-dataCopyright (c) 2016 GitHub Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. octocatalog-diff-1.5.4/scripts/0000755000004100000410000000000013413744025016450 5ustar www-datawww-dataoctocatalog-diff-1.5.4/scripts/git-extract/0000755000004100000410000000000013413744025020703 5ustar www-datawww-dataoctocatalog-diff-1.5.4/scripts/git-extract/git-extract.sh0000644000004100000410000000102113413744025023464 0ustar www-datawww-data#!/usr/bin/env bash # This script is called from lib/octocatalog-diff/catalog-util/git.rb and is used to # archive and extract a certain branch of a git repository into a target directory. if [ -z "$OCD_GIT_EXTRACT_BRANCH" ]; then echo "Error: Must declare OCD_GIT_EXTRACT_BRANCH" exit 255 fi if [ -z "$OCD_GIT_EXTRACT_TARGET" ]; then echo "Error: Must declare OCD_GIT_EXTRACT_TARGET" exit 255 fi set -euf -o pipefail git archive --format=tar "$OCD_GIT_EXTRACT_BRANCH" | ( cd "$OCD_GIT_EXTRACT_TARGET" && tar -xf - ) octocatalog-diff-1.5.4/scripts/puppet/0000755000004100000410000000000013413744025017765 5ustar www-datawww-dataoctocatalog-diff-1.5.4/scripts/puppet/puppet.sh0000644000004100000410000000047313413744025021642 0ustar www-datawww-data#!/usr/bin/env bash # Script to run Puppet. The default implementation here is simply to pass # through the command line arguments (which are likely to be numerous when # compiling a catalog). if [ -z "$OCD_PUPPET_BINARY" ]; then echo "Error: PUPPET_BINARY must be set" exit 255 fi "$OCD_PUPPET_BINARY" "$@" octocatalog-diff-1.5.4/scripts/env/0000755000004100000410000000000013413744025017240 5ustar www-datawww-dataoctocatalog-diff-1.5.4/scripts/env/env.sh0000644000004100000410000000017513413744025020367 0ustar www-datawww-data#!/usr/bin/env bash # This script echoes back the environment. This is used for spec testing # and possible debugging. env octocatalog-diff-1.5.4/lib/0000755000004100000410000000000013413744025015527 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff.rb0000644000004100000410000000041313413744025021257 0ustar www-datawww-data# These are all the classes we believe people might want to call directly, so load # them in response to a 'require octocatalog-diff'. loads = %w(api/v1 bootstrap catalog cli errors facts puppetdb version) loads.each { |f| require_relative "octocatalog-diff/#{f}" } octocatalog-diff-1.5.4/lib/octocatalog-diff/0000755000004100000410000000000013413744025020734 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/0000755000004100000410000000000013413744025023321 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/fileresources.rb0000644000004100000410000001530413413744025026523 0ustar www-datawww-data# frozen_string_literal: true require 'digest' module OctocatalogDiff module CatalogUtil # Used to convert file resources such as: # file { 'something': source => 'puppet:///modules/xxx/yyy'} # to: # file { 'something': content => $( cat modules/xxx/files/yyy )} # This allows the displayed diff to show differences in static files. class FileResources # Public method: Convert file resources to text. See the description of the class # just above for details. # @param obj [OctocatalogDiff::Catalog] Catalog object (will be modified) def self.convert_file_resources(obj, environment = 'production') return unless obj.valid? && obj.compilation_dir.is_a?(String) && !obj.compilation_dir.empty? _convert_file_resources(obj.resources, obj.compilation_dir, environment) begin obj.catalog_json = ::JSON.generate(obj.catalog) rescue ::JSON::GeneratorError => exc obj.error_message = "Failed to generate JSON: #{exc.message}" end end # Internal method: Locate a file that is referenced at puppet:///modules/xxx/yyy using the # module path that is specified within the environment.conf file (assuming the default 'modules' # directory doesn't exist or the module isn't found in there). If the file can't be found then # this returns nil which may trigger an error. # @param src_in [String|Array] A file reference: puppet:///modules/xxx/yyy # @param modulepaths [Array] Cached module path # @return [String] File system path to referenced file def self.file_path(src_in, modulepaths) valid_sources = [src_in].flatten.select { |line| line =~ %r{\Apuppet:///modules/([^/]+)/(.+)} } return unless valid_sources.any? valid_sources.each do |src| src =~ %r{\Apuppet:///modules/([^/]+)/(.+)} path = File.join(Regexp.last_match(1), 'files', Regexp.last_match(2)) modulepaths.each do |mp| file = File.join(mp, path) return file if File.exist?(file) end end nil end # Internal method: Parse environment.conf to find the modulepath # @param dir [String] Directory in which to look for environment.conf # @return [Array] Module paths def self.module_path(dir) environment_conf = File.join(dir, 'environment.conf') return [File.join(dir, 'modules')] unless File.file?(environment_conf) # This doesn't support multi-line, continuations with backslash, etc. # Does it need to?? if File.read(environment_conf) =~ /^modulepath\s*=\s*(.+)/ result = [] Regexp.last_match(1).split(/:/).map(&:strip).each do |path| next if path.start_with?('$') result << File.expand_path(path, dir) end result else [File.join(dir, 'modules')] end end # Internal method: Static method to convert file resources. The compilation directory is # required, or else this is a no-op. The passed-in array of resources is modified by this method. # @param resources [Array] Array of catalog resources # @param compilation_dir [String] Compilation directory (so files can be looked up) def self._convert_file_resources(resources, compilation_dir, environment = 'production') # Calculate compilation directory. There is not explicit error checking here because # there is on-demand, explicit error checking for each file within the modification loop. return unless compilation_dir.is_a?(String) && compilation_dir != '' # Making sure that compilation_dir/environments//modules exists (and by inference, # that compilation_dir/environments/ is pointing at the right place). Otherwise, try to find # compilation_dir/modules. If neither of those exist, this code can't run. env_dir = File.join(compilation_dir, 'environments', environment) modulepaths = module_path(env_dir) + module_path(compilation_dir) modulepaths.select! { |x| File.directory?(x) } return if modulepaths.empty? # At least one existing module path was found! Run the code to modify the resources. resources.map! do |resource| if resource_convertible?(resource) path = file_path(resource['parameters']['source'], modulepaths) if path.nil? # Pass this through as a wrapped exception, because it's more likely to be something wrong # in the catalog itself than it is to be a broken setup of octocatalog-diff. message = "Errno::ENOENT: Unable to resolve '#{resource['parameters']['source']}'!" raise OctocatalogDiff::Errors::CatalogError, message end if File.file?(path) # If the file is found, read its content. If the content is all ASCII, substitute it into # the 'content' parameter for easier comparison. If not, instead populate the md5sum. # Delete the 'source' attribute as well. content = File.read(path) is_ascii = content.force_encoding('UTF-8').ascii_only? resource['parameters']['content'] = is_ascii ? content : '{md5}' + Digest::MD5.hexdigest(content) resource['parameters'].delete('source') elsif File.exist?(path) # We are not handling recursive file installs from a directory or anything else. # However, the fact that we found *something* at this location indicates that the catalog # is probably correct. Hence, the very general .exist? check. else # This is probably a bug # :nocov: raise "Unable to find '#{resource['parameters']['source']}' at #{path}!" # :nocov: end end resource end end # Internal method: Determine if a resource is convertible. It is convertible if it # is a file resource with no declared 'content' and with a declared and parseable 'source'. # @param resource [Hash] Resource to check # @return [Boolean] True of resource is convertible, false if not def self.resource_convertible?(resource) return true if resource['type'] == 'File' && \ !resource['parameters'].nil? && \ resource['parameters'].key?('source') && \ !resource['parameters'].key?('content') && \ valid_sources?(resource) false end def self.valid_sources?(resource) [resource['parameters']['source']].flatten.select { |line| line =~ %r{\Apuppet:///modules/([^/]+)/(.+)} }.any? end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/bootstrap.rb0000644000004100000410000001564213413744025025673 0ustar www-datawww-data# frozen_string_literal: true require_relative '../bootstrap' require_relative '../errors' require_relative '../util/parallel' require_relative 'git' require 'fileutils' module OctocatalogDiff module CatalogUtil # Methods to bootstrap a directory. Intended to be called from cli. This handles # parallelization of bootstrap, and formats arguments as expected by the higher level bootstrap # script. class Bootstrap # Bootstrap directories specified by --bootstrapped-from-dir and --bootstrapped-to-dir # command line options. Bootstrapping occurs in parallel. This takes no parameters (options come # from options) and returns nothing (it raises an exception if something fails). def self.bootstrap_directory_parallelizer(options, logger) # What directories do we have to bootstrap? dirs = [] unless options[:bootstrapped_from_dir].nil? if options[:from_env] == '.' message = 'Must specify a from-branch other than . when using --bootstrapped-from-dir!' \ ' Please use "-f " argument.' logger.error(message) raise OctocatalogDiff::Errors::BootstrapError, message end opts = options.merge(branch: options[:from_env], path: options[:bootstrapped_from_dir], tag: 'from_dir', dir: options[:basedir]) dirs << opts end unless options[:bootstrapped_to_dir].nil? if options[:to_env] == '.' message = 'Must specify a to-branch other than . when using --bootstrapped-to-dir!' \ ' Please use "-t " argument.' logger.error(message) raise OctocatalogDiff::Errors::BootstrapError, message end opts = options.merge(branch: options[:to_env], path: options[:bootstrapped_to_dir], tag: 'to_dir') dirs << opts end # If there are no directories given, advise the user to supply the necessary options if dirs.empty? return unless options[:cached_master_dir].nil? message = 'Specify one or more of --bootstrapped-from-dir / --bootstrapped-to-dir / --cached-master-dir' \ ' when using --bootstrap_then_exit' logger.error(message) raise OctocatalogDiff::Errors::BootstrapError, message end # Bootstrap the directories in parallel. Since there are no results here that we # care about, increment the success counter for each run that did not throw an exception. tasks = dirs.map do |x| OctocatalogDiff::Util::Parallel::Task.new( method: method(:bootstrap_directory), description: "bootstrap #{x[:tag]} #{x[:path]} for #{x[:branch]}", args: x ) end logger.debug("Begin #{dirs.size} bootstrap(s)") parallel_tasks = OctocatalogDiff::Util::Parallel.run_tasks(tasks, logger, options[:parallel]) parallel_tasks.each do |result| if result.status logger.debug("Success bootstrap_directory for #{result.args[:tag]}") else # Believed to be a bug condition, since error should have already been raised if this happens. # :nocov: errmsg = "Failed bootstrap_directory for #{result.args[:tag]}: #{result.exception.class} #{result.exception.message}" raise OctocatalogDiff::Errors::BootstrapError, errmsg # :nocov: end end end # Performs the actual bootstrap of a directory. Intended to be called by bootstrap_directory_parallelizer # above, or as part of the parallelized catalog build process from util/catalogs. # @param options [Hash] Directory options: branch, path, tag # @param logger [Logger] Logger object def self.bootstrap_directory(options, logger) raise ArgumentError, ':path must be supplied' unless options[:path] FileUtils.mkdir_p(options[:path]) unless Dir.exist?(options[:path]) git_checkout(logger, options) if options[:branch] unless options[:bootstrap_script].nil? install_bootstrap_script(logger, options) run_bootstrap(logger, options) end end # Perform git checkout # @param logger [Logger] Logger object # @param options [Hash] Options (need to contain: basedir, branch, path) def self.git_checkout(logger, options) logger.debug("Begin git checkout #{options[:basedir]}:#{options[:branch]} -> #{options[:path]}") OctocatalogDiff::CatalogUtil::Git.check_out_git_archive(options.merge(logger: logger)) logger.debug("Success git checkout #{options[:basedir]}:#{options[:branch]} -> #{options[:path]}") rescue OctocatalogDiff::Errors::GitCheckoutError => exc logger.error("Git checkout error: #{exc}") raise OctocatalogDiff::Errors::BootstrapError, exc end # Install bootstrap script in the target directory. This allows proper bootstrapping from the # latest version of the script, not the script that was in place at the time that directory's branch # was committed. # @param logger [Logger] Logger object # @param opts [Hash] Directory options def self.install_bootstrap_script(logger, opts) # Verify that bootstrap file exists src = if opts[:bootstrap_script].start_with? '/' opts[:bootstrap_script] else File.join(opts[:basedir], opts[:bootstrap_script]) end raise OctocatalogDiff::Errors::BootstrapError, "Bootstrap script #{src} does not exist" unless File.file?(src) logger.debug('Begin install bootstrap script in target directory') # Create destination directory if needed dest = File.join(opts[:path], opts[:bootstrap_script]) dest_dir = File.dirname(dest) FileUtils.mkdir_p(dest_dir) unless File.directory?(dest_dir) # Copy file and make executable FileUtils.cp src, dest FileUtils.chmod 0o755, dest logger.debug("Success: copied #{src} to #{dest}") end # Execute the bootstrap. # @param logger [Logger] Logger object # @param opts [Hash] Directory options def self.run_bootstrap(logger, opts) logger.debug("Begin bootstrap with '#{opts[:bootstrap_script]}' in #{opts[:path]}") result = OctocatalogDiff::Bootstrap.bootstrap(opts) if opts[:debug_bootstrap] || result[:status_code] > 0 output = result[:output].split(/[\r\n]+/) output.each { |x| logger.debug("Bootstrap: #{x}") } end unless (result[:status_code]).zero? raise OctocatalogDiff::Errors::BootstrapError, "bootstrap failed for #{opts[:path]}: #{result[:output]}" end logger.debug("Success bootstrap in #{opts[:path]}") result[:output] end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/command.rb0000644000004100000410000001601413413744025025266 0ustar www-datawww-data# frozen_string_literal: true require 'fileutils' require 'open3' require 'shellwords' module OctocatalogDiff module CatalogUtil # Used to construct the command to run 'puppet' to construct the catalog. class Command # Constructor def initialize(options = {}, logger = nil) @options = options @logger = logger # Required parameters @compilation_dir = options[:compilation_dir] raise ArgumentError, 'Compile dir (:compilation_dir) must be a string' unless @compilation_dir.is_a?(String) raise Errno::ENOENT, "Compile dir #{@compilation_dir} doesn't exist" unless File.exist?(@compilation_dir) raise ArgumentError, "Compile dir #{@compilation_dir} not a directory" unless File.directory?(@compilation_dir) @node = options[:node] raise ArgumentError, 'Node must be specified to compile catalog' if @node.nil? || !@node.is_a?(String) # To be initialized on-demand @puppet_argv = nil @puppet_binary = nil end # Retrieve puppet_command, puppet_binary, puppet_argv def puppet_argv setup @puppet_argv end def puppet_binary setup @puppet_binary end def puppet_command setup [@puppet_binary, @puppet_argv].flatten.join(' ') end private # Build up the command line to run Puppet def setup return if @puppet_binary && @puppet_argv # Where is the puppet binary? @puppet_binary = @options[:puppet_binary] raise ArgumentError, 'Puppet binary was not supplied' if @puppet_binary.nil? raise Errno::ENOENT, "Puppet binary #{@puppet_binary} doesn't exist" unless File.file?(@puppet_binary) # Node to compile cmdline = [] cmdline.concat ['master', '--compile', Shellwords.escape(@node)] # storeconfigs? if @options[:storeconfigs] cmdline.concat %w(--storeconfigs --storeconfigs_backend=puppetdb) else cmdline << '--no-storeconfigs' end # enc? if @options[:enc] raise Errno::ENOENT, "Did not find ENC as expected at #{@options[:enc]}" unless File.file?(@options[:enc]) cmdline << '--node_terminus=exec' cmdline << "--external_nodes=#{Shellwords.escape(@options[:enc])}" end # Future parser? cmdline << '--parser=future' if @options[:parser] == :future # Path to facts, or a specific fact file? facts_terminus = @options.fetch(:facts_terminus, 'yaml') if facts_terminus == 'yaml' cmdline << "--factpath=#{Shellwords.escape(File.join(@compilation_dir, 'var', 'yaml', 'facts'))}" if @options[:fact_file].is_a?(String) && @options[:fact_file] =~ /.*\.(\w+)$/ fact_file = File.join(@compilation_dir, 'var', 'yaml', 'facts', "#{@node}.#{Regexp.last_match(1)}") FileUtils.cp @options[:fact_file], fact_file unless File.file?(fact_file) || @options[:fact_file] == fact_file end cmdline << '--facts_terminus=yaml' elsif facts_terminus == 'facter' cmdline << '--facts_terminus=facter' else raise ArgumentError, "Unrecognized facts_terminus setting: '#{facts_terminus}'" end # Some typical options for puppet cmdline.concat %w( --no-daemonize --no-ca --color=false --config_version="/bin/echo catalogscript" ) # Add environment - only make this variable if preserve_environments is used. # If preserve_environments is not used, the hard-coded 'production' here matches # up with the symlink created under the temporary directory structure. environ = @options.fetch(:environment, 'production') cmdline << "--environment=#{Shellwords.escape(environ)}" # For people who aren't running hiera, a hiera-config will not be generated when @options[:hiera_config] # is nil. For everyone else, the hiera config was generated/copied/munged in the 'builddir' class # and was installed into the compile directory and named hiera.yaml. unless @options[:hiera_config].nil? cmdline << "--hiera_config=#{Shellwords.escape(File.join(@compilation_dir, 'hiera.yaml'))}" end # Options with parameters cmdline << "--environmentpath=#{Shellwords.escape(File.join(@compilation_dir, 'environments'))}" cmdline << "--vardir=#{Shellwords.escape(File.join(@compilation_dir, 'var'))}" cmdline << "--logdir=#{Shellwords.escape(File.join(@compilation_dir, 'var'))}" cmdline << "--ssldir=#{Shellwords.escape(File.join(@compilation_dir, 'var', 'ssl'))}" cmdline << "--confdir=#{Shellwords.escape(@compilation_dir)}" # Other parameters provided by the user override_and_append_commandline_with_user_supplied_arguments(cmdline) # Return full command @puppet_argv = cmdline end # Private: Mutate the command line with arguments that were passed directly from the # user. This appends new arguments and overwrites existing arguments. # @param cmdline [Array] Existing command line - mutated by this method def override_and_append_commandline_with_user_supplied_arguments(cmdline) return unless @options[:command_line].is_a?(Array) @options[:command_line].each do |opt| # Validate format: Accept '--key=value' or '--key' only. unless opt =~ /\A--([^=\s]+)(=.+)?\z/ raise ArgumentError, "Command line option '#{opt}' does not match format '--SOME_OPTION=SOME_VALUE'" end key = Regexp.last_match(1) val = Regexp.last_match(2) # The key should not contain any shell metacharacters. Ensure that this is the case. unless key == Shellwords.escape(key) raise ArgumentError, "Command line option '#{key}' is invalid." end # If val is nil, then it's a '--key' argument. Else, it's a '--key=value' argument. Escape # the value to ensure it do not break the shell interpretation. new_setting = if val.nil? "--#{key}" else "--#{key}=#{Shellwords.escape(val.sub(/\A=/, ''))}" end # Determine if command line already contains this setting. If yes, the setting provided # here should override. If no, then append to the commandline. ind = key_position(cmdline, key) if ind.nil? cmdline << new_setting else cmdline[ind] = new_setting end end end # Private: Determine if the key (given by --key) is already defined in the # command line. Returns nil if it is not already defined, otherwise returns # the index. # @param cmdline [Array] Existing command line # @param key [String] Key to look up # @return [Integer] Index of where key is defined (nil if undefined) def key_position(cmdline, key) cmdline.index { |x| x == "--#{key}" || x =~ /\A--#{key}=/ } end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/builddir.rb0000644000004100000410000004317713413744025025460 0ustar www-datawww-data# frozen_string_literal: true require 'yaml' require_relative '../facts' require_relative 'enc' require_relative '../util/util' module OctocatalogDiff module CatalogUtil # Represents a directory that is created such that a catalog can be compiled # in it. This has the following major functions: # - Create the temporary directory that will serve as the puppet configuration directory # - Register a handler to remove the temporary directory upon exit # - Install needed configuration files within the directory (e.g. puppetdb.conf) # - Install the facts into the directory # - Install 'environments/(environment)' which is a symlink to the checkout of the puppet code class BuildDir # Allow the path to the temporary directory to be read attr_reader :tempdir, :enc, :fact_file # Constructor # Options for constructor: # :puppetdb_url [String] PuppetDB Server URLs # :puppetdb_server_url_timeout [Integer] Timeout (seconds) for puppetdb.conf # :facts [OctocatalogDiff::Facts] Facts object # :fact_file [String] File from which to read facts # :node [String] Node name # :basedir [String] Directory containing puppet code # :enc [String] ENC script file (can be relative or absolute path) # :pe_enc_url [String] ENC URL (for Puppet Enterprise node classification service) # :hiera_config [String] hiera configuration file (relative to base directory) # :hiera_path [String] relative path to hiera data files (mutually exclusive with :hiera_path_strip) # :hiera_path_strip [String] string to strip off the beginning of :datadir # :puppetdb_ssl_ca [String] Path to SSL CA certificate # :puppetdb_ssl_client_key [String] String representation of SSL client key # :puppetdb_ssl_client_cert [String] String representation of SSL client certificate # :puppetdb_ssl_client_password [String] Password to unlock SSL private key # @param options [Hash] Options for class; see above description def initialize(options = {}, logger = nil) @options = options.dup @tempdir = OctocatalogDiff::Util::Util.temp_dir('ocd-builddir-') @factdir = nil @enc = nil @fact_file = nil @node = options[:node] @facts_terminus = options.fetch(:facts_terminus, 'yaml') create_structure create_symlinks(logger) # These configurations are optional. Don't call the methods if parameters are nil. unless options[:puppetdb_url].nil? install_puppetdb_conf(logger, options[:puppetdb_url], options[:puppetdb_server_url_timeout]) install_routes_yaml(logger) end install_hiera_config(logger, options) unless options[:hiera_config].nil? @fact_file = install_fact_file(logger, options) if @facts_terminus == 'yaml' @enc = install_enc(logger) unless options[:enc].nil? && options[:pe_enc_url].nil? install_ssl(logger, options) if options[:puppetdb_ssl_ca] || options[:puppetdb_ssl_client_cert] end # Create common structure def create_structure %w(facts var var/ssl var/yaml var/yaml/facts).each do |dir| Dir.mkdir(File.join(@tempdir, dir)) FileUtils.chmod 0o755, File.join(@tempdir, dir) end end # Create symlinks. # # If the `--preserve-environments` option is used, the `environments` directory, plus `modules` and # `manifests` symlinks are created. Otherwise, `environments/production` is pointed at the base # directory. # # @param logger [Logger] Logger object def create_symlinks(logger = nil) if @options[:preserve_environments] install_directory_symlink(logger, File.join(@options[:basedir], 'environments'), 'environments') @options.fetch(:create_symlinks, %w(modules manifests)).each do |x| install_directory_symlink(logger, File.join(@options[:basedir], x), x) end else if @options[:create_symlinks] && @options[:environment] unless logger.nil? logger.warn '--create-symlinks with --environment ignored unless --preserve-environments is used' end elsif @options[:create_symlinks] logger.warn '--create-symlinks is ignored unless --preserve-environments is used' unless logger.nil? elsif @options[:environment] return install_directory_symlink(logger, @options[:basedir], "environments/#{@options[:environment]}") end install_directory_symlink(logger, @options[:basedir]) end end # Install puppetdb.conf file in temporary directory # @param server_urls [String] String for server_urls in puppetdb.conf # @param server_url_timeout [Integer] Value for server_url_timeout in puppetdb.conf def install_puppetdb_conf(logger, server_urls, server_url_timeout = 30) unless server_urls.is_a?(String) raise ArgumentError, "server_urls must be a string, got a: #{server_urls.class}" end server_url_timeout ||= 30 # If called with nil argument, supply default unless server_url_timeout.is_a?(Integer) raise ArgumentError, "server_url_timeout must be a fixnum, got a: #{server_url_timeout.class}" end puppetdb_conf = File.join(@tempdir, 'puppetdb.conf') File.open(puppetdb_conf, 'w') do |f| f.write "[main]\n" f.write "server_urls = #{server_urls}\n" f.write "server_url_timeout = #{server_url_timeout}\n" end logger.debug("Installed puppetdb.conf file at #{puppetdb_conf}") end # Install routes.yaml file in temporary directory # No parameters or return - thus just writes a file (and notes it to debugging log) # Note: catalog cache => json avoids sending the compiled catalog to PuppetDB # even if storeconfigs is enabled. def install_routes_yaml(logger) routes_yaml = File.join(@tempdir, 'routes.yaml') routes_hash = { 'master' => { 'facts' => { 'terminus' => @facts_terminus, 'cache' => 'yaml' }, 'catalog' => { 'cache' => 'json' } } } File.open(routes_yaml, 'w') { |f| f.write(routes_hash.to_yaml) } logger.debug("Installed routes.yaml file at #{routes_yaml}") end # Install the fact file in temporary directory # @param options [Hash] Options def install_fact_file(logger, options) unless @facts_terminus == 'yaml' raise ArgumentError, "Called install_fact_file but :facts_terminus = #{@facts_terminus}" end unless options[:node].is_a?(String) && !options[:node].empty? raise ArgumentError, 'Called install_fact_file without node, or with an empty node' end facts = if options[:facts].is_a?(OctocatalogDiff::Facts) options[:facts].dup elsif options[:fact_file] raise Errno::ENOENT, "Fact file #{options[:fact_file]} does not exist" unless File.file?(options[:fact_file]) fact_file_opts = { fact_file_string: File.read(options[:fact_file]) } fact_file_opts[:backend] = Regexp.last_match(1).to_sym if options[:fact_file] =~ /.*\.(\w+)$/ OctocatalogDiff::Facts.new(fact_file_opts) else raise ArgumentError, 'No facts passed to "install_fact_file" method' end if options[:fact_override].is_a?(Array) options[:fact_override].each do |override| keys = override.key.is_a?(Regexp) ? facts.matching(override.key) : [override.key] keys.each do |key| old_value = facts.fact(key) facts.override(key, override.value) logger.debug("Override #{key} from #{old_value.inspect} to #{override.value.inspect}") end end end fact_file_out = File.join(@tempdir, 'var', 'yaml', 'facts', "#{options[:node]}.yaml") File.open(fact_file_out, 'w') { |f| f.write(facts.facts_to_yaml(options[:node])) } logger.debug("Installed fact file at #{fact_file_out}") fact_file_out end # Install symbolic link to puppet environment # @param dir [String] Directory to link to # @param target [String] Where the symlink is created, relative to tempdir def install_directory_symlink(logger, dir, target = 'environments/production') raise ArgumentError, "Called install_directory_symlink with #{dir.class} argument" unless dir.is_a?(String) raise Errno::ENOENT, "Specified directory #{dir} doesn't exist" unless File.directory?(dir) symlink_target = File.join(@tempdir, target) if target =~ %r{/} parent_dir = File.dirname(symlink_target) FileUtils.mkdir_p parent_dir end FileUtils.rm_f symlink_target if File.exist?(symlink_target) FileUtils.symlink dir, symlink_target logger.debug("Symlinked #{symlink_target} -> #{dir}") end # Install ENC # @param enc [String] Path to ENC script, relative to checkout def install_enc(logger) raise ArgumentError, 'A node must be specified when using an ENC' unless @node.is_a?(String) enc_obj = OctocatalogDiff::CatalogUtil::ENC.new(@options.merge(tempdir: @tempdir)) enc_obj.execute(logger) raise "Failed ENC: #{enc_obj.error_message}" if enc_obj.error_message enc_path = File.join(@tempdir, 'enc.sh') File.open(enc_path, 'w') do |f| f.write "#!/bin/sh\n" f.write "cat <<-EOF\n" f.write enc_obj.content f.write "\nEOF\n" end FileUtils.chmod 0o755, enc_path logger.debug("Installed ENC to echo content, #{enc_obj.content.length} bytes") enc_path end # Install hiera config file # @param options [Hash] Options hash def install_hiera_config(logger, options) # Validate hiera config file hiera_config = options[:hiera_config] unless hiera_config.is_a?(String) raise ArgumentError, "Called install_hiera_config with a #{hiera_config.class} argument" end file_src = if hiera_config.start_with? '/' hiera_config elsif hiera_config =~ %r{^environments/#{Regexp.escape(environment)}/} File.join(@tempdir, hiera_config) else File.join(@tempdir, 'environments', environment, hiera_config) end raise Errno::ENOENT, "hiera.yaml (#{file_src}) wasn't found" unless File.file?(file_src) # Munge datadir in hiera config file obj = YAML.load_file(file_src) version = obj['version'] || obj[:version] || 3 if version.to_i == 5 update_hiera_config_v5(logger, options, obj) else update_hiera_config_v3(logger, options, obj) end # Write properly formatted hiera config file into temporary directory File.open(File.join(@tempdir, 'hiera.yaml'), 'w') { |f| f.write(obj.to_yaml.gsub('!ruby/sym ', ':')) } logger.debug("Installed hiera.yaml from #{file_src} to #{File.join(@tempdir, 'hiera.yaml')}") end # Install SSL certificate authority certificate, client key, and client certificate into the # expected locations within Puppet's SSL directory. Note that if the client key has a password, # this will write the key (without password) onto disk, because Puppet doesn't support unlocking # the private key. # @param logger [Logger] Logger object # @param options [Hash] Options hash def install_ssl(logger, options) return unless options[:puppetdb_ssl_client_cert] || options[:puppetdb_ssl_client_key] || options[:puppetdb_ssl_ca] # Create directory structure expected by Puppet %w(var/ssl/certs var/ssl/private var/ssl/private_keys).each do |dir| Dir.mkdir(File.join(@tempdir, dir)) FileUtils.chmod 0o700, File.join(@tempdir, dir) end # SSL client auth requested? if options[:puppetdb_ssl_client_cert] || options[:puppetdb_ssl_client_key] raise ArgumentError, '--puppetdb-ssl-ca must be provided for client auth' unless options[:puppetdb_ssl_ca] raise ArgumentError, '--puppetdb-ssl-client-cert must be provided' unless options[:puppetdb_ssl_client_cert] raise ArgumentError, '--puppetdb-ssl-client-key must be provided' unless options[:puppetdb_ssl_client_key] install_ssl_client(logger, options) end # SSL CA provided? install_ssl_ca(logger, options) if options[:puppetdb_ssl_ca] end private # Jump-off for hiera v3 (or earlier) # @param logger [Logger] Logger object # @param options [Hash] Options hash # @param obj [Hash] Parsed hiera.yaml file def update_hiera_config_v3(logger, options, obj) ([obj[:backends]].flatten || %w(yaml json)).each do |key| next unless obj.key?(key.to_sym) && !obj[key.to_sym][:datadir].nil? obj[key.to_sym][:datadir] = hiera_munge(options, obj[key.to_sym][:datadir]) # Make sure the directory exists. If not, log a warning. This is *probably* a setup error, but we don't # want it to be fatal in case (for example) someone is doing an octocatalog-diff to verify moving this # directory around or even setting up Hiera for the very first time. unless File.directory?(obj[key.to_sym][:datadir]) message = "WARNING: Hiera datadir for #{key} doesn't seem to exist at #{obj[key.to_sym][:datadir]}" logger.warn message end end end # Jump-off for hiera v5 # @param logger [Logger] Logger object # @param options [Hash] Options hash # @param obj [Hash] Parsed hiera.yaml file def update_hiera_config_v5(_logger, options, obj) defaults_key = obj.key?(:defaults) ? :defaults : 'defaults' hierarchy_key = obj.key?(:hierarchy) ? :hierarchy : 'hierarchy' # Fix defaults:datadir if obj[defaults_key].is_a?(Hash) [:datadir, 'datadir'].each do |key| next unless obj[defaults_key].key?(key) obj[defaults_key][key] = hiera_munge(options, obj[defaults_key][key]) end end # Fix hierarchy:datadir if obj[hierarchy_key].is_a?(Array) obj[hierarchy_key].each do |level| [:datadir, 'datadir'].each do |key| next unless level.key?(key) if options[:hiera_path_strip].is_a?(String) level[key] = hiera_munge(options, level[key]) elsif options[:hiera_path].is_a?(String) message = [ "Hierarchy item #{level.inspect} has a datadir.", '--hiera-path is not supported in this situation.', 'Please use --hiera-path-strip.' ].join(' ') raise ArgumentError, message end end end end end # Hiera munge - shared method to apply :hiera_path_strip and :hiera_path def hiera_munge(options, current_value) return if current_value.nil? if options[:hiera_path_strip].is_a?(String) rexp1 = Regexp.new('^' + options[:hiera_path_strip]) current_value.sub!(rexp1, @tempdir) elsif options[:hiera_path].is_a?(String) current_value = File.join(@tempdir, 'environments', environment, options[:hiera_path]) end rexp2 = Regexp.new('%{(::)?environment}') current_value.sub!(rexp2, environment) current_value end # Install SSL certificate authority certificate # @param logger [Logger] Logger object # @param options [Hash] Options hash def install_ssl_ca(logger, options) ca_file = options[:puppetdb_ssl_ca] raise Errno::ENOENT, 'SSL CA file does not exist' unless File.file?(ca_file) ca_content = File.read(ca_file) ca_outfile = File.join(@tempdir, 'var', 'ssl', 'certs', 'ca.pem') File.open(ca_outfile, 'w') { |f| f.write(ca_content) } logger.debug "Installed CA certificate in #{ca_outfile}" end # Install SSL keypair for client certificate authentication # @param logger [Logger] Logger object # @param options [Hash] Options hash def install_ssl_client(logger, options) # Since Puppet always looks for the key and cert in a file named after the hostname, determine the # hostname here for the purposes of naming the files. require 'socket' host = Socket.gethostname install_ssl_client_cert(logger, host, options[:puppetdb_ssl_client_cert]) install_ssl_client_key(logger, host, options[:puppetdb_ssl_client_key]) install_ssl_client_password(logger, options[:puppetdb_ssl_client_password]) end def install_ssl_client_cert(logger, host, content) cert_outfile = File.join(@tempdir, 'var', 'ssl', 'certs', "#{host}.pem") File.open(cert_outfile, 'w') { |f| f.write(content) } logger.debug "Installed SSL client certificate in #{cert_outfile}" end def install_ssl_client_key(logger, host, content) key_outfile = File.join(@tempdir, 'var', 'ssl', 'private_keys', "#{host}.pem") File.open(key_outfile, 'w') { |f| f.write(content) } logger.debug "Installed SSL client key in #{key_outfile}" end def install_ssl_client_password(logger, password) return unless password password_outfile = File.join(@tempdir, 'var', 'ssl', 'private', 'password') File.open(password_outfile, 'w') { |f| f.write(password) } logger.debug "Installed SSL client key password in #{password_outfile}" end def environment @options.fetch(:environment, 'production') end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/enc/0000755000004100000410000000000013413744025024066 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/enc/pe.rb0000644000004100000410000000761213413744025025025 0ustar www-datawww-data# frozen_string_literal: true require_relative 'pe/v1' require_relative '../../util/httparty' require_relative '../../errors' require_relative '../facts' module OctocatalogDiff module CatalogUtil class ENC # Support the Puppet Enterprise classification API. # Documentation: https://docs.puppet.com/pe/latest/nc_index.html class PE # Allow the main ENC object to retrieve these values attr_reader :content, :error_message # Constructor # @param options [Hash] Options - must contain the Puppet Enterprise URL and the node def initialize(options) # Make sure the node is in the options raise ArgumentError, 'OctocatalogDiff::CatalogUtil::ENC::PE#new requires :node' unless options.key?(:node) @node = options[:node] # Retrieve the base URL for the Puppet Enterprise ENC service raise ArgumentError, 'OctocatalogDiff::CatalogUtil::ENC::PE#new requires :pe_enc_url' unless options.key?(:pe_enc_url) # Save options @options = options # Get the object corresponding to the version of the API in use. # (Right now this is hard-coded at V1 because that is the only version there is. In the future # if there are different versions, this will need to be parameterized.) @api = OctocatalogDiff::CatalogUtil::ENC::PE::V1.new(@options) # Initialize the content and error message @content = nil @error_message = 'The execute method was never run' end # Executor # @param logger [Logger] Logger object def execute(logger) logger.debug "Beginning OctocatalogDiff::CatalogUtil::ENC::PE#execute for #{@node}" @options[:facts] ||= facts(logger) return unless @options[:facts] more_options = { headers: @api.headers, timeout: @options[:timeout] || 10 } post_hash = @api.body url = @api.url response = OctocatalogDiff::Util::HTTParty.post(url, @options.merge(more_options), post_hash, 'pe_enc') unless response[:code] == 200 logger.debug "PE ENC failed: #{response.inspect}" logger.error "PE ENC failed: Response from #{url} was #{response[:code]}" @error_message = "Response from #{url} was #{response[:code]}" return end logger.debug "Response from #{url} was #{response[:code]}" unless response[:parsed].is_a?(Hash) logger.error "PE ENC failed: Response from #{url} was not a hash! #{response[:parsed].inspect}" @error_message = "PE ENC failed: Response from #{url} was not a hash! #{response[:parsed].class}" return end begin @content = @api.result(response[:parsed], logger) @error_message = nil rescue OctocatalogDiff::Errors::PEClassificationError => exc @error_message = exc.message logger.error "PE ENC failed: #{exc.message}" return end logger.debug "Completed OctocatalogDiff::CatalogUtil::ENC::PE#execute for #{@node}" end # Facts # @param logger [Logger] Logger object # @return [OctocatalogDiff::Facts] Facts object def facts(logger) facts_obj = OctocatalogDiff::CatalogUtil::Facts.new(@options, logger) logger.debug "Start retrieving facts for #{@node} from #{self.class}" begin result = facts_obj.facts logger.debug "Success retrieving facts for #{@node} from #{self.class}" rescue OctocatalogDiff::Errors::FactRetrievalError, OctocatalogDiff::Errors::FactSourceError => exc @content = nil @error_message = "Fact retrieval failed: #{exc.class} - #{exc.message}" logger.error @error_message result = nil end result end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/enc/noop.rb0000644000004100000410000000056213413744025025371 0ustar www-datawww-data# frozen_string_literal: true module OctocatalogDiff module CatalogUtil class ENC # No-op ENC. class Noop # Constructor def initialize(_options) end # Retrieve content def content '' end # Error message def error_message nil end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/enc/pe/0000755000004100000410000000000013413744025024472 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/enc/pe/v1.rb0000644000004100000410000000400113413744025025340 0ustar www-datawww-data# frozen_string_literal: true require 'json' require 'uri' require 'yaml' require_relative '../../../errors' module OctocatalogDiff module CatalogUtil class ENC class PE # Support the Puppet Enterprise classification API. # Documentation: https://docs.puppet.com/pe/latest/nc_index.html # This is version 1 of the API class V1 # Constructor # @param options [Hash] All input options def initialize(options) @options = options end # Return the URL to the API # @return [String] API URL def url "#{@options[:pe_enc_url]}/v1/classified/nodes/#{@options[:node]}" end # Headers # @return [Hash] Headers for request def headers result = { 'Accept' => 'application/json', 'Content-Type' => 'application/json' } result['X-Authentication'] = @options[:pe_enc_token] if @options[:pe_enc_token] result end # POST body # @return [String] POST body data def body raise ":facts required (got #{@options[:facts].class})" unless @options[:facts].is_a?(OctocatalogDiff::Facts) { 'fact' => @options[:facts].facts }.to_json end # Parse response from ENC and return the final ENC data # @param parsed [Parsed response] Parsed response from ENC # @param logger [Logger] Logger.object # @return [String] ENC data as text def result(parsed, logger) %w(classes parameters).each do |required_key| next if parsed[required_key] logger.debug parsed.keys.inspect raise OctocatalogDiff::Errors::PEClassificationError, "Response missing: #{required_key}" end obj = { 'classes' => parsed['classes'], 'parameters' => parsed['parameters'] } obj.to_yaml end end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/enc/script.rb0000644000004100000410000000722213413744025025722 0ustar www-datawww-data# frozen_string_literal: true require 'fileutils' require 'open3' require 'shellwords' require 'tempfile' module OctocatalogDiff module CatalogUtil class ENC # Support an ENC that executes a script on this system which returns the ENC data on STDOUT. class Script attr_reader :content, :error_message, :script # Constructor # @param options [Hash] Options - must contain script name and node name, plus tempdir if it's a relative path def initialize(options) # Make sure the node is in the options raise ArgumentError, 'OctocatalogDiff::CatalogUtil::ENC::Script#new requires :node' unless options.key?(:node) @node = options[:node] # Determine path to ENC and make sure it exists raise ArgumentError, 'OctocatalogDiff::CatalogUtil::ENC::Script#new requires :enc' unless options.key?(:enc) @script = script_path(options[:enc], options[:tempdir]) # Other options we may recognize @pass_env_vars = options.fetch(:pass_env_vars, []) # Initialize the content and error message @content = nil @error_message = 'The execute method was never run' end # Executor # @param logger [Logger] Logger object def execute(logger) logger.debug "Beginning OctocatalogDiff::CatalogUtil::ENC::Script#execute for #{@node} with #{@script}" logger.debug "Passing these extra environment variables: #{@pass_env_vars}" if @pass_env_vars.any? # Copy the script and make it executable # Then run the command in the restricted environment raise Errno::ENOENT, "ENC #{@script} wasn't found" unless File.file?(@script) file = Tempfile.open('enc.sh') file.close begin FileUtils.cp @script, file.path FileUtils.chmod 0o755, file.path env = { 'HOME' => ENV['HOME'], 'PATH' => ENV['PATH'], 'PWD' => File.dirname(@script) } @pass_env_vars.each { |var| env[var] ||= ENV[var] } command = [file.path, @node].map { |x| Shellwords.escape(x) }.join(' ') out, err, status = Open3.capture3(env, command, unsetenv_others: true, chdir: File.dirname(@script)) logger.debug "ENC exited #{status.exitstatus}: #{out.length} bytes to STDOUT, #{err.length} bytes to STDERR" ensure file.unlink end # Analyze the output if status.exitstatus.zero? @content = out @error_message = nil logger.warn "ENC STDERR: #{err}" unless err.empty? else @content = nil @error_message = "ENC failed with status #{status.exitstatus}: #{out} #{err}" logger.error "ENC failed - Status #{status.exitstatus}" logger.error "Failed ENC printed this to STDOUT: #{out}" unless out.empty? logger.error "Failed ENC printed this to STDERR: #{err}" unless err.empty? end end private # Determine the script path for the incoming file -- absolute or relative # @param enc [String] Path to ENC supplied by user/config # @param tempdir [String] # @return [String] Full path to file on system def script_path(enc, tempdir) return enc if enc.start_with? '/' raise ArgumentError, 'OctocatalogDiff::CatalogUtil::ENC::Script#new requires :tempdir' unless tempdir.is_a?(String) return File.join(tempdir, enc) if enc =~ %r{^environments/[^/]+/} File.join(tempdir, 'environments', 'production', enc) end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/git.rb0000644000004100000410000000536513413744025024442 0ustar www-datawww-data# frozen_string_literal: true require 'rugged' require_relative '../errors' require_relative '../util/scriptrunner' module OctocatalogDiff module CatalogUtil # Class to perform a git checkout (via 'git archive') of a branch from the base git # directory into another targeted directory. class Git # Check out a branch via 'git archive' from one directory into another. # @param options [Hash] Options hash: # - :branch => Branch name to check out # - :path => Where to check out to (must exist as a directory) # - :basedir => Where to check out from (must exist as a directory) # - :logger => Logger object def self.check_out_git_archive(options = {}) branch = options.fetch(:branch) path = options.fetch(:path) dir = options.fetch(:basedir) logger = options.fetch(:logger) override_script_path = options.fetch(:override_script_path, nil) # Validate parameters if dir.nil? || !File.directory?(dir) raise OctocatalogDiff::Errors::GitCheckoutError, "Source directory #{dir.inspect} does not exist" end if path.nil? || !File.directory?(path) raise OctocatalogDiff::Errors::GitCheckoutError, "Target directory #{path.inspect} does not exist" end # Create and execute checkout script sr_opts = { logger: logger, default_script: 'git-extract/git-extract.sh', override_script_path: override_script_path } script = OctocatalogDiff::Util::ScriptRunner.new(sr_opts) sr_run_opts = { :working_dir => dir, :pass_env_vars => options[:pass_env_vars], 'OCD_GIT_EXTRACT_BRANCH' => branch, 'OCD_GIT_EXTRACT_TARGET' => path } begin script.run(sr_run_opts) logger.debug("Success git archive #{dir}:#{branch}") rescue OctocatalogDiff::Util::ScriptRunner::ScriptException raise OctocatalogDiff::Errors::GitCheckoutError, "Git archive #{branch}->#{path} failed: #{script.output}" end end # Determine the SHA of origin/master (or any other branch really) in the git repo # @param options [Hash] Options hash: # - :branch => Branch name to determine SHA of # - :basedir => Where to check out from (must exist as a directory) def self.branch_sha(options = {}) branch = options.fetch(:branch) dir = options.fetch(:basedir) if dir.nil? || !File.directory?(dir) raise Errno::ENOENT, "Git directory #{dir.inspect} does not exist" end repo = Rugged::Repository.new(dir) repo.branches[branch].target_id end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/cached_master_directory.rb0000644000004100000410000002127213413744025030520 0ustar www-datawww-data# frozen_string_literal: true require_relative 'bootstrap' require_relative 'git' require_relative '../util/catalogs' require 'fileutils' module OctocatalogDiff module CatalogUtil # Handle the bootstrapped and cached checkout of [master branch]. This is an optimization # targeted at local development environments, since a frequent pattern is "run a catalog-diff # between what I have here, and master." # # Please note that there could be a race condition here if this code was run in parallel (i.e., # the cached master directory is blown away and re-created when a Puppet catalog compile is in # progress). Do not introduce this code to an environment where catalog-diff may be running in # parallel unless you have accounted for this (or are willing to tolerate any errors). class CachedMasterDirectory # Set default branch. Can be overridden by options[:master_cache_branch]. DEFAULT_MASTER_BRANCH = 'origin/master'.freeze # Get the master branch based on supplied options. # @param options [Hash] Options hash # @return [String] Master branch configured (defaults to DEFAULT_MASTER_BRANCH) def self.master_branch(options = {}) options.fetch(:master_cache_branch, DEFAULT_MASTER_BRANCH) end # This is the entry point from the CLI (or anywhere else). Takes options hash and logger # as arguments, sets up the cached master directory if required, and adjusts options hash # accordingly. Returns nothing; raises exceptions for failures. # @param options [Hash] Options hash from CLI # @param logger [Logger] Logger object def self.run(options, logger) # If nobody asked for this, don't do anything return if options[:cached_master_dir].nil? # Verify that parameters are set up correctly and that at least one of the to-branch and # from-branch is [master branch]. If not, it's not worthwhile to do any of the remaining # tasks in this section. return unless cached_master_applicable_to_this_run?(options) # This directory was supposed to be created as part of the option setup. Make sure it exists # as a sanity check. Dir.mkdir options[:cached_master_dir], 0o755 unless Dir.exist?(options[:cached_master_dir]) # Determine if it's necessary to check out the git repo to the directory in question. git_repo_checkout_bootstrap(options, logger) unless git_repo_checkout_current?(options, logger) # Under --bootstrap-then-exit, don't adjust the options. (Otherwise code runs twice.) return if options[:bootstrap_then_exit] # Re-point any options to the cached directory. %w(from to).each do |x| next unless options["#{x}_env".to_sym] == master_branch(options) logger.debug "Setting --bootstrapped-#{x}-dir=#{options[:cached_master_dir]}" options["bootstrapped_#{x}_dir".to_sym] = options[:cached_master_dir] end # If a catalog was already compiled for the requested node, point to it directly to avoid # re-compiling said catalog. unless options[:node].nil? catalog_path = File.join(options[:cached_master_dir], '.catalogs', options[:node] + '.json') if File.file?(catalog_path) %w(from to).each do |x| next unless options["#{x}_env".to_sym] == master_branch(options) next unless options["#{x}_catalog".to_sym].nil? logger.debug "Setting --#{x}-catalog=#{catalog_path}" options["#{x}_catalog".to_sym] = catalog_path options["#{x}_catalog_compilation_dir".to_sym] = options[:cached_master_dir] end end end end # Determine if the cached master directory functionality is needed at all. # @param options [Hash] Options hash from CLI # @return [Boolean] whether to-branch and/or from-branch == [master branch] def self.cached_master_applicable_to_this_run?(options) return false if options[:cached_master_dir].nil? target_branch = master_branch(options) options.fetch(:from_env, '') == target_branch || options.fetch(:to_env, '') == target_branch end # Determine whether git repo checkout in the directory is current. # To consider here: (a) is anything at all checked out; (b) is the correct SHA checked out? # @param options [Hash] Options hash from CLI # @param logger [Logger] Logger object # @return [Boolean] whether git repo checkout in the directory is current def self.git_repo_checkout_current?(options, logger) shafile = File.join(options[:cached_master_dir], '.catalog-diff-master.sha') return false unless File.file?(shafile) bootstrapped_sha = File.read(shafile) target_branch = master_branch(options) branch_sha_opts = options.merge(branch: target_branch) current_master_sha = OctocatalogDiff::CatalogUtil::Git.branch_sha(branch_sha_opts) logger.debug "Cached master dir: bootstrapped=#{bootstrapped_sha}; current=#{current_master_sha}" bootstrapped_sha.strip == current_master_sha.strip end # Check out [master branch] -> cached directory and bootstrap it # @param options [Hash] Options hash from CLI # @param logger [Logger] Logger object def self.git_repo_checkout_bootstrap(options, logger) # This directory isn't current so kill it # Too dangerous if someone slips up on the command line: # FileUtils.rm_rf options[:cached_master_dir] if Dir.exist?(options[:cached_master_dir]) shafile = File.join(options[:cached_master_dir], '.catalog-diff-master.sha') target_branch = master_branch(options) branch_sha_opts = options.merge(branch: target_branch) current_master_sha = OctocatalogDiff::CatalogUtil::Git.branch_sha(branch_sha_opts) if Dir.exist?(options[:cached_master_dir]) && File.exist?(shafile) # If :cached_master_dir was set in a known-safe manner, safe_to_delete_cached_master_dir will # allow the cleanup to take place automatically. if options.fetch(:safe_to_delete_cached_master_dir, false) == options[:cached_master_dir] FileUtils.rm_rf options[:cached_master_dir] if Dir.exist?(options[:cached_master_dir]) else message = "To proceed, #{options[:cached_master_dir]} needs to be deleted, so it can be re-created."\ " I'm not yet deemed safe enough to do this for you though. Please jump out to a shell and run"\ " 'rm -rf #{options[:cached_master_dir]}' and then come back and try again. (Existing SHA:"\ " #{File.read(shafile).strip}; current master SHA: #{current_master_sha})" raise Errno::EEXIST, message end end # This logic is similar to 'bootstrap-then-exit' (without the exit part). The # bootstrap_then_exit handles creating this directory. fake_options = options.dup fake_options[:bootstrap_then_exit] = true fake_options[:bootstrapped_from_dir] = options[:cached_master_dir] fake_options[:bootstrapped_to_dir] = nil fake_options[:from_env] = master_branch(options) logger.debug 'Begin bootstrap cached master directory' catalogs_obj = OctocatalogDiff::Util::Catalogs.new(fake_options, logger) catalogs_obj.bootstrap_then_exit logger.debug 'Success bootstrap cached master directory' # Write the SHA of [master branch], so git_repo_checkout_current? works next time File.open(shafile, 'w') { |f| f.write(current_master_sha) } logger.debug "Cached master directory bootstrapped to #{current_master_sha}" # Create /.catalogs, to save any catalogs compiled along the way catalog_dir = File.join(options[:cached_master_dir], '.catalogs') Dir.mkdir catalog_dir unless File.directory?(catalog_dir) end # Save a compiled catalog in the cached master directory. Does not die fatally if # catalog is invalid or this isn't set up or whatever else. # @param node [String] node name # @param dir [String] cached master directory # @param catalog [OctocatalogDiff::Catalog] Catalog object # @return [Boolean] true if catalog was saved, false if not def self.save_catalog_in_cache_dir(node, dir, catalog) return false if dir.nil? || node.nil? return false if catalog.nil? || !catalog.valid? path = File.join(dir, '.catalogs') return false unless Dir.exist?(path) filepath = File.join(path, node + '.json') return false if File.file?(filepath) File.open(filepath, 'w') { |f| f.write(catalog.catalog_json) } true end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/facts.rb0000644000004100000410000000704713413744025024756 0ustar www-datawww-data# frozen_string_literal: true require_relative '../facts' module OctocatalogDiff module CatalogUtil # Helper class to construct a fact object based on options provided by # cli/options. Supports a direct fact file, looking up a YAML file based on # node name within Puppet fact directories, or retrieving from PuppetDB. class Facts # Constructor # @param options [Hash] Options from cli/options # @param logger [Logger] Logger object for debug messages (optional) def initialize(options, logger = nil) raise ArgumentError, "Argument to constructor must be Hash not #{options.class}" unless options.is_a?(Hash) @options = options.dup @logger = logger # Environment variable recognition @options[:puppetdb_url] ||= ENV['PUPPETDB_URL'] if ENV['PUPPETDB_URL'] @options[:puppet_fact_dir] ||= ENV['PUPPET_FACT_DIR'] if ENV['PUPPET_FACT_DIR'] end # Compute facts if needed and then return them # @return [Hash] Facts def facts @facts ||= compute_facts end private # Retrieve facts from a YAML file in the puppet facts directory # @param filename [String] Full path to file to read in # @return [OctocatalogDiff::Facts] Facts object def facts_from_file(filename) @logger.debug("Retrieving facts from #{filename}") unless @logger.nil? opts = { node: @options[:node], backend: :yaml, fact_file_string: File.read(filename) } OctocatalogDiff::Facts.new(opts) end # Retrieve facts from PuppetDB. Either options[:puppetdb_url] or ENV['PUPPETDB_URL'] # needs to be set for this to work. Node name must also be set in options. # @return [OctocatalogDiff::Facts] Facts object def facts_from_puppetdb @logger.debug('Retrieving facts from PuppetDB') unless @logger.nil? OctocatalogDiff::Facts.new(@options.merge(backend: :puppetdb, retry: 2)) end # Error message when the node is needed but not defined # :nocov: def error_node_not_provided message = 'Unable to determine facts. You must either supply "--fact-file FILENAME"' \ ' or a node name "-n NODENAME" to look up a set of node facts in a fact' \ ' directory or in PuppetDB.' raise ArgumentError, message end # :nocov: # Does the actual computation/lookup of facts. Seeks to return a OctocatalogDiff::Facts # object. Raises error if no fact sources are found. # @return [OctocatalogDiff::Facts] Facts object def compute_facts if @options.key?(:facts) && @options[:facts].is_a?(OctocatalogDiff::Facts) return @options[:facts] end if @options.key?(:fact_file) raise Errno::ENOENT, 'Specified fact file does not exist' unless File.file?(@options[:fact_file]) return facts_from_file(@options[:fact_file]) end error_node_not_provided if @options[:node].nil? if @options[:puppet_fact_dir] && File.directory?(@options[:puppet_fact_dir]) filename = File.join(@options[:puppet_fact_dir], @options[:node] + '.yaml') return facts_from_file(filename) if File.file?(filename) end return facts_from_puppetdb if @options[:puppetdb_url] message = 'Unable to compute facts for node. Please use "--fact-file FILENAME" option' \ ' or set one of these environment variables: PUPPET_FACT_DIR or PUPPETDB_URL.' raise ArgumentError, message end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-util/enc.rb0000644000004100000410000001050613413744025024415 0ustar www-datawww-data# frozen_string_literal: true require_relative 'enc/noop' require_relative 'enc/pe' require_relative 'enc/script' require 'stringio' require 'yaml' module OctocatalogDiff module CatalogUtil # Support a generic ENC. It must use one of the supported backends found in the # 'enc' subdirectory. class ENC attr_reader :builder # Constructor # @param :backend [Symbol] If set, this will force a backend # @param :enc [String] Path to ENC script (node_terminus = exec) # @param # FIXME: Add support for PE's ENC endpoint API def initialize(options = {}) @options = options # Determine appropriate backend based on options supplied @enc_obj = backend # Initialize instance variables for content and error message. @builder = @enc_obj.class.to_s # Set the executed flag to false, so that it can be executed when something is retrieved. @executed = false end # Retrieve content # @return [String] ENC content, or nil if there was an error def content(logger = nil) execute(logger) @content ||= @enc_obj.content end # Retrieve error message # @return [String] Error message, or nil if there was no error def error_message(logger = nil) execute(logger) @error_message ||= @enc_obj.error_message end # Execute the 'execute' method of the object, but only once # @param [Logger] Logger (optional) - if not supplied any logger messages will be discarded def execute(logger = nil) return if @executed logger ||= @options[:logger] logger ||= Logger.new(StringIO.new) @enc_obj.execute(logger) if @enc_obj.respond_to?(:execute) @executed = true override_enc_parameters(logger) end private # Override of ENC parameters with parameters specified on the command line. # Modifies structures in @enc_obj. # @param logger [Logger] Logger object def override_enc_parameters(logger) return unless @options[:enc_override].is_a?(Array) && @options[:enc_override].any? content_structure = YAML.load(content) @options[:enc_override].each do |x| keys = x.key.is_a?(Regexp) ? content_structure.keys.select { |y| x.key.match(y) } : [x.key] keys.each do |key| merge_enc_param(content_structure, key, x.value) logger.debug "ENC override: #{key} #{x.value.nil? ? 'DELETED' : '= ' + x.value.inspect}" end end @content = content_structure.to_yaml end # Merging behavior for ENC overrides # @param pointer [Hash] Portion of the content structure to modify # @param key [String] String representing structure, delimited by '::' # @param value [?] Value to insert at structure point def merge_enc_param(pointer, key, value) if key =~ /::/ first_key, the_rest = key.split(/::/, 2) if pointer[first_key].nil? pointer[first_key] = {} elsif !pointer[first_key].is_a?(Hash) raise ArgumentError, "Attempt to override #{pointer[first_key].class} with hash for #{key}" end merge_enc_param(pointer[first_key], the_rest, value) elsif value.nil? pointer.delete(key) else pointer[key] = value end end # Backend - given options, choose an appropriate backend and construct the corresponding object. # @return [?] Backend object def backend # Hard-coded backend if @options[:backend] return OctocatalogDiff::CatalogUtil::ENC::Noop.new(@options) if @options[:backend] == :noop return OctocatalogDiff::CatalogUtil::ENC::PE.new(@options) if @options[:backend] == :pe return OctocatalogDiff::CatalogUtil::ENC::Script.new(@options) if @options[:backend] == :script raise ArgumentError, "Unknown backend :#{@options[:backend]}" end # Determine backend based on arguments return OctocatalogDiff::CatalogUtil::ENC::PE.new(@options) if @options[:pe_enc_url] return OctocatalogDiff::CatalogUtil::ENC::Script.new(@options) if @options[:enc] # At this point we do not know what backend to use for the ENC raise ArgumentError, 'Unable to determine ENC backend to use' end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/version.rb0000644000004100000410000000037713413744025022755 0ustar www-datawww-data# frozen_string_literal: true module OctocatalogDiff # Determine the version of octocatalog-diff class Version version_file = File.expand_path('../../.version', File.dirname(__FILE__)) VERSION = File.read(version_file).strip.freeze end end octocatalog-diff-1.5.4/lib/octocatalog-diff/facts/0000755000004100000410000000000013413744025022034 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/facts/json.rb0000644000004100000410000000124213413744025023331 0ustar www-datawww-data# frozen_string_literal: true require_relative '../facts' require 'json' module OctocatalogDiff class Facts # Deal with facts in JSON files class JSON # @param options [Hash] Options hash specifically for this fact type. # - :fact_file_string [String] => Fact data as a string # @param node [String] Node name (overrides node name from fact data) # @return [Hash] Facts def self.fact_retriever(options = {}, node = '') facts = ::JSON.parse(options.fetch(:fact_file_string)) node = facts.fetch('fqdn', 'unknown.node') if node.empty? { 'name' => node, 'values' => facts } end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/facts/yaml.rb0000644000004100000410000000277513413744025023336 0ustar www-datawww-data# frozen_string_literal: true require_relative '../facts' require 'yaml' module OctocatalogDiff class Facts # Deal with facts in YAML files class Yaml # Manipulate a YAML file so it can be parsed and return the facts as a hash. # If we leave it as Puppet::Node::Facts then it will require us to load puppet # gems in order to parse it, and that's too heavy for simple fact retrieval. # @param options [Hash] Options hash specifically for this fact type. # - :fact_file_string [String] => Fact data as a string # @param node [String] Node name (overrides node name from fact data) # @return [Hash] Facts def self.fact_retriever(options = {}, node = '') fact_file_string = options.fetch(:fact_file_string) # Touch up the first line before parsing. fact_file_data = fact_file_string.split(/\n/) fact_file_data[0] = '---' if fact_file_data[0] =~ /^---/ # Load the parsed fact file. parsed = YAML.load(fact_file_data.join("\n")) # This is a handler for a YAML file that has just the facts and none of the # structure. For example if you saved the output of `facter -y` to a file and # are passing that in, this will work. result = if parsed.key?('name') && parsed.key?('values') parsed else { 'name' => node || parsed['fqdn'] || '', 'values' => parsed } end result['name'] = node unless node == '' result end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/facts/puppetdb.rb0000644000004100000410000000600413413744025024204 0ustar www-datawww-data# frozen_string_literal: true require_relative '../errors' require_relative '../facts' require_relative '../puppetdb' require 'yaml' module OctocatalogDiff class Facts # Deal with facts in PuppetDB class PuppetDB # Supporting multiple versions of the PuppetDB API. PUPPETDB_QUERY_FACTS_URL = { '3' => '/v3/nodes//facts', '4' => '/pdb/query/v4/nodes//facts' }.freeze # Retrieve facts from PuppetDB for a specified node. # @param :puppetdb_url [String|Array] => URL to PuppetDB # @param :retry [Integer] => Retry after timeout (default 0 retries, can be more) # @param node [String] Node name. (REQUIRED for PuppetDB fact source) # @return [Hash] Facts def self.fact_retriever(options = {}, node) # Set up some variables from options raise ArgumentError, 'puppetdb_url is required' unless options[:puppetdb_url].is_a?(String) raise ArgumentError, 'node must be a non-empty string' unless node.is_a?(String) && node != '' puppetdb_api_version = options.fetch(:puppetdb_api_version, 4) uri = PUPPETDB_QUERY_FACTS_URL.fetch(puppetdb_api_version.to_s).gsub('', node) retries = options.fetch(:retry, 0).to_i # Construct puppetdb object and options opts = options.merge(timeout: 5) puppetdb = OctocatalogDiff::PuppetDB.new(opts) # Use OctocatalogDiff::PuppetDB to pull facts exception_class = nil exception_message = nil obj_to_return = nil (retries + 1).times do begin result = puppetdb.get(uri) facts = {} result.map { |x| facts[x['name']] = x['value'] } if facts.empty? message = "Unable to retrieve facts for node #{node} from PuppetDB (empty or nil)!" raise OctocatalogDiff::Errors::FactRetrievalError, message end # Create a structure compatible with YAML fact files. obj_to_return = { 'name' => node, 'values' => {} } facts.each { |k, v| obj_to_return['values'][k.sub(/^::/, '')] = v } break # Not return, to avoid LocalJumpError in Ruby 2.2 rescue OctocatalogDiff::Errors::PuppetDBConnectionError => exc exception_class = OctocatalogDiff::Errors::FactSourceError exception_message = "Fact retrieval failed (#{exc.class}) (#{exc.message})" rescue OctocatalogDiff::Errors::PuppetDBNodeNotFoundError => exc exception_class = OctocatalogDiff::Errors::FactRetrievalError exception_message = "Node #{node} not found in PuppetDB (#{exc.message})" rescue OctocatalogDiff::Errors::PuppetDBGenericError => exc exception_class = OctocatalogDiff::Errors::FactRetrievalError exception_message = "Fact retrieval failed for node #{node} from PuppetDB (#{exc.message})" end end return obj_to_return unless obj_to_return.nil? raise exception_class, exception_message end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/0000755000004100000410000000000013413744025021503 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/0000755000004100000410000000000013413744025023176 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/ignore_tags.rb0000644000004100000410000000173013413744025026025 0ustar www-datawww-data# frozen_string_literal: true # Provide ability to set one or more tags, which will cause catalog-diff # to ignore any changes for any defined type where this tag is set. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:ignore_tags) do has_weight 400 def parse(parser, options) parser.on('--no-ignore-tags', 'Disable ignoring based on tags') do if options[:ignore_tags] raise ArgumentError, '--no-ignore-tags incompatible with --ignore-tags' end options[:no_ignore_tags] = true end parser.on('--ignore-tags STRING1[,STRING2[,...]]', Array, 'Specify tags to ignore') do |x| if options[:no_ignore_tags] raise ArgumentError, '--ignore-tags incompatible with --no-ignore-tags' end options[:ignore_tags] ||= [] options[:ignore_tags].concat x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/bootstrap_script.rb0000644000004100000410000000126413413744025027127 0ustar www-datawww-data# frozen_string_literal: true # Allow specification of a bootstrap script. This runs after checking out the directory, and before running # puppet there. Good for running librarian to install modules, and anything else site-specific that needs # to be done. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:bootstrap_script) do has_weight 40 def parse(parser, options) parser.on('--bootstrap-script FILENAME', 'Bootstrap script relative to checkout directory') do |file| options[:bootstrap_script] = file end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/display_datatype_changes.rb0000644000004100000410000000137613413744025030562 0ustar www-datawww-data# frozen_string_literal: true # Toggle on or off the display of data type changes when the string representation # is the same. For example with this enabled, '42' (the string) and 42 (the integer) # will be displayed as a difference. With this disabled, this is not displayed as a # difference. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:display_datatype_changes) do has_weight 280 def parse(parser, options) desc = 'Display changes in data type even when strings match' parser.on('--[no-]display-datatype-changes', desc) do |x| options[:display_datatype_changes] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/existing_catalogs.rb0000644000004100000410000000213313413744025027231 0ustar www-datawww-data# frozen_string_literal: true require 'json' # If pre-compiled catalogs are available, these can be used to short-circuit the build process. # These files must exist and be in Puppet catalog format. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:existing_catalogs) do has_weight 30 def parse(parser, options) these_options = { 'from' => :from_catalog, 'to' => :to_catalog } these_options.each do |tag, hash_key| parser.on("--#{tag}-catalog FILENAME", "Use a pre-compiled catalog '#{tag}'") do |catalog_file| path = File.absolute_path(catalog_file) raise Errno::ENOENT, "Invalid '#{hash_key} catalog' file provided" unless File.file?(path) options[hash_key] = path if options[:node].nil? x = JSON.parse(File.read(path)) options[:node] ||= x['data']['name'] if x['data'].is_a?(Hash) options[:node] ||= x['name'] if x['name'].is_a?(String) end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/hiera_config.rb0000644000004100000410000000203513413744025026140 0ustar www-datawww-data# frozen_string_literal: true # Specify a relative path to the Hiera yaml file # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:hiera_config) do has_weight 180 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'hiera-config', option_name: 'hiera_config', desc: 'Full or relative path to global Hiera configuration file', post_process: lambda do |opts| raise ArgumentError, '--no-hiera-config incompatible with --hiera-config' if opts[:no_hiera_config] end ) parser.on('--no-hiera-config', 'Disable hiera config file installation') do if options[:to_hiera_config] || options[:from_hiera_config] raise ArgumentError, '--no-hiera-config incompatible with --hiera-config' end options[:no_hiera_config] = true end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/pe_enc_ssl_client_cert.rb0000644000004100000410000000135713413744025030216 0ustar www-datawww-data# frozen_string_literal: true # Specify the client certificate for connecting to the Puppet Enterprise ENC. This must be specified along with # --pe-enc-ssl-client-key in order to work. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:pe_enc_ssl_client_cert) do has_weight 353 def parse(parser, options) parser.on('--pe-enc-ssl-client-cert FILENAME', 'SSL client certificate to connect to PE ENC') do |x| raise Errno::ENOENT, "--pe-enc-ssl-client-cert #{x} does not point to a valid file" unless File.file?(x) options[:pe_enc_ssl_client_cert] = File.read(x) end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppet_binary.rb0000644000004100000410000000112713413744025026405 0ustar www-datawww-data# frozen_string_literal: true # Set --puppet-binary, --to-puppet-binary, --from-puppet-binary # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppet_binary) do has_weight 300 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'puppet-binary', option_name: 'puppet_binary', desc: 'Full path to puppet binary' ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/include_tags.rb0000644000004100000410000000102313413744025026160 0ustar www-datawww-data# frozen_string_literal: true # Options used when comparing catalogs - tags are generally ignored; you can un-ignore them. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:include_tags) do has_weight 140 def parse(parser, options) parser.on('--[no-]include-tags', 'Include changes to tags in the diff output') do |x| options[:include_tags] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppetdb_ssl_client_password.rb0000644000004100000410000000134213413744025031507 0ustar www-datawww-data# frozen_string_literal: true # Specify the password for a PEM or PKCS12 private key on the command line. # Note that `--puppetdb-ssl-client-password-file` is slightly more secure because # the text of the password won't appear in the process list. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppetdb_ssl_client_cert) do has_weight 310 order_within_weight 35 def parse(parser, options) parser.on('--puppetdb-ssl-client-password PASSWORD', 'Password for SSL client key to connect to PuppetDB') do |x| options[:puppetdb_ssl_client_password] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/truncate_details.rb0000644000004100000410000000126113413744025027055 0ustar www-datawww-data# frozen_string_literal: true # When using `--display-detail-add` by default the details of any field will be truncated # at 80 characters. Specify `--no-truncate-details` to display the full output. This option # has no effect when `--display-detail-add` is not used. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:truncate_details) do has_weight 251 def parse(parser, options) parser.on('--[no-]truncate-details', 'Truncate details with --display-detail-add') do |x| options[:truncate_details] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/to_from_branch.rb0000644000004100000410000000131413413744025026504 0ustar www-datawww-data# frozen_string_literal: true # Set the 'from' and 'to' branches, which is used to compile catalogs. A branch of '.' means to use # the current contents of the base code directory without any git checkouts. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:to_from_branch) do has_weight 20 def parse(parser, options) parser.on('--from FROM_BRANCH', '-f', 'Branch you are coming from') do |env| options[:from_env] = env end parser.on('--to TO_BRANCH', '-t', 'Branch you are going to') do |env| options[:to_env] = env end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/pe_enc_token.rb0000644000004100000410000000140613413744025026155 0ustar www-datawww-data# frozen_string_literal: true # Specify the access token to access the Puppet Enterprise ENC. Refer to # https://docs.puppet.com/pe/latest/nc_forming_requests.html#authentication for # details on generating and obtaining a token. Use this option to specify the text # of the token. (Use --pe-enc-token-file to read the content of the token from a file.) # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:pe_enc_token) do has_weight 351 def parse(parser, options) parser.on('--pe-enc-token TOKEN', 'Token to access the Puppet Enterprise ENC API') do |token| options[:pe_enc_token] = token end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/pe_enc_ssl_ca.rb0000644000004100000410000000144313413744025026302 0ustar www-datawww-data# frozen_string_literal: true # Specify the CA certificate for the Puppet Enterprise ENC. If specified, this will enable SSL verification # that the certificate being presented has been signed by this CA, and that the common name # matches the name you are using to connecting. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:pe_enc_ssl_ca) do has_weight 352 def parse(parser, options) parser.on('--pe-enc-ssl-ca FILENAME', 'CA certificate that signed the ENC API certificate') do |x| raise Errno::ENOENT, "--pe-enc-ssl-ca #{x} does not point to a valid file" unless File.file?(x) options[:pe_enc_ssl_ca] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/pe_enc_token_file.rb0000644000004100000410000000177013413744025027160 0ustar www-datawww-data# frozen_string_literal: true # Specify the access token to access the Puppet Enterprise ENC. Refer to # https://docs.puppet.com/pe/latest/nc_forming_requests.html#authentication for # details on generating and obtaining a token. Use this option if the token is stored # in a file, to read the content of the token from the file. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:pe_enc_token_file) do has_weight 351 def parse(parser, options) parser.on('--pe-enc-token-file PATH', 'Path containing token for PE node classifier, relative or absolute') do |x| proposed_token_path = x.start_with?('/') ? x : File.join(options[:basedir], x) raise Errno::ENOENT, "Provided PE ENC token (#{proposed_token_path}) does not exist" unless File.file?(proposed_token_path) options[:pe_enc_token] = File.read(proposed_token_path) end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppet_master_timeout.rb0000644000004100000410000000152513413744025030164 0ustar www-datawww-data# frozen_string_literal: true # Specify a timeout for retrieving a catalog from a Puppet master / Puppet server. # This timeout is specified in seconds. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppet_master_timeout) do has_weight 329 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'puppet-master-timeout', option_name: 'puppet_master_timeout', desc: 'Puppet Master catalog retrieval timeout in seconds', validator: ->(x) { x.to_i > 0 || raise(ArgumentError, 'Specify timeout as an integer greater than 0') }, translator: ->(x) { x.to_i } ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/hostname.rb0000644000004100000410000000165013413744025025343 0ustar www-datawww-data# frozen_string_literal: true # Set hostname, which is used to look up facts in PuppetDB, and in the header of diff display. # This option can recieve a single hostname, or a comma separated list of # multiple hostnames, which are split into an Array. Multiple hostnames do not # work with the `catalog-only` or `bootstrap-then-exit` options. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:hostname) do has_weight 1 def parse(parser, options) parser.on( '--hostname HOSTNAME1[,HOSTNAME2[,...]]', '-n', 'Use PuppetDB facts from last run of a hostname or a comma separated list of multiple hostnames' ) do |hostname| options[:node] = if hostname.include?(',') hostname.split(',') else hostname end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppetdb_token_file.rb0000644000004100000410000000204213413744025027543 0ustar www-datawww-data# frozen_string_literal: true # Specify the PE RBAC token to access the PuppetDB API. Refer to # https://puppet.com/docs/pe/latest/rbac/rbac_token_auth_intro.html#generate-a-token-using-puppet-access # for details on generating and obtaining a token. Use this option to specify the text # in a file, to read the content of the token from the file. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppetdb_token_file) do has_weight 310 def parse(parser, options) parser.on('--puppetdb-token-file PATH', 'Path containing token for PuppetDB API, relative or absolute') do |x| proposed_token_path = x.start_with?('/') ? x : File.join(options[:basedir], x) unless File.file?(proposed_token_path) raise Errno::ENOENT, "Provided PuppetDB API token (#{proposed_token_path}) does not exist" end options[:puppetdb_token] = File.read(proposed_token_path) end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/facts_terminus.rb0000644000004100000410000000146713413744025026561 0ustar www-datawww-data# frozen_string_literal: true # Get the facts terminus. Generally this is 'yaml' and a fact file will be loaded from PuppetDB or # elsewhere in the environment. However it can be set to 'facter' which will run facter on the host # on which this is running. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:facts_terminus) do has_weight 310 def parse(parser, options) termini = %w(yaml facter) parser.on('--facts-terminus STRING', "Facts terminus: one of #{termini.join(', ')}") do |x| raise ArgumentError, "Invalid facts terminus #{x}; supported: #{termini.join(', ')}" unless termini.include?(x) options[:facts_terminus] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppetdb_api_version.rb0000644000004100000410000000123613413744025027746 0ustar www-datawww-data# Specify the API version to use for the PuppetDB. The current values supported are '3' or '4', and '4' is # the default. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppetdb_api_version) do has_weight 319 def parse(parser, options) parser.on('--puppetdb-api-version N', OptionParser::DecimalInteger, 'Version of PuppetDB API (3 or 4)') do |x| options[:puppetdb_api_version] = x raise ArgumentError, 'Only PuppetDB versions 3 and 4 are supported' unless [3, 4].include?(x) end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/quiet.rb0000644000004100000410000000066413413744025024660 0ustar www-datawww-data# frozen_string_literal: true # Quiet option # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:quiet) do has_weight 120 def parse(parser, options) parser.on('--[no-]quiet', '-q', 'Quiet (no status messages except errors)') do |x| options[:quiet] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppetdb_ssl_client_key.rb0000644000004100000410000000136013413744025030435 0ustar www-datawww-data# frozen_string_literal: true # Specify the client key for connecting to PuppetDB. This must be specified along with # --puppetdb-ssl-client-cert in order to work. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppetdb_ssl_client_key) do has_weight 310 order_within_weight 30 def parse(parser, options) parser.on('--puppetdb-ssl-client-key FILENAME', 'SSL client key to connect to PuppetDB') do |x| raise Errno::ENOENT, "--puppetdb-ssl-client-key #{x} does not point to a valid file" unless File.file?(x) options[:puppetdb_ssl_client_key] = File.read(x) end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/debug_bootstrap.rb0000644000004100000410000000116213413744025026706 0ustar www-datawww-data# frozen_string_literal: true # Option to print debugging output for the bootstrap script in addition to the normal # debugging output. Note that `--debug` must also be enabled for this option to have # any effect. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:debug_bootstrap) do has_weight 49 def parse(parser, options) parser.on('--debug-bootstrap', 'Print debugging output for bootstrap script') do options[:debug_bootstrap] = true end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/storeconfigs.rb0000644000004100000410000000101313413744025026223 0ustar www-datawww-data# frozen_string_literal: true # Set storeconfigs (integration with PuppetDB for collected resources) # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:storeconfigs) do has_weight 220 def parse(parser, options) parser.on('--[no-]storeconfigs', 'Enable integration with puppetdb for collected resources') do |x| options[:storeconfigs] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/filters.rb0000644000004100000410000000150213413744025025171 0ustar www-datawww-data# frozen_string_literal: true # Specify one or more filters to apply to the results of the catalog difference. # For a list of available filters and further explanation, please refer to # Filtering results. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:filters) do has_weight 199 def parse(parser, options) parser.on('--filters FILTER1[,FILTER2[,...]]', Array, 'Filters to apply') do |x| options[:filters] ||= [] options[:filters].concat x require_relative '../../catalog-diff/filter' options[:filters].each { |filter| OctocatalogDiff::CatalogDiff::Filter.assert_that_filter_exists(filter) } end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/override_script_path.rb0000644000004100000410000000150113413744025027737 0ustar www-datawww-data# frozen_string_literal: true # Provide an optional directory to override default built-in scripts such as git checkout # and puppet version determination. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:override_script_path) do has_weight 385 def parse(parser, options) parser.on('--override-script-path DIRNAME', 'Directory with scripts to override built-ins') do |dir| unless dir.start_with?('/') raise ArgumentError, 'Absolute path is required for --override-script-path' end unless File.directory?(dir) raise Errno::ENOENT, 'Invalid --override-script-path' end options[:override_script_path] = dir end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/fact_file.rb0000644000004100000410000000314113413744025025436 0ustar www-datawww-data# frozen_string_literal: true # Allow an existing fact file to be provided, to avoid pulling facts from PuppetDB. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:fact_file) do has_weight 150 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'fact-file', option_name: 'facts', desc: 'Override fact', datatype: '', validator: ->(fact_file) { File.file?(fact_file) && (fact_file =~ /\.ya?ml$/ || fact_file =~ /\.json$/) }, translator: lambda do |fact_file| local_opts = { fact_file_string: File.read(fact_file) } if fact_file =~ /\.ya?ml$/ OctocatalogDiff::Facts.new(local_opts.merge(backend: :yaml)) elsif fact_file =~ /\.json$/ OctocatalogDiff::Facts.new(local_opts.merge(backend: :json)) else # :nocov: # Believed to be a bug condition since the validator should kick this out before it ever gets here. raise ArgumentError, 'I do not know how to parse the provided fact file. Needs .yaml or .json extension.' # :nocov: end end, post_process: lambda do |opts| unless options[:node] %w[to_facts from_facts facts].each do |opt| next unless opts[opt.to_sym] && opts[opt.to_sym].node opts[:node] = opts[opt.to_sym].node break end end end ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppetdb_token.rb0000644000004100000410000000142613413744025026551 0ustar www-datawww-data# frozen_string_literal: true # Specify the PE RBAC token to access the PuppetDB API. Refer to # https://puppet.com/docs/pe/latest/rbac/rbac_token_auth_intro.html#generate-a-token-using-puppet-access # for details on generating and obtaining a token. Use this option to specify the text # of the token. (Use --puppetdb-token-file to read the content of the token from a file.) # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppetdb_token) do has_weight 310 def parse(parser, options) parser.on('--puppetdb-token TOKEN', 'Token to access the PuppetDB API') do |token| options[:puppetdb_token] = token end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/header.rb0000644000004100000410000000243413413744025024756 0ustar www-datawww-data# frozen_string_literal: true # Provide ability to set custom header or to display no header at all # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:header) do has_weight 260 def parse(parser, options) parser.on('--no-header', 'Do not print a header') do raise ArgumentError, '--no-header incompatible with --default-header' if options[:header] == :default raise ArgumentError, '--no-header incompatible with --header' unless options[:header].nil? options[:no_header] = true end parser.on('--default-header', 'Print default header with output') do raise ArgumentError, '--default-header incompatible with --header' unless options[:header].nil? raise ArgumentError, '--default-header incompatible with --no-header' unless options[:no_header].nil? options[:header] = :default end parser.on('--header STRING', 'Specify header for output') do |x| raise ArgumentError, '--header incompatible with --default-header' if options[:header] == :default raise ArgumentError, '--header incompatible with --no-header' unless options[:no_header].nil? options[:header] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/display_detail_add.rb0000644000004100000410000000103713413744025027323 0ustar www-datawww-data# frozen_string_literal: true # Provide ability to display details of 'added' resources in the output. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:display_detail_add) do has_weight 250 def parse(parser, options) parser.on('--[no-]display-detail-add', 'Display parameters and other details for added resources') do |x| options[:display_detail_add] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/hiera_path_strip.rb0000644000004100000410000000346013413744025027053 0ustar www-datawww-data# frozen_string_literal: true # Specify the path to strip off the datadir to munge hiera.yaml file # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:hiera_path_strip) do has_weight 182 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'hiera-path-strip', option_name: 'hiera_path_strip', desc: 'Path prefix to strip when munging hiera.yaml', post_process: lambda do |opts| if opts.key?(:to_hiera_path) && opts[:to_hiera_path] != :none if opts.key?(:to_hiera_path_strip) && opts[:to_hiera_path_strip] != :none raise ArgumentError, '--hiera-path and --hiera-path-strip are mutually exclusive' end end if opts.key?(:from_hiera_path) && opts[:from_hiera_path] != :none if opts.key?(:from_hiera_path_strip) && opts[:from_hiera_path_strip] != :none raise ArgumentError, '--hiera-path and --hiera-path-strip are mutually exclusive' end end if opts[:to_hiera_path_strip] == :none || opts[:from_hiera_path_strip] == :none raise ArgumentError, '--hiera-path-strip and --no-hiera-path-strip are mutually exclusive' end end ) parser.on('--no-hiera-path-strip', 'Do not use any default hiera path strip settings') do if options[:to_hiera_path_strip].is_a?(String) || options[:from_hiera_path_strip].is_a?(String) raise ArgumentError, '--hiera-path-strip and --no-hiera-path-strip are mutually exclusive' end options[:to_hiera_path_strip] = :none options[:from_hiera_path_strip] = :none end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/create_symlinks.rb0000644000004100000410000000133013413744025026714 0ustar www-datawww-data# frozen_string_literal: true # Specify which directories from the base should be symlinked into the temporary compilation # environment. This is useful only in conjunction with `--preserve-environments`. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:create_symlinks) do has_weight 503 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'create-symlinks', option_name: 'create_symlinks', desc: 'Symlinks to create', datatype: [] ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/bootstrap_then_exit.rb0000644000004100000410000000077013413744025027613 0ustar www-datawww-data# frozen_string_literal: true # Option to bootstrap directories and then exit # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:bootstrap_then_exit) do has_weight 70 def parse(parser, options) parser.on('--bootstrap-then-exit', 'Bootstrap from-dir and/or to-dir and then exit') do options[:bootstrap_then_exit] = true end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/color.rb0000644000004100000410000000075313413744025024646 0ustar www-datawww-data# frozen_string_literal: true # Color printing option # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:color) do has_weight 80 def parse(parser, options) parser.on('--[no-]color', 'Enable/disable colors in output') do |color| options[:colors] = color options[:format] = color ? :color_text : :text end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/compare_file_text.rb0000644000004100000410000000136613413744025027222 0ustar www-datawww-data# frozen_string_literal: true # When a file is specified with `source => 'puppet:///modules/something/foo.txt'`, remove # the 'source' attribute and populate the 'content' attribute with the text of the file. # This allows for a diff of the content, rather than a diff of the location, which is # what is most often desired. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:compare_file_text) do has_weight 210 def parse(parser, options) parser.on('--[no-]compare-file-text', 'Compare text, not source location, of file resources') do |x| options[:compare_file_text] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/from_puppetdb.rb0000644000004100000410000000105213413744025026367 0ustar www-datawww-data# frozen_string_literal: true # Set --from-puppetdb to pull most recent catalog from PuppetDB instead of compiling # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:from_puppetdb) do has_weight 300 def parse(parser, options) desc = 'Pull "from" catalog from PuppetDB instead of compiling' parser.on('--[no-]from-puppetdb', desc) do |x| options[:from_puppetdb] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/bootstrapped_dirs.rb0000644000004100000410000000170713413744025027257 0ustar www-datawww-data# frozen_string_literal: true # Allow (or create) directories that are already bootstrapped. Handy to allow "bootstrap once, build many" # to save time when diffing multiple catalogs on this system. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:bootstrapped_dirs) do has_weight 60 def parse(parser, options) these_options = { 'from' => :bootstrapped_from_dir, 'to' => :bootstrapped_to_dir } these_options.each do |tag, hash_key| parser.on("--bootstrapped-#{tag}-dir DIRNAME", "Use a pre-bootstrapped '#{tag}' directory") do |dir| options[hash_key] = File.absolute_path(dir) Dir.mkdir options[hash_key], 0o700 unless Dir.exist?(options[hash_key]) raise "Invalid bootstrapped-#{tag}-dir: does not exist" unless Dir.exist?(options[hash_key]) end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppet_master_ssl_client_cert.rb0000644000004100000410000000167213413744025031655 0ustar www-datawww-data# frozen_string_literal: true # Specify the SSL client certificate for Puppet Master. This makes it possible to authenticate with a # client certificate keypair to the Puppet Master. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppet_master_ssl_client_cert) do has_weight 320 order_within_weight 40 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'puppet-master-ssl-client-cert', option_name: 'puppet_master_ssl_client_cert', desc: 'Full path to certificate file for SSL client auth to Puppet Master', validator: ->(x) { File.file?(x) || raise(Errno::ENOENT, "Suggested certificate #{x} does not exist") }, translator: ->(x) { File.read(x) } ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/ignore_attr.rb0000644000004100000410000000112513413744025026037 0ustar www-datawww-data# frozen_string_literal: true # Specify attributes to ignore # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:ignore_attr) do has_weight 190 def parse(parser, options) parser.on('--ignore-attr "attr1,attr2,..."', Array, 'Attributes to ignore') do |res| options[:ignore] ||= [] res.each do |item| item_subst = item.gsub(/(\\f|::)/, "\f") options[:ignore] << { attr: item_subst } end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/debug.rb0000644000004100000410000000066213413744025024615 0ustar www-datawww-data# frozen_string_literal: true # Debugging option # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:debug) do has_weight 110 def parse(parser, options) parser.on('--[no-]debug', '-d', 'Print debugging messages to STDERR') do |x| options[:debug] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppetdb_url.rb0000644000004100000410000000150513413744025026231 0ustar www-datawww-data# frozen_string_literal: true require 'uri' # Specify the base URL for PuppetDB. This will generally look like https://puppetdb.yourdomain.com:8081 # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppetdb_url) do has_weight 310 order_within_weight 1 def parse(parser, options) parser.on('--puppetdb-url URL', 'PuppetDB base URL') do |url| # Test the format of the incoming URL. Only HTTPS should really be used, but we will # support HTTP begrudgingly as well. obj = URI.parse(url) raise ArgumentError, 'PuppetDB URL must be http or https' unless obj.is_a?(URI::HTTPS) || obj.is_a?(URI::HTTP) options[:puppetdb_url] = url end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/cached_master_dir.rb0000644000004100000410000000146713413744025027153 0ustar www-datawww-data# frozen_string_literal: true # Cache a bootstrapped checkout of 'master' and use that for time-saving when the SHA # has not changed. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:cached_master_dir) do has_weight 160 def parse(parser, options) parser.on('--cached-master-dir PATH', 'Cache bootstrapped origin/master at this path') do |path_in| path = File.absolute_path(path_in) unless Dir.exist?(path) begin Dir.mkdir path, 0o755 rescue Errno::ENOENT => exc raise Errno::ENOENT, "Invalid cached master directory path: #{exc}" end end options[:cached_master_dir] = path end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/command_line.rb0000644000004100000410000000117413413744025026153 0ustar www-datawww-data# frozen_string_literal: true # Provide additional command line flags to set when running Puppet to compile catalogs. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:command_line) do has_weight 510 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'command-line', option_name: 'command_line', desc: 'Command line arguments', datatype: [] ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/parallel.rb0000644000004100000410000000073213413744025025321 0ustar www-datawww-data# frozen_string_literal: true # Disable or enable parallel processing of catalogs. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:parallel) do has_weight 300 def parse(parser, options) parser.on('--[no-]parallel', 'Enable or disable parallel processing') do |x| options[:parallel] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/ignore.rb0000644000004100000410000000206213413744025025006 0ustar www-datawww-data# frozen_string_literal: true # Options used when comparing catalogs - set ignored changes. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:ignore) do has_weight 130 def parse(parser, options) descriptive_text = 'More resources to ignore in format type[title]' parser.on('--ignore "Type1[Title1],Type2[Title2],..."', Array, descriptive_text) do |res| options[:ignore] ||= [] res.each do |item| if item =~ /\A(.+?)\[(.+)\](.+)/ h = { type: Regexp.last_match(1), title: Regexp.last_match(2) } h[:attr] = Regexp.last_match(3).gsub(/(\\f|::)/, "\f") options[:ignore] << h elsif item =~ /^(.+?)\[(.+)\]$/ options[:ignore] << { type: Regexp.last_match(1), title: Regexp.last_match(2) } else raise ArgumentError, "Ignore #{item} must be in Type[Title] or Type[Title]::Attribute format!" end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/bootstrap_environment.rb0000644000004100000410000000161413413744025030166 0ustar www-datawww-data# frozen_string_literal: true # Allow the bootstrap environment to be set up via the command line. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:bootstrap_environment) do has_weight 50 def parse(parser, options) descriptive_text = 'Bootstrap script environment variables in key=value format' parser.on('--bootstrap-environment "key1=val1,key2=val2,..."', Array, descriptive_text) do |res| options[:bootstrap_environment] ||= {} res.each do |item| raise ArgumentError, "Bootstrap environment #{item} must be in key=value format!" unless item =~ /=/ key, val = item.split(/=/, 2) options[:bootstrap_environment][key] = Regexp.last_match(1) if val.strip =~ /^['"]?(.+?)['"]?$/ end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/preserve_environments.rb0000644000004100000410000000125113413744025030164 0ustar www-datawww-data# frozen_string_literal: true # Preserve the `environments` directory from the repository when compiling the catalog. Likely # requires some combination of `--to-environment`, `--from-environment`, and/or `--create-symlinks` # to work correctly. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:preserve_environments) do has_weight 501 def parse(parser, options) parser.on('--[no-]preserve-environments', 'Enable or disable environment preservation') do |x| options[:preserve_environments] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/safe_to_delete_cached_master_dir.rb0000644000004100000410000000136513413744025032172 0ustar www-datawww-data# frozen_string_literal: true # By specifying a directory path here, you are explicitly giving permission to the program # to delete it if it believes it needs to be created (e.g., if the SHA has changed of the # cached directory). # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:safe_to_delete_cached_master_dir) do has_weight 160 def parse(parser, options) parser.on('--safe-to-delete-cached-master-dir PATH', 'OK to delete cached master directory at this path') do |path_in| path = File.absolute_path(path_in) options[:safe_to_delete_cached_master_dir] = path end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppetdb_ssl_ca.rb0000644000004100000410000000146413413744025026677 0ustar www-datawww-data# frozen_string_literal: true # Specify the CA certificate for PuppetDB. If specified, this will enable SSL verification # that the certificate being presented has been signed by this CA, and that the common name # matches the name you are using to connecting. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppetdb_ssl_ca) do has_weight 310 order_within_weight 10 def parse(parser, options) parser.on('--puppetdb-ssl-ca FILENAME', 'CA certificate that signed the PuppetDB certificate') do |x| raise Errno::ENOENT, "--puppetdb-ssl-ca #{x} does not point to a valid file" unless File.file?(x) options[:puppetdb_ssl_ca] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/output_file.rb0000644000004100000410000000106513413744025026064 0ustar www-datawww-data# frozen_string_literal: true # Output file option # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:output_file) do has_weight 90 def parse(parser, options) parser.on('--output-file FILENAME', '-o', 'Output results into FILENAME') do |filename| path = File.absolute_path(filename) options[:output_file] = path options[:format] = :text options[:colors] = false end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/bootstrap_current.rb0000644000004100000410000000112413413744025027300 0ustar www-datawww-data# frozen_string_literal: true # Option to bootstrap the current directory (by default, the bootstrap script is NOT # run when the catalog builds in the current directory). # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:bootstrap_current) do has_weight 48 def parse(parser, options) parser.on('--bootstrap-current', 'Run bootstrap script for the current directory too') do options[:bootstrap_current] = true end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/environment.rb0000644000004100000410000000123413413744025026067 0ustar www-datawww-data# frozen_string_literal: true # Specify the environment to use when compiling the catalog. This is useful only in conjunction # with `--preserve-environments`. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:environment) do has_weight 502 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'environment', option_name: 'environment', desc: 'Environment for catalog compilation' ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppet_master.rb0000644000004100000410000000115713413744025026417 0ustar www-datawww-data# frozen_string_literal: true # Specify the hostname, or hostname:port, for the Puppet Master. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppet_master) do has_weight 320 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'puppet-master', option_name: 'puppet_master', desc: 'Hostname or Hostname:PortNumber for Puppet Master' ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/pe_enc_url.rb0000644000004100000410000000153413413744025025641 0ustar www-datawww-data# frozen_string_literal: true require 'uri' # Specify the URL to the Puppet Enterprise ENC API. By default, the node classifier service # listens on port 4433 and all endpoints are relative to the /classifier-api/ path. That means # the likely value for this option will be something like: # https://your-pe-console-server:4433/classifier-api # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:pe_enc_url) do has_weight 350 def parse(parser, options) parser.on('--pe-enc-url URL', 'Base URL for Puppet Enterprise ENC endpoint') do |url| obj = URI.parse(url) raise ArgumentError, 'PE ENC URL must be https' unless obj.is_a?(URI::HTTPS) options[:pe_enc_url] = url end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/validate_references.rb0000644000004100000410000000177713413744025027531 0ustar www-datawww-data# frozen_string_literal: true # Confirm that each `before`, `require`, `subscribe`, and/or `notify` points to a valid # resource in the catalog. This value should be specified as an array of which of these # parameters are to be checked. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:validate_references) do has_weight 205 def parse(parser, options) parser.on('--[no-]validate-references "before,require,subscribe,notify"', Array, 'References to validate') do |res| if res == false options[:validate_references] = [] else options[:validate_references] ||= [] res.each do |item| unless %w(before require subscribe notify).include?(item) raise ArgumentError, "Invalid reference validation #{item}" end options[:validate_references] << item end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/parser.rb0000644000004100000410000000374513413744025025030 0ustar www-datawww-data# frozen_string_literal: true # Enable future parser for both branches or for just one # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:parser) do has_weight 270 def parse(parser, options) supported_parsers = %w(default future) parser_str = supported_parsers.join(', ') # --parser sets both parser-to and parser-from parser.on('--parser PARSER_NAME', "Specify parser (#{parser_str})") do |x| unless supported_parsers.include?(x) raise ArgumentError, "--parser must be one of: #{parser_str}" end unless options[:parser_from].nil? || options[:parser_from] == x.to_sym raise ArgumentError, '--parser conflicts with --parser-from' end unless options[:parser_to].nil? || options[:parser_to] == x.to_sym raise ArgumentError, '--parser conflicts with --parser-to' end options[:parser_from] = x.to_sym options[:parser_to] = x.to_sym end # --parser-from sets parser for the 'from' branch parser.on('--parser-from PARSER_NAME', "Specify parser (#{parser_str})") do |x| unless supported_parsers.include?(x) raise ArgumentError, "--parser-from must be one of: #{parser_str}" end unless options[:parser_from].nil? || options[:parser_from] == x.to_sym raise ArgumentError, '--parser incompatible with --parser-from' end options[:parser_from] = x.to_sym end # --parser-to sets parser for the 'to' branch parser.on('--parser-to PARSER_NAME', "Specify parser (#{parser_str})") do |x| unless supported_parsers.include?(x) raise ArgumentError, "--parser-to must be one of: #{parser_str}" end unless options[:parser_to].nil? || options[:parser_to] == x.to_sym raise ArgumentError, '--parser incompatible with --parser-to' end options[:parser_to] = x.to_sym end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppet_master_api_version.rb0000644000004100000410000000172713413744025031020 0ustar www-datawww-data# frozen_string_literal: true # Specify the API version to use for the Puppet Master. This makes it possible to authenticate to a # version 3.x PuppetMaster by specifying the API version as 2, or for a version 4.x PuppetMaster by # specifying API version as 3. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppet_master_api_version) do has_weight 320 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'puppet-master-api-version', option_name: 'puppet_master_api_version', desc: 'Puppet Master API version (2 for Puppet 3.x, 3 for Puppet 4.x)', validator: ->(x) { x =~ /^[23]$/ || raise(ArgumentError, 'Only API versions 2 and 3 are supported') }, translator: ->(x) { x.to_i } ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppet_master_ssl_client_key.rb0000644000004100000410000000163713413744025031511 0ustar www-datawww-data# frozen_string_literal: true # Specify the SSL client key for Puppet Master. This makes it possible to authenticate with a # client certificate keypair to the Puppet Master. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppet_master_ssl_client_key) do has_weight 320 order_within_weight 50 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'puppet-master-ssl-client-key', option_name: 'puppet_master_ssl_client_key', desc: 'Full path to key file for SSL client auth to Puppet Master', validator: ->(x) { File.file?(x) || raise(Errno::ENOENT, "Suggested key #{x} does not exist") }, translator: ->(x) { File.read(x) } ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/retry_failed_catalog.rb0000644000004100000410000000120513413744025027664 0ustar www-datawww-data# frozen_string_literal: true # Transient errors can cause catalog compilation problems. This adds an option to retry # a failed catalog multiple times before kicking out an error message. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:retry_failed_catalog) do has_weight 230 def parse(parser, options) parser.on('--retry-failed-catalog N', OptionParser::DecimalInteger, 'Retry building a failed catalog N times') do |x| options[:retry_failed_catalog] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppetdb_ssl_client_cert.rb0000644000004100000410000000140313413744025030600 0ustar www-datawww-data# frozen_string_literal: true # Specify the client certificate for connecting to PuppetDB. This must be specified along with # --puppetdb-ssl-client-key in order to work. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppetdb_ssl_client_cert) do has_weight 310 order_within_weight 20 def parse(parser, options) parser.on('--puppetdb-ssl-client-cert FILENAME', 'SSL client certificate to connect to PuppetDB') do |x| raise Errno::ENOENT, "--puppetdb-ssl-client-cert #{x} does not point to a valid file" unless File.file?(x) options[:puppetdb_ssl_client_cert] = File.read(x) end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/suppress_absent_file_details.rb0000644000004100000410000000150713413744025031452 0ustar www-datawww-data# frozen_string_literal: true # If enabled, this option will suppress changes to certain attributes of a file, if the # file is specified to be 'absent' in the target catalog. Suppressed changes in this case # include user, group, mode, and content, because a removed file has none of those. # This option is DEPRECATED; please use --filters AbsentFile instead. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:suppress_absent_file_details) do has_weight 600 def parse(parser, options) parser.on('--[no-]suppress-absent-file-details', 'Suppress certain attributes of absent files') do |x| options[:suppress_absent_file_details] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/output_format.rb0000644000004100000410000000167413413744025026443 0ustar www-datawww-data# frozen_string_literal: true # Output format option. 'text' is human readable text, 'json' is an array of differences # identified by human readable keys (the preferred octocatalog-diff 1.x format), and 'legacy_json' is an # array of differences, where each difference is an array (the octocatalog-diff 0.x format). # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:output_format) do has_weight 100 def parse(parser, options) valid = %w(text json legacy_json) parser.on('--output-format FORMAT', "Output format: #{valid.join(',')}") do |fmt| raise ArgumentError, "Invalid format. Must be one of: #{valid.join(',')}" unless valid.include?(fmt) options[:format] = fmt.to_sym options[:format] = :color_text if options[:format] == :text && options[:colors] end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/pass_env_vars.rb0000644000004100000410000000171413413744025026377 0ustar www-datawww-data# frozen_string_literal: true # One or more environment variables that should be made available to the Puppet binary when parsing # the catalog. For example, --pass-env-vars FOO,BAR will make the FOO and BAR environment variables # available. Setting these variables is your responsibility outside of octocatalog-diff. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:pass_env_vars) do has_weight 600 def parse(parser, options) descriptive_text = 'Environment variables to pass' parser.on('--pass-env-vars VAR1[,VAR2[,...]]', Array, descriptive_text) do |res| options[:pass_env_vars] ||= [] res.each do |item| raise ArgumentError, "Environment variable #{item} must be in alphanumeric format!" unless item =~ /^\w+$/ options[:pass_env_vars] << item end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/basedir.rb0000644000004100000410000000117313413744025025136 0ustar www-datawww-data# frozen_string_literal: true # Option to set the base checkout directory of puppet repository # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:basedir) do has_weight 10 def parse(parser, options) parser.on('--basedir DIRNAME', 'Use an alternate base directory (git checkout of puppet repository)') do |dir| path = File.absolute_path(dir) raise Errno::ENOENT, 'Invalid basedir provided' unless Dir.exist?(path) options[:basedir] = path end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/enc.rb0000644000004100000410000000233413413744025024272 0ustar www-datawww-data# frozen_string_literal: true # Path to external node classifier, relative to the base directory of the checkout. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:enc) do has_weight 240 def parse(parser, options) parser.on('--no-enc', 'Disable ENC') do options[:no_enc] = true options[:enc] = nil options[:from_enc] = nil options[:to_enc] = nil end parser.on('--enc PATH', 'Path to ENC script, relative to checkout directory or absolute') do |x| unless options[:no_enc] proposed_enc_path = x.start_with?('/') ? x : File.join(options[:basedir], x) raise Errno::ENOENT, "Provided ENC (#{proposed_enc_path}) does not exist" unless File.file?(proposed_enc_path) options[:enc] = proposed_enc_path end end parser.on('--from-enc PATH', 'Path to ENC script (for the from catalog only)') do |x| options[:from_enc] = x unless options[:no_enc] end parser.on('--to-enc PATH', 'Path to ENC script (for the to catalog only)') do |x| options[:to_enc] = x unless options[:no_enc] end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/save_catalog.rb0000644000004100000410000000245313413744025026157 0ustar www-datawww-data# frozen_string_literal: true # Allow catalogs to be saved to a file before they are diff'd. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:save_catalog) do has_weight 155 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'save-catalog', option_name: 'save_catalog', desc: 'Save intermediate catalogs into files', datatype: '', validator: lambda do |catalog_file| target_dir = File.dirname(catalog_file) unless File.directory?(target_dir) raise Errno::ENOENT, "Cannot save catalog to #{catalog_file} because parent directory does not exist" end if File.exist?(catalog_file) && !File.file?(catalog_file) raise ArgumentError, "Cannot overwrite #{catalog_file} which is not a file" end true end, post_process: lambda do |opts| if opts[:to_save_catalog] && opts[:to_save_catalog] == opts[:from_save_catalog] raise ArgumentError, 'Cannot use the same file for --to-save-catalog and --from-save-catalog' end end ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/fact_override.rb0000644000004100000410000000150213413744025026335 0ustar www-datawww-data# frozen_string_literal: true # Allow override of facts on the command line. Fact overrides can be supplied for the 'to' or 'from' catalog, # or for both. There is some attempt to handle data types here (since all items on the command line are strings) # by permitting a data type specification as well. OctocatalogDiff::Cli::Options::Option.newoption(:fact_override) do has_weight 320 def parse(parser, options) # Set 'fact_override_in' because more processing is needed, once the command line options # have been parsed, to make this into the final form 'fact_override'. OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'fact-override', option_name: 'fact_override_in', desc: 'Override fact', datatype: [] ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/pe_enc_ssl_client_key.rb0000644000004100000410000000133013413744025030040 0ustar www-datawww-data# frozen_string_literal: true # Specify the client key for connecting to Puppet Enterprise ENC. This must be specified along with # --pe-enc-ssl-client-cert in order to work. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:pe_enc_ssl_client_key) do has_weight 354 def parse(parser, options) parser.on('--pe-enc-ssl-client-key FILENAME', 'SSL client key to connect to PE ENC') do |x| raise Errno::ENOENT, "--pe-enc-ssl-client-key #{x} does not point to a valid file" unless File.file?(x) options[:pe_enc_ssl_client_key] = File.read(x) end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/catalog_only.rb0000644000004100000410000000121413413744025026174 0ustar www-datawww-data# frozen_string_literal: true # When set, --catalog-only will only compile the catalog for the 'to' branch, and skip any # diffing activity. The catalog will be printed to STDOUT or written to the output file. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:catalog_only) do has_weight 290 def parse(parser, options) desc = 'Only compile the catalog for the "to" branch but do not diff' parser.on('--[no-]catalog-only', desc) do |x| options[:catalog_only] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/display_source_file_line.rb0000644000004100000410000000100713413744025030554 0ustar www-datawww-data# frozen_string_literal: true # Display source filename and line number for diffs # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:display_source_file_line) do has_weight 200 def parse(parser, options) parser.on('--[no-]display-source', 'Show source file and line for each difference') do |x| options[:display_source_file_line] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/enc_override.rb0000644000004100000410000000163113413744025026170 0ustar www-datawww-data# frozen_string_literal: true # Allow override of ENC parameters on the command line. ENC parameter overrides can be supplied for the 'to' or 'from' catalog, # or for both. There is some attempt to handle data types here (since all items on the command line are strings) # by permitting a data type specification as well. For parameters nested in hashes, use `::` as the delimiter. OctocatalogDiff::Cli::Options::Option.newoption(:enc_override) do has_weight 322 def parse(parser, options) # Set 'enc_override_in' because more processing is needed, once the command line options # have been parsed, to make this into the final form 'enc_override'. OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'enc-override', option_name: 'enc_override_in', desc: 'Override parameter from ENC', datatype: [] ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppetdb_ssl_client_password_file.rb0000644000004100000410000000134713413744025032513 0ustar www-datawww-data# frozen_string_literal: true # Specify the password for a PEM or PKCS12 private key, by reading it from a file. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppetdb_ssl_client_password_file) do has_weight 310 order_within_weight 37 def parse(parser, options) parser.on('--puppetdb-ssl-client-password-file FILENAME', 'Read password for SSL client key from a file') do |x| raise Errno::ENOENT, "--puppetdb-ssl-client-password-file #{x} does not point to a valid file" unless File.file?(x) options[:puppetdb_ssl_client_password] = File.read(x) end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/puppet_master_ssl_ca.rb0000644000004100000410000000170113413744025027736 0ustar www-datawww-data# frozen_string_literal: true # Specify the CA certificate for Puppet Master. If specified, this will enable SSL verification # that the certificate being presented has been signed by this CA, and that the common name # matches the name you are using to connecting. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:puppet_master_ssl_ca) do has_weight 320 order_within_weight 30 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'puppet-master-ssl-ca', option_name: 'puppet_master_ssl_ca', desc: 'Full path to CA certificate that signed the Puppet Master certificate', validator: ->(x) { File.file?(x) || raise(Errno::ENOENT, "SSL CA cert #{x} does not exist") } ) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/hiera_path.rb0000644000004100000410000000436113413744025025633 0ustar www-datawww-data# frozen_string_literal: true # Specify the path to the Hiera data directory (relative to the top level Puppet checkout). For Puppet Enterprise and the # Puppet control repo template, the value of this should be 'hieradata', which is the default. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:hiera_path) do has_weight 181 def parse(parser, options) OctocatalogDiff::Cli::Options.option_globally_or_per_branch( parser: parser, options: options, cli_name: 'hiera-path', option_name: 'hiera_path', desc: 'Path to hiera data directory, relative to top directory of repository', validator: lambda do |path| if path.start_with?('/') raise ArgumentError, '--hiera-path PATH must be a relative path not an absolute path' end end, translator: lambda do |path| result = path.sub(%r{/+$}, '') raise ArgumentError, '--hiera-path must not be empty' if result.empty? result end, post_process: lambda do |opts| if opts.key?(:to_hiera_path_strip) && opts[:to_hiera_path_strip] != :none if opts.key?(:to_hiera_path) && opts[:to_hiera_path] != :none raise ArgumentError, '--hiera-path and --hiera-path-strip are mutually exclusive' end end if opts.key?(:from_hiera_path_strip) && opts[:from_hiera_path_strip] != :none if opts.key?(:from_hiera_path) && opts[:from_hiera_path] != :none raise ArgumentError, '--hiera-path and --hiera-path-strip are mutually exclusive' end end if opts[:to_hiera_path] == :none || opts[:from_hiera_path] == :none raise ArgumentError, '--hiera-path and --no-hiera-path are mutually exclusive' end end ) parser.on('--no-hiera-path', 'Do not use any default hiera path settings') do if options[:to_hiera_path].is_a?(String) || options[:from_hiera_path].is_a?(String) raise ArgumentError, '--hiera-path and --no-hiera-path are mutually exclusive' end options[:from_hiera_path] = :none options[:to_hiera_path] = :none end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options/master_cache_branch.rb0000644000004100000410000000100313413744025027450 0ustar www-datawww-data# frozen_string_literal: true # Allow override of the branch that is cached. This defaults to 'origin/master'. # @param parser [OptionParser object] The OptionParser argument # @param options [Hash] Options hash being constructed; this is modified in this method. OctocatalogDiff::Cli::Options::Option.newoption(:master_cache_branch) do has_weight 160 def parse(parser, options) parser.on('--master-cache-branch BRANCH', 'Branch to cache') do |x| options[:master_cache_branch] = x end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/diffs.rb0000644000004100000410000000301313413744025023120 0ustar www-datawww-data# frozen_string_literal: true require_relative '../catalog-diff/differ' module OctocatalogDiff class Cli # Wrapper around OctocatalogDiff::CatalogDiff::Differ to provide the logger object, set up # ignores, and add additional ignores for items dependent upon the compilation directory. class Diffs # Constructor # @param options [Hash] Options from cli/options # @param logger [Logger] Logger object def initialize(options, logger) @options = options @logger = logger end # The method to call externally, passing in the catalogs as a hash (see parameter). This # sets up options and ignores and then actually performs the diffs. The result is the array # of diffs. # @param catalogs [Hash] { :to => OctocatalogDiff::Catalog, :from => OctocatalogDiff::Catalog } # @return [Array] Array of diffs def diffs(catalogs) @logger.debug 'Begin compute diffs between catalogs' diff_opts = @options.merge(logger: @logger) # Construct the actual differ object that the present one wraps differ = OctocatalogDiff::CatalogDiff::Differ.new(diff_opts, catalogs[:from], catalogs[:to]) differ.ignore(attr: 'tags') unless @options.fetch(:include_tags, false) differ.ignore(@options.fetch(:ignore, [])) differ.ignore_tags # Actually perform the diff diff_result = differ.diff @logger.debug 'Success compute diffs between catalogs' diff_result end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/options.rb0000644000004100000410000002235413413744025023531 0ustar www-datawww-data# frozen_string_literal: true require_relative '../cli' require_relative '../facts' require_relative '../version' require 'optparse' module OctocatalogDiff class Cli # This class contains the option parser. 'parse_options' is the external entry point. class Options # The usage banner. BANNER = 'Usage: catalog-diff -n [,...] [-f ] [-t ]'.freeze # An error class specifically for passing information to the document build task. class DocBuildError < RuntimeError; end # List of classes def self.classes @classes ||= [] end # Define the Option class and newoption() method for use by cli/options/*.rb files class Option def self.has_weight(w) # rubocop:disable Style/PredicateName @weight = w end def self.order_within_weight(w) # rubocop:disable Style/TrivialAccessors @order_within_weight = w end def self.weight if @weight && @order_within_weight @weight + (@order_within_weight / 100.0) elsif @weight @weight else # :nocov: raise ArgumentError, "Option #{name} does not have a weight specified. Add 'has_weight NNN' to control ordering." # :nocov: end end def self.name self::NAME end def self.newoption(name, &block) klass = Class.new(OctocatalogDiff::Cli::Options::Option) klass.const_set('NAME', name) klass.class_exec(&block) Options.classes.push(klass) end end # Method to call all of the other methods in this class. Except in very specific circumstances, # this should be the method called from outside of this class. # @param argv [Array] Array of command line arguments # @param defaults [Hash] Default values # @return [Hash] Parsed options def self.parse_options(argv, defaults = {}) options = defaults.dup Options.classes.clear ::OptionParser.new do |parser| parser.banner = "#{BANNER}\n\n" option_classes.each do |klass| obj = klass.new obj.parse(parser, options) end parser.on_tail('-v', '--version', 'Show version information about this program and quit.') do puts "octocatalog-diff #{OctocatalogDiff::Version::VERSION}" exit end end.parse! argv options end # Read in *.rb files in the 'options' directory and create classes from them. # Sort the classes according to weight and name and return the list of sorted classes. # @return [Array] Sorted classes def self.option_classes files = Dir.glob(File.join(File.dirname(__FILE__), 'options', '*.rb')) files.each { |file| load file } # Populates self.classes classes.sort do |a, b| [ a.weight <=> b.weight, a.name.downcase <=> b.name.downcase, a.object_id <=> b.object_id ].find(&:nonzero?) end end # Sets up options that can be defined globally or for just one branch. For example, with a # CLI name of 'puppet-binary' this will acknowledge 3 options: --puppet-binary (global), # --from-puppet-binary (for the from branch only), and --to-puppet-binary (for the to branch # only). The only options that will be created are the 'to' and 'from' variants, but the global # option will populate any of the 'to' and 'from' variants that are missing. # @param :datatype [?] Expected data type def self.option_globally_or_per_branch(opts = {}) opts[:filename] = caller[0].split(':').first datatype = opts.fetch(:datatype, '') return option_globally_or_per_branch_string(opts) if datatype.is_a?(String) return option_globally_or_per_branch_array(opts) if datatype.is_a?(Array) raise ArgumentError, "option_globally_or_per_branch not equipped to handle #{datatype.class}" end # See description of `option_globally_or_per_branch`. This implements the logic for a string value. # @param :parser [OptionParser object] The OptionParser argument # @param :options [Hash] Options hash being constructed; this is modified in this method. # @param :cli_name [String] Name of option on command line (e.g. puppet-binary) # @param :option_name [Symbol] Name of option in the options hash (e.g. :puppet_binary) # @param :desc [String] Description of option on the command line; will have "for the XX branch" appended def self.option_globally_or_per_branch_string(opts) parser = opts.fetch(:parser) options = opts.fetch(:options) cli_name = opts.fetch(:cli_name) option_name = opts.fetch(:option_name) desc = opts.fetch(:desc) flag = "#{cli_name} STRING" from_option = "from_#{option_name}".to_sym to_option = "to_#{option_name}".to_sym parser.on("--#{flag}", "#{desc} globally") do |x| validate_option(opts, x) translated = translate_option(opts[:translator], x) options[to_option] ||= translated options[from_option] ||= translated post_process(opts[:post_process], options) end parser.on("--to-#{flag}", "#{desc} for the to branch") do |x| validate_option(opts, x) options[to_option] = translate_option(opts[:translator], x) post_process(opts[:post_process], options) end parser.on("--from-#{flag}", "#{desc} for the from branch") do |x| validate_option(opts, x) options[from_option] = translate_option(opts[:translator], x) post_process(opts[:post_process], options) end end # See description of `option_globally_or_per_branch`. This implements the logic for an array. # @param :parser [OptionParser object] The OptionParser argument # @param :options [Hash] Options hash being constructed; this is modified in this method. # @param :cli_name [String] Name of option on command line (e.g. puppet-binary) # @param :option_name [Symbol] Name of option in the options hash (e.g. :puppet_binary) # @param :desc [String] Description of option on the command line; will have "for the XX branch" appended def self.option_globally_or_per_branch_array(opts = {}) parser = opts.fetch(:parser) options = opts.fetch(:options) cli_name = opts.fetch(:cli_name) option_name = opts.fetch(:option_name) desc = opts.fetch(:desc) flag = "#{cli_name} STRING1[,STRING2[,...]]" from_option = "from_#{option_name}".to_sym to_option = "to_#{option_name}".to_sym parser.on("--#{flag}", Array, "#{desc} globally") do |x| validate_option(opts, x) translated = translate_option(opts[:translator], x) options[to_option] ||= [] options[to_option].concat translated options[from_option] ||= [] options[from_option].concat translated end parser.on("--to-#{flag}", Array, "#{desc} for the to branch") do |x| validate_option(opts, x) options[to_option] ||= [] options[to_option].concat translate_option(opts[:translator], x) end parser.on("--from-#{flag}", Array, "#{desc} for the from branch") do |x| validate_option(opts, x) options[from_option] ||= [] options[from_option].concat translate_option(opts[:translator], x) end end # If a validator was provided, run the validator on the supplied value. The validator is expected to # throw an error if there is a problem. Note that the validator runs *before* the translator if both # a validator and translator are supplied. # @param opts [Hash] Options hash # @param value [?] Value to validate (typically a String but can really be anything) def self.validate_option(opts, value) # Special value to help build documentation automatically, since the source file location # for `option_globally_or_per_branch` is always this file. raise DocBuildError, opts[:filename] if value == :DOC_BUILD_FILENAME validator = opts[:validator] return true unless validator validator.call(value) end # If a translator was provided, run the translator on the supplied value. The translator is expected # to return the data type needed for the option (typically a String but can really be anything). Note # that the translator runs *after* the validator if both a validator and translator are supplied. # @param translator [Code] Translator function # @param value [?] Original input value # @return [?] Translated value def self.translate_option(translator, value) return value if translator.nil? translator.call(value) end # Code that can run after a translation and operate upon all options. This returns nothing but may # modify options that were input. # @param processor [Code] Processor function # @param options [Hash] Options hash def self.post_process(processor, options) return if processor.nil? processor.call(options) end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli/printer.rb0000644000004100000410000000416413413744025023520 0ustar www-datawww-data# frozen_string_literal: true require_relative '../catalog-diff/display' require_relative '../errors' module OctocatalogDiff class Cli # Wrapper around OctocatalogDiff::CatalogDiff::Display to set the options and # output to a file or the screen depending on selection. class Printer # Constructor # @param options [Hash] Options from cli/options # @param logger [Logger] Logger object def initialize(options, logger) @options = options @logger = logger end # The method to call externally, passing in diffs. This takes the appropriate action # based on options, which is either to write the result into an output file, or print # the result on STDOUT. Does not return anything. # @param diffs [Array] Array of differences # @param from_dir [String] Directory in which "from" catalog was compiled # @param to_dir [String] Directory in which "to" catalog was compiled def printer(diffs, from_dir = nil, to_dir = nil) unless diffs.is_a?(Array) raise ArgumentError, "printer() expects an array, not #{diffs.class}" end display_opts = @options.merge(compilation_from_dir: from_dir, compilation_to_dir: to_dir) diff_text = OctocatalogDiff::CatalogDiff::Display.output(diffs, display_opts, @logger) if @options[:output_file].nil? puts diff_text unless diff_text.empty? else output_to_file(diff_text) end end private # Output to a file, handling errors related to writing files. # @param diff_in [String|Array] Text to write to file def output_to_file(diff_in) diff_text = diff_in.is_a?(Array) ? diff_in.join("\n") : diff_in File.open(@options[:output_file], 'w') { |f| f.write(diff_text) } @logger.info "Wrote diff to #{@options[:output_file]}" rescue Errno::ENOENT, Errno::EACCES, Errno::EISDIR => exc @logger.error "Cannot write to #{@options[:output_file]}: #{exc}" raise OctocatalogDiff::Errors::PrinterError, "Cannot write to #{@options[:output_file]}: #{exc}" end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog/0000755000004100000410000000000013413744025022346 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/catalog/noop.rb0000644000004100000410000000065613413744025023655 0ustar www-datawww-data# frozen_string_literal: true require_relative '../catalog' require 'json' module OctocatalogDiff class Catalog # Represents a null Puppet catalog. class Noop < OctocatalogDiff::Catalog def initialize(options) super @catalog_json = '{"resources":[]}' @catalog = { 'resources' => [] } @error_message = nil @node = options.fetch(:node, 'noop') end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog/json.rb0000644000004100000410000000304513413744025023646 0ustar www-datawww-data# frozen_string_literal: true require_relative '../catalog' require_relative '../catalog-util/fileresources' require 'json' module OctocatalogDiff class Catalog # Represents a Puppet catalog that is read in directly from a JSON file. class JSON < OctocatalogDiff::Catalog # Constructor # @param :json [String] REQUIRED: Content of catalog, will be parsed as JSON # @param :node [String] Node name (if not supplied, will be determined from catalog) def initialize(options) super unless options[:json].is_a?(String) raise ArgumentError, "Must supply :json as string in options: #{options[:json].class}" end @catalog_json = options.fetch(:json) begin @catalog = ::JSON.parse(@catalog_json) @error_message = nil @node ||= @catalog['name'] if @catalog.key?('name') # Puppet 4.x @node ||= @catalog['data']['name'] if @catalog.key?('data') && @catalog['data'].is_a?(Hash) # Puppet 3.x rescue ::JSON::ParserError => exc @error_message = "Catalog JSON input failed to parse: #{exc.message}" @catalog = nil @catalog_json = nil end end # Convert file resources source => "puppet:///..." to content => "actual content of file". def convert_file_resources(dry_run = false) return @options.key?(:basedir) if dry_run return false unless @options[:basedir] OctocatalogDiff::CatalogUtil::FileResources.convert_file_resources(self, environment) end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog/puppetmaster.rb0000644000004100000410000001303713413744025025430 0ustar www-datawww-data# frozen_string_literal: true require_relative '../catalog' require_relative '../catalog-util/facts' require_relative '../external/pson/pure' require_relative '../util/httparty' require 'json' require 'securerandom' require 'stringio' module OctocatalogDiff class Catalog # Represents a Puppet catalog that is obtained by contacting the Puppet Master. class PuppetMaster < OctocatalogDiff::Catalog # Defaults DEFAULT_PUPPET_PORT_NUMBER = 8140 DEFAULT_PUPPET_SERVER_API = 3 PUPPET_MASTER_TIMEOUT = 60 # Constructor # @param :node [String] Node name # @param :retry_failed_catalog [Integer] Number of retries, if fetch fails # @param :branch [String] Environment to fetch from Puppet Master # @param :puppet_master [String] Puppet server and port number (assumed to be DEFAULT_PUPPET_PORT_NUMBER if not given) # @param :puppet_master_api_version [Integer] Puppet server API (default DEFAULT_PUPPET_SERVER_API) # @param :puppet_master_ssl_ca [String] Path to file used to sign puppet master's certificate # @param :puppet_master_ssl_verify [Boolean] Override the CA verification setting guessed from parameters # @param :puppet_master_ssl_client_pem [String] PEM-encoded client key and certificate # @param :puppet_master_ssl_client_p12 [String] pkcs12-encoded client key and certificate # @param :puppet_master_ssl_client_password [String] Path to file containing password for SSL client key (any format) # @param :puppet_master_ssl_client_auth [Boolean] Override the client-auth that is guessed from parameters # @param :timeout [Integer] Connection timeout for Puppet master (default=PUPPET_MASTER_TIMEOUT seconds) def initialize(options) super unless @options[:node].is_a?(String) && @options[:node] != '' raise ArgumentError, 'node must be a non-empty string' end unless @options[:branch].is_a?(String) && @options[:branch] != '' raise ArgumentError, 'Environment must be a non-empty string' end unless @options[:puppet_master].is_a?(String) && @options[:puppet_master] != '' raise ArgumentError, 'Puppet Master must be a non-empty string' end @timeout = options.fetch(:puppet_master_timeout, options.fetch(:timeout, PUPPET_MASTER_TIMEOUT)) @retry_failed_catalog = options.fetch(:retry_failed_catalog, 0) @options[:puppet_master] += ":#{DEFAULT_PUPPET_PORT_NUMBER}" unless @options[:puppet_master] =~ /\:\d+$/ end private # Build method def build_catalog(logger = Logger.new(StringIO.new)) facts_obj = OctocatalogDiff::CatalogUtil::Facts.new(@options, logger) logger.debug "Start retrieving facts for #{@node} from #{self.class}" @facts = facts_obj.facts logger.debug "Success retrieving facts for #{@node} from #{self.class}" fetch_catalog(logger) end # Returns a hash of parameters for each supported version of the Puppet Server Catalog API. # @return [Hash] Hash of parameters # # Note: The double escaping of the facts here is implemented to correspond to a long standing # bug in the Puppet code. See https://github.com/puppetlabs/puppet/pull/1818 and # https://docs.puppet.com/puppet/latest/http_api/http_catalog.html#parameters for explanation. def puppet_catalog_api { 2 => { url: "https://#{@options[:puppet_master]}/#{@options[:branch]}/catalog/#{@node}", parameters: { 'facts_format' => 'pson', 'facts' => CGI.escape(@facts.fudge_timestamp.without('trusted').to_pson), 'transaction_uuid' => SecureRandom.uuid } }, 3 => { url: "https://#{@options[:puppet_master]}/puppet/v3/catalog/#{@node}", parameters: { 'environment' => @options[:branch], 'facts_format' => 'pson', 'facts' => CGI.escape(@facts.fudge_timestamp.without('trusted').to_pson), 'transaction_uuid' => SecureRandom.uuid } } } end # Fetch catalog by contacting the Puppet master, sending the facts, and asking for the catalog. When the # catalog is returned in PSON format, parse it to JSON and then set appropriate variables. def fetch_catalog(logger) api_version = @options[:puppet_master_api_version] || DEFAULT_PUPPET_SERVER_API api = puppet_catalog_api[api_version] raise ArgumentError, "Unsupported or invalid API version #{api_version}" unless api.is_a?(Hash) more_options = { headers: { 'Accept' => 'text/pson' }, timeout: @timeout } post_hash = api[:parameters] response = nil 0.upto(@retry_failed_catalog) do |retry_num| @retries = retry_num logger.debug "Retrieve catalog from #{api[:url]} environment #{@options[:branch]}" response = OctocatalogDiff::Util::HTTParty.post(api[:url], @options.merge(more_options), post_hash, 'puppet_master') logger.debug "Response from #{api[:url]} environment #{@options[:branch]} was #{response[:code]}" break if response[:code] == 200 end unless response[:code] == 200 @error_message = "Failed to retrieve catalog from #{api[:url]}: #{response[:code]} #{response[:body]}" @catalog = nil @catalog_json = nil return end @catalog = response[:parsed] @catalog_json = ::JSON.generate(@catalog) @error_message = nil end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog/computed.rb0000644000004100000410000002231413413744025024515 0ustar www-datawww-data# frozen_string_literal: true require 'fileutils' require 'json' require 'stringio' require_relative '../catalog' require_relative '../catalog-util/bootstrap' require_relative '../catalog-util/builddir' require_relative '../catalog-util/command' require_relative '../catalog-util/facts' require_relative '../util/puppetversion' require_relative '../util/scriptrunner' require_relative '../util/util' module OctocatalogDiff class Catalog # Represents a Puppet catalog that is computed (via `puppet master --compile ...`) # By instantiating this class, the catalog is computed. class Computed < OctocatalogDiff::Catalog # Constructor # @param :node [String] REQUIRED: Node name # @param :basedir [String] Directory in which to compile the catalog # @param :pass_env_vars [Array] Environment variables to pass when compiling catalog # @param :retry_failed_catalog [Integer] Number of retries if a catalog compilation fails # @param :tag [String] For display purposes, the catalog being compiled # @param :puppet_binary [String] Full path to Puppet # @param :puppet_version [String] Puppet version (optional; if not supplied, it is calculated) # @param :puppet_command [String] Full command to run Puppet (optional; if not supplied, it is calculated) def initialize(options) super raise ArgumentError, 'Node name must be passed to OctocatalogDiff::Catalog::Computed' unless options[:node].is_a?(String) raise ArgumentError, 'Branch is undefined' unless options[:branch] # Additional class variables @pass_env_vars = options.fetch(:pass_env_vars, []) @retry_failed_catalog = options.fetch(:retry_failed_catalog, 0) @tag = options.fetch(:tag, 'catalog') @puppet_binary = options[:puppet_binary] @puppet_version = options[:puppet_version] @puppet_command = options[:puppet_command] @builddir = nil @facts_terminus = options.fetch(:facts_terminus, 'yaml') end # Get the Puppet version # @return [String] Puppet version def puppet_version raise ArgumentError, '"puppet_binary" was not passed to OctocatalogDiff::Catalog::Computed' unless @puppet_binary @puppet_version ||= OctocatalogDiff::Util::PuppetVersion.puppet_version(@puppet_binary, @options) end # Compilation directory # @return [String] Compilation directory def compilation_dir raise 'Catalog was not built' if @builddir.nil? @builddir.tempdir end # Environment used to compile catalog def environment @options.fetch(:environment, 'production') end # Convert file resources source => "puppet:///..." to content => "actual content of file". def convert_file_resources(dry_run = false) return @options.key?(:basedir) if dry_run return false unless @options[:basedir] OctocatalogDiff::CatalogUtil::FileResources.convert_file_resources(self, environment) end # Private method: Bootstrap a directory def bootstrap(logger) return if @builddir # Fill options for creating and populating the temporary directory tmphash = @options.dup # Bootstrap directory if needed if !@options[:bootstrapped_dir].nil? raise Errno::ENOENT, "Invalid dir #{@options[:bootstrapped_dir]}" unless File.directory?(@options[:bootstrapped_dir]) tmphash[:basedir] = @options[:bootstrapped_dir] elsif @options[:branch] == '.' if @options[:bootstrap_current] tmphash[:basedir] = OctocatalogDiff::Util::Util.temp_dir('ocd-bootstrap-basedir-') FileUtils.cp_r File.join(@options[:basedir], '.'), tmphash[:basedir] o = @options.reject { |k, _v| k == :branch }.merge(path: tmphash[:basedir]) OctocatalogDiff::CatalogUtil::Bootstrap.bootstrap_directory(o, logger) else tmphash[:basedir] = @options[:basedir] end else tmphash[:basedir] = OctocatalogDiff::Util::Util.temp_dir('ocd-bootstrap-checkout-') OctocatalogDiff::CatalogUtil::Bootstrap.bootstrap_directory(@options.merge(path: tmphash[:basedir]), logger) end # Create and populate the temporary directory @builddir ||= OctocatalogDiff::CatalogUtil::BuildDir.new(tmphash, logger) end # Private method: Build catalog by running Puppet # @param logger [Logger] Logger object def build_catalog(logger) if @facts_terminus != 'facter' facts_obj = OctocatalogDiff::CatalogUtil::Facts.new(@options, logger) logger.debug "Start retrieving facts for #{@node} from #{self.class}" @options[:facts] = facts_obj.facts logger.debug "Success retrieving facts for #{@node} from #{self.class}" end bootstrap(logger) result = run_puppet(logger) @retries = result[:retries] if (result[:exitcode]).zero? begin @catalog = ::JSON.parse(result[:stdout]) @catalog_json = result[:stdout] @error_message = nil rescue ::JSON::ParserError => exc @catalog = nil @catalog_json = nil @error_message = "Catalog has invalid JSON: #{exc.message}" end else @error_message = result[:stderr] @catalog = nil @catalog_json = nil end end # Get the command to compile the catalog # @return [String] Puppet command line def puppet_command puppet_command_obj.puppet_command end def puppet_command_obj @puppet_command_obj ||= begin raise ArgumentError, '"puppet_binary" was not passed to OctocatalogDiff::Catalog::Computed' unless @puppet_binary command_opts = @options.merge( node: @node, compilation_dir: @builddir.tempdir, parser: @options.fetch(:parser, :default), puppet_binary: @puppet_binary, fact_file: @builddir.fact_file, dir: @builddir.tempdir, enc: @builddir.enc ) OctocatalogDiff::CatalogUtil::Command.new(command_opts) end end # Private method: Actually execute puppet # @return [Hash] { stdout, stderr, exitcode } def exec_puppet(logger) # This is the environment provided to the puppet command. env = {} @pass_env_vars.each { |var| env[var] ||= ENV[var] } # This is the Puppet command itself env['OCD_PUPPET_BINARY'] = @puppet_command_obj.puppet_binary # Additional passed-in options sr_run_opts = env.merge( logger: logger, working_dir: @builddir.tempdir, argv: @puppet_command_obj.puppet_argv ) # Set up the ScriptRunner scriptrunner = OctocatalogDiff::Util::ScriptRunner.new( default_script: 'puppet/puppet.sh', override_script_path: @options[:override_script_path] ) begin scriptrunner.run(sr_run_opts) rescue OctocatalogDiff::Util::ScriptRunner::ScriptException => exc logger.warn "Puppet command failed: #{exc.message}" if logger end { stdout: scriptrunner.stdout, stderr: scriptrunner.stderr, exitcode: scriptrunner.exitcode } end # Private method: Make sure that the Puppet environment directory exists. def assert_that_puppet_environment_directory_exists target_dir = File.join(@builddir.tempdir, 'environments', environment) return if File.directory?(target_dir) raise Errno::ENOENT, "Environment directory #{target_dir} does not exist" end # Private method: Runs puppet on the command line to compile the catalog # Exit code is 0 if catalog generation was successful, non-zero otherwise. # @param logger [Logger] Logger object # @return [Hash] { stdout: , stderr: , exitcode: } def run_puppet(logger) assert_that_puppet_environment_directory_exists # Run 'cmd' with environment 'env' from directory 'dir' # First line of a successful result needs to be stripped off. It will look like: # Notice: Compiled catalog for xxx in environment production in 27.88 seconds retval = {} 0.upto(@retry_failed_catalog) do |retry_num| @retries = retry_num time_begin = Time.now logger.debug("(#{@tag}) Try #{1 + retry_num} executing Puppet #{puppet_version}: #{puppet_command}") result = exec_puppet(logger) # Success if (result[:exitcode]).zero? logger.debug("(#{@tag}) Catalog succeeded on try #{1 + retry_num} in #{Time.now - time_begin} seconds") first_brace = result[:stdout].index('{') || 0 retval = { stdout: result[:stdout][first_brace..-1], stderr: nil, exitcode: 0, retries: retry_num } break end # Failure logger.debug("(#{@tag}) Catalog failed on try #{1 + retry_num} in #{Time.now - time_begin} seconds") retval = result.merge(retries: retry_num) end retval end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog/puppetdb.rb0000644000004100000410000000564713413744025024532 0ustar www-datawww-data# frozen_string_literal: true require 'json' require 'stringio' require_relative '../catalog' require_relative '../errors' require_relative '../puppetdb' module OctocatalogDiff class Catalog # Represents a Puppet catalog that is read from PuppetDB. class PuppetDB < OctocatalogDiff::Catalog # Constructor - See OctocatalogDiff::PuppetDB for additional parameters # @param :node [String] Node name # @param :retry [Integer] Number of retries, if fetch fails def initialize(options) super unless @options[:node].is_a?(String) && @options[:node] != '' raise ArgumentError, 'node must be a non-empty string' end end private # Private method: Get catalog from PuppetDB. Sets @catalog / @catalog_json or @error_message # @param logger [Logger object] Logger object def build_catalog(logger) # Use OctocatalogDiff::PuppetDB to interact with puppetdb puppetdb_obj = OctocatalogDiff::PuppetDB.new(@options) # Loop to retrieve catalog from PuppetDB uri = "/pdb/query/v4/catalogs/#{@node}" retries = @options.fetch(:retry, 1) (retries + 1).times do @retries = -1 if @retries.nil? @retries += 1 begin # Fetch catalog from PuppetDB logger.debug "Retrieving #{@node} from #{uri}" time_start = Time.now result = puppetdb_obj.get(uri) time_it_took = Time.now - time_start # Validate received catalog raise "PuppetDB catalog for #{@node} failed: no 'resources' hash in object" unless result.key?('resources') raise "PuppetDB catalog for #{@node} failed: 'resources' was not a hash" unless result['resources'].is_a?(Hash) logger.debug "Catalog for #{@node} retrieved from PuppetDB in #{time_it_took} seconds" # Make this look like a generated catalog in Puppet 4.x @catalog = result.merge('resources' => result['resources']['data']) @catalog['resources'] = @catalog['resources'].map { |x| x.reject { |k, _v| k == 'resource' } } # Set the other variables @catalog_json = ::JSON.generate(@catalog) @error_message = nil rescue OctocatalogDiff::Errors::PuppetDBConnectionError => exc @error_message = "Catalog retrieval failed (#{exc.class}) (#{exc.message})" rescue OctocatalogDiff::Errors::PuppetDBNodeNotFoundError => exc @error_message = "Node #{node} not found in PuppetDB (#{exc.message})" rescue OctocatalogDiff::Errors::PuppetDBGenericError => exc @error_message = "Catalog retrieval failed for node #{node} from PuppetDB (#{exc.message})" rescue ::JSON::GeneratorError => exc @error_message = "Failed to generate result from PuppetDB as JSON (#{exc.message})" end break if @catalog end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/errors.rb0000644000004100000410000000205513413744025022577 0ustar www-datawww-data# frozen_string_literal: true module OctocatalogDiff # Contains error classes raised by this gem class Errors # Error classes for handled configuration file errors class ConfigurationFileNotFoundError < RuntimeError; end class ConfigurationFileContentError < RuntimeError; end # Error classes for building catalogs class BootstrapError < RuntimeError; end class CatalogError < RuntimeError; end class PuppetVersionError < RuntimeError; end class GitCheckoutError < RuntimeError; end # Error classes for retrieving facts class FactSourceError < RuntimeError; end class FactRetrievalError < RuntimeError; end # Errors for PuppetDB class PuppetDBNodeNotFoundError < RuntimeError; end class PuppetDBGenericError < RuntimeError; end class PuppetDBConnectionError < RuntimeError; end # Errors for Puppet Enterprise class PEClassificationError < RuntimeError; end # Miscellanous catalog-diff errors class DifferError < RuntimeError; end class PrinterError < RuntimeError; end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/util/0000755000004100000410000000000013413744025021711 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/util/scriptrunner.rb0000644000004100000410000001341213413744025024775 0ustar www-datawww-data# frozen_string_literal: true # Execute a built-in script (which can also be overridden with a user-supplied script) require 'fileutils' require 'open3' require 'shellwords' require_relative 'util' module OctocatalogDiff module Util # This is a utility class to execute a built-in script. class ScriptRunner # For an exception running the script class ScriptException < RuntimeError; end attr_reader :script, :script_src, :logger, :stdout, :stderr, :exitcode # Create the object - the object is a configured script, which can be executed multiple # times with different environment varibles. # # @param opts [Hash] Options hash # opts[:default_script] (Required) Path to script, relative to `scripts` directory # opts[:logger] (Optional) Logger object # opts[:override_script_path] (Optional) Directory where a similarly-named script MAY exist def initialize(opts = {}) @logger = opts[:logger] @script_src = find_script(opts.fetch(:default_script), opts[:override_script_path]) @script = temp_script(@script_src) @stdout = nil @stderr = nil @exitcode = nil end # Execute the script from a given working directory, with additional environment variables # specified in the options hash. # # @param opts [Hash] Options hash # opts[:working_dir] (Required) Directory where script is to be executed # opts[:argv] (Optional Array) Command line arguments # opts[:pass_env_vars] (Optional Array) Environment variables to pass (default: HOME, PATH) # opts[] (Optional) Environment variable def run(opts = {}) working_dir = opts.fetch(:working_dir) assert_directory_exists(working_dir) argv = opts.fetch(:argv, []) logger = opts[:logger] || @logger pass_env_vars = [opts[:pass_env_vars], 'HOME', 'PATH'].flatten.compact env = opts.select { |k, _v| k.is_a?(String) } pass_env_vars.each { |var| env[var] ||= ENV[var] } env['PWD'] = working_dir cmdline = [script, argv].flatten.compact.map { |x| Shellwords.escape(x) }.join(' ') log(:debug, "Execute: #{cmdline}", opts[:logger]) @stdout, @stderr, status = Open3.capture3(env, cmdline, unsetenv_others: true, chdir: working_dir) @exitcode = status.exitstatus @stderr.split(/\n/).select { |line| line =~ /\S/ }.each { |line| log(:debug, "STDERR: #{line}", logger) } log(:debug, "Exit status: #{@exitcode}", logger) return @stdout if @exitcode.zero? raise ScriptException, output end # All output from the latest execution of the command. # @return [String] Combined output of STDOUT and STDERR def output return if @exitcode.nil? [ 'STDOUT:', @stdout.split(/\n/).map { |line| " #{line}" }, 'STDERR:', @stderr.split(/\n/).map { |line| " #{line}" } ].flatten.compact.join("\n") end private # PRIVATE: Log a message, if logger is defined. Since this might be called under `parallel` # it's possible that the logger isn't defined, and if so the logged message is skipped. def log(priority, message, logger = @logger) return unless logger logger.send(priority, [message]) end # PRIVATE: Create a temporary file with the contents of the script and mark the script executable. # This is to avoid changing ownership or permissions on any user-supplied file. # # @param script [String] Path to script # @return [String] Path to tempfile containing script def temp_script(script) raise Errno::ENOENT, "Script '#{script}' not found" unless File.file?(script) temp_dir = OctocatalogDiff::Util::Util.temp_dir('ocd-scriptrunner') temp_file = File.join(temp_dir, File.basename(script)) File.open(temp_file, 'w') { |f| f.write(File.read(script)) } FileUtils.chmod 0o755, temp_file temp_file end # PRIVATE: Determine the path to the script to execute, taking into account the default script # location and the optional override script path. # # @param default_script [String] Path to script, relative to `scripts` directory # @param override_script_path [String] Optional directory with override script # @return [String] Full path to script def find_script(default_script, override_script_path = nil) script = find_script_from_override_path(default_script, override_script_path) || File.expand_path("../../../scripts/#{default_script}", File.dirname(__FILE__)) raise Errno::ENOENT, "Unable to locate script '#{script}'" unless File.file?(script) script end # PRIVATE: Find script from override path. # # @param default_script [String] Path to script, relative to `scripts` directory # @param override_script_path [String] Optional directory # @return [String] Override script if found, else nil def find_script_from_override_path(default_script, override_script_path = nil) return unless override_script_path script_test = File.join(override_script_path, File.basename(default_script)) if File.file?(script_test) log(:debug, "Selecting #{script_test} from override script path") script_test else log(:debug, "Did not find #{script_test} in override script path") nil end end # PRIVATE: Assert that a directory exists (and is a directory). Raise error if not. # # @param dir [String] Directory to test def assert_directory_exists(dir) return if File.directory?(dir) raise Errno::ENOENT, "Invalid directory '#{dir}'" end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/util/catalogs.rb0000644000004100000410000003070013413744025024033 0ustar www-datawww-data# frozen_string_literal: true require 'json' require 'open3' require 'yaml' require_relative '../catalog' require_relative '../errors' require_relative 'parallel' module OctocatalogDiff module Util # Helper class to construct catalogs, performing all necessary steps such as # bootstrapping directories, installing facts, and running puppet. class Catalogs # Constructor # @param options [Hash] Options # @param logger [Logger] Logger object def initialize(options, logger) @options = options @logger = logger @catalogs = nil raise '@logger must not be nil' if @logger.nil? end # Compile catalogs. This handles building both the old and new catalog (in parallel) and returns # only when both catalogs have been built. # @return [Hash] { :from => [OctocatalogDiff::Catalog], :to => [OctocatalogDiff::Catalog] } def catalogs @catalogs ||= build_catalog_parallelizer end # Handles the "bootstrap then exit" option, which bootstraps directories but # exits without compiling catalogs. def bootstrap_then_exit @logger.debug('Begin bootstrap_then_exit') OctocatalogDiff::CatalogUtil::Bootstrap.bootstrap_directory_parallelizer(@options, @logger) @logger.debug('Success bootstrap_then_exit') @logger.info('Successfully completed --bootstrap-then-exit action') end private # Parallelizes bootstrapping of directories and building catalogs. # @return [Hash] { :from => OctocatalogDiff::Catalog, :to => OctocatalogDiff::Catalog } def build_catalog_parallelizer # Construct parallel tasks. The array supplied to OctocatalogDiff::Util::Parallel is the task portion # of each of the tuples in catalog_tasks. catalog_tasks = build_catalog_tasks # Update any tasks for catalogs that do not need to be compiled. This is the case when --catalog-only # is specified and only one catalog is to be built. This will change matching catalog tasks to the 'noop' type. catalog_tasks.map! do |x| if @options["#{x[0]}_catalog".to_sym] == '-' x[1].args[:backend] = :noop elsif @options["#{x[0]}_catalog".to_sym].is_a?(String) x[1].args[:json] = File.read(@options["#{x[0]}_catalog".to_sym]) x[1].args[:backend] = :json end x end # Initialize the objects for each parallel task. Initializing the object is very fast and does not actually # build the catalog. result = {} catalog_tasks.each do |x| result[x[0]] = OctocatalogDiff::Catalog.create(x[1].args) @logger.debug "Initialized #{result[x[0]].builder} for #{x[0]}-catalog" end # Disable --compare-file-text if either (or both) of the chosen backends do not support it if @options.fetch(:compare_file_text, false) result.each do |_key, builder_obj| next if builder_obj.convert_file_resources(true) @logger.debug "Disabling --compare-file-text; not supported by #{builder_obj.builder}" @options[:compare_file_text] = false catalog_tasks.map! do |x| x[1].args[:compare_file_text] = false x end break end end # Inject the starting object into the catalog tasks catalog_tasks.map! do |x| x[1].args[:object] = result[x[0]] x end # Execute the parallelized catalog builds passed_catalog_tasks = catalog_tasks.map { |x| x[1] } parallel_catalogs = OctocatalogDiff::Util::Parallel.run_tasks(passed_catalog_tasks, @logger, @options[:parallel]) # If the catalogs array is empty at this point, there is an unexpected size mismatch. This should # never happen, but test for it anyway. unless parallel_catalogs.size == catalog_tasks.size # :nocov: raise "BUG: mismatch catalog_result (#{parallel_catalogs.size} vs #{catalog_tasks.size})" # :nocov: end # If catalogs failed to compile, report that. Prefer to display an actual failure message rather # than a generic incomplete parallel task message if there is a more specific message present. failures = parallel_catalogs.reject(&:status) if failures.any? f = failures.reject { |r| r.exception.is_a?(OctocatalogDiff::Util::Parallel::IncompleteTask) }.first f ||= failures.first raise f.exception end # Construct result hash. Will eventually be in the format # { :from => OctocatalogDiff::Catalog, :to => OctocatalogDiff::Catalog } # Analyze the results from parallel run. catalog_tasks.each do |x| # The `parallel_catalog_obj` is a OctocatalogDiff::Util::Parallel::Result. Get the first element from # the parallel_catalogs output. parallel_catalog_obj = parallel_catalogs.shift # Add the result to the 'result' hash add_parallel_result(result, parallel_catalog_obj, x) end # Things have succeeded if the :to and :from catalogs exist at this point. If not, things have # failed, and an exception should be thrown. return result if result.key?(:to) && result.key?(:from) # This is believed to be a bug condition. # :nocov: raise OctocatalogDiff::Errors::CatalogError, 'One or more catalogs failed to compile.' # :nocov: end # Get catalog compilation tasks. # @return [Array<[key, task]>] Catalog tasks def build_catalog_tasks [:from, :to].map do |key| # These are arguments to OctocatalogDiff::Util::Parallel::Task. In most cases the arguments # of OctocatalogDiff::Util::Parallel::Task are taken directly from options, but there are # some defaults or otherwise-named options that must be set here. args = @options.merge( tag: key.to_s, branch: @options["#{key}_env".to_sym] || '-', bootstrapped_dir: @options["bootstrapped_#{key}_dir".to_sym], basedir: @options[:basedir], compare_file_text: @options.fetch(:compare_file_text, true), retry_failed_catalog: @options.fetch(:retry_failed_catalog, 0), parser: @options["parser_#{key}".to_sym] ) args[:basedir] ||= args[:bootstrapped_dir] # If any options are in the form of 'to_SOMETHING' or 'from_SOMETHING', this sets the option to # 'SOMETHING' for the catalog if it matches this key. For example, when compiling the 'to' catalog # when an option of :to_some_arg => 'foo', this sets :some_arg => foo, and deletes :to_some_arg and # :from_some_arg. @options.keys.select { |x| x.to_s =~ /^(to|from)_/ }.each do |opt_key| args[opt_key.to_s.sub(/^(to|from)_/, '').to_sym] = @options[opt_key] if opt_key.to_s.start_with?(key.to_s) args.delete(opt_key) end # Skip reference validation in the from-catalog by saying we already performed it. args[:references_validated] = (key == :from) # The task is a OctocatalogDiff::Util::Parallel::Task object that contains the method to execute, # validator method, text description, and arguments to provide when calling the method. task = OctocatalogDiff::Util::Parallel::Task.new( method: method(:build_catalog), validator: method(:catalog_validator), validator_args: { task: key }, description: "build_catalog for #{@options["#{key}_env".to_sym]}", args: args ) # The format of `catalog_tasks` will be a tuple, where the first element is the key # (e.g. :to or :from) and the second element is the OctocatalogDiff::Util::Parallel::Task object. [key, task] end.compact end # Given a result from the 'parallel' run and a corresponding (key,task) tuple, add valid # catalogs to the 'result' hash and throw errors for invalid catalogs. # @param result [Hash] Result hash for build_catalog_parallelizer (may be modified) # @param parallel_catalog_obj [OctocatalogDiff::Util::Parallel::Result] Parallel catalog result # @param key_task_tuple [Array] Key, task tuple def add_parallel_result(result, parallel_catalog_obj, key_task_tuple) # Expand the tuple into variables key, task = key_task_tuple # For reporting purposes, get the branch name. branch = task.args[:branch] # Check the result of the parallel run on this object. if parallel_catalog_obj.status.nil? # The compile was killed because another task failed. @logger.warn "Catalog compile for #{branch} was aborted due to another failure" elsif parallel_catalog_obj.output.is_a?(OctocatalogDiff::Catalog) # The result is a catalog, but we do not know if it was successfully compiled # until we test the validity. catalog = parallel_catalog_obj.output if catalog.valid? # The catalog was successfully compiled. result[key] = parallel_catalog_obj.output if task.args[:save_catalog] File.open(task.args[:save_catalog], 'w') { |f| f.write(catalog.catalog_json) } @logger.debug "Saved catalog to #{task.args[:save_catalog]}" end else # The catalog failed, but a catalog object was returned so that better error reporting # can take place. In this error reporting, we will replace 'Error:' with '[Puppet Error]' # and remove the compilation directory (which is a tmpdir) to reveal only the relative # path to the files involved. dir = catalog.compilation_dir || '' dir_regex = Regexp.new(Regexp.escape(dir) + '/environments/[^/]+/') error_display = catalog.error_message.split("\n").map do |line| line.sub(/^Error:/, '[Puppet Error]').gsub(dir_regex, '') end.join("\n") message = "Catalog for #{branch} failed to compile due to errors:\n#{error_display}" raise OctocatalogDiff::Errors::CatalogError, message end else # Something unhandled went wrong, and an exception was thrown. Reveal a generic message. # :nocov: msg = parallel_catalog_obj.exception.message message = "Catalog for '#{key}' (#{branch}) failed to compile with #{parallel_catalog_obj.exception.class}: #{msg}" message += "\n" + parallel_catalog_obj.exception.backtrace.map { |x| " #{x}" }.join("\n") if @options[:debug] raise OctocatalogDiff::Errors::CatalogError, message # :nocov: end end # Performs the steps necessary to build a catalog. # @param opts [Hash] Options hash # @return [Hash] { :rc => exit code, :catalog => Catalog as JSON string } def build_catalog(opts, logger = @logger) logger.debug("Setting up Puppet catalog build for #{opts[:branch]}") catalog = opts[:object] logger.debug("Catalog for #{opts[:branch]} will be built with #{catalog.builder}") time_start = Time.now catalog.build(logger) time_it_took = Time.now - time_start retries_str = " retries = #{catalog.retries}" if catalog.retries.is_a?(Integer) time_str = "in #{time_it_took} seconds#{retries_str}" status_str = catalog.valid? ? 'successfully built' : 'failed' logger.debug "Catalog for #{opts[:branch]} #{status_str} with #{catalog.builder} #{time_str}" catalog end # The catalog validator method can indicate failure one of two ways: # - Raise an exception (this is preferred, since it gives a specific error message) # - Return false (supported but discouraged, since it only surfaces a generic error) # @param catalog [OctocatalogDiff::Catalog] Catalog object # @param logger [Logger] Logger object (presently unused) # @param args [Hash] Additional arguments set specifically for validator # @return [Boolean] Return true if catalog is valid, false otherwise def catalog_validator(catalog = nil, _logger = @logger, _args = {}) raise ArgumentError, "Expects a catalog, got #{catalog.class}" unless catalog.is_a?(OctocatalogDiff::Catalog) raise OctocatalogDiff::Errors::CatalogError, "Catalog failed: #{catalog.error_message}" unless catalog.valid? true end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/util/parallel.rb0000644000004100000410000002412513413744025024036 0ustar www-datawww-data# frozen_string_literal: true # A class to parallelize process executation. # This is a utility class to execute tasks in parallel, with our own forking implementation # that passes through logs and reliably handles errors. If parallel processing has been disabled, # this instead executes the tasks serially, but provides the same API as the parallel tasks. require 'stringio' require_relative 'util' module OctocatalogDiff module Util class Parallel # This exception is called for a task that didn't complete. class IncompleteTask < RuntimeError; end # -------------------------------------- # This class represents a parallel task. It requires a method reference, which will be executed with # any supplied arguments. It can optionally take a text description and a validator function. # -------------------------------------- class Task attr_reader :description attr_accessor :args def initialize(opts = {}) @method = opts.fetch(:method) @args = opts.fetch(:args, {}) @description = opts[:description] || @method.name @validator = opts[:validator] @validator_args = opts[:validator_args] || {} end def execute(logger = Logger.new(StringIO.new)) @method.call(@args, logger) end def validate(result, logger = Logger.new(StringIO.new)) return true if @validator.nil? @validator.call(result, logger, @validator_args) end end # -------------------------------------- # This class represents the result from a parallel task. The status is set to true (success), false (error), # or nil (task was killed before it could complete). The exception (for failure) and output object (for success) # are readable attributes. The validity of the results, determined by executing the 'validate' method of the Task, # is available to be set and fetched. # -------------------------------------- class Result attr_reader :output, :args attr_accessor :status, :exception def initialize(opts = {}) @status = opts[:status] @exception = opts[:exception] @output = opts[:output] @args = opts.fetch(:args, {}) end end # -------------------------------------- # Static methods in the class # -------------------------------------- # Entry point for parallel processing. By default this will perform parallel processing, # but it will also accept an option to do serial processing instead. # @param task_array [Array] Tasks to run # @param logger [Logger] Optional logger object # @param parallelized [Boolean] True for parallel processing, false for serial processing # @param raise_exception [Boolean] True to raise exception immediately if one occurs; false to return exception in results # @return [Array] Parallel results (same order as tasks) def self.run_tasks(task_array, logger = nil, parallelized = true, raise_exception = false) # Create a throwaway logger object if one is not given logger ||= Logger.new(StringIO.new) # Validate input - we need an array of OctocatalogDiff::Util::Parallel::Task. If the array is empty then # return an empty array right away. raise ArgumentError, "run_tasks() argument must be array, not #{task_array.class}" unless task_array.is_a?(Array) return [] if task_array.empty? invalid_inputs = task_array.reject { |task| task.is_a?(OctocatalogDiff::Util::Parallel::Task) } if invalid_inputs.any? ele = invalid_inputs.first raise ArgumentError, "Element #{ele.inspect} must be a OctocatalogDiff::Util::Parallel::Task, not a #{ele.class}" end # Initialize the result array. For now all entries in the array indicate that the task was killed. # Actual statuses will replace this initial status. If the initial status wasn't replaced, then indeed, # the task was killed. result = task_array.map { |x| Result.new(exception: IncompleteTask.new('Killed'), args: x.args) } logger.debug "Initialized parallel task result array: size=#{result.size}" # Execute as per the requested method (serial or parallel) and handle results. exception = parallelized ? run_tasks_parallel(result, task_array, logger) : run_tasks_serial(result, task_array, logger) raise exception if exception && raise_exception result end # Utility method! Not intended to be called from outside this class. # --- # Use a forking strategy to run tasks in parallel. Each task in the array is forked in a child # process, and when that task completes it writes its result (OctocatalogDiff::Util::Parallel::Result) # into a serialized data file. Once children are forked this method waits for their return, deserializing # the output from each data file and updating the `result` array with actual results. # @param result [Array] Parallel task results # @param task_array [Array] Tasks to perform # @param logger [Logger] Logger # @return [Exception] First exception encountered by a child process; returns nil if no exceptions encountered. def self.run_tasks_parallel(result, task_array, logger) pidmap = {} ipc_tempdir = OctocatalogDiff::Util::Util.temp_dir('ocd-ipc-') # Child process forking task_array.each_with_index do |task, index| # simplecov doesn't see this because it's forked # :nocov: this_pid = fork do ENV['OCTOCATALOG_DIFF_TEMPDIR'] ||= ipc_tempdir task_result = execute_task(task, logger) File.open(File.join(ipc_tempdir, "#{Process.pid}.dat"), 'w') { |f| f.write Marshal.dump(task_result) } Kernel.exit! 0 # Kernel.exit! avoids at_exit from parents being triggered by children exiting end # :nocov: pidmap[this_pid] = { index: index, start_time: Time.now } logger.debug "Launched pid=#{this_pid} for index=#{index}" logger.reopen if logger.respond_to?(:reopen) end # Waiting for children and handling results while pidmap.any? this_pid, exit_obj = Process.wait2(0) next unless this_pid && pidmap.key?(this_pid) index = pidmap[this_pid][:index] exitstatus = exit_obj.exitstatus raise "PID=#{this_pid} exited abnormally: #{exit_obj.inspect}" if exitstatus.nil? raise "PID=#{this_pid} exited with status #{exitstatus}" unless exitstatus.zero? input = File.read(File.join(ipc_tempdir, "#{this_pid}.dat")) result[index] = Marshal.load(input) # rubocop:disable Security/MarshalLoad time_delta = Time.now - pidmap[this_pid][:start_time] pidmap.delete(this_pid) logger.debug "PID=#{this_pid} completed in #{time_delta} seconds, #{input.length} bytes" next if result[index].status return result[index].exception end logger.debug 'All child processes completed with no exceptions raised' # Cleanup: Kill any child processes that are still running, and clean the temporary directory # where data files were stored. ensure pidmap.each do |pid, _pid_data| begin Process.kill('TERM', pid) rescue Errno::ESRCH # rubocop:disable Lint/HandleExceptions # If the process doesn't exist, that's fine. end end end # Utility method! Not intended to be called from outside this class. # --- # Perform the tasks in serial. # @param result [Array] Parallel task results # @param task_array [Array] Tasks to perform # @param logger [Logger] Logger def self.run_tasks_serial(result, task_array, logger) # Perform the tasks 1 by 1 - each successful task will replace an element in the 'result' array, # whereas a failed task will replace the current element with an exception, and all later tasks # will not be replaced (thereby being populated with the cancellation error). task_array.each_with_index do |ele, task_counter| result[task_counter] = execute_task(ele, logger) next if result[task_counter].status return result[task_counter].exception end nil end # Utility method! Not intended to be called from outside this class. # --- # Process a single task. Called by run_tasks_parallel / run_tasks_serial. # This method will report all exceptions in the OctocatalogDiff::Util::Parallel::Result object # itself, and not raise them. # @param task [OctocatalogDiff::Util::Parallel::Task] Task object # @param logger [Logger] Logger # @return [OctocatalogDiff::Util::Parallel::Result] Parallel task result def self.execute_task(task, logger) begin logger.debug("Begin #{task.description}") output = task.execute(logger) result = Result.new(output: output, status: true, args: task.args) rescue => exc logger.debug("Failed #{task.description}: #{exc.class} #{exc.message}") # Immediately return without running the validation, since this already failed. return Result.new(exception: exc, status: false, args: task.args) end begin if task.validate(output, logger) logger.debug("Success #{task.description}") else # Preferably the validator method raised its own exception. However if it # simply returned false, raise our own exception here. raise "Failed #{task.description} validation (unspecified error)" end rescue => exc logger.warn("Failed #{task.description} validation: #{exc.class} #{exc.message}") result.status = false result.exception = exc end result end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/util/colored.rb0000644000004100000410000000076313413744025023673 0ustar www-datawww-data# frozen_string_literal: true # Create colorizing methods in the 'String' class, but only if 'colors_enabled' # has been set. class String COLORS = { 'red' => 31, 'green' => 32, 'yellow' => 33, 'cyan' => 36 }.freeze COLORS.each do |color, _value| define_method(color) do @@colors_enabled ? "\e[0;#{COLORS[color]};49m#{self}\e[0m" : self end end def self.colors_enabled=(value) @@colors_enabled = value # rubocop:disable Style/ClassVars end end octocatalog-diff-1.5.4/lib/octocatalog-diff/util/puppetversion.rb0000644000004100000410000000262213413744025025163 0ustar www-datawww-data# frozen_string_literal: true # Helper to determine the version of Puppet require_relative 'scriptrunner' module OctocatalogDiff module Util # This is a utility class to determine the version of Puppet. class PuppetVersion # Determine the version of Puppet. # @param puppet [String] Path to Puppet binary # @param options [Hash] Options hash as defined in OctocatalogDiff::Catalog::Computed # @return [String] Puppet version number def self.puppet_version(puppet, options = {}) raise ArgumentError, 'Puppet binary was not supplied' if puppet.nil? raise Errno::ENOENT, "Puppet binary #{puppet} doesn't exist" unless File.file?(puppet) sr_opts = { default_script: 'puppet/puppet.sh', override_script_path: options[:override_script_path] } script = OctocatalogDiff::Util::ScriptRunner.new(sr_opts) sr_run_opts = { :logger => options[:logger], :working_dir => File.dirname(puppet), :pass_env_vars => options[:pass_env_vars], :argv => '--version', 'OCD_PUPPET_BINARY' => puppet } output = script.run(sr_run_opts) return Regexp.last_match(1) if output =~ /^([\d\.]+)\s*$/ # :nocov: raise "Unable to determine Puppet version: #{script.output}" # :nocov: end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/util/util.rb0000644000004100000410000000712213413744025023215 0ustar www-datawww-data# frozen_string_literal: true # Handy methods that are not tied to one particular class require 'fileutils' module OctocatalogDiff module Util # Helper class to construct catalogs, performing all necessary steps such as # bootstrapping directories, installing facts, and running puppet. class Util # Utility Method! # `is_a?(class)` only allows one method, but this uses an array # @param object [?] Object to consider # @param classes [Array] Classes to determine if object is a member of # @return [Boolean] True if object is_a any of the classes, false otherwise def self.object_is_any_of?(object, classes) classes.each { |clazz| return true if object.is_a? clazz } false end # Utility Method! # `.dup` can't be called on certain objects (Fixnum for example). This # method returns the original object if it can't be duplicated. # @param object [?] Object to consider # @return [?] Duplicated object if possible, otherwise the original object def self.safe_dup(object) object.dup rescue TypeError # :nocov: object # :nocov: end # Utility Method! # This does a "deep" duplication via recursion. Handles hashes and arrays. # @param object [?] Object to consider # @return [?] Duplicated object def self.deep_dup(object) if object.is_a?(Hash) result = {} object.each { |k, v| result[k] = deep_dup(v) } result elsif object.is_a?(Array) object.map { |ele| deep_dup(ele) } else safe_dup(object) end end # Utility Method! # This creates a temporary directory. If the base directory is specified, then we # do not remove the temporary directory at exit, because we assume that something # else will remove the base directory. # # prefix - A String with the prefix for the temporary directory # basedir - A String with the directory in which to make the tempdir # # Returns the full path to the temporary directory. def self.temp_dir(prefix = 'ocd-', basedir = ENV['OCTOCATALOG_DIFF_TEMPDIR']) # If the base directory is specified, make sure it exists, and then create the # temporary directory within it. if basedir unless File.directory?(basedir) raise Errno::ENOENT, "temp_dir: Base dir #{basedir.inspect} does not exist!" end return Dir.mktmpdir(prefix, basedir) end # If the base directory was not specified, then create a temporary directory, and # send the `at_exit` to clean it up at the conclusion. the_dir = Dir.mktmpdir(prefix) at_exit { remove_temp_dir(the_dir) } the_dir end # Utility method! # Remove a directory recursively that has been used as a temporary directory. This # should be called within an `at_exit` handler, and is only intended to be called via the # `temp_dir` method above. # # dir - A String with the directory to remove. def self.remove_temp_dir(dir) retries = 0 while File.directory?(dir) && retries < 10 retries += 1 begin FileUtils.remove_entry_secure(dir) rescue Errno::ENOTEMPTY, Errno::ENOENT # rubocop:disable Lint/HandleExceptions # Errno::ENOTEMPTY will trigger a retry because the directory exists # Errno::ENOENT will break the loop because the directory won't exist next time it's checked end end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/util/httparty.rb0000644000004100000410000001702613413744025024123 0ustar www-datawww-data# frozen_string_literal: true require 'httparty' require 'json' require_relative '../external/pson/pure' module OctocatalogDiff module Util # This is a wrapper around some common actions that octocatalog-diff does when preparing to talk # to a web server using 'httparty'. class HTTParty # Wrap the 'get' method in httparty with SSL options # @param url [String] URL to retrieve # @param options [Hash] Options # @param ssl_prefix [String] Strip "#{prefix}_" from the start of SSL options to generalize them # @return [Hash] HTTParty response and codes def self.get(url, options = {}, ssl_prefix = nil) httparty_response_parse(::HTTParty.get(url, options.merge(wrap_ssl_options(options, ssl_prefix)))) end # Wrap the 'post' method in httparty with SSL options # @param url [String] URL to retrieve # @param options [Hash] Options # @param post_body [String] Test to POST # @param ssl_prefix [String] Strip "#{prefix}_" from the start of SSL options to generalize them # @return [Hash] HTTParty response and codes def self.post(url, options, post_body, ssl_prefix) opts = options.merge(wrap_ssl_options(options, ssl_prefix)) httparty_response_parse(::HTTParty.post(url, opts.merge(body: post_body))) end # Common parser for HTTParty response # @param response [HTTParty response object] HTTParty response object # @return [Hash] HTTParty parsed response and codes def self.httparty_response_parse(response) # Handle HTTP errors unless response.code == 200 begin b = JSON.parse(response.body) errormessage = b['error'] if b.is_a?(Hash) && b.key?('error') rescue JSON::ParserError errormessage = response.body ensure errormessage ||= response.body end return { code: response.code, body: response.body, error: errormessage } end # Handle success if response.headers.key?('content-type') if response.headers['content-type'] =~ %r{/json} begin return { code: 200, body: response.body, parsed: JSON.parse(response.body) } rescue JSON::ParserError => exc return { code: 500, body: response.body, error: "JSON parse error: #{exc.message}" } end end if response.headers['content-type'] =~ %r{/pson} begin return { code: 200, body: response.body, parsed: PSON.parse(response.body) } rescue PSON::ParserError => exc return { code: 500, body: response.body, error: "PSON parse error: #{exc.message}" } end end return { code: 500, body: response.body, error: "Don't know how to parse: #{response.headers['content-type']}" } end # Return raw output { code: response.code, body: response.body } end # Wrap context-specific options into generally named options for the other methods in this class # @param options [Hash] Hash of all options # @param prefix [String] Prefix to strip from SSL options # @return [Hash] SSL options generally named def self.wrap_ssl_options(options, prefix) return {} unless prefix result = {} options.keys.each do |key| next if key.to_s !~ /^#{prefix}_(ssl_.*)/ result[Regexp.last_match[1].to_sym] = options[key] end ssl_options(result) end # SSL options to add to the httparty options hash # @param :ssl_ca [String] Optional: File with SSL CA certificate # @param :ssl_client_key [String] Full text of SSL client private key # @param :ssl_client_cert [String] Full text of SSL client public cert # @param :ssl_client_pem [String] Full text of SSL client private key + client public cert # @param :ssl_client_p12 [String] Full text of pkcs12-encoded keypair # @param :ssl_client_password [String] Password to unlock private key # @return [Hash] Hash of SSL options to pass to httparty def self.ssl_options(options) # Initialize the result result = {} # Verification of server against a known CA cert if ssl_verify?(options) result[:verify] = true raise ArgumentError, ':ssl_ca must be passed' unless options[:ssl_ca].is_a?(String) raise Errno::ENOENT, "'#{options[:ssl_ca]}' not a file" unless File.file?(options[:ssl_ca]) result[:ssl_ca_file] = options[:ssl_ca] else result[:verify] = false end # SSL client certificate auth. This translates our options into httparty options. if client_auth?(options) if options[:ssl_client_key].is_a?(String) && options[:ssl_client_cert].is_a?(String) result[:pem] = options[:ssl_client_key] + options[:ssl_client_cert] elsif options[:ssl_client_pem].is_a?(String) result[:pem] = options[:ssl_client_pem] elsif options[:ssl_client_p12].is_a?(String) result[:p12] = options[:ssl_client_p12] raise ArgumentError, 'pkcs12 requires a password' unless options[:ssl_client_password] result[:p12_password] = options[:ssl_client_password] else raise ArgumentError, 'SSL client auth enabled but no client keypair specified' end # Make sure there's not a password required, or that if the password is given, it is correct. # This will raise OpenSSL::PKey::RSAError if the key needs a password. if result[:pem] && options[:ssl_client_password] result[:pem_password] = options[:ssl_client_password] _trash = OpenSSL::PKey::RSA.new(result[:pem], result[:pem_password]) elsif result[:pem] # Ruby 2.4 requires a minimum password length of 4. If no password is needed for # the certificate, the specified password here is effectively ignored. # We do not want to wait on STDIN, so a password-protected certificate without a # password will cause this to raise an error. There are two checks here, to exclude # an edge case where somebody did actually put '1234' as their password. _trash = OpenSSL::PKey::RSA.new(result[:pem], '1234') _trash = OpenSSL::PKey::RSA.new(result[:pem], '5678') end end # Return result result end # Determine, based on options, whether SSL client certificates need to be used. # The order of precedence is: # - If options[:ssl_client_auth] is not nil, return it # - If (key and cert) or PEM or PKCS12 are set, return true # - Else return false # @return [Boolean] see description def self.client_auth?(options) return options[:ssl_client_auth] unless options[:ssl_client_auth].nil? return true if options[:ssl_client_cert].is_a?(String) && options[:ssl_client_key].is_a?(String) return true if options[:ssl_client_pem].is_a?(String) return true if options[:ssl_client_p12].is_a?(String) false end # Determine, based on options, whether SSL certificates should be verified. # The order of precedence is: # - If options[:ssl_verify] is not nil, return it # - If options[:ssl_ca] is defined, return true # - Else return false # @return [Boolean] see description def self.ssl_verify?(options) return options[:ssl_verify] unless options[:ssl_verify].nil? options[:ssl_ca].is_a?(String) end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/bootstrap.rb0000644000004100000410000000450313413744025023300 0ustar www-datawww-data# frozen_string_literal: true require 'open3' require 'shellwords' module OctocatalogDiff # Contains bootstrap function to bootstrap a checked-out Puppet environment. class Bootstrap # Bootstrap a checked-out Puppet environment # @param options [Hash] Options hash: # :path [String] => Directory to bootstrap # :bootstrap_script [String] => Bootstrap script, relative to directory # @return [Hash] => [Integer] :status_code, [String] :output def self.bootstrap(options = {}) # Options validation unless options[:path].is_a?(String) raise ArgumentError, 'Directory to bootstrap (:path) undefined or wrong data type' end unless File.directory?(options[:path]) raise Errno::ENOENT, "Non-existent directory '#{options[:path]}' in bootstrap" end unless options[:bootstrap_script].is_a?(String) raise ArgumentError, 'Bootstrap script (:bootstrap_script) undefined or wrong data type' end bootstrap_script = File.join(options[:path], options[:bootstrap_script]) unless File.file?(bootstrap_script) raise Errno::ENOENT, "Non-existent bootstrap script '#{options[:bootstrap_script]}'" end # 'env' sets up the environment variables that will be passed to the script. # This is a clean environment. env = { 'PWD' => options[:path], 'HOME' => ENV['HOME'], 'PATH' => ENV['PATH'], 'BASEDIR' => options[:basedir] } env.merge!(options[:bootstrap_environment]) if options[:bootstrap_environment].is_a?(Hash) # 'opts' are options passed to the Open3.capture2e command which are described # here: http://ruby-doc.org/stdlib-2.1.0/libdoc/open3/rdoc/Open3.html#method-c-capture2e # Setting { :chdir => dir } means the shelled-out script will execute in the specified directory # This natively avoids the need to shell out to 'cd $dir && script/bootstrap' opts = { chdir: options[:path], unsetenv_others: true } # Actually execute the command and capture the output (combined stdout and stderr). cmd = [bootstrap_script, options[:bootstrap_args]].compact.map { |x| Shellwords.escape(x) }.join(' ') output, status = Open3.capture2e(env, cmd, opts) { status_code: status.exitstatus, output: output } end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/0000755000004100000410000000000013413744025023254 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/display/0000755000004100000410000000000013413744025024721 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/display/json.rb0000644000004100000410000000200313413744025026212 0ustar www-datawww-data# frozen_string_literal: true require_relative '../display' require 'json' module OctocatalogDiff module CatalogDiff class Display # Display the output from a diff in JSON format. This is the new format, used in octocatalog-diff # 1.x, where each diff is represented by an hash with named keys. class Json < OctocatalogDiff::CatalogDiff::Display # Generate JSON representation of the 'diff' suitable for further analysis. # @param diff [Array] The diff which *must* be in this format # @param options [Hash] Options which are: # - :header => [String] Header to print; no header is printed if not specified # @param _logger [Logger] Not used here def self.generate(diff, options = {}, _logger = nil) result = { 'diff' => diff.map(&:to_h_with_string_keys) } result['header'] = options[:header] unless options[:header].nil? result.to_json end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/display/legacy_json.rb0000644000004100000410000000175313413744025027551 0ustar www-datawww-data# frozen_string_literal: true require_relative '../display' require 'json' module OctocatalogDiff module CatalogDiff class Display # Display the output from a diff in JSON format. This is the legacy format, used in octocatalog-diff # 0.x, where each diff is represented by an array. class LegacyJson < OctocatalogDiff::CatalogDiff::Display # Generate JSON representation of the 'diff' suitable for further analysis. # @param diff [Array] The diff which *must* be in this format # @param options [Hash] Options which are: # - :header => [String] Header to print; no header is printed if not specified # @param _logger [Logger] Not used here def self.generate(diff, options = {}, _logger = nil) result = { 'diff' => diff.map(&:raw) } result['header'] = options[:header] unless options[:header].nil? result.to_json end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/display/text.rb0000644000004100000410000006053713413744025026245 0ustar www-datawww-data# frozen_string_literal: true require_relative '../display' require_relative '../../util/colored' require_relative '../../util/util' require 'diffy' require 'json' module OctocatalogDiff module CatalogDiff class Display # Display the output from a diff in text format. Uses the 'diffy' gem to provide diffs in # blocks of text. Formats results in a logical Puppet output. class Text < OctocatalogDiff::CatalogDiff::Display SEPARATOR = '*******************************************'.freeze # Generate the text representation of the 'diff' suitable for rendering in a console or log. # @param diff [Array] The diff which *must* be in this format # @param options_in [Hash] Options which are: # - :color => [Boolean] True or false, whether to use color codes # - :header => [String] Header to print; no header is printed if not specified # - :display_source_file_line [Boolean] True or false, print filename and line (if known) # - :display_detail_add [Boolean] If true, print details of any added resources # @param logger [Logger] Logger object # @return [Array] Results def self.generate(diff, options_in = {}, logger = nil) # Empty? return [] if diff.empty? # We may modify this for temporary local use, but don't want to pass these changes # back to the rest of the program. options = options_in.dup # Enable color support if requested... String.colors_enabled = options.fetch(:color, true) previous_diffy_default_format = Diffy::Diff.default_format Diffy::Diff.default_format = options.fetch(:color, true) ? :color : :text # Strip out differences or update display where string matches but data type differs. # For example, 28 (the integer) and "28" (the string) have identical string # representations, but are different data types. Same for nil vs. "". adjust_for_display_datatype_changes(diff, options[:display_datatype_changes], logger) # Call the utility method to sort changes into their respective types only_in_new, only_in_old, changed = parse_diff_array_into_categorized_hashes(diff) sorted_list = only_in_old.keys | only_in_new.keys | changed.keys sorted_list.sort! unless logger.nil? logger.debug "Added resources: #{only_in_new.keys.count}" logger.debug "Removed resources: #{only_in_old.keys.count}" logger.debug "Changed resources: #{changed.keys.count}" end # Run through the list to build the result result = [] sorted_list.each do |item| # Print the header if needed unless options[:header].nil? result << options[:header] unless options[:header].empty? result << SEPARATOR options[:header] = nil end # A removed item appears only in the old hash. if only_in_old.key?(item) result.concat display_removed_item( item: item, old_loc: only_in_old[item][:loc], options: options, logger: logger ) # An added item appears only in the new hash. elsif only_in_new.key?(item) result.concat display_added_item( item: item, new_loc: only_in_new[item][:loc], diff: only_in_new[item][:diff], options: options, logger: logger ) # A change can appear either in the change hash, the nested hash, or both. # Therefore, changes and nested changes are combined for display. elsif changed.key?(item) result.concat display_changed_or_nested_item( item: item, old_loc: changed[item][:old_loc], new_loc: changed[item][:new_loc], diff: changed[item][:diff], options: options, logger: logger ) # An unrecognized change throws an error. This indicates a bug. else # :nocov: raise "BUG (please report): Unable to determine diff type of item: #{item.inspect}" # :nocov: end result << SEPARATOR end # Reset the global color-related flags String.colors_enabled = false Diffy::Diff.default_format = previous_diffy_default_format # The end result end # Display a changed or nested item # @param item [String] Item (type+title) that has changed # @param old_loc [Hash] File and line number of location in "from" catalog # @param new_loc [Hash] File and line number of location in "to" catalog # @param diff [Hash] Difference hash # @param options [Hash] Display options # @param logger [Logger] Logger object # @return [Array] Lines of text def self.display_changed_or_nested_item(opts = {}) item = opts.fetch(:item) old_loc = opts.fetch(:old_loc) new_loc = opts.fetch(:new_loc) diff = opts.fetch(:diff) options = opts.fetch(:options) logger = opts[:logger] result = [] info_hash = { item: item, result: result, old_loc: old_loc, new_loc: new_loc, options: options, logger: logger } add_source_file_line_info(info_hash) result << " #{item} =>" diff.keys.sort.each { |key| result.concat hash_diff(diff[key], 1, key, true) } result end # Display a removed item # @param item [String] Item (type+title) that has changed # @param old_loc [Hash] File and line number of location in "from" catalog # @param options [Hash] Display options # @param logger [Logger] Logger object # @return [Array] Lines of text def self.display_removed_item(opts = {}) item = opts.fetch(:item) old_loc = opts.fetch(:old_loc) options = opts.fetch(:options) logger = opts[:logger] result = [] add_source_file_line_info(item: item, result: result, old_loc: old_loc, options: options, logger: logger) result << "- #{item}".red end # Display an added item # @param item [String] Item (type+title) that has changed # @param new_loc [Hash] File and line number of location in "to" catalog # @param diff [Hash] Difference hash # @param options [Hash] Display options # @param logger [Logger] Logger object # @return [Array] Lines of text def self.display_added_item(opts = {}) item = opts.fetch(:item) new_loc = opts.fetch(:new_loc) diff = opts.fetch(:diff) options = opts.fetch(:options) logger = opts[:logger] result = [] add_source_file_line_info(item: item, result: result, new_loc: new_loc, options: options, logger: logger) if options[:display_detail_add] && diff.key?('parameters') limit = options.fetch(:truncate_details, true) ? 80 : nil result << "+ #{item} =>".green result << ' parameters =>'.green result.concat( diff_two_hashes_with_diffy( depth: 1, hash2: Hash[diff['parameters'].sort], # Should work with somewhat older rubies too limit: limit, strip_diff: true ).map(&:green) ) else result << "+ #{item}".green if diff.key?('parameters') && logger && !options[:display_detail_add_notice_printed] logger.info 'Note: you can use --display-detail-add to view details of added resources' options[:display_detail_add_notice_printed] = true end end result end # Generate info about the source of the change. Pass in parameters as a hash with indicated names. # @param item [Hash] Item that is added/removed/changed # @param result [Array] Result array (modified by this method) # @param old_loc [Hash] Old location hash { file => ..., line => ... } # @param new_loc [Hash] New location hash { file => ..., line => ... } # @param options [Hash] Options hash # @param logger [Logger] Logger object def self.add_source_file_line_info(opts = {}) item = opts.fetch(:item) result = opts.fetch(:result) old_loc = opts[:old_loc] new_loc = opts[:new_loc] options = opts.fetch(:options, {}) logger = opts[:logger] # Initialize any currently undefined settings empty_hash = { 'file' => nil, 'line' => nil } old_loc ||= empty_hash new_loc ||= empty_hash return if old_loc == empty_hash && new_loc == empty_hash # Convert old_loc and new_loc to strings old_loc_string = loc_string(old_loc, options[:compilation_from_dir], logger) new_loc_string = loc_string(new_loc, options[:compilation_to_dir], logger) # Debug log information and build up local_result with printable changes local_result = [] if old_loc == new_loc || new_loc == empty_hash || old_loc_string == new_loc_string logger.debug "#{item} @ #{old_loc_string || 'nil'}" if logger local_result << " #{old_loc_string}".cyan unless old_loc_string.nil? elsif old_loc == empty_hash logger.debug "#{item} @ #{new_loc_string || 'nil'}" if logger local_result << " #{new_loc_string}".cyan unless new_loc_string.nil? else logger.debug "#{item} -@ #{old_loc_string} +@ #{new_loc_string}" if logger local_result << "- #{old_loc_string}".cyan local_result << "+ #{new_loc_string}".cyan end # Only modify result if option to display source file and line is enabled result.concat local_result if options[:display_source_file_line] end # Convert { file => ..., line => ... } to displayable string # @param loc [Hash] file => ..., line => ... hash # @param compilation_dir [String] Compilation directory # @param logger [Logger] Logger object # @return [String] Location string def self.loc_string(loc, compilation_dir, logger) return nil if loc.nil? || !loc.is_a?(Hash) || loc['file'].nil? || loc['line'].nil? result = "#{loc['file']}:#{loc['line']}" if compilation_dir rex = Regexp.new('^' + Regexp.escape(compilation_dir + '/')) result_new = result.sub(rex, '') if result_new != result logger.debug "Removed compilation directory in #{result} -> #{result_new}" if logger result = result_new end end result end # Get the diff of two long strings. Call the 'diffy' gem for this. # @param string1 [String] First string (-) # @param string2 [String] Second string (+) # @param depth [Integer] Depth, for correct indentation # @return Array Displayable result def self.diff_two_strings_with_diffy(string1, string2, depth) # Single line strings? if single_lines?(string1, string2) string1, string2 = add_trailing_newlines(string1, string2) diff = Diffy::Diff.new(string1, string2, context: 2, include_diff_info: false).to_s.split("\n") return diff.map { |x| left_pad(2 * depth + 2, make_trailing_whitespace_visible(adjust_position_of_plus_minus(x))) } end # Multiple line strings string1, string2 = add_trailing_newlines(string1, string2) diff = Diffy::Diff.new(string1, string2, context: 2, include_diff_info: true).to_s.split("\n") diff.shift # Remove first line of diff info (filename that makes no sense) diff.shift # Remove second line of diff info (filename that makes no sense) diff.map { |x| left_pad(2 * depth + 2, make_trailing_whitespace_visible(x)) } end # Determine if two incoming strings are single lines. Returns true if both # incoming strings are single lines, false otherwise. # @param string_1 [String] First string # @param string_2 [String] Second string # @return [Boolean] Whether both incoming strings are single lines def self.single_lines?(string_1, string_2) string_1.strip !~ /\n/ && string_2.strip !~ /\n/ end # Add "\n" to the end of both strings, only if both strings are lacking it. # This prevents "\\ No newline at end of file" for single string comparison. # @param string_1 [String] First string # @param string_2 [String] Second string # @return [Array] Adjusted string_1, string_2 def self.add_trailing_newlines(string_1, string_2) return [string_1, string_2] unless string_1 !~ /\n\Z/ && string_2 !~ /\n\Z/ [string_1 + "\n", string_2 + "\n"] end # Adjust the space after of the `-` / `+` in the diff for single line diffs. # Diffy prints diffs with no space between the `-` / `+` in the text, but for # single lines it's easier to read with that space added. # @param string_in [String] Input string, which is a line of a diff from diffy # @return [String] Modified string def self.adjust_position_of_plus_minus(string_in) string_in.sub(/\A(\e\[\d+m)?([\-\+])/, '\1\2 ') end # Convert trailing whitespace to underscore for display purposes. Also convert special # whitespace (\r, \n, \t, ...) to character representation. # @param string_in [String] Input string, which might contain trailing whitespace # @return [String] Modified string def self.make_trailing_whitespace_visible(string_in) return string_in unless string_in =~ /\A((?:.|\n)*?)(\s+)(\e\[0m)?\Z/ beginning = Regexp.last_match(1) trailing_space = Regexp.last_match(2) end_escape = Regexp.last_match(3) # Trailing space adjustment for line endings trailing_space.gsub! "\n", '\n' trailing_space.gsub! "\r", '\r' trailing_space.gsub! "\t", '\t' trailing_space.gsub! "\f", '\f' trailing_space.tr! ' ', '_' [beginning, trailing_space, end_escape].join('') end # Get the diff of two hashes. Call the 'diffy' gem for this. # @param hash1 [Hash] First hash (-) # @param hash1 [Hash] Second hash (+) # @param depth [Integer] Depth, for correct indentation # @param limit [Integer] Maximum string length # @param strip_diff [Boolean] Strip leading +/-/" " # @return [Array] Displayable result def self.diff_two_hashes_with_diffy(opts = {}) depth = opts.fetch(:depth, 0) hash1 = opts.fetch(:hash1, {}) hash2 = opts.fetch(:hash2, {}) limit = opts[:limit] strip_diff = opts.fetch(:strip_diff, false) # Special case: addition only, no truncation return addition_only_no_truncation(depth, hash2) if hash1 == {} && limit.nil? json_old = stringify_for_diffy(hash1) json_new = stringify_for_diffy(hash2) # If stripping the diff, we need to make sure diffy does not colorize the output, so that # there are not color codes in the output to deal with. diff = if strip_diff Diffy::Diff.new(json_old, json_new, context: 0).to_s(:text).split("\n") else Diffy::Diff.new(json_old, json_new, context: 0).to_s.split("\n") end raise "Diffy diff empty for string: #{json_old}" if diff.empty? # This is the array that is returned diff.map do |x| x = x[2..-1] if strip_diff # Drop first 2 characters: '+ ', '- ', or ' ' truncate_string(left_pad(2 * depth + 2, x), limit) end end # Special case: addition only, no truncation # @param depth [Integer] Depth, for correct indentation # @param hash [Hash] Added object # @return [Array] Displayable result def self.addition_only_no_truncation(depth, hash) result = [] # Single line strings hash.keys.sort.map do |key| next if hash[key] =~ /\n/ result << left_pad(2 * depth + 4, [key.inspect, ': ', hash[key].inspect].join('')).green end # Multi-line strings hash.keys.sort.map do |key| next if hash[key] !~ /\n/ result << left_pad(2 * depth + 4, [key.inspect, ': >>>'].join('')).green result.concat hash[key].split(/\n/).map(&:green) result << '<<<'.green end result end # Limit length of a string # @param str [String] String # @param limit [Integer] Limit (0=unlimited) # @return [String] Truncated string def self.truncate_string(str, limit) return str if limit.nil? || str.length <= limit "#{str[0..limit]}..." end # Get the diff between two hashes. This is recursive-aware. # @param obj [diff object] diff object # @param depth [Integer] Depth of nesting, used for indentation # @return Array Printable diff outputs def self.hash_diff(obj, depth, key_in, nested = false) result = [] result << left_pad(2 * depth, " #{key_in} =>") if obj.key?(:old) && obj.key?(:new) if nested && obj[:old].is_a?(Hash) && obj[:new].is_a?(Hash) # Nested hashes will be stringified and then use 'diffy' result.concat diff_two_hashes_with_diffy(depth: depth, hash1: obj[:old], hash2: obj[:new]) elsif obj[:old].is_a?(String) && obj[:new].is_a?(String) # Strings will use 'diffy' to mimic the output seen when using # "diff" on the command line. result.concat diff_two_strings_with_diffy(obj[:old], obj[:new], depth) else # Stuff we don't recognize will be converted to a string and printed # with '+' and '-' unless the object resolves to an empty string. result.concat diff_at_depth(depth, obj[:old], obj[:new]) end else obj.keys.sort.each { |key| result.concat hash_diff(obj[key], 1 + depth, key, nested) } end result end # Get the diff between two arbitrary objects # @param depth [Integer] Depth of nesting, used for indentation # @param old_obj [?] Old object # @param new_obj [?] New object # @return Array Diff output def self.diff_at_depth(depth, old_obj, new_obj) old_s = old_obj.to_s new_s = new_obj.to_s result = [] result << left_pad(2 * depth + 2, "- #{old_s}").red unless old_s == '' result << left_pad(2 * depth + 2, "+ #{new_s}").green unless new_s == '' result end # Utility Method! # Indent a given text string with a certain number of spaces # @param spaces [Integer] Number of spaces # @param text [String] Text def self.left_pad(spaces, text = '') [' ' * spaces, text].join('') end # Utility Method! # Harmonize equivalent class names for comparison purposes. # @param class_name [String] Class name as input # @return [String] Class name as output def self.class_name_for_diffy(class_name) return 'Integer' if class_name == 'Fixnum' class_name end # Utility Method! # Given an arbitrary object, convert it into a string for use by 'diffy'. # This basically exists so we can do something prettier than just calling .inspect or .to_s # on object types we anticipate seeing, while not failing entirely on other object types. # @param obj [?] Object to be stringified # @return [String] String representation of object for diffy def self.stringify_for_diffy(obj) return JSON.pretty_generate(obj) if OctocatalogDiff::Util::Util.object_is_any_of?(obj, [Hash, Array]) return '""' if obj.is_a?(String) && obj == '' return obj if OctocatalogDiff::Util::Util.object_is_any_of?(obj, [String, Fixnum, Integer, Float]) "#{class_name_for_diffy(obj.class)}: #{obj.inspect}" end # Utility Method! # Implement the --display-datatype-changes option by: # - Removing string-equivalent differences when option == false # - Updating display of string-equivalent differences when option == true # @param diff [Array] Difference array # @param option [Boolean] Selected behavior; see description # @param logger [Logger] Logger object def self.adjust_for_display_datatype_changes(diff, option, logger = nil) diff.map! do |diff_obj| if diff_obj[0] == '+' || diff_obj[0] == '-' diff_obj[2] = 'undef' if diff_obj[2].nil? diff_obj else x2, x3 = _adjust_for_display_datatype(diff_obj[2], diff_obj[3], option, logger) if x2.nil? && x3.nil? # Delete this! Return nil and compact! will get rid of them. msg = "Adjust display for #{diff_obj[1].gsub(/\f/, '::')}: " \ "#{diff_obj[2].inspect} != #{diff_obj[3].inspect} DELETED" logger.debug(msg) if logger nil elsif x2 == diff_obj[2] && x3 == diff_obj[3] # Neither object changed diff_obj else # Adjust the display and return modified object msg = "Adjust display for #{diff_obj[1].gsub(/\f/, '::')}: " \ "old=#{x2.inspect} new=#{x3.inspect} "\ "(extra debugging: #{diff_obj[2].inspect} -> #{x2}; "\ "#{diff_obj[3].inspect} -> #{x3})" logger.debug(msg) if logger diff_obj[2] = x2 diff_obj[3] = x3 diff_obj end end end diff.compact! end # Utility Method! # Called by adjust_for_display_datatype_changes to compare an old value # to a new value and adjust as appropriate. # @param obj1 [?] First object # @param obj2 [?] Second object # @param option [Boolean] Selected behavior; see adjust_for_display_datatype_changes # @return [ or ] Updated values of objects def self._adjust_for_display_datatype(obj1, obj2, option, logger) # If not string-equal, return to leave untouched return [obj1, obj2] unless obj1.to_s == obj2.to_s # Delete if option to display these is false return [nil, nil] unless option # Delete if both objects are nil return [nil, nil] if obj1.nil? && obj2.nil? # If one is nil and the other is the empty string... return ['undef', '""'] if obj1.nil? return ['""', 'undef'] if obj2.nil? # If one is an integer and the other is a string return [obj1, "\"#{obj2}\""] if obj1.is_a?(Integer) && obj2.is_a?(String) return ["\"#{obj1}\"", obj2] if obj1.is_a?(String) && obj2.is_a?(Integer) # True and false return [obj1, "\"#{obj2}\""] if obj1.is_a?(TrueClass) && obj2.is_a?(String) return [obj1, "\"#{obj2}\""] if obj1.is_a?(FalseClass) && obj2.is_a?(String) return ["\"#{obj1}\"", obj2] if obj1.is_a?(String) && obj2.is_a?(TrueClass) return ["\"#{obj1}\"", obj2] if obj1.is_a?(String) && obj2.is_a?(FalseClass) # Unhandled case - warn about it and then return inputs untouched # Note: If you encounter this, please report it so we can add a handler. # :nocov: msg = "In _adjust_for_display_datatype, objects '#{obj1.inspect}' (#{obj1.class}) and"\ " '#{obj2.inspect}' (#{obj2.class}) have identical string representations but"\ ' formatting is not implemented to update display.' logger.warn(msg) if logger [obj1, obj2] # :nocov: end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/display.rb0000644000004100000410000001531013413744025025246 0ustar www-datawww-data# frozen_string_literal: true require_relative '../api/v1/diff' require_relative 'differ' require_relative 'display/json' require_relative 'display/legacy_json' require_relative 'display/text' module OctocatalogDiff module CatalogDiff # Prepare a display of the results from a catalog-diff. Intended that this will contain utility # methods but call out to a OctocatalogDiff::CatalogDiff::Display:: class to display in # the desired format. class Display # Display the diff in some specified format. # @param diff_in [OctocatalogDiff::CatalogDiff::Differ | Array] Diff to display # @param options [Hash] Consisting of: # - :header [String] => Header (can be :default to construct header) # - :display_source_file_line [Boolean] => Display manifest filename and line number where declared # - :compilation_from_dir [String] => Directory where 'from' catalog was compiled # - :compilation_to_dir [String] => Directory where 'to' catalog was compiled # - :display_detail_add [Boolean] => Set true to display parameters of newly added resources # @param logger [Logger] Logger object # @return [String] Text output for provided diff def self.output(diff_in, options = {}, logger = nil) diff_x = diff_in.is_a?(OctocatalogDiff::CatalogDiff::Differ) ? diff_in.diff : diff_in raise ArgumentError, "text_output requires Array; passed in #{diff_in.class}" unless diff_x.is_a?(Array) diff = diff_x.map { |x| OctocatalogDiff::API::V1::Diff.factory(x) } # req_format means 'requested format' because 'format' has a built-in meaning to Ruby req_format = options.fetch(:format, :color_text) # Options hash to pass to display method opts = {} opts[:header] = header(options) opts[:display_source_file_line] = options.fetch(:display_source_file_line, false) opts[:compilation_from_dir] = options[:compilation_from_dir] || nil opts[:compilation_to_dir] = options[:compilation_to_dir] || nil opts[:display_detail_add] = options.fetch(:display_detail_add, false) opts[:truncate_details] = options.fetch(:truncate_details, true) opts[:display_datatype_changes] = options.fetch(:display_datatype_changes, false) # Call appropriate display method case req_format when :json logger.debug 'Generating JSON output' if logger OctocatalogDiff::CatalogDiff::Display::Json.generate(diff, opts, logger) when :legacy_json logger.debug 'Generating Legacy JSON output' if logger OctocatalogDiff::CatalogDiff::Display::LegacyJson.generate(diff, opts, logger) when :text logger.debug 'Generating non-colored text output' if logger OctocatalogDiff::CatalogDiff::Display::Text.generate(diff, opts.merge(color: false), logger) when :color_text logger.debug 'Generating colored text output' if logger OctocatalogDiff::CatalogDiff::Display::Text.generate(diff, opts.merge(color: true), logger) else raise ArgumentError, "Unrecognized text format '#{req_format}'" end end # Utility method! # Construct the header for diffs # Default is diff / / # @param opts [Hash] Options hash from CLI # @return [String] Header in indicated format def self.header(opts) return nil if opts[:no_header] return opts[:header] unless opts[:header] == :default node = opts.fetch(:node, 'node') from_br = opts.fetch(:from_env, 'a') to_br = opts.fetch(:to_env, 'b') from_br = 'current' if from_br == '.' to_br = 'current' if to_br == '.' "diff #{from_br}/#{node} #{to_br}/#{node}" end # Utility method! # Go through the 'diff' array, filtering out ignored items and classifying each change # as an addition (+), subtraction (-), change (~), or nested change (!). This creates # hashes for each type of change that are consumed later for ordering purposes. # @param diff [Array] The diff which *must* be in this format # @return [Array a } } # Assign to appropriate variable diff = changed.key?(key) ? changed[key][:diff] : {} simple_deep_merge!(diff, result) changed[key] = { diff: diff, old_loc: diff_obj[4], new_loc: diff_obj[5] } else raise "Unrecognized diff symbol '#{diff_obj[0]}' in #{diff_obj.inspect}" end end [only_in_new, only_in_old, changed] end # Utility Method! # Deep merge two hashes. (The 'deep_merge' gem seems to de-duplicate arrays so this is a reinvention # of the wheel, but a simpler wheel that does just exactly what we need.) # @param hash1 [Hash] First object # @param hash2 [Hash] Second object def self.simple_deep_merge!(hash1, hash2) raise ArgumentError, 'First argument to simple_deep_merge must be a hash' unless hash1.is_a?(Hash) raise ArgumentError, 'Second argument to simple_deep_merge must be a hash' unless hash2.is_a?(Hash) hash2.each do |k, v| if v.is_a?(Hash) && hash1[k].is_a?(Hash) # We can only merge a hash with a hash. If hash1[k] is something other than a hash, say for example # a string, then the merging is NOT invoked and hash1[k] gets directly overwritten in the `else` clause. # Also if hash1[k] is nil, it falls through to the `else` clause where it gets set directly to the result # hash without needless iterations. simple_deep_merge!(hash1[k], v) else hash1[k] = v end end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/filter.rb0000644000004100000410000000665013413744025025075 0ustar www-datawww-datarequire_relative '../api/v1/diff' require_relative 'filter/absent_file' require_relative 'filter/compilation_dir' require_relative 'filter/json' require_relative 'filter/single_item_array' require_relative 'filter/yaml' require 'stringio' module OctocatalogDiff module CatalogDiff # Filtering of diffs, and parent class for inheritance. class Filter attr_accessor :logger # List the available filters here (by class name) for use in the validator method. AVAILABLE_FILTERS = %w(AbsentFile CompilationDir JSON SingleItemArray YAML).freeze # Public: Determine whether a particular filter exists. This can be used to validate # a user-submitted filter. # @param filter_name [String] Proposed filter name # @return [Boolean] True if filter is valid; false otherwise def self.filter?(filter_name) AVAILABLE_FILTERS.include?(filter_name) end # Public: Assert that a filter exists, and raise an error if it does not. # @param filter_name [String] Proposed filter name def self.assert_that_filter_exists(filter_name) return if filter?(filter_name) raise ArgumentError, "The filter #{filter_name} is not valid" end # Public: Apply multiple filters by repeatedly calling the `filter` method for each # filter in an array. This method returns nothing. # # @param result [Array] Difference array (mutated) # @param filter_names [Array] Filters to run # @param options [Hash] Options for each filter def self.apply_filters(result, filter_names, options = {}) return unless filter_names.is_a?(Array) filter_names.each { |x| filter(result, x, options || {}) } end # Public: Perform a filter on `result` using the specified filter class. # This mutates `result` by removing items that are ignored. This method # returns nothing. # # @param result [Array] Difference array (mutated) # @param filter_class_name [String] Filter class name (from `filter` subdirectory) # @param options [Hash] Additional options (optional) to pass to filtered? method def self.filter(result, filter_class_name, options = {}) assert_that_filter_exists(filter_class_name) filter_class_name = [name.to_s, filter_class_name].join('::') # Need to convert each of the results array to the OctocatalogDiff::API::V1::Diff object, if # it isn't already. The comparison is done on that array which is then applied back to the # original array. result_hash = {} result.each { |x| result_hash[x] = OctocatalogDiff::API::V1::Diff.factory(x) } obj = Kernel.const_get(filter_class_name).new(result_hash.values, options[:logger]) result.reject! { |item| obj.filtered?(result_hash[item], options) } end # Inherited: Constructor. Some filters require working on the entire data set and # will override this method to perform some pre-processing for efficiency. This also # sets up the logger object. def initialize(_diff_array = [], logger = Logger.new(StringIO.new)) @logger = logger end # Inherited: Construct a default `filtered?` method for the subclass via inheritance. # Each subclass must implement this method, so the default method errors. def filtered?(_item, _options = {}) raise "No `filtered?` method is implemented in #{self.class}" end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/filter/0000755000004100000410000000000013413744025024541 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/filter/json.rb0000644000004100000410000000320313413744025026035 0ustar www-datawww-data# frozen_string_literal: true require_relative '../filter' require 'json' module OctocatalogDiff module CatalogDiff class Filter # Filter based on equivalence of JSON objects for file resources with named extensions. class JSON < OctocatalogDiff::CatalogDiff::Filter # Public: Actually do the comparison of JSON objects for appropriate resources. # Return true if the JSON objects are known to be equivalent. Return false if they # are not equivalent, or if equivalence cannot be determined. # # @param diff_in [OctocatalogDiff::API::V1::Diff] Difference # @param _options [Hash] Additional options (there are none for this filter) # @return [Boolean] true if this difference is a JSON file with identical objects, false otherwise def filtered?(diff, _options = {}) # Skip additions or removals - focus only on changes return false unless diff.change? # Make sure we are comparing file content for a file ending in .json extension return false unless diff.type == 'File' && diff.structure == %w(parameters content) return false unless diff.title =~ /\.json\z/i # Attempt to convert the old value and new value into JSON objects. Assuming # that doesn't error out, the return value is whether or not they're equal. obj_old = ::JSON.parse(diff.old_value) obj_new = ::JSON.parse(diff.new_value) obj_old == obj_new rescue # Rescue everything - if something failed, we aren't sure what's going on, so we'll return false. false end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/filter/single_item_array.rb0000644000004100000410000000331013413744025030560 0ustar www-datawww-data# frozen_string_literal: true require_relative '../filter' module OctocatalogDiff module CatalogDiff class Filter # Filter out changes in parameters when one catalog has a parameter that's an object and # the other catalog has that same parameter as an array containing the same object. # For example, under this filter, the following is not a change: # catalog1: notify => "Service[foo]" # catalog2: notify => ["Service[foo]"] class SingleItemArray < OctocatalogDiff::CatalogDiff::Filter # Public: Implement the filter for single-item arrays whose item exactly matches the # item that's not in an array in the other catalog. # # @param diff [OctocatalogDiff::API::V1::Diff] Difference # @param _options [Hash] Additional options (there are none for this filter) # @return [Boolean] true if this should be filtered out, false otherwise def filtered?(diff, _options = {}) # Skip additions or removals - focus only on changes return false unless diff.change? old_value = diff.old_value new_value = diff.new_value # Skip unless there is a single-item array under consideration return false unless (old_value.is_a?(Array) && old_value.size == 1) || (new_value.is_a?(Array) && new_value.size == 1) # Skip if both the old value and new value are arrays return false if old_value.is_a?(Array) && new_value.is_a?(Array) # Do comparison if old_value.is_a?(Array) old_value.first == new_value else new_value.first == old_value end end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/filter/absent_file.rb0000644000004100000410000000520013413744025027336 0ustar www-datawww-data# frozen_string_literal: true require_relative '../filter' require 'set' module OctocatalogDiff module CatalogDiff class Filter # Filter out changes in parameters when the "to" resource has ensure => absent. class AbsentFile < OctocatalogDiff::CatalogDiff::Filter KEEP_ATTRIBUTES = (Set.new %w(ensure backup force provider)).freeze # Constructor: Since this filter requires knowledge of the entire array of diffs, # override the inherited method to store those diffs in an instance variable. # @param diffs [Array] Difference array # @param _logger [?] Ignored def initialize(diffs, _logger = nil) @diffs = diffs @results = nil end # Public: If a file has ensure => absent, there are certain parameters that don't # matter anymore. Filter out any such parameters from the result array. # Return true if the difference is in a resource where `ensure => absent` has been # declared. Return false if they this is not the case. # # @param diff [OctocatalogDiff::API::V1::Diff] Difference # @param _options [Hash] Additional options (there are none for this filter) # @return [Boolean] true if this difference is a YAML file with identical objects, false otherwise def filtered?(diff, _options = {}) build_results if @results.nil? @results.include?(diff) end private # Private: The first time `.filtered?` is called, build up the cache of results. # Returns nothing, but populates @results. def build_results # Which files can we ignore? @files_to_ignore = Set.new @diffs.each do |diff| next unless diff.change? && diff.type == 'File' && diff.structure == %w(parameters ensure) next unless ['absent', 'false', false].include?(diff.new_value) @files_to_ignore.add diff.title end # Based on that, which diffs can we ignore? @results = Set.new @diffs.reject { |diff| keep_diff?(diff) } end # Private: Determine whether to keep a particular diff. # @param diff [OctocatalogDiff::API::V1::Diff] Difference under consideration # @return [Boolean] true = keep, false = discard def keep_diff?(diff) return true unless diff.change? && diff.type == 'File' && diff.structure.first == 'parameters' return true unless @files_to_ignore.include?(diff.title) return true if KEEP_ATTRIBUTES.include?(diff.structure.last) false end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/filter/yaml.rb0000644000004100000410000000321113413744025026025 0ustar www-datawww-data# frozen_string_literal: true require_relative '../filter' require 'yaml' module OctocatalogDiff module CatalogDiff class Filter # Filter based on equivalence of YAML objects for file resources with named extensions. class YAML < OctocatalogDiff::CatalogDiff::Filter # Public: Actually do the comparison of YAML objects for appropriate resources. # Return true if the YAML objects are known to be equivalent. Return false if they # are not equivalent, or if equivalence cannot be determined. # # @param diff_in [OctocatalogDiff::API::V1::Diff] Difference # @param _options [Hash] Additional options (there are none for this filter) # @return [Boolean] true if this difference is a YAML file with identical objects, false otherwise def filtered?(diff, _options = {}) # Skip additions or removals - focus only on changes return false unless diff.change? # Make sure we are comparing file content for a file ending in .yaml or .yml extension return false unless diff.type == 'File' && diff.structure == %w(parameters content) return false unless diff.title =~ /\.ya?ml\z/ # Attempt to convert the old value and new value into YAML objects. Assuming # that doesn't error out, the return value is whether or not they're equal. obj_old = ::YAML.load(diff.old_value) obj_new = ::YAML.load(diff.new_value) obj_old == obj_new rescue # Rescue everything - if something failed, we aren't sure what's going on, so we'll return false. false end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/filter/compilation_dir.rb0000644000004100000410000000646113413744025030251 0ustar www-datawww-data# frozen_string_literal: true require_relative '../filter' require_relative '../../util/util' module OctocatalogDiff module CatalogDiff class Filter # Filter out changes that are due to the catalog compilation directory. class CompilationDir < OctocatalogDiff::CatalogDiff::Filter # Public: Filter the diff if the change is due to the catalog compilation directory. # Determine this by obtaining the compiilation directory from each of the catalogs # (supplied via options) and checking the differences. If the only thing different # is the compilation directory, filter it out with a warning. # # @param diff [OctocatalogDiff::API::V1::Diff] Difference # @param options [Hash] Additional options: # :from_compilation_dir [String] Compilation directory for the "from" catalog # :to_compilation_dir [String] Compilation directory for the "to" catalog # @return [Boolean] true if this difference is a YAML file with identical objects, false otherwise def filtered?(diff, options = {}) return false unless options[:from_compilation_dir] && options[:to_compilation_dir] dir1 = options[:to_compilation_dir] dir1_rexp = Regexp.escape(dir1) dir2 = options[:from_compilation_dir] dir2_rexp = Regexp.escape(dir2) dir = Regexp.new("(?:#{dir1_rexp}|#{dir2_rexp})") # Check for added/removed resources where the title of the resource includes the compilation directory if (diff.addition? || diff.removal?) && diff.title.match(dir) message = "Resource #{diff.type}[#{diff.title}]" message += ' appears to depend on catalog compilation directory. Suppressed from results.' logger.warn message return true end # Check for a change where the difference in a parameter exactly corresponds to the difference in the # compilation directory. if diff.change? o = remove_compilation_dir(diff.old_value, dir2) n = remove_compilation_dir(diff.new_value, dir1) if o != diff.old_value || n != diff.new_value message = "Resource key #{diff.type}[#{diff.title}] #{diff.structure.join(' => ')}" message += ' may depend on catalog compilation directory, but there may be differences.' message += ' This is included in results for now, but please verify.' @logger.warn message end if o == n message = "Resource key #{diff.type}[#{diff.title}] #{diff.structure.join(' => ')}" message += ' appears to depend on catalog compilation directory. Suppressed from results.' @logger.warn message return true end end false end def remove_compilation_dir(v, dir) value = OctocatalogDiff::Util::Util.deep_dup(v) traverse(value) do |e| e.gsub!(dir, '') if e.respond_to?(:gsub!) end value end def traverse(a) case a when Array a.map { |v| traverse(v, &Proc.new) } when Hash traverse(a.values, &Proc.new) else yield a end end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog-diff/differ.rb0000644000004100000410000007346513413744025025057 0ustar www-datawww-data# frozen_string_literal: true require 'diffy' require 'hashdiff' require 'json' require 'set' require 'stringio' require_relative '../catalog' require_relative '../errors' require_relative '../util/util' require_relative 'filter' module OctocatalogDiff module CatalogDiff # Calculate the difference between two Puppet catalogs. # ----------------------------------------------------- # It was necessary to write our own code for this, and not just use some existing gem, # for two main reasons: # # 1. There are things that we want to ignore when doing a Puppet catalog diff. For example # we want to ignore 'before' and 'require' parameters (because those affect the order of # operations only, not the end result) and we probably want to ignore 'tags' attributes # and all classes. No existing code (that I could find at least) was capable of allowing # you to skip stuff via arguments, without your own custom pre-processing. # # 2. When using the 'hashdiff' gem, there is no distinguishing between an addition of an entire # new key-value pair, or the addition of an element in a deeply nested array. By way of # further explanation, consider these two data structures: # # a = { 'foo' => 'bar', 'my_array' => [ 1, 2, 3 ] } # b = { 'foo' => 'bar', 'my_array' => [ 1, 2, 3, 4 ], 'another_key' => 'another_value' # # The hashdiff gem would report the differences between a and b to be: # + 4 # + another_key => another_value # # We want to distinguish (without a whole bunch of convoluted code) between these two situations. # One was a true addition (adding a key) while one was a change (adding element to array). This # distinction becomes even more important when considering top-level changes vs. changes to arrays # or hashes nested within the catalog. # # Therefore, the algorithm implemented here is as follows: # # 1. Pre-process the catalog JSON files to: # - Sort the 'tags' array, since the order of tags does not matter to Puppet # - Pull out additions of entire key-value pairs (above, 'another_key' => 'another_value') # # 2. Everything left consists of key-value pairs where the key exists in both old and new. Pass this # to the 'hashdiff' gem. # # 3. Filter any differences to remove attributes, types, or resources that have been explicitly ignored. # # 4. Reformat any '+' or '-' reported by hashdiff to be changes to the keys, rather than outright # additions. # # The heavy lifting is still handled by 'hashdiff' but we're pre-simplifying the input and post-processing # the output to make it easier to deal with later. class Differ # Constructor # @param catalog1_in [OctocatalogDiff::Catalog] First catalog to compare # @param catalog2_in [OctocatalogDiff::Catalog] Second catalog to compare def initialize(opts, catalog1_in, catalog2_in) @catalog1_raw = catalog1_in @catalog2_raw = catalog2_in @catalog1 = catalog_resources(catalog1_in, 'First catalog') @catalog2 = catalog_resources(catalog2_in, 'Second catalog') @logger = opts.fetch(:logger, Logger.new(StringIO.new)) @diff_result = nil @ignore = Set.new ignore(opts.fetch(:ignore, [])) @opts = opts end # Difference - calculates and then returns the diff of this objects # Each diff result is an array like this: # [ '+|-|~|!', Key name, Old object, New object ] # @return [Array] Results of the diff def diff @diff_result ||= catdiff end # Ignore - ignored items can be set by Type, Title, or Attribute; setting multiple in # a hash is interpreted as AND. The collection of all ignored items is interpreted as OR. # @param ignore [Hash] Ignore type/title/attr (can pass array also) # @return [OctocatalogDiff::CatalogDiff::Differ] This object, modified def ignore(ignores = []) ignore_array = ignores.is_a?(Array) ? ignores : [ignores] ignore_array.each do |item| raise ArgumentError, "Argument #{item.inspect} to ignore is not a hash" unless item.is_a?(Hash) unless item.key?(:type) || item.key?(:title) || item.key?(:attr) raise ArgumentError, "Argument #{item.inspect} does not contain :type, :title, or :attr" end item[:type] ||= '*' item[:title] ||= '*' item[:attr] ||= '*' # Support wildcards in title if item[:title].is_a?(String) && item[:title] != '*' && item[:title].include?('*') item[:title] = Regexp.new("\\A#{Regexp.escape(item[:title]).gsub('\*', '.*')}\\Z", 'i') end @ignore.add(item) end self end # Handle --ignore-tags option, the ability to tag resources within modules/manifests and # have catalog-diff ignore them. def ignore_tags return unless @opts[:ignore_tags].is_a?(Array) && @opts[:ignore_tags].any? # Go through the "to" catalog and identify any resources that have been tagged with one or more # specified "ignore tags." Add any such items to the ignore list. The 'to' catalog has the authoritative # list of dynamic ignores. @catalog2_raw.resources.each do |resource| next unless tagged_for_ignore?(resource) ignore(type: resource['type'], title: resource['title']) @logger.debug "Ignoring type='#{resource['type']}', title='#{resource['title']}' based on tag in to-catalog" end # Go through the "from" catalog and identify any resources that have been tagged with one or more # specified "ignore tags." Only mark the resources for ignoring if they do not appear in the 'to' # catalog, thereby allowing the 'to' catalog to be the authoritative ignore list. This allows deleted # items that were previously ignored to continue to be ignored. @catalog1_raw.resources.each do |resource| next if @catalog2_raw.resource(type: resource['type'], title: resource['title']) next unless tagged_for_ignore?(resource) ignore(type: resource['type'], title: resource['title']) @logger.debug "Ignoring type='#{resource['type']}', title='#{resource['title']}' based on tag in from-catalog" end end # Return catalog1 with filter_and_cleanups applied. # This is in the public section because it's called from spec tests as well # as being called internally. # @return [Array] Filtered resources in catalog def catalog1 filter_and_cleanup(@catalog1) end # Return catalog2 with filter_and_cleanups applied. # This is in the public section because it's called from spec tests as well # as being called internally. # @return [Array] Filtered resources in catalog def catalog2 filter_and_cleanup(@catalog2) end private # Determine if a resource is tagged with any ignore-tag. # @param resource [Hash] The resource # @return [Boolean] true if tagged for ignore, false if not def tagged_for_ignore?(resource) return false unless @opts[:ignore_tags].is_a?(Array) return false unless resource.key?('tags') && resource['tags'].is_a?(Array) @opts[:ignore_tags].each do |tag| # tag_with_type will be like: 'ignored_catalog_diff__mymodule__mytype' tag_with_type = [tag, resource['type'].downcase.gsub(/\W/, '_')].join('__') return true if resource['tags'].include?(tag) || resource['tags'].include?(tag_with_type) end false end # Actually perform the catalog diff. This implements the 3-part algorithm described in the # comment block at the top of this file. def catdiff @logger.debug "Entering catdiff; catalog sizes: #{@catalog1.size}, #{@catalog2.size}" # Compute '+' and '-' from resources that exist in one catalog but not another. # After this returns, # result = Array<'+|-', key, value> (Additions/subtractions of entire resources) # remaining1 & remaining2 = Hash (resources in each catalog) # Note that remaining1.keys == remaining2.keys after running this result, remaining1, remaining2 = preprocess_diff # Call the hashdiff gem. # After this returns, # initial_hashdiff_result = Array<'~', key, oldvalue, newvalue> # hashdiff_add_remove = Array initial_hashdiff_result, hashdiff_add_remove = hashdiff_initial(remaining1, remaining2) result.concat initial_hashdiff_result # Compute '!' which is elements of arrays or hashes within the 'hashdiff' change set that # have been added. See explanation in point #2 in main comment block at the top of this file. hashdiff_nested_changes_result = hashdiff_nested_changes(hashdiff_add_remove, remaining1, remaining2) result.concat hashdiff_nested_changes_result # Remove resources that have been explicitly ignored filter_diffs_for_ignored_items(result) # Legacy options which are now filters @opts[:filters] ||= [] add_element_to_array(@opts[:filters], 'CompilationDir') add_element_to_array(@opts[:filters], 'AbsentFile') if @opts[:suppress_absent_file_details] # Apply any additional pluggable filters. filter_opts = { logger: @logger, from_compilation_dir: @catalog1_raw.compilation_dir, to_compilation_dir: @catalog2_raw.compilation_dir } OctocatalogDiff::CatalogDiff::Filter.apply_filters(result, @opts[:filters], filter_opts) if @opts[:filters].any? # That's it! @logger.debug "Exiting catdiff; change count: #{result.size}" result end # Add an element to an array if it doesn't already exist in that array # @param array_in [Array] Array to have element added (**mutated** by this method) # @param element [?] Element to add def add_element_to_array(array_in, element) array_in << element unless array_in.include?(element) end # Filter the differences for any items that were ignored, by some combination of type, title, and # attribute. This modifies the array itself by selecting only items that do not meet the ignored # filter. def filter_diffs_for_ignored_items(result) result.reject! { |item| ignored?(item) } end # Pre-processing of a catalog. # - Remove 'before' and 'require' from parameters # - Sort 'tags' array, or remove the tags array if tags are being ignored # @param catalog_resources [Array] Catalog resources # @return [Array] Array of cleaned resources def filter_and_cleanup(catalog_resources) result = [] catalog_resources.each do |resource| # Exported resources are skipped (this is specifically testing that the value is # equal to the boolean true, not just that the value exists or something similar) next if resource['exported'] == true # This will be the modified hash added to result hsh = {} hsh['type'] = resource.fetch('type', '') hsh['title'] = resource.fetch('title', '') # Special case for something like: # file { 'my-own-resource-name': # path => '/var/lib/puppet/my-file.txt' # } # # The catalog-diff will treat the file above as "File\f/var/lib/puppet/my-file.txt" since the # name that was given to the resource has no effect on how the file is deployed. # # Note that if the file was specified like this: # file { '/var/lib/puppet/my-file.txt': } # # That also is "File\f/var/lib/puppet/my-file.txt" and that's what we want. if resource.fetch('type', '') == 'File' && resource.key?('parameters') && resource['parameters'].key?('path') hsh['title'] = resource['parameters']['path'] resource['parameters'].delete('path') end # Process each attribute in the resource resource.each do |k, v| # Title was pre-processed next if k == 'title' || k == 'type' # Handle parameters if k == 'parameters' cleansed_param = cleanse_parameters_hash(v) hsh[k] = cleansed_param unless cleansed_param.nil? || cleansed_param.empty? elsif k == 'tags' # The order of tags is unimportant. Sort this array to avoid false diffs if order changes. # Also if tags is empty, don't add. hsh[k] = v.sort if v.is_a?(Array) && v.any? elsif k == 'file' || k == 'line' # We don't care, for the purposes of catalog-diff, from which manifest and line this resource originated. # However, we may report this to the user, so we will keep it in here for now. hsh[k] = v else # Default case: just use the existing value as-is. hsh[k] = v end end result << hsh unless hsh.empty? end result end # Logic to match attribute regular expressions. Called by lambda function in attr_match_rule?. # @param operator [String] Either =~> (any regexp match) or =&> (all diffs must match regexp) # @param regex [Regexp] Regex object # @param old_val [String] Value from first catalog # @param new_val [String] Value from first catalog # @return [Boolean] True if condition is satisfied, false otherwise def regexp_operator_match?(operator, regex, old_val, new_val) # Use diffy to get only the lines that have changed in a text object. # As we iterate through the diff, jump out if we have our answer: either # true if '=~>' finds ANY match, or false if '=&>' fails to find a match. diffy_result = Diffy::Diff.new(old_val, new_val, context: 0) newline_alerts = diffy_result.count { |line| line.strip == '\\ No newline at end of file' } diffy_result.each do |line| if regex.match(line.strip) return true if operator == '=~>' elsif operator == '=&>' next if line.strip == '\\ No newline at end of file' && newline_alerts == 2 return false end end # At this point, we did not return out of the loop early. This means that for # '=~>' no matches were found at all, so we should return false. Or for '=&>' # every diff matched, so we should return true. operator == '=~>' ? false : true end # Determine whether a particular attribute matches a rule # @param rule [Hash] Rule # @param attrib [String] String representation of attribute # @param old_val [?] Old value # @param new_val [?] New value # @return [Boolean] True if attribute matches rule def attr_match_rule?(rule, attrib, old_val, new_val) matcher = ->(_x, _y) { true } rule_attr = rule[:attr].dup # Start with '+' or '-' indicates attribute was added or removed if rule_attr.start_with?('+') return false unless old_val.nil? rule_attr.sub!(/^\+/, '') elsif rule_attr.start_with?('-') return false unless new_val.nil? rule_attr.sub!(/^-/, '') end # Conditions that match the attribute value or regular expression # Operators supported include: # => String equality # =+> Attribute must have been added and equal this # =-> Attribute must have been removed and equal this # =~> Change must match regexp (one line of change matching is sufficient) # =&> Change must match regexp (all lines of change MUST match regexp) if rule_attr =~ /\A(.+?)(=[\-\+~&]?>)(.+)/m rule_attr = Regexp.last_match(1) operator = Regexp.last_match(2) value = Regexp.last_match(3) if operator == '=>' # String equality test matcher = ->(x, y) { x == value || y == value } elsif operator == '=+>' # String equality test only of the new value matcher = ->(_x, y) { y == value } elsif operator == '=->' # String equality test only of the old value matcher = ->(x, _y) { x == value } elsif operator == '=~>' || operator == '=&>' begin my_regex = Regexp.new(value, Regexp::IGNORECASE) rescue RegexpError => exc key = "#{rule[:type]}[#{rule[:title]}] #{rule_attr.gsub(/\f/, '::')} =~ #{value}" raise RegexpError, "Invalid ignore regexp for #{key}: #{exc.message}" end matcher = ->(x, y) { regexp_operator_match?(operator, my_regex, x, y) } end end if rule_attr =~ /\f/ beginning = rule_attr.start_with?("\f") ? '\A' : '(\A|\f)' ending = '(\f|\Z)' rule_attr.gsub!(/^\f+/, '') hash_attr_regexp = Regexp.new(beginning + Regexp.escape(rule_attr) + ending, Regexp::IGNORECASE) return attrib.match(hash_attr_regexp) && matcher.call(old_val, new_val) else s = attrib.downcase.split(/\f/) return s.include?(rule_attr.downcase) && matcher.call(old_val, new_val) end end # Determine if a particular item matches a particular ignore pattern # @param rule [Hash] Ignore rule # @param diff_type [String] One of +, -, ~, ! # @param hsh [Hash] { type: title: attr: } parsed resource name # @param old_val [?] Old value # @param new_val [?] New value # @return [Boolean] True if the item matched the rule def ignore_match?(rule_in, diff_type, hsh, old_val, new_val) rule = rule_in.dup # Type matches? if rule[:type].is_a?(Regexp) return false unless hsh[:type].match(rule[:type]) elsif rule[:type].is_a?(String) return false unless rule[:type] == '*' || rule[:type].casecmp(hsh[:type]).zero? end # Title matches? (Support regexp and string) if rule[:title].is_a?(Regexp) return false unless hsh[:title].match(rule[:title]) elsif rule[:title] != '*' return false unless rule[:title].casecmp(hsh[:title]).zero? end # If rule[:attr] is a regular expression, handle that case here. if rule[:attr].is_a?(Regexp) return false unless hsh[:attr].is_a?(String) return false unless rule[:attr].match(hsh[:attr]) return ignore_match_true(hsh, rule) end # Special 'attributes': Ignore specific diff types (+ add, - remove, ~ and ! change) if rule[:attr] =~ /\A[\-\+~!]+\Z/ return ignore_match_true(hsh, rule) if rule[:attr].include?(diff_type) return false end # Attribute matches? return ignore_match_true(hsh, rule) if hsh[:attr].nil? && rule[:attr].nil? return ignore_match_true(hsh, rule) if rule[:attr] == '*' return false if hsh[:attr].nil? # Attributes that match values if rule[:attr].is_a?(Array) rule[:attr].each do |attrib| return false unless attr_match_rule?(rule.merge(attr: attrib), hsh[:attr], old_val, new_val) end else return false unless attr_match_rule?(rule, hsh[:attr], old_val, new_val) end # Still here? Must be true. ignore_match_true(hsh, rule) end # Debugging for ignore_match: This logs a debug message for an ignored diff and then returns true. # @param hsh [Hash] Item that is being checked # @param rule [Hash] Ignore rule # @return [Boolean] Always returns true def ignore_match_true(hsh, rule) @logger.debug "Ignoring #{hsh.inspect}, matches #{rule.inspect}" true end # Determine if a given item is ignored # @param diff [Array] Diff # @return [Boolean] True to ignore resource, false not to ignore def ignored?(diff) key = diff[1] hsh = if key =~ /\A([^\f]+)\f([^\f]+)\Z/ { type: Regexp.last_match(1), title: Regexp.last_match(2) } else s = key.split(/\f/, 3) { type: s[0], title: s[1], attr: s[2] } end @ignore.each do |rule| return true if ignore_match?(rule, diff[0], hsh, diff[2], diff[3]) end false end # Cleanse parameters of filtered attributes. # @param parameters_hash [Hash] Hash of parameters # @return [Hash] Cleaned parameters hash (original input hash is not altered) def cleanse_parameters_hash(parameters_hash) result = parameters_hash.dup # 'before' and 'require' handle internal Puppet ordering but do not affect what # happens on the target machine. Don't consider these for the purpose of catalog diff. result.delete('before') result.delete('require') # Sort arrays for parameters where the order is unimportant %w(notify subscribe tag).each { |key| result[key].sort! if result[key].is_a?(Array) } # Return the result result end # Pre-process catalog resources by looking for additions and removals. This is required to distinguish between # top-level addition/removal of resources, and addition/removal of elements from arrays and hashes nested within # resources (those too will be reported as +/- by hashdiff, but we want to see them as changes). # @return [Array<['+|-', Key, Hash]>, Array<(catalog1 hashes)>, Array<(catalog2 hashes)>] Data def preprocess_diff @logger.debug "Entering preprocess_diff; catalog sizes: #{@catalog1.size}, #{@catalog2.size}" # Do the pre-processing: filter_and_cleanup catalogs of resources that do not matter, and then run # through each to tokenize the entries for initial comparison. # NOTE: 'catalog1' and 'catalog2' are methods above that call filter_and_cleanup(@catalogX) catalog1_result = resources_as_hashes_with_serialized_keys(catalog1) catalog1_resources = catalog1_result[:catalog] catalog2_result = resources_as_hashes_with_serialized_keys(catalog2) catalog2_resources = catalog2_result[:catalog] # Call out all added and removed keys, and delete these from further consideration. # (That way, 'hashdiff' will only be used to compare keys existing in both old and new.) result = [] added_keys = catalog2_resources.keys - catalog1_resources.keys removed_keys = catalog1_resources.keys - catalog2_resources.keys added_keys.each do |key| key_for_map = key.split(/\f/, 3)[0..1].join("\f") # Keep first two values separated by \f result << ['+', key, catalog2_resources[key], catalog2_result[:catalog_map][key_for_map]] catalog2_resources.delete(key) end removed_keys.each do |key| key_for_map = key.split(/\f/, 3)[0..1].join("\f") # Keep first two values separated by \f result << ['-', key, catalog1_resources[key], catalog1_result[:catalog_map][key_for_map]] catalog1_resources.delete(key) end @logger.debug "Exiting preprocess_diff; added #{added_keys.size}, removed #{removed_keys.size}" [result, catalog1_result, catalog2_result] end # This runs the remaining resources in the catalogs through hashdiff. # @param catalog1_resources [] Hash of catalog1's resources, tokenized # @param catalog2_resources [] Hash of catalog2's resources, tokenized # @return [Array, Array<(Token, Old, New)>] Input to next step def hashdiff_initial(catalog1_in, catalog2_in) catalog1_resources = catalog1_in[:catalog] catalog2_resources = catalog2_in[:catalog] @logger.debug "Entering hashdiff_initial; catalog sizes: #{catalog1_resources.size}, #{catalog2_resources.size}" result = [] hashdiff_add_remove = Set.new hashdiff_result = HashDiff.diff(catalog1_resources, catalog2_resources, delimiter: "\f") hashdiff_result.each do |obj| # Regular change if obj[0] == '~' key_for_map = obj[1].split(/\f/, 3)[0..1].join("\f") # Keep first two values separated by \f obj << catalog1_in[:catalog_map][key_for_map] obj << catalog2_in[:catalog_map][key_for_map] result << obj next end # Added/removed element to/from array if obj[1] =~ /^(.+)\[\d+\]/ hashdiff_add_remove.add(Regexp.last_match(1)) next end # Added a new key that points to some kind of data structure that we know how # to handle. classes = [String, Integer, Float, TrueClass, FalseClass, Array, Hash] if obj[1] =~ /^(.+)\f([^\f]+)$/ && OctocatalogDiff::Util::Util.object_is_any_of?(obj[2], classes) hashdiff_add_remove.add(obj[1]) next end # Any other weird edge cases need to be added and handled here. For now just error out. # :nocov: raise "Bug (please report): Unexpected data structure in hashdiff_result: #{obj.inspect}" # :nocov: end @logger.debug "Exiting hashdiff_initial; changes: #{result.size}, nested changes: #{hashdiff_add_remove.size}" [result, hashdiff_add_remove.to_a] end # This diffs nested changes deep in the data structure. Each item in hashdiff_add_remove # has been previously identified as being an addition or removal from a deeply nested element # that exists in both old and new. This code compares that deeply nested element in both the # old and new, and uses status '!' (rather than '+', '-', or '~') to indicate that the change # occurred in a deeply nested element. # @param hashdiff_add_remove [Array] Adds/removes from hashdiff # @param remaining1 [Hash] Serialized key / value pairs for catalog1 resources # @param remaining2 [Hash] Serialized key / value pairs for catalog2 resources # @return [Array<'!', key, old, new>] Change set def hashdiff_nested_changes(hashdiff_add_remove, remaining1, remaining2) return [] if hashdiff_add_remove.empty? catalog1 = remaining1[:catalog] catalog2 = remaining2[:catalog] catmap1 = remaining1[:catalog_map] catmap2 = remaining2[:catalog_map] result = [] hashdiff_add_remove.each do |key| key_split = key.split(/\f/) first_part_of_key = [key_split.shift, key_split.shift].join("\f") key_split.unshift first_part_of_key if catalog1[first_part_of_key].is_a?(Hash) && catalog2[first_part_of_key].is_a?(Hash) # At this point catalog1[first_part_of_key] might look like this: # { # "type"=>"Class", # "title"=>"Openssl::Package", # "exported"=>false, # "parameters"=>{"openssl_version"=>"1.0.1-4", "common-array"=>[1, 3, 5]} # } # and key_split looks like this: # [ "Class\fOpenssl::Package", 'parameters', 'common-array' ] # # We have to dig out remaining1["Class\fOpenssl::Package"]['parameters']['common-array'] # to do the comparison. obj0 = dig_out_key(catalog1, key_split.dup) obj1 = dig_out_key(catalog2, key_split.dup) result << ['!', key, obj0, obj1, catmap1[first_part_of_key], catmap2[first_part_of_key]] else # Bug condition # :nocov: raise "BUG (Please report): Unexpected resource: #{first_part_of_key.inspect} not a catalog resource" # :nocov: end end result end # From an array of keys [key1, key2, key3, ...] dig out the value of hash[key1][key2][key3]... # @param hash_in [Hash] Starting hash (or value passed in by recursion) # @param key_array [Array] Names of keys in order # @return [?] Value of hash_in[key1][key2][key3]..., or nil if any keys along the way don't exist def dig_out_key(hash_in, key_array) return hash_in if key_array.empty? return hash_in unless hash_in.is_a?(Hash) return nil unless hash_in.key?(key_array[0]) next_key = key_array.shift dig_out_key(hash_in[next_key], key_array) end # This is a helper for the constructor, verifying that the incoming catalog is an expected # object. # @param catalog [OctocatalogDiff::Catalog] Incoming catalog # @return [Hash] Internal simplified hash object def catalog_resources(catalog_in, name = 'Passed catalog') return catalog_in.resources if catalog_in.is_a?(OctocatalogDiff::Catalog) raise OctocatalogDiff::Errors::DifferError, "#{name} is not a valid catalog (input datatype: #{catalog_in.class})" end # Turn array of resources into a hash by serialized keys. For consistency with 'hashdiff' # the serialized key is the resource type and all components of the title (split on '::'), # joined with \f. # @param catalog Array Resource array from catalog # @return [Hash] See description above def resources_as_hashes_with_serialized_keys(catalog) result = { catalog: {}, catalog_map: {} } catalog.each do |item| i = item.dup result[:catalog_map]["#{item['type']}\f#{item['title']}"] = { 'file' => item['file'], 'line' => item['line'] } i.delete('file') i.delete('line') result[:catalog]["#{item['type']}\f#{item['title']}"] = i end result end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/external/0000755000004100000410000000000013413744025022556 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/external/pson/0000755000004100000410000000000013413744025023535 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/external/pson/version.rb0000644000004100000410000000041713413744025025551 0ustar www-datawww-datamodule PSON # PSON version VERSION = '1.1.9' VERSION_ARRAY = VERSION.split(/\./).map { |x| x.to_i } # :nodoc: VERSION_MAJOR = VERSION_ARRAY[0] # :nodoc: VERSION_MINOR = VERSION_ARRAY[1] # :nodoc: VERSION_BUILD = VERSION_ARRAY[2] # :nodoc: end octocatalog-diff-1.5.4/lib/octocatalog-diff/external/pson/pure.rb0000644000004100000410000000054513413744025025041 0ustar www-datawww-datarequire_relative 'common' require_relative 'pure/parser' require_relative 'pure/generator' module PSON # This module holds all the modules/classes that implement PSON's # functionality in pure ruby. module Pure $DEBUG and warn "Using pure library for PSON." PSON.parser = Parser PSON.generator = Generator end PSON_LOADED = true end octocatalog-diff-1.5.4/lib/octocatalog-diff/external/pson/README.md0000644000004100000410000000164713413744025025024 0ustar www-datawww-data# PSON Puppet uses a JSON variant called PSON to serialize data (e.g. facts) transmitting to/from a Puppet master. Documentation for PSON can be found here: https://docs.puppet.com/puppet/4.6/reference/http_api/pson.html The code in this directory was taken directly from Puppet and can be found at: https://github.com/puppetlabs/puppet/tree/master/lib/puppet/external/pson If you have found this code to deal with Puppet serialization, you should probably take the original and most up-to-date code from Puppet at the location above. This code contains the following modifications: - Change the `require` statements to `require_relative` statements so they work in this gem's directory structure - Change `$MATCH` to `$&` because `$MATCH` is undefined without `require 'english`` or equivalent This code is licensed by Puppet under the Apache 2.0 license. A copy of the Puppet license can be found in this directory. octocatalog-diff-1.5.4/lib/octocatalog-diff/external/pson/pure/0000755000004100000410000000000013413744025024510 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/external/pson/pure/parser.rb0000644000004100000410000002373513413744025026343 0ustar www-datawww-datarequire 'strscan' module PSON module Pure # This class implements the PSON parser that is used to parse a PSON string # into a Ruby data structure. class Parser < StringScanner STRING = /" ((?:[^\x0-\x1f"\\] | # escaped special characters: \\["\\\/bfnrt] | \\u[0-9a-fA-F]{4} | # match all but escaped special characters: \\[\x20-\x21\x23-\x2e\x30-\x5b\x5d-\x61\x63-\x65\x67-\x6d\x6f-\x71\x73\x75-\xff])*) "/nx INTEGER = /(-?0|-?[1-9]\d*)/ FLOAT = /(-? (?:0|[1-9]\d*) (?: \.\d+(?i:e[+-]?\d+) | \.\d+ | (?i:e[+-]?\d+) ) )/x NAN = /NaN/ INFINITY = /Infinity/ MINUS_INFINITY = /-Infinity/ OBJECT_OPEN = /\{/ OBJECT_CLOSE = /\}/ ARRAY_OPEN = /\[/ ARRAY_CLOSE = /\]/ PAIR_DELIMITER = /:/ COLLECTION_DELIMITER = /,/ TRUE = /true/ FALSE = /false/ NULL = /null/ IGNORE = %r( (?: //[^\n\r]*[\n\r]| # line comments /\* # c-style comments (?: [^*/]| # normal chars /[^*]| # slashes that do not start a nested comment \*[^/]| # asterisks that do not end this comment /(?=\*/) # single slash before this comment's end )* \*/ # the End of this comment |[ \t\r\n]+ # whitespaces: space, horicontal tab, lf, cr )+ )mx UNPARSED = Object.new # Creates a new PSON::Pure::Parser instance for the string _source_. # # It will be configured by the _opts_ hash. _opts_ can have the following # keys: # * *max_nesting*: The maximum depth of nesting allowed in the parsed data # structures. Disable depth checking with :max_nesting => false|nil|0, # it defaults to 19. # * *allow_nan*: If set to true, allow NaN, Infinity and -Infinity in # defiance of RFC 4627 to be parsed by the Parser. This option defaults # to false. # * *object_class*: Defaults to Hash # * *array_class*: Defaults to Array def initialize(source, opts = {}) source = convert_encoding source super source if !opts.key?(:max_nesting) # defaults to 19 @max_nesting = 19 elsif opts[:max_nesting] @max_nesting = opts[:max_nesting] else @max_nesting = 0 end @allow_nan = !!opts[:allow_nan] @object_class = opts[:object_class] || Hash @array_class = opts[:array_class] || Array end alias source string # Parses the current PSON string _source_ and returns the complete data # structure as a result. def parse reset obj = nil until eos? case when scan(OBJECT_OPEN) obj and raise ParserError, "source '#{peek(20)}' not in PSON!" @current_nesting = 1 obj = parse_object when scan(ARRAY_OPEN) obj and raise ParserError, "source '#{peek(20)}' not in PSON!" @current_nesting = 1 obj = parse_array when skip(IGNORE) ; else raise ParserError, "source '#{peek(20)}' not in PSON!" end end obj or raise ParserError, "source did not contain any PSON!" obj end private def convert_encoding(source) if source.respond_to?(:to_str) source = source.to_str else raise TypeError, "#{source.inspect} is not like a string" end if supports_encodings?(source) if source.encoding == ::Encoding::ASCII_8BIT b = source[0, 4].bytes.to_a source = case when b.size >= 4 && b[0] == 0 && b[1] == 0 && b[2] == 0 source.dup.force_encoding(::Encoding::UTF_32BE).encode!(::Encoding::UTF_8) when b.size >= 4 && b[0] == 0 && b[2] == 0 source.dup.force_encoding(::Encoding::UTF_16BE).encode!(::Encoding::UTF_8) when b.size >= 4 && b[1] == 0 && b[2] == 0 && b[3] == 0 source.dup.force_encoding(::Encoding::UTF_32LE).encode!(::Encoding::UTF_8) when b.size >= 4 && b[1] == 0 && b[3] == 0 source.dup.force_encoding(::Encoding::UTF_16LE).encode!(::Encoding::UTF_8) else source.dup end else source = source.encode(::Encoding::UTF_8) end source.force_encoding(::Encoding::ASCII_8BIT) else b = source source = case when b.size >= 4 && b[0] == 0 && b[1] == 0 && b[2] == 0 PSON.encode('utf-8', 'utf-32be', b) when b.size >= 4 && b[0] == 0 && b[2] == 0 PSON.encode('utf-8', 'utf-16be', b) when b.size >= 4 && b[1] == 0 && b[2] == 0 && b[3] == 0 PSON.encode('utf-8', 'utf-32le', b) when b.size >= 4 && b[1] == 0 && b[3] == 0 PSON.encode('utf-8', 'utf-16le', b) else b end end source end def supports_encodings?(string) # Some modules, such as REXML on 1.8.7 (see #22804) can actually create # a top-level Encoding constant when they are misused. Therefore # checking for just that constant is not enough, so we'll be a bit more # robust about if we can actually support encoding transformations. string.respond_to?(:encoding) && defined?(::Encoding) end # Unescape characters in strings. UNESCAPE_MAP = Hash.new { |h, k| h[k] = k.chr } UNESCAPE_MAP.update( { ?" => '"', ?\\ => '\\', ?/ => '/', ?b => "\b", ?f => "\f", ?n => "\n", ?r => "\r", ?t => "\t", ?u => nil, }) def parse_string if scan(STRING) return '' if self[1].empty? string = self[1].gsub(%r{(?:\\[\\bfnrt"/]|(?:\\u(?:[A-Fa-f\d]{4}))+|\\[\x20-\xff])}n) do |c| if u = UNESCAPE_MAP[$&[1]] u else # \uXXXX bytes = '' i = 0 while c[6 * i] == ?\\ && c[6 * i + 1] == ?u bytes << c[6 * i + 2, 2].to_i(16) << c[6 * i + 4, 2].to_i(16) i += 1 end PSON.encode('utf-8', 'utf-16be', bytes) end end string.force_encoding(Encoding::UTF_8) if string.respond_to?(:force_encoding) string else UNPARSED end rescue => e raise GeneratorError, "Caught #{e.class}: #{e}", e.backtrace end def parse_value case when scan(FLOAT) Float(self[1]) when scan(INTEGER) Integer(self[1]) when scan(TRUE) true when scan(FALSE) false when scan(NULL) nil when (string = parse_string) != UNPARSED string when scan(ARRAY_OPEN) @current_nesting += 1 ary = parse_array @current_nesting -= 1 ary when scan(OBJECT_OPEN) @current_nesting += 1 obj = parse_object @current_nesting -= 1 obj when @allow_nan && scan(NAN) NaN when @allow_nan && scan(INFINITY) Infinity when @allow_nan && scan(MINUS_INFINITY) MinusInfinity else UNPARSED end end def parse_array raise NestingError, "nesting of #@current_nesting is too deep" if @max_nesting.nonzero? && @current_nesting > @max_nesting result = @array_class.new delim = false until eos? case when (value = parse_value) != UNPARSED delim = false result << value skip(IGNORE) if scan(COLLECTION_DELIMITER) delim = true elsif match?(ARRAY_CLOSE) ; else raise ParserError, "expected ',' or ']' in array at '#{peek(20)}'!" end when scan(ARRAY_CLOSE) raise ParserError, "expected next element in array at '#{peek(20)}'!" if delim break when skip(IGNORE) ; else raise ParserError, "unexpected token in array at '#{peek(20)}'!" end end result end def parse_object raise NestingError, "nesting of #@current_nesting is too deep" if @max_nesting.nonzero? && @current_nesting > @max_nesting result = @object_class.new delim = false until eos? case when (string = parse_string) != UNPARSED skip(IGNORE) raise ParserError, "expected ':' in object at '#{peek(20)}'!" unless scan(PAIR_DELIMITER) skip(IGNORE) unless (value = parse_value).equal? UNPARSED result[string] = value delim = false skip(IGNORE) if scan(COLLECTION_DELIMITER) delim = true elsif match?(OBJECT_CLOSE) ; else raise ParserError, "expected ',' or '}' in object at '#{peek(20)}'!" end else raise ParserError, "expected value in object at '#{peek(20)}'!" end when scan(OBJECT_CLOSE) raise ParserError, "expected next name, value pair in object at '#{peek(20)}'!" if delim break when skip(IGNORE) ; else raise ParserError, "unexpected token in object at '#{peek(20)}'!" end end result end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/external/pson/pure/generator.rb0000644000004100000410000003201613413744025027025 0ustar www-datawww-datamodule PSON MAP = { "\x0" => '\u0000', "\x1" => '\u0001', "\x2" => '\u0002', "\x3" => '\u0003', "\x4" => '\u0004', "\x5" => '\u0005', "\x6" => '\u0006', "\x7" => '\u0007', "\b" => '\b', "\t" => '\t', "\n" => '\n', "\xb" => '\u000b', "\f" => '\f', "\r" => '\r', "\xe" => '\u000e', "\xf" => '\u000f', "\x10" => '\u0010', "\x11" => '\u0011', "\x12" => '\u0012', "\x13" => '\u0013', "\x14" => '\u0014', "\x15" => '\u0015', "\x16" => '\u0016', "\x17" => '\u0017', "\x18" => '\u0018', "\x19" => '\u0019', "\x1a" => '\u001a', "\x1b" => '\u001b', "\x1c" => '\u001c', "\x1d" => '\u001d', "\x1e" => '\u001e', "\x1f" => '\u001f', '"' => '\"', '\\' => '\\\\', } # :nodoc: # Convert a UTF8 encoded Ruby string _string_ to a PSON string, encoded with # UTF16 big endian characters as \u????, and return it. if String.method_defined?(:force_encoding) def utf8_to_pson(string) # :nodoc: string = string.dup string << '' # XXX workaround: avoid buffer sharing string.force_encoding(Encoding::ASCII_8BIT) string.gsub!(/["\\\x0-\x1f]/) { MAP[$&] } string rescue => e raise GeneratorError, "Caught #{e.class}: #{e}", e.backtrace end else def utf8_to_pson(string) # :nodoc: string.gsub(/["\\\x0-\x1f]/n) { MAP[$&] } end end module_function :utf8_to_pson module Pure module Generator # This class is used to create State instances, that are use to hold data # while generating a PSON text from a Ruby data structure. class State # Creates a State object from _opts_, which ought to be Hash to create # a new State instance configured by _opts_, something else to create # an unconfigured instance. If _opts_ is a State object, it is just # returned. def self.from_state(opts) case opts when self opts when Hash new(opts) else new end end # Instantiates a new State object, configured by _opts_. # # _opts_ can have the following keys: # # * *indent*: a string used to indent levels (default: ''), # * *space*: a string that is put after, a : or , delimiter (default: ''), # * *space_before*: a string that is put before a : pair delimiter (default: ''), # * *object_nl*: a string that is put at the end of a PSON object (default: ''), # * *array_nl*: a string that is put at the end of a PSON array (default: ''), # * *check_circular*: true if checking for circular data structures # should be done (the default), false otherwise. # * *check_circular*: true if checking for circular data structures # should be done, false (the default) otherwise. # * *allow_nan*: true if NaN, Infinity, and -Infinity should be # generated, otherwise an exception is thrown, if these values are # encountered. This options defaults to false. def initialize(opts = {}) @seen = {} @indent = '' @space = '' @space_before = '' @object_nl = '' @array_nl = '' @check_circular = true @allow_nan = false configure opts end # This string is used to indent levels in the PSON text. attr_accessor :indent # This string is used to insert a space between the tokens in a PSON # string. attr_accessor :space # This string is used to insert a space before the ':' in PSON objects. attr_accessor :space_before # This string is put at the end of a line that holds a PSON object (or # Hash). attr_accessor :object_nl # This string is put at the end of a line that holds a PSON array. attr_accessor :array_nl # This integer returns the maximum level of data structure nesting in # the generated PSON, max_nesting = 0 if no maximum is checked. attr_accessor :max_nesting def check_max_nesting(depth) # :nodoc: return if @max_nesting.zero? current_nesting = depth + 1 current_nesting > @max_nesting and raise NestingError, "nesting of #{current_nesting} is too deep" end # Returns true, if circular data structures should be checked, # otherwise returns false. def check_circular? @check_circular end # Returns true if NaN, Infinity, and -Infinity should be considered as # valid PSON and output. def allow_nan? @allow_nan end # Returns _true_, if _object_ was already seen during this generating # run. def seen?(object) @seen.key?(object.__id__) end # Remember _object_, to find out if it was already encountered (if a # cyclic data structure is if a cyclic data structure is rendered). def remember(object) @seen[object.__id__] = true end # Forget _object_ for this generating run. def forget(object) @seen.delete object.__id__ end # Configure this State instance with the Hash _opts_, and return # itself. def configure(opts) @indent = opts[:indent] if opts.key?(:indent) @space = opts[:space] if opts.key?(:space) @space_before = opts[:space_before] if opts.key?(:space_before) @object_nl = opts[:object_nl] if opts.key?(:object_nl) @array_nl = opts[:array_nl] if opts.key?(:array_nl) @check_circular = !!opts[:check_circular] if opts.key?(:check_circular) @allow_nan = !!opts[:allow_nan] if opts.key?(:allow_nan) if !opts.key?(:max_nesting) # defaults to 19 @max_nesting = 19 elsif opts[:max_nesting] @max_nesting = opts[:max_nesting] else @max_nesting = 0 end self end # Returns the configuration instance variables as a hash, that can be # passed to the configure method. def to_h result = {} for iv in %w{indent space space_before object_nl array_nl check_circular allow_nan max_nesting} result[iv.intern] = instance_variable_get("@#{iv}") end result end end module GeneratorMethods module Object # Converts this object to a string (calling #to_s), converts # it to a PSON string, and returns the result. This is a fallback, if no # special method #to_pson was defined for some object. def to_pson(*) to_s.to_pson end end module Hash # Returns a PSON string containing a PSON object, that is unparsed from # this Hash instance. # _state_ is a PSON::State object, that can also be used to configure the # produced PSON string output further. # _depth_ is used to find out nesting depth, to indent accordingly. def to_pson(state = nil, depth = 0, *) if state state = PSON.state.from_state(state) state.check_max_nesting(depth) pson_check_circular(state) { pson_transform(state, depth) } else pson_transform(state, depth) end end private def pson_check_circular(state) if state and state.check_circular? state.seen?(self) and raise PSON::CircularDatastructure, "circular data structures not supported!" state.remember self end yield ensure state and state.forget self end def pson_shift(state, depth) state and not state.object_nl.empty? or return '' state.indent * depth end def pson_transform(state, depth) delim = ',' if state delim << state.object_nl result = '{' result << state.object_nl result << map { |key,value| s = pson_shift(state, depth + 1) s << key.to_s.to_pson(state, depth + 1) s << state.space_before s << ':' s << state.space s << value.to_pson(state, depth + 1) }.join(delim) result << state.object_nl result << pson_shift(state, depth) result << '}' else result = '{' result << map { |key,value| key.to_s.to_pson << ':' << value.to_pson }.join(delim) result << '}' end result end end module Array # Returns a PSON string containing a PSON array, that is unparsed from # this Array instance. # _state_ is a PSON::State object, that can also be used to configure the # produced PSON string output further. # _depth_ is used to find out nesting depth, to indent accordingly. def to_pson(state = nil, depth = 0, *) if state state = PSON.state.from_state(state) state.check_max_nesting(depth) pson_check_circular(state) { pson_transform(state, depth) } else pson_transform(state, depth) end end private def pson_check_circular(state) if state and state.check_circular? state.seen?(self) and raise PSON::CircularDatastructure, "circular data structures not supported!" state.remember self end yield ensure state and state.forget self end def pson_shift(state, depth) state and not state.array_nl.empty? or return '' state.indent * depth end def pson_transform(state, depth) delim = ',' if state delim << state.array_nl result = '[' result << state.array_nl result << map { |value| pson_shift(state, depth + 1) << value.to_pson(state, depth + 1) }.join(delim) result << state.array_nl result << pson_shift(state, depth) result << ']' else '[' << map { |value| value.to_pson }.join(delim) << ']' end end end module Integer # Returns a PSON string representation for this Integer number. def to_pson(*) to_s end end module Float # Returns a PSON string representation for this Float number. def to_pson(state = nil, *) if infinite? || nan? if !state || state.allow_nan? to_s else raise GeneratorError, "#{self} not allowed in PSON" end else to_s end end end module String # This string should be encoded with UTF-8 A call to this method # returns a PSON string encoded with UTF16 big endian characters as # \u????. def to_pson(*) '"' << PSON.utf8_to_pson(self) << '"' end # Module that holds the extinding methods if, the String module is # included. module Extend # Raw Strings are PSON Objects (the raw bytes are stored in an array for the # key "raw"). The Ruby String can be created by this module method. def pson_create(o) o['raw'].pack('C*') end end # Extends _modul_ with the String::Extend module. def self.included(modul) modul.extend Extend end # This method creates a raw object hash, that can be nested into # other data structures and will be unparsed as a raw string. This # method should be used, if you want to convert raw strings to PSON # instead of UTF-8 strings, e.g. binary data. def to_pson_raw_object # create_id will be ignored during deserialization { PSON.create_id => self.class.name, 'raw' => self.unpack('C*'), } end # This method creates a PSON text from the result of # a call to to_pson_raw_object of this String. def to_pson_raw(*args) to_pson_raw_object.to_pson(*args) end end module TrueClass # Returns a PSON string for true: 'true'. def to_pson(*) 'true' end end module FalseClass # Returns a PSON string for false: 'false'. def to_pson(*) 'false' end end module NilClass # Returns a PSON string for nil: 'null'. def to_pson(*) 'null' end end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/external/pson/LICENSE0000644000004100000410000000127113413744025024543 0ustar www-datawww-data Puppet - Automating Configuration Management. Copyright (C) 2005-2016 Puppet, Inc. Puppet, Inc. can be contacted at: info@puppet.com Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. octocatalog-diff-1.5.4/lib/octocatalog-diff/external/pson/common.rb0000644000004100000410000003060413413744025025355 0ustar www-datawww-datarequire_relative 'version' module PSON class << self # If _object_ is string-like parse the string and return the parsed result # as a Ruby data structure. Otherwise generate a PSON text from the Ruby # data structure object and return it. # # The _opts_ argument is passed through to generate/parse respectively, see # generate and parse for their documentation. def [](object, opts = {}) if object.respond_to? :to_str PSON.parse(object.to_str, opts => {}) else PSON.generate(object, opts => {}) end end # Returns the PSON parser class, that is used by PSON. This might be either # PSON::Ext::Parser or PSON::Pure::Parser. attr_reader :parser # Set the PSON parser class _parser_ to be used by PSON. def parser=(parser) # :nodoc: @parser = parser remove_const :Parser if const_defined? :Parser const_set :Parser, parser end # Return the constant located at _path_. # Anything may be registered as a path by calling register_path, above. # Otherwise, the format of _path_ has to be either ::A::B::C or A::B::C. # In either of these cases A has to be defined in Object (e.g. the path # must be an absolute namespace path. If the constant doesn't exist at # the given path, an ArgumentError is raised. def deep_const_get(path) # :nodoc: path = path.to_s path.split(/::/).inject(Object) do |p, c| case when c.empty? then p when p.const_defined?(c) then p.const_get(c) else raise ArgumentError, "can't find const for unregistered document type #{path}" end end end # Set the module _generator_ to be used by PSON. def generator=(generator) # :nodoc: @generator = generator generator_methods = generator::GeneratorMethods for const in generator_methods.constants klass = deep_const_get(const) modul = generator_methods.const_get(const) klass.class_eval do instance_methods(false).each do |m| m.to_s == 'to_pson' and remove_method m end include modul end end self.state = generator::State const_set :State, self.state end # Returns the PSON generator modul, that is used by PSON. This might be # either PSON::Ext::Generator or PSON::Pure::Generator. attr_reader :generator # Returns the PSON generator state class, that is used by PSON. This might # be either PSON::Ext::Generator::State or PSON::Pure::Generator::State. attr_accessor :state # This is create identifier, that is used to decide, if the _pson_create_ # hook of a class should be called. It defaults to 'document_type'. attr_accessor :create_id end self.create_id = 'document_type' NaN = (-1.0) ** 0.5 Infinity = 1.0/0 MinusInfinity = -Infinity # The base exception for PSON errors. class PSONError < StandardError; end # This exception is raised, if a parser error occurs. class ParserError < PSONError; end # This exception is raised, if the nesting of parsed datastructures is too # deep. class NestingError < ParserError; end # This exception is raised, if a generator or unparser error occurs. class GeneratorError < PSONError; end # For backwards compatibility UnparserError = GeneratorError # If a circular data structure is encountered while unparsing # this exception is raised. class CircularDatastructure < GeneratorError; end # This exception is raised, if the required unicode support is missing on the # system. Usually this means, that the iconv library is not installed. class MissingUnicodeSupport < PSONError; end module_function # Parse the PSON string _source_ into a Ruby data structure and return it. # # _opts_ can have the following # keys: # * *max_nesting*: The maximum depth of nesting allowed in the parsed data # structures. Disable depth checking with :max_nesting => false, it defaults # to 19. # * *allow_nan*: If set to true, allow NaN, Infinity and -Infinity in # defiance of RFC 4627 to be parsed by the Parser. This option defaults # to false. def parse(source, opts = {}) PSON.parser.new(source, opts).parse end # Parse the PSON string _source_ into a Ruby data structure and return it. # The bang version of the parse method, defaults to the more dangerous values # for the _opts_ hash, so be sure only to parse trusted _source_ strings. # # _opts_ can have the following keys: # * *max_nesting*: The maximum depth of nesting allowed in the parsed data # structures. Enable depth checking with :max_nesting => anInteger. The parse! # methods defaults to not doing max depth checking: This can be dangerous, # if someone wants to fill up your stack. # * *allow_nan*: If set to true, allow NaN, Infinity, and -Infinity in # defiance of RFC 4627 to be parsed by the Parser. This option defaults # to true. def parse!(source, opts = {}) opts = { :max_nesting => false, :allow_nan => true }.update(opts) PSON.parser.new(source, opts).parse end # Unparse the Ruby data structure _obj_ into a single line PSON string and # return it. _state_ is # * a PSON::State object, # * or a Hash like object (responding to to_hash), # * an object convertible into a hash by a to_h method, # that is used as or to configure a State object. # # It defaults to a state object, that creates the shortest possible PSON text # in one line, checks for circular data structures and doesn't allow NaN, # Infinity, and -Infinity. # # A _state_ hash can have the following keys: # * *indent*: a string used to indent levels (default: ''), # * *space*: a string that is put after, a : or , delimiter (default: ''), # * *space_before*: a string that is put before a : pair delimiter (default: ''), # * *object_nl*: a string that is put at the end of a PSON object (default: ''), # * *array_nl*: a string that is put at the end of a PSON array (default: ''), # * *check_circular*: true if checking for circular data structures # should be done (the default), false otherwise. # * *allow_nan*: true if NaN, Infinity, and -Infinity should be # generated, otherwise an exception is thrown, if these values are # encountered. This options defaults to false. # * *max_nesting*: The maximum depth of nesting allowed in the data # structures from which PSON is to be generated. Disable depth checking # with :max_nesting => false, it defaults to 19. # # See also the fast_generate for the fastest creation method with the least # amount of sanity checks, and the pretty_generate method for some # defaults for a pretty output. def generate(obj, state = nil) if state state = State.from_state(state) else state = State.new end obj.to_pson(state) end # :stopdoc: # I want to deprecate these later, so I'll first be silent about them, and # later delete them. alias unparse generate module_function :unparse # :startdoc: # Unparse the Ruby data structure _obj_ into a single line PSON string and # return it. This method disables the checks for circles in Ruby objects, and # also generates NaN, Infinity, and, -Infinity float values. # # *WARNING*: Be careful not to pass any Ruby data structures with circles as # _obj_ argument, because this will cause PSON to go into an infinite loop. def fast_generate(obj) obj.to_pson(nil) end # :stopdoc: # I want to deprecate these later, so I'll first be silent about them, and later delete them. alias fast_unparse fast_generate module_function :fast_unparse # :startdoc: # Unparse the Ruby data structure _obj_ into a PSON string and return it. The # returned string is a prettier form of the string returned by #unparse. # # The _opts_ argument can be used to configure the generator, see the # generate method for a more detailed explanation. def pretty_generate(obj, opts = nil) state = PSON.state.new( :indent => ' ', :space => ' ', :object_nl => "\n", :array_nl => "\n", :check_circular => true ) if opts if opts.respond_to? :to_hash opts = opts.to_hash elsif opts.respond_to? :to_h opts = opts.to_h else raise TypeError, "can't convert #{opts.class} into Hash" end state.configure(opts) end obj.to_pson(state) end # :stopdoc: # I want to deprecate these later, so I'll first be silent about them, and later delete them. alias pretty_unparse pretty_generate module_function :pretty_unparse # :startdoc: # Load a ruby data structure from a PSON _source_ and return it. A source can # either be a string-like object, an IO like object, or an object responding # to the read method. If _proc_ was given, it will be called with any nested # Ruby object as an argument recursively in depth first order. # # This method is part of the implementation of the load/dump interface of # Marshal and YAML. def load(source, proc = nil) if source.respond_to? :to_str source = source.to_str elsif source.respond_to? :to_io source = source.to_io.read else source = source.read end result = parse(source, :max_nesting => false, :allow_nan => true) recurse_proc(result, &proc) if proc result end def recurse_proc(result, &proc) case result when Array result.each { |x| recurse_proc x, &proc } proc.call result when Hash result.each { |x, y| recurse_proc x, &proc; recurse_proc y, &proc } proc.call result else proc.call result end end private :recurse_proc module_function :recurse_proc alias restore load module_function :restore # Dumps _obj_ as a PSON string, i.e. calls generate on the object and returns # the result. # # If anIO (an IO like object or an object that responds to the write method) # was given, the resulting PSON is written to it. # # If the number of nested arrays or objects exceeds _limit_ an ArgumentError # exception is raised. This argument is similar (but not exactly the # same!) to the _limit_ argument in Marshal.dump. # # This method is part of the implementation of the load/dump interface of # Marshal and YAML. def dump(obj, anIO = nil, limit = nil) if anIO and limit.nil? anIO = anIO.to_io if anIO.respond_to?(:to_io) unless anIO.respond_to?(:write) limit = anIO anIO = nil end end limit ||= 0 result = generate(obj, :allow_nan => true, :max_nesting => limit) if anIO anIO.write result anIO else result end rescue PSON::NestingError raise ArgumentError, "exceed depth limit", $!.backtrace end # Provide a smarter wrapper for changing string encoding that works with # both Ruby 1.8 (iconv) and 1.9 (String#encode). Thankfully they seem to # have compatible input syntax, at least for the encodings we touch. if String.method_defined?("encode") def encode(to, from, string) string.encode(to, from) end else require 'iconv' def encode(to, from, string) Iconv.conv(to, from, string) end end end module ::Kernel private # Outputs _objs_ to STDOUT as PSON strings in the shortest form, that is in # one line. def j(*objs) objs.each do |obj| puts PSON::generate(obj, :allow_nan => true, :max_nesting => false) end nil end # Outputs _objs_ to STDOUT as PSON strings in a pretty format, with # indentation and over many lines. def jj(*objs) objs.each do |obj| puts PSON::pretty_generate(obj, :allow_nan => true, :max_nesting => false) end nil end # If _object_ is string-like parse the string and return the parsed result as # a Ruby data structure. Otherwise generate a PSON text from the Ruby data # structure object and return it. # # The _opts_ argument is passed through to generate/parse respectively, see # generate and parse for their documentation. def PSON(object, opts = {}) if object.respond_to? :to_str PSON.parse(object.to_str, opts) else PSON.generate(object, opts) end end end class ::Class # Returns true, if this class can be used to create an instance # from a serialised PSON string. The class has to implement a class # method _pson_create_ that expects a hash as first parameter, which includes # the required data. def pson_creatable? respond_to?(:pson_create) end end octocatalog-diff-1.5.4/lib/octocatalog-diff/cli.rb0000644000004100000410000002216613413744025022037 0ustar www-datawww-data# frozen_string_literal: true require_relative 'api/v1' require_relative 'catalog-util/cached_master_directory' require_relative 'cli/diffs' require_relative 'cli/options' require_relative 'cli/printer' require_relative 'errors' require_relative 'util/catalogs' require_relative 'util/util' require_relative 'version' require 'logger' require 'parallel' require 'socket' module OctocatalogDiff # This is the CLI for catalog-diff. It's responsible for parsing the command line # arguments and then handing off to appropriate methods to perform the catalog-diff. class Cli # Version number VERSION = OctocatalogDiff::Version::VERSION # Exit codes EXITCODE_SUCCESS_NO_DIFFS = 0 EXITCODE_FAILURE = 1 EXITCODE_SUCCESS_WITH_DIFFS = 2 # The default type+title+attribute to ignore in catalog-diff. DEFAULT_IGNORES = [ { type: 'Class' } # Don't care about classes themselves, only what they actually do! ].freeze # The default options. DEFAULT_OPTIONS = { from_env: 'origin/master', to_env: '.', colors: true, debug: false, quiet: false, format: :color_text, display_source_file_line: false, compare_file_text: true, display_datatype_changes: true, parallel: true, suppress_absent_file_details: true, hiera_path: 'hieradata' }.freeze # This method is the one to call externally. It is possible to specify alternate # command line arguments, for testing. # @param argv [Array] Use specified arguments (defaults to ARGV) # @param logger [Logger] Logger object # @param opts [Hash] Additional options # @return [Integer] Exit code: 0=no diffs, 1=something went wrong, 2=worked but there are diffs def self.cli(argv = ARGV, logger = Logger.new(STDERR), opts = {}) # Save a copy of argv to print out later in debugging argv_save = OctocatalogDiff::Util::Util.deep_dup(argv) # Are there additional ARGV to munge, e.g. that have been supplied in the options from a # configuration file? if opts.key?(:additional_argv) raise ArgumentError, ':additional_argv must be array!' unless opts[:additional_argv].is_a?(Array) argv.concat opts[:additional_argv] end # Parse command line options = parse_opts(argv) # Additional options from hard-coded specified options. These are only processed if # there are not already values defined from command line options. # Note: do NOT use 'options[k] ||= v' here because if the value of options[k] is boolean(false) # it will then be overridden. Whereas the intent is to define values only for those keys that don't exist. opts.each { |k, v| options[k] = v unless options.key?(k) } veto_options = %w(enc header include_tags) veto_options.each { |x| options.delete(x.to_sym) if options["no_#{x}".to_sym] } if options[:no_hiera_config] vetoes = %w[hiera_config to_hiera_config from_hiera_config] vetoes.each do |key| options.delete(key.to_sym) end end options[:ignore].concat opts.fetch(:additional_ignores, []) # Incorporate default options where needed. # Note: do NOT use 'options[k] ||= v' here because if the value of options[k] is boolean(false) # it will then be overridden. Whereas the intent is to define values only for those keys that don't exist. DEFAULT_OPTIONS.each { |k, v| options[k] = v unless options.key?(k) } veto_with_none_options = %w(hiera_path hiera_path_strip) veto_with_none_options.each { |x| options.delete(x.to_sym) if options[x.to_sym] == :none } # Fact and ENC overrides come in here - 'options' is modified setup_fact_overrides(options) setup_enc_overrides(options) # Configure the logger and logger.debug initial information # 'logger' is modified and used setup_logger(logger, options, argv_save) # --catalog-only is a special case that compiles the catalog for the "to" branch # and then exits, without doing any 'diff' whatsoever. Support that option. return catalog_only(logger, options) if options[:catalog_only] # Set up the cached master directory - maintain it, adjust options if needed. However, if we # are getting the 'from' catalog from PuppetDB, then don't do this. unless options[:cached_master_dir].nil? || options[:from_puppetdb] OctocatalogDiff::CatalogUtil::CachedMasterDirectory.run(options, logger) end # bootstrap_then_exit is a special case that only prepares directories and does not # depend on facts. This happens within the 'catalogs' object, since bootstrapping and # preparing catalogs are tightly coupled operations. However this does not actually # build catalogs. if options[:bootstrap_then_exit] catalogs_obj = OctocatalogDiff::Util::Catalogs.new(options, logger) return bootstrap_then_exit(logger, catalogs_obj) end # Compile catalogs and do catalog-diff node_set = options.delete(:node) node_set = [node_set] unless node_set.is_a?(Array) # run multiple node diffs in parallel catalog_diffs = if node_set.size == 1 [run_octocatalog_diff(node_set.first, options, logger)] else ::Parallel.map(node_set, in_threads: 4) { |node| run_octocatalog_diff(node, options, logger) } end # Return the resulting diff object if requested (generally for testing) # or otherwise return exit code return catalog_diffs.first if opts[:INTEGRATION] all_diffs = catalog_diffs.map(&:diffs) all_diffs.each do |diff| next unless diff.any? return EXITCODE_SUCCESS_WITH_DIFFS end EXITCODE_SUCCESS_NO_DIFFS end # Run the octocatalog-diff process for a given node. Return the diffs for a contribution to # the final exit status. # node - String with the node # options - All of the currently defined options # logger - Logger object def self.run_octocatalog_diff(node, options, logger) options_copy = options.merge(node: node) catalog_diff = OctocatalogDiff::API::V1.catalog_diff(options_copy.merge(logger: logger)) diffs = catalog_diff.diffs # Display diffs printer_obj = OctocatalogDiff::Cli::Printer.new(options_copy, logger) printer_obj.printer(diffs, catalog_diff.from.compilation_dir, catalog_diff.to.compilation_dir) # Return catalog-diff object. catalog_diff end # Parse command line options with 'optparse'. Returns a hash with the parsed arguments. # @param argv [Array] Command line arguments (MUST be specified) # @return [Hash] Options def self.parse_opts(argv) options = { ignore: OctocatalogDiff::Util::Util.deep_dup(DEFAULT_IGNORES) } Options.parse_options(argv, options) end # Generic overrides def self.setup_overrides(key, options) o = options["#{key}_in".to_sym] return unless o.is_a?(Array) return unless o.any? options[key] ||= [] options[key].concat o.map { |x| OctocatalogDiff::API::V1::Override.create_from_input(x) } end # Fact overrides come in here def self.setup_fact_overrides(options) setup_overrides(:from_fact_override, options) setup_overrides(:to_fact_override, options) end # ENC parameter overrides come in here def self.setup_enc_overrides(options) setup_overrides(:from_enc_override, options) setup_overrides(:to_enc_override, options) end # Helper method: Configure and setup logger def self.setup_logger(logger, options, argv_save) # Configure the logger logger.level = Logger::INFO logger.level = Logger::DEBUG if options[:debug] logger.level = Logger::ERROR if options[:quiet] # Some debugging information up front version_display = ENV['OCTOCATALOG_DIFF_CUSTOM_VERSION'] || VERSION logger.debug "Running octocatalog-diff #{version_display} with ruby #{RUBY_VERSION}" logger.debug "Command line arguments: #{argv_save.inspect}" logger.debug "Running on host #{Socket.gethostname} (#{RUBY_PLATFORM})" end # Compile the catalog only def self.catalog_only(logger, options) opts = options.merge(logger: logger) to_catalog = OctocatalogDiff::API::V1.catalog(opts) # If the catalog compilation failed, an exception would have been thrown. So if # we get here, the catalog succeeded. Dump the catalog to the appropriate place # and exit successfully. if options[:output_file] File.open(options[:output_file], 'w') { |f| f.write(to_catalog.to_json) } logger.info "Wrote catalog to #{options[:output_file]}" else puts to_catalog.to_json end return { exitcode: EXITCODE_SUCCESS_NO_DIFFS, to: to_catalog } if options[:INTEGRATION] # For integration testing EXITCODE_SUCCESS_NO_DIFFS end # --bootstrap-then-exit command def self.bootstrap_then_exit(logger, catalogs_obj) catalogs_obj.bootstrap_then_exit return EXITCODE_SUCCESS_NO_DIFFS rescue OctocatalogDiff::Errors::BootstrapError => exc logger.fatal("--bootstrap-then-exit error: bootstrap failed (#{exc})") return EXITCODE_FAILURE end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/puppetdb.rb0000644000004100000410000001745513413744025023120 0ustar www-datawww-data# frozen_string_literal: true require_relative 'errors' require_relative 'util/httparty' require 'uri' module OctocatalogDiff # A standard way to connect to PuppetDB from the various scripts in this repository. class PuppetDB DEFAULT_HTTPS_PORT = 8081 DEFAULT_HTTP_PORT = 8080 # Allow connections to be read (used in tests for now) attr_reader :connections # Constructor - will construct connection parameters from a variety # of sources, including arguments and environment variables. Supported # environment variables: # PUPPETDB_URL # PUPPETDB_HOST [+ PUPPETDB_PORT] [+ PUPPETDB_SSL] # # Order of precedence: # 1. :puppetdb_url argument (String or Array) # 2. :puppetdb_host argument [+ :puppetdb_port] [+ :puppetdb_ssl] # 3. ENV['PUPPETDB_URL'] # 4. ENV['PUPPETDB_HOST'] [+ ENV['PUPPETDB_PORT']], [+ ENV['PUPPETDB_SSL']] # When it finds one of these, it stops and does not process any others. # # When :puppetdb_url is an array, all given URLs are tried, in random order, # until a connection succeeds. If a connection succeeds, any errors from previously # failed connections are suppressed. # # Supported arguments: # @param :puppetdb_url [String or Array] PuppetDB URL(s) to try in random order # @param :puppetdb_host [String] PuppetDB hostname, when constructing a URL # @param :puppetdb_port [Integer] Port number, defaults to 8080 (non-SSL) or 8081 (SSL) # @param :puppetdb_ssl [Boolean] defaults to true, because you should use SSL # @param :puppetdb_ssl_ca [String] Path to file containing CA certificate # @param :puppetdb_ssl_verify [Boolean] Override the CA verification setting guessed from parameters # @param :puppetdb_ssl_client_pem [String] PEM-encoded client key and certificate # @param :puppetdb_ssl_client_p12 [String] pkcs12-encoded client key and certificate # @param :puppetdb_ssl_client_password [String] Path to file containing password for SSL client key (any format) # @param :puppetdb_ssl_client_auth [Boolean] Override the client-auth that is guessed from parameters # @param :puppetdb_token [String] PE RBAC token to authenticate to PuppetDB API # @param :timeout [Integer] Connection timeout for PuppetDB (default=10) def initialize(options = {}) @connections = if options.key?(:puppetdb_url) urls = options[:puppetdb_url].is_a?(Array) ? options[:puppetdb_url] : [options[:puppetdb_url]] urls.map { |url| parse_url(url) } elsif options.key?(:puppetdb_host) is_ssl = options.fetch(:puppetdb_ssl, true) default_port = is_ssl ? DEFAULT_HTTPS_PORT : DEFAULT_HTTP_PORT port = options.fetch(:puppetdb_port, default_port).to_i [{ ssl: is_ssl, host: options[:puppetdb_host], port: port }] elsif ENV['PUPPETDB_URL'] && !ENV['PUPPETDB_URL'].empty? [parse_url(ENV['PUPPETDB_URL'])] elsif ENV['PUPPETDB_HOST'] && !ENV['PUPPETDB_HOST'].empty? # Because environment variables are strings... # This will get the env var and see if it equals 'true'; the result # of this == comparison is the true/false boolean we need. is_ssl = ENV.fetch('PUPPETDB_SSL', 'true') == 'true' default_port = is_ssl ? DEFAULT_HTTPS_PORT : DEFAULT_HTTP_PORT port = ENV.fetch('PUPPETDB_PORT', default_port).to_i [{ ssl: is_ssl, host: ENV['PUPPETDB_HOST'], port: port }] else [] end @timeout = options.fetch(:timeout, 10) @options = options end # Wrapper around the httparty call in the private _get method. # Returns the parsed result of getting the provided URL and returns # a friendlier error message if there are network connection problems # to PuppetDB. # @param path [String] Path portion of the URL # @return [Object] Parsed reply from PuppetDB as an object def get(path) _get(path) rescue Net::OpenTimeout, Errno::ECONNREFUSED => exc raise OctocatalogDiff::Errors::PuppetDBConnectionError, "#{exc.class} connecting to PuppetDB (need VPN on?): #{exc.message}" end private # HTTP(S) Query - will attempt to retrieve URL from each connection # @param path [String] Path portion of the URL # @return [String] Parsed response def _get(path) # You need at least one connection or else this can't do anything raise ArgumentError, 'No PuppetDB connections configured' if @connections.empty? # Keep track of the latest exception seen exc = nil # Try each connection in random order. This will return the first successful # response, and try the next connection if there's an error. Once it's out of # connections to try it will raise the last exception encountered. @connections.shuffle.each do |connection| complete_url = [ connection[:ssl] ? 'https://' : 'http://', connection[:host], ':', connection[:port], path ].join('') begin headers = { 'Accept' => 'application/json' } headers['X-Authentication'] = @options[:puppetdb_token] if @options[:puppetdb_token] more_options = { headers: headers, timeout: @timeout } if connection[:username] || connection[:password] more_options[:basic_auth] = { username: connection[:username], password: connection[:password] } end response = OctocatalogDiff::Util::HTTParty.get(complete_url, @options.merge(more_options), 'puppetdb') # Handle all non-200's from PuppetDB unless response[:code] == 200 raise OctocatalogDiff::Errors::PuppetDBNodeNotFoundError, "404 - #{response[:error]}" if response[:code] == 404 raise OctocatalogDiff::Errors::PuppetDBGenericError, "#{response[:code]} - #{response[:error]}" end # PuppetDB can return 'Not Found' as a string with a 200 response code raise NotFoundError, '404 - Not Found' if response[:body] == 'Not Found' # PuppetDB can also return an error message in a 200; we'll call this a 500 if response.key?(:error) raise OctocatalogDiff::Errors::PuppetDBGenericError, "500 - #{response[:error]}" end # If we get here without raising an error, it will fall out of the begin/rescue # with 'result' non-nil, and 'result' will then get returned. raise "Unparseable response from puppetdb: '#{response.inspect}'" unless response[:parsed] result = response[:parsed] rescue => exc # Set response to nil so the loop repeats itself if there are retries left. # Also sets 'exc' to the most recent exception, in case all retries are # exhausted and this exception has to be raised. result = nil end # If the previous query didn't error, return result return result unless result.nil? end # At this point no query has succeeded, so raise the last error encountered. raise exc end # Parse a URL to determine hostname, port number, and whether or not SSL is used. # @param url [String] URL to parse # @return [Hash] { ssl: true/false, host: , port: } def parse_url(url) uri = URI(url) if URI.split(url)[3].nil? uri.port = uri.scheme == 'https' ? DEFAULT_HTTPS_PORT : DEFAULT_HTTP_PORT end raise ArgumentError, "URL #{url} has invalid scheme" unless uri.scheme =~ /^https?$/ parsed_url = { ssl: uri.scheme == 'https', host: uri.host, port: uri.port } if uri.user || uri.password parsed_url[:username] = uri.user parsed_url[:password] = uri.password end parsed_url rescue URI::InvalidURIError => exc raise exc.class, "Invalid URL: #{url} (#{exc.message})" end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/facts.rb0000644000004100000410000001123613413744025022364 0ustar www-datawww-data# frozen_string_literal: true require_relative 'errors' require_relative 'facts/json' require_relative 'facts/yaml' require_relative 'facts/puppetdb' require_relative 'util/util' require_relative 'external/pson/pure' module OctocatalogDiff # Deal with facts in all forms, including: # - In existing YAML files # - In existing JSON files # - Retrieved dynamically from PuppetDB class Facts # Constructor # @param options [Hash] Initialization options, varies per backend def initialize(options = {}, facts = nil) @node = options.fetch(:node, '') @timestamp = false @options = options.dup if facts @facts = OctocatalogDiff::Util::Util.deep_dup(facts) else case options[:backend] when :json @orig_facts = OctocatalogDiff::Facts::JSON.fact_retriever(options, @node) when :yaml @orig_facts = OctocatalogDiff::Facts::Yaml.fact_retriever(options, @node) when :puppetdb @orig_facts = OctocatalogDiff::Facts::PuppetDB.fact_retriever(options, @node) else raise ArgumentError, 'Invalid fact source backend' end @facts = OctocatalogDiff::Util::Util.deep_dup(@orig_facts) end end def dup self.class.new(@options, @orig_facts) end # Node - get the node name, either as set explicitly or as determined from the facts themselves. # @return [String] Node name, explicit or guessed def node return @node unless @node.nil? || @node.empty? return facts['name'] if facts.key?('name') return facts['values']['fqdn'] if facts.key?('values') && facts['values'].key?('fqdn') '' end # Facts - returned the 'cleansed' facts. # Clean up facts by setting 'name' to the node if given, and deleting _timestamp and expiration # which may cause Puppet catalog compilation to fail if the facts are old. # @param node [String] Node name to override returned facts # @return [Hash] Facts hash { 'name' => '...', 'values' => { ... } } def facts(node = @node, timestamp = false) raise "Expected @facts to be a hash but it is a #{@facts.class}" unless @facts.is_a?(Hash) raise "Expected @facts['values'] to be a hash but it is a #{@facts['values'].class}" unless @facts['values'].is_a?(Hash) f = @facts.dup f['name'] = node unless node.nil? || node.empty? f['values'].delete('_timestamp') f.delete('expiration') if timestamp f['timestamp'] = Time.now.to_s f['values']['timestamp'] = f['timestamp'] f['expiration'] = (Time.now + (24 * 60 * 60)).to_s end f end # Facts - Fudge the timestamp to right now and add include it in the facts when returned # @return self def fudge_timestamp @timestamp = true self end # Facts - remove one or more facts from the list. # @param remove [String|Array] Fact(s) to remove # @return self def without(remove) r = remove.is_a?(Array) ? remove : [remove] obj = dup r.each { |fact| obj.remove_fact_from_list(fact) } obj end # Facts - remove a fact from the list # @param remove [String] Fact to remove def remove_fact_from_list(remove) @facts['values'].delete(remove) end # Turn hash of facts into appropriate YAML for Puppet # @param node [String] Node name to override returned facts # @return [String] Puppet-compatible YAML facts def facts_to_yaml(node = @node) # Add the header that Puppet needs to treat this as facts. Save the results # as a string in the option. f = facts(node) fact_file = f.to_yaml.split(/\n/) fact_file[0] = '--- !ruby/object:Puppet::Node::Facts' if fact_file[0] =~ /^---/ fact_file.join("\n") end # Turn hash of facts into appropriate YAML for Puppet # @param node [String] Node name to override returned facts # @return [String] Puppet-compatible YAML facts def to_pson PSON.generate(facts) end # Get the current value of a particular fact # @param key [String] Fact key to override # @return [?] Value for fact def fact(key) @facts['values'][key] end # Override a particular fact # @param key [String] Fact key to override # @param value [?] Value for fact def override(key, value) if value.nil? @facts['values'].delete(key) else @facts['values'][key] = value end end # Find all facts matching a particular pattern # @param regex [Regexp] Regular expression to match # @return [Array] Facts that match the regexp def matching(regex) @facts['values'].keys.select { |fact| regex.match(fact) } end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/catalog.rb0000644000004100000410000003464313413744025022705 0ustar www-datawww-data# frozen_string_literal: true require 'json' require 'stringio' require_relative 'catalog/computed' require_relative 'catalog/json' require_relative 'catalog/noop' require_relative 'catalog/puppetdb' require_relative 'catalog/puppetmaster' require_relative 'catalog-util/fileresources' require_relative 'errors' module OctocatalogDiff # Basic methods for interacting with a catalog. Generation of the catalog is handled via one of the # supported backends listed above as 'require_relative'. Usually, the 'computed' backend # will build the catalog from the Puppet command. class Catalog attr_accessor :node attr_reader :built, :catalog, :catalog_json, :options # Constructor def initialize(options = {}) unless options.is_a?(Hash) raise ArgumentError, "#{self.class}.initialize requires hash argument, not #{options.class}" end @options = options # Basic settings @node = options[:node] @error_message = nil @catalog = nil @catalog_json = nil @retries = nil # The compilation directory can be overridden, e.g. when testing @override_compilation_dir = options[:compilation_dir] # Keep track of whether references have been validated yet. Allow this to be fudged for when we do # not desire reference validation to happen (e.g., for the "from" catalog that is otherwise valid). @references_validated = options[:references_validated] || false # Keep track of whether file resources have been converted. @file_resources_converted = false # Keep track of whether it's built yet @built = false end # Guess the backend from the input and return the appropriate catalog object. # @param :backend [Symbol] If set, this will force a backend # @param :json [String] JSON catalog content (will avoid running Puppet to compile catalog) # @param :puppetdb [Object] If set, pull the catalog from PuppetDB rather than building # @param :node [String] Name of node whose catalog is being built # @param :fact_file [String] OPTIONAL: Path to fact file (if not provided, look up in PuppetDB) # @param :hiera_config [String] OPTIONAL: Path to hiera config file (munge temp. copy if not provided) # @param :basedir [String] OPTIONAL: Base directory for catalog (default base directory of this checkout) # @param :pass_env_vars [Array] OPTIONAL: Additional environment vars to pass # @param :convert_file_resources [Boolean] OPTIONAL: Convert file resource source to content # @param :storeconfigs [Boolean] OPTIONAL: Pass the '-s' flag, for puppetdb (storeconfigs) integration # @return [OctocatalogDiff::Catalog::] Catalog object from guessed backend def self.create(options = {}) # Hard-coded backend if options[:backend] return OctocatalogDiff::Catalog::JSON.new(options) if options[:backend] == :json return OctocatalogDiff::Catalog::PuppetDB.new(options) if options[:backend] == :puppetdb return OctocatalogDiff::Catalog::PuppetMaster.new(options) if options[:backend] == :puppetmaster return OctocatalogDiff::Catalog::Computed.new(options) if options[:backend] == :computed return OctocatalogDiff::Catalog::Noop.new(options) if options[:backend] == :noop raise ArgumentError, "Unknown backend :#{options[:backend]}" end # Determine backend based on arguments return OctocatalogDiff::Catalog::JSON.new(options) if options[:json] return OctocatalogDiff::Catalog::PuppetDB.new(options) if options[:puppetdb] return OctocatalogDiff::Catalog::PuppetMaster.new(options) if options[:puppet_master] # Default is to build catalog ourselves OctocatalogDiff::Catalog::Computed.new(options) end # Build catalog - this method needs to be called to build the catalog. It is separate due to # the serialization of the logger object -- the parallel gem cannot serialize/deserialize a logger # object so it cannot be part of any object that is passed around. # @param logger [Logger] Logger object, initialized to a default throwaway value def build(logger = Logger.new(StringIO.new)) # If already built, don't build again return if @built @built = true # The resource hash is computed the first time it's needed. For now initialize it as nil. @resource_hash = nil # Invoke the backend's build method, if there is one. There's a stub below in case there's not. logger.debug "Calling build for object #{self.class}" build_catalog(logger) # Perform post-generation processing of the catalog return unless valid? validate_references return unless valid? convert_file_resources if @options[:compare_file_text] true end # Stub method if the backend does not contain a build method. def build_catalog(_logger) end # Compilation environment # @return [String] Compilation environment (if set), else 'production' by default def environment @environment ||= 'production' end # For logging we may wish to know the backend being used # @return [String] Class of backend used def builder self.class.to_s end # Set the catalog JSON # @param str [String] Catalog JSON def catalog_json=(str) @catalog_json = str @resource_hash = nil end # This retrieves the compilation directory from the catalog, or otherwise the passed-in directory. # @return [String] Compilation directory def compilation_dir @override_compilation_dir || @options[:basedir] end # The compilation directory can be overridden, e.g. during testing. # @param dir [String] Compilation directory def compilation_dir=(dir) @override_compilation_dir = dir end # Stub method for "convert_file_resources" -- returns false because if the underlying class does # not implement this method, it's not supported. def convert_file_resources(_dry_run = false) false end # Retrieve the error message. # @return [String] Error message (maximum 20,000 characters) - nil if no error. def error_message build return nil if @error_message.nil? || !@error_message.is_a?(String) @error_message[0, 20_000] end # Allow setting the error message. If the error message is set to a string, the catalog # and catalog JSON are set to nil. # @param error [String] Error message def error_message=(error) raise ArgumentError, 'Error message must be a string' unless error.is_a?(String) @error_message = error @catalog = nil @catalog_json = nil @resource_hash = nil end # Stub method to return the puppet version if the back end doesn't support this. # @return [String] Puppet version def puppet_version build @options[:puppet_version] end # This allows retrieving a resource by type and title. This is intended for use when a O(1) lookup is required. # @param :type [String] Type of resource # @param :title [String] Title of resource # @return [Hash] Resource item def resource(opts = {}) raise ArgumentError, ':type and :title are required' unless opts[:type] && opts[:title] build build_resource_hash if @resource_hash.nil? return nil unless @resource_hash[opts[:type]].is_a?(Hash) @resource_hash[opts[:type]][opts[:title]] end # This is a compatibility layer for the resources, which are in a different place in Puppet 3.x and Puppet 4.x # @return [Array] Resource array def resources build raise OctocatalogDiff::Errors::CatalogError, 'Catalog does not appear to have been built' if !valid? && error_message.nil? raise OctocatalogDiff::Errors::CatalogError, error_message unless valid? return @catalog['data']['resources'] if @catalog['data'].is_a?(Hash) && @catalog['data']['resources'].is_a?(Array) return @catalog['resources'] if @catalog['resources'].is_a?(Array) # This is a bug condition # :nocov: raise "BUG: catalog has no data::resources or ::resources array. Please report this. #{@catalog.inspect}" # :nocov: end # Stub method of the the number of retries necessary to compile the catalog. If the underlying catalog # generation backend does not support retries, nil is returned. # @return [Integer] Retry count def retries nil end # Determine if the catalog build was successful. # @return [Boolean] Whether the catalog is valid def valid? build !@catalog.nil? end # Determine if all of the (before, notify, require, subscribe) targets are actually in the catalog. # Raise a OctocatalogDiff::Errors::ReferenceValidationError for any found to be missing. # Uses @options[:validate_references] to influence which references are checked. def validate_references # If we've already done the validation, don't do it again return if @references_validated @references_validated = true # Skip out early if no reference validation has been requested. unless @options[:validate_references].is_a?(Array) && @options[:validate_references].any? return end # Puppet 5 has reference validation built-in and enabled, so there won't even be a valid catalog if # there were invalid references. It's pointless to perform validation of our own. return if puppet_version && puppet_version >= '5.0.0' # Iterate over all the resources and check each one that has one of the attributes being checked. # Keep track of all references that are missing for ultimate inclusion in the error message. missing = [] resources.each do |x| @options[:validate_references].each do |r| next unless x.key?('parameters') next unless x['parameters'].key?(r) missing_resources = resources_missing_from_catalog(x['parameters'][r]) next unless missing_resources.any? missing.concat missing_resources.map { |missing_target| { source: x, target_type: r, target_value: missing_target } } end end return if missing.empty? # At this point there is at least one broken/missing reference. Format an error message and raise. errors = format_missing_references(missing) plural = errors =~ /;/ ? 's' : '' self.error_message = "Catalog has broken reference#{plural}: #{errors}" end private # Private method: Format the name of the source file and line number, based on compilation directory and # other settings. This is used by format_missing_references. # @param source_file [String] Raw source file name from catalog # @param line_number [Fixnum] Line number from catalog # @return [String] Formatted source file def format_source_file_line(source_file, line_number) return '' if source_file.nil? || source_file.empty? filename = if compilation_dir && source_file.start_with?(compilation_dir) stripped_file = source_file[compilation_dir.length..-1] stripped_file.start_with?('/') ? stripped_file[1..-1] : stripped_file else source_file end "(#{filename.sub(%r{^environments/production/}, '')}:#{line_number})" end # Private method: Format the missing references into human-readable text # Error message will look like this: # --- # Catalog has broken references: exec[subscribe caller 1](file:line) -> subscribe[Exec[subscribe target]]; # exec[subscribe caller 2](file:line) -> subscribe[Exec[subscribe target]]; exec[subscribe caller 2](file:line) -> # subscribe[Exec[subscribe target 2]] # --- # @param missing [Array] Array of missing references # @return [String] Formatted references def format_missing_references(missing) unless missing.is_a?(Array) && missing.any? raise ArgumentError, 'format_missing_references() requires a non-empty array as input' end formatted_references = missing.map do |obj| # obj[:target_value] can be a string or an array. If it's an array, create a # separate error message per element of that array. This allows the total number # of errors to be correct. src_ref = "#{obj[:source]['type'].downcase}[#{obj[:source]['title']}]" src_file = format_source_file_line(obj[:source]['file'], obj[:source]['line']) target_val = obj[:target_value].is_a?(Array) ? obj[:target_value] : [obj[:target_value]] target_val.map { |tv| "#{src_ref}#{src_file} -> #{obj[:target_type].downcase}[#{tv}]" } end.flatten formatted_references.join('; ') end # Private method: Given a list of resources to check, return the references from # that list that are missing from the catalog. (An empty array returned would indicate # all references are present in the catalog.) # @param resources_to_check [String / Array] Resources to check # @return [Array] References that are missing from catalog def resources_missing_from_catalog(resources_to_check) [resources_to_check].flatten.select do |res| unless res =~ /\A([\w:]+)\[(.+)\]\z/ raise ArgumentError, "Resource #{res} is not in the expected format" end type = Regexp.last_match(1) title = normalized_title(Regexp.last_match(2), type) resource(type: type, title: title).nil? end end # Private method: Given a title string, normalize it according to the rules # used by puppet 4.10.x for file resource title normalization: # https://github.com/puppetlabs/puppet/blob/4.10.x/lib/puppet/type/file.rb#L42 def normalized_title(title_string, type) return title_string if type != 'File' matches = title_string.match(%r{^(?/|.+:/|.*[^/])/*\Z}m) matches[:normalized_path] || title_string end # Private method: Build the resource hash to be used used for O(1) lookups by type and title. # This method is called the first time the resource hash is accessed. def build_resource_hash @resource_hash = {} resources.each do |resource| @resource_hash[resource['type']] ||= {} title = normalized_title(resource['title'], resource['type']) @resource_hash[resource['type']][title] = resource if resource.key?('parameters') @resource_hash[resource['type']][resource['parameters']['alias']] = resource if resource['parameters'].key?('alias') @resource_hash[resource['type']][resource['parameters']['name']] = resource if resource['parameters'].key?('name') end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/api/0000755000004100000410000000000013413744025021505 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/api/v1/0000755000004100000410000000000013413744025022033 5ustar www-datawww-dataoctocatalog-diff-1.5.4/lib/octocatalog-diff/api/v1/catalog-compile.rb0000644000004100000410000000255013413744025025422 0ustar www-datawww-data# frozen_string_literal: true require_relative 'catalog' require_relative 'common' require_relative '../../util/catalogs' module OctocatalogDiff module API module V1 # This class allows octocatalog-diff to be used to compile catalogs. class CatalogCompile # Public: Compile a catalog given the options provided. # # Parameters are to be passed in a hash. # @param :logger [Logger] Logger object (be sure to configure log level) # @param :node [String] Node name (FQDN) # Other catalog building parameters are also accepted # @return [OctocatalogDiff::Catalog] Compiled catalogs def self.catalog(options = nil) # Validate the required options. unless options.is_a?(Hash) raise ArgumentError, 'Usage: #catalog(options_hash)' end pass_opts, logger = OctocatalogDiff::API::V1::Common.logger_from_options(options) logger.debug "Compiling catalog for #{options[:node]}" # Compile catalog catalog_opts = pass_opts.merge( from_catalog: '-', # Prevents a compile to_catalog: nil, # Forces a compile ) cat_obj = OctocatalogDiff::Util::Catalogs.new(catalog_opts, logger) OctocatalogDiff::API::V1::Catalog.new(cat_obj.catalogs[:to]) end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/api/v1/catalog-diff.rb0000644000004100000410000000547413413744025024712 0ustar www-datawww-data# frozen_string_literal: true require_relative 'catalog' require_relative 'common' require_relative 'diff' require_relative '../../util/catalogs' require_relative '../../catalog-util/cached_master_directory' require 'ostruct' module OctocatalogDiff module API module V1 # This class allows octocatalog-diff to be used to compile catalogs (if needed) # and then compute the differences between them. class CatalogDiff # Public: Run catalog-diff # # Parameters are to be passed in a hash. # @param :logger [Logger] Logger object (be sure to configure log level) # Other catalog-diff parameters are required # @return [OpenStruct] { :diffs (Array); :from (OctocatalogDiff::Catalog), :to (OctocatalogDiff::Catalog) } def self.catalog_diff(options = nil) # Validate the required options. unless options.is_a?(Hash) raise ArgumentError, 'Usage: #catalog_diff(options_hash)' end pass_opts, logger = OctocatalogDiff::API::V1::Common.logger_from_options(options) # Compile catalogs logger.debug "Compiling catalogs for #{options[:node]}" catalogs_obj = OctocatalogDiff::Util::Catalogs.new(pass_opts, logger) catalogs = catalogs_obj.catalogs logger.info "Catalogs compiled for #{options[:node]}" # Cache catalogs if master caching is enabled. If a catalog is being read from the cached master # directory, set the compilation directory attribute, so that the "compilation directory dependent" # suppressor will still work. %w(from to).each do |x| next unless options["#{x}_env".to_sym] == options.fetch(:master_cache_branch, 'origin/master') next if options[:cached_master_dir].nil? catalogs[x.to_sym].compilation_dir = options["#{x}_catalog_compilation_dir".to_sym] || options[:cached_master_dir] rc = OctocatalogDiff::CatalogUtil::CachedMasterDirectory.save_catalog_in_cache_dir( options[:node], options[:cached_master_dir], catalogs[x.to_sym] ) logger.debug "Cached master catalog for #{options[:node]}" if rc end # Compute diffs diffs_obj = OctocatalogDiff::Cli::Diffs.new(options, logger) diffs = diffs_obj.diffs(catalogs) logger.info "Diffs computed for #{options[:node]}" logger.info 'No differences' if diffs.empty? # Return diffs and catalogs in expected format OpenStruct.new( diffs: diffs.map { |x| OctocatalogDiff::API::V1::Diff.factory(x) }, from: OctocatalogDiff::API::V1::Catalog.new(catalogs[:from]), to: OctocatalogDiff::API::V1::Catalog.new(catalogs[:to]) ) end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/api/v1/diff.rb0000644000004100000410000001407613413744025023300 0ustar www-datawww-data# frozen_string_literal: true require_relative 'common' module OctocatalogDiff module API module V1 # This class represents a `diff` produced by a catalog-diff operation. This has traditionally # been stored as an array with: # [0] Type of change - '+', '-', '!', '~' # [1] Type, title, and maybe structure, delimited by "\f" # [2] Content of the "old" catalog # [3] Content of the "new" catalog # [4] File and line of the "old" catalog # [5] File and line of the "new" catalog # This object seeks to preserve this traditional structure, while providing methods to make it # easier to deal with. We recommend using the named options, rather than #raw or the indexed array, # as the raw object and indexed array are not guaranteed to be stable. class Diff attr_reader :raw, :diff_type, :type, :title, :structure # Public: Construct a OctocatalogDiff::API::V1::Diff object from many different types of # input. This includes passing a OctocatalogDiff::API::V1::Diff object and getting that # identical object back. # @param object_in [?] Object in # @return [OctocatalogDiff::API::V1::Diff] Object out def self.factory(object_in) return object_in if object_in.is_a?(OctocatalogDiff::API::V1::Diff) return new(object_in) if object_in.is_a?(Array) raise ArgumentError, "Cannot construct OctocatalogDiff::API::V1::Diff from #{object_in.class}" end # Constructor: Accepts a diff in the traditional array format and stores it. # @param raw [Array] Diff in the traditional format def initialize(raw) unless raw.is_a?(Array) raise ArgumentError, "OctocatalogDiff::API::V1::Diff#initialize expects Array argument (got #{raw.class})" end @raw = raw initialize_helper end # Public: Retrieve an indexed value from the array # @return [?] Indexed value def [](i) @raw[i] end # Public: Set an element of the array # @param [new_value] The value to set it to def []=(i, new_value) @raw[i] = new_value end # Public: Is this an addition? # @return [Boolean] True if this is an addition def addition? diff_type == '+' end # Public: Is this a removal? # @return [Boolean] True if this is an addition def removal? diff_type == '-' end # Public: Is this a change? # @return [Boolean] True if this is an change def change? diff_type == '~' || diff_type == '!' end # Public: Get the "old" value, i.e. "from" catalog # @return [?] "old" value def old_value return if addition? @raw[2] end # Public: Get the "new" value, i.e. "to" catalog # @return [?] "new" value def new_value return if removal? return @raw[2] if addition? @raw[3] end # Public: Get the filename from the "old" location # @return [String] Filename def old_file x = old_location x.nil? ? nil : x['file'] end # Public: Get the line number from the "old" location # @return [String] Line number def old_line x = old_location x.nil? ? nil : x['line'] end # Public: Get the filename from the "new" location # @return [String] Filename def new_file x = new_location x.nil? ? nil : x['file'] end # Public: Get the line number from the "new" location # @return [String] Line number def new_line x = new_location x.nil? ? nil : x['line'] end # Public: Get the "old" location, i.e. location in the "from" catalog # @return [Hash] of resource def old_location return if addition? return @raw[3] if removal? @raw[4] end # Public: Get the "new" location, i.e. location in the "to" catalog # @return [Hash] of resource def new_location return @raw[3] if addition? return if removal? @raw[5] end # Public: Convert this object to a hash # @return [Hash] Hash with keys set by these methods def to_h { diff_type: diff_type, type: type, title: title, structure: structure, old_value: old_value, new_value: new_value, old_file: old_file, old_line: old_line, new_file: new_file, new_line: new_line, old_location: old_location, new_location: new_location } end # Public: Convert this object to a hash with string keys # @return [Hash] Hash with keys set by these methods, with string keys def to_h_with_string_keys result = {} to_h.each { |key, val| result[key.to_s] = val } result end # Public: String inspection # @return [String] String for inspection def inspect to_h.inspect end # Public: To string # @return [String] Compact string representation def to_s raw.inspect end private # Private: Initialize further instance variables def initialize_helper unless ['+', '-', '~', '!'].include?(@raw[0]) raise ArgumentError, 'Invalid first element array: diff type needs to be one of: +, -, ~, !' end @diff_type = @raw[0] unless @raw[1].is_a?(String) raise ArgumentError, "Invalid second element array: type-title-structure needs to be a string not #{@raw[1].class}" end raw_1_split = @raw[1].split(/\f/) @type = raw_1_split[0] @title = raw_1_split[1] @structure = raw_1_split[2..-1] end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/api/v1/common.rb0000644000004100000410000000136713413744025023657 0ustar www-datawww-data# frozen_string_literal: true module OctocatalogDiff module API module V1 # Common functions for API v1 class Common def self.logger_from_options(options) # If logger is not provided, create an object that can have messages written to it. # There won't be a way to access these messages, so if you want to log messages, then # provide that logger! logger = options[:logger] || Logger.new(StringIO.new) # We can't keep :logger in the options due to marshal/unmarshal as part of parallelization. pass_opts = options.dup pass_opts.delete(:logger) # Return cleaned options and logger [pass_opts, logger] end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/api/v1/override.rb0000644000004100000410000001031613413744025024200 0ustar www-datawww-data# frozen_string_literal: true require 'json' module OctocatalogDiff module API module V1 # Sets up the override of a fact or ENC parameter during catalog compilation. class Override # Accessors attr_reader :key, :value # Constructor: Accepts a key and value. # @param input [Hash] Must contain :key and :value def initialize(input) key = input.fetch(:key) @key = key =~ %r{\A/(.+)/\Z} ? Regexp.new(Regexp.last_match(1)) : key @value = parsed_value(input.fetch(:value)) end # Initialize from a parsed command line # @param input [String] Command line parameter # @return [OctocatalogDiff::API::V1::Override] Initialized object def self.create_from_input(input, key = nil) # Normally the input will be a string in the format key=(data type)value where the data # type is optional and the parentheses are literal. Example: # foo=1 (auto-determine data type - in this case it would be a fixnum) # foo=(fixnum)1 (will be a fixnum) # foo=(string)1 (will be '1' the string) # If input is not a string, we can still construct the object if the key is given. # That input would come directly from code and not from the command line, since inputs # from the command line are always strings. # Also support regular expressions for the key name, if delimited by //. if key.nil? && input.is_a?(String) unless input.include?('=') raise ArgumentError, "Fact override '#{input}' is not in 'key=(data type)value' format" end k, v = input.strip.split('=', 2).map(&:strip) new(key: k, value: v) elsif key.nil? message = "Define a key when the input is not a string (#{input.class} => #{input.inspect})" raise ArgumentError, message else new(key: key, value: input) end end private # Guess the datatype from a particular input # @param input [String] Input in string format # @return [?] Output in appropriate format def parsed_value(input) # If data type is explicitly given if input =~ /^\((\w+)\)(.*)$/m datatype = Regexp.last_match(1) value = Regexp.last_match(2) return convert_to_data_type(datatype.downcase, value) end # Guess data type return input.to_i if input =~ /^-?\d+$/ return input.to_f if input =~ /^-?\d*\.\d+$/ return true if input.casecmp('true').zero? return false if input.casecmp('false').zero? input end # Handle data type that's explicitly given # @param datatype [String] Data type (as a string) # @param value [String] Value given # @return [?] Value converted to specified data type def convert_to_data_type(datatype, value) return value if datatype == 'string' return parse_json(value) if datatype == 'json' return nil if datatype == 'nil' if datatype == 'fixnum' || datatype == 'integer' return Regexp.last_match(1).to_i if value =~ /^(-?\d+)$/ raise ArgumentError, "Illegal integer '#{value}'" end if datatype == 'float' return Regexp.last_match(1).to_f if value =~ /^(-?\d*\.\d+)$/ return Regexp.last_match(1).to_f if value =~ /^(-?\d+)$/ raise ArgumentError, "Illegal float '#{value}'" end if datatype == 'boolean' return true if value.casecmp('true').zero? return false if value.casecmp('false').zero? raise ArgumentError, "Illegal boolean '#{value}'" end raise ArgumentError, "Unknown data type '#{datatype}'" end # Parse JSON value # @param input [String] Input, hopefully in JSON format # @return [?] Output data structure def parse_json(input) JSON.parse(input) rescue JSON::ParserError => exc raise JSON::ParserError, "Failed to parse JSON: input=#{input} error=#{exc}" end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/api/v1/catalog.rb0000644000004100000410000000477713413744025024011 0ustar www-datawww-data# frozen_string_literal: true require_relative 'common' require_relative '../../catalog' module OctocatalogDiff module API module V1 # This is a wrapper class around OctocatalogDiff::Catalog. This contains the methods we # are choosing to expose, and will be a compatibility layer should underlying methods # change in the future. The raw object will be available as `#raw` but this is not # guaranteed to be stable. class Catalog attr_reader :raw # Constructor: Accepts a raw OctocatalogDiff::Catalog object and stores it. # @param raw [OctocatalogDiff::Catalog] Catalog object def initialize(raw) unless raw.is_a?(OctocatalogDiff::Catalog) raise ArgumentError, 'OctocatalogDiff::API::V1::Catalog#initialize expects OctocatalogDiff::Catalog argument' end @raw = raw end # Public: Get the builder for the catalog # @return [String] Class of backend used def builder @raw.builder end # Public: Get the JSON for the catalog # @return [String] Catalog JSON def to_json @raw.catalog_json end # Public: Get the compilation directory # @return [String] Compilation directory def compilation_dir @raw.compilation_dir end # Public: Get the error message # @return [String] Error message, or nil if no error def error_message @raw.error_message end # Public: Get the Puppet version used to compile the catalog # @return [String] Puppet version def puppet_version @raw.puppet_version end # Public: Get a specific resource identified by type and title. # This is intended for use when a O(1) lookup is required. # @param :type [String] Type of resource # @param :title [String] Title of resource # @return [Hash] Resource item def resource(opts = {}) @raw.resource(opts) end # Public: Get the resources in the catalog # @return [Array] Resource array def resources @raw.resources end # Public: Determine if the catalog build was successful. # @return [Boolean] Whether the catalog is valid def valid? @raw.valid? end # Public: Return catalog as hash. # @return [Hash] Catalog as hash def to_h @raw.catalog end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/api/v1/config.rb0000644000004100000410000001245313413744025023632 0ustar www-datawww-data# frozen_string_literal: true require_relative 'common' require_relative '../../errors' module OctocatalogDiff module API module V1 # This class interacts with the configuration file typically named `.octocatalog-diff.cfg.rb`. class Config # Default directory paths: These are the documented default locations that will be checked # for the configuration file. DEFAULT_PATHS = [ ENV['OCTOCATALOG_DIFF_CONFIG_FILE'], File.join(Dir.pwd, '.octocatalog-diff.cfg.rb'), File.join(ENV['HOME'], '.octocatalog-diff.cfg.rb'), '/opt/puppetlabs/octocatalog-diff/octocatalog-diff.cfg.rb', '/usr/local/etc/octocatalog-diff.cfg.rb', '/etc/octocatalog-diff.cfg.rb' ].compact.freeze # Public: Find the configuration file in the specified path or one of the default paths # as appropriate. Parses the configuration file and returns the hash object with its settings. # Returns empty hash if the configuration file is not found anywhere. # # @param :filename [String] Specified file name (default = search the default paths) # @param :logger [Logger] Logger object # @param :test [Boolean] Configuration file test mode (log some extra debugging, raises errors) # @return [Hash] Parsed configuration file def self.config(options_in = {}) # Initialize the logger - if not passed, set to a throwaway object. options, logger = OctocatalogDiff::API::V1::Common.logger_from_options(options_in) # Locate the configuration file paths = [options.fetch(:filename, DEFAULT_PATHS)].compact config_file = first_file(paths) # Can't find the configuration file? if config_file.nil? message = "Unable to find configuration file in #{paths.join(':')}" raise OctocatalogDiff::Errors::ConfigurationFileNotFoundError, message if options[:test] logger.debug message return {} end # Load/parse the configuration file - this returns a hash settings = load_config_file(config_file, logger) # Debug the configuration file if requested. debug_config_file(settings, logger) if options[:test] # Return the settings hash logger.debug "Loaded #{settings.keys.size} settings from #{config_file}" settings end # Private: Print debugging details for the configuration file. # # @param settings [Hash] Parsed settings from load_config_file # @param logger [Logger] Logger object def self.debug_config_file(settings, logger) unless settings.is_a?(Hash) raise ArgumentError, "Settings must be hash not #{settings.class}" end settings.each { |key, val| logger.debug ":#{key} => (#{val.class}) #{val.inspect}" } end # Private: Load the configuration file from a given path. Returns the settings hash. # # @param filename [String] File name to load # @param logger [Logger] Logger object # @return [Hash] Settings def self.load_config_file(filename, logger) # This should never happen unless somebody calls this method directly outside of # the published `.config` method. Check for problems anyway. raise Errno::ENOENT, "File #{filename} doesn't exist" unless File.file?(filename) # Attempt to require in the file. Problems here will fall through to the rescued # exception below. logger.debug "Loading octocatalog-diff configuration from #{filename}" load filename # The required file should contain `OctocatalogDiff::Config` with `.config` method. # If this is undefined, raise an exception. begin loaded_class = Kernel.const_get(:OctocatalogDiff).const_get(:Config) rescue NameError message = 'Configuration must define OctocatalogDiff::Config!' raise OctocatalogDiff::Errors::ConfigurationFileContentError, message end unless loaded_class.respond_to?(:config) message = 'Configuration must define OctocatalogDiff::Config.config!' raise OctocatalogDiff::Errors::ConfigurationFileContentError, message end # The configuration file looks like it defines the correct method, so read it. # Make sure it's a hash. options = loaded_class.config unless options.is_a?(Hash) message = "Configuration must be Hash not #{options.class}!" raise OctocatalogDiff::Errors::ConfigurationFileContentError, message end options rescue Exception => exc # rubocop:disable Lint/RescueException logger.fatal "#{exc.class} error with #{filename}: #{exc.message}\n#{exc.backtrace}" raise exc end # Private: Find the first element of the given array that is a file and return it. # Return nil if none of the elements in the array are files. # # @param search_paths [Array] Paths to check def self.first_file(search_paths) search_paths.flatten.compact.each do |path| return path if File.file?(path) end nil end end end end end octocatalog-diff-1.5.4/lib/octocatalog-diff/api/v1.rb0000644000004100000410000000126013413744025022357 0ustar www-datawww-data# frozen_string_literal: true require_relative 'v1/catalog' require_relative 'v1/catalog-compile' require_relative 'v1/catalog-diff' require_relative 'v1/config' require_relative 'v1/diff' require_relative 'v1/override' module OctocatalogDiff module API # Call available methods for this version of the API module V1 def self.catalog(options = nil) OctocatalogDiff::API::V1::CatalogCompile.catalog(options) end def self.catalog_diff(options = nil) OctocatalogDiff::API::V1::CatalogDiff.catalog_diff(options) end def self.config(options = nil) OctocatalogDiff::API::V1::Config.config(options) end end end end octocatalog-diff-1.5.4/octocatalog-diff.gemspec0000644000004100000410000003314113413744025021535 0ustar www-datawww-data######################################################### # This file has been automatically generated by gem2tgz # ######################################################### # -*- encoding: utf-8 -*- # stub: octocatalog-diff 1.5.4 ruby lib Gem::Specification.new do |s| s.name = "octocatalog-diff".freeze s.version = "1.5.4" s.required_rubygems_version = Gem::Requirement.new(">= 0".freeze) if s.respond_to? :required_rubygems_version= s.require_paths = ["lib".freeze] s.authors = ["GitHub, Inc.".freeze, "Kevin Paulisse".freeze] s.date = "2018-12-13" s.description = "Octocatalog-Diff assists with Puppet development and testing by enabling the user to\ncompile 2 Puppet catalogs and compare them. It is possible to compare different\nbranches, different versions, and different fact values. This is intended to be run\nfrom a local development environment or in CI.\n".freeze s.email = "opensource+octocatalog-diff@github.com".freeze s.executables = ["octocatalog-diff".freeze] s.files = [".version".freeze, "LICENSE".freeze, "README.md".freeze, "bin/octocatalog-diff".freeze, "doc/CHANGELOG.md".freeze, "doc/advanced-bootstrap.md".freeze, "doc/advanced-cache-dir.md".freeze, "doc/advanced-catalog-only.md".freeze, "doc/advanced-catalog-validation.md".freeze, "doc/advanced-ci.md".freeze, "doc/advanced-dynamic-ignores.md".freeze, "doc/advanced-environment-variables.md".freeze, "doc/advanced-environments.md".freeze, "doc/advanced-filter.md".freeze, "doc/advanced-future-parser.md".freeze, "doc/advanced-hiera-path-stripping.md".freeze, "doc/advanced-ignores.md".freeze, "doc/advanced-output-formats.md".freeze, "doc/advanced-output-hacks.md".freeze, "doc/advanced-override-enc.md".freeze, "doc/advanced-override-facts.md".freeze, "doc/advanced-pe-enc.md".freeze, "doc/advanced-puppet-master.md".freeze, "doc/advanced-puppet-versions.md".freeze, "doc/advanced-script-override.md".freeze, "doc/advanced-storeconfigs.md".freeze, "doc/advanced-using-without-git.md".freeze, "doc/advanced.md".freeze, "doc/basic.md".freeze, "doc/configuration-enc.md".freeze, "doc/configuration-hiera.md".freeze, "doc/configuration-puppet.md".freeze, "doc/configuration-puppetdb.md".freeze, "doc/configuration.md".freeze, "doc/dev/README.md".freeze, "doc/dev/api.md".freeze, "doc/dev/api/v1.md".freeze, "doc/dev/api/v1/calls/catalog-diff.md".freeze, "doc/dev/api/v1/calls/catalog.md".freeze, "doc/dev/api/v1/calls/config.md".freeze, "doc/dev/api/v1/objects/catalog.md".freeze, "doc/dev/api/v1/objects/diff.md".freeze, "doc/dev/api/v1/objects/override.md".freeze, "doc/dev/coverage.md".freeze, "doc/dev/how-to-add-options.md".freeze, "doc/dev/integration-tests.md".freeze, "doc/dev/releasing.md".freeze, "doc/dev/run-from-branch.md".freeze, "doc/installation.md".freeze, "doc/limitations.md".freeze, "doc/optionsref.md".freeze, "doc/requirements.md".freeze, "doc/roadmap.md".freeze, "doc/similar.md".freeze, "doc/troubleshooting.md".freeze, "doc/versions/v1.md".freeze, "lib/octocatalog-diff.rb".freeze, "lib/octocatalog-diff/api/v1.rb".freeze, "lib/octocatalog-diff/api/v1/catalog-compile.rb".freeze, "lib/octocatalog-diff/api/v1/catalog-diff.rb".freeze, "lib/octocatalog-diff/api/v1/catalog.rb".freeze, "lib/octocatalog-diff/api/v1/common.rb".freeze, "lib/octocatalog-diff/api/v1/config.rb".freeze, "lib/octocatalog-diff/api/v1/diff.rb".freeze, "lib/octocatalog-diff/api/v1/override.rb".freeze, "lib/octocatalog-diff/bootstrap.rb".freeze, "lib/octocatalog-diff/catalog-diff/differ.rb".freeze, "lib/octocatalog-diff/catalog-diff/display.rb".freeze, "lib/octocatalog-diff/catalog-diff/display/json.rb".freeze, "lib/octocatalog-diff/catalog-diff/display/legacy_json.rb".freeze, "lib/octocatalog-diff/catalog-diff/display/text.rb".freeze, "lib/octocatalog-diff/catalog-diff/filter.rb".freeze, "lib/octocatalog-diff/catalog-diff/filter/absent_file.rb".freeze, "lib/octocatalog-diff/catalog-diff/filter/compilation_dir.rb".freeze, "lib/octocatalog-diff/catalog-diff/filter/json.rb".freeze, "lib/octocatalog-diff/catalog-diff/filter/single_item_array.rb".freeze, "lib/octocatalog-diff/catalog-diff/filter/yaml.rb".freeze, "lib/octocatalog-diff/catalog-util/bootstrap.rb".freeze, "lib/octocatalog-diff/catalog-util/builddir.rb".freeze, "lib/octocatalog-diff/catalog-util/cached_master_directory.rb".freeze, "lib/octocatalog-diff/catalog-util/command.rb".freeze, "lib/octocatalog-diff/catalog-util/enc.rb".freeze, "lib/octocatalog-diff/catalog-util/enc/noop.rb".freeze, "lib/octocatalog-diff/catalog-util/enc/pe.rb".freeze, "lib/octocatalog-diff/catalog-util/enc/pe/v1.rb".freeze, "lib/octocatalog-diff/catalog-util/enc/script.rb".freeze, "lib/octocatalog-diff/catalog-util/facts.rb".freeze, "lib/octocatalog-diff/catalog-util/fileresources.rb".freeze, "lib/octocatalog-diff/catalog-util/git.rb".freeze, "lib/octocatalog-diff/catalog.rb".freeze, "lib/octocatalog-diff/catalog/computed.rb".freeze, "lib/octocatalog-diff/catalog/json.rb".freeze, "lib/octocatalog-diff/catalog/noop.rb".freeze, "lib/octocatalog-diff/catalog/puppetdb.rb".freeze, "lib/octocatalog-diff/catalog/puppetmaster.rb".freeze, "lib/octocatalog-diff/cli.rb".freeze, "lib/octocatalog-diff/cli/diffs.rb".freeze, "lib/octocatalog-diff/cli/options.rb".freeze, "lib/octocatalog-diff/cli/options/basedir.rb".freeze, "lib/octocatalog-diff/cli/options/bootstrap_current.rb".freeze, "lib/octocatalog-diff/cli/options/bootstrap_environment.rb".freeze, "lib/octocatalog-diff/cli/options/bootstrap_script.rb".freeze, "lib/octocatalog-diff/cli/options/bootstrap_then_exit.rb".freeze, "lib/octocatalog-diff/cli/options/bootstrapped_dirs.rb".freeze, "lib/octocatalog-diff/cli/options/cached_master_dir.rb".freeze, "lib/octocatalog-diff/cli/options/catalog_only.rb".freeze, "lib/octocatalog-diff/cli/options/color.rb".freeze, "lib/octocatalog-diff/cli/options/command_line.rb".freeze, "lib/octocatalog-diff/cli/options/compare_file_text.rb".freeze, "lib/octocatalog-diff/cli/options/create_symlinks.rb".freeze, "lib/octocatalog-diff/cli/options/debug.rb".freeze, "lib/octocatalog-diff/cli/options/debug_bootstrap.rb".freeze, "lib/octocatalog-diff/cli/options/display_datatype_changes.rb".freeze, "lib/octocatalog-diff/cli/options/display_detail_add.rb".freeze, "lib/octocatalog-diff/cli/options/display_source_file_line.rb".freeze, "lib/octocatalog-diff/cli/options/enc.rb".freeze, "lib/octocatalog-diff/cli/options/enc_override.rb".freeze, "lib/octocatalog-diff/cli/options/environment.rb".freeze, "lib/octocatalog-diff/cli/options/existing_catalogs.rb".freeze, "lib/octocatalog-diff/cli/options/fact_file.rb".freeze, "lib/octocatalog-diff/cli/options/fact_override.rb".freeze, "lib/octocatalog-diff/cli/options/facts_terminus.rb".freeze, "lib/octocatalog-diff/cli/options/filters.rb".freeze, "lib/octocatalog-diff/cli/options/from_puppetdb.rb".freeze, "lib/octocatalog-diff/cli/options/header.rb".freeze, "lib/octocatalog-diff/cli/options/hiera_config.rb".freeze, "lib/octocatalog-diff/cli/options/hiera_path.rb".freeze, "lib/octocatalog-diff/cli/options/hiera_path_strip.rb".freeze, "lib/octocatalog-diff/cli/options/hostname.rb".freeze, "lib/octocatalog-diff/cli/options/ignore.rb".freeze, "lib/octocatalog-diff/cli/options/ignore_attr.rb".freeze, "lib/octocatalog-diff/cli/options/ignore_tags.rb".freeze, "lib/octocatalog-diff/cli/options/include_tags.rb".freeze, "lib/octocatalog-diff/cli/options/master_cache_branch.rb".freeze, "lib/octocatalog-diff/cli/options/output_file.rb".freeze, "lib/octocatalog-diff/cli/options/output_format.rb".freeze, "lib/octocatalog-diff/cli/options/override_script_path.rb".freeze, "lib/octocatalog-diff/cli/options/parallel.rb".freeze, "lib/octocatalog-diff/cli/options/parser.rb".freeze, "lib/octocatalog-diff/cli/options/pass_env_vars.rb".freeze, "lib/octocatalog-diff/cli/options/pe_enc_ssl_ca.rb".freeze, "lib/octocatalog-diff/cli/options/pe_enc_ssl_client_cert.rb".freeze, "lib/octocatalog-diff/cli/options/pe_enc_ssl_client_key.rb".freeze, "lib/octocatalog-diff/cli/options/pe_enc_token.rb".freeze, "lib/octocatalog-diff/cli/options/pe_enc_token_file.rb".freeze, "lib/octocatalog-diff/cli/options/pe_enc_url.rb".freeze, "lib/octocatalog-diff/cli/options/preserve_environments.rb".freeze, "lib/octocatalog-diff/cli/options/puppet_binary.rb".freeze, "lib/octocatalog-diff/cli/options/puppet_master.rb".freeze, "lib/octocatalog-diff/cli/options/puppet_master_api_version.rb".freeze, "lib/octocatalog-diff/cli/options/puppet_master_ssl_ca.rb".freeze, "lib/octocatalog-diff/cli/options/puppet_master_ssl_client_cert.rb".freeze, "lib/octocatalog-diff/cli/options/puppet_master_ssl_client_key.rb".freeze, "lib/octocatalog-diff/cli/options/puppet_master_timeout.rb".freeze, "lib/octocatalog-diff/cli/options/puppetdb_api_version.rb".freeze, "lib/octocatalog-diff/cli/options/puppetdb_ssl_ca.rb".freeze, "lib/octocatalog-diff/cli/options/puppetdb_ssl_client_cert.rb".freeze, "lib/octocatalog-diff/cli/options/puppetdb_ssl_client_key.rb".freeze, "lib/octocatalog-diff/cli/options/puppetdb_ssl_client_password.rb".freeze, "lib/octocatalog-diff/cli/options/puppetdb_ssl_client_password_file.rb".freeze, "lib/octocatalog-diff/cli/options/puppetdb_token.rb".freeze, "lib/octocatalog-diff/cli/options/puppetdb_token_file.rb".freeze, "lib/octocatalog-diff/cli/options/puppetdb_url.rb".freeze, "lib/octocatalog-diff/cli/options/quiet.rb".freeze, "lib/octocatalog-diff/cli/options/retry_failed_catalog.rb".freeze, "lib/octocatalog-diff/cli/options/safe_to_delete_cached_master_dir.rb".freeze, "lib/octocatalog-diff/cli/options/save_catalog.rb".freeze, "lib/octocatalog-diff/cli/options/storeconfigs.rb".freeze, "lib/octocatalog-diff/cli/options/suppress_absent_file_details.rb".freeze, "lib/octocatalog-diff/cli/options/to_from_branch.rb".freeze, "lib/octocatalog-diff/cli/options/truncate_details.rb".freeze, "lib/octocatalog-diff/cli/options/validate_references.rb".freeze, "lib/octocatalog-diff/cli/printer.rb".freeze, "lib/octocatalog-diff/errors.rb".freeze, "lib/octocatalog-diff/external/pson/LICENSE".freeze, "lib/octocatalog-diff/external/pson/README.md".freeze, "lib/octocatalog-diff/external/pson/common.rb".freeze, "lib/octocatalog-diff/external/pson/pure.rb".freeze, "lib/octocatalog-diff/external/pson/pure/generator.rb".freeze, "lib/octocatalog-diff/external/pson/pure/parser.rb".freeze, "lib/octocatalog-diff/external/pson/version.rb".freeze, "lib/octocatalog-diff/facts.rb".freeze, "lib/octocatalog-diff/facts/json.rb".freeze, "lib/octocatalog-diff/facts/puppetdb.rb".freeze, "lib/octocatalog-diff/facts/yaml.rb".freeze, "lib/octocatalog-diff/puppetdb.rb".freeze, "lib/octocatalog-diff/util/catalogs.rb".freeze, "lib/octocatalog-diff/util/colored.rb".freeze, "lib/octocatalog-diff/util/httparty.rb".freeze, "lib/octocatalog-diff/util/parallel.rb".freeze, "lib/octocatalog-diff/util/puppetversion.rb".freeze, "lib/octocatalog-diff/util/scriptrunner.rb".freeze, "lib/octocatalog-diff/util/util.rb".freeze, "lib/octocatalog-diff/version.rb".freeze, "scripts/env/env.sh".freeze, "scripts/git-extract/git-extract.sh".freeze, "scripts/puppet/puppet.sh".freeze] s.homepage = "https://github.com/github/octocatalog-diff".freeze s.licenses = ["MIT".freeze] s.required_ruby_version = Gem::Requirement.new(">= 2.0.0".freeze) s.rubygems_version = "2.5.2.1".freeze s.summary = "Compile Puppet catalogs from 2 branches, versions, etc., and compare them.".freeze if s.respond_to? :specification_version then s.specification_version = 4 if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then s.add_runtime_dependency(%q.freeze, [">= 3.1.0"]) s.add_runtime_dependency(%q.freeze, [">= 0.3.0"]) s.add_runtime_dependency(%q.freeze, [">= 0.11.0"]) s.add_runtime_dependency(%q.freeze, [">= 1.12.0"]) s.add_development_dependency(%q.freeze, ["= 2.7.1"]) s.add_development_dependency(%q.freeze, ["~> 5.5.8"]) s.add_development_dependency(%q.freeze, ["= 12.3.1"]) s.add_development_dependency(%q.freeze, ["~> 3.4.0"]) s.add_development_dependency(%q.freeze, ["= 0.5.0"]) s.add_development_dependency(%q.freeze, ["= 0.48.1"]) s.add_runtime_dependency(%q.freeze, [">= 0.25.0b2"]) s.add_development_dependency(%q.freeze, ["~> 0.14.1"]) s.add_development_dependency(%q.freeze, ["~> 0.1.1"]) else s.add_dependency(%q.freeze, [">= 3.1.0"]) s.add_dependency(%q.freeze, [">= 0.3.0"]) s.add_dependency(%q.freeze, [">= 0.11.0"]) s.add_dependency(%q.freeze, [">= 1.12.0"]) s.add_dependency(%q.freeze, ["= 2.7.1"]) s.add_dependency(%q.freeze, ["~> 5.5.8"]) s.add_dependency(%q.freeze, ["= 12.3.1"]) s.add_dependency(%q.freeze, ["~> 3.4.0"]) s.add_dependency(%q.freeze, ["= 0.5.0"]) s.add_dependency(%q.freeze, ["= 0.48.1"]) s.add_dependency(%q.freeze, [">= 0.25.0b2"]) s.add_dependency(%q.freeze, ["~> 0.14.1"]) s.add_dependency(%q.freeze, ["~> 0.1.1"]) end else s.add_dependency(%q.freeze, [">= 3.1.0"]) s.add_dependency(%q.freeze, [">= 0.3.0"]) s.add_dependency(%q.freeze, [">= 0.11.0"]) s.add_dependency(%q.freeze, [">= 1.12.0"]) s.add_dependency(%q.freeze, ["= 2.7.1"]) s.add_dependency(%q.freeze, ["~> 5.5.8"]) s.add_dependency(%q.freeze, ["= 12.3.1"]) s.add_dependency(%q.freeze, ["~> 3.4.0"]) s.add_dependency(%q.freeze, ["= 0.5.0"]) s.add_dependency(%q.freeze, ["= 0.48.1"]) s.add_dependency(%q.freeze, [">= 0.25.0b2"]) s.add_dependency(%q.freeze, ["~> 0.14.1"]) s.add_dependency(%q.freeze, ["~> 0.1.1"]) end end octocatalog-diff-1.5.4/doc/0000755000004100000410000000000013413744025015526 5ustar www-datawww-dataoctocatalog-diff-1.5.4/doc/advanced-cache-dir.md0000644000004100000410000000337213413744025021437 0ustar www-datawww-data# Enabling the cache directory If you are running `octocatalog-diff` on your workstation, enabling the cache directory can support faster runs by: - Bootstrapping the `master` branch and saving that in a directory, so that you do not need to bootstrap the `master` branch each time you run `octocatalog-diff`. - Saving the `master` catalog for each node, so that for the second and subsequent difference calculation, this catalog does not need to be re-computed. We recommend that you configure these settings in your [configuration file](/doc/configuration.md), although it is possible to specify these settings on the command line as well. ## Cache directory options There are two options that pertain to the cache directory: - `--cached-master-dir DIRECTORY_PATH` This is the full path to the directory where the bootstrapped master directory will reside. Please note that this directory will be created if it doesn't exist, but for the directory to be created, *its parent directory must already exist*. You will receive an error message if you specify a directory path that is not a directory, or cannot be created as a directory. Note that a subdirectory called `.catalogs` will be created within the chosen directory paths, and compiled `master` catalogs for nodes will be stored therein. - `--safe-to-delete-cached-master-dir DIRECTORY_PATH` If you want to allow `octocatalog-diff` to delete the cached master directory when it becomes stale, set this option. If you do not set this option, and the cached master directory becomes stale, an error will be raised. Historically, this was separated from `--cached-master-dir` to provide a separation between routine behavior (creating files and catalogs) and destructive behavior (deleting an entire directory). octocatalog-diff-1.5.4/doc/advanced-catalog-validation.md0000644000004100000410000000500013413744025023350 0ustar www-datawww-data# Catalog validation `octocatalog-diff` contains additional functionality to validate catalogs, based on configurable criteria. Catalog validation features include: - Validate references: Ensure resources targeted by `before`, `notify`, `require`, and/or `subscribe` exist in the catalog for Puppet 4 and below. ## Validate references `octocatalog-diff` includes the ability to validate references by ensuring resources targeted by `before`, `notify`, `require`, and/or `subscribe` parameters also exist in the catalog. Puppet 5 already has this checking built in, so the `--validate-references` option described in this section will be ignored if Puppet 5 is being used. The same exception (`OctocatalogDiff::Errors::CatalogError`) is raised for a missing reference, whether the problem was detected by octocatalog-diff or Puppet 5. Consider the following Puppet code: ``` file { '/usr/local/bin/some-script.sh': source => 'puppet:///modules/test/usr/local/bin/some-script.sh', notify => Exec['execute /usr/local/bin/some-script.sh'], } ``` The catalog for this code would build, whether or not the `exec { 'execute /usr/local/bin/some-script.sh': ... }` resource was part of the catalog. However, when the catalog is applied on the Puppet agent, it would fail if this resource is missing. With the `--validate-references` command line flag (or the `settings[:validate_references]` [configuration setting](/doc/configuration.md)), you can instruct `octocatalog-diff` to confirm that any resource targeted by a `before`, `notify`, `require`, and `subscribe` parameter actually exists. If the resource is missing from the catalog, an error will be raised, just as if the catalog failed to compile. The command line argument is demonstrated here: ``` # Validate all references: before,notify,require,subscribe octocatalog-diff ... --validate-references before,notify,require,subscribe # Validate some references: only before and require octocatalog-diff ... --validate-references before,require # Validate no references octocatalog-diff ... --no-validate-references ``` By default, no references are validated. Note as well, when using `octocatalog-diff` to compare two catalogs, the references in the "from" catalog are not checked. The reason for this design decision is as follows: the "from" catalog is generally what is considered to be stable and is perhaps already deployed, so it adds no value (and perhaps inhibits the ability to develop further) if `octocatalog-diff` fails just because references in the "from" catalog are broken. octocatalog-diff-1.5.4/doc/versions/0000755000004100000410000000000013413744025017376 5ustar www-datawww-dataoctocatalog-diff-1.5.4/doc/versions/v1.md0000644000004100000410000000276413413744025020257 0ustar www-datawww-data# What's new in octocatalog-diff 1.0 #### Summary The most significant change in version 1.0 is the addition of the V1 API, which permits developers to build catalogs (--catalog-only) and compare/diff catalogs using octocatalog-diff. Under the hood, we've rearranged the code to support these APIs, which should also improve the reliability and allow faster development cycles. There are also some additional capabilities including: - Better display of whitespace differences - Override ENC parameters with `--to-enc-override` and `--from-enc-override` #### Breaking changes The format of the output from --output-format json has changed. In version 0.x of the software, each difference was represented by an array. In version 1.x, each difference is represented by a hash with meaningful English keys. We have added an option --output-format legacy_json which outputs in the old format. If you have written code that calls the octocatalog-diff command line and expects JSON output, you will need to update your code to the new format, or add or update `--output-format legacy_json` in your call to octocatalog-diff. #### API stability The API is versioned, and `v1` will be valid for all 1.x releases. We may add capabilities but will preserve existing functionality as it exists in the initial 1.0 release. Each API version change will correspond to a new major release number. In other words, a `v2` API would be introduced in a 2.0 release. octocatalog-diff-1.5.4/doc/configuration-puppet.md0000644000004100000410000000467513413744025022246 0ustar www-datawww-data# Configuring octocatalog-diff to use Puppet The most common use of `octocatalog-diff` is to use `puppet` locally to compile catalogs. In order to successfully use Puppet to compile catalogs: 0. Puppet must be installed on the system. It is the goal of `octocatalog-diff` to support Puppet version 3.8 and higher, installed via any means supported by Puppet. This includes the [All-In-One agent package](https://docs.puppet.com/puppet/4.0/reference/release_notes.html#all-in-one-packaging) or installed as a Ruby gem. By default, `octocatalog-diff` will look for the Puppet binary in several common system locations. For maximum reliability, you can specify the full path to the Puppet binary in the configuration file. For example: ``` ############################################################################################## # puppet_binary # This is the full path to the puppet binary on your system. If you don't specify this, # the tool will just run 'puppet' and hope to find it in your path. ############################################################################################## # settings[:puppet_binary] = '/usr/bin/puppet' settings[:puppet_binary] = '/opt/puppetlabs/puppet/bin/puppet' ``` 0. Applies if you are using [exported resources](https://docs.puppet.com/puppet/latest/reference/lang_exported.html) from PuppetDB (i.e., the octocatalog-diff `--storeconfigs` option enabled): Your Puppet installation must have the `puppetdb-termini` feature available. This feature may not be included by default with the Puppet agent package. Consult the [Connecting Puppet masters to PuppetDB](https://docs.puppet.com/puppetdb/latest/connect_puppet_master.html#step-1-install-plug-ins) documentation for instructions on installing the `puppetdb-termini` gem. :warning: Attention Mac OS users: the [documentation](https://docs.puppet.com/puppet/latest/reference/puppet_collections.html#os-x-systems) states: > While the puppet-agent package is the only component of a Puppet Collection available on OS X, you can still use Puppet Collections to ensure the version of package-agent you install is compatible with the Puppet Collection powering your infrastructure. Unfortunately this means that you won't be able to enable `--storeconfigs` with the All-In-One Puppet Agent on Mac OS X, unless you manually install a gem-packaged version of `puppetdb-terminus`. The procedure for this is beyond the scope of this documentation. octocatalog-diff-1.5.4/doc/advanced.md0000644000004100000410000000444213413744025017621 0ustar www-datawww-data# Advanced usage With `--octocatalog-diff` supporting over 75 command line options (and counting), there's a little something for everyone. On this page, we document some interesting use cases that can be accomplished with creative combinations of options. If you find a creative use of `octocatalog-diff` that we haven't thought of, we encourage you to create a document named `advanced-SOMETHING.md` and link to it from here! See also: - [Basic usage](/doc/basic.md) - Common use cases to get you started - [Command line options reference](/doc/optionsref.md) - A list of *all* the options - [How to add new command line options](/doc/dev/how-to-add-options.md) - If you'd like to add an option of your own ## Advanced usage documentation ### Building catalogs - [Bootstrapping your Puppet checkout](/doc/advanced-bootstrap.md) - [Building catalogs instead of diffing catalogs](/doc/advanced-catalog-only.md) - [Enabling storeconfigs for exported resources in PuppetDB](/doc/advanced-storeconfigs.md) - [Fetching catalogs from Puppet Master / PuppetServer](/doc/advanced-puppet-master.md) - [Overriding ENC parameters](/doc/advanced-override-enc.md) - [Overriding facts](/doc/advanced-override-facts.md) - [Puppet Enterprise node classification service](/doc/advanced-pe-enc.md) - [Using `octocatalog-diff` without git](/doc/advanced-using-without-git.md) - [Catalog validation](/doc/advanced-catalog-validation.md) - [Environment setup](/doc/advanced-environments.md) - [Overriding built-in octocatalog-diff scripts](/doc/advanced-script-override.md) ### Controlling output - [Ignoring certain changes via command line options](/doc/advanced-ignores.md) - [Additional output filters](/doc/advanced-filter.md) - [Dynamic ignoring of changes via tags in Puppet manifests](/doc/advanced-dynamic-ignores.md) - [Output formats](/doc/advanced-output-formats.md) - [Useful output hacks](/doc/advanced-output-hacks.md) ### Using `octocatalog-diff` in CI - [Using `octocatalog-diff` in CI](/doc/advanced-ci.md) ### Using `octocatalog-diff` on your workstation - [Enabling the cache directory](/doc/advanced-cache-dir.md) ### Using `octocatalog-diff` to help you upgrade - [Compiling the same catalog with different Puppet versions](/doc/advanced-puppet-versions.md) - [Enabling the future parser](/doc/advanced-future-parser.md) octocatalog-diff-1.5.4/doc/configuration.md0000644000004100000410000000513513413744025020723 0ustar www-datawww-data# Configuration `octocatalog-diff` may require configuration to work correctly with your Puppet setup. 0. Download the [sample configuration file](https://raw.githubusercontent.com/github/octocatalog-diff/master/examples/octocatalog-diff.cfg.rb) and save it into one of the following directories. - In the base directory of your Puppet repository checkout (i.e., your current working directory): ``` .octocatalog-diff.cfg.rb ``` - In your home directory: ``` $HOME/.octocatalog-diff.cfg.rb ``` - In one of the following system locations: ``` /usr/local/etc/octocatalog-diff.cfg.rb /opt/puppetlabs/octocatalog-diff/octocatalog-diff.cfg.rb /etc/octocatalog-diff.cfg.rb ``` Note: If more than one of the above files is present, the first one found will be used (proceeding from top to bottom in that list). If you set an environment variable `OCTOCATALOG_DIFF_CONFIG_FILE` that will supersede all of the above paths, and allow you to specify the configuration file location however you wish. 0. Open the file in a text editor, and follow the comments within the file to guide yourself through configuration. The configuration file is pure ruby, allowing substantial flexibility in the configuration. To be minimally functional, you will almost certainly need to define at least the following settings: - `settings[:hiera_config]` as the absolute or relative path to your hiera configuration file - `settings[:hiera_path_strip]` as the prefix to strip when munging the hiera configuration file - `settings[:puppetdb_url]` as the URL to your PuppetDB instance so facts can be obtained For more information on these settings: - [Configuring octocatalog-diff to use Hiera](/doc/configuration-hiera.md) - [Configuring octocatalog-diff to use ENC](/doc/configuration-enc.md) - [Configuring octocatalog-diff to use PuppetDB](/doc/configuration-puppetdb.md) - [Configuring octocatalog-diff to use Puppet](/doc/configuration-puppet.md) 0. Test the configuration, which will indicate the location of the configuration file and validate the contents thereof. ``` octocatalog-diff --config-test ``` Note: If you [installed](/doc/installation.md) octocatalog-diff as a gem, the `octocatalog-diff` binary should be in your $PATH and the above command should work correctly. If you installed in a different way, you may need to provide the full path to where the `octocatalog-diff` binary was actually installed. Now that you have entered your configuration and confirmed proper reading of your configuration file, proceed to [Basic usage](/doc/basic.md) to see if it works! octocatalog-diff-1.5.4/doc/advanced-using-without-git.md0000644000004100000410000000214613413744025023225 0ustar www-datawww-data# Using `octocatalog-diff` without git `octocatalog-diff` is designed to be used with git, and is developed by GitHub. This means that on a daily basis, it is run thousands of times on repositories backed by GitHub for CI jobs reported to GitHub. If you do not manage your Puppet code with git, it is still possible to use `octocatalog-diff`, but you will not be able to take advantage of the git integrations and will need to do some things manually. ## Usage You will need to use these command line options (at minimum): - `--bootstrapped-from-dir`: The directory containing your checked out "from" Puppet code, and if there is any [bootstrapping](/doc/advanced-bootstrap.md) it must already be done. - `--bootstrapped-to-dir`: The directory containing your checked out "to" Puppet code, and if there is any [bootstrapping](/doc/advanced-bootstrap.md) it must already be done. You are responsible for the process by which you check out code from your version control system into two separate directories and run any necessary bootstrapping. You will need to script that step *before* invoking `octocatalog-diff`. octocatalog-diff-1.5.4/doc/basic.md0000644000004100000410000000774213413744025017143 0ustar www-datawww-data# Basic usage The most basic usage of octocatalog-diff is to compare catalogs built from two different git branches, for a node of your choosing. You should be aware of these defaults, all of which are [configurable](/doc/configuration.md). - octocatalog-diff will default to compiling catalogs based on the assumption that your Puppet code resides in a git repository. If your Puppet code does not reside in a git repository, head over to the [advanced instructions](/doc/advanced.md) for workarounds. - octocatalog-diff will compile the catalog produced from the `origin/master` branch of your repository as the "from" catalog, and the catalog produced from your current working directory as the "to" catalog. You can override these defaults with the `-f BRANCH` and `-t BRANCH` arguments, for the "from" and "to" branches, respectively. - octocatalog-diff will assume you are not using hiera or an external node classifier, unless you [configure](/doc/configuration.md) it accordingly, or use the appropriate command line arguments to point it at your hiera configuration and/or ENC script. You are required to provide the following information, either as a command line argument, in the [configuration](/doc/configuration.md), or in some cases, via the environment: - The node name whose catalogs you wish to compile. Use `-n HOSTNAME` on the command line. - Facts, which can either be retrieved from [PuppetDB](/doc/configuration-puppetdb.md) or via the `--fact-file` command line option. See the usage examples below. ## Examples ### From git repository with facts from PuppetDB ``` export PUPPETDB_URL="http://puppetdb.yourdomain.com:8080" cd Puppet_Checkout_Directory git checkout master git pull octocatalog-diff -n SomeNodeName.yourdomain.com ``` ### Using a fact file You can retrieve the fact file from your Puppet Master (3.x) typically in `/var/lib/puppet/yaml/facts/.yaml`, or your Puppet Server (4.x) typically in `/opt/puppetlabs/server/data/puppetserver/yaml/facts/.yaml`. We recommend using PuppetDB as a more convenient fact source, but you can copy the fact file for a node from your Puppet server onto the machine running octocatalog-diff for testing purposes. ``` # Copy the fact file for SomeNodeName.yourdomain.com into /tmp/SomeNodeName.yourdomain.com.yaml cd Puppet_Checkout_Directory git checkout master git pull octocatalog-diff -n SomeNodeName.yourdomain.com --fact-file /tmp/SomeNodeName.yourdomain.com.yaml ``` ## Using hiera This example demonstrates how to point octocatalog-diff at your Hiera configuration file. The Hiera configuration file for your site might be found in `/etc/puppet/hiera.yaml` (for Puppet 3.x) or `/etc/puppetlabs/puppet/hiera.yaml` (for Puppet 4.x). Note that you will either need to configure the PuppetDB URL or specify a `--fact-file` for this to work. ``` # Copy the fact file for SomeNodeName.yourdomain.com into /tmp/SomeNodeName.yourdomain.com.yaml # (or) # Set the PUPPETDB_URL variable as shown in the first example # # Also copy hiera.yaml from your Puppet master into the /tmp directory cd Puppet_Checkout_Directory git checkout master git pull octocatalog-diff -n SomeNodeName.yourdomain.com --hiera-config /tmp/hiera.yaml ``` Depending on your hiera configuration, you may also need to supply the `--hiera-path-strip` option (or set that option in your [configuration](/doc/configuration.md)). Consult the [configuring octocatalog-diff to use Hiera](/doc/configuration-hiera.md) document for details on this option. ## Next steps If you're ready to learn about additional command line flags to customize your experience, head to [Advanced usage](/doc/advanced.md). If you experience problems running octocatalog-diff even with these most basic arguments, please see [Troubleshooting](/doc/troubleshooting.md). If you are not using git to manage your Puppet source code, you will need to see the [Advanced usage](/doc/advanced.md) instructions to get your directories manually bootstrapped for use, or use one of the other supported methods to build catalogs. octocatalog-diff-1.5.4/doc/configuration-puppetdb.md0000644000004100000410000001616713413744025022553 0ustar www-datawww-data# Configuring octocatalog-diff to use PuppetDB octocatalog-diff can interact with PuppetDB in the following ways: - Retrieving the most recent set of facts for a node - Query for exported resources during a Puppet run with storeconfigs enabled - Retrieving a catalog for a node For this to work, you will need to configure or provide information about your PuppetDB server to octocatalog-diff. You can provide this information via a [configuration file](/doc/configuration.md), via environment variables, or via command line parameters. # Required information - **Version of PuppetDB**: octocatalog-diff supports PuppetDB's query API v4, which requires that you be running PuppetDB 2.3 or higher. - **URL to PuppetDB**: This is the URL with the host name and port number to reach your PuppetDB instance. If you have already set up your Puppet master to communicate with PuppetDB, you can see the URL by reviewing `/etc/puppetlabs/puppet/puppetdb.conf` (on Puppet Server) or `/etc/puppet/puppetdb.conf` (on Puppet Master 3.x). The URL (or URLs) to your PuppetDB installation are visible in the `server_urls` configuration setting. To use basic authentication, place the username and password in the URL, e.g.: ``` https://username:password@puppetdb.example.net:8081 ``` - **SSL Authentication Information**: Whether your PuppetDB instance requires clients to authenticate via SSL certificates. Unless you have made a special effort to configure your PuppetDB instance not to require client certificates, it is likely that client certificate authentication is required. Please see the separate section below concerning SSL certificates. NOTE: In certain situations, you may need to define or alter the `certificate-whitelist` setting in your PuppetDB configuration to whitelist the certificate used by octocatalog-diff. Please see [Configuring PuppetDB](https://docs.puppet.com/puppetdb/latest/configure.html#certificate-whitelist) in the Puppet documentation for additional information. ## Supplying necessary information via configuration files The following settings can be used in a [configuration file](/doc/configuration.md). | Setting | Description | | --- | --- | | `settings[:puppetdb_url]` | PuppetDB URL settings. If this is a string, it will set a single PuppetDB URL. If it is an array, it will set multiple URLs, which will be tried in a random order until one responds. | | `settings[:puppetdb_ssl_ca]` | Path to the certificate of the CA that signed PuppetDB's certificate. This file should contain only the public certificate, so it is safe to distribute to developer workstations or CI environments. | | `settings[:puppetdb_ssl_client_cert]` | TEXT of the certificate of the client SSL keypair used to authenticate to PuppetDB. Note: This variable is not set to a file path, which means you will likely want to use `File.read(...)` if you are configuring this to be read from a file. | | `settings[:puppetdb_ssl_client_key]` | TEXT of the private key of the client SSL keypair used to authenticate to PuppetDB. Note: This variable is not set to a file path, which means you will likely want to use means you will likely want to use `File.read(...)` if you are configuring this to be read from a file. | | `settings[:puppetdb_ssl_client_pem]` | Concatenation of the text of `puppetdb_ssl_client_key` and `puppetdb_ssl_client_cert` as previously described. This is a good alternative if your certificate chain is complex and it's easier just to put everything in a single place. Note: this option is second in precedence; if `settings[:puppetdb_ssl_client_cert]` and `settings[:puppetdb_ssl_client_key]` are both set, this will be ignored. | | `settings[:puppetdb_ssl_client_password]` | Plain text string containing the password to unlock the private key. For keys generated by the Puppet Master CA, this is not required and should be left undefined. | | `settings[:puppetdb_token]` | TEXT containing the PE RBAC token used to authenticate to PuppetDB. Note: This variable is not set to a file path, which means you will likely want to use `File.read(...)` if you are configuring this to be read from a file. | ## Supplying necessary information via the command line The following arguments can be used on the command line. | Setting | Description | | --- | --- | | --puppetdb-url https://puppetdb.example.net:8081 | PuppetDB URL. The argument should match the `server_urls` configuration setting as described previously. Please note that only one URL is supported via the command line method, so if you have multiple `server_urls` URLs specified, you can only choose one. To use multiple URLs for failover purposes, please configure via configuration files. | | --puppetdb-ssl-ca FILENAME | Path to the certificate of the CA that signed PuppetDB's certificate. This file should contain only the public certificate, so it is safe to distribute to developer workstations or CI environments. | | --puppetdb-ssl-client-cert FILENAME | Path to the certificate of the client SSL keypair. | | --puppetdb-ssl-client-key FILENAME | Path to the private key of the client SSL keypair. | | --puppetdb-ssl-client-password PASSWORD_STRING | Plain text string containing the password to unlock the private key. For keys generated by the Puppet Master CA, this is not required. | | --puppetdb-token STRING | String containing the PE RBAC token used to authenticate to PuppetDB. | | --puppetdb-token-file FILENAME | Path to the PE RBAC token file used to authenticate to PuppetDB. | ## Supplying necessary information via the environment :warning: While this method of configuring octocatalog-diff for use with PuppetDB is currently supported, we recommend that configuration is done via the command line or in configuration files. Set the environment variable `PUPPETDB_URL` to match the `server_urls` configuration setting as described previously. Please note that only one URL is supported via the environment variable method, so if you have multiple `server_urls` URLs specified, you can only choose one. To use multiple URLs for failover purposes, please configure via configuration files. Environment variable support is not currently available for SSL client authentication settings. # Notes about SSL certificates SSL support is enabled via any of the `--puppetdb-ssl-...` command line options or `puppetdb_ssl_...` configuration settings as described above. Please note the following concerning these SSL certificates. - The CA certificate should be the public certificate of the CA that signed your PuppetDB server's certificate. This file can be found in `/etc/puppetlabs/puppetdb/ssl/ca.pem` on a PuppetDB server. Since this is a public certificate, it is safe (and recommended) to distribute this file to any clients that may connect to this PuppetDB instance. - The client keypair (key, certificate, and optionally password) should be generated individually for each client. You should NOT copy SSL keypairs from your PuppetDB server (or anywhere else) to your clients. If you are using `octocatalog-diff` on a system that is managed by Puppet, you may wish to use the same SSL credentials that the system uses to authenticate to Puppet. With recent versions of the Puppet agent, those certificates are found in `/etc/puppetlabs/puppet/ssl`. octocatalog-diff-1.5.4/doc/dev/0000755000004100000410000000000013413744025016304 5ustar www-datawww-dataoctocatalog-diff-1.5.4/doc/dev/README.md0000644000004100000410000000005313413744025017561 0ustar www-datawww-data# octocatalog-diff developer documentation octocatalog-diff-1.5.4/doc/dev/run-from-branch.md0000644000004100000410000000347713413744025021641 0ustar www-datawww-data# Running octocatalog-diff from a branch When we are assisting with troubleshooting, or implementing a feature you've requested, we may ask you to run `octocatalog-diff` from a non-master branch to try it out. This document is intended for people who may not be familiar with git, GitHub, and/or ruby. If you already know how to do this in another way, feel free! ## Installation 1. Determine the branch name. If there's an open Pull Request, you can see the branch name near the top of the page. ![Pull Request branch](/doc/images/pull-request-identify-branch.png) 2. Clone the `octocatalog-diff` repository in your home directory. From the command line: ``` cd $HOME git clone https://github.com/github/octocatalog-diff.git ``` 3. Change into the directory created by your checkout: ``` cd $HOME/octocatalog-diff ``` 4. Check out the branch you wish to use, filling in the branch name you determined in the first step: ``` git checkout BRANCH_NAME_FROM_STEP_1 ``` 5. Bootstrap the repository to pull in dependencies: ``` ./script/bootstrap ``` 6. Optional but recommended - run the test suite: ``` rake ``` ## Use Now that you have `octocatalog-diff` checked out and bootstrapped, it's time to use it. We've created a wrapper script to make this easier for you. 1. Change directories to the location where you ordinarily run `octocatalog-diff` (for example: in your Puppet repository). ``` cd /etc/puppetlabs/code ``` 2. Run the `script/octocatalog-diff-wrapper` script from *this* checkout. For example, if you checked out `octocatalog-diff` to your home directory, you could use: ``` $HOME/octocatalog-diff/script/octocatalog-diff-wrapper ``` :warning: Note: If you are requesting our help, please use the debug option (`-d`) to display debugging information. octocatalog-diff-1.5.4/doc/dev/integration-tests.md0000644000004100000410000000612713413744025022317 0ustar www-datawww-data# Integration tests Integration tests are designed to run `octocatalog-diff` from beginning to end with options and fixtures to demonstrate a desired behavior. The integration tests are found in [`/spec/octocatalog-diff/integration`](/spec/octocatalog-diff/integration). ## Writing an integration test We recommend using the provided [`integration_helper.rb`](/spec/octocatalog-diff/integration/integration_helper.rb) which provides some handy functions to reduce duplicative code, and hopefully make integration tests easier to write. An integration test that compiles one or two catalogs from a repository will look like this: ```ruby describe 'whatever behavior' do before(:all) do @result = OctocatalogDiff::Integration.integration( spec_repo: 'fact-overrides', # The repository directory in /spec/octocatalog-diff/fixtures/repos spec_fact_file: 'valid-facts.yaml', # The fact file in /spec/octocatalog-diff/fixtures/facts argv: '--debug --from-fact-override somekey=somevalue', # Command line arguments ) # At this point @result is a hash containing these keys: # @result[:logs] is a String containing everything printed to STDERR (Logger) # @result[:output] is a String containing everything printed to STDOUT # @result[:diffs] is an Array of differences # @result[:exitcode] is an Integer representing the exit code: 0 = no changes, 1 = failure, 2 = success, with changes # @result[:exception] contains any exception that was thrown end it 'should do whatever' do # ... end end ``` An integration test that uses already-compiled catalogs from the fixtures directory will look like this: ```ruby describe 'whatever behavior' do before(:all) do @result = OctocatalogDiff::Integration.integration( spec_catalog_old: 'catalog-1.json', # The repository directory in /spec/octocatalog-diff/fixtures/catalogs spec_catalog_new: 'catalog-2.json', # The repository directory in /spec/octocatalog-diff/fixtures/catalogs argv: '--debug --display-format :color', # Command line arguments ) # At this point @result is a hash containing these keys: # @result[:logs] is a String containing everything printed to STDERR (Logger) # @result[:output] is a String containing everything printed to STDOUT # @result[:diffs] is an Array of differences # @result[:exitcode] is an Integer representing the exit code: 0 = no changes, 1 = failure, 2 = success, with changes # @result[:exception] contains any exception that was thrown end it 'should do whatever' do # ... end end ``` ## Hints for writing an integration test 0. If your integration test deals only with the calculation or display of differences, and not catalog compilation, and you are compiling catalogs multiple times, prefer `spec_catalog_old` and `spec_catalog_new` to pass in pre-compiled catalogs. This will make the test run faster. 0. It's a good idea to check the exit code in a test of its own, and then `pending` any subsequent tests if that exit code doesn't match. This way only one test, and not all the tests, will fail if the catalog compilation doesn't work. octocatalog-diff-1.5.4/doc/dev/coverage.md0000644000004100000410000000242013413744025020417 0ustar www-datawww-data# Test coverage We provide two types of tests for octocatalog-diff: - Unit tests (found in [/spec/octocatalog-diff/tests](/spec/octocatalog-diff/tests)) - Integration tests (found in [/spec/octocatalog-diff/integration](/spec/octocatalog-diff/integration)) The difference between these is as follows. Unit tests are designed to test the smallest bit of code that's practical to test. Integration tests are designed to run from end-to-end, starting via the invocation from the command line, exercising internals, and checking output from the end result. It's our goal to have as much test coverage as we can provide from both sides, so that we can have confidence in anything we release. ## Coverage report The `simplecov` gem is bundled with octocatalog-diff to produce coverage reports. To build a coverage report for unit tests only: ``` rake coverage:spec ``` To build a coverage report for integration tests only: ``` rake coverage:integration ``` To build a coverage report combining the results of unit tests and integration test: ``` rake coverage:all ``` Running any of these tests creates a `coverage` directory in the root of the project, and this contains an `index.html` file with a graphical report. Open this file in a browser to see the results. octocatalog-diff-1.5.4/doc/dev/how-to-add-options.md0000644000004100000410000001241713413744025022267 0ustar www-datawww-data# How to add new command line options This document contains a checklist and guidance to adding new command line options. ## Checklist Please copy and paste this text into your Pull Request. This will create boxes for each step along the way, which you can then check off when complete. ``` - [ ] REQUIRED: Add new file in `lib/octocatalog-diff/cli/options` - [ ] REQUIRED: Add corresponding test in `spec/octocatalog-diff/tests/cli/options` - [ ] OPTIONAL: Add default value in `lib/octocatalog-diff/cli.rb` - [ ] OPTIONAL: Add configuration example in `examples/octocatalog-diff.cfg.rb` - [ ] REQUIRED: Add code to implement your option in `lib` - [ ] REQUIRED: Add corresponding tests for code to implement your option in `spec/octocatalog-diff/tests` - [ ] OPTIONAL: Add an integration test to test your option in `spec/octocatalog-diff/integration` ``` ## Procedure ### Create option parser Option parsers are created in [`lib/octocatalog-diff/cli/options`](/lib/octocatalog-diff/cli/options). Your option should have a "long form" that contains dashes and not underscores. For example, you should prefer `--your-new-option` and NOT use `--your_new_option`. The file you create should reflect the name of your option. Generally the file name is the command line flag, with `-` converted to `_`. The key that your option creates in the options hash should generally be a symbol, named the same as the command line flag, with `-` converted to `_`. In this example, `:your_new_option`. If you are creating a binary (yes-no) option, please recognize both `--your-new-option` and `--no-your-new-option`. We recommend copying prior art as a template: - For a binary (yes-no) option, look at [`quiet.rb`](/lib/octocatalog-diff/cli/options/quiet.rb). - For an option that takes an integer parameter, look at [`retry_failed_catalog.rb`](/lib/octocatalog-diff/cli/options/retry_failed_catalog.rb). - For an option that takes an string parameter, look at [`bootstrap_script.rb`](/lib/octocatalog-diff/cli/options/bootstrap_script.rb). - For an option that takes an array or can be specified more than once, look at [`bootstrap_environment.rb`](/lib/octocatalog-diff/cli/options/bootstrap_environment.rb). If you can do simple validation of the argument, such as making sure the argument (if specified) matches a particular regular expression or is one of a particular set of values, please do that within the option file. For example, look at [`facts_terminus.rb`](/lib/octocatalog-diff/cli/options/facts_terminus.rb). ### Create test for option parser Option parser tests are created in [`spec/octocatalog-diff/tests/cli/options`](/spec/octocatalog-diff/tests/cli/options). If you used an existing option as a reference for your new code, consider using that option's test as a reference for your test. We have some methods, e.g. `test_with_true_false_option`, to avoid repetitive code for common patterns. If you have handled any edge cases, e.g. input validation, please add a test that expects an error when input is provided that does not match your validation. For example, look at [`parser_spec.rb`](/spec/octocatalog-diff/tests/cli/options/parser_spec.rb). ### Add default value (OPTIONAL) Unless specifically cleared with the project maintainers, adding a new option should not change the behavior of the program when that option isn't specified, and the new option should not be required. In other words, if someone invokes the program *without* specifying your option, it should behave in the same way as it did before your option was ever added. If you need to set a default value for your option, do so in [`lib/octocatalog-diff/cli.rb`](/lib/octocatalog-diff/cli.rb). Items should be added to DEFAULT_OPTIONS *only if* a value is required even if your option is not provided, or if you are defaulting something to *true* but providing an option to make it false. ### Add configuration example (OPTIONAL) If you believe your option is of general use to other users of octocatalog-diff such that they may wish to add it to their configuration file, update [`examples/octocatalog-diff.cfg.rb`](/examples/octocatalog-diff.cfg.rb) with some comments and example code for your option. Only the most commonly used options have entries in the example configuration file. If you are unsure whether or not to include your option there, please [open an issue](https://github.com/github/octocatalog-diff/issues/new) to discuss. As described in the default value section above, if your option is not configured in the template, the program should still work as it did before your new option was added. We want to avoid forcing users to update their configuration file unless there is a major update. ### Add code to implement your option This is for you to figure out! :smile_cat: ### Add corresponding tests for code Please add unit tests in [`spec/octocatalog-diff/tests`](/spec/octocatalog-diff/tests) that test the new behavior of any methods impacted by your changes. ### Add an integration test (OPTIONAL) Adding an integration test is optional, but very much appreciated. Integration tests are in [`spec/octocatalog-diff/integration`](/spec/octocatalog-diff/integration). Generally, if you're adding a new option, the integration test for that new option will go in its own file. Please see [integration tests](/doc/dev/integration-tests.md) for more. octocatalog-diff-1.5.4/doc/dev/releasing.md0000644000004100000410000000711513413744025020603 0ustar www-datawww-data# Releasing The project maintainers are responsible for bumping the version number, regenerating auto-generated documentation, tagging the release, and uploading to rubygems. ## Local testing *This procedure is performed by a GitHubber.* To test the new version of `octocatalog-diff` in the GitHub Puppet repository, check out `github/puppet` and: - Start a new branch based off master - Run `script/update-octocatalog-diff ` - Confirm and commit the result - Make sure all CI jobs pass - Run the `puppet-catalog-diff` CI job and make sure it passes and shows expected results ## Merging one PR This section is useful when releasing a new version based on one PR submitted by a contributor. Following this workflow is important so that the contributor gets appropriate credit in the revision history for his or her work. 0. In your local checkout, start a new branch based off master. 0. Add the contributor's repository as a remote. For example: ``` git remote add octocat https://github.com/octocat/octocatalog-diff.git ``` 0. Merge in the contributor's branch into your own. For example: ``` git merge octocat/some-branch ``` 0. Update `.version` and `doc/CHANGELOG.md` appropriately. In CHANGELOG you should link to the PR submitted by the contributor (not the PR you're creating now). 0. Commit your changes to `.version` and `doc/CHANGELOG.md`. 0. If necessary, auto-generate the build documentation, and commit the changes to your branch. ``` rake doc:build ``` 0. Open a Pull Request based on your branch. Confirm that the history is correct, showing the contributor's commits first, and then your commit(s) updating the version file, change log, and/or auto-generated documentation. 0. Ensure that CI tests are all passing. 0. Ensure that you've performed "local testing" within GitHub (typically, ~1 day) to confirm the changes. 0. Merge your PR and delete your branch. 0. Confirm that the contributor's PR now appears as merged, and any associated issues have been closed. ## Merging multiple PRs If multiple PRs will constitute a release, it's generally easier to merge each such PR individually, and then create a separate PR afterwards to update the necessary files. 0. Merge all constituent PRs and ensure that any associated issues have been closed. 0. Create your own branch based off master. 0. Update `.version` and `doc/CHANGELOG.md` appropriately. In CHANGELOG you should link to the PR submitted by the contributor (not the PR you're creating now). 0. Commit your changes to `.version` and `doc/CHANGELOG.md`. 0. If necessary, auto-generate the build documentation, and commit the changes to your branch. ``` rake doc:build ``` 0. Open a Pull Request based on your branch. 0. Ensure that CI tests are all passing. 0. Ensure that you've performed "local testing" within GitHub (typically, ~1 day) to confirm the changes. 0. Merge your PR and delete your branch. ## Releasing Generally, a new release will correspond to a merge to master of one or more Pull Requests. 0. Ensure that all changes associated with the release have been merged to master. - Merge all Pull Requests associated with release, including the version number bump, change log update, etc. - If necessary (for significant changes), complete a Pull Request to update the top-level README file. 0. Ensure the the master branch is checked out on your system. 0. Run the release procedure: ``` rake gem:release ``` This rake task handles the following: - Build the gem file (`rake gem:build`) - Tag the release in the repository (`rake gem:tag`) - Upload the gem file to rubygems (`rake gem:push`) octocatalog-diff-1.5.4/doc/dev/api.md0000644000004100000410000000042713413744025017402 0ustar www-datawww-data# octocatalog-diff API documentation The octocatalog-diff API allows developers to construct and work with certain octocatalog-diff internals in their own projects. The current API version is [API v1](/doc/dev/api/v1.md), which is available as of octocatalog-diff version 1.0. octocatalog-diff-1.5.4/doc/dev/api/0000755000004100000410000000000013413744025017055 5ustar www-datawww-dataoctocatalog-diff-1.5.4/doc/dev/api/v1/0000755000004100000410000000000013413744025017403 5ustar www-datawww-dataoctocatalog-diff-1.5.4/doc/dev/api/v1/objects/0000755000004100000410000000000013413744025021034 5ustar www-datawww-dataoctocatalog-diff-1.5.4/doc/dev/api/v1/objects/diff.md0000644000004100000410000002162513413744025022274 0ustar www-datawww-data# octocatalog-diff v1 API documentation: OctocatalogDiff::API::V1::Diff ## Overview `OctocatalogDiff::API::V1::Diff` is an object that represents a single difference between two catalogs. The return value from the `diffs` computed by the [`catalog-diff`](/doc/dev/api/v1/calls/catalog-diff.md) API call is an Array of these `OctocatalogDiff::API::V1::Diff` objects. ## Methods #### `#addition?` (Boolean) Returns true if this diff is an addition (resource exists in new catalog but not old catalog). #### `#change?` (Boolean) Returns true if this diff is a change (resource exists in both catalogs but is different between them). #### `#diff_type` (String) Returns the type of difference, which is one of the following characters: - `+` for addition (resource exists in new catalog but not old catalog) - `-` for removal (resource exists in old catalog but not new catalog) - `~` or `!` for change (resource exists in both catalogs but is different between them) See also: `#addition?`, `#change?`, `#removal?` Note: Internally `~` and `!` represent different types of changes, but when presenting the output, these can generally be considered equivalent. #### `#new_file` (String) Returns the filename of the Puppet manifest giving rise to the resource as it exists in the new catalog. Note that this is a pass-through of information provided in the Puppet catalog, and is not calculated by octocatalog-diff. If the Puppet catalog does not contain this information, this method will return `nil`. Note also that if the diff represents removal of a resource, this will return `nil`, because the resource does not exist in the new catalog. #### `#new_line` (String) Returns the line number within the Puppet manifest giving rise to the resource as it exists in the new catalog. (See `#new_file` for the filename of the Puppet manifest.) Note that this is a pass-through of information provided in the Puppet catalog, and is not calculated by octocatalog-diff. If the Puppet catalog does not contain this information, this method will return `nil`. Note also that if the diff represents removal of a resource, this will return `nil`, because the resource does not exist in the new catalog. #### `#new_location` (Hash) Returns a hash containing `:file` (equal to `#new_file`) and `:line` (equal to `#new_line`) when either is defined. Returns `nil` if both are undefined. #### `#new_value` (Object) Returns the value of the resource from the new catalog. - If a resource was added, this returns the data structure associated with the resource in the Puppet catalog. For example, if the resource was created as follows in the Puppet catalog, the `new_value` is as indicated. ``` # Resource in New Catalog { "type": "File", "title": "/etc/foo", "parameters": { "owner": "root", "content": "hello new world" } } # Demonstrates new_value diff.new_value #=> { 'parameters' => { 'owner' => 'root', 'content' => 'hello new world' } } ``` - If a resource was removed, this returns `nil` because there was no value of this resource in the new catalog. - If a resource was changed, this returns the portion of the data structure that is indicated by the `.structure` method. For example, if the resource existed as follows in both the old and new Puppet catalogs, the `new_value` is as indicated. ``` # Resource in Old Catalog { "type": "File", "title": "/etc/foo", "parameters": { "owner": "root", "content": "This is the old file" } } # Resource in New Catalog { "type": "File", "title": "/etc/foo", "parameters": { "owner": "root", "content": "This is the NEW FILE!!!!!" } } # Demonstrates structure and new_value diff.structure #=> ['parameters', 'content'] diff.new_value #=> 'This is the NEW FILE!!!!!' ``` #### `#old_file` (String) Returns the filename of the Puppet manifest giving rise to the resource as it exists in the old catalog. Note that this is a pass-through of information provided in the Puppet catalog, and is not calculated by octocatalog-diff. If the Puppet catalog does not contain this information, this method will return `nil`. Note also that if the diff represents addition of a resource, this will return `nil`, because the resource does not exist in the old catalog. #### `#old_line` (String) Returns the line number within the Puppet manifest giving rise to the resource as it exists in the old catalog. (See `#old_file` for the filename of the Puppet manifest.) Note that this is a pass-through of information provided in the Puppet catalog, and is not calculated by octocatalog-diff. If the Puppet catalog does not contain this information, this method will return `nil`. Note also that if the diff represents addition of a resource, this will return `nil`, because the resource does not exist in the old catalog. #### `#old_location` (Hash) Returns a hash containing `:file` (equal to `#old_file`) and `:line` (equal to `#old_line`) when either is defined. Returns `nil` if both are undefined. #### `#old_value` (Object) Returns the value of the resource from the old catalog. - If a resource was added, this returns `nil` because there was no value of this resource in the old catalog. - If a resource was removed, this returns the data structure associated with the resource in the Puppet catalog. For example, if the resource existed as follows in the Puppet catalog, the `old_value` is as indicated. ``` # Resource in Old Catalog { "type": "File", "title": "/etc/foo", "parameters": { "owner": "root", "content": "hello old world" } } # Demonstrates old_value diff.old_value #=> { 'parameters' => { 'owner' => 'root', 'content' => 'hello old world' } } ``` - If a resource was changed, this returns the portion of the data structure that is indicated by the `.structure` method. For example, if the resource existed as follows in both the old and new Puppet catalogs, the `old_value` is as indicated. ``` # Resource in Old Catalog { "type": "File", "title": "/etc/foo", "parameters": { "owner": "root", "content": "This is the old file" } } # Resource in New Catalog { "type": "File", "title": "/etc/foo", "parameters": { "owner": "root", "content": "This is the NEW FILE!!!!!" } } # Demonstrates structure and old_value diff.structure #=> ['parameters', 'content'] diff.old_value #=> 'This is the old file' ``` #### `#removal?` (Boolean) Returns true if this diff is a removal (resource exists in old catalog but not new catalog). #### `#structure` (Array) Returns the structure that has been changed, as an array. When a resource was added or removed, the result is an empty array. That's because all of the parameters and other metadata from the resource exist entirely in one catalog but not the other. When a resource has changed, one diff is created for each parameter that changed. For example, both the old and new catalogs contain the file resource `/etc/foo` but just the content has changed: ``` # Old file { '/etc/foo': owner => 'root', content => 'This is the old file', } # New file { '/etc/foo': owner => 'root', content => 'This is the NEW FILE!!!!!', } ``` Internally, the Puppet catalog for this resource will look like this in the catalogs (this has been abbreviated a bit for clarity): ``` # Old { "type": "File", "title": "/etc/foo", "exported": false, "parameters": { "owner": "root", "content": "This is the old file" } } # New { "type": "File", "title": "/etc/foo", "exported": false, "parameters": { "owner": "root", "content": "This is the NEW FILE!!!!!" } } ``` One diff will be generated to represent the change to the content of the file (which in the catalog is nested in the 'parameters' hash). The diff will be structured as follows: ``` diff.type #=> 'File' diff.title #=> '/etc/foo' diff.structure #=> ['parameters', 'content'] ``` #### `#title` (String) Returns the title of the resource from the Puppet catalog. For example, a diff involving `File['/etc/passwd']` would have: - `diff.title #=> '/etc/passwd'` - `diff.type #=> 'File'` #### `#type` (String) Returns the type of the resource from the Puppet catalog. For example, a diff involving `File['/etc/passwd']` would have: - `diff.title #=> '/etc/passwd'` - `diff.type #=> 'File'` Note that the type will be capitalized because Puppet capitalizes this in catalogs. ## Other methods These methods are available for debugging or development purposes but are not guaranteed to remain consistent between versions: - `#inspect` (String): Returns inspection of object - `#raw` (Array): Returns internal array data structure of the "diff" - `#to_h` (Hash): Returns object as a hash, where keys are above described methods - `#to_h_with_string_keys` (Hash): Returns object as a hash, where keys are above described methods; keys are strings, not symbols - `#[]` (Object): Retrieve indexed array elements from raw internal array object octocatalog-diff-1.5.4/doc/dev/api/v1/objects/catalog.md0000644000004100000410000001102713413744025022771 0ustar www-datawww-data# octocatalog-diff v1 API documentation: OctocatalogDiff::API::V1::Catalog ## Overview `OctocatalogDiff::API::V1::Catalog` is an object that represents a compiled catalog. It wraps the [`OctocatalogDiff::Catalog`](/lib/octocatalog-diff/catalog.rb) object. This object is the return value from the [`catalog`](/doc/dev/api/v1/calls/catalog.md) API call, and the `to` and `from` catalogs computed by the [`catalog-diff`](/doc/dev/api/v1/calls/catalog-diff.md) API call. ## Methods #### `#builder` (String) Returns the name of the class the built the catalog. ``` catalog.builder #=> 'OctocatalogDiff::Catalog::JSON' ``` #### `#compilation_dir` (String) Returns the temporary directory in which the catalog was compiled. ``` catalog.compilation_dir #=> '/var/folders/dw/5ftmkqk972j_kw2fdjyzdqdw0000gn/T/d20170108-95774-1r4ohjd' ``` This is only applicable for catalogs compiled by `OctocatalogDiff::Catalog::Computed`. This method will return `nil` for catalogs generated by other backends. #### `#error_message` (String) Returns the error message for a failed catalog. (If the catalog is valid, this returns `nil`.) ``` good_catalog.valid? #=> true good_catalog.error_message #=> nil bad_catalog.valid? #=> false bad_catalog.error_message #=> 'Failed to compile catalog for node ...' ``` #### `#puppet_version` (String) Returns the Puppet version used to compile the catalog. ``` catalog.puppet_version #=> '3.8.7' ``` This is only applicable for catalogs compiled by `OctocatalogDiff::Catalog::Computed`. This method will return `nil` for catalogs generated by other backends. #### `#resource()` (Hash) Returns the hash representation of the object identified by the type and title supplied. This method requires 1 argument, which is a hash containing `:type` and `:title` keys. ``` catalog.resource(type: 'File', title: '/etc/foo') #=> {"title"=>"/etc/foo", "type"=>"File", "parameters"=>{"content"=>"This is the file", "owner"=>"root"}, "file"=>"/etc/puppetlabs/code/environments/production/manifests/site.pp", "line"=>25} ``` Returns `nil` if the type+title resource was not present in the catalog. ##### Notes 1. The first call to this method will build a hash table (`O(N)` operation) from the resource array. Thereafter, each call to this method will look up the value in that hash table (`O(1)` operation). To perform lookups of known types and titles, it is faster to use this method than to use `#resources` with array operations when performing multiple lookups. 2. The structure of the returned hash is dependent on the representation of the resource in the Puppet catalog. It is therefore possible that different versions of Puppet could cause different data structures. It is the author's experience that Puppet 3.8.7 and Puppet 4.x produce similar (if not indistinguishable) resource representations. 3. This method will also locate a resource that was named using an alias. ``` # Puppet code file { '/etc/foo': alias => 'foo file for the win', ... } # octocatalog-diff API catalog.resource(type: 'File', title: 'foo file for the win') #=> {"title"=>"/etc/foo", "type"=>"File", ...} ``` #### `#resources` (Array<Hash>) Returns an array of catalog resources in a successful catalog. (If the catalog is not valid, this returns `nil`.) ``` bad_catalog.valid? #=> false bad_catalog.resources #=> nil good_catalog.valid? #=> true good_catalog.resources #=> [ { "title"=>"/etc/foo", "type"=>"File", ... }, { "title"=>"/bin/true", "type"=>"Exec", ... } ] ``` The structure of the returned hash is dependent on the representation of the resource in the Puppet catalog. It is therefore possible that different versions of Puppet could cause different data structures. It is the author's experience that Puppet 3.8.7 and Puppet 4.x produce similar (if not indistinguishable) resource representations. #### `#to_json` (String) Returns the JSON representation of a successful catalog. (If the catalog is not valid, this returns `nil`.) ``` bad_catalog.valid? #=> false bad_catalog.to_json #=> nil good_catalog.valid? #=> true good_catalog.to_json #=> '{ "document_type": "Catalog", ... }' ``` #### `#valid?` (Boolean) Returns `true` if the catalog is valid, and `false` if the catalog is not valid. ## Other methods These methods are available for debugging or development purposes but are not guaranteed to remain consistent between versions: - `#to_h` (Hash): Returns hash representation of parsed JSON catalog - `#raw` (OctocatalogDiff::Catalog): Returns underlying internal catalog object octocatalog-diff-1.5.4/doc/dev/api/v1/objects/override.md0000644000004100000410000000174313413744025023202 0ustar www-datawww-data# octocatalog-diff v1 API documentation: OctocatalogDiff::API::V1::Override ## Overview `OctocatalogDiff::API::V1::Override` is an object that represents a user-supplied fact or ENC parameter that will be used when compiling a catalog. ## Constructor #### `#new( { key: , value: })` The hash must contain the following keys: - `:key` (String) - The name of the fact or ENC parameter (e.g. `operatingsystem` or `parameters::fooclass::fooparam`) - `:value` (?) - The value of the fact or ENC parameter See also: `#create_from_input` ## Methods #### `#create_from_input( key=value)` (OctocatalogDiff::API::V1::Override) Parses the string (see [Overriding facts](/doc/advanced-override-facts.md) for the format to use). Returns a `OctocatalogDiff::API::V1::Override` object with key and value parsed from the string. #### `#key` (String) Returns the key as supplied in the constructor. #### `#value` (?) Returns the value as supplied in the constructor. octocatalog-diff-1.5.4/doc/dev/api/v1/calls/0000755000004100000410000000000013413744025020501 5ustar www-datawww-dataoctocatalog-diff-1.5.4/doc/dev/api/v1/calls/catalog.md0000644000004100000410000001231713413744025022441 0ustar www-datawww-data# octocatalog-diff v1 API documentation: catalog ## Overview `catalog` returns an `OctocatalogDiff::Catalog` object built with the octocatalog-diff compiler, obtained from a Puppet server, or read in from a file. This is analogous to using the `--catalog-only` option with the octocatalog-diff command-line script. ``` catalog_obj = OctocatalogDiff::API::V1.catalog() #=> OctocatalogDiff::API::V1::Catalog ``` The return value is an [`OctocatalogDiff::API::V1::Catalog`](/doc/dev/api/v1/objects/catalog.md) object. For an example, see [catalog-builder-local-files.rb](/examples/api/v1/catalog-builder-local-files.rb). ## Parameters The `catalog` method takes one argument, which is a Hash containing parameters. The list of parameters here is not exhaustive. The `.catalog` method accepts most parameters described in [Configuration](/doc/configuration.md), [Building catalogs instead of diffing catalogs](/doc/advanced-catalog-only.md), and [Command line options reference](/doc/optionsref.md). It is also possible to use the parameters from [OctocatalogDiff::API::V1.config](/doc/dev/api/v1/calls/config.md) for the catalog compilation. Simply combine the hash returned by `.config` with any additional keys, and pass the merged hash to the `.catalog` method. ### Global parameters #### `:logger` (Logger, Optional) Debugging and informational messages will be logged to this object as the catalog is built. If no Logger object is passed, these messages will be silently discarded. #### `:node` (String, Required) The node name whose catalog is to be compiled or obtained. This should be the fully qualified domain name that matches the node's name as seen in Puppet. ### Computed catalog parameters #### `:basedir` (String, Optional) Directory that contains a git repository with the Puppet code. Use in conjunction with `:to_branch` to specify the branch name that should be checked out. If your Puppet code is not in a git repository, or you already have the branch checked out via some other process, use `:bootstrapped_to_dir` instead. #### `:bootstrap_script` (String, Optional) Path to a script to run after checking out the selected branch of Puppet code from the git repository. See [Bootstrapping your Puppet checkout](/doc/advanced-bootstrap.md) for details. #### `:bootstrapped_to_dir` (String, Optional) Directory that is already prepared ("bootstrapped") and can have Puppet run against its contents to compile the catalog. Often this means that you have done the following to this directory: - Checked out the necessary branch of Puppet code from your version control system - Installed any third party modules (e.g. with librarian-puppet or r10k) #### `:enc` (String, Optional) File path to your External Node Classifier. See [Configuring octocatalog-diff to use ENC](/doc/configuration-enc.md) for details on using octocatalog-diff with an External Node Classifier. #### `:fact_file` (String, Optional) Path to the file that contains the facts for the node whose catalog you are going to compile. Generally, must either specify the fact file, or [configure octocatalog-diff to use PuppetDB](/doc/configuration-puppetdb.md) to retrieve node facts. #### `:hiera_config` (String, Optional) Path to the Hiera configuration file (generally named `hiera.yaml`) for your Puppet installation. Please see [Configuring octocatalog-diff to use Hiera](/doc/configuration-hiera.md) for details on Hiera configuration. #### `:hiera_path` (String, Optional) Directory within your Puppet installation where Hiera data is stored. Please see [Configuring octocatalog-diff to use Hiera](/doc/configuration-hiera.md) for details on Hiera configuration. If your Puppet setup is modeled after the [Puppet control repository template](https://github.com/puppetlabs/control-repo), the correct setting for `:hiera_path` is `'hieradata'`. #### `:puppet_binary` (String, Required) Path to the Puppet binary on your system. Please refer to [Configuring octocatalog-diff to use Puppet](/doc/configuration-puppet.md) for details of connecting octocatalog-diff to your Puppet installation. #### `:puppetdb_url` (String, Optional) URL to PuppetDB. See [Configuring octocatalog-diff to use PuppetDB](/doc/configuration-puppetdb.md) for instructions on this setting, and other settings that may be needed in your environment. #### `:to_branch` (String, Optional) Branch name in git repository to use for Puppet code. This option must be used in conjunction with `:basedir` so that code can be located. ## Exceptions The following exceptions may occur during the compilation of a catalog: - `OctocatalogDiff::Errors::BootstrapError` Bootstrapping failed. - `OctocatalogDiff::Errors::CatalogError` Catalog failed to compile. Please note that whenever possible, a `OctocatalogDiff::API::V1::Catalog` object is still constructed for a failed catalog, with `#valid?` returning false. It's also possible that the catalog contained broken references -- see [Catalog validation](/doc/advanced-catalog-validation.md). - `OctocatalogDiff::Errors::GitCheckoutError` Git checkout failed. - `OctocatalogDiff::Errors::PuppetVersionError` The version of Puppet could not be determined, generally because the Puppet binary was not found, or does not respond as expected to `puppet version`. octocatalog-diff-1.5.4/doc/dev/api/v1/calls/config.md0000644000004100000410000000334513413744025022275 0ustar www-datawww-data# octocatalog-diff v1 API documentation: config ## Overview `config` reads and parses an [octocatalog-diff configuration file](/doc/configuration.md). ``` options = OctocatalogDiff::API::V1.config( filename: "String", logger: Logger, test: ) ``` ## Options - **`:filename`** (String, optional): Full path to configuration file to read. If not provided, the configuration file will be searched as described in [Configuration](/doc/configuration.md). - **`:logger`** (Logger, optional): Logger object. If provided, debug messages and fatal errors will be logged to this object. - **`:test`** (Boolean, optional): Test mode, defaults to false. If true, the value of the configuration settings will be logged to the logger (with priority DEBUG) and an exception will be raised if the configuration file cannot be located. ## Return value If the configuration file is located and valid, the return value is a Hash consisting of the options defined in the configuration file. If the configuration file cannot be found, the return value is an empty Hash (`{}`). Except, with `:test => true`, an exception will be raised. ## Exceptions - `OctocatalogDiff::Errors::ConfigurationFileContentError` Raised if the configuration file could not be evaluated. A more specific error message will help identify the cause. Possible causes include the file not being valid ruby, the file not containing the expected structure or methods, or the method returning something other than a Hash. - `OctocatalogDiff::Errors::ConfigurationFileNotFoundError` Raised if the configuration file could not be found, *and* `:test => true` was supplied. (Note, if no configuration file is found, but `:test => false`, no error is raised, and `{}` is returned.) octocatalog-diff-1.5.4/doc/dev/api/v1/calls/catalog-diff.md0000644000004100000410000002455613413744025023357 0ustar www-datawww-data# octocatalog-diff v1 API documentation: catalog-diff ## Overview `catalog-diff` allows you compare two catalogs and obtain the differences between them. Catalogs can be built if necessary. This is analogous to using the default arguments with the octocatalog-diff command-line script. ``` catalog_diff_result = OctocatalogDiff::API::V1.catalog_diff() #=> { diffs: Array, from: OctocatalogDiff::API::V1::Catalog, to: OctocatalogDiff::API::V1::Catalog } ``` Return values: - [`OctocatalogDiff::API::V1::Diff`](/doc/dev/api/v1/objects/diff.md) - [`OctocatalogDiff::API::V1::Catalog`](/doc/dev/api/v1/objects/catalog.md) For an example, see [catalog-diff-local-files.rb](/examples/api/v1/catalog-diff-local-files.rb). ## Parameters The `catalog_diff` method takes one argument, which is a Hash containing parameters. The list of parameters here is not exhaustive. The `.catalog_diff` method accepts most parameters described in [Configuration](/doc/configuration.md), [Advanced options](/doc/advanced.md), and [Command line options reference](/doc/optionsref.md). It is also possible to use the parameters from [OctocatalogDiff::API::V1.config](/doc/dev/api/v1/calls/config.md) for the catalog-diff calculation. Simply combine the hash returned by `.config` with any additional keys, and pass the merged hash to the `.catalog_diff` method. ### Global parameters #### `:logger` (Logger, Optional) Debugging and informational messages will be logged to this object as the catalogs are built and differences are computed. If no Logger object is passed, these messages will be silently discarded. #### `:node` (String, Required) The node name whose catalogs are to be compiled and differences obtained obtained. This should be the fully qualified domain name that matches the node's name as seen in Puppet. ### Computed catalog parameters #### `:basedir` (String, Optional) Directory that contains a git repository with the Puppet code. Use in conjunction with `:to_branch` and `:from_branch` to specify the branch names that should be checked out. If your Puppet code is not in a git repository, or you already have the branches checked out via some other process, use `:bootstrapped_to_dir` and `:bootstrapped_from_dir` instead. #### `:bootstrap_script` (String, Optional) Path to a script to run after checking out the selected branch of Puppet code from the git repository. See [Bootstrapping your Puppet checkout](/doc/advanced-bootstrap.md) for details. #### `:bootstrapped_to_dir` / `:bootstrapped_from_dir` (String, Optional) Directories that is already prepared ("bootstrapped") and can have Puppet run against its contents to compile the catalog. `:bootstrapped_to_dir` is used for the "to" catalog, while `:bootstrapped_from_dir` is used for the "from" catalog. Often this means that you have done the following to this directory: - Checked out the necessary branch of Puppet code from your version control system - Installed any third party modules (e.g. with librarian-puppet or r10k) You may mix and match `:bootstrapped_XXX_dir` and `:basedir` + `:XXX_branch`. For example, you may specify `:bootstrapped_from_dir`, `:basedir`, and `:to_branch`, which will compile the "from" catalog in the bootstrapped "from" directory you specified, and the "to" catalog from the "to_branch" branch of the git repository found in the "basedir". #### `:enc` / `:to_enc` / `:from_enc` (String, Optional) File path to your External Node Classifier. See [Configuring octocatalog-diff to use ENC](/doc/configuration-enc.md) for details on using octocatalog-diff with an External Node Classifier. In most cases, you will use the `:enc` option to set the ENC for both the "to" and "from" catalogs. In the case that you are comparing two different ENCs, you may use `:to_enc` and/or `:from_enc` as needed. #### `:fact_file` / `:to_fact_file` / `:from_fact_file` (String, Optional) Path to the file that contains the facts for the node whose catalog you are going to compile. Generally, must either specify the fact file, or [configure octocatalog-diff to use PuppetDB](/doc/configuration-puppetdb.md) to retrieve node facts. In most cases, you will use the `:fact_file` option to set the fact file for both the "to" and "from" catalogs. In the case that you are comparing the results from two different sets of facts, you may use `:to_fact_file` and/or `:from_fact_file` as needed. #### `:hiera_config` (String, Optional) Path to the Hiera configuration file (generally named `hiera.yaml`) for your Puppet installation. Please see [Configuring octocatalog-diff to use Hiera](/doc/configuration-hiera.md) for details on Hiera configuration. #### `:hiera_path` (String, Optional) Directory within your Puppet installation where Hiera data is stored. Please see [Configuring octocatalog-diff to use Hiera](/doc/configuration-hiera.md) for details on Hiera configuration. If your Puppet setup is modeled after the [Puppet control repository template](https://github.com/puppetlabs/control-repo), the correct setting for `:hiera_path` is `'hieradata'`. #### `:puppet_binary` / `:to_puppet_binary` / `:from_puppet_binary` (String, Required) Path to the Puppet binary on your system. Please refer to [Configuring octocatalog-diff to use Puppet](/doc/configuration-puppet.md) for details of connecting octocatalog-diff to your Puppet installation. In most cases, you will use the `:puppet_binary` option to set the Puppet binary for both the "to" and "from" catalogs. In the case that you are comparing the catalogs produced by two different versions of Puppet, you may use `:to_puppet_binary` and/or `:from_puppet_binary` as needed. #### `:puppetdb_url` (String, Optional) URL to PuppetDB. See [Configuring octocatalog-diff to use PuppetDB](/doc/configuration-puppetdb.md) for instructions on this setting, and other settings that may be needed in your environment. #### `:to_branch` / `:from_branch` (String, Optional) Branch name in git repository to use for Puppet code. Each option must be used in conjunction with `:basedir` so that code can be located. If you have specified `:bootstrapped_from_dir` or `:from_catalog`, then `:from_branch` will be ignored. If you have specified `:bootstrapped_to_dir` or `:to_catalog`, then `:from_branch` will be ignored. #### `:to_catalog` / `:from_catalog` (String, Optional) If you have already compiled a catalog, set `:to_catalog` and/or `:from_catalog` to the full path to the catalog file. If you specify a `:XXX_catalog` setting, this will cause `:bootstrapped_XXX_dir` and `:XXX_branch` parameters to be ignored. ### Controlling diffs #### `:ignore` (Array<Hash>, Optional) Populating the `:ignore` array filters out matching differences from the overall result using the built-in logic. Please refer to [Ignoring certain changes from the command line](/doc/advanced-ignores.md) for details. The expected data structure for ignoring a type (e.g. `File` or `Exec`) or title is to construct a hash with a regular expression, like this. ``` # Simple definition, ignores File[/etc/foo] [ { type: Regexp.new('\AFile\z'), title: Regexp.new('\A/etc/foo\z') } ] # More complicated definition, ignores files in 'tmp' or '.tmp' directories wherever they exist [ { type: Regexp.new('\AFile\z'), title: Regexp.new('/\.?tmp/') } ] # Ignore all anchors [ { type: Regexp.new('\AAnchor\z') } ] # Ignore all resources of any type that contain 'foo' in the title [ { title: Regexp.new('foo') } ] ``` To ignore based on attributes, it is important to understand that each catalog resource is structured as a hash, sometimes with depth greater than 1. An example of Puppet code and the corresponding catalog structure is shown here: ``` # Puppet code file { '/etc/hostname': owner => 'root', notify => [ Service['postfix'], Exec['update hostname'] ], content => $::fqdn, } # In the catalog (somewhat abbreviated for brevity)... { "type": "File", "title": "/etc/hostname", "parameters": { "owner": "root", "notify": [ "Service[postfix]", "Exec[update hostname]" ], "content": "foo-bar.example.com" } } ``` In this case, "owner", "notify", and "content" are nested under "parameters". Internally in octocatalog-diff, this nesting is represented by the "\f" character. (We chose a character that you should have no reason to put in your resource titles or key names.) Thus, the following examples work: ``` # Ignore all changes to the `owner` attribute of a file. [ { type: Regexp.new('\AFile\z'), attr: Regexp.new("\\Aparameters\fowner\\z" } ] # Ignore changes to `owner` or `group` for a file or an exec. [ { type: Regexp.new('\A(File|Exec)\z'), attr: Regexp.new("\\Aparameters\f(owner|group)\\z" } ] ``` When using regular expressions, `\f` (form feed character) is used to separate the structure (e.g. `parameters\fowner` refers to the `parameters` hash, `owner` key). :bulb: Note that `\A` in Ruby matches the beginning of the string and `\z` matches the end, but these are not actual characters. Therefore, if you are using `\A` or `\z` in double quotes (`"`), be sure to heed the examples above and write your expression like: `Regexp.new("\\Aparameters\fowner\\z")`. #### `:validate_references` (Array<String>, Optional) Invoke the [catalog validation](/doc/advanced-catalog-validation.md) feature to ensure resources targeted by `before`, `notify`, `require`, and/or `subscribe` exist in the catalog. If this parameter is not defined, no reference validation occurs. You must set this parameter to an **Array** that contains one or more of the following values: - `before` - `notify` - `require` - `subscribe` ## Exceptions The following exceptions may occur during the compilation of a catalog within the catalog-diff operation: - `OctocatalogDiff::Errors::BootstrapError` Bootstrapping failed. - `OctocatalogDiff::Errors::CatalogError` Catalog failed to compile. Please note that whenever possible, a `OctocatalogDiff::API::V1::Catalog` object is still constructed for a failed catalog, with `#valid?` returning false. It's also possible that the catalog contained broken references -- see [Catalog validation](/doc/advanced-catalog-validation.md). - `OctocatalogDiff::Errors::GitCheckoutError` Git checkout failed. - `OctocatalogDiff::Errors::PuppetVersionError` The version of Puppet could not be determined, generally because the Puppet binary was not found, or does not respond as expected to `puppet version`. octocatalog-diff-1.5.4/doc/dev/api/v1.md0000644000004100000410000000301713413744025017726 0ustar www-datawww-data# octocatalog-diff v1 API documentation ## API calls #### catalog `catalog` allows you to build a catalog using the octocatalog-diff compiler or to obtain a catalog from a Puppet server. [Read more about the `catalog` call](/doc/dev/api/v1/calls/catalog.md) #### catalog-diff `catalog-diff` allows you compare two catalogs and obtain the differences between them. Catalogs can be built if necessary. [Read more about the `catalog-diff` call](/doc/dev/api/v1/calls/catalog-diff.md) #### config `config` allows you read and parse an [octocatalog-diff configuration file](/doc/configuration.md). [Read more about the `config` call](/doc/dev/api/v1/calls/config.md) ## Objects #### OctocatalogDiff::API::V1::Catalog The `OctocatalogDiff::API::V1::Catalog` object represents a compiled catalog and supports several methods to get information about the catalog. [Read more about the `OctocatalogDiff::API::V1::Catalog` object](/doc/dev/api/v1/objects/catalog.md) #### OctocatalogDiff::API::V1::Diff The `OctocatalogDiff::API::V1::Diff` object represents a difference between two catalogs and supports several methods to get information about the difference. [Read more about the `OctocatalogDiff::API::V1::Diff` object](/doc/dev/api/v1/objects/diff.md) #### OctocatalogDiff::API::V1::Override The `OctocatalogDiff::API::V1::Override` object represents a user-supplied fact or ENC parameter that will be used when compiling a catalog. [Read more about the `OctocatalogDiff::API::V1::Override` object](/doc/dev/api/v1/objects/override.md) octocatalog-diff-1.5.4/doc/similar.md0000644000004100000410000000236713413744025017520 0ustar www-datawww-data# Similar Projects We are aware of the following projects that do similar things to octocatalog-diff: - [Puppet's catalog_preview Puppet module](https://forge.puppet.com/puppetlabs/catalog_preview) Installs as a module into your Puppet codebase and helps with migration from older Puppet versions to newer ones, or from open source Puppet to Puppet Enterprise. Also provides the ability to compare environments (branches). Requires a full working Puppet installation. - [Zack Smith's catalog_diff Puppet module](https://forge.puppet.com/zack/catalog_diff) Installs as a module into your Puppet codebase, allowing you to diff catalogs created by different versions of Puppet. Requires a full working Puppet installation. - [camptocamp's puppet-catalog-diff-viewer](https://github.com/camptocamp/puppet-catalog-diff-viewer) A viewer for JSON reports produced by the catalog_diff Puppet module. `octocatalog-diff` differs from the above projects by running on a system without a fully configured Puppet installation (such as a developer workstation or CI server). This approach allows developers to run it without having access to the production Puppet servers, and it does not put any load on production Puppet masters when it compiles and compares catalogs. octocatalog-diff-1.5.4/doc/CHANGELOG.md0000644000004100000410000002273213413744025017345 0ustar www-datawww-data# octocatalog-diff change log
Version Date Description / Changes
1.5.4 2018-12-11
  • #190: (Enhancement) Additional filtered out cases for compilation directory
  • #195: (Enhancement) Parallel catalog-diff when multiple hostnames are passed
  • #198: (Bug Fix) Portability fixes
  • #200: (Bug Fix) Support name parameter when validating references
    		•
    1.5.3 2018-03-05
  • #176: (Enhancement) Normalize file resource titles in reference checks
    		•
    1.5.2 2017-12-19
  • #169: (Enhancement) Puppet Enterprise RBAC token to authenticate to PuppetDB
  • #170: (Enhancement) Filter to treat an object the same as a single array containing that object
  • #165: (Bug Fix) Override of fact file via CLI now has precedence over value set in configuration file
    		•
    1.5.1 2017-11-16
  • #159: (Enhancement) Add support for puppetdb behind basic auth
    		•
    1.5.0 2017-10-18
  • #151: (Enhancement) Support text differences in files where `source` is an array
  • #153: (Enhancement) Support for hiera 5
  • #152: (Internal) Better temporary directory handling
    		•
    1.4.1 2017-10-02
  • #149: (Internal) Set ports on PuppetDB URLs without altering constants
    		•
    1.4.0 2017-08-03
  • #135: (Enhancement) Puppet 5 compatibility
  • #140: (Internal) Prefix tmpdirs with ocd-
  • #138: (Internal) Refactor catalog class with proper inheritance
    		•
    1.3.0 2017-06-09
  • #121: (Enhancement) Allow different fact files for the "from" and "to" catalogs
  • #129: (Enhancement) Allow YAML facts in "facter -y" format
  • #126: (Enhancement) Allow saving of catalogs when catalog diffing
  • #122: (Bug) Handle File resources with no parameters
  • #125: (Bug) Fix error when parameters with integer values are added
  • #131: (Bug) Do not use override fact file for both catalogs when only `--to-fact-file` is specified
    		•
    1.2.0 2017-05-18
  • #112: Split arguments added for ENC
  • #113: (Enhancement) Override facts and ENC parameters using regular expressions
  • #111: Simplify parallel processing to solve some intermittent failures
  • #110: Ruby 2.4 compatibility
    		•
    1.1.0 2017-05-08
  • #108: (Bug) Support hiera.yaml backend declared as a string instead of array
  • #105: (Bug) Remove legacy exclusion of tags
  • #103: (Enhancement) Identify where the broken reference was declared
  • #98: (Enhancement) Separate scripts and commands and make override-able
    		•
    1.0.4 2017-03-17
  • #94: Make Puppet version check respect env vars
    		•
    1.0.3 2017-03-15
  • #86: Ability to use `--environment` without `--preserve-environments`
    		•
    1.0.2 2017-03-08
  • #91: `--no-truncate-details` option
    		•
    1.0.1 2017-02-14
  • #84: Add JSON equivalence filter
  • #83: Retries for Puppet Master retrieval
  • #82: Command line option for Puppet Master timeout
    		•
    1.0.0 2017-02-06 This is the first release of the 1.0 series. For more information please see What's new in octocatalog-diff 1.0.

    The most significant change in version 1.0 is the addition of the V1 API, which permits developers to build catalogs (--catalog-only) and compare/diff catalogs using octocatalog-diff. Under the hood, we've rearranged the code to support these APIs, which should improve the reliability and allow faster development cycles.

    Breaking Changes

    The format of the output from --output-format json has changed. In version 0.x of the software, each difference was represented by an array. In version 1.x, each difference is represented by a hash with meaningful English keys. We have added an option --output-format legacy_json for anyone who may depend on the old format.
    0.6.1 2017-01-07
    • #46: Add option to ignore whitespace in yaml file diff
    0.6.0 2017-01-04
    • #45: Support for alternate environments in hiera configuration
    • #43: Consider aliased resources in validation
    • #39: Pass command line arguments to Puppet during catalog compilation
    • #38: Preserve and select environments
    • #37: Consistent sorting of equally weighted options
    • #36: Validate before, notify, require, subscribe references
    • #34: Allow bootstrap script to start with /
    • #33: Double-escape facts passed to Puppet master
    • #32: Rewrite hiera data directory for multiple backends
    • #24: Support PuppetDB API v3
    0.5.6 2016-11-16
    • #20: Use modulepath from environment.conf to inform lookup directories for --compare-file-text feature
    0.5.5 - Unreleased internal version
    0.5.4 2016-11-07
    • #16: environment running puppet --version
    • #5: bootstrap debugging
    • #17: hiera simplification and --hiera-path option
    0.5.3 2016-10-31
    • #10: facts terminus optimization
    0.5.2 - Unreleased internal version
    0.5.1 2016-10-20 Initial release
    octocatalog-diff-1.5.4/doc/advanced-ignores.md0000644000004100000410000003352413413744025021270 0ustar www-datawww-data# Ignoring certain changes from the command line `octocatalog-diff` provides a means to ignore certain changes in the displayed output. You may choose to ignore a change because you know it has no effect on the system. Or perhaps you are aware that the change is being made (perhaps in many different places) and you want to suppress it so other changes will not be lost in the noise. ## Built-in ignores `octocatalog-diff` automatically ignores any changes to the following: - Classes and class parameters. Classes themselves are structures to contain resources, but do not themselves trigger actions on agents. Resources in classes are reported on, but the actual classes themselves, and parameters associated with the classes, are not. - Tags. Tags are useful for classification and collecting resources, but tags themselves do not trigger actions on agents. If you would like to see changes to tags, you can use `--no-ignore-tags`. Please note that tags are sorted alphabetically before comparison, so differences to the order of the tags will not ever show as a difference. - Resource attributes: `before` and `require`. These attributes control ordering on the agent, but we have found that displaying them in the diff has little value because the target resources are not often seen. If you are looking to visualize your infrastructure, Puppet Enterprise [now has that feature](https://puppet.com/blog/visualize-your-infrastructure-models). Please note: `octocatalog-diff` does display changes to `subscribe` and `notify` parameters. ## Ignoring via the `--ignore` command line option If you specify multiple `--ignore` options, they are OR'd. In other words, if a change matches *any* of the ignored conditions, it is ignored. ### Ignoring by type If you wish to ignore all changes to a certain resource type, use this syntax. For example, if you wanted to ignore all changes to 'exec' you would use: --ignore 'Exec[*]' Or to ignore changes to a custom defined type, you would use: --ignore 'Your::Custom::Type[*]' The matching is case insensitive, so `--ignore Exec[*]` and `--ignore exec[*]` are equivalent. ### Ignoring by type and title If you wish to ignore all changes to a certain resource identified by its type and title, use this syntax. For example, to ignore all changes to your `/etc/motd` file you would use: --ignore 'File[/etc/motd]' You can use wildcards in the title. Wildcards can be placed at the beginning, in the middle, or at the end of the title. You can also use multiple wildcards. `*` is the only wildcard supported, and it matches 0 or more characters. For example: --ignore 'File[*]' - Ignores all files --ignore 'File[/etc/foo/*]' - Ignores all files in or under '/etc/foo' --ignore 'File[*/foo]' - Ignores files named 'foo' anywhere in the file system --ignore 'File[/etc/*/foo]' - Ignores files named 'foo' in subdirectories of '/etc/' --ignore 'File[/etc/*/foo/*]' - Ignores all files under subdirectories named 'foo' under '/etc' Note that unlike on a unix system, `*` here matches any character, including "dot files." Therefore `--ignore File[/home/joe/*]` *would* ignore changes made, for example, to `/home/joe/.bashrc`. :warning: Do not put quotes of any kind in the ignore (e.g. `File['/etc/passwd']`) as these will be interpreted literally. ### Ignoring by attribute If you wish to ignore all changes to a particular attribute regardless of the resource with which it is associated, use this syntax. For example, if you wanted to ignore all changes to an attribute called `i_dont_care_about_this` you would use: --ignore-attr 'i_dont_care_about_this' That syntax will ignore a key `i_dont_care_about_this` *anywhere* it appears in the data structure -- top level key, last key, or something in between. If you want more control over where in the data structure the key appears, you can use '::' to separate multiple layers. For example, if your resource looked like this: { "type": "File", "title": "/tmp/foo", "parameters": { "owner": "root", "i_dont_care_about_this": "foo bar" } } You could use the following syntax to suppress `i_dont_care_about_this` only as it appears in the parameters hash using: --ignore-attr 'parameters::i_dont_care_about_this' If you want to specify that you are starting from the top of the data structure, prepend `::` to the attribute. For example: --ignore-attr '::parameters::i_dont_care_about_this' The difference between this syntax and the one appearing immediately before it is as follows. With the leading `::` it forces the attribute match to start at the top level of the data structure. Without the leading `::`, *any* place in the data structure where a key named `parameters` was a hash containing a key named `i_dont_care_about_this` would be matched. Typically there is not a deep level of nesting in Puppet catalogs so this distinction is minimal. However, multiple levels of nesting can occur when hashes are passed as parameters within Puppet manifests. TIP: For most attributes you wish to ignore, you should start with `::parameters` which is the standard top-level data structure within each catalog resource. As such, the remaining examples in this section will use this syntax. Functionally, `--ignore-attr 'FOO'` is equivalent to `--ignore '*[*]FOO'`, but that's less elegant. The prior example *could* be rewritten as: --ignore '*[*]::parameters::i_dont_care_about_this' ### Ignoring by type, title, and attribute `octocatalog-diff` allows you to ignore attributes that belong only to a specified type, or type + title. For example, maybe you want to ignore ownership changes to your `/tmp/foo` file. You could use the following syntax: --ignore 'File[/tmp/foo]::parameters::owner' You can use wildcards `*` in the resource title as described previously. For example, to ignore owner changes for all files in the `/tmp` directory you could use: --ignore 'File[/tmp/*]::parameters::owner' ### Ignoring by type, title, and attribute with conditions These functions allow you to ignore attributes, but only if the values of the attributes themselves or nature of the changes satisfy certain conditions. #### Ignoring additions and removals but not changes You can ignore resources that were added to the catalog. This syntax will *not* display an entry for any file in `/tmp` that was brand new in the new catalog. However, if the file resource existed in the old catalog and something changed, that's a change and not an addition, so it will display. For example: --ignore 'File[/tmp/*]+' The above will suppress this change because it's strictly an addition: + File[/tmp/brand-new-file] The above will NOT suppress this change because the resource existed before and changed: ~~~~~~~~ + File[/tmp/my-file] => parameters => owner => - root + new-owner ~~~~~~~~ Similarly, you can ignore resources that were entirely removed from the catalog. --ignore 'File[/tmp/*]-' And, you can ignore resources that were either entirely removed or entirely added: ~~~~~~~~ --ignore 'File[/tmp/*]+-' (or, equivalently) --ignore 'File[/tmp/*]+' --ignore 'File[/tmp/*]-' ~~~~~~~~ #### Ignoring for a specific value of an attribute You can ignore changes to an attribute when the value of the attribute matches a specific string or pattern. When you use this syntax, the tool will ignore the change if the specified value applies to the attribute in *either* the old catalog or the new catalog. To ignore all changes to file owners, if the file owner was root before or the file owner is now root, you could use the `=>` matcher: --ignore 'File[/tmp/*]::parameters::owner=>root' You can also use a regular expression. For example, to ignore changes if a file's content change includes "ice cream" you could use the `=~>` regular expression matcher: ---ignore 'File[/tmp/*]::parameters::content=~>ice cream' #### Ignoring attributes that were added or removed Similar to ignoring resources that were added or removed, you can also ignore attributes that were added or removed by prefixing the attribute name with a `+` or `-`. To ignore any new parameters named `foo` (i.e., where no attribute named `foo` was defined in the old catalog, but it is defined in the new catalog), you would use: --ignore 'My::Custom::Resource[*]+::parameters::foo' Similarly, to ignore the removal of the parameter named `foo` (i.e., where `foo` was defined in the old catalog, but is not defined in the new catalog), you would use: --ignore 'My::Custom::Resource[*]-::parameters::foo' It is possible to combine this condition with the attribute value check defined above. For example, to ignore a new parameter `foo` with value `bar`: --ignore 'My::Custom::Resource[*]+::parameters::foo=>bar' #### Ignoring attribute values but only in the new catalog or the old catalog Commonly, you will wish to suppress changes if an attribute is a certain value in the new catalog (or perhaps, a certain value in the old catalog). Say for example that you want to ignore any files that are now owned by root, but you will want to know about files that were owned by root and are now owned by somebody else. `=>root` would match files that were owned by root in the old catalog too but changed to somebody else in the new catalog, so that is too broad. The `=+>` operator performs a string match only in the new catalog, and `=->` performs a string match only in the new catalog. This will ignore any file resources under `/tmp` where the owner has changed, and the new owner is root: --ignore 'File[/tmp/*]::parameters::owner=+>root' As a similar example, perhaps you have terminated a user joe, and need to reassign ownership of his files to someone else. You want to ignore any files that were previously owned by joe, but you do want to know about files that you've accidentally reassigned *to* joe (because you don't believe there should be any). To ignore files owned by joe in the old catalog, where the new owner is someone other than joe: --ignore 'File[/tmp/*]::parameters::owner=->joe' If you need to use regular expressions, note that *changed lines* are preceded by `+` or `-` due to the `diff` implementation. More complicated, but equally functional, representations of the above commands are, respectively: --ignore 'File[/tmp/*]::parameters::owner=~>^\+root$' --ignore 'File[/tmp/*]::parameters::owner=~>^-joe$' :warning: Be sure to escape your literal '+' in the regular expression! Note that this syntax differs from `--ignore 'File[/tmp/*]+::...'` described in the prior section. The `+` or `-` between the title and the attribute indicates that the attribute must be brand new or completely removed; this will not ignore changes. Whereas using a `+` or `-` in the predicate of a string matcher will match changes between the catalogs. If you want to ignore changes where the attribute value exactly matches certain value in the old catalog, and exactly matches a certain other value in the new catalog, this is possible using the encompassing regular expressions described in the next section. You cannot combine `=+>` and `=->` in the same ignore condition. #### Ignoring attributes whose changes are encompassed by a regular expression It is possible to ignore changes only if *all* changed lines are matched by a regular expression. This is useful to suppress expected changes to an attribute, while still surfacing unexpected changes. As an example, consider that two catalogs defined the content of a file, which had this difference: ~~~~~~~~ File[/tmp/foo] => parameters => content => @@ -1,4 +1,4 @@ # This file is managed by Puppet. DO NOT EDIT. -This is the line in the old catalog that I do not care about +This is the line in the new catalog that I do not care about This line is very important ~~~~~~~~ Suppose that you do not care to see this change. You can use the `=&>` operator to specify *one* regular expression that must match *all* lines that are changed. (You do not need to worry about the `@@ -1,4 +1,4 @@` line, as that's an artifact of the `diff` process, and is not considered in the analysis.) One implementation of ignoring the line in question could be: --ignore 'File[/tmp/foo]::parameters::content=&>^(-This is the line in the old catalog that I do not care about)|(\+This is the line in the new catalog that I do not care about)$' If you aren't concerned about an edge case such as "This is the line in the new catalog that I do not care about" appearing in the old catalog, you could condense this to: --ignore 'File[/tmp/foo]::parameters::content=&>^[\-\+]This is the line in the (old|new) catalog that I do not care about$' Consider now that the change to the file looked like this instead: ~~~~~~~~ File[/tmp/foo] => parameters => content => @@ -1,4 +1,4 @@ # This file is managed by Puppet. DO NOT EDIT. -This is the line in the old catalog that I do not care about +This is the line in the new catalog that I do not care about -This line is very important ~~~~~~~~ In this case, the very important line was removed from the catalog, and you want to know about this. Ignoring `File[/tmp/foo]::parameters::content` would have suppressed this (because all changes to that attribute are ignored). Also ignoring `File[/tmp/foo]::parameters::content=~>This is the line in the new catalog that I do not care about$` would have also suppressed this (because the regular expression was matched for *one* of the lines). However, the two examples with `=&>` in this section would *not* have suppressed this change, because it is no longer the case that *all* changes in the file matched the regular expression. :warning: All lines are stripped of leading and trailing spaces before the regular expression match is tried. This stripping of whitespace is done *only* for this comparison stage, and does not affect the display of any results. octocatalog-diff-1.5.4/doc/configuration-hiera.md0000644000004100000410000001474113413744025022014 0ustar www-datawww-data# Configuring octocatalog-diff to use Hiera ## Hiera 5 Hiera 5 is included with Puppet 4.9 and higher. If there is a `hiera.yaml` file in the base directory of the environment that is in hiera 5 format, and you are running Puppet 4.9 or higher, then that file will be recognized by Puppet (and therefore, by octocatalog-diff). There is no special configuration for octocatalog-diff needed to make this work. Similarly, there is no command line option or setting to changed this behavior, because there is no corresponding option to change Puppet's behavior. If you are running Puppet 4.8 or lower, then the `hiera.yaml` file in the base directory of the environment will be ignored (unless you use `--hiera-config` to specify it as your global configuration file). If you have no global hiera configuration and you wish to rely on a `hiera.yaml` file in the base directory of your environment, make sure that you are *not* using any of the following command line options or [configuration settings](/doc/configuration.md): - `--hiera-path` or `settings[:hiera_path]` - `--hiera-path-strip` or `settings[:hiera_path_strip]` - `--hiera-config` or `settings[:hiera_config]` There is more information about Hiera 5 in Puppet's documentation: - [Enable the environment layer for existing Hiera data](https://puppet.com/docs/puppet/5.3/hiera_migrate_environments.html) ## Hiera global configuration If you are using Hiera 5 with a global configuration, or you are using Hiera 3 or before, then you must already have a [`hiera.yaml`](https://docs.puppet.com/puppet/latest/reference/config_file_hiera.html) file to configure it. These instructions will guide you through pointing octocatalog-diff at that configuration file. octocatalog-diff will automatically determine the version of your Hiera configuration file and treat it accordingly. (Hiera 5 configuration files are identified as such by a `version: 5` line in the file itself.) Before you start, please understand how octocatalog-diff compiles a catalog: - It creates a temporary directory (e.g. `/var/tmp/puppet-compile-dir-92347829847`) - It copies or creates hiera configuration, ENC, PuppetDB configuration, etc., in the temporary directory - It symlinks `/var/tmp/puppet-compile-dir-92347829847/environments/production` to your code - It compiles the catalog, based on the temporary directory, for environment=production - It removes the temporary directory ### Configuring the location of global hiera.yaml The command line option `--hiera-config PATH` allows you to set the path to the global hiera.yaml. You may specify this as either an absolute or a relative path. - As a relative path octocatalog-diff knows to use a relative path when the supplied path for `--hiera-config` does not start with a `/`. ``` bin/octocatalog-diff --hiera-config hiera.yaml ... ``` The path is relative to a checkout of your Puppet repository. With the setting above, it will use the file named `hiera.yaml` that is at the top level of your Puppet checkout. Perhaps your hiera.yaml file is in a subdirectory of your Puppet checkout. In that case, just use the relative directory path. Be sure not to add a leading `/` though, because you don't want octocatalog-diff to treat it as an absolute path. In the following example, suppose you have a top level directory called `config` and your `hiera.yaml` file is contained within it. You could then use: ``` bin/octocatalog-diff --hiera-config config/hiera.yaml ... ``` If you are specifying the hiera.yaml path in the [configuration file](/doc/configuration.md), you will instead set the variable like this: ``` settings[:hiera_config] = 'hiera.yaml' (or) settings[:hiera_config] = 'config/hiera.yaml' ``` octocatalog-diff will fail if you specify a hiera configuration location that cannot be opened. - As an absolute path octocatalog-diff knows to use a relative path when the supplied path for `--hiera-config` starts with a `/`. For example: ``` bin/octocatalog-diff --hiera-config /etc/puppetlabs/puppet/hiera.yaml ... ``` If you are specifying the hiera.yaml path in the [configuration file](/doc/configuration.md), you will instead set the variable like this: ``` settings[:hiera_config] = '/etc/puppetlabs/puppet/hiera.yaml' ``` Please note that octocatalog-diff will copy the file from the specified location into the compile directory. Since this hiera.yaml file is not copied from your Puppet repo, there is no way to compile the "to" and "from" branches using different hiera.yaml files. Furthermore, you are responsible for getting this file into place on any machine that will run octocatalog-diff. An absolute path may make octocatalog-diff work correctly on your Puppet master servers, but the structure may differ on other machines where you wish to run the utility. We strongly recommend that you version-control your hiera.yaml file within your Puppet repository, and use the relative path option described above. ### Configuring the directory in your repository in which hiera data files are found The command line option `--hiera-path PATH` allows you to set the directory path, relative to the checkout of your Puppet repository, of your Hiera YAML/JSON data files. If you are using the out-of-the-box Puppet Enterprise configuration, or the [Puppet Control Repo template](https://github.com/puppetlabs/control-repo), then the correct setting here is simply 'hieradata'. You must specify this as a relative path. octocatalog-diff knows to use a relative path when the supplied path for `--hiera-path` does not start with a `/`. ``` bin/octocatalog-diff --hiera-path hieradata ... ``` The path is relative to a checkout of your Puppet repository. With the setting above, it will look for Hiera data in a directory called `hieradata` that is at the top level of your Puppet checkout. If you are specifying the Hiera data path in the [configuration file](/doc/configuration.md), you will instead set the variable like this: ``` settings[:hiera_path] = 'hieradata' ``` octocatalog-diff will fail if you specify a path that is not a directory. ### Configuring the prefix path to strip This is a different, and potentially more complex, alternative to `--hiera-path` / `settings[:hiera_path]` described in the prior section. Unless you have a very good reason, you should prefer to use the instructions above. If you need to use the more complex path strip alternative, see: [Configuring octocatalog-diff to use Hiera path stripping](/doc/advanced-hiera-path-stripping.md). octocatalog-diff-1.5.4/doc/advanced-output-formats.md0000644000004100000410000000675713413744025022643 0ustar www-datawww-data# Output formats By default, when you run `octocatalog-diff`, it will display the results in color on STDOUT. An example of the output display is shown on the [main page](/README.md#example). Log messages will be written to STDERR. ## Writing to a file If you would like to write the results to a file instead, specify: octocatalog-diff -o OUTPUT_FILE_PATH [other options] When you output to a file, the default output format is the same as the on-screen format, but without the color. If you would like to redirect the debug/log output to a file, you can use shell redirection: octocatalog-diff [options] 2>/var/log/octocatalog-diff.log ## Alternate output formats You can specify the `--output-format FORMAT` command line options with one of the following values for FORMAT: | FORMAT | Explanation | | ------ | ----------- | | text | (Default) Traditional human readable output | | json | JSON output of the difference array | ## Color As previously noted, color text is enabled for on-screen display and disabled for file output. You can override the automatic selection via: - `--color` to turn colored text on - `--no-color` to turn colored text off Note: There is no color used for JSON output, regardless of whether the display is on-screen or redirected to a file, and regardless of whether you specify the `--color` command line option. ## JSON format ### Top level format The JSON result is in the following format: ``` { "diff": [ DIFFERENCE_ARRAY_1, DIFFERENCE_ARRAY_2, ... ], "header": "HEADER STRING" } ``` ### Format of difference arrays Each difference array is in one of the following formats: #### Addition or removal ``` [ (String) Change Type (+ or -), (String) Type, Title separated by \f, (Hash) Object as it exists in old or new catalog, (Hash) { file: "MANIFEST FILENAME", line: LINE_NUMBER } ] ``` When the change type is '+' this indicates that the resource was added (i.e., it exists in the new catalog and not the old). '-' indicates that the resource was removed (i.e., it exists in the old catalog, but not the new). It is important to note that a removed resource does not necessarily cause cleanup on the target system. #### Change ``` [ (String) Change Type (~ or !), (String) Type, Title, Attribute separated by \f, (?) Object as it exists in old catalog, (?) Object as it exists in new catalog, (Hash) { file: "MANIFEST FILENAME", line: LINE_NUMBER } from old catalog, (Hash) { file: "MANIFEST FILENAME", line: LINE_NUMBER } from new catalog ] ``` Note that `~` and `!` are used internally to signify different types of changes, but for display purposes they should be considered equivalent. The objects as they appear in the old and new catalogs can be many different data types. Strings, integers, and booleans are the most common. `null` represents that the attribute was not present in the catalog or that it was set to Puppet's `undef`. (`null` in JSON is equivalent to Ruby's `nil`.) ### Notes - In the second array field, the type, title, and attribute are separated by the form feed (`\f`) character. This was chosen because it is unlikely to be encountered in actual naming. You are guaranteed that when splitting on `\f`, the first item is the type, the second item is the title, and the third (and subsequent) items represent the attribute, with each key in the data structure also separated by `\f`. - If the manifest name and line number are not reported in the catalog, the hashes may have `nil` values for the file and line keys. octocatalog-diff-1.5.4/doc/optionsref.md0000644000004100000410000021075313413744025020250 0ustar www-datawww-data # Command line options reference ## Usage ``` Usage: octocatalog-diff [command line options] -n HOSTNAME1[,HOSTNAME2[,...]], Use PuppetDB facts from last run of a hostname or a comma separated list of multiple hostnames --hostname --basedir DIRNAME Use an alternate base directory (git checkout of puppet repository) -f, --from FROM_BRANCH Branch you are coming from -t, --to TO_BRANCH Branch you are going to --from-catalog FILENAME Use a pre-compiled catalog 'from' --to-catalog FILENAME Use a pre-compiled catalog 'to' --bootstrap-script FILENAME Bootstrap script relative to checkout directory --bootstrap-current Run bootstrap script for the current directory too --debug-bootstrap Print debugging output for bootstrap script --bootstrap-environment "key1=val1,key2=val2,..." Bootstrap script environment variables in key=value format --bootstrapped-from-dir DIRNAME Use a pre-bootstrapped 'from' directory --bootstrapped-to-dir DIRNAME Use a pre-bootstrapped 'to' directory --bootstrap-then-exit Bootstrap from-dir and/or to-dir and then exit --[no-]color Enable/disable colors in output -o, --output-file FILENAME Output results into FILENAME --output-format FORMAT Output format: text,json,legacy_json -d, --[no-]debug Print debugging messages to STDERR -q, --[no-]quiet Quiet (no status messages except errors) --ignore "Type1[Title1],Type2[Title2],..." More resources to ignore in format type[title] --[no-]include-tags Include changes to tags in the diff output --fact-file STRING Override fact globally --to-fact-file STRING Override fact for the to branch --from-fact-file STRING Override fact for the from branch --save-catalog STRING Save intermediate catalogs into files globally --to-save-catalog STRING Save intermediate catalogs into files for the to branch --from-save-catalog STRING Save intermediate catalogs into files for the from branch --cached-master-dir PATH Cache bootstrapped origin/master at this path --master-cache-branch BRANCH Branch to cache --safe-to-delete-cached-master-dir PATH OK to delete cached master directory at this path --hiera-config STRING Full or relative path to global Hiera configuration file globally --to-hiera-config STRING Full or relative path to global Hiera configuration file for the to branch --from-hiera-config STRING Full or relative path to global Hiera configuration file for the from branch --no-hiera-config Disable hiera config file installation --hiera-path STRING Path to hiera data directory, relative to top directory of repository globally --to-hiera-path STRING Path to hiera data directory, relative to top directory of repository for the to branch --from-hiera-path STRING Path to hiera data directory, relative to top directory of repository for the from branch --no-hiera-path Do not use any default hiera path settings --hiera-path-strip STRING Path prefix to strip when munging hiera.yaml globally --to-hiera-path-strip STRING Path prefix to strip when munging hiera.yaml for the to branch --from-hiera-path-strip STRING Path prefix to strip when munging hiera.yaml for the from branch --no-hiera-path-strip Do not use any default hiera path strip settings --ignore-attr "attr1,attr2,..." Attributes to ignore --filters FILTER1[,FILTER2[,...]] Filters to apply --[no-]display-source Show source file and line for each difference --[no-]validate-references "before,require,subscribe,notify" References to validate --[no-]compare-file-text Compare text, not source location, of file resources --[no-]storeconfigs Enable integration with puppetdb for collected resources --retry-failed-catalog N Retry building a failed catalog N times --no-enc Disable ENC --enc PATH Path to ENC script, relative to checkout directory or absolute --from-enc PATH Path to ENC script (for the from catalog only) --to-enc PATH Path to ENC script (for the to catalog only) --[no-]display-detail-add Display parameters and other details for added resources --[no-]truncate-details Truncate details with --display-detail-add --no-header Do not print a header --default-header Print default header with output --header STRING Specify header for output --parser PARSER_NAME Specify parser (default, future) --parser-from PARSER_NAME Specify parser (default, future) --parser-to PARSER_NAME Specify parser (default, future) --[no-]display-datatype-changes Display changes in data type even when strings match --[no-]catalog-only Only compile the catalog for the "to" branch but do not diff --[no-]from-puppetdb Pull "from" catalog from PuppetDB instead of compiling --[no-]parallel Enable or disable parallel processing --puppet-binary STRING Full path to puppet binary globally --to-puppet-binary STRING Full path to puppet binary for the to branch --from-puppet-binary STRING Full path to puppet binary for the from branch --facts-terminus STRING Facts terminus: one of yaml, facter --puppetdb-token TOKEN Token to access the PuppetDB API --puppetdb-token-file PATH Path containing token for PuppetDB API, relative or absolute --puppetdb-url URL PuppetDB base URL --puppetdb-ssl-ca FILENAME CA certificate that signed the PuppetDB certificate --puppetdb-ssl-client-cert FILENAME SSL client certificate to connect to PuppetDB --puppetdb-ssl-client-key FILENAME SSL client key to connect to PuppetDB --puppetdb-ssl-client-password PASSWORD Password for SSL client key to connect to PuppetDB --puppetdb-ssl-client-password-file FILENAME Read password for SSL client key from a file --puppetdb-api-version N Version of PuppetDB API (3 or 4) --fact-override STRING1[,STRING2[,...]] Override fact globally --to-fact-override STRING1[,STRING2[,...]] Override fact for the to branch --from-fact-override STRING1[,STRING2[,...]] Override fact for the from branch --puppet-master STRING Hostname or Hostname:PortNumber for Puppet Master globally --to-puppet-master STRING Hostname or Hostname:PortNumber for Puppet Master for the to branch --from-puppet-master STRING Hostname or Hostname:PortNumber for Puppet Master for the from branch --puppet-master-api-version STRING Puppet Master API version (2 for Puppet 3.x, 3 for Puppet 4.x) globally --to-puppet-master-api-version STRING Puppet Master API version (2 for Puppet 3.x, 3 for Puppet 4.x) for the to branch --from-puppet-master-api-version STRING Puppet Master API version (2 for Puppet 3.x, 3 for Puppet 4.x) for the from branch --puppet-master-ssl-ca STRING Full path to CA certificate that signed the Puppet Master certificate globally --to-puppet-master-ssl-ca STRING Full path to CA certificate that signed the Puppet Master certificate for the to branch --from-puppet-master-ssl-ca STRING Full path to CA certificate that signed the Puppet Master certificate for the from branch --puppet-master-ssl-client-cert STRING Full path to certificate file for SSL client auth to Puppet Master globally --to-puppet-master-ssl-client-cert STRING Full path to certificate file for SSL client auth to Puppet Master for the to branch --from-puppet-master-ssl-client-cert STRING Full path to certificate file for SSL client auth to Puppet Master for the from branch --puppet-master-ssl-client-key STRING Full path to key file for SSL client auth to Puppet Master globally --to-puppet-master-ssl-client-key STRING Full path to key file for SSL client auth to Puppet Master for the to branch --from-puppet-master-ssl-client-key STRING Full path to key file for SSL client auth to Puppet Master for the from branch --enc-override STRING1[,STRING2[,...]] Override parameter from ENC globally --to-enc-override STRING1[,STRING2[,...]] Override parameter from ENC for the to branch --from-enc-override STRING1[,STRING2[,...]] Override parameter from ENC for the from branch --puppet-master-timeout STRING Puppet Master catalog retrieval timeout in seconds globally --to-puppet-master-timeout STRING Puppet Master catalog retrieval timeout in seconds for the to branch --from-puppet-master-timeout STRING Puppet Master catalog retrieval timeout in seconds for the from branch --pe-enc-url URL Base URL for Puppet Enterprise ENC endpoint --pe-enc-token TOKEN Token to access the Puppet Enterprise ENC API --pe-enc-token-file PATH Path containing token for PE node classifier, relative or absolute --pe-enc-ssl-ca FILENAME CA certificate that signed the ENC API certificate --pe-enc-ssl-client-cert FILENAME SSL client certificate to connect to PE ENC --pe-enc-ssl-client-key FILENAME SSL client key to connect to PE ENC --override-script-path DIRNAME Directory with scripts to override built-ins --no-ignore-tags Disable ignoring based on tags --ignore-tags STRING1[,STRING2[,...]] Specify tags to ignore --[no-]preserve-environments Enable or disable environment preservation --environment STRING Environment for catalog compilation globally --to-environment STRING Environment for catalog compilation for the to branch --from-environment STRING Environment for catalog compilation for the from branch --create-symlinks STRING1[,STRING2[,...]] Symlinks to create globally --to-create-symlinks STRING1[,STRING2[,...]] Symlinks to create for the to branch --from-create-symlinks STRING1[,STRING2[,...]] Symlinks to create for the from branch --command-line STRING1[,STRING2[,...]] Command line arguments globally --to-command-line STRING1[,STRING2[,...]] Command line arguments for the to branch --from-command-line STRING1[,STRING2[,...]] Command line arguments for the from branch --pass-env-vars VAR1[,VAR2[,...]] Environment variables to pass --[no-]suppress-absent-file-details Suppress certain attributes of absent files ``` ## Detailed options description
    Option Description Extended Description 
    --basedir DIRNAME
    		 Use an alternate base directory (git checkout of puppet repository) Option to set the base checkout directory of puppet repository (basedir.rb) 
    --bootstrap-current
    		Run bootstrap script for the current directory too Option to bootstrap the current directory (by default, the bootstrap script is NOT run when the catalog builds in the current directory). (bootstrap_current.rb) 
    --bootstrap-environment "key1=val1,key2=val2,..."
    		 Bootstrap script environment variables in key=value format Allow the bootstrap environment to be set up via the command line. (bootstrap_environment.rb) 
    --bootstrap-script FILENAME
    		 Bootstrap script relative to checkout directory Allow specification of a bootstrap script. This runs after checking out the directory, and before running puppet there. Good for running librarian to install modules, and anything else site-specific that needs to be done. (bootstrap_script.rb) 
    --bootstrap-then-exit
    		Bootstrap from-dir and/or to-dir and then exit Option to bootstrap directories and then exit (bootstrap_then_exit.rb) 
    --bootstrapped-from-dir DIRNAME
    		 Use a pre-bootstrapped 'from' directory Allow (or create) directories that are already bootstrapped. Handy to allow "bootstrap once, build many" to save time when diffing multiple catalogs on this system. (bootstrapped_dirs.rb) 
    --bootstrapped-to-dir DIRNAME
    		 Use a pre-bootstrapped 'to' directory Allow (or create) directories that are already bootstrapped. Handy to allow "bootstrap once, build many" to save time when diffing multiple catalogs on this system. (bootstrapped_dirs.rb) 
    --cached-master-dir PATH
    		 Cache bootstrapped origin/master at this path Cache a bootstrapped checkout of 'master' and use that for time-saving when the SHA has not changed. (cached_master_dir.rb) 
    --catalog-only
--no-catalog-only
    		Only compile the catalog for the "to" branch but do not diff When set, --catalog-only will only compile the catalog for the 'to' branch, and skip any diffing activity. The catalog will be printed to STDOUT or written to the output file. (catalog_only.rb) 
    --color
--no-color
    		Enable/disable colors in output Color printing option (color.rb) 
    --command-line STRING1[,STRING2[,...]]
    		 Command line arguments globally Provide additional command line flags to set when running Puppet to compile catalogs. (command_line.rb) 
    --compare-file-text
--no-compare-file-text
    		Compare text, not source location, of file resources When a file is specified with `source => 'puppet:///modules/something/foo.txt'`, remove the 'source' attribute and populate the 'content' attribute with the text of the file. This allows for a diff of the content, rather than a diff of the location, which is what is most often desired. (compare_file_text.rb) 
    --create-symlinks STRING1[,STRING2[,...]]
    		 Symlinks to create globally Specify which directories from the base should be symlinked into the temporary compilation environment. This is useful only in conjunction with `--preserve-environments`. (create_symlinks.rb) 
    -d
--debug
--no-debug
    		Print debugging messages to STDERR Debugging option (debug.rb) 
    --debug-bootstrap
    		Print debugging output for bootstrap script Option to print debugging output for the bootstrap script in addition to the normal debugging output. Note that `--debug` must also be enabled for this option to have any effect. (debug_bootstrap.rb) 
    --default-header
    		Print default header with output Provide ability to set custom header or to display no header at all (header.rb) 
    --display-datatype-changes
--no-display-datatype-changes
    		Display changes in data type even when strings match Toggle on or off the display of data type changes when the string representation is the same. For example with this enabled, '42' (the string) and 42 (the integer) will be displayed as a difference. With this disabled, this is not displayed as a difference. (display_datatype_changes.rb) 
    --display-detail-add
--no-display-detail-add
    		Display parameters and other details for added resources Provide ability to display details of 'added' resources in the output. (display_detail_add.rb) 
    --display-source
--no-display-source
    		Show source file and line for each difference Display source filename and line number for diffs (display_source_file_line.rb) 
    --enc PATH
    		 Path to ENC script, relative to checkout directory or absolute Path to external node classifier, relative to the base directory of the checkout. (enc.rb) 
    --enc-override STRING1[,STRING2[,...]]
    		 Override parameter from ENC globally Allow override of ENC parameters on the command line. ENC parameter overrides can be supplied for the 'to' or 'from' catalog, or for both. There is some attempt to handle data types here (since all items on the command line are strings) by permitting a data type specification as well. For parameters nested in hashes, use `::` as the delimiter. (enc_override.rb) 
    --environment STRING
    		 Environment for catalog compilation globally Specify the environment to use when compiling the catalog. This is useful only in conjunction with `--preserve-environments`. (environment.rb) 
    --fact-file STRING
    		 Override fact globally Allow an existing fact file to be provided, to avoid pulling facts from PuppetDB. (fact_file.rb) 
    --fact-override STRING1[,STRING2[,...]]
    		 Override fact globally Allow override of facts on the command line. Fact overrides can be supplied for the 'to' or 'from' catalog, or for both. There is some attempt to handle data types here (since all items on the command line are strings) by permitting a data type specification as well. (fact_override.rb) 
    --facts-terminus STRING
    		 Facts terminus: one of yaml, facter Get the facts terminus. Generally this is 'yaml' and a fact file will be loaded from PuppetDB or elsewhere in the environment. However it can be set to 'facter' which will run facter on the host on which this is running. (facts_terminus.rb) 
    --filters FILTER1[,FILTER2[,...]]
    		 Filters to apply Specify one or more filters to apply to the results of the catalog difference. For a list of available filters and further explanation, please refer to Filtering results. (filters.rb) 
    -f FROM_BRANCH
--from FROM_BRANCH
    		 Branch you are coming from Set the 'from' and 'to' branches, which is used to compile catalogs. A branch of '.' means to use the current contents of the base code directory without any git checkouts. (to_from_branch.rb) 
    --from-catalog FILENAME
    		 Use a pre-compiled catalog 'from' If pre-compiled catalogs are available, these can be used to short-circuit the build process. These files must exist and be in Puppet catalog format. (existing_catalogs.rb) 
    --from-command-line STRING1[,STRING2[,...]]
    		 Command line arguments for the from branch Provide additional command line flags to set when running Puppet to compile catalogs. (command_line.rb) 
    --from-create-symlinks STRING1[,STRING2[,...]]
    		 Symlinks to create for the from branch Specify which directories from the base should be symlinked into the temporary compilation environment. This is useful only in conjunction with `--preserve-environments`. (create_symlinks.rb) 
    --from-enc PATH
    		 Path to ENC script (for the from catalog only) Path to external node classifier, relative to the base directory of the checkout. (enc.rb) 
    --from-enc-override STRING1[,STRING2[,...]]
    		 Override parameter from ENC for the from branch Allow override of ENC parameters on the command line. ENC parameter overrides can be supplied for the 'to' or 'from' catalog, or for both. There is some attempt to handle data types here (since all items on the command line are strings) by permitting a data type specification as well. For parameters nested in hashes, use `::` as the delimiter. (enc_override.rb) 
    --from-environment STRING
    		 Environment for catalog compilation for the from branch Specify the environment to use when compiling the catalog. This is useful only in conjunction with `--preserve-environments`. (environment.rb) 
    --from-fact-file STRING
    		 Override fact for the from branch Allow an existing fact file to be provided, to avoid pulling facts from PuppetDB. (fact_file.rb) 
    --from-fact-override STRING1[,STRING2[,...]]
    		 Override fact for the from branch Allow override of facts on the command line. Fact overrides can be supplied for the 'to' or 'from' catalog, or for both. There is some attempt to handle data types here (since all items on the command line are strings) by permitting a data type specification as well. (fact_override.rb) 
    --from-hiera-config STRING
    		 Full or relative path to global Hiera configuration file for the from branch Specify a relative path to the Hiera yaml file (hiera_config.rb) 
    --from-hiera-path STRING
    		 Path to hiera data directory, relative to top directory of repository for the from branch Specify the path to the Hiera data directory (relative to the top level Puppet checkout). For Puppet Enterprise and the Puppet control repo template, the value of this should be 'hieradata', which is the default. (hiera_path.rb) 
    --from-hiera-path-strip STRING
    		 Path prefix to strip when munging hiera.yaml for the from branch Specify the path to strip off the datadir to munge hiera.yaml file (hiera_path_strip.rb) 
    --from-puppet-binary STRING
    		 Full path to puppet binary for the from branch Set --puppet-binary, --to-puppet-binary, --from-puppet-binary (puppet_binary.rb) 
    --from-puppet-master STRING
    		 Hostname or Hostname:PortNumber for Puppet Master for the from branch Specify the hostname, or hostname:port, for the Puppet Master. (puppet_master.rb) 
    --from-puppet-master-api-version STRING
    		 Puppet Master API version (2 for Puppet 3.x, 3 for Puppet 4.x) for the from branch Specify the API version to use for the Puppet Master. This makes it possible to authenticate to a version 3.x PuppetMaster by specifying the API version as 2, or for a version 4.x PuppetMaster by specifying API version as 3. (puppet_master_api_version.rb) 
    --from-puppet-master-ssl-ca STRING
    		 Full path to CA certificate that signed the Puppet Master certificate for the from branch Specify the CA certificate for Puppet Master. If specified, this will enable SSL verification that the certificate being presented has been signed by this CA, and that the common name matches the name you are using to connecting. (puppet_master_ssl_ca.rb) 
    --from-puppet-master-ssl-client-cert STRING
    		 Full path to certificate file for SSL client auth to Puppet Master for the from branch Specify the SSL client certificate for Puppet Master. This makes it possible to authenticate with a client certificate keypair to the Puppet Master. (puppet_master_ssl_client_cert.rb) 
    --from-puppet-master-ssl-client-key STRING
    		 Full path to key file for SSL client auth to Puppet Master for the from branch Specify the SSL client key for Puppet Master. This makes it possible to authenticate with a client certificate keypair to the Puppet Master. (puppet_master_ssl_client_key.rb) 
    --from-puppet-master-timeout STRING
    		 Puppet Master catalog retrieval timeout in seconds for the from branch Specify a timeout for retrieving a catalog from a Puppet master / Puppet server. This timeout is specified in seconds. (puppet_master_timeout.rb) 
    --from-puppetdb
--no-from-puppetdb
    		Pull "from" catalog from PuppetDB instead of compiling Set --from-puppetdb to pull most recent catalog from PuppetDB instead of compiling (from_puppetdb.rb) 
    --from-save-catalog STRING
    		 Save intermediate catalogs into files for the from branch Allow catalogs to be saved to a file before they are diff'd. (save_catalog.rb) 
    --header STRING
    		 Specify header for output Provide ability to set custom header or to display no header at all (header.rb) 
    --hiera-config STRING
    		 Full or relative path to global Hiera configuration file globally Specify a relative path to the Hiera yaml file (hiera_config.rb) 
    --hiera-path STRING
    		 Path to hiera data directory, relative to top directory of repository globally Specify the path to the Hiera data directory (relative to the top level Puppet checkout). For Puppet Enterprise and the Puppet control repo template, the value of this should be 'hieradata', which is the default. (hiera_path.rb) 
    --hiera-path-strip STRING
    		 Path prefix to strip when munging hiera.yaml globally Specify the path to strip off the datadir to munge hiera.yaml file (hiera_path_strip.rb) 
    -n HOSTNAME1[,HOSTNAME2[,...]]
--hostname HOSTNAME1[,HOSTNAME2[,...]]
    		 Use PuppetDB facts from last run of a hostname or a comma separated list of multiple hostnames Set hostname, which is used to look up facts in PuppetDB, and in the header of diff display. This option can recieve a single hostname, or a comma separated list of multiple hostnames, which are split into an Array. Multiple hostnames do not work with the `catalog-only` or `bootstrap-then-exit` options. (hostname.rb) 
    --ignore "Type1[Title1],Type2[Title2],..."
    		 More resources to ignore in format type[title] Options used when comparing catalogs - set ignored changes. (ignore.rb) 
    --ignore-attr "attr1,attr2,..."
    		 Attributes to ignore Specify attributes to ignore (ignore_attr.rb) 
    --ignore-tags STRING1[,STRING2[,...]]
    		 Specify tags to ignore Provide ability to set one or more tags, which will cause catalog-diff to ignore any changes for any defined type where this tag is set. (ignore_tags.rb) 
    --include-tags
--no-include-tags
    		Include changes to tags in the diff output Options used when comparing catalogs - tags are generally ignored; you can un-ignore them. (include_tags.rb) 
    --master-cache-branch BRANCH
    		 Branch to cache Allow override of the branch that is cached. This defaults to 'origin/master'. (master_cache_branch.rb) 
    --no-enc
    		Disable ENC Path to external node classifier, relative to the base directory of the checkout. (enc.rb) 
    --no-header
    		Do not print a header Provide ability to set custom header or to display no header at all (header.rb) 
    --no-hiera-config
    		Disable hiera config file installation Specify a relative path to the Hiera yaml file (hiera_config.rb) 
    --no-hiera-path
    		Do not use any default hiera path settings Specify the path to the Hiera data directory (relative to the top level Puppet checkout). For Puppet Enterprise and the Puppet control repo template, the value of this should be 'hieradata', which is the default. (hiera_path.rb) 
    --no-hiera-path-strip
    		Do not use any default hiera path strip settings Specify the path to strip off the datadir to munge hiera.yaml file (hiera_path_strip.rb) 
    --no-ignore-tags
    		Disable ignoring based on tags Provide ability to set one or more tags, which will cause catalog-diff to ignore any changes for any defined type where this tag is set. (ignore_tags.rb) 
    -o FILENAME
--output-file FILENAME
    		 Output results into FILENAME Output file option (output_file.rb) 
    --output-format FORMAT
    		 Output format: text,json,legacy_json Output format option. 'text' is human readable text, 'json' is an array of differences identified by human readable keys (the preferred octocatalog-diff 1.x format), and 'legacy_json' is an array of differences, where each difference is an array (the octocatalog-diff 0.x format). (output_format.rb) 
    --override-script-path DIRNAME
    		 Directory with scripts to override built-ins Provide an optional directory to override default built-in scripts such as git checkout and puppet version determination. (override_script_path.rb) 
    --parallel
--no-parallel
    		Enable or disable parallel processing Disable or enable parallel processing of catalogs. (parallel.rb) 
    --parser PARSER_NAME
    		 Specify parser (default, future) Enable future parser for both branches or for just one (parser.rb) 
    --parser-from PARSER_NAME
    		 Specify parser (default, future) Enable future parser for both branches or for just one (parser.rb) 
    --parser-to PARSER_NAME
    		 Specify parser (default, future) Enable future parser for both branches or for just one (parser.rb) 
    --pass-env-vars VAR1[,VAR2[,...]]
    		 Environment variables to pass One or more environment variables that should be made available to the Puppet binary when parsing the catalog. For example, --pass-env-vars FOO,BAR will make the FOO and BAR environment variables available. Setting these variables is your responsibility outside of octocatalog-diff. (pass_env_vars.rb) 
    --pe-enc-ssl-ca FILENAME
    		 CA certificate that signed the ENC API certificate Specify the CA certificate for the Puppet Enterprise ENC. If specified, this will enable SSL verification that the certificate being presented has been signed by this CA, and that the common name matches the name you are using to connecting. (pe_enc_ssl_ca.rb) 
    --pe-enc-ssl-client-cert FILENAME
    		 SSL client certificate to connect to PE ENC Specify the client certificate for connecting to the Puppet Enterprise ENC. This must be specified along with --pe-enc-ssl-client-key in order to work. (pe_enc_ssl_client_cert.rb) 
    --pe-enc-ssl-client-key FILENAME
    		 SSL client key to connect to PE ENC Specify the client key for connecting to Puppet Enterprise ENC. This must be specified along with --pe-enc-ssl-client-cert in order to work. (pe_enc_ssl_client_key.rb) 
    --pe-enc-token TOKEN
    		 Token to access the Puppet Enterprise ENC API Specify the access token to access the Puppet Enterprise ENC. Refer to https://docs.puppet.com/pe/latest/nc_forming_requests.html#authentication for details on generating and obtaining a token. Use this option to specify the text of the token. (Use --pe-enc-token-file to read the content of the token from a file.) (pe_enc_token.rb) 
    --pe-enc-token-file PATH
    		 Path containing token for PE node classifier, relative or absolute Specify the access token to access the Puppet Enterprise ENC. Refer to https://docs.puppet.com/pe/latest/nc_forming_requests.html#authentication for details on generating and obtaining a token. Use this option if the token is stored in a file, to read the content of the token from the file. (pe_enc_token_file.rb) 
    --pe-enc-url URL
    		 Base URL for Puppet Enterprise ENC endpoint Specify the URL to the Puppet Enterprise ENC API. By default, the node classifier service listens on port 4433 and all endpoints are relative to the /classifier-api/ path. That means the likely value for this option will be something like: https://your-pe-console-server:4433/classifier-api (pe_enc_url.rb) 
    --preserve-environments
--no-preserve-environments
    		Enable or disable environment preservation Preserve the `environments` directory from the repository when compiling the catalog. Likely requires some combination of `--to-environment`, `--from-environment`, and/or `--create-symlinks` to work correctly. (preserve_environments.rb) 
    --puppet-binary STRING
    		 Full path to puppet binary globally Set --puppet-binary, --to-puppet-binary, --from-puppet-binary (puppet_binary.rb) 
    --puppet-master STRING
    		 Hostname or Hostname:PortNumber for Puppet Master globally Specify the hostname, or hostname:port, for the Puppet Master. (puppet_master.rb) 
    --puppet-master-api-version STRING
    		 Puppet Master API version (2 for Puppet 3.x, 3 for Puppet 4.x) globally Specify the API version to use for the Puppet Master. This makes it possible to authenticate to a version 3.x PuppetMaster by specifying the API version as 2, or for a version 4.x PuppetMaster by specifying API version as 3. (puppet_master_api_version.rb) 
    --puppet-master-ssl-ca STRING
    		 Full path to CA certificate that signed the Puppet Master certificate globally Specify the CA certificate for Puppet Master. If specified, this will enable SSL verification that the certificate being presented has been signed by this CA, and that the common name matches the name you are using to connecting. (puppet_master_ssl_ca.rb) 
    --puppet-master-ssl-client-cert STRING
    		 Full path to certificate file for SSL client auth to Puppet Master globally Specify the SSL client certificate for Puppet Master. This makes it possible to authenticate with a client certificate keypair to the Puppet Master. (puppet_master_ssl_client_cert.rb) 
    --puppet-master-ssl-client-key STRING
    		 Full path to key file for SSL client auth to Puppet Master globally Specify the SSL client key for Puppet Master. This makes it possible to authenticate with a client certificate keypair to the Puppet Master. (puppet_master_ssl_client_key.rb) 
    --puppet-master-timeout STRING
    		 Puppet Master catalog retrieval timeout in seconds globally Specify a timeout for retrieving a catalog from a Puppet master / Puppet server. This timeout is specified in seconds. (puppet_master_timeout.rb) 
    --puppetdb-api-version N
    		 Version of PuppetDB API (3 or 4) Specify the API version to use for the PuppetDB. The current values supported are '3' or '4', and '4' is the default. (puppetdb_api_version.rb) 
    --puppetdb-ssl-ca FILENAME
    		 CA certificate that signed the PuppetDB certificate Specify the CA certificate for PuppetDB. If specified, this will enable SSL verification that the certificate being presented has been signed by this CA, and that the common name matches the name you are using to connecting. (puppetdb_ssl_ca.rb) 
    --puppetdb-ssl-client-cert FILENAME
    		 SSL client certificate to connect to PuppetDB Specify the client certificate for connecting to PuppetDB. This must be specified along with --puppetdb-ssl-client-key in order to work. (puppetdb_ssl_client_cert.rb) 
    --puppetdb-ssl-client-key FILENAME
    		 SSL client key to connect to PuppetDB Specify the client key for connecting to PuppetDB. This must be specified along with --puppetdb-ssl-client-cert in order to work. (puppetdb_ssl_client_key.rb) 
    --puppetdb-ssl-client-password PASSWORD
    		 Password for SSL client key to connect to PuppetDB Specify the password for a PEM or PKCS12 private key on the command line. Note that `--puppetdb-ssl-client-password-file` is slightly more secure because the text of the password won't appear in the process list. (puppetdb_ssl_client_password.rb) 
    --puppetdb-ssl-client-password-file FILENAME
    		 Read password for SSL client key from a file Specify the password for a PEM or PKCS12 private key, by reading it from a file. (puppetdb_ssl_client_password_file.rb) 
    --puppetdb-token TOKEN
    		 Token to access the PuppetDB API Specify the PE RBAC token to access the PuppetDB API. Refer to https://puppet.com/docs/pe/latest/rbac/rbac_token_auth_intro.html#generate-a-token-using-puppet-access for details on generating and obtaining a token. Use this option to specify the text of the token. (Use --puppetdb-token-file to read the content of the token from a file.) (puppetdb_token.rb) 
    --puppetdb-token-file PATH
    		 Path containing token for PuppetDB API, relative or absolute Specify the PE RBAC token to access the PuppetDB API. Refer to https://puppet.com/docs/pe/latest/rbac/rbac_token_auth_intro.html#generate-a-token-using-puppet-access for details on generating and obtaining a token. Use this option to specify the text in a file, to read the content of the token from the file. (puppetdb_token_file.rb) 
    --puppetdb-url URL
    		 PuppetDB base URL Specify the base URL for PuppetDB. This will generally look like https://puppetdb.yourdomain.com:8081 (puppetdb_url.rb) 
    -q
--quiet
--no-quiet
    		Quiet (no status messages except errors) Quiet option (quiet.rb) 
    --retry-failed-catalog N
    		 Retry building a failed catalog N times Transient errors can cause catalog compilation problems. This adds an option to retry a failed catalog multiple times before kicking out an error message. (retry_failed_catalog.rb) 
    --safe-to-delete-cached-master-dir PATH
    		 OK to delete cached master directory at this path By specifying a directory path here, you are explicitly giving permission to the program to delete it if it believes it needs to be created (e.g., if the SHA has changed of the cached directory). (safe_to_delete_cached_master_dir.rb) 
    --save-catalog STRING
    		 Save intermediate catalogs into files globally Allow catalogs to be saved to a file before they are diff'd. (save_catalog.rb) 
    --storeconfigs
--no-storeconfigs
    		Enable integration with puppetdb for collected resources Set storeconfigs (integration with PuppetDB for collected resources) (storeconfigs.rb) 
    --suppress-absent-file-details
--no-suppress-absent-file-details
    		Suppress certain attributes of absent files If enabled, this option will suppress changes to certain attributes of a file, if the file is specified to be 'absent' in the target catalog. Suppressed changes in this case include user, group, mode, and content, because a removed file has none of those. This option is DEPRECATED; please use --filters AbsentFile instead. (suppress_absent_file_details.rb) 
    -t TO_BRANCH
--to TO_BRANCH
    		 Branch you are going to Set the 'from' and 'to' branches, which is used to compile catalogs. A branch of '.' means to use the current contents of the base code directory without any git checkouts. (to_from_branch.rb) 
    --to-catalog FILENAME
    		 Use a pre-compiled catalog 'to' If pre-compiled catalogs are available, these can be used to short-circuit the build process. These files must exist and be in Puppet catalog format. (existing_catalogs.rb) 
    --to-command-line STRING1[,STRING2[,...]]
    		 Command line arguments for the to branch Provide additional command line flags to set when running Puppet to compile catalogs. (command_line.rb) 
    --to-create-symlinks STRING1[,STRING2[,...]]
    		 Symlinks to create for the to branch Specify which directories from the base should be symlinked into the temporary compilation environment. This is useful only in conjunction with `--preserve-environments`. (create_symlinks.rb) 
    --to-enc PATH
    		 Path to ENC script (for the to catalog only) Path to external node classifier, relative to the base directory of the checkout. (enc.rb) 
    --to-enc-override STRING1[,STRING2[,...]]
    		 Override parameter from ENC for the to branch Allow override of ENC parameters on the command line. ENC parameter overrides can be supplied for the 'to' or 'from' catalog, or for both. There is some attempt to handle data types here (since all items on the command line are strings) by permitting a data type specification as well. For parameters nested in hashes, use `::` as the delimiter. (enc_override.rb) 
    --to-environment STRING
    		 Environment for catalog compilation for the to branch Specify the environment to use when compiling the catalog. This is useful only in conjunction with `--preserve-environments`. (environment.rb) 
    --to-fact-file STRING
    		 Override fact for the to branch Allow an existing fact file to be provided, to avoid pulling facts from PuppetDB. (fact_file.rb) 
    --to-fact-override STRING1[,STRING2[,...]]
    		 Override fact for the to branch Allow override of facts on the command line. Fact overrides can be supplied for the 'to' or 'from' catalog, or for both. There is some attempt to handle data types here (since all items on the command line are strings) by permitting a data type specification as well. (fact_override.rb) 
    --to-hiera-config STRING
    		 Full or relative path to global Hiera configuration file for the to branch Specify a relative path to the Hiera yaml file (hiera_config.rb) 
    --to-hiera-path STRING
    		 Path to hiera data directory, relative to top directory of repository for the to branch Specify the path to the Hiera data directory (relative to the top level Puppet checkout). For Puppet Enterprise and the Puppet control repo template, the value of this should be 'hieradata', which is the default. (hiera_path.rb) 
    --to-hiera-path-strip STRING
    		 Path prefix to strip when munging hiera.yaml for the to branch Specify the path to strip off the datadir to munge hiera.yaml file (hiera_path_strip.rb) 
    --to-puppet-binary STRING
    		 Full path to puppet binary for the to branch Set --puppet-binary, --to-puppet-binary, --from-puppet-binary (puppet_binary.rb) 
    --to-puppet-master STRING
    		 Hostname or Hostname:PortNumber for Puppet Master for the to branch Specify the hostname, or hostname:port, for the Puppet Master. (puppet_master.rb) 
    --to-puppet-master-api-version STRING
    		 Puppet Master API version (2 for Puppet 3.x, 3 for Puppet 4.x) for the to branch Specify the API version to use for the Puppet Master. This makes it possible to authenticate to a version 3.x PuppetMaster by specifying the API version as 2, or for a version 4.x PuppetMaster by specifying API version as 3. (puppet_master_api_version.rb) 
    --to-puppet-master-ssl-ca STRING
    		 Full path to CA certificate that signed the Puppet Master certificate for the to branch Specify the CA certificate for Puppet Master. If specified, this will enable SSL verification that the certificate being presented has been signed by this CA, and that the common name matches the name you are using to connecting. (puppet_master_ssl_ca.rb) 
    --to-puppet-master-ssl-client-cert STRING
    		 Full path to certificate file for SSL client auth to Puppet Master for the to branch Specify the SSL client certificate for Puppet Master. This makes it possible to authenticate with a client certificate keypair to the Puppet Master. (puppet_master_ssl_client_cert.rb) 
    --to-puppet-master-ssl-client-key STRING
    		 Full path to key file for SSL client auth to Puppet Master for the to branch Specify the SSL client key for Puppet Master. This makes it possible to authenticate with a client certificate keypair to the Puppet Master. (puppet_master_ssl_client_key.rb) 
    --to-puppet-master-timeout STRING
    		 Puppet Master catalog retrieval timeout in seconds for the to branch Specify a timeout for retrieving a catalog from a Puppet master / Puppet server. This timeout is specified in seconds. (puppet_master_timeout.rb) 
    --to-save-catalog STRING
    		 Save intermediate catalogs into files for the to branch Allow catalogs to be saved to a file before they are diff'd. (save_catalog.rb) 
    --truncate-details
--no-truncate-details
    		Truncate details with --display-detail-add When using `--display-detail-add` by default the details of any field will be truncated at 80 characters. Specify `--no-truncate-details` to display the full output. This option has no effect when `--display-detail-add` is not used. (truncate_details.rb) 
    --validate-references
--no-validate-references
    		References to validate Confirm that each `before`, `require`, `subscribe`, and/or `notify` points to a valid resource in the catalog. This value should be specified as an array of which of these parameters are to be checked. (validate_references.rb)
    ## Using these options in API calls Most of these options can also be used when making calls to the [API](/doc/dev/api.md). Generally, parameters for the API are named corresponding to the names of the command line parameters, with dashes (`-`) converted to underscores (`_`). For example, the command line option `--hiera-config` is passed to the API as the symbol `:hiera_config`. Each of the options above has a link to the source file where it is declared, should you wish to review the specific parameter names and data structures that are being set.octocatalog-diff-1.5.4/doc/advanced-bootstrap.md0000644000004100000410000000547713413744025021645 0ustar www-datawww-data# Bootstrapping your Puppet checkout For many implementations of Puppet, an intermediate step is required between checking out code from a repository and having that code be ready to be served via a Puppet Master server. For example, you may need to run `bundler` to install gems or `librarian-puppet` to download Puppet modules. This document will refer to this process -- whatever it may mean for your particular use case -- as *bootstrapping*. ## Bootstrapping with `octocatalog-diff` Since `octocatalog-diff` integrates closely with your git repository, we provide a mechanism to allow you to perform your bootstrapping between the checkout of the branch and the build of the catalog. The `--bootstrap-script` option takes a string parameter consisting of either: - An absolute path, starting with `/` - A path relative to your Puppet checkout, not starting with `/` For example, if you have a script named `script/bootstrap.sh` in a subdirectory of your Puppet repository, you could instruct `octocatalog-diff` to use this script for bootstrap by specifying: ``` octocatalog-diff --bootstrap-script script/bootstrap.sh ... ``` If you have your bootstrap script at a known location on the system (not stored in your Puppet repository), you can refer to it with an absolute path. ``` octocatalog-diff --bootstrap-script /etc/puppetlabs/repo-bootstrap.sh ... ``` ## Configuring bootstrapping via the configuration file The [example configuration file](/examples/octocatalog-diff.cfg.rb) contains an example setting for the bootstrap script. ``` # settings[:bootstrap_script] = '/etc/puppetlabs/repo-bootstrap.sh' # Absolute path # settings[:bootstrap_script] = 'script/bootstrap' # Relative path ``` ## Bootstrap environment When the bootstrap script runs, a limited set of environment variables are passed from the shell running octocatalog-diff. Only these variables are set: - `HOME` - `PATH` - `PWD` (set to the base directory of your Puppet checkout) - `BASEDIR` (as explicitly set with `--basedir` CLI option or `settings[:basedir]` setting) If you wish to set additional environment variables for your bootstrap script, you may do so via the `--bootstrap-environment VAR=value` command line flag, or by defining `settings[:bootstrap_environment] = { 'VAR' => 'value' }` in your configuration file. As an example, consider that your bootstrap script is written in Python, and needs the `PYTHONPATH` variable set to `/usr/local/lib/python-custom`. Even if this environment variable is set when octocatalog-diff is run, it will not be available to the bootstrap script. You may supply it via the command line: ``` octocatalog-diff --bootstrap-environment PYTHONPATH=/usr/local/lib/python-custom ... ``` Or you may specify it in your configuration file: ``` settings[:bootstrap_environment] = { 'PYTHONPATH' => '/usr/local/lib/python-custom' } ``` octocatalog-diff-1.5.4/doc/requirements.md0000644000004100000410000000441613413744025020600 0ustar www-datawww-data# Requirements To run `octocatalog-diff` you will need these basics: - An appropriate Puppet version and [corresponding ruby version](https://puppet.com/docs/puppet/5.4/system_requirements.html) - Puppet 5.x officially supports Ruby 2.4 - Puppet 4.x officially supports Ruby 2.1, but seems to work fine with later versions as well - Puppet 3.8.7 -- we attempt to maintain compatibility in `octocatalog-diff` to facilitate upgrades even though this version is no longer supported by Puppet - We don't officially support Puppet 3.8.6 or before - Mac OS, Linux, or other Unix-line operating system (Windows is not supported) - Ability to install gems, e.g. with [rbenv](https://github.com/rbenv/rbenv) or [rvm](https://rvm.io/), or root privileges to install into the system Ruby - Puppet agent for [Linux](https://docs.puppet.com/puppet/latest/reference/install_linux.html) or [Mac OS X](https://docs.puppet.com/puppet/latest/reference/install_osx.html), or installed as a gem We recommend that you also have the following to get the most out of `octocatalog-diff`, but these are not absolute requirements: - If your Puppet code stored in a git repository, `octocatalog-diff` can check out branches for you as it does its comparisons. Your git repository can be stored on [GitHub.com](https://github.com/), [GitHub Enterprise](https://enterprise.github.com/home), or similar. If your Puppet code is not stored in a git repository, you can still point the tool at "from" and "to" directories, but you'll have to check them out yourself. - If you have API access (HTTPS) to PuppetDB, `octocatalog-diff` can retrieve facts automatically and also support [exported resources](https://docs.puppet.com/puppet/latest/reference/lang_exported.html) if you use them. If you are not using PuppetDB or don't have access, the tool can still read facts from YAML files. - If your site uses an [external node classifier](https://docs.puppet.com/guides/external_nodes.html), `octocatalog-diff` can execute the ENC script as part of its catalog compiles. Depending on how your ENC is designed, this may require network access or credentials to some service. If you are not using an ENC, that's fine. If you have an ENC but don't have the requisite access, depending on your setup the tool could produce unexpected results. octocatalog-diff-1.5.4/doc/troubleshooting.md0000644000004100000410000000505213413744025021301 0ustar www-datawww-data# Troubleshooting Things not quite working as expected? This section will contain hints to help you get up and running. ### Make sure the tests pass If you are getting errors from ruby, we'd really like to know if the tests are passing on your platform. Please follow the [installation instructions](/doc/installation.md#installing-from-source) to install octocatalog-diff from source, if you have not already done so. Once the repository is checked out, change into the directory run `rake` to perform the tests. If you get test failures from a clean checkout of the master branch, please [open an issue](https://github.com/github/octocatalog-diff/issues/new) to let us know. ### Make sure your configuration file is found and error-free Run the following command to test for the existence and integrity of your configuration file. ``` octocatalog-diff --config-test ``` If you get an error indicating that the file can't be found, or you get errors arising from the content of the file, please review the [configuration instructions](/doc/configuration.md) to make sure you've set things up correctly. ### Run the command in debug mode Supplying `-d` on the command line, in addition to the node name and any other arguments, will provide a substantial amount of debugging information to the terminal window. If you ultimately end up requesting our help, we will need this debugging output. Example: ``` octocatalog-diff -d -n SomeNodeName.yourdomain.com ``` ### Run only certain components of the command To perform the bootstrapping and catalog compilation in separate steps, you can run octocatalog-diff with arguments asking it to do only one or the other. This will help you narrow down whether the problem is in the bootstrapping (first command) or catalog compilation (second command). Be sure you are in the directory where your Puppet code is checked out when you run these commands. To run just the bootstrapping code (do this within a checkout of your Puppet repository): ``` mkdir /tmp/octo-test octocatalog-diff -d --bootstrap-then-exit --bootstrapped-from-dir=/tmp/octo-test ``` To run just the catalog compilation code (do this within a checkout of your Puppet repository): ``` octocatalog-diff -d -n SomeNodeName.yourdomain.com -o /tmp/catalog.json --bootstrapped-to-dir=$PWD --catalog-only ``` ### Contact us Still having trouble? Please [open an issue](https://github.com/github/octocatalog-diff/issues/new) and we will do our best to help. Please follow the provided issue template, which will ask you for certain output that we need to diagnose the problem. octocatalog-diff-1.5.4/doc/advanced-storeconfigs.md0000644000004100000410000001065513413744025022327 0ustar www-datawww-data# Enabling storeconfigs for exported resources in PuppetDB The "storeconfigs" setting in Puppet is a feature related to [exported resources](https://docs.puppet.com/puppet/latest/reference/lang_exported.html). It is possible to enable the collection of exported resources when `octocatalog-diff` compiles catalogs, to give the most accurate representation possible of the catalogs before they are compared. ## Usage When you provide the `--storeconfigs` command line option, or set `settings[:storeconfigs] = true` in the [configuration file](/doc/configuration.md), the following behavior is triggered: - `octocatalog-diff` will create a `puppetdb.conf` file in its temporary compilation directory, using the [PuppetDB configuration settings](/doc/configuration-puppetdb.md) that you have specified, either as command line parameters or in a configuration file. - `octocatalog-diff` will install the SSL client certificates you have provided for PuppetDB, if any, in its temporary compilation directory, so that Puppet will pick these up and use them to connect to PuppetDB. This allows SSL client authentication to PuppetDB. (Please note: Puppet *must* connect to PuppetDB over an SSL connection, although not necessarily an authenticated SSL connection.) - `octocatalog-diff` will create a `routes.yaml` file in its temporary compilation directory so that Puppet does not try to send fact data, resource data, or reports back to PuppetDB. We have done our best to make this connection be "read only" although we do encourage you to set up a separate, read-only port for PuppetDB to ensure this. ## Caveats - Beware of load this may cause on PuppetDB, especially if you run `octocatalog-diff` simultaneously in a CI environment. At GitHub, we run `octocatalog-diff` distributed across 8 CI nodes, each of which is capable of performing 16 simultaneous catalog compilations. We have noticed performance degradation or outages when a "thundering herd" of 128 catalog compilations hit PuppetDB at the same time, leading us to implement a layer of caching on top of PuppetDB. - `octocatalog-diff` compiles a "before" and "after" catalog in mostly parallel fashion, but it is possible that the order of operations happens as follows: (a) "before" catalog compiles; (b) something else writes updated data to PuppetDB; (c) "after" catalog compiles. In this case, there can be false differences reported in the output. ## Advanced configuration This section contains tips on setting up a proxy in front of PuppetDB to control access and/or enable caching. You are not required to do this in order to use `--storeconfigs` but you may find these tips useful if simultaneous runs of `octocatalog-diff` put too much load on your PuppetDB instance. ### Making a read-only PuppetDB port It is possible to create a read-only endpoint to PuppetDB by setting up a proxy that only allows the desired URLs. This ensures that the Puppet runs cannot submit fact data, resource data, or reports back to PuppetDB from `octocatalog-diff` runs. (As noted previously, we do set up `routes.yaml` to prevent this, but the strategy in this section provides an extra layer of security.) To allow only the desired traffic, you should configure a port on your proxy that will pass to these URLs only: - /pdb/query - /pdb/meta ### Caching To reduce the impact of a "thundering herd" of simultaneous `octocatalog-diff` runs, you can set up a caching proxy in front of the `/pdb/query` endpoint. Here is a portion of the nginx configuration that has provided the best balance between performance and accuracy. We have chosen a 1 minute TTL on results, and incorporated the request body into the cache key. You will need to adjust and incorporate this configuration into your own nginx proxy. ``` upstream puppetdb { server localhost:8080; } proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=pdb_cache:10m max_size=25g inactive=2m; server { listen 10.0.0.1:8082; # Some SSL settings omitted. Configure as per your own needs. ssl_client_certificate /etc/nginx/ca.crt ssl_verify_client on; proxy_cache_key "$scheme$proxy_host$uri$is_args$args|$request_body"; deny all; location /pdb/query { proxy_cache pdb_cache; proxy_ignore_headers Cache-Control; proxy_cache_valid any 1m; proxy_pass http://puppetdb; proxy_redirect off; add_header X-Cache-Status $upstream_cache_status; allow all; } # Other settings omitted. } ``` octocatalog-diff-1.5.4/doc/advanced-environments.md0000644000004100000410000000667413413744025022357 0ustar www-datawww-data# Environment setup When building a catalog, the default behavior of `octocatalog-diff` is to: 1. Create a temporary directory 2. Create a symlink from `/environments/production` to the checkout of your code 3. Run Puppet using `environment=production` If you are using environment names to control the behavior of Puppet, this default behavior may not be suitable. In that case you can invoke the alternate behavior: preserving environments. ## Command line options ### Preserving the environments When you supply the command line argument `--preserve-environments` (or set `settings[:preserve_environments] = true` in your [configuration file](/doc/configuration.md)), `octocatalog-diff` will instead do the following: 1. Create a temporary directory 2. Create the following symlinks from `` to the corresponding directories in the checkout of your code: - `environments` - `manifests` - `modules` 3. Run Puppet using an environment you specify via the command line Note that you must have set `--preserve-environments` in order for the `--environment` and/or `--create-symlinks` options (described below) to have any effect. ### Changing the environment If you wish to use an environment name other than `production` you can use the `--environment ` command line option. This will set the environment for both the `to` and `from` compiles. ``` octocatalog-diff ... --preserve-environments --environment some-env-name ``` If you need to specify different environments for the `to` and `from` compiles, you can use `--to-environment ` and `--from-environment `. ``` octocatalog-diff ... --preserve-environments --to-environment first-environment --from-environment second-environment ``` ### Controlling symlinks that are created Within the temporary directory, the `environments` symlink will always be created. By default, `manifests` and `modules` will also be created from the temporary directory to the corresponding directories in your Puppet code base. If you need to customize the symlinks that are created, you can use the `--create-symlinks ,,...` to list the symlinks that you need. For example, if you have some code stored in a directory called `modules` and more code stored in a directory called `site`, you could do the following to create the symlinks as desired: ``` octocatalog-diff ... --preserve-environments --create-symlinks manifests,modules,site ``` ## Examples Consider that your Puppet code base is organized as follows: ``` - /opt/puppet - environments - old - environment.conf - manifests - site.pp - modules - module_zero - new - environment.conf - manifests - site.pp - modules - module_zero - modules - module_one - module_two - site - module_three - module_four ``` To calculate the difference between the "old" and "new" environment, you could use: ``` octocatalog-diff \ --bootstrapped-from-dir /opt/puppet \ --bootstrapped-to-dir /opt/puppet \ --preserve-environments \ --from-environment old \ --to-environment new \ --create-symlinks modules,site ``` (Note that `--bootstrapped-from-dir` and `--bootstrapped-to-dir` are used to specify the directory path to your code, and `-t` and `-f` are not used. That's because the difference in the catalog is derived from the environment used, and not the branch from a git repository.) octocatalog-diff-1.5.4/doc/installation.md0000644000004100000410000000422013413744025020547 0ustar www-datawww-data# Installation Before you get started, please make sure that you have the following: - Ruby 2.0 or higher - Mac OS, Linux, or other Unix-line operating system (Windows is not supported) - Ability to install gems, e.g. with [rbenv](https://github.com/rbenv/rbenv) or [rvm](https://rvm.io/), or root privileges to install into the system Ruby - Puppet agent for [Linux](https://docs.puppet.com/puppet/latest/reference/install_linux.html) or [Mac OS X](https://docs.puppet.com/puppet/latest/reference/install_osx.html), or installed as a gem - required if you are going to compile Puppet catalogs locally without querying a master ## Installing from rubygems.org `octocatalog-diff` is published on [rubygems](https://rubygems.org/gems/octocatalog-diff). On a standard system with internet access, installation may be as simple as typing: ``` gem install octocatalog-diff ``` Once the gem is installed, please proceed to [Configuration](/doc/configuration.md). For general information on installing gems, see: [RubyGems Basics](http://guides.rubygems.org/rubygems-basics/#installing-gems). ## Installing from source To install from source, you'll need a git client and internet access. 0. Clone the repository ``` git clone https://github.com/github/octocatalog-diff.git ``` 0. Bootstrap the repository (this will install dependent gems in the project) ``` cd octocatalog-diff ./script/bootstrap ``` 0. RECOMMENDED: Make sure the tests pass on your machine ``` rake ``` Note: If tests fail on your machine with a clean checkout of the master branch, we would definitely appreciate if you would report it. Please [open an issue](https://github.com/github/octocatalog-diff/issues/new) with the output and some information about your system (e.g. OS, ruby version, etc.) to let us know. Once the code is downloaded and bootstrapped, please proceed to [Configuration](/doc/configuration.md). ## Running from an alternate branch We have prepared specific instructions for running `octocatalog-diff` from a non-master branch, for testing changes that may be requested by the developers. - [Running octocatalog-diff from a branch](/doc/dev/run-from-branch.md) octocatalog-diff-1.5.4/doc/advanced-environment-variables.md0000644000004100000410000000600613413744025024127 0ustar www-datawww-data# Environment variables The following environment variables have special meaning to octocatalog-diff: ### `OCTOCATALOG_DIFF_CONFIG_FILE` This environment variable is used to locate the configuration file for the CLI. The use of configuration files is described generally in: - [Configuration](/doc/configuration.md) ### `OCTOCATALOG_DIFF_CUSTOM_VERSION` When set, the `octocatalog-diff` CLI will display this as the version number within debugging, instead of the version number in the package. This is most useful if you want to use or include a git SHA in the version number. ``` $ export OCTOCATALOG_DIFF_CUSTOM_VERSION="@$(git rev-parse HEAD)" $ octocatalog-diff -d ... D, [2017-10-12T08:57:46.454738 #35205] DEBUG -- : Running octocatalog-diff @504d7f3c91267e5193beb103caae5d4d8cebfee3 with ruby 2.3.1 ... ``` ### `OCTOCATALOG_DIFF_DEVELOPER_PATH` When set, instead of loading libraries from the system or bundler location, libraries will be loaded from the specified value of this environment variable. This is used internally for development as we point users to unreleased code for debugging or testing. ``` $ export OCTOCATALOG_DIFF_DEVELOPER_PATH=$HOME/git-checkouts/octocatalog-diff $ octocatalog-diff ... ``` ### `OCTOCATALOG_DIFF_TEMPDIR` When set: - `octocatalog-diff` will create all of its temporary directories within the specified directory. - `octocatalog-diff` will not attempt to remove any temporary directories it creates. This is useful in the following situations: - You are calling `octocatalog-diff` from within another program which is highly parallelized, and `at_exit` handlers are difficult to implement. Instead of figuring that all out (if it can even be figured out), you create a temporary directory before the parallelized logic, and remove it afterwards. - You wish to debug intermediate output. For example, you may be instructed to set this variable and send some of the output to the project maintainers if you request assistance. This variable is used internally for the parallelized logic for catalog compilation, but the value set from the environment will override any internal usage. ### `OCTOCATALOG_DIFF_VERSION` This variable is used when building the gem, to override the default version. This is used for internal testing of `octocatalog-diff` before public releases. This variable is not useful outside the build context. ### `PUPPETDB_HOST` This variable specifies the fully qualified domain name or IP address of the PuppetDB server. Note: If `PUPPETDB_URL` is specified, then `PUPPETDB_HOST` is not consulted. ### `PUPPETDB_PORT` This variable specifies the port number of the PuppetDB server. Note: If `PUPPETDB_URL` is specified, then `PUPPETDB_PORT` is not consulted. ### `PUPPETDB_URL` This variable specifies the URL to the PuppetDB server. Example: `https://puppetdb.example.net:8081` ### `PUPPET_FACT_DIR` This variable specifies the directory path where puppet fact files are stored. (Fact files must be named `.yaml` where `` is specified when running `octocatalog-diff`.) octocatalog-diff-1.5.4/doc/limitations.md0000644000004100000410000000552513413744025020413 0ustar www-datawww-data# Limitations Testing of Puppet catalogs is faster than running the agent, but you need to be careful of the following limitations: 0. Facts are not taken from a live agent run octocatalog-diff by default uses the facts reported from a node's more recent Puppet run. If you have made changes to custom facts, catalog testing will **NOT** be an adequate test of whether your custom facts worked. (You can still use octocatalog-diff to help predict changes to nodes based on changes to facts, by overriding facts on the command line.) 0. Agents handle depenency ordering and implementation details The catalog defines the state of the system, but it's up to the agent to determine how to bring the system to a point that matches the catalog. The agent is responsible for order of operations and actually making the change. Two specific situations that catalog testing does **NOT** detect are: - Dependency loops (e.g., you have made A require B, B require C, and C require A). - Operations not supported by the provider. For example, assume that in your current Puppet manifests, you set the size of a file system to 100 GB. You change this in your new branch to 50 GB. octocatalog-diff will dutifully report this change to you. However, the agent will fail to make the change, because it is not possible to shrink a file system from 100 GB to 50 GB. 0. Changes in underlying providers may not be noticed Consider that you are using a Puppet module that creates a file system. The current implementation of that module checks to see if *any* file system is present on the device, and creates a new file system there if no file system was present. You upgrade the module, and the new version checks to see if *the specified* file system is present on the device, and reformats the device with the specified file system (regardless of whether there was no file system or if there was an existing file system of a different type). There would be no catalog changes (hence octocatalog-diff would report nothing) because the catalog simply instructs the agent to create a file system of the specified type at the defined location. However, the actual implementation of those instructions has changed dramatically. In general catalog testing is great for: - Refactoring classes and defined types (which do not have custom providers) - Moving information around in hiera - Generally adding, removing, or modifying standard Puppet resources - Checking the net effect of any custom functions (`lib/puppet/parser/functions`) since these execute during catalog compilation Catalog testing in general (including octocatalog-diff and similar tools) is generally inadequate for: - Changes to custom facts - Changes to any providers (includes upgrading any Puppet modules from Puppet Forge or other sources) - Changes to order of operations (before, require, subscribe, notify) octocatalog-diff-1.5.4/doc/configuration-enc.md0000644000004100000410000001110213413744025021455 0ustar www-datawww-data# Configuring octocatalog-diff to use ENC If you use an [External Node Classifier (ENC)](https://docs.puppet.com/guides/external_nodes.html) on your Puppet master, you should configure octocatalog-diff to use the same ENC when it compiles catalogs. Before you start, please understand how octocatalog-diff compiles a catalog: - It creates a temporary directory (e.g. `/var/tmp/puppet-compile-dir-92347829847`) - It copies or creates hiera configuration, ENC, PuppetDB configuration, etc., in the temporary directory - It symlinks `/var/tmp/puppet-compile-dir-92347829847/environments/production` to your code - It compiles the catalog, based on the temporary directory, for environment=production - It removes the temporary directory NOTE: If you are using the built-in node classification in Puppet Enterprise, you don't need to worry about any of this. Instead, please read about [Puppet Enterprise as your ENC](/doc/advanced-pe-enc.md). ## Configuring the path to the ENC The command line option `--enc PATH` allows you to set the path to your ENC script. You may specify this as either an absolute or a relative path. - As a relative path octocatalog-diff knows to use a relative path when the supplied path for `--enc` does not start with a `/`. ``` bin/octocatalog-diff --enc config/enc.sh ... ``` The path is relative to a checkout of your Puppet repository. As per the example in the introduction, say that octocatalog-diff is using a temporary directory of `/var/tmp/puppet-compile-dir-92347829847` when compiling a Puppet catalog. With the setting above, it will copy `config/enc.sh` (relative to your Puppet checkout) into the temporary directory. If you are specifying the ENC path in the [configuration file](/doc/configuration.md), you will instead set the variable like this: ``` settings[:enc] = 'config/enc.sh' ``` octocatalog-diff will fail if you specify a ENC location that cannot be opened. - As an absolute path octocatalog-diff knows to use a relative path when the supplied path for `--enc` starts with a `/`. For example: ``` bin/octocatalog-diff --enc /etc/puppetlabs/puppet/enc.sh ... ``` If you are specifying the ENC path in the [configuration file](/doc/configuration.md), you will instead set the variable like this: ``` settings[:enc] = '/etc/puppetlabs/puppet/enc.sh' ``` Please note that octocatalog-diff will copy the file from the specified location into the compile directory. Since this ENC file is not copied from your Puppet repo, there is no way to compile the "to" and "from" branches using different ENC scripts. Furthermore, you are responsible for getting this file into place on any machine that will run octocatalog-diff. We strongly recommend that you version-control your ENC script within your Puppet repository, and use the relative path option described above. ## Executing the ENC When Puppet runs the ENC, it will do so with one argument (the node name for which you are compiling the catalog). For example, when compiling the catalog for `some-node.github.net`, Puppet will effectively execute this command: ``` /etc/puppetlabs/puppet/enc.sh some-node.github.net ``` Sometimes the ENC script requires credentials or makes other assumptions about the system on which it is running. To be able to run the ENC script on systems other than your Puppet master, you will need to ensure that any such credentials are supplied and other assumptions are met. ## Environment When the ENC is executed, the following environment variables are set to match the environment of the shell in which octocatalog-diff executes: - `HOME` - `PATH` - `PWD` (set to the temporary directory as previously described) No other environment variables are passed from the shell. If you wish to pass additional environment variables, you must explicitly list them with the `--pass-env-vars` CLI flag or `settings[:pass_env_vars]` array in your configuration file. As an example, consider that your ENC is written in Python, and needs the `PYTHONPATH` variable set to `/usr/local/lib/python-custom`. Even if this environment variable is set when octocatalog-diff is run, it will not be available to the ENC script. You may pass the variable via the command line: ``` octocatalog-diff --pass-env-vars PYTHONPATH ... ``` Or you may specify it in your configuration file: ``` settings[:pass_env_vars] = [ 'PYTHONPATH' ] ``` If you wish to specify multiple environment variables to pass: ``` octocatalog-diff --pass-env-vars PYTHONPATH,SECONDVAR,THIRDVAR ... ``` or ``` settings[:pass_env_vars] = [ 'PYTHONPATH', 'SECONDVAR', 'THIRDVAR' ] ``` octocatalog-diff-1.5.4/doc/advanced-pe-enc.md0000644000004100000410000000507713413744025020773 0ustar www-datawww-data# Puppet Enterprise node classification service If you are using Puppet Enterprise, `octocatalog-diff` can use the node classifier service API instead of an external node classifier. ## Basics To use the Puppet Enterprise node classifier service instead of an ENC, you must supply the URL to your node classifier service API endpoint, and either an authentication token or a whilelisted SSL client keypair. It is recommended to supply the SSL Certificate Authority (CA) file as well, to verify the identity of the server to which you are connecting. To use Puppet Enterprise node classifier service with an authentication token: ``` octocatalog-diff \ --pe-enc-url https://your.pe.console.server:4433/classifier-api \ --pe-enc-token-file /path/to/token/file.txt \ --pe-enc-ssl-ca /path/to/ca.crt \ [other options] ``` To use Puppet Enterprise node classifier service with a whitelisted SSL client keypair: ``` octocatalog-diff \ --pe-enc-url https://your.pe.console.server:4433/classifier-api \ --pe-enc-ssl-ca /path/to/ca.crt \ --pe-enc-ssl-client-cert /path/to/client.crt \ --pe-enc-ssl-client-key /path/to/client.key \ [other options] ``` ## Details ### Requirements - Your PE console server must be accessible to the machine from which you are running `octocatalog-diff` on the appropriate port (by default, 4433). This port is *not* required to run Puppet agents, so it's possible that you have it firewalled off. ### Authentication token Please see [Authentication token](https://docs.puppet.com/pe/latest/nc_forming_requests.html#authentication-token) in the official Puppet documentation. The referenced document contains links to generate a token with the `puppet-access` command. Note that if you wish to hard-code an authentication token in your [configuration file](/doc/configuration.md), the internal variable key is `:pe_enc_token` and the content is a string containing the entire token. (The `--pe-enc-token-file` option simply reads the provided file and stores the content in the `:pe_enc_token` key. See [source](/lib/octocatalog-diff/cli/options/pe_enc_token_file.rb).) ### SSL client keypair If you wish to use a SSL client keypair instead of a token, please see [Whilelisted certificate](https://docs.puppet.com/pe/latest/nc_forming_requests.html#whitelisted-certificate) in the official Puppet documentation. The referenced document contains instructions to add the certificate name to the whitelist file and restart the necessary services. ### Further reading [Puppet documentation on node classifier endpoints](https://docs.puppet.com/pe/latest/nc_index.html) octocatalog-diff-1.5.4/doc/advanced-puppet-versions.md0000644000004100000410000000144113413744025022776 0ustar www-datawww-data# Compiling the same catalog with different Puppet versions `octocatalog-diff` can be a valuable tool when upgrading from one Puppet version to another. By instructing `octocatalog-diff` to compile the "from" catalog with one version of Puppet, and the "to" catalog with another version of Puppet, you can look for changes that arise due to different Puppet versions. To use this feature, simply point `octocatalog-diff` at the Puppet binary you wish to use for each catalog. - `--from-puppet-binary DIRECTORY_PATH/puppet` will compile the "from" catalog with the specified Puppet binary - `--to-puppet-binary DIRECTORY_PATH/puppet` will compile the "to" catalog with the specified Puppet binary - `--puppet-binary DIRECTORY_PATH/puppet` will compile both catalogs with the specified Puppet binary octocatalog-diff-1.5.4/doc/advanced-override-enc.md0000644000004100000410000000357513413744025022207 0ustar www-datawww-data# Overriding ENC parameters One powerful feature of `octocatalog-diff` allows you to override ENC parameters when compiling the catalogs, to predict the effect of an ENC parameter change on the catalog. This is useful to simulate a change in agent node configuration without actually setting up an agent to do so. ## Usage To override an ENC parameter in both catalogs: ``` --enc-override parameters::some_class::some_param=value ``` To override an ENC parameter in the "to" catalog: ``` --to-enc-override parameters::some_class::some_param=value ``` To override an ENC parameter in the "from" catalog: ``` --from-enc-override parameters::some_class::some_param=value ``` You may use as many of these arguments as you wish to adjust as many ENC parameters as you wish. ## Limitations As presently implemented, this only works on ENCs that supply their results as YAML. Is your ENC doing something different? Please [let us know](https://github.com/github/octocatalog-diff/issues/new) so we can enhance octocatalog-diff to handle it! ## Examples Simulate a change to a top-level parameter named "role" only in the "to" catalog: ``` octocatalog-diff -n some-node.example.com -f master -t master \ --to-enc-override parameters::role=some_new_role ``` Simulate a change in a class parameter between the catalogs: ``` octocatalog-diff -n some-node.example.com -f master -t master \ --from-enc-override parameters::my_class::my_value=value_in_old \ --to-enc-override parameters::my_class::my_value=value_in_new ``` Note that each of the examples specified the from branch and to branch to be `master`. There is no requirement that you do this, but you can generally obtain the most accurate test results by changing only one variable at a time. ## Advanced usage The format for declaring overrides with data types is the same as [overriding facts](/doc/advanced-override-facts.md#advanced-usage). octocatalog-diff-1.5.4/doc/advanced-filter.md0000644000004100000410000001017213413744025021101 0ustar www-datawww-data# Additional output filters It is possible to enable additional filters for output results via the `--filters` command line option. This command line option accepts a comma-separated list of additional filters, and applies them to the results in the order you specify. The default behavior is not to use any of these filters. Please note that there are other options to ignore specified diffs, including: - [Ignoring certain changes via command line options](/doc/advanced-ignores.md) - [Dynamic ignoring of changes via tags in Puppet manifests](/doc/advanced-dynamic-ignores.md) Here is the list of available filters and an explanation of each: - [Absent File](/doc/advanced-filter.md#absent-file) - Ignore parameter changes of a file that is declared to be absent - [JSON](/doc/advanced-filter.md#json) - Ignore whitespace differences if JSON parses to the same object - [SingleItemArray](/doc/advanced-filter.md#SingleItemArray) - Ignore differences between object and array containing only that object - [YAML](/doc/advanced-filter.md#yaml) - Ignore whitespace/comment differences if YAML parses to the same object ## Absent File #### Usage ``` --filters AbsentFile ``` #### Description When the `AbsentFile` filter is enabled, if any file is `ensure => absent` in the *new* catalog, then changes to any other parameters will be suppressed. Consider that a file resource is declared as follows in two catalogs: ``` # Old catalog file { '/etc/some-file': ensure => present, owner => 'root', group => 'nobody', content => 'my content here', } # New catalog file { '/etc/some-file': ensure => absent, owner => 'bob', } ``` Since the practical effect of the new catalog will be to remove the file, it doesn't matter that the owner of the (non-existent) file has changed from 'root' to 'bob', or that the content and group have changed from a string to undefined. Consider the default output without the filter: ``` File[/etc/some-file] => parameters => ensure => - present + absent group => - nobody owner => - root + bob content => - my content here ``` Wouldn't it be nice if the meaningless information didn't appear, and all you saw was the transition you actually care about, from present to absent? With `--filters AbsentFile` it does just this: ``` File[/etc/some-file] => parameters => ensure => - present + absent ``` ## JSON #### Usage ``` --filters JSON ``` #### Description If a file resource has extension `.json` and a difference in its content is observed, JSON objects are constructed from the previous and new values. If these JSON objects are identical, the difference is ignored. This allows you to ignore changes in whitespace, comments, etc., that are not meaningful to a machine parsing the file. Note that changes to files may still trigger Puppet to restart services even though these changes are not displayed in the octocatalog-diff output. ## Single Item Array #### Usage ``` --filters SingleItemArray ``` #### Description When enabling the future parser or upgrading between certain versions of Puppet, the internal structure of the catalog for certain parameters can change as shown in the following example: ``` Old: { "notify": "Service[foo]" } New: { "notify": [ "Service[foo]" ] } ``` This filter will suppress differences for the value of a parameter when: - The value in one catalog is an object, AND - The value in the other catalog is an array containing *only* that same object ## YAML #### Usage ``` --filters YAML ``` #### Description If a file resource has extension `.yml` or `.yaml` and a difference in its content is observed, YAML objects are constructed from the previous and new values. If these YAML objects are identical, the difference is ignored. This allows you to ignore changes in whitespace, comments, etc., that are not meaningful to a machine parsing the file. Please note that by filtering these changes, you are ignoring changes to comments, which may be meaningful to humans. Also, changes to files may still trigger Puppet to restart services even though these changes are not displayed in the octocatalog-diff output. octocatalog-diff-1.5.4/doc/advanced-output-hacks.md0000644000004100000410000000406513413744025022247 0ustar www-datawww-data# Output hacks This document describes command line options that are useful to format the output of the human readable text format (which is the default). Note that if you are [outputting in JSON](/doc/advanced-output-formats.md#json-format), none of these options will have any effect. See also [Output formats](/doc/advanced-output-formats.md) for details on text vs. JSON and color vs. non-color. ## Displaying the detail of added resources (`--display-detail-add`) By default, `octocatalog-diff` will display an added resource on one line, without displaying all of the parameters. For example: ``` + File[/tmp/my-file] ``` If you would like to see the detail (all parameters and other settings), provide the `--display-detail-add` command line argument. Then the display will look more like: ``` + File[/tmp/my-file] => parameters => "content": "This is my amazing new file", "ensure": "file", "group": "root", "mode": "0755", "owner": "root" ``` If any line is too long, it will be abbreviated with `...`. ## Displaying the file and line giving rise to a resource (`--display-source`) To get help tracking down the file and line giving rise to a resource, provide the `--display-source` command line argument. If the file name and line number are known in the catalog, they will be displayed with the resource. If this information is the same between the old and new catalogs, it will be displayed just once. If the information differs, it will be displayed twice in a `diff` format: the `-` represents the location in the old catalog, and `+` represents the location in the new catalog. ## Tuning the level of debugging (`--debug`, `--quiet`) `octocatalog-diff` comes with 3 levels of log output. From least to most: | Command line option | Description | | ------------------- | ----------- | | `--quiet` (or `-q`) | Only critical errors are displayed to STDERR | | (default) | Critical errors and informational messages (statistics mainly) are displayed to STDERR | | `--debug` (or `-d`) | Full debugging messages are displayed to STDERR | octocatalog-diff-1.5.4/doc/advanced-ci.md0000644000004100000410000000122213413744025020203 0ustar www-datawww-data# Using `octocatalog-diff` in CI ## Catalog compilation check Compile the catalog for several important hosts, and ensure that all catalogs successfully compile before permitting the merge. This is a pass/fail check. - [Sample rspec file for catalog compilation test](/examples/ci/puppet_catalogs_spec.rb) ## Catalog difference analysis Compute the difference between the proposed code and the base branch across a number of hosts. This is more than just a pass/fail check, as a human should review the results to ensure that the differences are expected. - [Sample rspec file for catalog difference analysis](/examples/ci/puppet_catalog_diff_spec.rb) octocatalog-diff-1.5.4/doc/advanced-puppet-master.md0000644000004100000410000001046013413744025022422 0ustar www-datawww-data# Fetching catalogs from Puppet Master / PuppetServer `octocatalog-diff` can fetch catalogs from a Puppet Master, PuppetServer, or Puppet Enterprise server by calling their HTTPS API, just as a node would when it fetches its catalog. For simplicity, this document will refer only to "Puppet Master" but unless otherwise noted, the instructions apply equally to the open source PuppetServer and Puppet Enterprise PuppetServer as well. Please note the following caveats: 0. This method will put some load on your Puppet Master to build the catalog. Depending on how you use `octocatalog-diff` you should ensure that this extra load will not overwhelm your Puppet Master (especially if you create a "thundering herd" by launching several instances of `octocatalog-diff` simultaneously). 0. You will need to deploy your Puppet code to an environment on your Puppet Master prior to running `octocatalog-diff` for that environment. `octocatalog-diff` does not deploy code for you. 0. You will need to configure authorization for one or more whitelisted certificates on your Puppet Master. The default permissions allow a node to retrieve its own catalog via the API, but you need a certificate for `octocatalog-diff` that permits it to retrieve any catalog. See the [Certificate authorization](#certificate-authorization) section below. ## Command line options The following command line options are used to retrieve a catalog from a Puppet Master: | Option | Description | | ------ | ----------- | | `-f ENVIRONMENT` | Environment name to use for the "from" catalog | | `-t ENVIRONMENT` | Environment name to use for the "to" catalog | | `--puppet-master HOSTNAME:PORT | The hostname and port number of the Puppet Master. (By default the port used by Puppet Master is 8140.) | | `--puppet-master-api-version VERSION | The API version used by the Puppet Master. API versions 2 and 3 are supported. Puppet Master 3.x uses API version 2, and the PuppetServer for Puppet 4.x uses API version 3. By default, API version 3 is used, so you only need to set this option if you are using Puppet Master 3.x. | | `--puppet-master-ssl-ca PATH` | Path to the CA certificate (public portion of certificate only) for your Puppet Master. This file will be on your Puppet Master and all Puppet agents. You can find it by running `puppet config print cacert` on any Puppet-managed host. | | `--puppet-master-ssl-client-cert PATH` | Path to the client certificate. Please see the section below on certificate authentication. | | `--puppet-master-ssl-client-key PATH` | Path to the client private key. Please see the section below on certificate authentication. | If you wish to use a different Puppet Master to compile the "to" and "from" catalogs, you may prefix any of the `--puppet-master...` options with `to` or `from`. For example, perhaps you are testing an upgrade from Puppet 3.x to 4.x. You could use: ``` ... --from-puppet-master puppet3-x.yourdomain.com:8140 --from-puppet-master-api-version 2 --to-puppet-master puppet4-x.yourdomain.com:8140 ... ``` It is possible to "mix and match" catalog generation methods. For example, you could retrieve a "from" catalog from a Puppet Master using `--from-puppet-master` while compiling a "to" catalog from local code. Please note that some enhanced options of `octocatalog-diff`, such as comparing file text instead of file source location, may not be available for all such combinations. ## Certificate authorization In order to use `octocatalog-diff` you will need to create one or more certificates that are empowered to retrieve all catalogs. This requires both creating the certificate, and reconfiguring your Puppet Master to expand the scope of authorization for that certificate. Puppet Masters use the [legacy auth.conf file](https://docs.puppet.com/puppet/latest/reference/config_file_auth.html) and/or [PuppetServer auth.conf file](https://docs.puppet.com/puppetserver/latest/config_file_auth.html) to control access to HTTPS API. In particular, the following entry in the legacy auth.conf permits a particular agent to retrieve its own catalog: ``` # allow nodes to retrieve their own catalog (ie their configuration) path ~ ^/catalog/([^/]+)$ method find allow $1 ``` Please follow the instructions for the version of Puppet Master, PuppetServer, or Puppet Enterprise that you are using in order to generate and authorize the certificates. octocatalog-diff-1.5.4/doc/advanced-hiera-path-stripping.md0000644000004100000410000000352613413744025023660 0ustar www-datawww-data# Configuring octocatalog-diff to use Hiera path stripping This is a different, and potentially more complex, alternative to `--hiera-path` / `settings[:hiera_path]` described in the [Configuring octocatalog-diff to use Hiera](/doc/configuration-hiera.md) document. Unless you have a very good reason, you should prefer to use the instructions in that document instead of using the more complicated option that is described herein. The command line option `--hiera-path-strip PATH` allows you to manipulate directory paths for the JSON or YAML hiera backends. This setting only has an effect on the copy of hiera.yaml that is copied into the temporary compilation directory. This does not make any changes to the actual source hiera.yaml file on your system or in your checkout. For example, perhaps your production hiera.yaml file has entries such as the following: ``` --- :backends: - yaml :hierarchy: - "nodes/%{::trusted.certname}" - common :yaml: :datadir: /etc/puppetlabs/code/environments/%{environment}/hieradata ``` However, when you run octocatalog-diff on a machine that is not a Puppet master, the hiera data will not actually be found in `/etc/puppetlabs/code/environments/production/hieradata`, but rather in a directory called `hieradata` relative to the checkout of your Puppet code. Specifying `--hiera-path-strip PATH` causes octocatalog-diff to munge the datadir for the YAML and JSON configuration. The correct command in this case is now: ``` bin/octocatalog-diff --hiera-config hiera.yaml --hiera-path-strip /etc/puppetlabs/code ``` ``` --- :backends: - yaml :hierarchy: - "nodes/%{::trusted.certname}" - common :yaml: :datadir: /var/tmp/puppet-compile-dir-92347829847/environments/%{environment}/hieradata ``` :warning: Be sure that you do NOT include a trailing slash on `--hiera-path-strip` or `settings[:hiera_path_strip]`. octocatalog-diff-1.5.4/doc/advanced-script-override.md0000644000004100000410000000427713413744025022746 0ustar www-datawww-data# Overriding external scripts ## Background During normal operation, `octocatalog-diff` runs certain scripts or commands from the underlying operating system. For example, it may run `git` to check out a certain code branch, and run `puppet` to build a catalog. Each external script is found within the [`scripts`](/scripts) directory. ## How to override scripts ### Command line option It is possible to override these scripts with customized versions. To do this, specify a directory that contains replacement scripts via the command line: ``` octocatalog-diff [other options] --override-script-path /path/to/scripts ... ``` ### Configuration file You can also specify this option via a [configuration file](/doc/configuration.md) setting: ``` settings[:override_script_path] = '/path/to/scripts' ``` ### Writing replacement scripts Within the override script path you've configured, place a file with the same name as the built-in script. For example, if you wish to override the `git-extract.sh` script with a custom version, also name your script `git-extract.sh`. (Do NOT create subdirectories within the override directory.) If you specify an override script path but a particular script is not present there, octocatalog-diff will default to the built-in script. This means that you do not need to create unmodified copies of the built-in scripts. Only override the scripts you need to change. ### Notes Please note that these scripts are considered part of octocatalog-diff, and not part of your Puppet codebase. Therefore, the path to your scripts must be an absolute path, and we do not support (or intend to support) using multiple script directories during the same run of octocatalog-diff. ## Explanation of scripts This is an explanation of the [existing scripts supplied by octocatalog-diff](/scripts): - [`env.sh`](/scripts/env) Prints out the environment. This is currently only used for spec tests. - [`git-extract.sh`](/scripts/git-extract) Extracts a specified branch from the git repository into a specified target directory. - [`puppet.sh`](/scripts/puppet) Runs puppet (with additional command line arguments), generally used to compile a catalog or determine the Puppet version. octocatalog-diff-1.5.4/doc/advanced-dynamic-ignores.md0000644000004100000410000000710513413744025022706 0ustar www-datawww-data# Dynamic Ignoring Based On Tags Using the `--ignore-tags` command line option, it is possible to ignore all resources with particular Puppet tags. This allows dynamic ignoring of wrappers or other resources that are not of interest. NOTE: This option is separate and distinct from `--include-tags`, which controls whether differences in tags themselves will appear as a difference. For more on `--include-tags`, consult the [options reference](/doc/optionsref.md). ## Getting Started To use ignored tags, you first need to decide what the name of your tag will be. The standard is `ignored_octocatalog_diff`. When you are writing Puppet code, you can tag a particular resource as being of no interest to `octocatalog-diff`. ``` class foo { file { '/etc/foo': ensure => file, source => 'puppet:///modules/foo/etc/foo', tag => [ 'ignored_octocatalog_diff' ], } } ``` You can also tag a resource that is a custom defined type. ``` class foo { foo::customfile { '/etc/foo': source => 'puppet:///modules/foo/etc/foo', tag => [ 'ignored_octocatalog_diff__foo__customfile' ], } } define foo::customfile ( String $source, ) { file { $name: ensure => file, source => $source, } } ``` Finally, you can tag an entire defined type. ``` class foo { foo::customfile { '/etc/foo': source => 'puppet:///modules/foo/etc/foo', } } define foo::customfile ( String $source, ) { tag 'ignored_octocatalog_diff__foo__customfile' file { $name: ensure => file, source => $source, } } ``` When octocatalog-diff processes the ignore-tag, it will ignore a resource if either of the following is true: - The resource has a tag exactly matching the ignore-tag. For the default tag name, this means a resource has the tag `ignored_octocatalog_diff`. - The resource has a tag that matches the ignore-tag joined to the type with two underscores (where the type is in lower case and non-alphanumeric characters are replaced with underscores). This means that when the ignore-tag is `ignored_octocatalog_diff`, octocatalog-diff would ignore a file resource with a tag of `ignored_octocatalog_diff__file`, but would not ignore an exec resource with that same tag. The reasoning for the second syntax is explained in [caveats](#caveats). ## Usage To ignore one tag: ``` octocatalog-diff --ignore-tags ignored_octocatalog_diff ... ``` To ignore multiple tags: ``` octocatalog-diff --ignore-tags ignored_octocatalog_diff --ignore-tags second_tag ... ``` To disable all ignoring of tags: ``` octocatalog-diff --no-ignore-tags ... ``` ## Caveats When you tag a resource or defined type, Puppet will propagate that tag to *all* descendent resources. In this example, the tag `ignored_octocatalog_diff__foo__customfile` is propagated to the `foo::customfile` resource and to the file resource. However, octocatalog-diff will ignore only the `foo::customfile`, and will not ignore the file resource. ``` define foo::customfile ( String $source, ) { tag 'ignored_octocatalog_diff__foo__customfile' file { $name: ensure => file, source => $source, } } ``` :warning: If you were to do the following, not only would changes to `foo::customfile` parameters be ignored, but changes to the file resource would be ignored as well! That's because both `foo::customfile` and the file would have the tag `ignored_octocatalog_diff`, because the tag set in the defined type propagates to all descendent resources. ``` define foo::customfile ( String $source, ) { # DO NOT DO THIS!!! tag 'ignored_octocatalog_diff' file { $name: ensure => file, source => $source, } } ``` octocatalog-diff-1.5.4/doc/advanced-override-facts.md0000644000004100000410000000630413413744025022533 0ustar www-datawww-data# Overriding facts One powerful feature of `octocatalog-diff` allows you to override facts when compiling the catalogs, to predict the effect of a fact change on the catalog. This is useful to simulate a change in agent node configuration without actually setting up an agent to do so. ## Usage To override a fact in the "to" catalog: ``` --to-fact-override factname=value ``` To override a fact in the "from" catalog: ``` --from-fact-override factname=value ``` You may use as many of these arguments as you wish to adjust as many facts as you wish. ## Examples Simulate a change to its IP address in the "to" branch: ``` octocatalog-diff -n some-node.example.com -f master -t master \ --to-fact-override ipaddress=10.0.0.1 ``` Simulate a change in operating system version (in this case, from Ubuntu trusty to xenial): ``` octocatalog-diff -n some-node.example.com -f master -t master \ --from-fact-override lsbdistcodename=trusty --to-fact-override lsbdistcodename=xenial ``` Simulate changes to multiple facts, in this case the effect of moving a physical machine into EC2: ``` octocatalog-diff -n some-node.example.com -f master -t master \ --to-fact-override ec2=true --to-fact-override ec2_ami_id=ami-abcdef01 \ --to-fact-override ec2_hostname=ip-172-16-0-1.internal --to-fact-override ec2_instance_id=i-ba987654 \ --to-fact-override ec2_instance_type=c4.2xlarge ...... \ --to-fact-override virtual=xenhvm ``` Note that each of the examples specified the from branch and to branch to be `master`. There is no requirement that you do this, but you can generally obtain the most accurate test results by changing only one variable at a time. ## Advanced usage The `octocatalog-diff` parser will attempt to guess the data type based on the input. However, you can force the data type using the following syntax: ``` octocatalog-diff -n some-node.example.com -f master -t master \ --to-fact-override some_fact='(string)42' fact_to_delete='(nil)' ``` The following data types in parentheses are supported: | Data type in parentheses | Description | | ------------------------ | ------------| | `(string)` | Treat the input as a string | | `(fixnum)` | Treat the input as an integer (calls the `.to_i` method in ruby) | | `(float)` | Treat the input as an integer (calls the `.to_f` method in ruby) | | `(json)` | Treat the input as a JSON string (calls `JSON.parse` in ruby) | | `(boolean)` | Treat the input as a boolean -- it must be `true` or `false`, case-insensitive | | `(nil)` | Ignore any characters after `(nil)` and deletes the fact if the fact exists | ## Regular expressions If you wish to match multiple facts by pattern, specify the regular expression in place of the key name. For example: ``` octocatalog-diff -n some-node.example.com -f master -t master \ --to-fact-override /^ipaddress/=10.11.12.13 ``` In this example, `$::ipaddress`, `$::ipaddress_eth0`, `$::ipaddress_bond0`, and any other facts starting with "ipaddress" would be overridden. However, a fact named `$::additional_ipaddress` would not be overridden, because it does not match the regular expression. Please note that you cannot *add* a fact with a regular expression -- when using regular expressions you can only modify or delete facts. octocatalog-diff-1.5.4/doc/advanced-future-parser.md0000644000004100000410000000153113413744025022417 0ustar www-datawww-data# Enabling the future parser The [future parser](https://docs.puppet.com/puppet/3.8/reference/experiments_future.html) is a feature in Puppet 3.8 designed to provide functionally identical to the Puppet language in Puppet 4.0. You can use these options to enable the future parser for the "from" catalog, the "to" catalog, or both catalogs: - `--parser-from future` will enable the future parser for the "from" catalog - `--parser-to future` will enable the future parser for the "to" catalog - `--parser future` will enable the future parser for both the "from" catalog and the "to" catalog Note that you can also enable the future parser by creating a file named `environment.conf` in the base directory of your Puppet checkout. When `octocatalog-diff` computes the catalog from this directory, Puppet will read this file and act upon it accordingly. octocatalog-diff-1.5.4/doc/roadmap.md0000644000004100000410000000273713413744025017504 0ustar www-datawww-data# Roadmap This document outlines our philosophy and goals for the continued development of `octocatalog-diff`. ## Goals - Work on a system without a full Puppet installation - Cause no added load on production Puppet masters (unless you specifically choose to do so with non-default options) - Offer a command line tool to help make developers more efficient - Offer the ability to in a Continuous Integration (CI) environment - Be compatible with Puppet 3.8.7, 4.5, and later versions - Provide flexibility to build and compare catalogs even in esoteric Puppet codebases ## Areas for future development We are considering these areas for possible future development: - Improved display of diffs, perhaps a web interface - Additional CI use cases - CI output display to summarize a change and a list of affected hosts, rather than listing all changes host-by-host ## Antipatterns These are ideas we've evaluated and decided not to pursue. (If you are considering a [contribution](/.github/CONTRIBUTING.md) along these lines, please [open an issue](https://github.com/github/octocatalog-diff/issues/new) before start working on it. We would feel badly if you did a bunch of work that we could not accept.) - Making this into a Puppet module with a "face" so that it can be run with `puppet octocatalog-diff ...` or similar. (We have specifically designed this tool to run without a full Puppet installation. There are [similar projects](/doc/similar.md) that are distributed as Puppet modules.) octocatalog-diff-1.5.4/doc/advanced-catalog-only.md0000644000004100000410000000234613413744025022211 0ustar www-datawww-data# Building catalogs instead of diffing catalogs `octocatalog-diff` is designed primarily to build two catalogs and compare them. However, it can also simply generate the catalog without performing comparisons. ## Usage The `--catalog-only` command line flag triggers the following behavior: - The compiled catalog (not the difference) is written to the screen or stored in a file - Only the "to" branch is relevant (the "from" branch is not touched) - Options that control [output formats](/doc/advanced-output-formats.md), such as color and JSON format, do not apply - The `-o FILENAME` option will write the catalog to the indicated file rather than displaying it on screen ## Examples Building a catalog for a node from the current working directory and displaying on screen: ``` octocatalog-diff -n some-node.example.com --catalog-only ``` Building a catalog for a node from a specific branch and saving to a file: ``` octocatalog-diff -n some-node.example.com -t my-branch -o /tmp/some-node.json --catalog-only ``` As part of a CI job, testing whether a catalog for a particular host compiles: ``` octocatalog-diff -n some-node.example.com -o /dev/null --catalog-only if [ $? -eq 0 ]; then echo "Pass" else echo "Fail" fi ``` octocatalog-diff-1.5.4/.version0000644000004100000410000000000613413744025016443 0ustar www-datawww-data1.5.4