Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(wip) some patches to get things working with puppet-unbound #1

Open
wants to merge 2 commits into
base: add-hiera-overrides
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions lib/puppet-strings/hiera.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# frozen_string_literal: true

module PuppetStrings
module Hiera
require_relative 'hiera/hierarchy_data_path'
require_relative 'hiera/data'

def self.load_config
PuppetStrings::Hiera::Data.new('hiera.yaml')
end
end
end
132 changes: 132 additions & 0 deletions lib/puppet-strings/hiera/data.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# frozen_string_literal: true

module PuppetStrings::Hiera
class Data
attr_reader :config_path, :interpolated_paths, :uninterpolated_paths, :defaults

def initialize(config_path)
@config_path = config_path
@interpolated_paths = []
# This will probably always be ['common.yaml'] jut make it an array just incase
@uninterpolated_paths = []

load_config
end

def files
@files ||= begin
result = {}

interpolated_paths.each do |dp|
dp.matches.each do |file, interpolations|
unless result.key?(file)
result[file] = interpolations
end
end
end

result
end
end

# @return [Hash[String, Hash[String, Any]]]
# Full variable (class::var) -> filename: value
def overrides
@overrides ||= begin
overrides = {}
files.each_key do |file|
data = YAML.load(File.read(file))
data.each do |key, value|
overrides[key] ||= {}
overrides[key][file] = value
end
end
overrides
end
end

# @return [Hash[String, Hash[String, Any]]]
# Full variable (class::var) -> filename: value
def defaults
@defaults ||= begin
defaults = {}
uninterpolated_paths.each do |file|
data = YAML.load(File.read(file))
data.each do |key, value|
defaults[key] = value.nil? ? 'undef' : value.inspect
end
end
defaults
end
end

# @return [Hash[String, Hash[String, Any]]]
# variable -> filename: value
def overrides_for_class(class_name)
filter_mappings(class_name, overrides)
end

# @return [Hash[String, Hash[String, Any]]]
# variable -> filename: value
def defaults_for_class(class_name)
filter_mappings(class_name, defaults)
end

def to_s
config_path
end

private

# @return [Hash[String, Hash[String, Any]]]
# variable -> filename: value
def filter_mappings(class_name, mappings)
result = {}
mappings.each do |key, value|
mapped_class_name, _, variable = key.rpartition('::')
if mapped_class_name == class_name
result[variable] = value
end
end
result
end

# TODO: this should be a class method not an instance method
def load_config
return unless File.exist?(config_path)

config = YAML.load(File.read(config_path))

unless config['version'] == 5
log.warn("Unsupported version '#{config['version']}'")
return
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this achieves "Graceful fallback if legacy Hiera data is used" although see the todo above, currently we load the config for every instance

end

hierarchy = config['hierarchy']
return unless hierarchy

hierarchy.each do |level|
data_hash = level['data_hash'] || config['defaults']['data_hash']
next unless data_hash == 'yaml_data'

datadir = level['datadir'] || config['defaults']['datadir']

if level['path']
if level['path'] =~ /%{[^}]+}/
interpolated_paths << PuppetStrings::Hiera::HierarchyDataPath.new(datadir, level['path'])
else
uninterpolated_paths << File.join(datadir, level['path'])
end
elsif level['paths']
level['paths'].each do |path|
if path =~ /%{[^}]+}/
interpolated_paths << PuppetStrings::Hiera::HierarchyDataPath.new(datadir, path)
else
uninterpolated_paths << File.join(datadir, path)
end
end
end
end
end
end
end
60 changes: 60 additions & 0 deletions lib/puppet-strings/hiera/hierarchy_data_path.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# frozen_string_literal: true

module PuppetStrings::Hiera
class HierarchyDataPath
attr_reader :datadir, :path, :regex, :mapping

def initialize(datadir, path)
@datadir = datadir
@path = path
@regex, @mapping = HierarchyDataPath.path2regex(path)
end

def matches
result = {}

Dir.glob(File.join(datadir, '**', '*.yaml')).each do |entry|
next unless File.file?(entry)

regex.match(entry) do |match|
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had the chdir there so the globbed paths were relative to datadir rather than absolute. This reduced the risk of any oddities with the regex here caused by the unused part.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm im not sure thats the best idea. 1) im not convinced there is a problem to solve (do you have some examples, even contrived ones) 2) if there is a problem at best we are just reducing the risk of triggering it by using chdir

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The alternative is to normalize it again. For example (using pathname):

data_dir = Pathname.new(datadir)
data_dir.glob('**').each do |entry|
  next unless x.file?

  regex.match(entry.relative_path_from(data_dir)) do |match|
    # ...
  end
end

Note that I also chose to glob everything so that we don't have to think about extensions (yaml vs yml). The regex matching will take care of that.

interpolations = {}

mapping.each do |name, interpolation|
interpolations[interpolation] = match.named_captures[name]
end

result[entry] = interpolations
end
end

result
end

def self.path2regex(path)
mapping = {}

intermediate_result = path

# First pass - intermediate replacements
path.scan(/%{[^}]+}/).each_with_index do |interpolation, i|
replacement = "X_INTERPOLATION_#{i}_X"
mapping[replacement] = interpolation[2..-2]
intermediate_result = intermediate_result.sub(interpolation, replacement)
end

# Second pass - escape any special chars
escaped = Regexp.escape(intermediate_result)

# Third pass - replacement intermediates with regex
mapping.each_key do |replacement|
escaped = escaped.sub(replacement, "(?<#{replacement}>.+)")
end

[Regexp.new(escaped), mapping]
end

def to_s
File.join(datadir, path)
end
end
end
12 changes: 12 additions & 0 deletions lib/puppet-strings/markdown/base.rb
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,18 @@ def defaults
@registry[:defaults] unless @registry[:defaults].nil?
end

# Overrides from Hiera
#
# Hiera overrides only apply to classes. Each entry is a tuple of the
# filename it's defined in, a mapping of interpolations that were applied
# in the filename and the value inside the file.
#
# @return [Array[Tuple[String, Hash[String, String], Any]]]
# Any overrides from Hiera.
def hiera_overrides
[]
end

# @return [Hash] information needed for the table of contents
def toc_info
{
Expand Down
23 changes: 23 additions & 0 deletions lib/puppet-strings/markdown/puppet_class.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,29 @@ class PuppetClass < Base
def initialize(registry)
@template = 'classes_and_defines.erb'
super(registry, 'class')
@hiera = PuppetStrings::Hiera.load_config
update_defauls
end

def hiera_overrides
@hiera_overrides ||= begin
overrides = @hiera.overrides_for_class(name)

result = {}

overrides.each do |variable, files|
result[variable] = files.map do |filename, value|
interpolations = @hiera.files[filename]
[filename, interpolations, value]
end
end

result
end
end

def update_defauls
@registry[:defaults].merge!(@hiera.defaults_for_class(name))
end

def render
Expand Down
13 changes: 13 additions & 0 deletions lib/puppet-strings/markdown/templates/classes_and_defines.erb
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,19 @@ Options:
<% if defaults && defaults[param[:name]] -%>
Default value: `<%= value_string(defaults[param[:name]]) %>`

<% end -%>
<% if !hiera_overrides.empty? && hiera_overrides[param[:name]] -%>
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't have the empty check because in Ruby a missing key in a hash returns nil so it wouldn't be triggered. I think this change is redundant.

Copy link
Author

@b4ldr b4ldr Apr 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hiera_overrides by default returns [] and defined as @return [Array[Tuple[String, Hash[String, String], Any]]] not sure if this is by design or not but i got a stack trace when it was empty: unable to cast string to int (or something like that)

<details>
<summary>Hiera overrides in a detailed table</summary>

| Filename | Interpolations | Value |
|----------|----------------|-------|
<% hiera_overrides[param[:name]].each do |filename, interpolations, value| -%>
| `<%= filename %>` | <%= interpolations.map { |i, v| "`#{i}`: `#{v}`" }.join("<br>") %> | `<%= value %>` |
<% end -%>

</details>

<% end -%>
<% end -%>
<% end -%>
1 change: 1 addition & 0 deletions lib/puppet-strings/yard.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ module PuppetStrings::Yard
require 'puppet-strings/yard/handlers'
require 'puppet-strings/yard/tags'
require 'puppet-strings/yard/parsers'
require 'puppet-strings/hiera'
require 'puppet-strings/monkey_patches/display_object_command'

# Sets up YARD for use with puppet-strings.
Expand Down
1 change: 1 addition & 0 deletions lib/puppet-strings/yard/code_objects/class.rb
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ def to_hash
hash[:docstring] = PuppetStrings::Yard::Util.docstring_to_hash(docstring)
defaults = Hash[*parameters.reject{ |p| p[1].nil? }.flatten]
hash[:defaults] = defaults unless defaults.nil? || defaults.empty?
#hash[:hiera_overrides] = hiera_overrides if hiera_overrides.any?
hash[:source] = source unless source.nil? || source.empty?
hash
end
Expand Down