yaml.representer.RepresenterError with nested inventory classes and indirect array item references

[jperville]

The issue

I tried reclass an inventory with one yaml value referencing another value nested inside an Array and nested classes are involved.

The issue is very difficult for me to describe since I am not familiar with the "reclass" ontology, so I describe it with full reproduction instructions (see below).

I am using the develop branch from https://github.com/salt-formulas/reclass , that is version 1.7.0 of reclass.
My system is Ubuntu 18.04 on amd64 with python 3.6 (Python 3.6.9 according to python -v).

Here is the stacktrace that I get when I run python reclass.py -b mytest --inventory in my virtualenv :
Running:

Traceback (most recent call last):
  File "reclass.py", line 15, in <module>
    reclass.cli.main()
  File "/home/julien/reclass/reclass/cli.py", line 48, in main
    print(output(data, options.output, options.pretty_print, options.no_refs))
  File "/home/julien/reclass/reclass/__init__.py", line 28, in output
    return outputter.dump(data, pretty_print=pretty_print, no_refs=no_refs)
...
  File "/home/julien/reclass/myenv/lib/python3.6/site-packages/PyYAML-5.3.1-py3.6-linux-x86_64.egg/yaml/representer.py", line 58, in represent_data
    node = self.yaml_representers[None](self, data)
  File "/home/julien/reclass/myenv/lib/python3.6/site-packages/PyYAML-5.3.1-py3.6-linux-x86_64.egg/yaml/representer.py", line 231, in represent_undefined
    raise RepresenterError("cannot represent an object", data)
yaml.representer.RepresenterError: ('cannot represent an object', Value(RefItem([ScaItem('kubemanual:hosts:0:interfaces:public:ip')])))

I can work around by duplicating the indirect reference to use the direct reference everywhere in my inventory,
but I'd rather have a shorter and more flexible inventory.

How to reproduce

In my reproduction scenario, the directory structure is as follows:

julien:mytest(79078m|develop?) $ find mytest -type f
mytest/classes/kubemanual.yml
mytest/classes/kubespray.yml
mytest/nodes/mynode.yml

The files are created as follows:

# install some dependencies needed to install the bundle
sudo apt-get install libyaml-dev

# checkout the reclass project and cd into it
git clone https://github.com/salt-formulas/reclass myreclass
cd myreclass

# install everything into a venv, activate it
virtualenv -p python3.6 myenv
source myenv/bin/activate
python setup.py install

# prepare the bug fixtures
mkdir -p mytest/classes mytest/nodes

# create mytest/nodes/mynode.yml
echo -n '---
classes:
  - kubemanual
parameters:
  kubemanual:
    hosts:
      - name: one
        interfaces:
          public:
            name: eth0
            ip: 8.8.8.8
          private:
            name: eth1
            ip: 192.168.0.1
        access-ip: ${kubemanual:hosts:0:interfaces:public:ip}
' | tee mytest/nodes/mynode.yml

# create mytest/classes/kubemanual.yml
echo -n '---
classes:
  - kubespray
parameters:
  kubemanual:
    hosts: []

  kubespray:
    access-ip-using-indirect-item: ${kubemanual:hosts:0:interfaces:public:ip}
    access-ip-using-direct-item: ${kubemanual:hosts:0:access-ip}
' | tee mytest/classes/kubemanual.yml

# create mytest/classes/kubespray.yml
echo -n '---
parameters:
  kubespray: {}
' | tee mytest/classes/kubespray.yml

This finishes setting up the bug fixture directory.

Some tests and comments

Running with both direct and indirect references and the kubespray class imported

Run:

python reclass.py -b mytest --inventory

Should trigger the exception reported above.

Running with only the indirect reference and the kubespray class imported

Run:

sed -i -e 's/^.*- kubespray/  - kubespray/' mytest/classes/kubemanual.yml
sed -i -e 's/^.*access-ip-using-direct-item:/    #access-ip-using-direct-item:/' mytest/classes/kubemanual.yml
sed -i -e 's/^.*access-ip-using-indirect-item:/    access-ip-using-indirect-item:/' mytest/classes/kubemanual.yml

egrep '(access-ip|kubespray)' mytest/classes/kubemanual.yml

python reclass.py -b mytest --inventory

Should result in a functional inventory

Running with only the direct reference and the kubespray class imported

Run:

sed -i -e 's/^.*- kubespray/  - kubespray/' mytest/classes/kubemanual.yml
sed -i -e 's/^.*access-ip-using-direct-item:/    access-ip-using-direct-item:/' mytest/classes/kubemanual.yml
sed -i -e 's/^.*access-ip-using-indirect-item:/    #access-ip-using-indirect-item:/' mytest/classes/kubemanual.yml

egrep '(access-ip|kubespray)' mytest/classes/kubemanual.yml

python reclass.py -b mytest --inventory

Should also trigger the exception reported above.
So, does it mean that the indirect reference alone triggers the bug?

Running with both direct and indirect references and the kubespray class commented

Run:

sed -i -e 's/^.*- kubespray/  #- kubespray/' mytest/classes/kubemanual.yml
sed -i -e 's/^.*access-ip-using-direct-item:/    access-ip-using-direct-item:/' mytest/classes/kubemanual.yml
sed -i -e 's/^.*access-ip-using-indirect-item:/    access-ip-using-indirect-item:/' mytest/classes/kubemanual.yml

egrep '(access-ip|kubespray)' mytest/classes/kubemanual.yml

python reclass.py -b mytest --inventory

Should result in a functional inventory.
Looks like the problem is in the kubespray class, not in the indirect reference !

Running the inventory with an empty kubespray class

Run:

sed -i -e 's/^.*- kubespray/  #- kubespray/' mytest/classes/kubemanual.yml
sed -i -e 's/^.*access-ip-using-direct-item:/    access-ip-using-direct-item:/' mytest/classes/kubemanual.yml
sed -i -e 's/^.*access-ip-using-indirect-item:/    access-ip-using-indirect-item:/' mytest/classes/kubemanual.yml

# this is the important part
sed -i -e s/kubespray/notkubespray/ mytest/classes/kubespray.yml

egrep '(access-ip|kubespray)' mytest/classes/kubemanual.yml

python reclass.py -b mytest --inventory

This last example also results in a functional inventory.
The bug does not trigger !

Conclusion

The bug occurs when:

• the kubespray class is imported from within the kubemanual class
• the node file declares parameters.kubespray.access-ip-using-indirect-item which indirectly references a value which is located inside an array item
• the kubespray class has already defined parameters.kubespray (even if it is an empty dictionary)

The bug does not occur when either:

• using the direct reference instead of the indirect one
• the parameters.kubespray is undefined at the time the indirect reference within parameters.kubespray is evaluated

[AndrewPickford]

The bug comes from a combination of how reclass handles references to references and paths to list elements. If a parameter is reference a check is made to see if the referenced parameter itself is also a reference. If this nested reference value has not yet been evaluated then it is evaluated first and then the original reference is evaluated.

This check to see if a referenced path itself needs dereferencing fails for list elements because of an inconsistency in how reclass represents the path to elements in a list. Essentially the type of the index number of an element in a list is set inconsistency, in one part of the code it's an integer in another it's a string. So when the check to see is a reference itself needs dereferencing happens there's a check to see if the path to the referenced parameter is in a set of parameters that have not yet been dereferenced. For references to lists this check involves an equally test of the list element number, this always fails due to the type mismatch, i.e. there are tests such as '0' == 0, which of course always fail.

I'll give this a little thought over the next week or so, but sorting out the type mismatch will fix the issue.